Why so many mediocre-review games suddenly have very high reviews?

[Turbo Nozomix]

Many games on Steam which were somewhat commercial flops and have had mixed reviews since they released suddenly have a huge influx of 90%+ or thereabouts reviews in the previous 30 days. In some cases, the influx of 90%+ positive reviews has suddenly lifted the game out of its Mixed review status.

I've been noticing this all over the place on Steam.


Here are some examples:

https://store.steampowered.com/app/637090/BATTLETECH/

https://store.steampowered.com/app/690790/DiRT_Rally_20/

https://store.steampowered.com/app/421020/DiRT_4/

https://store.steampowered.com/app/272270/Torment_Tides_of_Numenera/

https://store.steampowered.com/app/1017900/Age_of_Empires_Definitive_Edition/

https://store.steampowered.com/app/640820/Pathfinder_Kingmaker__Enhanced_Edition/



What's going on? These games haven't all received new patches and nothing has changed about their basic experience.

They could all be legitimate, but at face value, and right before the Steam Autumn sale, it's suspicious, IMO. It sort of looks like titles are being massaged ahead of the sale.



Edit:

The cause is expected to be from Steam prompting people who have been actively playing a game to leave a review.

A problem I see with that approach is that, just as people who keep going to a restaurant obviously like the experience of that restaurant, people who actively play a game are also likely to be playing it because they like it.

Discussion of the issues of that type of system start around here:

https://steamcommunity.com/discussions/forum/0/1744512174759305803/?ctp=7#c1744512496198191837

[Erebus]

Początkowo opublikowane przez Turbo Nozomix:

I didn't say that not soliciting current players for reviews leads results in having the full picture. I said that soliciting only recurring players for reviews results in a less-accurate picture than the previous situation.

How pray tell is most reviews being old and out of date from people not playing a game "better"? The bulk of reviews previously came in solely at the launch window. Beyond that they'd trickle in when someone felt particularly strongly about something.

Games are not static products anymore. They change over time, recent players is an important font of knowledge. As is getting them to update their reviews as time moves on. Just the other day one of the people on my friends list (likely prompted by the library update) changed the review for a game he has 1400+ hours in to a very large very descriptive negative review based on how the game development and updates have flowed.

It was a way more useful read than the typical 1-2 sentences you see from players of stuff with little playtime in things.

And obviously I don't like the outcome if the review consensus and score ends up being a skewed picture.

You're overly obsessed with the %.

[Turbo Nozomix]

Początkowo opublikowane przez Erebus:

Początkowo opublikowane przez Turbo Nozomix:

The issue potentially posed by the the system's review trend is not down to what people write in their review, but is down to the selectivity of who is solicited to write those review. That shouldn't have to be restated at this point as it's been emphasized many times already.

So what's your proposal? That people not playing the games, with no hours in them, and no idea of the current state also be prompted for review?

My proposal at this time would be to not solicit reviews from one demographic of players as that biases the pool of reviews that is received and consequently the score a game has on Steam.

Soliciting reviews only from a demographic that continues to play a game can be expected to receive favourable reviews for the game and that creates an imbalance in the reviews that are coming in.

I think that the Steam review system as it was was just fine and gave more a more usable reading than the average 85 - 99% positive reviews for tons of games that have been coming in since the change in the system.

Forget the %, when I'm buying I'm looking for the opinions of those that have actually played the game and played it lately. Not the guy that played it 3 hours and decided "oh ♥♥♥♥ I can't refund this". Worth noting too just cause someone is forced into a thumbs up or thumbs down does not mean the written reviews don't carry criticisms while being positive and compliments while being negative. You're forced to pick one or the other whether you just slightly feel it's worth it or just slightly feel it's over the line of not worth it.

And now you have all the same elements and players are still forced into the same dichotomy only now, due to solicitation of reviews from current players, there's over-representation from players who continue to play the game because, in most cases, they like it. So, the quality of the review presentation and summarization has declined.

Trying to shift a binary option into being this nuanced "guide" is totally ridiculous.

Over a large enough pool, without additional influences, it does give somewhat of a balanced report. And regardless of how nuanced it ends up being, the previous method of not soliciting that binary feedback specifically from recurring players gave a more measured result than the current implementation appears to be doing.

[Erebus]

Początkowo opublikowane przez Turbo Nozomix:

My proposal at this time would be to not solicit reviews from one demographic of players as that biases the pool of reviews that is received and consequently the score a game has on Steam.

Customer reviews are already biased.

As explained prior people with issues are more likely to speak up than people without issues. And the demographic of people willing to say anything at all is a specific one, not a general one representative of the greater population.

Optional reviews are not a proper statistical study, they never are ever. It's about as "valid" as using forum posts to decide a consensus. The % is no more meaningful now than it was before. And in fact the amount of information in the reviews themselves might be of higher quality and content then before by actually prompting some people with hands on experience.

You're so focused on that stupid % you're not looking at anything else.

[Turbo Nozomix]

Początkowo opublikowane przez Erebus:

Początkowo opublikowane przez Turbo Nozomix:

I didn't say that not soliciting current players for reviews leads results in having the full picture. I said that soliciting only recurring players for reviews results in a less-accurate picture than the previous situation.

How pray tell is most reviews being old and out of date from people not playing a game "better"?

In what scenario does not having review prompts for current players lead to having out of date reviews? If nobody is leaving a review for a game, then the game isn't popular and isn't receiving updates. So, reviews can't go out of date unless the game changes to make pre-existing reviews non longer relevant. If a game undergoes a change that impacts the reviews that are given, it's only because people are actually still playing it and so still leaving reviews. And when a patch comes out, old players return to check out what's different and more reviews get less.

Reviews don't go out of date just by sitting there.

The bulk of reviews previously came in solely at the launch window. Beyond that they'd trickle in when someone felt particularly strongly about something.

It depends on which game you're looking at. But many games continue to receive a lot of reviews years after they release. Games that don't probably aren't being played by that many people.

Games are not static products anymore. They change over time, recent players is an important font of knowledge. As is getting them to update their reviews as time moves on. Just the other day one of the people on my friends list (likely prompted by the library update) changed the review for a game he has 1400+ hours in to a very large very descriptive negative review based on how the game development and updates have flowed.

The recent player-base is gaugable, and better so, by methods other than reviews. There are Steam stats for current and historical players. There are forums to give reports on how many people are still playing a game. Steam reviews don't cover that ground very well.

So, the information you are talking about already existed without review prompts. And as I said above, when a new patch comes out that changes something there's always been a wave of reviews covering it.

And obviously I don't like the outcome if the review consensus and score ends up being a skewed picture.

You're overly obsessed with the %. [/quote]

It's not the full picture but it's an important part of it. Why would I not talk about it? This goes back to the point of: If this subject doesn't interest you, why post in a thread about it? If it interests other people enough for them to discuss it, who are you to criticize people for talking about their interests? Nobody has forced you to reply in this thread.

[Erebus]

Początkowo opublikowane przez Turbo Nozomix:

The recent player-base is gaugable, and better so, by methods other than reviews. There are Steam stats for current and historical players. There are forums to give reports on how many people are still playing a game. Steam reviews don't cover that ground very well.

Better so? Neither of those tell you what the players think.

Hell some people stick with things solely because of friends or because of "sunken cost fallacy". Those opinions are actually quite valuable to read. Especially when people are wanting to buy something for the "long haul" and not a few hours and forget.

[Turbo Nozomix]

Początkowo opublikowane przez Erebus:

Początkowo opublikowane przez Turbo Nozomix:

My proposal at this time would be to not solicit reviews from one demographic of players as that biases the pool of reviews that is received and consequently the score a game has on Steam.

Customer reviews are already biased.

This is rationalizing and saying that because they're already biased that it doesn't matter if they become more biased. But it matters. And if it doesn't matter to you, why are you saying so much about it?

As explained prior people with issues are more likely to speak up than people without issues.

I previously responded to that assertion in regards to Steam by saying this:

Początkowo opublikowane przez Turbo Nozomix:

It's well known that people who enjoy thongs are less likely to actually speak up. For this reason alone I don't look at ratings, but at what's written.

It's not proven that's the case in an environment like Steam where everybody knows the review option is on the store page for every game and is easily accessible to use and reliably from the same place. And even the article posted earlier by Spawn of Totoro it states that the trend in reviews is moving towards leaving positive reviews.

It's actually predictable that there would be less of a difference between those willing to leave positive and those willing to leave negative reviews on Steam because the system to do so is streamlined and reliable and always just a couple of clicks away. The tendency to leave positive or negative reviews through Steam will not be analogous to a person going out of their way to leave a review for some physical store or restaurant, or where they have to go off-site to leave a review.

It is likely that there has been no greater inclination to leave a negative review on Steam. And it is obvious that there is now a large bias towards eliciting positive reviews on Steam.

Optional reviews are not a proper statistical study, they never are ever. It's about as "valid" as using forum posts to decide a consensus. The % is no more meaningful now than it was before. And in fact the amount of information in the reviews themselves might be of higher quality and content then before by actually prompting some people with hands on experience.

You're so focused on that stupid % you're not looking at anything else.

Actually, I'm looking at it all. I don't see a positive trade-off to losing a more accurate feedback pool. More positive reviews? That's not a positive trade-off because the purpose of reviews is to have accurate and comprehensive information to use to base a purchasing decision on.

A mountain of glowing reviews masking the problems that might affect your own experience is not helpful. And a mountain of positive reviews will possibly be very redundant. And then there's the detail that the overall score isn't accurate or usable is all overall scores become homogenized into 85%+ ratings despite games being of various relative qualities.

[Turbo Nozomix]

Początkowo opublikowane przez Erebus:

Początkowo opublikowane przez Turbo Nozomix:

The recent player-base is gaugable, and better so, by methods other than reviews. There are Steam stats for current and historical players. There are forums to give reports on how many people are still playing a game. Steam reviews don't cover that ground very well.

Better so? Neither of those tell you what the players think.

Hell some people stick with things solely because of friends or because of "sunken cost fallacy". Those opinions are actually quite valuable to read. Especially when people are wanting to buy something for the "long haul" and not a few hours and forget.

Neither of what tells readers what the players think? Forums and player stats? Well, forums can, but they don't have to because that's why reviews are there.

And reviews don't tell you how many current players there are. But there are Steam stats and Steam Charts, which do and people in forums can give more insight into that matter than looking at the number of recent Steam reviews can.

And Steam stats don't let people ask other players for help with issues or discuss aspects of the game. But the forums do.

All the bases are covered. Using reviews to gauge active player-base is not going to work.

In the previous situation, there is a more balanced response. In the current situation, there is over-representation of certain types of feedback resulting in players missing out on criticisms and not getting an accurate look at the larger picture. The current system encourages sycophantic representation in the reviews, and that's not a good thing.

People who stick with a game were by no means unable to leave a review before review prompts appeared. The previous system took nothing away from people. But I think that the current system takes away accuracy of the review pool and the overall review rating and relevance of feedback from readers.

[Erebus]

Początkowo opublikowane przez Turbo Nozomix:

Początkowo opublikowane przez Erebus:

Better so? Neither of those tell you what the players think.

Hell some people stick with things solely because of friends or because of "sunken cost fallacy". Those opinions are actually quite valuable to read. Especially when people are wanting to buy something for the "long haul" and not a few hours and forget.

Neither of what tells readers what the players think? Forums and player stats? Well, forums can, but they don't have to because that's why reviews are there.

In the previous situation, there is a more balanced response. In the current situation, there is over-representation of certain types of feedback resulting in players missing out on criticisms and not getting an accurate look at the larger picture. The current system encourages sycophantic representation in the reviews, and that's not a good thing.

People who stick with a game were by no means unable to leave a review before review prompts appeared. The previous system took nothing away from people. But I think that the current system takes away accuracy of the review pool and the overall review rating and relevance of feedback from readers.

People are unlikely to give feedback in general. Forums are not statistically valid ever. # of players at a given time is more about popularity than enjoyment.

https://www.google.com/search?q=people+are+more+likely+to+give+negative+feedback&oq=people+are+more+likely+to+give+negative+feedback&ie=UTF-8

Pick a link. Look into psychology a bit. Look at anything really other than the stupid % which is only off because you claim it's "off". Maybe previously negative reviews were over-represented. It's entirely possible 50+% of negative opinions were posted but maybe only 20% of positive opinions.

You're claiming things are off even more than they were before solely because you don't like them... and your grand argument was a comparison to metacritic (which is a steaming pile of garbage as shown previously).

[Turbo Nozomix]

Początkowo opublikowane przez Erebus:

https://www.google.com/search?q=people+are+more+likely+to+give+negative+feedback&oq=people+are+more+likely+to+give+negative+feedback&ie=UTF-8

Pick a link. Look into psychology a bit. Look at anything really other than the stupid % which is only off because you claim it's "off". Maybe previously negative reviews were over-represented. It's entirely possible 50+% of negative opinions were posted but maybe only 20% of positive opinions.

The likelihood of people leaving negative reviews over positive has been discussed in this thread. Spawn of Totoro posted a link that said that while people are more likely to leave a negative review that the trend is moving closer towards leaving positive reviews.

And I gave you my response to that argument:

"It's not proven that's the case in an environment like Steam where everybody knows the review option is on the store page for every game and is easily accessible to use and reliably from the same place. And even the article posted earlier by Spawn of Totoro it states that the trend in reviews is moving towards leaving positive reviews.

It's actually predictable that there would be less of a difference between those willing to leave positive and those willing to leave negative reviews on Steam because the system to do so is streamlined and reliable and always just a couple of clicks away. The tendency to leave positive or negative reviews through Steam will not be analogous to a person going out of their way to leave a review for some physical store or restaurant, or where they have to go off-site to leave a review.

It is likely that there has been no greater inclination to leave a negative review on Steam. And it is obvious that there is now a large bias towards eliciting positive reviews on Steam."

You're claiming things are off even more than they were before solely because you don't like them... and your grand argument was a comparison to metacritic (which is a steaming pile of garbage as shown previously).

No, I have not associated my claims with what I like and don't like. There are games in the OP which I really like. I am not giving them a free pass and my criticism is directed towards biased review results and not specific games being reviewed in a way different than I would review them.

Metacritic was not anything resembling a grand argument here. But if you're appealing to that idea and arguing that Metacritic is full of crap reviews (and it is), then you're not helping out an argument in favour of Steam's new review-prompt system because the review-prompt system is biasing ratings just as badly as any on Metacritic and even worse (if comparing to the professional critic reviews) by homogenizing game scores into a high bracket, while making lots of games you regard as bad and which previously had Steam ratings reflecting that to have scores equal to or higher than games which you like.

[Tito Shivan]

Początkowo opublikowane przez Turbo Nozomix:

Soliciting reviews only from a demographic that continues to play a game can be expected to receive favourable reviews for the game and that creates an imbalance in the reviews that are coming in.

Tell that to the people who played TF2 when it went F2P. Or the Payday players with hundreds of hours when the game added game-altering microtransactions... Or any patch that changes a game meta or mechanic.

Assuming that long-time players mostly leave positive reviews is a mistake. The hardcore subset of any game playerbase is often the most critical of the game.

[Turbo Nozomix]

Początkowo opublikowane przez Tito Shivan:

Początkowo opublikowane przez Turbo Nozomix:

Soliciting reviews only from a demographic that continues to play a game can be expected to receive favourable reviews for the game and that creates an imbalance in the reviews that are coming in.

Tell that to the people who played TF2 when it went F2P. Or the Payday players with hundreds of hours when the game added game-altering microtransactions... Or any patch that changes a game meta or mechanic.

Assuming that long-time players mostly leave positive reviews is a mistake. The hardcore subset of any game playerbase is often the most critical of the game.

I would say that my comment that players who keep coming back to a game are more likely to leave positive reviews isn't a mistake but is logical and evidence-based going by the results of the new Steam review-prompt, and that disruptive changes to a game are a separate factor that doesn't challenge what I said.

Those upset when TF2 when F2P were, as identified in your comment, were upset for a specific reason which is because the game they paid for went F2P. Without that change they would have remained more favourable towards the game. The same is true regarding Payday's adding of MTX.

And I have mentioned in my comments that changes to a game elicit new waves of reviews regardless of whether people are prompted to leave a review or not.

[Erebus]

Początkowo opublikowane przez Turbo Nozomix:

And I have mentioned in my comments that changes to a game elicit new waves of reviews regardless of whether people are prompted to leave a review or not.

Not always. Especially before the library update it wasn't very visible when something got a major patch unless you were already actively engaged. Go look at the titles you posted and cross reference with dates patches went live. I bet there wasn't much of a change in the rate of review coming in or anything, except for cases when the patch literally broke stuff... which would then be negative reviews.

[Start_Running]

Początkowo opublikowane przez Tito Shivan:

Początkowo opublikowane przez Turbo Nozomix:

Soliciting reviews only from a demographic that continues to play a game can be expected to receive favourable reviews for the game and that creates an imbalance in the reviews that are coming in.

Tell that to the people who played TF2 when it went F2P. Or the Payday players with hundreds of hours when the game added game-altering microtransactions... Or any patch that changes a game meta or mechanic.

Assuming that long-time players mostly leave positive reviews is a mistake. The hardcore subset of any game playerbase is often the most critical of the game.

The DOta2 Oyutlanders update has rankled a few feathers too.

Początkowo opublikowane przez Turbo Nozomix:

Początkowo opublikowane przez Tito Shivan:

Tell that to the people who played TF2 when it went F2P. Or the Payday players with hundreds of hours when the game added game-altering microtransactions... Or any patch that changes a game meta or mechanic.

Assuming that long-time players mostly leave positive reviews is a mistake. The hardcore subset of any game playerbase is often the most critical of the game.

I would say that my comment that players who keep coming back to a game are more likely to leave positive reviews isn't a mistake but is logical and evidence-based going by the results of the new Steam review-prompt, and that disruptive changes to a game are a separate factor that doesn't challenge what I said.

And? WHat is wrong with that? You're collecting the views of customers. And keep in mind these reminders pop no matter how much time you have in the game.

[Turbo Nozomix]

Początkowo opublikowane przez Start_Running:

Początkowo opublikowane przez Turbo Nozomix:

I would say that my comment that players who keep coming back to a game are more likely to leave positive reviews isn't a mistake but is logical and evidence-based going by the results of the new Steam review-prompt, and that disruptive changes to a game are a separate factor that doesn't challenge what I said.

And? WHat is wrong with that? You're collecting the views of customers. And keep in mind these reminders pop no matter how much time you have in the game.

We have many pages in this thread detailing what I think is wrong with that and why I think that. It skews the readily-available feedback and the overall rating for games, rendering the reviews not useful for accurate information. When all games become 85%+ and higher due to the solicitation of reviews from recurring players, then the % of positive reviews becomes a meaningless statistic.

[Kelthorian]

Początkowo opublikowane przez Tito Shivan:

Tell that to the people who played TF2 when it went F2P. Or the Payday players with hundreds of hours when the game added game-altering microtransactions... Or any patch that changes a game meta or mechanic.

Which ironically enough would nowadays result in the automated system deeming it a review bombing and filter it out.

So taking the whole system into account those examples are useless now with the current biased structure.