Restartt Aug 7, 2014 @ 8:02am
How ratings work - Can a Mod or someone edit the sticky "Understanding the Steam Workshop"?
I like it, but it falls flat on the rating system, suggesting false information.

On first read it seems "Oh, the rating just just based on the likes vs dislikes".

This is not true. Or rather, it's not true until there are a LOT of votes. Specifically, a workshop item can have a 96% approval rating (lives vs dislikes) or higher and still be rated at 3/5 stars, if a certain threshold of votes isn't passed yet.

This is because the rating is a Bayesian average, not a "normal" average as most people assume and the tone of the post suggests, so the actual rating system is factored in with some other criteria (Valve developer TomB confirmed this, and said the total number of posts was a factor, but wouldn't confirm specifics).

Assuming a similarity to most other popular Bayesian systems, the actual rating is given a weighting based on number of users that have voted, then averaged with either a 2.5 rating or the average workshop gaming. So basically, until the mod gets a lot of votes, the rating system tells you nothing. A workshop item with 40 upvotes and no downvotes will be rated the same 3/5 stars as a mod with 40 downvotes and no upvotes until the number of votes passes some threshold to be statistically significant.

this means that a workshop item with a "few" votes (in steam terms, they've never defined what they think a "few" is) is but very positive ones (say, 90% upvote) would be rated like such:

--------------------------
90% item upvote rating * (0.1 wieghting for few votes)
+
50% average weighting * (1.0 weighting for average)
--------------------------
/1.1 for total weighting

would give the game a 54% rating(3/5 stars


A very negative item (90% downvote) that doesn't have a lot of votes would also be rated similarly

--------------------------
10% item upvote rating * (0.1 weighting for few votes)
+
50% average weighting * (1.0 weighting for average)
--------------------------
/1.1 for total weighting

would give the game a 46% rating (3/5 stars)


It's only once steam decides the workshop item has enough votes to outweigh the average (the 0.1 weighting above starts to become much bigger than the 1.0 weighting the average is given) that the games rating becomes meaningful (and even then for a while it can be pulled up or down a little until it has a LOT of votes)


So yeah, the rating system only works if there are a lot of ratings. If there aren't a lot of ratings, the X/5 stars you see has no meaning whatsoever. Once there are a modest number of ratings, the workshop items displayed rating starts to get closer to, but is still decently far from, the actual upvote vs downvote rating. and until the item gets an ASTRONOMICAL amount of ratings, the rating will still be pulled slightly towards average.


NOTE i think this is an awesome system. There have been studies about how this is much more descriptive of how good a game is than a simple average or a 1-10 star rating system. I just think that if an explanation of how the rating system works is said to be given, it should accurately describe that system!

I was very confused by this at first, because like most people i thought the X/5 stars system was a simple average. The sticky seemed to confirm this, and hey, it's Moderator approved, it has to be right. But it's actually a lot more in depth than that.
Last edited by Restartt; Aug 7, 2014 @ 9:11am
Date Posted: Aug 7, 2014 @ 8:02am
Posts: 0