Ratings
It's not a big problem in my day-to-day life, I must admit. It doesn't keep me awake at night. It's just that sometimes, when it comes up, it gives me pause: I don't always know how to rate things.
Sometimes it's unambiguous. When rating a seller, I rate 5 stars when the product arrives without a hitch, or when refunds are handled promptly. Do you ever give a seller 3 stars? When rating a hotel, the problems start. I am supposed to rate it out of 10, but does that include the price? Should I give that overpriced hotel an 8 out of 10, because it has most of the amenities and a nice location, or should I rate it a 6 because you should expect just a little more for that price?
When rating a movie, should I listen to my inner Siskel and Ebert and rate whether the movie was good, or should I just give it a thumbs up because I had an okay time? There's a difference. If I can give the movie up to four stars, I would be inclined to do the former.
Netflix used to have percentage ratings, and that communicated something to me. Now they have thumbs up, and that communicates something else. If I give a thumbs-up, Netflix will show me more movies of that type. It's only really for me. I can't thumbs-down a bad movie in order to communicate that it's bad like I can give one star only to a product on Amazon. They went from one kind of rating system to another, or from a rating system to a recommendation system, but they kept it in the same place in the UI.
Meaning
Giving something a thumbs-up or five stars or 95% can mean any one of the following: Show me more like this; show this to more people; add this to my list; adjust the score upwards; this is a good product; the product is irrelevant as long as the transaction went smoothly and it did; this hotel is clean and in a nice location.
I can usually distinguish between "This is good" and "Show me more like this". I appreciate that Steam has both, with reviews and discovery as separate systems. I also understand that you aren't supposed to grade Uber drivers on a curve.
It's all good as long as users understand what a good rating means, or what a bad rating means. If users don't know whether ratings are public, or which ones will be used in a recommendation system, then problems arise. They might give high ratings to movies they don't want to watch again, and get confused by the recommendations.
Problems will also arise when users disagree about the meaning of votes and ratings. If a downvote on Reddit is supposed to express disagreement, that's okay as long as users disagree, but if it's meant to be only a punishment for dangerous, false, or bad-faith comments, then users might argue whether they "really deserved" that downvote.
The arguments about meaning don't matter when raters agree about the meaning of the rating, or when they agree that they don't need to agree. For this to work they both need to agree that the rating is subjective, and they need to be correct in that assessment. If one rater thinks the meaning of a thumbs-down is "The Godfather is objectively a bad film", and the other thinks "Do not show me this movie again", then there's still a problem, and potential for conflict. If all raters think the meaning is "Do not show me this movie again", then you better not make any pronouncements about the quality of The Godfather based on aggregate data.
Distributions
Even when users/raters agree on the meaning of ratings, you often do not see nice unimodal rating distributions. Different users value different things, and even if they agree on what rating a product on Amazon 4 stars out of 5 should mean, they disagree about the ratings of individual products. Naively, you might expect there to be a distribution with a peak around the "true" quality of a product, falling off monotonically in both directions, with most products being of average quality.
If a product is "really" a three and a half star product, you may expect a distribution like 05-10-40-30-15, meaning 5% gave one star, 10% game two stars, and do on.
If you have ever spent any time at all researching your options on Amazon, you know this is not the case. Most products have a bimodal distribution of star ratings. The most common is 40-5-10-15-30. There is a peak at one star, and a peak at 5 stars, with a low ramp-up in between. The product is cheap, and it does the job, except for half the users, it broke after one or two uses. The people who didn't get a defective unit rate it higher. There are variations of this ratings distribution, depending on how quickly it breaks, how good customer service is, how well it works before breaking, and so on.
Other bimodal distributions, like 40-10-5-15-30, are also common. If you see a one-sided distribution like 60-20-10-5-5, you know you should avoid the product, no matter what the written reviews say. If you see something with a peak around three or four stars, like 10-15-25-40-10, you cannot possibly know why the product wasn't rated five stars without reading some of the reviews. It's probably something that does the job, but has a major ergonomic flaw, but it also could be a product that does the job, but badly. There must be a drawback or more people would have rated it five stars. Maybe it's too bulky. Maybe it's too expensive.
Likewise, if a product is rated 10-75-5-5-5, you don't know why it doesn't have a peak at one star without reading the text.
The underlying problem with the meaning of Amazon ratings is that "product quality" is not one-dimensional, even if raters all agree more or less on what product quality is, what it means, and that star ratings should somehow reflect product quality. I suspect that for the average Amazon buyer, even stupid-sounding reviews like "I don't know whether it works because I haven't tried it yet. I bought the life jacket for emergencies" or "This is too big to fit on my shelf" fulfil some important function.
Many Amazon review distributions are a composite of two of these: Bell curve, fat-tailed one-sided distribution, uniform with a single outlier peak, one-sided distribution, bimodal/bathtub curve. Many Amazon listings contain products that are updated, or use old product pages with good ratings to sell new products.
The five-star range is just about granular enough. If you look at professional reviews of games, the difference between 76% and 82% is highly meaningful, and the step from 66% to 72% is a world of difference. On the other hand, the difference between two out of ten and four out of ten is completely meaningless, and 1/10 might as well not exist. Everything that is 69% and below, or 6/10 and below, means "don't buy", and everything below 80% means "wait for a sale". Sometimes even games that are reviewed as "deeply flawed" or "uninspired" routinely get an 8/10. Out of five stars, 8/10 means 3 stars, and on a four star scale with half-stars it means 2½. AAA games like Detroit: Become Human get enough benefit of the doubt to still get an 8/10, with reviewers panning the gameplay, writing, cliched premise, pacing, and characters of a primarily story-driven game, but grasping for positives to justify a B+ for effort. Only something truly baffling like The Quiet Man can break through the 7/10 sound barrier and get a 4/10.
With that in mind, it's no wonder that Steam has limited your choices to thumbs-up and thumbs-down. In practice
I just wish Amazon would let me see "recent reviews" or a graph over time, a Steam feature that ostensibly was introduced to fight review bombs, but in practice one which helps buyers filter out games that were initially good, but which recently got replaced with pay-to-win games, like in those zombie Amazon listings.
The Theme Category
All of this brings me to Ludum Dare. If you follow my work there, you have probably guessed where I am going with this. I wrote a post on the LDJAM.com site about the Ludum Dare rating system. I'm not a fan. I thought, now that the LDJAM site is offline and PoV is in hiding, rather than trying to re-write my post from memory, I'd try to approach the same topic from the opposite direction. I knew I would still end up in the same spot.
Ludum Dare has these rating categories: "overall", "fun", "innovation", "graphics", "audio", "humour", "mood", and "theme".
Some are harder to grasp, some are self-explanatory. But some people think "graphics" means "this game has impressive/detailed/elaborate graphics", and some people think it means "the graphics serve the main point or main gameplay loop of the game very well". Some people think that humour means "this game is funny and has a lot fo jokes" and some people think "this game uses humour well, if at all".
The "theme" category is the worst offender, because it means either "this game sticks close to the theme and incorporates it into the mechanics" or "this game reinterprets the theme in a creative way" or "this game interprets the theme in a unique and novel fashion" or "this game really digs into what [whatever this theme is this time] means in our day and age".
So if the theme is "Dating Simulator", then some people will make "Kitchen appliance dating simulator" (I am very confident that this already exists) and some people will make "Carbon Dating Simulator", a game where (psych!) you really just date fossils, and somebody will interpret the same theme to make a game also titled "Carbon Dating Simulator", except (double psych!) you actually go on dates with fossils. Maybe there will also be a game where you're really old and you date other old people and the humour doesn't land... What can you say, it's a game jam, and they can't all be winners.
If, on the other hand, I went to itch.io and joined the next lesbian dating sim jam (again, without Googling, I think I can confidently say that this exists in multiple forms), and if I uploaded a game where you're a straight woman from the Greek island Lesbos dating men (possibly from other parts of Greece) on the island of Lesbos, then the people running the jam wouldn't congratulate me for my creative interpretation of the theme, even if I put in a lot of work and took photos of real places on Lesbos in Greece. They would probably think I was taking the piss out of them, even if the game was actually good, even if it was well-written and played (on my re-read of this paragraph, I realise that for the first time ever, a pun was actually not intended) completely straight.
In the old rule set of Ludum Dare, the rules were roughly "the theme is optional, but if you don't use it at all, others will give you a bad overall score, and if you use it well, they may give you a good theme score."
In the new rule set of Ludum Dare, the rules around theme are roughly "The theme is not really optional, but also not mandatory. You are supposed to use it, but this will not be enforced. Also, if somebody is doing something really inventive with the theme, you may want to reward that with a high theme score."
When I tried to point out my problems with the Ludum Dare ratings, I got some interesting feedback. Different people on the Ludum Dare site actually interpret the theme question differently.
Some think means "adhering to the theme", and they rate accordingly, and consistently so. To some of them, five stars in theme means a definitive and comprehensive exploration of the theme. To others, it means a mechanical interpretation of the theme, or to have the theme deeply integrated into the gameplay in another way.
Some think the theme category means "interpreting the theme in a far-fetched, non-literal or non-obvious way", and they give games low ratings when they take on the theme in a straightforward or mechanical way. The most important thing is not how much you rely on the theme, but how unique and clever your interpretation is.
Some think the theme may not be strictly mandatory, but they give games a low overall rating if they do not incorporate the theme enough, and a high theme rating when they interpret the theme in a far-fetched way.
Developers from group #1 were most likely to be annoyed by the difference in interpretation of the theme category. Developers from group #2 mostly seemed unbothered, because it's all subjective anyway. Group #3 seemed to have made their peace with the rating system.
I wasn't surprised by the people who didn't care about the disagreement. It's just for fun, you aren't supposed to take it seriously, what matters is participation, yada yada. I've heard all of that before. I also wasn't surprised by the insistence that ratings are subjective "anyway", so there is no difference between "this game is objectively good, therefore I give it five stars" and "this game is objectively good, but I don't like it, one star". If ratings are supposed to be subjective, then disagreement about the meaning of the rating categories is just noise – I disagree with sentiment. I find it annoying, but I expected to encounter it.
What I didn't see coming is that it was mostly people like me, those who agreed with my own interpretation of the theme category, who felt the same annoyance about the disagreement. People in #2 and #3 were not concerned, or interested in clarifying what theme ratings actually mean, or how you should rate games. I should have seen it coming, I guess.
Solutions
Even though it is not an agreed-upon problem, we can still apply solutions that work for everybody. At the very least, we could apply unobtrusive solutions that won't alienate people who don't think there is anything wrong.
Mandatory Randomly Assigned Ratings: This won't solve the problems of the theme category, but it would make Ludum Dare ratings "more fair". Every player/developer gets a list of five games that run on the same platform. If you develop a game for windows, you are assigned five games that run on windows. You have to rate them. This is in addition to the 20 ratings your game needs. If you don't rate these five games, your game won't be counted. This would make the ratings be more evenly calibrated and less skewed by popularity and selection effects, but it would also be a highly obtrusive change.
Sample Ratings: There is a list of games that have been rated by a panel of experts, with each rating in each category present at least once. If you want to see whether a game deserves two stars in graphics, you can compare it to a canonical list of two-star graphics games. Of course this would be highly obtrusive, take a lot of time, and you could never get a panel to agree on star ratings. Players/raters/developers would also need to play these games, and not just look at their names in the list. It would be quite harsh on the devs of canonical one-star games. It could solve the problem though.
Explanations of Categories: This is not that obtrusive. All you would need to do is to write a long version of the guide/rules, revise the current guide/rules so it isn't in contradiction with the long version, and then link to the long version from the short version. What is fun? What is mood? When do games deserve a mood rating? Is it ambition or execution? Can a game be five stars on fun and zero overall? What do these ratings mean? I suspect many developers would find taking a definitive and opinionated stance on matters of taste distasteful, and they would probably protest.
Rank Ranking: Instead of letting raters give star ratings, let's do pairwise comparison or best-to-worst sorting. This might be highly contentious, and it would be an obtrusive change to the rating system, but here is a tweak: Instead of making raters list games best to worst, add a view where raters are shown each category sorted best to worst. The rater can drag games around in this view. If a game with more stars is dragged below a game with fewer stars, the star ratings of these games are swapped. Alternatively, give the rater a tier list view, where S tier is five stars, and D tier is one star. These views would be optional and unobtrusive.
Two Theme Rankings, Three Graphics Rankings, Five Golden Rings, Seven Samurai: Specify multiple distinct interpretations of rating categories, and let some raters loose on old Ludum Dare games. Tell them to rate "How closely does this follow the theme" or "How detailed/elaborate are the graphics" and "How realistic are the graphics" and "How pretty are the graphics" and "How well do the graphics serve gameplay compare to more detailed/more realistic/prettier graphics". Is there that much difference in practice? Which one predicts Ludum Dare ratings best?
Recommendation System: After the jam is over, instead of showing the games in ranked order, go full Netflix, and show logged-in users games they might also like. Under games, show related games, based on a simple "people who liked this game also liked..." recommendation engine. You could also allow logged-out users to apply fine-grained metrics and tags, but these would not be subjective or value-laden. Instead, they would be information like "number of minutes to complete" or "how many times did I die before I beat the game" or "sentiment on a scale from fluffy to grimdark".
Histograms: This one is the easiest to implement, unobtrusive, and definitely the most bang for your buck. Instead of telling players the average "fun" or "overall" score, show them the histogram/bar chart. You know, like on Amazon.











