Five stars are a lie — Charles Bine

On March 17, thumbs-up/down feedback went live in Amoura.io. Naturally, my first inclination was to nerd out on how to collect feedback and what the science has to say about the various methods.

The lay of the land

Explicit feedback: thumbs, stars, numbers, reactions, pairwise comparisons.
Behavioural feedback: clicks, skips, rewatches, purchases, returns.

Collection and display needn’t match either: Rotten Tomatoes collects binary judgments from critics and shows the percentage classified as positive.¹

Netflix had collected more than 10 billion ratings before switching from stars to thumbs. I wanted to understand why it gave up the extra resolution.

The J-curve, or: five stars is a lie

Nan Hu, Paul Pavlou, and Jie Zhang looked at Amazon product ratings and instead of analysing average product ratings, they analysed the distribution of stars given. They found that, across the product categories they studied, the distribution of ratings was J-shaped: the mode was five stars, there were fewer one-star ratings, and few ratings in between.²

They explained this with two biases. People who are neutral about a product have less incentive to review it: under-reporting bias, skewing ratings towards both ends of the scale, into a U. The people reviewing it also bought it, presumably expecting to like it: purchasing bias, skewing that U towards the positive end, into a J.

To test this, they ran an experiment where participants were given some products and all of them were obliged to rate them. As expected, the distribution of ratings was much more unimodal than the J-shaped curve of voluntary product ratings.³

These ratings are missing not at random: whether someone responds depends on the opinion they would report.⁴ Averaging the ratings we received won’t recover the ones we didn’t. We’d need more data or assumptions about the non-responding group.

In 2009, YouTube product manager Shiva Rajaraman already referred to YouTube star ratings as more of a “seal of approval” than an indicator of how much the YouTube community liked a video.⁵ Shortly after, YouTube replaced its star rating system with likes/dislikes.

The Netflix arc

Netflix’s rating system looked successful: in 2017, Netflix had more than 10 billion ratings and more than half of its members had given at least 50 ratings.⁶ Still, Netflix VP of Product at the time Todd Yellin noticed that although members rated thought-provoking documentaries well, they watched lighter entertainment they rated lower much more often.⁷

Of course, these aren’t necessarily contradictory: you can really enjoy a movie but not feel like watching another movie like that one. A rating tells you how good someone thinks something is; it doesn’t necessarily tell you how interested you are in something similar.

Another issue with Netflix’s star rating system was that they displayed personalised star ratings (i.e. what Netflix predicted you would rate a movie) and many members mistakenly thought they were seeing aggregated ratings (like you do on review websites).⁸

When Netflix switched to thumbs-up/down in 2017, rating activity went up by 200 percent: three times as much feedback.⁸ Fewer buttons may have helped, but the purpose was clearer too: instead of reviewing movies for other consumers, you’re giving feedback to a recommendation algorithm. That test doesn’t tell us how much came from each change.

As a statistician, this makes sense to me: I care about predicting whether you’ll like something, not asking you to approximate an objective rating. And as a user, I want to tell Netflix what I liked, not decide how many stars a film deserves.

In 2022, Netflix decided that thumbs-up/down wasn’t enough granularity and introduced Two Thumbs Up – but instead of going back to a five star system, they added a way to distinguish “like” from “love”.⁹

So what does thumbs-up/down actually measure?

Although a thumbs-up/down system takes away some ambiguity as to what feedback options there are, it doesn’t resolve the ambiguity as to what exactly you’re rating. Did you like the response? Was it accurate/helpful? Would you like to receive similar responses in the future? For Amoura, this last question can be especially tricky: what if the character behaved realistically, but you disagreed with their response? Would you thumbs-down it?

If Amoura users thumbs-down a response, I’d like to know whether this was because the response was bad or because they disagreed with the character. Adding two or three more buttons probably won’t get me that information, but an optional follow-up question might.

The statistics

A star rating scale is an ordinal scale. Five stars is higher than four stars, but we can’t assume that the difference between four and five stars is comparable to the difference between two and three stars. Still, taking averages of ordinal scales isn’t completely invalid, as long as you’re aware of this assumption (and thus careful about basing decisions on small differences in averages).¹⁰

With binary feedback, you can calculate proportions and make inferences with binomial confidence intervals and logistic regression.¹¹ But this still doesn’t solve the problem of an ambiguous feedback question or selection bias.

With a balanced binary scale you receive one bit of information per response; with five equally likely stars, log₂(5) ≈ 2.32 bits.¹² If most people give five stars, you get less than that maximum.

In a simulation of multi-item psychoeducational scales, Lee and Paek found similar performance with four, five and six response options, with the biggest deterioration going from three to two.¹³ But that isn’t a test of whether distracted users will bother rating one response in a chat app.

A hundred thumbs up out of a thousand times shown isn’t the same as a hundred out of ten thousand. I want to log impressions as well as ratings. What about people who didn’t give feedback? Maybe they didn’t care one way or the other, didn’t notice the button, or were annoyed and left. I’d want to know which; null doesn’t tell me.

When the labels stop meaning what they say

Consider a rideshare app: for drivers, receiving anything less than five stars might be seen as receiving a complaint (and negatively influence their status in the app), whereas for passengers four stars might be a perfectly acceptable indication of “good”. In situations like these where politeness or people’s livelihood are at stake, five stars can indicate acceptability and anything less problematic. The interpretation of a feedback scale depends on the users’ behaviour on that scale.

What I picked for Amoura

For Amoura I’ve gone with thumbs-up/down. People are coming to chat, not participate in a study, so I don’t want to overly tax their attention by asking them to rate us. Explicit feedback is useful for sanity-checking behavioural feedback: how often people return, how in-depth their conversations are, etc. But a long conversation could mean they’re particularly engaged or can’t work out how to get what they want.

My implementation notes:

It should be clear what exactly the thumbs-up/down refer to.
Consider asking (optional) follow-up questions; collect enough context to understand the feedback, within a clear privacy boundary.

If I introduce a third response option for Amoura, I’d like to be able to explain what I want people to express and why I can’t get that from the existing feedback. Netflix had a use for “love” as well as “like”. I don’t have an equivalent distinction in mind yet.

Rotten Tomatoes defines the Tomatometer as the percentage of approved critic reviews classified as positive; at least 60 percent is considered Fresh. About Rotten Tomatoes. ↩
Nan Hu, Paul A. Pavlou, and Jie Zhang, “Why Do Online Product Reviews Have a J-Shaped Distribution?” Published as “Overcoming the J-Shaped Distribution of Product Reviews,” Communications of the ACM 52, no. 10 (2009), 144–147. DOI: 10.1145/1562764.1562800. ↩
Hu, Pavlou, and Zhang also report a controlled experiment in which all participants reviewed assigned products; unlike voluntary Amazon reviews, those responses were approximately unimodal. SSRN working paper. ↩
Missing data is “not at random” when the probability of being missing depends on the unseen value itself, even after accounting for what was observed. Here, people with middling opinions are systematically less likely to leave a rating. ↩
Shiva Rajaraman, Five Stars Dominate Ratings, Official YouTube Blog, September 22, 2009. ↩
Contemporary reporting on Netflix’s 2017 product briefing states that members had submitted more than ten billion ratings and that more than half had rated at least fifty titles. MacRumors, March 17, 2017. ↩
Todd Yellin contrasted highly rating a serious documentary with watching an Adam Sandler movie ten times more frequently. The Verge, March 16, 2017. ↩
Cameron Johnson, Goodbye Stars, Hello Thumbs, Netflix, April 5, 2017. The announcement reports a 200 percent increase in rating activity; it does not isolate reduced effort from the changed interpretation of the control. ↩ ↩²
Christine Doig-Cardet, Double the Thumbs for Even Better Recommendations, Netflix, April 11, 2022. ↩
Brad Verhulst and Michael C. Neale, Best Practices for Binary and Ordinal Data Analyses, Behavior Genetics 51 (2021), 204–214. ↩
A binomial interval expresses uncertainty around a positive-response proportion. Logistic regression estimates how predictors change the odds of a binary outcome. Neither repairs a badly defined question or a biased sample. ↩
Information entropy measures uncertainty in a distribution. For equally likely outcomes, the maximum is log₂(n) bits: log₂(2) = 1, while log₂(5) ≈ 2.32. Claude Shannon introduced the measure in A Mathematical Theory of Communication (1948). Entropy alone doesn’t tell us whether a rating is relevant to the outcome we care about. ↩
Jihyun Lee and Insu Paek, In Search of the Optimal Number of Response Categories in a Rating Scale, Journal of Psychoeducational Assessment 32, no. 7 (2014), 663–673. The study used simulated item-response data, varying response categories, scale length, and item discrimination. ↩