This post is by A Crowded Space from A Crowded Space
Click here to view on the original site: Original Post
I’m sure everyone here has received several hundred of these emails: “How would you rate this book / freelancer / coffee maker /_etc_, on a scale of 1 through 5 stars?”
If you know me, you know that I’ve ranted about how these questions don’t get to a true understanding of high quality. They might help identify some bad actors, but they are terrible at identifying the highest quality in a marketplace.
However, I recently learned that some marketplaces are using the quantitative score in addition to the response rate to get a more interesting signal of quality. See this excellent paper by Chris Nosko and Steven Tadelis for a thorough academic perspective. (hat tip to Andrei Hagiu for pointing this out to me)
I was excited to learn about this – it makes a ton of sense. The mental reaction when I get one of those “how would you
xyz?” emails is to almost always immediately ignore it. The only time I will take the time to respond is if I had a truly terrible or a truly wonderful experience.
It’s no surprise that many marketplaces have a response rate to that feedback question in the single digit percentages. I’d guess that a typical response rate is between 5-20% for email questions. Marketplaces get much higher response rates if the question is asked via SMS or in-app. What I did not appreciate until recently is that the response rate will vary widely based on the quality of the good or service.
Consider this scenario:
“How would you rate Game of Thrones?” – result: 4.9 stars, 500 ratings out of 1,000 possible responses.
“How would you rate Lord of the Rings?” – result: 4.9 stars, 500 ratings out of 5,000 possible responses.
Which one is better?
Most platforms would probably say that they are the same. However, I think it’s reasonable to say that Game of Thrones is much “higher quality” based on the response rate. If 50% of people that watched thought it was worth spending a minute to rate the show, that is an incredibly strong indication that it’s a well-loved show. It is creating that exceptionally wonderful experience that causes people to invest additional time to provide a high rating.
On the other hand, a very low response rate is likely an indicator that there are actually a lot more silent people that would rate things somewhere in the middle, perhaps a 3 or 4 star rating, but didn’t think it was worth the time to enter a rating. 3 stars, just ok, not worth a single click.
Consider this scenario:
4.9 stars, 100 ratings, 20% response rate
5.0 stars, 100 ratings, 10% response rate
This is where I think things really start to get interesting. I could easily see the higher response rate being a more important indicator of quality than the actual star rating itself.
As usual, these platforms are all special snowflakes and everyone will need to decide if and how to implement a measure of response rate into your overall quality and matching algorithms.
I put together this trusty 2×2 to help think about the categories.
Please reach out if you have any more experience using response rate for quality. Would love to hear from you.