Response Rate is a Quality Signal


This post is by A Crowded Space from A Crowded Space


Click here to view on the original site: Original Post




I’m sure everyone here has received several hundred of these emails: “How would you rate this book / freelancer / coffee maker /_etc_, on a scale of 1 through 5 stars?” 

If you know me, you know that I’ve ranted about how these questions don’t get to a true understanding of high quality.  They might help identify some bad actors, but they are terrible at identifying the highest quality in a marketplace

However, I recently learned that some marketplaces are using the quantitative score in addition to the response rate to get a more interesting signal of quality.  See this excellent paper by Chris Nosko and Steven Tadelis for a thorough academic perspective. (hat tip to Andrei Hagiu for pointing this out to me)

I was excited to learn about this – it makes a ton of sense.  The mental reaction when I get one of those “how would you

image

xyz?” emails is to almost always immediately ignore it.  The only time I will take the time to respond is if I had a truly terrible or a truly wonderful experience.  

It’s no surprise that many marketplaces have a response rate to that feedback question in the single digit percentages.  I’d guess that a typical response rate is between 5-20% for email questions. Marketplaces get much higher response rates if the question is asked via SMS or in-app.  What I did not appreciate until recently is that the response rate will vary widely based on the quality of the good or service

Consider this scenario: 

“How would you rate Game of Thrones?”  – result: 4.9 stars, 500 ratings out of 1,000 possible responses.   

“How would you rate Lord of the Rings?” – result: 4.9 stars, 500 ratings out of 5,000 possible responses. 

Which one is better? 

Most platforms would probably say that they are the same.  However, I think it’s reasonable to say that Game of Thrones is much “higher quality” based on the response rate.  If 50% of people that watched thought it was worth spending a minute to rate the show, that is an incredibly strong indication that it’s a well-loved show.  It is creating that exceptionally wonderful experience that causes people to invest additional time to provide a high rating.  

On the other hand, a very low response rate is likely an indicator that there are actually a lot more silent people that would rate things somewhere in the middle, perhaps a 3 or 4 star rating, but didn’t think it was worth the time to enter a rating.  3 stars, just ok, not worth a single click.  

Consider this scenario:

4.9 stars, 100 ratings, 20% response rate

5.0 stars, 100 ratings, 10% response rate

This is where I think things really start to get interesting.  I could easily see the higher response rate being a more important indicator of quality than the actual star rating itself. 

As usual, these platforms are all special snowflakes and everyone will need to decide if and how to implement a measure of response rate into your overall quality and matching algorithms.  

I put together this trusty 2×2 to help think about the categories. 

image

Please reach out if you have any more experience using response rate for quality.  Would love to hear from you.  

CategoriesUncategorized