Pink Hearts, Orange Stars, Yellow Moons


Earlier today Stowe Boyd posted a quote to Tumblr:

How Airbnb Evolved To Focus On Social Rather Than Searches – Cliff Kuang via Co.Design

For a couple years, registered Airbnb users have been able to star the properties they browse, and save them to a list. But Gebbia’s team wondered whether just a few tweaks here and there could change engagement, so they changed that star to a heart. To their surprise, engagement went up by a whopping 30%. The star, they realized, was a generic web shorthand and a utilitarian symbol that didn’t carry much weight. The heart, by contrast, was aspirational. “It showed us the potential for something bigger,” Gebbia tells Co.Design. And in particular, it made them think about the subtle limitations of having a search-based service. “You have to have search,” Gebbia says. “But what if you don’t know where you want to go?”

I noted elsewhere that I think there are other factors in play here as well, but it got me thinking. A 30% increase in engagement is a good thing, right? Probably, but let’s move into gedankenexperiment territory for a moment.

While “user engagement” seems like something that you want headed up and to the right under all circumstances, the truth is that it’s like any other metric: it’s only meaningful in the context of clear goals.

Consider two possible scenarios:

A. Users who spend more than X minutes on the site are more likely to book through Airbnb. Changing the star to a heart eliminated a psychological barrier that some users felt, and the resulting increased engagement takes a non-trivial number of users over that X minute mark, so Airbnb gets more bookings. That’s good.

B. Users who “star” listings were more likely to convert to booking through Airbnb when they get a reminder notification. Changing the star to a heart eliminated the psychological barrier that some users felt, but the new “heart” people don’t convert to booking at the same rate the star people did. It’s actually a little harder to identify high-potential users now. That’s bad.

Since Airbnb seems like a well run company I assume that their situation is closer to the former case, but it’s a good reminder in any case: no matter how important the metric is, don’t confuse the metric itself for the goal that it supports.

  • Reminds me of my conversations with call center executive that were enormously proud of their reduction in call times and subsequent decrease in call wait times. Then those results were correlated with customer satisfaction and churn numbers however, it told a very different story.  Customers complained about wait times, but the focus on wait times had the effect of frustrating customers even more.

    Moral of the story; do not fall in love with one metric at the expense of all the other metrics.

  • Exactly so. A friend who works in the analytics has two precepts:

    1. If you didn’t measure it, it didn’t happen.
    2. Even if you measured it, it may not matter.