Earlier today Stowe Boyd posted a quote to Tumblr:
How Airbnb Evolved To Focus On Social Rather Than Searches – Cliff Kuang via Co.Design
For a couple years, registered Airbnb users have been able to star the properties they browse, and save them to a list. But Gebbia’s team wondered whether just a few tweaks here and there could change engagement, so they changed that star to a heart. To their surprise, engagement went up by a whopping 30%. The star, they realized, was a generic web shorthand and a utilitarian symbol that didn’t carry much weight. The heart, by contrast, was aspirational. “It showed us the potential for something bigger,” Gebbia tells Co.Design. And in particular, it made them think about the subtle limitations of having a search-based service. “You have to have search,” Gebbia says. “But what if you don’t know where you want to go?”
I noted elsewhere that I think there are other factors in play here as well, but it got me thinking. A 30% increase in engagement is a good thing, right? Probably, but let’s move into gedankenexperiment territory for a moment.
While “user engagement” seems like something that you want headed up and to the right under all circumstances, the truth is that it’s like any other metric: it’s only meaningful in the context of clear goals.
Consider two possible scenarios:
A. Users who spend more than X minutes on the site are more likely to book through Airbnb. Changing the star to a heart eliminated a psychological barrier that some users felt, and the resulting increased engagement takes a non-trivial number of users over that X minute mark, so Airbnb gets more bookings. That’s good.
B. Users who “star” listings were more likely to convert to booking through Airbnb when they get a reminder notification. Changing the star to a heart eliminated the psychological barrier that some users felt, but the new “heart” people don’t convert to booking at the same rate the star people did. It’s actually a little harder to identify high-potential users now. That’s bad.
Since Airbnb seems like a well run company I assume that their situation is closer to the former case, but it’s a good reminder in any case: no matter how important the metric is, don’t confuse the metric itself for the goal that it supports.