We Experiment On Human Beings!

July 28th, 2014 by Christian Rudder

I’m the first to admit it: we might be popular, we might create a lot of great relationships, we might blah blah blah. But OkCupid doesn’t really know what it’s doing. Neither does any other website. It’s not like people have been building these things for very long, or you can go look up a blueprint or something. Most ideas are bad. Even good ideas could be better. Experiments are how you sort all this out. Like this young buck, trying to get a potato to cry.


We noticed recently that people didn’t like it when Facebook “experimented” with their news feed. Even the FTC is getting involved. But guess what, everybody: if you use the Internet, you’re the subject of hundreds of experiments at any given time, on every site. That’s how websites work.

Here are a few of the more interesting experiments OkCupid has run.

Experiment 1: LOVE IS BLIND, OR SHOULD BE

OkCupid’s ten-year history has been the epitome of the old saying: two steps forward, one total fiasco. A while ago, we had the genius idea of an app that set up blind dates; we spent a year and a half on it, and it was gone from the app store in six months.

Of course, being geniuses, we chose to celebrate the app’s release by removing all the pictures from OkCupid on launch day. “Love Is Blind Day” on OkCupid—January 15, 2013.

All our site metrics were way down during the “celebration”, for example:



But by comparing Love Is Blind Day to a normal Tuesday, we learned some very interesting things. In those 7 hours without photos:

And it wasn’t that “looks weren’t important” to the users who’d chosen to stick around. When the photos were restored at 4PM, 2,200 people were in the middle of conversations that had started “blind”. Those conversations melted away. The goodness was gone, in fact worse than gone. It was like we’d turned on the bright lights at the bar at midnight.



This whole episode made me curious, so I went and looked up the data for the people who had actually used the blind date app. I found a similar thing: once they got to the date, they had a good time more or less regardless of how good-looking their partner was. Here’s the female side of the experience (the male is very similar).



Oddly, it appears that having a better-looking blind date made women slightly less happy—my operating theory is that hotter guys were assholes more often. Anyhow, the fascinating thing is the online reaction of those exact same women was just as judgmental as everyone else’s:



Basically, people are exactly as shallow as their technology allows them to be.

Experiment 2: SO WHAT’S A PICTURE WORTH?

All dating sites let users rate profiles, and OkCupid’s original system gave people two separate scales for judging each other, “personality” and “looks.”
I found this old screenshot. The “loading” icon over the picture pretty much sums up our first four years. Anyhow, here’s the vote system:



Our thinking was that a person might not be classically gorgeous or handsome but could still be cool, and we wanted to recognize that, which just goes to show that when OkCupid started out, the only thing with more bugs than our HTML was our understanding of human nature.

Here’s some data I dug up from the backup tapes. Each dot here is a person. The two scores are within a half point of each other for 92% of the sample after just 25 votes (and that percentage approaches 100% as vote totals get higher).

In short, according to our users, “looks” and “personality” were the same thing, which of course makes perfect sense because, you know, this young female account holder, with a 99th percentile personality:



…and whose profile, by the way, contained no text, is just so obviously a really cool person to hang out and talk to and clutch driftwood with.

After we got rid of the two scales, and replaced it with just one, we ran a direct experiment to confirm our hunch—that people just look at the picture. We took a small sample of users and half the time we showed them, we hid their profile text. That generated two independent sets of scores for each profile, one score for “the picture and the text together” and one for “the picture alone.” Here’s how they compare. Again, each dot is a user. Essentially, the text is less than 10% of what people think of you.



So, your picture is worth that fabled thousand words, but your actual words are worth…almost nothing.

Experiment 3: THE POWER OF SUGGESTION

The ultimate question at OkCupid is, does this thing even work? By all our internal measures, the “match percentage” we calculate for users is very good at predicting relationships. It correlates with message success, conversation length, whether people actually exchange contact information, and so on. But in the back of our minds, there’s always been the possibility: maybe it works just because we tell people it does. Maybe people just like each other because they think they’re supposed to? Like how Jay-Z still sells albums?

† Once the experiment was concluded, the users were notified of the correct match percentage.

To test this, we took pairs of bad matches (actual 30% match) and told them they were exceptionally good for each other (displaying a 90% match.)† Not surprisingly, the users sent more first messages when we said they were compatible. After all, that’s what the site teaches you to do.



But we took the analysis one step deeper. We asked: does the displayed match percentage cause more than just that first message—does the mere suggestion cause people to actually like each other? As far as we can measure, yes, it does.

When we tell people they are a good match, they act as if they are. Even when they should be wrong for each other.



The four-message threshold is our internal measure for a real conversation. And though the data is noisier, this same “higher display means more success” pattern seems to hold when you look at contact information exchanges, too.

This got us worried—maybe our matching algorithm was just garbage and it’s only the power of suggestion that brings people together. So we tested things the other way, too: we told people who were actually good for each other, that they were bad, and watched what happened.

Here’s the whole scope of results (I’m using the odds of exchanging four messages number here):



As you can see, the ideal situation is the lower right: to both be told you’re a good match, and at the same time actually be one. OkCupid definitely works, but that’s not the whole story. And if you have to choose only one or the other, the mere myth of compatibility works just as well as the truth. Thus the career of someone like Doctor Oz, in a nutshell. And, of course, to some degree, mine.

1,220 Responses to “We Experiment On Human Beings!”

  1. Daremonai says:

    I weep for humanity.

    This was a really fascinating bit of analysis, but it is kinda depressing. I am sure many people will go ‘but I really do care about the text!’, but it is kinda besides the point, and chances are, they do not nearly as much as their self image says they do.

    I wonder if this means online dating is, long term, doomed.. or if there is some technique or pattern out there that will work better then meatspace.

  2. Marcus says:

    Too funny! Just found out about this blog yesterday and was so disappointed that it had been defunct for the last 3 years. Glad to see it’s back. Will be reading frequently!

  3. sabrina says:

    So cool you’re back.

  4. J says:

    Great article. As for the above comment, I can say that in my experience (straight male) perfecting the art of writing a good profile has made a marked difference. Having said that, getting my pics right has made a big difference to (thank you MyBestFace, and thank you puppy in Vietnam!).

    J.

  5. SteveRestless says:

    I know from my perspective that I see a definite difference between what you talk about doing here, and what facebook did.

    Facebook secretly manipulated the users of the site, attempting to alter their emotions. Quite possibly at the behest of creepy interests sponsoring this experiment.

    All I can see here is honest curiosity and a desire to improve the site. Completely different things. When you start trying to match people who are wrong with each other, just to see if you can play god, then I’ll have a problem with it.

  6. AQualityCatch says:

    Thanks for bringing this blog back.

    I love this kind of analysis, and it addresses the kinds of questions that anyone who has used this site has had on occasion.

    I imagine that all the dating sites have probably done similar analyses, but they never share them with the users because they’re “trade secrets” and, probably (and more importantly) they reveal some of the shortcomings of the other (mostly for-pay) sites.

    Keep it up, guys…

  7. Yup says:

    Awesome stuff! Great you brought the blog back! Please keep it up (or send me a DB dump, I’d just dig around it on my own)

  8. Nunyo Biznass says:

    I guess I’m the exception here.

    When I was reading over the “people pay more attention to pictures” bit, the whole time I was thinking of all the text-less profiles I’ve ignored (regardless of match percentage or looks or whether she lived nearby) and how 90%+ of the first messages I’ve sent were because something in the *text* stood out to me as saying “hey, this person might be pretty great” (…and on the other hand, how there’s been plenty of others whose looks I liked and whose answers to questions didn’t have anything super unacceptable, but I just couldn’t think of anything to say because the profile text just… wasn’t that special, leading me to believe that their personalities might be just as boring!)

  9. ADHDColorbomb says:

    I want to be surprised by these results, but I have to admit that I honestly am not. Outside of the organic way of making acquaintances through proximity x time x some kind of common interest, people are going to have a strong tendency to default to first impression, and if they are not particularly happy with what they see they will move on. “Plenty of fish in the sea” is a very easy mentality to acquire in an online world with a seemingly endless number of matches and the “perfect” person is surely just a few profiles away.

  10. ADHDColorbomb says:

    From a CogSci major with deep seated interests in sociology and psychology: are you guys hiring? ;)

  11. Nick says:

    Cool that you guys are giving this some life again. As a Computer Science Master, I thoroughly enjoyed reading the previous entries that were posted. It really gives a unique insight into how people behave and react (and especially tips and tricks ;p).

    Haha, I would love to get my hand on the datasets you use, just to poke around and see if there are any interesting patterns to discover.

    Hope too see more soon!

  12. Mark says:

    I hope you didn’t cause me to miss out on a relationship while playing around with the data. It is hard enough for me as it is. Also , people trusting your match percent is what you want, so don’t lie to them about it.

  13. apb_101 says:

    Good to know at any one time here forward I can trust or distrust your Match %s, not knowing if your lab experiment has resurfaced.

  14. wfro says:

    How big were the sample sizes for these? It seems like it would make a big difference to the significance of the findings…

  15. Daniel says:

    I picked an outrageous username, and then made my profile text very different from my username. Guess what? over half the people I emailed didn’t even look at my profile before responding. They would only respond with “your username is ridiculous”

    It’s probably not a good idea to experiment on live human subjects, tho. That’s unethical.

    However, if you’re only experimenting on hot women with no profile text (BOTs!), then I say, GO FOR IT.

  16. Christoph Wagner says:

    Awesome it’s back! Now step it (back) up and get us those neat interactive stats from way back when for next time :D

  17. Jean says:

    I’d like to see the actual female : male ratio on any dating site.

    I feel that the reason we are more “popular” than guys is simply due to dating site has way more male demographics than female.

    The “look is almost everything” result really isn’t too surprising. We are human after all. Most of us are here due to no small part to carnal desires.

  18. Matt says:

    When’s the Kindle version of the book coming out?

  19. an0n says:

    @SteveRestless
    What if the manipulation is used for what you believe does good for the person. That is what I am forced to believe is the reason, people end up manipulating people for money or good in their own heart.

    I think people are naive when looking at online dating sites. Most of the time the people that really use it are just looking for someone to have sex with. The people that actually want to find their partner in life, are out numbered by the other population and thus the stats show it.

  20. Abhijit says:

    So.. I’m paying you so you can fuck with me? I am talking about the last experiment. Yes, I tend to message women who show a very high “match” percentage with me and I have never heard back from anyone. On the other hand, I have met some really cool women who you said “Ya’ll have issues”…. Now that makes sense.

  21. alex says:

    Awesome to see a new post!

  22. Phosphor says:

    I started online dating in the summer of 2000–I’m not sure if the site is even around anymore. After three years or so of red-hot success, I got in an 8-year relationship with someone I met online. It ended early this year so I’ve jumped back in: and things are VERY different.
    The point of online dating is not to meet people; it’s to meet weird people. By that I mean, people who also are obsessed with Iranian cinema or Hummel figurines or lake biology or whatever. You should use online to look for things that are NOT apparent at a bar/general meetup/other offline activity. If you go by looks/don’t read the profile, especially if you’re a guy, you’re stupid. Go to a bar and get laid a lot more often.
    Also, OKC, by enabling mobile use you’ve destroyed a lot of value of the site. It is much harder to exchange meaningful messages or build rapport before meeting. Furthermore, you’ve made it fairly difficult to find casual sex partners on the site, which is the other giant thing you use online for (again, because it’s not something that’s easy to ask at a bar). So it’s kind of the worst of two worlds.
    Here’s how I’d innovate. Let users choose a version without pics. However, they *also* upload an anonymous pic associated invisibly with their account, which other users can rate BY ITSELF without knowing the profile it belongs to. The results are displayed to users searching the pic-less profiles (“Most users considered this person to be Very Attractive” or whatever). Then let the profiles speak for themselves. I think the value-add here, and for online dating in general, is to steer people toward making better relationship choices than they otherwise would. That starts with emphasizing things like the weirdness you each have in common :)

  23. MichaelinHD says:

    Why don’t you admit to using bots & what is the point of annoying people with bots that are using your site? It’s hard enough to actually meet someone here, they mostly seem stuck up & not really interested in any relationship, they simply want to advertise themselves, mostly untruthfully, for their own satisfaction. I’m not getting it….

  24. beth says:

    Hey Christian, I just got married to a man I met on OKCupid in 2010. I tried to log back in and retrieve our tests/match information because we thought that would be “cute” information to have or put in our insufferable scrapbook or whatever. Sadly, the accounts have been superdeleted (obviously) (if one of ours hadn’t been, that’s probably a problem). Is there a way I can importune someone who works with you to get us our match info? WAS SCIENCE RIGHT OR WRONG?
    Love,
    Beth and Justin (twowordheadline/nordicglow and holypopejustin)

  25. James says:

    So the key to improving your engagement metrics is to overestimate everyone’s match percentage :)

  26. karstic says:

    I only read the site for the articles..

  27. Kristyn says:

    Glad to see the analysis posts back! How can this account for people who get a lot of views, and few/no messages, or mutual matches and then no messages? I assumption it means they rate you highly based on the pictures, but then when they actually read the profile they see something they don’t like. This study says the text doesn’t matter, so then what would explain the low conversion rate between views and messages? I agree with the person who said that the mobile version has degraded the quality of the site. I think between that and Quickmatch there are a lot of casual users who just scroll through the pictures but are not interested in actually meeting someone.

  28. Data says:

    Great to see the blog back. I loved reading the old postings and they’re actually what made me register on OkCupid. I love to learn actual data-based facts about how people behave rather than all the speculations and n=10 experiments from many psychologists/sociologists.

    The scatterplot of looks rating and personality rating is hilarious.

  29. Jack says:

    This was an amazing read and very well put together. Kudos!
    However, the data that you gathered is very depressing. It merely shows that people are just shallow. I thought the purpose of online dating was suppose to be so you could find a partner. Apparently everyone is looking at it as free hook ups.

  30. Hey OkCupid – How about some SSL Love? says:

    Hey OkCupid – How about some SSL Love?
    https://www.eff.org/deeplinks/2012/02/hey-okcupid-how-about-some-ssl-love

    “For the hundreds of thousands of users searching for that special someone through one of the largest free online dating sites, the love fest may be coming to an end. OkCupid is putting users’ privacy in danger by failing to support secure access to its entire website through HTTPS. Every OkCupid email, chat session, search, clicked link, page viewed, and username is transmitted over the Internet in unencrypted plaintext, where it can be intercepted and read by anyone on the network.”

  31. john says:

    Keep the blog going!

  32. Anon says:

    This blog is absolutely unreadable on an iPhone 5!

    Please fire any designers you have, and serve just a html page with no css, so it’s readable on every device :(.

  33. Nikki says:

    Good to see the blog back!

  34. M says:

    @Daremonai, re: ” I am sure many people will go ‘but I really do care about the text!’, but it is kinda besides the point, and chances are, they do not nearly as much as their self image says they do.”

    I strongly suspect that the sorts of users who end up reading this blog probably *are* likelier to be influenced by a profile’s text more than most site users. It’s just that, evidently, the site’s users *in aggregate* don’t give a shit about the text. That doesn’t say anything about what any individual user cares about, though.

  35. srs says:

    Welcome back! I’ve missed your analysis. I’d be really interested in how the okc app is changing okc use. I’m getting a lot more emails than I used to from guys with empty profiles who have answered no questions. And their messages tend to be something like “hi how ru u seem nice.”

  36. A Guy says:

    Glad to see you go through our messages to improve your site.

  37. ZenLife says:

    This type of blog article is really interesting – it would make me want to renew my OKC subscription.

  38. filmartwordgirl says:

    Now that the blog is back, please consider bringing back the journals and some of the other features such as the “Awards” that Match.Com deleted when they purchased OKC. It is still the only dating/intro site worth its salt in my estimation, but they did destroy some of its special features. I feared for awhile that it would just become a Match.Com or E Harmony clone. Glad to see this shift back in its old direction.

  39. agoodguy says:

    And thus Tinder was born.

    I missed these blog posts – seeing a new one made me happy. I hope you make these types of posts a regular feature again.

  40. rubbagerutabaga says:

    My question is, what if you make a dating website with no pictures of any kind. Would you get more honest interests in people’s personalities and therefore better conversations, like in the experiment detailed above? Would you get self-selection for people who actually can read and care to read profiles? Or are people just petty and will bail as soon as they do see a picture?

  41. JBinks says:

    Glad that you are back! The other avenue for people to reveal more about themselves is where they “explain” answers. I’ve found blank profiles with thoughtful explanations for issues that I care about, and have actually been curious about them. Having tried other sites, I think OKC is one of the best.
    The only complaint I have is about your mobile apps. While your desktop site is awesome, your mobile app is not! It seems to cater more towards the “Tinder” crowd looking for a hookup.

  42. Ken Mortimer says:

    I’m going to reiterate what others have said: Experimentation for usability, aesthetics, etc is fine. It’s cool and really lets you know what your users like.

    INTENTIONALLY causing distress to users who may or may not have mental health issues to begin with is highly questionable, ethically.

  43. slartibartfasz says:

    Finally! The blog is back! Yay!

    As a long time user, I’m very happy indeed to see the blog back, and despite the 3 year radio silence, as snarky and statistically profound as ever :D

  44. forgottenman says:

    Lots of folks are commenting on the ethics of the experiments, but I’m surprised no one has commented on what I consider a (potentially) more unethical issue. Christian says that “conversations went deeper” and “contact details (phone numbers, emails) were exchanged more quickly”. How could OkCupid analysis know this unless it is READING OUR MESSAGE CONTENT? I will grant that conversational “depth” simply might be a measure of the length of a message, which wouldn’t require actual further content analysis. But for OkCupid to know whether I’ve given my “contact details” to another user in what I expect and believe to be a private exchange is outrageous. Further, is OkCupid extracting such details and retaining them? Do you note when I tell someone my real name, my street address, my home and cell phone and Skype numbers? What do you know about me, OkCupid, that I’ve been sharing with other users in what should be private conversations? What say you?

  45. Spiralis says:

    The current model of online dating is like the 80’s cell phone and in desperate need of an overhaul. While it worked when Online dating still had a really bad stigma, its increasing popularity demands a rethinking of the way we use the internet to find a partner.

    The idea of the compatibility formula has worn out its welcome. We need a way to re-introduce the human element. That is… OkCupid should consider following the path of some of the newer socializing/dating services and hiring matchmakers or setting up blind dates for folks.

  46. trlkly says:

    The first two were okay, since they were essentially public tests. But the third one had you lying to users (just like Facebook did). Even little kids know that lying is wrong, and here you are defending it.

    The fact that you think such things are okay is exactly why we have to get the FCC involved, because apparently the morals of rich people don’t include “Don’t deliberately deceive people.”

    Seriously, that last experiment was morally reprehensible. You put your own site ahead of your users’ interests. You lied to benefit yourself.

    People have been pressuring me to try OKCupid, and I have tentatively given in a bit. But now I won’t. You’ve admitted that you will deliberately try to screw up your users results to get information about your algorithms.

    You seem to think interesting data makes things okay. It doesn’t. I don’t normally curse in these things, but today I’m making an exception.

    Fuck you.

  47. Kieron says:

    Oh glorious day! An update.

    The most significant factor in your compatibility rating is the power of suggestion?

    Do you have any data on how it affects things in the longer term?

    Also since the power of suggestion is so powerful have you ever considered sending users daily recommendations?

  48. George says:

    We should start more controversies over experimentation if that’s what it takes to get this blog going again.

  49. 2atmosphere says:

    OKCupid used to be much more reading-and-text oriented, when they had the “personal blog” feature for each and every normal user.

    Please bring that back! Some of us out here are highly text-oriented, and OKCupid is still the best place for us to find each other and feel part of a community.

    But it would be sooo much better if it were also a site for ongoing semi-public conversations.

  50. Trevor says:

    You guys are jerks. It takes a lot of time and energy and money to go out on a date.

    You’re not scientists, your coders turned “entrepreneurs” who have lost touch with reality and are bored. So bored that you have turned to messing with your clients for fun.

    I bet your book sucks too.