We Experiment On Human Beings!

July 28th, 2014 by Christian Rudder

I’m the first to admit it: we might be popular, we might create a lot of great relationships, we might blah blah blah. But OkCupid doesn’t really know what it’s doing. Neither does any other website. It’s not like people have been building these things for very long, or you can go look up a blueprint or something. Most ideas are bad. Even good ideas could be better. Experiments are how you sort all this out. Like this young buck, trying to get a potato to cry.


We noticed recently that people didn’t like it when Facebook “experimented” with their news feed. Even the FTC is getting involved. But guess what, everybody: if you use the Internet, you’re the subject of hundreds of experiments at any given time, on every site. That’s how websites work.

Here are a few of the more interesting experiments OkCupid has run.

Experiment 1: LOVE IS BLIND, OR SHOULD BE

OkCupid’s ten-year history has been the epitome of the old saying: two steps forward, one total fiasco. A while ago, we had the genius idea of an app that set up blind dates; we spent a year and a half on it, and it was gone from the app store in six months.

Of course, being geniuses, we chose to celebrate the app’s release by removing all the pictures from OkCupid on launch day. “Love Is Blind Day” on OkCupid—January 15, 2013.

All our site metrics were way down during the “celebration”, for example:



But by comparing Love Is Blind Day to a normal Tuesday, we learned some very interesting things. In those 7 hours without photos:

And it wasn’t that “looks weren’t important” to the users who’d chosen to stick around. When the photos were restored at 4PM, 2,200 people were in the middle of conversations that had started “blind”. Those conversations melted away. The goodness was gone, in fact worse than gone. It was like we’d turned on the bright lights at the bar at midnight.



This whole episode made me curious, so I went and looked up the data for the people who had actually used the blind date app. I found a similar thing: once they got to the date, they had a good time more or less regardless of how good-looking their partner was. Here’s the female side of the experience (the male is very similar).



Oddly, it appears that having a better-looking blind date made women slightly less happy—my operating theory is that hotter guys were assholes more often. Anyhow, the fascinating thing is the online reaction of those exact same women was just as judgmental as everyone else’s:



Basically, people are exactly as shallow as their technology allows them to be.

Experiment 2: SO WHAT’S A PICTURE WORTH?

All dating sites let users rate profiles, and OkCupid’s original system gave people two separate scales for judging each other, “personality” and “looks.”
I found this old screenshot. The “loading” icon over the picture pretty much sums up our first four years. Anyhow, here’s the vote system:



Our thinking was that a person might not be classically gorgeous or handsome but could still be cool, and we wanted to recognize that, which just goes to show that when OkCupid started out, the only thing with more bugs than our HTML was our understanding of human nature.

Here’s some data I dug up from the backup tapes. Each dot here is a person. The two scores are within a half point of each other for 92% of the sample after just 25 votes (and that percentage approaches 100% as vote totals get higher).

In short, according to our users, “looks” and “personality” were the same thing, which of course makes perfect sense because, you know, this young female account holder, with a 99th percentile personality:



…and whose profile, by the way, contained no text, is just so obviously a really cool person to hang out and talk to and clutch driftwood with.

After we got rid of the two scales, and replaced it with just one, we ran a direct experiment to confirm our hunch—that people just look at the picture. We took a small sample of users and half the time we showed them, we hid their profile text. That generated two independent sets of scores for each profile, one score for “the picture and the text together” and one for “the picture alone.” Here’s how they compare. Again, each dot is a user. Essentially, the text is less than 10% of what people think of you.



So, your picture is worth that fabled thousand words, but your actual words are worth…almost nothing.

Experiment 3: THE POWER OF SUGGESTION

The ultimate question at OkCupid is, does this thing even work? By all our internal measures, the “match percentage” we calculate for users is very good at predicting relationships. It correlates with message success, conversation length, whether people actually exchange contact information, and so on. But in the back of our minds, there’s always been the possibility: maybe it works just because we tell people it does. Maybe people just like each other because they think they’re supposed to? Like how Jay-Z still sells albums?

† Once the experiment was concluded, the users were notified of the correct match percentage.

To test this, we took pairs of bad matches (actual 30% match) and told them they were exceptionally good for each other (displaying a 90% match.)† Not surprisingly, the users sent more first messages when we said they were compatible. After all, that’s what the site teaches you to do.



But we took the analysis one step deeper. We asked: does the displayed match percentage cause more than just that first message—does the mere suggestion cause people to actually like each other? As far as we can measure, yes, it does.

When we tell people they are a good match, they act as if they are. Even when they should be wrong for each other.



The four-message threshold is our internal measure for a real conversation. And though the data is noisier, this same “higher display means more success” pattern seems to hold when you look at contact information exchanges, too.

This got us worried—maybe our matching algorithm was just garbage and it’s only the power of suggestion that brings people together. So we tested things the other way, too: we told people who were actually good for each other, that they were bad, and watched what happened.

Here’s the whole scope of results (I’m using the odds of exchanging four messages number here):



As you can see, the ideal situation is the lower right: to both be told you’re a good match, and at the same time actually be one. OkCupid definitely works, but that’s not the whole story. And if you have to choose only one or the other, the mere myth of compatibility works just as well as the truth. Thus the career of someone like Doctor Oz, in a nutshell. And, of course, to some degree, mine.

1,220 Responses to “We Experiment On Human Beings!”

  1. Allie says:

    OKC,
    no worries, no complaints.
    Taking a broad view at this I would say, as long as you are not imitating Pavlov
    keep working at making the matches better and keeping it free.

  2. e hall says:

    If you do not experiment, you never know for sure; also, experiments engender and speed growth. But being told of false data will diminish confidence and comfort, which will reduce use and success of the site.

  3. Moonlight Driver says:

    You sound proud–but your callous manipulation of the system to ‘experiment’ on us like a bunch of lab rats is UTTERLY DESPICABLE. Take your experiment, and shove it!

  4. Aaron says:

    Fuck. You. I am outraged and intend to sue.

  5. garden girl says:

    Cry me a river: your f*¢king hilarious! You took the words straight out of my mouth

  6. Dan says:

    Thanks for bringing back the blog. Honestly, I think reading these is one of the most entertaining parts of the site.

  7. bill says:

    Life’s too short to be experimented on once in a while. ..let’s do it more often. ..

  8. Andrew says:

    Thank you OKC for writing another blog! These blogs have always been so useful for me to understand the dating scene and how people think. Your scope is large enough that it is as powerful as any peer-reviewed journal study, which makes it incredibly useful.

    Our own biology and psychology work against us most of the time, and so being aware of our internal subconscious biases makes these analysis worth it.

  9. Jesse says:

    I am more or less okay with this type of research. The interactions are largely based on the compatibility scores. In the case of Facebook, they are messing with existing relationships between you, your family, and your friends. OkCupid is messing with your relationship with strangers.

  10. e hall says:

    After reading some of the comments, I saw some misunderstanding. No where in the experiment was there anything analagous to a “postal worker opening my mail.” No private information was divulged. At worst, a few people who would not have been in contact had a conversation. The put-up profile hurt absolutely no one, and might (though not likely) have awakened someone to their shallow, though maybe valid for them, judjements. I think this was a worthy and valuable social experiment with negligable down side.

  11. Ben says:

    Er, has anyone pointed out that OKCupid is basically free? You get what you pay for. If you were *paying* OKCupid to give you amazingly accurate matches (something of course no dating site can do) you might have grounds for complaint.

    But if you’re not paying, then anything OKCupid does for you is a big favour in return for diddly-squat – if you don’t like it, you can lump it.

  12. bigsexydan says:

    the ‘matches’ i’m getting are always so very dissapointing!

  13. keith says:

    My question is why I keep getting “matches” that are more enemy than friend. The majority don’t even fit the characteristics I specified as my preference.

  14. Ben says:

    awesome to see another of these blog post! I love reading about the experiments you guys do!

  15. Dale says:

    This is actually a very interesting experiment. For those who are whining about informed consent; do you really think the results would be the same if you knew what was happening? There have actually been studies proving that when people know the purpose of an experiment(or even if it is happening at all) the results are drastically different from those when they have no idea.

    The simple fact is the only way to get accurate results from an experiment like this is to keep the test subjects in the dark.

  16. Barbara Jones says:

    A lot of men who show as 90% and above for me…I would not date. What I usually do is look at the questions that are in the “Unacceptable answers” group. if I see no deal breakers, he’s a good match…but usually in the high percentage matches…they have horrible unacceptables…that I just can not accept. My 2 cents worth. I never just go by the match percentage.

  17. Philip Snyder says:

    Every website runs experiments on their users. Anyone who doesn’t realize that is only fooling themselves. Generally these experiments are used to figure out how to get people to buy something. I think a dating site should be encouraged to experiment in order tune the algorithm and create better results. I can’t say I’m really surprised about any of the results … seems about right to me. I was surprised at the degree your tests went though. Makes me wonder about the tests you didn’t tell us about!

  18. Aeyr says:

    For anyone complaining about information being collected, see:

    http://www.okcupid.com/legal/privacy

    In particular, see “How we use the information we collect.” The ‘Terms and Conditions’ which you accepted when you created your account on the website expressly gives them permission to use data that you willingly submit to the website to be used in the ways they state.

    P.S. Only the manipulation of ‘match %’ is really an “experiment”. The rest is simply a correlation analysis.

  19. Ron says:

    I pay no attention to the match percentage but by their profile and photo. I have noticed that your matching me up with women I would NOT even consider meeting. For example, you have sent me many matches of Black and obese woman. Not of interest to me. You need to create a filter so we can select them by ethnic and size. I find that looking at woman who are….Let’s say…..fat and ugly is a waste of my time and a turn off to OK Cupid.
    Thats just my opinion.

  20. Margaret Tellez says:

    Now I get it! Your “experiment” must be responsible for the succession of trolls you send me on a daily basis. Do me a favor and send a few who aren’t QUITE so hard on the eyes. It’s true that beauty is skin deep– but ugly is to the bone.

  21. Teresa Brown says:

    I didn’t like what you did at all. I remember writing to you more then once saying you couldn’t be more wrong with the matching system. I absolutely didn’t agree. I knew something was real wrong with this. I’m rarely here. I almost went off entirely and still may.

  22. Yup says:

    People here are funny. How should they go about finding out how to find good matches except by trying it out? Do they think that that once programmed the algorithm years ago and never touched since?

    This isn’t facebook after all, this isn’t emotional manipulation within a supposedly scientific context with unaware subject (with testers who should know better).

    But a dating website that uses human made algorithms to try and match people.

    And the blog (and the data that come out of it) is one of the best parts of it. A very honest look at the human nature.

    If you don’t want anybody to look at, interpret and guide what you’re doing while dating the only option you have is to move your ass back into the bars and actually talk to people.

  23. Myrddwn says:

    Forget all these people who are upset, I suspect they are simply unhappy with the reminder of just how much control over their lives and information they have surrendered. This is how the internet works, folks. I also have to wonder how many of these people calling you ‘despicable’ are on Facebook, spreading their vitriol, when FB did the same thing on a much more egregious scale. For profit.
    I for one am not the least bit upset. I also had it figured out, when the same girls started showing up with and without text, at both high and low percentages.
    Calm yourselves and enjoy the FREE ride.

  24. eric says:

    This just proves how shallow people are and how nasty and trollish they can be on the interwebz.

    Good going everyone! :P

    Best of luck to the real and to the phonies, may you rot alone forever. :]

  25. Dan Twaddell says:

    Why can’t I zoom in on the mobile site? I can’t see the graphs (iphone 5 on safari). Frustrated.

  26. Teresa says:

    I didn’t like it one bit! Yet I knew something was up. I emailed you more then once telling you how OFF I thought your matching system was. I never got a response either Yet I doubted myself a few times, ok this guy sounds compatible, but with a ten per cent are you kidding me?? So yes I let your rating system keep me from contacting possibly a good match.?? It’s cruel to manipulate people this way. I’m rarely on here because it seemed so OFF. Someone must have wanted to be an author pretty bad.

  27. Chimer says:

    This summer I finally met 2 men I’d met on okcupid about 2 years ago. One is still on the site, as am I, and the other went off of it over a year ago. I finally took the initiative to meet them on a 2800 mile motorcycle trip I took. (saw one on day 2 and one on day 4 & 5)
    I tried to use the blind date thing when it was introduced, found it an exciting idea, but no takers…bummer. I sure do miss the “Notes” feature…I can’t remember their details/names. I hope you consider reviving that.
    I like the way things get tweaked and changed. Been on here for..umm..5 years?

  28. Teresa says:

    I didn’t like it one bit! Yet I knew something was up. I emailed you more then once telling you how OFF I thought your matching system was. I never got a response either Yet I doubted myself a few times, ok this guy sounds compatible, but with a ten per cent are you kidding me?? So yes I let your rating system keep me from contacting possibly a good match.?? It’s cruel to manipulate people this way. I’m rarely on here because it seemed so OFF. Someone must have wanted to be an author pretty bad. Yes this bares repeating twice. I didn’t see it go through before any way, I had more to get out here.

  29. Tom says:

    Not real happy that I may have been false numbers, but not mad either.
    Besides the false numbers thing, I have no problem with them running stats on my usage, as long as personal info isn’t disclosed.

  30. CR says:

    What IS it with guys who think their body type is “average?” I’ve found average to be very much overweight and often obese. I guess they think that since most of America IS overweight, by comparison they’re “just about right.”
    Thanks for letting us write back.
    c

  31. MG says:

    MORE!!!!!

    I love reading about lots of data. Especially when it comes to dating. This is incredibly interesting stuff. PLEASE post more often.

  32. Virginia (The Heartographer) says:

    I enjoyed this post, even though I kind of pick it apart on my blog (http://theheartographer.com/the-mere-myth-of-compatibility/). Really, I think you’re doing your users a HUGE favor when you’re transparent about how little weight those match percentage numbers actually carry. I wish you’d make it clearer!

    As an online dating coach, I send the bulk of my clients to you, because you’re the site that makes the most sense for them. But if I could get you to experiment with not showing match percentage at all, or to show it less (maybe hidden by default?), now THAT would be a victory. :)

  33. Blue says:

    I have to question the integrity of the results a little bit.

    I started a conversation BECAUSE I noticed that the match % had changed dramatically for what had been, up to that point, an extremely good match. (Can be verified by the content of my messages.)

    Either way, I’ve now had a somewhat meaningful conversation because of the experiment. So, thank you, OKC!

    (If you’re wondering why I hadn’t already messaged the other person, it’s because they had a clearly stated requirement that I could not fulfill.)

  34. Willy says:

    This is pretty much why I try to ignore the match-rating unless it’s below 40%. At that point I almost always see several deal breakers in their profile.

    The match/friend/enemy rating system is very confusing to me….

  35. Rose Darlin says:

    I don’t know if I’ve been experimented on, but I really find it hard to believe that a fat 56yr old lady can get over 90 likes. Humble bragging? Maybe. But thanks!

  36. beargirl says:

    First off, I have always assumed OKC has been running statistical metrics…..to me that’s what you do. Guess I didn’t expect you were manipulating it’s users but it’s your site. I joined freely. However, I have never looked at the match % as anything other than a faulty metric. People lie all the time. lol. I regularly receive messages from scammers. How do I know? Because of how they construct their sentences. They aren’t American and these are users who say they are American. These scammers usually disable their accounts rather quickly. They also have a similar pattern in the message they send to me.

    I have no issue with anything you’ve done because I don’t think it’s affected me at all. I left another free dating site because they decided the age difference that was allowed to contact or be contacted by someone. That is controlling.

    I wasn’t thrilled with OKC getting involved in the gay issue but I figured hey it’s your site. If you want to get political it’s your decision as a company. Things like that generally have consequences but obviously you made this as a calculated decision and hopefully not on emotion.

  37. Ape says:

    I am in agreement that testing is done on most websites to some degree. What I disagree with is dicking about wit aid subscribers while not being told that they are being tested on.
    It’s all bullshit anyway. Web based dating agencies are just a visual tool to decide a potential partner. To make things really happen you need to be in the same room to actually read and understand a person.
    No more experiments needed.

  38. choppergirl says:

    My quezstion for OkC developers… : why are you trying to encourage more courtship and breeding?

    The world is frightfully overpopulated as it is and is only able to sustain this insane overpopulation because of the Haber-Beyer process which is energy intensive. Eventually the stored energy from millions of years in the form of fossil fuels is going to run out and the planet is going to find itself woefully unable to fed everyone. And lots of people are going to end up dying…

    Stop breeding. You irresponsible stupid fuckers are doing massive ecological damage to the planet…

  39. STFU Whiners says:

    Get over yourselves and stop whining, losers.

    I personally couldn’t care less that OKC targeted me for one of these “experiments” – in fact, I’d be honored! You, know, “for science” and all that. I would like to take it one step further and say that if you OBJECT to being the test subject of an experiment which will cause virtually no harm to you and is being conducted for the good of society (in this case, it’s to make the whole site better, and not, as Andy points out, to bombard you with crap ads) – then you are a bad, selfish human and I would be much happier seeing you grovel for specks of meth under the freeway every morning. I believe the technical term is “gerping” or perhaps “gurping”.

    Anyone that whines about this is a superficial thinker, plain & simple.

  40. Clyde says:

    Thank you Christian for your blog and for your upcoming book. I have loved your blog since yous started it, and this new post is simply fantastic. In fact, to offset all of the people who are now refusing to pay you, I’m signing up for A-List.

    You’re the best! Keep experimenting.

  41. Michael Maner says:

    I would like to do this experiment.

  42. Aeromot says:

    Can you connect me with Ms. Driftwood? I know a guy she’d be perfect for. His name is Groot.

  43. ChillinDillon11 says:

    Are people really getting upset about this? Who cares. The site is for meeting people, and the experiment helped you meet more people. Good on ya and kind of hillarious. Lab rat the shit out of us!

    P.s. Who is dumb enough to actually trust these percentages with finding you “true love?” Come on meow, get real.

  44. FlagGreenGirl says:

    This is pretty interesting data. I have no issues with being experimented with in this way. I think it’s rather funny, actually.

    Thanks for sharing the results!

  45. Monkey says:

    Experementing is great. Like, just the tip.

  46. Chris Gold says:

    I’m pretty OK with having psychological experiments performed on me, but other people aren’t. Which is why the American Psychological Association and the American Sociological Association have both established codes of ethics that require licensed practitioners who conduct these studies to establish consent from their subjects. Had you or Facebook contracted professionals to aid in these studies, they surely would have told you to gain informed consent from your subjects first. Next time, please do.

  47. dbreeze says:

    Keep up the good work.

  48. cesar marcelo alvarenga says:

    im on your site and I really like it…. but for me foreing girls are better ( im in guatemala) but i do preffer the personality of girls in other places definitely…. loved your article,,,, feel free to experiment with me please the email of my account is pcplanetguatemala@hotmail.com sometimes it`s cool to be a guinea pig and see where other possibilities take you

  49. scott says:

    I like that OKC as a company is interested in employing these experiments. From this post, it appears that this particular blogger is actually excited about working for/with a dating website and is curious to know more about end-user behavior and match success rates. It’s nice to see that!!

  50. cs says:

    I am just so happy that OKC blog is open for business again!!