We Experiment On Human Beings!

July 28th, 2014 by Christian Rudder

I’m the first to admit it: we might be popular, we might create a lot of great relationships, we might blah blah blah. But OkCupid doesn’t really know what it’s doing. Neither does any other website. It’s not like people have been building these things for very long, or you can go look up a blueprint or something. Most ideas are bad. Even good ideas could be better. Experiments are how you sort all this out. Like this young buck, trying to get a potato to cry.


We noticed recently that people didn’t like it when Facebook “experimented” with their news feed. Even the FTC is getting involved. But guess what, everybody: if you use the Internet, you’re the subject of hundreds of experiments at any given time, on every site. That’s how websites work.

Here are a few of the more interesting experiments OkCupid has run.

Experiment 1: LOVE IS BLIND, OR SHOULD BE

OkCupid’s ten-year history has been the epitome of the old saying: two steps forward, one total fiasco. A while ago, we had the genius idea of an app that set up blind dates; we spent a year and a half on it, and it was gone from the app store in six months.

Of course, being geniuses, we chose to celebrate the app’s release by removing all the pictures from OkCupid on launch day. “Love Is Blind Day” on OkCupid—January 15, 2013.

All our site metrics were way down during the “celebration”, for example:



But by comparing Love Is Blind Day to a normal Tuesday, we learned some very interesting things. In those 7 hours without photos:

And it wasn’t that “looks weren’t important” to the users who’d chosen to stick around. When the photos were restored at 4PM, 2,200 people were in the middle of conversations that had started “blind”. Those conversations melted away. The goodness was gone, in fact worse than gone. It was like we’d turned on the bright lights at the bar at midnight.



This whole episode made me curious, so I went and looked up the data for the people who had actually used the blind date app. I found a similar thing: once they got to the date, they had a good time more or less regardless of how good-looking their partner was. Here’s the female side of the experience (the male is very similar).



Oddly, it appears that having a better-looking blind date made women slightly less happy—my operating theory is that hotter guys were assholes more often. Anyhow, the fascinating thing is the online reaction of those exact same women was just as judgmental as everyone else’s:



Basically, people are exactly as shallow as their technology allows them to be.

Experiment 2: SO WHAT’S A PICTURE WORTH?

All dating sites let users rate profiles, and OkCupid’s original system gave people two separate scales for judging each other, “personality” and “looks.”
I found this old screenshot. The “loading” icon over the picture pretty much sums up our first four years. Anyhow, here’s the vote system:



Our thinking was that a person might not be classically gorgeous or handsome but could still be cool, and we wanted to recognize that, which just goes to show that when OkCupid started out, the only thing with more bugs than our HTML was our understanding of human nature.

Here’s some data I dug up from the backup tapes. Each dot here is a person. The two scores are within a half point of each other for 92% of the sample after just 25 votes (and that percentage approaches 100% as vote totals get higher).

In short, according to our users, “looks” and “personality” were the same thing, which of course makes perfect sense because, you know, this young female account holder, with a 99th percentile personality:



…and whose profile, by the way, contained no text, is just so obviously a really cool person to hang out and talk to and clutch driftwood with.

After we got rid of the two scales, and replaced it with just one, we ran a direct experiment to confirm our hunch—that people just look at the picture. We took a small sample of users and half the time we showed them, we hid their profile text. That generated two independent sets of scores for each profile, one score for “the picture and the text together” and one for “the picture alone.” Here’s how they compare. Again, each dot is a user. Essentially, the text is less than 10% of what people think of you.



So, your picture is worth that fabled thousand words, but your actual words are worth…almost nothing.

Experiment 3: THE POWER OF SUGGESTION

The ultimate question at OkCupid is, does this thing even work? By all our internal measures, the “match percentage” we calculate for users is very good at predicting relationships. It correlates with message success, conversation length, whether people actually exchange contact information, and so on. But in the back of our minds, there’s always been the possibility: maybe it works just because we tell people it does. Maybe people just like each other because they think they’re supposed to? Like how Jay-Z still sells albums?

† Once the experiment was concluded, the users were notified of the correct match percentage.

To test this, we took pairs of bad matches (actual 30% match) and told them they were exceptionally good for each other (displaying a 90% match.)† Not surprisingly, the users sent more first messages when we said they were compatible. After all, that’s what the site teaches you to do.



But we took the analysis one step deeper. We asked: does the displayed match percentage cause more than just that first message—does the mere suggestion cause people to actually like each other? As far as we can measure, yes, it does.

When we tell people they are a good match, they act as if they are. Even when they should be wrong for each other.



The four-message threshold is our internal measure for a real conversation. And though the data is noisier, this same “higher display means more success” pattern seems to hold when you look at contact information exchanges, too.

This got us worried—maybe our matching algorithm was just garbage and it’s only the power of suggestion that brings people together. So we tested things the other way, too: we told people who were actually good for each other, that they were bad, and watched what happened.

Here’s the whole scope of results (I’m using the odds of exchanging four messages number here):



As you can see, the ideal situation is the lower right: to both be told you’re a good match, and at the same time actually be one. OkCupid definitely works, but that’s not the whole story. And if you have to choose only one or the other, the mere myth of compatibility works just as well as the truth. Thus the career of someone like Doctor Oz, in a nutshell. And, of course, to some degree, mine.

1,220 Responses to “We Experiment On Human Beings!”

  1. Magic says:

    Dear OKC Robot,

    I never EVER used the matching percentages to decide if I would contact someone.
    The matching algorithm that sends suggested matches my way is so way off I do not think it makes a difference if you faked the numbers or not.

    m

  2. HotFlash says:

    “And, of course, to some degree, mine.”

    Only some degree?

  3. beargirl says:

    Oh and I do have a question. Why is it there’s no link on the site for the blog? At least I don’t see one.

  4. Dale Kamp says:

    As a person that often spends time reading other’s profile questions out of fascination, I found this post welcoming. Thank you for sharing the data.

    I’ve learned from OkCupid just how inconsistent people are in their ideologies.

  5. Evelyn says:

    Thank you for sharing all this research data with us. It’s interesting from various aspects of the geeky side, and offers transparency on your processes — as well as ours. ;-)

    I am motivated to go take some more flattering photos to upload.

    Nother thing, though: I refuse to rate other people. I don’t feel any desire to rate someone else or their profile. I don’t like the concept.

    Also, I don’t respond to people who supposedly want to meet me, which I think means they’ve rated my profile. So there’s a gap where perhaps HE thinks he has “sent me some kind of message” by pushing a button rating my profile, and I do NOT think he’s sent me a message, because he has not written me a note.

    So perhaps my lack of response to messages that were sent-but-not-sent means I have missed out on some (lazy) men. Don’t know.

    Worth thinking about?

  6. flow says:

    ok cupid is great. you know that the internet is the mind of god, shortly before the thing goes hyperdimensional? at any rate, putting stuff about yourself and who you are looking for up on here is a sure way to get god to notice and send you the right thing in real life. so experiment away. i’ve never met anyone i’d date through here, but the people i have talked to have all been fun. so thanks

  7. Rick Maiman says:

    No wonder why your suggestions suck. You keep pushing women in NY, that’s code for Manhattan. I hate the fucking borough. Bad enough having to go there for work, but for a date? I live in fucking Queens. Skip the NY too full of themselves unyielding women there.
    Your experiments, your robot suggestions are flawed and innacurate.

  8. Howard Crane says:

    This article was great! I really enjoyed it. I so would buy that book if I had the cash. I’ll put it on my wishlist for later.

  9. Ummm says:

    So why were the quizzes and tests and other random fun things taken away? Where is the treasure chest? Why wasn’t I contacted after scoring so high on that test that matches very personal questions to people’s photographs?

  10. alx says:

    Let me guess,
    Women who keep in shape have more messages han those who don’t. Men who are 6ft get more replies than those who are 5’7.
    Genius experimenting.

  11. I despise you says:

    you guys are horrible wastes of human organs. you should be deeply ashamed of manipulating match percentages. we all knew we were signing up for data mining, but actual out-and-out lies are deeply unethical. What if some guy went nuts and starting shooting people because of your bs? Would you sound even a titch less self congratulatory then?

  12. SMaggs says:

    This is pretty awesome. I like your My Best Face app as well, data driven analytics are pretty cool. I do miss Crazy Blind Date though. I met some awesome people from that, a bunch of hilarious theater majors. The best hotel they sent us to was the Hudson Hotel, they had pictures of cows with hats and we just couldn’t stop laughing.

  13. OldLaura says:

    Too funny! Very well written! I so enjoyed reading that. It’s hysterical. First off, I don’t go exclusively by whether I’m a good match for someone or not. It says I am, I look for the reason why. If it says I’m not I look for the reason why. Then I make my own decision. Anyway all you whiners, this is free, suck it up.

  14. Just Me says:

    OK Cupid’s latest experiment:

    Create a funny blog about something terrible you just did and see how many people “like” and tweet about it. Share the data and add some cute pictures.That’s sure to soften the blow from the negative media attention and angry members.

    I’m not saying this is a huge deal but if you are actually happy with this blog you are one of the members who fell for the “we better bring back our blog and nip this in the bud before our members start dropping like flies” meeting OK Cupid must have had this week.

  15. Okcupid guinea pig says:

    You dumb fucks at Okcupid, it’s not cool to do experiments on people without their knowledge. Sign me up if there is a class action lawsuit being taken against you.

  16. Anon says:

    This article has prompted me to delete my account. The tone disgusts me. I expect apologies not gloating on this issue.

  17. Denise says:

    Well I hate you done this!!!! Leaving the site!!! Who knows if I’m even talking to a real person or not. Don’t trust you!!!!!

  18. robbt says:

    Youre welcome.

  19. VespaChef says:

    Does it say anything about volunteering to be experimented on in the user agreement on OKcupid? I didn’t see anything in it about experimentation when I agreed to it. I don’t think that any of the experiments helped me at all. I use your site and check it on a regular basis a few times a week. I rarely get messages and haven’t been on a date in over 9 months. I haven’t “gotten lucky” in about double that time and my desert is getting a little dry. I send messages and respond to over 90% of them that I receive. Ladies usually send me an email or two back and fourth then they stop. I’m not being rude or forward in messages I’m not sure what the problem is. If you’re going to experiment could you at least help me out a little bit, it seems like you have skewed match percentages before, I could use a little help and some company. Honestly I am lonely and it sucks. Could you help me please.

  20. J says:

    Wow. So basically you’re saying your site has been worthless because you’re been screwing with people this whole time.

    Who knows how many missed opportunities I’ve had, along with everyone else.

    This is completely unethical. I didn’t sign up for this. If OkCupid was a country they would be in violation of the Geneva Conventions. Where is my compensation for being an unwilling guinea pig?

  21. Mark says:

    Hey, can I have some of your employees’ info? I just want to conduct an experiment or two.
    Knee-jerk reaction : Of course you won’t.
    DON’T DO IT TO US!

  22. Lois Oda says:

    I liked your site better than all the rest until you started the quick match swiping thing
    It has no names one a number for reference –
    I have reported a scammer 3 times and he is back on there again but unless I go through 100’s of face on the regular match section to find him I can’t report him ( I have reported him 4 times already

    You do us all a disservice by not putting a name or at least a number on each person who is on that – I also saw 3 guys whose photo was of their penis only on it.

    I know you did that so we would buy the upgraded membership instead of only the free one BUT – it makes your site much lower ranking compared to other s now

  23. Phillip says:

    I think your site is pretty good . I met a women on her a few years ago and we really hit it off and dated for three yrs . Found out she was a cheater . I’ve met other woman on here that are after money and I’ve turned them in . But for the most part I really like the site accept not being able to see who likes me with out having to pay . I know nothing is 100% free in the world though . Keep up the good work .
    Thanks Phillip

  24. Just my opinion says:

    This may explain why this senior is sent 20-somethings as good matches, even when they are clearly out of my range for preferences. Also, I believe the Europeans are clearly outside the geographic range I specified. And just what IS an enemy percentage? I am in a hard to match category for several reasons (age, gender, search criteria, etc.). That’s the only reason I would resort to trying to make friends this way. I would appreciate honesty at this stage of the game. It’s hard enough to be old and lonely. I would prefer not to be the mouse to someone’s “cat.” You should ask for permission and those adventurous enough to enjoy this could participate without putting the rest of us through an extra layer of duress. And yeah the microscopic warning in the pages we all sign off on isn’t good enough to meet that bar.

  25. Ian says:

    I’ve been using OK for some years – not very frequently I must admit. But I used to love the ‘weird shit’ things like the ‘stink foot’ experiment. That used to bring me back to find out what silliness you guys had been up to. As to dating algorithms … they’re probably as much fun as the stink-foot ‘speriment! ;) Keep up the silliness.

  26. Phillip says:

    I also read their profiles and their questions and don’t look at the percentage . I do look at the pics though . Looks aren’t everything but they do help

  27. Nameless says:

    Both me and my twin are on this site, he lives in CT and I in ME. He’s traveled a bunch, is arrogant, kind of immature, and has nicer pictures. I’m basically the opposite and more fun to be around.

    Woman actually contact him, reply to his messages, and he has frequent dates.

    I’m closing in on 40 messaged users between OKC and POF, I’ve only had one woman even reply to a message, have had zero “conversations”, and have never been messaged first regardless of of match % or how much I have in common with them via their profile/questions…

    Perhaps it could be that Maine is just a crappy place for online dating, but I’m banking on the better pictures/shallow people factor. Particularly so after he let me log in to his account and I got to see what he was doing “differently”.

  28. Oonagirl says:

    Your robot is so off that I ignore what you or it thinks about who might be a match for me. I ignore it. So say what you want, study what you want. When it comes down to it, I use my criteria like writing skills, attitude toward my questions, things only I can decide. It doesn’t matter what you or your robot thinks.

  29. Gc says:

    In the next blog post will you talk about the “hot list?” I have friends who use the site who aren’t “hot” and get shown completely different matches in the same area. What if I want to see profiles or date someone who’s not hot listed?

  30. Drumsong4u says:

    Maybe I’m the only one who actually reads the Q & A, but I do! The match % is not my only measure. But, I notice that most every man just looks at the pictures. I’m looking for compatibility first, so I read…

    OKC is all very interesting. In general, I like it better than any other dating site because I CAN find out more about prospective dates.

    By the way, 4 email conversations does not a relationship make.

    I have to “rate” one of your features. Quickmatch…Rating people just from their pictures and a sliver of their profile is lame. Why not let us dig deeper before we decide? And, why can’t we actually email those people?

  31. Mina says:

    The problem with messing with the match percentages is that you are opening people up to a whole lot of harassment/danger if their social values are different.

  32. debra says:

    I dont feel u should charge to view anyone or to get to know anyone..Im not sure ur site works..It seems u match me up with some ugly fat guys n thats not rite at all…

  33. BelmontReportFail says:

    I have two major problems with what you have done:

    1. It is patently unethical by any common ethical code of research, including the one that guides research rules here in the U.S., the Belmont Report:

    http://www.hhs.gov/ohrp/humansubjects/guidance/belmont.html

    The Belmont Report clearly indicates that conducting research on people without their consent shows a profound lack of respect for their autonomy as individuals. I will buy that to a certain extent all websites will use and analyze the data they hold to improve the quality of their product, but this went way beyond that to creating and testing interventions that willfully deceived people. That strays way outside of anything that could reasonably be considered normal process evaluation for a dating website.

    2. I see no discussion of the basis or methodology used to conduct these experiments. Were thee studies appropriately powered? Are the results statistically significant? Without this relevant information, the results are pseudo-science at best.

    So in summary, you violated people’s fundamental right to decide whether they want to take part in research, and you did it to conduct some lousy science.

  34. jay says:

    Thanks! The girl who “liked” me leftyour site before I could talk to her thaks to your PR blunder!

  35. Greg Griswold says:

    I’ve bitched for years, to deaf ears, that your %enemy rating is totally backwards. . . why would I want to see who I am a perfect enemy to? It needs to show persons who are the least % enemy to be of any possible value, or don’t bother having it as an option.

    I’ve found that those persons to whom I match higher than 95% are for the most part, ones that I thought I could easily fall in love with, if there was even the slightest physical attraction, such that I am at this point absolutely convinced that your algorithm is nothing short of simply just fantastic for predicting who I would want to consider becoming more attached to.

    Had one false match, less than 30% who I wrote to anyways, and she too thought we had much in common, later, received a message from you guys that there had been a glitch, and instead, we were matched higher than 94% – confirming what I had so strongly first suspected.

    Please keep these insights coming, they are fascinating to digest and priceless in value.

    Greg

  36. Pat Fobair says:

    Thanks for your blog…

    Tell us more as new info. develops…

    Pat

  37. s. seaman says:

    Speaking as an actual social sciences researcher (I have to go through an IRB process and everything!), you can all go Fuck yourselves. Not only are you conducting unethical research in the absence of informed consent, but it’s BAD research. Well, whoop-de-doo, you’ve discovered the halo effect! No shit! When exactly will you be publishing this in a peer reviewed journal so we can double-check your methodology? Oh right never. Take your sanctimonious, defensive smugness and shove it. Needless to say I’m deleting my account.
    PS- the stinky feet project back in the day was better run, and better reported.

  38. Blair says:

    OKC keeps matching me with geminis, and I’m a gemini. I suppose we’re more detail oriented and like lots of options and things, buuuuut, yeah, the system isn’t perfect.

    Not saying that its bad either. OKC is worlds better than many dating sites.

  39. VadimDiver says:

    Too much mathematics and statistics ! Nobody can measure feelings and relationships.I received messages,when a Lady wrote:”We match in 92%”. Stupidity ! The bestway is meeting in person and talking, walking,have common activity.

  40. Joe says:

    As a research scientist and a former member of an IRB as well as an OKcupid member, I’m rather appalled by these studies. Informed consent is a foundation for modern ethical research and that was clearly not present. This was either unethical research or false advertising depending on whether you view it as being science or a corporate service. I already go into profiles with a very big dose of healthy skepticism but hearing that other users may have been intentionally misled about my own posting is very disturbing and is causing me to rethink my use of this service. I am not a lab rat, except when I give my approval for it and know what I am getting into.

  41. jony says:

    you should really get a Statistics Professional if you do these experiments, because I don’t have more then highschool math, and already I saw some flaws in your conclusions of the data.
    if you’re going to get this data, better get a profesional to get it right, then to believe you got it right when in fact you’re wrong, it would probably even be better to not do anything and know that you don’t know, then to do this and believe in something that is wrong.

  42. CCHGN01 says:

    Well, you haven’t done anything that sociologists haven’t already done. Human nature is human nature.

    We, as human animals, use our senses to find a mate. Our first sense is sight. Our mate has to look appealing to us, first and foremost. I work out and maintain my body weight. I don’t want to get involved with someone who obviously doesn’t, because there’s other women who do, that are just as nice and can cook and likes to shop for antiques, etc. Then we use smell and sound to make decisions about a mate. I can talk to a woman one time on the phone and tell if I want to continue. Alot of it is the sound of her vice. It has to be appealing to me. Then of course, how she handles herself. Then by the first date, how she smells and makes herself up. I like confident, not too flashy, subtle sexiness, not slutty. definitely appropriate for the date.

    So first, I look at the pics, then I look at the answers to the questions.

    Afa the profiles, What I’ve found is that most folks don’t know themselves enough to write a good profile…lol I’ve found that women write who they’d LIKE to be, not who they are.

  43. annoyed says:

    Dear OKC –

    Experimenting like this for research purposes without consent is illegal or at the very least immoral.

    It’s one thing to evaluate how people us the service – that’s standard metrics and good practice. But to knowingly manipulate people without their consent is wrong.

    Like Facebook you can join the ranks of sites I’ve left for behaving so poorly.

  44. Peter says:

    Hmmm, I seem to buck the trend as usual. When I see an empty profile, that is without text, I immediately move on. Never mind how pretty the picture is. Not interested in inarticulate people.

    If, no when (and why ferchrissakes) OKC sends me a ‘match’ with a sub 60% ‘match rate’ I move on, in fact practically anything below 80% gets ignored immediately.

    Whatever is > 90% gets scrutinized very carefully for what this person has to say about themselves. And yes, that’s where the picture comes into place also – someone with a funny, thoughtful, mischievous picture is much more likely to make me want to try to talk to this person than somebody who looks like a stone-faced gargoyle.

    Lastly, if I like a profile, I attempt to take it to ‘proper email’ almost immediately. I do not like to communicate via editor panels on a website. If a person is not prepared to join a ‘proper email’ exchange, I assume they have trust issues, and my interest and reciprocal trust go down several notches immediately —- what I am trying to say here is that your metric of checking how many messages are exchanged between two people on your site may be way way off, if others behave the way I do in this respect.

  45. Jim Sprake says:

    That people are ‘outraged’, and the comparisons to FB fly, is utterly predictable. However, I can’t see how anyone was harmed. Or, more to the point, how such a thing could be quantifyed. Explaining affairs of the heart through data is a real shot in the dark, and it seems that even the most simple person can tell you there is no formula. Suppose your numbers are poor, but looking at photo three twists you just so, wouldn’t you think…opposites attract, and post her anyway. Surely whether or not she likes documentarys, has no bearing on whether you like her, or not. Now if you LOVE some one, it truly is ‘warts and all’. OKC complained that I marked too many questions irrelevant, and I was no help because I know… If the sex is good, I am willing to overlook just about anything.(except punishing a child for masterbating, that’s just sick) If I fall in love for real, and we have devastating sex, who knows, I got punished at least once as a kid when I didn’t deserve it and it didn’t kill me…you see where I am going and I feel several of you rising indignantly. Ideology, is a trickbag and had you been in jaws, the shark would have killed you instead of you, it. Do whatever you want OKC, won’t say it makes no difference, just don’t see that I can blame or thank you, for what it is that I do.

  46. Terran says:

    Social studies say a lot of what we really think. Shows us there may be a hint of hope when using this site. Thumbs up OKC

  47. TeaDidikai says:

    It’s interesting research, but there isn’t enough of a control sample.
    Let’s put it this way: OKC is great for several reasons including its
    flexibility in intention and its global diversity; however, that does mean the sample may not relate directly to compatibility.

    Since casual sex is a valid option on this site (compared to other sites which filter out people looking for casual sex, members of the LGBTQ community, non-monogamous peoples etc) it is a possibility that data clusters that surround different demigraphics would reflect different weighted values in matching.

    For example, if OKC filters someone as high compatibility based on some issues and not others, but the difference between how many questions asked is too great, it don’t account for “deal breaker” disagreements in some respects.

    That would be a neat function that would address the multicultural dating pool OKC provides- a button that goes beyond “extremely important” to actually disqualify matches based on hard limits.

  48. BelmontReportFail says:

    I have two major problems with what you have done:

    1. It is patently unethical by any common ethical code of research, including the one that guides research rules here in the U.S., the Belmont Report.

    The Belmont Report clearly indicates that conducting research on people without their consent shows a profound lack of respect for their autonomy as individuals. I will buy that to a certain extent all websites will use and analyze the data they hold to improve the quality of their product, but this went way beyond that to creating and testing interventions that willfully deceived people. That strays way outside of anything that could reasonably be considered normal process evaluation for a dating website.

    2. I see no discussion of the basis or methodology used to conduct these experiments. Were thee studies appropriately powered? Are the results statistically significant? Without this relevant information, the results are pseudo-science at best.

    So in summary, you violated people’s fundamental right to decide whether they want to take part in research, and you did it to conduct some lousy science.

  49. Richard Long says:

    You said it all…’ You dont know what you’re doing, and neither does any other site’. Mix ‘em up, mix ‘em all up! Opposites attract…sometimes. Personally, I’d like to view as many different people as possible. Just keep doing what youre doing… its fine!

  50. firefly416 says:

    I heard people discussing this today on the news and they were all saying how terrible what you did was. I actually like your site the best and that’s comparing it to a few free and a couple paid sites. To me what makes OKC better is the questions. I’m one of the question addicts and have answered 3,000 of them. It does annoy me that you’ve lied about the percentage match which is one of the things I consider when deciding to write to someone. I will never respond or write to anyone without a photo. I’ve only dated one person I met here. I love him, he doesn’t feel the same way about me. I’m still looking for someone who does.