We Experiment On Human Beings!

July 28th, 2014 by Christian Rudder

I’m the first to admit it: we might be popular, we might create a lot of great relationships, we might blah blah blah. But OkCupid doesn’t really know what it’s doing. Neither does any other website. It’s not like people have been building these things for very long, or you can go look up a blueprint or something. Most ideas are bad. Even good ideas could be better. Experiments are how you sort all this out. Like this young buck, trying to get a potato to cry.


We noticed recently that people didn’t like it when Facebook “experimented” with their news feed. Even the FTC is getting involved. But guess what, everybody: if you use the Internet, you’re the subject of hundreds of experiments at any given time, on every site. That’s how websites work.

Here are a few of the more interesting experiments OkCupid has run.

Experiment 1: LOVE IS BLIND, OR SHOULD BE

OkCupid’s ten-year history has been the epitome of the old saying: two steps forward, one total fiasco. A while ago, we had the genius idea of an app that set up blind dates; we spent a year and a half on it, and it was gone from the app store in six months.

Of course, being geniuses, we chose to celebrate the app’s release by removing all the pictures from OkCupid on launch day. “Love Is Blind Day” on OkCupid—January 15, 2013.

All our site metrics were way down during the “celebration”, for example:



But by comparing Love Is Blind Day to a normal Tuesday, we learned some very interesting things. In those 7 hours without photos:

And it wasn’t that “looks weren’t important” to the users who’d chosen to stick around. When the photos were restored at 4PM, 2,200 people were in the middle of conversations that had started “blind”. Those conversations melted away. The goodness was gone, in fact worse than gone. It was like we’d turned on the bright lights at the bar at midnight.



This whole episode made me curious, so I went and looked up the data for the people who had actually used the blind date app. I found a similar thing: once they got to the date, they had a good time more or less regardless of how good-looking their partner was. Here’s the female side of the experience (the male is very similar).



Oddly, it appears that having a better-looking blind date made women slightly less happy—my operating theory is that hotter guys were assholes more often. Anyhow, the fascinating thing is the online reaction of those exact same women was just as judgmental as everyone else’s:



Basically, people are exactly as shallow as their technology allows them to be.

Experiment 2: SO WHAT’S A PICTURE WORTH?

All dating sites let users rate profiles, and OkCupid’s original system gave people two separate scales for judging each other, “personality” and “looks.”
I found this old screenshot. The “loading” icon over the picture pretty much sums up our first four years. Anyhow, here’s the vote system:



Our thinking was that a person might not be classically gorgeous or handsome but could still be cool, and we wanted to recognize that, which just goes to show that when OkCupid started out, the only thing with more bugs than our HTML was our understanding of human nature.

Here’s some data I dug up from the backup tapes. Each dot here is a person. The two scores are within a half point of each other for 92% of the sample after just 25 votes (and that percentage approaches 100% as vote totals get higher).

In short, according to our users, “looks” and “personality” were the same thing, which of course makes perfect sense because, you know, this young female account holder, with a 99th percentile personality:



…and whose profile, by the way, contained no text, is just so obviously a really cool person to hang out and talk to and clutch driftwood with.

After we got rid of the two scales, and replaced it with just one, we ran a direct experiment to confirm our hunch—that people just look at the picture. We took a small sample of users and half the time we showed them, we hid their profile text. That generated two independent sets of scores for each profile, one score for “the picture and the text together” and one for “the picture alone.” Here’s how they compare. Again, each dot is a user. Essentially, the text is less than 10% of what people think of you.



So, your picture is worth that fabled thousand words, but your actual words are worth…almost nothing.

Experiment 3: THE POWER OF SUGGESTION

The ultimate question at OkCupid is, does this thing even work? By all our internal measures, the “match percentage” we calculate for users is very good at predicting relationships. It correlates with message success, conversation length, whether people actually exchange contact information, and so on. But in the back of our minds, there’s always been the possibility: maybe it works just because we tell people it does. Maybe people just like each other because they think they’re supposed to? Like how Jay-Z still sells albums?

† Once the experiment was concluded, the users were notified of the correct match percentage.

To test this, we took pairs of bad matches (actual 30% match) and told them they were exceptionally good for each other (displaying a 90% match.)† Not surprisingly, the users sent more first messages when we said they were compatible. After all, that’s what the site teaches you to do.



But we took the analysis one step deeper. We asked: does the displayed match percentage cause more than just that first message—does the mere suggestion cause people to actually like each other? As far as we can measure, yes, it does.

When we tell people they are a good match, they act as if they are. Even when they should be wrong for each other.



The four-message threshold is our internal measure for a real conversation. And though the data is noisier, this same “higher display means more success” pattern seems to hold when you look at contact information exchanges, too.

This got us worried—maybe our matching algorithm was just garbage and it’s only the power of suggestion that brings people together. So we tested things the other way, too: we told people who were actually good for each other, that they were bad, and watched what happened.

Here’s the whole scope of results (I’m using the odds of exchanging four messages number here):



As you can see, the ideal situation is the lower right: to both be told you’re a good match, and at the same time actually be one. OkCupid definitely works, but that’s not the whole story. And if you have to choose only one or the other, the mere myth of compatibility works just as well as the truth. Thus the career of someone like Doctor Oz, in a nutshell. And, of course, to some degree, mine.

1,220 Responses to “We Experiment On Human Beings!”

  1. Paint says:

    When I use match %’s to filter out potential sexist womanizing rapey bastards I get pretty upset that my efforts to utilize their matching algorithm to my advantage has been undermined, and my physical/psychological safety put at risk.

    My rant: http://youtu.be/yadL9RRC-k0

  2. FU OKC says:

    Playing with people’s lives? Wasting countless of thousands of people’s time? Giving false hope

    OKC I am ZERO respect for your business and yourselves.

    Well played you probably just knocked yourselves to the bottom of the totem pole. I am looking forward to Plenty of Fish beating you in subscribers and view count alike.

    You make me sick OKC.

  3. Tim says:

    I love this. You seem obviously very interested in the human condition, in human compatibility and in creating a good product. And yet you’re not afraid to show your warts. And judging by some of the reactions here, there’s plenty to fear. People don’t seem to like such candor, whereas I find it very refreshing. I like your site a lot – I like the percentage matches. And this article make me realize that I’d better get better looking, because my brilliant profile is not going to do shit for me! Needless to say, I have not found love on this site, or sex, or really much of anything. But your article was very entertaining.

  4. Ron says:

    Christian – Good that you are disclosing this stuff. I commend you for that. I think you should try to disclose this sort of testing up front, though, especially the type where you are purposefully manipulating or deceiving people. Informed consent was invented for a reason. Human beings are not to be used, even if YOU think it may benefit them as well as yourself. I think if you keep the level of experimentation low (like one in a thousand users per experiment) and are using it to improve the service, people will accept being subjects.

  5. Simone says:

    Conclusion: We all want to think we aren’t superficial, but we are. At the same time, we are optimistically invested in actually finding someone we’re compatible with and having a good time with them. Lastly, if you are an especially hot person, don’t even bother with writing a clever, witty, scintillating profile, because your looks will bring you success on OkCupid as they do everywhere else in life.

    There. Saved you some reading.

    http://greatnydatingexperiment.wordpress.com/2014/07/30/okcupid-stole-my-idea/

  6. Chris says:

    None of the research shows whether differences are significant or not. Start adding your p-values guys if you want people to take these findings with more credence. It’s good work though.

  7. eclecticjazzgirl says:

    Now I know why many times when you said I ‘matched’ with someone I was thinking NOT! Some of my own filtering is much better executed than yours because of your ‘black/white’ approach to your questions. I haven’t found the scale of ‘Personality’ at all very accurate when compared to who I am and the person I’ve taken time to message or talk to. I take the time to read someone’s profile and based on location, interests and much more whether I’ll send or respond to someone else. People can post any kind of photo or words, but until you have a phone and in-person contact you don’t really know if what they’ve posted is at all appealing or honest. Your ‘experiment’ in my experience is a failure. Most of the time when given a ‘Match’ either here or on POFish, it has been a small percentage of actual ‘attraction’ material. I’ll continue to respond to people who meet my own criteria and hopefully they too have the good sense and intelligence to see through your misguided experiments to contact me based on my profile and pictures for discovery.

  8. Rob says:

    Find a better way to use your time.

  9. Friend4youforever says:

    I remember this event. Shameful! LOL…..It was an unfortunate occurence that was corrected. A message was put out with an apology if I remember correctly. But yeah, so what, it is what is…or was what it was….deal with it…or go in a corner and cry.

    Using the information that was created AFTER the fact is fine with me. Its data, it was mined. Some interesting, but unrelevant facts came out.

    The algorithm is screwed up anyway! I don’t know how many times I review questions with only two answers and neither one is really important. Some are just plain stupid, but you get “graded” on these idiotic questions. Questions need to be set up that when they are created four answers are required….not less than three. I just don’t answer stupid questions, like…..”Would the world be a better place without stupid people?” Yeah Mr. Author….that includes your stupid idiotic ass! I answer true questions about relationships. Sexual and non-sexual. But I do something most people dont do…I read the answers and if they are two question answers and acceptable to me, then I write that question off as a low priority. I have emaild people with a match rate of 30-40% and gotten better feedback from them then some higher rated people. It’s about reading a profile, looking at questions and then formulatiing your message to someone in a polite, inquiring, but yet serious manner. Not “Hey baby…wanna fuck?” That will get you nowhere fast.

    Well, back to the true world…I reign over all!

  10. xxx says:

    Hey, don’t you also use bots? I was corresponding with a “match” that was sending me computer generated messages…it was obvious he was not a real person.

  11. Feet says:

    Honesty ftw! They never had to tell people and we all do stuff like that. Kudos for gaving balls okc.

  12. pat says:

    Actually, OkCupid does NOT work well. One of your fellow Math geniuses, Chris McKinlay, proved this by the numbers in an article with Wired magazine. You can find the article here

    http://www.wired.com/2014/01/how-to-hack-okcupid/all/.

    The experiments you have been performing demonstrate that while you may be mathematically gifted, you certainly lack the social skills needed to understand the subtleties of human interaction. World of Warcraft has a better match making record than OkCupid, and they are not even trying to make matches. The game psychologists are so adept at creating environments to reward social behavior that their imaginary worlds bring more people into dating relationships than your real world algorithms. Doesn’t that sting?

  13. Sharon Cassidy says:

    I would wonder if age would make a difference. I’m 70 and I find the important things are much different from what they were in my 50s and my 30s. I now know that political and religious incompatibilities are deal breakers–they would not have been so in my 30s, etc.

    p.s.: I’m new to OKCupid. Just joined 3 days ago. I was surprised to find that there were men who answered political and social questions exactly like I did. I’m excited to see what lies ahead on OKCupid.

  14. sheltontom says:

    It’s despicable you did this and then rationalize it by saying “that’s what everyone on the internet does”. You didn’t disclose this and that is misleading, you cannot get around it.

    You’ve lost a client and besmirched your name.

    “A mistake is always forgiven but any kind of dodge or untruth will always rankle”. Winston Churchill

  15. Robert Pike says:

    Glad you’re using science to test the reactions; as a scientist-teacher, I appreciate the effort!

    If I can give a few tips; you have a flaw in the “questions” section that needs to be fixed.

    When You Ask “how important is it that the other person agrees” if you check “not at all” then their answer shows up in red; in other words, even if we both agree, if one of us says it’s “not important” then we’re scored as not compatible (or our answer is in red?).

    But I’ve recommended it to several dates; keep up the good work; I think your questions give you a definite advantage over POF. (both of you have free email; thanks for that!).

    Rob

  16. Aaron says:

    Go fuck yourselves you arrogant fuckers

  17. Jogobu says:

    It is extremely disgusting, after the fact, for you to say that you were experimenting with my emotions after the fact, and you want to make it into a joke.
    I do research work myself and I find it intolerable that people with any professional ethics in research would go so low as to manipulate us that way

  18. Spirecat says:

    Soo… you thinking of maybe showing the pictures last or something? Or maybe replacing the texts boxes with something a little more direct and less SAT-dependent?

  19. TheWord says:

    I find it amusing so many people are upset about the candor of the okc experiment. This whole thing is an experiment. If you’re not paying attention, or in other words,!collecting your own data, you’re missing out.

    Let me help you. This is a numbers game not a percentage one. It gives us the opportunity to reach out to way more people then you would be able to otherwise. Of course you do better meeting people in person. Don’t take it personal, or blame okc for a low percentage of responses. One reason was stated by a woman is that security is a concern. Here are some other reasons; 1. Some girls are on here just for the attention. 2. Some girls get so many emails they would have to live to 100 to reply to all of them. If a good looking woman mentions sex or has provocative photos, she falls into both reasons for not replying. 3. Woman with a long list of how you should be and all of the things they don’t like about men they have met. These woman are poison anyway. Don’t bother trying to convince them you are different. Give that attention to women who have a positive attitude.

    The only woman you should email are ones who’s profiles are pleasant, succinct with a positive message. And have normal pictures where you can see the face and body. Look close, because some girls will mix recent photos with old ones. Sometimes it’s sutttle and sometimes it’s obvious.

    Okc is on the mark with the notion that people respond to being told they are a match. Whether it’s astrology, a friend’s suggestion, or the belief in fate, it’s been that way since the beginning. Meeting online is just about getting past the filters. You’re percentages of responses will always be low. Your percentages of success once you meet someone in person will be high if you’re normal, nice and look like your pictures or better. I have found if you send an email to a woman who is currently online you will increase your chances. Also, stay in your league!

    Don’t take this dating system so seriously. Be patient and have fun.

    My only complaint of okc is they leave profiles in the system long after the person was last online. On match they put that info right up front. I go throught the trouble of reading a profile on Quickmatch then find out she hasn’t been online since 2009. Come on okc, take them out of the rotation!

    Good luck to all

  20. Paul Harrison says:

    WE MISSED YOU!!!! PLEASE GIVE US MORE
    Your old blogs are fascinating, informative and funny.
    Please trickle out some more.

    How about something on the over 50s?
    What are their distinctive behaviors, preferences and lies?
    You must have enough data for that by now.

    How about the “your best face” experiment?
    I was once asked to choose between a beautiful woman and a terrier dog.

    Paul Harrison Ph D

  21. Grant says:

    Here I was doing my best to find someone through your site. another waste of time. I do appreciate your honesty about the experiment. like you said other site do the same without telling anyone. maybe its time for people to go back to meeting in person.

  22. Susan says:

    At first when I heard about this I thought it was despicable. I still think so, but to a lesser degree. I think it would have been fine if you’d somehow notified people in advance that they were rodents in an experiment, perhaps giving them a chance to opt out.

    But the interesting thing to me is you. How long did you wait between when the news broke of your experiments and when you penned your arrogant response? Traditional corporate damage control takes place very quickly and, importantly, is always conciliatory in tone. By contrast, your response was puzzlingly militant, delivering a swift and emphatic “fuck you” to anyone who felt put off by what you’d done. It seems to be incredibly bad business to have handled it this way, especially judging by some of the comments. Obviously if I’d ever considered paying for A List, I am no longer interested.

  23. Auntbessie says:

    I think you experimented on me cos everyone of my matches are not what I’m looking to find
    I don’t really rate the site as I haven’t found any one who I like

  24. TigerTiger says:

    OkCupid –

    I think it’s “cute” that you are attempting to present yourself as harmless by conducting psychological reaseaech on hour users. Perhaps it’s bc you justify your methodoly on the fact that us ‘rabbits’ that use your free dating service should be subjected to unknown tests. Luckily, I never took your site seriously. So even though you may thing you’ve collected data – you may be mislead by the variable that not everyone who is signed up on OkCupid actually cares!

  25. Chrisfs says:

    Calling me and the people who use your service and (and pay your bills) guinea pigs, does make me appreciate you any more. It’s really sort of demeaning and arrogant. I would choose a different graphic.

  26. ivladi says:

    Bullshit.

  27. Everett says:

    Haha how amusing. Haters goin’ hate, but I respect what you’re doing. Online dating is becoming so prolific, I think it’s a good thing to take such a scientific and statistical approach to better understand how humans date through the internet. I’d love to see the raw data behind your analyses and graphs.

  28. WI_Guy_in_KS says:

    Got my refund, cancelled my A-list membership, signed up for membership at POF.com. That’s what I think of your experiment that wasted my time and money.

  29. Silky says:

    And so, the question they asked themselves is: how important is that number, really? All they did was test the validity of their match scores. Do match scores make a difference? I think it’s entirely fair, and even welcomed that they are testing their system. You know, just because the rooster crows and the sun rises, doesn’t mean that the rooster makes the sun rise.

    And in the end, OK Cupid still works. Now maybe they can come up with an “I feel violated” score, and those who feel violated by this test can find each other.

  30. Juan says:

    When Do I start?:)

  31. Waiting says:

    I like the OkCupid site; it’s easy to use, cheap, and has some great features.

    Nice to see the blog back; what a way to start off with a bang. I’m still thinking that OkCupid is pulling our leg, though.

    Are there any legal ramifications if two people were mismatched, and something bad happened? I know that there are all those legal disclaimers, but if customers are mislead on purpose, and people can sue over anything, what then?

    The real downside to online dating in general is that people suck for the most part. One can write to a beautiful woman, and get rejected. It’s understandable…. but why are almost all of the allegedly beautiful women with apparently great lives on this site for a time span of literally years? Are there are no really handsome, successful men on the site at all that they can date?

    The answer is that “online dating” has become a hobby or pastime in and of itself. The majority of people on the site don’t really want to meet anybody, ever. They’re just there for the messaging, the affirmation, “the game” so to speak.

    Good site… but, the people that are on it suck for the most part.

  32. suckyourtoes says:

    Please remove the pornographic ads from your site!

  33. McKluskie says:

    This is interesting. Thanks for sharing. I think we should also consider that despite match percentage the success of a date depends on people moods an attitudes at any given moment. Even with the power of persuasion a first online date can be a crapshoot. I have had both bad and good dates with high match % and good dates with low match %. I suppose it also has to do with people’s approach to creating a profile. If they answer match questions sarcastically(which some people do) than they could miss out on someone cool and get some bad suggestion of a person from the site. At the end of the day we should continue to base our choices on pictures and conversation and not be afraid of the occasional awkward date.

  34. kate says:

    After looking at all the data you gathered I realized that what I thought about dating sites is true – people ignore whatever the site says is matching criteria because all anyone ever cares about is their own attraction to the persons picture. That’s not shallow it’s reality, no one falls in love with a personality or similarities in music taste. I never agreed with your matching ratings you gave to me because it’s based on the wrong things. If someone is attractive to me the interests are worked out later. That’s why E-Harmony is the worst site, we have to choose ourselves, hopefully wisely.

  35. tiffany says:

    you could have asked us. Even if you didn’t disclose what the experiment entailed, i”m sure thousands upon thousands of users would have consented. the fact that you didn’t makes me think there are ulterior motives, like maybe you’re selling us out.

  36. Michael says:

    I do not like being lied to. I am considering dropping OK cupid based on this experimentation.

  37. Hindira carias says:

    Hey, I have fished jersey on this site, one that almost took me for a ride a soldier in Libya who was looking for the love of his life and ended up wanted me to help him with a shipment and send him money 3,000 to pay it off. Search for real persons identity and found the man is a captain in Florida married. It hurt and has cause so much pain and depression in my life. I think the site should work on scamming to protect other individuals. Soldiers pics are the biggest scamming women for money or leaving them with weddings and never showing up but taking money or properties. Wow maybe the research should start on scamming.

  38. Jeannie Graham says:

    I personally never went by your compatibility rate. Anyone with even half a brain would be able to see if you put a high compatibility rate on a person that had little in commen or say for instants a good girl with good morals and you put a guy that looked like he just got out of prison as a highly rated match is really stupid on your part. I also go by my own judgment on all aspects of a person. Not those of okcupids. I will say this, if you thought by doing this to those that took your bate and it caused them to feel your site was not helping them or they got hurt by trusting your opinions. Than that just hurt your site because if they closed out of it do to what you had been doing. You might have lost clients that were helping you build your business as a dating site which eventually you could’ve started charging for it once it was popular enough like match.com. I’m a business owner and know there was a lot at stake for you to lose by messing with people’s heads instead of worrying about yourself and what you could do to really make yourself a trustworthy company that could make you and your employees money from this. You should of contacted me. I teach people how to run a successful business. You must not need the money. Apparently trying to play God was more important. Good luck and if you need help with damage control and building back what you could of had call me.

  39. native100 says:

    Your graphs and charts make it all scientifically reasonable almost as acceptable as the sites match percentages. Without the wow factor of your graphs and with realistic expectations I believe men and women are the same, we just have different ways of doing the same thing. We’re looking for two things, packaging and resources.Time after time this has proven true for me and others I know on this site.If all your research works and gets people together then it really doesn’t matter how you did it.
    Please save the paper for something more useful by putting it behind you.

  40. Bryant Caplan says:

    Playing with match percentage is a major problem. I want my upgraded membership money back!

    Bryant Caplan

  41. Tim Brown says:

    Can’t get in to my massages wants to delete

  42. Sean m says:

    Good article, very thorough data. I’m a psyche major so I read the whole thing like a journal article. Seems persuasion is very easy.

  43. Chow says:

    Whoa, what’s with all the hate? It’s just a game people.
    It’s interesting results too.
    Because they said you matched higher with someone for a second you get all mad? Suck it up.

  44. 999aboveaverage says:

    I have been a free member of POF for a long time and as you say I have suspected and known for a long time that they also do experiments, play games and I believe even sabotage relationships.
    POF has different rules for men and women in every part of their setup and it continues to get worse for men.
    On POF men are treated as though woman are shopping for a horse and will often get a man who may last just through the breeders cup before the guy gets away as quickly as possible.
    All websites are only as good as their users, those who are truely sincere and those who are sincerely dispicable. If you aren’t true in your search you will only get what you ask for, a lie.
    Keep experimenting, maybe someday the human race can really say we have evolved.
    Lets just hope it still takes two to tango in another 1,000+ years.

  45. Soggy says:

    I generally don’t pay much attention at all to percentages so this matters very little to me. I’m open to meeting whoever as long as their profile is interesting enough. You can’t really get to know someone until you see them IRL anyway so it’s not like I’m going to put much faith in OKC’s matching algorithm.

    I do find that most of the people I message are in the 70%-90% range anyway, but I’m not using those numbers to weed people out.

  46. Andrew says:

    This seems like very little inconvenience for everyone’s benefit. Imagine if they just kept guessing at how to improve the site.
    I voluntarily joined this site to find people. Experiments such as these are exactly the kind of thing I like to see in these kind of sights and validate my joining. It is not much of a bother to me as I am the chooser of my friends, OkCupid is simply the one who recommends. If OkCupid makes fake or virtual profiles, I still will not mind. Plus being on this site is totally voluntary and in my case free. I can consider fictitious results the same kind of inconvenience as ads.
    I’m glad the system is being improved.

  47. 8in baby...........Around. says:

    Funny but I had this hunch that these 5000 questions or whatever would be used to categorize people and in theory these however many people just took what I consider to be a “national survey.” It’s sorta like saying there are people out there that would say effff no I’m not taking a survey ever but then they’re either lonely, honey or whatever and then decide “efff it! I’ll answer 500 questions hehe.” I take online dating with a grain of salt. My personality shines through meaningful conversation with people so matching and 5000 questions for me is just the tip of the iceberg plus I have never gotten emotionally attached to anyone here even on a few dates I went on……….Unfortunately pictures even if not photo shopped can be a bit deceiving and some people still conceal certain areas so you really don’t know until you meet them in person. Was I disappointed no but a little frusterated but I think it comes with the territory. Did we still have a good date? Yes and I’m a gent not an asshole. One thing I’m doing is match or not I’m making sure I’m at least physically interested. The thing is that the world of dating has flipped and has been turned inside out. #1 we want it now and #2 some of us maybe don’t get out very much right now. Anyway seeing that I don’t do as much grocery shopping as the average Joe and prefer not to meet my long term at a bar, I go online.

  48. Pete says:

    We all joined expecting to get good results not false ones because of this I am deleting my account.

  49. Kit says:

    This explains why I have said so many times, “What the hell are you thinking, OKC?”

  50. Gustav says:

    Bring on the butt hurt. This is a non issue. I commend you for thinking outside the box. Your going to have the PC morons all wadded up over this tho. For no real reason either.