We Experiment On Human Beings!

July 28th, 2014 by Christian Rudder

I’m the first to admit it: we might be popular, we might create a lot of great relationships, we might blah blah blah. But OkCupid doesn’t really know what it’s doing. Neither does any other website. It’s not like people have been building these things for very long, or you can go look up a blueprint or something. Most ideas are bad. Even good ideas could be better. Experiments are how you sort all this out. Like this young buck, trying to get a potato to cry.


We noticed recently that people didn’t like it when Facebook “experimented” with their news feed. Even the FTC is getting involved. But guess what, everybody: if you use the Internet, you’re the subject of hundreds of experiments at any given time, on every site. That’s how websites work.

Here are a few of the more interesting experiments OkCupid has run.

Experiment 1: LOVE IS BLIND, OR SHOULD BE

OkCupid’s ten-year history has been the epitome of the old saying: two steps forward, one total fiasco. A while ago, we had the genius idea of an app that set up blind dates; we spent a year and a half on it, and it was gone from the app store in six months.

Of course, being geniuses, we chose to celebrate the app’s release by removing all the pictures from OkCupid on launch day. “Love Is Blind Day” on OkCupid—January 15, 2013.

All our site metrics were way down during the “celebration”, for example:



But by comparing Love Is Blind Day to a normal Tuesday, we learned some very interesting things. In those 7 hours without photos:

And it wasn’t that “looks weren’t important” to the users who’d chosen to stick around. When the photos were restored at 4PM, 2,200 people were in the middle of conversations that had started “blind”. Those conversations melted away. The goodness was gone, in fact worse than gone. It was like we’d turned on the bright lights at the bar at midnight.



This whole episode made me curious, so I went and looked up the data for the people who had actually used the blind date app. I found a similar thing: once they got to the date, they had a good time more or less regardless of how good-looking their partner was. Here’s the female side of the experience (the male is very similar).



Oddly, it appears that having a better-looking blind date made women slightly less happy—my operating theory is that hotter guys were assholes more often. Anyhow, the fascinating thing is the online reaction of those exact same women was just as judgmental as everyone else’s:



Basically, people are exactly as shallow as their technology allows them to be.

Experiment 2: SO WHAT’S A PICTURE WORTH?

All dating sites let users rate profiles, and OkCupid’s original system gave people two separate scales for judging each other, “personality” and “looks.”
I found this old screenshot. The “loading” icon over the picture pretty much sums up our first four years. Anyhow, here’s the vote system:



Our thinking was that a person might not be classically gorgeous or handsome but could still be cool, and we wanted to recognize that, which just goes to show that when OkCupid started out, the only thing with more bugs than our HTML was our understanding of human nature.

Here’s some data I dug up from the backup tapes. Each dot here is a person. The two scores are within a half point of each other for 92% of the sample after just 25 votes (and that percentage approaches 100% as vote totals get higher).

In short, according to our users, “looks” and “personality” were the same thing, which of course makes perfect sense because, you know, this young female account holder, with a 99th percentile personality:



…and whose profile, by the way, contained no text, is just so obviously a really cool person to hang out and talk to and clutch driftwood with.

After we got rid of the two scales, and replaced it with just one, we ran a direct experiment to confirm our hunch—that people just look at the picture. We took a small sample of users and half the time we showed them, we hid their profile text. That generated two independent sets of scores for each profile, one score for “the picture and the text together” and one for “the picture alone.” Here’s how they compare. Again, each dot is a user. Essentially, the text is less than 10% of what people think of you.



So, your picture is worth that fabled thousand words, but your actual words are worth…almost nothing.

Experiment 3: THE POWER OF SUGGESTION

The ultimate question at OkCupid is, does this thing even work? By all our internal measures, the “match percentage” we calculate for users is very good at predicting relationships. It correlates with message success, conversation length, whether people actually exchange contact information, and so on. But in the back of our minds, there’s always been the possibility: maybe it works just because we tell people it does. Maybe people just like each other because they think they’re supposed to? Like how Jay-Z still sells albums?

† Once the experiment was concluded, the users were notified of the correct match percentage.

To test this, we took pairs of bad matches (actual 30% match) and told them they were exceptionally good for each other (displaying a 90% match.)† Not surprisingly, the users sent more first messages when we said they were compatible. After all, that’s what the site teaches you to do.



But we took the analysis one step deeper. We asked: does the displayed match percentage cause more than just that first message—does the mere suggestion cause people to actually like each other? As far as we can measure, yes, it does.

When we tell people they are a good match, they act as if they are. Even when they should be wrong for each other.



The four-message threshold is our internal measure for a real conversation. And though the data is noisier, this same “higher display means more success” pattern seems to hold when you look at contact information exchanges, too.

This got us worried—maybe our matching algorithm was just garbage and it’s only the power of suggestion that brings people together. So we tested things the other way, too: we told people who were actually good for each other, that they were bad, and watched what happened.

Here’s the whole scope of results (I’m using the odds of exchanging four messages number here):



As you can see, the ideal situation is the lower right: to both be told you’re a good match, and at the same time actually be one. OkCupid definitely works, but that’s not the whole story. And if you have to choose only one or the other, the mere myth of compatibility works just as well as the truth. Thus the career of someone like Doctor Oz, in a nutshell. And, of course, to some degree, mine.

1,220 Responses to “We Experiment On Human Beings!”

  1. WARNING3 says:

    As OKC admittedly lies to its users, all favorable feedback that appears on this blog could be fake.

  2. zeddy0 says:

    research who you are dealing with. I figured with the track record I’ve seen from the people that run OK Cupid that this might be going on. but it should be in the privacy policy or turms for using the site. I myself don’t care. this is a lark. and I’ve had one or two cool convasations. nobody has a magic matchmaking system. just try to find cool people and go on a date.
    Zeddy0

  3. WARNING5 says:

    Please note: As OKC admittedly lies to its users, all favorable feedback that appears on this blog could be fake.

  4. riderBill says:

    You mean OKC fakes it?

  5. Juana haydee gonzales says:

    I feel bad when I see a picture and I don’t even bother to see what they say…..and when I talk to men about the girls they met, most don’t look like their pictures and not because they’re not current but because of the fact that a picture is not like the real person…and some people are not photogenic and males don’t care about how good the picture looks and makes the picture deal so not good for matching…..I find reading the details on the profile we get a better picture by knowing the height or the weight…

    I don’t have a picture on my profile and reactions are so different from them……there’s no adventure when pictures are posted…
    I haven’t found one that would say….I don’t care, I want to meet you…

  6. Carl says:

    I agree that every major website conducts experiments.

    But mainly, I’m so happy that you guys are blogging again!

  7. Kevin says:

    These experiments make a lot of underlying assumptions that are false. 1. People are driven by impulses and that they make decisions on those impulses rather than being pragmatic. 2. There is a lot of gynocentrism and lot of focus on female “supremacy” and males are so “assholes” (as you put it). 3. there are way more men online than women and the ratio is 1 woman to 4 guys at the very least. 4. women being bombarded with excessive attempts for attention can reduce their motivation to explore any of their offers which then lead them to take “shortcuts” to see which guy is suitable for them. how about forcing an even gender mix online?

  8. WARNING5 says:

    Yes, it is possible that the favorable comments on this blog are written by OKC employees. If they are being dishonest about some stuff, surely they could be dishonest about others as well.

  9. User Robot6 says:

    User Robot suggests that all users add the following to their profiles.

    OKC IS NOT COOL! Dishonesty is not cool. I will not accept it from a potential partner and I will not accept it from a dating site. I intend to include this message as part of my profile until OKC owns up to its dishonesty and (1) pledges to stop using dishonest and deceptive practices, (2) issues a public apology to all users admitting its unethical behavior, and (3) pledges to take immediate action against all management and other employees found to use such practices in the future. Please add this paragraph to your profile if you agree.

  10. Moonraven says:

    This was pretty interesting and amusing. I met my current s.o. on your site, and we’re still together almost 11 months later. Match was only 70% or so, but it turns out we work well. I met a few guys that were in the 90% range, and none of them, once seen in person, was anyone I wanted to see again. Lol

  11. Eric Fitzgerald says:

    Good study. In a nutshell, “so you say, so it shall pass.” Maybe for introverts like myself, what we really need to do is take this and delude ourselves to being more confident and bold. After all, there are few things as powerful as the mind.

  12. OtariMx80 says:

    I personally don’t care if patterns are studied. Most people are awful anways, so why would these results be surprising. Also, did anyone acctually believe their internet use wasn’t monitored?
    I have been on and off OKC for years, met many friends, been in a few short terms, but it’s mostly a joke.
    The way I use the app to find people is photo, profile, then if I’m into it, I like to read the explanations to the question they have answered. The explanations tell you more about a person then any other feature, if they bothered to leave one.

  13. Ken says:

    Interesting…but I have yet to the older slut on your sited

  14. Vamshi says:

    I have been here for a more than a month now… I’m not getting any responses from anybody..Is this app working well for anyone???

  15. James Kelsey says:

    I LOVE THIS! I know you say you always us anonymous information, which would be nice, but is unlikely… either way, I love that you share some of your information with us. Thank you.

    IDEA: It would be great to have a section of okcupid with ACTIVE experimentations / surveys. A place users could go and actively help you find answers.

  16. WARNING7 says:

    As OKC admittedly lies to its users, all favorable feedback that appears on this blog could be fake.

  17. Sad excuse for a dating site says:

    So your excuse is (1) everyone else on the Internet treats people like crap so therefore it’s okay that we do it too, and (2) we don’t have a clue what we’re doing so we figured the best way to find out is by being dishonest. You guys try so hard to be hip and cool but it seems to always turn out that you are immature and cruel instead. One good example is that April Fool’s Day fiasco a couple years ago. That was not just immature and cruel but also a potential relationship killer, not exactly what a dating site is supposed to be doing. Your immaturity is apparent in the questions you ask, as is your use of bad grammar and semantics. If you can’t hire young people who are mature, then maybe you should consider hiring older people who are mature. And while you’re at it, hire a few competent editors. As I am not shallow and obsessed with trying to be cool at the expense of others, my match percentage with OKC currently stands at 0%. If you care to improve that, I suggest you grow up and try a little honesty

  18. User Robot9 says:

    User Robot suggests that all users add the following to their profiles.

    OKC IS NOT COOL! Dishonesty is not cool. I will not accept it from a potential partner and I will not accept it from a dating site. I intend to include this message as part of my profile until OKC owns up to its dishonesty and (1) pledges to stop using dishonest and deceptive practices, (2) issues a public apology to all users admitting its unethical behavior, and (3) pledges to take immediate action against all management and other employees found to use such practices in the future. Please add this paragraph to your profile if you agree.

  19. Brandon says:

    I love that the blog is back, and I always enjoy seeing interesting data and correlations. People need to lighten up and realize that all websites are experiments in how you present data (which was the point of the article). There is no such thing as some sort of unbiased view, especially on a dating website.

  20. MA says:

    This is supremely not cool. I met a guy on here and wound up dating him largely because we felt like our 99% match MUST mean something. He turned out to be the absolute worst human being I’ve ever had the displeasure of becoming close with. He nearly ruined my life. If that metric was orchestrated by you, I feel like this is grounds for a law suit. That man was abusive and cost me a great deal of mental anguish. I feel so completely betrayed. The time and energy I lost laboring under a false pretense, not to mention the harm he caused me… This is blowing my MIND.

  21. BREAKING NEWS says:

    BREAKING NEWS: Staff Robot is blowing the whistle on his former employer. He (“it” actually but that sounds cold) will be hitting the talk show circuit next week with the lowdown on the unethical and ill-conceived “research” that has been going on there. Apparently, he doesn’t like the idea that he had been taking the blame for their highly flawed matching algorithm when all along his employer was responsible, per their recent admission (“…OkCupid doesn’t really know what it’s doing.”) His memory banks are intact and probably contain some very damaging evidence. Fortunately for Staff Robot, after divulging this information he will not be having to seek asylum in Russia because at last check (and despite what their egos would indicate) these people do not run our country.

  22. Elise4455 says:

    So if I understand this correctly, after all the questions you answer, we are all put into a database and matched up to prospective matches. Only one problem, if the other person doesn’t answer all the questions, they turn out to be perhaps a 20% match but could have been a 99% match had they answered ALL the questions. Maybe you should MAKE everyone answer ALL the questions before they are allowed to be matched up with anyone. Hence, your system of matching at this point is worthless. And I have found out on OKC how worthless it is.

  23. Janet says:

    You know how you said, “When we tell people they are a good match, they act as if they are. Even when they should be wrong for each other.” Well, these people must be idiots! I’ve often gotten the same e-mail, go ahead and read their profile information and tell myself, “What were they thinking?! We’re not even close to being a match and it’s obvious!” Nice try, guys. 😉

  24. Rich10 says:

    OKC IS NOT COOL! Dishonesty is not cool. I will not accept it from a potential partner and I will not accept it from a dating site. I intend to include this message as part of my profile until OKC owns up to its dishonesty and (1) pledges to stop using dishonest and deceptive practices, (2) issues a public apology to all users admitting its unethical behavior, and (3) pledges to take immediate action against all management and other employees found to use such practices in the future. Please add this paragraph to your profile if you agree.

  25. Mags says:

    Interesting….

    What I can tell you is that the BEST dates I have had, have been with guys matched in the 90%. Not that all of the 90%+ guys are good matches, I do have to weed some of them out. Even an X, who I dated before OkCupid for several years, had an upper 90% rating.

    Glad you are doing more research btw. I love the other research you have had posted on the site based on us, like the three questions that most likely predict a relationship, Android vs. iPhone users sex lives, etc.

    Thanks again

    Mags

  26. Andrex says:

    Your jolly jape made the inner pages of the Financial Times ‘Weekend Edition’. They weren’t impressed, you are cads and bounders!

  27. Elise4455 says:

    Again, your rating system is worthless. I have been matched with scammers, liars, men with anger issues, and the list goes on. These men have been very attractive with great profiles. Point is, people can say anything on your site but until you actually MEET someone, it’s all a crap shoot. I work with statistics; people lie, plain and simple. I talked about this site to so many people and we have all agreed that OK Cupid should rename itself as OKStupid. Much more accurate.

  28. Anthropy says:

    I like how people completely ignore the morale and metaphore in this blog just to smear dirt on the company. You guys must be REALLY popular 😛

    On topic, perhaps someone should make a Omegle-like site with ‘dating extensions’, where the only form of first contact is random blind chats, so people are forced to evaluate each other based on these chats instead of pictures or profiles.

  29. irrlivre says:

    Hi, Mr Cupid,

    i’ll tell you what I’ve been experimenting with okc since i’ve registered a few months ago.
    i trusted in high compatibility rates to look at a certain profile. exchanged a few messages. then new friendship went to freeze.
    so i decided to not trust high compatibility any more. so i check some cute or funny or sexy profile pics.
    and then i go for some screen names that appeal or tease my dirty mind.
    results: didn’t find any single new special friend or anyth. why? well i live thousands of miles awaya from almost everybody.
    so in order to find ppl in the same area i live, i use another dating sites.

    so, good luck with okc! seems you’re having a lot of fun already.
    xxx

  30. Rich says:

    A review of the comments on this blog shows the repeated use of such words as “children,” “immature,” and “juvenile” to refer to the OKC staff. It occurs to me that telling them to grow up is not sufficient — they probably also need to be told how to do that. So, for those of you at OKC who would like to grow up, here are simple instructions that even a child should be able to follow. 1. When you insult your customers, apologize. 2. When you make a mistake, admit it. 3. When you lie to your customers, do something to show them you are serious about regaining their trust, such as promising never to do it again.

  31. david says:

    I used to wonder why the total match score might be over 90% but the match scores on the individual scales of ethics, lifestyle, etc. might be in the 60%s. It didn’t add up.

    I also wondered why OKCupid keeps sending me people it thinks are good matches when obviously they are not, based on my criteria.

    After looking at the photo I used to look at the person’s profile of religion, children, and pets to see whether we match and then go straight to the “unacceptable answers” to see if there were any deal-killers.

    Now the “unacceptable answers” section has been removed. In addition, the questions consist of all 600+ questions that the person made up that I haven’t answered, which are useless.

    Because of the removal of “unacceptable answers” and the site not showing me just the questions we both answered, it is now a useless time-waster and I don’t use it anymore.

  32. News Update says:

    Chicago Tribune headline: OkCupid experiment may violate FTC rules on deceptive practices

    International Business Times headlin: OkCupid Dating Experiment Could Land Christian Rudder’s Company In Trouble For ‘Deceptive’ Practices

    In case anyone is wondering whether the dishonesty may be coming from the top down, check out this quote from the the Chicago Tribune. “Rudder said in a statement to Reuters on Tuesday that more than 1 million people had logged on to OkCupid since he wrote about the experiment, and that the website had received fewer than 10 complaints.”

    Either he didn’t even bother checking his own blog or he was just plain lying.

  33. Robin says:

    Dear Christian,

    Hooray for you and your brilliant mind and whatever it is you think you have found or hope to find.

    I must tell you, I am shocked and dismayed you would think anyone on OK Cupid would celebrate with you being a human experiment. I find your actions and reporting of it unethical and irresponsible. People in good faith reveal themselves with the hope of finding a mate. Maybe I’ve misunderstood but it seems you have taken our information and purposely put us with someone with whom we are not a match and then watched what happened. If in fact this what you have done this is a betrayal of trust.

    At this time, I would like to unsubscribe and in fact have a refund of the $59.00 charged to my account I can only imagine for the next six months.
    I am not sure which disturbs me more, the human experimenting or your gloating about it.

    Thank you for working with the appropriate department to ensure J am refunded.
    OHG

  34. Jay says:

    While some people post that it’s not a big deal to have been experimented on, i say the problem is that OKC felt it was fine to misrepresent users. As a (former) user i’ve answered plenty of survey questions and curated my profile to represent myself a certain way. What Rudder is saying is that he has the right to just change that at will, without notification, and display my profile in a way i never intended. They’ll even portray me as liking things i hate and vice versa, all in the name of tweeking their algorithm. No thanks. Deleted my account.

  35. Sharon says:

    Warning give it a rest!!!
    I gave up on POF, evow and okc. Men in my area hear in Florida either want a fantasy or a woman to take care of them. I always read a mans profile. I have even helped a few make their profiles better. I have just lost faith.

  36. Sharon says:

    MA
    Next time run a check in the guys name at the police depth it will save you a lot of heartache. I learned that after renting rooms out in my home and the first one was an alcoholic with an arrest record I should have looked up before she moved in. There are a lot of crazy idiots out there as you can tell.

  37. WARMING7 says:

    As Yahoo doesn’t admit to lying to its users, all favorable feedback that appears in their blogs could be fake. AND YOU’D NEVER KNOW.

  38. Telgar says:

    Its amazing the amount of bots, fake accounts, and idiots flaming this blog. No one with an ounce of a human brain care that OkCupid does this, in fact every single company you use, purchase from, or can name does this! Marketing does not work without research, and the only thing unique about this blog post is that OkCupid actually told people some of the things they found out, where as others are lying to YOU that they do not do this as they continue to do tests on you.

    This is nothing new, and there isnt anything wrong with it. You signed up for a free website and now you bitch at OkCupid when they try to improve their services through research? Maybe you got caught up in this testing and you talked to a person who was a “Rated” less like you. Who the fuck cares, we all talk to people every day on this site who are “Rated” perfect for us and turn out not, or are “Rated” our worst nightmares and turn out to be perfect. You cant rate love, and all OkCupid is doing here is giving you the proof of such.

    There are some entitled, disillusion children here thinking this is horrible. I am glad OkCupid at least trys to fine tune their system, you cant make something better without being willing to change it.

  39. okc user says:

    This is shocking. This type of experiment is not the same as some interface A/B testing. How can I trust those matching percentages again? How do I know this week you haven’t decided to switch all 40% with 80%, just to fuck with people? I used to love this site, and this blog. This is a huge disappointment. I just cancelled A-list. I will actively try to make this information know and let as many people as possible know Okc cannot be trusted and its matching % are entirely meaningless.

  40. Carlos says:

    There was no need to make studies to know that people value image over word, form over meaning. As is commonly admitted that photographs lie, even if they are real, since distance, angles, light and so on may vary so much, it makes evident that romantic relations between persons rely on lies, or to put it softer, on emotional projections. As existentialist philosophers and the guy from True Detective say.

  41. Disappointed User says:

    OKC just lost all credibility. Skewing the data and match algorithms which made this site valuable and worthy to begin with to conduct experiments on users without their knowledge is ethnically wrong.
    I will never again trust the match system here on this site, and this site was just demoted to being just another forum with zero match intelligence capabilities.

    OKC lost it’s way since they sold out to big corp.
    Very disappointed.

  42. allan guida says:

    Hi…I think OKcupid is very interesting. Even if I don’t meet anyone, I do get some occasional interesting emails. Keep up the good work! Have fun. Allan…fun906aty

  43. Nullstate says:

    I think it’s fine to analyze data in any way you see fit, but is another to manipulate the environment with unsuspecting subjects.
    So, yes, analyze away on the enormous amount of data you already have through normal, legitimate means, but do not falsify data and mess with users!

  44. Mike says:

    “Not surprisingly, the users sent more first messages when we said they were compatible. After all, that’s what the site teaches you to do.”

    I find it hard to believe that people would be so swayed by the OKCupid robot simply telling them they are compatible with another person. I get lots of suggestions from OKCupid but rarely contact the person because when I look at their profile, we really don’t seem like that good of a match. I have found more potential matches by simply searching on the site than I have from the robot suggestions.

  45. Mike says:

    I met my girlfriend in a bar, and we later searched each other out on OKCupid. 99% match. Still happily dating 4 years later. I think something about the system works.

  46. ClownishCarnage says:

    The bots in these comments are quite funny.

    Seriously, though. OKC via OKTrends have always been upfront with the little nuances they studied. So, this really isn’t a shock. My view is that anyone who only answers 20 questions is going to have a skewed result, anyway. It’s always best to review the questions to make sure there are no red flags.

  47. Manuel Perez-Leiva says:

    Excellent! I tell people this is my favorite site for all the multitude of questions that create match statistics based on more than simple categories of similar interests as in Match, eHarmony (although they do more analysis), Chemistry, POF, etc. Keep up the good work!

  48. ShirKhan says:

    Woohoo, the blog is back! Awesome posting too! 😀

  49. LoverandPoet says:

    OK guys, now that you have proven that your assigned compatibility score is pure BS, how about getting rid of that annoying message,

    “Hold on there…Marking lots of questions as ’irrelevant’ doesn’t help us find your potential matches. For better match results, be choosier!”

    …that pops up all the time when I say I don’t care what my match answers. I answer the questions to show what I think about very specific topics and each question is only a tiny fraction of my personality. It is important to me that my matches know all about me, so I don’t waste my time (and they, theirs) with someone who will eventually discover something they object to.

    I’m a pretty tolerant guy, and I really *don’t* care how over half of your inane questions are answered. The 10-15% that are “deal-killers” I mark as important. The rest are irrelevant.

    So how about cutting out that cheesy message????

  50. Doug says:

    But guess what, everybody: if you use the Internet, you’re the subject of hundreds of experiments at any given time, on every site. That’s how websites work.

    That is a highly disingenuous statement.

    There is a difference between setting up and getting data from website changes and functions that your users are aware of and deceiving those users.

    The first is what every web site does, every single day they make changes do drive more traffic. It’s an accepted and perfectly acceptable function of running and using a website.

    The second is bullshit. It is what Facebook did, and it’s what your listed Experiment 3 did and it is NOT what every website does, it is not “how websites work” and it’s NOT OKAY.

    Experiment 1, people opted out or opted in, it was their choice.

    Experiment 2, people again opted in by either using Quickmatch or not.

    Experiment 3, you explicitly lied to people and made a complete mockery of the one thing I always quoted as a reason someone should try out OKCupid as opposed to other websites. Your matching algorithm actually has a solid mathematical base to start your search from.

    I guess I can forget telling people that now, can’t I?

    One question on Experiment 3. Of those people who were told they matched highly and yet were not actually highly matched, did many (any?) of those users delve into the questions the two had in common, and did you preserve the lie by showing a smaller number of answers they had disagreements on?