We Experiment On Human Beings!

July 28th, 2014 by Christian Rudder

I’m the first to admit it: we might be popular, we might create a lot of great relationships, we might blah blah blah. But OkCupid doesn’t really know what it’s doing. Neither does any other website. It’s not like people have been building these things for very long, or you can go look up a blueprint or something. Most ideas are bad. Even good ideas could be better. Experiments are how you sort all this out. Like this young buck, trying to get a potato to cry.


We noticed recently that people didn’t like it when Facebook “experimented” with their news feed. Even the FTC is getting involved. But guess what, everybody: if you use the Internet, you’re the subject of hundreds of experiments at any given time, on every site. That’s how websites work.

Here are a few of the more interesting experiments OkCupid has run.

Experiment 1: LOVE IS BLIND, OR SHOULD BE

OkCupid’s ten-year history has been the epitome of the old saying: two steps forward, one total fiasco. A while ago, we had the genius idea of an app that set up blind dates; we spent a year and a half on it, and it was gone from the app store in six months.

Of course, being geniuses, we chose to celebrate the app’s release by removing all the pictures from OkCupid on launch day. “Love Is Blind Day” on OkCupid—January 15, 2013.

All our site metrics were way down during the “celebration”, for example:



But by comparing Love Is Blind Day to a normal Tuesday, we learned some very interesting things. In those 7 hours without photos:

And it wasn’t that “looks weren’t important” to the users who’d chosen to stick around. When the photos were restored at 4PM, 2,200 people were in the middle of conversations that had started “blind”. Those conversations melted away. The goodness was gone, in fact worse than gone. It was like we’d turned on the bright lights at the bar at midnight.



This whole episode made me curious, so I went and looked up the data for the people who had actually used the blind date app. I found a similar thing: once they got to the date, they had a good time more or less regardless of how good-looking their partner was. Here’s the female side of the experience (the male is very similar).



Oddly, it appears that having a better-looking blind date made women slightly less happy—my operating theory is that hotter guys were assholes more often. Anyhow, the fascinating thing is the online reaction of those exact same women was just as judgmental as everyone else’s:



Basically, people are exactly as shallow as their technology allows them to be.

Experiment 2: SO WHAT’S A PICTURE WORTH?

All dating sites let users rate profiles, and OkCupid’s original system gave people two separate scales for judging each other, “personality” and “looks.”
I found this old screenshot. The “loading” icon over the picture pretty much sums up our first four years. Anyhow, here’s the vote system:



Our thinking was that a person might not be classically gorgeous or handsome but could still be cool, and we wanted to recognize that, which just goes to show that when OkCupid started out, the only thing with more bugs than our HTML was our understanding of human nature.

Here’s some data I dug up from the backup tapes. Each dot here is a person. The two scores are within a half point of each other for 92% of the sample after just 25 votes (and that percentage approaches 100% as vote totals get higher).

In short, according to our users, “looks” and “personality” were the same thing, which of course makes perfect sense because, you know, this young female account holder, with a 99th percentile personality:



…and whose profile, by the way, contained no text, is just so obviously a really cool person to hang out and talk to and clutch driftwood with.

After we got rid of the two scales, and replaced it with just one, we ran a direct experiment to confirm our hunch—that people just look at the picture. We took a small sample of users and half the time we showed them, we hid their profile text. That generated two independent sets of scores for each profile, one score for “the picture and the text together” and one for “the picture alone.” Here’s how they compare. Again, each dot is a user. Essentially, the text is less than 10% of what people think of you.



So, your picture is worth that fabled thousand words, but your actual words are worth…almost nothing.

Experiment 3: THE POWER OF SUGGESTION

The ultimate question at OkCupid is, does this thing even work? By all our internal measures, the “match percentage” we calculate for users is very good at predicting relationships. It correlates with message success, conversation length, whether people actually exchange contact information, and so on. But in the back of our minds, there’s always been the possibility: maybe it works just because we tell people it does. Maybe people just like each other because they think they’re supposed to? Like how Jay-Z still sells albums?

† Once the experiment was concluded, the users were notified of the correct match percentage.

To test this, we took pairs of bad matches (actual 30% match) and told them they were exceptionally good for each other (displaying a 90% match.)† Not surprisingly, the users sent more first messages when we said they were compatible. After all, that’s what the site teaches you to do.



But we took the analysis one step deeper. We asked: does the displayed match percentage cause more than just that first message—does the mere suggestion cause people to actually like each other? As far as we can measure, yes, it does.

When we tell people they are a good match, they act as if they are. Even when they should be wrong for each other.



The four-message threshold is our internal measure for a real conversation. And though the data is noisier, this same “higher display means more success” pattern seems to hold when you look at contact information exchanges, too.

This got us worried—maybe our matching algorithm was just garbage and it’s only the power of suggestion that brings people together. So we tested things the other way, too: we told people who were actually good for each other, that they were bad, and watched what happened.

Here’s the whole scope of results (I’m using the odds of exchanging four messages number here):



As you can see, the ideal situation is the lower right: to both be told you’re a good match, and at the same time actually be one. OkCupid definitely works, but that’s not the whole story. And if you have to choose only one or the other, the mere myth of compatibility works just as well as the truth. Thus the career of someone like Doctor Oz, in a nutshell. And, of course, to some degree, mine.

1,220 Responses to “We Experiment On Human Beings!”

  1. Penny says:

    Not impressed with okcupid at all.

  2. a_dude says:

    I always thought the matching on okcupid was thet I’ve ever seen sie a lot about the persons info would often appeal to to me, but I always found it funny when okcupid said they found a match and the people wer about 20% or less of a match and 70% or above of an enemy. Despite this, I went by the photo next, en the “about me” info, then the questions. Now I wonder if any of the numbers are true. Anyway, I like the site. It’s pretty solid.

  3. sheri young says:

    guess you’ve lost some credibility!

  4. Tom Hamann says:

    I enjoy reading most of the comments Even the negative ones.
    I’m not too sure But I think eHarmony Was the first on the Internet I have been on numerous websites I even met a girl got married off one And divorced lol I think we all try to find our true love I don’t think it matters what percentages on page If I’m not physically turned on to someone I don’t waste my time But I’m not a mean person. I have gone on dates With women I would never even think of going on dates with I think we should all step outside the box once in a while We all might learn something about ourselves

  5. Josh says:

    Interesting data but I think you’re missing some things. First thing…Does using a 4 message conversation take in to account the percentage of people that looked at the comparative questions, when during the course of the exchange did they check and did it make a difference? You said you changed the actual match numbers but I assume you didn’t change the answers they gave. I think that most of the users of okCupid assume that the questions at least play a part in coming up with the percentages. When people share a high match percentage almost all the answers to the questions are the same meaning that similar values are shared to some degree. Under the assumption that you guys aren’t just screwing with for the sake of checking whether your algorithms work, the match percentage is sort of a metric that we use to help make decisions and plays a part in searches. I may not know how you’ve come up with your math but it comes from a logical place, right….? I actually have a lot more to say but I’m too busy. I will say that at least some of us understand that the internet as a whole is always collecting data and using it for who the hell knows. At this point in history, I’m surprised when people are surprised by this practice. Over all, I think you guys are do a pretty good job.

  6. Hollis says:

    I’m glad someone is finally letting the consumers know how it really is. How could you NOT know you are part of hundreds of thousands of experiments daily? I also love how people are bashing OKC for doing this, when nearly ever site that has ever existed does the same thing. Anyways, good on you for being truthful and not messing with my privacy OKC.

  7. Chris says:

    FOR A FUTURE BLOG…The Williams Institute analyzed racial dating trends in the US. They did a study called “Marrying-Out” that analyzed U.S. census data about interracial marriages. I’d like to see how closely OKCupid users follow the trends purported by that report. Yeah. I know racial info kinda annoys some people, but it’d be really interesting to me! Also…they did a short bit about GLBT. Are their trends the same as the general pop? I’m gonna guess yes. I also believe that OKCupid users might be more inclined to date outside of their race/ethnic background than may be suggested by “Marrying-Out”.

  8. misterfeisty says:

    BULL SHIT!!!!!

    What about “matching” people that weren’t matches????

    When you use people for “experimental” purposes, you tell them FIRST!!!

    That’s why I’ve never answered you’re questions.

    Your PR response to your unethical behavior is pathetic

  9. Jeff Trigger says:

    I like how you think that this isn’t fucked up. People signed up expecting you all to be truthful. You weren’t. It’s not funny, and even worse it could have been dangerous. If you want to perform science experiments, go to school.

    Sincerely,
    One Pissed Off Former Customer.

  10. Ray says:

    As an African-American living in New Hampshire, it difficult to get into serious with anyone so far I’ve met on this site. It’s like something happened to women over the past few years that leaves me very confused. If I had a dime for every time I’ve been asked how big it is or ones that wants to keep anything we do in secret is very passe but women have become very aggressive lately and are acting more like horndogs more than I have ever done. But I still have hope that I will find the right one here and so my hunt is still on. One other thing. I’m into short women; the shorter the better but this site doesn’t take things like that into focus.

  11. Improvman says:

    It is so easy to say you have no freedom because you use the internet. And for that reason the people making that argument make it impossible to avoid. You hide behind ToS Agreements so long, that if someone took the time to read them all. Then it would eat three months out of every year.

    You maybe are protected by those, however your methods are unethical. Had a psychologist tried this they would loose their license. Your word is worthless now and nothing you say can be trusted from this point on.

  12. arron mohr says:

    Well I understand the sites understanding of Data, however the rating system i personally beleive should be in some heirchy. Just natural biology tends to predispose those with symmetrical faces versus none. I think the site should invest research into the understanding of biofeedback, neuroscience, and then their own theis of internal experiments of metadata of users. But of course im 20 soo im sure these things are already being thought in some kind of thinktank

  13. Joanna says:

    I do believe i was in the Experiment, i read the profile and then make a diction only after i have talked to them a few time then decide if it is a match,but unfortunately their are a lot of people on here that are fake and want money or other thins,Did you Experiment with that.I have gotten so hurt by people like this and i can’t believe you used people and their feeling to experiment with,i am a little up set about it..

  14. Renee says:

    Measure.duration of conversation for success. If you can capture when it goes off line would be best. Then duration off line. Look for commonalities that contributed to the new success benchmark. 4 messages could be a thank you that never ends.

  15. Achillesbluesky says:

    I’ve thought this place was creepy for a while now. Now that we see what’s been going on in the kitchen, there’s just no way we can be expected to think anything we find there isn’t making fools or statistics out of us.

  16. Kelli says:

    Oh but we have learned so much about you at okcupid as well. Your study tells me that you are a bunch of corporate whores. I am a Psych major . I hope that you prostitutes at okcupid got paid well by your .owners Here is hoping a swarm of a thousand Anons bestow you with their “blessings”. We do not forget. We do not forgive. Expect us.

  17. Suffrajester says:

    Did you just change the match percentage and leave everything else, or did you change the question responses that appeared for the purposes of calculating it?

    For example, the question “Is it ever okay to hit a partner?”. Suppose person A had said that they’d only accept the answer “No” and this was “Very Important” to them. And suppose person B answered “Yes, it is sometimes okay”. Normally, they’d appear with a low match % and high enemy %.

    Now suppose A and B were in your testing sample and A is told they have a 90% match with B. A goes to B’s profile and looks at that question: what do they see? Does A see straightaway that B said “Yes”, or has the response been changed to “No” to suit the scores? That could have very serious safety implications for A.

  18. Daryl says:

    From what I’m seeing, I will never again trust anything on a profile, and can’t honestly say that I would believe in OKC, in general. How do I know the replies aren’t being tampered with too? There are quite a few, with the same name…or very close, with essentially the same post. Someone mad, or amused enough…to post multiple times? Or, are these phony too…just thrown in …to see if anyone would notice? Does profile rating go on, at ALL online dating sites? Some, but not all. Some out there… have huge numbers of profiles, that they made up. They claim it’s to entertain and encourage user interaction. Does OKC pull that one, too? Don’t bother answering…. I can’t know if your answer would be true, anyhow. See how this business works,now? It can go many ways…not just what you planned it to

  19. Martin says:

    I’ve been on and off dating services over 20 years, I’ve met many ladies and have just corresponded with many. It all boils down to physical chemistry, life styles and internal chemistry in that order.
    All humans are visual, men more so, it all starts there and then what you do with your time makes you more or less desirable and then the inner person is the glue that holds it all together or not.
    Not everyone has these priorities in this order but I find most do.
    My son was on E-Harmony and met his wife site unseen and he is much more physically attractive than her but looks for him was not a priority, I truly believe he is the exception and most agree with me.
    I’ve had women just look at my profile and when I tell them I don’t make the kind of income they are looking for in their profile I’ve been told I didn’t have as much in common as they thought at first glance. So I know men are not the only ones looking at profiles without reading all the facts first.
    I just met a lady on OkCupid and neither of us recognized the other at first, most everyone posts pictures that are more flattering than we look in person. I got over it quickly after we talked and hugged good-by. I emailed her the next day and she said I was fun and interesting to talk to but she didn’t feel chemistry. We stayed friends and exchanged emails and are meeting again for laughs.
    We both got a message saying our original match was 31% and should have been 91%, we both laughed about it when we know it’s the chemistry that matters.
    All dating sites can do is lead the horses to water and let nature take it’s course.
    Martin

  20. ElectricCatHerder says:

    So that explains why I got so many highly-ranked religious single moms when I’m an agnostic who doesn’t want children.

    Congratulations, OKC, you just destroyed online dating.

  21. nix says:

    good job getting ahead of it. nice writeup

  22. Doug says:

    I’ve been on OKC for nearly a decade. I miss the friend % as I use OKC to meet cool new people. I’ve always enjoyed OKC’s experiments! I’ve learned more about myself on OKC than anyplace else! OKC is full of dry humor!!! Love You OKC!!!

  23. Jonathan says:

    I’m not your typical user.

    I’m looking for an intellectual / sexual connection.

    We can’t measure chemistry online, but I know that I’m attracted to women who are fit and slim like myself, and not overweight, hence the need for a full length photo. If I’m attracted to the photo and the face, I read the profile.

    If the profile is interesting, shows signs of intelligence, and ticks some of my boxes, there is hope. If it contains ‘red lines’ or is cliched, bland, boring, can’t-be-bothered, I move on.

    I’m looking for someone sexual and liberated, and find the questionnaire useful in giving me an insight into a woman’s sexuality, and how liberated and keen she is. This is vital information omitted by many dating sites.

    I respond to women who look good, sound interesting, and show a likelihood of being matched sexually.

    Experience has taught me there’s a 10% chance of chemistry if I follow these rules.

  24. Felix says:

    I actually entered on okcupid for experimental purposes too. I found that women here think of themselves so high that only a prince or a king might be their match ( and it seems they forget to look into the mirror ). “Gosh, where did all the good men go?”

  25. Brian says:

    Normally…. I’m very anal about being a guinea pig. But I like these experiment analysis’ and I like the approach you took to be open about it.

    I’ve found on your site, that a 30% rating was a direct reflection of someone’s self esteem, as compaired to mine. That a 90+% rating was likely a moral compatibility, and level of real happiness with ones-self (as compared to me).

    I’m not a stick in the mud…. But….. A lot of the sexual questions I have refused to answer, Generally due to the fact that Internet and past never disappear. Where others will willingly answer the sexual questions and care less about a job interview. I also find when reading others answers to questions about the sexual questions, I can evaluate their perspective of what a relationship means to them. Does relationship mean the interaction of communication, trust, respect and commitment….. Or does it mean orgasm? I can find sex anywhere. But I can’t find relationships anywhere.

    What I also find funny…. Even your admission and this reply/feedback session is going to be evaluated as an experiment! Hahaha!

    Anyway…. I believe your product probably works better in the urban setting than the rural. And I’m glad your experiment is free…. Charging me to be a guinea pig is almost like paying taxes….. Although I’m currently paying on another site…. Which I believe your owners also own…. You sneaky devils! Match.com. It’s nice to live in America right?!

    Wished your product worked better in rural areas! Truly! But you can’t choose volunteers I guess.

    Thanks guys! I’ll continue using your site. If nothing else, it allows me to conduct my own personality experiments and evaluations.

    Brian

  26. Truffs says:

    No kidding? Online for-profit site with little/no regulation not entirely honest with customers?

    I just died of shock…

  27. Jason says:

    My wife managed to find me on OKCupid. It works for some of us. My guess is that people whose reading comprehension and composition ability are of roughly equal levels, whose values are reasonably similar, and who will be attracted to each other when they meet are people who will probably be a good match. That third one is why we actually go on the dates.

    As for the match percentage manipulation thing, it’s not as if anyone knows what goes into that algorithm anyway, so why would anyone care if you decided 90 = 30 one day? At least you told them later.

  28. Todd says:

    When was the bad match percentages done? You’re attempt at humor is insulting. You shouldn’t mess with people like that. Shame on you okcupid!

  29. David says:

    I have advanced degrees in predictive analytics. In the past I had offered my services to assist you in getting better results in all facets of your data collection.

    I got no reply.

    Viewing the conclusions you have drawn I will state the following: while interesting at first glance I doubt the results based on many factors.

    How many people are on your site period vs. Your sample groups, not knowing this means that there I s no way to know if your sample is applicable to all users.

    Further, correlation does not equal causality. Just because more people exchanged contact info due to no photos means jack. You have no way of really knowing if photos truly played a significant factor, advanced regression analysis would assist you with this.

    Surveys, feedback, etc. Yield little without cluster analysis.

    In short I could seriously assist you in making OKC one of the most accurate sites ever. But you would need to contact me and we can discuss particular details.

    Lastly, I always get the exact same potential matches regardless of search criteria. This leads me to conclude that either A you do not have that many users or more likely B. You only grant paid members a larger listing. Thus your results are highly skwed at best.

    Given the average knowledge of your audience I bet many will be stunned by your findings.

    A statistically educated audience however will know that the results are open to debate.

  30. mike o says:

    It really doesn’t matter what kind of test you do,all in all its still a dating site,hopefully the people on here are real and really looking for someone.its all how you as a person act on here,how true you are to the ones your trying to get to meet.you can tell by the looks and what the people write in their profile how they are and go from there.who cares what percent of a match you are,you make your own judgment……good luck to all,have some fun,relax……

  31. Bette Davis says:

    I love your site, not too crazy about my matches, but yes, I recommend this site to all my friends, that are looking for a connection , I have met many friends, that I still talk with, thank you, for the awesome free site, online dating, I love. It.

  32. Dave says:

    It’s not a big deal… I can think for myself”no harm was done,probably to anyone!!!

  33. roberto feliciano says:

    I found it very interesting. I’m an average looking guy who has had a rough time dating. I must have gotten a good angle per picture when one person told me I don’t look like my own pictures. I was like wtf, you look heavier than your photos and I didn’t say that. I also was told without asking that my dates had a great time, only to receive a message saying, I’m a great guy but….. we, that’s my 2 cents comment. Wish me luck!

  34. Raxmiga says:

    I’d love to see some stats on how much the questionnaire affected results. I see someone I might like the first thing I do after that is go mining their answers for deal breakers. That’s what makes this site better than the others, its all based on user input.

  35. mobycone says:

    How can you act so nonchalant about this deceit? Most people come on this site, trusting that the profiles and match percentages are on the up and up. I may have been unknowingly part of this experiment of yours, while the very person I was supposed to meet, passed me by. Shame on you for hiding behind the ” everybody does it” pathetic excuse. If I wanted to be part of an experiment, I should have been given a choice, and a chance to take it or not. You coming forward is not going to excuse your reckless behavior with the FCC. How much emotional damage have you caused unsuspecting people? How dare you treat us like mice in a cage. Get the picture, you’re disgusting!

  36. Bill says:

    Ha ha ha ha ha, gosh what a funny jape. I always thought that your ratings were probably bollocks, but am surprised by the number of women that ask for a high percentage match in their profiles. You really have shown no respect for your members at all

  37. J says:

    All of the comments here that are praising OkCupid are very suspicious. I’ve had regular access to the internet for about 15 years now, and at no time have I ever seen anyone so enthusiastic about a blog post.

    OkCupid is pathetic. Everything about them is fake. They admitted to faking matches, and now they have to have paid commenters. I believe that all the legitimate comments are the ones from people who are pissed off, and the rest are fake.

  38. Jessie says:

    Awesome. Thanks for the honesty.
    Who knows, really, how love and relationships work? I’m enjoying meeting people, whether we are ‘supposed’ to be perfect for each other or not. If I could truly and accurately describe myself (TO myself, much less through a website questionnaire), I would probably not be human.

  39. RJ says:

    I love the statistics and they pretty much verify what I would intuitively have guessed. The picture is so important that I do not write nor respond without one. Keeps me from hurting someone’s feelings later when/if I am not physically drawn to them.

  40. Reynolds says:

    In any relationship without trust you have nothing..
    and you violated our trust.. and your not even remorseful!
    I thought you were one of the better dating sites..

  41. Tyler Adams says:

    I always thought the match % number was kinda shady. I always wondered how you determine two people’s match %. Cause some people I felt I matched way higher than what my match % said.

    Also I find that the majority of users male or female don’t bother to read your profile. They just go by your pocture. They never bother to read and get to know the person.

  42. Tasha says:

    I wouldn’t be surprised if I was part of the power of suggestion experiment – in September last year, my now-boyfriend messaged me, and we had something well over a 90% match rating, and we were always a little excited when we answered new questions and it went up slightly. When we met, and for the following few weeks, we came to realise we don’t have too many hobbies in common, and we have a million different opinions – yet, we still love each other very much and our anniversary is just on the horizon. I don’t know how your deliberate attempts to sabotage your users managed to bring us together, OKC, but I’m very grateful!

  43. Leon says:

    We’re just going to love your book because you told us we would, not because it’s really good, right?

  44. buckminster says:

    So you’re telling us that we ugly folks stand a better chance of getting laid if we date blind people. Great. Just. Great

  45. Tyler Adams says:

    I always thought the match % number was kinda shady. I always wondered how you determine two people’s match %. Cause some people I felt I matched way higher than what my match % said.

    Also I find that the majority of users male or female don’t bother to read your profile. They just go by your picture. They never bother to read and get to know the person. It drives me nuts that people do that. I always read people’s profiles because looks and personality are two totally different things. A person might not look the greatest but they might be a really cool person and thus the looks either get better or doesn’t matter.

  46. dignity in a box says:

    Sleazy and sooooo smug about it. Actually the more I see on the Internet the more misanthropic I become. Surprise! Hopefully someone will screw with your heads soon and you will see what it feels like. This warning came a little too late for more than a few of us it would seem so put that little bit of data in your next book punks.It is said that confession is good for the soul, assuming you are so equipped but if you expressed remorse you could very well be lying about THAT as well.

  47. Paul says:

    Experimenting on human beings without their permission puts OKCupid in the company of the Nazis. What you are doing is not a joke.

  48. sono says:

    Omg thank you OKC. I’ve always felt like sites like these are for bottom of the barrel scraping. I was surprised to find that the women were so shallow and the men were so disrespectful. You have proven online dating is a huge mockery of a lost art, conversation, compassion and open mindedness! To all my ladies, there is no Christian Grey. To my fellas, conduct yourselves with some class. To OKC respect for the honesty! Lmao!!!

  49. texanbear says:

    I never really gave a rat’s ass how you rated who is my match/enemy! I know what I like and what I’m looking for, so I base my choices on the profiles, not your match-ups. unless of course, you somehow edited those too! however, I do understand how your other customers are PISSED for screwing around with their lives.

  50. Brian says:

    What kind of scum lies to people looking for relationships, then tries to justify it by comparing themselves to scientists? Lol….Does it get more pathetic?