We Experiment On Human Beings!

July 28th, 2014 by Christian Rudder

I’m the first to admit it: we might be popular, we might create a lot of great relationships, we might blah blah blah. But OkCupid doesn’t really know what it’s doing. Neither does any other website. It’s not like people have been building these things for very long, or you can go look up a blueprint or something. Most ideas are bad. Even good ideas could be better. Experiments are how you sort all this out. Like this young buck, trying to get a potato to cry.


We noticed recently that people didn’t like it when Facebook “experimented” with their news feed. Even the FTC is getting involved. But guess what, everybody: if you use the Internet, you’re the subject of hundreds of experiments at any given time, on every site. That’s how websites work.

Here are a few of the more interesting experiments OkCupid has run.

Experiment 1: LOVE IS BLIND, OR SHOULD BE

OkCupid’s ten-year history has been the epitome of the old saying: two steps forward, one total fiasco. A while ago, we had the genius idea of an app that set up blind dates; we spent a year and a half on it, and it was gone from the app store in six months.

Of course, being geniuses, we chose to celebrate the app’s release by removing all the pictures from OkCupid on launch day. “Love Is Blind Day” on OkCupid—January 15, 2013.

All our site metrics were way down during the “celebration”, for example:



But by comparing Love Is Blind Day to a normal Tuesday, we learned some very interesting things. In those 7 hours without photos:

And it wasn’t that “looks weren’t important” to the users who’d chosen to stick around. When the photos were restored at 4PM, 2,200 people were in the middle of conversations that had started “blind”. Those conversations melted away. The goodness was gone, in fact worse than gone. It was like we’d turned on the bright lights at the bar at midnight.



This whole episode made me curious, so I went and looked up the data for the people who had actually used the blind date app. I found a similar thing: once they got to the date, they had a good time more or less regardless of how good-looking their partner was. Here’s the female side of the experience (the male is very similar).



Oddly, it appears that having a better-looking blind date made women slightly less happy—my operating theory is that hotter guys were assholes more often. Anyhow, the fascinating thing is the online reaction of those exact same women was just as judgmental as everyone else’s:



Basically, people are exactly as shallow as their technology allows them to be.

Experiment 2: SO WHAT’S A PICTURE WORTH?

All dating sites let users rate profiles, and OkCupid’s original system gave people two separate scales for judging each other, “personality” and “looks.”
I found this old screenshot. The “loading” icon over the picture pretty much sums up our first four years. Anyhow, here’s the vote system:



Our thinking was that a person might not be classically gorgeous or handsome but could still be cool, and we wanted to recognize that, which just goes to show that when OkCupid started out, the only thing with more bugs than our HTML was our understanding of human nature.

Here’s some data I dug up from the backup tapes. Each dot here is a person. The two scores are within a half point of each other for 92% of the sample after just 25 votes (and that percentage approaches 100% as vote totals get higher).

In short, according to our users, “looks” and “personality” were the same thing, which of course makes perfect sense because, you know, this young female account holder, with a 99th percentile personality:



…and whose profile, by the way, contained no text, is just so obviously a really cool person to hang out and talk to and clutch driftwood with.

After we got rid of the two scales, and replaced it with just one, we ran a direct experiment to confirm our hunch—that people just look at the picture. We took a small sample of users and half the time we showed them, we hid their profile text. That generated two independent sets of scores for each profile, one score for “the picture and the text together” and one for “the picture alone.” Here’s how they compare. Again, each dot is a user. Essentially, the text is less than 10% of what people think of you.



So, your picture is worth that fabled thousand words, but your actual words are worth…almost nothing.

Experiment 3: THE POWER OF SUGGESTION

The ultimate question at OkCupid is, does this thing even work? By all our internal measures, the “match percentage” we calculate for users is very good at predicting relationships. It correlates with message success, conversation length, whether people actually exchange contact information, and so on. But in the back of our minds, there’s always been the possibility: maybe it works just because we tell people it does. Maybe people just like each other because they think they’re supposed to? Like how Jay-Z still sells albums?

† Once the experiment was concluded, the users were notified of the correct match percentage.

To test this, we took pairs of bad matches (actual 30% match) and told them they were exceptionally good for each other (displaying a 90% match.)† Not surprisingly, the users sent more first messages when we said they were compatible. After all, that’s what the site teaches you to do.



But we took the analysis one step deeper. We asked: does the displayed match percentage cause more than just that first message—does the mere suggestion cause people to actually like each other? As far as we can measure, yes, it does.

When we tell people they are a good match, they act as if they are. Even when they should be wrong for each other.



The four-message threshold is our internal measure for a real conversation. And though the data is noisier, this same “higher display means more success” pattern seems to hold when you look at contact information exchanges, too.

This got us worried—maybe our matching algorithm was just garbage and it’s only the power of suggestion that brings people together. So we tested things the other way, too: we told people who were actually good for each other, that they were bad, and watched what happened.

Here’s the whole scope of results (I’m using the odds of exchanging four messages number here):



As you can see, the ideal situation is the lower right: to both be told you’re a good match, and at the same time actually be one. OkCupid definitely works, but that’s not the whole story. And if you have to choose only one or the other, the mere myth of compatibility works just as well as the truth. Thus the career of someone like Doctor Oz, in a nutshell. And, of course, to some degree, mine.

1,220 Responses to “We Experiment On Human Beings!”

  1. LadyIdunaStorm says:

    Experimentation is what everyone does every day of their life. We try “this” to see what happens. if it works we keep it; if is is a dud, we move on. One suggestion…. open up your religious parameters a little. “New Age” really is not a religion. It’s a PC BS way of saying pagan, wiccan, asatru, druid, etc. How about an “Other” pulldown with a fill in line? I’d really like to meet spiritually compatible folks, or at least some that I needn’t explain everything to. Thanks. :)

  2. Eveproc says:

    Well, that’s weird cos I’d never date somebody I couldnt see and didn’t know what they looked like! I never respond to messages without pics.
    First I look at the pic and then the profile. I’ve been on the site a year and had a few chats but never met anyone! My location is a big problem.
    I’ m not happy with how many young boys contact me when I specifically said age 40plus.

  3. sanjoysaw says:

    M wnt to a figical satisfication form u in bed. Call me8479038613.

  4. K2 says:

    Match.com bought over OKC … That would explain why many of the great functions on this site (e.g. Notes, Journals, Forums to provide feedback) have been killed, and the lack of the great blogs it used to have.

    This site’s becoming more like the other lame mainstream sites like POF and Tinder where users judge based on looks alone …

    This just attracts more POF and Tinder users who don’t bother to fill in their profile … and less discerning users … People who don’t understand the algorithms, don’t pay attention to how readily a person responds to message, and yet demand a response.

    And you’re surprised with the results of experiment 2?

    Stop treating all users as uniform …

    I strongly disagree that profile text means nothing.

    Try doing a poll on functions that users like, and then compare this to when they joined the site, and also how many questions they have answered, how many messages they send, and how much they’ve filled in their profile …

    You probably have this info at hand already … but you’re ignoring it to drive user numbers up and sell advertising space on the site, at the expense of the substance and quality of the service …

    OKC’s slowly loosing all it’s accolades … and I don’t use it as often as I used to.

    The decision-making person/s doesn’t give a toss about the true development of this site. Otherwise, they’d jump through hoops so ensure the site remains and retains all of it’s unique functions.

    Or perhaps, all the persons that made the site what it was, have left due to frustration brought on by the short-sightedness of new management …

  5. Greg Jenkins says:

    No worries, whatever you can do to make the site better works for me!

  6. Alexander says:

    People like pictures, but they reduce the quality of conversations, so how can we encourage conversation? How about a discussion forum? A place to have interesting conversations with groups of people – to break the ice…

  7. sanjoysaw says:

    M very hard hot sexy playboy,wnt to be a figical satisfication form u in bed. Call me8479038613.

  8. Hannah says:

    Wow, there are a lot of overly sensitive men in the comments here.

    For those who never get responses from women, ask yourselves a few questions:
    1) Do I have my profile mostly or fully completed?
    2) Have I answered at least 100 questions (to help with match percentages and to let women learn a little bit about me)?
    3) Do I have at least two pictures that are decently well taken (only one will make you look lazy/suspicious and no pictures will get you no responses pretty much ever)?
    *** 4) When I message a girl do I mention something I liked about her profile and try to start an actual conversation with more than “hi/how r u/ur hot”?

    Biggest tip for any man messaging women in OkC – be polite, open, positive, and fill out your profile. Most men don’t put any effort into their profile. Women get dozens of messages a week (when I first opened my account I had over 100 in less than 24 hours). So stand out. Put in the effort and you’ll find yourself a lot more success.

    This is not a popularity contest, it’s about exposing yourself to enough people in a positive and useful way that you’ll find someone that fits well with you.

  9. shane says:

    Do you bitches and assholes really think and calling yourself geniouses when you are really putting mens and especially womens lives at risk? Do you stupid motherfuckers really not see any dangers in your tests? I hope you get sued so royally that you are forced to suck your own cocks and pussies for money. And I hope your dicks are cheesy when sucking them and your pussy smells like 2week old salmon in the sun.

  10. Paul says:

    I always suspected men and women were shallow and hypocritical beasts… I do it myself and it depresses me when I skip someone because the fall outside my criteria, even more when they message me… to respond and kick up a conversation or not ?

    More women need to flaunt their brass and start messaging the guys. What are the stats on that ? How many women start conversations ? Has it been increasing over the years of okcupids operation ?

    I can tell you one thing though… Your inexhaustible list of questions need a cleanup of duplicates and such forth… I feel there are missed matches and opportunities due to the plethora of poorly worded, obscure and oddly weighted questions. And there needs to be some encouragement to add written answers to questions. I have messaged more than one after busting a gut over their answers.

  11. Justin says:

    Profiles must be completed and at least to pictures

  12. Mohamed Esssa Ashoor says:

    Hallo

  13. ally says:

    good experiment

  14. JJ says:

    The realities of internet dating …

    Even if you and another user have a 90% match, chances that either will send a message to each other is 20% …

    Do bear this in mind when you message someone (in terms of what and how much you write, both in your profile and in your message) …

    and temper your expectations … Life’s complicated … OKC alone can’t find you Mr or Mrs Right … but it’s better than most other sites … for now

  15. Chris says:

    I’m glad to have read this. If there’s any way that the website can become better to help other men like me, who are trying to do this the right way and have our profiles completed with up to date and good pictures and send messages trying to spark a conversation, I’m all for it. Getting one response for every 50 messages is heartbreaking.

  16. Brad Martin says:

    I knew it was an experiment with all the questions. I have no problem with that the site is free. I would have not have gone on any dates with out it. Hey you have to make money too, right. I just hope she will find me one of these days it has been 9 years and still looking.

  17. jose says:

    very interesting and true. it’s all about the looks to start the fire, it’s all about personality to make it real.

  18. Luke M says:

    You can strongly disagree that profile text means something, but it may have a stronger correlation with location/demographic than it suggests. Moreover, if an entire city is in a similar socioeconomic position, people are less driven by desire to be “somewhere’ else (in their fantasy).

    What I’m saying is, where I live, looks are 100% and bio-feedback is 0%. Everyone will ignore their feelings and base their real selves on how they want to perceive their thoughts.

    It sounds skewed, but it’s real.

  19. dean says:

    Co-incidence, timing or damage limitation, as the press leaked this story last week .. Oh surprise but we already knew this .. lol

  20. john says:

    Your analysis can’t distinguish between whether having no photos actually caused people to behave in deeper, less shallow ways or simply weeded out all of the shallow people and left only the less shallow ones remaining. Either way the percentage of deeper contacts would increase but the actual change in human behaviour is quite different.

    If you’re having a hard time with it, contact me if you want to figure this out.

  21. RV Bond says:

    I would like a some kind of category where people can be pen pals via email or snail mail. Lots of people enjoy corresponding with new and interesting people without having romance be any issue.

    RV

  22. samuel says:

    Just a suggestion: OKCupid should have a status feed, where everyone can see it. Like for ex: I can just post on my status feed saying” hey I just came from work message me or something ” you know something like that

  23. Elsie Kahan says:

    I already new all this because it was on almost every channell on TV

  24. Elherrawy says:

    fingers of the same hand doesnt match . :)

  25. Karen says:

    I don’t find OKCupid’s “experiment” at all interesting. The website seems to claim no harm, no foul…especially since this kind of data-gathering is done to people via the Internet all the time anyway. But I don’t like wondering whether the “matches” I get are for-real or if OKCupid’s just screwing with me. This doesn’t inspire any trust in your system.

  26. Frank Williams says:

    Nice. As in accurate. Very perceptive. Thanj you !

  27. Enygma says:

    Have to agree with what user K2 said…. all the best stuff about OKC is gone, reduced to little more than a profile (the censored, edited version of self) and a picture… All that experimenting just to clone all the other boring, lifeless dating sites? Pity, it had potential to be something special… a social network for actively dating adults or somesuch, but noooo, you just had to reduce it to a virtual meat market :/

    Also not really using as it was intended to be used, it’s more like an entertaining social science read, giggles at the good, bad or ugly side of human personalities…

  28. HeadHeartBody says:

    Yeah, we’re being *totally* unreasonable, objecting to OKC screwing about with results without telling anyone. You know what result it had on me? It made me think OKC hasn’t a clue what it’s doing and treats its customers like dirt.

  29. fixurproblem says:

    if u have time to do the experiment, fix your ‘technical issue’ so i can log in. this is the second time that i CAN’T log in

  30. Siilk says:

    So OKC is just another f*cked-up company that thinks it can do anything with it’s clients simply because “it’s internets”. To hell with you then, I’m removing my profile.

  31. Daniel says:

    This is the best evidence that human relationships aren’t something you can express in numbers.
    Download Elimi App from http://www.ElimiApp.com/ and check how people should match :) #HaveFun

  32. peter says:

    If ypu get older love gets more realistic and people tend not to travel lhuge distances for a lover. Sure some will but most not.
    So practical selection a better geo matching system.

    And if you dont want to live in a huge town get those of your search results.

    Also if you like to experiment try to do face matching. Its just my observation that people often have some face feature in common.its the looks thing I guess.

  33. BB says:

    I found everything in the article very interesting, pictures are the thing I’ve always looked at first but I’m also looking for that spark that need for something more to connect with something and someone a girl that is beautiful inside and out. That is extremely hard-to-find sometimes by starting blind may be a great way to get to know somebody. I believe whatever you can do to enhance the site to help others find that person is a great opportunity.

  34. Dalekasaurus says:

    I’m all for the text. I’ve always thought of OKCupid as the site for nerds and introverts (Sadly it’s being flooded with POF rejects) and a lot of us, myself included jump to hobbies, likes, dislikes trying to find another who shares in our passions.

    Keep up the good work OKC.

  35. Dominic says:

    I don’t know why anyone would be mad about this-It’s sociology which is the study of humans-it will only make it work better

  36. RQRQ1 says:

    How thoroughly sad and depressing. A bunch of shallow horndogs that can’t figure out that plain people can be wonderful and hot people can be complete assholes. That speaks volumes about our society and what we value.

  37. salah says:

    hi how r u ?
    well I cant thanks to u with words coz u dont how much the okcupid help me I was wondering how I meat my soul meet and how I find her
    and when I start okcupid I meet so many people and make lat of friends and I find my soul meet in okcupid .
    soo thank you soo much .
    and I really love u.

  38. A says:

    Against my better judgment I clicked and opened your blog link. I could hardly get through your stats but your commentary kept me engaged all the way to the bitter end. It’s astonishing that your list of questions has any value at all– to call them merely irritating and insulting would be a compliment. Thanks for everything else though.

  39. Anonymous says:

    Wow, good job mentioning Facebook’s experiment. How many people do you -want- to have /ragequit on you?

    And good job on saying you’re part of the problem! How many people do you -want- to have /ragequit on you?

    And good job for in fact explaining HOW you are part of the problem!

    … I think I might /ragequit on you.

  40. Kindlingwood says:

    I’ve always felt that your method of matching people is really ineffective. I rarely have any interest in the matches that are e-mailed to me and the match percentages are equally poor. I guess that the 1% to 6% margin for messages turning into conversations, between 30% matches and 90%, matches proves my point. At this point, I log onto Okcupid for my amusement, more than I do for the prospect of meeting anyone with relationship potential. I guess inflated ego’s are how companies sell themselves too, just like most of the people on dating sites.

  41. EowynB5 says:

    I only look at the match percentage to see if the guy actually filled out his profile. So many get on here and just post a pic and then troll for women. If I wanted a bar scene pick up, I would go to a bar. If the guy didn’t fill out his profile, I don’t bother. If he did, I read the profile to see if we have similar interests or compatible ones. I also look at the details since certain things are important to me like not wanting to have kids or liking cats. Looking at the photos comes last but I won’t deny that what a man looks like will influence me on whether or not I want to date him. Appearance is subjective though. Everyone has a different idea of what they find attractive.

    Btw, I only ever met one person on this site in person. Every other man I talked to and we intended to meet, he either stood me up or came up with a lame reason why he had to cancel at the very last minute. The one I actually did meet decided to not speak to me after t he coffee meet up. Perhaps it is me but I am confident that through the talking here, on the phone, and in emails, anyone I meet through here has a pretty good idea of who I am before we actually meet. I do take my time getting to know someone before actually meeting him. At least I know ofnone person who met a woman through here and now they are married so it gives me hope that it is possible.

  42. Apurva Shrivastava says:

    Interesting read.

    I have found women are more attracted to looks than to any other thing. While they profess to be atrracted to a “mund”, they may be like their male counterparts. So, gender may not play anything in determining criteria for likeness.

    It may be a question of age as well. People at an older age they see there is lesser runway ahead than behind them. So, the need to experience all carnal needs before thay burn out. :-)

    At an earlier age one ia more idealistic and as one grows older one griws more practical. One has more resources to be inward looking only too.

  43. Scott says:

    That explains why all the people who supposedly matched me, and the two to three dates a week failed to be fun. I met one awesome chick, so thanks, but I had to have gone on at least 90 dates where I didn’t have a mutual connection.

    I’ve finally found someone to love, but it was rough, and I am a playful, goofy guy. I don’t know exactly what was up, but I want to take a survey or something to add my voice to the data.

  44. Kathleen Kite-Powell says:

    As a writer, I do pay attention to what is written. In fact, to avoid any reaction by men to photos, I have none up there, just to see if I get better people that way. No idea yet. The reason for that was that I was in a dating service years ago and was often confused by the men who made contact…they appeared not to have read my profile/not be compatible, so eventually I guessed that they did what they seem to do in life: like looks and attach fantasy person, rather than be interested in the real one that came with the face/body. I remain a bit flummoxed about how much weight they give the human under the body. But as a picky person, I notice that I notice pictures for their ‘Would this ever work at all for me’ type of potential. If I can’t feel tempted by the picture, that man would likely never be more than a friend anyway. For me I know that even if pic and words went well, there would still be the whole in-person chemistry hurdle. So I consider my beginning attempts at the electronic version of things entirely experimental, and simply holding potential that should be explored . I do think all the matching so far that I have encounted at different websites is not good at all. Why are certain things not ending up as filters? How can I make them become bottleneck filters, instead of seeming to receive totally random and incompatible guys? When will matching ever become truly believable, trustworthy? Right now my most frequent reaction to matches given (by this and other sites so far, I reiterate) is, “Whaaaaaat? No way!” But then again, maybe they ARE more compatible than the rest…heh heh

  45. saleh says:

    I do believe it works yes would love to be in one

  46. delbert smith says:

    Being told youre a match doesnt matter , if you have a mind of your own .I use the match % as a starting point for getting aling , i takevit as ” how many of the same things do you like etc .. .hell two people mightbe beyond perfect for one another andv once one of them speakks , they shri

  47. rsgrafiker says:

    Fascinating! I appreciate opening up your analytics for comment, especially as to the relative importance of photos. Since “Every picture sells a story” I’m surprised how little thought goes into most of those selected to represent the person. Going beyond the “looks” of the person, first the the photo quality itself: in focus? color correct? person large enough in frame? Then the background: most people unconsciously “read” a picture for personality/shared interest clues. Vacation site? athletic props? books? posters on walls? pets? And avoid group pics. Finally, it would be refreshing to see a creative photo that spontaneously intrigued the imagination of the viewer.
    Having said that, I still read the profile. But the photos say a lot.

  48. Antoine says:

    Very good read!! I actually prefer OKC over POF and for many reasons. More of these articles please!!

  49. Anthony says:

    Interesting research, but, after looking at te picture and the match%, I always read the profile and look for matching questions/answers

    1. The Match is important. I don’t even look at women who have
    less than a 75% match as I know that there is no chance at all
    of a matching lifestyle/outlook. The only exception is looking
    at new people who have 0%.

    2. Just like people don’t show up in Browse Matches without a
    picture (on the computer, they do on the iphone app), I think
    the same limitation should be applied to people who have not:
    (a) written a minimum profile, *and*
    (b) answered at least some minimum number of questions.
    All three of thee conditions should be clearly stated when a
    new member joins.

    3. There are way too many questions, and too many asking
    essentially the same question. This results in two people
    each answering a different “same” question and so missing
    a match. There needs to be a massive cleanup.

    4. There is, in my opinion, a mistake in the matching algorithm.
    As far as I understand, the matching is calculated based on
    questions answered by both sides.
    I would suggest that if I have marked a question as very
    important, and some one has not answered that question, this
    should actively *reduce* the match, and not be simply ignored
    as non-common question.

    Still, OkCupid is still the best and most fun site that I have found (where I live).