I know a person who runs lots of field experiments. Literally prize winning stuff. Their estimate of how likely it is for a field experiment to happen successfully once a partner firm or organization agrees: 10% (Success here means actually carried out, correctly)

Feb 27, 2023 · 4:52 PM UTC

17
37
13
289
Replying to @dggoldst
Am quite far from being a prize winning field experimenter, but this is exactly my experience too. (And I'm curious if this is the same person who said 1 in 8 to me!)
1
12
Could be! Maybe they updated by 2.5 percentage points
6
Replying to @dggoldst
What are the usual failure modes in that 90%?
1
2
Replying to @dggoldst
I’m curious if folks charge? 100% of studies we spin up run. We ask for $. Companies pay us. So they aren’t backing out. 50% of those we get approval to write about /share. This prob results in same amount of studies run, given top of funnel is low, but saves our time!
3
Replying to @dggoldst
That seems pretty accurate!
1
Replying to @dggoldst
my field experiment N is small but 10% seems a good estimation
1
Replying to @dggoldst
Sure. I suspect it’s way lower than 10% for the average economist. I also suspect that there are some things economists can do to get these numbers to be better.
1
Replying to @dggoldst
This is actually quite comforting
1
Replying to @dggoldst @jonmladd
I have a Drive folder full of never run experiments and I've been genuinely pleased with the amount that have been completed
Replying to @dggoldst
That’s reassuring. I’ve invested many hours building relationships with many orgs (but <10) and only one “success” (w/ Uber) but was able to get a couple other papers with proprietary data from other firms even if the proposed experiment didn’t happen.
4
Load more