“They Might Never Tell You It’s Broken”, 2019-11-02 (; backlinks; similar):
As part of my PhD, I developed Higgs, an experimental JIT compiler…I developed it on GitHub, completely in the open, and wrote about my progress on this blog. Pretty soon, the project had 300 stars on GitHub, a handful of open source contributors, and I was receiving some nice feedback.
…One day, someone I had been exchanging with on the chat room for two weeks reached out to me to signal a strange bug. They couldn’t get the tests to pass and were getting a segmentation fault. I was puzzled. They asked me if Higgs had MacOS support. I explained that I’d never tested it on MacOS myself, but I couldn’t see any reason why it wouldn’t work.
I told this person that the problem was surely on their end. Higgs had been open source for over a year. It was a pretty niche project, but I knew for a fact that at least 40–60 people must have tried it, and at least 50% of these people must have been running MacOS. I assumed that surely, if Higgs didn’t run on MacOS at all, someone would have opened a GitHub issue by now. Again, I was wrong.
…It’s a horrifying thought, but it could be that for every one person who opens an issue on GitHub, 100 or more people have already tried your project, run into that same bug, and simply moved on.
[One developer observes that “Despite having just 5.8% sales, over 38% of bug reports come from the Linux community”, of which only 3⁄400 were Linux-specific—affected users on other platforms simply didn’t report them. Firefox likewise. Data science example. Bill Gates Windows rant. Scott Hanselman listed 19 categories of issues in his previous week (iTunes/iPhoto simply being categories of their own listed: ‘everything’.) Soren Bjornstad nearly failed an online exam due to a cascade of 15+ problems & attempted fixes, which fixes he estimates as requiring knowledge of at least 12 different pieces of tech esoterica that no user should have to know. tzs notes Japanese users are so inured to bad software they’ll let something install for 30 hours. Dan Luu chronicles a week of bugs he observed, large & small (so many he couldn’t report them all). Fintech support staff don’t report bugs because “complaining about this is like complaining about gravity”. Sometimes you have to bounce between browsers even for high-value transactions like selling a car. I’ve observed quite a few bugs just from cats walking on my keyboard. (A IRL version of fuzz testing, itself notorious for finding endless bugs in any software it’s used on.) Sometimes, the wrong book get published and despite many angry readers, no one knows or fixes it for months. My own experience reporting website bugs in particular has been that many of them were unknown to the authors (and they appreciate the reports). Gwern.net examples of this include: 400,000+ Chinese visitors to This Waifu Does Not Exist not mentioning that the mobile version was horribly broken; Apple users not mentioning that 80% of Gwern.net videos didn’t play for them or that Apple won’t support OGG music; the Anime Faces page loading 500MB+ of files on each page load… Another fun example: popups on all Wikipedias worldwide for ~5 months (September 2020–January 2021), could be disabled but not re-enabled (affecting ~24 billion page views per month or ~120 billion page views total); no one mentioned it until we happened to investigate the feature while cloning it for Gwern.net.
How can devs miss so many bugs, and so many be unreported by users? Many reasons. Users don’t know it’s supposed to not hurt, and have low expectations; they also develop horrifying workflows (obligatory XKCD), (ab)using features in ways designers never thought of. Developers (particularly ones with mechanical sympathy) undergo decades of operant conditioning teaching them subconsciously how to use software in the safest possible way (eg. not typing or mousing when software lags, cf. Luu), illusions of transparency in how clear it is to use something, and suffer from a curse-of-expertise in knowing how their software should work, while they need to unsee it in order to break it. (Testing is a completely different mindset; for an amusing fictional demonstration, read “Stargate Physics 101”.) Alan Kay: “…in many ways one of the most difficult things in programming is to find end user sensibilities. I think the general programmer personality is somebody who has, above all other things, learned how to cope with an enormous number of disagreeable circumstances.” “You can’t tell people anything”/“You can’t reach the brain through the ears”/inferential distance, in part because students are silent.]
View HTML: