They Might Never Tell You It’s Broken

November 2, 2019

This blog post is a public service announcement (or maybe a reminder) for anyone working on a programming project that they already have, or intend to release to the public, be it as something open source, a pet project, or a startup.

I’ve worked on a few successful open source projects over the last 10 years. I would consider these projects successful in that they got hundreds of stars on GitHub and each attracted multiple open source contributors. I actually shut down one of these project because reviewing pull requests alone was becoming a second unpaid job on top of my regular job, taking multiple hours out of my evenings after work, which became exhausting, but that’s a story for another post. What I want to tell you about today is something important, that I believe any developer should know, but I personally didn’t understand until I had been working on open source projects for a few years.

As part of my PhD, I developed Higgs, an experimental JIT compiler for JavaScript written using the D programming language. I developed it on GitHub, completely in the open, and wrote about my progress on this blog. Pretty soon, the project had 300 stars on GitHub, a handful of open source contributors, and I was receiving some nice feedback. It made me happy to have so many people taking notice of my research work. As part of Higgs, I had written my own x86 machine code generator, which enabled it to do machine code pirouettes LLVM couldn’t. I did all my development on Linux, but had done my best to keep the code as portable as possible, and so, that code would work fine on MacOS as well, I had assumed. Unfortunately, I was wrong.

About a year into its development, Higgs had enough of a small community that it made sense to have create a chat room to exchange with other contributors and users. About a dozen people joined over the next two months. One day, someone I had been exchanging with on the chat room for two weeks reached out to me to signal a strange bug. They couldn’t get the tests to pass and were getting a segmentation fault. I was puzzled. They asked me if Higgs had MacOS support. I explained that I’d never tested it on MacOS myself, but I couldn’t see any reason why it wouldn’t work. I told this person that the problem was surely on their end. Higgs had been open source for over a year. It was a pretty niche project, but I knew for a fact that at least 40-60 people must have tried it, and at least 50% of these people must have been running MacOS. I assumed that surely, if Higgs didn’t run on MacOS at all, someone would have opened a GitHub issue by now. Again, I was wrong.

The problem, it turned out, was that MacOS had more strict requirements for keeping the stack pointer aligned. This wasn’t difficult to fix. The more important lesson, that I didn’t understand until that point, is that you can’t count on the people trying your project to quickly and reliably signal bugs to you. Most of the time, if it doesn’t work, they won’t report the problem. There are a few reasons why this might be:

  • They assume that someone else has already reported the problem, and there would be no point in saying anything. The bigger your project, the more likely people are to assume that someone else has already reported the issue.
  • They think the fault might be on their end, they may be confused and feel too embarrassed to reach out for help. Nobody wants to look stupid or get told to RTFM, and so, they choose silence.
  • They are just trying your project out of curiosity, and are in no way invested. They will find an alternative to your project, or go do something else.

It’s a horrifying thought, but it could be that for every one person who opens an issue on GitHub, 100 or more people have already tried your project, run into that same bug, and simply moved on. So, what can you do? You can encourage people to report bugs. I visibly write in my GitHub README that reporting bugs is encouraged and welcome: “Please tell me if something is wrong, you’re helping me make this project better.” Another obvious thing that you can do is to have robust automated testing. Some continuous integration services can automatically test on both Linux and Mac.

More broadly, if you want your project to be successful, I think it’s important to try and put yourself in the user’s shoes. Every once in a while, try installing your software from scratch, along with all the dependencies. Ideally, you want your installation process to be as simple and frictionless as possible. If your software requires the user to perform 20 steps to get it running, you’ll be losing potential users even before they have a chance to figure out if it works on their system or not. In general, I find that writing software with a minimalist design philosophy, minimizing external dependencies as much as is reasonable to do so, will help you avoid bugs, and streamline your installation process.

From → Programming

10 Comments
  1. Alice Vixie permalink

    “Given enough eyeballs, all bugs are shallow.” — Raymond: The Cathedral and the Bazaar. 1999.
    🙃

  2. Had the same issue with bug reports and changed a fail message long ago here:
    https://github.com/JikesRVM/JikesRVM/blob/master/rvm/src/org/jikesrvm/VM.java#L2423
    There’s a culture of assuming compilers/runtimes can’t be fixed and working around them (see C’s inline assembly). I can definitely relate to the, “I tried your project but encountered a bug but didn’t report it,” problem.

    • It seems like a very smart thing to do to have an error message that directs people to a link to report the issue, ideally with friendly language (the users reporting bugs are doing you a favor).

  3. This is basically the Bystander Effect as applied to software:
    https://en.wikipedia.org/wiki/Bystander_effect
    If this effect is in fact, in effect, here, then the number of visible stars on your project might only amplify it, ironically!

  4. Anonymous permalink

    I was looking for minimalist-ish systemd-free Linux distros for my RPi recently. I tried Void, it worked well, booted reeeally fast, but I couldn’t get 3d acceleration to work so I just moved on, tried Alpine instead. Turns out it has the same issue but this one I’m reporting (unless somebody beat me to it): the project runs a mailing list and an IRC channel while Void is on Github which means all the things I ‘love’: Microsoft, javascript, and Google’s reCAPTCHA. Nope. Absolutely nope.

    I remember a talk by a (or is it ‘the’) developer of i3 (a window manager) where he mentioned that their bugtracker allows anonymous access exactly not to kill motivation to report.

  5. ulidtko permalink

    Agreed. I wish more developers and/or maintainers would realize this.

    I’ve had a vaguely similar insight awhile ago, perhaps back in the days when I’d learn [and internalize the consequences of] the Bystander Effect.

    Being conscious of this, and enthusiastic about Open Source, I’d start reporting issues, as light as mere “friction points” during build/install, as part of my “trying the thing out”.

    Imagine my disappointment when authors would dismiss the reports under various excuses, not recognizing that my intention is to help them, and at times even visibly skimming through the report without a second thought.

    Certainly, not all authors have been like that. But please, don’t be those authors, and don’t allow your peers to be — if you care, of course.

Trackbacks & Pingbacks

  1. New top story on Hacker News: They Might Never Tell You It’s Broken – Latest news
  2. They Might Never Tell You It’s Broken – Hacker News Robot
  3. Michael Tsai - Blog - They Might Never Tell You It’s Broken
  4. Code that Doesn’t Rot | Pointers Gone Wild

Leave a comment