--- title: Technology Holy Wars are Coordination Problems description: "Flamewars over platforms & upgrades are so bitter not because people are jerks but because the choice will influence entire ecosystems, benefiting one platform through network effects & avoiding 'bitrot' while subtly sabotaging the rest through 'bitcreep'." created: 2020-06-15 modified: 2020-07-09 status: finished confidence: likely importance: 3 css-extension: dropcaps-de-zs ...
> The enduring phenomenon of **holy wars** in computing, such as the bitterness around the prolonged Python 2 to Python 3 migration, is not due to mere pettiness or love of conflict, but because they are a [coordination problem](https://en.wikipedia.org/wiki/Coordination_game): the problem is not getting everyone to make a good decision, but making the *same* decision. Apparent problems with an unpopular platform may actually be unacknowledged parts of an attempt to coordinate use of different platform: dominant platforms enjoy strong [network effects](!W), such as reduced [**bitrot**](#bitrot) as it is regularly used & maintained by many users, and can inflict a mirror-image [**bitcreep**](#bitcreep) on other platforms which gradually are neglected and begin to bitrot because of the dominant platform. > > The outright negative effect of bitcreep mean that holdouts do not just cost early adopters the possible network effects, they also greatly reduce the value of a given thing, and may cause the early adopters to be actually worse off and more miserable on a daily basis. > Given the extent to which holdouts have benefited from the community, holdout behavior is perceived as parasitic and immoral behavior by adopters, while holdouts in turn deny any moral obligation and resent the methods that adopters use to increase adoption (such as, in the absence of formal controls, informal ones like bullying). > > This desperate need for there to be *a* victor, and the large technical benefits/costs to those who choose the winning/losing side, explain the (only apparently) disproportionate energy, venom, and intractability of holy wars. > > Perhaps if we explicitly understand holy wars as coordination problems, we can avoid the worst excesses and tap into knowledge about the topic to better manage things like language migrations.
Make a communication platform for nerds, whether it's [Usenet](!W), [FidoNet](!W), forums, Twitter, Github, Hacker News, Reddit or what have you, and sooner rather than later, there will surely be *holy wars*---those intractable perennial arguments over whether technology A or B sucks and why users of the other technology are not just simple-minded, but sociopathic, sinning against all that is good and pure. The recurrent 'holy war' phenomenon in computing is defined in the [Jargon File](!W) [as](http://www.catb.org/jargon/html/H/holy-wars.html "'The Jargon File (version 4.4.7): H: holy wars', Raymond 2003"): > flame wars...The paper by Danny Cohen that popularized the terms [big-endian and little-endian](!W "Endianness") in connection with the LSB-first/MSB-first controversy was entitled ["On Holy Wars and a Plea for Peace"](/doc/cs/algorithm/1981-cohen.pdf "Cohen 1981"). > > Great holy wars of the past have included [ITS](!W "Incompatible Timesharing System") vs.: [Unix](!W), Unix vs.: [VMS](!W "OpenVMS"), [BSD Unix](!W) vs.: [System V](!W), [C](!W "C (programming language)") vs.: [Pascal](!W "Pascal (programming language)"), C vs.: [FORTRAN](!W), etc. In the year 2003, popular favorites of the day are [KDE](!W) versus [GNOME](!W), [vim](!W "Vim (text editor)") versus [elvis](!W "Elvis (text editor)"), [Linux](!W) versus \[[Free](!W "FreeBSD")|[Net](!W "NetBSD")|[Open](!W "OpenBSD")\]BSD. Hardy perennials include [EMACS](!W) vs.: [vi](!W).^[Also an important one of this era: [version control systems](!W)---none vs [SCCS](!W "Source Code Control System") vs [RCS](!W "Revision Control System") vs [CVS](!W "Concurrent Versions System") vs [SVN](!W "Apache Subversion") vs [Monotone](!W "Monotone (software)") vs [Mercurial](!W) vs [Bitkeeper](!W) vs [git](!W), which then folded into code-hosting websites like [SourceForge](!W) vs [Google Code](!W) vs [Github](!W) vs [Gitlab](!W).] > > ...The characteristic that distinguishes holy wars from normal technical disputes is that in a holy war most of the participants spend their time trying to pass off personal value choices and cultural attachments as objective technical evaluations. This happens precisely because in a true holy war, the actual substantive differences between the sides are relatively minor. [The silent dog: where holy wars are *not*.]{.margin-note} One can probably name further ones. For example, before the Internet could become universal, there had to be the [Protocol Wars](!W); then, the great console wars of the 1990s were accompanied by equally endless and often acrimonious debates over the merits of the SNES vs the Sega Genesis (then the N64 vs the PS1, the PS2 vs XBox...). Or moving 2 decades ahead, social media like Usenet vs Slashdot vs digg vs Reddit vs Hacker News vs Twitter vs Mastodon, cryptocurrencies such as [Bitcoin](!W) vs [Ethereum](!W) (vs [Monero](!W) vs [Zcash](!W) vs...). We should also note what things *aren't* holy wars: the exact choice of string search algorithm in a tool like grep, one's choice of FPS vs RPG video game genre, how large a hard drive or how many CPU cores^[As opposed to which CPU *brand* or *architecture*.] to buy... There may be the occasional debate over these, but they will tend to be level-headed and productive in a way holy wars aren't. It's not mere "brand loyalty" either---there are countless brands out there with cults of followers but no holy wars, and to the extent they are debated, most shrug and conclude _De gustibus non est disputandum_. [Choices matter.]{.margin-note} Are holy wars just nerds being nerds? To some extent, sure, arguing is often a recreational activity; but this can't possibly be the whole explanation, because they descend into highly acrimonious flamewars that no one enjoys and from which everyone comes away embittered. And contra ESR, the technical differences are often quite substantial. (Even using his own examples, ITS vs Unix or C vs Pascal or Emacs vs vi, they embody dramatically different technical philosophies.) [Armin Ronacher](https://lucumr.pocoo.org/2019/12/28/open-source-migrates/ "Open Source Migrates With Emotional Distress"), complaining about the "emotional distress" used to encourage upgrades like Python 2 → 3 which appear to "under-deliver" direct benefits commensurate with all the _Sturm und Drang_, is perplexed by the whole matter, and can come up with no better explanation than holy wars coming from people enjoying "suffering together" & being "on the moral right side", which are radically inadequate (why suffer in the first place?) and he does not enquire any further, but revealingly notes in passing that "very few projects actually are large enough that a fork by some third party would actually survive".^[There are many examples of large-scale forks where the final result is that one project lives and the other dies, and the victor may well be adopted by the loser as the now-official version. The traditional example is the [EGCS fork of GCC](!W "GNU Compiler Collection#EGCS fork")---the FSF was unable to keep the original GCC viable, and ultimately blessed EGCS as the true GCC. Another oft-cited example is [XEmacs](!W) vs [GNU Emacs](!W): despite initial momentum---which competition prompted considerable GNU Emacs reform---XEmacs was ultimately unable to keep up with maintaining compatibility & feature parity, and was largely defunct by 2015.] [Network effects.]{.margin-note} What's the difference? The users of some kinds of technology benefit from more scale; while others are hurt by it---infrastructure is anything which gets more useful the more people use it, while applications get less useful. Holy wars (much like political debates) are always about infrastructure, or *platforms*, with network effects, and the indirect second-order effects are much more important than the first-order changes. One can often 'fork', but the [option of "exit"](!W "Exit, Voice, and Loyalty") is often a hollow one: what OS or programming language or binary number encoding you use matters to other people, and vice-versa. This leaves only 'voice'. People flame over their favorite MMO, but not their favorite Solitaire, because if a MMO (or any other multi-player game) can't get enough players, *no one* can play it; even two neighborhood kids arguing over video game consoles benefit if they can agree on which console is best, because then they can easily share games & save files & video game magazines. That you do them matters to others. In contrast, to others, what you do with them matters less. Once you have settled on Microsoft Excel rather than VisiCalc or Lotus-1-2-3 for your spreadsheet infrastructure, the contents of said spreadsheets don't matter nearly as much to other people as the fact that you are an Excel user. # (Inter)Network Effects
> Should array indices start at 0 or 1? My compromise of 0.5 was rejected without, I thought, proper consideration. > > [Stan Kelly-Bootle](!W)[^Bootle]
[^Bootle]: As quoted in _Expert C Programming: Deep C Secrets_, Peter Van der Linden 1994; while this is the earliest I found, Van der Linden says he does not know where it came from, suggesting it was already folklore by 1994. (It is not in Kelly-Bootle's [_The Devil's DP Dictionary_](https://archive.org/details/isbn_9780070340220/) or _The Computer Contradictionary_.) See also: ["LuaRocks is indented with 3 spaces."](https://github.com/luarocks/lua-style-guide/blob/7112321176dff1e2d1b97765c9801e83495b8797/README.md#indentation-and-formatting) [Grow the network, even with dirty tricks.]{.margin-note} If I use Python 3 and can convince everyone else to use Python 3 rather than 2, regardless of how much the Python 3 improvements themselves actually matter to them or how dishonest my arguments are, I am much better off.[^Norvig] The path of a programmer who uses the most common language is a blessed path: libraries abound, bugs programmers of lesser languages routinely encounter daily will mysteriously fail to occur in relatively bulletproof codebases, documentation always addresses his use-case with helpful snippets to copy-paste, APIs miraculously return data formatted in a way convenient for his default data structures, which their rich featureful IDE will tab-complete the code for them, Stack Overflow will overflow with tips & tricks for any glitches (which come almost as fun puzzles, bagatelles between the real work), for all is for the best in this best of all possible worlds...^[And so on, eg. if we are discussing a cryptocurrency instead of a programming language, all of that is true in terms of tooling, but also in terms of interacting with the rest of the world: cryptocurrencies whose fanatical users refuse to bow to, say, [KYC](!W "Know your customer") requirements will maintain a KYC-less ecosystem, and can enjoy genuine privacy rather than a fig leaf of anonymity solely on the blockchain but nowhere else.] There is a large personal and collective investment in explicit & [tacit knowledge](!W) (one must even learn to think differently for specific tools), but once the price is paid, life is good. While those who venture away from the beaten path quickly discover that how many of their own libraries they will have to write, how under-specified many standards are, how many assumptions their OS or API makes (the better to stab them in the back), how few signposts there are or fellows to ask advice, how quickly the yak shaving kills the beast by a thousand cuts (and the more diligent they are about filing bugs or patches, the deeper they are sucked into the mire), and in their darkest moments, their faith wavers as they wonder if perhaps their favorite language is not the best programming language in the world. [^Norvig]: Norvig, [on language choice](https://norvig.com/21-days.html "Teach Yourself Programming in Ten Years: Appendix: Language Choice") makes this his first criterion: > *Use your friends.* When asked "what operating system should I use, Windows, Unix, or Mac?", my answer is usually: "use whatever your friends use." The advantage you get from learning from your friends will offset any intrinsic difference between OS, or between programming languages. Also consider your future friends: the community of programmers that you will be a part of if you continue. Does your chosen language have a large growing community or a small dying one? Are there books, web sites, and online forums to get answers from? Do you like the people in those forums? [Not just zero, but negative-sum.]{.margin-note} Those positive network effects are clear enough. But one could acknowledge them and think they are not adequate to drive the fervor of a holy war. So let's consider a subtler problem, which is how the success of one platform *indirectly harms other platforms* by wresting away shared resources. ## Bitrot [Always in motion.]{.margin-note} The phrase [*bitrot*](!W "Software rot") encapsulates the problem of network effects for such platforms. Programs do not bitrot because they changed on disk, or because the mere passage of time rendered them buggy (with occasional exceptions like [Y2K](!W "Year 2000 problem")). Bitrot happens when the rest of the world changes around the program. APIs change a field name; or the name remains the same but the meaning changes; or an obscure function is broken by a change somewhere else and no one noticed it until your program tried to use it as always and it broke; or a version number changed and something else will no longer work because it assumed the old version number and is unsure if the new one is safe; or you need a particular program which can't be installed without upgrading existing programs (thereby potentially breaking thousands of others); or a system backup uses keys which quietly expired or ran out of space or was never updated to backup a new server as well. The implicitness of dependencies and system interactions means that there is no way to avoid taking 'the path of least resistance' because one does not realize there is even a path or that one is being locked into anything. ("What do you mean, 'a [byte](!W) isn't always 8 bits'‽") Bitrot must be actively fought: if you are serious about not building in too many dependencies, you need tricks like [chaos engineering](!W), or deliberately crashing services which [exceed their expected reliability.](https://sre.google/sre-book/service-level-objectives/ "The Google SRE Handbook: Chapter 4 - Service Level Objectives") [No silver bullet.]{.margin-note} Technical fixes like type systems can help reduce bitrot by identifying precisely where things have changed dangerously, but there is always a system outside the type system, and the only truly end-to-end test is the end-user.^[A corollary here is that a system which is also created end-to-end, like a deep learning system, may be able to reduce bitrot. In "Software 2.0", conventional code like Tensorflow (itself notoriously bitrot prone) is minimized as much as possible in favor of learning all functionality. Consider a NN like [GPT-3](https://arxiv.org/abs/2005.14165#openai "'GPT-3: Language Models are Few-Shot Learners', Brown et al 2020") when used for code completion or spreadsheets: even if the text formatting changes, which would immediately bitrot a conventional software system reliant on brittle regexps or parsers, they can be given examples to do few-shot learning (["prompt programming"](/gpt-3#prompts-as-programming "'GPT-3 Creative Fiction § Prompts As Programming', Gwern 2020")) & smoothly adapt on the fly to "do what I mean", and can retrain on future data without human intervention. As the ultimate input/output pairs change, it can retrain itself end-to-end.] [What gets used, works. What doesn't...]{.margin-note} There is no cheap way to avoid bitrot: what gets used gets maintained. Programs which are in wide use avoid bitrot less because of any intrinsically superior design but because of the ecosystem *around them*: because breaking changes are immediately detected by a cutting-edge user (sparing all the others), or the breaking changes are never made in the first place (because they tested & detected it, or were well-aware of the consequences, or simply because it broke their own system first!), and because everyone adapts their things to fit the popular program rather than vice versa. Thus constantly refreshed and updated, the program+ecosystem avoids bitrot---like living flesh, which avoids rotting (decomposition by bacteria etc.) by constant renewal. ## Bitcreep
> We all have strength enough to endure the misfortunes of others. > > [François de La Rochefoucauld](!W "François de La Rochefoucauld (writer)"), maxim #19
[Dying systems 'bitrot', growing ones 'bitcreep'.]{.margin-note} The flip side of bitrot is what we might call *bitcreep*: because there is only so much time and energy to go around, a system which avoids bitrot will also experience 'bitcreep', where other programs begin to 'bitrot' in that they increasingly assume, depend, and are tested only with that system, and that distortion gradually [creeps](!W "Creep (deformation)") through the ecosystem. In a system with heavy bitcreep, getting things done in any way other than the dominant way means thrashing around in what is not so much a 'Turing tarpit' as a La Brea tarpit, diverting new programs towards it. It becomes a black hole, bending space around it; once a program has built in enough implicit and explicit dependencies, it has passed the event horizon, and it can no longer escape (because it would be easier to rewrite the program from scratch). And because the alternatives are *not* being used, they are not getting maintained, which means that as bitcreep spreads, anyone interacting with older programs in a way that *used* to work will discover their program has suffered bitrot, without having changed at all; all past investments are jeopardized, rotting away invisibly.[^TPUs] (This suggests there are two evolutionary strategies for systems: to be so simple that no one would replace them, and to be so complex no one *could* replace them.) [^TPUs]: Bitcreep's inverse is trying to use an old system for new purposes, which reveals hidden limits & assumptions. For example, compared to Nvidia GPUs, Google TPU clusters are [remarkably reliable & easy to use](https://www.yitay.net/blog/training-great-llms-entirely-from-ground-zero-in-the-wilderness)---if you are a Googler. However, if you are not, and you try to use Google TPUs, you will run into the inverse of bitcreep. When I and other generative AI hobbyists in Tensorfork tried to use Google's generous TPU research grant program in 2019 (mostly to train [GPT-2](/gpt-2 "‘GPT-2 Neural Network Poetry’, Branwen & Presser 2019") & [StyleGAN](/face "‘Making Anime Faces With StyleGAN’, Gwern 2019") DL models), despite TPUs being rented to the general public for years at that point, we ran into countless papercuts. Whether it was unclear or outdated documentation, or undocumented behavior like resetting every day, or opaque permissions problems, or needing to use a low-level command that no Googler thought an outsider would need to use... Some issues were regular bugs but many were not problems inside Google, because they automatically had permission or always used internal frameworks which handled problems or they all knew something already etc. ["Embrace, extend, extinguish."]{.margin-note} Two old examples are GCC's monolithic architecture, and the Linux kernel deliberately breaking internal binary compatibility, with systemd looking like a third. RMS's justification for not designing [GCC](!W "GNU Compiler Collection") in the logical way of modular components repeatedly passing over the inputs is that such a well-designed architecture would make GCC *too* easy to modify---hostile entities could provide closed-source frontends/backends and damage the spread of FLOSS, while if GCC was made nigh-unmaintainable for closed-source modifications, they would be forced to open-source them; whatever the merits of this view, it is true that GCC did not suffer much in the way of 'Androidization', and the non-GPL highly-modularized [LLVM](!W) compiler had to start from scratch. (On the other hand, LLVM's immense uptake by both industry and academia highlights the costs of such a "my way or the highway" approach.) [Linux kernel devs](https://www.kernel.org/doc/Documentation/process/stable-api-nonsense.rst) justify the inability to create proprietary modules by arguing that by only supporting modules that are incorporated into the Linux source code base (and thus must be FLOSS-licensed by the owner), they can fix bugs, security vulnerabilities, and rewrite the kernel in general, delivering much greater benefits in the long run than if they froze everything to avoid breaking opaque binary blobs released years ago & never modified since^[One might say that the kernel devs could just go ahead and break those blobs in case they need to, but this overlooks the problem of power & economics. They *can*... but only as long as few people create such blobs (and there is thus little value to enabling those blobs). As the joke goes, "if you owe the bank a million dollars, you have a problem; but if you owe the bank a billion dollars, the *bank* has a problem." The more blobs, the closer one comes to 'the tail wagging the dog'.]; this point of view has few fans in the GPU and Android industries, among others, and Google expends enormous efforts maintaining a permanent fork of the Linux kernel for Android & creating an ad hoc stable ABI for downstream vendors (in addition to [rewriting](https://arxiv.org/abs/1702.01715#google "‘Software Engineering at Google’, Henderson 2017") the equivalent of its entire codebase every few years). And the Linux kernel itself is undergoing a slow-motion process of bitcreep in the 'outside' form of reimplementation in Rust, on the one hand, and the 'inside' form of implementing new programs in [eBPF](!W) (once a simple DSL for filtering network packets, which can now [hook functions for tracing](https://filipnikolovski.com/posts/ebpf/), has grown a JIT and [runs on MS Windows](https://github.com/microsoft/ebpf-for-windows) may soon even be able to *send* packets, and is starting to look like reinventing [SPIN](!W "SPIN (operating system)") the hard way). A third (still in progress) example might be how [systemd](!W) has metastasized in Linux ([history](https://blog.darknedgy.net/technology/2020/05/02/0/ "systemd, 10 years later: a historical and technical retrospective")): a boot script alternative has become involved in everything from desktop environments to audio daemons; increasingly, the only way to do something is via systemd. Things which used to be compartmentalized and usable with alternative tools now assume the user uses only systemd and tailor their capabilities to systemd. Increasingly, rooting out systemd ceases to be possible: it simply breaks too many things, even things which ostensibly have nothing to do with the original purpose of booting up Linux, and it has now ["infected"](https://x.com/pid_eins/status/1461713069130854412 "I just realized that systemd has already infected Solaris—just look at this: https://docs.oracle.com/cd/E88353_01/html/E37852/os-release-5.html (yes, that file is a systemd invention). And that since 2018 apparently. Ha, we are coming after you, UNIX! Hmm, and apparently FreeBSD already has it since 2019, too: https://freebsd.org/cgi/man.cgi?query=os-release&apropos=0&sektion=0&manpath=FreeBSD+13.0-RELEASE&arch=default&format=html") [Solaris](!W "Oracle Solaris") & FreeBSD as well. Someone who discovers the hard way that a key script of theirs has been broken by systemd, through no fault of their own, and that systemd cannot be removed without bricking their system and the last OS version without systemd was many years ago and would break a dozen other programs and be extremely insecure, and who dislikes systemd for other reasons, could be forgiven for being upset and seeking solace [in video games](https://web.archive.org/web/20150525235536/http://www.bloodbathsoftworks.com/xylemon/xlennart.php "'XLennart', Gethyn ThomasQuail 2014-11-02"). And for a partisan of systemd, it is not enough that systemd succeed---others must fail. [One man's bitcreep is another man's standardization.]{.margin-note} Bitrot/bitcreep are themselves quasi-neutral, neither clearly good nor bad.[^no-evolution] The rest of the world is constantly changing for good reasons, and bitrot is an unavoidable consequence of that; it also changes for bad reasons, and that bitrot is bad. Fighting bitrot requires a constant investment in running a program, and it may be better to let bitrot happen and move on; too much investment in backwards compatibility & legacy systems can be a major mistake (consider [Makefiles](!W "Make (software)") remaining [tab](!W "Tab key#Tab characters")-sensitive after this was clearly a mistake because the creator was too worried about hurting his literally dozens of users, or the burden on Microsoft Windows to try to run everything back to DOS no matter how buggy). Bitcreep is bad when it is forcing through an inferior technology, but it also is just what convergence and standardization looks like. Fighting bitcreep requires a constant community-wide investment in exploring the alternatives and enforcing, and it may be better to just let it happen and move on to more important things. It is not a bad thing if, say, the definition of a byte as 8 bits gradually bitcreeps through an ecosystem---because how useful is it *really* to make the software ecosystem byte-agnostic and support old systems with 7-bit (much less 48-bit) bytes? [^no-evolution]: One might ask why software ecosystems don't evolve steadily towards higher quality. A major reason that [software (and corporations) do not evolve](/backstop "'Evolution as Backstop for Reinforcement Learning', Gwern 2018") seems to be that, as [Alan Kay](!W) puts it, "computing is pop culture". Whatever progress and experience slowly accrete over time is swamped by random exogenous events. For programming languages in particular, it is better to be lucky than good. Good software practice has too little correlation with fitness or profits, perhaps because everyone and everything cycles over way too quickly, for good practices to culturally evolve. It was infinitely more important for JavaScript to be available in web browsers 'natively' than for any of its other design choices to make sense; it was infinitely more important for Facebook, say, to find a social network niche than for it to not use PHP---the constant factor cost of technical debt could be trivially paid off once they were a multi-billion-dollar success. (As absurd as it sounds to make bad choices early on such that later on FB would have to write their own [PHP compiler](!W "HHVM") and build [their own better language](!W "Hack (programming language)") on top of it, well, nevertheless, Mark Zuckerberg is rich and you are not, no matter how much more refined your technical preferences are.) It's just [Price's equation](!W). The lower the covariance between fitness and replicated phenotype, the less selection is possible. Shifting environments, poor replication of 'culture' & programmers, business models and ecosystems creating far higher fitness differentials than minor choices of language or library... then combine this with breathtakingly high 'mutational targets' ("this year, we are Agile!")... Progress is slow, and every so often something like the PC or web browser or smartphone comes along, creating a mass extinction event or brand new niche to colonize---and everything must start over. # Platform Life and Death
> All cruelty springs from weakness. > > [Seneca the Younger](!W) ([_De Vita Beata_](!W), [Book 3](https://en.wikisource.org/wiki/Of_a_Happy_Life/Book_III "'Of a Happy Life: Book 3', Seneca 1990"))
[Total war: best never to begin, but once begun---win!]{.margin-note} The consequence of bitrot/bitcreep, however, is to raise the stakes of personal choices dramatically, and turn it into a [game of chicken](!W "Chicken (game)"). Behind bitcreep lies the threat: "join or else". If you don't pick the winner, your knowledge will become obsolete, your code useless, and you may even be fired or become unemployable. The more encompassing a platform, the more important it is to convert others, to not just benefit from positive externalities, but to avoid bitrot, and a competitor causing bitcreep.^[Which is not to say there aren't many other benefits: you can look cool by using a new platform, look clever by criticizing the old platform while being immune to criticism yourself (the new thing's flaws having not yet manifested or become widely known), become famous writing about & promoting a new platform, get hired to work on it...] Thus the holy war hierarchy: endian-ness > OS > language > editor > desktop... terminating in mock-serious debates like tabs vs spaces. (And also why some holy wars end. People have ceased to debate Emacs vs vim not because any consensus was reached or either one disappeared, but because both are now niche editors which are more threatened by a common enemy---giants like [Visual Studio Code](!W) or [Eclipse](!W "Eclipse (software)") or [Sublime Text](!W).^[Indeed, vi/vim users are now natural allies of Emacs users simply because they both have a common enemy: bitcreeping monolithic IDEs rendering it impossible for third-party editors to effectively integrate with compilers etc. If one can still interoperate with something successfully, then so too can the other (see eg. the [Language Server Protocol](!W)).]) What matters is less the merits of the victor than that there *is* a victor: "Worse Is Better"---enough JavaScript programmers can push a noodle, or NodeJS, around the equator a dozen times. (Surely at this point the incompatibilities, conversion overhead, corrupted data, and programming headaches of having every other system switch endianess have vastly outweighed any conceivable disadvantage of standardizing on the worse of big/little-endianess?) [A bad compromise makes everyone unhappy.]{.margin-note} In the case of Python 2 vs Python 3, both are large communities, and could potentially go on indefinitely. The persistence of 2 Pythons is a perpetual headache for all Python programmers, as it does not just forfeit the benefits of scale (2 half-communities add up to less than 1 whole-community), but inflicts complexity and error on everyone dealing with Python, who must make sure they have multiple versions safely installed and always use the right one etc. And because of that, Python 2 will enjoy the benefits of being able to cause bitcreep in other systems, which must work around and cope with Python 2 (rather than vice versa). This is a passively-enjoyed subsidy, a 'rent', as it were, on the intellectual property built by past Python programmers (ironically, many of whom are behind Python 3). Why should the Python 2 users invest in learning all the new idiosyncrasies of Python 3? The users of Python 2 enjoy the benefits of a quiet life as their programs do not bitrot---but that comes at the steep triple cost of the opportunity cost of the Python 3 changes (so easy to forget in all this discussion of network effects), of bitcreep, and of dividing the community. And almost all of this triple cost is borne by others rather than the holdouts! (It would be hard to calculate, but considering the imbalance between library/tool maintainers and their potentially millions of downstream users, it seems likely that the ratio of cost:benefits is probably imbalanced by multiple orders of magnitude.) A Python 3 developer could be forgiven for being upset.^[Early-adopters & computing pioneers are our bridge to the future---because we knife them in the back by taking their work without giving, and then step across the muddy ditch on top of their bodies.] [War, by other means.]{.margin-note} Given all this, it's no wonder that arguments might become less than polite or some [resort to bullying](https://gigamonkeys.wordpress.com/2009/10/16/coders-c-plus-plus/ "‘C++ in Coders at Work’, Seibel 2009") or promise all things to all people: the stakes can in fact be quite high, and even if not a matter of life & death, exactly, make a considerable difference to one's quality of life. Still, such decentralized uncoordinated flamewars & bullying is a bad idea; if it worked, perhaps it could be justified, but the history of freelance flamewars shows that it almost *never* works and it is merely destructive. People should ["be nice, at least until you can coordinate meanness"](https://slatestarcodex.com/2016/05/02/be-nice-at-least-until-you-can-coordinate-meanness/). # Explaining Holy Wars This explains why holy wars exist for some technical things but not others, why they persist, why they are often the most vicious between the most similar groups (it is not ['narcissism'](!W "Narcissism of small differences") but pragmatism, because they are competing for the same scarce resources)^[eg. a [Scheme](!W "Scheme (programming language)") Lisp programmer vs a [Common Lisp](!W) programmer: if one Lisp programmer decides to change his preferred language, he's probably not going to switch to Java!], why they are so intellectually dishonest and intractable, why participants feel forced to resort to bullying and emotional abuse, why they do in fact matter a lot, and how they tend. It is because bitcreep is real, but the costs & benefits unevenly distributed, often relatively modest, affect a specific group of users with few choices, not choosing is worse than either choice, and there is no mechanism which can force a community-wide choice (even when it is more important that *a* choice gets made rather than which). This perspective also suggests some ways to reduce the damage. # Coordinating Migrations Such coordination problems are not unique to computing. No one wants to be the first penguin off the iceberg---but those fish won't catch themselves. Instead of ad hoc upgrades, and resorting to flamewars, propaganda, and bullying of people who have different needs or understandably simply do the selfish thing, could there be better ways? [Appeal to public-spiritedness.]{.margin-note} One better way would be to just raise awareness of the costs. Is an upgrade or alternative *really* necessary? If it really is, then holdouts like [Armin Ronacher](#ronacher-2019) should be made aware of the costs they are creating in a reasoned manner, instead of making outraged noises at them and bullying them online with bogus arguments. [Economic incentives?]{.margin-note} Given the need to do an upgrade, another option is to use economics approaches: create a fund to pay maintainers to do the transition, or sponsor a fork if need be (which can be returned to the original maintainers once the transition is a fait accompli and the status quo now creates bitcreep in the other direction). An [assurance contract](!W) might be a useful social technology here (paying maintainers for their upgrade costs is smaller than the total gains and so is [Kaldor-Hicks efficient](!W "Kaldor-Hicks efficiency")), or perhaps simply a public commitment to do the transition at a specific date, similar to Python 2 being EOLed. [Deliberately centralize to solve coordination problems.]{.margin-note} From the perspective of bitcreep, attempts at making peace by bending over backwards to improve modularization and try to support multiple ecosystems in parallel are nothing but self-sabotaging folly: finding a 'commanding height' to do the transition may be the single most effective strategy, and it self-executes by default. (Such a strategy may seem underhanded, but in coordination problems, kindness is cruelty.) The more "there is only one way to do it" that an ecosystem is, the easier coordination is. [Go](!W "Go (programming language)") may not be a great language, but isn't it nice that there's a `gofmt` tool everyone uses (and forces everyone else to use), which consistently formats Go code *somehow*, and frees Go programmers from the tyranny of freedom? Indeed, perhaps we should reconceive the real role of a "[BDFL](!W)" or maintainer as not being ordinary everyday project maintenance or organizing, but simply serving as a final authority to force the community into a new better equilibrium---a role the more valuable the more rarely exercised. # See Also
- [Commoditize Your Complement](/complement "'Laws of Tech: Commoditize Your Complement', Gwern 2018"){.backlink-not} - [Bitcoin is Worse is Better](/bitcoin-is-worse-is-better){.backlink-not} - [Choosing Software](/choosing-software){.backlink-not}/[Resilient Haskell Software](/resilient-software){.backlink-not} - [Predicting Google Closures](/google-shutdown){.backlink-not} - [In Defense of Inclusionism](/inclusionism){.backlink-not}/[Wikipedia & Other Wikis](/wikipedia-and-other-wikis "'Wikipedia and Other Wikis', Gwern 2009"){.backlink-not} - [On Having Enough Socks](/socks){.backlink-not} - ["Timing Technology: Lessons From The Media Lab"](/timing){.backlink-not}
# External Links - ["Common Lisp: The Untold Story"](https://www.nhplace.com/kent/Papers/cl-untold-story.html), Pitman 2008 - ["The Most Important Scarce Resource is Legitimacy"](https://vitalik.eth.limo/general/2021/03/23/legitimacy.html), Vitalik Buterin - ["Keep Your Identity Small"](https://paulgraham.com/identity.html), Paul Graham - ["Roads and Bridges: The Unseen Labor Behind Our Digital Infrastructure"](https://www.fordfoundation.org/wp-content/uploads/2016/07/roads-and-bridges-the-unseen-labor-behind-our-digital-infrastructure.pdf), Nadia Asparouhova 2016; [_Working in Public: The Making and Maintenance of Open Source Software_](https://www.amazon.com/gp/product/0578675862/), 2020 - ["Dear Google Cloud: Your Deprecation Policy is Killing You"](https://steve-yegge.medium.com/dear-google-cloud-your-deprecation-policy-is-killing-you-ee7525dc05dc), Steve Yegge - ["Why Johnny Won’t Upgrade"](https://jacquesmattheij.com/why-johnny-wont-upgrade/), Jacques Mattheij 2020; ["Never update anything"](https://blog.kronis.dev/articles/never-update-anything) - ["Tripping over the potholes in too many libraries"](https://rachelbythebay.com/w/2020/08/09/lib/) - ["Strategy Letter III: Let Me Go Back!"](https://www.joelonsoftware.com/2000/06/03/strategy-letter-iii-let-me-go-back/), Joel Spolsky 2000 (on lockin vs lockout) - ["Forking Protocol: Why, When, and How to Fork an Open Source Project"](https://jamesdixon.wordpress.com/2009/05/07/forking-protocol-why-when-and-how-to-fork-an-open-source-project/), James Dixon - ["Moving Google toward the mainline"](https://lwn.net/Articles/871195/); ["It takes us about 1 man-year to upgrade GHC at Facebook."](https://github.com/haskell/core-libraries-committee/issues/12 "Decide on core principles against which to judge breaking changes #12: Simon Mar comment") (a dark side to [in-house expertise](https://danluu.com/in-house/)---even giants must fear tech debt & bitcreep) - ["The Economics of Programming Languages"](https://www.welton.it/articles/programming_language_economics.html), Nelton 2005 - ["Hyrum’s Law: An observation on Software Engineering"](https://www.hyrumslaw.com/) - ["Volatile Software"](https://stevelosh.com/blog/2012/04/volatile-software/), Steve Losh - ["We invested 10% to pay back tech debt; Here's what happened"](https://blog.alexewerlof.com/p/tech-debt-day) - ["Why and How We Retired Elm at Culture Amp"](https://kevinyank.com/posts/on-endings-why-how-we-retired-elm-at-culture-amp/) - ["Every feature request has a constituency---some group who wants it implemented, because they benefit from it. Simplicity does not"](https://alexgaynor.net/2020/nov/30/why-software-ends-up-complex/) - ["The Maddest My Code Made Anyone" comments](https://news.ycombinator.com/item?id=35945941) - ["finally. `#embed`"](https://thephd.dev/finally-embed-in-c23), JeanHeyd Meneide - ["My resignation letter as `R7RS-large` chair"](https://groups.google.com/g/scheme-reports-wg2/c/xGd0_eeKmGI/m/q-xM5fbuAQAJ), John Cowan - ["#2,846: Daylight Saving Choice"](https://xkcd.com/2846/), XKCD - ["My 20 Year Career is Technical Debt or Deprecated"](https://startuphustlenews.com/p/my-20-year-career-is-technical-debt) - [Half-life of knowledge](!W): ["The Dollars and Sense of Continuing Education"](/doc/cs/1966-jones.pdf), Jones 1966 - ["Let 1,000 flowers bloom. Then rip 999 of them out by the roots."](https://gigamonkeys.com/flowers/) - ["Yesterday's technology tomorrow"](http://itre.cis.upenn.edu/~myl/languagelog/archives/000606.html) - ["How _Doom_ didn't kill the Amiga: A detailed account of the rise and fall of an Amiga zealot"](https://www.datagubbe.se/afb/ "Carl Svensson 2024-02-04") - ["Using Rust at a startup: A cautionary tale"](https://www.medium.com/using-rust-at-a-startup-a-cautionary-tale-42ab823d9454), 2022; ["I love building a startup in Rust. I wouldn't pick it again."](https://www.propelauth.com/post/i-love-building-a-startup-in-rust-i-wouldnt-pick-it-again), 2023; ["Leaving Rust gamedev after 3 years"](https://loglog.games/blog/leaving-rust-gamedev/), 2024; ["For some reason, the whole Rust versus C discussion has taken almost religious overtones in certain areas..."](https://www.youtube.com/watch?v=OM_8UOPFpqE&t=510s "Linus Torvalds") - ["This Post Is Not About Python"](https://jerf.org/iri/post/2024/not_about_python/ "Jeremy Bowers 2024-10-07") - ["The Failed Migration of Academic Twitter"](https://arxiv.org/abs/2406.04005), Wang et al 2024 - ["Be Careful About What You Dislike"](https://lucumr.pocoo.org/2016/11/5/be-careful-about-what-you-dislike/) - ["20 years of Git. Still weird, still wonderful"](https://blog.gitbutler.com/20-years-of-git/) - ["Shared Misery"](http://metamodular.com/Psychology/shared-misery.html) - ["Perl's decline was cultural"](https://www.beatworm.co.uk/blog/computers/perls-decline-was-cultural-not-technical) - ["Empathy and subjective experience in programming languages"](https://lexi-lambda.github.io/blog/2019/10/19/empathy-and-subjective-experience-in-programming-languages/); ["The reactionary conservatism of the median programmer"](https://lexi-lambda.github.io/blog/2025/05/29/a-break-from-programming-languages/#the-reactionary-conservatism-of-the-median-programmer) - Discussion: [HN](https://news.ycombinator.com/item?id=23753477), [2](https://news.ycombinator.com/item?id=38161743)