“The Existential Risk of Math Errors § Unreliability of Programs”, 2012-07-20 (similar):
Mathematical mistake/error-rates limit our understanding of rare risks and ability to defend against them
I have never written an equation or line of code that I was 100% confident of, or which I thought had less than a 1-in-trillions chance of it being wrong in some important way. Software & real-world systems are too complex & fragile.
Every part of my understanding, the hardware, or the real-world context is less reliable than 1-in-trillions.
Let’s consider potential problems with our understanding of even the most trivial seeming arithmetic comparison checking that ‘x + x = 2x’.