--- title: "On Seeing Through and Unseeing: The Hacker Mindset" description: "Defining the security/hacker mindset as extreme reductionism: ignoring the surface abstractions and limitations to treat a system as a source of parts to manipulate into a different system, with different (and usually unintended) capabilities." thumbnail: /doc/cs/shell/2024-01-17-cmatrix-matrixstylescreenscroll.png thumbnailText: "Terminal screenshot of cmatrix utility running in rxvt-unicode, emulating the famous The Matrix dropping green-text screen effect." thumbnailCSS: "outline invert-not" created: 2012-12-09 modified: 2021-05-04 status: finished previous: /turing-complete next: /wifi confidence: highly likely importance: 6 cssExtension: dropcaps-kanzlei ...
> To draw some parallels here and expand [Dullien 2017](/turing-complete#dullien-2017 "Weird machines, exploitability, and provable unexploitability"), I think [unexpected Turing-complete systems and weird machines](/turing-complete "'Surprisingly Turing-Complete', Branwen 2012") have something in common with heist movies or cons or stage magic: they all share a specific paradigm we might call the *security mindset* or *hacker mindset*. > > What they/OP/security/[speedrunning](https://en.wikipedia.org/wiki/Speedrunning)/hacking/[social-engineering](https://en.wikipedia.org/wiki/Social_engineering_\(security\)) all have in common is that they show that the much-ballyhooed 'hacker mindset' is, fundamentally, a sort of reductionism run amok, where one ['sees through'](/doc/philosophy/epistemology/2012-sistery-tryingtoseethrough.html) abstractions to a manipulable reality. > Like Neo in the _Matrix_---a deeply cliche analogy for hacking, but cliche because it resonates---one achieves enlightenment by seeing through the surface illusions of objects and can now see the endless lines of green code which make up the Matrix, and vice-versa. (It's maps all the way down!) > > In each case, the fundamental principle is that the hacker asks: "here I have a system _W_, which pretends to be made out of a few [Xs](https://github.com/kdeldycke/awesome-falsehood "Falsehoods Programmers Believe About X"); however, it is **really** made out of many _Y_, which form an entirely different system, _Z_; I will now proceed to ignore the _X_ and understand how _Z_ works, so I may use the _Y_ to thereby change _W_ however I like".
Abstractions are vital, but like many living things, dangerous, because [abstractions always leak](https://www.joelonsoftware.com/2002/11/11/the-law-of-leaky-abstractions/ "The Law of Leaky Abstractions"). ("You're very clever, young man, but it's reductionism all the way down!") This is in some sense the opposite of a mathematician: a mathematician tries to 'see through' a complex system's accidental complexity up to a simpler more-abstract more-true version which can be understood & manipulated---but for the hacker, all complexity is essential, and they are instead trying to *un*see the simple abstract system down to the more-complex less-abstract (but also more true) version.^['Thinking outside the box' can be this, but often isn't. This is a specific pattern of reductionism, and many instances of 'thinking outside the box' are other patterns, like putting on another layer, or eliminating the systems in question entirely.] (A mathematician might try to transform a program up into successively more abstract representations to eventually show it is trivially correct; a hacker would prefer to compile a program down into its most concrete representation to [brute force all execution paths](/forking-path "'Technology Forecasting: The Garden of Forking Paths', Branwen 2014") & find an exploit trivially proving it incorrect.) # Confirmation Bias
> [Uncle Milton Industries](!W "Milton Levine") has been selling [ant farms](!W "Formicarium") to children since 1956. Some years ago, I remember opening one up with a friend. There were no actual ants included in the box. Instead, there was a card that you filled in with your address, and the company would mail you some ants. > > My friend expressed surprise that you could get ants sent to you in the mail. I replied: 'What's really interesting is that these people will send a tube of live ants to anyone you tell them to.' > > [Bruce Schneier](!W), ["The Security Mindset"](https://www.schneier.com/blog/archives/2008/03/the_security_mi_1.html) (2008); cf. [DNS](https://www.tbray.org/ongoing/When/202x/2022/06/02/Dangerous-Gift), [Mormons/JVs](https://twitter.com/_JeanLannes/status/1687649736356982784)
![](/doc/philosophy/epistemology/1888-flammarion-latmospheremetereorologiepopulaire-theflammarianengraving.jpg "The 1888 Flammarion engraving by Camille Flammarion (WMF), illustrating, using a pastiche of a Renaissance woodblock engraving, a medieval missionary peeking out from Earth under the celestial sphere and seeing the mechanics of the universe for the first time."){.float-right .invert-not} Ordinary users ask only that all their everyday examples of Ys transforms into _Z_ correctly; they forget to ask whether all and *only* correct examples of Ys transform into correct Zs, and whether only correct Zs can be constructed to become Ys. Even a single 'anomaly', apparently trivial in itself, can indicate the everyday mental model is not just a little bit wrong, but *fundamentally* wrong, in the way that Newton's theory of gravity is not merely a little bit wrong and just needs a quick patch with a fudge factor to account for [Mercury](!W "Two-body problem in general relativity#Anomalous precession of Mercury") or that NASA management's mental model of O-rings was [not merely](!W "Space Shuttle Challenger disaster") in need of a minor increase in the thickness of the rubber gaskets[^Feynman]. [^Feynman]: [Feynman](https://www.nasa.gov/history/rogersrep/v2appf.htm "Appendix F: Personal Observations on the Reliability of the Shuttle"): > The phenomenon of accepting for flight, seals that had shown erosion and blow-by in previous flights, is very clear. The Challenger flight is an excellent example. There are several references to previous flights; the acceptance and success of these flights are taken as evidence of safety. But erosion and blowby are not what the design expected. They are warnings that something is wrong. The equipment is not operating as expected, and therefore there is a danger that it can operate with even wider deviations in the unexpected and not thoroughly understood way. The fact that this danger did not lead to catastrophe before is no guarantee that it will not the next time, unless it is completely understood. When playing Russian roulette the fact that the first shot got off safely is little comfort for the next. The origin and consequences of the erosion and blow-by were not understood. They did not occur equally on all flights and all joints; sometimes more, and sometimes less. Why not sometime, when whatever conditions determined it were right, still more leading to catastrophe? > > In spite of these variations from case to case, officials behaved as if they understood it, giving apparently logical arguments to each other often depending on the "success" of previous flights... # Atoms
> Every drop of blood has great talent; the original cellule seems identical in all animals, and only varied in its growth by the varying circumstance which opens now this kind of cell and now that, causing in the remote effect now horns, now wings, now scales, now hair; and the same numerical atom, it would seem, was equally ready to be a particle of the eye or brain of man, or of the claw of a tiger...The man truly conversant with life knows, against all appearances, that there is a remedy for every wrong, and that every wall is a gate. > > [Ralph Waldo Emerson](!W), "Natural History Of Intellect", 1893^[[pg441--442](https://quod.lib.umich.edu/e/emerson/4957107.0012.001/1:15.1?rgn=div2;view=fulltext), _The complete works of Ralph Waldo Emerson: Natural history of intellect, and other papers_, Vol. 12]
It's all "atoms and the void"^["By convention sweet is sweet, bitter is bitter, hot is hot, cold is cold, color is color; but in truth there are only atoms and the void." Incidentally, [Democritus's](!W "Democritus") other famous quote on atomism is a pun: "For 'Tragedy' [_τρ**α**γωδία_] and 'Comedy' [_τρ**υ**γωδία_] come to be out of the same letters." (As quoted/paraphrased by Aristotle, Book 1, [_On Generation and Corruption_](!W); for defense of the interpretation that this is wordplay & not merely a generic observation about alphabetic writing, see [West 1969](/doc/philosophy/ontology/1969-west.pdf "An Atomist Illustration In Aristotle").)]: - In hacking, a computer pretends to be made out of things like 'buffers' and 'lists' and 'objects' with rich meaningful semantics, but really, it's just made out of bits which mean nothing and only accidentally can be interpreted as things like 'web browsers' or 'passwords', and if you move some bits around and rewrite these other bits in a particular order and read one string of bits in a different way, now you have bypassed the password. - In speed running (particularly TASes), a video game pretends to be made out of things like 'walls' and 'speed limits' and 'levels which must be completed in a particular order', but it's really again just made out of bits and memory locations, and messing with them in particular ways, such as deliberately overloading the RAM [to cause](https://threadreaderapp.com/thread/1148361355130527748.html) [memory allocation](https://www.halopedia.org/Overload_Glitch_\(Halo_3\)) errors, can give you infinite 'velocity' or shift you into [alternate coordinate systems in the true physics](https://www.youtube.com/watch?v=wjge1bVobN0 "How Mario 64 was solved using parallel universes - Super Mario 64 Tool-Assisted Speedrun Explained"), allowing enormous movements in the supposed map, giving shortcuts to the 'end'^[A fictional example from _[Ender’s Game](!W)_ is worth noting: if victory in Battle School is defined by 4 soldiers at the corner of the enemy gate & someone passing through, then why not---shades of [Eurisko](!W)---skip fighting entirely & go straight for the gate?] of the game. - In robbing a hotel room, people see 'doors' and 'locks' and 'walls', but really, they are just made out of atoms arranged in a particular order, and you can move some atoms around more easily than others, and instead of going through a 'door' you can just cut a hole in the [wall](!W "Drywall")[^burglar] (or ceiling) and obtain access to a space. At Los Alamos, Richard Feynman, among other tactics, [obtained classified papers by reaching in underneath drawers](/doc/cs/cryptography/1985-feynman-surelyyourejokingmrfeynman-ch18-safecrackermeetsafecracker.pdf "_Surely You're Joking, Mr. Feynman!_: Safecracker Meets Safecracker") & ignoring the locks entirely. - One analysis of the movie _[Die Hard](!W)_, ["Nakatomi space"](https://bldgblog.com/2010/01/nakatomi-space/), highlights how it & the Israel military's [mouse-holing](!W) in the [Battle of Nablus](!W) treat buildings as kinds of machines, which can be manipulated in weird ways to move around to attack their enemies. - That example reminds me of the [Carr & Adey](https://bodiesfromthelibrary.com/2017/10/23/seven-types-of-locked-room-mystery-part-15/) anatomy of [locked room murder mysteries](!W "Locked-room mystery"), laying out a taxonomy of all the possible solutions which---like a magician's trick---violate one's assumptions about the locked room: whether it was always locked, locked at the right time, the murder done while in the room, the murder done *before* everyone entered the room, it being murder rather than suicide, the supposed secure room with locked-doors having a *ceiling* etc.^[Particularly in office buildings, 'ceilings' are more of [a suggestion](!W "Dropped ceiling") than a structure; in many other buildings, like data centers, so are [the floors](!W "Raised floor").] (These tricks inspired [_Umineko_'s](!W "Umineko When They Cry") mysteries ([review](/review/umineko "'Umineko: The Hopium Of The Magics', Branwen 2018")), although in it a lot of them turn out to just involve [conspirators/lying](https://07th-expansion.fandom.com/wiki/Willard's_Truths).) - In lockpicking, copying a key or reverse-engineering its cuts are some of the most difficult ways to pick a lock. One can instead simply use a [bump key](!W "Lock bumping") to brute-force the positions of the pins in a lock, or kick the door in, or [among other door lock bypasses](https://www.youtube.com/watch?v=4YYvBLAF4T8?t=330 "'The Search for the Perfect Door', Deviant Ollam 2016"), wiggle the bolt, or reach through a crack to open from the inside, or drill the lock. (How do you know someone hasn't already? You *assume* it's the same lock as yesterday?) Locks & safes have many other interesting vulnerabilities; I particularly like [Matt Blaze's](!W "Matt Blaze") [master-key](!W "Master keying") vulnerability ([Blaze 2003](https://www.mattblaze.org/papers/mk.pdf "Cryptology and Physical Security: Rights Amplification in Master-Keyed Mechanical Locks")/[Blaze 2004a](https://www.mattblaze.org/papers/safelocks.pdf "Safecracking for the computer scientist")/[Blaze 2004b](https://www.mattblaze.org/papers/humancambridgepreproc.pdf "Toward a Broader View of Security Protocols")), which uses the fact that a master-key lock is actually opening for any *combination* of master+ordinary key cuts (ie. 'master OR ordinary' rather than 'master XOR ordinary'), and so it is like a password which one can guess one letter at a time. (These papers made locksmiths so mad [they harrassed Blaze into quitting](https://twitter.com/mattblaze/status/1553254965870841856).) - In [stage magic](!W "Magic (illusion)") (especially close-up/card/coin/pickpocketing), one believes one is continuously seeing single whole objects which must move from one place to another continuously; in reality, one is only seeing, occasionally, surfaces of many (possibly duplicate) objects, which may be moving only when you are not looking, in the opposite direction, or not moving at all. By hacking [object permanence](!W) and limited [attentional](!W "Misdirection (magic)") [resources](!W "Change blindness"), the stage magician shows the 'impossible' ([Macknik et al 2008's Table 1](/doc/psychology/cognitive-bias/illusion-of-depth/2008-macknik.pdf "Attention and awareness in stage magic: turning tricks into research") lists many [folk physics](!W) assumptions which can be hacked). Stage magic works by exploiting our implicit beliefs that no adversary would take the trouble to so precisely exploit our heuristics and shortcuts.[^Teller][^Teller2] - In weird machines, you have a 'protocol' like SSL or x86 machine code which appear to do simple things like 'check a cryptographic signature' or 'add one number in a register to another register', but in reality, it's a layer over far more complex realities like processor states & optimizations like speculative execution reading other parts of memory and then quickly erasing it, and these can be pasted together to execute operations and reveal secrets without ever running 'code' (see again Mcilroy et al 2019). Similarly, in finding hidden examples of Turing completeness, one says, 'this system appears to be a bunch of dominoes or whatever, but actually, each one is a computational element which has unusual inputs/outputs; I will now proceed to wire a large number of them together to form a Turing machine so I can play Tetris in Conway’s Game of Life or use heart muscle cells to implement Boolean logic or run arbitrary computations in a game of _Magic: The Gathering_'. Or in side channels, you go below bits and say, 'these bits are only approximations to the actual flow of electricity and heat in a system; I will now proceed to measure the physical system' etc. - In social engineering/pen testing, people see social norms and imaginary things like 'permission' and 'authority' and 'managers' which 'forbid access to facilities', but in reality, all there is, is a piece of laminated plastic or a clipboard or certain magic words spoken; the people are merely non-computerized ways of implementing rules like 'if laminated plastic, allow in', and if you put on a blue piece of plastic to your shirt and you incant certain words at certain times, you can walk right past the guards.^[Speaking of 'social engineering', why was Facebook's success in spreading from a niche of college students to much of the world by offering such superficial social networking so surprising to so many? Perhaps its success is a hint that the underlying logic of social interactions are much more abstractable than, and not as rich & subtle as, we'd prefer to think.] - Many financial or economic strategies have a certain flavor of this; [Alice Maz's _Minecraft_ economics exploits](https://www.alicemaz.com/writing/minecraft.html "Playing to Win") strongly reminds me of 'seeing through', as do many clever financial trades based on careful reading of contractual minutiae or taking seriously what are usually abstracted details like 'taking delivery' of futures etc - and while we're at it, why are puns so [irresistible to hackers](http://www.catb.org/jargon/html/H/hacker-humor.html)? (Consider how omnipresent they are in _[Gödel, Escher, Bach](!W)_ or the [Jargon File](!W) or text adventures or...) Computers are nothing but puns on bits, and languages are nothing but puns on letters. Puns force one to drop from the abstract semantic level to the raw syntactic level of sub-words or characters, and back up again to achieve some semantic twist---they are literally hacking language. And so on. These sorts of things can seem magical ('how‽'), shocking ('but---but---but that's *cheating*!' [the scrub](https://www.sirlin.net/articles/playing-to-win "'Playing to Win Overview', Sirlin 2006") says), or hilarious (in the '[violation of expectations](!W "Theories of humor#Incongruity theory") followed by [understanding](https://people.idsia.ch/~juergen/creativity.html "'Formal Theory of Creativity & Fun & Intrinsic Motivation (1990–2010)', Jürgen Schmidhuber 2010")' theory of humor) because the abstract system _W_ & our verbalizations are so familiar and useful that we quickly get trapped in our dreams of abstractions, and forget that it is merely a map and not the territory, while inevitably the map has made gross simplifications and it fails to document various paths from one point to another point which we don't want to exist. Indeed, these 'backdoors' *must* exist unless carefully engineered away, because the high-level properties we rely on have no existence at the lower levels. If we explain things like 'permission' in terms of sequences of digital bits, we must at some point reach a level where the bits no longer express this 'permission', in the same way that if we explain 'color' or 'smell' by atoms, we must do so by eventually describing entities which do not look like they have any color nor have any smell; at some point, these properties must *disintegrate* into brute facts like a circuit going one way rather than another.[^disintegration] [^disintegration]: [Heisenberg](!W "Werner Heisenberg") (as quoted in [Hanson 1962](/doc/philosophy/ontology/1962-hanson.pdf "The Dematerialization of Matter")): > It is impossible to explain...qualities of matter except by tracing these back to the behavior of entities which themselves no longer possess these qualities. If atoms are really to explain the origin of color and smell of visible material bodies, then they cannot possess properties like color and smell...Atomic theory consistently denies the atom any such perceptible qualities. Hofstadter sums it up as ["Greenness disintegrates."](/doc/philosophy/ontology/1981-hofstadter.pdf#page=21 "‘Heisenberg's Uncertainty Principle and the Many Worlds Interpretation of Quantum Mechanics’, Hofstadter 1981") [^Teller]: Stage magician [Teller](!W "Teller (magician)"), of [Penn & Teller](!W), puts this well in interviews: what makes stage magic work is *hard work*. Teller, ["Teller Reveals His Secrets: The smaller, quieter half of the magician duo Penn & Teller writes about how magicians manipulate the human mind" 2012](https://www.smithsonianmag.com/arts-culture/teller-reveals-his-secrets-100744801/ "'Teller Reveals His Secrets: The smaller, quieter half of the magician duo Penn & Teller writes about how magicians manipulate the human mind', Teller 2012") > I think you'll see what I mean if I teach you a few principles magicians employ when they want to alter your perceptions...Make the secret a lot more trouble than the trick seems worth. You will be fooled by a trick if it involves more time, money and practice than you (or any other sane onlooker) would be willing to invest. My partner, Penn, and I once produced 500 live cockroaches from a top hat on the desk of talk-show host [David Letterman](!W). To prepare this took weeks. We hired an entomologist who provided slow-moving, camera-friendly cockroaches (the kind from under your stove don't hang around for close-ups) and taught us to pick the bugs up without screaming like preadolescent girls. Then we built a secret compartment out of foam-core (one of the few materials cockroaches can't cling to) and worked out a devious routine for sneaking the compartment into the hat. More trouble than the trick was worth? To you, probably. But not to magicians. Or in his [Huttson 2015 interview](http://www.magicalthinkingbook.com/2015/07/teller-of-penn-teller-on-explaining-magic-tricks/ "Teller (of Penn & Teller) on Revealing Magic Tricks"): > **Matt**: "So why don't you explain all your tricks?" > > **Teller**: "Because the short explanation---the explanation that you'd have to do during a theatrical or TV performance---is dull and no fun. The greatest secret to making a deceptive piece of magic is you do it by the ugliest possible means. It's complex, it's unromantic, it's unclever. Because there are no big secrets. There is no safe full of magic secrets somewhere. [Jim Steinmeyer](!W) said he thinks most of the public believes there's a big safe that contains all the magic secrets. The biggest job for a magician, he says, is to conceal the fact that that safe is empty. Because every magic secret is just a minor modification of something that you fully understand in everyday life. Take suspending something with a thread, for example. Everybody's not been able to see a piece a thread when they were trying to put it through a needle. What makes it difficult to find is lighting and background. If a magician's using a thread on stage, say, to levitate a ball, he must use lighting and background to conceal the thread. There's no obscure secret in that. You learned that playing in your grandmother's sewing box. Every magic 'secret' is hiding in plain sight in the everyday world. It's not news, and eminently drab." [^Teller2]: [Houdini's trick of Sir Arthur Conan Doyle](/doc/psychology/cognitive-bias/2006-polidoro-houdinisimpossibledemonstration.html "'Notes on a Strange World: Houdini’s Impossible Demonstration', Polidoro 2006") exemplifies these strategies. No one would expect Houdini to renovate a room just for a trick, to have learned a steganographic code to communicate the phrase Doyle wrote on a piece of phrase to the assistant without Doyle noticing, or the assistant to manipulate a magnetic pole behind a small suspended slate board, hiding it in the viewers' *precise* blind spot in order to make it appear as if the chalk were hovering in mid-air & writing by itself. Doyle did not, and disbelieved Houdini's statement it was merely a trick. But Doyle should have remembered Hume's dictum: which is more likely---witnessing the paranormal, or that [somewhere in the world](/littlewood "'Littlewood’s Law and the Global Media', Branwen 2018") there was a man as cunning, careful, & compulsive as Houdini? The latter! [Olson & Raz 2020](/doc/psychedelic/lsd/2020-olson-2.pdf "Applying insights from magic to improve deception in research: The Swiss cheese model") give further examples, and demonstrate how this can be useful for running psychology experiments. [^burglar]: pg356 of [_A Burglar's Guide to the City_](https://burglarsguide.com/), Geoff Manaugh 2016: > [Schatz's](!W "Andy Schatz") exhortation to [players](!W "Monaco: What’s Yours Is Mine") to move *against* the architecture, not with it, to uncover a scene's possible crimes, is useful not only in the world of games. Ignoring the paths laid out by architects and even remaking a space from within are some of the most fundamental ways in which burglars misuse the built environment...In one of the most interesting moments in [Bill Mason's](!W "Bill Mason (jewel thief)") [memoir](https://www.amazon.com/Confessions-Master-Jewel-Thief-Mason/dp/0375760717 "_Confessions of a Master Jewel Thief_, Mason 2005"), he sees that architecture can be made to do what he wants it to do; it's like watching a character in _Star Wars_ learn to use the Force. > > ...he explains that his intended prize was locked inside a room whose door was too closely guarded for him to slip through. Then he realizes the obvious: he has been thinking the way the hotel wanted him to think---the way the architects had hoped he would behave---looking for doors and hallways when he could simply carve a new route where he wanted it. The ensuing realization delights him. "Elated at the idea that I could cut my own door right where I needed one," he writes, Mason simply breaks into the hotel suite adjacent to the main office. There, he flings open the closet, pushes aside the hangers, and cuts his way from one room into the other using a drywall knife. In no time at all, he has cut his "own door" through to the manager's office, where he takes whatever he wants---departing right back through the very "door" he himself made. It is architectural surgery, pure and simple. > > Later, Mason actually mocks the idea that a person would remain reliant on doors, making fun of anyone who thinks burglars, in particular, would respect the limitations of architecture. "*Surely if someone were to rob the place*," he writes in all italics, barbed with sarcasm, "*they'd come in as respectable people would, through the door provided for the purpose. Maybe that explains why people will have 4 heavy-duty locks on a solid oak door that's right next to a glass window*." People seem to think they should lock-pick or kick their way through solid doors rather than just take a [$10]($2016) drywall knife and carve whole new hallways into the world. Those people are mere slaves to architecture, spatial captives in a world someone else has designed for them. > > Something about this is almost unsettlingly brilliant, as if it is *nonburglars* who have been misusing the built environment this whole time; as if it is nonburglars who have been unwilling to question the world's most basic spatial assumptions, too scared to think past the tyranny of architecture's long-held behavioral expectations...Because doors are often the sturdiest and most fortified parts of the wall in front of you, they are a distraction and a trap. By comparison, the wall itself is often more like tissue paper, just drywall and some 2×4s, without a lock or a chain in sight. Like clouds, apartment walls are mostly air; seen through a burglar's eyes, they aren't even there. Cut a hole through one and you're in the next room in seconds. # Curse of Expertise
> The question is", said Alice, "whether you *can* make words mean so many different things."
"The question is", said Humpty Dumpty, "which is to be master---that's all. > > Lewis Carroll, [_Through the Looking-Glass, and What Alice Found There_](!W "Through the Looking-Glass") (1872)^["48\. The best book on programming for the layman is _Alice in Wonderland_; but that's because it's the best book on anything for the layman." ---["Epigrams on Programming"](/doc/cs/algorithm/1982-perlis.pdf), Perlis 1982.]
Perversely, the more educated you are, and the more of the map you know, the worse this effect can be, because you have more to unsee (eg. in [fiction](/story-of-your-life "'‘Story Of Your Life’ Is Not A Time-Travel Story', Branwen 2012")). One must always maintain a certain contempt for [words](/language "On the Existence of Powerful Natural Languages") & [spooks](https://en.wikipedia.org/wiki/Max_Stirner#Philosophy). The fool can walk right in because he was too ignorant to know that's impossible. This is why atheoretical optimization processes like animals (eg. [cats engaged in](/review/cat#fuzz-testing "'Cat Psychology & Domestication: Are We Good Owners? § Fuzz Testing', Branwen 2018") [fuzz testing](!W)) or [SMT solvers](!W "Satisfiability modulo theories") or [evolutionary AI](https://arxiv.org/abs/1803.03453 "'The Surprising Creativity of Digital Evolution: A Collection of Anecdotes from the Evolutionary Computation and Artificial Life Research Communities', Lehman et al 2018") are so dumb to begin with, but in the long run can be so good at surprising us and finding 'unreasonable' inputs or [reward hacks](/tank#alternative-examples) (analogous to the [bias-variance tradeoff](!W)): being unable to understand the map, they can't benefit from it like we do, but they also can't overvalue it, and, forced to explore the territory directly to get what they want, discover new things. # Learning To Unsee
> I don't even see the code. All I see is blonde, brunette, redhead. > > Cypher, _The Matrix_
> Whoa. > > Neo
To escape our semantic illusions can require a determined effort to unsee them, and use of techniques to [defamiliarize](!W "Defamiliarization") the things. For example, you can't find typos in your own writing without a great deal of effort because you know what it's *supposed* to say; so copyediting advice runs like 'read it out loud' or 'print it out and read it' or 'wait a week' or [recite until gibberish](!W "Semantic satiation") or even 'read it upside down' (easier than it sounds). That's the sort of thing it takes to force you to read what you actually wrote, and not what you thought you wrote. Similar tricks are used for learning drawing: a face is too familiar, so instead you can flip it in a mirror and try to copy it. # External Links - ["Security Mindset and Ordinary Paranoia"](https://www.lesswrong.com/posts/8gqrbnW758qjHFTrH/security-mindset-and-ordinary-paranoia); ["Security Mindset and the Logistic Success Curve"](https://www.lesswrong.com/posts/cpdsMuAHSWhWnKdog/security-mindset-and-the-logistic-success-curve) - ["How did so many _Dungeon Crawl: Stone Soup_ players miss such an obvious bug?"](https://desystemize.substack.com/p/desystemize-7) - ["Stargate Physics 101"](https://archiveofourown.org/works/3673335) - ["The Line of Death"](https://textslashplain.com/2017/01/14/the-line-of-death/) - ["Movie-Plot Threats"](https://www.schneier.com/tag/movie-plot-threat-contests/) - ["Security is Mathematics"](https://www.daemonology.net/blog/2008-03-21-security-is-mathematics.html), Colin Percival; ["On Exactitude in Science"](https://kwarc.info/teaching/TDM/Borges.pdf), Jorge Luis Borges - ["No general method to detect fraud"](https://calpaterson.com/fraud.html), Cal Peterson - [_Red Teaming: How Your Business Can Conquer the Competition by Challenging Everything_](https://www.amazon.com/Red-Teaming-Competition-Challenging-Everything/dp/1101905972), Hoffman - [_Baba Is You_](!W): ["No Really, There Are No Rules!"](https://www.lesswrong.com/posts/gvCwotnq2cBTYqEsS/no-really-there-are-no-rules) - [_The City & the City_](!W) - [Homograph attacks](https://en.wikipedia.org/wiki/IDN_homograph_attack) - **Discussion**: Reddit: [1](https://www.reddit.com/r/slatestarcodex/comments/c0nqg7/people_seem_to_think_thieves_should_lockpick_or/er6huvz/), [2](https://www.reddit.com/r/DepthHub/comments/c0uutk/ugwern_talks_about_the_hacker_mindset_in/); [Twitter](https://twitter.com/sonyaellenmann/status/1139752544761081858)