…But such difficulties do not much obscure Vinge’s vision, for he seems to regard present day forms of programming—with their stiff, formal, inexpressive languages—as but an early stage of how better programs will be made in the future.

I too am convinced that the days of programming as we know it are numbered, and that eventually we will construct large computer systems not by anything resembling today’s meticulous but conceptually impoverished procedural specifications. Instead, we’ll express our intentions about what should be done in terms of gestures and examples that will be better designed for expressing our wishes and convictions. Then these expressions will be submitted to immense, intelligent, intention-understanding programs that then will themselves construct the actual, new programs. We shall no longer need to understand the inner details of how those programs work; that job will be left to those new, great utility programs, which will perform the arduous tasks of applying the knowledge that we have embodied in them, once and for all, about the arts of lower-level programming. Once we learn better ways to tell computers what we want them to accomplish, we will be more able to return to our actual goals—of expressing our own wants and needs. In the end, no user really cares about how a program works, but only about what it does—in the sense of the desired effects it has on things which the user cares about.

In order for that to happen, though, we will have to invent and learn to use new technologies for “expressing intentions”. To do this, we will have to break away from our old, though still evolving, programming languages, which are useful only for describing processes. But this brings with it some serious risks!

The first risk is that it is always dangerous to try to relieve ourselves of the responsibility of understanding exactly how our wishes will be realized. Whenever we leave the choice of means to any servants we may choose then the greater the range of possible methods we leave to those servants, the more we expose ourselves to accidents and incidents. When we delegate those responsibilities, then we may not realize, before it is too late to turn back, that our goals have been misinterpreted, perhaps even maliciously. We see this in such classic tales of fate as Faust, the Sorcerer’s Apprentice, or the Monkey’s Paw by W. W. Jacobs.

A second risk is exposure to the consequences of self-deception. It is always tempting to say to oneself, when writing a program, or writing an essay, or, for that matter, doing almost anything, that “I know what I would like to happen, but I can’t quite express it clearly enough”. However, that concept itself reflects a too-simplistic self-image, which portrays one’s own self as existing, somewhere in the heart of one’s mind (so to speak), in the form of a pure, uncomplicated entity which has well-defined wishes, intentions, and goals. This pre-Freudian image serves to excuse our frequent appearances of ambivalence; we convince ourselves that clarifying our intentions is merely a matter of straightening-out the input-output channels between our inner and outer selves. The trouble is, we simply aren’t made that way. Our goals themselves are ambiguous.

The ultimate risk comes when our greedy, lazy, master-minds attempt to take that final step—of designing goal-achieving programs that are programmed to make themselves grow increasingly powerful, by self-evolving methods that augment and enhance their own capabilities. It will be tempting to do this, both to gain power and to decrease our own effort toward clarifying our own desires. If some genie offered you 3 wishes, would not your first one be, “Tell me, please, what is it that I want the most!”? The problem is that, with such powerful machines, it would require but the slightest accident of careless design for them to place their goals ahead of ours—as it were. The machine’s goals may be allegedly benevolent, as with the robots of With Folded Hands, by Jack Williamson, whose explicit purpose was allegedly benevolent: to protect us from harming ourselves, or as with the robot in Colossus, by D. F. Jones, who itself decides, at whatever cost, to save us from an unsuspected enemy. [cf. Darrach1970] In the case of Arthur C. Clarke’s HAL, the machine decides that the mission we have assigned to it is one we cannot properly appreciate. And in Vernor Vinge’s computer-game fantasy, True Names, the dreaded Mailman (who teletypes its messages because it cannot spare the time to don disguises of dissimulated flesh) evolves new ambitions or its own.

See Also: