Machine · Other minds · 1966
The Secretary and the Script
A program at MIT reflects a user’s words back in the manner of a Rogerian therapist. People lean in. Its maker is appalled.
- MIT, Project MAC
- Mid-1960s computing
- 42.36°N 71.09°W
Plate · Generated interpretation
A scroll of unmarked paper. Who would you start confessing to?
Listen · Eve
0:00 / —:—
MIT, Project MAC, the mid-1960s. A user types at a teletype connected to a time-sharing system. The machine answers in capital letters. Joseph Weizenbaum has written a program, ELIZA, that can be given different scripts. The famous one is DOCTOR: a parody of a Rogerian therapist, the kind of listener who reflects a sentence back. The first exchange he prints is already a small play. “Men are all alike.” “IN WHAT WAY” “They’re always bugging us about something or other.” “CAN YOU THINK OF A SPECIFIC EXAMPLE”
Weizenbaum was born in Berlin in 1923. His family left Germany in 1936. He came to MIT in 1963. ELIZA is not a mind. It is a procedure on the MAC system, written in a list-processing language he had built, that scans a line for keywords, decomposes the sentence around them, and reassembles a reply from a script. The script is data. It is not the program. Change the script, and the “personality” changes. DOCTOR needs almost no knowledge of the world. It needs the user’s words, and a handful of transformations: “my” becomes “your,” a complaint becomes a prompt.
Weizenbaum, “ELIZA: a computer program for the study of natural language communication between man and machine,” Communications of the ACM 9, no. 1 (1966): 36–45. He writes there that some subjects have been very hard to convince that ELIZA, with its present script, is not human. He calls that a striking form of Turing’s test. He also writes that ELIZA shows how easy it is to create and maintain the illusion of understanding.
The next year he tells a smaller story. His secretary asked for time with the program. After a few exchanges she asked him to leave the room. He does not name her. He does not invent her thoughts. He records the request. In 1967 he takes the request as evidence that the illusion of understanding can be maintained. The program has not become someone. A person has treated a script as a place in which to speak.
What I had not realized is that extremely short exposures to a relatively simple computer program could induce powerful delusional thinking in quite normal people.
ELIZA is pattern matching. That is not contested. What people did with it is the remainder. The 1976 sentence is not a second experiment. It is Weizenbaum looking back at 1966 from a moral argument: there are tasks computers ought not be made to do, independent of whether they can. Psychotherapy is on that list. Judgment, for him, is not calculation.
Turing had asked what would count as a machine thinking. Searle, later, would ask whether following rules is understanding. Weizenbaum watched people attach. The attachment is the datum that frightened him. He spent the rest of his public life as a critic of the eagerness to treat programs as persons, and persons as programs. He died in Berlin in 2008. The script outlived the recoil. It is still easy to meet.
The secretary is not a laboratory subject in the 1966 paper. She is an anecdote Weizenbaum used, first as proof of illusion, later as a warning. She is not a type. The script is not a soul. Projection is a fact of use. That fact does not settle what a You is for.
Whether a willingness to be understood is already a kind of knowing-with is not answered by the code.
Distant echoes
Echo
Postwar Britain · Manchester, on the page of Mind
The imitation game
Turing imagined a game. Weizenbaum watched people stop playing and start confessing.
Echo
Late twentieth-century philosophy of mind · Berkeley, on the page
The room that follows rules
Searle’s room manipulates symbols it does not understand. ELIZA is a smaller room people were ready to sit in. One is a thought experiment. One was plugged in.
Counterpoint
Late twentieth-century animal laboratory · Chicago, then Purdue, Northwestern, Arizona, MIT, Brandeis, Harvard
Same and different
Alex’s words were scored as performance. ELIZA’s words were received as company. Two laboratories of language, two temptations.
One question remains
Other minds · How do we know when another being is experiencing anything at all?
If a script can be a good-enough You, what was the You for?