June 22, 2003

Modes of AI-based art

by Michael Mateas · , 10:13 pm

Harold’s post inspired me to post on what I see as the modes or genres of AI-based art. The modes described here are not mutually exclusive; a single piece may simultaneously explore multiple modes or genres. I would love to hear any comments describing a new mode (with an example) or an alternative categorization scheme.

Procedural Portraits of Human Meaning-Making

A procedural portrait is a dynamic representation of some psychological, social or cultural process. For example, Terminal Time (Domike, Mateas, Vanouse) is a procedural portrait of the ideologically-biased construction of mainstream historical documentaries. While the constructed documentary, and the relationship between this documentary and the audience input, are the directly sensed portion of the audience experience, the mechanisms by which the system constructs documentaries are also part of the piece. Terminal Time’s architecture is a caricature of totalizing thought, of the mono-maniacal pursuit of an ideological position. How Terminal Time constructs biased histories is as important a part of the conceptual exploration of the piece as are the histories that are actually constructed.

Harold Cohen’s Aaron is another example of procedural portraiture. But, as Harold described in his post, he was not motivated by the end result, by the idea of a painting produced by a machine, but rather by the process of painting. Writing Aaron was a prolonged meditation on his own mental processes when painting, an attempt to understand his own image-making behavior.

“What the computer provided was a way of externalizing, stabilizing my speculations about image-making behavior: not only my own behavior, but what I thought I could see operating in drawings generally, and especially in children’s drawings and so-called primitive art.”[McCorduck 1991:78]

The notion of procedural portraiture is similar to longstanding research approaches in AI that build models of psychological processes, such as models of human problem solving behavior when faced with laboratory tasks. But in traditional AI research these models are taken as simulations, possessing a one-to-one correspondence with the system being simulated and thus, in a functionalist framework, identical to the system being simulated. But simulation is just one of many possible relationships that can exist between a representation and the world. Representations can simplify, embellish, highlight, subvert, satirize – procedural portraits do not make claims of identity with that which they represent, but are rather thought experiments, dynamic representations of some aspect of human activity.


AI-based characters are dynamic portrayals of humans or animals, with the focus not on realism but believability. Two of the earliest (and best known) AI-based characters are the Doctor script of Eliza [Weizenbaum 1966], and Parry [Colby 1975]. Both Doctor and Parry are chatterbots, programs that process textual natural language input and produce text output. Doctor used simple pattern matching techniques to respond to user input in the guise of a non-directive Rogerian psychotherapist. Parry portrayed a paranoid schizophrenic. Technically, Parry was more advanced than Doctor in that it maintained internal state beyond individual player utterances, pursuing its own delusions as well as maintaining a simple emotional state. Both programs demonstrated that relatively simple techniques can support the suspension of disbelief, allowing users to participate in the illusion.

Doctor and Parry are in some sense the first believable agents, autonomous characters that convey rich personality, emotion and social behavior. Contemporary work on believable agents tends to focus on real-time, animated representations. For example, Naoko Tosa has built a series of screen-based characters that respond to the participant’s emotional state. GTxA’s own Andrew Stern worked for a number of years on the Petz and Babyz projects.

Alien Presence

Characters leverage viewer expectations of human and animal forms and behaviors in order to achieve believability, making use of known character types from other media representations, such as movies, animated films or comics. Alien presences, like characters, are dynamic depictions of living creatures. But unlike characters, alien presences have novel forms and/or behavioral repertoires. While perhaps borrowing bits of form and behavior from the animal or human world, they primarily have their own perceptions, reactions, concerns and thoughts, divorced from the everyday human world. For the participant, the pleasure of interacting with an alien presence is to enter its alien world, to discover how it perceives and acts. Office Plant #1 (Böhlen, Mateas), a desktop robot that responds to the social and emotional tone of the email received by its owner, is an example of an alien presence; it offers an ambient commentary the email stream.

One of the earliest examples is cyberneticist Gordon Pask’s Colloquy of Mobiles, presented at the Cybernetic Serendipity show in 1968 [Pask 1971]. The mobiles form a simple social system. Communication takes place by modulating colored lights and audible tones. The mobiles attempt to maintain a “territory” – this results in a combination of competition and cooperation between the mobiles. Left to their own devices, the mobiles pursue their own goals, rotating, flashing lights and tones at each other, and exhibiting various forms of social organization. Audience members entering the environment were provided with devices to generate the lights and tones of the mobile’s language, and were thus able to participate in the alien society.

Simon Penny’s Petit Mal [Penny 1997] is an autonomous robotic being that interacts in physical space with audience members. Penny explicitly sought to avoid anthropomorphism, biomorphism, or zoomorphism, wanting instead “… to present the viewer with a phenomenon which was clearly sentient, while also being itself, a machine, not masquerading as a dog or president” [Penny 2000]. Petit Mal’s body consists of a double pendulum system that dynamically stabilizes its two-wheeled stance. Audience members project rich and complex readings of Petit Mal’s internal life – much of the complexity of its “personality” arises from the physical dynamics of its alien form.

David Rokeby’s Giver of Names [Huhtamo 1998] is a computer-vision based work that offers poetic commentary on the objects placed in front of its camera. The audience is provided with a collection of objects that can be placed on a podium in front of the camera. The colors, forms and juxtapositions of the objects stimulate nodes in a semantic network; the poetic commentary results from the activated concepts in the semantic net. After listening to the commentary for awhile, audience members begin to discern the private, idiosyncratic structure of Giver of Names’ conceptual space.


Story-telling and narrative are fundamental to human experience [Mateas & Sengers 2003]. As children, we are immersed in stories and learn to approach the world via narrative frameworks. As adults, we order events and find meaning by assimilating them to more-or-less familiar narratives. AI-based narrative art explores narrative meanings by generating, modeling and understanding narrative structures and meanings.

Façade (Mateas, Stern), with its incorporation of autonomous character and player activity into a story framework, and Terminal Time, with its generation of ideologically biased documentary histories, are examples of AI-based narrative art. There is a rich tradition of story generation work in the AI research literature; this literature offers a useful collection of ideas and technical approaches for AI-based narrative art.

Robotic Art

Robotic art is concerned with building physical, often sculptural, systems whose behavioral responses are a function of the system’s perception of the environment. Robotic art is concerned with the aesthetics of physical behavior. Jack Burnham described the possibility of an aesthetics based on cybernetics and artificial intelligence in 1968 [Burnham 1968].

Edward Inahtowicz’s The Senster is an early example of robotic art. Inahtowicz was interested in how humans ascribe intention to physical motion and behavior:

“This is not the place to present the general argument that all perception is dependent in some way on an interpretation of physical movement, but all the pieces I have made so far and all that I am planning to make aim ultimately at making the spectator aware of just how refined our appreciation of motion is and how precisely we are capable of interpreting the intention behind even the simplest motion. For an artificial system to display a similar sense of purpose, it is necessary for it to have a means of observing and interpreting the state of its environment.” [Inahtowicz: 4-5]

Robotic art often explores the responsive behavior of abstract sculptural forms that have little or no relationship to human or animal forms. For this reason, robotic art is often an alien presence, an abstract responsive “life form” with its own idiosyncratic behaviors and drives appropriate to its physical form.


Meta-art systems produce art as output; that is, they autonomously create writing, visual imagery, music, etc., that might be considered art in its own right. The goal of meta-art is often conceptual, an exploration of the dynamics of creativity within a specific genre; the generated output itself is often not the primary goal.

One body of work has explored the generation of writing, such as poetry or stories. Masterson’s haiku generator [Masterson 1971] is an early example of poetry generation. More recent examples include Loss Glazier’s sound poetry generator (sound poetry is interested purely in the sound of often nonsense words, rather than in semantic meaning), Kurzweil’s Cybernetic Poet, which learns language models from example poems and generates poetry in that style, and Manurung’s work, which generates poetry using a natural language generation framework that makes use of both syntactic and semantic knowledge [Manurung, Ritchie & Thompson 2000]. Story generators include the first generator, Tale-Spin [Meehan 1976 (Warren Sack’s rewrite of original) (Michael Cox’s drug-addicted Elvis version)], a character-based generator that generates stories by pursuing character goals and plans, Universe [Lebowitz 1984; Lebowitz 1985], a generator that coordinates multiple characters via authorial plans, and Minstrel [Turner 1994], a case-based generator that simultaneously pursues character and author goals within a single model of creative problem solving.

There is a vast body of work in music generation. Notable examples include David Cope’s EMI (Experiments in Musical Intelligence), a composition system that analyzes musical examples in terms of their stylistic components and recombines these components to generate new music in the same style [Cope 1996], and George Lewis’ Voyager, an improvisation system that performs with human musicians [Roads 1985].

There is a similarly vast body of work on systems that generate drawings or paintings, with Aaron being a prominent example. Visual grammars have been developed to describe the painting style of painters such as Wassily Kandinsky, Joan Miro, Juan Gris and Richard Diebenkorn; programs implementing these visual grammars are then able to generate endless variations on these styles. Similarly, shape grammars have been implemented that can generate endless architectural forms in the style of various well-known architects.

Note that not all meta-art is procedural portraiture. Meta-art may employ methods that are not dynamic representations of human activity. For example, shape grammars or n-gram poetry generation models offer little interesting commentary on human processes of architectural design or poetry writing, yet can serve as a generative framework for producing an interesting output space.

Burnham, J. 1968. Beyond Modern Sculpture: The Effects of Science and Technology on the Sculpture of This Century. New York: G. Braziller.

Colby, K. M. 1975. Artificial Paranoia: A Computer Simulation of Paranoid Processes. New York: Pergamon Press.

Cope, D. 1996. Experiments in Musical Intelligence. Madison, WI: A-R Editions.

Huhtamo, E. 1998. Silicon Remembers Ideology, or David Rokeby’s Meta-Interactive Art. Catalog essay for The Giver of Names exhibit at McDonald-Stewart Art Center.

Lebowitz, M. 1985. Story Telling as Planning and Learning. Poetics 14, pp. 483-502.

Lebowitz, M. 1984. Creating Characters in a Story-Telling Universe. Poetics 13, pp. 171-194.

Manurung, H. M., Ritchie, G., Thompson, H. 2000. Towards a Computational Model of Poetry Generation. In Proceedings of AISB Symposium on Creative and Cultural Aspects and Applications of AI and Cognitive Science, 79-86, Birmingham, April 2000.

Masterson, M. 1971. Computerized haiku. In J. Reichardt (Ed.), Cybernetics, Art and Ideas. Greenwich, CT: New York Graphic Society Ltd. 175-183

Mateas, M and Sengers, P (Eds.). 2003. Narrative Intelligence. Amsterdam: John Benjamins

McCorduck, P. 1991. Aaron’s Code: Meta-art, Artificial Intelligence, and the Work of Harold Cohen. New York, NY: W. H. Freeman and Co.

Meehan, J. 1976. The Metanovel: Writing Stories by Computer. Ph.D. Dissertation. Yale University.

Pask. G. 1971. A comment, a case history, and a plan. In J. Reichardt (Ed.), Cybernetics, Art and Ideas. Greenwich, CT: New York Graphic Society Ltd. 76-99.

Penny, S. 2000. Agents as Artworks and Agent Design as Artistic Practice. In K. Dautenhahn (Ed.), Human Cognition and Social Agent Technology. Amsterdam: John Benjamins

Penny, S. 1997. Embodied Cultural Agents at the Intersection of Robotics, Cognitive Science and Interactive Art. In K. Dautenhahn (Ed)., Working notes of the Socially Intelligent Agents Symposium. AAAI Fall Symposium Series. Menlo Park: Calif.: AAAI Press.

Roads, C. (1985). Improvisation with George Lewis. In C. Roads (Ed.), Composers and the Computer, 75-88. William Kaufmann, Inc.

Turner, S. R. 1994. The Creative Process: A Computer Model of Storytelling and Creativity. Lawrence Erlbaum Associates.

Weizenbaum, J. 1966. Eliza – A computer program for the study of natural language communication between man and machine. Communications of the ACM 9(1):36-45.