October 5, 2005
How Stella Got Her Text Back
This is a version of my Elective Affinities talk “How Stella Got Her Text Back: Trajectories of Word and Image in Creative Computing” given on September 26, 2005, at the University of Pennsylvania.
1.
To begin with a very big question: What is the relationship between word and image in creative computing – a field also known as new media or digital media, which encompasses all of the expressive uses of the computer, including video games, digital art that is shown in galleries, low-brow Flash animations, electronic literature, and other digital works?
The slide shows Textmode Quake (Bob Zimbinski) and Arteroids (Jim Andrews), one a work of high art and the other a work of high nerdity, created for the enjoyment of the technically proficient. Arteroids is an Asteroids-like game that replaces the asteroids with words and your ship with the word “poetry”; Textmode Quake uses the text console to “render” the graphical game Quake in typographical characters. These are some extreme examples of word-image collision and conflation in new media, but even the typical use of text and image in mainstream video games is quite interesting. (Title images from Jim Andrews and Bob Zimbinski.)
My claim is that the relationship is a complex one that depends on social, cultural, and technological or computing contexts. Absolute claims about how text and image relate in computing are not very useful. Rather, scholars need to do the same sort of work that keynote speaker Peter Stallybrass has done in investigating early modern printing; it’s necessary to apply the same methods and the same attention to detail in understanding how creative computer systems are developed.
2.
World of Warcraft (Blizzard Entertainment, 2004-) actually uses a great deal of text, for chat, status information, and annotation of characters and objects in the world. (Screenshot source.)
3.
Gridrunner++ (Jeff Minter, 2002) is a completely different sort of game, abstract rather than realistic in its use of image, but it also uses text for annotation of points gained, announcing the completion of a level and entry into a new level, and status information such as the current score.
4.
Dark Mage (Greg Troutman, c. 1998) is even more curious: an all-text adventure game for the Atari VCS, a very graphically-oriented system that used a chipset code named “Stella.” How did Stella get her text back? How could this game exist?
5.
One idea is that, on the computer, word and image are equal to each other and to all other media. The computer’s digital representation scheme makes it either a “multimedia” system or, as Alan Kay said, a “metamedium” that can imitate anything, and the things it can present or imitate can be manipulated in a variety of ways. This is a purely formal position based on the nature of digital representation; it suggests that there is no hierarchy that would hold word over image or vice-versa.
Another idea is that text is primary, that words are more important or essential than images. John Cayley argued this in his essay “Literal Art,” in which he argued that literary practice, working with a finite set of elements, prefigured digital art, and that “poetics provides a preferred and even paradigmatic underlying or critical framework for what is now called digital art.” There are several ways to argue for the primacy of text, some of which Cayley employs. We could say that web surfers almost always read and that they often type search queries and emails, and the typical experience of computing is essentially textual. Or we argue that programmers use textual programming languages to put systems of all sorts together.
6.
There is also plenty of evidence that images are dominant, however. From many perspectives, it seems that creative computing is only interesting because of the graphics capabilities of computers. This is certainly the perspective of the dominant commercial force in creative computing, the video game industry. The picture element or pixel could be said to have a more straightforward representation in the digital computer than the character does. Our mapping of a byte onto a particular alphabet depends on the language and character set we are using, but images are seemingly universal.
Although some of these philosophical and practical arguments are interesting, they won’t be able to establish that creative computing is either “text-based” or “image-based,” because creative computing is not monolithic. Historically, the computer’s capabilities as a manipulator and presenter of media have been different at different times, and people’s interest in using these capabilities has also differed.
7.
Even at the same point in history and in the same place in the world – say, in the United States in the late 1970s – different traditions of creative computing have existed, some rooted in text, word, writing, and language, and others based on images, figure, drawing, and visual effect. Gridrunner++, World of Warcraft, and Dark Mage are very different sorts of systems, visually and in terms of how they work, but they are both part of the same broad tradition of computer gaming, so they have at least this in common. Looking at computer gaming in the late 1970s, however, shows that even this specific thread of creative computing, in a single country, at a single time, had different traditions, such that textual practices and the display of words dominated in the hobbyist BASIC programming tradition and images were dominant in another, the commercial video gaming tradition. (Image source.)
8.
By the late 1970s there were a sizable handful of people writing text-based games in BASIC (Beginners All-Purpose Symbolic Instruction Code, developed at Dartmouth starting in 1964). BASIC games might take place on a grid or on some more unusual playing field, and this might be represented explicitly in some sort of table (as in the early game Star Wars) or it might be described in sentences, as in the game Hunt the Wumpus (shown) and in the text adventures that followed. It was common for BASIC programs to be transmitted via program listings in books and magazines, such as the one shown here. Readers were expected to type in the program themselves, probably making mistakes along the way and needing the debug the program when they were done. Not all such programs were written in BASIC, but it was a popular language for short creative programs.
9.
A different tradition of creative computing, one that would become very economically and culturally significant, was also well underway by the late 1970s. This was commercial video gaming, already a huge success in the arcades at this point and poised to break completely into the home market as well. The company that pioneered arcade video games and that dominated the industry would market its major cartridge-based system beginning in 1972. This was the Atari VCS. The cartridge that came with the system was Combat. Although very primitive by today’s standards and dominated by the image, you can see in this screenshot that some text was used to keep score for the two players.
10.
The information that is stored in the Combat cartridge takes up only 2k, and can be represented in hexadecimal on a single page. This is the “whole game” or at least the whole cartridge – obviously some of what the game did relied on the Stella chipset, controllers, a television, and so on.
11.
We can look at some of the binary code that makes up Combat. What you can see from looking at the code this way is that numbers were stored as data within the cartridge. The programmer had to draw each number dot by dot to keep score; there was no built-in text provided. The Atari VCS could draw pixels on the screen and make sounds, but programmers had to build whatever text they wanted themselves. So the way Stella got her text back was -arduously-. A text adventure on the Atari 2600, completed in the late 1990s, was a proof that the system could be put to very strange purposes and that it could me made to do things which it was not designed to do.
More generally, how did the video game tradition get its text back? We’ve already seen where this commercial video game tradition has gone. Programmers of modern-day, mainstream game systems do not have to do what Greg Troutman did to add text to their creations. As graphical power improved and was highly touted, additional support for text was also built into system, making it much easier for programmers to use text with consoles such as today’s Playstation 2, Xbox, and Game Cube – and of course developing games on general-purpose computers allows access to the standard text-display and text-processing tools that developers of all sorts can access.
12.
Having seen where the video gaming tradition went, where did the scrappy, low-brow, and innovative BASIC programming tradition go? One recent system that may have enable modern-day programming of the sort that I’ve discussed is the commercial system Flash, by Macromedia, which has been adopted by an interesting subset of the world’s youth as a platform for short movies and as a game-making system. ROFL Copter, one product of just-in-spare-time Flash development, gives the player a helicopter made of ASCII characters. People running around below, composed in part of “LOL”, can be blown away.
13.
My hope is that those of us interested in word and image in the digital age will be able to look past “the capabilities and limitations of the computer” to start to deal with some more subtle matters. First of all, while there are fundamentals of computing, video game hardware does not have the same sort of capabilities as does a server set up for networked access by multiple users, and even platforms created for the same purpose differ in important ways, so we should be looking past the idea of “the computer,” recognizing that there is not just one. Additionally, questions about computing – whether they are questions about word and image, questions about communication and computation, or questions about interaction and responsiveness – do not just rest on what computers can and cannot do. Computer scientists know very well that this sort of “binary” is not useful; it is also necessary to take into account how quickly computers can do certain things, which functions have been optimized and which can be achieved, but only slowly. To understand the place of text and image within computing, we should bring humanistic concern for history, social and cultural contexts, the computational equivalent of intertextuality, and all the rest – and we should bring an appropriate technical understanding of the systems we are considering, and a grasp of the basic nature of computing.
October 7th, 2005 at 2:33 pm
Nice piece, Nick, and I’m sorry I missed hearing it live.
You might want to have a look at my essay “The Word as Image in an Age of Digital Reproduction” in MIT’s Eloquent Images volume. Contrary to the kind of currently popular argument that celebrates the collapse of boundaries between words and images in electronic environments, I demonstrate some of the ways they can be leveraged as very distinct data types.