January 30, 2007

Media Machines

by Noah Wardrip-Fruin · , 3:13 pm

Earlier this month I posted an excerpt from my in-process manuscript — currently titled Expressive Processing — on the topic of process intensity. Interesting discussion ensued, I decided to post further excerpts, and I realized what excerpt I’d post next: the part of the book that comes just before the section on process intensity.

This is a central passage for the book. I’m laying out my basic perspective — what it is that draws me toward examining processes — and starting to work through the practice-oriented part of that perspective.

I appreciated the comments from people last time (they’ve already resulted in manuscript revisions) and I’ll be interested to hear any thoughts on the moves this excerpt makes.

Media Machines

A computer is a strange type of machine. While most machines were developed for particular purposes — washing machines, forklifts, movie projectors, typewriters — modern computers are designed specifically to be able to simulate the operations of many different types of machines, depending on the computer’s current instructions.

This is why a computer can simulate a movie projector: showing a set of image frames in quick succession. It’s also why a computer can act like a tape player: reading and amplifying a stream of sound data.

And it is for this same reason that computers can be instructed to act like previously-impossible types of machines. A computer can simulate a typewriter — getting input from the keyboard and arranging pixels on the screen to shape the corresponding letters — but it can also go far beyond a typewriter, offering many fonts, automatic spelling correction, painless movement of manuscript sections (through simulations of “cut” and “paste”), programmable transformations (such as “find and replace”), and even collaborative authoring by large, dispersed groups (as with projects like Wikipedia). This is what modern computers (more lengthily called “stored-program electronic digital computers”) are designed to make possible: the continual creation of new machines, opening new possibilities, through the definition of new sets of computational processes.

“Digital media” are the media enabled by this possibility. This includes web projects, like Wikipedia, and also all computer games. The first computer games were created on the earliest stored-program computers, and since then we have seen a major cultural impact from the fact that a computer can not only simulate a pinball machine but also act like game machines never seen before: a Tetris machine, a Doom machine, a SimCity machine, and more.

Personally, I am fascinated by the possibilities that digital media open for fiction. A blossoming of new models of character, story, and language is being enabled by computational processes. From computer games with epic structures to experimental interactive films, digital fictions are providing diverse experiences for a wide range of audiences. From ambitious artificial intelligence experiments to straightforward uses of weblogs and email, authors are creating digital fictions at a wide range of technical complexity. The field is already too vast to cover in a single book.

Luckily, quite a number of books have already been written about digital literature, and many more have been written about digital media more generally. However, almost all of these have focused on what the machines of digital media look like from the outside: their output. Sometimes the output is considered as an artifact, and interpreted as a literary scholar or art historian might. Sometimes the output is seen in relation to the audience and the wider culture, as an educator or sociologist might. And there are, of course, a variety of other perspectives. But, regardless of perspective, writings on digital media almost all ignore something crucial: the actual processes that make digital media work, the computational machines that make digital media possible.

On one hand, there is nothing wrong with this. Output-focused approaches have brought many valuable insights for those who seek to understand and create digital media. But, on the other hand, it leaves a big gap.

This book is my attempt to help bridge the gap. As far as I know, it is the first book focused on computational processes that comes from the perspective of media, games, and fiction (rather than software engineering or computer science). It is a first passage across the gap, and we will want to move much more weight across over time. But hopefully it demonstrates that there is something to be gained by being able to move between the gap’s two sides, being able to see the inside and outside of digital media’s machines.

At this point I do a certain amount of signposting, laying out some of the chapter organization — which doesn’t seem necessary to include here. Then the chapter continues with a new section…

Authoring Processes

Above, I said that the possibility of creating new simulated machines, of defining new computational behaviors, is the great opportunity that digital media offers. Seizing this opportunity requires a bit of a shift. It is common to think of the work of authoring, the work of creating media, as the work of writing text, composing images, arranging sound, and so on. But now one must think of authoring new processes as an important element of media creation.

In undertaking this shift, it may be helpful to think of the creation of a piece of digital media as being made up of data and process, with a somewhat fuzzy line between them. The data are mostly pre-created media elements (text, still images, video and animation, sound and music) and the sorts of things that are stored in spreadsheets (lists and tables of information, with varying degrees of structure).

The processes, on the other hand, are the working parts of the simulated machine. Some are dedicated to tasks with simple structures, such as displaying a series of video images on a screen. But many of digital media’s tasks are more complex in structure, requiring processes capable of performing in a range of different ways. Even a simple piece of digital media such as Pong has processes that define behaviors much more complex than showing a series of images in quick succession. The processes of Pong define and calculate simple rules of physics (how the ball bounces off the paddles and walls) and simple game rules (who receives each serve, how points are scored, and how winning is achieved) that, when well-tuned, can combine to create a compelling experience of gameplay — even in the face of remarkably primitive graphics.

Of course, the idea of creating media through the authoring of novel processes is not new. Tristan Tzara’s Dada cut-up technique was presented, in the wake of World War One, as a process for turning a chosen newspaper article into a poem. On a more technological level, the pioneers of early cinema had to develop novel processes (embodied in physical machinery) to capture and display their sets of image data. And, on a longer-term level, the creation of board and card games has always primarily been the development of process definitions, embodied in game rules, that determine how play moves forward.

In important ways the non-computational media processes mentioned above are like the processes of digital media: they are defined previously, but (at least in part) carried out during the time of audience experience. This is true as Tzara pulls a paper scrap from his sack, as the Zoetrope image flickers, as the poker hand goes through another round of betting, and as the image of a Pong ball bounces off the image of a Pong paddle. The processes of digital media are, however, separated from non-computational media processes by their potential numerousness, repetition, and complexity. For example, we might play a game of tennis using the rules of Pong — they’re simpler than the normal rules of tennis. But we wouldn’t want to play Pong as a board game, having to hand-execute all the processes involved even in its (extremely simplified) modeling of physics. It is the computer’s ability to carry out processes of significant magnitude (at least in part during the time of audience experience) that enables digital media that create a wide variety of possible experiences, respond to context, evolve over time, and interact with audiences.

From here the text continues into the already-excerpted section on process intensity, from there continues into Expressive AI, and then transitions to some of the reasons I want to focus on processes from a critical (rather than practice-oriented) perspective. After that I go on to discuss issues related to those in my recent receiver article. Any requests for what excerpt comes next?