May 14, 2003
Expressive AI
Hi, I’m Michael Mateas. My particular interest in the topics of this blog is thinking about, and making, artificial intelligence-based interactive experiences. AI techniques enable interactive art to be more porous to human meanings, to generate responses that more deeply incorporate interaction. In my work I’m interested in developing AI techniques and architectures that enable new forms of interactive experience. I’m currently working with Andrew Stern on the interactive drama Façade, which he described in more detail in his first post. Previous work includes Terminal Time, an interactive video piece that constructs ideologically-biased documentary histories in response to audience feedback, and Office Plant #1, a desktop robot that responds to the social and emotional tone of the email you receive.
One strand of work I’m interested in is interactive narrative. The well-known conundrum, which has already been raised in the opening comments on this blog, is how to combine interaction, which seems to be about the freedom to do what you want, with narrative, which seems to be about tightly crafted sequences. I believe AI techniques will be vital in creating interactive story systems that can generate large numbers of tightly crafted sequences that incorporate player interaction. That is, such systems support the author in capturing their authorial intention at a more abstract level than handcrafted sequences – this more abstract description is then used to generate the concrete sequences that result during any one play of the story. For example, in Façade we have authored a collection of dramatic beats, individual dramatic moments or episodes, that can sequence together in a fairly large number of ways. Whenever a beat is chosen to be sequenced, the exact choice made depends on what’s happened in the story so far, including player interaction. The beats are described using a custom language to specify the various conditions under which it is appropriate to choose a beat. Each beat is itself a mini story machine, a collection of character behaviors that can incorporate player interaction within the beat to accomplish the dramatic moment of the beat in a number of ways. The behaviors are written in another custom language designed for authoring real-time, reactive characters. There is of course more to the Façade architecture than this, but it gives an example of what I mean by specifying a story at a more abstract level. Incidentally, Andrew and I in no way feel that we have “solved” the interactive story problem. After having spent almost four years building Façade, we are keenly aware of the many limitations of our approach, which I’m sure we’ll discuss on this blog. There are many approaches to capturing story structure at a more abstract level – we’re both excited to move forward with new projects that build on the lessons learned in Façade. (For more on Façade, see Architecture, Authorial Idioms and Early Observations of the Interactive Drama Façade).
Skeptics may feel that I’m saying that all the problems of interactive art have a technical solution – just use artificial intelligence and presto, you have art. By this way of thinking, art becomes an application area of AI. Every art project should start with the step “First, build a general purpose AI” (watch out, that first step is a doozy). But AI-based art and entertainment is not simply an “application” of AI. Combining an art practice and AI research results in an new agenda, one I call Expressive AI. Expressive AI raises AI research questions that would never come up unless doing AI research in the context of art making, and opens up new opportunities in interactive art that would never arise unless making art in the context of a research practice. I’m looking forward to talking more about Expressive AI on this blog. If you would like to read more now (probably much more than you want to read) see my dissertation, Interactive Drama, Art and Artificial Intelligence.
May 15th, 2003 at 1:37 am
One thing that I think would be interesting is if a “drama manager” sort of AI could be more introspective–to have goals that it is trying to accomplish and to “understand” how its actions are approaching that goal, rather than just having a bag of things to try and a fitness function to choose between them. (But maybe the latter is the former, who knows?)
I think Douglas Hofstadter’s AI research–published in the book Fluid Concepts and Creative Analogies–might be of interest, since it is an attempt to model certain kinds of creative processes that achieve some sort of aesthetics. I can’t summarize the book here, but two important aspects are that they’re working in microworlds, and that they’re primarily modelling an unconscious creative process, not a conscious one.
I got a chance to talk to him about this a little, since games as microworlds seem a pretty good fit, but he wasn’t at all interested in that side of things–he’s less interested in application, more just in pursuing AI to understand human mental function. But he did mention that one of the projects, Letter Spirit, he felt was too automatic, that it was using his unconscious-process modelling paradigm solely, and needed a certain amount of introspecting in what it was trying to accomplish and how to go about it; but I couldn’t get a clear idea from him what he thought this looked like from an implementation standpoint. Is it just a planner? I don’t know.
May 15th, 2003 at 11:50 am
I completely agree with the fact that Interative Drama systems should catpure “authorial intention at a more abstract level than handcrafted sequences”, and to support this idea, I would like to refer to my own project on Interactive Drama. The project, called IDtension, have globally the same goal as Façade, in terms of combining interactivity and narrative. The underlying models are very different: it does not use the concept of beats, but works at the more basic level of actions and tasks. The author is writing actions, tasks, goals, obstacles, ethical values and the system is responsible of aranging this in terms of sequences of events. But as in Façade, there is this need to write at an abstract level. So I believe this is a very general property of “expressive AI”.
May 15th, 2003 at 12:44 pm
Michael has talked some to me in the past about Hofstadter’s work and how it might apply to interactive narrative, so it’s definitely on the radar. We’re each very interested in exploring AI techniques for creativity. Michael is more up to speed on it than me, I’ve only read a few papers/books on it so far, such as Scott Turner’s MINSTREL and Kim Binsted’s JAPE. I’m really looking forward to reading all of the literature in the field once I find some free time. For sure, it will be technically intense stuff.
The biggest lesson we’ve learned from Facade is the need to take an even more generative approach to interactive story. The drama manager approach has been fruitful, but it only works well if there is a big enough bag of content for the drama manager to choose from. The process of more-or-less manually writing slews of various hierarchies of behaviors has been the most laborious part of the project for us, and the reason the project has gone on at least a year longer than we originally planned for. Once the architecture was complete, it’s taken over 2 person-years of work just to create a short, varied but still limited, dramatic situation about a marriage falling apart.
Nicolas’ work, mentioned in the previous comment, is an example of new research in generative, interactive narrative; definitely check his papers out. From what I understand, among other things, his research goals include ways for the system to begin trying to “understand” the dramatic qualities of the experience in real-time, and somehow modulate the system’s response to enhance these qualities.
It would be great to have a general discussion of the merits of various AI & creativity techniques, in future threads of this blog.
May 15th, 2003 at 1:59 pm
Hi,
I will just briefly introduce myself first. I am Magy Seif El-Nasr, just finished my PhD on Expressive lighting for Interactive Narrative. My view of interactive narrative has been more of a filmmakers’ view, where the drama manager combines (a) (similar to Michael and Andrew’s work) techniques for selecting story beats and (b) a visual plan to visually present them integrating camera, lighting, blocking, and postures/gestures.
I agree with Nicolas, tension is important to promote in an interactive narrative. I worked with many artists to produce Mirage (an interactive drama based on the Greek Tragedy Electra), we tried many techniques to keep the drama and tension and promote beats and character goals that aid in establishing the right tension. The technique that we finally adopted was a user-centric architecture based on Benedetti’s acting theory where beats are further divided into tactics that are selected by characters, and where the characters monitor the effect of the tactic on the user’s actions (mouse clicks, time to respond, etc.).. I didn’t yet test that with users, but I thought it was an interesting idea to try…
Also related to tension, from my experience writing the story, it seems, at least to me, that the success of tension or drama heavily rely on the setup and the existence of a clock. For example, in Mirage the clock was ‘Orestes’s arrival’. In the story, Orestes is the user’s double, his position or the meaning of the clock really depend on the user’s position and actions… But the clock was always helpful to increase the tension whenever needed.
Of course, investement is a very important element to keep the tension and drama going.
Another issue/comment that is worth mentioning also, is user modeling. In Mirage, the user’s personality was modeled as a point in a multi-dimentional scale describing his character as a coward, self-interested, passive, violent, or a reluctant hero. This, I think, helped us define the interactivity model within Mirage… As the story unfolds, the user’s character becomes more defined within this scale and his choices become more limited and the character’s reactions towards him starts to form depending on the personality. I thought that was also an interesting point, because I think user modeling is an important technique that may open up many doors for interactive narrative design.
Anyway, that was my two cents…
May 16th, 2003 at 11:28 am
Andrew says, in a comment above, “The biggest lesson we’ve learned from Facade is the need to take an even more generative approach to interactive story.” I can see how it might look that way to one of the authors right at the end of the massive two-person project. But that may not be the lesson that the field takes away.
Instead, much of the field may think, “The lesson here is that this human-authored approach worked while none of the attempts at ’emergent’ narrative have.” And, frankly, hiring a roomful of writers (even good ones, with track records) is quite a bit easier (and more sensible) than tackling unsolved AI research problems when one is trying to create a major art/entertainment project (the kind with budgets and deadlines).
In other words, I think the Facade approach, with the massive human authoring effort involved, may soon be seen as the only ‘production quality” approach we have to interactive drama.
May 17th, 2003 at 10:13 am
Thanks Magy for updates on your project. Many projects on Interactive drama seem to be based on a kind of user model. This is not that surprising: if we want to make AI expressive, then a natural way for AI people is to “calculate” this expressiveness. Two kinds of user models however can be distinguished: in Magy’s example (coward, self-interested, passive, violent, reluctant), it is the model of the user-as-a-character; but when the user model tries to catch the “dramatic qualities of the experience”, then it is more the model of the user-as-a-spectator. I implicitely use both in IDtension, but the relation between the two is still an open question to me, and I would be happy to discuss that.
To answer Noah’s comment, I would say that AI-based generativity is not just a matter of saving writing time. With an authored approach, the user feels like reading what has been written by an author, while with a generative approach, the user feels he is producing a situation, and that the system, on behalf of the writer, reacts to this situation, in a unique way. For example, in Façace, which is already generative up to a certain point, at any moment, the user can walk to the painting on the wall and comment it; the author has not predicted that the user would do that action at that precise moment, but he (with the programmer) has found a way to react to that event in a generic way that should work everytime. But in Façade, the user might perceive that his action “just” triggers some pre-written beats… or he might not!! We’ll see…
May 21st, 2003 at 3:07 am
Since you might be watching this thread I thought I might ask —
When will the rest of the world get to play with Facade? And if not the whole enchilada how about some of the pieces?
adam…
May 21st, 2003 at 10:08 am
We’re planning on releasing Facade in December 2003. We’re working hard to reach alpha in the next couple of months.
May 1st, 2005 at 4:21 pm
Michael how would you say Facade compares to the 1960’s ELIZA?
January 11th, 2007 at 10:42 am
[…] From here the chapter transitions into a discussion of concepts such as Michael’s Expressive AI (on which, perhaps, more later). If there’s interest, I may share m […]