September 27, 2006
Approaches to Visual Programming
As I mentioned I’m currently developing authoring tools to be used for creating character behavior for interactive drama. Depending on how this is approached, this can overlap quite a bit with visual programming. I’m in the middle of researching people’s various approaches to visual programming, paying particular attention to user interfaces. (I’m still experimenting with what extent our authoring tools will expose the programming layer below, but I’d like to see how others have attempted to visually represent programs.)
In doing so I came across an very interesting ongoing project called Subtext by a research fellow at MIT named Jonathan Edwards, whose goal is nothing less than a complete revolution of how we program, to make a “fundamentally better programming language”. He’s created a pair of geekily entertaining demo movies that demonstrate a UI for programming in Subtext, definitely worth 30 minutes of your time.
Here’s a screen capture of Subtext from the movie, just to give you a flavor:
Here are three other (more conservative) visual programming examples.
Alice, by Randy Pauch and team at CMU:
(Here’s a recent discussion by Jonathan Edwards about making Subtext more Alice-like.)
The above Alice is not to be confused with Alicebot, which now has a new GUI authoring tool:
Behave!, by Michael Travers while he was a grad student at MIT, developed for use by museum-goers at the Virtual Fishtank exhibit at the Boston Computer Museum:
Anyone know of any other examples worth looking at? I’m particularly interested in how procedures are represented visually; I’m not very interested in seeing flowchart or state machine-like editors, since our character behavior goes beyond state machines.
September 27th, 2006 at 7:22 am
Alan Kay’s Squeak environment for Smalltalk – see http://www.squeakland.org and http://www.squeak.org – has been doing this sort of thing for a long time. I think there are a couple of demos & walkthroughs at the Squeakland site.
September 27th, 2006 at 9:06 am
I’ve been having my intro to computational media students use Quartz Composer, which is an interesting visual programming tool for graphical effects. There’s of course Max/Jitter and other vfx tools. You might check out Massive (the film vfx agent simulator). No new approaches to visual programming really in these, but some interesting approaches to the process of doing it.
September 27th, 2006 at 10:56 am
The original Lego Mindstorm robotics kit had an interesting drop and drag kind of thing to it. Too bad I can’t find any reference pictures of it.
September 27th, 2006 at 11:23 am
Is Max/MSP too flowchart-like to be of interest here?
September 27th, 2006 at 1:47 pm
Thanks for these suggestions.
Both Quartz Composer and Max/MSP/Jitter seem to be data flow programming, ie, plug the output of this unit/object into the input of that (see pic), eventually ending in display or sound — yeah, too flowchart-like to be useful for our purposes.
I had heard of Squeak and saw it listed on the Wikipedia visual programming page (I added Alice and Behave! to the list, btw), but have been having trouble finding documentation showing the interface; I’m trying to run the web demos via plugins now, but their server seems slow. Oops, now it seems Internet Explorer has denied the plugin installation because “it can’t verify the publisher”. (Anyone know why Firefox rarely seems to know how to automatically install such plugins? It never works.) Here’s a page with some sort of screenshot, and a page with Alan Kay demoing it.
Here’s a flowchart language for Lego Mindstorm, however I assume the RCX visual programming environment is what you remember Scott? (Note the description says is has “some serious limitations. The biggest problem is a lack of variables.” More variations found here.)
This page has an image of Virtools’ editor, for physics at least. I think the AI editor is state machine-based.
Here’s a nice survey from 1997 of visual programming approaches, with a set of references of papers from the 80’s and 90’s.
Here’s a short survey from 2002 of alternative programming paradigms.
Here’s a survey from 1999 of dialog management authoring tools, akin to the Alicebot editor shown above.
September 27th, 2006 at 1:59 pm
I just watched the subtext presentation. It’s fascinating. I have some thoughts and comments but I’m not sure I’m ready to vocalize them yet.
One thing that does strike me is the different approaches to making programming better or more accessible. Often, attempts to teach programming (or teach it better, or increase people’s facility for it) are tied to new programming paradigms or forms. I find this abstract approach quite different from Andrew’s goals here (or even Inform 7), which really have to do with creating specific new kinds of computational expression.
September 27th, 2006 at 4:26 pm
Right, the goal of making authoring tools for a specific purpose isn’t the same as the more general goal of a general-purpose visual programming language, e.g. Subtext. For example, the computational expressivity of patching-together of input/output units in Max/MSP/Jitter (from what I understand about them) is limited, but sufficient for the applications of manipulating audio and video.
Character behavior, though, can at times require pretty complex logic, sometimes requiring the full expressive power of a “general purpose” programming language, in my experience. ABL (see pdf), used to program the characters in Façade and branched/spun-off for commercial work at Procedural Arts (a new language name to be announced), has features tuned to the needs of creating characters, but it’s still got general-purpose power to it. One could program a word processor or utility application in ABL.
Much of the work in Façade was developing idioms for using ABL that targeted the needs of our approach to interactive drama (the kind of thing ABL and its parent language Hap were designed to create, even if they could be used for other things). That is, we developed particular techniques and patterns of usage of ABL; these new authoring tools will be designed around those idioms, with several new extensions / refinements.
Our idioms break down the problem pretty well, often to the point of reducing complexity in any one block of authored content, sort of divide-and-conquer strategy. It’s somewhat akin to creating smaller blocks of code in object-oriented programming, the complexity emerging from how the objects interact with each other. The idiom refinements over the past few months have gone even farther in this direction.
But, even those idioms often occasionally require bits of non-trivial programming logic within the smaller blocks of code. So… a solution to creating an visual programming tool for creating character behavior within our idioms, if effective, might in the end contribute some insight into how to design a general-purpose visual programming language. Likewise, I can steal ideas from other people’s work on general-purpose visual programming languages.
The “example-centric programming” (pdf) (sort of combining coding and debugging together) and “first class copy and paste” concepts are two of the very interesting ideas in Subtext (the author describes how these ideas are derived from a history of related research). Realistically though, these ideas are too radical and time-consuming to implement for the first pass of these authoring tools, but are great food-for-thought and inspiration for the inevitable future round-2 of development, and/or another phase of research.
September 27th, 2006 at 7:08 pm
Andrew hit on the main comment I was going to make—I’m most familiar with the dataflow style, having done a good bit of Puredata hacking (the open-source version of Max/MSP), and the main problem I’ve encountered is that in any sufficiently complex project you do end up needing “the full expressive power of a ‘general purpose’ programming language”. More to the point, you need it in some way that isn’t extremely painful to use.
Puredata (and Max/MSP) give you a Turing-complete programming language, so the lack of expressive power per se isn’t the problem; the problem is that if you try to do anything that doesn’t neatly fit the dataflow paradigm, it quickly gets complicated. A moderately simple example is keeping a “score” in a sequence of note-info in an array, and then iterating over the array to output the proper noise in sequence. Iterating over an array is trivial in “normal” programming languages, but in Puredata, you end up simulating it with a monstrous assortment of delay lines feeding into cells that hold pointers until a trigger line causes the pointer to be sent out, like this. Not impossible, and at this level even manageable if you’re a Pd hacker, but things more complicated than that would quickly get very unpleasant.
So I’d be interested to see how other visual programming languages would solve that problem. Is it something like letting you “escape” into old-fashioned text code if necessary?
September 27th, 2006 at 7:17 pm
Oops, I was going to add that the above reasons are why I’ve switched to using ChucK, an old-fashioned text-based language, but specialized for music and dataflow. There are pros and cons. If you’re doing heavily dataflow stuff, it’s somewhat less pleasant. You can still “connect” objects: for example, “sinosc x => filterfoo => dac” instantiates a sine-wave oscillator at frequency x, runs it through filterfoo, and then pipes that to the soundcard output. But if you have complicated graphs of connectivity, you have to build them up line by line like that, instead of visually placing units and connecting them. But things like iteration and perfectly simple, and I find that the slightly harder dataflow in ChucK is much easier to live with than the much harder non-dataflow in Pd. Some hybrid might be nice; I’m currently screwing around with a programming aid that will output a dataflow-like graph visualizing the connectivity of ChucK objects at any given time, though admittedly that’s a visual aid to textual programming rather than visual programming.
But that’s been my general experience so far: It seems more difficult, to me at least, to “escape” from a visual convention that doesn’t fit my needs than it is to approximate a visual convention in text. Admittedly I don’t have experience with much of a range of visual-programming languages, though.
September 27th, 2006 at 10:16 pm
As Dan mentioned above, this work looks most similar the smalltalk/squeak tradition – visual, “alive” editors for code rather than dead text, ability to make on the fly changes to a running programming (no distinction between code, compilation and running), etc. I remember stumbling across a smalltalk book 1985 – having only programmed in textual languages at that point (basic, pascal, cobol, fortran, assembly), I was blown away. It was like programming from the future.
Symbolics lisp workstations also had that “something from the future dropped back in time” feel. Lisp (and other functional languages with a Top Level) have that incremental “putting little pieces together dynamically and seeing what they do” feel that comes across in the Subtext video.
Regarding other visual programming approaches, two that fit in the “fill in the form” model of the Alice, Alicebot and Behave editors are:
The Never Winter Nights ScriptEase authoring tool (auto-generates NWN script).
Chris Crawford’s SWAT for the Storytron.
I can see that fill-in-the-form approaches can be successful for supporting specific idioms (this is the approach the ScriptEase folk take), but not as the sole interface for general purpose programming.
Mark points out the main problem that has plagued the visual programming community: complexity management. Nobody has figured out a way to manage really large programs visually. Textual code representation still remains the best way to manage big codebases. The intuition that “a picture is worth a thousand words” sure sounds compelling – if a picture could replace a thousand lines of code, than imagine how much complexity could be represented in a screen full of pictures. Unfortunately, a picture is only worth a thousand words in the connotative sense – and code is depressingly denotative.
In 1984, Scientific American ran a cover article on Visual Programming Languages, using an image from an experimental language developed by Jaron Lanier. A few minutes of googling have not been able to retrieve this image, but what I remember (from 22 years ago) was a musical staff with all kinds of cool icons, like flying birds, laid out on the staff. I remember thinking (again coming from a basic-pascal-fortran-cobol-yadayada background) “Cool! This will be the way we program computers!” Too bad the promise of visual languages never really worked out.
September 27th, 2006 at 11:57 pm
Smalltalk itself doesn’t appear to be an inherently visual programming language; are you saying the IDEs for Smalltalk tend to be GUI like, like Subtext is? And the reflective nature of Smalltalk lends itself to that? (I saw a bit of Squeak’s visual editor in the Alan Kay video I linked to above, but there’s virtually no Squeak editor screenshots I can find…)
“fill in the form” model of the Alice, Alicebot and Behave editors
Alice seems like real coding with training wheels — more than just “fill in the form” , that Behave! seems to be like. Based on the screenshot of Alice it looks like you can (almost?) write arbitrary code, create whatever functions you want, etc. The visual editor is essentially just graphically showing you the structure and scoping of the code you’re writing, like color-coded auto-indentation, and not allowing you to create syntax errors. It also seems to have some macros / hidden function calls custom to the domain of 3D virtual world programming.
I think several of these principles could be applied to a character behavior authoring tool.
Generally speaking, training wheels like this seem like a good way to learn to program. Once you learn it however, you’re ready to take off the cumbersome training wheels; writing code in Alice is probably slower than freeform typing.
September 28th, 2006 at 12:19 am
Yes, Smalltalk itself is a textual language, but smalltalk programming environments tend to provide dynamic GUI object browsers.
I agree that form-like interfaces can scaffold programming, and within a specific idiom, may give you a good chunk of the expressive power you need. For our current Procedural Arts project, the goal is not to support non-programmers in building believable characters in general, but rather to speed up our own production pipeline, which is a different goal than learning to program.
September 28th, 2006 at 12:52 am
Right, although the more accessible our authoring tools are to non-programmer (yet technically-oriented) designers, the better, since designer/programmers are hard to find.
Looking at the Swat tutorial, it looks rather, shall we say, complex.
September 28th, 2006 at 1:35 am
I don’t have proof of this, but my gut feeling is that it’s very difficult to simultaneously design for ease of novice use and efficiency of expert use all in one package. I may be somewhat biased as a command-line/Unix bigot, but I find that the things I think are “easy to use” are nearly the exact opposite of the things my less-technically-oriented friends find “easy to use”, mainly because they optimize for how easy it is to do something the first or second time, while I optimize for how easy it is to repeatedly do something hundreds or thousands of times. Not to say it’s impossible, but it would be quite a feat, since I think each problem is independently still fairly open. :)
September 28th, 2006 at 1:38 am
Swat looks as complicated as Erasmatron (some of my students played around with Eras in my interactive narrative class a few years ago).
Ian and I are IMing right now about form-like interfaces (like Alice’s interface). Ian takes the hard-line position that form-like interfaces aren’t good for anything, even teaching people to program. In working with undergrads with very little programming, he has them writing Atari 2600 programs in BASIC within a couple of weeks. But when he tries to use visual environments, like Quartz, the student’s don’t produce good work.
The main thing that form-like interfaces do (not the “expose a few parameters” type, but the “GUIify code” type), is eliminate syntax errors. You can only make syntactically well-formed statements; you still have to worry about the semantics (the interesting part), but not syntax at the same time. This can be effective pedagogically – in teaching new programmers in my Computation as an Expressive Medium class (in Processing, which is Java), some people really would get syntax and semantics confused. Because they had to worry about both at the same time, they had a hard time learning either. But the price the form-like interfaces make you pay is lots of cumbersome interface manipulation. Further, just like I’ve argued before about natural language input vs. menus for games, by “menuizing” programming, such interfaces make you look at the possible choices you could make at every choice point. You’re working on some code that, say, has nothing to do with multiplying, but there’s that damn multiply operator saying “you could multiply here, you could multiply here, you could multiply here” (just like “you could fart here, you could fart here, you could fart here” in Fable). Subtly turns programming into a “choose from menus” exercise rather than the open-ended writing it really is. As you (Andrew) says, perhaps useful training wheels, but, like training wheels, something to then abandon.
In arguing against form-like interfaces even for pedagogical purposes, Ian says “hey, we don’t teach writing that way. Students learn syntax and semantics at the same time.” But, our natural language compilers (our brains) are extremely syntax lenient. Imagine if a English 101 teacher, when grading essays, stopped at the first ungrammatical sentence or misspelling, put a big red-mark there, and handed back the paper with a 0 saying “completely unreadable, try again.” It would be hard to learn to write. But this is what compilers do to us. So perhaps what we really want is a super syntax-lenient programming language, something that tries to get the gist as best it can. Then you’d have the rapid expressiveness of a textual language without the brittleness of syntax. Like if we applied Façade’s noise-tolerant semantic-parsing technique to a high-level scripting language for characters. But you will need some AI heavy lifting to fill in the gaps. And sometimes it would completely misunderstand what you’re trying to do, something generally not considered OK in a programming language.
September 28th, 2006 at 1:42 am
Re: Mark’s comment: I see our design goals more as making the authoring system understandable yet powerful, tools that allow you to create sophisticated character behavior in a way understandable to programmer-lite designers. Perhaps this is somewhere between novice and expert; you need to learn how the system and architecture operate in order to author, which is non-trivial (more than novice). But once you learn that, you’re empowered, and in fact there may not be a whole lot more to learn, where “experts” aren’t doing much more than regular authors. I don’t know, we’ll see.
Also, I should add, the more accessible the authoring tools are to a generative AI, that is, the simpler the “hooks” are in editor, the closer we are to creating an interactive drama generator AI in the future…
September 29th, 2006 at 5:53 am
I didn’t look too closely at all the previous discussion, but one interesting language/environment I’ve come across is called impromptu. ( http://impromptu.moso.com.au ) It’s a Scheme IDE specialized for “live-coding” and so has a very tight writing/testing loop, as well as special facilities for time scheduling and dataflow graph management. I have no idea how well any of this would map to your problem.
Also, I’ve had this paper (“Usability Analysis of Visual Programming Environments” — Green, Petre) sitting around since a seminar I took on languages. Still have to read it, so I’m not sure how good it is, but maybe it would help raise some important issues regardless.
October 1st, 2006 at 9:18 pm
For a simpler take on how to put together a powerful interface, check out Ableton Live – http://www.ableton.com/. It’s very good at making music tools accessible and is usable for both composition and real-time performance. It’s simple, consistent and works the way you think it should. Everything works as the program is running so you can ‘perform’ and ‘compose’ simultaneously…we always look at this before building new applications.
October 2nd, 2006 at 1:36 am
The main goal of a development environment is to increase productivity. If, as has been said, the target is not newcomers with little or no programming experience, but people in the team working for months or years with the toolset, the focus has to be different.
I don’t think that a visual interface can compete with a text-based one today, when an experienced programmer is driving it. Attempts like Swat, ScriptEase or the others mentioned above try to simplify the interface so that newbies can tinker with it, mostly using trial and error, and generate something reasonable. They can’t do that if they are staring at a blank page and a thick help file. They need the list of available choices, the menus that Michael talked about.
For experienced people, you don’t spend most of your time writing code. You are either thinking about it, debugging it (fixing errors), or modifying it slightly until you get the expected result. A good environment has therefore to be able to:
– Structure your code properly, so that you know what to place where, and you can find it quickly and easily when you are looking for it. It means enforcing some high-level patterns. You could argue that UML class diagrams are like that for OOP. Imagine an editor where you could click on a class representation and edit the code for that member function. The same could be said for a state machine or a petri net editor. If you click on a state or a transition, then you can add a snippet of textual code to that node, and you keep the general structure without losing expressivity. Event-based editors like the Starcraft and Warcraft ones (can’t find a picture right now) also follow this pattern: provide places to insert conditions and actions, and let them be as general as needed.
– Make iterations quick. To me, the greatest feature of Subtext is the ability to see the result of your program as you are writing it. It lets you find errors quickly, and correct them immediately. Of course, without a functional language it’s harder to do what Subtext does, but having an editor that is connected with the game, and being able to reset and test the script without restarting is nearly as good. The focus here is in how dependencies work in the script, so you can isolate the piece you are working on and test it without having to reload everything else. Again it’s very language dependant. For a functional language it’s easy, there is no state. For a state machine, you probably would have to save some kind of checkpoint and be able to go back to it to test, being careful to properly update it if you modify the script code that defines that checkpoint (like adding a new variable).
– Have good error reporting and diagnostics. If you can write script and test it immediately, and the environment will warn you with a nice message explaining what’s wrong when you make a mistake, then you don’t really need an editor that will prevent you from making syntax errors. Any good programmer makes reasonably few mistakes and can correct them quickly. Of course if you have enough time and resources, a development environment that will let you catch mistakes sooner or even prevent them altogether is very nice, but I’d rather spend my time building a good reporting system, that can catch not only static syntax errors, but runtime ones. You’ll make mistakes anyway that the GUI might not catch.
I guess that what I’m trying to say is don’t be afraid of text. You already have a programming language. Text is easy to read and easy to write, easy to find and replace, to copy and paste. Any visual interface has to offer all these features and that requires a lot of work. And it’s never as quick as doing the same thing with text. Find a way to organize that text, and provide good tools for iteration and bug fixing, and experienced people will be much more productive. Then you can worry about inexperienced people, and create a new layer on top of everything (like ScriptEase) building nice forms for them to fill, and not ask them to learn to program.
October 2nd, 2006 at 6:12 am
(I posted this 3 days ago, but it did not appear…)
Very interesting discussion!
As suggested by Mark, it seems that “easy to use” progamming languages do not exist. First it depends on the user, in particular the difference between novices and experts. Second most languages who pretend to be easy to use are finally either limited when you want do advanced stuffs or too complex for novices anyway.
I believe that the most promising track is to work on the programing environment rather than the programing language. I played recently with Alice, and beyond its “form-like” feature, it is also an environment where you see directly the results of your programs. The “no distinction between code, compilation and running” mentioned by Michael is a great feature to work on.
I have been designing lately an authoring tool for Interactive Drama (on paper only), and I came to the point where I would need an environment where you debug before programming (as mentioned above), a situation which is exactly opposite from classical “serious” programming: design carefully your programs to avoid bugs!
Without going as far as “noise-tolerant semantic-parsing” (Michael), the following features are of interest:
– immediate testing of the program
– testing even with missing parts (unitialized variables, 3D character unfinished, etc.)
– following in real-time the execution of the program
– modifications on the fly
– support of “shortcut programming” (I am making up the term): if a part is not developed, enable to hardcode it in a special case before programming it really.
– programming from the concrete to the abstract (from the linear too the generative): start with an execution and induce the code, either manually or automatically (cf “example-centric programming” mentioned above).
All these feature certainly exist in some environments. They are particularly relevant in artistic creation: An artist need to experiment things. The challenge is to have these features alltogether for a complex language like ABL.
October 3rd, 2006 at 3:08 pm
Sergio, Nicolas, your suggestions are good ones, thanks. (Nicolas, sorry if one us accidentally deleted your comment; we often delete spam comments en masse, sometimes deleting ham in the process.)
I agree that one of Subtext’s most attractive features (which isn’t exclusive to Subtext or its approach) is the ability to see the result of the function you’re currently writing, aka immediate testing of the program, aka to what Michael termed as no distinction between code/compilation/execution. But, with interactive drama, there is *a lot* of state, and so it gets pretty tricky to achieve this.
November 6th, 2006 at 5:18 am
Have you seen Quest3D? It’s a realtime visual programming environment for realtime 3D applications. We make The Endless Forest with it and are using it for Drama Princess as well, the latter also being focussed on autonous behaviour of virtual characters.
Blender also has an interesting visual programming system in its game engine, called called Logic Bricks. And it appears that the Unreal engine now has a visual scripting system called Kismet.