April 5, 2008
Blog-Based Peer Review: Some Preliminary Conclusions, part 2
[This is a continuation of part 1]
The version of the Expressive Processing manuscript used for both forms of peer review begins with an introductory chapter composed, in part, in response to a desire to let people know what is at stake right up front. I wrote it to let readers know, from the beginning, what I am advocating and why it matters to me. I also wanted a first chapter that could be assigned as a stand-alone class reading (as so many monograph chapters are) and function to make my case.
In the blog-based review I got a number of important comments on this chapter, especially on my discussion of process intensity and The Sims. In the course of that discussion I also learned a number of things about the blog-based review form that still hold true in my conclusions about this project. I outlined them first in my “meta” post between chapters four and five. The relevant section reads:
In most cases, when I get back the traditional, blind peer review comments on my papers and book proposals and conference submissions, I don’t know who to believe. Most issues are only raised by one reviewer. I find myself wondering, “Is this a general issue that I need to fix, or just something that rubbed one particular person the wrong way?” I try to look back at the piece with fresh eyes, using myself as a check on the review, or sometimes seek the advice of someone else involved in the process (e.g., the papers chair of the conference).
But with this blog-based review it’s been a quite different experience. This is most clear to me around the discussion of “process intensity” in section 1.2. If I recall correctly, this began with Nick’s comment on paragraph 14. Nick would be a perfect candidate for traditional peer review of my manuscript — well-versed in the subject, articulate, and active in many of the same communities I hope will enjoy the book. But faced with just his comment, in anonymous form, I might have made only a small change. The same is true of Barry’s comment on the same paragraph, left later the same day. However, once they started the conversation rolling, others agreed with their points and expanded beyond a focus on The Sims — and people also engaged me as I started thinking aloud about how to fix things — and the results made it clear that the larger discussion of process intensity was problematic, not just my treatment of one example. In other words, the blog-based review form not only brings in more voices (which may identify more potential issues), and not only provides some “review of the reviews” (with reviewers weighing in on the issues raised by others), but is also, crucially, a conversation (my proposals for a quick fix to the discussion of one example helped unearth the breadth and seriousness of the larger issues with the section).
On some level, all this might be seen as implied with the initial proposal of bringing together manuscript review and blog commenting (or already clear in the discussions, by Kathleen Fitzpatrick and others, of “peer to peer review”). But, personally, I didn’t foresee it. I expected to compare the recommendation of commenters on the blog and the anonymous, press-solicited reviewers — treating the two basically the same way. But it turns out that the blog commentaries will have been through a social process that, in some ways, will probably make me trust them more.
Conversations such as these let me know that I would need to revise some sections of the introductory chapter. In the peer-reviewed version of the manuscript, the rest of the book, rather than focusing on high-level arguments, was shaped by my preference for engaging deeply with a relatively small number of examples. I hoped to show, from the inside out, how a path through the particular digital media that interests me could support the argument I made in the first chapter. I also hoped to show how one would discuss such work differently, after accepting the argument made in the first chapter. In this way, the particular examples were important (of deep interest to me) but at the same time incidental (example-focused chapters with the same relationship to the first chapter’s broad argument could have been written, I felt, by someone with an interest in computer music or generative visual art, rather than my particular interests).
In this section of the manuscript I continued to have helpful conversations that pointed toward revisions. At the same time, the conversations also took a new turn, because the centrality of certain examples meant it was possible to get comments from people with expertise of a particular, exciting, kind: the project creators. This led to some of the best conversations, involving people like Scott Turner (Minstrel), Jeff Orkin (F.E.A.R. and The Restaurant Game), and Andrew Stern (Facade). The last person mentioned in the previous sentence, of course, is also a GTxA blogger. The mix between GTxA reader comments and GTxA co-author comments was also present for examples I touched on more briefly, as with GTxA reader Richard Evans (Black & White) and GTxA co-author Nick Montfort (nn).
It was also during this period that I dropped the ball on some discussions. If I remember correctly, McKenzie Wark told me that he had a non-teaching term when he was responding to the comments on the GAM3R 7H30RY project. It would have been useful for me to have had the same — or even a single course release. But it would certainly take a shift in academic attitudes to allow faculty to apply for release time to participate in community discussions around already-drafted, not-yet-published manuscripts. I suspect this will remain true for a while, even if we continue to find, as I have, that such discussions can lead to valuable insights for improvement in the book before it goes to press.
On a related note, people have asked me if one feature of the discussions around this portion of Expressive Processing, the involvement of project authors, is something close to idiosyncratic, possible only for someone writing on a topic like mine. Certainly I feel that digital media is a lucky area in which to work, on this front, because so many of the authors of field-defining projects are still alive (and online). But I think the same sort of blog-based review involving project creators could happen for authors in many other areas. For example, during the recent Writers Guild of America strike a light was shown on the involvement of many movie and television writers in blog communities. I would not be at all surprised if such writers, already engaged in reading and writing on blogs, took an interest in academic writing that discusses their work — especially as part of a blog-based peer review that might generate revisions before the text is put into print. But only further experimentation will reveal if I am correct in this.
Another question, posed in response to the same “meta” post mentioned earlier, is whether this form of review only works because I have already developed some reputation in my field (e.g., from editing other books). My belief is that my personal reputation is not the primary issue. Rather, it is Grand Text Auto‘s reputation that matters. It makes sense to do a blog-based review because we have, in blogs, already-existing online communities that attract university-based experts, industry-based experts, and interested members of the public. The way we use blogs also already encourages discussion and questioning. This connects to my main critique of the GAM3R 7H30RY project: starting a new site for each manuscript reviewed, and gathering a new community to that site, is not a sustainable model for review, and not likely to result in an optimal signal-to-noise ratio. Inserting a new form of review into the ongoing flow of conversation on an existing blog makes much more sense.
Of course, this experience has also indicated that inserting review into an ongoing blog conversation can be problematic in some ways. I will discuss that, and more, in the next part of these preliminary conclusions.
April 6th, 2008 at 9:34 am
I’m not sure those critiques are all that different, though. Someone not well established in the field who writes a new book will probably also not be the author of a well-established blog in the field on which to post the manuscript, so they’ll have to go a different route.
April 6th, 2008 at 12:06 pm
That’s definitely true. Right now it feels to me like academic blogging is gaining momentum, so that many people in digital media who would have a book under blind peer review from an academic press would also be part of the community of a blog in the field. I can imagine Grand Text Auto hosting a blog-based review for someone who is a member of the community but not one of the primary authors (e.g., a regular reader whose work we admire). I can also imagine us hosting a review for someone who is a colleague of ours who doesn’t have an established blog (e.g., if Kate Hayles had wanted to do this kind of review for her recent Electronic Literature book).
But maybe a better model (than expanding the role of existing blogs) is something like the planned MediaCommons network. This would have an ongoing flow of conversation, and an established community, but not as small a circle of authors as a typical academic blog. People who are authors of established blogs might connect their efforts to the MediaCommons network, but host them on their own blogs. Those who are not authors of established blogs might host them directly with MediaCommons.
April 7th, 2008 at 10:22 am
A flip side of this is that examples that didn’t seem to particularly grab anyone and which didn’t attract engagement from their creators (like Brutus) didn’t produce much conversation. I was a bit surprised by this, given that some of what I wrote in these sections seems controversial. For example, Brutus has previously been treated as a major story generation system, and I characterized it as something close to a literary hoax…
November 4th, 2008 at 3:48 pm
[…] cf. Wardrip-Fruin’s concern that “starting a new site for each manuscript reviewed, and gathering a new community to that […]