February 16, 2008
EP Meta: Chapter Four
This week, when I was talking with Jessica Bell about her story for the Daily Pennsylvanian, I realized one of the most important things, for me, about the blog-based peer review form. In most cases, when I get back the traditional, blind peer review comments on my papers and book proposals and conference submissions, I don’t know who to believe. Most issues are only raised by one reviewer. I find myself wondering, “Is this a general issue that I need to fix, or just something that rubbed one particular person the wrong way?” I try to look back at the piece with fresh eyes, using myself as a check on the review, or sometimes seek the advice of someone else involved in the process (e.g., the papers chair of the conference).
But with this blog-based review it’s been a quite different experience. This is most clear to me around the discussion of “process intensity” in section 1.2. If I recall correctly, this began with Nick’s comment on paragraph 14. Nick would be a perfect candidate for traditional peer review of my manuscript — well-versed in the subject, articulate, and active in many of the same communities I hope will enjoy the book. But faced with just his comment, in anonymous form, I might have made only a small change. The same is true of Barry’s comment on the same paragraph, left later the same day. However, once they started the conversation rolling, others agreed with their points and expanded beyond a focus on The Sims — and people also engaged me as I started thinking aloud about how to fix things — and the results made it clear that the larger discussion of process intensity was problematic, not just my treatment of one example. In other words, the blog-based review form not only brings in more voices (which may identify more potential issues), and not only provides some “review of the reviews” (with reviewers weighing in on the issues raised by others), but is also, crucially, a conversation (my proposals for a quick fix to the discussion of one example helped unearth the breadth and seriousness of the larger issues with the section).
On some level, all this might be seen as implied with the initial proposal of bringing together manuscript review and blog commenting (or already clear in the discussions, by Kathleen Fitzpatrick and others, of “peer to peer review”). But, personally, I didn’t foresee it. I expected to compare the recommendation of commenters on the blog and the anonymous, press-solicited reviewers — treating the two basically the same way. But it turns out that the blog commentaries will have been through a social process that, in some ways, will probably make me trust them more.
Turning to other topics, the last week wrapped up a fourth Expressive Processing chapter: “Making Models.” Next week will begin, but not complete, another chapter: “The Tale-Spin Effect.” The last week also saw a crop of interesting new commenters — including people such as Richard Evans and John Cayley, whose work I very much admire. I’m looking forward to continuing the conversation next week, and welcome here any broader discussion of chapter four and/or the three chapters that form EP’s major discussion of “the Eliza effect” (chapters two, three, and four).
February 18th, 2008 at 2:57 pm
I completely agree with your conclusion here. In response to the many voices that have argued that with blog-based reviewing the essences of an authority is lost, I feel that the authority is not lost at all but rather from where it comes has been shifted. Instead of looking to a handful of academics to correct and improve the work we now can look to the world that actively choses to involved itself with this work. Thus you have intelligent individuals sharing their views, and should those views be incorrect or not as well organized and developed as possible, other commentators will correct them. Or perhaps you as the author can through commenting as well. Therefore as you said, the work improves not through the views of the few who may or may not be 100% correct on their suggestions, but through the views of the many who through the conversation will have a higher percentage of always being the best advice.
But I wonder, is the opposite true? Does such a democratic process prevent true change, since the canon of tradition is always very powerful and influential. But then again perhaps it is naive to think that change occurs through one or two individuals and that in reality those figure heads of change are just that, they represent the larger population that in a traditional sense cannot express itself.
Keep the conversation going,
Huysmans
February 18th, 2008 at 4:43 pm
Thanks for your supportive note. My hope is that we can find ways to draw on the best aspects of both models, with Expressive Processing as just one early experiment toward that goal.
February 18th, 2008 at 4:57 pm
[…] thought-provoking “meta-post” from Noah Wardrip-Fruin on Grand Text Auto reflecting on the blog-based review of his new book […]
February 18th, 2008 at 8:37 pm
I think it is a choice, not either or. It is up to whether you need peer review or open public review. I think the age of long tail is ultimately a good thing, but in the short term people will feel pain having to make much more decisions. Don’t we already feel pressure to read more, because there are more interesting blogs that other people seem to read? I think our brain is still in mass production age, where an educated person was expected to know what other educated people know. For a while, tools will help make us more productive. But ultimately, we will give up and define what we follow narrower.
By the way, I am writing a book(?), which will include a chapter on this, at http://paragraphr.com/pages/show/11 (right now it is a mess). And I am actually doing a similar experiment of open feedback. But only one comment so far. That’s probably another challenge for a less known writer.
February 19th, 2008 at 8:50 am
Hyokon, I’m glad you’re interested in exploring open feedback as well.
My belief, put into practice with this project, is that open feedback is a more sustainable model if it is situated within an already-existing community. That’s why I’m doing this review at Grand Text Auto, instead of on a site created for the project. Maybe there’s an existing community within which you could situate your review?
February 21st, 2008 at 5:04 pm
I agree with your conclusion about the value of networked review. A logical next step is to question the exclusive role that closed peer review plays in academic recognition, and to offer alternatives for the network age. I’ve ranted about the former problem in essays like “Three Threats to the Survival of New Media.”
http://newmedia.umaine.edu/interarchive/three_threats.html
On a more practical level, U-Me’s Still Water program recently released “New Criteria for New Media,” a white paper and set of criteria for promotion and tenure of new media practitioners.
http://newmedia.umaine.edu/interarchive/new_criteria_for_new_media.html
http://newmedia.umaine.edu/interarchive/promotion_tenure_redefinitions.html
Ironically but appropriately, our field’s oldest peer-reviewed journal, Leonardo, will be publishing these in a future issue and has begun a discussion of the future of scholarly evaluation on Leonardo Electronic Forum. We invite anyone to join in the debate.
http://artsci.ucla.edu/LEF/node/104
jon
March 11th, 2008 at 12:17 pm
[…] and using CommentPress to engage in a conversation with readers. In reading over Wardruip-Fruin’s meta-reflections on blog-based peer review, I was struck by his observation that getting feedback from multiple reviewers helps him to figure […]