Readers will be interested in a series of essays on the evaluation of digital scholarship edited by Susan Schreibman, Laura Mandell, and Stephen Olsen and published in the recent issue of MLA’s Profession.
These essays are freely available as PDF files. Their titles are as follows:
“Introduction,” Susan Schreibman, Laura Mandell, and Stephen Olsen
“Engaging Digital Scholarship: Thoughts on Evaluating Multimedia Scholarship,” Steve Anderson and Tara Mcpherson
“On the Evaluation of Digital Media as Scholarship,” Geoffrey Rockwell
“Where Credit Is Due: Preconditions for the Evaluation of Collaborative Digital Scholarship,” Bethany Nowviskie
“On Creating a Usable Future,” Jerome McGann
“Peer Review, Judgment, and Reading,” Kathleen Fitzpatrick
In introducing the essays, the editors point to national calls for clearer guidelines for evaluating digital scholarship:
National scholarly organizations such as the Modern Language Association and the American Council of Learned Societies have called for department and institutions to “recognize the legitimacy of scholarship produced in new media, whether by individuals or in collaboration, and create procedures for evaluating these forms of scholarhsip” (Report of the MLA Task Force).
This publication provides an opportunity for emob’s readers to discuss how digital scholarship might best be evaluated and to raise questions about the process of evaluation.
Early Modern Online Bibliography 
December 17, 2011 at 8:10 am |
To access an individual article, click on the “Profession” link above. Scroll down the TOC; click on the essay’s author; click on “abstract”; click on PDF.
LikeLike
December 18, 2011 at 11:15 am |
Thanks for this, Anna. I’m glad that the contributors address the larger issues of knowledge production and academic labor, but I think this follows from the welcome emphasis on evaluation. The crux I think stems from encompassing the diversity of practices and approaches within a single discipline, if we can say that English departments or literary studies represent a discipline. But I do think it’s high time to learn the cost of eliminating bibliography from the official grad curriculum. This could bring such practices back on a new basis.
LikeLike
December 18, 2011 at 10:31 pm |
Thanks, Dave. I’m interested in Kathleen Fitzpatrick’s argument about tenure and promotion reviews. She argues that
I like the idea that we should read our colleagues’ work and engage with it. But does reading that work automatically make us adequate evaluators?
This seems like an important question.
LikeLike
December 18, 2011 at 11:32 pm |
i’ll look at fitzpatrick’s piece, but in the meantime I think the engagement issue is really crucial to dealing with the evaluation issues raised by a diverse set of research approaches, so that faculty develop a collective sense of their work (Wergin). Departments can’t run on autopilot, or expect others to take care of these issues.
LikeLike
December 19, 2011 at 10:05 am |
I agree. But reading for engagement differs from reading for evaluation. The latter requires acquiring familiarity with the state of the field in order to measure a given work’s contribution. There is much to be gained from such evaluative reading: faculty, departments, and the profession would be enriched, as Fitzpatrick argues. But whether faculty will invest in such evaluative reading and whether they can do it successfully are two separate and important questions.
In Enemies of Promise, Lindsay Waters argues forcefully that evaluation should not be outsourced and that faculty have an obligation to use their judgment. I buy his argument. But I can also imagine objections to it.
LikeLike
December 19, 2011 at 11:25 pm |
Sure, but the logical conclusion is that specialists can only evaluate fellow specialists, in fragmented fields already rife with internal disagreements. And most sub-fields are too small to be guaranteed representation in departmental settings. I think if you’re the lone bibliographer or linguist or 18c person in a department, then there should of course be some effort to have the work evaluated by people in the “same” field, but the variety of work happening in English departments has always made this an aspiration rather than an expectation. And why shouldn’t scholars who depend on editors and translators and digital designers everyday at least participate in an evaluation process?
LikeLike
December 20, 2011 at 8:29 am |
Good points. I also think that Fitzpatrick encourages literary scholars to push beyond sub-disciplinary boundaries. The move toward becoming a generalist, someone more knowledgeable about the variety of pursuits that literary studies might include, seems healthy, as I stated above.
But that move requires labor, just as Fitzpatrick’s peer-to-peer review requires labor. And it’s materially uncompensated labor, though one could argue that it brings about the best compensation: greater understanding of the field, broadly conceived. Thus, not only is she asking us to rethink our sub-disciplinary boundaries; she is also asking us to rethink our system of labor and rewards.
I can imagine someone objecting that we have a first duty to master our sub-fields in all their complexity, and that that process has become more intensive. What we’re really discussing is the nature of our expertise.
LikeLike
December 20, 2011 at 9:30 am |
I don’t think we’re talking about entirely new practices, but acknowledging and conceptualizing the diversity always contained in departments. (We’ve been discussing this at long 18th, too) We already have rhet/comp and ethnic or cultural studies alongside creative writing etc. etc., plus the period-based sub specialties. I don’t see any benefit from an assumption that no one can read or evaluate another’s work in our depts. Perhaps the academic presses supported this illusion at one time, but no longer. But yes, this is about how we define and recognize expertise. I just think this happens in a local and departmental context as well as one’s PhD program.
LikeLike
December 21, 2011 at 9:55 am |
But it is it safe to say that literary studies now encompasses so many different kinds of studies and practices that appreciating intellectual labor (its intensity, its complexity, its intelligence) is complicated? Fitzpatrick’s call for “engagement” requires an appreciation of different kinds of labor. Can tribalism complicate fair judgment?
LikeLike
December 21, 2011 at 12:45 pm
Of course. It already does, once you recognize how p&t operates within a college or university, which by no means take peer review as the last word. But if you read Tony Becher’s work, a discipline is always constituted in two dimensions, by its objects and practices, but also as a community of very human members, with all the complexities that that fact entails (conflicts, disagreements, etc.)
LikeLike
January 2, 2012 at 1:30 pm |
Sorry to join this fruitful conversation so late. You both make excellent points. There are obstacles to evaluating work in other fields and subfields, yet I worry about the practice of devolving the responsibility for judging this work on putative experts, as the problem of “tribalism,” as you nicely put it, Anna, is often counterpoised by the problem of turf battles within a given field or subfield. As a book review editor, I am struck by how many puffs and pans litter the world of reviewing. In many instances, of course, the reviewers have a stake in the argument, which is scarcely conducive to objective evaluation. On the other hand, “interloping” reviewers may not have the background necessary to render an informed judgment. “Peer-to-peer” review holds promise as a way of including a wide range of voices in the conversation, but ultimately, such problems may be insoluble.
The MLA’s emphasis on evaluating online resources is, however, most welcome. As Kathleen Fitzpatrick, Geoffrey Rockwell, Bethany Nowviskie, and others observe, digital media often entail different evaluative criteria from traditional media: hypertext is radically different from linear text; databases and digital tools are different from monographs. Moreover, collaborative scholarship is more common–or at least more evident–in digital projects than it is in print publications, so the question of allotting responsibility is a pressing one. Although they raise as many questions as they answer, the contributors to the recent PMLA volume usefully highlight important features of the changing landscape.
LikeLike
January 3, 2012 at 10:44 am |
Rrhumanist, you raise an excellent point about book reviews. The intense communal feeling that characterizes a sub-specialty could just as easily enforce a group-think; conversely, what might seem trivial to the unitiated could be crucial information for the sub-specialist. No form of scholarly evaluation can be free of these kinds of risks, even if everyone works in good faith.
The best principle to counteract this phenomenon, I believe, is to try to gather information from as many sources as possible, to make sure that the form of the evaluation is appropriate to the process or product being evaluated, and to acknowledge that different kinds of activities may demand different kinds of evaluation to distinguish the best work.
LikeLike
January 4, 2012 at 9:16 am |
The issue of book reviews always interests me. Blogs offer interesting possibilities, including the kind of group reviews Dave has started on the Long 18th, a practice we have adopted here. In this model, each reviewer takes a chapter, but I wonder whether it would be productive to include several reviews of the entire volume by different reviewers. The range of responses might tame panning instincts while encouraging a serious articulation of differing points of view.
This kind of collective review might also help us articulate the accomplishments of new DH projects, an articulation that might be helpful to members of t&p committees. 18thConnect already provides letters for their editing contributors that articulate the accomplishments of a given digital edition.
This new pressure on reviewing–whether it is reviewing DH projects or scholarly activity of t&p candidates or book reviews–requires the best judgment and research. For this to happen, we need to weigh reviewing more heavily in the review process. Are we moving away from a celebrity model of intellectual labor to a more modest but perhaps more serious and effective model?
LikeLike
January 4, 2012 at 11:07 am |
Now that I’ve had the opportunity to read most of these articles, I would add that Geoffrey Rockwell’s essay on evaluating digital scholarship is really good at describing the kinds of protocols that would help departments to assess their members’ work. And there’s the added benefit that evaluating DH work also makes us more aware of elements like scholarly genre and form, more appreciative of collaboration and process, less inclined to treat our own field and its conventions as settled.
But of course the precondition for all of this is a department engaged enough in its members’ own work that members feel obliged to do this kind of work, and do it well.
LikeLike