Archive for the ‘Annotation Tools’ Category

T-PEN: A New Tool for Transcription of Digitized Manuscripts

October 22, 2012

One of the exciting turn of events for scholars has been the growing number of unpublished, hand-written documents now available on the world wide web. Textual scholars no longer have to travel to distant countries for view the essential manuscript(s) for their research. Instead, they can now sit themselves down in front of their laptop and display each successive page. This has moved many sources that were once difficult to access into the “completely accessible” category.

But does that make them usable?  Despite the desire to make many manuscript collection freely accessible, many digital repositories use “tiled-based” viewers in order to protect unauthorized copying of the collection. This is completely understandable, but those viewers sometimes place limits on how a digital surrogate can be viewed. They can even make it difficult for scholars to extract what they often want most: a transcription of the manuscript’s content. Moreover, the current practice of transcribing from digitized pages can easily permit mistakes to occur. Transcribers currently move from the image to a word processing application in another display window (either on the same screen or on a different monitor). That process can easily mimic the same mistakes that the original scribe could make: haplography (omission of content between similar or identical words; “saut du même au meme”), dittography (repetition of letters or syllables), duplication or omission (of letters, words, or lines), often caused by homoearcton and homoeoteleuton (similar beginnings and endings of words), and transpositions. Could it then be possible to make these digital manuscripts both accessible and highly usable?

T-PEN (Transcription for Paleographical and Editorial Notation) seeks to address both the accessibility and usability of digital repositories. Developed by the Center for Digital Theology of Saint Louis University, in collaboration with the Carolingian Canon Law Project of the University of Kentucky, this new digital tool is a sophisticated web-based application that assists scholars in transcribing these manuscripts. To reduce the likelihood of transcription errors, we took advantage of digital technology to place both the transcription and the exemplar in a manner that minimized the visual movement between the two as much as possible. We accomplished this with a simple but novel visualization of the lines of script in the exemplar, which we integrated with interactive transcription spaces. To build the tool, we developed an algorithm for “parsing” the lines of script in an image, and a data model that connected the image delivery of manuscript repositories with the actions of transcribers.

But we wanted T-PEN to offer more than just a means to ensure good transcription. We had, in fact,  three goals in mind:

  1. To build a tool useful for any kind of scholar, from the digital Luddite to those obsessed with text encoding;
  2. To provide as many tools as possible to enhance the transcription process;
  3. To help scholars make their transcriptions interoperable so that those transcriptions would never be locked into the world of T-PEN alone.

After two years of design, development, and intensive testing this tool is now available to the wider public. It was built in the first instance for those working with pre-modern manuscripts, but there is nothing in its design that would prevent early modern scholars from exploiting T-PEN for their purposes. T-PEN is a complex application and to explain every function would take several posts. Instead, I want to provide a brief overview of how someone can set up a transcription project, how they can use T-PEN to produce high-quality work and finally how to get transcriptions out of T-PEN and into other applications or contexts.

Choosing your Manuscript

T-PEN is meant to act as a nexus between digital repositories and the scholar. To date, we have negotiated access to over 3,000 European manuscripts and we are working on further agreements to expand that list. Our aim is to have a minimum of 10,000 pre-modern European manuscripts available for transcription. Even with that number, we will never be able to satisfy all potential users. We therefore enabled private uploads to extend T-PEN’s usability. Many scholars have obtained digital images of a manuscript and they have permission to make use of them for research purposes. Private uploads to T-PEN are an extension of that “fair use.”  Users zip the JPG images into a single file and then upload them to T-PEN. These type of projects can only add five additional collaborators (see project management, below), and they can never become public projects. Currently T-PEN can support around 300 private projects, and we are expanding our storage capacity for more.

T-PEN's Catalog of Available Manuscripts

Transcribing your Manuscript

Once you select your manuscript you can immediately begin your transcription work. T-PEN does not store any permanent copies of the page images, so each time you request to see a page T-PEN loads the image from the originating repository. If you have never transcribed the page before, T-PEN takes you to the line parsing interface. This adds a little time to the image loading as T-PEN parses the image in real time. When it finishes, you will see a page that looks like this:

T-PEN's Line Parsing Interface

T-PEN attempts to identify the location of each line on the page and then uses alternating colors to display those coordinates. As you can see, we make no claim of absolute perfection. We worked on this algorithm for  almost two and half years and after extensive testing, we’ve been able to promise, on average, an 85% success rate. There are a number of factors that prohibit complete accuracy and so we offer a way for the transcriber to introduce corrections herself. You can add, delete or re-size columns; and insert or merge lines as well. You can even adjust the width of individual lines if they vary in length. You can even combine a number of lines if you want to have them grouped together for your  transcription. Sometimes, manuscripts don’t merge well in our modern, rectilinear world: many handwritten texts were written at an angle or were so tightly bound that the page could not be photographed as flat. T-PEN ultimately doesn’t care: what really matters for connecting transcription to a set of coordinates on a digital image. What really matters is that the left side of the line box aligns with the written text. That’s the anchor.

When you are satisfied with the line parsing, you can start transcribing. The transcription interface looks like this:

T-PEN Transcription User Interface

This interface allows you to transcribe line by line, with the current line surrounded by a red box. There are some basic features to note. First, as you transcribe the previous line is noted above because so often sentence units are split across lines. Transcription input is stored in Unicode and T-PEN will take whatever language set the user has enabled his computer to type. If there are special characters in the manuscript, the transcriber can insert them either by clicking on the special character button (the first ten are hot-keyed to CTRL+1 through 0).

Second, users can encode their transcription as they go. On this aspect, T-PEN is both innovative and provocative. Many scholarly projects that include text encoding often adopt a three-step process: the scholar transcribes the text and then hands it to support staff to complete the encoding, which is finally vetted by the scholar. However, there are many times in which semantic encoding of transcriptions has to include how the text is presented on the page. T-PEN innovatively allows scholars to integrate transcription (with the manuscript wholly in view) and encoding into one step. Often the best encoder is the transcriber herself. That innovation comes with a provocative concept, however. In digital humanities where TEI is the reigning orthodoxy, T-PEN is at least heterodox if not openly heretical. T-PEN’s data model does not expect,  nor require, a transcription to be encoded much less utilize TEI as the basis of structured text. Instead, T-PEN treats all XML elements as simply part of the character stream. T-PEN can support transcribers who don’t want to encode at all as well as those who are wholly committed to the world of TEI. For those who want to encode, a schema can be linked to a project to produce a set of XML buttons that can be used in the transcription interface.

Project Management

For those who simply want to start transcribing, project management will not be that important. For those who envisage a more sustained project (and perhaps a collaborative one at that), it will be vital. There are a number of components in managing a T-PEN project, but here I want to highlight two of them.

Collaboration. Like most digital tools, T-PEN allows you to invite collaborators to join your project. All members of a project have to be registered on T-PEN (but that’s free and requires only providing your full name and an email address). Managing collaboration has three features, of which only a few projects will use all three. There is first adding and deleting project members. Any member of a project can see who is also a member, but only the project leader can add or delete members. A project leader can even have T-PEN send an invitation to a non-T-PEN person and invite them to join (and once they do, they automatically become part of that project).

Collaboration in Project Management

Second, there is a project log to inspect. This log records any activity that changes the content or parameters of the project. This can be particularly helpful when tracking down how a transcription has changed in a shared project (and a user can display the history of each line in the Trasnscription UI). Finally, projects can make use of T-PEN’s switchboard feature. This is for transcription projects that may be part of a larger project, and where the transcriptions will be aggregated in another digital environment. Switchboard does two things for a project: (1) it allows different projects to share the same XML schema so that all transcriptions will conform to the larger project’s standards; and (2) it will expose the transcription through a web service to permit easy export to the larger project.

Project Options. The two more important options are button management and setting the transcription tools. As seen in the screen shot of the transcription interface, users can use buttons to insert both XML elements and special characters. Those buttons are created and modified as part of the project options. If there is an XML schema for the project, a project leader can link it to the project. Then in button management, the elements in that schema populate the XML button list. The button populator does not discern between metadata elements and elements found in the body of an encoding schema. Users then have to modify the button list to cull the elements that won’t be used during transcription. There’s an additional advantage to editing that list: each button can gain a more readable title. This can be helpful if the encoding schema exploits the varying use of the <seg>  or the <div> elements in TEI. When the possible deployment of the tag might be unclear to those with less experience with TEI, a more straightforward title can become a better guide to its use.

Special characters allow the user to identify characters in the UTF-8 system which may not be represented on a standard keyboard. These can be created by entering the correct Unicode value for the character. The first 10 characters are mapped to hotkeys CTRL+1 through 0.

Finally, the set of tools that are available on the transcription interface are set in project options. T-PEN has thirteen tools built-in and most of them were included to assist transcribers of pre-modern manuscripts. Some will be helpful to editors of modern texts. If those tools are unhelpful, then the user can expand that list of tools: all that is needed a name of the tool and its URL. Once attached to the project, the user will be able to access that tool in the transcription interface.

Getting your Transcription out of T-PEN

Digital tools often fall into one of two categories. “Thinking” tools are ones that allow users to manipulate and process datasets in order to test a certain idea or to visualize an abstract concept. They can also allow the user to annotate a resource as a way of processing the scholar’s conception of the object’s meaning or the hermeneutical framework it may require. These tools are invaluable, but they do not easily produce results that can be integrated into a print or digital publication. The second type is what I call the production tool. With these applications, the final objective is to produce something that can be integrated in other contexts. T-PEN falls firmly into this second category—although it has its own annotation tool with which a user can record observations about each manuscript page (and it is compliant with the W3C standard, the Open Annotation Collaboration). Scholars transcribe normally one of three reasons; to create a scholarly edition; to place those transcriptions in footnotes or in the appendices of a monograph; or to integrate an encoded text into a larger resource.

T-PEN supports four basic export formats: XML/plaintext, where the user can filter out one or more XML tags; PDF; RTF which is compatible with most word processors; and finally, basic HTML. For the first one, if the user has attached a header to the project, that header can be included in the export. There is an important caveat here:  T-PEN was not designed to be an XML editor. We do offer a basic, well-formedness check (which stops at the first error), but T-PEN does not offer full validation services. Most scholars who encode with T-PEN export their transcriptions to an XML editor for full validation of the file. The last three export formats include some simple transformation for text decoration (italics, bold, etc.). Users can also identify the whole transcription or specify a range based on the pagination (or foliation) of the manuscript.

T-PEN's Export Options

This post only covers the basics of T-PEN. There are more features available to the user. There is a demonstration video on YouTube  where you can walk with one of T-PEN’s research fellows as she begins a transcription project.  T-PEN is freely available, thanks to a major investment from the Andrew W. Mellon Foundation and a Level 2 Start-up grant from the National Endowment for the Humanities. So go to t-pen.org and register for an account.

“The Past Has Arrived”: NYU’s Conference on Digital Media, Teaching, and Scholarship

May 5, 2012

Martha Rust (NYU) recently organized an inspiring conference on digital tools called “The Past Has Arrived: The Digital Middle Ages and the Renaissance.”  The tools discussed usefully supplement books in both teaching and scholarship.

Annotation tools like Digital MappaeMundi–now re-branded as DM–allow users to annotate and link images and texts.  In the image below, downloaded from the DM web site, purple annotation selects material on the twelfth-century world map from Sawley Abbey in Yorkshire (left screen) and links it to text on the right screen.  The text can similarly be formatted or annotated to include links to relevant sites, images, or glosses, such as entries in the Dictionary of Old English.  Martin Foys and Shannon Bradshaw (Drew) and Asa Mittman (Cal. State, Chico) presented an introduction, a technological context, and an application of this tool.

Image from Digital MappaeMundi

Visualization tools, such as Mapping Gothic France, allow users to view representations of medieval buildings in staggering detail.  MGF presents twelfth- or thirteenth-century cathedrals in France “in terms of sameness and difference found in the forms of multiple buildings within a defined period of time and space that corresponds to the emergence of France as a nation state,” according to its web site.  The photographs–and there are tens of thousands of them–are strikingly clear and the site is interactive, so that one can navigate the interior of cathedrals as if one were flying through them.  Those raised on Harry Potter will be particularly happy with this feature.  The views would once have been considered nearly unobtainable.  Click on the following screen shot for a larger image.

Screen shot of Mapping Gothic France home page

What’s striking about this project is that it supplements book technology.  “Architecture doesn’t fit tidily into the pages of a book,” co-administrator Andrew Tallon (Vassar) explains in an interview with Chronogram.  Indeed, this five-year project designed by Tallon and Michael Murray (Columbia) demonstrates how digital media can provide features that a book can’t or rarely offers.  Using MGF, students can manipulate maps to see the sequence in which Cathedrals were built, zoom in on architectural details, view floor plans, read narratives associated with a building, and even use a simulation tool to experiment with the physics of stone arches.

Michael Witmore’s keynote talk “What Is Access?” provided an overview of the history of Docu-Scope, which was designed to help teach freshman English but functions in surprisingly innovative ways to annotate texts.  It categorizes words into types, and generates charts of word strings that force a re-consideration of texts, such as Shakespeare’s plays, in new ways.  Witmore distinguished between archives–that maze of material books shelved within a given collection–and the archive available by a digitized database or tool.  Docu-scope might be said to re-shelve Shakespeare’s oeuvre by suggesting surprising points of contact between plays divided generically.  As noted in an earlier emob entry, Witmore finds points of contact between Othello and the comedies.  It also exposes what is odd about a given play.  The following screen shot downloaded, not from Witmore’s NYU talk, but from his blog, Wine Dark Sea, shows how Docu-scope found a high frequency of words denoting motion and spatial relations in A Midsummer Night’s Dream.  Those words appear underlined in yellow below (click on image to enlarge and clarify):

 

The preponderance of such motion words, once we see them, makes immediate sense in a play featuring fairies and other supernatural creatures that move in ways that humans cannot.  One of Docu-scope’s gifts is to help us see formal aspects of a text that we might not otherwise see.  In this sense, digital tools can provide access to linguistic features of a text less likely to be found by human reading.

Cataloguing tools was the topic of my discussion of EEBO InteractionsEEBO Interactions facilitates “relational cataloguing,” allowing entries to link to ODNB entries, or to related texts within EEBO, or to articles, or to spaces where bibliographical  and critical issues can be discussed.  In the past, an EEBO user might have found the following entry to be something of a dead end:

Clicking on the text bubble by the author’s name calls up the corresponding EEBO Interactions page, which identifies J.V.C. as a Catholic priest and provides brief biographical information.

Scrolling down the EEBO Interactions page, one would also find relevant links.  Because users can add pertinent information for either the author or the text title, those working on little known work can, if they wish, share their expertise and enhance catalogue entries. This kind of relational cataloguing capitalizes on current technology and points the way to the future.

Pedagogical tools were the focus of several talks. These included the Medieval Narrative Project, designed by Evelyn Birge Vitz and Marilyn Lawrence (both at NYU), which collects video clips of performances of medieval texts.  Other teaching aids included Second Life, which Martha Driver (Pace) had her students use to construct avatars engaged in medieval contexts.  The afterlives of these projects, which continued to be used beyond the end of a given course, suggest that students enjoyed imagining medieval life through this technology.

Theoretical and practical issues were also probed.  On the practical side, Consuelo Dutschke, Curator of Medieval and Renaissance Manuscripts at Columbia University, argued eloquently for the value of projects like the Digital Scriptorium, which, in addition to collecting images segregated by disparate archives into one database, also allows a “diverse community of medievalists, classicists, musicologists, paleographers, diplomats and art historians” to help strengthen cataloguing.  Similarly, Stephen Nichols (Johns Hopkins) and Nadial Altschul (Johns Hopkins), editors of Digital Philology: A Journal of Medieval Cultures, discussed some of the technical and pragmatic issues that emerged regarding digital publication, including the difference between a link to a work of art and its printed reproduction, or how royalties affect what can be included in digital publications.   More theoretical speculations included concerns expressed by Alan Galey (University of Toronto) regarding textual variation: how can interface design help organize text, textual notes, and commentary?  The Visualizing Variation project demonstrates how digital media provides innovative features, such as animated variants, for textual editing.  Nicola Masciandaro mediated on how digital tools produced “textual shapes” other than the article or the monograph.  Bill Blake discussed keywords, a topic he broached at the ASECS meeting in April and developed further here.  How can searching be conceptualized so as to explore, rather than reproduce an archive?

A final keynote delivered by Stephen Nichols on “The Anxiety of Irrelevance: Digital Humanities and Medieval Literary Scholarship” probed the ambivalence prompted by digital humanities projects.  He argued that there need not be a disconnect between the goals of Digital Humanities projects and those of traditional humanists, but that more attentive listening and understanding of questions at hand is necessary.  The day-long conference and the discussions that it fostered well into the evening, including at a lively dinner, helped advance that needed conversation.

This entry was posted on May 5, 2012 at 1:47 pm and is filed under Uncategorized. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site. Edit this entry.


Follow

Get every new post delivered to your Inbox.

Join 121 other followers