joewiz

Even if you don't know XQuery, if you've only heard of it, you know XQuery is a language built for querying, or searching, XML. And if you've started learning XQuery for a digital humanities project, you use XQuery because it helps you search the text that you've put inside your XML. Given your interest in searching text, it's likely that the first function you learn in an XQuery class or tutorial is contains(). Take this simple XPath expression:

//book[contains(title, "arm")]

This economical little query searches through all the books in your XML for those with a title that contains the phrase "arm" — you know, all of your books about armadillo shwarma. Tasty, right?

Then in the next lesson you learn about the matches() function, which can do the same simple searches as contains() but can also handle patterns, expressed using a pattern matching language called regular expressions:

//book[matches(title, "[A-Za-z]\d{2}T!")]

This finds titles like "W00T!" and "L33T!" — an upper or lower case letter, two digits, and a capital T, and an exclamation mark. Slick!

Then, naturally, you learn the highlight-matches() function, which turns highlights the phrase or pattern that you searched for:

//book[highlight-matches(title, "[PT]ickl[a-z]+")]

This highlights the matching portion of the book titles: "The Art of Pickling" and "How to Tickle the Ivories like a Pro." Super! 

...

But wait! The highlight-matches() function never appears in your lessons or class materials. It's not in the XQuery spec. Not the 1.0 version, not the 3.0 version. Surely, this must be a mistake. No, your teacher says. You google for it. You click through the links. Stuff about indexes? Proprietary functions? The disappointment sets in. Really? No standard way to highlight the search results?

This was my experience, lasting several years, until today. I realized that I could combine two features of XQuery 3.0 — the analyze-text() function and higher order functions — to write a simple, implementation-independent highlight-matches() function, allowing us to write queries like this:

local:highlight-matches(     //book/title,     "[PT]ickl[a-z]+",      function($word) { <b>{$word}</b> }     )

To make this easier to read, I've split the expression onto several lines. Here's what's going on:

This should look pretty familiar: we return all the pickling and tickling titles. But instead of applying contains() or matches(), we use local:highlight-matches(). And instead of putting the function inside a predicate (i.e., [inside square brackets]), we put the function outside. This is because our function doesn't merely serve as a condition (i.e., return all books whose title matches); it actually creates an in-memory copy of the nodes that meet the condition with the highlight function applied.

Whereas we gave contains() and matches() two parameters (title and the phrase/pattern), we pass local:highlight-matches() a third parameter: a function. You may not have ever used a function as a parameter, but this is a perfectly valid thing to do in XQuery 3.0. It's the idea of "higher order functions" - or functions that can take other functions. The advantage of letting you define your own highlighting function is that you might not want to highlight with <b> tags. Rather, you might want to surround matches with a <person key="#smith_michael"> tag. In other words, you might use highlight-matches() to do more than "highlight."

Submit the query. The local:highlight-matches() function finds the matching books and works its magic, returning the titles with the properly bolded phrases <b>Pickling</b> and <b>Tickle</b>.

...

But wait! If this highlight-matches() function isn't part of the XQuery specification, where can you get it?

I've posted the source code to highlight-matches() as a GitHub gist. Copy and paste the code into any XQuery 3.0-compliant engine. Need one? Try eXist-db, which has a handy online sandbox called eXide that you can access from any web browser.

Once you get the sample code working, try writing your own highlighting function to return underlined text or text with a yellow background—or find instances of people whose names you know appear in the text and tag them using <person> or proper TEI.

And enjoy!

Another day, another gist. Today's was prompted by a question on the eXist-db mailing list about how to access OAuth-based services like the Google API with XQuery. I happened to have just been working on accessing the OAuth-based Twitter v1.1 API for the new social media section of my office's homepage, so I posted the code and some pointers. Like the gist I posted yesterday, I hope others can use these bits of XQuery code.

But there's a back story and, dare I say, some illustrative lessons, to this latest addition to my series of posts and gists on XQuery.

Until recently, writing a program to retrieve one's latest tweets was as simple as going to the Twitter homepage is: you just made a basic, unauthenticated HTTP request to Twitter's servers to get the data you needed. But with version 1.1 of Twitter's API, Twitter announced a new requirement - that all requests to its API be signed and authenticated using the OAuth 1.0 protocol. This complicated the task of getting data from Twitter exponentially. The OAuth protocol, while not rocket science, requires one to jump through a rather intricate sequence of steps to compose the parameters of your request, and then cryptographically sign the request with a hashing function. (I'm not complaining about the protocol; it does a great job providing an authentication layer to the web. I'm just saying that requiring OAuth to retrieve tweet imposes a pretty heavy burden on users and developers.) If that weren't enough, Twitter also ended support for the XML-based Atom format, leaving just JSON as the format it returned results as. That left me with two problems.

First, XQuery's rich function library does not include the HMAC-SHA1 cryptographic hashing algorithm needed to sign OAuth requests. So I turned to Claudius Teodorescu, who applied his considerable Java skills to the task of creating an HMAC-SHA1 function for eXist-db, the XQuery-based server that powers history.state.gov. We took it a step further, releasing Claudius's work to the EXPath community in the form of an specification: the EXPath Crypto Module. The EXPath community builds up common standards for XPath and XQuery implementations. Claudius also released his module as an EXPath package for eXist-db, which is now available in the eXist-db Public Package Repository for anyone to download and install (to do so, go to eXist-db's Dashboard, click on Package Manager, and find "EXPath Cryptographic Module Implementation" in the list of packages"). Look at the prolog of the OAuth module I posted in today's gist, and you'll see that it imports Claudius's module.  

So I was able to check OAuth off my list of problems.  

But besides handling OAuth, I also needed a way to deal with JSON. JSON is increasingly ubiquitous data format in the world of APIs, but its data model is subtly incompatible with XML, and XML-based software like eXist-db has a difficult time ingesting or searching JSON data. Luckily, there were a number of XQuery libraries for me to choose from, and I decided to use one that John Snelson wrote for XQilla. With his permission, I updated it a bit, using some new features in XQuery 3.0 to make his library implementation-independent, and released the updated library on GitHub. Thanks to GitHub's mechanisms for code contributions ("pull requests"), the library has already received several improvements from the community. The package is also available in eXist-db's public package repository and the CXAN package repository. (I'm also eagerly following the JSONiq project which is working on extending XQuery to deal natively with JSON, obviating the need to convert JSON to XML to deal with it.)

So I was able to check JSON off my list of problems.  

This paved the way for yesterday's addition of social media links to the homepage of history.state.gov, and coming soon, a complete, searchable social media archive. 

All in all, a story — albeit not unique — of open source communities working together to build solutions to common challenges.

Helps trim phrases of arbitrary length to a maximum length, without cutting off words, as the substring() function would inevitably do, or ending on unwanted words

Where does each sentence in this post start and end? Given some schooling and well punctuated text, our brains handle this task pretty easily, but it turns out that telling a computer how to split a text into sentences is a bit tricky. In modern English we have a general rule: sentences begin with a capitalized word and end with a period. But there are plenty of exceptions to account for in writing a program to isolate sentences: other words in the sentence might be capitalized, and abbreviations can contain and end with those periods, whether they’re at the end of a sentence or not.

For some time, I’ve wondered about how to find the start and end of sentences, but couldn’t ever devise an approach that worked. Then, recently, inspired by my friend Josh’s comment during a course we were taking on XQuery ("Could we use XQuery to pull out all topic sentences in a manuscript to help ensure the narrative flows logically and smoothly?"), I decided to return to the challenge. After some research I found a hint in this post on stackoverflow, which unlocked a core insight: if you look at the individual words in the text, one at a time and in order, you can look for signs of a sentence break, and then apply logic against surrounding words and account for known exceptions to the rule, such as abbreviations or stock phrases. Proceeding this way through a text, you can isolate each sentence.

So, should you ever have the need for splitting text into sentences—perhaps for looking at all topic sentences in a chapter, or for counting sentences in a paragraph—check out https://gist.github.com/joewiz/5889711. It’s a pair of XQuery functions for analyzing a chunk of text and identifying the sentences within. It’s a naive approach (see my notes at the top of that page), but it does a pretty good job with newspaper articles and other edited English prose.

It takes text like this paragraph (from FRUS):

154613. You should arrange to deliver following note to North Vietnamese Embassy. If in your opinion it can be done without creating an issue, we would prefer that you ask North Vietnamese Charge to come to your Embassy to receive note. “The U.S. Government agrees with the statement of the Government of the DRV, in its note of April 27, that it is necessary for Hanoi and Washington to engage in conversations promptly. The U.S. Government notes that the DRV has now agreed that representatives of the two countries should hold private discussions for the sole purpose of agreeing on a location and date. The U.S. Government notes that the DRV did not respond to its suggestion of April 23 that we meet for this limited purpose in a ‘capital not previously considered by either side.’ The U.S. Government suggested the DRV might wish to indicate three appropriate locations suitable for this limited purpose. The U.S. Government does not consider that the suggestion of Warsaw is responsive or acceptable. The U.S. Government is prepared for these limited discussions on April 30 or several days thereafter. The U.S. Government would welcome the prompt response of the DRV to this suggestion.”

and returns this:

154613.

You should arrange to deliver following note to North Vietnamese Embassy.

If in your opinion it can be done without creating an issue, we would prefer that you ask North Vietnamese Charge to come to your Embassy to receive note.

“The U.S. Government agrees with the statement of the Government of the DRV, in its note of April 27, that it is necessary for Hanoi and Washington to engage in conversations promptly.

The U.S. Government notes that the DRV has now agreed that representatives of the two countries should hold private discussions for the sole purpose of agreeing on a location and date.

The U.S. Government notes that the DRV did not respond to its suggestion of April 23 that we meet for this limited purpose in a ‘capital not previously considered by either side.’

The U.S. Government suggested the DRV might wish to indicate three appropriate locations suitable for this limited purpose.

The U.S. Government does not consider that the suggestion of Warsaw is responsive or acceptable.

The U.S. Government is prepared for these limited discussions on April 30 or several days thereafter.

The U.S. Government would welcome the prompt response of the DRV to this suggestion.”

I chose this text because of the many capitalized words and abbreviations throughout and the variations in punctuation. I also tested against several New York Times and Boston Globe articles, a tricky portion from Moby Dick that threw off some other utilities, and some made up sentences with edge cases.

If you want to give it a try with your own text, you can copy the entire gist and paste it into eXide, the XQuery sandbox for eXist-db; click “Run" to see the results. (Should work with any XQuery implementation though.)

Two weeks ago several of my colleagues and I were lucky enough to take a course on "XQuery for Documents" taught by Michael Sperberg-McQueen.  

I've taken courses on XQuery before—all excellent—but this one was absolutely unique.  

While Michael wasn't directly involved in the working group that produced the XQuery language, he wasn't far from it—given his involvement with the W3C and the creation of XML and XSD, not to mention the creation of TEI.  Thus, besides just giving an expert introduction to XQuery, he also shed light on the communities who came together, with different interests, personalities, politics, and intellectual frameworks, to create this remarkable language.  

My colleagues and I—all historians, some who specialize in the history of technology—were fascinated by this aspect of the course.  We came away with a solid foundation in the language, and an appreciation for the milestone it marks in the history of programming languages and technology.  

(The creation of XQuery, as well as the creation of TEI, would be two worthy dissertation topics.)  

Inspired by Michael's course, I've begun thinking about ways to both share my own appreciation for XQuery and related technologies, tools, and tips that I have discovered in my own work, and to give this a human dimension, rather than just a purely instructional one.  I've heard it said that the best camera is the one that you have with you.  In that spirit, I'll start writing here, in a blog that I already have.  

So, let's begin with a brief introduction about how I came to learn XQuery.  

In mid-2007, as a freshly minted history PhD, my new job presented me with the challenge of revamping a website for a group of venerable historical publications.  (I've written an article about the project.)  After researching the formats in use for encoding books and historical documents, I decided to adopt TEI as the format for the publications.   TEI P5—the standard's 5th major version—was released in October 2007, just in time for my project to adopt it and benefit from its many advances from the start.

This left the question: how to turn the huge volumes of TEI into a website that allow historians and the public to view, browse, and search the publications?  In a superb stroke of luck, I met James Cummings at the TEI conference at U Maryland College Park in October 2007 and told him about my project.  James encouraged me to look into native XML databases.  

James' suggestion led me to eXist-db.  eXist-db's Shakespeare demos impressed me with their speed and precision.  Moreover, it supported XSLT, which would allow me to use the stylesheets I had adapted from TEI community for turning my XML into HTML for the web.  

But besides XSLT, eXist-db also used XQuery for its search and scripting operations.  XQuery 1.0 had just achieved recommendation status in January 2007, and Priscilla Walmsley's O'Reilly book on XQuery was published in June—the month I graduated.  

As I taught myself XQuery with eXist-db (not to mention the indispensable oXygen XML Editor), I found myself ever more comfortable with XQuery.  Thanks to an invaluable hint in the right direction by David Sewell (using the same typeswitch-based approach he outlined in this mailing list posting of his), I migrated all of my XSLT routines to XQuery.  Now, rather than having to master two languages, I could focus on the one that did everything I needed: XQuery.  

Ever since that moment, I've used XQuery almost exclusively, and together with TEI, eXist-db, and oXygen, it has been my gateway into the world of digital humanities and software development.  XQuery was challenging to learn but still very accessible.  The successes have been rewarding.  Even after five years I am still learning new aspects of the language and uncovering new ways to apply it.

After several years of being the sole user of XQuery in my office, I'm thrilled that so many of my colleagues are learning XQuery.  It makes perfect sense given the amount of TEI and XML that we now have created.  Thanks to Michael for getting us started, and linking us intellectually to the shared sense of purpose and possibilities that led to the creation of XML, TEI, XQuery and the other foundational standards that have enabled and empowered us to do our work.