Instead of unemployment, I chose to examine whether family income influences students’ choices about enrollment in public versus private colleges and universities in the United States. Given the substantial gap in net price between public and private institutions, you would think that family finances might play an important role in the public vs private decision.

One caveat as always. Throughout these use cases, I’ve been less interested in the research question than in how xquery performs. I chose research questions that I thought would be interesting, but I was more concerned about the tool than the actual research.

In this post I won’t give away the story-line about the research. For that, please refer to the 4-page PDF that describes the project.

However, I will say once again that I enjoyed working with xquery. There are several implementations of the W3C standards. I used zorba.

This project did place stress on xquery performance at times, but I was able to concoct workarounds that I’ll be able to use in similar situations if they occur in the future.

Also, I’d like to highlight once again the very intricate nature of data analysis. I’m occasionally uncomfortable in this age of transparency and big data and semantic linked data that ease-of-use is the assumed norm and expectation. It has never been my experience that data analysis is easy or fast. It requires living in the data for considerable periods of time. It requires thought and play and more thought. It requires failure and more failure and finally small steps of success. I hope this will be evident in the PDF that describes this project.

I also tried again to provide links to all important programs I wrote as part of this project. It’s not exactly what some call open research, but it certainly is a step in that direction. If anyone feels motivated to duplicate what I did or, better yet, to modify and improve upon what I did, then I think you’ll find the source links useful. And I’d be happy to assist if you have questions.

With the conclusion of this project, I no longer feel the need to equivocate about xquery. I will definitely use it.

Here’s a sample graph from the report. It shows the annual percentage change in enrollments from all U.S. degree-granting institutions of higher education from 1968 to 2007. The grayscale background shows unemployment rates and periods of recessions. The report is available in this pdf that includes better resolution graphics.

Here is an abbreviated version of the summary section from the report.

1. The new XQuery use cases included screen scraping enrollment data embedded in html, and the pipelining of enrollment and economic data through several staged transformations.
2. The evidence that higher education in the United States is countercyclical appears weak based on the exploratory analysis done here.
3. The evidence varies somewhat by institutional control and type, but mostly the general observation of a weak association between enrollment change and recession cycles holds true.
4. The tools used in this project, XQuery and R, are wonderful research tools but are not suitable for more general use without wrappers that mute their complexity.
5. The screen-scraping technique used in this project provides another access route to a vast amount of U.S. Department of Education data on the web.
6. None of the data sources used in this project included semantic or linked data markup. Whether XQuery can be used successfully with RDFa or microformats seems worthy of investigation.

Here are links to various pieces of this project:
a. Final report
b. Documentation for XQuery source programs
c. XQuery to generate enrollment data
d. XQuery to generate unemployment and recession data
e. XQuery to merge enrollment, unemployment, and recession data
f. R analysis history

I remain quite satisfied with XQuery.

Since then I installed Oracle’s BerkeleyDB XML database and duplicated one of the performance tests I’d done with the FRED data. Response time in the test dropped from 195 seconds to 2 seconds. Not bad; much more consistent with what I’d expect from SQL queries against a similarly sized relational database.

These are not precise performance tests by any means. For example, the XQuery processor I used with the original FRED data was Zorba, but XQilla is the XQuery processor that comes bundled with BerkeleyDB XML. So, for certain, I changed at least two important factors in the tests, and unless you make controlled changes in factors it is pretty much impossible to attribute causation.

However, I rather doubt that the choice of the XQuery processor had much to do with the performance improvement. Performance only improved when I created database indexes for the XML.

At this point I am still very much playing; just trying to use XQuery in a variety of environments to better understand what it can and cannot accomplish. If you’re a researcher, for example, it doesn’t much matter if it takes 195 seconds for a data integration step in an entire research process that may take days, weeks or longer. However, 2 seconds is much preferred if you’re doing ad hoc data mashups in a browser.

From what little I’ve seen of Oracle’s BerkeleyDB XML database, it’s an interesting product, available with either a commercial or free open source license. It’s an embedded database accessed through a programmatic API and linked libraries. It is not a relational database, database management system, or a database server. It does, however, offer a lightweight solution when performance and data persistence are critical. Disclaimer: I have no association with Oracle, BerkeleyDB, or BerkeleyDB XML.

I realize this blog post will only interest a very few people, but from time-to-time I may write similar posts that document my playing with XQuery. I continue to like what I see. Truthfully I’m not certain where it’s headed, other than to say that the driver is free learning for everyone everywhere.