Making a Dent in a Steep Learning Curve

In a previous post called Web Query Tools – Part 2, I said:

I’m now ready to start work on another XQuery data integration, this time using the new FRED API from the Federal Reserve Bank of St. Louis. Not that I’m particularly interested in banking-related data, but FRED uses a REST web service architecture and will allow me to play more  thoroughly with Zorba’s REST capability. And the volume of the data will allow me to stress test the performance of Zorba’s XQuery.

The siren’s song, of course, is that the web may one day become a giant database of interrelated data.

I’ve now finished the XQuery/FRED project. You can see the details here (pdf), but I’ll touch on a few of the points.

The FRED database includes more than 20,000 economic time series. The folks at the Economic Research Division at the Federal Reserve Bank of St. Louis have done a nice job making this data available via an API. With so much data to choose from, however, it helps if you know at the outset what kind of data you’d like to retrieve. I didn’t.

So I set myself two tasks: 1) Build the XML documents to provide flexible overviews of the 20,000 time series; and 2) Choose two of these times series and conduct a small analysis.

I did the first job using Zorba XQuery and the second job using XQuery and the R statistics package. Again, the details are here (pdf).

Basically I came away feeling like I’d made some progress. The Zorba REST calls to the FRED API worked flawlessly. Some of the XQuery programs with joins to multiple XML just chugged away slowly, but then I  was using a creaky old server to run Zorba and serve up the XML files. Otherwise it was a wonderful opportunity to learn more about XQuery as a web query tool. I continue to like what I saw.

Since that’s a pretty positive statement for me, perhaps this is the place to say that I have no relationship with Zorba or any of the other XQuery implementations. I’m simply looking for a tool that will utilize some of my strengths in pursuit of the main reason I write this blog … which, if you don’t yet know, is to help nudge us closer to the day when learning will be free for everyone everywhere.

Next I plan to look at Zorba in a variety of other settings. Options include the XQuery in the Browser project (XQIB), XQuery scripting and Sausilito to construct web sites, and some serious performance tests of  XQuery  on EC2 and S3 (Amazon Web Services).

Below is a screen shot of the R analysis. It shows a not unexpected relationship between the unemployment rate in the United States over the last 55 years and the rate of change in outstanding household credit market debt.  The current recession seems somewhat atypical, however. Details and a legible graphic are also in the PDF, as is souce code for the XQuery programs I wrote.