Here’s a sample graph from the report. It shows the annual percentage change in enrollments from all U.S. degree-granting institutions of higher education from 1968 to 2007. The grayscale background shows unemployment rates and periods of recessions. The report is available in this pdf that includes better resolution graphics.

Here is an abbreviated version of the summary section from the report.

1. The new XQuery use cases included screen scraping enrollment data embedded in html, and the pipelining of enrollment and economic data through several staged transformations.
2. The evidence that higher education in the United States is countercyclical appears weak based on the exploratory analysis done here.
3. The evidence varies somewhat by institutional control and type, but mostly the general observation of a weak association between enrollment change and recession cycles holds true.
4. The tools used in this project, XQuery and R, are wonderful research tools but are not suitable for more general use without wrappers that mute their complexity.
5. The screen-scraping technique used in this project provides another access route to a vast amount of U.S. Department of Education data on the web.
6. None of the data sources used in this project included semantic or linked data markup. Whether XQuery can be used successfully with RDFa or microformats seems worthy of investigation.

Here are links to various pieces of this project:
a. Final report
b. Documentation for XQuery source programs
c. XQuery to generate enrollment data
d. XQuery to generate unemployment and recession data
e. XQuery to merge enrollment, unemployment, and recession data
f. R analysis history

I remain quite satisfied with XQuery.

As a way of identifying a “web query tool,” I use a very simple measurement device … it is something that can answer the following question: “Please give me a list of <a> about <b>” (eg, a list of web tutorials about python programming). This sounds like a search, but it is more than search. It is an ad hoc query run against a web of data instead of a database.

I don’t think the time is at hand when I’ll find a real web query tool.

But recently I’ve been playing quite a bit with XQuery and have been pleased (eg, see Making a Dent in a Steep Learning Curve). I believe XQuery will do nicely  as a tool for mashups of web data and as a front-end into a technology stack that runs from data integration to transformation to analysis to visualization and then to presentation.

However I wanted a reality check from people far more experienced with XQuery than me. So I posed a question to a list that serves people focused on XQuery. The question contained a twist about the future as well.

The respondents were uniformly gracious, and I found the conversation very helpful. Below is an edited version of that conversation organized as threads rather than in time sequence. Enjoy.

Notes:
1. I’m reluctant to quote the words of other people. So originally I thought I would summarize the conversation. But this made it lifeless. So I left the respondents anonymous but included their (slightly edited) direct quotes. I also wrote the list where the original conversation happened and included a link to a preview of this blog post, and asked if anyone objected if I included the transcript in the post.
2. In the transcript, indents denote threads. So, for example, something indented once means it is a response to my original question. Something indented twice is a response to the immediately preceding response indented once. And so on.
3. I omitted side conversations not directly relevant to my original question. As I write this, the discussion continues on the list but it has long since left my original topic. This still evolving conversation is not included in the transcript.

Transcript

Gary Lewis, 10:13am
Can anyone comment on the possibility that xquery could be used as a data integration/mashup tool running in a browser that included a web server? There’s been quite a bit of excitement the past couple days about the Opera Unite announcement of its “web server in a web browser” concept. Is it possible that xquery, as  query/integration/transformation tool, might someday be a plugin to a browser (eg, Chrome; Chrome/Wave)?

Respondent #1, 10:25am
sure why not … XSLT support in the browser was/is a plugin and sets a precedent for this kind of thing

XQuery in the browser would be great, anything that reduces the hegemony that is javascript … which reminds me, I know it sounds strange but perhaps browsers will turn into *the* personal ‘database’ for the individual someday and it will be natural to have something like xquery to help process the data it contains.

Respondent #2, 10:32am
This may be of interest -
http://www.zorba-xquery.com/index.php/xquery-in-the-browser-xqib/

Respondent #3, 10:42am
There are a lot of people keen on the idea of xquery in the web browser.

Unfortunately, though, I think there’s a bit of a conflict. A lot of the benefits of the web architecture come from the fact that the browser is small, ubiqitous, and predictable. The more goodies you put in it, the less that remains true. And the less it is true, the harder it is to write applications that will run anywhere and talk to anything. This is why, after ten years, writing applications that rely on XSLT-in-the-browser can still be problematic, as recently discussed on the xml-dev list.

Respondent #5, 1:25pm
Well, one way to achieve the same result is to build on that small and predictable foundation. This rules out plugins, but there’s other possibility – that of “compiling” XQuery to cross-browser JavaScript. We already have GWT that does a similar thing, so the general concept is proven, and I don’t see why doing the same to XQuery would be any harder. The main problem that I see there is that browser XML manipulation APIs aren’t standardized yet, but this is going to change with HTML5 (and meanwhile there are already “good enough” ways to work around this by detecting and using the mush-mash of the existing browser-specific APIs).

Respondent #4, 11:21am
http://xqib.org

Gary Lewis, 11:30am
A couple people have mentioned zorba. I’ve used and quite like zorba xquery; even wrote a blog post about it and included some analysis of Federal Reserve Economic Data time series accessed via an API with zorba.

I’m also familiar with the zorba XQIB project but do not use a Windows OS so have not played with it yet. But my impression (probably wrong) is that XQIB is cast mostly as an alternative to JavaScript, using scripting extensions to XQuery. If true (please let me know otherwise), I’m not so much interested in this aspect. As I said earlier, I’m mostly interested in a data integration & transformation tool that sits in a browser and gets used as a precursor to analysis & graphics tools.

Respondent #7, 5:00pm
the xqib people can answer you in more details, but in general, what they do have is a full embedding of a Zorba xquery processor in the browser — this includes all the functionalities of Zorba, full XQuery with updates and scripting, XQuery 1.1 with groupby, windowing, try-catch, etc, plus the REST, function libraries.

What users use this XQuery for is there own decision: can be very well data integration, data transformation, RSS integration, etc, everything is good, as long as it can be expressed in XQuery.

Respondent #6, 1:26pm
I surely see your point about XQuery in the browser – as stated before by others: check out our project at http://www.xqib.org/

At the moment, the release works only with Internet Explorer – but the next release (which is nearly finished – we hope to release it this month) will also run with Firefox and will be much faster. [...]

But I wanted to ask you, where exactly you see the usecases of having a web server in the browser? To be honest: the last thing I want running in my browser is a server (as [Respondent #3] stated because of complexity, but also because of security). Also, in most companies (and also on most routers), all ports are normally closed, which makes a webserver useless. The only benefit I can see, is that this way it would be easy to implement some kind of RPC from the server to the client. But in my oppinion, this would be a too small advantage to legitimate a webserver.

Gary Lewis, 4:29pm
Let’s start with this question. In a p2p web … as Google Chrome/Wave and Opera Unite suggest might happen … is there a new role for xquery? I guess I’d answer that question with a tentative “Yes.” Here’s a simple example. There’s a lot of buzz around transparency now … government transparency, data transparency, etc. In the U.S., data.gov is a recent case. The idea is simple enough. Make government data sets available to the public and let them mash it up and then share it. [In some ways this scenario gives me the creeps, because I know from years of experience how very difficult it is to insure data integrity during analysis.] I’d love to use xquery as a data integration and transformation tool as preparation for analysis, visualization, & presentation. And then to share the data with others in a kind of social network of data reuse. I was guessing when I suggested that xquery in a browser with a web server might make this future possible. I’ve no way of knowing. But maybe one of you might.

Respondent #8, 6:53pm
Why? To me, the best thing about XQuery is that it can run natively (and more efficiently than the more encompassing XSL) on an XML DB providing all the speed and memory efficiency afforded by the different and incompatible XML DBs. Caveat emptor: I fall into the camp that doesn’t believe XQuery is a good styling language or a good PHP-like webapp templating language. (if there even is a camp, rather than me leaning over a bic lighter).

Running XQuery in the browser seems like a huge waste of bandwidth where you would have to download everything you want to query against, perhaps to find only a tiny fragment. I would create a URL and just GET it from a server where XQuery creates the XML or JSON to be styled on the client with XSL or JavaScript.

Respondent #4, 1:36am
i think browser can act as proxy server to localhost-based eXist database (or AMP if somebody still love it).

I’m now ready to start work on another XQuery data integration, this time using the new FRED API from the Federal Reserve Bank of St. Louis. Not that I’m particularly interested in banking-related data, but FRED uses a REST web service architecture and will allow me to play more  thoroughly with Zorba’s REST capability. And the volume of the data will allow me to stress test the performance of Zorba’s XQuery.

The siren’s song, of course, is that the web may one day become a giant database of interrelated data.

I’ve now finished the XQuery/FRED project. You can see the details here (pdf), but I’ll touch on a few of the points.

The FRED database includes more than 20,000 economic time series. The folks at the Economic Research Division at the Federal Reserve Bank of St. Louis have done a nice job making this data available via an API. With so much data to choose from, however, it helps if you know at the outset what kind of data you’d like to retrieve. I didn’t.

So I set myself two tasks: 1) Build the XML documents to provide flexible overviews of the 20,000 time series; and 2) Choose two of these times series and conduct a small analysis.

I did the first job using Zorba XQuery and the second job using XQuery and the R statistics package. Again, the details are here (pdf).

Basically I came away feeling like I’d made some progress. The Zorba REST calls to the FRED API worked flawlessly. Some of the XQuery programs with joins to multiple XML just chugged away slowly, but then I  was using a creaky old server to run Zorba and serve up the XML files. Otherwise it was a wonderful opportunity to learn more about XQuery as a web query tool. I continue to like what I saw.

Since that’s a pretty positive statement for me, perhaps this is the place to say that I have no relationship with Zorba or any of the other XQuery implementations. I’m simply looking for a tool that will utilize some of my strengths in pursuit of the main reason I write this blog … which, if you don’t yet know, is to help nudge us closer to the day when learning will be free for everyone everywhere.

Next I plan to look at Zorba in a variety of other settings. Options include the XQuery in the Browser project (XQIB), XQuery scripting and Sausilito to construct web sites, and some serious performance tests of  XQuery  on EC2 and S3 (Amazon Web Services).

Below is a screen shot of the R analysis. It shows a not unexpected relationship between the unemployment rate in the United States over the last 55 years and the rate of change in outstanding household credit market debt.  The current recession seems somewhat atypical, however. Details and a legible graphic are also in the PDF, as is souce code for the XQuery programs I wrote.

Two quick examples. Google researchers describe “a unified Web knowledge base” as the “holy grail of web information extraction” (PDF), and the company energetically applies their research to harvest additional structured data from natural language text, HTML-embedded tables, and the so-called deep-web of databases sheltered behind form-only access.

Tim Berners-Lee, whose legacy is the web, provides another example. In a video from TED, he calls for a “new reframing” that will release the “unlocked  potential” of the web through linked data. The idea is that people and computers could traverse data related to other data, from A to B to … wherever you want to go as long as the relationships, the tuple linkings, are available. Berners-Lee’s new reframing translates into a web-of-data that could be used by scientists, citizens, social reformers, businesses, entrepreneurs, and governments in innovative new ways.

I applaud these efforts even though I realize that the potential for good and for ill seems equally strong. But my focus is nothing so grand as a web-of-data. I’d be happy if I could just answer a profoundly simple question: “Please give me a list of all web tutorials on python programming.” More symbolically I want to answer questions of the form: “Give me a list of <a> about <b>.” From there, of course, the possible questions become far more interesting.

For certain my query would be trivial if a web-of-data existed. But it does not, even though progress is being made. So, in the interim I’m looking for a substitute.

My latest excursions have taken me into the world of XML, and specifically into XQuery and, to a lesser extent, into the XQuery relatives XSLT and XPath. All of these are W3C recommendations that have been implemented in both open source and proprietary products.

Why XQuery? Well, because XML is part of the fabric of the web, and because data in XML format can be queried with XQuery, and because I stumbled upon Zorba via an O’Reilly xml.com article called “Something Tells Me You Need to Pay Attention to This.” How could I resist a title like that?

I started playing with Zorba’s XQuery about two months ago. Maybe I’ve written a couple hundred queries now. Like the learning curve for SQL, it’s clear that a couple hundred is at least an order of magnitude too small to become skilled. But it’s great fun and I’m encouraged by what I see.

One example. Tony Hirst at the Open University has done some cool things recently with The Guardian’s new data API. In one of these projects, Tony used data from The Guardian’s university guide to do a mashup on student satisfaction in architecture and planning programs at various UK universities. It featured a very nice use of DabbleDB database.

Data integration is one of the strengths of XQuery, so I set about following Tony’s lead to see if I could duplicate his mashup but by using Zorba. It was very fun and I learned tons. You can see the results in this PDF.

I’m now ready to start work on another XQuery data integration, this time using the new FRED API from the Federal Reserve Bank of St. Louis. Not that I’m particularly interested in banking-related data, but FRED uses a REST web service architecture and will allow me to play more  thoroughly with Zorba’s REST capability. And the volume of the data will allow me to stress test the performance of Zorba’s XQuery.

If you are interested at all by Zorba, I’d recommend you read some of the technical documents where you can catch glimpses of longer term development objectives and experience some of the chutzpah that must exist in the development team. A recent example is XQuery in the Browser, which was presented this week at the 18th International World Wide Web Conference in Madrid. The article basically takes aim at JavaScript. As another example, check out the plans of a 3 year-old startup called 28msec and some of their technical papers. I particularly enjoyed the architecture discussion in Donald Kossmann’s slide presentation on Building Web Applications without a DBMS (PDF).

Ok, I better stop. It’s already beginning to sound like an infomercial. Hopefully some of my sincere enthusiasm comes through, however. It’s a hopeful time. And now it’s back to making a dent in that order of magnitude learning curve.

So, here is my advice on what not to build. Actually, it’s a bit more that that: it’s a list of what not to build, a list of some things that people are working on now, some fads to avoid, and some indication of what’s out there for the taking, if you can get your act together in a hurry. And what lies beyond that? The domain of real innovation and progress.

I left this comment on Stephen’s post:

Here are just 2 quick points.
1. “How” is just as important as “what”. There’s a parallel to What Not to Build that might be called How Not to Build.
2. There’s one feature of the web that nobody has come close to solving. That feature is it’s monstrous size and it’s pulsating rate of growth. For example, consider search. Google Search is woefully inadequate in the face of the web’s size and growth rate. Alternatives like the federated searches (eg, on science.gov) are better but still provide inadequate coverage and features. There must be a hundred or more companies in the search business, and I’m certainly not suggesting that anyone create another one (this might be worse even than building another OS). But the web desperately needs some kind of middle layer of data systems that can be used by ordinary people. I’d say that the major entrepreneurial and socially constructive opportunities await builders of the web infrastructure.

In some ways the web today parallels the situation with enterprise databases several decades ago. It was easy to put data into these relational databases, but it was dreadfully difficult to get reports, analysis, and information out in any useful form. Development work proceeded along two fronts: i) creating reporting and analysis tools that simplified the user interface and seemingly made SQL unnecessary; and ii) creating a middle layer of data abstracted from the full complexity of the relational databases but with a simplified architecture so that the new tools would actually work.

Google search and other current web search tools are over matched by the complexity of the web, just as reporting and analysis tools were once over matched by enterprise databases. I suspect that it will take a similar architectural change to make the web truly useful. But this is not going to come easily. It was devilishly difficult with enterprise databases, but at least developers had SQL as a query tool that could uncover even the most inaccessible of the data. There is no equivalent query language for the web, so simplifying the web architecture for ease of use will be orders of magnitude more difficult than enterprise databases. It’s possible that the semantic web may eventually play this role, but I’m not very sanguine that it will happen any time soon. We need something that will work today.

Here’s how Yahoo describes YQL:

YQL is a new web service API that lets you access other web services using a SQL-like language rather than typical programmatic access. You can think of it as a command line version of Pipes. Its goal is to make data from Yahoo! as well as from across the internet universally accessible through a single common interface.

I’ve lamented previously that search tools provide only meager help for web queries. For example, the following query cannot be accomplished in a web search: “Give me a list of all JavaScript tutorials available online.” This is a trivial query as queries go.

So when I heard about the YQL announcement, I promised myself I’d have a look at it as soon as I could. This post provides a preliminary reaction to YQL based on several readings of the documentation and perhaps 6 hours of play at the YQL console that Yahoo provides (very nice, thank you!).

YQL is definitely a step beyond search. For example, you can get tantalizingly close to an answer for the query on JavaScript tutorials. The YQL looks like:

select * from search.web(0) where query = “javascript tutorial” | unique(field=”url”)

This is definitely SQL-like. There are select, from, and where clauses.  The search.web notation refers to one of about 50 XML  or JSON data collections presented by Yahoo as a “table.” Each table has the equivalent of SQL data columns that can be used for filtering or for display in the select clause.

The (0) after the table name signifies that I requested an unbounded query. A setting of (1000) exposes only a subset of 1000 entries in the source web service.

The | pipe is a post-query operation that discards non-unique url’s. You can also perform sorts in a post-query operation.

There are several other nice features of YQL that I won’t go into here. One example is sub-query capability. But take a look at the YQL documentation for examples. Or better yet, play with the YQL console and experiment for yourself.

Compared to any standard SQL, YQL is a very simple query tool. This is not necessarily a bad thing. For example, when someone designs large data warehouses from source data stored in relational tables,the design almost always removes the complexity from the data so that users and analysts can focus on the business questions and not on the database architecture. Simplifying the data structure means that the stress on the query and analysis tools is reduced. Simple tools work fine when the data architecture is also simple.

The YQL documentation that I saw does not contain architectural schemas, so I cannot comment on whether the “tables” exposed by Yahoo are adequately served by a simple query tool like YQL. Given the early stage in the lifecycle of this project, I suspect the answer is no. But I’d also expect additional work on both the architecture and the language will occur as people use it.

The JavaScript query that I wrote in YQL failed to complete. At present, Yahoo has a governor on YQL queries. I believe the max is 30 seconds or 50,000 items. Run-away queries are every DBA’s nightmare, so it is not surprising that Yahoo would place limits on YQL queries.

For web queries to be truly useful, however, we’ll need a way to relax the governor restrictions and actually do complete queries. But for now, I’m very grateful to see the precursors of tomorrow’s web query tools emerging.

Part 2 of this series will look at another approach to analysis of very large datasets. It’s called Pig (no joke) and is described by Yahoo and Apache Hadoop as an “infrastructure” or “platform” to conduct ad-hoc data analysis. Apparently “the highest abstraction layer in Pig is a query language interface, whereby users express data analysis tasks as queries, in the style of SQL or Relational Algebra.”

Bingo. If only it’s true. I’ll let you know what I find out.

After all these years using Google, I finally read through the stuff on the Google Web Search Help Center. Duh, should have done this long ago. For example, I had no idea you could do synonym searches.

But what surprised me the most was Google’s statement “keep in mind that the order in which the terms are typed will affect the search results.” How many thousands of searches have I made and never noticed this? But it’s true. The search [vacation paris] produces 560,000 results, while the search [paris vacation] produces 913,000 results. Note: I’ve used square brackets [] for readability only. In an actual search, of course, the brackets would not be used.

I figured an AND would be commutative like an INTERSECT set operator. You get the vacation result set (call it V) and the paris result set (call it P). V intersect P would give the same results as P intersect V. But V AND P != P AND V in a search.

I didn’t see any explanation of this in Google Web Search Help, but found a pretty helpful discussion that makes sense. Search word order matters because bakers dozen means something different from dozen bakers, and factoring word order into the relevance algorithm improves search results.

Well, ok, I get the bakers dozen idea. But vacation paris and paris vacation? Why do these two searches produce such different results?

Which brings me to a couple peeves about search.

It should be possible to string search result sets together. Here’s an illustration. Suppose I wanted to find out how the search V AND P differs from P AND V. I could do the V AND P search to get a result set (call it R1). Then I could do the P AND V search to get another result set (call it R2). Having these 2 result sets, I could produce R2 MINUS R1 to find out what results were in R2 but not in R1. Also interesting would be R1 MINUS R2 and R1 INTERSECT R2. In other words, I’d like to produce search result sets that could be refined with other search result sets. This would certainly help with complex queries.

Which brings me to a second but related peeve. Right now we do searches. But to use the web as a research tool, we need to do queries. For example, here’s a very simple query that you cannot now translate into a Google search: “give me a list of all the web design courses”. The query ["web design" ~course] produces roughly 22 million results. It’s great if you’re looking for a few examples. But it’s lousy if you want a reasonably complete universe of unique web design courses.

In his book Ambient Findability, Peter Morville has an interesting discussion about different types of searches (pp 49-50). He distinguishes between sample, existence, and exhaustive searches. In a sample search, you’re looking for a few examples. Google does this great. An existence search is just a binary yes/no search (does document x exist)? Google also does this great. An exhaustive search should return all of the relevant items. As Morville discusses, the effectiveness of exhaustive searches falls rapidly with collection size. So it’s no wonder that this type of query/search is not available.

I suppose you could get something like an exhaustive search using Amazon Web Services and the Alexa search engine. I think you can get up to 10 million results returned that can be saved in a file for further manipulation. And maybe for a specific high return-value query, I would do this. But not for day-to-day work.

ok, that’s it for this search novice.