XQuery as a Web Query Tool

Those of you who have followed this blog know that I’ve long searched for something that I call a “web query tool” (click on the Query Tools category to see this history).

As a way of identifying a “web query tool,” I use a very simple measurement device … it is something that can answer the following question: “Please give me a list of <a> about <b>” (eg, a list of web tutorials about python programming). This sounds like a search, but it is more than search. It is an ad hoc query run against a web of data instead of a database.

I don’t think the time is at hand when I’ll find a real web query tool.

But recently I’ve been playing quite a bit with XQuery and have been pleased (eg, see Making a Dent in a Steep Learning Curve). I believe XQuery will do nicely  as a tool for mashups of web data and as a front-end into a technology stack that runs from data integration to transformation to analysis to visualization and then to presentation.

However I wanted a reality check from people far more experienced with XQuery than me. So I posed a question to a list that serves people focused on XQuery. The question contained a twist about the future as well.

The respondents were uniformly gracious, and I found the conversation very helpful. Below is an edited version of that conversation organized as threads rather than in time sequence. Enjoy.

Notes:
1. I’m reluctant to quote the words of other people. So originally I thought I would summarize the conversation. But this made it lifeless. So I left the respondents anonymous but included their (slightly edited) direct quotes. I also wrote the list where the original conversation happened and included a link to a preview of this blog post, and asked if anyone objected if I included the transcript in the post.
2. In the transcript, indents denote threads. So, for example, something indented once means it is a response to my original question. Something indented twice is a response to the immediately preceding response indented once. And so on.
3. I omitted side conversations not directly relevant to my original question. As I write this, the discussion continues on the list but it has long since left my original topic. This still evolving conversation is not included in the transcript.

Transcript

Gary Lewis, 10:13am
Can anyone comment on the possibility that xquery could be used as a data integration/mashup tool running in a browser that included a web server? There’s been quite a bit of excitement the past couple days about the Opera Unite announcement of its “web server in a web browser” concept. Is it possible that xquery, as  query/integration/transformation tool, might someday be a plugin to a browser (eg, Chrome; Chrome/Wave)?

Respondent #1, 10:25am
sure why not … XSLT support in the browser was/is a plugin and sets a precedent for this kind of thing

XQuery in the browser would be great, anything that reduces the hegemony that is javascript … which reminds me, I know it sounds strange but perhaps browsers will turn into *the* personal ‘database’ for the individual someday and it will be natural to have something like xquery to help process the data it contains.

Respondent #2, 10:32am
This may be of interest -
http://www.zorba-xquery.com/index.php/xquery-in-the-browser-xqib/

Respondent #3, 10:42am
There are a lot of people keen on the idea of xquery in the web browser.

Unfortunately, though, I think there’s a bit of a conflict. A lot of the benefits of the web architecture come from the fact that the browser is small, ubiqitous, and predictable. The more goodies you put in it, the less that remains true. And the less it is true, the harder it is to write applications that will run anywhere and talk to anything. This is why, after ten years, writing applications that rely on XSLT-in-the-browser can still be problematic, as recently discussed on the xml-dev list.

Respondent #5, 1:25pm
Well, one way to achieve the same result is to build on that small and predictable foundation. This rules out plugins, but there’s other possibility – that of “compiling” XQuery to cross-browser JavaScript. We already have GWT that does a similar thing, so the general concept is proven, and I don’t see why doing the same to XQuery would be any harder. The main problem that I see there is that browser XML manipulation APIs aren’t standardized yet, but this is going to change with HTML5 (and meanwhile there are already “good enough” ways to work around this by detecting and using the mush-mash of the existing browser-specific APIs).

Respondent #4, 11:21am
http://xqib.org

Gary Lewis, 11:30am
A couple people have mentioned zorba. I’ve used and quite like zorba xquery; even wrote a blog post about it and included some analysis of Federal Reserve Economic Data time series accessed via an API with zorba.

I’m also familiar with the zorba XQIB project but do not use a Windows OS so have not played with it yet. But my impression (probably wrong) is that XQIB is cast mostly as an alternative to JavaScript, using scripting extensions to XQuery. If true (please let me know otherwise), I’m not so much interested in this aspect. As I said earlier, I’m mostly interested in a data integration & transformation tool that sits in a browser and gets used as a precursor to analysis & graphics tools.

Respondent #7, 5:00pm
the xqib people can answer you in more details, but in general, what they do have is a full embedding of a Zorba xquery processor in the browser — this includes all the functionalities of Zorba, full XQuery with updates and scripting, XQuery 1.1 with groupby, windowing, try-catch, etc, plus the REST, function libraries.

What users use this XQuery for is there own decision: can be very well data integration, data transformation, RSS integration, etc, everything is good, as long as it can be expressed in XQuery.

Respondent #6, 1:26pm
I surely see your point about XQuery in the browser – as stated before by others: check out our project at http://www.xqib.org/

At the moment, the release works only with Internet Explorer – but the next release (which is nearly finished – we hope to release it this month) will also run with Firefox and will be much faster. [...]

But I wanted to ask you, where exactly you see the usecases of having a web server in the browser? To be honest: the last thing I want running in my browser is a server (as [Respondent #3] stated because of complexity, but also because of security). Also, in most companies (and also on most routers), all ports are normally closed, which makes a webserver useless. The only benefit I can see, is that this way it would be easy to implement some kind of RPC from the server to the client. But in my oppinion, this would be a too small advantage to legitimate a webserver.

Gary Lewis, 4:29pm
Let’s start with this question. In a p2p web … as Google Chrome/Wave and Opera Unite suggest might happen … is there a new role for xquery? I guess I’d answer that question with a tentative “Yes.” Here’s a simple example. There’s a lot of buzz around transparency now … government transparency, data transparency, etc. In the U.S., data.gov is a recent case. The idea is simple enough. Make government data sets available to the public and let them mash it up and then share it. [In some ways this scenario gives me the creeps, because I know from years of experience how very difficult it is to insure data integrity during analysis.] I’d love to use xquery as a data integration and transformation tool as preparation for analysis, visualization, & presentation. And then to share the data with others in a kind of social network of data reuse. I was guessing when I suggested that xquery in a browser with a web server might make this future possible. I’ve no way of knowing. But maybe one of you might.

Respondent #8, 6:53pm
Why? To me, the best thing about XQuery is that it can run natively (and more efficiently than the more encompassing XSL) on an XML DB providing all the speed and memory efficiency afforded by the different and incompatible XML DBs. Caveat emptor: I fall into the camp that doesn’t believe XQuery is a good styling language or a good PHP-like webapp templating language. (if there even is a camp, rather than me leaning over a bic lighter).

Running XQuery in the browser seems like a huge waste of bandwidth where you would have to download everything you want to query against, perhaps to find only a tiny fragment. I would create a URL and just GET it from a server where XQuery creates the XML or JSON to be styled on the client with XSL or JavaScript.

Respondent #4, 1:36am
i think browser can act as proxy server to localhost-based eXist database (or AMP if somebody still love it).

1. Cefn Hoile — June 20, 2009 @ 10:40 am

   I’ve been looking into the use of a browser based webserver to enable ordinary people to edit and process schema-driven XML (their own data authored according to their own schema) on their local machine. Current tools are expensive and over-engineered for this job. I aim to make this open source and free.

   A firefox plugin acting as a webserver seems like an ideal approach – as a cross-browser and easy-to-install component which is easily hackable and extensible with widely understood HTML-oriented libraries for presentation and data entry as well as dynamic AJAX-like interactions with a backing data set which lives on the hard drive.

   I see this being used in order to create data-driven websites or for non-techies to hack their own personal information managers.

   XQuery in the browser would be a useful facility, but for now, I’m planning to use something like Saxon running XQueries, triggered by POW – David Kellogg’s Plain Old Webserver.
   http://davidkellogg.com/wiki/Main_Page

   Some of the XML processing tasks are doable with XPath driven by Javascript directly in the page without any additional plugins in Firefox at least.

   It may be that the facilities required for this project will need direct access to the OS capabilites anyway (i.e. writing files onto the harddrive), so would not be able to run inside a browser sandbox even if XQuery was available to it.

2. Gary Lewis — June 20, 2009 @ 11:25 am

   Hi Cefn – That sounds very cool. I hope we can stay in touch; I’d like to hear more. Thanks for the comment. … Gary