Educational Imaginations » Query Tools

Knowing When to Say No

Gary Lewis — Mon, 01 Feb 2010 13:46:12 +0000

In a previous post, I expressed reservations about privacy in Google’s new operating system. But that reservation was based on impressions rather than actual use. Here I describe how my impressions held up during 10 hours of hands-on testing of the OS and browser [1].

I downloaded the source code for Google’s Chromium OS and then built and installed the image on an Asus EeePC 1000 netbook [2]. Included was a pre-built binary of the Chromium-based browser.

First a few words about product names, because unfortunately they are a bit confusing here. Chromium OS and Chromium are Google’s names for the open source developer versions of the proprietary Chrome operating system (OS) and Chrome browser. The Chrome browser is currently in beta and is available for Mac, Windows, and Linux as binary downloads. But the OS is only available in the open source version. It differs from the Chrome OS that OEMs will ship on netbooks in late 2010. According to a Google FAQ:

Specifically, Google Chrome OS will run on specially optimized hardware in order to get enhanced performance and security.

When Chrome OS gets released, it will be based on Google firmware that is not part of the open source versions of Chromium OS. So I was unable to test several important features like fast boot, verified boot (a security measure), and system recovery.

As you’d expect in a developer release, the version of Chromium OS that I tested was a little buggy (eg, wifi did not work) and incomplete (eg, no JSON for bookmark imports). But these are really minor matters with simple work-arounds (eg, ethernet instead of wifi and HTML instead of JSON).

Bugs, partially complete features, and even missing features don’t really concern me at this stage. I was mostly interested in design and architecture decisions and whether I as a user would feel safe when using the OS and browser.

By safe I do not merely mean secure. Google has engineered an impressive product with security as a foundational principle. I’d expect Chrome OS to excel in this area, although I’ll reserve final judgement until I can test the Google firmware that will provide much of the security foundation.

By safe I also mean private. This I could and did test, although the tight integration of Chromium the browser and Chromium the operating system sometimes obscures exactly which Chromium is being tested.

When Chromium OS boots, you’re presented with a login screen. At present the login requires a Google account password. But in a Login design document, Google mentions OpenID as a future possibility and says they “want to ensure that people can fully use Chromium OS without needing a Google login.” Login design appears to be under active discussion, re-design, and development.

Chromium OS devices are cloud-based devices meant to run web services. The login requirement facilitates single signon to multiple web applications. By default, Chromium opened my Gmail and Calendar apps in separate tabs. I could logout of these apps without logging out from the OS. I could also easily browse to other Google apps via a Chromium icon that appears prominently in the far left of the top panel. Judging from the Google partner applications that also appear via this icon, single signon across multiple applications is an important design goal.

Single signon is a compelling benefit for the user. No one likes to re-enter authentication information multiple times a day. But the user also makes an implicit bargain with Google for this convenience, security, and price (ie, free). The entire time you use Chromium OS, you are logged in and therefore identifiable.

I encourage Google to allow users to opt-out of the initial OS login.

I spent the first 3 hours testing Chromium OS and Chromium with default configurations. This is the path of least resistance that many users may take. The test involved normal browsing of web pages bookmarked from RSS feeds. At the end of the test I’d accumulated 105 cookies from 38 different sites, many of which were third-party sites that I did not initiate. My browsing history recorded 63 events, all datetime stamped and all including the associated url.

However, this is just the tip of an iceberg of personal data collected in standard browser mode. The Google Chrome Privacy Notice provides more detail than I can include here, about information collected by Google and by other website owners. It also includes warnings about use of third-party Chrome extensions and an eye-opening description of information stored locally on your computer when using Chrome. This latter category includes cached text of pages visited; IP addresses obtained via DNS pre-fetching of links on pages you visit; a searchable index of pages you visit; thumbnail screenshots of pages; and, of course, browsing history, download history, and cookies set. If you enable Chrome’s synchronization feature, your browser settings (including bookmarks) will also be stored on Google servers.

Is all this data acceptable use? Well, it depends on each person’s level of tolerance for scrutiny of personal behavior by a private company. For me, it is not acceptable. As I mentioned in my previous post, I use the web energetically but very cautiously.

So as a second test I activated Incognito mode. In this mode, web pages and downloaded files are not recorded in browsing or download history. All new cookies are also deleted when closing the Incognito window. In a Security Overview design document, Google describes Incognito mode this way:

• Users can initiate a completely stateless session, which does not sync or cache data.
• All system settings would be kept out of this session, including networking config.

After many hours of Incognito use, I had no browsing history, no download history, and a single Google cookie. This is a happy result for a cautious web user. It is, however, quite easy to accidentally leave Incognito mode and bounce back to standard mode, so cautious users will also want to change Chrome’s default browser settings to opt-out of several features. See the Privacy Notice or Chrome Help for notes on how to opt-out and disable features.

It’s not clear to me what Google means by the term stateless and how it relates to common definitions (eg, see Wikipedia and stateless server). The word gets used when Google describes Incognito mode for Chromium (the browser), yet the entire occurrence appears in the context of a design document for Chromium OS (the operating system). I interpreted this to mean that stateless was a feature of the both the browser and the OS, but this is just a guess on my part. One of the Chromium OS design documents lists the stateless feature as “coming soon,” which also supports the notion that the term refers to something larger than just Incognito mode in the browser.

I encourage Google to clarify the stateless concept. It’s too important to leave it open to individual interpretation.

One final comment. You may have noticed in the block-quote about Incognito mode, the use of the term cached data. This is another area where Google could provide greater detail. What exactly does cached data mean? And what data does it include?

Some cached data is identified in the above discussion of Chrome data stored locally. But there appears to be an OS as well as a browser component to cached data. I got this sense from a Chromium OS design document called Protecting Cached User Data, where the following statement appears:

Chromium OS devices should provide privacy protection for user data stored on the local disk. In particular, some subset of a user’s email, pictures, and even HTTP cookies will be stored on the local drive to enhance the online experience and ensure Chromium OS devices are useful even when an Internet connection isn’t available.

I understand the offline rationale. But I wonder if it isn’t more complicated than that. Google’s Native Client technology will allow web-based applications to run native (ie, local) code from the Chrome browser, thereby using local resources to boost the performance of web apps.

To my knowledge, Google isn’t talking and the role of Native Client in Chrome OS is currently unclear. This merely encourages speculation that Native Client and local data are actually strategies in the competition between Google and makers of desktop applications.

How would that affect user privacy? Again, it’s impossible to tell from any of the Google design documents I reviewed. But it doesn’t leave me with a warm and fuzzy feeling.

I encourage Google to prepare a privacy notice for Chrome OS as it has for Chrome and other individual products. Without one, we’re left making mostly uninformed guesses.

Am I likely to use Chrome OS when it arrives in late 2010? It’s too early to provide a firm answer. I’ll re-evaluate the operating system at release time. Currently there are too many unanswered privacy questions. But my guess is that I will selectively use the Chrome browser but not the Chrome operating system.

Notes:
[1] If you’d like screen shots and a fair summary of Chromium OS features, I recommend Dedoimedo’s Google Chrome OS – Is this the future?
[2] The version of Chromium OS was 0.5.25.201001051736 Developer Build aa10e830.F

Threaded Bookmarks 29-January-2009

Gary Lewis — Fri, 29 Jan 2010 14:29:54 +0000

In this post I’m experimenting with a variation on bookmark posts by including only items connected in some fashion by a common thread.

Today’s theme is one of design. Is there a way to use data analysis and visualization to engage people enough that they follow their curiosity into exploration of complex issues?

I very much like the problem orientation of the auto technician interviewed in the New York Times article. Don’t treat symptoms; look for the root causes; and start with the money. His question “why does it cost so much?” is equally valid for higher education.

In the second post, a designer provides a five-stage model of how to create participatory experiences within museums. The model uses a me-to-we flow that starts with “individual consumes content” and progresses to “individuals engage with each other socially.” Step two (“individuals interact with content”) seems particularly important.

Data analysis and visualization may provide an entry point. But interaction with the content is essential to provoke curiosity and the urge for more exploration. Beyond curiosity a transition must occur and stories may provide help. I’m a big fan of storytelling for precisely the reasons that John Hagel mentions in the third post.

Do you see where I’m headed? Suppose we take an issue like the high cost of higher education. There is absolutely tons of data available on this topic and no end to the research done on it. But is there a way to design visual analysis as an entry point for individual interaction with the topic and then provide a way for people to explore the complexities with stories?

If any of you are aware of web sites designed in a similar manner (around any topic), I’d very much appreciate it if you would provide links in the comments. Or email me with links. Many thanks!

In Colorado, Craving Reform of Health Care and Congress
Robert Pear. New York Times. 10-January-2010

“I am an automotive diagnostician,” Mr. Seyfer said. “We look for the root cause of problems. If we treat the symptoms, the problem always comes back. With health care, we are not treating the root cause: Why does it cost so much?”

A Revised Theory of Social Participation via a “Me-to-We” Design
Nina Simon. Museum 2.0. 25-Janurary-2010.

[I]f you want to support social engagement among people, especially in an unfacilitated setting (i.e. no tour guides or game masters), you need to start by designing personal services for users, then linking up users through shared interests or objects to promote interpersonal connections. You don’t start by designing “for the crowd.” Instead, you design ways for each person to feel acknowledged and valued as an individual. You make them comfortable interacting on their own, and then start providing opportunities to connect with others.

From Research Monographs to Story-Telling: New Forms of Communication in the Big Shift
John Hagel. Edge Perspectives. 22-January-2010.

Stories provide powerful filters that help us to orient ourselves in complex and rapidly changing worlds. On a daily basis, we are bombarded by an ever expanding array of stimuli that spread our attention ever more thinly and risk disorienting us in terms of a sense of what matters and what is simply noise. Stories help to focus our attention. The task of the story teller is to reduce a complex situation to its essence, making difficult decisions about what matters and what is simply extraneous while still preserving the relationships and textures that drive forward movement. The result can be very helpful to listeners in terms of communicating what is really important in a world that distracts and diminishes our ability to focus. At the same time, stories also encourage listeners to use their own imagination to enrich the context of the story – they pull listeners in and invite them to co-create the world at hand.

Playing with Google Chromium OS

Gary Lewis — Tue, 19 Jan 2010 21:42:33 +0000

I downloaded the source code for Google’s Chromium OS and then built and installed the image on an Asus EeePC 1000 netbook [2]. Included was a pre-built binary of the Chromium-based browser.

Specifically, Google Chrome OS will run on specially optimized hardware in order to get enhanced performance and security.

I encourage Google to allow users to opt-out of the initial OS login.

• Users can initiate a completely stateless session, which does not sync or cache data.
• All system settings would be kept out of this session, including networking config.

I encourage Google to clarify the stateless concept. It’s too important to leave it open to individual interpretation.

Chromium OS devices should provide privacy protection for user data stored on the local disk. In particular, some subset of a user’s email, pictures, and even HTTP cookies will be stored on the local drive to enhance the online experience and ensure Chromium OS devices are useful even when an Internet connection isn’t available.

How would that affect user privacy? Again, it’s impossible to tell from any of the Google design documents I reviewed. But it doesn’t leave me with a warm and fuzzy feeling.

I encourage Google to prepare a privacy notice for Chrome OS as it has for Chrome and other individual products. Without one, we’re left making mostly uninformed guesses.

Bellylaughs Amid Technical Complexity

Gary Lewis — Thu, 24 Dec 2009 16:31:25 +0000

Recently I’ve been up to my eyeballs in HTML5 and SVG (Scalable Vector Graphics). Not exactly bed-time reading. So imagine my surprise today when I read an article on SVG that had me laughing out loud. Here’s one of the milder examples:

Space travelers encountering new species should not necessarily think that stealing a dictionary and plying an informant with single-malt scotch will automatically produce a Rosetta stone.

The author is David Dailey. He’s a professor of computer science at Slippery Rock University in Pennsylvania. The article is Why is SVG going to be REALLY BIG?. Dailey presented the paper at the SVG Open 2009 conference held from October 2-4 in Mountain View, California.

Here’s a taste of the main message:

In short, the fundamental metaphor of SVG for the expression of human meaning is closer to the realm in which meaning exists (at least in some abstract mathematical sense). SVG has not suffered from the setbacks that millenia of bad assumptions about how to communicate thoughts have saddled onto speech, text, hypertext and, hence, HTML.

Those of you with philosophical inclinations will likely be engaged by the article. All of us can take heart from the hope that HTML5 and SVG may open new opportunities for deeper understanding of complex policy issues.

Final XQuery Use Case

Gary Lewis — Wed, 09 Dec 2009 23:04:58 +0000

For a finale in my run of testing xquery in different data situations, I chose to deliberately use xquery where it would be slightly disadvantaged. So I decided to analyze moderately complex survey data in the form of the October 2007 School Enrollment Supplement to the Current Population Survey. The CPS is sponsored jointly by the U.S. Bureau of Census and the Bureau of Labor Statistics. It is the data source used to calculate those dreadful monthly unemployment rates we’ve seen in the current recession.

Instead of unemployment, I chose to examine whether family income influences students’ choices about enrollment in public versus private colleges and universities in the United States. Given the substantial gap in net price between public and private institutions, you would think that family finances might play an important role in the public vs private decision.

One caveat as always. Throughout these use cases, I’ve been less interested in the research question than in how xquery performs. I chose research questions that I thought would be interesting, but I was more concerned about the tool than the actual research.

In this post I won’t give away the story-line about the research. For that, please refer to the 4-page PDF that describes the project.

However, I will say once again that I enjoyed working with xquery. There are several implementations of the W3C standards. I used zorba.

This project did place stress on xquery performance at times, but I was able to concoct workarounds that I’ll be able to use in similar situations if they occur in the future.

Also, I’d like to highlight once again the very intricate nature of data analysis. I’m occasionally uncomfortable in this age of transparency and big data and semantic linked data that ease-of-use is the assumed norm and expectation. It has never been my experience that data analysis is easy or fast. It requires living in the data for considerable periods of time. It requires thought and play and more thought. It requires failure and more failure and finally small steps of success. I hope this will be evident in the PDF that describes this project.

I also tried again to provide links to all important programs I wrote as part of this project. It’s not exactly what some call open research, but it certainly is a step in that direction. If anyone feels motivated to duplicate what I did or, better yet, to modify and improve upon what I did, then I think you’ll find the source links useful. And I’d be happy to assist if you have questions.

With the conclusion of this project, I no longer feel the need to equivocate about xquery. I will definitely use it.

Another XQuery Use Case: Is Higher Education Countercyclical?

Gary Lewis — Mon, 10 Aug 2009 17:52:12 +0000

I’ve completed another XQuery demonstration project (for earlier ones, see XQuery). In this version I used XQuery to assemble data to consider anew the conventional wisdom that higher education is countercyclical (ie, enrollments grow during recessions).

Here’s a sample graph from the report. It shows the annual percentage change in enrollments from all U.S. degree-granting institutions of higher education from 1968 to 2007. The grayscale background shows unemployment rates and periods of recessions. The report is available in this pdf that includes better resolution graphics.

Here is an abbreviated version of the summary section from the report.

1. The new XQuery use cases included screen scraping enrollment data embedded in html, and the pipelining of enrollment and economic data through several staged transformations.
2. The evidence that higher education in the United States is countercyclical appears weak based on the exploratory analysis done here.
3. The evidence varies somewhat by institutional control and type, but mostly the general observation of a weak association between enrollment change and recession cycles holds true.
4. The tools used in this project, XQuery and R, are wonderful research tools but are not suitable for more general use without wrappers that mute their complexity.
5. The screen-scraping technique used in this project provides another access route to a vast amount of U.S. Department of Education data on the web.
6. None of the data sources used in this project included semantic or linked data markup. Whether XQuery can be used successfully with RDFa or microformats seems worthy of investigation.

Here are links to various pieces of this project:
a. Final report
b. Documentation for XQuery source programs
c. XQuery to generate enrollment data
d. XQuery to generate unemployment and recession data
e. XQuery to merge enrollment, unemployment, and recession data
f. R analysis history

I remain quite satisfied with XQuery.

XQuery and an XML Database

Gary Lewis — Wed, 01 Jul 2009 19:30:26 +0000

Recently I wrote about using XQuery with the Federal Reserve Economic Data (FRED) API. I also complained a bit about sluggish performance (pdf), but noted by way of explanation that I was using a creaky old server and that XQuery was processing XML files and not a database.

Since then I installed Oracle’s BerkeleyDB XML database and duplicated one of the performance tests I’d done with the FRED data. Response time in the test dropped from 195 seconds to 2 seconds. Not bad; much more consistent with what I’d expect from SQL queries against a similarly sized relational database.

These are not precise performance tests by any means. For example, the XQuery processor I used with the original FRED data was Zorba, but XQilla is the XQuery processor that comes bundled with BerkeleyDB XML. So, for certain, I changed at least two important factors in the tests, and unless you make controlled changes in factors it is pretty much impossible to attribute causation.

However, I rather doubt that the choice of the XQuery processor had much to do with the performance improvement. Performance only improved when I created database indexes for the XML.

At this point I am still very much playing; just trying to use XQuery in a variety of environments to better understand what it can and cannot accomplish. If you’re a researcher, for example, it doesn’t much matter if it takes 195 seconds for a data integration step in an entire research process that may take days, weeks or longer. However, 2 seconds is much preferred if you’re doing ad hoc data mashups in a browser.

From what little I’ve seen of Oracle’s BerkeleyDB XML database, it’s an interesting product, available with either a commercial or free open source license. It’s an embedded database accessed through a programmatic API and linked libraries. It is not a relational database, database management system, or a database server. It does, however, offer a lightweight solution when performance and data persistence are critical. Disclaimer: I have no association with Oracle, BerkeleyDB, or BerkeleyDB XML.

I realize this blog post will only interest a very few people, but from time-to-time I may write similar posts that document my playing with XQuery. I continue to like what I see. Truthfully I’m not certain where it’s headed, other than to say that the driver is free learning for everyone everywhere.

XQuery as a Web Query Tool

Gary Lewis — Fri, 19 Jun 2009 11:52:03 +0000

Those of you who have followed this blog know that I’ve long searched for something that I call a “web query tool” (click on the Query Tools category to see this history).

As a way of identifying a “web query tool,” I use a very simple measurement device … it is something that can answer the following question: “Please give me a list of about ” (eg, a list of web tutorials about python programming). This sounds like a search, but it is more than search. It is an ad hoc query run against a web of data instead of a database.

I don’t think the time is at hand when I’ll find a real web query tool.
But recently I’ve been playing quite a bit with XQuery and have been pleased (eg, see Making a Dent in a Steep Learning Curve). I believe XQuery will do nicely as a tool for mashups of web data and as a front-end into a technology stack that runs from data integration to transformation to analysis to visualization and then to presentation.

However I wanted a reality check from people far more experienced with XQuery than me. So I posed a question to a list that serves people focused on XQuery. The question contained a twist about the future as well.

The respondents were uniformly gracious, and I found the conversation very helpful. Below is an edited version of that conversation organized as threads rather than in time sequence. Enjoy.

Notes:
1. I’m reluctant to quote the words of other people. So originally I thought I would summarize the conversation. But this made it lifeless. So I left the respondents anonymous but included their (slightly edited) direct quotes. I also wrote the list where the original conversation happened and included a link to a preview of this blog post, and asked if anyone objected if I included the transcript in the post.
2. In the transcript, indents denote threads. So, for example, something indented once means it is a response to my original question. Something indented twice is a response to the immediately preceding response indented once. And so on.
3. I omitted side conversations not directly relevant to my original question. As I write this, the discussion continues on the list but it has long since left my original topic. This still evolving conversation is not included in the transcript.

Transcript

Gary Lewis, 10:13am
Can anyone comment on the possibility that xquery could be used as a data integration/mashup tool running in a browser that included a web server? There’s been quite a bit of excitement the past couple days about the Opera Unite announcement of its “web server in a web browser” concept. Is it possible that xquery, as query/integration/transformation tool, might someday be a plugin to a browser (eg, Chrome; Chrome/Wave)?

Respondent #1, 10:25am
sure why not … XSLT support in the browser was/is a plugin and sets a precedent for this kind of thing

XQuery in the browser would be great, anything that reduces the hegemony that is javascript … which reminds me, I know it sounds strange but perhaps browsers will turn into *the* personal ‘database’ for the individual someday and it will be natural to have something like xquery to help process the data it contains.

Respondent #2, 10:32am
This may be of interest -
http://www.zorba-xquery.com/index.php/xquery-in-the-browser-xqib/

Respondent #3, 10:42am
There are a lot of people keen on the idea of xquery in the web browser.

Unfortunately, though, I think there’s a bit of a conflict. A lot of the benefits of the web architecture come from the fact that the browser is small, ubiqitous, and predictable. The more goodies you put in it, the less that remains true. And the less it is true, the harder it is to write applications that will run anywhere and talk to anything. This is why, after ten years, writing applications that rely on XSLT-in-the-browser can still be problematic, as recently discussed on the xml-dev list.

Respondent #5, 1:25pm
Well, one way to achieve the same result is to build on that small and predictable foundation. This rules out plugins, but there’s other possibility – that of “compiling” XQuery to cross-browser JavaScript. We already have GWT that does a similar thing, so the general concept is proven, and I don’t see why doing the same to XQuery would be any harder. The main problem that I see there is that browser XML manipulation APIs aren’t standardized yet, but this is going to change with HTML5 (and meanwhile there are already “good enough” ways to work around this by detecting and using the mush-mash of the existing browser-specific APIs).

Respondent #4, 11:21am
http://xqib.org

Gary Lewis, 11:30am
A couple people have mentioned zorba. I’ve used and quite like zorba xquery; even wrote a blog post about it and included some analysis of Federal Reserve Economic Data time series accessed via an API with zorba.

I’m also familiar with the zorba XQIB project but do not use a Windows OS so have not played with it yet. But my impression (probably wrong) is that XQIB is cast mostly as an alternative to JavaScript, using scripting extensions to XQuery. If true (please let me know otherwise), I’m not so much interested in this aspect. As I said earlier, I’m mostly interested in a data integration & transformation tool that sits in a browser and gets used as a precursor to analysis & graphics tools.

Respondent #7, 5:00pm
the xqib people can answer you in more details, but in general, what they do have is a full embedding of a Zorba xquery processor in the browser — this includes all the functionalities of Zorba, full XQuery with updates and scripting, XQuery 1.1 with groupby, windowing, try-catch, etc, plus the REST, function libraries.

What users use this XQuery for is there own decision: can be very well data integration, data transformation, RSS integration, etc, everything is good, as long as it can be expressed in XQuery.

Respondent #6, 1:26pm
I surely see your point about XQuery in the browser – as stated before by others: check out our project at http://www.xqib.org/

At the moment, the release works only with Internet Explorer – but the next release (which is nearly finished – we hope to release it this month) will also run with Firefox and will be much faster. [...]

But I wanted to ask you, where exactly you see the usecases of having a web server in the browser? To be honest: the last thing I want running in my browser is a server (as [Respondent #3] stated because of complexity, but also because of security). Also, in most companies (and also on most routers), all ports are normally closed, which makes a webserver useless. The only benefit I can see, is that this way it would be easy to implement some kind of RPC from the server to the client. But in my oppinion, this would be a too small advantage to legitimate a webserver.

Gary Lewis, 4:29pm
Let’s start with this question. In a p2p web … as Google Chrome/Wave and Opera Unite suggest might happen … is there a new role for xquery? I guess I’d answer that question with a tentative “Yes.” Here’s a simple example. There’s a lot of buzz around transparency now … government transparency, data transparency, etc. In the U.S., data.gov is a recent case. The idea is simple enough. Make government data sets available to the public and let them mash it up and then share it. [In some ways this scenario gives me the creeps, because I know from years of experience how very difficult it is to insure data integrity during analysis.] I’d love to use xquery as a data integration and transformation tool as preparation for analysis, visualization, & presentation. And then to share the data with others in a kind of social network of data reuse. I was guessing when I suggested that xquery in a browser with a web server might make this future possible. I’ve no way of knowing. But maybe one of you might.

Respondent #8, 6:53pm
Why? To me, the best thing about XQuery is that it can run natively (and more efficiently than the more encompassing XSL) on an XML DB providing all the speed and memory efficiency afforded by the different and incompatible XML DBs. Caveat emptor: I fall into the camp that doesn’t believe XQuery is a good styling language or a good PHP-like webapp templating language. (if there even is a camp, rather than me leaning over a bic lighter).

Running XQuery in the browser seems like a huge waste of bandwidth where you would have to download everything you want to query against, perhaps to find only a tiny fragment. I would create a URL and just GET it from a server where XQuery creates the XML or JSON to be styled on the client with XSL or JavaScript.

Respondent #4, 1:36am
i think browser can act as proxy server to localhost-based eXist database (or AMP if somebody still love it).

Making a Dent in a Steep Learning Curve

Gary Lewis — Mon, 08 Jun 2009 20:14:56 +0000

In a previous post called Web Query Tools – Part 2, I said:

I’m now ready to start work on another XQuery data integration, this time using the new FRED API from the Federal Reserve Bank of St. Louis. Not that I’m particularly interested in banking-related data, but FRED uses a REST web service architecture and will allow me to play more thoroughly with Zorba’s REST capability. And the volume of the data will allow me to stress test the performance of Zorba’s XQuery.

The siren’s song, of course, is that the web may one day become a giant database of interrelated data.

I’ve now finished the XQuery/FRED project. You can see the details here (pdf), but I’ll touch on a few of the points.

The FRED database includes more than 20,000 economic time series. The folks at the Economic Research Division at the Federal Reserve Bank of St. Louis have done a nice job making this data available via an API. With so much data to choose from, however, it helps if you know at the outset what kind of data you’d like to retrieve. I didn’t.

So I set myself two tasks: 1) Build the XML documents to provide flexible overviews of the 20,000 time series; and 2) Choose two of these times series and conduct a small analysis.

I did the first job using Zorba XQuery and the second job using XQuery and the R statistics package. Again, the details are here (pdf).

Basically I came away feeling like I’d made some progress. The Zorba REST calls to the FRED API worked flawlessly. Some of the XQuery programs with joins to multiple XML just chugged away slowly, but then I was using a creaky old server to run Zorba and serve up the XML files. Otherwise it was a wonderful opportunity to learn more about XQuery as a web query tool. I continue to like what I saw.

Since that’s a pretty positive statement for me, perhaps this is the place to say that I have no relationship with Zorba or any of the other XQuery implementations. I’m simply looking for a tool that will utilize some of my strengths in pursuit of the main reason I write this blog … which, if you don’t yet know, is to help nudge us closer to the day when learning will be free for everyone everywhere.

Next I plan to look at Zorba in a variety of other settings. Options include the XQuery in the Browser project (XQIB), XQuery scripting and Sausilito to construct web sites, and some serious performance tests of XQuery on EC2 and S3 (Amazon Web Services).

Below is a screen shot of the R analysis. It shows a not unexpected relationship between the unemployment rate in the United States over the last 55 years and the rate of change in outstanding household credit market debt. The current recession seems somewhat atypical, however. Details and a legible graphic are also in the PDF, as is souce code for the XQuery programs I wrote.

Web Query Tools – Part 2

Gary Lewis — Fri, 24 Apr 2009 13:58:06 +0000

There’s huge interest in harnessing the web as a giant database, and the motivations for this are easily as diverse as the interest is large. I’m just looking for a precursor.

Two quick examples. Google researchers describe “a unified Web knowledge base” as the “holy grail of web information extraction” (PDF), and the company energetically applies their research to harvest additional structured data from natural language text, HTML-embedded tables, and the so-called deep-web of databases sheltered behind form-only access.

Tim Berners-Lee, whose legacy is the web, provides another example. In a video from TED, he calls for a “new reframing” that will release the “unlocked potential” of the web through linked data. The idea is that people and computers could traverse data related to other data, from A to B to … wherever you want to go as long as the relationships, the tuple linkings, are available. Berners-Lee’s new reframing translates into a web-of-data that could be used by scientists, citizens, social reformers, businesses, entrepreneurs, and governments in innovative new ways.

I applaud these efforts even though I realize that the potential for good and for ill seems equally strong. But my focus is nothing so grand as a web-of-data. I’d be happy if I could just answer a profoundly simple question: “Please give me a list of all web tutorials on python programming.” More symbolically I want to answer questions of the form: “Give me a list of about .” From there, of course, the possible questions become far more interesting.

For certain my query would be trivial if a web-of-data existed. But it does not, even though progress is being made. So, in the interim I’m looking for a substitute.
My latest excursions have taken me into the world of XML, and specifically into XQuery and, to a lesser extent, into the XQuery relatives XSLT and XPath. All of these are W3C recommendations that have been implemented in both open source and proprietary products.

Why XQuery? Well, because XML is part of the fabric of the web, and because data in XML format can be queried with XQuery, and because I stumbled upon Zorba via an O’Reilly xml.com article called “Something Tells Me You Need to Pay Attention to This.” How could I resist a title like that?

I started playing with Zorba’s XQuery about two months ago. Maybe I’ve written a couple hundred queries now. Like the learning curve for SQL, it’s clear that a couple hundred is at least an order of magnitude too small to become skilled. But it’s great fun and I’m encouraged by what I see.

One example. Tony Hirst at the Open University has done some cool things recently with The Guardian’s new data API. In one of these projects, Tony used data from The Guardian’s university guide to do a mashup on student satisfaction in architecture and planning programs at various UK universities. It featured a very nice use of DabbleDB database.

Data integration is one of the strengths of XQuery, so I set about following Tony’s lead to see if I could duplicate his mashup but by using Zorba. It was very fun and I learned tons. You can see the results in this PDF.

I’m now ready to start work on another XQuery data integration, this time using the new FRED API from the Federal Reserve Bank of St. Louis. Not that I’m particularly interested in banking-related data, but FRED uses a REST web service architecture and will allow me to play more thoroughly with Zorba’s REST capability. And the volume of the data will allow me to stress test the performance of Zorba’s XQuery.

If you are interested at all by Zorba, I’d recommend you read some of the technical documents where you can catch glimpses of longer term development objectives and experience some of the chutzpah that must exist in the development team. A recent example is XQuery in the Browser, which was presented this week at the 18th International World Wide Web Conference in Madrid. The article basically takes aim at JavaScript. As another example, check out the plans of a 3 year-old startup called 28msec and some of their technical papers. I particularly enjoyed the architecture discussion in Donald Kossmann’s slide presentation on Building Web Applications without a DBMS (PDF).

Ok, I better stop. It’s already beginning to sound like an infomercial. Hopefully some of my sincere enthusiasm comes through, however. It’s a hopeful time. And now it’s back to making a dent in that order of magnitude learning curve.