Here’s how Yahoo describes YQL:

YQL is a new web service API that lets you access other web services using a SQL-like language rather than typical programmatic access. You can think of it as a command line version of Pipes. Its goal is to make data from Yahoo! as well as from across the internet universally accessible through a single common interface.

I’ve lamented previously that search tools provide only meager help for web queries. For example, the following query cannot be accomplished in a web search: “Give me a list of all JavaScript tutorials available online.” This is a trivial query as queries go.

So when I heard about the YQL announcement, I promised myself I’d have a look at it as soon as I could. This post provides a preliminary reaction to YQL based on several readings of the documentation and perhaps 6 hours of play at the YQL console that Yahoo provides (very nice, thank you!).

YQL is definitely a step beyond search. For example, you can get tantalizingly close to an answer for the query on JavaScript tutorials. The YQL looks like:

select * from search.web(0) where query = “javascript tutorial” | unique(field=”url”)

This is definitely SQL-like. There are select, from, and where clauses.  The search.web notation refers to one of about 50 XML  or JSON data collections presented by Yahoo as a “table.” Each table has the equivalent of SQL data columns that can be used for filtering or for display in the select clause.

The (0) after the table name signifies that I requested an unbounded query. A setting of (1000) exposes only a subset of 1000 entries in the source web service.

The | pipe is a post-query operation that discards non-unique url’s. You can also perform sorts in a post-query operation.

There are several other nice features of YQL that I won’t go into here. One example is sub-query capability. But take a look at the YQL documentation for examples. Or better yet, play with the YQL console and experiment for yourself.

Compared to any standard SQL, YQL is a very simple query tool. This is not necessarily a bad thing. For example, when someone designs large data warehouses from source data stored in relational tables,the design almost always removes the complexity from the data so that users and analysts can focus on the business questions and not on the database architecture. Simplifying the data structure means that the stress on the query and analysis tools is reduced. Simple tools work fine when the data architecture is also simple.

The YQL documentation that I saw does not contain architectural schemas, so I cannot comment on whether the “tables” exposed by Yahoo are adequately served by a simple query tool like YQL. Given the early stage in the lifecycle of this project, I suspect the answer is no. But I’d also expect additional work on both the architecture and the language will occur as people use it.

The JavaScript query that I wrote in YQL failed to complete. At present, Yahoo has a governor on YQL queries. I believe the max is 30 seconds or 50,000 items. Run-away queries are every DBA’s nightmare, so it is not surprising that Yahoo would place limits on YQL queries.

For web queries to be truly useful, however, we’ll need a way to relax the governor restrictions and actually do complete queries. But for now, I’m very grateful to see the precursors of tomorrow’s web query tools emerging.

Part 2 of this series will look at another approach to analysis of very large datasets. It’s called Pig (no joke) and is described by Yahoo and Apache Hadoop as an “infrastructure” or “platform” to conduct ad-hoc data analysis. Apparently “the highest abstraction layer in Pig is a query language interface, whereby users express data analysis tasks as queries, in the style of SQL or Relational Algebra.”

Bingo. If only it’s true. I’ll let you know what I find out.