Aggregate your searches with OpenSearch
by Jacob on Nov.10, 2006, under Technical
The rise of RSS and Atom feeds brought increased usage of the word, aggregate, which meant to gather many articles together to one destination. This turns out to be a really great thing. No longer do I have to go from site to site to check on all my favorite news. I can use an aggregator, and bring all the news from all my favorite sites together to one point. Not only can I do this with news, but also blog posts, photos, audio clips, events, alerts, and the list goes on.
Now the time has arrived to bring aggregation to search. Why should I have to go to all of my various different search engines to search for a term, when I should be able to have all the results from all my my favorite search engine aggregated into one single application?
Search result aggregation is not a new thing. The MetaCrawler search engine has been doing this sort of thing for years. Unfortunately, the search information is bounded and limited to just a couple of pre-determined set of engines.
OpenSearch is a set of standards which opens up search aggregation. Any search engine can share their search results using the OpenSearch standards, so that search aggregators can pull it all together. All the OpenSearch information and documentation is actually pretty well written (unlike other specifications which are dozens of pages long and too difficult to get through) and are available at www.OpenSearch.org.
One example is that blog searches can be aggregated. I operate a few different blogs, each one with its own search function. If I were to want to look for the word “oregon” in my blogs, I would have to visit each one individually and use their individual search engines. I could try using a third party search engine limited to the scope of only my blogs, but if that search engine hasn’t visited all the posts in all my blogs, then I’m not getting great search results back. Now suppose that each of my blogs’ search functions supported OpenSearch. Then I could simply use a search aggregator and search all of them at once, and all their results displayed all at once. Bingo.
OpenSearch version 1.1 uses standard technology and is composed of two main parts. There is the OpenSearch description file which tells an aggregator or search application how to perform a search. This is what Firefox 2 and IE7 use for their search plugin format. The description file is XML with a fairly simple vocabulary. The second part is the search results, which uses RSS and Atom format standards along with a few custom tags to return the search data. Using RSS and Atom makes it easier to write search clients and servers which use OpenSearch because there are already libraries for the standards and developers are already familiar with the technology.
So when you use a search aggregator or application, such as A9.com, the application uses the description files to properly format search queries. The queries are sent to the search servers, which return the search results as an Atom or RSS formated file. The application then parses and displays in some meaningful way the search results from the various sources.
In the past we were limited to using either one generic monstrous search engine like Google, or bouncing around to many different tiny little specific search engines. With OpenSearch, we should be able to instead search many specific search engines without having to bounce around.
Related posts:
November 10th, 2006 on 2:48 am
[...] On my technical blog, I talked about aggregating search results with OpenSearch. Below is a religious case example for use of the OpenSearch technology. (You should read what I’ve written about OpenSearch before continuing to read this.) Something I’ve been working on has been my LDSsearch.com project. LDSsearch.com is a search engine which has indexed a selection of web sites which have been identified as friendly to the LDS faith. It is good, but it fails at completely indexing all LDS content out there. However, it does provide its search results in an OpenSearch format. Now there is another site called Scripture Tag which tags verses from scriptures. I’ve talked with the developer of the site about returning the scripture tag information in an OpenSearch format. I’m also hoping that someday we can get a search engine for general conference talks which can provide its search results in an OpenSearch format. [...]
November 12th, 2006 on 1:57 pm
While not an open solution, a solution that works right now is to create your own Google Custom Search. You provide the list of sites that will be searched and you get all the goodness of a Google search. You can put your search box anywhere you like and you can have people that collaborate with you on the list of sites that are indexed. The search on the front page of the Mormon Archipelago does this.
November 12th, 2006 on 2:57 pm
Yes, I’m aware of Google Custom Search, but it has the problem of using a single algorithm, with only a single way of looking at things (web page view).
With aggregated searches, you could query against say, a number of different quote collections, where the methods for searching and indexing quotes may use different algorithms, such as human tagging, than a general web indexer might use.
Aggregated searches do more than limit a search to a particular site, but they can allow for specializations of each individual search source.