In September, JSTOR released a new XML Gateway to facilitate federated searching or 'metasearching' of the JSTOR archive. The Gateway is the culmination of JSTOR efforts to advocate for and adopt a standards-based approach to metasearching. The aim is to improve the interactions between JSTOR and metasearch engines, ultimately resulting in a better search experience for users.

What is metasearching?

Metasearching enables a user to simultaneously search multiple content sources (databases, websites, indexes, and other search engines) through a single interface. The metasearch engine sends the user's query out to a variety of sources, retrieves and aggregates the corresponding results, and then presents a final set of results to the user. Metasearching can both dramatically expand the scope of a search and decrease the time spent searching. While this may be a very convenient tool for users, the set of results returned by metasearch engines can vary in quality and relevance. A key factor is the ability of the metasearch engine to successfully search the databases targeted by the user.

How is it done?

To date, metasearch engines have relied on two methods to search various content sources. The first is the Z39.50 protocol, a standard designed to enable communications between computer systems that has been widely used for years by libraries to search electronic resources. While the Z39.50 protocol can be employed effectively for metasearching, it has limitations and can be difficult to support because different resources implement Z39.50 differently, which can lead to interoperability problems.

An alternative method, widely used by metasearch providers, is HTML parsing or 'screen-scraping.' In this approach, the metasearch engine mimics a user by submitting search requests to a database, retrieving an HTML result page, and then parsing the information from the page into the aggregate result set passed onto the user. HTML parsing is attractive because it enables metasearch engines to search a wide variety of data sources that do not have Z39.50 deployments without establishing a formal relationship or "connection." But screen-scraping introduces other problems. Because there is no agreed-upon method for query and data exchange, getting accurate results can be challenging. Success is heavily dependent on the metasearch engine's ability to keep up with data sources that are continually changing. If JSTOR updates its search interface, for example, the HTML parsing mechanism employed by metasearch engines no longer works.

A new approach

In 2003, JSTOR witnessed the growing use of metasearch engines among libraries and scholars and recognized the benefits of establishing more reliable connections between the archive and metasearch engines. JSTOR began to participate in community discussion of solutions through the ongoing National Information Standards Organization (NISO)'s Metasearch Initiative. Among other recommendations, the NISO Metasearch Initiative concluded that the best means for improving the communication between content providers and metasearch engines would be through the deployment of XML Gateways.

JSTOR began development of its XML Gateway in April 2005. The Gateway is based on the Search and Retrieve URL (SRU) Service, a Web Services-based protocol currently under review as a part of the NISO Metasearch Initiative. The SRU Service uses the Common Query Language (CQL) as the format for submitting search queries to content providers. CQL allows for both very simple and complex, powerful queries that involve Boolean operators, nesting, etc. The search results that are returned in response to these queries are organized as XML records in the Dublin Core format and are listed in order of relevancy ranking. The JSTOR XML Gateway was designed using SRW/U Open Source Software, developed by Online Computer Library Center (OCLC) Research. Please visit www.oclc.org/research/projects/webservices/ for more information.

This new approach makes it possible for search requests to be received, translated, and their result sets delivered within an agreed-upon framework, while eliminating the main shortcomings associated with earlier methods. For JSTOR, it provides a way for the archive to be available to users that wish to rely on these search alternatives and gives a greater level of assurance that the results returned for JSTOR by the metasearch engine will be accurate.

Community partners

This past summer JSTOR worked with Auto-Graphics Inc., Endeavor Information Systems Inc., ExLibris, and WebFeat to develop and test the XML Gateway. JSTOR has recently begun working with several additional vendors and hopes to hear from others over time. For information on JSTOR's XML Gateway visit http://www.jstor.org/about/xml_gateway.html. For detailed information on JSTOR's current Federated Search partners visit http://www.jstor.org/about/metasearch_partners.html.

New Linking Partners

JSTOR continues to expand its linking relationships with publishers, libraries, and other online resource providers. In this year alone, 14% of article views in JSTOR resulted from links to the archive from other websites. We expect this figure will grow over time as our efforts to enable pathways for users continue. Set-up and configuration information, as well as a complete list of linking partners and instructions, is available at www.jstor.org/resources/linking.partners.html.

Last updated on September 8, 2006