Search: sophisticated yet simple

Enterprise search solutions have traditionally been ahead of Internet search in terms of sophistication and effectiveness. Now some of those technologies are being directed toward the Web, with much success.

The library at U.S. News and World Report receives more than 800 requests per month from its reporters and other staffers. Its mission is to provide support for background research conducted by reporters covering a large range of topics, and also to answer questions from business users in the company.

To provide top-quality information to its customers, the library pushes out about 70 licensed subscription services, including LexisNexis and Proquest. However, users require training in order to access some of those resources effectively, and in a fast-paced environment not everyone can reach the ideal level of proficiency. As a result, the rate of usage for some of the paid subscriptions was low.

Researchers, including librarians and journalists, were also turning to the Internet to augment the information they were able to glean from the subscription services.

“We conducted a survey that confirmed our users preferred the simplicity of a typical Internet search engine,” says Jill Konieczko, director of information and research services at U.S. News. “Moreover, these engines have no password requirements or learning curves.”

Yet the Internet search engines also had drawbacks; for example, users often did not click past the first few results in a list of thousands of hits, and many hits were not relevant. To simplify the search process for subscription services and improve the accuracy of Internet searching, U.S. News & World Report selected Vivisimo’s Velocity platform, which includes a search engine, content integrator and clustering engine. The search engine provides a simple interface, and the content integrator allows searchers to view results comprehensively, no matter where the target information originated, including Web content. Behind the scenes, role-based authorization allows access to the subscriptions without requiring sign-on for each database.

The Velocity clustering engine groups the results into meaningful categories. “We liked the nesting of topical buckets,” says Konieczko. “We are now using Velocity to search the Internet more effectively and also to help our users access the databases from our subscription services in a simpler way.”

The deployment process was smooth; with a modest amount of help from Vivisimo’s support team, the library staff was able to customize Velocity successfully once the IT department had installed it.

Vivisimo was founded in 2000 as a spinoff of technology developed at Carnegie Mellon University, and initially focused on enterprise search.

“Our architecture is now a full-fledged XML-based search engine that is often used in Web-facing applications where the user interface is critical,” says Raul Valdes-Perez, CEO of Vivisimo. “Velocity is best recognized for its innovative clustering of search results.” The consumer version demonstrates that technique, and offers views by source, date and other parameters.

Velocity generates metadata in order to determine categories for clustering.

“We use diverse strategies to establish the metadata,” says Valdes-Perez. “Sometimes it depends on the source Web site, if the site is focused on politics or the arts, for example. Other times we can reverse engineer the database that generated the contents and derive the metadata.”

Often, he points out, more structure can be inferred than is initially apparent. For example, date, author and title can be found even if they are not explicitly tagged on the original page. In this sense, Velocity is creating the equivalent of a Semantic Web in situations where it does not yet exist.

Enterprise to Internet

Another search solution that is finding increasing use in searching the Internet is Exalead’s one:search platform. Developed with the aim of making search both powerful and simple, and widely used in Europe, one:search has been applied mainly for enterprise search (in the one:enterprise product version) and as the search tool on Web sites. For example, the Sanger Institute, which conducts genome research, uses one:search to search its databases and scientific materials. AOL France uses one:search for content on its site. The Exalead one:websearch product was developed specifically for searching the Internet.

One of the motivations for development of that version was the clear need. “While popular Web search engines have helped raise awareness of the importance of search,” says Francois Bourdoncle, CEO of Exalead, “consumer search engines do not meet the standards required for business use, including security and relevancy.”

Prior to founding Exalead, Bourdoncle developed technology at AltaVista that helped users make sense of their search results by displaying topic maps. “People should be able to search the way they think,” maintains Bourdoncle, “and to be able to find what they need even if they can’t formulate their search query properly.”

Exalead has a version of one:websearch on its exalead.com Web site that demonstrates the functionality of the product. After a search is conducted, the hits are shown with thumbnails of the target Web site. A sidebar shows related terms, multimedia options (audio and video), languages and other categories that the user can select to refine the search. The related terms allow the user to select a particular group of hits that reflects the search. The list can be expanded, and any item on the related terms list that is not relevant can be excluded from future searches. Exalead relaunched its Web search engine late last year with a new user interface, and is planning additional improvements in the near term.

Value-added video

Another important trend in search technology is the ability to search for and analyze multimedia information. Increasingly, information is being generated in Webcasts and video. Compared to print, those formats are often less dense in content and more difficult to search and analyze. For functions such as competitive intelligence, however, video and other multimedia content provide unique insights.