Gazing into the crystal ball:Search technology gets smarter and more personal

By Kim Ann Zimmermann

It has been a long-standing joke among users that of the 1,000 results a search engine might produce, you're lucky if one is useful, and that could be the last file presented on the list. And if you type in a common word such as “sweater,” you'll have to dig through a pile of Web sites—selling everything from winter clothing to knitting products to dog accessories—to get to the item for which you are looking.

As companies build their knowledge databases, more accurate search engines are crucial to produce the best results for the customer—whether that customer is an internal user looking for the latest marketing report or a paying client surfing the Web for a size 6 pair of jeans.

One way search engines are becoming more effective is by using natural language queries. Instead of just typing sweaters, you can type, "Where can I buy women's red sweaters?" and get more accurate results.

Search engines are also working to address an annoyance often stated by search users, particularly those in large corporations with numerous databases throughout the organization: Before they can perform a search, they have to know where the item is located. The common response is that if they knew in which database last year’s sales forecast was stored, they wouldn’t need a search engine. Many search engines are addressing that problem by searching across multiple databases as well as the Web to find the appropriate documents.

"We really are stepping up the capabilities of search to reduce the frustration of the user," says Stephen Brown, VP of product strategies for Vality, to be acquired by Ascential Software). "Search is starting to move into a next generation of product expectation and market requirements."

One key way that search engines are changing is through personalization, Brown says. The search engine begins to learn the patterns of an individual's searches and to present them with the information that most closely matches their needs.

In another related trend that Brown refers to as semantic mediation, the search engine acts as an interpreter of the request based on certain models of behavior. "We all have our own syntax and choice of slang and terminology. The search engine acts as a go-between. In the early years, the burden was on the user to try out different words and do complex Boolean searches to determine how to adopt his or her language to a particular search engine," Brown says.

One technique that Vality uses to address those needs is called skimming. For example, the search engine recognizes various forms of the word bike, including bicycling and biking.

Brown says that search engine developers have come to recognize that search is really not about finding one thing. It is about navigation. Users rarely stop after one inquiry. One search leads to another. There is progressive discovery as the user navigates through large databases.

The option for natural language queries is becoming more popular, according to Beth Krasna, CEO of Albert Albert.

"Essentially, the idea is to be sure that what you mean is what you get," she says. "As we move forward, we're going to put more and more information online, and we have to improve use for those who are not so computer savvy. There will be computers in libraries, government agencies and train stations. The people who will be using these computers to look for an online form at the DMV or find the latest Tom Clancy novel at the library need to access information in a very simple way and in a way they are used to speaking."

"The need to perform natural language queries extends to the corporate world," Krasna adds. "In the last few months, almost all of our RFPs included a request for natural language option."

The goal of natural language queries and personalization efforts is to reduce user dissatisfaction with searches. Krasna says, "A lot of people are frustrated with search. If they are on a Web site, 70% of people drop out after doing a search because they can't find what they want and the results are so overwhelming. Corporate users are facing layoffs and downsizing and not everyone files things in the same way. The document you need might be in the sales database or the R&D database. The idea is to be able to search the way people are most comfortable and do it in a way that they can search various databases with one search."

Krasna also talks about "softening" what she calls the brittleness of search methodologies. "Search engines have a tendency to take the user's query very literally. We need to need to relax that constraint," she says. For example, someone searching for women's slacks might type "ladies pants," but that is not how the item has been identified in the database of a large online retailer. "We have to normalize what the user has asked for and translate that into nomenclature that has been used by the person creating and indexing the database," Krasna says.

Search engines can have influence on increasing revenue or reducing cost. "If an online user can search your database and get what they want, that's increased revenue. If your corporate users who are creating the database that the customer is searching can spend less time maintaining the database through tagging and keyword indexing, your costs are reduced," says Krasna.

The use of search to generate revenue is certainly driving innovations in technology, according to Laust Sondergaard, CEO of Mondosoft. The company uses behavior matching to determine search patterns. For example, he says, the company worked with a European drug company to determine that when users entered a particular phrase, they were actually looking for information on a particular drug. “Prior to this, unless these folks were diligent, they didn’t find what they wanted. Forty to 50% of users went away unsatisfied, and that was not good for sales.”

The search engine should almost be able to read the mind of the searcher, Sondergaard says. “We’re getting closer to that. The more information the search engine can retain on how you do searches, the more successful your search will be. Based on history, we can know if you use a nickname or other shorthand to search for a particular result. We can also know that when you search for one thing, you’re likely to search on another item for your next query, based on the patterns that have developed.”

The global nature of business is also pushing search engines to bridge the boundaries of language, Sondergaard says.

“There is a growing need to be able to have search be language-independent,” he says. “In the past, there has been a tendency to put all of the English documents in one database, for example, and the German documents in another database. But there may be an English-speaking user looking for a document that was created in German. We’re getting more requests to include more languages and be able to type a search in any language and pull up documents in that language, regardless of the language they were created in.”

Kim Ann Zimmermann is a free-lance writer, 732-636-3612, e-mail kimzim2764@yahoo.com