One size does not fit all with search engines

At Enterprise Search Summit East in May 2007 and again at Enterprise Search Summit West in November 2007, I asserted that enterprise search is dead. I will probably say the same thing in 2008.

The remark underscores the crisis in search and retrieval behind an organization’s firewall. The phrase "enterprise search" is misleading but ubiquitous. Dissatisfaction with enterprise search hovers in the 60 percent range—a figure that has been increasing since I started keeping records in 2002. The reason is simple: Keyword retrieval is useful some of the time, but most users need suggestions, tools to find previously accessed documents or answers to questions like, "What’s the telephone number of Fujiwara Deliveries?"

Most enterprise search systems do a lousy job of delivering immediately useful information. Reviewing a laundry list of results is not helpful. In fact, sticky notes on monitors are proof that an organization’s enterprise search systems are not cutting it.

I’ve tallied more than 150 vendors of search systems. They range from startups like PolySpot to vendors with technology that is 25 years old, such as Open Text’s BRS engine or Autonomy/Verity. In between those two extremes are keyword retrieval vendors (dtSearch); rich text processing vendors (Cognition Technologies); free search systems (Lucene); specialist vendors (Blossom Software); utility vendors (Data Harmony); semantic engines (Siderean); super-platform vendors (IBM, Microsoft and Oracle); and the Big Three (Autonomy, Endeca and Fast Search & Retrieval [FAST]). The enterprise search spectrum is more overloaded than wireless bandwidth.

Into this crowded field comes Google’s Appliance. Does Google’s enterprise division have a recipe that is superior to the other 150-plus vendors? Recent pronouncements from experts, pundits and industry observers assert that the Google Appliance is a weak sister in the behind-the-firewall search market. In a horse race of 150 stallions, Google’s running dead last caught my attention.

The Appliance is a search toaster. It’s similar to other search appliance offerings from EPI Thunderstone, Index Engines and Planet Technologies. Each of those products is designed to be unpacked, plugged in, configured and made available to users. Each is a hardware device, and the customer is relieved of some of the muss and fuss associated with procuring hardware, installing software and getting thesystem up and running. The appliance vendors offer convenience, but the tradeoffs for some organizations may be unattractive.

Appliances can be customized, but their purpose is fast deployment of search. A licensee can customize any appliance because the vendors know that some customers will "learn" what their users need after getting the system up and running. Thunderstone offers a wide range of configuration options plus a software development kit, a driven API and fast indexing. But a novice is likely to spend some time figuring out the system.

Google takes a similar approach. The basic Appliance can be serving queries in less than a day, longer if there are hundreds of millions of documents and hundreds of servers to configure. Index Engines and Planet Technologies take a similar approach.

Comparing the appliance solutions to a hosted service where the customer does nothing except open a port is useful. Hosted services are easier to deploy, faster to get up and running, and less hassle than any on-premises installation.

Comparing the appliance solutions to the box of parts delivered, customized, tuned and deployed is also helpful. The search systems available from such companies as Autonomy, Endeca, FAST, IBM, Microsoft, Oracle and many other firms require considerable care and feeding.

Isolating a single vendor’s system and identifying it as lacking this or that feature, or suggesting one vendor’s approach is better than another’s is unhelpful. The reason is that each system does some things well and others things differently. Depending on the specific situation, certain systems may be more or less well suited.