The Evolution of Enterprise Search

Specialist Applications Move to the Top of the “Food Chain”

Enterprise search and enterprise search platforms have been around for many years. Now, though, questions exist about the actual benefit of these types of solutions. Do organizations really need large-scale enterprise search capabilities, and—if so—is a true “enterprise search solution” actually affordable? Are no other, more cost-effective solutions available?

Given the number of specific search capabilities needed within most organizations, it’s becoming clear that before implementing an enterprise search solution, every organization should carefully answer the following questions:

• What is the benefit of an enterprise search platform to the organization?
• What level of organizational commitment is needed to optimize this solution?
• What level of complexity is required to integrate relevant search capabilities for a particular business solution?
• How soon is a deployed solution needed?
• What level of licensing, deployment and maintenance costs are acceptable given the perceived search needs?
• Is there a future for certain enterprise search vendors now that search is offered for free by many large players in the market? Can free search solutions actually meet most organizational needs?

But the most important question is whether an enterprise search platform is needed at all. Can a set of specialist search applications be used, or can the search capability included in off-the-shelf applications or more specialized business solutions (for business intelligence, ERP or CRM) be used?

Search isn’t a whole solution—evolving toward search-driven applications. If enterprise search solutions attempt to accommodate all the various types of searches people need to conduct, simplicity and overall usability can be severely compromised. In most cases, specialized applications do a better job for particular tasks than a
specially developed deployment of an enterprise search platform. For example, if users want to handle all the intricacies
inherent in conducting detailed e-discovery activities, they will typically need to add a lot of functionality to any existing enterprise search solution. Similarly, how does email archiving functionality—particularly in the context of compliance—get integrated? Rebuilding all such specialist functionality on top of an enterprise search platform can take a long time and be less cost-effective than deploying a specialist solution.

After all, in many cases, search is only part of a required process. A need exists for capabilities to capture information, convert it to something searchable (OCR, speech recognition, video analysis), store it on one platform and disclose it, all of which are outside of the realistic reach of enterprise search. Of course, enterprise search platforms often provide basic components of these capabilities, but that doesn’t mean that users will have a comprehensive, full-blown application for, as an example, e-discovery and e-disclosure, SOX-compliant contract management or historical archiving.

Bottom line: Given the diversity of organizational tasks, coupled with market demand for better information sharing and connectivity among business units, claims that a “one-size-fits-all” enterprise search solution is the best or only choice for most organizations is simply not true.

Survival, an issue of functional diversity and awareness. Understanding why specialized search solutions are desirable over the deployment of large enterprise search platforms is often easier to understand if viewed from a technical standpoint. Consider the following instances:

• Relevance ranking. When search engines index the Internet, they typically only want to find the best sites, or the most popular ones, as opposed to finding all potentially relevant sites. Enterprise search platforms that are derived from these Internet search engines focus on 100% precision and use deeply embedded relevance ranking algorithms, which can be based upon, for instance, popularity ranking (in traffic and in links).
  In many legal, intelligence or investigative environments, though, “popular” hits aren’t always desired; for example, consider the complex financial transactions in a money laundering case. Users in this instance cannot miss a document that contains a “smoking gun” or provides critical details that could influence the case. All potentially relevant hits must be found, meaning 100% recall is needed, which is difficult to achieve if an engine is optimized for precision across an implementation.
• Fast hit highlighting and hit navigation. For performance reasons, Web-search technology does not store word co-occurrences, hit information, optimizations for hit navigation, text mining information or intelligent structures for fuzzy search. These engines just need to quickly index as much data as possible. Auxiliary functionality for viewing and navigation is calculated at viewing time, which works fine for Web pages because most of them are relatively small.

However, search capabilities for law enforcement officials, (corporate) lawyers and auditors are compromised by Web-search technology. Fast navigation and advanced hit highlighting to determine whether a document is relevant or not are essential search features. If information on work and hit occurrences isn’t stored in the search index, large-document search, hit highlighting and hit navigation will be very slow, as will be general document transfer. Because these professionals obtain extensive amounts of confiscated materials, they are not always in a position to divide a binder into logical documents. In many cases, a single document could consist of an entire binder or even a roomful of material. Similarly, many legal contracts, historical archives or research reports only exist on paper and contain 100+ pages.

Many other examples can be given. The fundamental differences that exist among data structures and algorithms, as well as overall functional approaches, make some enterprise search platforms completely unsuitable for other applications. Finding