The next generation of search

By Alan Pelz-Sharpe “Presumably man's spirit should be elevated if he can better review his shady past and analyze more completely and objectively his present problems . . . if he can reacquire the privilege of forgetting the manifold things he does not need to have immediately at hand, with some assurance that he can find them again if they prove important.” —Vannevar Bush, “As We May Think,” 1945

As Web and Internet technologies move us closer to Vannevar Bush’s vision of a universal repository of knowledge, search technology has become a crucial, if often undervalued, part of today’s connected world.

Search engines helped colonize the Web for the general users through sites such as AltaVista, Excite and Yahoo. A search engine is also the first enhancement to be made to most intranets: Search vendors were the first to offer knowledge management products, and search technology is one of the core elements of any enterprise portal. That ubiquity means that search vendors—in their many forms—will generate revenues of over $1.2 billion in 2001 [discussed in more detail in Ovum’s report “Next-Generation Search: Building The Smart Portal”].

But the limitations of “traditional” search methods when set against the sheer size and diversity of sources and the variation and uncertainty in user requirements are well recognized. They have spurred wide-scale interest (and significant investment) in technologies and products that are bringing new levels of intelligence, order and personalization to the search process—what Ovum defines as next-generation search.

While the role of search in a Web-based environment may be well recognized, search also carries an impression of being a technology of last resort. Vendors of next-generation search technology are trying to change that impression by increasing the intelligence--and more importantly the utility—of search technology.

The goal of next-generation search is to provide guides and filters in a world of information glut. That is being achieved through innovations such as conceptual search, document categorization and user profiling. But there is great confusion about how the various technologies fit and also about the size of the market opportunity they present. In the enterprise market, search technology is no longer being seen as a minor component within the corporate architecture. Great opportunities exist for search vendors to establish a strong position in the rapidly evolving market for infrastructure tools for the management of unstructured information. Knowledge management, content management and enterprise portals are all booming markets—and search technology is key to each of them.

And on the horizon are further disruptions, as Internet and Web technologies continue to develop at astonishing rates. Peer-to-peer computing and Tim Berners-Lee’s vision of a Semantic Web both offer very different visions of how search might evolve in the next decade.

Next-generation search technology is about providing more intelligent and more proactive tools to allow users to take control of the vast amounts of information that the networked world makes available--to allow them to swim rather than drown in this sea of information.

Next-generation search is not a precise category. It overlaps with terms such as information retrieval, knowledge discovery and intelligent search, and it encompasses a wide variety of technologies and products.

In knowledge management or intranet applications, the primary focus is on accessing internal resources (corporate file systems, databases, document repositories, for example) and external resources for internal purposes (for example, newsfeeds or Web sites). With many knowledge management initiatives now moving beyond the firewall—so that internal corporate information is made accessible to partners or customers, companies are looking for greater integration between their internal and their online infrastructure—and that includes the search technology they use.

Next-generation search vendors are competing for a slice of an overall market for knowledge management and business intelligence software, which Ovum estimates is worth $15 billion in 2001 (discussed in “Ovum Forecasts: Global Software Markets”). In particular, search vendors are targeting the rapidly expanding enterprise requirement for tools to manage and exploit vast volumes of unstructured information. In that area, their products both complement and compete with vendors of content management, enterprise portal and collaboration software.

The biggest opportunity for “search” vendors in coming years will be to exploit the increasing demand for knowledge and information integration by embedding their technologies even more deeply into the corporate application infrastructure. In the last two years, as many intranets have evolved into enterprise portals, the opportunity open to search vendors has increased, with no decline in sight over the next five years as the Web architecture continues to colonize corporate IT. The evolution of the intranet into more sophisticated applications and greater integration of applications will continue to create new opportunities for next-generation search tools.

Search technology is typically the first enhancement organizations make to their initial intranet development. However, many organizations have now realized that simply adding search capabilities is not enough to truly exploit the benefits of Web and Internet platforms for internal and external information sharing. The rapid growth of both the content management and the enterprise portal markets reflects the urgency with which organizations are trying to bring greater order to their sites and their content. Search vendors have benefited from this through OEM relationships with content management and portal vendors, but also through reinventing themselves as portal vendors. But the rise of the portal market has led to more than rebranding and relabeling, it has also driven significant technology and product developments, such as:

• categorization and taxonomy building tools, ;

• personalization services,;

• closer integration with other products, and ;

• new packaged products such as “out-of-the-box” portal software.;

The portal market continues to gain momentum, driven by the twin forces of dissatisfaction with current intranet deployments and recognition of the potential of the Web for information delivery and knowledge sharing. As the Web platform becomes critical to many more core business processes, information infrastructure technologies must be more closely integrated with business applications. The move of application vendors such as SAP and PeopleSoftinto the portal market is one example of that. This means that there will be greater emphasis on integration capabilities for search technology. Search vendors are increasingly positioning themselves as infrastructure players—hoping to establish themselves as a core component of enterprise IT architectures.

To establish leadership of the next phase in the evolution of the search market, a vendor must offer a more diverse set of services and close integration with application and infrastructure vendors. The aim must be to establish a role as an enterprise information or knowledge broker—mediating between diverse users, diverse sources and diverse delivery mechanisms.

Over the next couple of years, the enterprise search market will continue to grow, albeit at a slower rate than in recent years. In response to user demand and to provide technical differentiation, vendors will begin to introduce advanced technologies such as concept-based search, auto-classification, user profiling and support for multimedia and voice.

That will be followed by a period of consolidation, with the emergence of one or two key players in the overall knowledge management and information infrastructure market. The hype surrounding peer-to-peer search today, will become reality both through the availability of standalone products and extensions to existing search solutions, while the promised benefits of metadata standards will begin to materialize as XML- and RDF-based initiatives reach critical mass.

Beyond 2005, next-generation search capabilities will become mainstream technology, embedded in a wide range of applications with a consequent blurring of the distinction between different knowledge management/intranet/extranet technologies and tools.

Vendor implications

Next-generation search is becoming a core requirement for organizations, in both a knowledge management and e-business context, and will drive revenue growth from $525 million in 2001 to just over $1.5 billion in 2005. Search vendors must effectively position themselves as a key part of the required solutions, both as a tactical fix and as a strategic part of the information infrastructure, if they are to realize the potential of the opportunity.

Success will depend on embedding advanced search or discovery services within the corporate architecture, through close integration with core applications and processes. Combining unstructured and structured search, with support for multimedia, will become increasingly important, while the evolving peer-to-peer market must be monitored, for competitive threats and user requirements. Although the Web search market does not represent a significant competitive threat, the potential synergies resulting from alliances should not be ignored.

For the larger and more aggressive players in the market, the ultimate goal is to establish a role as the central knowledge broker within the enterprise, providing users with multiple search capabilities that can adequately deal with any request, understand its purpose, execute against any relevant resources, and display in the appropriate format. Smaller players and new entrants must decide whether to make a play for that role, or focus on niche opportunities or become a component in the strategies of the larger players.

User considerations

User organizations must carefully consider the demands that will be placed on search technology—in both the short and long term. Why does it need better search? Which areas of the business will gain most?

Appropriate investment in people and processes is essential to get the maximum benefit. Users tend to have low threshold levels when it comes to search, so if they do not know how to search properly, or have false expectations, they will stop using it. Early adopters have discovered that calculating return on investment for search technologies can be difficult, especially in an internal environment where there is no direct revenue connection. The most common method of judging the success of an advanced search implementation is through user feedback which is, by its very nature, not quantitative. However, other methods are available such as calculating the time saved per employee per day. Not forgetting the potential returns to be derived from integrating search technology with other applications.

Search vendors are eager to explain the inferiority of competitors’ search algorithms. It is important to avoid that “algorithm war” and instead to focus on buying a solution capable of meeting business requirements, especially because few of the major vendors rely on a single approach.

One of the great advantages when considering search technology is that it is relatively easy to pilot a product. Initial capabilities and likely fit with your requirements can therefore be seen much more quickly than is possible, for example, with a complex business application or middleware technology.

Alan Pelz-Sharpe is a senior consultant at Ovum (ovum.com), e-mail aps@ovum.com.