What Are You Looking For? An Overture to "Intelligent Search"

The following is an overture article, by Andy Moore, for "Best Practices in Intelligent Search," a KMWorld white paper available for download.

"Seek and ye shall find..." Yeah, right. Maybe about half the time, if you're lucky.

It's true that most searches end in failure. And what's even more troubling is that the person making the search may never know that it was a failure. Let's say: A hypothetical knowledge worker enters a search query. May be a good one; may be a vague one. Doesn't (and shouldn't) matter. Then that person gets a results page. There is a long list of possible answers to his query, and there might even be some interesting stuff on it.

But the nagging doubt comes from "not knowing what you don't know," as Donald Rumsfeld once put it. MindMetre did a serious research report. They surveyed approximately 2,000 business directors and managers from all over the world. Their key conclusion? More than half (52%) said they "cannot find the information they are seeking using their own organization's enterprise search facility," within a reasonable amount of time.

There's plenty of blame for this to go around. The first one most people cite is: "information overload." I'm so sick of hearing about information overload, because I don't believe it for a second. Hear me out... There are more than 53 million individual pieces of information (books, articles, reports, videos, recordings) in the New York Public Library system. That's an overload. And yet, I guarantee you I can find any single thing I want within "a reasonable amount of time."

It's not information overload that's vexing those 2,000 business managers; it's "information under-managed" that's the problem. There can NEVER be too much information. That's absolutely contrary to the basic principles of knowledge work.

We set out to discuss "enterprise search" and "intelligent search" in this White Paper. And that we shall. But let's get something straight: Enterprise search is not what you think it is. It is not a single unified piece of software that can magically scour through the dozens of business applications that contain that piece of information our hypothetical guy was looking for. And, much less, it is not a single tool that can seek, discover and deliver an important piece of information from the hundreds and thousands of repositories from which it may emerge. In this day of thinly sliced Web content stores, SharePoint farms, mobile storage devices and cloud repositories, the hope of controlling the location and nature of a piece of information has been quite dashed.

Some people say it's "like finding a needle in a haystack." Heck, that's easy. The hard part is finding a needle in a needlestack.

What Do You Expect?

There's a certain aspect of "user expectation" that has emerged in enterprise search, and there really is little justification for it. A lot of times this is referred to as the "Googleization" factor. This is the flawed notion that corporate search should work the same way Google does. Type in a query, get a results list with "the best" answers at the top. Nothing could be further from the truth when it comes to enterprise search, and we'll touch on that later, but for now let's return to that MindMetre survey. They found a couple interesting things in this regard. One, there seems to be a cultural bias: "Respondents from the US have the highest expectations when it comes to search, with 71% expecting to achieve the desired result within two minutes," says the report. But it then goes on to add: "They (US respondents) also demonstrate the best performance for internal search facilities, with 53% saying that they can find what they want in that time."

Really, 53% is "the best performance?" Wow. There must be something about American optimism that allows a success rate of only a little over half the time to be considered "the best." So be it.

But it shouldn't be easily dismissed as "just the way it is; live with it." There are hidden as well as direct costs in play as well: "Non-productive" information work, such as reformatting documents or reentering documents into computers, consumes more than $1.5 trillion in US salaries. Solutions must be brought to the table to improve those ghastly numbers.

Such as? The other participants in the White Paper certainly are a good place to start. But in general, some of the key components of a reliable and effective enterprise search solution would be:

• Accurate concept searching;
• Contextual navigation; and
• Precise automated classification.

And those activities can be deployed through the effective application of:

• Taxonomies and ontologies;
• Automatic classification;
• Better metadata;
• Better navigation; and
• A user experience that is intuitive and frictionless.

The Way Out

How did we get into this mess? Well, considering the nature of information creation and delivery, it was unavoidable. All enterprises both generate andaccumulate huge amounts of data in all sorts of formats. Much of it is textual, such as word-processing documents, HTML pages, email, social sites, newsgroups and forums, etc. But complicating matters, there's also numeric data (referred to as "structured data") often kept in relational databases.

Worrying about whether data is structured or unstructured has to stop. The location and format of the information a knowledge worker needs is irrelevant to him. The goal is to hide the sausage-making from the worker, and get him to the kielbasa as fast and easily as possible. But we've artificially, I think, made that impossible. Information assets are information assets, whether they're content or data. But the tools available historically haven't brought those two sources together; search engines have targeted documents and other forms of content while business intelligence tools have queried data. And there's a "stubborn reality," as John Mancini, president of AIIM, put it in his great blog, Digital Landfill, "that unstructured information is the red-headed stepchild of the equation-and the source of so much untapped value and intelligence in organizations."

Back to the "Googleization" issue, and the unmet expectations of many enterprise search users. Web search and enterprise search are nothing alike. Let me repeat that: Web search and enterprise search are nothing alike. Web search engines deal with information at huge scales; measured both by the amount of data and by the number of users and queries.

Enterprise engines, on the other hand, need to collect, index and rank content from more varied data sources than Web search engines, and need to do a better job of limiting the result set sizes. Enterprise users often require advanced search features, as mentioned above, to accommodate scenarios that are rarely encountered on the Web. Also, in enterprise search, users are often not allowed to see all documents. This challenge doesn't exist on the Web, where search engines only index data that is open to the general public.

So, it's different.

 

The entire "Best Practices in Intelligent Search" white paper is available for download.