The Purpose-Driven Search Life

Writing about enterprise search is not the cakewalk it used to be. With customers demanding more business value, and vendors responding by becoming more "purpose-driven" and specialized, the search market has fragmented into a series of business applications that only opaquely rely on "the search engine" to accomplish their tasks.

I often call it "the technology arc." At first, all you have to do is say "enterprise search," and you have the attention of the users and the investors. Then after a while, you have to ask, "What can this new technology do for me?" Then after a while and the shine is off the lily (or however that expression goes), you need to ask, "Where is the business process improvement I should expect for my (fill in the blank) financial services/manufacturing/healthcare/gardening shop... it no longer is about the technology underpinnings. It's about the work you need to get done."

No better example is enterprise search over the last few years. And no better interview could have fallen into my lap than the opportunity to speak with Jerome Levadoux, senior vice president for products at (what is now officially known as) HP Autonomy.

Now, here's what I can tell you about that. Not much. The little bit I know, anyway. Autonomy was once the powerhouse software license godhead for enterprise search. Still is an impressive player in the market, for certain. But in the meantime, many smaller startups, many of them based on open-source software and thus carrying a "hipper than thou" aesthetic, rolled onto the scene. Autonomy (like FAST Search) found itself members of a much larger, much more complex marketplace. And, being as respected and enduring as they are, they were natural targets for acquisition.

Which they became. I will not dwell here on any of the fall-out regarding that acquisition. It remains to smarter people to sort that out. In fact, when Jerome and I talked, the subject didn't even come up.

But what DID come up was the vastly and rapidly changing role that search plays in the information management landscape. New markets opening up... new companies experiencing happy upticks in their business... new entries into the space (as I said before...)—I wanted to know from Jerome which of these things seemed to matter most to him?

"The enterprise search market is entering its third wave," he began. And once Jerome begins, it's best just to lean in and listen. "The market was created in the early 2000s, and was driven by the adoption of portals and the arrival of more and more sophisticated websites. You need search engines for those things! The second wave happened around information compliance and e-discovery... there was a recession around 2007, and compliance was considered a great way (by the vendors) to drive business. There was a need that could be matched with a budget," he explained.

"We are now at the beginning of the third wave, and it is being driven by two things: One is big data. I know it's a buzzword, but every buzzword reflects an underlying truth. And the underlying trend to big data is that people are trying now to analyze news kinds of data in novel ways. And the old techniques don't work to get real insights into data," he said.

"The second truth—and we're really just at the beginning of this—is the appearance of mobile, social and cloud. Information is becoming much more abundant on one hand, but also much more siloed, and a lot harder to find, and that's causing a lot of headache in terms of productivity. That's driving a whole new need to integrate those silos and allow people to get value from the information. That's a great new role for search.

"The thing is this," he continued. "As an information worker, I have my usual SharePoint and fileshares and content management systems, etc., but on top of that I have SalesForce and WorkDay and DropBox and Box.net and GoogleDocs... it's a flood of silos! I can't connect those sources of information. The same is true of all my social media and collaboration apps. Yammer and LinkedIn... I'm using all these things at once, trying to extract knowledge from this very siloed world. That is the next opportunity for search."

He continued, and I'm still leaned in. "And on the subject of big data... big data is such a big deal because there are novel forms of information that people want to analyze. In the ‘old days' (he means like 10 years ago, tops, I'm thinking) people would look primarily at financial data in rows and columns, and use BI to draw pretty charts, and maybe use some basic level of analytics to gain value from the information. Today, for many different reasons, and mainly because of the proliferation of non-database data, people now feel the need to apply analytics to things like social data, data on the Web, input from customers on their websites, and things like that.  It is coming in totally unstructured, in random formats and random languages, sometimes in slang..."

He thought for a minute... "Oh, and then there's video," he added. "Whether it's security footage or cameras from surveillance drones, the amount of video is beyond anybody's ability to process. Universities are streaming their courses, and they need that material to be usable (and thus searchable). Same goes for images and voice. So all these forms of data that nobody bothered to analyze before is suddenly very important to look at, analyze and unlock the value hidden within.

It makes former database operations pale in comparison. "For example, ERP databases are not really that big. After they're compressed, they're usually less than a terabyte," he claims. (Jerome comes from a background at SAP, so I take his word on this.) "I don't think there's any company in the world that has a petabyte of ERP data. So the real ‘big' data these days are things like sensor data from machines, or click-stream data from the Web at large. Then there's also what I call ‘human data'—text, social feeds, etc. That's where big data really comes into play. The irony is that the online repositories are actually easier for knowledge workers to access than many of the legacy tools that were never easy to access!" And in this I agree. BI systems and financial analytic tools have always been cumbersome and "non-democratic." And that was a problem. But now we have the opposite problem—information is now TOO damn democratic. You can't swing a cat in the average organization without hitting a "content provider" of one kind or another.

A Brave New Market

Here's how Jerome describes it in his great article on the following pages:

"Today's workers are increasingly on-the-go, embracing the newest mobile technologies to stay connected and productive, as they continue to fuel the migration of content to the cloud. In addition to driving the great shift of data to the cloud, there is also fragmentation of knowledge among multiple systems and repositories. Information today has many addresses. It lives in email, on mobile devices, in Dropbox and Evernote, and in whatever applications people may choose to install. The consumerization of content has also meant that devices and applications are used for both professional and personal purposes," he writes.

"An important distinction to remember about information growth is that it is not just about documents. We are becoming a multimedia-focused world watching and listening more and reading less. Many meetings are now conducted remotely over video, and training sessions are often recorded. This means your search technology is required to handle these new and pervasive content formats. The number of files, images, records and other digital information is predicted to grow by a factor of 67 from 2009 to 2020, with corresponding growth of IT professionals globally by a wimpy 1.4."