In Search of Search
What Does the Marketplace See When it Looks in the Mirror?

I usually make it a rule for these interviews: No inside baseball. I'm normally much more interested in the so-called 50,000-foot view...how customers perceive their business needs and how these various KMWorld-y technologies can address them. Or how buyers organize themselves into multi-disciplinary teams to spearhead enterprise technology purchases. Or even how CEOs feel about risk, like taking the perp walk on the 6 O'clock news.

But I almost never ask vendors what it's like to be a vendor. Enterprise search, it turns out, is different. It's been my observation that enterprise search has behaved like a micro-economic model of the typical technology adoption and maturity cycle...except, like, on crack.

So I tested my incredibly mature theory on a group of enterprise-search vendors in a series of conversations in February, 2006. Here's what I learned:

The Cast of Characters

Jeff Dirks
President and CEO
Schemalogic (www.schemalogic.com)

Bob MacDonald
Chief Marketing Officer
InQuira (www.inquira.com)

Andy Moore
Editorial Director
KMWorld Specialty Publishing Group

Steve Papa
Founder and Chairman
Endeca (www.endeca.com)

Johannes Scholtes
President and CEO
ZyLAB (www.zylab.com)

Silvija Seres
Vice President of Strategic Market Development
Fast Search & Transfer (www.fastsearch.com)

Chris Weitz
Managing Director
BearingPoint (www.bearingpoint.com)

Andy Moore: So, am I right? Is enterprise search on crack, or what? (I'm paraphrasing a little here, for the sake of brevity.)

Bob MacDonald: When a new technology—any technology—first hits, it has a singular definition, and people talk about it in a singular way. Search was like that, too. Then products start emerging, both expanding and breaking apart that original perception. Search started with just such a simple concept: You need to access the information you have. Then product vendors created solutions that focused on different aspects. Now—all of a sudden—you have those focused on e-commerce, those that are for legal or security requirements, or those for customer-facing applications in call center or service environments.

Jeff Dirks: It's a classic boom-bust-boom-again cycle, where the front-runners get out early and grab the IT budgets of the Global 5000, followed by consolidation and a narrowing of the leaders in the particular segment. I think that's what's happening in search.
But I also think it's a much bigger addressable market than some of the analysts. We're on the edge of a tectonic shift in the way big companies, especially, think about information management, and search is at the center of this.

Johannes Scholtes: Our customers have come to understand search much better than just five years ago. "Enterprise search" has changed, and has become a series of solutions optimized for specific customers' needs. Some of us vendors are better in certain applications than in others...once you sell to a few customers in that market, you gain references and it becomes easier to sell in that market.

Chris Weitz: The adoption cycles among the customers and maturity cycles of the vendors are more accelerated due to the availability of Internet technology and rapid software development. Search will be assumed to be a platform going forward, not just a weird thing used by the IT department.

Silvija Seres: We have a single platform, and on top of that platform we create several, focused, search-derivative applications. At the heart of these specialized applications, there is a shared set of core capabilities—a scalable index; the flexible indexing of content and queries. Then you add the linguistics and media add-ons that allow you to create focused products.

Steve Papa: In the early days of search, there wasn't much barrier to entry; a couple guys in a university could create a keyword search product. It wasn't really any different than what you could do from a DOS prompt. Now the hard work and innovation are being done.

Silvija: We are now in this incredibly sweet spot in the market. It's going so fast that we are all running to keep up. There's an explosion in the volume of data, of course, plus there's an explosion in the complexity of the data types and media. Users are becoming more demanding and sophisticated, they know what they want to do, and they want to do it on a huge scale.
So we're going deeper into places where search is already in use. But we're also going broader, into more markets. The technology analysts see it this way, too. IDC is now adding structured data processes, such as business intelligence, into their analysis of this marketspace. So instead of predicting it as a $700 million market, with this broader view they now think of it as an aggregate $25 billion market.

Andy: They put a man on the moon 37 years ago. But it's only the last few years that we've had decent search. What gives?

Johannes: People have been trying to optimize search for more than 50 years now, with mostly disappointing results until recently. The more technology that got thrown at it, the less efficiently it seemed to work.

Bob: I am amazed at how much of a blend of technology and intellectual theory there is in search. Back in the day of database development, everyone pretty much knew how to do it. There was an industry blueprint. You could pull together the people with computer skillsets and pretty much get into that business. But you cannot be effective in the search space without a high understanding of computational linguistics and semantic analysis, as well as computer science capabilities.

Johannes: On the Internet, you want to find the single answer that solves your problem...it's what we call precision. But in auditing, discovery, disclosure or investigations, you want to find everything. The problem is, once you get to that 100% level of recall, your precision goes down the drain. So for those high-recall applications, you need to apply linguistic technology or visualization techniques to reach the level of precision AND recall you need.
Due to this, search is changing many markets. Beyond just the database market—where databases are becoming too large to be effectively managed by the database-providers' tools—other markets such as advertising, auditing, internal discovery, law enforcement, legal...search is at the heart of major revolutions in all those markets.

Bob: Customers have come to realize that keyword search does not give them the specificity they need to respond to, say, a consumer's question to the call center. They need the computational linguistics necessary to index their content for meaning, allowing them to pull an excerpt out of relevant document from a thousand such documents, rather than just point to the document. That is a huge productivity jump.

Silvija: Search technology is built for huge volumes of complex data. Users who don't necessarily know precisely how to "ask the right questions" are still being served. We're just where we need to be.
Maybe it's not an information overload; maybe it's information underload. If you think about all the nature around us, we humans have learned how to handle that volume and complexity. That is where we are getting now with our computers. If we provide them with enough of the right information, with all the necessary context, and the right tools to filter and abstract, we will have a very intelligent tool. We're just about there; it's incredibly exciting.

Bob: The greater power of computing, and the lowering of those costs, have made it feasible for businesses to apply these tools. Our stuff came out of an 8 1/2 year study at MIT. They weren't trying to implement it for business; they were just trying to find a way—any way—to index and retrieve information with a completely automated system. The computing power needed to implement this stuff had to get to a price point where storing and retrieving it made sense. Now we're there.

Andy: Let's talk about the elephant in the corner—Google...

Chris: Search is badly defined and badly understood across the board. If you talk to business leaders and say "search," they either draw a blank, or they say "Google" (It should be noted that this conversation took place the day following BearingPoint's and Google's announcement of a major alliance.)
If you're eight or 80 years old, you now know what "Search," with a capital S, means. And that's because of Google. But enterprise search is currently the domain of the nerd in the IT department who is not making any business decisions. That's changing, but it's true.

Steve: It depends on the company, and who you're talking to. If they're using it to serve their customers, they're taking it very seriously. The head of e-commerce at a large on-line retailer sees search as mission-critical.

Chris: Things migrate from features, to applications, to strategic levers, and we're not anywhere close to the far right of that curve. Yet. We're a huge systems integrator. We follow the money, and you'd have to use a magnifying glass to find the money in search. But we're, to use an old adage, skating where the puck is going. I am clear about the fact that's changing. That's why we've invested in this search practice.

Jeff: It's true that Google has captured the hearts and minds of the public. But I think of Google as a fairly straightforward search algorithm and a good way to monetize advertising in a Web setting. There couldn't be a larger chasm of difference between the way the information stack is organized in companies, and the way Google works. We think of it as information-enabling key processes; search and discovery is a key part of that. But the use case might be global brand management across 1,000 content silos, or global customer self-help. So much is driven by the flow of information, where 15-20 years ago it would have been just the physical assets of the company.

Johannes: Everyone understands how the Google "popularity algorithm" works. If you search for the best hotel in New York, you'll get an answer—the one that most people THINK is the best website for the best hotel. But in law enforcement, you don't necessarily want the most popular website...you want, perhaps, all of them or the ones that are most relevant.

Chris: For most companies, pretty good is good enough. A lot of niche search vendors have overshot the market, and they're doing more than people need.

Andy: Why is search such a big deal now?

Bob: Whether it's an internal or external application, all the factors are in place to drive growth for the search market. For one, there's not a single major corporation that doesn't want to leverage Internet-type technologies for internal use to distribute information. And then on the external, customer-facing side, we're going through a next-generation of corporate websites, through which the businesses are trying to drive interactions more effectively. And the most effective way to do both is through the search paradigm.

Steve: When people invest in new technology, they are increasing additional relationships among data. The irony is, the more you invest, the less accessible those relationships become. It's not really about search, it's about those relationships. The other challenge for search is open source stuff, such as Lucene. It's even harder now for those two guys to compete with something that's free!

Jeff: It's not uncommon for us to hear from customers—the ultimate customers—who can't find what they want on the website, and so they go to the call center and say "I didn't want to call you, but I had no choice. Why don't you save yourself some money, and me some time, and fix your site?" Steve: The more people who are aware, the better. We find, though, that as people find the potential locked in the information assets, they then start to look for how to do a better job. With search high up on the agenda, higher people in the organization have gotten interested.

Silvija: It's to the point where we are helping write business plans around search. We help companies understand how their whole strategy changes because of search. We help them operate in a new way.