Dr. Strangelove, or,
How I Learned to Stop Worrying and Love BI

This begins to sound as though we're advocating that enterprise search replace traditional business intelligence tools. That's not quite true. While search plays an increasingly important role in the functionality of BI, there are still certain functions, such as statistical analysis, visualization, regression analysis, that search does not (currently) provide. Those functions are still the bread and butter of the Business Objects and the Cognoses of the world.

The opportunity is for search and BI to partner together, and leverage each others' various strengths and functionality. For example, search, because it understands text AND data, can be used to determine the frequency and relationships between certain phrases, words or data strings. Then you export that information to a BI tool to create visualization models and "what-if" analyses of the resulting information. This information could then be used in a business context to, say, allocate resources better, or to identify revenue opportunities, or to locate good places to drill for oil. "Integrating the internal with the external, integrating structured with unstructured, doing scalable text mining, then exporting the results to visualization and reporting packages is the way the world is going to go," predicts Davor.

And Then There's the Money
It's becoming more and more clear that the intention of the enterprise search vendors is to become the data integration layer, doing the heavy lifting previously provided only by large-scale application integration and data aggregation efforts. If this is the case, the savings available to an organization, derived from avoiding a significant portion of its EAI work, can be enormous. I always fantasize that, if all the money spent on integration and data aggregation simply in order to create BI repositories could be retrieved and put in the bank, we could feed a starving nation!

There are so many ways that a search/BI partnership could save money, it's practically self-evident. But, of course, CFOs don't want to hear "self-evident." They want hard ROI models.

Well, one is just plain old efficiency gains. Lots of companies are using search tools to replace portions of their HR department's workload, for instance. And there's a BI payoff, too, because the search tool provides not only standard retrieval for the user at the front-end, but also creates analytics for the business managers in the back-end, too.

Davor has more: "When the majority of the top 20 online Yellow Page providers move to search instead of database, there are two drivers: One is the quality of the search experience; Yellow Pages is all address. It's basically a big database. But with search you can integrate other, more unstructured information." This leads to greater targeted advertising opportunities, integration with maps, etc.

But more to the strictly budgetary issue, there is a great "total cost of ownership" argument to be made. Databases demand expensive servers. IBM and others make a lot of their revenue from companies who are in the "database escalation" cycle I described earlier. But the advantage of switching to a searchbased model is that you can move from highend servers to commodity hardware.

The cost of supporting a large Oracle database can be well over what it costs to implement a search engine that uses low-cost off-the-shelf hard drives. And it's only getting worse. Despite all the talk about unstructured data growing at exponential rates, you should see the database side! Structured data is growing faster than Moore's Law. The number of rows in very large databases has increased by a factor of five over the last 48 months. That means the size of data stores is going up a lot faster than the cost of repositories is coming down (due to improvements in chip design, memory costs, production techniques and all the other Moore's Law stuff).

So—contrary to popular opinion—the cost of storage is actually going up; you need to replace that cost center with something more efficient. If you're maintaining a data warehouse of about a terabyte in size, you could be spending $2 million, $21/2 million. Half of that is ongoing integration and maintenance, and we've already talked about how search reduces that. The other half is proprietary hardware. Depending on the number of queries per second you need to support, you might be able to store that on a bank of Dells, dude, and save almost all that hardware budget! A terabyte of SCSI disks is about $10,000, tops!

Just Asking
Don't get me wrong; search vendors are not foolishly ambitious. They know that they need to partner with—not compete against—the big BI and ERP vendors. There is, and always will be (maybe) certain functions that remain the domain of the big BI guys. There's a similar corollary, by the way, with content and document management vendors: CM and DM also have certain functions that search does not provide, such as workflow, revision control and collaboration.

Clearly, the search kernel beneath all these functions is fundamental, but nobody is suggesting that enterprise search can take over the world.

Are they?