What You Know...and What You Don't
A Brief Foray Into Text Analytics As We Know It

Whew. That’s lot a to dig through. But I had to ask Jan whether this technology was ready for primetime.

Andy Moore: By anybody’s measure, text analytics isn’t exactly an everyday business tool yet. Question: So far, which has had more impact on the acceptance of text analytic tools—the pace of development of the technology? Or the true business demand? In other words, of these two drivers, has the development of viable text analytics technol-ogy lagged behind? Or has business demand been slow to have an impact? Either? Both?

Jan Scholtes: "Text analytics is going through a typical ‘crossing the chasm’ technology marketing lifecycle, as described in the famous book by Geoffrey Moore. Initially new technology is adopted by early adopters, primarily tech wizards who don’t mind using peer solutions that crash once in awhile and that need a lot of tuning. But at a certain moment, the technology gets accepted and moves to a majority of buyers. These users want a full solution that works as required and is also a market leader," explained Jan.

"Because the later groups of buyers are more managers and other non-technical people, they do not accept the references from the early adopter techies. They also insist on market leaders, which leads to a chicken and egg situation, often referred to as the ‘chasm’ that needs to be crossed in selling and marketing technology. Text analytics has crossed that chasm and it now needs to be part of a full platform that solves a business problem in total. So, it needs to be sold with more than just the text mining technology. Now it is business demand driving the sales, but—unfortunately—buyers want more than just text analytics," he said.

Pondering this, I felt I needed some concrete examples I could wrap my brain around.

AM: What other factors are now making text analytics move closer to mass acceptance? You can discuss anything from e-discovery to sentiment analysis to information overload... just provide some examples, please.

JS: "Of course, the 9/11 events really bootstrapped the text-mining industry. Later, law enforcement, legal, medical, life sciences, production, service and customer opinions followed. Now, we expect e-discovery, compliance and other fall-outs of the credit crisis will turbocharge text mining and text analytics even more," Jan said.

"But," he continued, "for full acceptance we’ll need:

• Understanding of the technology and awareness of the technical possibilities by the masses;
• Success of usage in well-known and successful cases or environments such as within social networks, and—for a specific example—the UN War Crimes Tribunals; and
• Integration in business platforms."

Getting to the Crux
It’s probably unavoidable to ignore current economic realities when discussing a cutting-edge technology such as text analytics. After all, we’ve lived without it for a long time, and given the choice between making payroll and developing new technology platforms, the decision is pretty easy. So I asked Jan what he thought it would take to reach the "tipping point" (as long as we’re name-checking business-management bestsellers) for text analytics to become an everyday reality. His first three responses were pretty predictable:

• Lower price;
• Easier to use; and
• Better integration with existing platforms and applications.

But I pressed harder for information regarding the adoption of text analytics. One element in the mix that he hadn’t mentioned was cost.

AM: Is text analytics a costly technology to apply? What are the cost factors? How much "human" resource is required, in either terms of application development; on-going content management (tagging, etc); interpretation after the fact...?

JS: "Pricing will go down very rapidly in the next months. There is too much competition in this field. Typically, when there are more than three leading players (as is the case in text analytics) in a technology, margins will erode VERY quickly."

That’s good news for consumers, I suppose, but is there a silver lining for the developers of these tools? I wondered. "Through better integration into existing platforms, the technical challenges will go away. Deploying text-mining no longer as a general technology framework, but by deploying it within the context of an application such as intelligence, e-discovery, customer opinions, sentiment mining, product quality or clinical research, it will be easier to use and there is less requirement to tune the systems," Jan said.

And theoretically, it follows, text analysis will be easier to cost-justify if there’s an undeniable business goal at which it is targeted.

AM: One of the analyst groups conducted a market survey in the summer of 2007, but already says it’s obsolete; the market has "significantly matured" since then, they contend. In what ways, do YOU think, the market has changed/matured in these short two years?

JS: "Text analytics technology is well known. Everybody applies the same technological principles. There is even a variety of open-source software available that can do a very good job for the technical specialists. There are few secrets left," said Jan.

"The difference in quality is in the support and knowledge of different languages and different application areas, and less and less in the technology. In text analytics the only good data is more data, although many vendors still focus on the main languages such as English, Spanish and some typical intelligence languages and forget about 190 other languages spoken."

AM: Have you seen any decline or slow-down in projects due to recent economic pressures?

JS: "Not really. There may be saturation in the intelligence space. But other typical text analytics markets—such as life sciences—do not suffer from the crisis that much." The most growth, said Jan, for text analytics, is currently in segments where "the most money can be made," namely e-discovery and compliance. Further down the road, Jan expects to see increasing activity in consumer applications and social networks. "That’s where we will see the technology grow," he predicted.