Text analytics finds dynamic growth in e-discovery and customer feedback

One specific insight companies are seeking is a customer's propensity for purchasing a product or service. "It's possible to do deep text analytics and then use the results of those analyses to create predictive models of likely customer behaviors," Kobielus says. "There may be a correlation between a certain price and propensity to buy, and the company can then develop a campaign to validate the propensity model and possibly identify areas where the models needs tweaking." That process, often called "next best offer" modeling, helps companies invest their marketing resources more wisely.

Text analysis meets business analytics

Text analyses also can be combined with quantitative data from business analytics systems. "We are seeing a lot of traction in these types of analyses," says Dan Lahl, senior director of product marketing at Sybase. "For example, if discussions in customer support forums indicate that a product is hard to use, it's helpful to know if it's your top customers who are commenting." However, identifying the top customers depends on certain criteria such as buying patterns and historical revenue amounts that are stored in structured databases.

Sybase IQ is a business analytics solution that indexes and stores unstructured data in the same system as structured data. Therefore, queries that combine both can be run, analyzed and correlated in the same system by Sybase IQ. For example, analysis of the customer comments, which originated as unstructured data, would be coded and stored in the database. Sybase IQ analyzes the coded comments relative to other factors, such as the number of years the customer has used the product.

Processing the raw text data is necessary for Sybase IQ to carry out its analyses. Sybase has partnered with ISYS Search Software to parse text for text indexing, and with Kapow Software to extract and transform data from the Web so that it can be consumed by Sybase IQ. "The processing power of Sybase IQ allows us to analyze data in real time so our customers can react quickly to changing circumstances that may be reflected in different types of data," Lahl explains. "Analysis of structured and unstructured data in combination offers insights that neither can provide alone."

Text analytics takes on bribery

In addition to providing more sophisticated e-discovery capability, text analytics technology is being used proactively by companies to search their records for vulnerabilities. "Text analytics can be useful at several stages," says Howard Sklar, senior counsel at Recommind. Recommind offers a family of products for information access, governance, e-discovery and compliance based on its Content Optimized Relevancy Engine (CORE) technology.

Bribery scandals in the mid-1970s prompted the passage of the Foreign Corruption Practices Act (FCPA) of 1977, which outlawed payments to foreign officials by U.S. companies in order to obtain business. In fact, bribery was quite widespread. A survey conducted by the Securities and Exchange Commission (SEC) indicated that many public companies, including about 20 percent of Fortune 500 companies, bribed foreign officials. In Europe, those payments were not only allowed but were tax-deductible as "ordinary and reasonable" business expenses until anti-bribery laws were passed there in the late 1990s.

The incentives for companies to put an end to those payments are now many and varied. Around 2005, the SEC began prosecuting cases more aggressively, and recently the Department of Justice (DOJ) began prosecuting individuals rather than just corporations. Financial penalties include not only fines, but also legal costs that may far exceed the fines. Shareholders can also sue the corporations, based on violations of the portion of the FCPA that requires accurate financial records. "General counsels at the firms began going to their compliance officers and saying, ‘What are we doing about this?'" Sklar explains. "That began a move toward a more proactive approach."

Through the use of text analytics combined with analysis of business intelligence (BI) data, companies can analyze information centered on high-risk employees and high-risk geographic areas and look for key indicators of problems. Detecting issues early is far better than having them discovered through an audit. "Over the next few years companies doing business globally will face increasing needs for risk management," Sklar says, "and text analysis is part of the solution."