Text Analytics and Natural Language Processing: Knowledge Management’s Next Frontier

Text analytics and natural language processing are not new concepts. Most knowledge management professionals have been grappling with these technologies for years. From the KM perspective, these technologies share the same fundamental purpose: They help get the right information to employees at the right time.

Yet many organizations still struggle to deploy these tools effectively. The primary challenge 10 years ago was that the tools themselves were in their infancy. Most businesses didn’t see the value, particularly relative to the risk of adopting a new technology. Today, the challenge is scale. We are creating data at astonishing rates. In fact, some researchers predict that we will be producing 44 zettabytes of data per day by 2025.

This means the typical KM professional faces an unprecedented amount of data to make sense of, and there isn’t a human being on the planet with the capacity to do it. To find signals in the loud noise of big data, you need intelligent machines and intelligent interfaces.

That’s where text analytics and natural language processing (NLP) come in. If your organization is interested in deploying these tools, here are a few best practices to help you get started.

Data Quality Is Key

Before you can begin any other work, you need to assess and optimize the quality of your data.

While data quality is not a new concept, it has quickly become a big problem. Some researchers estimate that up to 96 percent of enterprises report running into trouble with data quality in the process of incorporating AI in their projects. And this isn’t just an IT problem. There are estimates suggesting that bad data costs the US over $3 trillion per year.

If you want to succeed with text analytics and NLP, data quality has to be the starting point. Further, it has to be a business priority. The most successful enterprises have an interdisciplinary team dedicated to this work.

Of course, with both text analytics and NLP, we’re often working with unstructured data. Unfortunately, organizations often believe that because they are working with unstructured data, the quality doesn’t matter. Nothing could be further from the truth. It’s precisely because you’re working with unstructured data that great care needs to be placed into the data you put into and derive from these technologies.

The good news is that we have international standards for data quality assessment that many companies have adopted. And the business case for this work has never been stronger.