Data and our future: too much of a good thing? Not enough? How will we know?
Not very long ago, thoughtful analysts of digital information were bemoaning the problems of too much data. They described “data smog,” “information anxiety,” “data overload,” “infoglut,” and other creative descriptors for a circumstance in which the sheer quantity of data and its continuing rapid growth made it physically impossible for regular humans to get at what they wanted to know.
In today’s AI-exploded world, the chorus has changed. Analysts and businesspeople now loudly call for more data, as much as possible, complaining that cognitive computing and other AI applications need more raw material to build better models and more accurate predictions. There’s a broad consensus that AI tools perform at a higher level when provided with more datapoints from which to develop more (and deeper) networks of connections.
(We should note here that there is interesting research ongoing looking to develop strong networks with fewer datapoints—a rich topic for another day.)
The drive for more
So what exactly happened to shift our perspective from too much data to not enough? The difference maker can be stated in this way: machine learning and neural networks.
A reasonable question would be: Why all the fuss about machine learning? Machine learning is an ancient technique by technology industry standards, and recent successes in putting it on faster hardware and feeding it historically unprecedented amounts of data isn’t changing it in any fundamental way, is it?
Perhaps it is. And the change may presage a fundamentally different relationship between us and the way we view our world.
Understanding the world
Fellow KMWorld columnist and long-time technologist and philosopher David Weinberger builds his latest book—Everyday Chaos: Technology, Complexity, and How We’re Thriving in a New World of Possibility—around this change in the relationship among us, our way of understanding the world, and the new directions into which technology is taking us.
Weinberger sets up the challenge with the following observation:
[N]ow that our new tools, especially machine learning and the internet, are bringing home to us the immensity of the data and information around us, we’re beginning to accept that the true complexity of the world far outstrips the laws and models we devise to explain it. Our newly capacious machines can get closer to understanding it than we can, and they, as machines, don’t really understand anything at all.
Perhaps this is no more than a restatement of one of the oldest “processes” in knowledge management: The more we learn about something, the more we appreciate the many elements that go into it and the complexity inherent in its operations. But Weinberger is grappling with something larger. He asks: How do we humans adjust to new understanding that upends and will eventually replace conventional modes of thinking and analysis—particularly when the new normal will be dependent on machines and our relationship with them?
An analogy might be the development of the theories of health and disease that have driven several millennia of Western medicine. In the Greek world, Hippocrates helped solidify thinking about the importance of vital bodily fluids, described and categorized in the theory of the four humors. In healthy people, the four humors (each a distinct type of bodily fluid) were thought to be in balance. Illness, however, was an indication of an imbalance of those fluids. Medical people need only understand the characteristics of each humor, and consult various local experts on herbs, minerals, and their efficacy, apply a few leeches to draw off the blood containing the vital fluids, and wait for balance to be restored.