Big opportunities in small data

With the world cranking out over two exabytes on a daily basis, it’s no wonder big data is all the rage. It’s a natural consequence of networked access to any and all types of data sources. And with the emergence of the Internet of Things (IoT), the volume is rapidly growing from exabytes to zettabytes.

Data analytics goes hand in hand with big data. Its goal is to more accurately predict what you’re likely to do next, so the things you need will be waiting for you precisely when you need them.

At some point down the road as the cloud becomes an inescapable part of our world, the matching of need and availability will become even more transparent and instantaneous. This will be enabled in part by a technology that’s quietly emerging on the other end of the spectrum, known as small data.

Big and small data differ not only with regard to volume, format and structure. Approaches to analyzing and acting on the two types are also markedly different. Let’s do a comparison and see where KM fits in.

The cosmos of big data

Big data uses open source platforms such as Hadoop, often in a peer-networked configuration, to store massive volumes of data. Tools like MapReduce extract the data, and an array of statistics software, pattern matching and machine learning algorithms are applied.

As a result, upward of trillions of data points are reduced, often into a simple mean and standard deviation or into various categorical representations, i.e., “buckets.” Rule sets are produced that generate recommendations aimed at specific segments of the population. By doing that, big data seeks to help organizations make better business decisions based on the aggregate behavioral tendencies of the population segments in each bucket.

But that approach has its drawbacks. The greater the reduction, the more context and individuality are lost. What might be an abnormal reading for some (such as a prolonged, low-grade fever) could be completely normal for others. Another drawback is as the number of rules for various conditions and circumstances grows, combinatorics makes the rule sets increasingly more difficult to maintain.

Perhaps the greatest shortfall is the lack of capacity for measuring unintended consequences. As is often the case in so-called “data-driven” decisions, the resulting damage may not be discovered until it’s too late.

On the other hand, there are many positive benefits to big data. If your home address is in the path of a severe storm, the entire supply network mobilizes to help you quickly stock up on whatever emergency supplies you might need.

The law of large numbers also comes into play. Direct marketers are delighted if data analytics improves their response rates by even half a percent. The same goes for politics. If you can garner support from 25 percent of one group, 15 percent of another and 11 percent of the rest, congratulations—you’ve won a majority of the vote!

The microworld of small data

Small data also uses the power of peer-networked computing. In this case, the goal is to create and maintain millions, even billions of much smaller, highly individualized buckets.

The emerging array of small data analysis tools consists of graph or other NoSQL databases, personal ontologies, state and goal spaces and generative rule sets. Instead of maintaining large data repositories, small data uses those tools to build a computationally based model of the state of an individual (physical health, financial health, education, etc.) as the primary record. It then uses a stream of individualized data inputs to update and adjust the model.

At that point, unlike in big data, input data are discarded. Then the state changes of the model are analyzed, focusing on aspects such as learning behaviors, lifestyle habits, etc. Instead of attempting to identify and act on mass-market trends, rules generate recommendations that focus on influencing individual behaviors at the deep structure level (e.g., memory engrams). Over time, the rules are adjusted as well. Once in place, small data rule sets are more stable and less complex than those typically associated with big data.

All of that dramatically reduces storage requirements while producing resources that can be more easily de-individualized and shared. That translates into a different and much more manageable set of governance policies and responsibilities.

Opportunities for KM

For decades, we’ve been promoting the notion of mass customization. From building a playlist to configuring your own tablet/laptop, it’s become commonplace in our consumer-based society. But we haven’t even scratched the surface of what small data can do. Here are three simple steps you can take:

• Look for any market niche in which the current one-size-fits-all model might be replaced with something built for “you and only you.”
• Replace the whole notion of databases and analytics with personal ontologies that monitor changes in the state of the user. Note that this can be applied to devices and systems as well. Complex ontologies may not always be necessary. In many circumstances, a simple concept space or topic map will suffice.
• Generate and maintain a set of rules for (a) analyzing the current state of the person, device or system and (b) recommending behavioral changes aimed at advancing toward the goal state.

For example, in the area of personal health and fitness, every person has a unique physiology, the attributes of which are represented in the model. Vital signs and other data are continuously streamed from devices such as Fitbit, along with records that may include food and liquid intake, medicines, air quality, periods of rest and activity, etc. All can be used to update the state of the model and generate recommendations to correct negative tendencies and reinforce positive behaviors.

The same principles apply to education, in which every person has unique mental models and learning styles. Similarly, consumers can be guided to help choose and use products more efficiently and effectively.

Action to take

Look for anything resembling a “one-size-fits-all” approach, or that lumps customers, products and services into groups or categories. Rather than attempting to pick and choose among groups, or worse yet “playing the averages,” put your brain trust to work at designing new business models that create the best, most unique experience possible for each individual customer.

Don’t hesitate to consider complex systems like the human physiology, the environment or a nation’s economy. For these, think state, not data.

Big data has its place. But by obsessing over data we’ve missed the true essence of knowledge representation and how we might influence behaviors through deep structure modeling. The added benefit is that this can be accomplished within an extremely small digital footprint.

In the age of “black swans,” where dramatic changes can appear overnight, your best bet may very well be to spread your risk across as many unique signatures as possible. Small data may be the disruptive technology you’re looking for to propel you to the forefront of the next economic wave.