A limit to business intelligence?

In a post at CMS Wire, Greg Silverman, the CEO of Concentric, makes a good point about the new ability of market researchers to do their own analysis, rather than relying on big firms to do it for them. This, says Silverman, changes not only the economics of research, but also the nature of the results:

The marketers' relationships with their analytics providers are currently strained as a service-based, methodologically undisclosed and one-off delivery of insights. These providers and methods are pitted against a new generation of managers and executives who are "data natives"—professionals who rose to the top by having full control of their answering techniques, who like to be empowered and in charge of their own destinies, and who understand the world as a continuous, adaptive place that may have constantly changing answers. This new generation of leaders likes to identify tradeoffs and understand the "grayness" of insight rather than the clarity being marketed by the service providers.

When you're doing it yourself, you can see through the confidence with which old school black box analytics houses deliver their results. That confidence is there to drive up the perceived value of their product, but the truth is always probabilistic and almost always ambiguous all the way down. So, from my point of view, Silverman's comments are very positive.

But I hope more than that is going on. If not, business analytics will not get as smart as it can in the Age of the Net.

The "more" is not just the distributing of research but also the networking of it. It's great that a "data native" now has the skills and tools to take control of the analytics process. But the real power comes when research and researchers are able to use one another's work, and to elaborate on that work in public. That's how you get a data commons that not only aggregates data but that enables researchers to bang their models together to discover weaknesses and to create sparks.

We see this sort of commons arising in some of the sciences. The Human Genome project, for example, made lots of data public, which enabled and encouraged dramatic advances in genetics and related sciences. Lots of environmental data is public. Physicists, among others, are rapidly developing a culture of openness. Networking knowledge is the only way to enable it to scale to the size of the Internet.

But it's different in business. Business has a Prime Directive: Maximize shareholder value. Or at least: Make as much money as possible. Business views its data as a valuable asset, a conception that knowledge management has all too often abetted. Business thus is not going to share its data with the world, because the world contains the business' competitors. But, if networking knowledge is how we get really smart, then businesses are structurally unable to get as smart as they could be. Which seems a shame.

The trade-off

It's not entirely hopeless, of course. We know that people from competitive companies will engage in surprisingly frank shop talk if put into the same room, preferably one with some food and beverages. People generally have enough judgment to hold these conversations without spilling any important secrets. Such conversations help everyone get smarter. Entire companies have been built to provide these conversational opportunities within industries.

Such occasions create social channels through which knowledge flows, and that's important. But it's not enough. If we really want to understand our markets and customers, we'd make our data about them publicly available precisely so lots of people with lots of different ideas and interests can try to find new pathways through the chaos. We'd let everyone with a computer come up with a model and try it out. We'd encourage debates about the results, and enable public tinkering with the models.

But competitive pressures don't allow this, at least not at scale. That's too bad. The extent to which businesses protect their data assets is the extent to which business is limiting its own intelligence. The trade-off may be worth it. But we should at least recognize that it's a trade-off.