Progress and knowledge

When it comes to knowledge, are we making progress? And if we are, why does it so often seem like we're  sliding backward as we try to climb Mt. Smartypants?

From the gitgo, the question of progress has a ton of assumptions packed into it. For example, Sidney Pollard in 1968 wrote that "a belief in progress implies the assumption that a pattern of change exists in the history of mankind, that this pattern is known, that it consists of irreversible changes in one general direction only, and that this direction is toward improvement."

Now, we don't have to think that all of human history exhibits progress to wonder if we are now making progress in knowledge in the current age. But we do have to face the fundamental question buried in the concept of "improvement?" If we say (as Charles van Doren did in 1967) that progress is improvement from "a less to a more desirable state of affairs," we're pretty much at the nub of the problem: What do we desire? And what if we don't all agree about what we desire? And, most of all, what if the changes we're experiencing are changes in what we desire?

In a rough sense, we know what we want knowledge to do. From the beginning, we've wanted to believe true things rather than false ones. We want this knowledge to be sufficient for making good decisions, which means we want knowledge to be manageable by our brains and our technology. In fact, we limit the scope of our decisions based on our capacity to manage knowledge; that's why a company might make a one-year plan, a five-year plan, or even a 10-year plan, but not a 30,000-year plan.

Degraded methodology?

So, we can put the question of progress differently: If our desire is to have enough knowledge that we can make good decisions, are we making progress? We still make plenty of mistakes. In fact, it often feels like we make more than ever. But our ability to analyze physical phenomena has progressed without question, so that we now can decide on investments in research and development that will change the lives of billions of people. And we now can know some things that we simply couldn't before. For example, patterns of human behavior are emerging from ginormous clouds of data, enabling us to make better decisions about design and marketing.

If it often seems like we're not making progress, or that our new always-on, always-chattering communications infrastructure is in fact distracting and confusing us, it may be that we feel that the methodology of knowledge has degraded. The old methodology was well known and seemed to guarantee good results. To become knowledge, something had to be filtered by experts. Those experts had qualifications and often degrees. Now the "methodology" seems to be simply who tweets loudest.

New and tasty

Yet that's an oversimplification to the point of simple wrongness. There's no doubt that more people are speaking more freely—without credentials, authorization or permission—but we are evolving new methodologies of knowledge to deal with this. We're discovering that we can filter after publication, not before. We're discovering that there are ways in which the voices of many can filter as well as the hands of a few credentialed experts. We're discovering the power of iteration at scale—many hands making many small tweaks can accomplish knowledge tasks that the old methodology would never even have attempted. The standard example of this is Wikipedia, but StackOverflow's question-answering service is another, as is any discussion site capable of an extended conversation.

So, yes, we're making progress in knowing our world, if progress means moving us to the desired state of making better decisions. But we're doing so by using methods that our old ideas of knowledge assume must be flawed and degraded. Still, the proof of the pudding is indeed in the taste. And our new ways of knowing are tasty indeed.