Re-decentralize knowledge

There was a time when the Web was flat. People created sites, and all sites were created equal.

But flatness does not scale. By the mid 1990s, you knew what you wanted was probably on the Web, but you didn’t know how to find it. The search engines were young and incomplete. Why, I remember back when I worked for Open Text and its search engine was heading up toward 100,000 pages. Imagine that! At the time it was an awesome number, but the number of sites was bigger than any search engine could encompass.

That issue inspired an early centralizer of the Web: Yahoo. It started as a multilevel categorization of the favorite Web sites of two Stanford grad students. With geek humor they called it “Yet Another Hierarchical Officious Oracle.” While the search engines let you search its categorical listing of sites, Yahoo only let you browse, at least until it started using Open Text’s search engine. Yahoo only worked because it only listed “the best of the Web.” If you wanted your Web site to be found, you had to hope that Yahoo would accept it.

This has been the play of decentralization and centralization at scale ever since. Anyone can create a Web site and add it to the Web’s distributed architecture. But to be found, it’s got to show up somewhere centralized, such as Yahoo in the old days and Google and Bing in the new ones. Of course, just showing up in a search engine isn’t enough. Your site really needs to be on the first page of results. So the centralized gods of the Web were born and now have grown into seemingly unshakeable dominance: It’s a grand decentralized Web except for the commercial entities that exert tremendous power over what we see of it.

Search engine power

There are lots of reasons to be troubled by this, but let’s stick to what it does to knowledge.

When it comes to simple facts, it’s pretty amazing. The search engines generally give you correct answers to questions about the height of movie stars or the weight of a pound of feathers. (WolframAlpha surprisingly does not understand that question.) As matters become more complex, controversial and chewy, the power exerted by the search engines becomes more troubling.

But what is the alternative? How can we decentralize knowledge?

It turns out that we’ve already invented a number of ways, and most decentralize knowledge by decentralizing power.

For example, at Quora and Stackoverflow, anyone can ask a question or answer one. Because these are now big public sites, the problem of scale once again raises its scaly head. The solution for both sites is to give everyone a vote, spreading the power. That’s nice and democratic, but the aim of these two sites, as I understand it, is not to be an experiment in social theory. It’s to try to get to the best answer; if people are being misled, eventually they’ll stop coming to these sites. So, I assume if there were some non-democratic way of getting better answers—the more cred you’ve established, the more your vote counts, perhaps—these sites would adopt them.

Long tail of small sites

But there’s another way that we have kept knowledge decentralized on the Web: sites and links. I know that I am an old-fashioned (and just plain old) blogger, but one of the key ways we’re keeping knowledge decentralized is by having our own sites that we then link to other sites. We’ve known since the early 2000s that this results in a power law distribution—a relative handful of big sites and a long tail of small sites—but so long as that long tail is there and active, knowledge is distributed. So long as the ecosystem is healthy, the big sites provide the sort of concentration we need in order to coalesce knowledge, which gives those sites power, but there is also power spread out in the long tail where smaller groups can speak more frankly (because they’re out of the bright public glare), can generate local knowledge about smaller topics and can spark ideas that percolate to the larger sites.

This is an imperfect system, but the problem of scale means there is no perfect system possible. If we want to avoid giving a handful of gigantic corporations sole power over knowledge, then we need to keep our multiple imperfect systems alive, vital and distributed.