The summer of transparency

Google has been and remains a secretive company. Part of the firm’s reluctance to engage in orgies of public relations is common sense. Mountain View, Calif., is open but also closed. The culture spawned Andy Grove’s best-selling Only the Paranoid Survive: How to Exploit the Crisis Points that Challenge Every Company. Dr. Grove popularized the importance of chaos, which obscures underlying intent. When published, Messrs. Brin and Page [Google co-founders Sergey Brin and Larry Page] were revving Google’s engines, and too much chatter about the Google technical secret ingredients was unnecessary.

The other reason is the nature of bright engineers and mathematicians. Talking to people who did not understand the jargon and mindset was a great deal of work. Simplifying certain concepts is difficult. Sound bites are not the core competency of an individual who can explain the importance of Kolmogorov-Smirnov statistics for small samples.

My research suggests another reason for Google’s early cone of silence about its technology. Back when Google was still called BackRub, Hewlett-Packard (HP) was busy turning the Digital Equipment Corp. (DEC) engineers who developed AltaVista.com into digital orphans.

Despite its reputation and technical resources, HP did not appreciate the AltaVista.com search system or its remarkable technology. AltaVista.com showcased Digital Equipment’s Alpha chip technology and sported some clever in memory tricks to speed up some functions.

In the pre-Google era, AltaVista.com was arguably the best Web search engine. Most of its features were far ahead of the systems offered by Excite, HotBot, Lycos and Yahoo.

Google’s founders and their advisers were smarter and more prescient than HP’s management mavens. Google corralled a number of those AltaVista.com engineers and scientists. Once they were on the Google team, their learnings provided Google’s founders with a turbo boost in three key technical areas.

First, the AltaVista.com engineers learned what not to do when indexing Web content and serving queries. For example, scarce, exotic CPUs and proprietary memory sticks were a bad thing. Commodity CPUs, commodity memory and economical scaling were more important than state-of-the-art hardware. Exotic gear translated to higher costs, technical headaches and often long, long delays between placing an order and receiving the devices.

Second, the AltaVista.com engineers understood the importance of what we now call supercomputer performance on a Taco Bell diet. Today, we use the term "green" to refer to this type of efficiency-centric engineering. Let me give you an example. The Alpha chips were little welding torches. Put some DEC Alphas in a basic server room, and the Alphas created an instant sauna. From its inception, Google’s infrastructure benefited from practical hardware engineering considerations such as eliminating the need for humans to fiddle with dead servers. Google’s plumbing ignores a dead machine so downtime does not require a human to get a system back online.

Third, the AltaVista.com experience made clear the importance of finding ways to eliminate bottlenecks that choke search systems. Reading data from a magnetic storage device costs time. In a high demand situation, time is a precious commodity. The speed up, however, cannot come from paying Dell, HP or IBM for bleeding-edge hardware. AltaVista.com’s engineers focused on software methods to minimize bottlenecks. Brin and Page understood the importance of that line of thought.

The AltaVista.com engineers contributed to such innovations as the Google File System (smart enough to avoid the message passing that can choke a parallelized system), MapReduce (a framework supporting parallel computations to be shared among Google’s server clusters for processing large data sets), Chubby (a record locking and unlocking service that jumped over traditional relational database methods of managing cell and row locks), and other innovations that continue to give Google a performance and cost advantage 10 years after Google opened for business.

And the AltaVista.com engineers brought to Google fresh thinking about data management. When you index the Web, the indexes are huge. But when data about system processes, user behavior and other system-derived data are considered, data management is a very big deal.

Google’s 2008 messaging focuses on topics that I find remarkable for two reasons: First, the comments are not about the deep technical decisions that allow Google to maintain and increase its lead over its competitors, and, second, the subjects are the equivalent of a masseuse’s kneading a client’s shoulders. The human touch is defined by a cold, clinical, professional demeanor. Let’s look at three recent examples of Google’s public statements.

ITEM: Eric Schmidt, Google’s president, told New Yorker writer Ken Auletta at a conference: "The goal of the company is not to monetize anything. The goal is to change the world—and monetization is a technique to do that." (Source: Wired.com Web site, June 11, 2008.)

ITEM: Sergey Brin at the June 23, 2008, expansion of Google’s New York office, was asked about the Google mobile phone. Brin replied, "I’ve been playing with some prototypes." His latest application allowed a Googler to throw an Android phone "up in the air and it would measure the amount of time until he caught it." (Source: New York Times Web log, June 23, 2008.)

ITEM: Marisa Mayer, Google VP of search products and user experience, said, "We’re constantly looking at the user experience—what they want, what they need ... We’re constantly looking at user intent." (Source: Freep.com, June 26, 2008.)

Add to those statements the avalanche of information provided at the Google I/O conference. You can see most of the presentations here: http://code.google.com/events/io. More technical papers are available at http://research.google.com/pubs/papers.html. Google continues to boost its output of technical data and sample code. Comparatively few people read that information because the documents are stuffed with equations and code.

Yet in the popular media, it is Google, Google, Google: market share of Web search over 70 percent, market share of Web advertising over 60 percent, market share of online video over 70 percent.

A disconnect between Google’s technical information and its public persona exists. The technical Google is the company’s core competitive advantage. But Wall Street analysts and some competitors have a superficial view of Google as a giant college dorm with a fridge stocked with free Odwalla juice.

Google’s transparency is more like the brightly polished surface on a Chip Foos show automobile. The apparent depth of the finish, the liquid look of the auto’s paint and dazzling reflections inhibit most people from crawling under the vehicle to see its hidden engineering.