AI: The issue is execution

Another coup

In Jeopardy!, IBM marketing engineered another coup. Watson appeared to exhibit deep understanding of people, things, events, and the relationships among them—all while operating in a context constrained by the unusual structure of the Jeopardy! game itself, in which in the first place, contestants are effectively fed answers and have to identify questions, and second, have to do this in a free flow of language that often contains slang, arcane terms, and the use of synonyms or nicknames designed to confuse all but the most wary contestants.

The Watson victory happened over 3 days in February, and we were pleased to be part of the celebration that IBM hosted on the last evening of the contest. Within 3 years, in 2014, IBM announced a new Watson business unit, whose mission would be to bring the knowledge and technology required to deploy cognitive computing (AI was deemed too strong a term at the time) to the enterprise. The era of “cognitive business” was upon us, according to CEO Ginny Rometty in speeches and in IBM’s annual reports.

To its credit, IBM made a point of emphasizing that cognitive computing applications would be “local”—driven by the accumulated knowledge and expertise of each area of business and each specific field. In 2015, it announced that a new Watson business unit, Watson Health, dedicated to health and wellness, would be the primary initial business target for Watson’s intelligent capabilities. In particular, the focus of the new division was to be on oncology and assisting the oncology community in training Watson to be an “oncology assistant,” an intelligent tool to assist doctors in the attempt to improve cancer diagnosis and treatment. Early IBM partners announced as part of this effort were Memorial Sloan Kettering Cancer Center (www.mskcc.org) in New York and the University of Texas MD Anderson Cancer Center (www.mdanderson.org). This was a time of much optimism and hope for medicine assisted by computer smarts, not to mention a multi-billion-dollar bet for IBM.

So how is it going? Fast-forward to the present day. By mid-2018, Watson Health was suffering under more and more negative news about the lack of effectiveness of its cancer solutions, and IBM was being subjected to increasing skepticism about the overall health of its Watson business. Watson Health executed a round of layoffs toward the end of the second quarter, and while executives claimed these were normal adjustments carried out to align staff across several acquired entities, the headlines implied a darker story. Financial analysts have been pointing to data difficulties and service engagement costs as high hurdles to Watson deployments since 2016.

In August, Daniela Hernandez and Ted Greenwald would ask, in The Wall Street Journal (www.wsj.com): “Can Watson cure cancer? That’s what IBM asked soon after its AI system beat humans at the quiz show Jeopardy! in 2011. Watson could read documents quickly and find patterns in data. Could it match patient information with the latest in medical studies to deliver personalized treatment recommendations? Big Blue promised its AI platform would be a big step forward in treating cancer. But after pouring billions into the project, the diagnosis is gloomy. … Sometimes, it didn’t say more than doctors already knew. In other cases, Watson was tripped up by a lack of data in rare or recurring cancers and rapidly evolving treatments.”

Model versus reality

As early as 2015, the company’s work with MD Anderson had ended in acrimony when an oncology diagnostic assistant effort had been written off after $39 million in progress payments to IBM and a $70 million overall investment. In June this year, a report from Stat News (www.statnews.com) quoted doctors from another lead partner, Sloan Kettering, saying that the oncology assistant IBM had been working on since 2013 often gave the wrong advice. To add insult to injury, the former deputy head of Watson Health declared in a presentation in 2017 that for a number of reasons related to the difficulty of obtaining validated data streams, the deployment team had had to resort to using “hypothetical data” as opposed to real patient data to train the system.

So what is going on here? Seven years after Jeopardy! where are the doctors claiming that they could not do their jobs today without the help of their Watson Health computer partners?

I propose that IBM, with all its research chops and marketing acumen, has fallen badly into the “fallacy of misplaced concreteness.” Articulated by British philosopher Alfred North Whitehead in 1929, the fallacy arises from the fact that “people tend to mistake abstract concepts for accurate descriptions of reality,” according to Psychology Concepts (www.psychologyconcepts.com). Wikipedia (www.wikipedia.org) adds that “it is the error of treating something that is not concrete, such as an idea, as a concrete thing. A common case of the fallacy is the confusion of a model with reality: the map is not the territory.”

So the idea of cognitive computing as a cancer-fighting tool is not the same as a real cancer-fighting tool. Every cancer patient has a unique genome. Every cancer has its own genetic variations that can change from patient to patient. Therapies work for some and not others. And do doctors know why? Even if they could access perfect patient, research, and disease data (impossible today), do they understand the underlying dynamics that result in success or failure? We have personal experience that indicates that the answer to those simple questions is “no.”