Analytics: unlocking insights
For the organization licensing the system, Agilex reports, "Phanero addresses a variety of advanced analytics and discovery requirements. Phanero Investigations is a Web-based user interface optimized for the unique requirements of the intelligence and law enforcement communities. For example, it provides automated risk assessments based on strength of relationship. It also integrates directly and automatically populates a number of custom and commercial data visualization and case management tools, such as Palantir (palantir.com) Government."
A slower revolution
Those three examples make clear a progression in enterprise analytics. TIBCO was an early leader in integrating advanced analytics within a broader messaging capability of its bus architecture. The founder of Spotfire shifted his focus from trimmed or constrained data sets to the rapidly growing and unbounded environment of public Web content. Agilex combined management tools, applications and mathematical methods into integrated solutions that support other specialist vendors' tools such as the Palantir discovery application.
The progress in advanced analytics, however, is not moving at the pace of the mobile handset sector or social networking. The mathematics revolution is underway but it works at a slower pace. One of the friction points in advanced analytics is the computational capabilities of today's computer systems. Most of the individuals with training in advanced mathematics know about the Big O problem. The snappy name means that some types of numerical recipes cannot be calculated given the time and resource constraints in an organization. If you are technically savvy, you know that programmers have to dodge the class of decision problems known as NP-complete, where there are no known fast solutions. Too ambitious recipes can leave the computer huffing and puffing without finishing the calculation.
A second point of friction for analytics systems is the graphical interface. The math used in most of the systems, including open source solutions like Talend is secondary to the interface and presentation layer. A person with math expertise can use most systems after a short period of familiarization. However, a product manager requires an interface that works more like an automobile's steering wheel and less like a RoboPhilo Humanoid Robot Kit. The interface makes the difference between an analytics solution that sells and one that does not.
In the present big data environment, analytics is the only way to try to make sense of the volumes of digital information available. In separate presentations I attended last year, two professionals said that the volume of big data would be eight zettabytes by 2015, and another expert, this one from IBM, said that the volume of data would be 35 zettabytes in 2020. A zettabyte is a thousand exabytes or a one followed by 21 zeroes. The point is that there are not enough humans on earth to look at those quantities of information. The catch, of course, is that computing power is limited, so chopping a data set down to size requires considerable mathematical acumen and an even larger amount of creativity and cleverness.
Special effects
Much of what passes for analytics in the enterprise is pretty pictures. One firm that has garnered much media attention is Palantir, which has attracted more than $300 million in venture funding since its founding in 2004. Palantir is one of the firms putting considerable effort into interface. On one hand, interface makes the difference in making sales. On the other, interface makes it difficult for the average analytics user to know much about what happens under the hood. An eye-catching graphic display may be more important than the data the visualization presents. The image here shows an output generated by Palantir. I have difficulty figuring out what's important in that type of output, which dozens of companies are racing to emulate.
In my opinion, Palantir is trying to be the leader in the Hollywood-grade special effects approach to analytics. The approach is visually stunning. The market wants a quick way to get from raw data to insights. Other companies are taking the same approach, and I think we have officially entered the era of special effects graphics. There is no turning back because mobile devices change the game for data interaction.
Popular books about mathematics and analytics signal an appetite for numerical literacy. As formal education takes shots to the jaw from venture firms wanting 18-year-olds innovating in incubators, the interest in mathematics is a very positive indication that it is important. There is a shortage of talent for available jobs in analytics. A popular book may not equip a person to compete successfully for a data scientist position.
I heard a senior military professional say to his adjutant, "Find out how that guy created his visuals. I want to have those types of images in my next presentation." The power of graphics may be the point of advanced analytics systems. The user, whether a general or a sales manager, may not come to grips with the underlying data, the assumptions used to trim the data set or the math selected to generate the output. I worked in a company where numbers in an Excel worksheet were changed until the graph looked the way the boss wanted. I changed jobs because art was taking precedence over the craft of analysis. Modern analytics systems may put visual sizzle before understanding.
The need for analytics solutions is rising more rapidly than vendors can respond. We are now facing an analytics gap, the intellectual equivalent of standing on one side of the Grand Canyon and yelling for a person on the other side to come on over right now. When trying to make sense of big or small data, most people do not have access to a jet-powered helicopter, a trained pilot and the foresight to have the aircraft in a holding pattern.