Big data: hype or transformation?

Recent Gartner surveys indicate that fewer than 10 percent of organizations have big data applications in production, although 30 percent have invested in it. Will the investment pay off? Right now the answer seems to be a strong yes. According to a report from Bain & Company, early adopters of big data analytics are twice as likely to be in the top quartile of their industries, and five times as likely to make more rapid decisions than their peers.

"Big data is not just more data," says Forrester's Hopkins. "Because scale fundamentally changes the way that business gets done, only organizations that are able to change their mindset will be able to experience the transformation that big data offers."

Big data: social media meets GIS

When social media streams such as Twitter are enhanced with geographical information system (GIS) data and visualized on a map, the tweets can become much more meaningful. The IBM Research Accelerated Discovery Lab (research.ibm.com/client-programs/accelerated-discovery-lab) has been exploring the use of big data analytics in different industries. Esri, which develops geographical information systems, partnered with the lab to build a Social Monitor to better understand brand customer sentiment and brand management but with a geographic component.

Esri developed an interactive map that showed tweets for eight national clothing retailers by location and time, and IBM used social media analytics to determine sentiment and location, as well as some psychological analyses. The results indicated that GIS could show whether an issue was localized or had become national in scope. This information helps determine how the retailers should respond to the issue. The GIS data also helped retailers better understand their image by region.

Workload automation for big data

MetaScale was formed to support companies in deploying big data solutions. It evolved as a separate entity after developing in-house expertise in Hadoop in Sears Holdings Corporation. MetaScale uses Control-M, a product from BMC Software, to manage the complex batch processing typical in big data solutions. Control-M works with any existing enterprise workflows for processing information, and can integrate information from numerous data sources.

"About 60 to 70 percent of data processing overall is in batch form, but with Hadoop, the figure jumps to nearly 100 percent," says Robin Reddick, senior solutions marketing manager at BMC, "so organizations need tools to help them manage batch processing. In addition, most work environments are hybrid, so a tool that works with other data sources and platforms is important."

For companies starting out with big data, Reddick suggests that experimentation on a small scale is the best approach. "It's fine to put data into Hadoop and then see what kinds of questions can be asked," she says. Often companies can benefit in both time and money from moving data to Hadoop for their regular transactions, too. "One insurance company was running an ETL job that took 14 hours, and by moving it to Hadoop, reduced the time to seven minutes," Reddick adds.