BIG DATA interviews with the experts
Q Lamont: What obstacles stand in the way of big data applications?
A Bakshi: To start with, the sheer volume, variety and velocity of big data would be a challenge to keep up. Hence, from an enterprise and agency strategy perspective, one needs to develop a big data strategy starting point. Issues of data governance such as data privacy, ownership and lifecycle need to be reconsidered in the context of new and growing sources of big data. Second, the notion of enterprise data is going to be broader than traditional sources. It will include a lot more unstructured data, which generally does not have a well defined schema. To extract the most value from the different types of data, the agencies will need to adopt an integrated approach for analyzing traditional structured and unstructured data, which would imply extracting that information from multiple silos.
Bhambhri: One of the biggest challenges businesses face is finding enough data scientists to make sense of the petabytes of available data. Businesses need marketers, managers and IT staff with data analysis skills to allow them to leverage data and use it to improve business management and marketing. IBM is training the next generation of data scientists by helping universities integrate big data into their curriculums and creating project-focused case studies through which students can gain hands-on experience .
Q Lamont: What is the most common mistake when organizations first start working with big data?
A Zedlewski: A common error from the IT department is not identifying a first use case that has compelling business value. The technology sweet spot should align with the business sweet spot.
Vesset: One problem is that the IT department may look at the technology first and become enthusiastic about how much information they can store, without thinking about how they can use it. IT groups need to work hand in hand with the business side to identify specific use cases and proceed from there.
Q Lamont: How should an organization get started with big data?
A Zedlewski: Hadoop is an open source product that can be downloaded and installed from http://hadoop.apache.org, and the documentation is available. Users can try it out. There is a preinstalled virtual node on the website or users can build their own node, which is an individual computer in a cluster of computers. Start with a relatively narrow project to demonstrate what it can do. A good use case to try out is a recommendation engine for a product based on consumer preferences, or analyze an event log that records data from network performance to look for abnormalities. These applications are ideal for Hadoop.
Bhambhri: The organization should start an experiment in its own shop. Companies should take a look at the sources of data that they care about but are not tapping into now. Then, bring the data into a platform and start the discovery process, with a few people dedicated to the job. The analysts can then see what the data is telling them, and present the insights to their lines of business. If there is value added, they can expand beyond the pilot and build out their infrastructure.
Q Lamont: What do you see in the future of big data applications?
A Bhambhri: We expect big data to continue to grow at a very rapid rate. So many things are instrumented today, such as traffic information-GPS devices, traffic lights, sensors in the roads, for example-and it generates a tremendous amount of information. This information can be used by city planners to solve traffic problems and improve transportation efficiency. Consumer devices such as set-top cable boxes are generating logs. The number of users on smart devices has exploded in the last couple of years, along with data derived from social media. Meanwhile, technology breakthroughs have come along that allow organizations to make use of the information.