CEO Guide to Big Data and Real-time Analytics

Aimee Lee WebberNovember 21, 2013October 21, 2019Education and Career

Big data and the open-source Hadoop technology behind it are a couple of the hottest topics in the corporate world at the moment. Hadoop gives companies a cheap way to store and process huge amounts of raw data culled from Web applications, server logs and social media —and often fosters hopes and expectations among CEOs that their companies can be the next Google, Amazon or Facebook if only they can find innovative ways to harness that data for competitive advantage.

It’s important for CEOs to realize that Hadoop technology, with few exceptions, is batch-oriented and doesn’t have the capability to do real-time queries.

In this respect, although Hadoop is modern, distributed and parallel, Hadoop reports are similar to traditional business intelligence (BI) charts and reports that are generated using historical data and are only useful in describing business processes that have already been completed.

While historical data has value, for many business managers trying to steer their company through complicated commercial, retail or regulatory environments, navigating this way may be a bit like trying to drive a car by looking in the rear-view mirror. Real-time insight into high-volume streaming data sources, current events, and ongoing business processes provides a bigger and more complete picture of your business environment, especially if your goal to streamline or simplify business processes that drive customer experience.

Operational, Organizational and Business Intelligence

Collecting the data is only part of work you need to do in developing Big Data applications, you still need know-how or intelligence about what needs to be extracted from that data. Two types of intelligence are useful in situations that require massive data collection efforts and real-time analysis: Operational Intelligence and Organizational Intelligence. Operational Intelligence is often linked to or compared with real-time business intelligence (BI) since both delivers visibility and insight into business operations.

But there are fundamental differences: Operational Intelligence is primarily activity-centric, whereas BI is primarily data-centric and relies on a database (or Hadoop cluster) as well as after-the-fact and report-based approaches to identifying patterns in data.

Operational Intelligence transforms unstructured Big Data streams—from log file, sensor, network and service data—into real-time, actionable intelligence. While Operational Intelligence is activity-focused and BI is data-focused, Organizational Intelligence differs from these other approaches in being workforce- or organization-focused.

In his book King Arthur’s Round Table, Harvard Professor David Perkins defines organizational intelligence as “how well people put their heads together in a group, team, organization, or community.” Perkins notes that pooling physical effort is easy, but pooling mental effort is hard—something he calls the lawnmower paradox since “it’s a lot easier for 10 people to collaborate on mowing a large lawn than for 10 people to collaborate on designing a lawnmower.”

Core Take-Away/Summary

Instead of looking at historical data to try and figure out what happened in the past, companies use Organization Intelligence and real-time analytics to look at data in terms of flows and processes, to understand their business environment at a more granular level, to create new products and services, and, finally, to respond more quickly to change.

Availability of data isn’t a problem in building Big Data applications. The problem is filtering out noise to find the data you need: finding the proverbial needles in the haystack and getting those needles into the right hands of frontline and back-office employees (including executives) who can determine the right course of action so that this intelligence becomes part of the complex underlying structure or fabric of your organization.

(Writing by Roger Smith; editing by Amy Canter, Janina Energin, and Hendrik L Clarke) Roger Smith, Technology Marketing Specialist at Hypersoft Information Systems, a 20-year-old software consulting firm specializing in Organizational and Operational Intelligence.

Related article by Roger Smith: “Big Data Requires More than Seat-of-Your-Pants Business Leadership.”