Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxi
The Business Transformation Imperative . . . . . . . . . . . . . . . . . . . . . . . . . . . .3
Walmart Case Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3
The Big Data Business Model Maturity Index . . . . . . . . . . . . . . . . . . . . . . . .5
Business Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7
Business Insights . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7
Business Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9
Data Monetization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
Business Metamorphosis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
Big Data Business Model Maturity Observations . . . . . . . . . . . . . . . . . . . . . 16
This chapter introduced you to the business drivers behind the big data movement. I talked about the bevy of new data sources available covering structured, semi- structured (for example, log files generated by sensors), and unstructured (e.g., text documents, social media postings, physician comments, service logs, consumer comments) data. I also discussed the growing sources of publicly available data that reside outside the four walls of an organization.
This chapter also briefly covered why traditional data warehousing and business intelligence technologies are struggling with the data volumes, the wide variety of new unstructured data sources and the high-velocity data that shrinks the latency between when a data event occurs and when that data is available for analysis and actions.
Probably most importantly, you learned how leading organizations are leveraging big data to transform their businesses—moving from a retrospective view of the business with partial chunks of data in batch to monitor their business performance, to an environment that integrates predictive analytics with real-time data feeds that leverage all available data in order to optimize the business.
Finally, you were introduced to the concept of the Big Data Business Model Maturity Index as a vehicle for helping your organization identify where they are today, and map out where they could be with respect to leveraging big data to uncover new monetization and business metamorphosis opportunities. Several “How To” guides were included in this chapter to help your organization move from one phase to the next in the maturity index
Consumer Package Goods and Retail Industry Pre-1988 . . . . . . . . . . . . . . . 19
Lessons Learned and Applicability to Today’s Big Data Movement . . . . . . .23
This chapter covered the history lesson from the late 1980s, where retail POS scanner data created an earlier “big data” revolution. POS scanner data volumes quickly jumped from megabytes to gigabytes and ultimately to terabytes of data, replacing the bimonthly store audit data that had previously been used to make marketing, promotional, product, pricing, and placement decisions.
You reviewed how the volume, diversity, and velocity of this POS data broke existing data management and analytical technologies. EIS analytic software that ran on mainframes could not handle the volume of data, which gave birth to new data processing technologies such as specialized data management platforms (Red Brick, Teradata, Britton Lee, Sybase IQ) and new analytic software packages (Brio, Cognos, Microstrategy, Business Objects).
Finally, the chapter covered how the ultimate winners were those companies who were able to create new analytics-driven business applications, such as category management and demand-based forecasting. Suddenly, retailers with immediate access to POS scanner data coupled with customer loyalty data knew more about customer shopping behaviors and product preferences that they used to change the industry balance of power and dictate terms to CPG manufacturers with respect to pricing, packaging, promotion, and in-store product placement.
Big Data Impacts: The Questions Business Users Can Answer . . . . . . . . . . .26
Managing Using the Right Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .27
Data Monetization Opportunities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .30
Digital Media Data Monetization Example . . . . . . . . . . . . . . . . . . . . . .30
Digital Media Data Assets and Understanding Target Users . . . . . . . . . 31
Data Monetization Transformations and Enrichments . . . . . . . . . . . . .32
Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .34
Model Planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .44
Model Building . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .44
Communicate Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .45
Operationalize . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .46
New Organizational Roles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .46
User Experience Team . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .46
New Senior Management Roles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
Liberating Organizational Creativity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .49
This chapter covered the organizational impact of big data, specifically the organizational impact of adding the data scientist to the organization’s existing analytics lifecycle process. I laid out an analytic lifecycle where the roles, responsibilities, and expectations of each key stakeholder—business users, DBA/data warehouse managers, data scientists, and B1 analysts—are clearly defined to ensure tighter collaboration against a targeted business process.
The chapter also dove deeply into the specific roles and responsibilities of the data scientist as part of the data science lifecycle. I described each of the key tasks within the different data science lifecycle stages, and also identified specific areas where close collaboration with the data warehouse, ETL, and BI teams could be beneficial to the data scientist.
Next, we covered new organizational roles that are being dictated by the needs and potential of big data. We discussed the importance of the user experience team, and that team’s role with respect to the other members of the big data team. We also discussed new senior management roles—the chief data officer and the chief analytics officer—and the critical nature of those roles to capture, augment, preserve, and even legally protect the growing portfolio of corporate big data assets.
Finally, we covered the liberating effect of embracing a culture of experimentation—empowering organizational “what if” thinking—and how the concept of experimentation can free up the creative juices of both individuals and the organization as a whole.
Business Intelligence Challenge . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
The Death of Why . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .55
Big Data User Interface Ramifi cations . . . . . . . . . . . . . . . . . . . . . . . . . . . . .56
The Human Challenge of Decision Making . . . . . . . . . . . . . . . . . . . . . . . . .58
Traps in Decision Making . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .58
What Can One Do? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .62
This chapter covered how access to massive and diverse data sets is changing the way analysis is performed—organizations are spending less time understanding what caused something to happen and instead are acting more quickly on what the data is telling them to do. This approach does not work in all situations, as there are cases where it is important to understand why something happened (think medical care or triaging a major accident). But in many cases, the speed of making a decision is more important than getting the “perfect” decision (think use cases such as pricing, yield management, markdown management, ad serving, or fraud detection). To quote General George S. Patton, “A good plan violently executed now is better than a perfect plan executed next week.”
Next you considered how the insights and recommendations that can be driven by big data and advanced analytics could impact the business user experience and the user interface. Instead of the traditional Business Intelligence model of giving users access to their data and hoping that they can slice and dice their way to insights (like trying to find the silver needle in a haystack of chrome needles), you can now leverage predictive analytics and real-time data feeds to uncover and publish insights and recommendations directly to the business users. That can dramatically impact the way the business users interact with data, and dramatically improve their productivity and business effectiveness.
Finally, you had a bit of fun considering how the human mind works against the best decision-making intentions. You considered human tendencies that short-circuit desired decision processes and lead to suboptimal, wrong or even fatal decisions. The chapter presented some techniques and processes that your organization can leverage to ensure that these traps in decision-making don’t hinder your organization’s ability to embrace data- or analytics-driven decision making.
The Big Data Strategy Document . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .66
Customer Intimacy Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
Turning the Strategy Document into Action . . . . . . . . . . . . . . . . . . . . .69
Starbucks Big Data Strategy Document Example . . . . . . . . . . . . . . . . . . . .70
San Francisco Giants Big Data Strategy Document Example . . . . . . . . . . . .73
This chapter covers several topics. It goes into detail on the use of the Big Data Strategy Document to ensure alignment between your big data initiatives and what the business thinks is materially important—the overall business strategy. The Big Data Strategy Document drives alignment between the business stakeholders and IT, while also acting as a guide against which data and technology decisions can be made. The strategy document helps prioritize the technology requirements by gauging them against their ability to support the targeted business strategy, key business initiatives, critical success factors, and key tasks.
Two examples of Big Data Strategy Document development were used—Starbucks and the San Francisco Giants—to help you understand how to build your own Big Data Strategy Document. It is not a hard process, but it does require the cooperation of key business stakeholders and their supporting IT organization in order to complete the document, which likely is the first test for how serious an organization is in leveraging big data to materially transform their business operations and rewire their value creation processes.
Understanding the Big Data Value Creation Drivers . . . . . . . . . . . . . . . . . .81
Driver #1: Access to More Detailed Transactional Data . . . . . . . . . . . . .82
Driver #2: Access to Unstructured Data . . . . . . . . . . . . . . . . . . . . . . . .82
Driver #3: Access to Low-latency (Real-Time) Data . . . . . . . . . . . . . . . .83
Driver #4: Integration of Predictive Analytics . . . . . . . . . . . . . . . . . . . .84
Big Data Envisioning Worksheet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .85
Big Data Business Drivers: Predictive Maintenance Example . . . . . . . . .86
Big Data Business Drivers: Customer Satisfaction Example . . . . . . . . . . 87
Big Data Business Drivers: Customer
Micro-segmentation Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .89
Michael Porter’s Valuation Creation Models . . . . . . . . . . . . . . . . . . . . . . . .91
Michael Porter’s Five Forces Analysis . . . . . . . . . . . . . . . . . . . . . . . . . .91
Michael Porter’s Value Chain Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . 93
Value Creation Process: Merchandising Example . . . . . . . . . . . . . . . . .94
This chapter provided several detailed, hands-on techniques for leveraging big data to impact your value creation processes. You were introduced to the four big data business drivers:
Access to more detailed, structured, transactional (dark) data
You worked through several generic examples across different industries to comprehend of how the four big data business drivers could impact the business.
This chapter then introduced the Big Data Envisioning Worksheet as a tool for brainstorming how the four big data business drivers could be applied to a specific business initiative. You walked through several business examples—predictive maintenance, customer satisfaction, and customer micro-segmentation—where you applied the four big data business drivers to identify areas of the business where big data could impact an organization’s value creation processes.
Next, you were introduced to Michael Porter’s Value Chain Analysis and Five Forces Analysis as two additional value creation frameworks that you can use to ascertain how the four big data business drivers could be applied to a specific business initiative.
Finally, you walked through a real-world example of how Foot Locker could leverage the four big data business drivers and the three different value creation models—Big Data Envisioning Worksheet, Porter’s Value Chain Analysis, and Porter’s Five Forces Analysis—to improve their merchandising effectiveness initiative.
All in all, I hope that this chapter felt like an introduction to the Big Data MBA.
The Unintelligent User Experience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
Understanding the Key Decisions to Build a Relevant User Experience . . . 107
Using Big Data Analytics to Improve Customer Engagement . . . . . . . . . . 108
Uncovering and Leveraging Customer Insights . . . . . . . . . . . . . . . . . . . . . 110
Rewiring Your Customer Lifecycle Management Processes . . . . . . . . . 112
Using Customer Insights to Drive Business Profi tability . . . . . . . . . . . 113
Big Data Can Power a New Customer Experience . . . . . . . . . . . . . . . . . . 116
B2C Example: Powering the Retail Customer Experience . . . . . . . . . . 116
B2B Example: Powering Small- and Medium-Sized Merchant
Effectiveness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
This chapter introduced you to the “unintelligent” user experience. Many organizations do not take advantage of the volumes of data they have captured about their customers—through initiatives like customer loyalty programs—to uncover customer and product insights that could power a more relevant and meaningful customer experience. Organizations that don’t take the time to learn, understand, and leverage those customer and product insights risk delivering irrelevant, confusing, and even frustrating user experiences that in the long-run damage the organization’s brand and decrease customer satisfaction and loyalty.
I defined a simple, pragmatic technique for identifying the information necessary to ensure a relevant, meaningful, and actionable user experience. The methodology ensures that your organization is providing the relevant data and insights to help your customers make the “right” decisions in their interactions with your organization.
Next, you considered how to leverage big data to improve your customer engagement processes. I discussed the importance of identifying where and how to apply big data through the entirety of your customer lifecycle process—from profiling, segmentation, targeting, acquisition, maturation, retention, and advocacy. Your organization has the opportunity to leverage customer and product insights to rewire your customer engagement processes which will improve customer profitability and drive long-term customer loyalty, as well as nurture customer advocacy.
Finally, you looked at a couple of mockups that demonstrate how your organization can leverage relevant, meaningful, and actionable insights to improve the customer experience. The mockups provided examples of how your organization can leverage insights and recommendations gleaned from big data sources to increase the value and “stickiness” of your customers’ relationships with your organization.
The Big Data Envisioning Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
Step 1: Research Business Initiatives . . . . . . . . . . . . . . . . . . . . . . . . . . 127
Step 2: Acquire and Analyze Your Data . . . . . . . . . . . . . . . . . . . . . . . 129
Step 3: Ideation Workshop: Brainstorm New Ideas . . . . . . . . . . . . . . . 132
Step 4: Ideation Workshop: Prioritize Big Data Use Cases . . . . . . . . . . 138
Step 5: Document Next Steps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
The Prioritization Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
The Prioritization Matrix Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142
Prioritization Matrix Traps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
Using User Experience Mockups to Fuel the Envisioning Process . . . . . . . . 145
This chapter reviewed in detail the vision workshop or envisioning process. I described each of the five steps in the vision workshop methodology and provided details on each step using real-world examples.
You spent quite a bit of time on the data preparation and analysis work required to transform business initiative-specific data into an envisioning exercise that can be used as part of the ideation workshop. This is an important part of the vision workshop methodology because it helps the envisioning process come to life for the workshop participants. I provided several examples of creating customer-specific envisioning exercises.
You learned about the brainstorming and aggregation process of the ideation workshop. You also reviewed how to use the Michael Porter value creation processes—-Value Chain and Five Forces Analysis—as well as the business initiative- specific envisioning exercise to tease out new business opportunities as part of the envisioning process.
You also learned how to use the Prioritization Matrix to drive agreement between the business and IT stakeholders around the right use cases on which to start the big data journey.
The chapter concluded with a discussion on how to leverage user experience mockups in the ideation workshop to further enhance the brainstorming process. I provided a couple of sample mockups and demonstrated how to use those mockups to drive the “what if” creative thinking process.
The Solution Engineering Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151
Step 1: Understand How the Organization Makes Money . . . . . . . . . 153
Step 2: Identify Your Organization’s Key Business Initiatives . . . . . . . . 155
Step 3: Brainstorm Big Data Business Impact . . . . . . . . . . . . . . . . . . . 156
Step 4: Break Down the Business Initiative Into Use Cases . . . . . . . . . 157
Step 5: Prove Out the Use Case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158
Step 6: Design and Implement the Big Data Solution. . . . . . . . . . . . . 159
Solution Engineering Tomorrow’s Business Solutions . . . . . . . . . . . . . . . . 161
Customer Behavioral Analytics Example . . . . . . . . . . . . . . . . . . . . . . . 162
Predictive Maintenance Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
Marketing Effectiveness Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
Fraud Reduction Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166
Network Optimization Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166
Reading an Annual Report . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
Financial Services Firm Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168
Retail Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169
Brokerage Firm Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
This chapter introduced you to the concept of solution engineering and provided a six-step process for going from opportunity identification to solution implementa¬tion. 1 provided several examples across different industries, highlighting how a business solution could leverage new sources of data and new big data technology innovations.
You then learned how to read an annual report (and other publicly available data sources) to identify an organization’s business initiatives where big data can provide material financial impact. You then reviewed several examples of review¬ing annual reports across different industries to identify how big data could impact those organizations’ key business initiatives.
Big Data: Time for a New Data Architecture . . . . . . . . . . . . . . . . . . . . . . . 173
Introducing Big Data Technologies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175
Apache Hadoop . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176
Hadoop MapReduce . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177
Apache Hive. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178
Apache HBase . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178
Pig . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178
New Analytic Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179
New Analytic Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180
Bringing Big Data into the Traditional Data Warehouse World . . . . . . . . . 181
Data Enrichment: Think ELT, Not ETL . . . . . . . . . . . . . . . . . . . . . . . . . 181
Data Federation: Query is the New ETL . . . . . . . . . . . . . . . . . . . . . . . 183
Data Modeling: Schema on Read . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184
Hadoop: Next Gen Data Staging and Prep Area . . . . . . . . . . . . . . . . . 185
MPP Architectures: Accelerate Your Data Warehouse . . . . . . . . . . . . . 187
In-database Analytics: Bring the Analytics to the Data . . . . . . . . . . . . 188
Cloud Computing: Providing Big Data Computational Power . . . . . . 190
This chapter began with a discussion of the transition from a traditional ETL, data warehouse, and B1 environment to a modern, big data-ready data management and analytics environment.
Next, you were introduced to some of the key big data technologies (Hadoop, MapReduce, Hive, HBase, and Pig) and considered some of the new data management and analytics capabilities being enabled by these new technologies. The chapter wrapped up with a discussion of how some of these new big data technologies, capabilities, and approaches can be used today to extend and enhance an organization’s existing investment in ETL, data warehousing, BI, and advanced analytics.
Explosive Data Growth Drives Business Opportunities . . . . . . . . . . . . . . . 194
Traditional Technologies and Approaches Are Insuffi cient . . . . . . . . . . . . . 195
The Big Data Business Model Maturity Index . . . . . . . . . . . . . . . . . . . . . . 197
Driving Business and IT Stakeholder Collaboration . . . . . . . . . . . . . . . . . . 198
Operationalizing Big Data Insights . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199
Big Data Powers the Value Creation Process . . . . . . . . . . . . . . . . . . . . . . .200
The Big Data Storymap provides a comprehensive and engaging metaphor around which to end this book (go here: www.wi 1 ey. com/go/bi gdataforbusi ness if you would like to download a PDF version of the Big Data Storymap). It helps to nurture that natural curiosity about what big data can mean to your organization, and helps you to envision the realm of what’s possible through a visual story. It summarizes many of the key big data best practices in a single graphic (see Figure 12-1) that you can share with your key stakeholders as you build the organizational support for your big data journey. You are now ready to launch your own big data journey. Go forth and be fruitful!
Identify Your Organization’s Key Business Initiatives . . . . . . . . . . . . . . . . . 203
Start with Business and IT Stakeholder Collaboration . . . . . . . . . . . . . . . .204
Formalize Your Envisioning Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .204
Leverage Mockups to Fuel the Creative Process . . . . . . . . . . . . . . . . . . . .205
Understand Your Technology and Architectural Options. . . . . . . . . . . . . .205
Build off Your Existing Internal Business Processes . . . . . . . . . . . . . . . . . .206
Uncover New Monetization Opportunities . . . . . . . . . . . . . . . . . . . . . . .206
Understand the Organizational Ramifi cations . . . . . . . . . . . . . . . . . . . . . . 207
Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209
