OCTOBER 2018

A Forrester Consulting Thought Leadership Paper Commissioned By IBM

Don’t Get Caught Waiting On Fast Data

How Fast Data Solutions Are Pointing To The Future Of Data Analytics

Executive Summary

In today’s digital world where enterprises are awash in critical information, one thing is certain: Data drives everything — fast. Business stakeholders at the executive and line-of-business levels need their data faster to keep up with customers, competitors, and partners.

However, effectively collecting and analyzing data is not as simple as it once was. With the rise of fast data, modern enterprises must now handle data in higher quantities, frequencies, and variations. Current implementations simply aren’t working quickly enough for firms to remain competitive — so what’s next? To reap the benefits these insights can offer, enterprises require solutions that will ingest and analyze data at quicker speeds, resulting in higher-quality data insights and thus higher levels of customer satisfaction.

In September 2018, IBM commissioned Forrester Consulting to evaluate the state of data and analytics strategies. Forrester conducted an online survey of 253 US and EMEA enterprises to explore this topic. We found that effective data management has become more complicated in recent years. Enterprises look to fast data solutions to help, but challenges prevent them from achieving the necessary functionalities to reap business benefits.

Key Findings

  • Over 25% of companies use fast data for most or all of their applications, and more than 75% use it for at least some of their applications.

  • Enterprises look to fast data solutions to help, but they often lack the required functionalities to reap business benefits.

  • Modern enterprises thrive on fast data.

    The use of fast data is on the rise: Over one-quarter of firms are using fast data for most or all of their applications, and more than three-quarters use fast data for at least some of their applications. Firms realize that their data is becoming both more strategically crucial and more difficult to manage.

  • Fast data solutions are the answer to this complexity, but many miss the mark

    The majority of enterprises surveyed say they need to analyze their data within a day or less, yet many current solutions process data at a much slower rate than these demands require. What’s more, these solutions don’t support enterprises’ necessary actions, which can lead to detrimental business and financial issues. Considering that almost 90% of organizations report that they typically require their data to be ingested and analyzed within one day or less, and that 88% need to perform analytics in near-real time on stored streamed data, it’s clear that businesses cannot afford to wait around for the insights that fuel growth.

  • Enterprises know they must rethink their fast data strategy to remain competitive.

    Respondents surveyed are aware of the need to revamp their data strategies to remain competitive and prefer commercially enhanced open source, hybrid cloud solutions to do so. Furthermore, machine learning is already a feature of many enterprise solutions, and its use is set to expand in the near future.

Fast Data Powers The Modern Enterprise

This study defines fast data as:

Any type of data that originates in applications and devices and is streamed, stored, and immediately analyzed by applications that detect patterns, automate decisions, and immediately initiate actions executed through a business or operations process and/or customer-facing applications.

Modern enterprises live in a real-time world, where the moment between capturing data and having an opportunity to apply it is both shrinking and becoming more crucial to business success. All data originates in real time, whether from the click of a mouse, a sensor reading on a device, a transaction in a database, or a market feed.1 In this study, we found that businesses are looking to match this need with fast data solutions that can help them keep up with the pace of modern, digital business.

AS DATA BECOMES A STRATEGIC PRIORITY, IT ALSO BECOMES INCREASINGLY VARIED AND COMPLICATED TO MANAGE

Three-quarters of study respondents say that data from mobile sources and data from internet-of-things (IoT) sources are a high or critical priority for their firms’ data strategies. Nearly three-quarters agree that internal data sources from enterprise or operational applications are a high priority, and more than two-thirds agree that data from external sources like third-party data providers or customers is highly or critically important.

At the same time, managing all of these data sources has become more complicated over the past three years, as the variety of data sources, the volume of data coming in, and the velocity of that data have all increased significantly for most firms (See Figure 1).

Figure 1: “How much of a strategic priority are each of the following data sources for your firm’s data science, structured data, and analytics strategy?”

As data variety, volume, and velocity rise, most enterprises say that data from an eclectic mix of internal and external sources must be treated as a high or even critical strategic priority.

Click to see data by region

Base: 253 data/analytics decision makers at US/EMEA enterprises
Source: A commissioned study conducted by Forrester Consulting on behalf of IBM, July 2018

Figure 1 (Cont.): “How much do you agree or disagree with the following statements about data management?”

Click to see data by region

Base: 253 data/analytics decision makers at US/EMEA enterprises
Source: A commissioned study conducted by Forrester Consulting on behalf of IBM, July 2018


ENTERPRISES FEEL THAT PROFITABILITY AND REVENUE ARE NOT FULLY SUPPORTED BY CURRENT DATA STRATEGIES

The organizations in our study are focused on driving bottom-line objectives this year — including improving profitability, growing revenues, and increasing the number of and driving more satisfaction for customers. Success depends on data speed, so firms must explore solutions that enable their businesses to keep up with the influx of data that powers their activities. More than three-quarters of the enterprises in our study say they are using fast data for at least some of their applications today, with 26% saying fast data is in use for most or all of their applications (see Figure 2 ).

While study participants by and large feel their current data and analytics strategies support many of these key business objectives, there is some room for improvement on how well their strategies drive profitability and revenue support:

  • Improving profitability.

    Only 40% believe profitability is very well supported.

  • Growing revenues.

    Only 27% of organizations believe revenue growth is very well supported.

  • Improving customer satisfaction.

    Just over half of organizations (52%) believe customer satisfaction goals are very well supported.

  • Increasing the number of customers.

    Nearly half of organizations (49%) believe increasing the number of customers is very well supported.

Instant Poll

To what extent is your organization leveraging fast data today?

-----------------------------------------------------------------------------------------------------------------------------

To what extent is your organization leveraging fast data today? (Select one)

Thank you for your response, please see below for how you compare to the study’s respondents.

DID YOU KNOW?

Organizations suggest their current data/analytics strategies support their customer-centric objectives but provide less robust support for profitability and revenue objectives.

Base: 253 data/analytics decision makers at US/EMEA enterprises
Source: A commissioned study conducted by Forrester Consulting on behalf of IBM, July 2018

Figure 2: “To what extent is your organization leveraging fast data today?”

Click to see data by region

Base: 253 data/analytics decision makers at US/EMEA enterprises
Source: A commissioned study conducted by Forrester Consulting on behalf of IBM, July 2018


Complexity, Lack Of Integration Hinder Effectiveness Of Fast Data Solutions

Enterprises are awash in data, which is arriving from more sources, in greater quantity, and more quickly than ever before. This makes speed incredibly important to firms managing and using their data, but current fast data implementations may not be providing the support these enterprises need. For example, 88% of firms say they need to perform analytics in near-real time on stored streamed data, and almost 80% report that they have streamed data in data storage currently.

SPEED IS KING AS ORGANIZATIONS CAN’T WAIT ON DATA — BUT IT’S NOT THE ONLY CONSIDERATION

This study asked what organizations think are the most important considerations for their fast data solution and found that, while speed was the leading factor, it wasn’t the only one. Just over a third report that speed, or a solution that could analyze data in real time or near-real time, is the most important consideration, followed by the ability to use fast data analysis in applications, storage volume, and ingestion rate. However, the second-ranked option shows the demands placed on fast data solutions often transcend speed: 21% say that they effectively couldn’t choose just one area, that all of these considerations were equally critical for them (see Figure 3).

Furthermore, enterprises can’t afford to wait around on their data. Nearly nine in 10 organizations report that they typically require their data to be ingested and analyzed within one day or less, and that delays in data processing and analysis lead to significant business issues, including:

  • Missed opportunities to act on perishable insights.

    Over 50% of enterprises say that delays in data ingestion and analysis would cost them the opportunity to react in near-real time to significant business situations.

  • Lower customer satisfaction.

    Forty-one percent say that they worry data delays would negatively impact customer satisfaction.

  • Longer and more frequent downtime.

    Thirty-seven percent of firms are concerned about idling resources caused by data ingestion or analysis delays.

  • Falling behind trends.

    Quickly responding to insights drives the power to identify and plan a response to market trends. Thirty-six percent are concerned data delays would cost them the opportunity to be ready.

  • Loss of competitive advantage.

    Falling behind on data and insights means falling behind your competition. Thirty-five percent say that data delays would lead to loss of competitive advantage.

Figure 3: “Which of these would you define as most important for your fast data solution?”

While speed is paramount, many organizations need their fast data solution to excel at a blend of speed, application use, storage volume, and ingestion rate.

Click to see data by region

Base: 253 data/analytics decision makers at US/EMEA enterprises
Source: A commissioned study conducted by Forrester Consulting on behalf of IBM, July 2018

MANY ORGANIZATIONS FEEL THEY RECEIVE PARTIAL SUPPORT FROM PARTIALLY INTEGRATED DATA AND ANALYSIS TOOLS
85% of enterprises say their data and analytics tools are only “somewhat integrated” at best.

Challenges associated with delays in data ingestion and analysis can arise in part out of data and analytics tools that aren’t properly integrated with one another. Indeed, less than half of organizations in our study feel their tools are somewhat integrated at best, and only 15% believe their tools are completely integrated today. Poor integration leads to missed opportunities for timely, effective data usage; lower productivity; and other concerns (see Figure 4).

Figure 4: “What issues have you experienced or would you expect to see as a result of having tools that are not integrated?”

Click to see data by region

Base: 253 data/analytics decision makers at US/EMEA enterprises
Source: A commissioned study conducted by Forrester Consulting on behalf of IBM, July 2018

MANY FAST DATA SOLUTIONS OFFER ONLY PARTIAL SUPPORT FOR KEY CAPABILITIES

Fast data solutions are intended to help enterprises match the intensity of modern enterprises’ data input and analytics needs, but respondents tell us that solutions are still developing to the point where their higher- level needs will be met. For example, less than half feel their solution is able to deliver data analysis, real-time analytics, or the ability to view results in real time. And less than a third say their solutions provide data storage on the same platform, continuous integration, data ingestion/ processing, or data throughput (see Figure 5).

Furthermore, more than half of respondents say the support they do receive on these capabilities has room for improvement or even that the capabilities are poorly supported.

Figure 5: “Which of the following functions/capabilities does your current fast data solution support for high volumes of data?”

More than half of enterprises who use fast data solutions say there is room for improvement.

Click to see data by region

Base: 253 data/analytics decision makers at US/EMEA enterprises
Source: A commissioned study conducted by Forrester Consulting on behalf of IBM, July 2018


Enterprises Will Focus On Refining Their Data Strategies To Improve The Business

As the importance of fast data solutions rises for enterprises, many are realizing they need to reexamine their data strategies to keep up. Nearly nine of 10 enterprises plan to either set or reevaluate their fast data analytics strategy over the next two years (see Figure 6).

Organizations that are looking to set up a fast data strategy for the first time tell us they want to leverage these capabilities to make better-informed decisions, improve their data quality and consistency, and drive financial benefits like increased revenue and reduced operational costs.

Interestingly, firms that have implemented fast data strategies but are looking to reevaluate them report that their chief benefits include gaining deeper insights from their data, enjoying improved data quality and consistency, creating a better customer experience, and being able to make better-informed business decisions using insights from analytics.

Instant Poll

Do you plan to adopt or reevaluate your company’s fast data analytics strategy within the next two years?

Do you plan to adopt or reevaluate your company’s fast data analytics strategy within the next two years? (Select one)

Thank you for your response, please see below for how you compare to the study’s respondents.

DID YOU KNOW?

The vast majority of organizations are planning to either adopt or reevaluate their fast data analytics strategies in the near future.

Base: 253 data/analytics decision makers at US/EMEA enterprises
Source: A commissioned study conducted by Forrester Consulting on behalf of IBM, July 2018

Figure 6: “Do you plan to adopt or reevaluate your company’s fast data analytics strategy within the next two years?”

Click to see data by region

Base: 253 data/analytics decision makers at US/EMEA enterprises
Note: Percentages do not total 100 because of rounding
Source: A commissioned study conducted by Forrester Consulting on behalf of IBM, July 2018


ENTERPRISES PREFER TO GEAR UP FOR FAST DATA WITH COMMERCIALLY ENHANCED OPEN-SOURCE, HYBRID CLOUD SOLUTIONS

As enterprises prepare to start or reapproach their fast data capabilities in the coming years, they will necessarily turn to new and more powerful solutions that can provide the support and access to benefits that will help these programs pay off. Study participants have specific preferences for how these tools should be set up (see Figure 7):

  • Enterprises prefer commercially enhanced open source solutions.

    Leading the available options, 49% of enterprises say they would prefer to use a commercially enhanced open source solution that comes with support and full integration when choosing a fast data solution.

  • They are most interested in a hybrid cloud and on-premises approach.

    Enterprises also tell us they are most likely to prefer a hybrid could and on-premises approach to their fast data solutions, as it would best meet their various needs.

Figure 7: “What deployment options would your firm prefer for a fast data solution?”

Enterprises are turning to solutions that feature commercially enhanced open source delivery, with a hybrid cloud and on-premises architecture to meet their fast data needs.

Open source / Commercial

Click to see data by region

Cloud / On premises

Click to see data by region

Base: 253 data/analytics decision makers at US/EMEA enterprises
Source: A commissioned study conducted by Forrester Consulting on behalf of IBM, July 2018


MACHINE LEARNING TO PUSH THE FRONTIER OF FAST DATA EFFICIENCY FOR STORED AND STREAMED DATA

Study participants have big plans for applying machine learning capabilities to their fast data analytics. More than three-quarters of firms are either currently (45%) or planning (32%) to apply machine learning to their stored data today, whereas 73% are either currently (33%) or planning (40%) to use it for streamed data in the near future (see Figure 8). Organizations would do well to ensure that they factor in machine learning as a consideration when they create or revamp their fast data strategies in the near term (see Figure 8).

Figure 8: “Which of the following best describes how your organization is approaching machine learning for each component of fast data analytics?”

Machine learning has arrived for many firms and will be in place for both stored and streamed data analytics in the very near future.

Click to see data by region

Base: 253 data/analytics decision makers at US/EMEA enterprises
Source: A commissioned study conducted by Forrester Consulting on behalf of IBM, July 2018


Key Recommendations

Fast data matters. It is fuel for real-time intelligence (and artificial intelligence) for next-gen digital business processes and customer experiences. The demands on both hardware and software infrastructure will expand exponentially due to the real-time nature of digital services; the compute intensity of analytics and machine learning algorithms; and the volume of data that must be stored. To meet the demands of fast data, enterprises must implement fast data solutions strategically by:

  • Establishing realistic requirements for storage and compute.

    Most enterprises understand storage and compute requirements for traditional analytics — reports and dashboards —since that has been the main use of data for at least 30 years. Fast data is different because it powers real-time insights, decisions, and machine learning. Hardware and software infrastructure must expand to handle fast data workloads. IT professionals should consult with data science, AI engineering, and application development teams to forecast storage and compute needs for next-gen, real-time digital services.

  • Evolving to a real-time architecture.

    Enterprise architects must use fast data solutions as both the engine and the glue to evolve existing data and application architectures to enable application developers, data scientists, and AI engineers to create intelligent, real-time services.

  • Choosing a fast data solution that leverages open source.

    The open source community continues to be a major source of innovation for both big data and machine learning. Enterprise architects should choose fast data solutions that leverage open source innovation, but that are enhanced to easily manage, scale, and mesh with existing data and application architectures.

  • To learn about new solutions for fast data architectures, click here for resources from IBM, or go to ibm.biz/fastdata.


Appendix A: Methodology

In this study, Forrester conducted an online survey of 253 US and EMEA enterprises to evaluate the state of data and analytics strategies. Survey participants included decision makers in IT and business user roles. Questions provided to the participants asked about their current data and analysis strategies and use of fast data. Respondents were offered a small monetary incentive as a thank you for time spent on the survey. The study began in March 2018 and was completed in September 2018.


Appendix B: Demographics/Data


Appendix C: Endnotes

1 Source: “The Forrester Wave: Streaming Analytics, Q3 2017,” Forrester Research, Inc., September 7, 2017.