As AI/ML Use Cases Expand, Firms Must Put Workloads Where They Belong
We are in the nascent stages of AI and ML, both as an architectural and business discipline. Many organizations have started experimenting with AI/ ML on a small scale, with use cases such as customer recommendations, targeted advertising, and predictive analysis. These organizations run on either commodity hardware or a few machines with devoted GPU hardware in the data center; alternatively, third-party cloud providers offer the needed systems. In both cases, while easing setup, this takes much of the control and strategy out of the hands of enterprise architects.
As more advanced use cases such as image recognition, speech recognition, self-automation, and others become widespread, the hardware needed to efficiently and effectively run these use cases — as well as workload placement — must evolve. To find out how architecture will change as use cases advance, we surveyed 200 IT and business professionals that manage architecture or strategy for AI and ML at large enterprises in the US and China. We found that:
-
Advanced analytics are being run across the enterprise today.
Seventy-one percent of surveyed banking providers said that consumers choose to use multiple products/services from them because of the trusted relationship they have together — a sentiment shared by only 37% of consumers. The bottom line is that consumers want simplicity and convenience, both of which are available by expanding their relationship with a single banking provider, rather than having to be onboarded as a new customer somewhere else.
-
Far fewer organizations run dedicated hardware for AI training and inference.
Only 28% of surveyed organizations are using hardware for training AI/ML models on-premises or mixed with third-party cloud providers (see Figure 2). Far more common, 42% exclusively use third-party cloud providers for their current AI/ML model training, due to ease of use, tool kits, reduced maintenance, etc. A further 29% are not using specialized hardware or using it at a very small scale.
-
Those who leverage specific hardware solutions are finding that some assembly is required.
Components of dedicated hardware for AI require customization that off-the-shelf offerings can’t yet provide. Close to half of respondents say they source, or plan to source, AL/ ML hardware components by buying and customizing or integrating them to address this.
PUT THE WORKLOAD WHERE IT BELONGS: THE IMPORTANCE OF AI/ML LOCALITY
When architecting hardware specifically for AI/ML, survey respondents say that the location of compute and memory is a crucial component of performance and success. Eighty-nine percent of respondents say that it is important that compute and memory are architecturally close together (see Figure 3). This is even more essential for organizations that analyze data outside of their own data centers or the cloud — 51% of companies analyzing data sets at the edge say locality is critical, compared to just 37% who are not. As these more advanced use cases continue to grow in popularity, this idea will play a greater role in how hardware solutions are architected.
Memory And Storage Bottlenecks Limit AI/ML Analytics
Throughput and performance are two of the most important aspects of memory and storage when it comes to both AI/ML training and inference. While most organizations are satisfied with the kind of analytics their current hardware can support today, there are some architectural bottlenecks that limit throughput and performance and as a result, bottleneck AI/ML analytics. Our survey shows that:
-
Enterprises recognize a need to upgrade or rearchitect their memory and storage.
For AI/ML training, 79% say upgrading or rearchitecting memory is important or critical, while 76% say it is important or critical to do the same with storage (see Figure 4).
-
Storage and memory limit both training and inference performance and throughput.
Available memory and storage are the top two hardware-related challenges for AI/ML training and inference today, with close to two-thirds of respondents indicating that they are bottlenecks to performance and throughput (see Figure 5).
-
These challenges escalate as AI/ML moves more to the edge.
Data centers are not always the ideal location for all AI/ML workloads. Respondents understand this and predict that they will move workloads away from data centers to the public cloud and edge in the next three years (see Figure 6). When this happens, the location of workloads’ compute and memory resources becomes even more important and firms must pay greater attention to how their hardware is architected.
HARDWARE CHALLENGES ARE NOT THE ONLY BARRIER TO AI/ ML SUCCESS
Architects are running into issues managing infrastructure, as well as working with data privacy and governance challenges (see Figure 7).
-
Most lack skills to implement and manage infrastructure.
As firms tackle more advanced use cases for AI and ML, the standard toolkits and base models provided by cloud vendors and third parties are not sufficient. Firms need to implement custom infrastructure to support these use cases, and the skills to do that are in low supply. Over 50% of firms say that they do not have the skills to implement or manage the hardware for both AI/ML training and inference.
-
Data privacy and governance challenges grow.
Across the globe, there is growing concern about how data, especially personal data, is used by companies. Regulations like GDPR highlight this and foreshadows additional regulations to come. Many current AI/ ML practices are based on potentially sensitive data and further regulations may hinder those use cases. Concerns over privacy and security requirements hindering the effective use of AI/ML is the number one concern when it comes to training AI/ML. Organizations need to review their data governance practices for potential data privacy challenges and make changes that will allow for the effective use of AI/ML.
Rearchitect Memory And Storage To Reap Rewards Of Better AI/ML Architecture
Despite the current challenges, the future is bright for AI/ML analytics. New use cases and models are being built every day, and firms are just scratching at the surface of neural network potential. While many are in the experimentation phases with AI/ML today, almost all plan to expand their footprint. In the next three years, 89% of survey respondents plan to run mixed complex data set workloads — a sign of growing maturity in handling diverse data sets. Rearchitecting memory and storage for AI/ML is seen as a core component of advancing AI/ML practices and will lead to success for the future. Our survey shows:
-
Organizations will adopt a balance between on-premises and cloud architecture for AI/ML.
While today, close to half (42%) of organizations use only third-party cloud for their analytics, most plan to move some of those workloads on-premises, with hardware designed for AI/ML (see Figure 8). Fifty-six percent of organizations see themselves using hardware built for AI/ML both on-premises and in the cloud in the next three years. This follows the emerging trend of “putting the workloads where they belong” — placing workloads in the ideal location depending on the data used, the use case, and other determining factors. This also means that many organizations will be building hardware designed for AI/ML for their data centers in the near future.
-
Organizations understand the importance of moving memory and computing closer.
Moving memory and compute closer together for AI/ML workloads is seen as essential to success by our respondents. Ninety percent of firms plan to move computing and memory closer together to improve AI/ML workloads in the future (see Figure 9). This is especially true of organizations that are experimenting with AI/ML outside of the cloud or data center. Of respondents who are currently running advanced analytics at the edge, this rises to 95%.
