GPUs can be more than 200% faster than CPUs when parallel processing
Datadog, Inc., the monitoring and security platform for cloud applications, announced its new report, the State of Cloud Costs 2024. The report found organizations that use graphics processing unit (GPU) instances have increased their average spending on those instances by 40% in the last year. This growth in spend on GPU instances comes as more companies are experimenting with AI and large language models (LLMs). GPUs’ capacity for parallel processing makes them critical for training LLMs and executing other AI workloads, where they can be more than 200% faster than CPUs.
Also Read: Success of Dell Technologies: The Leadership Journey of Michael Dell
“Today, the most widely used type of GPU-based instance is also the least expensive. This suggests that many customers are still in the experimentation phase with AI and applying the GPU instance to their early efforts in adaptive AI, machine learning inference and small-scale training,” said Yrieix Garnier, VP of Product at Datadog. “We expect that as organizations expand their AI activities and move them into production, they will be spending a larger proportion of their cloud compute budget as they use more expensive types of GPU-based instances.”
In addition to more companies spending compute on AI projects, the report found that containers were a common theme of wasted spend among organizations. In fact, 83% of container costs were associated with idle resources. About 54% of this wasted spend was on cluster idle, which is the cost of overprovisioning cluster infrastructure, while 29% was associated with workload idle, which comes from resource requests that are larger than their workloads require. This wasted spend comes as organizations allocate more of their EC2 compute to running containers, up to 35% compared to 30% a year ago.
Other key findings from the report include:
- Outdated Technologies Are Widely Used: AWS’s current infrastructure offerings commonly both outperform their previous-generation versions and cost less, but 83% of organizations still spend an average of 17% of their EC2 budgets on previous-generation technologies.
- Fewer Organizations Are Taking Advantage of Discounts: Cloud service providers offer commitment-based discounts on many of their services—for example, AWS has d******* programs for Amazon EC2, Amazon RDS, Amazon SageMaker and others—but only 67% of organizations are participating in these discounts, down from 72% last year.
- Green Technology Is on the Rise for Better Performance and Cost: On average, organizations that use Arm-based instances spend 18% of their EC2 compute budget on them—twice as much as they did a year ago. Instance types based on the Arm processor use up to 60% less energy than similar EC2s and often provide better performance at a lower cost.
- For the report, Datadog analyzed AWS cloud cost data from hundreds of organizations and explored how their use of emerging and previous-generation technologies, patterns of cloud resource usage, and participation in AWS d******* programs all contributed to their cloud costs.
Also Read: How CIOs Can Simplify CRM Transformation
[To share your insights with us as part of editorial or sponsored content, please write to psen@itechseries.com]
