Groundbreaking Solution Enables Customers to Orchestrate and Run HPC and AI Workloads Across Decentralized Compute Clusters
Hammerspace, the company orchestrating the next data cycle, and Parallel Works, provider of the ACTIVATE control plane for AI and HPC resources, today unveiled a unified solution for compute and data orchestration. This groundbreaking solution enables IT infrastructure teams, research computing teams and AI teams to orchestrate compute, storage and data workloads seamlessly across sites and clouds, making it easy to scale cloud compute resources for burst capacity.
Also Read: From Afterthought to Agenda: Why Data Security Now Dominates the Boardroom
“Innovators need the agility to move complete workloads across different compute clusters – adding flexibility to use CPUs for data preparation, GPUs for processing, or accessing remote clusters for extra cycles or cost-efficient cloud regions. Seamlessly orchestrating both compute and data across locations and cloud regions is a game-changer”
“Innovators need the agility to move complete workloads across different compute clusters – adding flexibility to use CPUs for data preparation, GPUs for processing, or accessing remote clusters for extra cycles or cost-efficient cloud regions. Seamlessly orchestrating both compute and data across locations and cloud regions is a game-changer,” said David Flynn, Founder and CEO of Hammerspace. “Our unified solution with Parallel Works simplifies operations, reduces overhead and accelerates innovation by ensuring data is always where it needs to be, with minimal latency.”
“This partnership marks a major advancement in managing compute resources and data across hybrid and multi-cloud environments,” said Matthew Shaxted, CEO of Parallel Works. “By integrating our expertise in HPC and AI workload orchestration with Hammerspace’s parallel file system, global namespace and data objectives, we’re enabling application portability and achieving agile task placement. This approach empowers organizations to efficiently allocate resources where needed, driving performance, innovation, and simplified hybrid infrastructure management.”
AI and HPC workloads demand significant compute resources and low-latency access to large datasets. However, both data and compute resources are often distributed across different locations, making it challenging to process data efficiently. Leveraging alternate data centers or cloud infrastructure provides a scalable, elastic option for burst compute capacity, enabling on-demand scaling. Yet, historically, orchestrating workloads across multiple, geographically dispersed clusters – especially when data is remote from the compute – has been highly complex. This complexity arises from the need to relocate the application while managing data transfer, network latency and synchronization, leading to error-prone and time-consuming setups that most teams have been unwilling to take on the burden of managing.
Hammerspace and Parallel Works have co-developed a unified, integrated solution for compute and data orchestration that automates the provisioning and management of local and cloud compute resources and orchestrates the flow of data to those compute resources.
In this solution, Parallel Works ACTIVATE acts as the unified control plane that creates, provisions, and manages the compute resources located in multiple data centers and clouds. Parallel Works also automates workflows, such as spinning up Slurm instances to manage job scheduling and the associated HPC nodes.
Hammerspace acts as the data platform that provides high-performance file and object storage, creates a Global Namespace that spans sites and clouds, and automates the flow of data within that namespace with Data Orchestration services. Hammerspace delivers the data to the HPC servers regardless of where they reside, with orchestration policies set at the share, directory, and even individual file levels.
This integrated solution enables administrators to create, configure, and manage storage clusters from the Parallel Works ACTIVE Control Plane console.
By making it easier to use elastic cloud resources for burst computing capacity, organizations can “right size” their infrastructure with the ease of use of a fully managed cloud service but at the cost point of standard cloud infrastructure services. This approach can deliver significant cloud cost savings, as shown in the cloud storage cost comparison chart (see image) that compares the annual costs of Hammerspace on standard cloud virtual machines to Managed Lustre and shows the savings potential of the Hammerspace standards-based architecture.
