Today’s encryption methods – from AES to TLS – were designed long before AI and high-speed, data-driven computation became central to business operations. They excel at protecting data at rest (stored on disk) and in motion (transmitted across networks), but they weren’t built to safeguard data in use, which includes data actively processed by AI systems.
This distinction matters profoundly. To achieve truly secure or compliant AI, organizations need protection that extends beyond the traditional “before and after.” They need technology that keeps data secure even while it’s being computed on, akin to how Secure Sockets Layer (SSL) technology secured the early Internet. Without such an advancement, the potential for catastrophic data leakage is enormous.
Also Read: CIO Influence Interview with Carl Froggett, Chief Information Officer (CIO) at Deep Instinct
The Fatal Flaw: The Memory Exposure Problem
In a typical AI workload, data usually remains in memory at all times. For performance reasons, this data is not encrypted. As a result, while the data is stored in the GPU’s memory (VRAM)—which is essentially the entire time the AI system is running—it stays in plaintext and is therefore vulnerable.
This creates an opportunity for attackers to steal your AI model, prompts, related documents, and even the AI’s responses.
The memory exposure problem runs deeper than many executives realize. During AI processing, plaintext data and model weights are loaded into the system’s Random Access Memory (RAM) or Video Random Access Memory (VRAM) on a Graphics Processing Unit (GPU) for computation. Unlike traditional applications that may clear memory after use, most AI frameworks keep this data resident in memory long after inference completes to maximize performance and efficiency. This performance shortcut creates a severe exposure in two ways:
- Lurking Data: Prompts, embeddings, and intermediate activations can linger in GPU VRAM for hours or even days. An attacker who gains control can dump memory (copy the contents of volatile memory to persistent storage) and recover sensitive data long after the transaction is complete. Memory scraping doesn’t have to be real-time; historical secrets are still there for the taking.
- Interconnect Backdoors: In multi-GPU and distributed inference setups, most AI frameworks still do not encrypt data as it moves between GPUs on the same server or across nodes. High-speed interconnects like NVLink and PCIe transfer raw, unencrypted tensors between devices for maximum throughput.
That performance shortcut comes with a heavy security cost. In most cloud or multi-tenant environments, system administrators, cloud operators, and third-party service engineers have access to bus traffic or driver-level hooks. A single compromised account at that level could capture model weights or input data in cleartext as it flows through the cluster.
In essence, traditional encryption only protects data before and after computation. During inference, when the data is actually being used, it remains dangerously exposed.
The New Standard: Cryptographic Guarantees for AI
To close this persistent vulnerability, protection must extend into the runtime environment so sensitive data and models never appear in plaintext at any stage. This requirement is driving the development and adoption of advanced cryptographic frameworks, known broadly as Confidential AI.
Confidential AI leverages technologies like Fully Homomorphic Encryption (FHE) and Trusted Execution Environments (TEEs) to achieve this “in-use” security:
- FHE for Computation on Encrypted Data: FHE is a groundbreaking technique that enables computation on encrypted data. The data never needs to be decrypted into plaintext, even when running through the GPU. This means that even if the entire machine is compromised, the attacker only sees encrypted noise.
- TEEs for Key and Code Isolation: TEEs (hardware-enforced secure enclaves) are used to isolate the core encryption keys and model execution logic from the underlying operating system and hypervisor. This prevents cloud operators or malicious system administrators from accessing the environment’s most critical secrets.
By implementing these technologies, organizations achieve true “in-use” encryption:
- Model and Inputs Remain Encrypted: All proprietary model weights, user inputs, and intermediate calculations remain in ciphertext while residing in memory.
- Key Isolation: Encryption keys are never exposed in the general operating system space.
- End-to-End Protection: In multi-GPU setups, data moving between processors remains encrypted, eliminating the interconnect as a viable attack surface.
Why This Matters for Business and Compliance
The demand for “in-use” security is driven by more than just technical best practice; it’s a strategic necessity for compliance and preserving enterprise value. Consider the following:
- Compliance: Regulations such as GDPR, HIPAA, and CCPA require organizations to protect data throughout its lifecycle, including processing. Without cryptographic guarantees during inference, compliance is fundamentally at risk.
- Intellectual Property (IP) Protection: For businesses whose primary value lies in their proprietary models (or the data used to fine-tune them), in-use encryption prevents model inversion and theft by insiders or malicious partners in multi-tenant cloud environments.
- Market Confidence: Early adopters of robust Confidential AI frameworks gain a competitive edge, demonstrating to customers and partners that they are willing to apply the highest cryptographic standard to sensitive data, wherever it resides.
“In use” encryption is not an optional extra; it is the logical next step in AI security. It is the only way to deploy AI in the cloud, across borders confidently, and in multi-tenant settings – knowing your secrets stay secret, always.
Catch more CIO Insights: AI Governance: The Catalyst for Safe and Confident Innovation
[To share your insights with us, please write to psen@itechseries.com ]
