A lakehouse architecture for unstructured logs that enables organizations to search first, optimize later, and dramatically reduce observability and SIEM infrastructure costs
Imply announced Lumi Loglake, an industry-first capability for Imply Lumi that enables enterprises to search unstructured logs directly in object storage.
Also Read: CIO Influence Interview with Hugo Dozois-Caouette, CTO and Co-founder at MaintainX
As AI systems, cloud infrastructure, and security workloads generate larger volumes of telemetry data, organizations are increasingly adopting lakehouse architectures and open storage environments to control infrastructure costs and retain more data. But observability and security teams work with unstructured machine data, where investigations often begin without knowing in advance which fields, patterns, or events will matter.
Showcased this week at the Databricks Data + AI Summit, Lumi Loglake enables organizations to search and investigate unstructured logs directly where they are stored—including AWS S3, Delta Lake, and Apache Iceberg—without requiring data catalogs, schema definitions, or rehydration workflows.
Built for Today’s Economics of AI-scale Telemetry
“AI is generating more telemetry than organizations can afford to index,” said Eric Tschetter, Chief Architect at Imply. “Lumi Loglake gives teams a new way to retain, search, and investigate that data without the cost and complexity of always-on indexing architecture. By bringing interactive log search directly to open storage, Loglake applies the economics and flexibility to lakehouse architectures to operationalize log data.”
Traditional observability architectures often require logs to be cataloged, structured, and indexed before they become searchable. That model made sense when data volumes were manageable. At AI scale, it forces teams to make upfront decisions about what to retain and index before they know what will matter, creating operational overhead and making costs difficult to control.
Lumi Loglake separates compute from object storage, allowing organizations to scale telemetry retention independently from infrastructure costs. Compute resources provision dynamically based on query activity, eliminating the need for always-on indexing infrastructure while keeping historical telemetry instantly accessible.
Teams can retain significantly more data in object storage, search it immediately, and apply indexing only where it delivers value.
Lower Infrastructure Costs and a Simpler Path Beyond Traditional Observability Architectures
Organizations using Lumi Loglake can reduce software costs by 70% or more and hardware costs by 40% or more by moving data off always-on indexed storage and into low-cost open storage environments such as Amazon S3.
For Splunk customers, that represents a practical migration path. Rather than keeping large volumes of data inside Splunk’s indexing tier, organizations can retain historical telemetry in S3 while continuing to query it using SPL and receive native Splunk events. Analysts keep the experience they know while organizations gain the flexibility and economics of open storage.
Teams can query the same datasets across multiple platforms without duplicating storage, including Splunk using SPL, Databricks using Spark SQL, Grafana using LogQL, and AI and BI platforms using ANSI SQL/JDBC.
Lumi Loglake Reflects a Broader Shift in Observability
As teams rethink how telemetry data is stored, retained, and operationalized at scale, industry analysts see growing demand for more flexible observability architectures.
“Traditional observability pricing models are forcing teams into visibility tradeoffs at the exact moment AI systems are driving unprecedented telemetry growth,” said Stephen Catanzano, Principal Analyst at Omdia. “As infrastructure complexity and data volumes continue to rise, organizations are looking for more scalable approaches that improve operational flexibility without significantly increasing costs.”
“Observability and SIEM require analyzing mountains of data without prohibitive cost or complexity,” said Kevin Petrie, Vice President of Research at BARC. “Lumi reduces this tradeoff by searching logs in object storage without the overhead of indexing, cataloging, or schemas. Collecting all these logs in a cloud lakehouse also creates opportunities beyond observability, because telemetry data can enrich other analytics and AI initiatives.”
Loglake addresses that challenge by making data stored in object storage operationally accessible for observability and SIEM investigations without requiring teams to build and maintain additional indexing infrastructure.
Catch more CIO Insights: What Does “Job-Ready” Really Mean in IT and Cybersecurity?
[To share your insights with us, please write to psen@itechseries.com ]
