LakehouseIQ is a knowledge engine that uses generative AI to learn what makes an organization’s data unique and provides useful, contextual answers to natural language questions
At the sold-out Data + AI Summit, Databricks, the Data and AI company, announced LakehouseIQ, a knowledge engine that learns what makes an organization’s data, culture and operations unique. LakehouseIQ uses generative AI to understand jargon, data usage patterns, organizational structure, and more to answer questions within the context of a business. Anyone in an organization can interact with LakehouseIQ using natural language to search, understand, and query data. LakehouseIQ is fully integrated with Databricks Unity Catalog to help ensure that democratizing access to data adheres to internal security and governance rules.
The AI-Powered Knowledge Engine that Uniquely Understands your Business
Today, every business must be data-driven. Yet, when employees need access to internal data to help complete their tasks, it can be difficult to quickly find and analyze the data they need. Whether the CEO is trying to build quarterly sales forecasts or a marketer is attempting to analyze campaign performance, knowledge workers rely on a small team of over-worked data scientists and programmers to find and query the relevant data sets. This bottleneck prevents businesses from truly embracing data and AI. Large language models (LLMs) promised to fix this problem, but so far, the results have been disappointing. General purpose models don’t understand the unique language of every business: they cannot process jargon or internal acronyms; they are not trained on the company’s unique data sets; and they do not understand organizational charts or know which teams should have access to what information.
CIO INFLUENCE News: Thales Partners with Google Cloud to Build Generative AI-Powered Security Capabilities
LakehouseIQ learns from these signals within an organization using schemas, documents, queries, popularity, lineage, notebooks, and BI dashboards to gain intelligence as it answers more queries. LakehouseIQ helps employees get immediate answers to questions without requiring that they possess the technical skill required by traditional data analysis tools. The engine understands their unique business jargon and context to more accurately interpret the intent of the question, and can even generate additional insights that could spur new questions or lines of thinking. With LakehouseIQ, every employee — not just data scientists — can unlock the full potential of internal corporate data to make better, more informed decisions and drive new innovations.
“LakehouseIQ will help democratize data access for every company to improve better decision-making and accelerate innovation. With LakehouseIQ, an employee can simply write a question and find the data they need for a project, or get answers to questions relevant to their company’s operations. It removes the roadblocks inherent in traditional data tools and doesn’t require programming skills,” said Ali Ghodsi, Co-Founder and CEO at Databricks. “Every employee knows the questions to ask to improve their day-to-day work and ultimately their business. With LakehouseIQ, they have the power to quickly and accurately discover the answers.”
CIO INFLUENCE News: Matillion Announces the End of Slow, Fragmented, Expensive Data Pipelines
Securely Democratizing Data Access
LakehouseIQ is governed by Unity Catalog, Databricks’ flagship solution for unified search and governance across data, analytics and AI. Organizations can be confident that their employees will only have access to the data they are authorized to use, so increasing data accessibility doesn’t increase risk.
“LakehouseIQ solves two of the biggest challenges that businesses face in using AI: getting employees the right data while staying compliant and keeping data private when it should be,” said Ghodsi. “It alleviates time-strapped engineers, eases the burden of data management, and empowers employees to take advantage of the AI revolution without jeopardizing the company’s proprietary information.”
Databricks continues to expand the Lakehouse Platform, recently announcing Lakehouse Apps and the general availability of Databricks Marketplace, Delta Lake 3.0, new governance capabilities, and a suite of data-centric AI tools for building and governing LLMs on the lakehouse.
CIO INFLUENCE News: BigID Brings Privacy and Security Context, Powered by Snowflake, to the Data Cloud
[To share your insights with us, please write to sghosh@martechseries.com]
