For the first time, an open data lake delivered better performance than traditional data warehouses, making the lakehouse vision a reality.
Databricks, the Data and AI company, announced that it has set a world record for the official 100 terabyte TPC-DS benchmark, the gold standard to evaluate the performance of data warehouse systems. Unlike most other benchmark results, the results were audited and made public by the official Transaction Processing Performance Council (TPC) that organizes TPC-DS. According to the council, Databricks outperformed the previous world record holder by 2.2x. A separate research study conducted by Barcelona Supercomputing Center (BSC) compared Databricks and Snowflake and found that Databricks was 2.7X faster and more than an order of magnitude cheaper on the same workload. Read Databricks blog on how the team built the engine that could achieve these results.
For the first time, Databricks has shown that the data lakehouse architecture built on top of vast amounts of data stored in open data lakes can deliver better data warehousing performance than traditional data warehouses using proprietary data formats. This is a major validation for the lakehouse paradigm and helps prove why the data warehouse as we know it today will either cease to exist or look vastly different in the coming decade.
Top iTechnology Networking News: RTI Awarded $1Million U.S. DoD Research Contract to Explore 5G-NextG Enhancements for Real-Time Communications
Traditionally organizations have maintained two separate data stacks – data lake for data science and machine learning, and data warehouse for BI and SQL analytics. This has led to cost overruns, data duplication, and governance issues. To avoid these issues, increasingly organizations are pointing BI tools directly to the data lake to power their analytics, as most of the data in the enterprise is already in the data lake. But the performance on data lakes has not been on par with the expectations of the analyst and business community.
Databricks has been rapidly developing full blown data warehousing capabilities directly on data lakes, bringing the best of both worlds in one data architecture dubbed the data lakehouse. We announced our full suite of data warehousing capabilities as Databricks SQL in November 2020. The open question since then has been whether an open architecture based on the Lakehouse can provide the performance, speed, and cost of the classic data warehouses. This result proves beyond any doubt that this is possible and achieved by the Lakehouse architecture.
Top iTechnology Cloud News: CyberloQ Technologies Inc. Leadership Outlines Importance of Recently Announced Collaboration with Pannovate Limited
[To share your insights with us, please write to sghosh@martechseries.com]
