With Incident IQ, data teams using Monte Carlo’s Data Observability Platform can now easily and collaboratively identify, alert on, and remediate the root cause of data issues before they impact the business.
Monte Carlo, the data reliability company, released Incident IQ, a new suite of capabilities that help data engineers better pinpoint, address, and resolve data downtime at scale through the Monte Carlo Data Observability Platform. Incident IQ automatically generates rich insights about critical data issues through root cause analysis, giving teams unprecedented visibility into the end-to-end health and trust of their data beyond the scope of traditional data quality solutions.
On average, companies lose over $15 million per year on bad data, with data engineers spending upwards of 40 percent – or 120 hours per week – of their time tackling broken data pipelines. In the same way that New Relic, DataDog, and other Application Performance Management (APM) solutions ensure reliable software and keep application downtime at bay, Data Observability solves the costly problem of data downtime, in other words, periods of time when data is missing, inaccurate, or otherwise unreliable.
Recommended ITech News: Fastest 5G ✓ Most Available 5G ✓ New Data From Opensignal Confirms T-Mobile 5G Leadership
“In my conversations with senior executives, a theme that emerges time and again is that having control over data is fundamental for any organization wanting to compete in today’s digital economy,” says Stewart Bond, research director, Data Integration and Intelligence Software research at IDC. “As data systems become increasingly distributed, diverse and dynamic, trusting this data to deliver enterprise intelligence has never been more difficult. As architectures to control modern data environments continue to evolve, data observability solutions such as Monte Carlo will make trustworthy data more accessible and scalable.”
To help companies eliminate data downtime, Monte Carlo built Incident IQ, the first end-to-end solution that conducts root cause analysis for data issues at each stage of the pipeline, from ingestion in the data warehouse or lake to analytics in your business intelligence dashboards. Incident IQ automatically generates historical insights about your data to identify patterns in query logs, trigger investigative follow-on query results, and monitor upstream dependency changes to pin-point exactly what caused the issue to occur, reducing the amount of data incidents by 90 percent at each stage of the pipeline.
Recommended ITech News: Code-X Introduces First Commercially Available Lattice-Based Data Protection Platform Validated by the United States Department of Defense
Automatic root cause analysis and incident resolution in one UI
Developed after reviewing thousands of real data incidents from our customers, Incident IQ gives data engineers access to insights about their code, their data, and their operational environment that allows them to quickly and collaboratively get to the root cause of data problems — all in a single UI.
With Incident IQ, everything related to the data issue is captured in an elegant timeline with easy commenting, documentation, and collaboration features to create rich post-mortems. This level of detail, common in software engineering and DevOps tooling, helps data teams learn from past incidents and determine where to allocate future investment. Additionally, Incident IQ makes it easy to create and share high-level incident reporting with CTOs and CDOs, fostering greater data trust and ownership across the company.
Core capabilities of Incident IQ include:
- Central UI that connects the dots between correlated causes of data incidents, and surfaces a historical collection of data incidents for quick comparison.
- Access to example queries that pull sample data, as well as rich query logs, historical incidents, and quick links to Monte Carlo’s Lineage and Catalog features, making it easy to identify, root cause, and fix data issues all from the same interface.
- Automatic insights based on the statistical correlation between table fields in anomalous records (for instance, Incident IQ can surface if an increase in order_id null values correlates with a specific order source).
- Automatic, end-to-end lineage that maps impacted downstream BI dashboards to the furthest upstream tables, helping teams narrow the focus of root cause investigations.
- Automatic runbooks and workflows to make the incident resolution and triaging process easy, fast, and collaborative between data engineers and analysts.
- Comprehensive query logs that reveal periodic vs. ad hoc queries, changes in query patterns, and more.
Recommended ITech News: Digi International Unveils Enterprise-Caliber Turnkey Starter Kit for LoRaWAN Connectivity
“As companies become more data driven, it’s fundamental that organizations not only understand the health of their data, but also have the data observability necessary to trust it from end to end,” said Lior Gavish, CTO, Monte Carlo. “As the data stack fragments to incorporate new tools, it’s becoming increasingly difficult to identify when data pipelines break and take action to fix them. With Incident IQ, data practitioners and leaders alike can holistically understand and respond to issues faster, before they become a serious problem for the business. We believe these features will help customers eliminate hundreds of hours of data downtime and thousands to millions of dollars in savings each month, as well as enable data platform teams to scale with rich post-mortems that track performance and facilitate greater learning.”
One such customer is ThredUp, who leverages Monte Carlo’s Incident IQ capabilities to know where and how their data pipelines break, in real time. Monte Carlo has enabled ThredUp to immediately identify broken data before it affects the business, saving them time and resources on manually firefighting data downtime.
“With Monte Carlo, my team can understand what data is important for the business, as well as whether or not this data can be trusted,” said Satish Rane, Head of Data Engineering, ThredUp. “Monte Carlo’s end-to-end lineage helps me draw these connections between critical data tables and the reports the company relies on to make decisions and power our product roadmap.”
Monte Carlo is the leading Data Observability partner for the FinTech, e-commerce, media, B2B software, and retail industries, counting data teams at Fox, Vimeo, ThredUp, and PagerDuty among their customers.
Recommended ITech News: Virtual Mobile Network Software is Heating up Market Value will hit $6B by 2026
