To further strengthen our commitment to providing industry-leading coverage of data technology, VentureBeat is excited to welcome Andrew Brust and Tony Baer as regular contributors. Watch for their articles in the Data Pipeline.
After Snowflake and MongoDB’s product fireworks a couple of weeks ago, Databricks joined the party. At its ongoing Data and AI summit, the San Francisco-headquartered data lakehouse company has made a number of notable announcements, starting from Project Lightspeed aimed at improving streaming data processing to a more open Delta Lake and improved MLFlow.
However, the summit hasn’t just been about platform improvements from Databricks. Multiple players forming a part of the modern data stack have also announced new and improved integrations to help their customers get the most out of their lakehouse investment.
Below is a rundown of key new integrations.
Monte Carlo
Data observability provider Monte Carlo first announced quick, no-code integrations to help enterprise users get end-to-end data observability for Databricks data pipelines. The company said it will let enterprises plug Monte Carlo into Databricks meta-stores, unity catalog or delta lake and use them to gain out-of-the-box visibility into data freshness, volume, distribution, schema and lineage – and the anomalies associated with them. This way, teams will be able to quickly detect structured and unstructured data incidents, starting from ingestion in Databricks down to the business intelligence (BI) layer, and resolve them well before they affect downstream users.
Event
Transform 2022
Join us at the leading event on applied AI for enterprise business and technology decision makers in-person July 19 and virtually from July 20-28.
Acceldata
Acceldata, Monte Carlo’s competitor in the data observability space, also announced an integration for end-to-end data pipeline visibility. This solution will track pipeline quality inside and outside Databricks to flag incidents and also include performance optimization capabilities such as automated stability tracking and cost intelligence.
“Data observability offers visibility into the entire data pipeline to help customers observe the overall quality and health of their data end-to-end to help predict potential issues and prevent costly data disasters,” Rohit Choudhary, founder and CEO of Acceldata, said. “With this integration, Acceldata data observability cloud (also) offers customers an added layer of cost intelligence to help detect and decrease inefficiencies to optimize performance and maximize their Databricks investment.”
Decodable
Data engineering company Decodable debuted a new delta lake connector to enable the ingestion of streaming data into Databricks in a simple and cost-effective way.
The current process of ingesting streaming data involves batching and is cost-prohibitive and complex. However, the new connector, which is now in general availability, ingests from any source in any cloud at the bronze and silver stages of the Databricks medallion data layer architecture, enabling application developers and data engineers to quickly connect streaming data. It is available for use with a free Decodable developer account and can unlock a host of powerful AI and analytics capabilities on Databricks.
Sigma Computing
San Francisco-based Sigma Computing announced an integration under which its no-code spreadsheet-like analytics interface will be available on Databricks. This will enable any business user, who is working with a company leveraging Databricks, to analyze cloud-scale, live data at a granular level. The integration just requires one-time deployment and allows users to build sophisticated pivot tables, create and iterate dashboards, and aggregate or free-drill into dynamic, live data in the data lakehouse.
The Databricks Data + AI Summit concludes today, June 30.
VentureBeat’s mission is to be a digital town square for technical decision-makers to gain knowledge about transformative enterprise technology and transact. Learn more about membership.
Source: Read Full Article