The partnership combines the Databricks Lakehouse Platform with Prefect’s visibility into data workflows for greater reliability and confidence
Prefect Technologies, Inc., the company behind the open-source data workflow automation platform Prefect, announced its partnership with Databricks, the Data and AI company, to provide data engineers, data scientists and analytics engineers the ability to automate dataflows for even the most data-intensive organisations and applications.
Together, organisations will have greater control and confidence in their workflows and the ability to connect various resources across the modern data stack on the lakehouse. The Databricks Lakehouse Platform combines the best elements of data lakes and data warehouses, enabling users to unify their data, analytics and AI, build on open source technology, and maintain a consistent platform across clouds. With Prefect’s data workflow orchestration platform, data teams will have greater performance, reliability, and visibility into their data.
“As the Databricks Lakehouse Platform becomes the hub of data-driven organisations, workflow orchestration ensures that accurate, reliable data is ready for critical use cases – with minimal human intervention,” said Roger Murff, VP of Technology Partners at Databricks. “Prefect saves data engineers time, offering end-to-end visibility and increasing confidence in the accuracy and availability of data.”
For users migrating to Databricks, Prefect allows them to work with tools across the data stack and operate across multiple infrastructures simultaneously. Additionally, data engineers can access all of the tools they currently use while developing in Python.
“Through the Databricks and Prefect integration, we are able to automate our workflows with greater efficiency than ever before,” said Elizabeth Hollinger, Director of Insights, Aggreko. “We have greater confidence in our data’s performance and have raised productivity through the unmatched visibility Prefect provides.”
The Prefect and Databricks partnership is driven by the companies’ shared vision to provide engineers with flexibility and open-source support.
“As data engineers encounter a growing number of tools across the stack, we’re excited to add Databricks as a partner to help companies continue to coordinate their dataflows,” said Jeremiah Lowin, CEO of Prefect. “Databricks’ validation of Prefect’s technology also helps further our growth as the leading dataflow automation platform.”
