This webinar on ‘AIRflow at Scale’ covered the walkthrough of how Qubole extended Apache AIRflow to manage the operational inefficiencies that arise when you manage data pipelines in a multi-tenant environment. And also the process of making the data pipelines robust by adding data quality checks using CheckOperators.
Key Takeaways:
Listed below are the other major key takeaways:
- Overview of major type of data pipelines
- How Qubole manages deployments and upgrades of data pipelines in a multi-tenant environment
- The Data quality issues which Qubole faces in process of data ingestion or transformation.
- The approach that Qubole has adopted using Apache Airflow Check operators.
- Enhancements we had to make to Check operators.
- Integration of Apache Airflow Check operators with our ETLs.
- Challenges Qubole faced in developing the alerting framework.
- The best practices in using Apache Airflow for data quality checks.
AIRflow at Scale from Digital Vidya
Did you find the webinar interesting? Register here for more.
