Practical notes on reliable data platforms

Orchestration

Orchestration is where data platforms become predictable systems or scheduled chaos.

This category explores DAG design, scheduling, dependencies, retries, backfills, parsing performance, failure isolation, and pipeline reliability.

The focus is on building workflows that run quietly, fail clearly, and stay understandable in production.

  • Optimising Airflow parse time by configuration

    Optimising Airflow parse time by configuration

    Slow DAG parsing is not always a code problem. Tune Airflow’s configuration so changes appear faster and the scheduler has room to do its real job.

  • Optimising Airflow parse time by code

    Optimising Airflow parse time by code

    Small inefficiencies in DAG code can overwhelm Airflow at scale. Remove unnecessary imports, database calls and top-level work to cut parse times and keep the scheduler responsive.