Data Engineering Practices
Most data engineering problems are not solved by choosing another tool. They are solved by better engineering habits.
This category collects practical notes on pipeline design, code organization, SQL and Python patterns, CI/CD, testing, naming, reviews, maintainability, and operational thinking.
The focus is on everyday decisions that make data systems easier to change without quietly turning them into platform debt.
-
Optimising Airflow parse time by configuration
In the first part of this series, I focused on code-level optimisations: reducing top-level imports, avoiding unnecessary Variable.get() calls during DAG parsing, replacing heavy custom Python logic with provider operators, moving configuration into lightweight Python dictionaries, and…
-
Optimising Airflow parse time by code
Ever stared at your Airflow UI, waiting for DAGs to load, or noticed your scheduler seems to be perpetually lagging? You might be suffering from slow DAG parse times, a common but often overlooked bottleneck that…