Data Engineering Practices
Most data engineering problems are not solved by choosing another tool. They are solved by better engineering habits.
This category collects practical notes on pipeline design, code organization, SQL and Python patterns, CI/CD, testing, naming, reviews, maintainability, and operational thinking.
The focus is on everyday decisions that make data systems easier to change without quietly turning them into platform debt.
-
How considering an Iceberg migration made me rethink BigQuery costs
An Iceberg migration assessment exposed unexpected trade-offs between storage size, query speed, and slot consumption. Here is how those findings changed my approach to BigQuery costs.
-
How to make Data Engineers actually write documentation
Most data teams don’t have a documentation discipline problem. They have a system that makes documentation expensive to create and easy to ignore. Here’s how to turn it into a natural byproduct of shipping.
-
Optimising Airflow parse time by configuration
Slow DAG parsing is not always a code problem. Tune Airflow’s configuration so changes appear faster and the scheduler has room to do its real job.
-
Optimising Airflow parse time by code
Small inefficiencies in DAG code can overwhelm Airflow at scale. Remove unnecessary imports, database calls and top-level work to cut parse times and keep the scheduler responsive.