Practical Data Platform Engineering
-
How considering an Iceberg migration made me rethink BigQuery costs
An Iceberg migration assessment exposed unexpected trade-offs between storage size, query speed, and slot consumption. Here is how those findings changed my approach to BigQuery costs.
-
How to make Data Engineers actually write documentation
Most data teams don’t have a documentation discipline problem. They have a system that makes documentation expensive to create and easy to ignore. Here’s how to turn it into a natural byproduct of shipping.
-
Optimising Airflow parse time by configuration
Slow DAG parsing is not always a code problem. Tune Airflow’s configuration so changes appear faster and the scheduler has room to do its real job.
-
Optimising Airflow parse time by code
Small inefficiencies in DAG code can overwhelm Airflow at scale. Remove unnecessary imports, database calls and top-level work to cut parse times and keep the scheduler responsive.