Optimising Airflow parse time by code

Ever stared at your Airflow UI, waiting for DAGs to load, or noticed your scheduler seems to be perpetually lagging? You might be suffering from slow DAG parse times, a common but often overlooked bottleneck that can cripple your data orchestration. While parsing may seem like a background task, its frequency and performance directly affect how quickly new DAG definitions are picked up, how reliably task states are updated, and how fast your scheduler reacts to changes.

In a recent project in one of my previous companies, we faced exactly this: average DAG parse times had crept up to over 60 seconds, with outliers reaching over 2 minutes. This meant our parse times were exceeding Airflow’s dag_dir_list_interval (which is dag_processor.refresh_interval in Airflow 3.0+), also set to 30 seconds in our case. Essentially, the system couldn’t even finish parsing all DAGs before it was time to start again! This led to many pain points:

Delayed task scheduling: New DAG runs and task instances were being scheduled with a noticeable lag, sometimes several minutes after their expected execution time. That was unacceptable for some business-critical DAGs that required no downtime.
Inconsistent UI updates: The Airflow UI would show stale or missing DAGs, confusing developers and operators who weren’t sure if their changes had been picked up.
Increased scheduler CPU and memory usage: Because the scheduler never had idle time between parse cycles, CPU and memory utilization spiked, sometimes leading to process restarts or crashes.
Developer frustration: Teams deploying new DAGs or updating existing ones had to wait much longer than expected to see their changes reflected, slowing down development and delivery cycles.

This article dives into the practical steps we took to diagnose and drastically reduce these parse times to just a few milliseconds, and how you can apply these principles to your own Airflow setup.

These optimisations were performed in 2023 on an Airflow 2.x (Composer 2) environment. Most tweaks remain relevant in 2.7+ and even 3.0, and I flag the few that have changed.

Levels Of Optimization

There are many ways to optimize DAG parse performance, depending on:

How you deploy Airflow (on-prem, Docker, Cloud Composer, MWAA)
Your underlying infrastructure (CPU, memory, disk speed, filesystem, OS, etc.)

However, since, depending on your role, infrastructure and deployment decisions might be out of your control in your company, this article will focus purely on code-level optimizations — the things you can influence as a DAG developer or maintainer. In the next article, I’ll cover infrastructure and deployment-level strategies to help you fully unlock Airflow’s scheduler performance.

Without further ado, let’s dive into the code-level strategies that brought our parse times back under control.

Code Optimizations

Embrace DRY (Don’t Repeat Yourself) with Gusto

Problem: We found a lot of boilerplate code for task definitions, sensor configurations, and common utility logic repeated across many DAGs. Each instance of this repeated code adds to the overall size and complexity of the DAG files that the Airflow parser needs to process. While a small repetition in one DAG might seem trivial, across hundreds of DAGs, this redundancy significantly bloats the total code and increases the total code to parse, leading to slower interpretation and increased memory footprint during parsing. For example, we saw similar Slack notification functions or custom retry logic were copied and pasted across numerous files or Operators being defined in individual DAG files. This not only increased parse times but also made maintenance a nightmare—a small change in a common piece of logic required updating it in dozens of places, risking inconsistency.

Solution: We created a centralized helpers/ directory (ensuring it was in PYTHONPATH for Airflow by placing it in the dags/ folder). This new directory became home to:

Utility Functions: Functions for common tasks like sending Slack notifications were centralized. Instead of each DAG defining its own send_slack_alert function, a single, well-tested version was available.

# helpers/notifications.py
from airflow.providers.slack.operators.slack_webhook import SlackWebhookOperator

def get_slack_failure_alert_task(slack_conn_id='slack_default'):
    return SlackWebhookOperator(
        task_id='slack_failure_alert',
        slack_webhook_conn_id=slack_conn_id,
        message="DAG {{ dag.dag_id }} failed on task {{ ti.task_id }}",
        # ... other params
    )

Task Group Generators: For recurring patterns of tasks, we created functions that would return a TaskGroup or a list of tasks. For instance, many of our DAGs were responsible for triggering Google Cloud Dataform pipelines. Each Dataform invocation typically involved a sequence: first, creating a “Compilation Result” from a specific branch or commit, and second, using that compilation to create and monitor a “Workflow Invocation,” often with specific tags or targets. Manually defining these two (or more) Dataform operators with their intricate configurations, dependencies, and XCom handling in every relevant DAG was verbose, error-prone, and a nightmare for updates. To address this, we created a dataform_workflow_task_group generator. Instead of copy-pasting Dataform operator definitions, a DAG author could now simply call this function, which would construct the entire sequence of Dataform tasks, neatly packaged as a TaskGroup, with just a single call in their DAG definition file.
Custom Operator Classes (Lightweight): If we needed a slight variation of an existing operator or a simple custom operator used in many places, defining it once in a dedicated helpers/operators/ subdirectory was very beneficial (importable as from helpers.operators import ... since helpers/ was in PYTHONPATH). However, we were careful not to create overly complex custom operators that would become another source of parsing overhead and maintenance burden.

While the primary win here was a massive improvement in DAG maintainability and consistency, the resulting leaner DAG files also contributed positively to reducing overall parse times, most importantly setting the stage for further optimizations.

Leverage official Airflow provider operators

Problem: We had a lot of custom-written functions performing operations with GCP services (like submitting BigQuery jobs or transferring data to GCS), which were invoked using the PythonOperator. This solution led to importing heavy client libraries (e.g., google-cloud-bigquery, google-cloud-storage) at the top level of the DAG file. These libraries were then loaded every single time the DAG was parsed, significantly increasing parse duration, even if the operator itself wasn’t running. Besides that, the DAG file itself contained more Python code for interacting with external services, making it larger and taking longer for the Airflow parser to process.

Solution: Recognizing that this flexible approach introduced performance issues and code duplication, we adopted the best practice of replacing custom PythonOperator logic with official provider operators wherever possible. We systematically identified custom PythonOperator tasks that performed standard interactions with GCP services and replaced them with official Airflow Provider Operators (e.g., BigQueryInsertJobOperator, GCSToBigQueryOperator, GCSDeleteObjectsOperator). This shift made our DAGs more declarative, focusing on what tasks to run rather than how to run them in detail within the DAG file. Refactoring our DAGs to use official provider operators achieved the following:

Reduced Parse Times: By eliminating a lot of unnecessary top-level imports of heavy GCP client libraries, DAG parse times were noticeably reduced. The provider operators handle their imports internally, typically at task runtime (within their execute method), meaning these heavy libraries are not loaded when Airflow merely parses the DAG file structure.
Simpler and Cleaner DAGs: DAG files became shorter, easier to read, and more focused on workflow orchestration, as the complex interaction logic was abstracted away into the provider operators.
Improved Maintainability & Reliability: Leveraging official, community-vetted operators reduced our custom code maintenance burden and benefited from ongoing updates and bug fixes within the provider packages.

Efficient Handling of Airflow Variables & Connections

Airflow 2.7+ note: With the new cached-DAG mechanism the DB hit is smaller, but deferring Variable look-ups to runtime still saves you the first-parse penalty.

Problem: Our investigation into slow DAG parse times revealed a significant bottleneck related to how Airflow Variables and Connections were being accessed within our DAG files. We found numerous instances where Variable.get("var_name") or BaseHook.get_connection("conn_id") were called directly at the top level of DAG Python files. Because fetching these often involves a synchronous database lookup against the Airflow metastore, these top-level calls were executed every single time a DAG file was parsed by the scheduler or webserver. With a large number of DAGs, or even a few DAGs making multiple such calls, this practice resulted in a high volume of unnecessary database queries during each parsing cycle. This cumulative effect overloaded the metastore significantly. For example, if you have 100 DAGs each making just two top-level Variable/Connection calls, that sums to 200 database queries per parse cycle, repeating every dag_dir_list_interval.

Solution: To mitigate the performance impact of Variable and Connection lookups during DAG parsing, we implemented the following strategies:

Defer Lookups to Task Runtime — For every PythonOperator and other callables instead of calling Variable.get() or BaseHook.get_connection() at the top level, we moved these lookups inside the python_callable function or any function that is executed at task runtime. It’s worth noting that well-designed official Provider Operators typically fetch connection details or variables within their execute method (as mentioned in previous point), thus avoiding this pitfall during DAG parsing.

# Bad: Top-level call in DAG file
from airflow.models import Variable
my_api_key = Variable.get("my_api_key_secret") # DB call during DAG parsing

def my_task_function(**kwargs):
    # ... uses my_api_key (which was loaded at parse time) ...
    pass

# Good: Lookup inside the callable
def my_task_function_optimized(**kwargs):
    from airflow.models import Variable
    my_api_key_runtime = Variable.get("my_api_key_secret") # DB call only when task runs
    # ... use my_api_key_runtime ...
    pass

Utilize Jinja Templating — Many operator parameters are templatable (if they aren’t, you can create your own version of the operator to make it templatable). This means you can pass references to Variables or Connection attributes that Airflow will resolve at runtime.

from airflow.providers.slack.operators.slack_webhook import SlackWebhookOperator

send_notification = SlackWebhookOperator(
  task_id='send_slack_notification',
  slack_webhook_conn_id='my_slack_connection', # Connection ID already known at parse time
  message="Today's important value is: {{ var.value.my_important_var }}", # Resolved at runtime
 )

Note: Accessing var.json.my_json_var.key or var.value.my_var in Jinja templates is generally more performant during parsing than direct Variable.get() calls if the variable is not already cached, as Airflow can sometimes optimize these. However, the primary win is deferring any potential DB hit to runtime.

Environment Variables for Configuration — For non-sensitive configuration that might otherwise go into an Airflow Variable, consider using environment variables. These can be read with os.getenv("MY_CONFIG_VALUE") at the top level of a DAG file with negligible performance impact, as it doesn’t involve a database call. Connection details, especially passwords, should use the Airflow connection system, ideally backed by a secrets backend.

This optimization was huge. By implementing these changes, we drastically reduced the number of database queries during parsing, alleviating metastore load and cutting down DAG parse times almost 80%!

Minimize Top-Level Code and Imports

Problem: Any Python code written at the top level of your DAG file (i.e., not inside a function or class method that is only called at task runtime) is executed every time the DAG is parsed. This includes not just direct Variable.get() calls or heavy library imports, but also any complex computations, I/O operations, or extensive object instantiations. If these top-level operations are slow, they will directly contribute to slow parsing for that DAG.

Solution: The core principle here is to keep your DAG files declarative, as emphasized earlier. Think of your DAG file primarily as a static blueprint defining the workflow’s structure and dependencies. Any actual ‘work’ — be it data processing, interacting with external systems, or complex calculations — should be deferred to task execution time. This means:

Move Computations and I/O into Tasks: Any complex calculations, data transformations, or I/O operations (like reading from files or making network calls) performed at the top level of your DAG file should be shifted into python_callable functions or the execute method of custom operators. Our next optimization of inlining per-DAG JSON configurations into Python dictionaries is a prime example of this principle effectively reducing file I/O.
Minimize Top-Level Logic: If dynamic task generation (mentioned in the first point) is necessary, ensure the logic driving it is extremely lightweight and avoids external dependencies or slow computations during the parsing phase.
Lazy Loading for Task-Specific Heavy Libraries (as covered previously): if a heavy library is only needed for a specific task, it should be imported within that task’s runtime scope, not globally in the DAG file.

Decouple DAG Configurations: From JSON files to In-DAG Dictionaries

Problem: In our setup, many DAGs had their configurations stored in individual, dedicated JSON files (e.g., my_dag_a_config.json for my_dag_a.py, my_dag_b_config.json for my_dag_b.py, and so on). While this approach kept configurations neat for each DAG, allowing to tweak them without changing the DAG files themselves, it meant that each time a specific DAG file was parsed by Airflow, its corresponding JSON configuration file had to be read from the disk and then parsed using json.load(). This introduced two distinct overheads for each DAG:

Disk I/O: And as you already know — it always hurts performance.
JSON Parsing: While not that CPU-intensive itself, when multiplied across a large number of DAGs, these seemingly small, per-DAG overheads accumulated, contributing to slower overall DAG processing times.

Solution: To eliminate this per-DAG file I/O and JSON parsing overhead, we migrated the configurations from these individual JSON files directly into their respective Python DAG files as native Python dictionaries.

Embedded Configuration Dictionaries: For each DAG, we took the contents of its <dag-id>_config.json file and defined them as a Python dictionary, typically named DAG_CONFIG, at the top of the DAG’s Python script.
Performance Gain: Python’s interpreter handles dictionary literals much more efficiently (as part of its bytecode compilation) than the explicit steps of opening a file, reading its contents, and then calling json.load(). As an added bonus, this also eliminated the need to import the json library in these DAG files. This change effectively removed both the disk I/O and the JSON deserialization steps from the parse path of each DAG.

Old way:

import json
import os
from airflow import DAG

DAG_FILE_DIR = os.path.dirname(os.path.abspath(__file__))
CONFIG_FILE_PATH = os.path.join(DAG_FILE_DIR, 'my_dag_a_config.json')

with open(CONFIG_FILE_PATH, 'r') as f:
    DAG_CONFIG =  json.load(f) # Reads and parses this DAG's specific JSON

with DAG(dag_id='my_dag_a', schedule=DAG_CONFIG.get('schedule'), ...) as dag:
  image = DAG_CONFIG.get('k8s_settings', {}).get('image')
  ...

Evolved into:

from airflow import DAG
from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import KubernetesPodOperator
import pendulum

# Configuration is now specific to this DAG, directly embedded,
# and parsed efficiently by the Python interpreter.
DAG_CONFIG = {
    'dag_id': 'my_dag_a',
    'schedule': '0 7 * * 1',
    'start_date': [2024, 1, 1],
    'catchup': False,
    'tags': ['data_processing', 'k8s'],
    'owner': 'data_platform_team',
    'k8s_pod_config': {
        'image': 'custom-processor:1.5.2',
        'name_prefix': 'dag-a-k8s-task',
        # ...
    },
}

with DAG(
    dag_id=DAG_CONFIG['dag_id'],
    schedule=DAG_CONFIG['schedule'],
    start_date=pendulum.datetime(*DAG_CONFIG['start_date'], tz='UTC'),
    catchup=DAG_CONFIG['catchup'],
    tags=DAG_CONFIG['tags'],
    default_args={'owner': DAG_CONFIG['owner']}
) as dag:

    run_kubernetes_processing = KubernetesPodOperator(
        task_id='run_heavy_processing',
        name=f'{DAG_CONFIG["k8s_pod_config"]["name_prefix"]}-processing',
        image=DAG_CONFIG['k8s_pod_config']['image'],
        # ...
    )

    # ...

The impact of this single optimization was genuinely huge, especially at our scale. Moving these configurations in-DAG dramatically reduced the cumulative I/O and CPU load during parsing cycles.

Strategically Use `.airflowignore`

Problem: The Airflow scheduler and webserver periodically scan the DAGs folder (and any other folders specified in dags_folder) to parse Python files and discover DAG objects. If your DAGs folder contains many non-DAG Python files—such as utility scripts, archived DAGs or temporary development files — the parser might waste valuable time attempting to inspect them. This becomes particularly problematic with an accumulation of such files. In our case, we had many leftover development DAG files and helper scripts cluttering the DAGs directory. While cleaning up (deleting unnecessary files) and adopting firm naming conventions (e.g., suffixing development scripts with _dev.py or _temp.py) helped organize things, Airflow would still attempt to look at these files. A more explicit mechanism was needed to tell Airflow, “Don’t even try to parse these”.

Solution: We leveraged the .airflowignore file, a powerful feature that functions much like .gitignore. By creating an .airflowignore file in our DAGs directory (or any subdirectory you want to apply rules to), we could specify patterns, filenames, or directory names that Airflow should completely disregard during its DAG discovery and parsing process. We’ve added entries there to tell Airflow what to skip during DAG parsing, and the list included:

Directories containing helper utilities that didn’t define DAG objects themselves (e.g., helpers/, utils/ – assuming these were added to PYTHONPATH separately if needed by DAGs).
Directories for tests (e.g., tests/).
Patterns matching our naming conventions for temporary or development files (e.g., *_dev.py, temp_*.py).
Specific archived DAG files or subfolders containing old versions.

It turned out that, by strategically leveraging the .airflowignore file, we were able to omit over 50 legacy DAGs and numerous utility scripts from the parsing process entirely. This single change significantly reduced the clutter for the parser, directly contributing to faster scan times and a noticeable improvement in the overall DAG processing loop duration.

How to measure your DAGs parse performance?

Before you start to optimize, you need to measure. Understanding your current DAG parse performance is key to identifying bottlenecks and quantifying the impact of any changes you make. The methods for measurement can vary slightly depending on your Airflow deployment. Our Airflow instance was running on Composer, so in GCP you have these options:

Cloud Monitoring: Composer environments are deeply integrated with Google Cloud Monitoring. Look for metrics related to the Airflow scheduler and DAG processing. Key metrics often include dag_processing_time.last_duration.<dag-file> (for a single DAG) or similar metrics like dag_processing.total_parse_time which might show the duration of a full parsing loop. Also, monitor the CPU and memory utilization of your scheduler instances, as high resource consumption during parsing is a strong indicator of inefficiency.
Cloud Logging: The logs from your Airflow scheduler and DAG processor (if using a separate DAG processor, common in Composer 2+) are available in Cloud Logging. Search for messages indicating the start and end of DAG parsing cycles and any specific errors or timeouts related to parsing individual DAG files.
Airflow CLI via gcloud: You can execute Airflow CLI commands directly against your Composer environment. Running the following command will give you a detailed breakdown of each DAG, including its last parse time.

gcloud composer environments run <YOUR_ENV_NAME> \
    --location <YOUR_REGION> dags report

For on-prem deployments

Airflow Logs: Directly inspect the logs generated by your Airflow scheduler and/or DAG processor processes. Use tools like grep to search for specific patterns related to individual DAG parse times (often logged per file).
Airflow UI: Modern Airflow versions (2.x+) provide valuable insights directly in the UI. Navigate to Browse > DAG Runs and look for the “DAG Processing” entries (or Admin > DAG Processing in some views/versions). This page often shows the duration of recent DAG parsing loops, the number of DAGs parsed, and any errors encountered. While individual DAG pages show “Last Parsed” timestamps, the DAG Processing view gives a better aggregate.
airflow dags report CLI Command: This is one of the most direct ways to get performance data. Running airflow dags report in your Airflow environment’s shell will output a table listing all your DAGs, their owners, and critically, their individual parse times (duration column). This helps pinpoint specific problematic DAGs.
External Monitoring Systems (e.g., Prometheus, Grafana, Datadog): If you have integrated Airflow with an external monitoring system, look for metrics such as airflow_dag_processing_duration, airflow_dag_file_refresh_duration, or specific timers for DAG parsing loops. Also, monitor scheduler/DAG processor CPU and memory usage. The exact metric names can vary based on the exporter or agent used (e.g., airflow.dag_processing.total_parse_time).

Regardless of your environment, aim to understand both the total duration of a full DAG parsing cycle (how long it takes Airflow to scan and process all DAG files) and the individual parse times for each DAG. The former tells you if you’re exceeding your dag_dir_list_interval (or dag_processor.refresh_interval in Airflow 3.0+), while the latter helps you identify which specific DAGs are contributing most to any slowdowns.

Strategic Measurement and Performance Goals

Armed with the methods to measure DAG parse performance, it’s crucial to apply them strategically. I recommend performing these checks before you begin any optimization to establish a clear baseline. This initial snapshot is invaluable for understanding the scale of the problem and identifying the most egregious offenders. Then, during your optimization efforts, re-measure after applying each significant change. This iterative approach allows you to see the direct impact of specific fixes — for instance, you might discover that refactoring one complex DAG or fixing a common anti-pattern in your helper utilities yielded the biggest improvement. Finally, conduct a thorough measurement after all planned optimizations are complete to quantify the overall success and establish your new, improved baseline.

What’s an Acceptable DAG Parse Time?

The general rule is — the shorter the better. While there’s no single “magic number” for acceptable DAG parse time that fits all scenarios, we can define some aspirational goals based on operational stability and responsiveness:

Individual DAG Parse Time: Aim for parse times in the milliseconds range (e.g., under 100–200ms) for most DAGs. Simpler DAGs should be even faster. If an individual DAG consistently takes several seconds to parse, it warrants immediate investigation.
Total Parse Cycle Time: This is the most critical metric for scheduler health. Your total time to parse all DAGs in a cycle must be significantly less than your Airflow’s DAG directory refresh interval (controlled by dag_dir_list_interval in airflow.cfg). This interval often defaults to 30, 120 seconds or sometimes 5 minutes. A good rule of thumb is to keep your total parse cycle below 50% of this interval. For example, if your interval is 30 seconds, a total parse cycle under 10-15 seconds is a healthy target. If it takes 25 seconds to parse DAGs with a 30-second interval, your scheduler has very little breathing room.
Impact of DAG Count: Naturally, more DAGs will lead to a longer total parse cycle. This emphasizes that as your DAG count grows, maintaining extremely low per-DAG parse times becomes even more critical to stay within a healthy total cycle time for your chosen refresh interval.

The ultimate “gold standard” is an Airflow environment where DAGs appear quickly in the UI after changes, tasks are scheduled promptly, and the scheduler operates with ample idle capacity between parse cycles. If your total parse time consistently exceeds your refresh interval, you’re in a state where the system can’t keep up, leading to the problems outlined at the beginning of this article.

Monitor — Keeping Parse Times Under Control

Optimization isn’t a one-time task; it requires ongoing vigilance. To ensure your DAG parse times remain healthy long-term, implement continuous monitoring and alerting:

Alert on Total Parse Cycle Duration: Configure alerts in your monitoring system (Cloud Monitoring for Composer, Prometheus/Alertmanager, Datadog, etc.) to notify you if the total DAG parse cycle duration exceeds a critical threshold (e.g., 75% of your dag_dir_list_interval).
Monitor Individual DAG Parse Time Spikes: Where possible, track the parse times of individual DAGs. A sudden increase in a specific DAG’s parse time after a deployment is a strong signal of a newly introduced inefficiency. Tools like airflow dags report can be used periodically, or its data can be scraped if your monitoring solution allows.
Track Failed DAG Parses: An increase in the number of DAGs failing to parse often points to syntax errors or problematic code changes. Set up alerts for num_failed_dag_parses or similar metrics.
Scheduler Health & Resource Utilization: Continuously monitor the CPU and memory utilization of your Airflow scheduler (and DAG processor, if separate). Sustained high utilization, especially correlated with parsing cycles, indicates the system is struggling.
Regular Audits: Periodically review the output of airflow dags report (e.g., quarterly, or after significant new DAG deployments) to proactively identify any creeping performance degradation.
Integrate with CI/CD: For a more proactive approach, consider adding a step to your CI/CD pipeline that runs basic DAG validation and parsing checks (e.g., airflow dags list --report or simply trying to import the DAG files) on changed DAG files to catch obvious parsing issues before they reach your production environment.

By proactively measuring, setting clear performance targets, and continuously monitoring, you can ensure that your Airflow environment remains robust, responsive, and capable of handling your evolving orchestration needs efficiently.

Conclusion

The journey from multi-minute DAG parse outliers and a perpetually lagging scheduler to a consistently responsive and efficient Airflow environment, as we experienced, underscores a vital truth: seemingly small inefficiencies within your DAG code can have a massive cumulative impact on parse performance. Throughout this article, we’ve explored a range of practical, code-level optimizations you can implement — from embracing DRY principles and strategically leveraging provider operators, to the careful handling of Airflow Variables, external configuration files and top-level imports.

Each of these techniques is designed to minimize the workload on the Airflow parser during its critical discovery phase, ensuring it only processes what’s essential, at precisely the right time. By diligently applying these strategies, establishing robust measurement practices, and committing to ongoing monitoring, you won’t just alleviate the immediate pain points of slow parsing; you’ll be building a more resilient, scalable, and developer-friendly Airflow foundation.

While infrastructure configurations and deployment choices — topics I aim to delve into in Part 2 of this article — certainly play their part in overall system performance, the discipline and best practices you instill in your DAG development today are your first and most powerful line of defense. Investing in parse-friendly DAG code will pay dividends not only in raw performance and resource savings but also in improved developer productivity and the long-term stability of your data orchestration platform.

While we ran this on Airflow 2.x, the same principles continue to shave milliseconds off parses in 2.7+ and will serve you well when you eventually jump to Airflow 3.