Why your Airflow DAG processes yesterday's data (logical date explained)

In Airflow 2, a daily DAG that runs at midnight on 15 May has the logical date 14 May. It is not a bug: the run processes the day that has just ended and is named after the start of that day. Airflow 3 changed the default, and the same run is now dated 15 May. A query that filters on ds therefore loads different days in the two versions.

Copy and paste

Airflow 3 with the default schedule: load the day that just ended

sql = "SELECT * FROM orders WHERE order_date = '{{ macros.ds_add(ds, -1) }}'"

Keep the Airflow 2 behaviour in one DAG

import pendulum
from airflow.sdk import dag, CronDataIntervalTimetable

@dag(
    schedule=CronDataIntervalTimetable("0 0 * * *", timezone="UTC"),
    start_date=pendulum.datetime(2026, 1, 1, tz="UTC"),
    catchup=False,
)
def daily_orders():
    ...

daily_orders()

With that schedule, the run at midnight on 15 May covers 14 May:

sql = """
    SELECT * FROM orders
    WHERE order_ts >= '{{ data_interval_start }}'
      AND order_ts <  '{{ data_interval_end }}'
"""

Keep the Airflow 2 behaviour in every DAG. Set it before the first Airflow 3 runs exist:

export AIRFLOW__SCHEDULER__CREATE_CRON_DATA_INTERVALS=True

Two ways to date a run

In the data interval model every run covers a period. It starts when the period is over, so that the data is complete, and it is named after the start of the period: the run for 14 May happens on 15 May.

In the trigger model, the default since Airflow 3, a run has no period. It is named after the moment it starts, and the data interval has the same start and end. A plain timedelta schedule follows the same rule, because Airflow 3 also turns it into a trigger timetable by default.

What changes after an upgrade

• A query with WHERE order_date = '{{ ds }}' in a midnight DAG loaded the complete previous day in Airflow 2. In Airflow 3 it loads the day that has just started, which is empty or partial.
• A filter between data_interval_start and data_interval_end returns nothing in the trigger model, because the two values are equal.
• Changing create_cron_data_intervals after some Airflow 3 runs exist skips one scheduled run, so decide before the upgrade.

Manual runs

A manual run triggered with a logical date has that date as logical_date and ds. The Airflow 3 documentation warns not to assume that its data interval is derived from that date: the trigger form can infer the interval from the timetable or take it as input. When a job must process the date the user picked, base it on logical_date or ds.

A run triggered without a logical date has no ds at all; see how to use date macros.

Tested with Airflow 3.3.2.