How to run dbt models as Airflow tasks with Cosmos

Running dbt build in a BashOperator makes the whole project a single task. Cosmos, an open-source package from Astronomer, reads the project and creates one Airflow task per seed, model and snapshot, with the dependencies taken from ref(). A failed model is visible in the graph and can be retried on its own.

Copy and paste

1. Install Cosmos in the Airflow environment, and dbt in its own:

pip install astronomer-cosmos                          # in the Airflow environment

python3 -m venv /opt/dbt/venv                          # dbt in its own environment
/opt/dbt/venv/bin/pip install dbt-core dbt-snowflake

2. The DAG:

import pendulum
from airflow.sdk import dag
from cosmos import DbtTaskGroup, ExecutionConfig, ProfileConfig, ProjectConfig

DBT_PROJECT_DIR = "/opt/dbt/shop"

profile_config = ProfileConfig(
    profile_name="shop",
    target_name="prod",
    profiles_yml_filepath=DBT_PROJECT_DIR + "/profiles.yml",
)

@dag(schedule="0 5 * * *", start_date=pendulum.datetime(2026, 1, 1, tz="UTC"), catchup=False)
def dbt_cosmos():
    DbtTaskGroup(
        group_id="dbt",
        project_config=ProjectConfig(DBT_PROJECT_DIR),
        profile_config=profile_config,
        execution_config=ExecutionConfig(dbt_executable_path="/opt/dbt/venv/bin/dbt"),
        operator_args={"vars": {"run_date": "{{ ds }}"}},
    )

dbt_cosmos()

Only some models:

from cosmos import RenderConfig

DbtTaskGroup(
    ...,
    render_config=RenderConfig(select=["tag:daily"]),
)

Credentials from an Airflow connection instead of profiles.yml:

from cosmos.profiles import SnowflakeEncryptedPrivateKeyFilePemProfileMapping

profile_config = ProfileConfig(
    profile_name="shop",
    target_name="prod",
    profile_mapping=SnowflakeEncryptedPrivateKeyFilePemProfileMapping(
        conn_id="snowflake_default",
        profile_args={"database": "ANALYTICS", "schema": "MARTS"},
    ),
)

This mapping signs in with a key pair: it reads the path of the private key from the connection’s private_key_file extra and the key’s passphrase from its password field. Snowflake no longer accepts plain passwords for service users; see key-pair authentication. Cosmos ships mappings for the common warehouses, in cosmos.profiles.

What Cosmos creates

For a project with a seed orders and a model daily_sales that has a not_null test, the task group contains:

Each model gets a small group with its run and its tests, so a downstream model starts only after the tests of the models it depends on have passed. operator_args go to every task: here each dbt command receives --vars with the date of the run. DbtDag turns the whole project into a DAG instead of a task group, with the same configuration objects.

How Cosmos reads the project

To build the task list, Cosmos runs dbt ls when the DAG file is parsed, and caches the result. The DAG processor therefore needs the project folder and a dbt executable:

• ExecutionConfig.dbt_executable_path is the dbt used to run the tasks. If no dbt is on the PATH of the Airflow environment, Cosmos also uses it to parse the project. With dbt in its own virtual environment this is all you need.
• RenderConfig.dbt_executable_path sets the dbt used for parsing. Setting it to an executable that is not the one on the PATH raises an error; see how to fix “RenderConfig.dbt_executable_path is set”.
• For large projects, point Cosmos at a compiled manifest.json with ProjectConfig(manifest_path=...) and RenderConfig(load_method=LoadMode.DBT_MANIFEST), so parsing does not call dbt at all.

When Cosmos cannot find any dbt executable, the DAG fails to load with an error that starts with Unable to find the dbt executable. Set ExecutionConfig.dbt_executable_path to the full path.

Tested with Airflow 3.3.2, Cosmos 1.15.1 and dbt Core 1.12 (DuckDB adapter).