An incremental model adds or updates only the new rows on each run. When its logic changes, or its data
went wrong, the existing table has to be rebuilt from all the source data. --full-refresh does that:
it ignores the existing table and runs the model as if for the first time.
Copy and paste
dbt run -s fct_orders --full-refresh # rebuild one incremental model
dbt build -s fct_orders+ --full-refresh # the model, everything after it, and the tests
dbt seed --full-refresh # recreate the seed tables tooProtect a large table from accidental full refreshes:
{{ config(materialized='incremental', unique_key='order_id', full_refresh=false) }}What changes in the run
| Run | is_incremental() |
What dbt does |
|---|---|---|
| Normal incremental run | true | selects the new rows and merges them into the table (merge on Snowflake) |
With --full-refresh |
false | rebuilds the table from the full query (create table ... as) |
With --full-refresh, the block inside {% if is_incremental() %} disappears from the query, so the
model reads every row of its sources. On a large table that is a long and expensive query.
When you need it
- The logic changed. New business rules apply only to new rows until the old ones are rebuilt.
- Columns changed. By default an incremental model ignores new columns in its query. Either run
--full-refresh, or seton_schema_change: append_new_columns(orsync_all_columns) so that dbt alters the table itself. - Data went wrong. Duplicates or missing rows from a bug are fixed by rebuilding, once the bug is fixed.
- The first run after a rename. A model that writes to a new table name is built in full anyway.
full_refresh: false
With full_refresh=false in the configuration, dbt ignores --full-refresh for that model and runs it
incrementally as usual. It protects tables whose history no longer exists in the sources, such as event
tables loaded from files that were deleted after loading. To rebuild such a model on purpose, remove
the setting for one run, or drop the table by hand.
full_refresh can also be set for a folder in dbt_project.yml with +full_refresh: false.
Tested with dbt Core 1.12.
