Dev Setup – dbt Core 1.9.0 with Airflow 3.0 Orchestration

Hello Data Engineers 👋

I’ve been scouting on the internet for the best and easiest way to setup dbt Core 1.9.0 with Airflow 3.0 orchestration. I’ve followed through many tutorials, and most of them don’t work out of the box, require fixes or version downgrades, and are broken with recent updates to Airflow and dbt.

I’m here on a mission to find and document the best and easiest way for Data Engineers to run their dbt Core jobs using Airflow, that will simply work out of the box.

Disclaimer: This tutorial is designed with a Postgres backend to work out of the box. But you can change the backend to any supported backend of your choice with little effort.

So let’s get started.

Prerequisites

• Docker desktop (https://docs.docker.com/desktop/setup/install/mac-install/)
• Python 3.12 or higher (https://www.python.org/downloads/)
• Code repo (https://dbtengineer.com/airflow-with-dbt-core-tutorial/#code-repo-video-tutorial)

Video Tutorial

Setup

1. Clone the repo in prerequisites.
2. Create a data folder in the root folder on your local.
3. Rename .env-example to .env and create new values for all missing values. Instructions to create the fernet key at the end of this Readme.
4. Rename airflow_settings-example.yaml to airflow_settings.yaml and use the values you created in .env to fill missing values in airflow_settings.yaml.
5. Rename servers-example.json to servers.json and update the host and username values to the values you set above.

Running Airflow Locally

1. Run docker compose up and wait for containers to spin up. This could take a while.
2. Access pgAdmin web interface at localhost:16543. Create a public database under the postgres server.
3. Access Airflow web interface at localhost:8080. Trigger the dag.

Running dbt Core Locally

Create a virtual env for installing dbt core

python3 -m venv dbt_venv
source dbt_venv/bin/activate

Optional, to create an alias

alias env_dbt='source dbt_venv/bin/activate'

Install dbt Core

python -m pip install dbt-core dbt-postgres

Verify Installation

dbt --version

Create a profile.yml file in your /Users/<yourusernamehere>/.dbt directory and add the following content.

default:
  target: dev
  outputs:
    dev:
      type: postgres
      host: localhost
      port: 5432
      user: your-postgres-username-here
      password: your-postgres-password-here
      dbname: public
      schema: public

You can now run dbt commands from the dbt directory inside the repo.

cd dbt/hello_world
dbt compile

Cleanup

Run Ctrl + C or Cmd + C to stop containers, and then docker compose down.

FAQs

Generating fernet key

python3 -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"

I hope this tutorial was useful. Let me know your thoughts and questions in the comments section.

Happy Coding!