Setting up dbt using Poetry — A clean and effective approach for your team

Setting up dbt using the Python Poetry dependency manager ensures that all users are working with the same version of dbt, and the right version of dbt. By initializing their poetry environment using a common configuration file (pyproject.toml - which we will describe later on in this post), users will not need to install dbt directly on their machines. Poetry will manage this for them! Add to that being able to manage any additional Python dependencies that your project might need, saving your users the headache of installing each of these dependencies themselves and saving them from managing potential conflicts.

Here are the steps to follow:

1- Install Poetry

At the time of writing this article, the recommended Python version for using poetry is 3.7+. You can always refer to this link for the latest installation instructions. Here is the recommended way to install Poetry:

Linux, macOS, or Windows WSL:

curl -sSL https://install.python-poetry.org | python3 -

Windows Powershell:

(Invoke-WebRequest -Uri https://install.python-poetry.org -UseBasicParsing).Content | py -

2- Add Poetry to your PATH

When you install Poetry using the above approach, the installer will print out the location to your shell, for example: … Python/3.9/bin

Add:

export PATH=$HOME/Library/Python/3.9/bin:$PATH

to ~/.profile or ~/.zprofile or ~/.zshrc

Make sure that Poetry is properly installed by running the below command:

poetry --version

3- Create a pyproject.toml file

The third step is to create a pyproject.toml file in which you will be able to define all the Python requirements of the project, along with the needed version. Here’s a sample pyproject.toml file would eventually install dbt Core version 1.0.2 using Snowflake as a database.

[tool.poetry]name = "my-dbt-project"version = "0.1.0"description = "My dbt project"authors = ["John Doe"][tool.poetry.dependencies]python = "^3.7"snowflake-connector-python = "2.4.6"dbt-snowflake = "1.0.0"dbt-core = "1.0.2"[tool.poetry.dev-dependencies][build-system]requires = ["poetry-core==1.1.8"]build-backend = "poetry.core.masonry.api"

The ^ in front of 3.7 means that Poetry can install any version that matches the leftmost non-zero digit of the version string, ie: “3.”.

You can add additional dependencies to the file such as sqlfluff and other packages that you might need for your dbt project.

4- Install the dependencies

Run the below command to install all of the Poetry dependencies

poetry install

Your shell should output something similar to the below:

You might notice that when running the above command, Poetry creates a file called poetry.lock in your project’s home directory. This file keeps a log of all packages and the exact versions you are using for the project.

5- Verify that dbt is installed

If the above step is successful, run the below command to make sure that dbt is installed:

poetry run dbt --version

6- Run dbt via Poetry

You can now run dbt using Poetry. In order to do so, precede your usual dbt commands with the word poetry run. For example:

poetry run dbt run

poetry run dbt test

That’s it! In case you face any Python version conflicts, run the poetry shell command at the beginning of your session. What this command will basically do is spawn a shell, according to the $SHELL environment variable, within the virtual environment. If one doesn’t exist yet, it will be created.

References:

1- https://python-poetry.org/docs/master/