Introduction
Dbt [1] is a tool for transforming data in data warehouses using Python and SQL.
dbt enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.
The idea is that Data Engineers make source data available to an environment where dbt projects run, for example with Debezium or with Airflow. Afterwards, Data Analysts can run their dbt projects against this data to produce models (tables and views) that can be used with a number of BI tools.
dbt’s Features
The data abstraction layer provided by dbt allows the decoupling of the models on which reports and dashboards rely from the source data. When business rules or source systems change, you can still maintain the same models as a stable interface.
Some of the things that dbt can do include:
- Import reference data from CSV files
- Track changes in source data with different strategies so that downstream models do not need to be built every time from scratch
- Run tests on data, to confirm assumptions remain valid, and to validate any changes made to the models’ logic
dbt and CrateDB
Due to its unique capabilities, CrateDB is an excellent warehouse choice for data transformation projects. It offers automatic indexing, fast aggregations, easy partitioning, and the ability to scale horizontally.
Getting started
Please refer to CrateDB setup | dbt Developer Hub and using dbt with CrateDB.
These dbt features have been tested successfully:
- models with view, table, and ephemeral materializations
- dbt source freshness
- dbt test
- dbt seed
- Incremental materializations (with
incremental_strategy='delete+insert'and without involving OBJECT columns)
We hope you find this useful. CrateDB is continuously adding new features and we will very happy to hear about your experience using CrateDB with dbt.