local data stack template

This template brings together some of the best open-source tools in data engineering and Rust-based tooling for ensuring code quality:

• dagster, dbt, duckdb
• pixi
• docker
• ruff, pyright, taplo, pytest
• cruft

While the stack runs locally, it's designed with scale in mind. Data and state accumulate over time (as data engineers say in partitions). Slow data - similar to slow food, is healthy and allows the fast delivery of business value efficiently.

The idea of slow data refers to:

• focusing on the reduction of complexity

- allows to reduce complex data pipelines down to a single developers notebook - making complex data pipeliens testable

• shipping real busines value fast
• scaling as needed via partitions and a scalable resource provider

The template directly can be deployed via docker-compose. With only minimal adaptations you can instead deploy to kubernetes in any cloud provider.

Additionally included for secure handling of secrets:

• sops
• age

usage

Prerequisites:

• git
• an installation of cruft pip install cruft jinja2-ospath
• an installation of pixi https://pixi.sh/latest/#installation curl -fsSL https://pixi.sh/install.sh | bash
• text editor of choice such as vscode
• docker

curl -fsSL https://pixi.sh/install.sh | bash
git clone https://github.com/l-mds/local-data-stack.git
cd local-data-stack
pixi run tpl-init-cruft
alternatively:
pip install cruft jinja2-ospath
cruft create https://github.com/l-mds/local-data-stack.git
cd <<your project name>> git init git add . git commit -m "initial commit"
pixi update git commit -m "set dependencies"
git commit -m "reformatting" pixi run fmt
set dagster home to some directory
your logs and run results will be persistet there
mkdir /path/to/mydagsterhome
export DAGSTERHOME=/path/to/mydagster_home
start the data stack
pixi run start-dev

Post install:

• update the secrets in the .env files by executing: openssl rand -base64 32 and setting a suitable secret
• ensure the .env.enc can be created by following the instructions in [documentation/secops]

Developing the template

To render an instance of the project:

pixi run render-dev