End-to-end data platform leveraging the Modern data stack
Last updated Oct 21, 2025
52
Stars
6
Forks
0
Issues
0
Stars/day
Attention Score
10
Language breakdown
Python 54.9%
HCL 33.4%
Makefile 9.0%
Dockerfile 2.7%
▸ Files
click to expand
README
Overview
Article
Check my blog article for more details : https://medium.com/datadriveninvestor/how-i-built-this-data-platform-in-one-week-13b457d7c323Setup
- Before running the terraform apply
- Snowflake structure
RAW : database to store raw data coming from Airbyte (schemas : postgres_airbyte )
ANALYTICS : the production database (schemas: staging, intermediate, marts(finance))
DBT_DEV: the dev database (the same schemas as the production database)
DATAENGINEER : A role to allow usage of RAW database and ownership of ANALYTICS AND DBTDEV AIRBYTEROLE : used by airbyte to write in the RAW database (postgresairbyte schema)
- Ingest data (daily) from RDS to Snowflake : Airbyte
DBT structure
Source data model
Warehouse data model 
Airflow :
- When the CD to deploy the airflow code is done, we need to execute this on the EC2 instance:
cd airflow
docker exec (webservercontainerid) /bin/bash -c "source /opt/airflow/dbtvenv/bin/activate && cd /opt/airflow/dags/dbt/dbttransformation/ && dbt deps && deactivate"- Then create the snowflake_conn
Dashboard
What need to be improved ?
- Terraform resource to create the tables on snowflake
- Unit test airflow dag
