#Etl-pipeline
Showing 60 of 73 repositories tagged #etl-pipeline, ranked by stars
Event streaming platform for agentic AI. Continuously ingest, transform, and serve event streams in real time, at scale.
Build data pipelines, the easy way 🛠️
Apache Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.
Implementing best practices for PySpark ETL jobs and applications.
Few projects related to Data Engineering including Data Modeling, Infrastructure setup on cloud, Data Warehousing and Data Lake development.
An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.
The open document intelligence platform for builders and hackers - DMS for the agentic world
A scalable general purpose micro-framework for defining dataflows. THIS REPOSITORY HAS BEEN MOVED TO www.github.com/dagworks-inc/hamilton
A comprehensive guide to building a modern data warehouse with SQL Server, including ETL processes, data modeling, and analytics.
A Clojure high performance data processing system
A blazingly fast general purpose blockchain analytics engine specialized in systematic mev detection
Integrate LLM in any pipeline - fit/predict pattern, JSON driven flows, and built in concurency support.
The Supabase of AI era. A modular, open-source backend for building AI-native software — designed for knowledge, not static data.
A Task-Based Workflow Orchestration Engine
An end-to-end data engineering pipeline that orchestrates data ingestion, processing, and storage using Apache Airflow, Python, Apache Kafka, Apache Zookeeper, Apache Spark, and Cassandra. All components are containerized with Docker for easy deployment and scalability.
sync/async iterable streams for Python
Service for bulk-loading data to databases with automatic schema management (Redshift, Snowflake, BigQuery, ClickHouse, Postgres, MySQL)
This is a template you can use for your next data engineering portfolio project.
A simple Spark-powered ETL framework that just works 🍺
Jayvee is a domain-specific language and runtime for automated processing of data pipelines
Regular practice on Data Science, Machien Learning, Deep Learning, Solving ML Project problem, Analytical Issue. Regular boost up my knowledge. The goal is to help learner with learning resource on Data Science filed.
The goal of this project is to track the expenses of Uber Rides and Uber Eats through data Engineering processes using technologies such as Apache Airflow, AWS Redshift and Power BI.
Data pipelines from re-usable components
This repository will help you to learn about databricks concept with the help of examples. It will include all the important topics which we need in our real life experience as a data engineer. We will be using pyspark & sparksql for the development. At the end of the course we also cover few case studies.
High-performance streaming SQL query engine designed for real-time data processing. Use cases include event-driven architectures, ETL pipelines, and modern data-intensive applications.
Conductor OSS SDK for Python programming language
Watchmen Platform is a low code data platform for data pipeline, meta data management , analysis, and quality management
Prism is the easiest way to develop, orchestrate, and execute data pipelines in Python.
Near real time ETL to populate a dashboard.
A production-ready PySpark project template with medallion architecture, Python packaging, unit tests, integration tests, CI/CD automation, Databricks Asset Bundles, and DQX data quality framework.
Data streaming runtime focused on performance, consistency, and extensibility. Write plugins in Rust or WASM and process data with data guarantees.
Ethereum Analytical Database - Ethereum data access solution that can be used for analytics and application development. The solution works on a fast DB - Clickhouse.
Building data processing pipelines for documents processing with NLP using Apache NiFi and related services
:mens: 💾 Script to import issues from a JIRA instance into a database.
A repository defining a simple data pipleine for ETL jobs relating to media metadata.
Data Engineer with Python lecture notes from #datacamp.
ETL / ELT / Reverse ETL Framework powered by DuckDB, designed to seamlessly integrate and process data from diverse sources. It leverages Markdown as a configuration medium, where YAML blocks define metadata for each data source, and embedded SQL blocks specify the extraction, transformation, and loading logic.
A Pub/Sub for Tables based data integration platform, to discover, publish, modify and consume data effortlessly.
End-to-end ETL pipeline in the Microsoft Azure cloud - (Jun '24 - Jul '24)
Stellar ETL will enable real-time analytics on the Stellar network
Solution for IBM Data Engineering Professional Certificate
Portfolio of projects and studies conducted in data engineering.
The goal of this project is to illustrate Extract Transform Load (ETL) using Python and SQL. ETL is a process commonly done in computing, which takes raw data, cleans it and stores it for later use. The extraction phase targets and retrieves the data. Transform manipulates and cleans the data. Then load stores the data, typically in a data warehouse.
Frontend & BFF (Backend for frontend) for Olake. This includes the UI code and backend code for storing the configuration of sync and orchestrating it.
A Python PySpark Projet with Poetry
Project was based on an interest in Data Engineering, ETL pipeline. It also provided a good opportunity to develop skills and experience in a range of tools. As such, project is more complex than required, utilising dbt, airflow, docker and cloud based storage.
OpenSource data platform to build event-driven systems. It's like Deebezium for golang :)
A end-to-end real-time stock market data pipeline with Python, AWS EC2, Apache Kafka, and Cassandra Data is processed on AWS EC2 with Apache Kafka and stored in a local Cassandra database.
End-to-end data engineering pipeline with various technologies to ingest real time data.
CryptoDataPy is a python library that makes it easy to build high quality data pipelines for the analysis of cryptoassets
A Data Engineering Project that implements an ETL data pipeline using Dagster, Apache Spark, Streamlit, MinIO, Metabase, Dbt, Polars, Docker. Data from kaggle and youtube-api
This project demonstrates how to build and automate an ETL pipeline written in Python and schedule it using open source Apache Airflow orchestration tool on AWS EC2 instance.
Starter project for building an ETL pipeline using SSIS in Visual Studio 2019
A Streamlit + SQLite + Python dashboard for monitoring content performance KPIs. Tracks impressions, clicks, conversions, CTR, and conversion rates across categories and time. Includes an automated ETL from CSV → SQL view, interactive charts, and filters for data-driven content optimization.
Blog post on ETL pipelines with Airflow
Real-Time M-Pesa Transaction Streaming Pipeline built using modern data engineering technologies to ingest, process, stream, and analyze transaction data in real time. Demonstrates event-driven architecture, stream processing, data pipelines, and scalable analytics for fintech applications.
velib-v2: An ETL pipeline that employs batch and streaming jobs using Spark, Kafka, Airflow, and other tools, all orchestrated with Docker Compose.
An end-to-end data engineering pipeline that fetches real-time YouTube analytics and streams them through Kafka for processing with ksqlDB. The processed analytics data is then sent to Telegram for real-time notifications.
📊 ETL pipeline for YouTube competitor analytics. Orchestrated with Airflow, Docker, and AWS. Features sentiment analysis and a Power BI dashboard.
A collection of data engineering projects: data modeling, ETL pipelines, data lakes, infrastructure configuration on AWS, data warehousing, containerization, and a dashboard to monitor data pipeline KPIs