#Data-pipeline
Showing 60 of 116 repositories tagged #data-pipeline, ranked by stars
Open-source data movement for ELT pipelines and AI agents โ from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.
Data processing for and with foundation models! ๐ ๐ ๐ฝ โก๏ธ โก๏ธ๐ธ ๐น ๐ท
High-performance AI pipeline engine with a C++ core and 50+ Python-extensible nodes. Build, debug, and scale LLM workflows with 13+ model providers, 8+ vector databases, and agent orchestration, all from your IDE. Includes VS Code extension, TypeScript/Python SDKs, and Docker deployment.
Privacy and Security focused Segment-alternative, in Golang and React
A list of useful resources to learn Data Engineering from scratch
ingestr is a CLI tool to copy data between any databases with a single command seamlessly.
Local-first workflow engine with a Web UI for small teams. Define DAGs in a declarative YAML format. Self-contained and no DBMS required. Use any AI agent to manage your DAGs.
Memphis.dev is a highly scalable and effortless data streaming platform
An open-source data logging library for machine learning models and data pipelines. ๐ Provides visibility into data quality & model performance over time. ๐ก๏ธ Supports privacy-preserving data collection, ensuring safety & robustness. ๐
The dbt-native data observability solution for data & analytics engineers. Monitor your data pipelines in minutes. Available as self-hosted or cloud service with premium features.
Open-source inference server and production cluster for all the models your agent needs.
CLI task management & automation tool
๐ฅ๐ฅ๐ฅ Open source Reverse ETL - alternative to hightouch and census.
Source code accompanying book: Data Science on the Google Cloud Platform, Valliappa Lakshmanan, O'Reilly 2017
Example end to end data engineering project.
Modern SeaTunnel Web UI with visual DAG pipelines, batch & streaming sync, connector management, built-in metrics, and runtime logs.
๐ Continuously synchronize the systems where your data lives, to the systems where you _want_ it to live, by managing your data flows with Estuary. ๐
Pythonic tool for orchestrating machine-learning/high performance/quantum-computing workflows in heterogeneous compute environments.
Practical Data Engineering: A Hands-On Real-Estate Project Guide
Local-first ETL/ELT studio: a drag-and-drop visual pipeline designer that compiles to SQL and runs on DuckDB. Tiny desktop app, no servers, git-friendly workspaces.
Conduit streams data between data stores. Kafka Connect replacement. No JVM required.
A curated list of open source tools used in analytics platforms and data engineering ecosystem
Augmentation pipeline for rendering synthetic paper printing, faxing, scanning and copy machine processes
Executable memory system for tabular data work
Code review for data in dbt
Streaming reactive and dataflow graphs in Python
Code for "Efficient Data Processing in Spark" Course
DataMate is an enterprise-level data processing platform designed for model fine-tuning and RAG retrieval.
End-to-end Data Lakehouse project built on Databricks, following the Medallion Architecture (Bronze, Silver, Gold). Covers real-world data engineering and analytics workflows using Spark, PySpark, SQL, Delta Lake, and Unity Catalog. Designed for learning, portfolio building, and job interviews.
An end-to-end data engineering pipeline that orchestrates data ingestion, processing, and storage using Apache Airflow, Python, Apache Kafka, Apache Zookeeper, Apache Spark, and Cassandra. All components are containerized with Docker for easy deployment and scalability.
Use LLMs to robustly extract web data
Use SQL to build ELT pipelines on a data lakehouse.
A SQL transformation engine that type-checks your whole pipeline and catches breaking changes before they run โ branches, replay, column-level lineage, compile-time contracts, per-model cost. Adapters: Databricks, Snowflake, BigQuery, DuckDB. Single static Rust binary. Apache 2.0.
Genblaze is an open source Python SDK for orchestrating generative AI media pipelines across video, audio, and image providers with built in provenance for every output.
A Clojure machine learning library
Agentic Data Engineering Harness for building data pipelines, data products, data APIs, and data lakes autonomously
Declarative text based tool for data analysts and engineers to extract, load, transform and orchestrate their data pipelines.
Jayvee is a domain-specific language and runtime for automated processing of data pipelines
Automated Tool for Optimized Modelling
Content for architecting a data science platform for products using Luigi, Spark & Flask.
A ่กใๆ่ดงใๆๆๆฐๆฎๆฌๅฐๅ็ฎก้๏ผTushare Pro ๆๅ โ Parquet ๅๅบๅญๅจ โ DuckDB ๆฌๅฐๆฅ่ฏข๏ผๆฏๆๅข้ๅๆญฅไธๅฎๆถ่ฐๅบฆ
The Data Engineering Book - เธซเธเธฑเธเธชเธทเธญเธงเธดเธจเธงเธเธฃเธฃเธกเธเนเธญเธกเธนเธฅ เธเธญเธเธเธเนเธเธข เนเธเธทเนเธญเธเธเนเธเธข
Data pipelines from re-usable components
SQLpipe makes it easy to move the result of one query from one database to another.
Cortex | Data Frameworkโa cutting-edge SDK that simplifies real-time data processing with intuitive operators, robust state management, and seamless telemetry for efficient, scalable pipelines.
A cross-platform tool for data pipelines.
๐ฆ Batch data pipeline with Airflow, DuckDB, Delta Lake, Trino, MinIO, and Metabase. Full observability and data quality.
Watchmen Platform is a low code data platform for data pipeline, meta data management , analysis, and quality management
(project & tutorial) dag pipeline tests + ci/cd setup
Routines-based, event-driven workflow orchestration for Pythonโcompose complex data/AI pipelines and run concurrent workflows across distributed systems with durable state and robust error handling.
Job board aggregator indexing 1,000,000+ active positions from 20,000+ companies across Greenhouse, Lever, Ashby, Workday, and other major ATS platforms. Multithreaded Python ETL pipeline, daily automated refreshes via GitHub Actions, filterable client-side search.
๐ก Real-time data pipeline with Kafka, Flink, Iceberg, Trino, MinIO, and Superset. Ideal for learning data systems.
OpenSnowcat Collector, an open source fork of Snowplow (Apache 2.0 License)
Found a data engineering challenge or participated in a selection process ? Share with us!
mloda.ai - Open Data Access for AI and ML. Plugin-based. Traceable. Framework-agnostic.
Indonesian Stock Exchange API wrapper for trading data integration.
End to end data engineering project
3DCF / doc2dataset: token-efficient document layer with NumGuard numeric integrity and multi-framework exports for RAG & fine-tuning.
Copy Pandas DataFrames and HDF5 files to PostgreSQL database
ไธไธชๅบไบๆต่งๅจ็ฏๅข็ๆฐๆฎ้้SDK