#Data-pipelines
Showing 60 of 80 repositories tagged #data-pipelines, ranked by stars
Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
An orchestration platform for the development, production, and observation of data assets.
Convert documents to structured data effortlessly. Unstructured is open-source ETL solution for transforming complex documents into clean, structured formats for language models. Visit our website to learn more about our enterprise grade Platform product for production grade workflows, partitioning, enrichments, chunking and embedding.
Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code
Kedro is a toolbox for production-ready data science. It uses software engineering best practices to help you create data engineering and data science pipelines that are reproducible, maintainable, and modular.
🧙 Build, run, and manage data pipelines for integrating and transforming data.
Easy Data Preparation with latest LLMs-based Operators and Pipelines.
🦀 event stream processing for developers to collect and transform data in motion to power responsive data intensive applications.
Preswald is a WASM packager for Python-based interactive data apps: bundle full complex data workflows, particularly visualizations, into single files, runnable completely in-browser, using Pyodide, DuckDB, Pandas, and Plotly, Matplotlib, etc. Build dashboards, reports, and notebooks that run offline, load fast, and share like a document.
Build data pipelines, the easy way 🛠️
A system for agentic LLM-powered data processing and ETL
Maestro: Netflix’s Workflow Orchestrator
Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.
The dbt-native data observability solution for data & analytics engineers. Monitor your data pipelines in minutes. Available as self-hosted or cloud service with premium features.
The best place to learn data engineering. Built and maintained by the data engineering community.
The Feldera Incremental Computation Engine
Fast and efficient unstructured data extraction. Written in Rust with bindings for many languages.
Build data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end flows.
Concurrent Python made simple
A comprehensive Python package template to kickstart and standardize your MLOps initiatives and data pipelines.
First open-source data discovery and observability platform. We make a life for data practitioners easy so you can focus on your business.
visual data prep powered by python
Optimus is an easy-to-use, reliable, and performant workflow orchestrator for data transformation, data modeling, pipelines, and data quality management.
This dbt package captures metadata, artifacts, and test results so you can detect anomalies, monitor data quality, and build metadata tables. It powers Elementary OSS and feeds the wider context layer used by Elementary Cloud’s full Data & AI Control Plane.
One framework to develop, deploy and operate data workflows with Python and SQL.
AI agent tooling for data engineering workflows.
Dataplane is an Airflow inspired unified data platform with additional data mesh and RPA capability to automate, schedule and design data pipelines and workflows. Dataplane is written in Golang with a React front end.
Work with your web service, database, and streaming schemas in a single format.
A curated list of awesome projects and resources related to Kubeflow (a CNCF incubating project)
A lightweight CLI tool for versioning data alongside source code and building data pipelines.
Runtime for building and managing AI agents and Workflows. Easy to learn, fast to build, High Performance, Reliable by design, Intuitive UI, Production Ready.
A lightweight workflow engine
Relational Workflows: where database schemas define executable data pipelines.
graph based stream processing framework
Cloud-native, data onboarding architecture for Google Cloud Datasets
An Open Source PHP Reporting Framework that helps you to write perfect data reports or to construct awesome dashboards in PHP. Working great with all PHP versions from 5.6 to latest 8.0. Fully compatible with all kinds of MVC frameworks like Laravel, CodeIgniter, Symfony.
The open core .NET job orchestrator that we've been missing
Data pipelines from re-usable components
Conductor OSS SDK for Python programming language
Developed a data pipeline to automate data warehouse ETL by building custom airflow operators that handle the extraction, transformation, validation and loading of data from S3 -> Redshift -> S3
Beneath is a serverless real-time data platform ⚡️
The developer-friendly ETL platform for transforming data in real-time. Based on Apache Kafka® and Kubernetes®.
The REST API and execution engine for the Didact Platform.
Classwork projects and home works done through Udacity data engineering nano degree
A robust (🐢) and fast (🐇) MLOps tool for managing data and pipelines in Rust (🦀)
Found a data engineering challenge or participated in a selection process ? Share with us!
Easiest way to monitor asynchronous data pipelines
Building data processing pipelines for documents processing with NLP using Apache NiFi and related services
A high-performance, extremely flexible, and easily extensible universal workflow engine.
Udacity Data Engineering Nanodegree Program
The practical use-cases of how to make your Machine Learning Pipelines robust and reliable using Apache Airflow.
Databricks DLT Apparel Pipeline Project: Learn medallion architecture, streaming, and data engineering with Delta Live Tables. Includes synthetic data, step-by-step guide, and certification prep.
A Pub/Sub for Tables based data integration platform, to discover, publish, modify and consume data effortlessly.
Example of an ETL Pipeline using Airflow
Best practices for data workflows, integrations with the Modern Data Stack (MDS), Infrastructure as Code (IaC), Cloud Provider Services
A configuration-driven framework for building Dagster pipelines that enables teams to create and manage data workflows using YAML/JSON instead of code
Never sift through endless dbt™ logs again. dbt Command Center is a free, open-source, local web application that provides a user-friendly interface to monitor and manage dbt runs.
Data Engineering portfolio projects, resources used to study data tools...
A framework for rapid development of robust data pipelines following a simple design pattern