#Data-engineering
Showing 60 of 849 repositories tagged #data-engineering, ranked by stars
Apache Superset is a Data Visualization and Data Exploration Platform
Learn how to develop, deploy and iterate on production-grade ML applications.
Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
Data Engineering Zoomcamp is a free 9-week course on building production-ready data pipelines. The next cohort starts in January 2026. Join the course here ๐๐ผ
๐ Papers & tech blogs by companies sharing their work on data science & machine learning in production.
Event Driven Orchestration & Scheduling Platform for Mission Critical Applications
Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
Open-source data movement for ELT pipelines and AI agents โ from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.
Turns Data and AI algorithms into production-ready web applications in no time.
Workflow Engine for Kubernetes
An orchestration platform for the development, production, and observation of data assets.
The Data Engineering Cookbook
Roadmap to becoming a data engineer in 2021
Always know what to expect from your data.
Incremental engine for long horizon agents ๐ Star if you like it!
๐ Python-powered shell. Full-featured, cross-platform and AI-friendly.
Event streaming platform for agentic AI. Continuously ingest, transform, and serve event streams in real time, at scale.
๐ง Build, run, and manage data pipelines for integrating and transforming data.
Fancy stream processing made operationally mundane
Open Source Feature Flags, Experimentation, and Product Analytics
The Open Source Feature Store for AI/ML
LLM-Driven Extraction of Unstructured Data โ Built for API Deployments & ETL Pipeline Workflows
Business intelligence as code: build fast, interactive data visualizations in SQL and markdown
Data pipelines for cloud config and security data. Build cloud asset inventory, CSPM, FinOps, and vulnerability management solutions. Extract from AWS, Azure, GCP, and 70+ cloud and SaaS sources.
High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale
data load tool (dlt) is an open source Python library that makes data loading easy ๐ ๏ธ
lakeFS - Data version control for your data lake | Git for data
Privacy and Security focused Segment-alternative, in Golang and React
DeepAnalyze is the first agentic LLM for autonomous data science. ๐ไฝ ็AIๆฐๆฎๅๆๅธ๏ผ่ชๅจๅๆๅคง้ๆฐๆฎ๏ผไธ้ฎ็ๆไธไธๅๆๆฅๅ๏ผ
SQL Translator is a tool for converting natural language queries into SQL code using artificial intelligence. This project is 100% free and open source.
Data Science Roadmap from A to Z
pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).
A list of useful resources to learn Data Engineering from scratch
Maestro: Netflixโs Workflow Orchestrator
Blazing-fast Data-Wrangling toolkit
A collection of scientific methods, processes, algorithms, and systems to build stories & models.
The fastest โก๏ธ way to build data pipelines. Develop iteratively, deploy anywhere. โ๏ธ
Memphis.dev is a highly scalable and effortless data streaming platform
Learn how to design, develop, deploy and iterate on production-grade ML applications.
An Awesome List of Open-Source Data Engineering Projects
Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.
Apache DevLake is an open-source dev data platform to ingest, analyze, and visualize the fragmented data from DevOps tools, extracting insights for engineering excellence, developer experience, and community growth.
Compare tables within or across databases
Learn to build your Second Brain AI assistant with LLMs, agents, RAG, fine-tuning, LLMOps and AI systems techniques.
Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.
Apache Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.
A low code Machine Learning personalized ranking service for articles, listings, search results, recommendations that boosts user engagement. A friendly Learn-to-Rank engine
Data Contracts engine for the modern data stack. https://www.soda.io
Datart is a next generation Data Visualization Open Platform
Implementing best practices for PySpark ETL jobs and applications.
Python Stream Processing
The best place to learn data engineering. Built and maintained by the data engineering community.
Few projects related to Data Engineering including Data Modeling, Infrastructure setup on cloud, Data Warehousing and Data Lake development.
Feathr โ A scalable, unified data and AI engineering platform for enterprise
An document retrieval project built on SAG
A curated list of awesome dbt resources
More than 2000+ Data engineer interview questions.
MLRun is an open source MLOps platform for quickly building and managing continuous ML applications across their lifecycle. MLRun integrates into your development and CI/CD environment and automates the delivery of production data, ML pipelines, and online applications.
๐ฅ๐ฅ๐ฅ Open source Reverse ETL - alternative to hightouch and census.
A comprehensive list of 180+ YouTube Channels for Data Science, Data Engineering, Machine Learning, Deep learning, Computer Science, programming, software engineering, etc.