#Elt
Showing 53 of 53 repositories tagged #elt, ranked by stars
Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.
dbt enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.
SeaTunnel is a multimodal, high-performance, distributed, massive data integration tool.
🧙 Build, run, and manage data pipelines for integrating and transforming data.
Data pipelines for cloud config and security data. Build cloud asset inventory, CSPM, FinOps, and vulnerability management solutions. Extract from AWS, Azure, GCP, and 70+ cloud and SaaS sources.
data load tool (dlt) is an open source Python library that makes data loading easy 🛠️
Privacy and Security focused Segment-alternative, in Golang and React
A system for agentic LLM-powered data processing and ETL
Maestro: Netflix’s Workflow Orchestrator
Scalable and efficient data transformation framework - backwards compatible with dbt.
Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.
Open-source BI for engineers
🌊 Continuously synchronize the systems where your data lives, to the systems where you _want_ it to live, by managing your data flows with Estuary. 🌊
Sling is a CLI tool that extracts data from a source storage/database and loads it in a target storage/database.
Kuwala is the no-code data platform for BI analysts and engineers enabling you to build powerful analytics workflows. We are set out to bring state-of-the-art data engineering tools you love, such as Airbyte, dbt, or Great Expectations together in one intuitive interface built with React Flow. In addition we provide third-party data into data science models and products with a focus on geospatial data. Currently, the following data connectors are available worldwide: a) High-resolution demographics data b) Point of Interests from Open Street Map c) Google Popular Times
Local-first ETL/ELT studio: a drag-and-drop visual pipeline designer that compiles to SQL and runs on DuckDB. Tiny desktop app, no servers, git-friendly workspaces.
Optimus is an easy-to-use, reliable, and performant workflow orchestrator for data transformation, data modeling, pipelines, and data quality management.
One framework to develop, deploy and operate data workflows with Python and SQL.
The data-validation toolkit for enhanced dbt (data build tool) PR review
Astro SDK allows rapid and clean development of {Extract, Load, Transform} workflows using Python and SQL, powered by Apache Airflow.
PyAirbyte brings the power of Airbyte to every Python developer. Powers the Airbyte Cloud Replication MCP.
Use SQL to build ELT pipelines on a data lakehouse.
Airbyte made simple (no UI, no database, no cluster)
Data processing and modelling framework for automating tasks (incl. Python & SQL transformations).
Collection of Sample Databricks Spark Notebooks ( mostly for Azure Databricks )
Airbyte's CLI for managing local Airbyte installations
Sample project that use Dagster, dbt, DuckDB and Dash to visualize car and motorcycle Spanish market
git push your data stack with Airbyte, Airflow, and dbt - 2022 Airflow Summit
Extract Load Transform (ELT) framework is a metadata based batch orchestration framework for modern data platforms. Implemented using Azure PaaS data services. Common ingestion and transformation patterns available out of box. Reusable code can be easily extended to cater to custom patterns.
ETL / ELT / Reverse ETL Framework powered by DuckDB, designed to seamlessly integrate and process data from diverse sources. It leverages Markdown as a configuration medium, where YAML blocks define metadata for each data source, and embedded SQL blocks specify the extraction, transformation, and loading logic.
Python based Wikidata framework for easy dataframe extraction
Companion repository to the ETL & ELT Pipelines with Apache Airflow® eBook
Repositório de desenvolvimento (MVP) para estruturação analítica do CadÚnico municipal. Sem dados pessoais — apenas código, dicionários, esquemas e amostras sintéticas. Alinhado à LGPD e boas práticas de governança.
Arcane Insight is a data analytics project designed to harness the power of SQLMesh & DuckDB to collect, transform, and analyze data from Blizzard’s Hearthstone API. Focused on card statistics and attributes, this project reveals detailed insights into card mechanics, strengths, and trends to support BI and strategic analysis.
A configuration-driven framework for building Dagster pipelines that enables teams to create and manage data workflows using YAML/JSON instead of code
Frontend & BFF (Backend for frontend) for Olake. This includes the UI code and backend code for storing the configuration of sync and orchestrating it.
Never sift through endless dbt™ logs again. dbt Command Center is a free, open-source, local web application that provides a user-friendly interface to monitor and manage dbt runs.
Spark data pipeline that processes movie ratings data.
A Python package to load complex XML files into a relational database
ELT Code for your Data Warehouse
A project for exploring how Great Expectations can be used to ensure data quality and validate batches within a data pipeline defined in Airflow.
A batch processing data pipeline, using AWS resources (S3, EMR, Redshift, EC2, IAM), provisioned via Terraform, and orchestrated from locally hosted Airflow containers. The end product is a Superset dashboard and a Postgres database, hosted on an EC2 instance at this address (powered down):
This project demonstrates how to build and automate an ETL pipeline written in Python and schedule it using open source Apache Airflow orchestration tool on AWS EC2 instance.
End-to-end ELT data engineering project
DBConvert Streams: Database IDE, Federated SQL, Real-time CDC & AI assistants via MCP — explore, query, and replicate data across databases and files
Data Brewery is an ETL (Extract-Transform-Load) program that connect to many data sources (cloud services, databases, ...) and manage data warehouse workflow.
One platform to build, run, and operate every data workflow your team touches. Source-available, self-hosted.
Metadata-driven framework for Databricks Spark Declarative Pipelines. Config-driven, pattern based approach to batch & streaming across the medallion architecture. Deploys via Declarative Automation Bundles. Built for simplicity, extensibility, and alignment with the Databricks product roadmap.
Extract transform load CLI tool for extracting small and middle data volume from sources (databases, csv files, xls files, gspreadsheets) to target (databases, csv files, xls files, gspreadsheets) in free combination.
An analytics engineering sandbox focusing on real estates prices in Cook County, IL
Airflow POC demo : 1) env set up 2) airflow DAG 3) Spark/ML pipeline | #DE
MCP server for the Python library dlt