#Data-integration
Showing 49 of 49 repositories tagged #data-integration, ranked by stars
Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.
Turns Data and AI algorithms into production-ready web applications in no time.
An orchestration platform for the development, production, and observation of data assets.
SeaTunnel is a multimodal, high-performance, distributed, massive data integration tool.
🧙 Build, run, and manage data pipelines for integrating and transforming data.
Data pipelines for cloud config and security data. Build cloud asset inventory, CSPM, FinOps, and vulnerability management solutions. Extract from AWS, Azure, GCP, and 70+ cloud and SaaS sources.
🦀 event stream processing for developers to collect and transform data in motion to power responsive data intensive applications.
Jitsu is an open-source Segment alternative. Fully-scriptable data ingestion engine for modern data teams. Set-up a real-time data pipeline in minutes, not days
Privacy and Security focused Segment-alternative, in Golang and React
Community-curated list of software packages and data resources for single-cell, including RNA-seq, ATAC-seq, etc.
ingestr is a CLI tool to copy data between any databases with a single command seamlessly.
Apache DevLake is an open-source dev data platform to ingest, analyze, and visualize the fragmented data from DevOps tools, extracting insights for engineering excellence, developer experience, and community growth.
superglue (YC W25) builds integrations and tools from natural language. Get production-grade tools for long tail and enterprise systems.
Modern SeaTunnel Web UI with visual DAG pipelines, batch & streaming sync, connector management, built-in metrics, and runtime logs.
🌊 Continuously synchronize the systems where your data lives, to the systems where you _want_ it to live, by managing your data flows with Estuary. 🌊
Kuwala is the no-code data platform for BI analysts and engineers enabling you to build powerful analytics workflows. We are set out to bring state-of-the-art data engineering tools you love, such as Airbyte, dbt, or Great Expectations together in one intuitive interface built with React Flow. In addition we provide third-party data into data science models and products with a focus on geospatial data. Currently, the following data connectors are available worldwide: a) High-resolution demographics data b) Point of Interests from Open Street Map c) Google Popular Times
Local-first ETL/ELT studio: a drag-and-drop visual pipeline designer that compiles to SQL and runs on DuckDB. Tiny desktop app, no servers, git-friendly workspaces.
Continuously updated paper list on advancements in Data Agents. Companion repo to our paper "A Survey of Data Agents: Emerging Paradigm or Overstated Hype?"
Conduit streams data between data stores. Kafka Connect replacement. No JVM required.
A curated list of open source tools used in analytics platforms and data engineering ecosystem
Dataplane is an Airflow inspired unified data platform with additional data mesh and RPA capability to automate, schedule and design data pipelines and workflows. Dataplane is written in Golang with a React front end.
Categorical Query Language IDE
Work with your web service, database, and streaming schemas in a single format.
Use SQL to build ELT pipelines on a data lakehouse.
🎼 Integrate multiple high-dimensional datasets with fuzzy k-means and locally linear adjustments.
Powerful RDF Knowledge Graph Generation with RML Mappings
Declarative text based tool for data analysts and engineers to extract, load, transform and orchestrate their data pipelines.
PANDORA :computer:
The Data Engineering Book - หนังสือวิศวกรรมข้อมูล ของคนไทย เพื่อคนไทย
Cortex | Data Framework—a cutting-edge SDK that simplifies real-time data processing with intuitive operators, robust state management, and seamless telemetry for efficient, scalable pipelines.
Build complete API integrations with YAML and SQL. Rapid development without vendor lock-in and per-row costs.
Prism is the easiest way to develop, orchestrate, and execute data pipelines in Python.
Intuitive guide to multi-omics integration with toy examples: supervised latent components (DIABLO), unsupervised shared/partial/unique structure (DIVAS), VAEs/conditional VAEs, and key baselines (MOFA, JIVE, SNF) with practical tips + code.
Building data processing pipelines for documents processing with NLP using Apache NiFi and related services
A high-performance, extremely flexible, and easily extensible universal workflow engine.
A Pub/Sub for Tables based data integration platform, to discover, publish, modify and consume data effortlessly.
Oracle AI Data Platform Workbench Samples
A configuration-driven framework for building Dagster pipelines that enables teams to create and manage data workflows using YAML/JSON instead of code
Research data management in biomedical and machine learning applications
DBConvert Streams: Database IDE, Federated SQL, Real-time CDC & AI assistants via MCP — explore, query, and replicate data across databases and files
3xtudio is a web-based visual editor for RML mappings
Code for the paper "Match, Compare, or Select? An Investigation of Large Language Models for Entity Matching" (COLING 2025)
An Airflow-based pipeline that fetches data from a free API, cleans and transforms it, and saves it to a database—ready for downstream machine learning.
One platform to build, run, and operate every data workflow your team touches. Source-available, self-hosted.
CloverDX Docker container for CloverDX Server deployment including examples.
Data-Splitter is a Python script designed to split a large CSV file containing data into three different formats: JSON, a database table, and another CSV file. The script ensures a random distribution of data across the three output formats based on custom-defined ratios.
A comprehensive educational resource hub dedicated to mastering Microsoft Fabric, offering in-depth tutorials, real-world use cases, and hands-on guides for seamless end-to-end analytics
Automate your data pipelines using Apache Airflow with this ready-to-use DAG for data integration, ETL and workflow automation.