#Data-ingestion
Showing 16 of 16 repositories tagged #data-ingestion, ranked by stars
SeaTunnel is a multimodal, high-performance, distributed, massive data integration tool.
ingestr is a CLI tool to copy data between any databases with a single command seamlessly.
Build data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end flows.
Copy to/from Parquet in S3, Azure Blob Storage, Google Cloud Storage, http(s) stores, local files or standard inout stream from within PostgreSQL
The Supabase of AI era. A modular, open-source backend for building AI-native software — designed for knowledge, not static data.
Orbital automates integration between data sources (APIs, Databases, Queues and Functions). BFF's, API Composition and ETL pipelines that adapt as your specs change.
Use SQL to build ELT pipelines on a data lakehouse.
Apache Paimon Rust The rust implementation of Apache Paimon.
The Data Engineering Book - หนังสือวิศวกรรมข้อมูล ของคนไทย เพื่อคนไทย
Build complete API integrations with YAML and SQL. Rapid development without vendor lock-in and per-row costs.
Oracle AI Data Platform Workbench Samples
Claude Skills for connecting Claude.ai to local Weaviate vector databases - manage collections, ingest data, and query with RAG
Download and warehouse historical trading data
K2I - Kafka to Iceberg streaming ingestion engine. A Rust CLI tool inspired by Moonlink architecture that consumes from Kafka, buffers with Apache Arrow for sub-second query freshness, and writes to Apache Iceberg tables.
DCEE is a lightweight Python framework for validating data against contracts and enforcing SLA rules. Built on pandas and boto3, it provides simple, fast data validation without heavy dependencies.
IDPS-ESCAPE: Intrusion Detection and Prevention System - Enhanced Security through a Cooperative Anomaly Prediction Engine. SOAR via a Risk-aware Anomaly Detection-based Automated Response (RADAR) subsystem and a deep learning-based AD subsystem (SONAR), integrated with Wazuh, Flowintel, Suricata