#Data-lake
Showing 38 of 38 repositories tagged #data-lake, ranked by stars
data load tool (dlt) is an open source Python library that makes data loading easy 🛠️
lakeFS - Data version control for your data lake | Git for data
Apache Kyuubi is a distributed and multi-tenant gateway to provide serverless SQL on data warehouses and lakehouses.
Few projects related to Data Engineering including Data Modeling, Infrastructure setup on cloud, Data Warehousing and Data Lake development.
An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.
Lakekeeper is an Apache-Licensed, secure, fast and easy to use Apache Iceberg REST Catalog written in Rust.
Personal Data Engineering Projects
Data API Framework for AI Agents and Data Apps
This repository has been merged into Canner/WrenAI under the core/ directory
Enterprise-grade, production-hardened, serverless data lake on AWS
Real Time Big Data / IoT Machine Learning (Model Training and Inference) with HiveMQ (MQTT), TensorFlow IO and Apache Kafka - no additional data store like S3, HDFS or Spark required
Use SQL to build ELT pipelines on a data lakehouse.
Resources for video demonstrations and blog posts related to DataOps on AWS
Data Forge — a modern data stack playground to practice flows and best practices, not just tools. Spark, Trino, Kafka, Iceberg, ClickHouse, Airflow, MinIO, Superset — all wired together locally with Docker Compose.
🧊 A curated list of OLAP databases, data lake tools, columnar engines, and analytics frameworks for data engineers.
Icebird: JavaScript Iceberg Client
Road to Azure Data Engineer Part-I: DP-200 - Implementing an Azure Data Solution
A Declarative framework for Building, Maintaining, and Analyzing Graph Data
Udacity Data Engineering Nanodegree Program
Data Engineer with Python lecture notes from #datacamp.
ETL / ELT / Reverse ETL Framework powered by DuckDB, designed to seamlessly integrate and process data from diverse sources. It leverages Markdown as a configuration medium, where YAML blocks define metadata for each data source, and embedded SQL blocks specify the extraction, transformation, and loading logic.
rtdl makes it easy to build and maintain a real-time data lake
📒(GitBook) A curated list of awesome Data Engineering resources
Breaking Cloud Native Web APIs in their natural habitat.
Portfolio of projects and studies conducted in data engineering.
Data Engineering - Metropolitan Transportation Authority (MTA) Subway Data Analysis
DatAasee - A Metadata-Lake for Libraries
JobAnalytics system consumes data from multiple sources and provides valuable information to both job hunters and recruiters.
Open Data Stack Platform: a collection of projects and pipelines built with open data stack tools for scalable, observable data platform.
DCEE is a lightweight Python framework for validating data against contracts and enforcing SLA rules. Built on pandas and boto3, it provides simple, fast data validation without heavy dependencies.
DS525
Rust-native time-series table format with gap/overlap tracking and SQL queries
An end-to-end data pipeline for building Data Lake and supporting report using Apache Spark.
A collection of data engineering projects: data modeling, ETL pipelines, data lakes, infrastructure configuration on AWS, data warehousing, containerization, and a dashboard to monitor data pipeline KPIs
a simple lakeFS webhook for pre-commit and pre-merge validation of data objects
a literature review analyzing ClickHouse's compute-storage separation landscape, comparing the available offerings to assess the production readiness of stateless CH deployments, and identify future prospects for the architecture
DataHut-DuckHouse is a modern, modular, and multi-tenant analytics platform that combines DuckDB, Apache Iceberg, Arrow Flight, dbt, and Trino to build a hybrid, lightweight, and scalable data stack ready for SaaS.
Extract transform load CLI tool for extracting small and middle data volume from sources (databases, csv files, xls files, gspreadsheets) to target (databases, csv files, xls files, gspreadsheets) in free combination.