#Delta-lake
Showing 33 of 33 repositories tagged #delta-lake, ranked by stars
Official repository of Trino, the distributed SQL query engine for big data, formerly known as PrestoSQL (https://trino.io)
Create full-fledged APIs for slowly moving datasets without writing a single line of code.
A native Rust library for Delta Lake, with bindings into Python
Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.
Real-time analytics on Postgres tables
An open protocol for secure data sharing
Python framework for building efficient data pipelines. It promotes modularity and collaboration, enabling the creation of complex pipelines from simple, reusable components.
Analytical database for data-driven Web applications 🪶
The Lakehouse Engine is a configuration driven Spark framework, written in Python, serving as a scalable and distributed engine for several lakehouse algorithms, data flows and utilities for Data Products.
Quick start: pip install jsoniq ⛈️ RumbleDB 2.1.0 "Cedrus Libani" 🌳 for Apache Spark | Run queries on your large-scale, messy datasets (JSON, text, CSV, Parquet, Delta...) | Data Lakehouse with Updates, Scripting, Declarative Machine Learning and more
Practice Databricks coding skills with hands-on exercises. Import into Databricks Free Edition, write code, run assertions, check pass/fail. Covers Delta Lake, Spark SQL, PySpark, Auto Loader, medallion architecture, window functions, and more.
Sample project to demonstrate data engineering best practices
A Minimalistic Rust Implementation of Delta Sharing Server.
🦆 Batch data pipeline with Airflow, DuckDB, Delta Lake, Trino, MinIO, and Metabase. Full observability and data quality.
A production-ready PySpark project template with medallion architecture, Python packaging, unit tests, integration tests, CI/CD automation, Databricks Asset Bundles, and DQX data quality framework.
Books and Papers in Mathematics, Econometrics, Machine Learning, Finance etc for different levels that can be useful for Data Scientists, Developers and everyone whoo is interesting in STEM.
100+ data engineering projects from scratch — streaming, CDC, table formats, query engines, consensus, governance. 2,500+ tests, mypy strict.
Exercícios do módulo 1 - Bootcamp EDC - IGTI 2021
600+ free patterns and concepts for data engineers on Azure Databricks, Microsoft Fabric, and PySpark. 12 books covering the full stack. Free forever.
Open Source data engineering demo project using dbt, DuckDB, dlt, Dagster and Metabase. Two storage modes for the delta tables are supported: local and Microsoft Fabric Onelake.
100 essential Databricks concepts for data engineers, organized by category with difficulty levels and self-assessment scoring
170+ curated resources every Databricks Data Engineer should bookmark - tools, courses, creators, labs, and communities
Data Engine for AI/Algo Trading: Download/Stream -> Clean -> Store. Supports Data Lakehouse Architecture. Clean Once and Forget.
Production-style real-time e-commerce lakehouse with Kafka, Airflow, Databricks, Medallion architecture, data quality, quarantine, Terraform, and Dash analytics.
Template to spin up delta lake locally using docker
Big Data CONFERENCE Europe 2025 🇱🇹 AI, Cloud and Data Conference - Build Your First End-to-End Lakehouse Solution (aka.ms/fabconlake)
PawMark is a platform for developers to build, schedule and monitor data pipelines.
Delta Lake Optimization Project: Hands‑on lab to explore partitioning, Z‑Ordering, compaction (manual & auto), Liquid Clustering, and VACUUM using a synthetic sales dataset in Databricks. Includes a step‑by‑step notebook to measure file scans, bytes read, and query performance for each optimization.
OpenAutoLoader: A lightweight, open-source alternative to Databricks Auto Loader. Built with Polars and SQLite for efficient, incremental file ingestion.
Metadata-driven framework for Databricks Spark Declarative Pipelines. Config-driven, pattern based approach to batch & streaming across the medallion architecture. Deploys via Declarative Automation Bundles. Built for simplicity, extensibility, and alignment with the Databricks product roadmap.
Databricks Real-Time Fintech Monitoring Pipeline: Hands-on lab to build a streaming fraud detection system using Auto Loader, watermarked deduplication, stream-static joins, and windowed rules engines in Databricks. Covers dual-SLA architecture for real-time alerts and batch compliance reporting.
Building a poor man's data lake: Exploring the Power of Polars and Delta Lake
A comprehensive educational resource hub dedicated to mastering Microsoft Fabric, offering in-depth tutorials, real-world use cases, and hands-on guides for seamless end-to-end analytics