#Apache-spark
Showing 60 of 73 repositories tagged #apache-spark, ranked by stars
The open source AI engineering platform for agents, LLMs, and ML models. MLflow enables teams of all sizes to debug, evaluate, monitor, and optimize production-quality AI applications while controlling costs and managing access to models and data.
lakeFS - Data version control for your data lake | Git for data
Simple and Distributed Machine Learning
Interactive and Reactive Data Science using Scala and Spark.
Kubernetes operator for managing the lifecycle of Apache Spark applications on Kubernetes.
Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.
.NET for Apache® Spark™ makes Apache Spark™ easily accessible to .NET developers.
Apache Spark docker image
Feathr – A scalable, unified data and AI engineering platform for enterprise
Oryx 2: Lambda architecture on Apache Spark, Apache Kafka for real-time large scale machine learning
SQL data analysis & visualization projects using MySQL, PostgreSQL, SQLite, Tableau, Apache Spark and pySpark.
An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.
R interface for Apache Spark
Fully managed Apache Parquet implementation
This repository contains the development code for sparkMeasure, an Apache Spark performance analysis and troubleshooting library. It simplifies collecting, aggregating, and exporting Spark task/stage metrics, and is designed for practical use by developers and data engineers in interactive analysis, testing, and production monitoring workflows.
Distributed Deep Learning, with a focus on distributed training, using Keras and Apache Spark.
Code for Agile Data Science 2.0, O'Reilly 2017, Second Edition
Infrastructures™ for Machine Learning Training/Inference in Production.
Code for "Efficient Data Processing in Spark" Course
End-to-end Data Lakehouse project built on Databricks, following the Medallion Architecture (Bronze, Silver, Gold). Covers real-world data engineering and analytics workflows using Spark, PySpark, SQL, Delta Lake, and Unity Catalog. Designed for learning, portfolio building, and job interviews.
A Spark UI and Spark History Server alternative with CPU and Memory metrics! Delight is free, cross-platform, and open-source.
An end-to-end data engineering pipeline that orchestrates data ingestion, processing, and storage using Apache Airflow, Python, Apache Kafka, Apache Zookeeper, Apache Spark, and Cassandra. All components are containerized with Docker for easy deployment and scalability.
Use SQL to build ELT pipelines on a data lakehouse.
A pure Python implementation of Apache Spark's RDD and DStream interfaces.
A guide on how to set up Jupyter with Pyspark painlessly on AWS EC2 clusters, with S3 I/O support
MCP Server and CLI for Apache Spark History Server. Debug Spark applications from AI agents, scripts, or the terminal.
Scalable Data Science, course sets in big data Using Apache Spark over databricks and their mathematical, statistical and computational foundations using SageMath.
Big Data Modeling, MapReduce, Spark, PySpark @ Santa Clara University
PySpark Tutorial for Beginners - Practical Examples in Jupyter Notebook with Spark version 3.4.1. The tutorial covers various topics like Spark Introduction, Spark Installation, Spark RDD Transformations and Actions, Spark DataFrame, Spark SQL, and more. It is completely free on YouTube and is beginner-friendly without any prerequisites.
Experiment tracking server focused on speed and scalability
REST API for Apache Spark on K8S or YARN
Flowman is an ETL framework powered by Apache Spark. With its declarative approach, Flowman simplifies the development of complex data pipelines.
JupyterLab extension that enables monitoring launched Apache Spark jobs from within a notebook
Big Data essentials: Hadoop, MapReduce, Spark. Explore tutorials and demos in Jupyter notebooks—most are self-contained and live, ready to run with a click.
⛳️ PASS: Amazon Web Services Certified (AWS Certified) Machine Learning Specialty (MLS-C01) by learning based on our Questions & Answers (Q&A) Practice Tests Exams.
curated list of awesome tools and libraries for specific domains
A lightweight, declarative PySpark framework for data quality validation — check columns, rows, and entire datasets directly in your Spark pipelines
A production-ready PySpark project template with medallion architecture, Python packaging, unit tests, integration tests, CI/CD automation, Databricks Asset Bundles, and DQX data quality framework.
Data engineering interview prep - PySpark notebooks, theory docs, quizzes, and company-specific patterns. Built around Zephyr Coffee Co., a fictional 200-store chain with messy data.
Use Kafka and Apache Spark streaming to perform click stream analytics
The Internals of Spark on Kubernetes
Methods for the parallel and distributed analysis and mining of the Protein Data Bank using MMTF and Apache Spark.
SparkER: an Entity Resolution framework for Apache Spark
Ansible roles to install an Spark Standalone cluster (HDFS/Spark/Jupyter Notebook) or Ambari based Spark cluster
Structural Bioinformatics Training Workshop & Hackathon 2018
Udacity Data Engineering Nanodegree Program
Powerful rapid automatic EDA and feature engineering library with a very easy to use API 🌟
A framework for Spatio-Temporal Data Analytics on Spark
A re-implementation of Hadoop DistCP in Apache Spark
PyJaws: A Pythonic Way to Define Databricks Jobs and Workflows
This project serves as a comprehensive guide to building an end-to-end data engineering pipeline using TCP/IP Socket, Apache Spark, OpenAI LLM, Kafka and Elasticsearch. It covers each stage from data acquisition, processing, sentiment analysis with ChatGPT, production to kafka topic and connection to elasticsearch.
Stream processing guidelines and examples using Apache Flink and Apache Spark
Apache Spark Guide
100 essential Databricks concepts for data engineers, organized by category with difficulty levels and self-assessment scoring
170+ curated resources every Databricks Data Engineer should bookmark - tools, courses, creators, labs, and communities
Data Engineering portfolio projects, resources used to study data tools...
ELT Data Pipeline implementation in Data Warehousing environment
By Smart Shaped s.r.l. (https://www.smartshaped.com/)
End-to-end data engineering pipeline with various technologies to ingest real time data.
