#Pyspark
Showing 60 of 117 repositories tagged #pyspark, ranked by stars
the portable Python dataframe library
Simple and Distributed Machine Learning
State of the Art Natural Language Processing
Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.
Implementing best practices for PySpark ETL jobs and applications.
Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, and PySpark and can be used from pure Python code.
SQL data analysis & visualization projects using MySQL, PostgreSQL, SQLite, Tableau, Apache Spark and pySpark.
Lightweight and extensible compatibility layer between dataframe libraries!
Apache Spark & Python (pySpark) tutorials for Big Data Analysis and Machine Learning as IPython / Jupyter notebooks
:truck: Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark
Jupyter magics and kernels for working with remote Spark clusters
Hopsworks - Data-Intensive AI platform with a Feature Store
Sparkling Water provides H2O functionality inside Spark cluster
80+ DevOps & Data CLI Tools - AWS, GCP, GCF Python Cloud Functions, Log Anonymizer, Spark, Hadoop, HBase, Hive, Impala, Linux, Docker, Spark Data Converters & Validators (Avro/Parquet/JSON/CSV/INI/XML/YAML), Travis CI, AWS CloudFormation, Elasticsearch, Solr etc.
Kuwala is the no-code data platform for BI analysts and engineers enabling you to build powerful analytics workflows. We are set out to bring state-of-the-art data engineering tools you love, such as Airbyte, dbt, or Great Expectations together in one intuitive interface built with React Flow. In addition we provide third-party data into data science models and products with a focus on geospatial data. Currently, the following data connectors are available worldwide: a) High-resolution demographics data b) Point of Interests from Open Street Map c) Google Popular Times
Python framework for building efficient data pipelines. It promotes modularity and collaboration, enabling the creation of complex pipelines from simple, reusable components.
🐍 Quick reference guide to common patterns & functions in PySpark.
Pandas, Polars, Spark, and Snowpark DataFrame comparison for humans and more!
Learn Apache Spark in Scala, Python (PySpark) and R (SparkR) by building your own cluster with a JupyterLab interface on Docker. :zap:
Code for "Efficient Data Processing in Spark" Course
End-to-end Data Lakehouse project built on Databricks, following the Medallion Architecture (Bronze, Silver, Gold). Covers real-world data engineering and analytics workflows using Spark, PySpark, SQL, Delta Lake, and Unity Catalog. Designed for learning, portfolio building, and job interviews.
Open Source LeetCode for PySpark, Spark, Pandas and DBT/Snowflake
A tool for building feature stores.
MorphL Community Edition uses big data and machine learning to predict user behaviors in digital products and services with the end goal of increasing KPIs (click-through rates, conversion rates, etc.) through personalization
Possibly the fastest DataFrame-agnostic quality check library in town.
Practice Databricks coding skills with hands-on exercises. Import into Databricks Free Edition, write code, run assertions, check pass/fail. Covers Delta Lake, Spark SQL, PySpark, Auto Loader, medallion architecture, window functions, and more.
Sample project to demonstrate data engineering best practices
Code repository for the "PySpark in Action" book
HandySpark - bringing pandas-like capabilities to Spark dataframes
Data pipeline performing ETL to AWS Redshift using Spark, orchestrated with Apache Airflow
Big Data Modeling, MapReduce, Spark, PySpark @ Santa Clara University
Microsoft Fabric Unified Data Foundation with Databricks & Purview 2026
PySpark Tutorial for Beginners - Practical Examples in Jupyter Notebook with Spark version 3.4.1. The tutorial covers various topics like Spark Introduction, Spark Installation, Spark RDD Transformations and Actions, Spark DataFrame, Spark SQL, and more. It is completely free on YouTube and is beginner-friendly without any prerequisites.
This repository will help you to learn about databricks concept with the help of examples. It will include all the important topics which we need in our real life experience as a data engineer. We will be using pyspark & sparksql for the development. At the end of the course we also cover few case studies.
Custom PySpark Connectors
Big Data for Data Engineers Coursera Specialization from Yandex
Spark 2.0 Python Machine Learning examples
JupyterLab extension that enables monitoring launched Apache Spark jobs from within a notebook
PySpark Algorithms Book: https://www.amazon.com/dp/B07X4B2218/ref=sr_1_2
A data engineering project (Twitter monitor app)
Big Data essentials: Hadoop, MapReduce, Spark. Explore tutorials and demos in Jupyter notebooks—most are self-contained and live, ready to run with a click.
Hadoop-Hive-Spark cluster + Jupyter on Docker
The goal of this project is to build a docker cluster that gives access to Hadoop, HDFS, Hive, PySpark, Sqoop, Airflow, Kafka, Flume, Postgres, Cassandra, Hue, Zeppelin, Kadmin, Kafka Control Center and pgAdmin. This cluster is solely intended for usage in a development environment. Do not use it to run any production workloads.
:globe_with_meridians: Interactive Workshop on GeoAnalysis using PySpark
A lightweight, declarative PySpark framework for data quality validation — check columns, rows, and entire datasets directly in your Spark pipelines
A production-ready PySpark project template with medallion architecture, Python packaging, unit tests, integration tests, CI/CD automation, Databricks Asset Bundles, and DQX data quality framework.
Anovos - An Open Source Library for Scalable feature engineering Using Apache-Spark
This repository provides a comprehensive implementation of a deep neural network-based recommendation system similar to YouTube's. The repo is organized to incl
Methods for the parallel and distributed analysis and mining of the Protein Data Bank using MMTF and Apache Spark.
pyspark-cassandra is a Python port of the awesome @datastax Spark Cassandra connector. Compatible w/ Spark 2.0, 2.1, 2.2, 2.3 and 2.4
Word2Vec models with Twitter data using Spark. Blog:
Soda Spark is a PySpark library that helps you with testing your data in Spark Dataframes
This repo contains commands that data engineers use in day to day work.
Structural Bioinformatics Training Workshop & Hackathon 2018
A simple VS Code devcontainer setup for local PySpark development
Detailed notes and homeworks from 2025 Data Engineering Zoomcamp by Datatalks.Club
This repo collects the open-source work of the Analytics Service within NHS Digital Data Services
Ultimate AWS Data & AI Platform: Real-time flight delay predictions with complete DE, DS, MLOps, Web App & Multi-Agent LLM - All deployed via CDK self-mutating pipelines
Notes on Data Engineering with Pandas, PySpark, Dask, Ray, Arrow DataFusion, Polars etc.