Topics

#arrow #dask #dask-distributed #data-engineering #datafusion #jupyter #numpy #pandas #polars #pyspark #ray #spark

Language breakdown

Jupyter Notebook 100.0%

Rust 0.0%

CSS 0.0%

▸ Files click to expand

README

Data Munging Using \X\ in Python, Rust & Julia

Data Engineering Workshops on some of the more popular libraries, frameworks and tech circa 2023-2024. Data Wrangling with Python, Rust and Julia, Image © Shaurya Agarwal, created using Dalle and GIMP

Data Wrangling with Python, Rust and Julia, Image © Shaurya Agarwal, created using Dalle and GIMP

Data Engineers working with Python, Rust and Julia :P

---

Notebooks

00 Python Collections

This set of notebooks works through examples of how some pretty sophisticated data engineering can be done using Python Collections, Itertools and Functools. It uses the small MovieLens dataset.

Basic Collections and the
Module: Notebook also

01 Numpy

NumPy vs Python Collections Notebook also

02 Pandas

Wrangling MovieLens with Pandas - Part 1: Getting Started, Load the MovieLens dataset: Notebook also
Wrangling MovieLens with Pandas - Part 2: Playing with the Movies and Ratings data: Notebook also

03 Spark

01 - Toy introduction to the basics

01 - Setting up Spark locally (on Windows): Notebook also

02 - How to run Apache Spark based notebooks in Google Colab: Notebook%20GoogleColabsetupSparkdownloaddata.ipynb) also %20GoogleColabsetupSparkdownloaddata.ipynb)

02 - A set of notebooks exploring data wrangling in depth using the MovieLens dataset

Part 01: Overview, Starting Spark and Loading the data: Notebook or

Part 02: Data Analysis basics using tags.csv from the MovieLens dataset: Notebook or

03 - Harder to solve problems - trying to capture a variety of "problem types" at appear hard to solve in the wild

01: Just the questions: Notebook.ipynb) or .ipynb)

02: Added hints, patterns etc. to the questions, read through if you are not really doing anything this afternoon: Notebook.ipynb) or .ipynb)

04 Dask

Distributed Data Analysis with Dask - Part 1: Getting Started, Load the MovieLens dataset: Notebook also
Distributed Data Analysis with Dask - Part 2: Playing with the Movies data: Notebook also

05 Polars

Polars with the MovieLens dataset - Getting Started, Load the MovieLens dataset, A quick look at Arrow, and some analysis: Notebook also

06 Apache Arrow and DataFusion

01 - 10+ minutes to Arrow+DataFusion+Ballista [WIP]: Notebook also

07 Ray

[WIP]

99 Static: The TPC Benchmark Queries

[WIP]

Note

The "10+ minutes to XX" notebooks are just references, not to be run as actual workshop material. These are there to carry toy examples that "getting started" pages for XX carry. I have tried to ensure there's a 10+ minutes notebook for each data engineering library/framework considered here. While it may be interesting to go through these to quickly refresh the syntax and other idiosyncracies, the actual data munging happens in other notebooks.

References

01 Numpy

02 Pandas

03 Spark

* This is also available as live binder notebooks: * Live Notebook: DataFrame * Live Notebook: Pandas API on Spark

Spark SQL and Built-in Functions Reference
weak references, some dated but interesting

* Spark Python Notebooks * Data Science Engineering, your way * A scalable on-line movie recommender using Spark and Flask * The SparkLearning Repo

04 Dask

The approach is different: Dask focuses on Task scheduling vs Spark's Map-Reduce

10 minutes to Dask
90-minute Dask tutorial video
Talks and tutorials page
The Dask tutorial notebooks
The SciPy 2022 tutorial talk
Journey of a Task
High level performance of Pandas, Dask, Spark, and Arrow - from Dask Working Notes Blog
Dask distributed
Dask Task Graphs
Tornado - used by Dask distributed
For some Dask exercises, we may need GraphViz or Cytoscape and ipycytoscape

05 Polars

Polars User Guide and Getting Started

06 Arrow, Arrow DataFusion and Ballista

07 Ray

Future State / Miscellany

Datasets we use:

There's a lot of interesting (interesting to me) tools, datasets and papers out there. When there's time or need, we'll get to them as well. Arrow and pyArrow really warrant a deeper study. Maybe a gateway to Rust based data processing. Not really emerging* anymore, a lot of very cool stuff is being done with this and datafusion, very interesting to explore.