Spark Structured Streaming data pipeline that processes movie ratings data in real-time.
Last updated Jul 3, 2026
14
Stars
7
Forks
0
Issues
0
Stars/day
Attention Score
53
Topics
Language breakdown
Python 72.0%
Makefile 28.0%
▸ Files
click to expand
README
Spark Structured Streaming Demo
Spark Structured Streaming data pipeline that processes movie ratings data in real-time.Consumes events from a Kafka topic in Avro, transforms and writes to an Apache Iceberg table.
The pipeline handles updates and duplicate events by merging to the destination table based on the event_id.
The output table is partitioned by days(ratingtimestamp) (leveraging Iceberg's hidden partitioning for optimal querying)
Data Architecture
Local setup
We spin up a local Kafka cluster with Schema Registry based on the Docker Compose file provided by Confluent.We install a local Spark Structured Streaming app using uv.
Dependency management
Dependabot is configured to periodically upgrade repo dependencies. See dependabot.yml.Running instructions
Run the following commands in order:make setupto install the Spark Structured Streaming app on a local Python env.make kafka-upto start local Kafka in Docker.make kafka-create-topicto create the Kafka topic we will use.make kafka-produce-test-eventsto start writing messages to the topic.
make streaming-app-runto start the Spark Structured Streaming app.
$ make pyspark
>>> df = spark.read.table("movie_ratings")
>>> df.show()
+--------------------+--------------------+--------------------+------+-----------+-------------------+
| eventid| userid| movieid|rating|isapproved| rating_timestamp|
+--------------------+--------------------+--------------------+------+-----------+-------------------+
|ad8f6fa4-f2bf-11f...|ad8f6fb8-f2bf-11f...|ad8f6fc2-f2bf-11f...| 4.1| false|2026-01-16 09:42:31|
|ad8fe38a-f2bf-11f...|ad8fe39e-f2bf-11f...|ad8fe3a8-f2bf-11f...| 9.7| true|2026-01-16 09:42:31|
|ad900496-f2bf-11f...|ad9004aa-f2bf-11f...|ad9004b4-f2bf-11f...| 3.0| false|2026-01-16 09:42:31|
|ad901670-f2bf-11f...|ad90167a-f2bf-11f...|ad901684-f2bf-11f...| 3.0| false|2026-01-16 09:42:31|
+--------------------+--------------------+--------------------+------+-----------+-------------------+Table internal maintenance
The streaming microbatches can produce many small files and constant table snapshots.In order to tackle these issues, the recommended Iceberg table maintenance operations can be used, see doc.
