#Data-engineering-pipeline
Showing 27 of 27 repositories tagged #data-engineering-pipeline, ranked by stars
Few projects related to Data Engineering including Data Modeling, Infrastructure setup on cloud, Data Warehousing and Data Lake development.
An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.
One framework to develop, deploy and operate data workflows with Python and SQL.
Data pipeline performing ETL to AWS Redshift using Spark, orchestrated with Apache Airflow
Data Engineering Project with Hadoop HDFS and Kafka
Project demonstrating how to automate Prefect 2.0 deployments to AWS ECS Fargate
Code examples showing flow deployment to various types of infrastructure
Classwork projects and home works done through Udacity data engineering nano degree
Let your pipe lines flow thru the Python code in Xonsh shell.
Agentic Data Integrator that helps you build production-ready data pipelines so you can connect to more systems, faster. You run it in your terminal as a workflow wizard.
Apache Spark Guide
Deploy a Prefect flow to serverless AWS Lambda function
Distributed Data Processing Pipeline for MCP.
A end-to-end real-time stock market data pipeline with Python, AWS EC2, Apache Kafka, and Cassandra Data is processed on AWS EC2 with Apache Kafka and stored in a local Cassandra database.
End-to-end data engineering pipeline with various technologies to ingest real time data.
A Python package of end-to-end weather data clients & raw data clients with VPN/PROXY support, data processors that decode variable keys from GRIB format into a plain-language format & various tools for assisting Python automated workflows, querying meteorological datasets and filling gaps in meteorological data.
A Data Engineering Project that implements an ETL data pipeline using Dagster, Apache Spark, Streamlit, MinIO, Metabase, Dbt, Polars, Docker. Data from kaggle and youtube-api
Reusable data engineering toolkit My personal data infrastructure
An end-to-end data pipeline for building Data Lake and supporting report using Apache Spark.
Challenge to job: Data Scientist
An end-to-end Twitter Data Pipeline that extracts data from Twitter and loads it into AWS S3.
Data Engineering pipeline hosted entirely in the AWS ecosystem utilizing DocumentDB as the database
The goal of this project is to analyse the impact of Covid-19 on the Aviation industry through data engineering processes using technologies such as Apache Airflow, Apache Spark, Tableau and couple of AWS services
Marshmallow serializer integration with pyspark
Building a modern data warehouse with Microsoft SQL Server, including ETL processes with Bronze Layer, Silver Layer and the Gold Layer, data modeling and as well as analytics.
Building a next-generation hybrid data pipeline architecture that combines the power of Microsoft Fabric, Azure Cloud, and Power BI. This pipeline is engineered to tackle the challenges of real-time data ingestion, multi-layered processing, and analytics, delivering business-critical insights.
Social Media Analysis, scalable solution, flexible deployment that analyses social media contents