#Airflow
Showing 60 of 144 repositories tagged #airflow, ranked by stars
Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
Workflow Engine for Kubernetes
Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code
Build data pipelines, the easy way ๐ ๏ธ
Docker Apache Airflow
Elyra extends JupyterLab with an AI centric approach.
Few projects related to Data Engineering including Data Modeling, Infrastructure setup on cloud, Data Warehousing and Data Lake development.
More than 2000+ Data engineer interview questions.
An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.
Example end to end data engineering project.
A Data Engineering & Machine Learning Knowledge Hub
Personal Data Engineering Projects
๐ ๐ง๐ต๐ฒ ๐๐๐น๐น ๐ฆ๐๐ฎ๐ฐ๐ธ ๐ณ-๐ฆ๐๐ฒ๐ฝ๐ ๐ ๐๐ข๐ฝ๐ ๐๐ฟ๐ฎ๐บ๐ฒ๐๐ผ๐ฟ๐ธ | ๐๐ฒ๐ฎ๐ฟ๐ป ๐ ๐๐ & ๐ ๐๐ข๐ฝ๐ for free by designing, building and deploying an end-to-end ML batch system ~ ๐ด๐ฐ๐ถ๐ณ๐ค๐ฆ ๐ค๐ฐ๐ฅ๐ฆ + 2.5 ๐ฉ๐ฐ๐ถ๐ณ๐ด ๐ฐ๐ง ๐ณ๐ฆ๐ข๐ฅ๐ช๐ฏ๐จ & ๐ท๐ช๐ฅ๐ฆ๐ฐ ๐ฎ๐ข๐ต๐ฆ๐ณ๐ช๐ข๐ญ๐ด
A data engineering project with Kafka, Spark Streaming, dbt, Docker, Airflow, Terraform, GCP and much more!
Optimus is an easy-to-use, reliable, and performant workflow orchestrator for data transformation, data modeling, pipelines, and data quality management.
The User-Community Airflow Helm Chart is the standard way to deploy Apache Airflow on Kubernetes with Helm. Originally created in 2017, it has since helped thousands of companies create production-ready deployments of Airflow on Kubernetes.
Beginner data engineering project - batch edition
Simple but powerful DAG scheduler and dashboard
Code for Agile Data Science 2.0, O'Reilly 2017, Second Edition
TorchX is a universal job launcher for PyTorch applications. TorchX is designed to have fast iteration time for training/research and support for E2E production ML pipelines when you're ready.
AI agent tooling for data engineering workflows.
Astro SDK allows rapid and clean development of {Extract, Load, Transform} workflows using Python and SQL, powered by Apache Airflow.
Auto-generated Diagrams from Airflow DAGs. ๐ฎ ๐ช
Dataplane is an Airflow inspired unified data platform with additional data mesh and RPA capability to automate, schedule and design data pipelines and workflows. Dataplane is written in Golang with a React front end.
A Task-Based Workflow Orchestration Engine
Kubernetes custom controller and CRDs to managing Airflow
A Helm chart to install Apache Airflow on Kubernetes
Pipeline that extracts data from Crinacle's Headphone and InEarMonitor databases and finalizes data for a Metabase Dashboard. The dashboard is then used to support a purchasing decision of which Headphone / IEM to get.
A web frontend for scheduling Jupyter notebook reports
Construct a modern data stack and orchestration the workflows to create high quality data for analytics and ML applications.
๐ณ๐๐คCookiecutter template to launch an awesome dockerized Data Science toolstack (incl. Jupyster, Superset, Postgres, Minio, AirFlow & API Star)
A collection of Airflow operators, hooks, and utilities to elevate dbt to a first-class citizen of Airflow.
Airflow Deployment on AWS ECS Fargate Using Cloudformation
Resources for video demonstrations and blog posts related to DataOps on AWS
The most popular ClickHouse plugin for Airflow. ๐ Top-1% downloads on PyPI: https://pypi.org/project/airflow-clickhouse-plugin! Based on mymarilyn/clickhouse-driver.
Cloud-native, data onboarding architecture for Google Cloud Datasets
Reference framework for building data workflows provided by Google. Accelerates authentication, logging, scheduling, and deployment of solutions using GCP. To borrow a tagline.. "The framework for professionals with deadlines."
Great Expectations Airflow operator
ๆฌๅฐๅ AI ๅฐ่ฏด็ฒพไฟฎๅทฅไฝๅฐ๐ใๅ ็ฝฎ RAG ้ฟๆถ่ฎฐๅฟ๐ง ใไธๆจกๆๅไฝ๏ผไฝๅฎถ/ไธป็ผ/ๅฉๆ๏ผ๐๏ธใๅก็ๆตๅผ็ผ่พๅจโกใๆฏๆ้ ้ฆๅกๅฏผๅ ฅใๅ จไนฆ่ชๅจๅ็ฒพไฟฎไธๆฉๅ๐ญใAIๅฐ่ฏดๅ ๆ ๅฐ่ฏด็ฒพไฟฎ AIๆๆฌไฟฎๆน ๅฐ่ฏดๆฉๅ ๅฐ่ฏดๆป็ป
Data pipeline performing ETL to AWS Redshift using Spark, orchestrated with Apache Airflow
Run Airflow in AWS ECS(Elastic Container Service) using Fargate tasks
๐๐ป๐ All material from the PyCon.DE 2018 Talk "Beyond Jupyter Notebooks - Building your own data science platform with Python & Docker" (incl. Slides, Video, Udemy MOOC & other References)
This is a repository to demonstrate my details, skills, projects and to keep track of my progression in Data Analytics and Data Science topics.
Apache Liminals goal is to operationalise the machine learning process, allowing data scientists to quickly transition from a successful experiment to an automated pipeline of model training, validation, deployment and inference in production. Liminal provides a Domain Specific Language to build ML workflows on top of Apache Airflow.
A full data warehouse infrastructure with ETL pipelines running inside docker on Apache Airflow for data orchestration, AWS Redshift for cloud data warehouse and Metabase to serve the needs of data visualizations such as analytical dashboards.
HashiQube - The Ultimate Hands on DevOps Lab running All the HashiCorp Products in a Github Codespace or a Docker Container using Vagrant or Docker Compose
๐ A scalable, production-ready data pipeline for real-time streaming & batch processing, integrating Kafka, Spark, Airflow, AWS, Kubernetes, and MLflow. Supports end-to-end data ingestion, transformation, storage, monitoring, and AI/ML serving with CI/CD automation using Terraform & GitHub Actions.
Viewflow is an Airflow-based framework that allows data scientists to create data models without writing Airflow code.
A lightweight tool to get an AI Infrastructure Stack up in minutes not days. K3ai will take care of setup K8s for You, deploy the AI tool of your choice and even run your code on it.
Any Airflow project day 1, you can spin up a local desktop Kubernetes Airflow environment AND one in Google Cloud Composer with tested data pipelines(DAGs) :desktop_computer: >> [ :rocket:, :ship: ]
ETL (extract, transform and load) tools for ingesting Polygon blockchain data to Google BigQuery and Pub/Sub
๐ฆ Batch data pipeline with Airflow, DuckDB, Delta Lake, Trino, MinIO, and Metabase. Full observability and data quality.
Developed a data pipeline to automate data warehouse ETL by building custom airflow operators that handle the extraction, transformation, validation and loading of data from S3 -> Redshift -> S3
(project & tutorial) dag pipeline tests + ci/cd setup
Ansible role to install Apache Airflow
Full-stack Highly Scalable Cloud-native Machine Learning system for demand forecasting with realtime data streaming, inference, retraining loop, and more
A Python package that creates fine-grained dbt tasks on Apache Airflow
Magniv Core - A Python-decorator based job orchestration platform. Avoid responsibility handoffs by abstracting infra and DevOps.
Terraform module to deploy an Apache Airflow cluster on AWS, backed by RDS PostgreSQL for metadata, S3 for logs and SQS as message broker with CeleryExecutor
The goal of this project is to build a docker cluster that gives access to Hadoop, HDFS, Hive, PySpark, Sqoop, Airflow, Kafka, Flume, Postgres, Cassandra, Hue, Zeppelin, Kadmin, Kafka Control Center and pgAdmin. This cluster is solely intended for usage in a development environment. Do not use it to run any production workloads.