Airflow tutorial

This is the code for Apache Airflow Tutorials playlist by Tuan Vu on Youtube

Contents

| Part | Title | Git Tag | |------|---------------------------|---------| | 1 | Introduction to Apache Airflow (blog post) | v0.1 | | 2 | Set up airflow environment with docker (blog post) | v0.2 | | 3 | Set up airflow environment using Google Cloud Composer (blog post) | N/A | | 4 | Writing your first pipeline (blog post) | N/A | | 5 | Airflow concept (blog post) | N/A | | 6 | Build a data pipeline using Google Cloud Bigquery (blog post) | v0.6 | | 7 | Airflow variables (blog post) | v0.7 |

Getting Started

These instructions will get you a copy of the project up and running on your local machine for development and testing purposes.

• Clone this repo
• Install the prerequisites
• Run the service
• Check http://localhost:8080
• Done! :tada:

Prerequisites

• Install Docker
• Install Docker Compose
• Following the Airflow release from Python Package Index

Usage

Run the web service with docker

docker-compose up -d
Build the image
docker-compose up -d --build

Check http://localhost:8080/

• docker-compose logs - Displays log output
• docker-compose ps - List containers
• docker-compose down - Stop containers

Other commands

If you want to run airflow sub-commands, you can do so like this:

• docker-compose run --rm webserver airflow list_dags - List dags
• docker-compose run --rm webserver airflow test [DAGID] [TASKID] [EXECUTION_DATE] - Test specific task

• docker-compose run --rm webserver python /usr/local/airflow/dags/[PYTHON-FILE].py - Test python script

Connect to database

If you want to use Ad hoc query, make sure you've configured connections: Go to Admin -> Connections and Edit "postgres_default" set this values:

• Host : postgres
• Schema : airflow
• Login : airflow
• Password : airflow

Credits

• Apache Airflow
• docker-airflow