Airflow Valohai Plugin

Integration between Airflow and Valohai that allow Airflow tasks to launch executions in Valohai.

Installation

Install this package directly from pypi.

pip install airflow-valohai-plugin

You can read more about the use cases of Airflow plugins in the official docs.

Usage

Get a Valohai API Token.

Create a Valohai Connection in the Airflow UI with:

• Conn Id: valohai_default
• Conn Type: HTTP
• Host: app.valohai.com
• Password: REPLACEWITHAPI_TOKEN

from airflowvalohaiplugin.operators.valohaisubmitexecution_operator import ValohaiSubmitExecutionOperator
from airflowvalohaiplugin.operators.valohaidownloadexecutionoutputsoperator import ValohaiDownloadExecutionOutputsOperator

You can then create tasks and assign them to your DAGs.

ValohaiSubmitExecutionOperator

train_model = ValohaiSubmitExecutionOperator(
    taskid='trainmodel',
    project_name='tensorflow-example',
    step='Train model (MNIST)',
    dag=dag,
)

ValohaiDownloadExecutionOutputsOperator

from airflowvalohaiplugin.operators.valohaidownloadexecutionoutputsoperator import ValohaiDownloadExecutionOutputsOperator
download_model = ValohaiDownloadExecutionOutputsOperator(     taskid='downloadmodel',     outputtask=trainmodel,     output_name='model.pb',     dag=dag )

And create dependencies between your tasks.

trainmodel >> downloadmodel

Examples

You can find the complete code of example Airflow DAGs in the examples folder.

Contributing

Check out the contributing page.