prefect-saturn

prefect-saturn is a Python package that makes it easy to run Prefect Cloud flows on a Dask cluster with Saturn Cloud. For a detailed tutorial, see "Fault-Tolerant Data Pipelines with Prefect Cloud ".

Installation

prefect-saturn is available on PyPi.

pip install prefect-saturn

prefect-saturn can be installed directly from GitHub

pip install git+https://github.com/saturncloud/prefect-saturn.git@main

Getting Started

prefect-saturn is intended for use inside a Saturn Cloud environment, such as a Jupyter notebook.

import prefect
from prefect import Flow, task
from prefect_saturn import PrefectCloudIntegration
@task def hello_task():     logger = prefect.context.get("logger")     logger.info("hello prefect-saturn")
flow = Flow("sample-flow", tasks=[hello_task])
project_name = "sample-project" integration = PrefectCloudIntegration(     prefectcloudprojectname=projectname ) flow = integration.registerflowwith_saturn(flow)
flow.register(     projectname=projectname,     labels=["saturn-cloud"] )

Customize Dask

You can customize the size and behavior of the Dask cluster used to run prefect flows. prefectsaturn.PrefectCloudIntegration.registerflowwithsaturn() accepts to arguments to accomplish this:

daskclusterkwargs: keyword arguments to pass to the constructor dask_saturn.SaturnCluster.
daskadaptkwargs: keyword arguments used to configure "Adaptive Scaling"

For example, the code below tells Saturn that this flow should run on a Dask cluster with 3 xlarge workers, and that prefect should shut down the cluster once the flow run has finished.

flow = integration.registerflowwith_saturn(
    flow=flow,
    daskclusterkwargs={
        "n_workers": 3,
        "worker_size": "xlarge",
        "autoclose": True
    }
)
flow.register(     projectname=projectname,     labels=["saturn-cloud"] )

Contributing

See CONTRIBUTING.md for documentation on how to test and contribute to prefect-saturn.

🔗 More in this category