Prefect 2.0b12 removes flow runners in favor of more flexible Infrastructure blocks

Archive
, , ,
1

1. Introduction

This page explains how you can define runtime-infrastructure for your flows as part of your Deployment.

The syntax shown in section #2 represents the latest state of Deployments in the General Availability release of Prefect 2.0. In contrast, section #3 illustrates the concept of flow runners that was used in the Beta stage (all releases with the letter “b” up to and including 2.0b11).

We show the syntax of both:
:green_circle: (new) infrastructure blocks, and
:red_circle: (old) flow runners to help you recreate your flow deployments.

Important note

:warning: Recreating deployments is required to make your deployments compatible with the latest state of the Deployments API used in the stable release of Prefect 2.0.

2. Infrastructure blocks

Here is how you can set the infrastructure on your Deployments in 2.0b12 or later:

2.1 Process

Functionally equivalent to the SubprocessFlowRunner:

from prefect.infrastructure import Process
from prefect.deployments import Deployment
from prefect import flow


@flow
def hello_world():
    print("Hello world!")


Deployment(
    name="example",
    flow=hello_world,
    infrastructure=Process(),
)

2.2 DockerContainer

from prefect.infrastructure import DockerContainer
from prefect.deployments import Deployment
from prefect import flow


@flow
def hello_world():
    print("Hello world!")


Deployment(
    name="example_docker",
    flow=hello_world,
    infrastructure=DockerContainer(),
)

2.3 KubernetesJob

from prefect.infrastructure import KubernetesJob
from prefect.deployments import Deployment
from prefect import flow


@flow
def hello_world():
    print("Hello world!")


Deployment(
    name="example_kubernetes",
    flow=hello_world,
    infrastructure=KubernetesJob(),
)

2.4 More advanced examples

For more examples showing various Deployment and Infrastructure configuration, see the following Discourse topic:

3. Old concept: Flow Runners (do NOT use it - deprecated! :warning: )

To deploy a flow to some remote infrastructure, in Prefect 2.0b11 and lower, you would need to configure a flow runner as an argument to your Deployment. There were three flow runners that you could use:

  1. SubprocessFlowRunner
  2. DockerFlowRunner
  3. KubernetesFlowRunner

All of them were located in the prefect.flow_runners module:

from prefect.flow_runners import (
    SubprocessFlowRunner,
    DockerFlowRunner,
    KubernetesFlowRunner,
)

Related Docs:
https://orion-docs.prefect.io/concepts/flow-runners/

The below sections show the old syntax that matches the same concepts in the beta versions.

1. SubprocessFlowRunner

from prefect.flow_runners import SubprocessFlowRunner
from prefect.deployments import Deployment
from prefect import flow

@flow
def hello_world():
    print("Hello world!")

Deployment(
    name="example",
    flow=hello_world,
    flow_runner=SubprocessFlowRunner(condaenv="testconda"),
)

2. DockerFlowRunner

from prefect.flow_runners import DockerFlowRunner
from prefect.deployments import Deployment
from prefect import flow

@flow
def hello_world():
    print("Hello world!")

Deployment(
    name="example_docker",
    flow=hello_world,
    flow_runner=DockerFlowRunner(),
)

3. KubernetesFlowRunner

from prefect.flow_runners import KubernetesFlowRunner
from prefect.deployments import Deployment
from prefect import flow

@flow
def hello_world():
    print("Hello world!")

Deployment(
    name="example_docker",
    flow=hello_world,
    flow_runner=KubernetesFlowRunner(),
)
1 Like