How can I run my flow in a Docker container?

Prefect 2.0

Using the flow_runner argument in a DeploymentSpec, you can dynamically allocate infrastructure for your flow runs. Since the code must be retrieved on the created infrastructure, configuring flow runners is possible only for deployed flows. Here is an example deploying flow runs as Docker containers:

# docker_deployment.py
from prefect.flow_runners import DockerFlowRunner
from prefect.deployments import DeploymentSpec

DeploymentSpec(
    name="deployment_name",
    flow_location="s3://bucket_name/path/to/my_flow.py",
    flow_runner=DockerFlowRunner(image="prefecthq/prefect:2.0"),
)

You can leverage the CLI to create a deployment from that file:

prefect deployment create docker_deployment.py

To execute that deployment locally, you can use the following syntax:

prefect deployment run 'flow_name/deployment_name'

For more details, check out the DockerFlowRunner tutorial.

Prefect 1.0

The same can be accomplished in Prefect 1.0 using storage and run configuration attached to the Flow object.

from prefect.run_configs import DockerRun
from prefect.storage import S3
from prefect import Flow

with Flow(
    "flow_name",
    run_config=DockerRun(image="prefecthq/prefect:1.0"),
    storage=S3(bucket="bucket_name"),
) as flow:
    ...