In Prefect 2.0, deployments are extremely flexible and allow you to use any type of infrastructure - from a local process, a Docker container to a job running on a remote Kubernetes cluster.
Here is the syntax to build and create dockerized flow run deployments:
- Build step:
prefect deployment build path/to/flow_script.py:flow_name \ --name deployment_name --tag dev -sb storage_block_type/storage_block_name \ -ib infrastructure_block_type/infrastructure_block_name
In practice, this could look as follows:
prefect deployment build flows/hello.py:hello \ --name docker-custom --tag dev \ -sb s3/dev -ib docker-container/docker-custom-image
- Apply step:
prefect deployment apply deployment.yaml
To determine the type of infrastructure for build, you can use either:
The first one will generate a deployment manifest providing a placeholder you can fill with extra information. The latter requires that you create the infrastructure block beforehand, either from code or from the UI.
DockerContainer infrastructure blocks:
Looking at this project template, here is how you can create a deployment for a simple flow that doesn’t require any custom pip packages.
As long as you have some custom pip packages, Prefect base image won’t be enough - you need to package your dependencies into a custom image and push it to some registry. The code below shows how you can do all that programmatically by building a custom image and pushing it to Dockerhub.
Once the image is built, you can create a custom
DockerContainer block and pass it to your deployment build:
flow_runner argument in a
DeploymentSpec, you can dynamically allocate infrastructure for your flow runs. Since the code must be retrieved on the created infrastructure, configuring flow runners is possible only for deployed flows. Here is an example deploying flow runs as Docker containers:
# docker_deployment.py from prefect.flow_runners import DockerFlowRunner from prefect.deployments import DeploymentSpec DeploymentSpec( name="deployment_name", flow_location="s3://bucket_name/path/to/my_flow.py", flow_runner=DockerFlowRunner(image="prefecthq/prefect:2.0"), )
You can leverage the CLI to create a deployment from that file:
prefect deployment create docker_deployment.py
To execute that deployment locally, you can use the following syntax:
prefect deployment run 'flow_name/deployment_name'
For more details, check out the DockerFlowRunner tutorial.
from prefect.run_configs import DockerRun from prefect.storage import S3 from prefect import Flow with Flow( "flow_name", run_config=DockerRun(image="prefecthq/prefect:1.0"), storage=S3(bucket="bucket_name"), ) as flow: ...