With the General Availability release of Prefect 2.0, we modified the approach to creating deployments.
This approach is:
more intuitive - you can create your deployments from the UI and CLI,
compatible with CI/CD frameworks,
and more comprehensive, allowing you to use storage blocks not only for the flow code, but also for your custom modules and files, such as Python modules, configuration files, SQL scripts and dbt models! the documentation .
This Discourse topic includes plenty of practical deployment CLI commands :
Each of the following GitHub repositories contains a file that allows to easily create deployments for all projects:
Manually from any terminal - using deploy_flows.bash
From CI/CD - using the GitHub Actions workflows in the directory .github
We assume that an agent is spun up with a work queue name dev in all the examples below.:
prefect agent start -q dev
Platform
Storage Block
Infrastructure Block
End Result
CLI Build Command for hello.py flow with flow function hello
Local/VM
N/A
N/A
Local storage and local process on the same machine from which you created a deployment
prefect deployment build hello.py :hello -n implicit -q dev
Local/VM
N/A
N/A
Local storage and local process on the same machine from which you created a deployment — but with version and storing the output YAML manifest with the given file name in the deploy directory
p…
Yes, but only if the parameter schema for the flow remains unchanged!
This is why the manifest file exists - to detect changes that could break a deployment.
build step need to evaluate the flow object?
To extract parameter schema and import paths for the entrypoint.
apply step be executed from an environment that doesn’t have access to the flow code?
Yes, e.g., from a CI/CD tool or build server, as long as the YAML file has been already generated.
Storing the YAML file in your project repository with flows is entirely optional. If you want to, you can version-control those YAML files as a scaffold of deployment information for each flow.
One common use case we anticipate is using the YAML files as build artifacts of the CI/CD process, and relying purely on blocks and CLI as a way to continuously send deployment metadata to the Prefect backend.
Here are some repository templates and resources to help you build a CI/CD pipeline for your flow deployments.
Yes, that’s possible by using the prefect.deployments.Deployment class.
from flows.healthcheck import healthcheck
from prefect.deployments import Deployment
from prefect.filesystems import S3
deployment = Deployment.build_from_flow(
flow=healthcheck,
name="pythonic_s3",
description="hi!",
version="v1",
work_queue_name="dev",
tags=["myproject"],
storage=S3.load("dev"),
infra_overrides=dict(env={"PREFECT_LOGGING_LEVEL": "DEBUG"}),
output="your.yaml"
)
if __na…
The following diagram helps illustrate the relationships between different objects, including the underlying schema (at the time of writing).
[q]
There are several ways you can solve that problem:
Install the required file-system subpackage in your environment, e.g.:
pip install s3fs
pip install gcsfs
pip install adlfs
Include it as environment variable called EXTRA_PIP_PACKAGES - works only for DockerContainer block - docs
Build a custom image with those dependencies - you can use this GitHub Actions workflow template to build and push such custom DockerHub image in a single click:
You can configure and trigger the image bu…
It loads the deployment’s infrastructure block; if no block is associated with this deployment, the agent will use a preconfigured default configuration that can be configured on a per-agent basis
It executes the command python -m prefect.engine $FLOW_RUN_ID within this infrastructure
It then loads the deployment’s storage block, which has the ability to download the flow’s definition along with any supporting files into a temporary directory from which execution proceeds
TL;DR it’s the state of flow deployment on the API backend, not the YAML file, Python declaration, or CLI command - those are only used to create or modify a deployment, but the API representation is the single source of truth.
Deployment is an API-first concept. Therefore the API is a single source of truth rather than the YAML manifest or CLI command.
This means that anytime you modify any attribute of deployment from the UI, CLI, or via an API call (incl. changing default parameters or schedules, storage, and infrastructure blocks), any of those actions trigger a change in the corresponding deployment attribute.
Relying purely on YAML for that can only serve as a single source of truth if the YAML file is the only…
After we introduced the new way of deploying flows in the Prefect 2.0 General Availability release, many users requested a setup where they don’t need to have a separation between storage and infrastructure blocks and can bake all flow code and module dependencies into a single Docker image.
With this PR , we introduced:
--apply flag on build CLI and a corresponding apply kwarg on Deployment.build_from_flow allowing you to build and apply the deployment in one step
automatic detection of d…
Some users like to operate in a Terraform-style and have each CI/CD workflow deleting all existing flow deployments and creating new ones.
Here is how you can delete all deployments existing in your workspace (note, this is irreversible
There are several ways to do it.
[image]
Currently, there is no special task for that. You would need to
retrieve the deployment ID corresponding to the the flow you try to trigger, e.g. using prefect deployment ls
[image]
use Orion client as follows:
import asyncio
from prefect.client import get_client
async def main():
async with get_client() as client:
depl_id = "074db2e5-229a-460e-85ad-fca31b379fd2"
response = await client.create_flow_run_from_deployment(depl_id)
print(response)
if __name_…
In certain scenarios, you may want to continue processing your flow only once you’ve checked the status of previous processing.
Yes! Your run will still be executed.
The toggle is only used for scheduling. This means that even if you set the schedule as inactive, the deployment still exists and can be triggered ad-hoc from the UI, CLI, run_deployment Python function or via an API call.
[image]
The UI makes it easy to distinguish between scheduled runs and ad-hoc runs from deployment thanks to the auto-scheduled tag applied only to the scheduled runs:
[image]
2 Likes