With the General Availability release of Prefect 2.0, we modified the approach to creating deployments.
This approach is:
more intuitive - you can create your deployments from the UI and CLI,
compatible with CI/CD frameworks,
and more comprehensive, allowing you to use storage blocks not only for the flow code, but also for your custom modules and files, such as Python modules, configuration files, SQL scripts and dbt models! the documentation .
This Discourse topic includes plenty of practical deployment CLI commands :https://discourse.prefect.io/t/how-to-deploy-all-flows-from-a-project-at-once/1566
We assume that an agent is spun up with a work queue name dev in all the examples below.:
prefect agent start -q dev
Platform
Storage Block
Infrastructure Block
End Result
CLI Build Command for hello.py flow with flow function hello
Local/VM
N/A
N/A
Local storage and local process on the same machine from which you created a deployment
prefect deployment build hello.py :hello -n implicit -q dev
Local/VM
N/A
N/A
Local storage and local process on the same machine from which you created a deployment — but with version and storing the output YAML manifest with the given file name in the deploy directory
p…
Yes, but only if the parameter schema for the flow remains unchanged!
This is why the manifest file exists - to detect changes that could break a deployment.
build step need to evaluate the flow object?To extract parameter schema and import paths for the entrypoint.
apply step be executed from an environment that doesn’t have access to the flow code?Yes, e.g., from a CI/CD tool or build server, as long as the YAML file has been already generated.
Storing the YAML file in your project repository with flows is entirely optional. If you want to, you can version-control those YAML files as a scaffold of deployment information for each flow.
One common use case we anticipate is using the YAML files as build artifacts of the CI/CD process, and relying purely on blocks and CLI as a way to continuously send deployment metadata to the Prefect backend.
Here are some repository templates and resources to help you build a CI/CD pipeline for your flow deployments.
Yes, that’s possible by using the prefect.deployments.Deployment class.
from flows.healthcheck import healthcheck
from prefect.deployments import Deployment
from prefect.filesystems import S3
deployment = Deployment.build_from_flow(
flow=healthcheck,
name="pythonic_s3",
description="hi!",
version="v1",
work_queue_name="dev",
tags=["myproject"],
storage=S3.load("dev"),
infra_overrides=dict(env={"PREFECT_LOGGING_LEVEL": "DEBUG"}),
output="your.yaml"
)
if __na…
The following diagram helps illustrate the relationships between different objects, including the underlying schema (at the time of writing).
[q]
There are several ways you can solve that problem:
Install the required file-system subpackage in your environment, e.g.:
pip install s3fs
pip install gcsfs
pip install adlfs
Include it as environment variable called EXTRA_PIP_PACKAGES - works only for DockerContainer block - docs
Build a custom image with those dependencies - you can use this GitHub Actions workflow template to build and push such custom DockerHub image in a single click:
https://github.com/anna-geller/prefect-deplo…
https://discourse.prefect.io/t/what-happens-when-agent-picks-up-a-deployed-flow-run-from-a-work-queue/1464
TL;DR it’s the state of flow deployment on the API backend, not the YAML file, Python declaration, or CLI command - those are only used to create or modify a deployment, but the API representation is the single source of truth.
https://discourse.prefect.io/t/i-created-a-deployment-then-i-added-a-schedule-from-the-ui-when-i-then-reran-the-build-step-the-yaml-file-had-the-same-schedule-as-i-defined-from-the-ui-even-though-i-didn-t-specify-that-schedule-manually-from-cli-or-in-yaml-why-is-that-happening/1460
After we introduced the new way of deploying flows in the Prefect 2.0 General Availability release, many users requested a setup where they don’t need to have a separation between storage and infrastructure blocks and can bake all flow code and module dependencies into a single Docker image.
With this PR , we introduced:
--apply flag on build CLI and a corresponding apply kwarg on Deployment.build_from_flow allowing you to build and apply the deployment in one step
automatic detection of d…
https://discourse.prefect.io/t/how-to-use-prefect-client-to-delete-all-deployments/1335
https://discourse.prefect.io/t/how-can-i-toggle-the-schedule-off-for-my-flow-deployment-pause-a-schedule/1474
Currently, there is no special task for that. You would need to
retrieve the deployment ID corresponding to the the flow you try to trigger, e.g. using prefect deployment ls
[image]
use Orion client as follows:
import asyncio
from prefect.client import get_client
async def main():
async with get_client() as client:
depl_id = "074db2e5-229a-460e-85ad-fca31b379fd2"
response = await client.create_flow_run_from_deployment(depl_id)
print(response)
if __name_…
https://discourse.prefect.io/t/implementing-waiting-for-approval-in-prefect-2/1558
https://discourse.prefect.io/t/if-i-toggle-my-deployment-off-set-schedule-inactive-and-then-trigger-an-ad-hoc-flow-run-will-this-ad-hoc-run-be-executed/1769