Prefect 2.1.0 has just arrived! It includes Python-based Deployments, improvements to work queues, tons of new integrations and features!

Announcements
,
1

Hi everyone! :wave:

We are excited to announce the 2.1 release!

Build Deployments in Python :ship:

The prefect deployment build CLI provides a simple way of creating deployments and it has been designed for reliable engineering workflows that integrate well with CI/CD systems.

However, some users requested an additional feature to also allow deployment creation purely using Python. With this release, you can do the same from Python using the syntax:

from flows.healthcheck import healthcheck
from prefect.deployments import Deployment
from prefect.filesystems import S3


deployment = Deployment.build_from_flow(
    flow=healthcheck,
    name="pythonics3",
    description="hi!",
    version="1",
    work_queue_name="prod",
    tags=["myproject"],
    storage=S3.load("prod"),
    infra_overrides=dict(env={"PREFECT_LOGGING_LEVEL": "DEBUG"}),
    output="pythonics3.yaml",
)

if __name__ == "__main__":
    deployment.apply()

See the documentation to learn more.

Simplified Agents & Work Queues :rocket:

Agents and work queues give you control over where and how flow runs are executed. Now, creating an agent (and corresponding work queue) is even easier. Why?

  • Work queues now operate strictly by name, not by matching tags.
  • Deployments, and the flow runs they generate, are explicitly linked to a single work queue - this means that deployments map 1:1 to work queues rather than this relationship being many-to-many
  • the work queue is automatically created whenever a deployment references it - this means that you no longer have to worry about creating a work queue manually; assign a new work queue name to a deployment when you need it and just start a new agent polling from the same work queue name. Here is how this looks like in practice:
prefect deployment build myflow.py:myfunc -n mydeployment  -q prod -o my.yaml
prefect deployment apply my.yaml
prefect deployment run myfunc/mydeployment # triggers a run
prefect agent start -q prod

This “prod” work queue will pick up the run since it matches with the -q prod specified on this deployment.

Default work queue :raised_hands:

If you forget a work queue name on your deployment, Prefect will automatically assign a “default” work-queue name. This way, the runs generated by such deployments can be picked up by an agent you can start using:

prefect agent start -q default

Benefits of the work queue changes :blue_heart:

Most users will not have to interact directly with work queues at all – you only need to interact with deployments and agents. You can use work queues only once you need to configure concurrency limits or similar more advanced configurations.

Work queue changes are backward compatible! :gear:

These changes are fully backward compatible. See the documentation to learn more.

New Collections :package:

Improvements and bug fixes :computer:

  • Added three new exceptions to improve errors when parameters are incorrectly supplied to flow runs in #6091
  • Fixed a task dependency issue where unpacked values were not being correctly traced in #6348
  • Added the ability to embed BaseModel subclasses as fields within blocks, resolving an issue with the ImagePullPolicy field on the KubernetesJob block in #6389
  • Added comments support for deployment.yaml to enable inline help in #6339
  • Added support for specifying three schedule types - cron, interval and rrule - to the deployment build CLI in #6387
  • Added error handling for exceptions raised during the pre-transition hook fired by an OrchestrationRule during state transitions in #6315
  • Updated visit_collection to be a synchronous function in #6371
  • Revised loop service method names for clarity in #6131
  • Modified deployments to load flows in a worker thread in #6340
  • Resolved issues with capture of user-raised timeouts in #6357
  • Added base class and async compatibility to DockerRegistry in #6328
  • Added max_depth to visit_collection, allowing recursion to be limited in #6367
  • Added CLI commands for inspecting and deleting Blocks and Block Types in #6422
  • Added a Server Message Block (SMB) file system block in #6344 - Special thanks to @darrida for this contribution!
  • Removed explicit type validation from some API routes in #6448
  • Improved robustness of streaming output from subprocesses in #6445

Next steps

As always, to upgrade to the latest version:

pip install prefect -U

Cloud 2.0 is already up to date! :cloud:

Release notes on GitHub

Happy engineering!

3 Likes