Our biggest Prefect 2.0 release yet: 2.0b8 including Blocks, Notifications, Flow Run Retries, Flow Packaging module, refinements to Deployments including serializers, packagers to automate building Docker images and more!

Announcements
, , , , , , , ,
1

Hey Prefectionists! :wave:

We’re psyched to share our biggest Prefect 2.0 release yet - 2.0b8! It’s full of exciting new features and refinements to existing concepts. As always, you can view release notes here on GitHub or in the description below.

Blocks :rock:

We’ve been working on Blocks behind the scenes for a while. Whether you know it or not, if you’ve used the past few releases, you’ve used them. Blocks enable you to securely store configuration with the Prefect Orion server and access it from your code later with just a simple reference.

image
image1129×289 46.6 KB

Think of Blocks as secure, UI-editable, type-checked configurations to external systems and an easy-to-use Key-Value Store - all configurable in one place and provided to you as part of the open-source Prefect 2.0.

We’re starting with just a few Blocks - mostly storage, but over time we’ll expand this pattern to include every tool and service in the growing modern data stack. You’ll be able to set up access to your entire stack once in just a few minutes, then manage access forever without editing your code.

Flow Run Retries :arrows_counterclockwise:

Flow run retries have been one of our most requested features, especially given how easy it is to run a flow as a “subflow” or “child flow” with Prefect 2.0. Flow run retries are configured just as task retries are - with

  • retries
  • and retry_delay_seconds parameters.

If both a task and its flow have retries configured, tasks within the flow will retry up to their specified task retry limit for each flow run.

For example, if you have a flow configured with a limit of 2 retries (up to 3 total runs, including the initial attempt), and a task in the flow configured with 3 retries (up to 4 attempts per flow run, including the initial attempt). The task could run-up to a total of 12 attempts, since task retry limits are reset after each flow run or flow run attempt.

Notifications :email:

You can always visit the Prefect UI to get a comprehensive view of the state of all of your flows, but when something goes wrong with one of them, you need that information immediately. Notifications can alert you and your team when any flow enters any state you specify, with or without specific tags.

image
image1123×290 46.4 KB

To create a notification, go to the new Notifications page via the sidebar navigation and select “Create Notification.” Notifications are structured just as you would describe them to someone. For example, if I want to get a Slack message every time my daily-ETL flow fails, my notification will simply read:

If a run of any flow with any tag enters a failed state, send a notification to Slack

Currently, notifications can only be sent to a Slack webhook (or email addresses if you are using Prefect Cloud 2.0).

Over time, notifications will support additional messaging services. Let us know which messaging services you’d like to send your notifications to!

Flow packaging and deployment :package:

We’ve revisited our flow packaging and deployment UX, making it both more powerful and easier to use. DeploymentSpecs are now just Deployments. Most of the fields are unchanged, but there are a few differences:

  • The flow_storage field has been replaced with a packager field.
  • The flow_location, flow_name, and flow parameters are now just flow.

You can deploy a flow directly, or you can package it with the new packaging module. Packaging enables you to bundle a flow with its dependencies, including Python packages. Packaging produces a Package, the artifact that contains your flow and its dependencies, as well as a Manifest, a JSON description of the package and what is inside of it.

There are several ways to create a package, with more to come. For instance, you can package your flow as a Docker image containing the flow and the runtime environment necessary to run it and register it with Docker. Alternatively, you could package your flow as a reference to the import path of a package stored directly in the Prefect Orion database. Learn more in the Deployment concept documentation.You can continue to use your existing DeploymentSpecs, but they are deprecated and will be removed in the coming weeks.

Breaking Changes :point_down:

This release removes the deprecated DaskTaskRunner and RayTaskRunner from the core library, breaking existing references to them. You can find them in their respective collections prefect-ray and prefect-dask.

It also removes the previously deprecated restart policy for the KubernetesFlowRunnner.

Most importantly, there are new breaking changes to the Deployments interface described below.

Special thanks to our contributors! :heart:

Happy Engineering!

3 Likes
2

Some cool changes! Especially notifications. Within the month we were planning to implement notifications ourselves as we get our Prefect pipelines ready for production, so that we could be notified of failures immediately indeed.
Our preference would be to have notification possibilities for Microsoft Teams (and also configurable email addresses if that is easier). Either through a direct integration with Microsoft Teams, or through the ability of calling some Azure Function that sends notifications to a Teams channel.
Because sadly, we do not use Slack

1 Like
3

If by chance you are looking at using Prefect Cloud, could you use a Teams Channel email address?

(Found in the channel options menu)
Screenshot 2022-07-11 at 15.27.39

1 Like
4

Sadly we are not using Prefect Cloud, and to be honest I did not know about the email feature. In that case, if Prefect’s email feature would also work when not using Prefect Cloud, that would seem to do the trick as well.

1 Like