The second launch week initiative - announcing Prefect 2.0!

Hi Prefectionists! :wave:

We’re excited to introduce the second launch week initiative which is Prefect 2.0 powered by Orion - our second-generation orchestration engine! :rocket:

This release solidifies many features we’ve been working on during Orion’s technical preview and marks an important milestone in making Prefect 2.0 production-ready. For more details on that milestone, check out this blog post by our CEO @jeremiah:

We will continue to develop Prefect 2.0 on the orion branch of the prefect repo. Both the Orion engine and Prefect 2.0 as a whole will remain under active development in beta for the next several months, with a number of major features yet to come. But for now, let’s look at the features introduced in this release! :point_down:

1. Expanded UI :chart_with_upwards_trend:

Through our technical preview phase, our focus has been on establishing the right concepts and making them accessible through the CLI and API. Now that some of those concepts have matured, we’ve made them more accessible through the UI.

Flows and deployments

If you’ve ever created a deployment without a schedule, you know it can be difficult to find that deployment in the UI. This release gives flows and their deployments a dedicated home on the growing left sidebar navigation. The dashboard continues to be the primary interface for exploring flow runs and their task runs.

Work queues

With the 2.0a13 release, we introduced work queues, which could only be created through the CLI. Now, you can create and edit work queues directly from the UI, then copy-paste, and run a command that starts an agent that pulls work from that queue.

2. Collections :books:

Prefect Collections are groupings of pre-built tasks and flows used to quickly build data flows with Prefect. You may think of them as equivalent to the task library in Prefect 1.0 (even though they include much more than just tasks!).

Collections are grouped around the services with which they interact. For example, to download data from an S3 bucket, you could use the s3_download task from the prefect-aws collection, or if you want to send a Slack message as part of your flow you could use the send_message task from the prefect-slack collection.

By using Prefect Collections, you can reduce the amount of boilerplate code that you need to write for interacting with common services, and focus on the outcome you’re seeking to achieve.

Learn more about Collections in the docs with more info to come later this week!

3. Profile switching :gear:

We’ve added the prefect profile use <name> command to allow you to easily switch your active profile.

The format for the profiles file has changed to support this. Existing profiles will not work unless their keys are updated.

For example, the profile “foo” must be changed to “profiles.foo” in the file profiles.toml:

[foo]
SETTING = "VALUE"

to

[profiles.foo]
SETTING = "VALUE"

4. Other enhancements :rocket:

  • It’s now much easier to explore Prefect 2.0’s major entities, including flows, deployments or flow runs through the CLI with the ls command, which produces consistent, beautifully formatted tables for each entity.
  • Improved error handling for issues that the client commonly encounters, such as network errors, slow API requests, etc.
  • The UI has been polished throughout to be sleeker, faster, and even more intuitive.
  • We’ve made it even easier to access file storage through fsspec, which includes many useful built-in implementations.

We’ve also included a brand new prefect version command which is going to be incredibly valuable for troubleshooting! If you face any issues with Orion and want to ask a question about it here via Discourse, via a GitHub issue or via our Community Slack, please include the output of this command for easier troubleshooting :pray:

image

Give it a try and give us feedback!

Install the Prefect 2.0 Beta release with the command below, and try the new features with your flows!

pip install -U "prefect>=2.0b1"

Thank you for all the valuable feedback you provided us through the technical preview! :heart:

Please continue to provide feedback through the beta phase as well!

Happy engineering! :technologist:

1 Like