New Prefect 2.0 release: 2.0b4 including flow and task runner improvements, performance optimizations, numerous fixes and new Prefect Collections!

Announcements
, , , , , , , , , , , , , , , ,
1

Hi Prefectionists! :wave:

We’ve been really busy over here at Prefect, getting lots of great feedback from early adopters. Thank you all for that! :heart:

There’s a lot of work going on behind the scenes as we work on building some exciting new features that will be exclusive to the General Availability of Prefect 2.0, but we want to keep the enhancements flowing to you. In that spirit, there are a lot of quality-of-life improvements here!

Community contributions :raised_hands:

While most of the development of Prefect 2.0 is still happening internally, we’re incredibly excited to be getting contributions in our open source repository. Big shoutout to our contributors for this last release:

  • @dannysepler
  • @ColeMurray
  • @albarrentine
  • @mkarbo
  • @AlessandroLollo

FlowRunner improvements :running_man:

  • Flow runners now pass all altered settings to their jobs instead of just the API key and URL
  • The KubernetesFlowRunner supports configuration of a service account name
  • The SubprocessFlowRunner streams output by default to match the other flow runners

TaskRunner improvements :man_running:

  • The DaskTaskRunner has improved the display of task keys in the Dask dashboard
  • The DaskTaskRunner now submits the execution graph to Dask allowing optimization by the Dask scheduler

New collections for Dask and Ray coming soon :sparkles:

Note that the Dask and Ray task runners will be moving out of the core Prefect library into dedicated prefect-ray and prefect-dask collections with the next release. Why?

  • This will reduce the number of code dependencies we require for most use cases.
  • Since we now have concurrent execution built into the core library (via ConcurrentTaskRunner), these packages do not need to be bundled with Prefect.
  • Lastly, we’re looking forward to building additional tasks and flows specific to Ray and Dask in their respective collections.

Prefect Collections :package:

We’ve received our first user-contributed collection. It includes tasks for Cube.js, check it out!

The following collections have also been recently released:

You can see a list of all available collections in the Prefect Collections Catalog.

Windows compatibility :window:

We’re excited to announce that we’ve begun work on Windows compatibility. Our full test suite isn’t passing yet, but we have core features working on Windows. We expect the majority of the edge cases to be addressed in an upcoming release.

Documentation improvements :open_book:

We’ve added some new documentation and made lots of improvements to existing documentation and tutorials:

  • Added documentation for associating conda environments with separate Prefect profiles
  • Added storage steps and advanced examples to the Deployments tutorial
  • Expanded documentation of storage options
  • Added workspace details to the Prefect Cloud :cloud: documentation
  • Improved schedules documentation with examples
  • Revised the Kubernetes tutorial to include work queue setup
  • Improved tutorial examples of task caching

CLI :technologist:

  • Deployments can be deleted from the CLI
  • The CLI displays help by default
  • prefect version is robust to server connection errors
  • prefect config view shows sources by default
  • prefect deployment create exits with a non-zero exit code if one of the deployments fails to be created
  • prefect config set allows setting values that contain equal signs
  • prefect config set validates setting types before saving them
  • prefect profile inpect displays settings in a profile instead of duplicating prefect config view behavior
  • prefect storage create trims long descriptions

Bug squashing :bug:

We’ve eradicated some bugs, replacing them with good behavior:

  • Flow runs are now robust to log worker failure
  • Deployment creation is now robust to ObjectAlreadyExists errors
  • Futures from async tasks in sync flows are now marked as synchronous
  • Tildes (~) in user-provided paths for PREFECT_HOME are expanded
  • Fixed parsing of deployments defined in YAML
  • Deployment deletion cleans up scheduled runs

Optimizations and reactors :gear:

You might not see these fixes in your day-to-day, but we’re dedicated to improving performance and maintaining our reputation as maintainers of an approachable and clean project.

  • The state_name is attached to run models for improved query performance
  • Lifespan management for the ephemeral Orion application is now robust to deadlocks
  • The hello route has moved out of the admin namespace so it is available on Prefect Cloud
  • Improved readability and performance of profile management code
  • Improved lower-bounds dependency parsing
  • Tests are better isolated and will not run against a remote API
  • Improved representation of Prefect Setting objects
  • Added extensive tests for prefect config and prefect profile commands
  • Moved testing utilities and fixtures to the core library for consumption by collections

Give it a try :rocket: 

pip install -U "prefect>=2.0b4"

Docs to get started: Prefect 2.0

Happy Engineering!

2 Likes
2

Looking forward to trying this out! Thank you to the team and contributors :clap:t5:

2 Likes
3

A blog post about the same:

4

:point_right: Note to anyone using DaskTaskRunner or RayTaskRunner:

from prefect version 2.0b8 onwards, those task runners were moved to the respective Prefect Collections for better code dependency management (the core library no longer requires dask or ray as dependencies - now, those can be installed sepataely when needed).

The correct imports are now:

from prefect_dask import DaskTaskRunner
from prefect_ray import RayTaskRunner