New Orion release 2.0a13 featuring work queues, agents, profiles, remote object storage and distributed parallel processing with Ray!

We are extremely excited to announce our biggest Orion release so far - Orion 2.0a13 introducing the concept of work queues, profiles, remote storage, and Ray task runner! :rocket:

Work queues

Work queues determine the work that needs to be done, i.e. your scheduled flow run deployments.

You may wonder - shouldn’t agents do it? That’s indeed how the problem was approached in Prefect 1.0, but Orion makes this process even more flexible, configurable, observable, and intuitive! :tada:

The role of agents in Orion

Agents poll a specific work queue for new work. Previously, agents would poll for any scheduled flow run. Now, scheduled flow runs are added to work queues that can filter flow runs by:

  • tags,
  • specific deployments,
  • and flow runner type.

New features made possible thanks to work queues

Work queues enable some exciting new features, including:

  • Filtering: Each work queue can target a specific subset of work. This filtering can be adjusted without restarting your agent.
  • Concurrency limits: Each work queue can limit the number of flows that run at the same time.
  • Pausing specific work: Each work queue can be paused independently. This prevents agents from submitting additional work.

Check out the work queue documentation for more details.

:point_right: Note that: prefect agent start now requires you to pass a work queue identifier and prefect orion start no longer starts an agent by default.

Profiles

Prefect now supports profiles for configuration. You can store settings in profiles and switch between them. For example, this allows you to quickly switch between using a local and hosted API.

View all of the available commands with prefect config --help and check out our settings documentation for a full description of how to use profiles.

We’ve also rehauled our settings reference to make it easier to see all the available settings. You can override any setting with an environment variable or prefect config set .

Remote storage

Prior to this release, the Orion server would store your flow code and results in its local file system. Now, we’ve introduced storage with external providers including AWS S3, Google Cloud Storage, and Azure Blob Storage. :bucket:

There’s an interactive command, prefect storage create , which walks you through the options required to configure storage. Your settings are encrypted and stored in the Orion database.

:point_right: Note that: you will no longer be able to use the Kubernetes or Docker flow runners without configuring storage. While automatically storing flow code in the API was convenient for early development, we’re focused on enabling the hybrid model as a core feature of Orion.

Running tasks on Ray

We’re excited to announce a new task runner with support for Ray. You can run your tasks on an existing Ray cluster, or dynamically create one with each flow run.

Ray has powerful support for customizing runtime environments, parallelizing tasks to make use of your full compute power, and dynamically creating distributed task infrastructure.

An overview of using Ray can be found in our documentation.

Try it yourself and give us feedback

Download the latest version, and give those new features a try.

pip install -U "prefect>=2.0a"

We appreciate your feedback.

Have a fantastic weekend and as always, happy engineering!

3 Likes

Super excited to try out this release. One question though: what about Docker storage? Is this something that needs to be added in a later release along with git storage?

2 Likes

@Matthias I’m pretty sure Docker and GitHub storage options will be added in the future, but to be 100% sure about the details here I asked the team.

If you don’t see any follow-up from me on this, it means it’s on the roadmap :slight_smile:

1 Like

Congratulations on the Rocketship Award! I’m working on a project to transfer our ETL pipeline from a collection of Jupyter Notebooks executed manually and with Papermill over to Prefect. The Orion documentation states that there will be a stable release in early 2022 and then it will become the default version. Considering that we’re starting the transition to Prefect now and it will take us a couple of months before we are using it in earnest, do you recommend working with Orion instead of Prefect 1.0? Thanks!

2 Likes

@rupert That’s definitely a valid option and you could indeed start directly with Orion in that case!

But note that even if you start transforming your Jupyter notebooks to Prefect flows using Prefect 1.0, it should be quite seamless to then move it over to Orion later since the basic Prefect Core syntax is quite similar. For instance, check out this thread:

Regardless of which option you choose, let us know if you have any questions along the way.

1 Like