We are extremely excited to announce our biggest Orion release so far - Orion 2.0a13 introducing the concept of work queues, profiles, remote storage, and Ray task runner!
Work queues
Work queues determine the work that needs to be done, i.e. your scheduled flow run deployments.
You may wonder - shouldn’t agents do it? That’s indeed how the problem was approached in Prefect 1.0, but Orion makes this process even more flexible, configurable, observable, and intuitive!
The role of agents in Orion
Agents poll a specific work queue for new work. Previously, agents would poll for any scheduled flow run. Now, scheduled flow runs are added to work queues that can filter flow runs by:
- tags,
- specific deployments,
- and flow runner type.
New features made possible thanks to work queues
Work queues enable some exciting new features, including:
- Filtering: Each work queue can target a specific subset of work. This filtering can be adjusted without restarting your agent.
- Concurrency limits: Each work queue can limit the number of flows that run at the same time.
- Pausing specific work: Each work queue can be paused independently. This prevents agents from submitting additional work.
Check out the work queue documentation for more details.
Note that: prefect agent start
now requires you to pass a work queue identifier and prefect orion start
no longer starts an agent by default.
Profiles
Prefect now supports profiles for configuration. You can store settings in profiles and switch between them. For example, this allows you to quickly switch between using a local and hosted API.
View all of the available commands with prefect config --help
and check out our settings documentation for a full description of how to use profiles.
We’ve also rehauled our settings reference to make it easier to see all the available settings. You can override any setting with an environment variable or prefect config set
.
Remote storage
Prior to this release, the Orion server would store your flow code and results in its local file system. Now, we’ve introduced storage with external providers including AWS S3, Google Cloud Storage, and Azure Blob Storage.
There’s an interactive command, prefect storage create
, which walks you through the options required to configure storage. Your settings are encrypted and stored in the Orion database.
Note that: you will no longer be able to use the Kubernetes or Docker flow runners without configuring storage. While automatically storing flow code in the API was convenient for early development, we’re focused on enabling the hybrid model as a core feature of Orion.
Running tasks on Ray
We’re excited to announce a new task runner with support for Ray. You can run your tasks on an existing Ray cluster, or dynamically create one with each flow run.
Ray has powerful support for customizing runtime environments, parallelizing tasks to make use of your full compute power, and dynamically creating distributed task infrastructure.
An overview of using Ray can be found in our documentation.
Try it yourself and give us feedback
Download the latest version, and give those new features a try.
pip install -U "prefect>=2.0a"
We appreciate your feedback.
Have a fantastic weekend and as always, happy engineering!