Prefect 2.8.0 is here with work pools, webhooks, and more!

Prioritize flow runs with work pools :swimming_man:

Work pools allow you to organize and prioritize work

With this release, flow runs can now be prioritized among work queues via work pools! Work pools allow you to organize and prioritize work by grouping related work queues together. Within work pools, you can assign a priority to each queue, and flow runs scheduled on higher priority work queues will be run before flow runs scheduled on lower priority work queues. This allows agents to prioritize work that is more important or time-sensitive even if there is a large backlog of flow runs on other work queues in a given work pool.

All existing work queues will be assigned to a default work pool named default-agent-pool. Creating a new work pool can be done via the Work Pools page in the UI or via the CLI.

To create a new work pool named “my-pool” via the CLI:

prefect work-pool create "my-pool"

Each work pool starts out with a default queue. New queues can be added to a work pool via the UI or the CLI.

To create a new work queue in a work pool via the CLI:

prefect work-queue create "high-priority" --pool "my-pool"

Deployments can now be assigned to a work queue in a specific work pool. Use the --pool flag to specify the work pool and the --queue flag to specify the work queue when building a deployment.

prefect deployment build \ 
--pool my-pool \
--queue high-priority \
--name high-priority \

Once a deployment has been created and is scheduling flow runs on a work queue, you can start an agent to pick up those flow runs by starting an agent with the --pool flag.

prefect agent start --pool my-pool

Starting an agent with the --pool command allows the agent to pick up flow runs for the entire pool even as new queues are added to the pool. If you want to start an agent that only picks up flow runs for a specific queue, you can use the --queue flag.

prefect agent start --pool my-pool --queue high-priority

A few other important parts of this release:

  • The ability to create a flow run from the UI with parameters from a previous run
  • A generic Webhook block
  • Override customizations functionality for CLI deployments v
  • You can now reset concurrency limits via the CLI to purge existing runs from taking concurrency slots

To learn more about work pools, and everything else we’ve added or updated, check out the docs or see the relevant pull requests in the release notes.


Hey, thanks for the amazing work you guys do in developing this tool. I am learning so much from this tool being provided to me for free and I am grateful for it. The following comment is just a concern I want to raise.

I feel that this change is quite sudden, it is not only a change to the UI but a significant change to the deployment process. If I were to adopt the Prefect Cloud in a production setting, this would worry me as I may be tied up with other tasks and this sudden change would cause some of my pipelines to break. I understand that as Prefect is still in its early stage of development we want to introduce new releases quickly but this would be a significant red flag for adoption in the industry.

I might be mistaken as I am still in the process of learning Prefect, please feel free correct me if I am wrong.

Suggestion: It might be useful to create a feature where significant changes of this level has to be opt-ed in through the UI, giving users more time to react to these changes.

1 Like

I’m so glad to hear that you’re enjoying using Prefect, @Good_Role_Model!

I’m sorry that the UI changes felt a bit jarring. Prefect Cloud raised an in app notification to let users know that their work queues had moved, but we did not have the capability to raise such a notification in open source Prefect. We’re investigating how we might be able to do that.

Regarding the change to the deployment process, we should have more strongly emphasized that this change is fully backwards compatible. Specifying a work queue behaves just as before - the work queue will just be part of the default-agent-pool. We will never make breaking changes to any part of Prefect, especially the deployment process, without ample notice.

1 Like