There’s a lot in it to be excited about in the latest Orion release, 2.0a10, but we’re particularly excited to introduce the new concurrent task runner and task run concurrency limits. Together, these features enable you to take advantage of Orion’s native
anyio ) support, harnessing the power of concurrency without complex software patterns such as thread pools.
The concurrent task runner enables tasks that are blocked on input/output to yield to other tasks, whether your code is synchronous or asynchronous.
When running a flow using a task runner that enables concurrent execution, or running many flows across multiple execution environments, you may want to limit the number of certain tasks that can run at the same time. For example, perhaps you want to ensure that, across all of your flows, there are no more than three open connections to your production database at once. You can do so by creating a “prod-db” tag and applying it to all of the tasks that open a connection to that database. Then, you can create a concurrency limit with:
prefect concurrency-limit create prod-db 3
Orion will now ensure that no more than 3 task runs with the “prod-db” tag will run at the same time.
Previously, this feature was only available in a paid tier of Prefect Cloud, our hosted commercial offering. We’re very happy to move it to the open-source domain, furthering our goal of making Orion the most capable workflow orchestration tool ever
Breaking change: If you have previously installed any version of Orion, you’ll need to delete the existing database with
rm ~/.prefect/orion.db or stamp it with
prefect orion database stamp to successfully upgrade.