New Orion release 2.0a10 including concurrency limits, ConcurrentTaskRunner, CLI refinements and improved database migrations

There’s a lot in it to be excited about in the latest Orion release, 2.0a10, but we’re particularly excited to introduce the new concurrent task runner and task run concurrency limits. Together, these features enable you to take advantage of Orion’s native asyncio (and anyio ) support, harnessing the power of concurrency without complex software patterns such as thread pools.

ConcurrentTaskRunner

The concurrent task runner enables tasks that are blocked on input/output to yield to other tasks, whether your code is synchronous or asynchronous.

Concurrency limits

When running a flow using a task runner that enables concurrent execution, or running many flows across multiple execution environments, you may want to limit the number of certain tasks that can run at the same time. For example, perhaps you want to ensure that, across all of your flows, there are no more than three open connections to your production database at once. You can do so by creating a “prod-db” tag and applying it to all of the tasks that open a connection to that database. Then, you can create a concurrency limit with:

prefect concurrency-limit create prod-db 3

Orion will now ensure that no more than 3 task runs with the “prod-db” tag will run at the same time.

Great news for open-source users

Previously, this feature was only available in a paid tier of Prefect Cloud, our hosted commercial offering. We’re very happy to move it to the open-source domain, furthering our goal of making Orion the most capable workflow orchestration tool ever :muscle:

Other improvements

There is so much more in this release, including greater flow parameter flexibility, database migrations, CLI refinements, and much more. Read all about it in the release notes!

:rotating_light: Breaking change: If you have previously installed any version of Orion, you’ll need to delete the existing database with rm ~/.prefect/orion.db or stamp it with prefect orion database stamp to successfully upgrade.

1 Like