Concurrency limits in Prefect 2.0 on a task-level vs. on a flow-level

View in #prefect-community on Slack

Jared_Robbins @Jared_Robbins: Also, is it possible to limit concurrency at the deployment level? I am implementing a job that I don’t want running twice at the same time. It seems tags in a DeploymentSpec don’t work for concurrency limits. Right now I might move the tag to the main flow

Kevin_Kho @Kevin_Kho: Tags should work on the deployment. I’ll try this in a bit

Anna_Geller @Anna_Geller: @Jared_Robbins there are two levels of concurrency limits.

#1 Currently, using the prefect concurrency-limit command, you can set task-level concurrency limits.

2# The alternative to task-level concurrency limits are work-queue concurrency limits which allow you to set limits on a deployment level - this seems to be exactly what you want! To create that, you can create a work-queue for a specific deployment UUID:

prefect work-queue create -d 'uuid' test_queue

this will return WORK_QUEUE_ID which you can use to set a limit for that to 1:

prefect work-queue set-concurrency-limit WORK_QUEUE_ID 1

Jared_Robbins @Jared_Robbins: I suppose that will work… will need a work queue and agent for every job though
@Kevin_Kho Was that possible? I think it would be a really useful feature. I’m sure I’m not the only one who doesn’t want the same job to run twice

Kevin_Kho @Kevin_Kho: That I know for sure will be a built-in feature where the current run will just wait for the previous run to finish. It just hasn;’t been developed yet

Jared_Robbins @Jared_Robbins: Dang. I’ll keep an eye on the changelogs. will put this on the backburner for now