Enable Task-run concurrency limit based on task_keys + enable 0/negative concurrency limit for complete pausing of task runs

Help
,
1

Hi,

We are trying to find a good way to set a concurrency limit for tasks identified by their unique task_keys. We know that Prefect has this nice concurrency limit concept, but it would be weird if we add the task_key to all tasks as a tag, as it will make the UI really crowded, and it introduces unnecessary repetition of task identifiers.

Is it possible to enable a concurrency limit that accepts a task key? Also, since we want to not only have a positive concurrency limit to avoid race conditions, but we also want to completely pause the execution of some task runs, is it possible to allow a concurrency limit = 0 to not fail task runs? Or is it possible to make that behavior configurable for a given concurrency limit rule?

Thanks!

2

Hi @PPPSDavid here is a github issue for adding key:value properties in tags.

Automations can be used to pause a workqueue under certain conditions. Also you can pause a flow by an API call.

3

Thanks @taylor_curran ! The progress on having a structured tagging system is well noted and appreciated :slight_smile: However, it still feels a bit repetitive we will have to add the task key for all the tasks as a tag to achieve the task-key concurrency limit. It could be some minor things that we just need to get use to (or come up with a properly named tag for the tasks we want to set concurrency on )

On the second point, your thought on pausing the flow and work queue is well noted. We are considering other toolings that use these APIs to achieve pausing of flow runs. The case above is actually more granular: suppose we have some 10 tasks hitting API “A”, and they typically have a concurrency limit of 2. These tasks could be (conditionally) used in 20 different flows. Now suppose we noticed an issue with API “A” and want to pause any calls to it for 10 minutes until we deploy a quick fix, having the option of changing that existing concurrent limit from 2 to zero, or “off” will be much more straightforward and fast than figuring our which flow runs to pause and pause those.

Please let me know if this case story makes sense, and if you think this is a reasonable usage!