Crashed tasks doesnt free up resources when setting concurrency limit

Wingmore · January 12, 2023, 10:59pm

I have set a concurrency limit, however whenever a task/flow crashes, it seems to hog up that process and the resources arent freed. So any and all future tasks get the “prefect.engine - Received wait instruction for 30s: Concurrency limit for the ___ tag has been reached”.

This seems like a bug, and I’m wondering if there’s a fix in progress or a workaround? As in Is there any way to free up the concurrency limit workers? I thought maybe changing the state of the task would help but to to avail…

This might be related but it doesnt help my current case.

github.com/PrefectHQ/prefect

Clean up up zombie task runs caused by an infrastructure crash to free up concurrency slots

opened 12:02PM - 12 Jul 22 UTC

marvin-robot

from:slack status:accepted v2 priority:high api team:orchestration

## Opened from the [Prefect Public Slack Community](https://prefect.io/slack) *…*emil.ostergaard**: Hello, I have problems with prefect cloud 2.0. We use kubernetes flow runner, and a dask task runner. Friday (8/7-2022), I had a flow run which I wanted to abort. I attempted to use the `delete` functionality in the UI, thinking it would delete all resources related to the flow_run, including the kubernetes job etc. It did not remove the kubernetes job, so I removed this manually. The issue is concurrency-limits: The tasks launched by this flow has a tag, with a concurrency limit. It appears the task data associated with the deleted flow run was not removed from prefect storage. For instance, if I try: ``` prefect concurrency-limit inspect my-tag ``` It shows a bunch of active task ids, even though nothing is running in k8s. This causes an unfortunate issue where any new flow runs, for this flow, will never start tasks, because prefect thinks the concurrency-limit is hit, due to these zombie tasks. However, I can not seem to find a way to manually clean up these task ids, which means this flow is dead. Any help is appreciated! **anna**: Deleting a flow run will delete only the flow run, it will not terminate any external resources Due to a hybrid model, Prefect doesn't have direct access to your infra, which is why terminating resources this way is difficult Let me open an issue to investigating the best approach for such zombie tasks <@ULVA73B9P> open "Investigate the right approach for cleaning up zombie task runs caused by an infrastructure crash to free up concurrency limit slots" Original thread can be found [here](https://prefect-community.slack.com/archives/CL09KU1K7/p1657620079287959?thread_ts=1657620079.287959&cid=CL09KU1K7).

anna_geller · January 13, 2023, 7:11pm

I agree that it seems you didn’t do anything wrong and it’s a bug. Generally, bug reports should be reported as GitHub issues. I’d encourage you to add comments to the issue you linked or open a new GitHub issue and show a minimal reproducible example (MRE) + add more details. If you could e.g. open a new issue with MRE and link that to this issue, that would be great!