Hi all, this is my first post on this forum, so please let me know if I am missing anything. Thank you!
For context:
My Prefect flow starts by crawling through a REST API using requests.get. Everything runs smoothly until the repackaging tasks get queued up, this is where all of the repackaging tasks get stuck in a state of Pending(). This is with the DaskTaskRunner. I have also tried it with the RayTaskRunner, same issue.
I am not sure how to get these tasks out of the Pending() state and I feel like I have exhausted many options for this, increase memory limit, increase amount of workers, change task runner type, decrease batch size, no upstream/downstream dependencies, use the wait() function(which doesn’t do anything to change the state, program just gets stuck at this point as it waiting for the task to finish, but it never starts running). I also persist results for all the tasks and have retries for them. Overall, this flow has 18k-20k tasks when it runs.
According to Prefect docs, pending means:
The run has been submitted to run, but is waiting on necessary preconditions to be satisfied.
What are the necessary preconditions that need to be satisfied, and are there any ways to convert this Pending() state into a Running() state with code?
More context:
This is running in a local Airflow instance using AWS ECS to run these tasks. Could it possibly be an ECS issue?
Thank you very much, please let me know if you have any questions!!