Tasks are not running in parallel when using DaskTaskRunner() and returning state on the task.submit() call

[yaron]

Hi (-:

prefect == 2.7.7

I have a flow defined with a task_runner=DaskTaskRunner()

As you can see, this flow iterates on several cats and for each creates a new Task using the submit() function. The task is:

But when looking at the logs, I can see that the tasks run sequentially:

The agent that serves this run sits on an instance on Render.

The resources this instance has are:

Any idea why the tasks don’t run in parallel? Where should I start looking for the problem?

[yaron]

got it (-:
it was that return_state=True!
after I removed it, we are back to parallel.

[anna_geller]

That makes sense, states make those calls blocking and without it, you operate on Prefect Futures

thanks so much for sharing the solution and great writeup!

[yaron]

Thanks (-:
There is an issue in prefect-dask about that.