Hi, I have a very specific problem. I need to run a dynamic number of subflows (number decided by parent tasks) in a sequence. Indeed, I need those subflows to be run one at a time (in order to not overload the dask scheduler with too many tasks at once). I created a LOOP task that create the su…

[image] smartan007: I need to run a dynamic number of subflows (number decided by parent tasks) This sounds like a use case for mapping, rather than looping. If you opted for LOOP instead of mapping because you need to run it sequentially, then the solution would be running this flow with a L…

Persist context of LOOP tasks for flow restarts

Archive

anna_geller July 1, 2022, 12:14pm 8

@smartan007 sorry for late reply, I was on vacation

so it looks like the problem you are facing is concurrency limiting? this is something available out of the box in Prefect Cloud:

in Server, you can do:

Topic		Replies	Views
Can my subflow use a different task runner than my parent flow? Archive migration-guide , prefect-1-0 , prefect-2-0 , flow-of-flows , executor , task-runner , create_flow_run , orchestrator-pattern , subflows , wait_for_flow_run	0	935	January 31, 2022
We want to create a mapped task that retries N times and then proceeds to the downstream task if "any_successful" - Is it possible? Archive prefect-1-0 , parallel-processing , state-dependencies , mapping , local-dask-executor , slack-notifications , notifications , failure	0	849	March 6, 2022
Run different Prefect Tasks in different clusters Archive dask-executor	1	403	June 24, 2022
Trouble executing subflows concurrently with DaskTaskRunner Help prefect-2-0 , dask-task-runner , async	0	710	February 27, 2023
Using Dask client persistence within a Prefect Flow Help prefect-2-0 , dask , dask-dataframes	0	424	August 28, 2023

Persist context of LOOP tasks for flow restarts

Related Topics