Running subflows in parallel with downstream dependency

Have been working hard on some migration efforts recently. Really love the new perks with 2.0 :hugs: but still have some minor issues I found hard to migrate:

Previously in 1.0 we can:

  1. Define flow starter tasks like other tasks, and they will be orchestrated and run in parallel if necessary
  2. Let downstream tasks/subflows to have dependence linked to subflow runs.

Understood from the examples that in 2.0 we can use async to run subflows in parallel, but how would we ensure parallel subflow execution if we are using subflows in a flow with rather complex upstream/downstream tasks dependencies?

In another word, is it possible to make flows return PrefectFutures and adhere to the same submit(wait_for=[xxx]) signature?

What I want to achieve:

def master_flow():
    task1 = some_task.submit()
    task2 = some_task.submit()
    subflow_1_result = subflow_1.submit(wait_for=[task_1])
    subflow_2_result = subflow_2.submit(wait_for=[task_2])
    task_final = final_task.submit(wait_for=[subflow_1_result , task_2])

And in this case, we should see subflow 1&2 running in parallel if task1&2 finish at around the same time, we should also see task_final being run once both subflow_1 and task2 finish.

Also stumbled upon these GitHub request that seems related:

Is there an ETA on this feature? Are there any possible workarounds? :face_with_monocle:

2 Likes

I fully understand the problem and Iā€™m with you on that, we should add that. No ETA, but Iā€™d recommend following up on this GitHub issue. I will link this topic to it.

1 Like