Run different Prefect Tasks in different clusters

Archive
1

Is it possible to run tasks within a flow in different environments (specifically for my use case in separate Dask Clusters)?

The motivation behind this is that I would want different Tasks to have access to different resources, but want to keep it in the same flow to allow for Prefect to help manage some state and context between the Task Runs.

Example flow:

task1 = Task1()
task2 = Task2()

with Flow() as flow:
    task1_results = task1.map(params) # execute this in Cluster 1
    task2_results = task2.map(task1_results) # execute this in Cluster 2
2

So far the answer here is maybe. See this. I am thinking that you can use a resource manager to send that work to another cluster, but another user reported to me it did not work. I suggest you try, and if it fails, just use a flow of flows with each subflow on a different executor