Can I run a flow-of-flows that triggers each child flow on a different machine?

View in #prefect-community on Slack

Question

@Carmen_Marcos: Hello everyone! During the past few weeks I have been learning how to use Prefect locally. However, now I am trying to deploy a flow using 4 OpenStack machines and I have some questions about the possible implementation:

• Is it possible to make a flow of flows in which each flow executes on a specific machine of the 4 I have (without using dask)? Machines would share volumes between them.
• In one of the machines I want to run a flow (triggered by other flow on a different machine) that must run a docker with a custom image when certain criteria is meet. Could I use the Docker Agent with the Local Executor for that flow?
Thank you in advance! :pray:

Answer

@Anna_Geller: Hi! This is definitely possible by leveraging different run configs for each child flow. And to define a specific condition, you can use the case task, as shown here:

from prefect import Flow, task, case
from prefect.tasks.prefect import create_flow_run, wait_for_flow_run
from prefect.run_configs import DockerRun


@task
def check_the_infrastructure():
    return "Docker with label machine-1"


with Flow("parent_flow") as flow:
    infra = check_the_infrastructure()
    with case(infra, "Docker with label machine-1"):
        child_flow_run_id = create_flow_run(
            flow_name="child_flow_name",
            run_config=DockerRun(
                labels=["machine-1"]
            ),  # with a specific condition like image
        )
        child_flowrunview = wait_for_flow_run(
            child_flow_run_id, raise_final_state=True, stream_logs=True
        )
    with case(infra, "Docker with label machine-2"):
        child_flow_run_id = create_flow_run(
            flow_name="child_flow_name",
            run_config=DockerRun(
                labels=["machine-2"]
            ),  # with another different condition like image, label etc.
        )
        another_child_flowrunview = wait_for_flow_run(
            child_flow_run_id, raise_final_state=True, stream_logs=True
        )

:point_right: Note that: