@Carmen_Marcos: Hello everyone! During the past few weeks I have been learning how to use Prefect locally. However, now I am trying to deploy a flow using 4 OpenStack machines and I have some questions about the possible implementation:
• Is it possible to make a flow of flows in which each flow executes on a specific machine of the 4 I have (without using dask)? Machines would share volumes between them.
• In one of the machines I want to run a flow (triggered by other flow on a different machine) that must run a docker with a custom image when certain criteria is meet. Could I use the Docker Agent with the Local Executor for that flow?
Thank you in advance!
@Anna_Geller: Hi! This is definitely possible by leveraging different run configs for each child flow. And to define a specific condition, you can use the
case task, as shown here:
from prefect import Flow, task, case from prefect.tasks.prefect import create_flow_run, wait_for_flow_run from prefect.run_configs import DockerRun @task def check_the_infrastructure(): return "Docker with label machine-1" with Flow("parent_flow") as flow: infra = check_the_infrastructure() with case(infra, "Docker with label machine-1"): child_flow_run_id = create_flow_run( flow_name="child_flow_name", run_config=DockerRun( labels=["machine-1"] ), # with a specific condition like image ) child_flowrunview = wait_for_flow_run( child_flow_run_id, raise_final_state=True, stream_logs=True ) with case(infra, "Docker with label machine-2"): child_flow_run_id = create_flow_run( flow_name="child_flow_name", run_config=DockerRun( labels=["machine-2"] ), # with another different condition like image, label etc. ) another_child_flowrunview = wait_for_flow_run( child_flow_run_id, raise_final_state=True, stream_logs=True )
- this requires running multiple agents - one per machine.
- each agent should have a unique label: Local Agent | Prefect Docs