View in #prefect-community on Slack
Question
@Carmen_Marcos: Hello everyone! During the past few weeks I have been learning how to use Prefect locally. However, now I am trying to deploy a flow using 4 OpenStack machines and I have some questions about the possible implementation:
• Is it possible to make a flow of flows in which each flow executes on a specific machine of the 4 I have (without using dask)? Machines would share volumes between them.
• In one of the machines I want to run a flow (triggered by other flow on a different machine) that must run a docker with a custom image when certain criteria is meet. Could I use the Docker Agent with the Local Executor for that flow?
Thank you in advance!
Answer
@Anna_Geller: Hi! This is definitely possible by leveraging different run configs for each child flow. And to define a specific condition, you can use the case
task, as shown here:
from prefect import Flow, task, case
from prefect.tasks.prefect import create_flow_run, wait_for_flow_run
from prefect.run_configs import DockerRun
@task
def check_the_infrastructure():
return "Docker with label machine-1"
with Flow("parent_flow") as flow:
infra = check_the_infrastructure()
with case(infra, "Docker with label machine-1"):
child_flow_run_id = create_flow_run(
flow_name="child_flow_name",
run_config=DockerRun(
labels=["machine-1"]
), # with a specific condition like image
)
child_flowrunview = wait_for_flow_run(
child_flow_run_id, raise_final_state=True, stream_logs=True
)
with case(infra, "Docker with label machine-2"):
child_flow_run_id = create_flow_run(
flow_name="child_flow_name",
run_config=DockerRun(
labels=["machine-2"]
), # with another different condition like image, label etc.
)
another_child_flowrunview = wait_for_flow_run(
child_flow_run_id, raise_final_state=True, stream_logs=True
)
Note that:
- this requires running multiple agents - one per machine.
- each agent should have a unique label: Local Agent | Prefect Docs