Run each task in a different Docker container using Prefect 2

Help
2

Hi @nss0204 :wave: Prefect lets you schedule flows in different infrastructures (such as containers). I imagine it’s possible to do something similar for tasks with Dask or Ray, but the most supported way is via flows. Here’s an example of what that looks like using Docker: GitHub - EmilRex/multi-container-demo

Notably, we use the run_deployment utility to schedule the additional flows which are then spawned in different Docker containers by the agent,

@task
def run_deployment_task(deployment_name):
    """Run the deployment in a task for async execution"""
    run_deployment(
        name=f"version/{deployment_name}",
    )

@flow
def parent():
    """Run three deployments in different docker containers"""
    deployment_names = [
        "main-2-8-5-python3-10",
        "main-2-8-6-python3-10",
        "main-2-8-7-python3-10",
    ]
    for deployment_name in deployment_names:
        run_deployment_task.submit(
            deployment_name,
        )