Run each task in a different Docker container using Prefect 2

Help
1

Is there a way to run a Prefect 2 flow with a number of tasks that each run in separate Docker container and can each be assigned different resource requirements (CPU, memory, etc) ? I looked at https://discourse.prefect.io/t/can-prefect-run-each-task-in-a-different-docker-container/434, however that is using Prefect 1 and the same functions have changed significantly in Prefect 2.

How do you do the same using Prefect 2?

2

Hi @nss0204 :wave: Prefect lets you schedule flows in different infrastructures (such as containers). I imagine it’s possible to do something similar for tasks with Dask or Ray, but the most supported way is via flows. Here’s an example of what that looks like using Docker: GitHub - EmilRex/multi-container-demo

Notably, we use the run_deployment utility to schedule the additional flows which are then spawned in different Docker containers by the agent,

@task
def run_deployment_task(deployment_name):
    """Run the deployment in a task for async execution"""
    run_deployment(
        name=f"version/{deployment_name}",
    )

@flow
def parent():
    """Run three deployments in different docker containers"""
    deployment_names = [
        "main-2-8-5-python3-10",
        "main-2-8-6-python3-10",
        "main-2-8-7-python3-10",
    ]
    for deployment_name in deployment_names:
        run_deployment_task.submit(
            deployment_name,
        )