Mounting Volume to DockerRun for Local Storage

Archive
, , , , ,
1

When using Local Storage (with the flow stored on the host machine) with DockerRun, the Flow needs to be accessible from inside the container. There are two ways to do this:

  1. Create a new image that contains the Flow file. The LocalStorage can then point to the location of the file inside the container. An example can be found here

  2. For faster prototyping, you can develop locally and just mount the local directory to the DockerRun Container. In order to do this, just mount the local directory on the agent.

prefect agent docker start --volume /Users/kevinkho/Work/scratch/:/usr/src/prefect

And then the Flow storage can point to the mount

from prefect import Flow, task
from prefect.storage import Local
from prefect.run_configs import DockerRun

@task
def do_something(x: int):
    return x

with Flow(
    "docker-test",
    storage=Local(stored_as_script=True, path="/usr/src/prefect/param_test.py"),
    run_config=DockerRun()
) as flow:
    do_something(1)

flow.storage.add_default_labels = False
flow.register("project_name")

But this means that the agent has to add the volume to every flow run it deploys. To mount a volume for a specific flow, the Run Configuration can be used.

The file in this setup lives in:
/Users/kevinkho/Work/scratch/param_test.py

with Flow(
    "docker-test",
    storage=Local(stored_as_script=True, path="/usr/src/prefect/param_test.py"),
    run_config=DockerRun(
        host_config={
            "binds": {
                "/Users/kevinkho/Work/scratch/": {
                    "bind": "/usr/src/prefect/",
                    "mode": "rw",
                }
            }
        }
    ),
) as flow:
    do_something(1)
3 Likes
2

Can you do something like this with a cifs mount that has authentication credentials? Essentially a volume created with docker volume create

3

Joe, check this Low-level API — Docker SDK for Python 5.0.3 documentation

looks like the answer is Yes, if you leverage this arg instead of binds:

  • volumes_from (list) – List of container names or IDs to get volumes from.