How to bind mount a volume to a flow run container - for example, mounting a volume with AWS credentials to a Docker agent

Archive
, , , ,
1

Prefect 2

To mount your AWS credentials to your DockerFlowRunner:

from prefect import flow, get_run_logger
from prefect.deployments import DeploymentSpec
from prefect.flow_runners import DockerFlowRunner


@flow
def docker_flow():
    logger = get_run_logger()
    logger.info("Hello from Docker!")


DeploymentSpec(
    name="example",
    flow=docker_flow,
    flow_runner=DockerFlowRunner(
        image="prefecthq/prefect:2.0b6-python3.9",
        volumes=["/Users/anna/.aws:/root/.aws"],
    ),
)

Note that you need to mount credentials to the Docker root user. Otherwise, it won’t work.

Prefect 1

On the agent

For local development with Docker, you can bind mount the AWS credentials directly to your agent - note that you must mount the credentials to the root user. Otherwise, it won’t work.

prefect agent docker start --label AGENT_LABEL --volume ~/.aws:/root/.aws

Usage example in a flow of flows

Kubernetes agent

If each machine on a given cluster has access to that network file system, then mounting it on the run_config’s job template would be the right place to do that rather than on the agent:

from prefect.run_configs import KubernetesRun

with Flow(
    name="your_flow", run_config=KubernetesRun(job_template=your_job_template)
) as flow:

But what would probably be easier in such an on-prem setup would be to use a Docker agent. You could then mount the volume directly on the agent, and this would make it available to all flow run containers (.aws is just an example):

prefect agent docker start --label AGENT_LABEL --volume ~/.aws:/root/.aws

In a Docker storage

Custom modules can be packaged up during build by attaching the files and setting the PYTHONPATH to the location of those files. Otherwise, the modules can be set independently when using a custom base image prior to the build here.

Docker(
    files={
        # absolute path source -> destination in image
        "/Users/me/code/mod1.py": "/modules/mod1.py",
        "/Users/me/code/mod2.py": "/modules/mod2.py",
    },
    env_vars={
        # append modules directory to PYTHONPATH
        "PYTHONPATH": "$PYTHONPATH:modules/"
    },
)