How to mount a folder on my local filesystem to a flow run pod on Kubernetes?

[anna_geller]

View in #prefect-community on Slack

@Jacqueline_Riley_Garrahan: I am testing Prefect deployed into a kind kubernetes cluster using the helm chart. I’d like to mount a folder on my local filesystem to a running job. I’ve configured this in the job template using:

apiVersion: batch/v1
kind: Job
metadata:
  name: prefect-job
  labels: {}
spec:
  template:
    metadata:
      labels: {}
    spec:
      containers:
        - name: flow
          image: prefecthq/prefect:latest
          imagePullPolicy: IfNotPresent
          ...
          volumeMounts:
            - name: filesystem-dir
              mountPath: /job-files
      volumes:
        - name: filesystem-dir
          hostPath:
            # directory location on host
            path: /path/to/my/dir
            # this field is optional
            type: Directory

I’m unable to mount the path which I believe is a function of the volume not being available on the agent. Is there a straightforward way for me to mount a volume to the agent in the helm chart deployment?

@Anna_Geller: Hi @Jacqueline_Riley_Garrahan, I wonder whether mounting a volume is the right approach here. Even if we manage to do that successfully, what happens when you want to deploy your Server and your flows to a real distributed Kubernetes cluster? It wouldn’t be possible to mount the files from your local file system then.

What may be actually much easier and better for your PoC would be to copy your files to the docker image and specify the image on your Kubernetes job template. Here is how it could look like:

#1 Create a Dockerfile

FROM prefecthq/prefect:1.0.0-python3.8
WORKDIR /opt/prefect
COPY your_dir/ /opt/prefect/your_dir/

#2 Build and push to some container registry:

docker build -t yourorg/your_image:latest .
docker push yourorg/your_image:latest

#3 Use this image in your job template:

apiVersion: batch/v1
kind: Job
metadata:
  name: prefect-job
  labels: {}
spec:
  template:
    metadata:
      labels: {}
    spec:
      containers:
        - name: flow
          image: yourorg/your_image:latest
          imagePullPolicy: IfNotPresent
          ...

Following this approach, there is no confusion about where to mount a file directory (flow run pod vs agent pod) and your image can be even easier tested locally before deploying it to Kubernetes

@Jacqueline_Riley_Garrahan: @Anna_Geller in our case, we’ll be deploying into an on-prem cluster and each resource will have access to the file system. Do you have any guidance on how to mount the volume using the helm chart?

@Anna_Geller: if each machine on that cluster has access to that network file system, then mounting it on the run_config’s job template would be the right place to do that rather than on the agent:

from prefect.run_configs import KubernetesRun

with Flow(
    name="your_flow", run_config=KubernetesRun(job_template=your_job_template)
) as flow:

But what would be probably easier in such on-prem setup would be to use a Docker agent? You could then mount the volume directly on the agent and this would make it available to all flow run containers (.aws is just an example):

prefect agent docker start --label AGENT_LABEL --volume ~/.aws:/root/.aws

@Jacqueline_Riley_Garrahan: Thanks for your help and time @Anna_Geller. In my case the issue was that I’m deploying into a kind cluster on my laptop, which requires the mount as well and was causing a directory not found error on the job deployment

@Anna_Geller: So is it solved now? Did the same job template work for you without the kind cluster?

@Jacqueline_Riley_Garrahan: I had to modify the template by dropping the command as mentioned here: https://github.com/PrefectHQ/prefect/issues/4525

GitHub: setting command in custom job template with seemingly-default setting breaks flow · Issue #4525 · PrefectHQ/prefect