Set the job_template in Kubernetes

Help
,
1

Hi,

I am trying to set a custom job template in kubernetes, as suggested here (Worker - prefect-kubernetes) but I don’t understand where to do it. I can successfully set the variables in the template in my prefect.yaml by doing something like:

definitions:
  tags: &common_tags
  - k8s
  work_pool: &common_work_pool
    name: kubernetes-pool
    job_variables:
      image: '{{ build-image.image }}'
      namespace: my-namespace-default

But where can I set a custom template? It seems like it would be a property of the work-pool, but I can’t seem to figure out how to make it work.

We setup the prefect-server and prefect-worker using the Helm charts.

Thanks,

Matt

2

Update: So I see that the documentation says:

For advanced use cases, users can create work pools with fully customizable job templates. This customization is available when creating or editing a work pool on the 'Advanced' tab within the UI.

Is that the only way? I am able to make the changes I want to add nodeSelector and tolerations to the job template via the UI, but I’d really like to do it programmatically when creating a new work pool. Anyone have a suggestion on how to do this?

I’ve looked through the code examples here, prefect-kubernetes, but it does not seem that that is exactly what I need. It looks like I need to maybe create a KubernetesWorkerJobConfiguration but then I am not sure how I would apply that to a work pool.

Maybe I need to switch back to agents? It seemed that the trend (recommendation) was to move away from agents, so I’d figured I would try that, but maybe for certain things agents are the way to go?

3

i would like to know this too. How do i version control the workpool job templates?

I tried to get the manifest from prefect kubernetes manifest flow-run-job
edit the manifest to change the environment variables but when i try to apply it by kubectl apply -f job-manifest.yaml I get the error

error: error when retrieving current configuration of:
Resource: "batch/v1, Resource=jobs", GroupVersionKind: "batch/v1, Kind=Job"
Name: "", Namespace: "data"

my manifest looks like this

apiVersion: batch/v1
kind: Job
metadata:
  # labels are required, even if empty
  labels: {}
  namespace: data
  stream_output: true
spec:
  template:
    spec:
      completions: 1
      containers:
        - name: prefect-job
          env:
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  key: DB_PASSWORD
                  name: etl
          image: xxxxx.dkr.ecr.eu-central-1.amazonaws.com/data/prefect_agent:latest
      parallelism: 1
      restartPolicy: Never
4

Did you find how to set the custom job manifest?

5

Hi Matthew,

This effort is currently in progress to make this possible!
At the moment, the work pool template can be modified in the UI only, but once released, it will be possible to create one programmatically via the Python SDK.

The prefect kubernetes manifest command is explicitly for the agent, and not related to the worker.

I keep a few templated work-pools as local files, that I can just copy and paste into my work pool definition as needed, for example if I want to set up taints and tolerations, or initContainers, or side car logging.

6

Are there any updates on the ability to programmatically control the base template?

7

This should be available to create and update custom job templates since 2.13.3 -

8

Ah! Thanks for the info!

9

Hi All, anyone found a way for this?