What is the difference between base job template of work pool and base job manifest of kubernetes job block?

Help
, ,
1

I am a k8s user, and I recently tried to deploy the prefect on my local cluster.

  • The Prefect worker is deployed in my local cluster via prefect-worker helm chart.
  • I use Prefect Cloud as my prefect-server.
  • I apply the deployment via my local terminal.

Although I’ve successfully applied and run the flow, I am confused about the difference between the base job template of work pool and base job manifest of kubernetes job block?

According to the result, I found base job manifest of Kubernetes job block is useless since the worker pools the work from the work pool and creates jobs based on the base job template of work pool

Here are some questions I want to ask or need to be clarified in the document.

  • When will base job template of work pool be used?
  • What is the best practice to deploy/setup a k8s worker/work pool with the deployment?
    • I think the documents currently cause confusion that the newbie cannot find a steps to deploy the Prefect in the Kubernetes.
2

hi @Leo_Lin

the steps for setting up a worker / work pool for k8s is as follows:

  • create a kubernetes type work pool (no configuration required yet)
  • follow this guide for deploying a worker on your k8s cluster using the helm chart
  • go back and customize your work pool’s base job configuration template as needed
  • create deployment definitions in your prefect.yaml, overriding the base job template as needed on a deployment-by-deployment basis

let us know if you develop any questions!

1 Like
3

Hi @nate

Thank you for sharing the best practice! I want to confirm some detail since it looks like a little different from the post in Prefect medium blog. Don’t we need to set up an infra block such as a Kubernetes-job block?
ref: How to use Kubernetes with Prefect: Part 2 | by Jeff Hale | The Prefect Blog | Medium

4

hi @Leo_Lin - using an infra and storage block to specify a deployment was our first iteration of deployment management.

Just last week, we took Workers and Work Pools out of beta (after a couple months) and is now our main recommendation for creating and managing Prefect Deployments.

to quickstart, make a new folder for your project and run

prefect init

write a simple flow

from prefect import flow

@flow
def hello_marvin():
   return "hello Marvin!"

and

prefect deploy

Going through this process will help things make sense I think, and then you could return to my list above to setup your kubernetes runtime!

more docs

5

Hi @nate,

Really appreciate the clarification! I think it is worth mentioning the different iterations of deployment management more straightforwardly in the official documents.

1 Like
6

thank you for the feedback @Leo_Lin - I will advocate for making that more clear!