How many Kubernetes pods does Prefect spin up for a flow run? How does it change when using Dask's KubeCluster?

How does Prefect deploy flow runs on Kubernetes?

Prefect creates a Kubernetes job for each backend-triggered flow run. When this Kubernetes job gets deployed to your Kubernetes cluster (at flow runtime e.g. when your flow is scheduled to run), you will see one pod being deployed for the flow run.

So in short: by default, you get one pod per flow run.

How does Prefect deploy task runs on Kubernetes?

A short answer: one pod since the agent is a lightweight process responsible for polling for work and creating other pods for scheduled flow runs.

For the task run execution (not flow run!), you may leverage a DaskExecutor. When you use it with a KubeCluster creating an on-demand Kubernetes cluster, you will see additional task run pods created by Dask.

For more on KubeCluster check out:
https://discourse.prefect.io/tag/kube-cluster

How many pods are needed for my agent?

The Kubernetes agent itself is usually set up through a Kubernetes deployment. You can create and apply a deployment manifest using the following command:

prefect agent kubernetes install --rbac --key API_KEY | kubectl apply --namespace=YOUR_NAMESPACE -f - 

The above command generates a YAML file with the agent definition and deploys the agent to your cluster.

Step by step agent creation

You can do the same as above step by step:

  1. Create the YAML deployment manifest
prefect agent kubernetes install --rbac --key API_KEY > kubernetes_agent.yaml
  1. Validate and manually adjust the file kubernetes_agent.yaml to be sure everything is set up correctly (e.g. your Prefect base image) - you may e.g. add some custom environment variables to this file.
  2. Apply the deployment to your cluster within the desired namespace:
kubectl apply --namespace=YOUR_NAMESPACE -f kubernetes_agent.yaml

Once you finished, you should see one new pod created for your Kubernetes agent.