How does Prefect deploy flow runs on Kubernetes?
Prefect creates a Kubernetes job for each backend-triggered flow run. When this Kubernetes job gets deployed to your Kubernetes cluster (at flow runtime e.g. when your flow is scheduled to run), you will see one pod being deployed for the flow run.
So in short: by default, you get one pod per flow run.
How does Prefect deploy task runs on Kubernetes?
A short answer: one pod since the agent is a lightweight process responsible for polling for work and creating other pods for scheduled flow runs.
For the task run execution (not flow run!), you may leverage a DaskExecutor. When you use it with a KubeCluster
creating an on-demand Kubernetes cluster, you will see additional task run pods created by Dask.
For more on KubeCluster
check out:
https://discourse.prefect.io/tag/kube-cluster
How many pods are needed for my agent?
The Kubernetes agent itself is usually set up through a Kubernetes deployment. You can create and apply a deployment manifest using the following command:
prefect agent kubernetes install --rbac --key API_KEY | kubectl apply --namespace=YOUR_NAMESPACE -f -
The above command generates a YAML file with the agent definition and deploys the agent to your cluster.
Step by step agent creation
You can do the same as above step by step:
- Create the YAML deployment manifest
prefect agent kubernetes install --rbac --key API_KEY > kubernetes_agent.yaml
- Validate and manually adjust the file
kubernetes_agent.yaml
to be sure everything is set up correctly (e.g. your Prefect base image) - you may e.g. add some custom environment variables to this file. - Apply the deployment to your cluster within the desired namespace:
kubectl apply --namespace=YOUR_NAMESPACE -f kubernetes_agent.yaml
Once you finished, you should see one new pod created for your Kubernetes agent.