Deploying Prefect Agents on Kubernetes

Archive
, , , , ,
1

Summary

Deploying Prefect locally is fast and simple to get up and running.

When you’re in a production environment, and you can’t tolerate an agent failure - deploying Prefect into your Kubernetes environment provides that availability and resiliency.

This article will discuss a few common ways you might consider deploying Prefect in Kubernetes.

Videos

Configuring Prefect Agent in a Default Namespace -

Configuring Prefect Agent in a Custom Namespace -

Configuring Prefect Agent in Custom Namespace and Cloud API -

Configuration Options

There are three main configurations you might consider deploying your agent.

  1. Deploying the default prefect kubernetes manifest with no changes to the default namespace.
  2. Creating a modified prefect manifest, and deploying to a different namespace.
  3. Creating a modified prefect manifest, and deploying the agent to connect to the Cloud API.

Options 1 and 2 are great for development, small workloads, testing, and if you have a local Prefect Server (that is, you are running a Prefect Orion instance locally and not using the Cloud API.

Option 3 is ideal if you are using Prefect Cloud.

Requirements

The following steps assume you already have a valid KUBECONFIG and can successfully connect to your cluster using kubectl.

Additionally, an up-to-date local prefect installation > 2.0.0 is required.

Deploying to a Default Namespace

  1. Generate the manifest and apply to your cluster.
    prefect kubernetes manifest agent | kubectl apply -f -
  2. Configure your PREFECT_API_URL to point to localhost:4200 - this step directs your Prefect traffic locally.
    prefect config set PREFECT_API_URL=http://localhost:4200/api
  3. Configure port-forwarding, so your local prefect traffic (on 4200) is passed to the Prefect deployment in Kubernetes.
    kubectl port-forward deployments/orion 4200:4200
  4. Create a generic work-queue - by default, the standard manifest is checking for a ‘kubernetes’ work-queue.
    prefect work-queue create kubernetes

Deploying to a Custom Namespace

  1. Create a new namespace (unless it already exists).
    kubectl create namespace prefect2
  2. Generate the manifest and save the output to a file - here we are saving it to orion.yaml
    prefect kubernetes manifest agent > orion.yaml
  3. Edit orion.yaml - here we will specify the namespace.
    In each definition in the manifest, we will need to either add or update the namespace.
    Below are the changes from default manifest generated above.
    As the changes are in a code block, I have highlighted these changes with ** ** in the snippet below.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prefect-agent
  namespace: **prefect2**
  labels:
    app: prefect-agent
spec:
  selector:
    matchLabels:
      app: prefect-agent
  replicas: 1
  template:
    metadata:
      labels:
        app: prefect-agent
    spec:
      containers:
      - name: agent
        image: prefecthq/prefect:2.4.5-python3.9
        command: ["prefect", "agent", "start", "-q", "kubernetes"]
        imagePullPolicy: "IfNotPresent"
        env:
          - name: PREFECT_API_URL
            value: 
          - name: PREFECT_API_KEY
            value:
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: **prefect2**
  name: prefect-agent
rules:
- apiGroups: [""]
  resources: ["pods", "pods/log", "pods/status"]
  verbs: ["get", "watch", "list"]
- apiGroups: ["batch"]
  resources: ["jobs"]
  verbs: [ "get", "list", "watch", "create", "update", "patch", "delete" ]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: prefect-agent-role-binding
  namespace: **prefect2**
subjects:
- kind: ServiceAccount
  name: default
  namespace: **prefect2**
roleRef:
  kind: Role
  name: prefect-agent
  apiGroup: rbac.authorization.k8s.io
  1. Apply the modified manifest.
    kubectl apply -f orion.yaml
  2. Configure your PREFECT_API_URL to point to localhost:4200 - this step directs your Prefect traffic locally.
    prefect config set PREFECT_API_URL=http://localhost:4200/api
  3. Configure port-forwarding, so your local prefect traffic (on 4200) is passed to the Prefect deployment in Kubernetes. Since this deployment is now in another namespace, we need to specify that in the port-forward command via -n prefect2.
    kubectl port-forward deployments/prefect-agent 4200:4200 -n prefect2
  4. Create a generic work-queue - by default, the standard manifest is checking for a ‘kubernetes’ work-queue.
    prefect work-queue create kubernetes

Deploying to Custom Namespace using Prefect Cloud API.

For this configuration, we will re-use the existing orion.yaml definition created above.

  1. Configuring the following steps assumes you have your PREFECT_API_URL handy already. This appears like:
https://api.prefect.cloud/api/accounts/<removed>/workspaces/<removed>

If you do not have your API url handy, it can be retrieved via:
prefect config view

If it is not set locally, when you login to the Cloud UI, there should be “Set your workspace”:
prefect cloud workspace set --workspace "<account>/<workspace>"
Once set, your API can be retrieved via prefect config view

  1. Additionally, a PREFECT_API_KEY must already be created and handy.
  2. Create a secret in the prefect2 namespace containing your API key. Replace 12345 with your actual API key.
kubectl create secret generic api-key --from-literal='prefect_api_key=12345' -n prefect2
# api-key is the name of the secret, and visible via kubectl get secrets -n prefect2
# prefect_api_key is the key to access the value inside the secret.
# This is how we will reference the secret in the manifest without displaying.
  1. Edit the existing orion.yaml manifest file, where we will specify the new API key and URL.
    Values that have changed or added are bolded.
				
	env:
          - name: PREFECT_API_URL 
            value: **https://app.prefect.cloud/account/<>/workspace/<>**
        **- name: PREFECT_API_KEY** 
          **valueFrom:**
            **secretKeyRef:**
              **name: api-key** # This is the secret name from step 3
              **key: prefect_api_key** # This is the secret key from step 3
  1. Deploy the manifest.
    kubectl apply -f orion.yaml
  2. Since this is a Cloud pointed agent, we do not need to update or change or local API_URL - any changes we make locally , will update the configuration in the cloud, and will be retrieved by the agent.

Deploying Flows as Kubernetes Jobs

  • List item
5 Likes