How to deploy a flow to Kubernetes in Azure AKS with Prefect

Archive
, , , , , , ,
1

Question from StackOverflow

" I’m new to Prefect and would like to deploy a Kubernetes Job at an Azure Kubernetes Service using Prefect. I’ve set up a Linux Container including Prefect, kubectl. I could establish a connection to the AKS cluster and by using kubectl, I could deploy a Kubernetes Job at the AKS. But how does it work with Prefect? At the Container, I’ve stored the .py file and a YAML file, which defines the Kubernetes Job.

enter image description here

→ kubectl apply -f deploytestcontainer.yaml → works

Running attached example Prefect code also works (but the Job is not getting deployed at the AKS)

enter image description here

That’s the content of “firstk8sjob.py”

import prefect
from prefect import task, Flow
from prefect.run_configs import KubernetesRun
 
@task
def hello_task():
    flow.run_config = KubernetesRun(job_template_path="deploytestcontainer.yaml")
    logger = prefect.context.get("logger")
    logger.info("Hello world!")
 
with Flow("hello-flow") as flow:
    hello_task()
 
flow.run()

Thank you a lot in advance for your advice!"

Answer

There are two steps to deploy your flow to Azure AKS:

  1. Deploy your Kubernetes agent to this cluster
  2. Register your flow

First, you need to deploy a Kubernetes agent. You can generate a manifest file using:

prefect agent kubernetes install --rbac --key YOUR_API_KEY --label YOUR_LABEL > agent.yaml

Then you can check the file, and modify it as needed (e.g. change image version to match your desired Python and Prefect version, add environment variables when needed, etc).

Note that the API key is for Prefect Cloud - are you a Prefect Cloud or Prefect Server user? If you’re on Prefect Server, you need additionally those env variables.

Once your manifest is ready, you can apply it to your AKS cluster:

kubectl apply -f agent.yaml # optionally set: -n yournamespace

Then, once the agent is running, you can deploy your flow simply by specifying the label on your KubernetesRun .

Note that you were setting the run config within a task which is not recommended and may be the cause of your issue. You should attach the run config to your flow object as follows:

import prefect
from prefect import task, Flow
from prefect.run_configs import KubernetesRun


@task
def hello_task():
    logger = prefect.context.get("logger")
    logger.info("Hello world!")


with Flow("hello-flow", run_config=KubernetesRun(labels=["YOUR_LABEL"])) as flow:
    hello_task()

if __name__ == "__main__":
    flow.register("YOUR_PROJECT_NAME")

You can also register within your CLI and get rid of this "__main__" block:

prefect register --project YOUR_PROJECT_NAME -p path/to/flow.py

Once the flow is registered, you can trigger a run on AKS using:

prefect run --name "hello-flow" --project YOUR_PROJECT_NAME --watch

Source