" I’m new to Prefect and would like to deploy a Kubernetes Job at an Azure Kubernetes Service using Prefect. I’ve set up a Linux Container including Prefect, kubectl. I could establish a connection to the AKS cluster and by using kubectl, I could deploy a Kubernetes Job at the AKS. But how does it work with Prefect? At the Container, I’ve stored the .py file and a YAML file, which defines the Kubernetes Job.
→ kubectl apply -f deploytestcontainer.yaml → works
Running attached example Prefect code also works (but the Job is not getting deployed at the AKS)
That’s the content of “firstk8sjob.py”
import prefect from prefect import task, Flow from prefect.run_configs import KubernetesRun @task def hello_task(): flow.run_config = KubernetesRun(job_template_path="deploytestcontainer.yaml") logger = prefect.context.get("logger") logger.info("Hello world!") with Flow("hello-flow") as flow: hello_task() flow.run()
Thank you a lot in advance for your advice!"
There are two steps to deploy your flow to Azure AKS:
- Deploy your Kubernetes agent to this cluster
- Register your flow
First, you need to deploy a Kubernetes agent. You can generate a manifest file using:
prefect agent kubernetes install --rbac --key YOUR_API_KEY --label YOUR_LABEL > agent.yaml
Then you can check the file, and modify it as needed (e.g. change image version to match your desired Python and Prefect version, add environment variables when needed, etc).
Note that the API key is for Prefect Cloud - are you a Prefect Cloud or Prefect Server user? If you’re on Prefect Server, you need additionally those env variables.
Once your manifest is ready, you can apply it to your AKS cluster:
kubectl apply -f agent.yaml # optionally set: -n yournamespace
Then, once the agent is running, you can deploy your flow simply by specifying the label on your
Note that you were setting the run config within a task which is not recommended and may be the cause of your issue. You should attach the run config to your flow object as follows:
import prefect from prefect import task, Flow from prefect.run_configs import KubernetesRun @task def hello_task(): logger = prefect.context.get("logger") logger.info("Hello world!") with Flow("hello-flow", run_config=KubernetesRun(labels=["YOUR_LABEL"])) as flow: hello_task() if __name__ == "__main__": flow.register("YOUR_PROJECT_NAME")
You can also register within your CLI and get rid of this
prefect register --project YOUR_PROJECT_NAME -p path/to/flow.py
Once the flow is registered, you can trigger a run on AKS using:
prefect run --name "hello-flow" --project YOUR_PROJECT_NAME --watch