How to deploy a Prefect 2.0 agent to an Azure Kubernetes Cluster and connect to Azure Blob Storage?

This tutorial will guide you through setting up a local AKS setup, provisioning Azure storage, and setting up the Prefect 2.0 Agent.
– At this time, the original configuration of storage used in this doc has been deprecated in favor of blocks. I have removed references to the previous storage configuration - configuring storage can now be done via Storage - Prefect 2.0

More documentation and terraform recipes can be found here:

There are a few pre-requisites necessary to proceed:
azure-cli
kubectl
kubelogin

Additionally, prefect 2.0 must be installed locally to interact and forward configurations to the cluster.

Lastly, this document anticipates you have a valid Service Principal or User Authorization to perform the necessary roles and steps. As this is provisioning compute, network, and storage, the “Contributor” role should have sufficient permissions necessary.
More details can be found here:

  1. Install pre-requisites.
brew install azure-cli
az aks install-cli --kubelogin-install-location mykubetools/kubelogin

===========+==================+==================+
Provision the infrastructure
===========+==================+==================+

  1. Login with your service principal; Sign in with Azure CLI — Login and Authentication | Microsoft Docs
    az login --service-principal -u <app-id> -p <password-or-cert> --tenant <tenant>

  2. Register the AKS provider in Azure ; Resource providers and resource types - Azure Resource Manager | Microsoft Docs

az provider register -n Microsoft.ContainerService

  1. Create a resource group, and export the value for re-use later
export rg="prefect_aks-rg"
az group create --name $rg --location eastus
  1. Create a vnet and subnet. For simplicity in this tutorial, we will allow AKS to use “kubenet” networking, which is by default, and requires no additional steps or configuration.
    If provisioning through Terraform, Azure CNI will be used - more details can be found here.
    Configure Azure CNI networking in Azure Kubernetes Service (AKS) - Azure Kubernetes Service | Microsoft Docs
az network vnet create -g $rg -n prefectvnet --address-prefix 10.1.0.0/16 \
    --subnet-name prefectsubnet --subnet-prefix 10.1.1.0/24
  1. Export and create an AKS cluster. Here we are creating a minimal configuration with 2 nodes for tutorial purposes.
    NOTE - Standard_B2s nodes might not be available in the eastus region if you are using a free-tier Azure account. See the following articles for more help in determining suitable locations / sku’s if you are using a free tier.
    Resource Skus - List - REST API (Azure Compute) | Microsoft Docs
    SKU not available errors - Azure Resource Manager | Microsoft Docs
export aks="myprefectAKSCluster"
az aks create --resource-group $rg --name "$aks" --node-count 2 --node-vm-size "Standard_B2s"
  1. Retrieve the connection kubeconfig from the cluster to interface. Here we are setting the output kubeconfig to an alternate location, to not merge with any existing contexts you might already have.
    az aks get-credentials --resource-group $rg --name "$aks" -f "~/.kube/$aks_config"

  2. Apply the prefect agent manifest. This step requires prefect orion installed locally. Alternatively, a template prefect.yaml is available in the PrefectRecipes github repository.

prefect kubernetes manifest orion | kubectl apply -f -

===========+==================+==================+
Configure the Prefect Agent
===========+==================+==================+

  1. Open a new terminal in your shell that will remain running, and execute the following to forward prefect agent traffic to the cluster.
kubectl port-foward deployment/orion 4200:4200
  1. In your original shell session, configure the agent - setting the API url to localhost, creating a work-queue.
prefect config set PREFECT_API_URL=http://127.0.0.1:4200/api
prefect work-queue create kubernetes
  1. At this point, your agent is successfully configured locally - a separate tutorial will be provided detailing connecting to Prefect Cloud as it requires additional AKS configuration to provision the ingress controller.

  2. A deployment can be created at this time. Deployments have changed from the previous revision of this article.
    First create a basic prefect flow, then we will create a deployment from it.

cat << EOF > my_k8s_deployment.py
from prefect import flow, get_run_logger
from prefect.deployments import DeploymentSpec
from prefect.filesystems import Azure

azure_block = Azure.load("boyd-block")

@flow(name="my_kubernetes_flow")
def my_kubernetes_flow():
    logger = get_run_logger()
    logger.info("Hello from Kubernetes!")
EOF

prefect deployment build ./my_k8s_deployment.py:my_kubernetes_flow -n demo_flow -t kubernetes -i kubernetes-job --storage-block azure/boyd-block

===========+==================+==================+
Optional Monitoring and Configuration
===========+==================+==================+

The following steps are entirely optional to get you started with monitoring , but are not required and at your own discretion.

  1. Use k8slens to provide a GUI for interfacing with your cluster. This is a fantastic tool for interfacing with any K8s cluster.
brew install lens
  1. Install helm for managing and provisioning manifests.
brew install helm
  1. Install prometheus for addition cluster monitoring and details
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack
5 Likes