How to Run a Prefect 2 Agent in an Azure Container Instance

[ryan_peden]

This post demonstrates how to run a Prefect 2 agent using Azure Container Instances (ACI).

Prerequisites

• The latest version of the Azure CLI installed for your operating system and then authenticated.

• A Prefect Cloud account. If you don’t have one yet, follow these instructions to sign up, create an API key, and sign the Prefect CLI into your cloud account.

• A Prefect deployment built using the process infratructure type and Azure Blob Storage. Although you can use a different Storage type such as S3, Blob Storage is the ideal storage option when running Prefect flows on Azure.

You might wonder why your deployment needs process infrastructure rather than docker-container infrastructure when you are deploying a Docker image to ACI.

A Prefect deployment’s infrastructure type describes how you want Prefect agents to run flows for the deployment. With docker-container infrastructure, the agent will try to use Docker to spin up a new container for each flow run

Since you’ll be starting your own container on ACI, you don’t need Prefect to do it for you. Using a process deployment tells Prefect you want to agent to run flows by starting a process in your ACI container.

Create a Resource Group

Like most Azure resources, Azure Container Instance applications must live in a resource group. If you don’t already have a resource group you’d like to use, create a new one by running:

az group create --name prefectAgentsGroup --location eastus

Feel free to change the group name to match your use case. You can also run az account list-locations -o table to see all available resource group locations.

Running a Prefect Agent

It only takes one more Azure CLI command to run a Prefect agent on ACI. First, though, you’ll need a little more information: your Prefect API key, and the API URL for your Prefect workspace.

The Prefect CLI can give you both. Run:

prefect config view

and the output will be similar to:

PREFECT_PROFILE='your-profile'
PREFECT_API_KEY='your-api-key' (from profile)
PREFECT_API_URL='https://api.prefect.cloud/api/accounts/abc-123-456/workspaces/xyz-987-654' (from profile)

You now have everything you need to run a Prefect agent. Examine the following command:

bash/zsh

az container create \
-g prefectAgentsGroup \
--name prefect-agent \
--image prefecthq/prefect:2-python3.10 \
--secure-environment-variables PREFECT_API_URL=YOUR_API_URL PREFECT_API_KEY=YOUR_API_KEY \
--command-line "/bin/bash -c 'pip install adlfs; prefect agent start -q YOUR_WORK_QUEUE_NAME'"

Powershell

az container create `
-g prefectAgentsGroup `
--name prefect-agent `
--image prefecthq/prefect:2-python3.10 `
--secure-environment-variables PREFECT_API_URL=YOUR_API_URL PREFECT_API_KEY=YOUR_API_KEY `
--command-line "/bin/bash -c 'pip install adlfs; prefect agent start -q YOUR_WORK_QUEUE_NAME'"

Let’s start at the top and explore what is happening.

• The az container create command creates a new ACI container.
• The -g flag tells Azure which resource group the new container belongs in.
• --name determines the container name you will see in the Azure Portal. You can set any name you’d like here to suit your use case.
• --image tells ACI which Docker images to run. The script above pulls a public Prefect image from Docker Hub.
  • You can also build custom images and push them to a public container registry so ACI can access them.
  • Or you can push your image to a private Azure Container Registry and use it to create a container instance.
• --secure-environment-variables sets environment variables that are only visible from inside the container. They do not show up when viewing the container’s metadata — perfect for keeping app secrets safe. Make sure you update the values PREFECT_API_URL and PREFECT_API_KEY to match the ones you got earlier using the Prefect CLI.
• --command-line lets you override the container’s normal entry point and run a command instead. The script above uses this section to install the adlfs pip package so it can read flow code from Azure Blob Storage. It then runs the Prefect agent.
  • Update YOUR_WORK_QUEUE_NAME to match the name of the Prefect work queue you want the agent to connect to.

And that’s it! When you run the full az container create command shown above, ACI pulls and runs the Docker image. Once the container is up and running, the Prefect agent runs, connects to Prefect Cloud, and is ready to run flows.