This post demonstrates how to run a Prefect 2 agent using Azure Container Instances (ACI).
Prerequisites
-
The latest version of the Azure CLI installed for your operating system and then authenticated.
-
A Prefect Cloud account. If you don’t have one yet, follow these instructions to sign up, create an API key, and sign the Prefect CLI into your cloud account.
-
A Prefect deployment built using the
process
infrastructure type and Azure Blob Storage. Although you can use a different Storage type such as S3, Blob Storage is the ideal storage option when running Prefect flows on Azure.
You might wonder why your deployment needs process
infrastructure rather than docker-container
infrastructure when you are deploying a Docker image to ACI.
A Prefect deployment’s infrastructure type describes how you want Prefect agents to run flows for the deployment. With docker-container
infrastructure, the agent will try to use Docker to spin up a new container for each flow run
Since you’ll be starting your own container on ACI, you don’t need Prefect to do it for you. Using a process
deployment tells Prefect you want to agent to run flows by starting a process in your ACI container.
Create a Resource Group
Like most Azure resources, Azure Container Instance applications must live in a resource group. If you don’t already have a resource group you’d like to use, create a new one by running:
az group create --name prefectAgentsGroup --location eastus
Feel free to change the group name to match your use case. You can also run az account list-locations -o table
to see all available resource group locations.
Running a Prefect Agent
It only takes one more Azure CLI command to run a Prefect agent on ACI. First, though, you’ll need a little more information: your Prefect API key, and the API URL for your Prefect workspace.
The Prefect CLI can give you both. Run:
prefect config view
and the output will be similar to:
PREFECT_PROFILE='your-profile'
PREFECT_API_KEY='your-api-key' (from profile)
PREFECT_API_URL='https://api.prefect.cloud/api/accounts/abc-123-456/workspaces/xyz-987-654' (from profile)
You now have everything you need to run a Prefect agent. Examine the following command:
bash/zsh
az container create \
-g prefectAgentsGroup \
--name prefect-agent \
--image prefecthq/prefect:2-python3.10 \
--secure-environment-variables PREFECT_API_URL=YOUR_API_URL PREFECT_API_KEY=YOUR_API_KEY \
--command-line "/bin/bash -c 'pip install adlfs; prefect agent start -q YOUR_WORK_QUEUE_NAME'"
Powershell
az container create `
-g prefectAgentsGroup `
--name prefect-agent `
--image prefecthq/prefect:2-python3.10 `
--secure-environment-variables PREFECT_API_URL=YOUR_API_URL PREFECT_API_KEY=YOUR_API_KEY `
--command-line "/bin/bash -c 'pip install adlfs; prefect agent start -q YOUR_WORK_QUEUE_NAME'"
Let’s start at the top and explore what is happening.
- The
az container create
command creates a new ACI container. - The
-g
flag tells Azure which resource group the new container belongs in. -
--name
determines the container name you will see in the Azure Portal. You can set any name you’d like here to suit your use case. -
--image
tells ACI which Docker images to run. The script above pulls a public Prefect image from Docker Hub.- You can also build custom images and push them to a public container registry so ACI can access them.
- Or you can push your image to a private Azure Container Registry and use it to create a container instance.
-
--secure-environment-variables
sets environment variables that are only visible from inside the container. They do not show up when viewing the container’s metadata — perfect for keeping app secrets safe. Make sure you update the valuesPREFECT_API_URL
andPREFECT_API_KEY
to match the ones you got earlier using the Prefect CLI. -
--command-line
lets you override the container’s normal entry point and run a command instead. The script above uses this section to install theadlfs
pip package so it can read flow code from Azure Blob Storage. It then runs the Prefect agent.- Update
YOUR_WORK_QUEUE_NAME
to match the name of the Prefect work queue you want the agent to connect to.
- Update
And that’s it! When you run the full az container create
command shown above, ACI pulls and runs the Docker image. Once the container is up and running, the Prefect agent runs, connects to Prefect Cloud, and is ready to run flows.