This post demonstrates how to run a Prefect 2 agent using Azure Container Instances (ACI).
A Prefect Cloud account. If you don’t have one yet, follow these instructions to sign up, create an API key, and sign the Prefect CLI into your cloud account.
A Prefect deployment built using the
processinfrastructure type and Azure Blob Storage. Although you can use a different Storage type such as S3, Blob Storage is the ideal storage option when running Prefect flows on Azure.
You might wonder why your deployment needs
process infrastructure rather than
docker-container infrastructure when you are deploying a Docker image to ACI.
A Prefect deployment’s infrastructure type describes how you want Prefect agents to run flows for the deployment. With
docker-container infrastructure, the agent will try to use Docker to spin up a new container for each flow run
Since you’ll be starting your own container on ACI, you don’t need Prefect to do it for you. Using a
process deployment tells Prefect you want to agent to run flows by starting a process in your ACI container.
Like most Azure resources, Azure Container Instance applications must live in a resource group. If you don’t already have a resource group you’d like to use, create a new one by running:
az group create --name prefectAgentsGroup --location eastus
Feel free to change the group name to match your use case. You can also run
az account list-locations -o table to see all available resource group locations.
It only takes one more Azure CLI command to run a Prefect agent on ACI. First, though, you’ll need a little more information: your Prefect API key, and the API URL for your Prefect workspace.
The Prefect CLI can give you both. Run:
prefect config view
and the output will be similar to:
PREFECT_PROFILE='your-profile' PREFECT_API_KEY='your-api-key' (from profile) PREFECT_API_URL='https://api.prefect.cloud/api/accounts/abc-123-456/workspaces/xyz-987-654' (from profile)
You now have everything you need to run a Prefect agent. Examine the following command:
az container create \ -g prefectAgentsGroup \ --name prefect-agent \ --image prefecthq/prefect:2-python3.10 \ --secure-environment-variables PREFECT_API_URL=YOUR_API_URL PREFECT_API_KEY=YOUR_API_KEY \ --command-line "/bin/bash -c 'pip install adlfs; prefect agent start -q YOUR_WORK_QUEUE_NAME'"
az container create ` -g prefectAgentsGroup ` --name prefect-agent ` --image prefecthq/prefect:2-python3.10 ` --secure-environment-variables PREFECT_API_URL=YOUR_API_URL PREFECT_API_KEY=YOUR_API_KEY ` --command-line "/bin/bash -c 'pip install adlfs; prefect agent start -q YOUR_WORK_QUEUE_NAME'"
Let’s start at the top and explore what is happening.
az container createcommand creates a new ACI container.
-gflag tells Azure which resource group the new container belongs in.
--namedetermines the container name you will see in the Azure Portal. You can set any name you’d like here to suit your use case.
--imagetells ACI which Docker images to run. The script above pulls a public Prefect image from Docker Hub.
- You can also build custom images and push them to a public container registry so ACI can access them.
- Or you can push your image to a private Azure Container Registry and use it to create a container instance.
--secure-environment-variablessets environment variables that are only visible from inside the container. They do not show up when viewing the container’s metadata — perfect for keeping app secrets safe. Make sure you update the values
PREFECT_API_KEYto match the ones you got earlier using the Prefect CLI.
--command-linelets you override the container’s normal entry point and run a command instead. The script above uses this section to install the
adlfspip package so it can read flow code from Azure Blob Storage. It then runs the Prefect agent.
YOUR_WORK_QUEUE_NAMEto match the name of the Prefect work queue you want the agent to connect to.
And that’s it! When you run the full
az container create command shown above, ACI pulls and runs the Docker image. Once the container is up and running, the Prefect agent runs, connects to Prefect Cloud, and is ready to run flows.