How to spin up a Docker agent on Azure VM - a full walkthrough

Create an Azure Resource group

To start in a clean new environment, we’ll first create a new resource group. You can do that either from the CLI or from the Azure portal.

Using Azure CLI:

az group create --name PrefectDockerAgent --location eastus

From the Azure portal:


Click Review and create and then confirm once again:


Create an Azure VM

Within the resource group, click “Create”:

Then, select Ubuntu Server 20.04 LTS. If you don’t see it directly, you can find it under the “Compute” category.

:bangbang: Important note: choose Ubuntu 20.04 LTS as we do in this tutorial, otherwise you’ll likely get into trouble with the Python version when trying to use Prefect >= 1.0.0. More on that later.

Set your VM name - this should populate the name for your SSH key that you will be able to download later. You can accept the defaults for the rest - it’s important to keep port 22 open so that you can SSH to the instance.

Finally, confirm by “Create”:

This will prompt you to download the SSH key - download it (.pem file).
image

SSH to your VM

Go to “Connect”:

You should see the exact two commands to connect to your instance:

You can type those in your terminal and confirm with yes to connect:

cd ~/Downloads/
chmod 400 PrefectDockerAgent_key.pem
ssh -i PrefectDockerAgent_key.pem azureuser@public-ip-of-the-vm

Then, switch to the root user and you are ready to start installing everything you need for your Prefect setup!

Installation

First, run the following commands - you can paste those and run sequentially or execute them as a bash script:

sudo apt-get update -y
sudo apt-get upgrade -y
sudo apt-get install software-properties-common -y
sudo apt-get install python3-dateutil -y
sudo apt install python3-pip -y
sudo apt install docker.io -y
# the curl command installs Azure CLI: 
curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash
PATH="$HOME/.local/bin:$PATH"
export PATH

# python3 is installed by default with version 3.8.10
pip3 install pytz tabulate supervisor prefect

Once all that’s in progress, you can go to your Prefect Cloud UI and generate an API key.

Once you have the API key, you can log in to your Prefect Cloud account from the terminal using:

prefect auth login -k YOUR_API_KEY
export PREFECT__CLOUD__USE_LOCAL_SECRETS=false

The Secrets environment variable :point_up: is useful if you plan on using Secrets stored in Prefect Cloud e.g. for Azure storage.

Prefect, Python and Ubuntu OS version

By default, it installed the latest Prefect 1.0 version, which at the time of writing is 1.0.0 (using the default Python 3.8.10 version that ships with this VM at the time of writing).

If you use an Ubuntu version that is lower than 20.04 LTS, you would have to install Python manually because:

  • Ubuntu <= 18.04 LTS requires their default Python 3.6 version for the VM to work properly
  • Prefect 1.0 requires Python >= 3.7 so with Python 3.6 you could only install Prefect <= 0.15.13, or you would likely face difficulties managing multiple Python 3 installations on the same VM.

The VM setup and installation are now complete! :tada: All that’s left is to:

  • authenticate with Azure CLI
  • and start our Prefect agent.

Authenticate with Azure

Let’s now look at how you can authenticate with Azure.

We’ve already installed the Azure CLI. You can now run this command to log into your Azure account from the terminal:

az login

You should see a message similar to this one:

Follow the link Sign in to your account and complete the login with the provided code. You’ll then be prompted to sign into your Azure account:


Now that your CLI is authenticated, we can authenticate with ACR - Azure’s container registry.

Authenticate with ACR

In order to generate long-lived permissions for your Docker agent, you will need to create a service principal, which is Azure’s fancy way of saying: a username and password :smile:

We will follow the instructions from this documentation page to create the credentials:

Modify the bash script below to include your ACR registry name (replace my prefectcommunity registry name with your ACR registry name). I recommend keeping the acrpush permission scope so that you can also register your flows and push new images to ACR using the same set of credentials, but feel free to configure it depending on your needs.

Then, run those commands in your Ubuntu terminal - all that it’s really doing is generating the long-lived username and password, storing those as environment variables, and printing those to the console:

#!/bin/bash
# This script requires Azure CLI version 2.25.0 or later. Check version with `az --version`.

# Modify for your environment.
# ACR_NAME: The name of your Azure Container Registry
# SERVICE_PRINCIPAL_NAME: Must be unique within your AD tenant
ACR_NAME=prefectcommunity
SERVICE_PRINCIPAL_NAME=acr-service-principal-prefect-docker-agent-demo

# Obtain the full registry ID for subsequent command args
ACR_REGISTRY_ID=$(az acr show --name $ACR_NAME --query "id" --output tsv)

# Create the service principal with rights scoped to the registry.
# Default permissions are for docker pull access. Modify the '--role'
# argument value as desired:
# acrpull:     pull only
# acrpush:     push and pull
# owner:       push, pull, and assign roles
PASSWORD=$(az ad sp create-for-rbac --name $SERVICE_PRINCIPAL_NAME --scopes $ACR_REGISTRY_ID --role acrpush --query "password" --output tsv)
USER_NAME=$(az ad sp list --display-name $SERVICE_PRINCIPAL_NAME --query "[].appId" --output tsv)

# Output the service principal's credentials; use these in your services and
# applications to authenticate to the container registry.
echo "Service principal ID: $USER_NAME"
echo "Service principal password: $PASSWORD"

This will print the username and password:

Now, let’s get to a more exciting part - let’s use those beautiful long-lived Docker credentials to sign in to ACR so that our Docker agent will be able to push and pull docker images :fire:

docker login prefectcommunity.azurecr.io -u $USER_NAME -p $PASSWORD

Again, replace prefectcommunity with your ACR registry name.

You should see: Login succeeded.

Troubleshooting tips

If you:

  • don’t see Login Succeeded
  • see errors such as Values of identifierUris property must use a verified domain of the organization or its subdomain: 'http://anna-prefect-docker-agent'

Then, check your Azure CLI version and make 100% sure you are using Azure CLI version 2.25.0 or later. To check that, run:

az --version

For example, if you happen to have installed the Azure CLI using apt:

sudo apt install azure-cli -y

You will get an outdated version. The only way to get the right up-to-date Azure CLI version on Ubuntu is to use:

curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash

or other methods mentioned in this official Azure CLI install guide:

Set up your Prefect agent

Let’s now get to the most exciting part - setting up our Prefect Docker agent! :tada:

Feel free to replace the label “docker” with some other label or add new labels using the --label flag multiple times.

prefect agent local install --label docker > supervisord.conf
sed -i 's/agent local/agent docker/g' supervisord.conf
supervisord -c ./supervisord.conf

Now you should see an agent in your UI. To give it a try with a sample flow, you can register and run this flow (replace your ACR name as always):

Ensure agent restarts on a VM reboot

Let’s now make sure that even if our machine fails for some reason, or when we stop the VM for the night and resume later, our Docker agent spins up automatically on VM startup. To accomplish that, we need… :drum: drumroll please… the good old CRON tab!

Run this single line in your terminal:

echo "@reboot root supervisord -c /home/azureuser/supervisord.conf -l /home/azureuser/supervisord.log -u root" >> /etc/crontab

This line will add a crontab that starts your supervisor process every time your VM boots up.

Once you’ve done that, go to your Azure portal and stop the VM.

Make sure to select the checkbox to keep the IP address so that you can easily SSH using the same command as before:

You should see that the agent stopped querying for your flow runs.

Now start the VM again.

As soon as your machine boots up, you should see that your agent comes back up automatically - magic! :tada: All that we had to do to accomplish that was a single CRON tab.

This concludes our demo.

Questions?

If you have any questions or issues following this setup, hit reply as a comment and tell us where did you get stuck.