How to run Prefect in Azure Container Instances (ACI)

Help
, , ,
1

Azure Container Infrastructure Reproduction

Follow along with the video tutorial on YouTube here - How to run Prefect in Azure Container Instances (ACI) with Prefect agents - YouTube

Building and Pushing an Image

  1. Create a Private docker Registry (simulate private repo) – chaboy/private_test:latest

  2. Create a new directory -
    cd ~/ && mkdir this_is_only_a_test
    cd this_is_only_a_test

  3. Write a hello world flow:

cat << EOF > flow.py
from prefect import task, flow
from prefect import get_run_logger


@task
def log_platform_info():
    import platform
    import sys
    logger = get_run_logger()
    logger.info("Host's network name = %s", platform.node())
    logger.info("Python version = %s", platform.python_version())
    logger.info("Platform information (instance type) = %s ", platform.platform())
    logger.info("OS/Arch = %s/%s", sys.platform, platform.machine())


@flow(log_prints=True)
def healthcheck():
    log_platform_info()

if __name__ == "__main__":
    healthcheck()
EOF
  1. Create a requirements file (optional)
cat <<EOF > requirements.txt
adlfs ## For Azure Storage Usage
prefect-azure ## For AzureContainerInstance Jobs on the ACI Agent
EOF
  1. Create a Dockerfile:
cat <<EOF > Dockerfile
FROM prefecthq/prefect:2-python3.9

COPY flow.py /opt/prefect/flows/
COPY requirements.txt /opt/prefect/flows/

RUN pip install -r /opt/prefect/flows/requirements.txt
EOF

  1. Build the docker image:
    docker build --platform linux/amd64 -t chaboy/private_test:latest .

  2. Push to registry:
    docker push chaboy/private_test:latest

  3. Export the image tag for re-use in the later command
    export image_tag='chaboy/private_test:latest'

Setting up ACI

Requires a user, or service principal with appropriate permissions.

  1. Export a resource group name, as we will be re-using in later commands
    export boyd_rg=BoydACIPrefectAgent

  2. Create a resource group using exported RG variable
    az group create --name $boyd_rg --location eastus

  3. Verify using exported RG variable:
    az group exists --resource-group $boyd_rg

  4. Have your prefect api URL, and prefect api key on hand.
    Verify that they are functional BEFORE configuring as a secure environment variable to save troubleshooting headache if they are not valid.

prefect config view
export PREFECT_API_URL=https://api.prefect.cloud/api/accounts/<replaced>/workspaces/<replaced>
export PREFECT_API_KEY=abc_12345678910

Test Verification first on the /health endpoint (should return “true”)
curl -i -H "Authorization: Bearer $PREFECT_API_KEY" "$PREFECT_API_URL/health"
401 indicates a valid API key, but invalid or incorrect workspace
404 a bad URL
403 indicates an invalid API key

  1. Having verified your api url and key, we can now create the agent.
az container create \
-g $boyd_rg \
--name prefect-agent \
--image index.docker.io/chaboy/private_test:latest \
--secure-environment-variables PREFECT_API_URL=$PREFECT_API_URL PREFECT_API_KEY=$PREFECT_API_KEY \
--registry-login-server 'index.docker.io' \
--registry-password '<REDACTED>' \
--registry-username '<REDACTED>' \
--command-line "/bin/bash -c 'prefect agent start -q aci-test'"

Alternatively we can create a file:

cat << EOF > container.yaml
apiVersion: 2018-10-01
location: eastus
name: prefect-agent
properties:
  containers:
  - name: prefect-agent
    properties:
      image: index.docker.io/chaboy/private_test:latest
      environmentVariables:
        - name: PREFECT_API_URL
          secureValue: https://api.prefect.cloud/api/accounts/<>/workspaces/<>
        - name: PREFECT_API_KEY
          secureValue: <>
        - name: PREFECT_API_ENABLE_HTTP2
          value: 'false'
      command: ["/bin/bash", "-c", "prefect agent start -q aci-test"]
      ports:
      - port: 80
      resources:
        requests:
          cpu: 1
          memoryInGB: 1.5
  osType: Linux
  restartPolicy: OnFailure
  imageRegistryCredentials:
  - server: index.docker.io
    username: <docker user>
    password: 'docker pass'
type: Microsoft.ContainerInstance/containerGroups

EOF

az container create -g $boyd_rg --file container.yaml

Process Based Deployment:

Create a deployment:

cat <<EOF > deployment.py
from os import environ, path
import sys

#Prefect 2
from prefect.deployments import Deployment

#Without a storage block for remote storage, the PATH of the flow, and the ENTRYPOINT are required to locate the flow in docker.
#Docker flow is at /opt/prefect/flows/healtcheck.py

deployment = Deployment(
    name="ProcessACI",
    version="latest",
    flow_name="healthcheck",
    work_queue_name="test-aci",
    path="/opt/prefect/flows",
    entrypoint="flow.py:healthcheck"
)


deployment.apply()
EOF

Azure Container Instance Job based Deployment

  1. Installing required packages locally:
    pip install prefect-azure
    prefect block register -m prefect_azure

  2. Navigate to create an Azure Container Instance Job in Prefect Cloud:
    Create Blocks → Azure → Azure Container Instance Job

  3. The following are required to configure your block:
    Minimum Required:
    name
    image → fully qualified registry.url/repository/image:tag
    Docker Registry Block (to pull the private image) - for docker, can be retrieved via docker info
    server: index.docker.io
    username:
    password:
    ACI Credentials Block (to create new container instances)
    ARM_CLIENT_ID
    ARM_CLIENT_SECRET
    ARM_TENANT_ID
    Azure Subscription ID
    Azure Resource Group Name

  4. Create the new Deployment:

cat << EOF > deployment2.py

from os import environ, path
import sys

#Prefect 2
from prefect.deployments import Deployment

from prefect_azure.container_instance import AzureContainerInstanceJob

#Replace with your block name
azure_container_instance_job_block = AzureContainerInstanceJob.load("test-aci-block2")

#Without a storage block for remote storage, the PATH of the flow, and the ENTRYPOINT are required to locate the flow in docker.
#The flow is at /opt/prefect/flows/flow.py from step 4 when we built the image

deployment = Deployment(
    name="InstanceJobACI",
    version="latest",
    flow_name="healthcheck",
    infrastructure=azure_container_instance_job_block,
    work_queue_name="aci-test",
    path="/opt/prefect/flows",
    entrypoint="flow.py:healthcheck"
)
deployment.apply()
  1. Run and apply the newest deployment:
    python deployment2.py

Troubleshooting ACI Issues:

2 Likes