Scaling docker flow runners from 0 to N and back? (Google Cloud preferred)

[acgourley]

In prefect 1.0 we use the Vertex Agent to achieve this. Other than the persistent agent looking for new tasks sitting in a tiny VM, it scales to a very low cost steady state.

In prefect 2.0 I don’t see vertex support. Any advice on how to set things up so when our (infrequent!) jobs hit the queue, a new docker vm can be provisioned on GCS, run, and shut down when it is done?

(context we do video processing jobs (ffmpeg, x264) and sometimes it’s just one a month and sometimes it’s 200, and the VMs we need are heavy, so we don’t want to leave them running.)

[anna_geller]

@acgourley, I believe the easiest solution to the VM problem would be to start an agent automatically on a VM boot so that when you want to resume your flows, the agent processes are running.

VM set up

This part of the Azure tutorial addresses the issue (works the same way on any Ubuntu VM incl. GCP VM):

How to spin up a Docker agent on Azure VM - a full walkthrough

Create an Azure Resource group To start in a clean new environment, we’ll first create a new resource group. You can do that either from the CLI or from the Azure portal. Using Azure CLI: az group create --name PrefectDockerAgent --location eastus From the Azure portal: [image] [image] Click Review and create and then confirm once again: [image] [image] Create an Azure VM Within the resource group, click “Create”: [image] Then, select Ubuntu Server 20.04 LTS. If you don’t see it …

Quote from the shared topic:

Run this single line in your terminal:

echo "@reboot root supervisord -c /home/azureuser/supervisord.conf -l /home/azureuser/supervisord.log -u root" >> /etc/crontab

This line will add a crontab that starts your supervisor process every time your VM boots up.

Pause schedule

For the flow itself, you could pause the schedule once you want to put everything on hold and stop the VM.