Scaling docker flow runners from 0 to N and back? (Google Cloud preferred)

In prefect 1.0 we use the Vertex Agent to achieve this. Other than the persistent agent looking for new tasks sitting in a tiny VM, it scales to a very low cost steady state.

In prefect 2.0 I don’t see vertex support. Any advice on how to set things up so when our (infrequent!) jobs hit the queue, a new docker vm can be provisioned on GCS, run, and shut down when it is done?

(context we do video processing jobs (ffmpeg, x264) and sometimes it’s just one a month and sometimes it’s 200, and the VMs we need are heavy, so we don’t want to leave them running.)

@acgourley, I believe the easiest solution to the VM problem would be to start an agent automatically on a VM boot so that when you want to resume your flows, the agent processes are running.

VM set up

This part of the Azure tutorial addresses the issue (works the same way on any Ubuntu VM incl. GCP VM):

Quote from the shared topic:

Run this single line in your terminal:

echo "@reboot root supervisord -c /home/azureuser/supervisord.conf -l /home/azureuser/supervisord.log -u root" >> /etc/crontab

This line will add a crontab that starts your supervisor process every time your VM boots up.

Pause schedule

For the flow itself, you could pause the schedule once you want to put everything on hold and stop the VM.