In prefect 1.0 we use the Vertex Agent to achieve this. Other than the persistent agent looking for new tasks sitting in a tiny VM, it scales to a very low cost steady state.
In prefect 2.0 I don’t see vertex support. Any advice on how to set things up so when our (infrequent!) jobs hit the queue, a new docker vm can be provisioned on GCS, run, and shut down when it is done?
(context we do video processing jobs (ffmpeg, x264) and sometimes it’s just one a month and sometimes it’s 200, and the VMs we need are heavy, so we don’t want to leave them running.)
1 Like
@acgourley, I believe the easiest solution to the VM problem would be to start an agent automatically on a VM boot so that when you want to resume your flows, the agent processes are running.
VM set up
This part of the Azure tutorial addresses the issue (works the same way on any Ubuntu VM incl. GCP VM):
https://discourse.prefect.io/t/how-to-spin-up-a-docker-agent-on-azure-vm-a-full-walkthrough/407#ensure-agent-restarts-on-a-vm-reboot-10
Quote from the shared topic:
Run this single line in your terminal:
echo "@reboot root supervisord -c /home/azureuser/supervisord.conf -l /home/azureuser/supervisord.log -u root" >> /etc/crontab
This line will add a crontab that starts your supervisor process every time your VM boots up.
Pause schedule
For the flow itself, you could pause the schedule once you want to put everything on hold and stop the VM.
Thanks. The solution of stopping and starting a VM isn’t ideal for us because sometimes we do need to scale up to 200 instances. If we did that manually, turning them all off when done would be labor-intensive. Maybe if I figured out a “self terminating” VM which, when it noticed it’s CPU usage went down, fully delete itself off the cloud. That’s a bit strange, though.
Luckily, I noticed VertexAI support is coming to prefect2 GitHub - PrefectHQ/prefect-gcp: Prefect integrations with Google Cloud Platform. (although it’s still experimental)