In prefect 1.0 we use the Vertex Agent to achieve this. Other than the persistent agent looking for new tasks sitting in a tiny VM, it scales to a very low cost steady state.
In prefect 2.0 I don’t see vertex support. Any advice on how to set things up so when our (infrequent!) jobs hit the queue, a new docker vm can be provisioned on GCS, run, and shut down when it is done?
(context we do video processing jobs (ffmpeg, x264) and sometimes it’s just one a month and sometimes it’s 200, and the VMs we need are heavy, so we don’t want to leave them running.)