We have a couple of large jobs that run around 2-3 times per week on a predefined CRON expression and we want to schedule them in Prefect. We also use Render as our hosting service.
As far as I understand, currently, if you are using a PassS like Heroku or Render, there are two ways to achieve this on Prefect:
- Keep an instance running 24/7 in Render with an agent.
- Use a “parallel” CRON job in Render (it will wake up at the same time and start the agent which will process the flow)
The first option is obviously very costly and not efficient. Those jobs need a significant amount of memory, and keeping Render instances running 24/7 is pour money down the drain.
The second option is doable, but it’s super brittle and takes away a big part of the robustness that Prefect brings.
I know about the “serverless” AWS solution and read Prefect’s blog about it (this), but I am specifically talking about high-level PaaS like Heroku and Render.
Currently, we use Render (as Heroku is slowly dying ) and it would be amazing if we could define in a couple of clicks an integration to run Prefect CRON schedules on Render instances automatically.
I would happily help to write such an integration with Render if possible and try to make it happen. This will be super helpful for us!
Also, I strongly believe that if such an option will exist out-of-box, many (many) users will use this since currently there is just no simple and “codeless” way to run infrequent CRON jobs on Prefect.
There are a huge amount of devs (solo and teams) that use Heroku/Render (most of them moving to Render), that don’t want to use AWS and get their hand “dirty”.
Another option will be for Prefect’s cloud to actually provision such instances, but I believe it’s quite a big defocus for Prefect’s mission.
Would love to hear your thoughts about the issue, and again, would love to help and contribute regarding the Render integration.