Pausing Flow Group Schedule with GraphQL

Gotcha, thanks for explaining that in more detail. You asked basically two questions, one about the “catchup” and another one about the PAUSE signal - I tried to structure the response, hope it doesn’t seem overwhelming this way :smile:

1. Catchup issue

The problem with “catchup”

The issue with “catchup” is that you would need to manually create “missed” flow runs because Prefect never creates flow runs with the schedule being in the past because:

  • this pattern is generally used by legacy workflow orchestration solutions which are entirely schedule based, while Prefect decouples scheduling from flow run execution, allowing flow runs to be triggered e.g. event-based
  • for most use cases we’ve seen this leads to confusion and hard to debug issues.

This scheduler documentation explains it this way:

“If you pause a schedule, any future auto-scheduled runs that have not started will be deleted. Reactivating the schedule will cause them to be recreated, as long as they are scheduled to start in the future. The scheduler will never create runs that were scheduled to start in the past.”

Backfilling

But why would you need to create flow runs for the past? If you need it for backfilling, you may solve that by manually creating flow runs using this approach:

System-wide pause

If you use Prefect Cloud and you would be open to temporarily pause all work during your maintenance window, the easy solution would be to also hit the system-wide pause button (but this wouldn’t solve the issue of pausing specific flows and the catchup/backfill problem is also unaddressed this way):


Orion!

The good news for you is that in Orion your use case is that much easier! You may simply create a separate work queue for the respective flow tags (exactly what you do!) and then you may pause and resume the work queue when needed.

This documentation page explains it best:

2. PAUSE signal

I think you’re right that the PAUSE-resume pattern may be useful in your use case because it would allow your DBAs to manually resume once they are done with maintenance.

This page explains that more:

Regarding passing some string that can be retrieved in downstream tasks after the approval, you could leverage the KV Store if you are on Prefect Cloud or some other stateful mechanism if you are on Prefect Server (say S3 or Redis?):