What are various ways to introduce a delay between flow and task runs?

View in #prefect-community on Slack

Matt_Yamkowy @Matt_Yamkowy: Hi. I’m just starting to look at Prefect 2.0. Is there a suggested way of handling a waiting period mid flow? For example if I start a flow, and after an initial task I want to wait for 5 minutes before continuing, do I just sleep for 5 minutes in a task? What about if I wanted to wait for days?

Kevin_Kho @Kevin_Kho: You could do that with a sleep yeah. What is the use case for waiting days? Because maybe at that point to can trigger a sub flow scheduled for a later time

Matt_Yamkowy @Matt_Yamkowy: In some cases we would detect an alarm, then start gathering data, days later we’d check if the gathered data met certain criteria before continuing with tasks. So are you saying I could dynamically schedule a follow up flow to check on my gathered data?

Kevin_Kho @Kevin_Kho: Ah I guess not yet. We do have Client.create_flow_run_from_deployment here but I guess it doesn’t take in a scheduled start time yet. The equivalent in Prefect 1 does

Matt_Yamkowy @Matt_Yamkowy: Hmm, can you suggest a way to do this? or will it be coming as 2.0 comes out of beta?

Kevin_Kho @Kevin_Kho: it will be coming, for now though you’d either need to poll or use Prefect 1.0. I did leave an internal message to confirm and i’ll get back to you

Matt_Yamkowy @Matt_Yamkowy: Thanks, I appreciate it!

Andrew_Huang @Andrew_Huang: Or maybe I was wondering if you could set up a scheduled deployment every hour to check if there’s new data; if not, stop the flow run prematurely and recheck in another hour? to prevent clobbering, could also set concurrency limit = 1 Schedules - Prefect 2.0

Kevin_Kho @Kevin_Kho: I think at that point, you can just use the task level retry to achieve it can it will be simpler

Matt_Yamkowy @Matt_Yamkowy: It could be possible to setup a schedule to check if any jobs are complete, but we could potentially have a hundreds of different flows so we’d have to plan it out well. I was hoping prefect would have something built in to handle it.
I know temporal can do async sleep built in, I was hoping there was something similar in prefect.

Kevin_Kho @Kevin_Kho: Not exactly the same issue, but Add start and end dates to schedules · Issue #5620 · PrefectHQ/prefect · GitHub is heavily related. Definitely in the list of things to do for Prefect 2.0

Matt_Yamkowy @Matt_Yamkowy: This is great.