Hello,
We have the same problematic in my company.
We are looking in a new orchestrator, and prefect2 seems like a very good candidate but the lack of a restart command is really annoying.
When you run a flow, you have entry parameters but also sub results that you reuse in other tasks.
I you recreate a full run from the begining these sub results can change (so also the behaviour and the result of the flow).
For example, with this sequential flow (on a microservice architecture) :
- API_logistic_send_package() => return $pkg_ref
- API_notif_send_mail_pkg_sent($pkg_ref)
- API_logistic_wait_delivery($pkg_ref)
- API_notif_send_mail_pkg_delivered($pkg_ref)
When you have an error on the task API_logistic_wait_delivery, your flow run is killed. But obviously we need to play the missing steps.
If we run a new flow from the beginning, we will resend a new package to the client (and the associated mail) which is not possible.
How are you debuging your complex flow runs when you have an error ?
We can try to hack the code to save somewhere the value of the pkg_ref with the flow run ref to allow to run a new flow with params set in entry (and the task skipped if pkg_ref is already set) but it’s really complicated and impratical (and I don’t imagine if you have 20 steps to manage…).
The caching method seems also tricky and it’s only working for crons.
A solution could be to allow to directly set the result of some tasks when you instantiate a new flow run. The restart would be just getting the result of the success tasks on a flow run in error and instanciate a new flow run with these values.
It would also be amazing to easily test/mock the deployments with bypassing some steps with a specific result !
Perhaps it’s already possible ?
Best regards,
Max