View in #prefect-community on Slack
@Marwan_Sarieddine: Hi folks, question about the lazarus process. Why would lazarus try to reschedule a flow run if it reaches a successful state ?
please see a screenshot of our flow run’s logs
We are running on prefect cloud, with a kubernetes agent and dask executor on EKS
@Anna_Geller: Could you DM me the flow run ID of this flow so that I can cross-check? In general, Lazarus is typically only rescheduling flow runs in case of some infrastructure issues (e.g. if your pod crashed). This is indeed a bit puzzling given that the flow run seems to have been successfully completed.
@Marwan_Sarieddine: (sent) thank you for taking the time to look into this
@Anna_Geller: as a workaround, you could disable Lazarus for this flow as described here
Flows | Prefect Docs
@Marwan_Sarieddine: yes sure
I should note that this is a rare occurrence, this flow run has been running successfully without being rescheduled by lazarus
@Anna_Geller: Gotcha. Could it be that there are some unclosed DB connections or other resources like HTTP clients that you use in your flow? I saw a similar issue occurring due to resources failing to close/shut down.
I’m also wondering if the concurrency limits you use may be related to this.
I don’t see any clear indicators why this may be happening and would open a Github issue for now and suggest to disable Lazarus for this process as a workaround until then
@Marwan_Sarieddine: > Could it be that there are some unclosed DB connections or other resources like HTTP clients that you use in your flow
there are no DB connections, just a call to the prefect KV store to add a key and then remove it
> I’m also wondering if the concurrency limits you use may be related to this.
there are no concurrency limits set