Flows not resuming from point of failure

We have been running Prefect flows for months and the latest release we have things running off of for the long-term is 2.14.17.

We have configured our Prefect settings and enabled the PREFECT_RESULTS_PERSIST_BY_DEFAULT=true

This has allowed us to resume flows from tasks that have failed. For a simple example, if we have a linear flow of
Task A → Task B

If Task A succeeds but Task B fails, if we go into the UI and hit “Retry”, we are picking up directly on Task B rather than running Task A again, which is perfect.

We recently upgraded our test Prefect environment to 2.19.1 to test new features, and after upgrading, the restart capability no longer works. All flows will restart from the top of the flow, rather than from the failed task within the flow. Meaning, instead of picking up at failed Task B in my example above, we now re-run Task A before getting to B again.

No settings have changed on our side other than updating the package. We’ve verified everything is still matching our Production environment where the restart works. Nothing has changed in our code/flow/task setup. But for some reason, it’s like the results are not persisting, or Prefect isn’t properly scanning the results and recognizing the successful tasks that can be passed over.

We have tried forcing the results to persist even at the task/flow level by adding those flags in the task/flow decorators, and the flows will still not resume from the failed task. It’s always back to the start of the flow.

Not sure if functionality changed somehow and we need to return Result states or anything like that in our Tasks now. Just looking for some help if anyone has any so we can get the newest features while still being able to resume flows from the point of failure.

We did some more digging and it looks like the issue with the Restarts no longer properly functioning begins in version 2.16.5

In version 2.16.4, we are still able to have flows resume from the point of failure with the PREFECT_RESULTS_PERSIST_BY_DEFAULT=True

Hi, were you able to find the reason for this? Facing the same issue on 2.20.4 and would love to be able to resume and only run tasks that failed.