TL;DR: set flow.reference_tasks=[your_important_mapped_task] and ensure that you pass upstream tasks with the unmapped keyword when used with a mapped task. Otherwise, the flow run state is determined by the state of the last task run.
[View in #prefect-community on Slack]unm(https://slack.com/archives/CL09KU1K7/p1650467978229159)
@Constantino_Schillebeeckx: I have a flow that runs 121 mapped tasks - two of those tasks failed however the overall flow was nor marked as failed
Both tasks failed because:
> No heartbeat detected from the remote task; marking the run as failed.
@Kevin_Kho: So for the heartbeat, it might be more stable with threads. Have you done that?
With the flow marked as failed, are these mapped tasks the last tasks in your flow? The final state is determined by the terminal_tasks so you can explicit make them terminal tasks by doing:
flow.terminal_tasks = [...]
Frequently Asked Questions | Prefect Docs
@Constantino_Schillebeeckx: no, these tasks run in the middle of the flow
in general though, even if the task fails, shouldn’t the flow be marked as failed?
@Kevin_Kho: not really unless you explicitly specify. for example if you have a flow with A->B->C and B fails but you force C to run and it’s successful, I think this is considered a success by default because C is the terminal task in the graph
normally failure would propagate and fail the last task as well but in the event it doesn’t, you have to be explicit about what task makes the flow fail
@Constantino_Schillebeeckx: i see - i noticed in this particular case, the failed tasks didn’t report failure until the all other tasks had run in the flow (which confuses me) - so technically the terminal task was a success
to be a bit more explicit, i have mapped tasks L1, Ln and then I have a task do_dbt which is a downstream dependencies of all the mapped tasks L- one of those mapped tasks stopped reporting a heartbeat back and eventually do_dbt just started running and passed. then Ln was finally set as failed
@Kevin_Kho: it sounds like the dependency was not set right. how did you make t a downstream?
@Constantino_Schillebeeckx:
@Kevin_Kho: That looks perfectly right though. I would try setting
flow.terminal_tasks = [L]
but I’m not 100% sure on the mapped task behavior
@Constantino_Schillebeeckx: but L aren’t terminal tasks, and do-dbt should only run after all of L is finished
is it bad form to set all tasks as terminal tasks to ensure this behavior doesn’t happen?
@Kevin_Kho: My bad not terminal task
flow.reference_tasks=[…]
@Constantino_Schillebeeckx: i see - that does seem like a valid workaround, i’m curious though why do_dbt would run if one of the mapped L tasks has a missing heartbeat
@Kevin_Kho: Yeah I am not sure about that at the moment either
@Constantino_Schillebeeckx: I’ve just bumped up my container resources and it looks like i’m not getting heartbeat issues anymore (thank you docs!) - i’m wondering if there’s a bug for up/down stream task dependencies when this heartbeat issue comes up
@Kevin_Kho: Potentially, I can see that.