When flows are stuck in a Scheduled state, it’s usually due to a label mismatch between the Flow and agent.
Labels are the mechanism that dictates which Flows an agent can pick up. If there is no agent that can pick up a flow, the flow remains stuck in a Scheduled state.
Tips for debugging:
- Check that the Flow labels are a subset of the Agent labels, i.e. your agent can have more labels but all labels that you set on your Flow’s run configuration must exist on the agent in order to be picked up by this agent. For more information, check out Run Configuration | Prefect Docs
- Check that there is a healthy agent that can pick up the Flow.
- Make sure that the RunConfig takes in a
List[str]
as opposed to just a string. Settinglabels="prod"
will be treated as four labels(["p", "r", "o", "d"])
because strings are iterable in Python. - Flows with no labels are picked up by agents with no labels only.
- There is a default label on the LocalAgent that contains the hostname.
- There is a default label on the default Local Storage.
- Check if there is a Flow Concurrency Limit preventing the Flow from being picked up.
There is an edge case where the scheduler gets overwhelmed if there are more than 750 late runs in a tenant. In this case, the late runs need to be cleared for more work to be executed.
See also:
If your flow is instead stuck in a Submitted state, check out this thread: