Locking Mechanism used by the Agents to pick up and execute the Flow from Work Queue

Help
, , ,
1

In this documentation about the agent, the following is mentioned-

It is possible for multiple agent processes to be started for a single work pool. Each agent process sends a unique ID to the server to help disambiguate themselves and let users know how many agents are active.

This means that multiple agents can listen to or poll to the same queue in a work pool. I am curious to know, how Prefect avoids the race condition where two agents can pick up the same Flow present in the queue.

It must be using some sort of locking mechanism to lock the flow once an agent has picked it up in order to prevent another agent from picking up the same flow. I don’t see any Redis or Zookeeper dependency, so will someone please explain to me what is going on behind the scenes over here?

2

Hey Pratik,

Thanks for the question. Usually we do recommend one agent per work queue. You can definitely use multiple agents for a single work queue or single work pool with one queue, but I would say there are better patterns you can implement partially due to the reasons you noted above. Both these two articles explain more about this front.

3

Hi @Matt_Conger , thanks for the reply.
I went through the links and found out this link is no longer working https://discourse.prefect.io/t/what-happens-when-agent-picks-up-a-deployed-flow-run-from-a-work-queue/1464

While going through this link I found out the below statement and I am still confused and curious about how does the Prefect handle the case where there are multiple agents polling the same work queue.

  • the idempotency key on each flow run guarantees that even if multiple agents poll for runs from the same work queue, each run can be picked up only once by a single agent.

I didn't get an exact answer to my question in the above links and still have these two questions-

  1. Let’s say we have a flow run (say Flow_Run_A) in the work queue and there are two Agents (Agent A and Agent B) polling this work queue. Will there be a case where both Agents A and B pick up the Flow_Run_A and execute it?

  2. In the above scenario, if only one Agent is allowed to pick up the Flow_Run_A, then how does Prefect manages this race condition? What locking mechanism or semaphore does the Prefect use so that multiple agents do not pick up the same flow run?