What is the relationship between an agent, work queue and a deployment?

The following diagram helps illustrate the relationships between different objects, including the underlying schema (at the time of writing).

:bulb: Note that the manifest_path is marked red since this argument is no longer used, and it’s maintained in the schema only for backward compatibility with Prefect ≤ 2.0.3.

Deployments and work queues

Deployments and work queues have a one-to-many relationship.

  • Any given deployment can have only one work queue name assigned (e.g. -q dev ).
  • Any given work queue can simultaneously pick up multiple deployments.

Agents and work queues

Agents and work queues have a many-to-many relationship. This allows for a highly customized agent configuration.

  • Any given agent can poll from one or more work queues
  • Any given work queue can simultaneously be polled for flow runs by multiple agents.

Even though the mapping of agents to work queues is so flexible, there are two constraints that guarantee reliability:

  • the constraint that any given deployment may belong to only one work queue makes it easy to understand which work queue will pick up any given flow run from a deployment,

  • the idempotency key on each flow run guarantees that even if multiple agents poll for runs from the same work queue, each run can be picked up only once by a single agent.

Examples of agent and work-queue configurations

  1. prefect agent start -q etl = a single agent polling from a single work queue
  2. prefect agent start -q etl -q ml = a single agent polling simultaneously from the “etl” and “ml” work queues
  3. prefect agent start -q etl = agent #1 and also prefect agent start -q etl -q ml = agent #2; here, the work queue “etl” is simultaneously polled for flow runs by two agent processes

Agents and work queues FAQ