The following diagram helps illustrate the relationships between different objects, including the underlying schema (at the time of writing).
Note that the
manifest_pathis marked red since this argument is no longer used, and it’s maintained in the schema only for backward compatibility with Prefect ≤ 2.0.3.
Deployments and work queues
Deployments and work queues have a one-to-many relationship.
- Any given deployment can have only one work queue name assigned (e.g.
-q dev). - Any given work queue can simultaneously pick up multiple deployments.
Agents and work queues
Agents and work queues have a many-to-many relationship. This allows for a highly customized agent configuration.
- Any given agent can poll from one or more work queues
- Any given work queue can simultaneously be polled for flow runs by multiple agents.
Even though the mapping of agents to work queues is so flexible, there are two constraints that guarantee reliability:
-
the constraint that any given deployment may belong to only one work queue makes it easy to understand which work queue will pick up any given flow run from a deployment,
-
the idempotency key on each flow run guarantees that even if multiple agents poll for runs from the same work queue, each run can be picked up only once by a single agent.
Examples of agent and work-queue configurations
prefect agent start -q etl= a single agent polling from a single work queueprefect agent start -q etl -q ml= a single agent polling simultaneously from the “etl” and “ml” work queuesprefect agent start -q etl= agent #1 and alsoprefect agent start -q etl -q ml= agent #2; here, the work queue “etl” is simultaneously polled for flow runs by two agent processes