I’ve seen these terms used in different places but can’t figure out the relationship. Are they the same thing?
No they are not the same. Work pools contain work queues and they can be configured to add concurrency limits on how many flow runs are supposed to run in that pool.
Work queues are the actual Prefect workers/agents that do the work on whatever machine/VM/technology they have been deployed. Hope it helps clarify the different terms a bit.
Below you may find attached a screenshot from the default Prefect work pool and that it contains a tab titled work queues:
2 Likes
Am I right for understanding this as Agent = an abstraction of the Work Queue.
So the Work Pool is akin to an “Agent Pool”?
hi @duckdatum
Am I right for understanding this as Agent = an abstraction of the Work Queue .
So the Work Pool is akin to an “Agent Pool”?
workers are the new agents. you assign them to a work pool (a specific queue if you want, but no need) and let them pick up flow runs that get sent to the work pool from deployments.
The way I like to think about a work pool, is twofold:
- its a group of work queues that deployments send flow runs to when they should be executed
- its the place that a worker looks to find details how it should submit the flow run (e.g. what env should it have? what image should I pull if its docker based? etc). you can also add extra details to the deployment itself, which will override any defaults on the work pool
usually, you’ll have 1 work pool for each unique kind of execution environment - for example in my workspace, I have:
- 1 process work pool for testing deployments locally
- 1 docker work pool for simulating containerized deployments locally on docker desktop
- 1 kubernetes work pool for my “prod” deployments
One last thing, is that if you happen to want just the easiest way of running a flow as a container without worrying about running a worker, you can use something like the Cloud Run Push Pool, which handles running the worker process for you when its time for a flow run.