What's the role of agents and work queues, and how the concept of agents differ between Prefect 1.0 and 2.0?

Archive
, , , , ,
1

Work Queues and Agents

Work queues and agents bridge the Prefect Orion server’s orchestration environment with a user’s execution environment. Work queues define the work to be done and agents poll a specific work queue for new work.

More specifically:

  • You create a work queue on the server. Work queues collect scheduled runs for deployments that match their filter criteria.
  • You run an agent in the execution environment. Agents poll a specific work queue for new work, take scheduled work from the server, and deploy it for execution.

To run orchestrated deployments, you must configure at least one work queue and agent.

To configure a work queue and agent for orchestrated deployments:

  1. Create a work queue
  2. Start an agent

Differences between agents in Prefect 1.0 and 2.0

The role of agents has changed from their implementation in Prefect 1.0. It requires one more step in the setup but offers much greater control and flexibility with how deployments are executed.

Key changes:

  • Work queues contain all the logic about what flows run and how. Agents pick up work from queues and execute the flows.
  • There is no global agent that picks up orchestrated work by default. You must configure a work queue and agent.
  • In Prefect 1.0, agents are required to see the flow run history in the UI.
  • In Prefect 2.0, agents and work queues are only required if you want to trigger flows via UI or API.

Related discussion

View in #prefect-server on Slack

Egil_Bugge @Egil_Bugge: Hey all!

Tested out Prefect Orion in the cloud yesterday and had the following scenario:
• Made a simple hello world flow with a deployment spec

@flow
def hello_world(name="world"):
    print(f"Hello {name}!")

DeploymentSpec(
    flow = hello_world,
    name="hello-world",
)

• Made a work queue and connected the work queue to this deployment
• Span up an agent locally and connected it to this work queue
• Clicked into the Deployments tab in the UI and selected “Quick run” which scheduled a run of my flow
Then nothing happened, my local agent didn’t pick up the flow. I then made a new work queue which I did not connect to the deployment and connected my agent to this new work queue instead. Now it immediately picked up the flow that I had started earlier and ran it.

Is there something more I need to do to get an agent to pick up flows connected to a deployment?

@Anna_Geller: If you are switching from Prefect 1.0 agents to Prefect 2.0 work-queues and agents, a good mental model would be treating a work-queue as a way to assign a label to your agent. For a local agent, you could assign a tag called local:

prefect work-queue create local -t local
prefect agent start QUEUE_ID

and then, on your DeploymentSpec use the same tag local:

DeploymentSpec(
    flow=hello_world,
    name="sandbox",
    tags=["local"]
)

Egil_Bugge @Egil_Bugge: Ah, okey! Thank you :slight_smile:

Related topics