Prefect 2 setup for a use case w/ few and long running flows

Help
, ,
1

The setup/configuration of agents, queues, etc. in Prefect 2 seems quite flexible. I imagine there’s optimal configurations for certain use cases. I wonder what that optimal configuration is for our use case - perhaps others have a similar use case.

We have a relatively small number of flows (~30; at least I imagine that’s relatively small), but many of the flows run for many hours (e.g., 7 hrs).

Clearly we don’t want the long running flows to fail due to some sort of exception related to the flow running for such a long period.

Is there some combination of number of agents, queues, etc. that will best support this use case?

The setup we’re currently using is 3 agents running on a single large VM, with each agent listening to 4 queues each. I can’t speak to how this performs under load because we haven’t completely moved from Prefect 1 yet.

1 Like
2

actually no, it’s very open-ended and up to you to figure out how you would like it to work

we generally try to provide building blocks so that you can build things exactly as you want

your setup looks good :+1:

3

Thanks, but I’m guessing you’ve seen what works for this kind of use case and could offer some suggestion?

4

sure, the ideal architecture is one that is as simple as possible: if you could make it work with just one agent and work queue, that would be ideal because you have fewer moving parts to manage