How to use prefect for load balancing with dynamic parameters

Archive
1

Hi, I don’t know much about prefect but we are using it in our project to run ETL pipelines. Currently we have an algorithm that needs to be run by changing variables A and B, each of which has 3 and 17 different settings respectively for a total of 51 different combinations. Currently we are instantiating some virtual machines and registering N combinations in each machine, for example: VM_1 have A=1 and B=1,2,3,4 ; then this VM will run algorithm A for various B parameters. Then we just register one agent in the prefect and have it run this parameter combination every 1 hour for example and this has been working so far.

But I wonder if there is a smarter way to use prefect, so that instead of having to register each combination on each machine we could make a list of parameters that will be run and prefect would dynamically allocate the resources between the VMs that we have running the prefect agent. It seems that we are using this tool only as a remote cron that allows you to monitor the state of the flows.

For example, use the prefect dashboard or api to pass a list:[ (A=1, B=1), (A=1, B=2), (A=2, B=1), (A=2, B=2),… and the prefect dynamically executes (A=1, B=1) in VM_3, (A=1, B=2) in VM_1 and so on.

1 Like
2

wow, at a first glance it looks like you are looking more for Kubernetes rather than Prefect? allocating work to machines in a resource-aware way sounds like a great task for Kubernetes

you could leverage work queues to do the same by setting up an agent on each of those VMs (e.g. prefect agent start -q vm1) and creating deployments pointing to each of those queues (e.g. prefect deployment build myflow.py:myflow -n mydeploy -q vm1 -sb github/yourblock)

regarding config file and avoiding boilerplate, check out this post and the related deploy scripts on the repo for that post:

3

Thanks for the tip!
Would it be possible to deploy and schedule a flow with custom parameters through Prefect UI? So for example: i got two VM running agents with the same “-q VM” and the user can deploy multiple flows with custom parameters in the prefect UI in that queue or something like that.

1 Like
4

Yes, that’s possible. You could either create multiple (possibly scheduled) deployments for the same flow, but using different parameter values, or you can trigger a custom ad hoc run from the UI with custom parameter values, or run deployment from the UI

all of those methods are explained in this tutorial series:

1 Like