Related Slack Discussion about HA setup
View in #prefect-community on Slack
@Tom_Manterfield: Hey everyone! Can anyone tell me if the Orion API and/or Scheduler and/or UI support running multiple replicas? Is anyone running them in an HA config already?
@Anna_Geller: Great question! One engineer from our team thinks about this problem a lot and we may address this in the second half of the year, but due to the complexity of doing it well it’s not on an immediate roadmap
in general, the OSS Prefect 2.0 orchestration layer will be easier to scale than e.g. Prefect 1.0 due to simpler architecture - you should be able to deploy multiple API servers talking to the same Postgres backend and load balance the incoming requests
We definitely welcome your input if you are willing to contribute and share your solution with the community
@Tom_Manterfield: > You should be able to deploy multiple API servers talking to the same Postgres backend and load balance the incoming requests
To clarify, is this saying you can do that right now and it’s just the scheduler that needs work, or are you describing the target state for later development?
because if the API is already good then it solves my need, the scheduler I feel like I can get by with vertical scaling for quite a while.
@Anna_Geller: I don’t want to give you a wrong answer, but generally speaking, a scheduler service is part of the Orion API server and you could in theory (to achieve HA) have multiple uvicorn servers and load balance the requests
what I meant with complexity is e.g. ensuring how those multiple replicas work together e.g. I know that we use idempotency keys to avoid duplicate work (the same run being picked up twice) but I don’t know whether this would be enough to prevent duplicate work when running multiple Orion servers
This is definitely a hard problem (as anything distributed/HA) and my main point is that I don’t want to give you a sense that you can just deploy multiple replicas, add a load balancer on top and you’re done, there is a lurking complexity here to ensure it all works well together
> I feel like I can get by with vertical scaling for quite a while.
you’re right that vertical scaling seems to be the best solution right now - especially ensuring that your database scales and performs well because this is the only stateful component
@Tom_Manterfield: Okay, from the sounds of things you’ve at least baked in a concept of idempotency. I do appreciate the caution, definitely better than overconfidence in this area. I might just give it a shot in dev for a bit and see what happens.
I was under the impression I could run the scheduler and the API separately though. I saw the --no-scheduler and --scheduler flags and assumed I could run Orion as the API only and then one with the scheduler only.
@Anna_Geller: Indeed, it only shows that you already know more about it than I do 
it seems you can run a separate Orion server without UI, scheduler, mark late runs and analytics services
if you give it a try, could you share your work somehow e.g. via a public repo or even a public Gist with README? would be a great way to push the topic forward and continue iterating on that together, I’m sure others from the community would benefit from that too and could use it to contribute as well
@Tom_Manterfield: Anything I make that’s shareable I definitely will. Once I’ve got my current setup working nicely and proven in a prod like environment then I’m planning to wrap most of it in a Kubernetes operator and open source it.
Don’t want to do that until I know it actually works though as I don’t want to push out rubbish and have some poor unsuspecting dev pick it up.
@Michael_Adkins: All of the services can be run separately from the API to scale them individually
I think your biggest issue in a setup with replicas will be Postgres database connection limits
@Tom_Manterfield: Good info, thanks. I guess I can pool them with pgbouncer to ease that. I’m more comfortable having that problem than I am having something I can’t allow to scale.
@Michael_Adkins Is it possible to run the scheduler without the API? I can’t actually see how that one would be done. I can run the API without anything else, but I can’t see a way to do an equivalent of prefect orion start --scheduler-only
Seems like I’d have to run one API standalone and then one with the scheduler, and just scale the standalone one for API traffic and accept the scheduler always comes with an API attached. I can live with that if so, but thought I’d check if I missed something.
@Michael_Adkins: You can call python -m "prefect.orion.services.scheduler"
@Tom_Manterfield: Oh cool! Thanks!
@Michael_Adkins: And similarly for any of the other services. I don’t think logging will be set up correctly yet though as we’ve not polished this.
should serve as your main source about the Prefect orchestration layer explaining all components and how they work together:
documentation.
