I want to run Prefect flows using Docker containers deployed on different VMs through Docker Swarm, my infrastructure consists of the following:
- 1 container as Prefect Orion server
- 2 containers as Prefect Agents, picking up jobs from various queues
Each of these containers has its own
PREFECT_HOME where files are stored (i.e.,
storage/, etc.), but I may set up Docker volumes so that either:
PREFECT_HOMEpoints to a shared location where all files are seen by all containers (Orion or Agents alike)
- each container has its own
PREFECT_HOMEin different locations, without sharing any file
I looked at the Settings docs, but it didn’t clarify the best practice in the case of Docker containers deployed on different nodes and if having a shared file system is considered best practice or not.
I think it is part of a more general question: are all Prefect components supposed to share the same settings and storage locations, or are there cases where this shouldn’t be like this? Thanks for your help.