View in #prefect-community on Slack
@Daniil_Ponizov: Hi! is it possible to pass source of flow that stores in local or docker storage on one machine to agent on the other machine?
@Kevin_Kho: You can copy the file over but Prefect has no mechanism for that
@Daniil_Ponizov: @Kevin_Kho so the only one way to do that is use cloud storages like git or s3?
@Kevin_Kho: that’s the easiest but you can use Local Storage plus DockerRun if the flow files are in the Docker container
@Daniil_Ponizov: @Kevin_Kho and how agent on differrent machine will see the image from another server?
@Kevin_Kho: Ah I see what you mean. Yeah you’d need to upload the image to a registry
@Vincent_Chéry: Hi, I’d like to use Local Storage + DockerRun + Docker Agent to run my flows in containers. All my sources are in the image so that seems to correspond to the setup you are suggesting @Kevin_Kho.
But I’m not sure how to do that. Also the docs suggest Local Storage can only be used with Local Agent, so I’m a little confused.
At the moment I tried to do this :
from time import sleep
import prefect
from prefect import task, Flow
from prefect.agent.docker import DockerAgent
from prefect.run_configs import DockerRun
from prefect.schedules import CronSchedule
from prefect.storage.local import Local
@task
def ping():
logger = prefect.context.get("logger")
for n in range(20):
logger.info(f"Ping n°{n}")
sleep(3)
with Flow("dummy flow") as flow:
ping()
flow.schedule = CronSchedule("* * * * *")
flow.storage = Local()
flow.run_config = DockerRun(
image="http://ghcr.io/mtes-mct/monitorfish/monitorfish-pipeline:v1.6.4|ghcr.io/mtes-mct/monitorfish/monitorfish-pipeline:v1.6.4"
)
flow.register(project_name=PROJECT_NAME, build=False)
agent = DockerAgent(show_flow_logs=True)
agent.start()
In the server UI I see the flow that correctly gets registered, the Docker Agent queries the API, and flow runs are scheduled but never submitted nor executed.
Any hint would be welcome !
What I’m guessing is I would need to store my serialized flows in the .prefect/flows folder inside the image… Ideally I’d like have the serialized flows stored outside of the image, on the machine that runs the Docker Agent. When starting a container to run a flow, the .prefect/flows folder of the host machine would therefore need to be bind mounted inside the flow runner container. Is there any way to do that with Prefect ?
@Kevin_Kho: If you are using DockerRun + LocalStorage, the ideal is to do Local(stored_as_script=True, path =…) so you then point Prefect to the path inside the container. I have an example here
If you want to mount, yes you need to bind. Try mounting the volume on the agent side and I think that should be enough
@Vincent_Chéry: Hi, I forgot to reply but thanks, I got this working, exactly what I wanted !
In case an example cna help someone :
• Flows configuration : https://github.com/MTES-MCT/monitorfish/blob/master/datascience/src/pipeline/flows_config.py
• DockerFile to build the image that will run the flows : https://github.com/MTES-MCT/monitorfish/blob/master/infra/docker/Dockerfile.DataPipeline
• Run script to register flows and start the DockerAgent : https://github.com/MTES-MCT/monitorfish/blob/master/datascience/main.py