Change where data from an S3 block is copied to for execution

Help
, , ,
1

Consider a flow being deployed as follows:

deployment = Deployment.build_from_flow(
        flow = myflow,
        name = "my-deployment",
        skip_upload = False,
        work_queue_name = "special-queue",
        work_pool_name = "default-agent-pool",
        storage = S3.load("my-s3-block"),
        path = "",
        entrypoint = f"deploy.py:myflow",
        infrastructure = Process(
            env = {"user": "prefect"}
        )
    )
run_deployment(deployment.apply())

From my understanding, when the deployment is built and run, the folder containing the code will be uploaded to my S3 bucket, and then pulled to a /tmp/blabla folder and executed. Is this correct? If so, how can we execute from a different folder? The goal would be to have different flows, running at different times, but always executing from the same working directory, which would be a great convenience for persisting the results of a flow without having to coordinate pulling and pushing from S3 all the time.

Question: How can I change the location where my files are pulled to (from S3) and then executed by Prefect? I basically don’t want my code to run from /tmp/blabla when using an S3 storage block.

Failed Attempts:
I tried updating working_dir in the infrastructure Process:

...
infrastructure = Process(
            working_dir = "~/not_tmp/",
            env = {"user": "prefect"}
        )

However, Prefect crashed and I couldn’t find any logs to debug.

Thanks for helping!!!

2 Likes
2

Seems like you need to create a local storage block and copy to it. or set it as a working dir (not sure if possible)