Deployment to Prefect 3 with Minio storage

Hi All,

Thank you for being that generous with your time and work.

I’m migrating from prefect 2 to prefect 3 and I need to switch to the new deployment mechanism. Unfortunately it doesn’t work as I expect.

The old deployment was:

Deployment.build_from_flow(
    name="...",
    flow=my_flow,
    parameters={..},
    storage=RemoteFileSystem.load('minio'),
    path='...',
    version=os.getenv("GIT_COMMIT_SHA"),
    infrastructure=DockerContainer(
        auto_remove=True,
        image='prefect-default-worker:2.20.9',
        image_pull_policy='NEVER',
        network_mode='bridge',
    ),
).apply()

Where ‘minio’ is a Remote File System block containing Minio connection data, prefect-default-worker:2.20.9 is an image based on prefecthq/prefect:2.20.9-python3.10 with all the flow’s dependencies installed. I have another container prefect-agent:2.20.9 with entrypoint: ["prefect", "agent", "start", "-q", "default"]

Reading the migration guide I get that I need to 1) Replace ‘infrastructure’ with ‘work_pool_name’ and 2) Replace ‘storage’ with a call to ‘flow.from_source’ 3) replace entrypoint: ["prefect", "agent", "start", "-q", "default"] with entrypoint: ["prefect", "worker", "start", "-p", "default"]

The new deployment script looks like this:

my_flow.from_source(
    source=RemoteFileSystem.load("minio"),
    entrypoint='path to the file with @flow definition'
).deploy(
    name="same",
    work_pool_name='default',
    parameters={same},
    version=os.getenv("GIT_COMMIT_SHA"),
    build=False,
    push=False,
    job_variables=dict(pull_policy="NEVER", network_mode="bridge", auto_remove=True)
)

Before the migration, the script uploads the flow’s python file (and the whole folder) to minio and then create/update the deployment.

Now it doesn’t seem to do the upload to Minio. The script throws an error: FileNotFoundError: [Errno 2] No such file or directory: ‘/var/folders/l9/q51p6spx7sb6j3gqdq5q7jhr0000gn/T/tmpi8gx9s0t/remote-file-system-minio/…’

If I manually put the files in Minio manually It seems to load the flow. I can see that a ScriptError is raised loading the flow’s python file. I’ll share the error after I clarify this issue.

So I wonder if I can keep the previous behaviour, or I should upload the flow’s definition to Minio manually.