Path Option Not Working in Deployment with GitHub Storage Block

Help
, , , , ,
1

I can’t seem to get a flow to run on a deployment with a GitHub storage block that has a sub-directory path specified. It works fine if the flow folder is in the root directory of my GitHub repo. I’ve reviewed the documentation, watched videos on Prefect’s YouTube channel, and searched for similar issues reported by other users, but haven’t found a solution. Any help would be greatly appreciated.

My flow is stored in GitHub on a private repo within an organization:

https://github.com/My-Org/My-Repo/proj1/flows/my_flow.py

I created a GitHub storage block with a fine-grained personal access token:

Block Name: my-github-sb
Reference: main
Repository: https://github.com/My-Org/My-Repo.git

I successfully build and apply the deployment in a virtual environment from the following working directory:

(prefect-venv) PS C:/Users/Me/Documents/GitHub/My-Repo/proj1>

with the following CLI command:

prefect deployment build flows/my_flow.py:my_function -n my-deployment -sb github/my-github-sb --path proj1 -a

I’ve also tried:

prefect deployment build flows/my_flow.py:my_function -n my-deployment -sb github/my-github-sb/proj1 -a

After starting the agent and triggering the flow from the Prefect Cloud UI, I get this output:

16:00:04.015 | INFO    | prefect.agent - Submitting flow run 'b9ecb2dc-3362-4a81-a348-8563ede1da42'
16:00:05.001 | INFO    | prefect.infrastructure.process - Opening process 'wine-bittern'...
16:00:05.293 | INFO    | prefect.agent - Completed submission of flow run 'b9ecb2dc-3362-4a81-a348-8563ede1da42
16:00:16.164 | INFO    | Flow run 'wine-bittern' - Downloading flow code from storage at 'proj1'
16:00:18.075 | ERROR   | Flow run 'wine-bittern' - Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "<frozen importlib._bootstrap_external>", line 846, in exec_module
  File "<frozen importlib._bootstrap_external>", line 982, in get_code
  File "<frozen importlib._bootstrap_external>", line 1039, in get_data
FileNotFoundError: [Errno 2] No such file or directory: 'flows\\my_flow.py'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Users\Me\Documents\GitHub\My-Repo\venv\prefect\lib\site-packages\prefect\engine.py", line 294, in retrieve_flow_then_begin_flow_run
    flow = await load_flow_from_flow_run(flow_run, client=client)
  File C:\Users\Me\Documents\GitHub\My-Repo\venv\prefect\lib\site-packages\prefect\client\utilities.py", line 40, in with_injected_client
    return await fn(*args, **kwargs)
  File "C:\Users\Me\Documents\GitHub\My-Repo\venv\prefect\lib\site-packages\prefect\deployments.py", line 219, in load_flow_from_flow_run
    flow = await run_sync_in_worker_thread(load_flow_from_entrypoint, str(import_path))
  File "C:\Users\Me\Documents\GitHub\My-Repo\venv\prefect\lib\site-packages\prefect\utilities\asyncutils.py", line 91, in run_sync_in_worker_thread
    return await anyio.to_thread.run_sync(
  File "C:\Users\Me\Documents\GitHub\My-Repo\venv\prefect\lib\site-packages\anyio\to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "C:\Users\Me\Documents\GitHub\My-Repo\venv\prefect\lib\site-packages\anyio\_backends\_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "C:\Users\Me\Documents\GitHub\My-Repo\venv\prefect\lib\site-packages\anyio\_backends\_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "C:\Users\Me\Documents\GitHub\My-Repo\venv\prefect\lib\site-packages\prefect\flows.py", line 875, in load_flow_from_entrypoint
    flow = import_object(entrypoint)
  File "C:\Users\Me\Documents\GitHub\My-Repo\venv\prefect\lib\site-packages\prefect\utilities\importtools.py", line 201, in import_object
    module = load_script_as_module(script_path)
  File "C:\Users\Me\Documents\GitHub\My-Repo\venv\prefect\lib\site-packages\prefect\utilities\importtools.py", line 164, in load_script_as_module
    raise ScriptError(user_exc=exc, path=path) from exc
prefect.exceptions.ScriptError: Script at 'flows\\my_flow.py' encountered an exception: FileNotFoundError(2, 'No such file or directory')
16:00:21.414 | INFO    | prefect.infrastructure.process - Process 'wine-bittern' exited cleanly.
2

Sometimes writing it all out really helps the troubleshooting process. I had to make two changes to get it to work:

  1. Change my working directory up one level from:
    (prefect-venv) PS C:/Users/Me/Documents/GitHub/My-Repo/proj1>
    to:
    (prefect-venv) PS C:/Users/Me/Documents/GitHub/My-Repo>

  2. Change the path preceding the flow entrypoint from:
    prefect deployment build flows/my_flow.py:my_function -n my-deployment -sb github/my-github-sb --path proj1 -a
    to:
    prefect deployment build proj1/flows/my_flow.py:my_function -n my-deployment -sb github/my-github-sb --path proj1 -a

Super Troopers Oops GIF by Searchlight Pictures

1 Like