Flow could not be retrieved from deployment

Help
, ,
1

Some of my flow runs are randomly returning this error and failing. The Python scripts are stored in GitHub and accessed via storage blocks. Here’s my setup:

  • Work Queue
    • Work Pool 1
      • Deployment A (runs hourly M-F between 8:00a and 4:00p)
    • Work Pool 2
      • Deployment B (runs daily at 3:00a)
    • Work Pool 3
      • Deployment C (runs daily at 8:00a)
      • Deployment D (runs hourly M-F between 8:30a and 4:30p)

Deployments A and D are the ones that are failing at various times throughout the day. I have 11 failures out of 99 flow runs over the past week, so nothing significant, but would like to try understanding why this is happening. Possible related to this?

Here’s the error I’m seeing:

Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "C:\Users\King\Documents\GitHub\Syteline\venv\prefect\lib\site-packages\prefect\engine.py", line 294, in retrieve_flow_then_begin_flow_run
    flow = await load_flow_from_flow_run(flow_run, client=client)
  File "C:\Users\King\Documents\GitHub\Syteline\venv\prefect\lib\site-packages\prefect\client\utilities.py", line 40, in with_injected_client
    return await fn(*args, **kwargs)
  File "C:\Users\King\Documents\GitHub\Syteline\venv\prefect\lib\site-packages\prefect\deployments.py", line 197, in load_flow_from_flow_run
    await storage_block.get_directory(from_path=deployment.path, local_path=".")
  File "C:\Users\King\Documents\GitHub\Syteline\venv\prefect\lib\site-packages\prefect\filesystems.py", line 966, in get_directory
    raise OSError(f"Failed to pull from remote:\n {err_stream.read()}")
OSError: Failed to pull from remote:
 Cloning into 'C:\Users\King\AppData\Local\Temp\2\tmpy_agj3p8prefect'...
fatal: unable to access 'https://github.com/Con-Forms/Business-Intelligence.git/': Recv failure: Connection was reset
2

FOLLOW UP #1

I think the root cause is an unstable internet connection within my company’s network. I pinged GitHub to see if I could consistently get a reply. I did this both inside and outside of my company’s network with the following entry in the command line:

ping github.com -t > c:\users\king\downloads\ping.txt

I had 0% packet loss running this outside of my company’s network and 22% packet loss running inside the network.

Is it possible to automatically retry pulling the repo from GitHub when this happens? I assume this would need to be set within the deployment configuration, but I don’t see an option for this in the YAML file.

The alternative would be storing the flow locally, but I would really prefer to use GitHub.

4

Brought this up with the Prefect team during an Associate Certification Course and they recommended creating an issue for this in GitHub. Issue #10725 has been added to their backlog for future consideration.