Intermittent Error with Prefect Deployments

Hello everyone,

This is my first post, and I’m quite desperate to find a solution to an intermittent error I’ve been encountering. Despite extensive web searches, consulting the documentation, and reaching out to Marvin on Slack, I haven’t been able to resolve it.

Here are the details:

  • Prefect Version: 2.19.4
  • Project Structure:
    • I have an RDP server with a prefect-sba project folder containing a flows subfolder.
    • Some of my deployments require reading and writing files within a data folder. While this data isn’t uploaded to GitHub, it resides in the project folder on the RDP server.
  • Deployment Command: I use the following command for my deployments:
prefect deployment build -n zbooksInvoices -q test flows/zbooks.py:update_zbooks_invoices -sb github/prefect-repo --apply
  • Working Directory Configuration:
    • After creating the deployment, I navigate to the Prefect Cloud UI, select the deployment, go to “Infrastructure,” and set the working directory to C:/prefect-sba.
    • This allows me to successfully read the code from GitHub and write reports to my RDP folder.
  • The Issue:
    • Occasionally, the deployment runs fail with the following error:
Downloading flow code from storage at ''
...
shutil.Error: [('C:\\Users\\SEBAST~1\\AppData\\Local\\Temp\\2\\tmpi2cv1c7cprefect\\flows\\src\\calendly_http\\calendly_http.py', '.\\flows\\src\\calendly_http\\calendly_http.py', '[WinError 32] The process cannot access the file because it is being used by another process')]
  • Interestingly, this error doesn’t occur for functions without a custom “working dir” in infrastructure.
  • Prefect Agent Invocation:
    • I invoke my Prefect agent as follows (from within the virtual environment at C:/prefect-sba):
prefect agent start --pool "default-agent-pool" --work-queue "test"

Any insights or assistance would be greatly appreciated. Thank you in advance! :pray: