Prefect.deployments.steps.pip_install_requirements cannot find my requirements.txt file

Help
1

I am using Prefect 2.16.8 with GitHub as my storage and a Vertex AI worker. My (truncated) repository structure looks like this:

requirements.txt
...
projects/
├── csv_upload_flow
│   ├── README.md
│   ├── flows
│   │   ├── main.py
│   ├── prefect.yaml
│   ├── requirements.txt
│   └── tests
│       └── sample_test.py

Where the requirements.txt file at the root of the repository is for base requirements and the project level one is for each specific project. I am running my deployment from the projects/csv_upload_flow folder. My deployment file looks like this:

definitions:
  actions:
    github_pull: &github_pull
      id: github_pull
      repository: ##################
      branch: new-prefect-infra
      credentials: "{{ prefect.blocks.github-credentials.github-token }}"
    github_prod: &github_prod
      <<: *github_pull
      branch: main
    pip_install: &pip_install
      requires: github_pull
      requirements_file: requirements.txt
    work_pools:
      vertex_work_pool: &vertex_work_pool
        name: vertex-dev-pool
        work_queue_name: default
        job_variables:
          image: ################

deployments:
  - name: csv_upload_flow-dev-test
    tags: [dev]
    entrypoint: flows/main.py:csv_upload_flow
    schedules: []
    parameters:
      #######: ########
    pull:
      - prefect.deployments.steps.git_clone: *github_pull
      - prefect.deployments.steps.pip_install_requirements: *pip_install
    work_pool: *vertex_work_pool

  - name: csv_upload_flow-prod-test
    tags: [prod]
    entrypoint: flows/main.py:csv_upload_flow
    schedules: []
    pull:
      - prefect.deployments.steps.git_clone: *github_prod
      - prefect.deployments.steps.pip_install_requirements: *pip_install
    work_pool:
      <<: *vertex_work_pool
      name: vertex-prod-pool

The error I get when trying to run either of the flows is this:

RuntimeError: pip_install_requirements failed with error code 1: ERROR: Could not open requirements file: [Errno 2] No such file or directory: 'requirements.txt'

I have tried everything I can think of, but I’m not sure why at least one of the requirements.txt files was not found. I would anticipate that it would find the project level one, because that’s where the deployment was created from, but I would understand if it found the root file, because it is using GitHub storage, and how does it know to path into the correct project folder after cloning? Regardless, it finds neither, and I am baffled.

Any help is appreciated!

2

@desertaxle this is blocking us from upgrading to workers, any thoughts?

3

I’ll answer my own question. Adding this line to the step fixed it:

directory: "{{ github_pull.directory }}/projects/csv_upload_flow"