How to get a Prefect flow to run inside a conda environment

[kzathey]

I’m new to Prefect and I’m trying to get my flow to run inside a conda environment. I’m trying to follow the yaml manifest deployment outlined here:

How to run my deployment in a virtual environment such as conda?

There are several ways how you can accomplish that. Run the agent directly in the virtual environment from which you want to execute the flow runs Assuming that your agent is already running in the virtual environment that has all required dependencies. In that case, you don’t need to do anything else on your infrastructure block or deployment - the default Process block will already work out of the box. Adjust the YAML file Another option is to adjust the YAML deployment file before apply…

I have 2 conda environments, the Prefect server just running under a conda environment called prefect (only has Prefect installed) and I have an environment called flow-test that the flow should be running in (has prefect and pandas installed).

My manifest yaml looks like:

### hello-deployment.yaml
###
### A complete description of a Prefect Deployment for flow 'hello'
###
name: my-flow-deployment
description: null
version: 84cb14085d1234c5cf60c13e352b0c9e
# The work queue that will handle this deployment's runs
work_queue_name: my-queue
work_pool_name: my-process-pool
tags: []
parameters: {}
schedule: null
is_schedule_active: null
infra_overrides: {}
infrastructure:
  type: process
  env: {}
  labels: {}
  name: null
  command:
    - conda
    - run
    - -n
    - flow-test
    - python
    - -m
    - prefect.engine
  stream_output: true
  working_dir: null
  block_type_slug: process
  _block_type_slug: process

###
### DO NOT EDIT BELOW THIS LINE
###
flow_name: hello
manifest_path: null
storage: null
path: C:\Users\{MY_USER_NAME}\Documents\prefect\flows\hello
entrypoint: hello.py:hello
parameter_openapi_schema:
  title: Parameters
  type: object
  properties:
    name:
      title: name
      position: 0
      type: string
  required:
    - name
  definitions: null
timestamp: "2023-07-26T14:04:25.905180+00:00"
triggers: []

I deploy this using prefect deployment apply ./hello-deployment.yaml from inside the prefect environment.

My hello.py looks like:

import datetime
from prefect import flow
import pandas as pd
from pandas.tseries.offsets import BDay


@flow(log_prints=True)
def hello(name: str):
    print(f"Hello {name}, yesterday was {(datetime.datetime.today() - BDay(1)).strftime(r'%Y-%m-%d')}!")


if __name__ == "__main__":
    hello(name="John Smith")

When I trigger this flow from the UI I’m getting the error:

Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "C:\Users\{MY_USER_NAME}\AppData\Local\Temp\tmp3eia6u6vprefect\hello.py", line 3, in <module>
    import pandas as pd
ModuleNotFoundError: No module named 'pandas'

...

prefect.exceptions.ScriptError: Script at 'hello.py' encountered an exception: ModuleNotFoundError("No module named 'pandas'")
03:54:57 PM
prefect.flow_runs
Process 17228 exited cleanly.

I’ve verified that if a activate the conda environment flow-test and run python that I can import pandas without issue. I’m not sure what I’m doing wrong. Any help is appreciated.