Hi all, I am having some trouble with warnings when I run deployments. I have created an MRE based on the Prefect Deployments tutorial, which will help explain this.
I created the deployment using Python. The deployments actually run as expected and the flow produces the expected output, but the warnings are always present. The warnings are related to
- an unrecognized Prefect profile (I did set a custom profile and Prefect does recognize it)
- the order of Python imports in
deployment.py('prefect.engine' found in sys.modules after import of package 'prefect'- more details below)
I am using a custom Prefect profile file myprofiles.toml. All my deployment runs are local. Details and MRE code below.
Problem / Error (Warning)
The following are the two warnings that appear when I run any deployment
WARNING: Active profile 'local' set by environment variable not found. The default profile will be used instead.
/usr/lib/python3.10/runpy.py:126: RuntimeWarning: 'prefect.engine' found in sys.modules after import of package 'prefect', but prior to execution of 'prefect.engine'; this may result in unpredictable behaviour
warn(RuntimeWarning(msg))
Steps to Reproduce with MRE
Code for MRE
My deployment.py file contents are
# deployment.py
from log_flow import log_flow
from prefect.deployments import Deployment
deployment = Deployment.build_from_flow(
flow=log_flow,
name="log-simple",
parameters={"url": "https://catfact.ninja/fact"},
infra_overrides={
"env": {
"PREFECT_LOGGING_LEVEL": "DEBUG",
"PREFECT_PROFILES_PATH": "myprofiles.toml",
"PREFECT_PROFILE": "local",
}
},
work_queue_name="demo",
)
if __name__ == "__main__":
deployment.apply()
and the flow file contents are
# log_flow.py
import requests
from prefect import flow, task
@task
def call_api(url):
response = requests.get(url)
print(response.status_code)
return response.json()
@task
def parse_fact(response):
fact = response["fact"]
print(fact)
return fact
@flow
def log_flow(url):
fact_json = call_api(url)
fact_text = parse_fact(fact_json)
return fact_text
if __name__ == "__main__":
log_flow("https://catfact.ninja/fact")
and the custom Prefect profiles file is also located in the working directory. It contains
active = "local"
[profiles.local]
PREFECT_API_URL = "http://127.0.0.1:4200/api"
Contents of Working directory
The contents of the working dir are
$ ls
deployment.py log_flow.py myprofiles.toml
There is also a blank .prefectignore file in the working directory.
Versions
Python==3.10.6
prefect==2.7.8
requests==2.28.1
Workflow for running MRE
I started by specifying to use the 'local' profile
export PREFECT_PROFILES_PATH="myprofiles.toml"
export PREFECT_PROFILE="local"
prefect profile use local
and verified that this was selected
prefect profile use local
This produces the expected output
β Connecting...
Connected to Prefect Orion using profile 'local'
When I list profiles
prefect profile ls
'local' is shown as active, which is also the expected output
βββββββββββββββββββββββ
β Available Profiles: β
β‘ββββββββββββββββββββββ©
β default β
β * local β
βββββββββββββββββββββββ
* active profile
I then followed the docs to create the deployment with Python by runing
python deployment.py
Then (in a separate terminal) start the agent and create a work-queue
prefect agent start -q demo
Back in the deployments terminal, run the deployment
prefect deployment run log-flow/log-sample
After this
- the two warnings from above are printed to the screen
- the deployment runs and the flow produces the expected output
The full output is
Starting v2.7.8 agent connected to http://127.0.0.1:4200/api...
___ ___ ___ ___ ___ ___ _____ _ ___ ___ _ _ _____
| _ \ _ \ __| __| __/ __|_ _| /_\ / __| __| \| |_ _|
| _/ / _|| _|| _| (__ | | / _ \ (_ | _|| .` | | |
|_| |_|_\___|_| |___\___| |_| /_/ \_\___|___|_|\_| |_|
Agent started! Looking for work from queue(s): demo...
17:41:31.726 | INFO | prefect.agent - Submitting flow run 'ee7bde5a-3fb2-4fce-b8e6-7a7169231ead'
17:41:31.786 | INFO | prefect.infrastructure.process - Opening process 'cheerful-poodle'...
17:41:31.802 | INFO | prefect.agent - Completed submission of flow run 'ee7bde5a-3fb2-4fce-b8e6-7a7169231ead'
WARNING: Active profile 'local' set by environment variable not found. The default profile will be used instead.
/usr/lib/python3.10/runpy.py:126: RuntimeWarning: 'prefect.engine' found in sys.modules after import of package 'prefect', but prior to execution of 'prefect.engine'; this may result in unpredictable behaviour
warn(RuntimeWarning(msg))
17:41:33.070 | DEBUG | prefect.client - Connecting to API at http://127.0.0.1:4200/api/
17:41:33.088 | INFO | Flow run 'cheerful-poodle' - Downloading flow code from storage at '/home/<...>/prefect-tutorial'
17:41:33.089 | DEBUG | Flow run 'cheerful-poodle' - Importing flow code from 'log_flow.py:log_flow'
17:41:33.107 | DEBUG | Flow run 'cheerful-poodle' - Starting 'ConcurrentTaskRunner'; submitted tasks will be run concurrently...
17:41:33.108 | DEBUG | prefect.task_runner.concurrent - Starting task runner...
17:41:33.189 | DEBUG | Flow run 'cheerful-poodle' - Executing flow 'log-flow' for flow run 'cheerful-poodle'...
17:41:33.190 | DEBUG | Flow run 'cheerful-poodle' - Beginning execution...
17:41:33.213 | INFO | Flow run 'cheerful-poodle' - Created task run 'call_api-8b7f2ba2-0' for task 'call_api'
17:41:33.213 | INFO | Flow run 'cheerful-poodle' - Executing 'call_api-8b7f2ba2-0' immediately...
17:41:33.246 | DEBUG | Task run 'call_api-8b7f2ba2-0' - Beginning execution...
17:41:33.464 | INFO | Task run 'call_api-8b7f2ba2-0' - Finished in state Completed()
17:41:33.485 | INFO | Flow run 'cheerful-poodle' - Created task run 'parse_fact-7d0f7c7e-0' for task 'parse_fact'
17:41:33.486 | INFO | Flow run 'cheerful-poodle' - Executing 'parse_fact-7d0f7c7e-0' immediately...
17:41:33.533 | DEBUG | Task run 'parse_fact-7d0f7c7e-0' - Beginning execution...
17:41:33.556 | INFO | Task run 'parse_fact-7d0f7c7e-0' - Finished in state Completed()
17:41:33.579 | DEBUG | prefect.task_runner.concurrent - Shutting down task runner...
17:41:33.580 | INFO | Flow run 'cheerful-poodle' - Finished in state Completed()
17:41:33.625 | DEBUG | prefect.client - Connecting to API at http://127.0.0.1:4200/api/
200
Many cats cannot properly digest cow's milk. Milk and milk products give them diarrhea.
17:41:33.809 | INFO | prefect.infrastructure.process - Process 'cheerful-poodle' exited cleanly.
Questions
-
It looks like the Prefect profile that I set is not being recognized
WARNING: Active profile 'local' set by environment variable not found. The default profile will be used instead.As I showed above, I have set the profile to
'local'and defined this profile in the.tomlfile.prefect profile lsis correctly picking up this'local'profile. Is there some additional configuration I need for the'local'profile to be correctly picked up by the deployment? How can I fix the issue that is causing this warning? -
The
RuntimeWarningerror message is suggesting the order of Python imports in thedeployment.pyfile should be reversed/usr/lib/python3.10/runpy.py:126: RuntimeWarning: 'prefect.engine' found in sys.modules after import of package 'prefect', but prior to execution of 'prefect.engine'; this may result in unpredictable behaviour warn(RuntimeWarning(msg))I have tried reversing the order. Change
# deployment.py from log_flow import log_flow from prefect.deployments import Deploymentto
# deployment.py from prefect.deployments import Deployment from log_flow import log_flowbut there is no difference in the output - the
RuntimeWarningmessage still appears. The order in my initialdeployment.pyfile was the same as in the docs example for Deployment. I am concerned aboutthis may result in unpredictable behaviour(for more complicated flow files). Is this an expected warning, or is there a way to fix this issue about the order of the imports?Many thanks in advance!
