Can I use .env files with Prefect 1.0 run configurations, e.g. using Python dotenv package?

Archive
, , ,
1

View in #prefect-community on Slack

@Daniel_Saxton: according to the docs it’s possible to define individual environment variables at runtime for a given agent by using the --env flag, but is there also something like an --env-file flag if you have several environment variables defined in a file (kind of like you have with docker-compose)?
https://docs.prefect.io/orchestration/getting-started/flow-configs.html#configure-environment-variables

Flow Configuration | Prefect Docs

Kevin_Kho @Kevin_Kho: There is none unfortunately. What is your agent type? You could consider doing it at the flow runconfig side and that might be easier. You also can maybe use the Python interface you spin up the agent and programmatically insert the env there?

@Anna_Geller: @Daniel_Saxton You can do that by using python-dotenv package - this way, you can load your .env file as a dict (as correctly suggested by @Kevin_Kho):

CONFIG = dotenv_values(".env")

and pass it to your run_config:

flow.run_config=DockerRun(env=CONFIG)

Here is a full flow example:

"""
The example contents of .env file in this root project dir (.env file is obviously not committed):
ENV_1=value_1
ENV_2=value_2

Usage in CLI:
prefect agent docker start -l local_docker
prefect register --project xyz -p flows/docker_pickle_docker_run_envfile_example.py
prefect run --name docker_pickle_docker_run_envfile_example --watch
"""
import os
from dotenv import dotenv_values
from prefect import Flow, task
from prefect.run_configs import DockerRun
from prefect.storage import Docker

# Note: those env vars are read from root project dir and loaded to run config "env" at flow registration time
CONFIG = dotenv_values(".env")

FLOW_NAME = "docker_pickle_docker_run_envfile_example"
DOCKER_STORAGE = Docker(
    image_name=FLOW_NAME, image_tag="latest", python_dependencies=["python-dotenv"],
)


@task(log_stdout=True)
def print_env_vars_from_env_file():
    env1 = os.environ.get("ENV_1")
    env2 = os.environ.get("ENV_2")
    print(env1)
    print(env2)


with Flow(
    FLOW_NAME,
    run_config=DockerRun(
        image=f"{FLOW_NAME}:latest", labels=["local_docker"], env=CONFIG
    ),
    storage=DOCKER_STORAGE,
) as flow:
    print_env_vars_from_env_file()

and usage in a repo: https://github.com/anna-geller/packaging-prefect-flows/blob/master/flows/docker_pickle_docker_run_envfile_example.py

GitHub: packaging-prefect-flows/docker_pickle_docker_run_envfile_example.py at master · anna-geller/packaging-prefect-flows

@Daniel_Saxton: thanks @Kevin_Kho (i was using a local agent) and @Anna_Geller, could a flag like this make for a useful enhancement to the prefect CLI? wanting to specify many environment variables for a given agent is probably a fairly common situation

Kevin_Kho @Kevin_Kho: I have seen someone else request this and I think we’d accept a PR for it.
@Marvin open “Accept env variables as a file with agent start using the CLI”

Marvin @Marvin: Accept env variables as a file with agent start using the CLI · Issue #5513 · PrefectHQ/prefect · GitHub