Handling Environment Configuration Across Tasks in Prefect 2 with DaskTaskRunner

Help
, , ,
1

Hello!

I’m currently in the process of migrating my pipelines from Prefect 1 to Prefect 2, and I’ve run into an issue with configuring the DaskTaskRunner for my flow.
Here’s the structure of my flow:

from dotenv import load_dotenv


@flow(task_runner=DaskTaskRunner())
def myflow(env_path: str = ".env"):
    load_dotenv(dotenv_path=env_path)
    raw_data = download_data() # Downloads batches of data and holds the path to the files
    adapt = adapt.map(raw_data)
    load = load.map(adapt)

I pass the .env file path as an argument, and the adapt task recognizes and uses the loaded .env file. However, when it reaches the load task, it doesn’t recognize it.
I suspect this might be because each subprocess has its own environment, and I’m seeking a way to configure the .env file for all tasks.
How can I achieve this?

Thank you for your help!