Parameterizing Secrets in Flows and Tasks

Archive
, ,
1

View in #prefect-community on Slack

Justin_Burchard @Justin_Burchard: Hello everyone.

I’m a recent adopter of Prefect 1.0. I have a use case in which I have a flow that I would like to reuse for multiple projects, with different credentials and database schemas. I’d like to use a config file to publish the flows. It seems like Parameters would be great for this, but the documentation seems to suggest can only assign them for specific flow runs. Does anyone have any suggestions as to how I could structure this?

Kevin_Kho @Kevin_Kho: You can Parameterize the Secret name to pull to create the Database connection. If you want to do it as a Flow, you can do

with Flow('parent') as flow:
   create_flow_run(flow_name=.., project_name=..., parameters={'env': "DEV"})

and then in the child flow:

@task
def get_secret_name(env):
    return env+"_DATABASE"
with Flow('child') as flow:
    env = Parameter('env')
    secret_name = get_secret_name(env)
    connection = PrefectSecret()(secret_name)
    ....

If you want to reuse a task across flows:

@task
def get_db(env):
    connection = Secret(env+"DATABASE").get()
    do_something(connection)

with Flow("...") as flow:
    env = Parameter('env')
    get_db(env)

so here the Secret is pulled inside the task and the Parameter can be changed on the Flow level.

Does that give ideas?

Justin_Burchard @Justin_Burchard: Yes, that is great. Thank you!

1 Like