Creating Volume Mounts or Files from Prefect Secrets

Archive
, ,
1

I would like to mount a Prefect Secret into a file on my flow.

I have done this using a shell task but there is no write file task in the files tasks File and Filesystem Tasks | Prefect Docs. Is there a task I should be using instead to do this?

Dealing with keys and permissions would be easier in some cases if we could just assume that a secret is written to a given file either as the flow starts. It would also be cool to do this for environment variables, but maybe that is possible already?

I would also settle for best practices configuring ~/.dbt/profiles.yaml for orchestrating DBTShellTasks. How are people providing or overriding profiles in their flows?

1 Like
2

Welcome back @Tyler_Wanner, great to see you here! :wave:

For 1.0, one pattern you may follow is described here:

Essentially you would store all sensitive dbt profile information in your Prefect Secrets and reference those in the DbtShellTask.

–
For Prefect 2.0 though, we provide much easier building blocks to do that, which are literally called Blocks. You can store all your sensitive data there and reference it on your prefect-dbt tasks as shown here:

For reference here is dbt collection:

Another alternative, although not secure, would be baking your dbt profile file into your container image and reference the path on the DbtShellTask

3

Hi Anna! For Prefect1, It would seem then that the suggested practice is to, rather than using a file at all, set a json secret for the params that would normally go into the profile and then pass them to dbt_kwargs along with overwrite_profiles=True in my DbtShellTask?

For Prefect2, it seems like the collection doesn’t have a docs page like the others here: Collections Catalog - Prefect 2.0
However, I think the second example in the docstring here is what I’m looking for
prefect-dbt/commands.py at main · PrefectHQ/prefect-dbt · GitHub

            dbt_cli_profile = DbtCliProfile(
                name="jaffle_shop",
                target="dev",
                target_configs=target_configs,
            )
            result = trigger_dbt_cli_command(
                "dbt debug",
                overwrite_profiles=True,
                dbt_cli_profile=dbt_cli_profile
            )
1 Like
4

Yup, you’re spot on!

You’re right, it’s because this collection hasn’t been officially released. We are working on it and will announce it once it’s published - if you follow the dbt tag, you can get an email notification once we release that - Topics tagged dbt

5

ooh awesome! Looking forward to that release and I am just researching for an upcoming project at this point so timeline is no problem for me and my team :+1: Thanks for your help, I feel prepared now :muscle:

1 Like