Obscure Secret in Prefect UI with Pydantic

View in #prefect-community on Slack

Nora_Myer @Nora_Myer: Hey hi - prefect noob here - we’re using Prefect 2.0 w/ Prefect Cloud in a new system were building, and I have a couple Q’s around secrets. Details in the thread - thanks in advance!!
Right now we have a task that grabs secrets from Vault (lets call Task A), and a task that uses those secrets to make api calls to internal services (Task B). Since you cant call tasks within tasks, the higher level flow that calls Task A, then Task B. The issue is by passing these secrets into the task as a parameter, I think we’d be exposing them in logs and in the Prefect Cloud 2.0 UI

So my q’s:

  1. Does using a secret Block always require uploading the Block to Prefect Cloud 2.0 (if thats the route were going)? We dont want any secrets to ever leave our own infra, so wouldnt want to use Blocks if thats the case
  2. Is there any other way to pass secrets around within a flow and not expose them?
  3. Obviously another option is, only grab the secrets from within the task level that is actually using them. So make Task A, not be a task. If this is the best route, also totally fine mostly just curious

Kalise_Richmond @Kalise_Richmond: Hi Nora, in order to ensure that your secrets never leaves your infrastructure option 3 would be best. Blocks do require uploading to Prefect Cloud 2.0. You could definitely make a github request for feature enhancement to have a way to obscure parameters in the UI.


Ryan_Peden @Ryan_Peden: If you decide to pass them as parameters, consider using pydantic.SecretStr like many of our blocks do. For example:

from prefect import flow, get_run_logger
from pydantic import SecretStr

def flow1():
    secret = SecretStr("password")
def flow2(secret: SecretStr):
    logger = get_run_logger()
    logger.info(f"secret: {secret}")

if __name__ == "__main__":

As this example shows, the secret remains obscured in the flow run parameters UI (screenshot attached), and if you accidentally log a secret, it will be hidden there as well:

12:30:30.654 | INFO    | prefect.engine - Created flow run 'quiet-boobook' for flow 'flow1'
12:30:30.826 | INFO    | Flow run 'quiet-boobook' - Created subflow run 'crystal-sponge' for flow 'flow2'
12:30:30.856 | INFO    | Flow run 'crystal-sponge' - secret: **********
12:30:30.877 | INFO    | Flow run 'crystal-sponge' - Finished in state Completed()
12:30:30.891 | INFO    | Flow run 'quiet-boobook' - Finished in state Completed('All states completed.')