Obscure Secret in Prefect UI with Pydantic

Archive
, ,
1

View in #prefect-community on Slack

Nora_Myer @Nora_Myer: Hey hi - prefect noob here - we’re using Prefect 2.0 w/ Prefect Cloud in a new system were building, and I have a couple Q’s around secrets. Details in the thread - thanks in advance!!
Right now we have a task that grabs secrets from Vault (lets call Task A), and a task that uses those secrets to make api calls to internal services (Task B). Since you cant call tasks within tasks, the higher level flow that calls Task A, then Task B. The issue is by passing these secrets into the task as a parameter, I think we’d be exposing them in logs and in the Prefect Cloud 2.0 UI

So my q’s:

  1. Does using a secret Block always require uploading the Block to Prefect Cloud 2.0 (if thats the route were going)? We dont want any secrets to ever leave our own infra, so wouldnt want to use Blocks if thats the case
  2. Is there any other way to pass secrets around within a flow and not expose them?
  3. Obviously another option is, only grab the secrets from within the task level that is actually using them. So make Task A, not be a task. If this is the best route, also totally fine mostly just curious

Kalise_Richmond @Kalise_Richmond: Hi Nora, in order to ensure that your secrets never leaves your infrastructure option 3 would be best. Blocks do require uploading to Prefect Cloud 2.0. You could definitely make a github request for feature enhancement to have a way to obscure parameters in the UI.

GitHub

Ryan_Peden @Ryan_Peden: If you decide to pass them as parameters, consider using pydantic.SecretStr like many of our blocks do. For example:

from prefect import flow, get_run_logger
from pydantic import SecretStr

@flow
def flow1():
    secret = SecretStr("password")
    flow2(secret)
    
@flow
def flow2(secret: SecretStr):
    logger = get_run_logger()
    logger.info(f"secret: {secret}")

if __name__ == "__main__":
    flow1()

As this example shows, the secret remains obscured in the flow run parameters UI (screenshot attached), and if you accidentally log a secret, it will be hidden there as well:

12:30:30.654 | INFO    | prefect.engine - Created flow run 'quiet-boobook' for flow 'flow1'
12:30:30.826 | INFO    | Flow run 'quiet-boobook' - Created subflow run 'crystal-sponge' for flow 'flow2'
12:30:30.856 | INFO    | Flow run 'crystal-sponge' - secret: **********
12:30:30.877 | INFO    | Flow run 'crystal-sponge' - Finished in state Completed()
12:30:30.891 | INFO    | Flow run 'quiet-boobook' - Finished in state Completed('All states completed.')

Screen Shot 2022-10-26 at 12.37.36 PM
Screen Shot 2022-10-26 at 12.37.36 PM1896×536 23.4 KB

3 Likes
2

Hey Kalise, thanks for the feedback. However, I still cannot figure this out. We’re using a local Prefect server with some deployments. In one of them, I tried using a SecretStr (from Pydantic) parameter to allow the same deployment to be run by different users. The intent here is for them to be able to pass their username and password using the “Custom Run” option in the deployment and keep the password string obscure in the UI.

So, I tried the following:

from pydantic import SecretStr
from prefect import flow

@flow(name='example')
def main(username:str, password:SecretStr):
    # do stuff with the "password" variable
    pass

After the deployment, when I access the “Custom Run” option, the password is indeed obscured in the UI (*********** instead of mysecretpass).

However, the password is visible when I access the Parameters tab in the Flow Run.

Is this the desired behaviour? Am I doing something wrong here?

Thanks in advance!