Why am I getting this error and how to fix it? - "KeyError('The secret <SECRET NAME> was not found. Please ensure that it was set correctly in your tenant"

Archive
, ,
1

It happens when you retrieve a Secret separately in each mapped task. There are two ways to solve it:

  1. Changing the flow structure so that Secret is retrieved only once and passed to the mapped task as unmapped data dependency.
  2. Adding the Secret additionally to storage as shown below

[quote]
View in #prefect-community on Slack

Ryan_Brennan @Ryan_Brennan: Hi All - I’m running into a weird issue with Secrets. Sometimes a task will fail with the error message below even though the Secret is stored correctly in the tenant:

KeyError('The secret <SECRET NAME> was not found.  Please ensure that it was set correctly in your tenant

This seems to happen the most when I’m using task mapping but it seems pretty random. 100 iterations of the mapped task will work, then the 101st fails with the error above, then the next 100 are successful. Has anyone ran into this issue or have any ideas on what might be happening?

Kevin_Kho @Kevin_Kho: Hey @Ryan_Brennan, this is because the Secret is an API call and this is an API call failing (maybe rate limiting). It would help if you add the secret to storage ahead of that so that it’s only pulled once.

flow.storage = Storage(…,secrets=[…])

Ryan_Brennan @Ryan_Brennan: Oh awesome! thanks for the quick response @Kevin_Kho