How to use the AWSParametersManager task?

Archive
, , , ,
1

View in #prefect-community on Slack

@Vasco_Leitão: Hello! I’m exploring Prefect integration with AWS and trying to extract values from a AWS Parameter Store inside a task. From what I understand from docs, I can run the method in the AWSParametersManager class, however, I’m getting an error from the defaults_from_attrs helper decorator. What am I missing? :sweat_smile: (Code in the thread)

from prefect.tasks.aws.parameter_store_manager import AWSParametersManager

...
@task(name="Fetch Invoices from FTP Server")
def fetch_invoice_files() -> list:
    
    logger = prefect.context.get('logger')
    try:
        with paramiko.SSHClient() as s:
            s.set_missing_host_key_policy(paramiko.AutoAddPolicy())
            UPS_USERNAME = AWSParametersManager.run('ftp_user')
            UPS_PW = AWSParametersManager.run('ftp_password')
            s.connect(URL, 10022, username=UPS_USERNAME, password=UPS_PW)
    ...
    return files


...
with Flow(
    'parameter-test',
    run_config=flow_run_config,
) as flow:
    files = fetch_invoice_files()
    ...

Error:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/prefect/engine/task_runner.py", line 880, in get_task_run_state
    value = prefect.utilities.executors.run_task_with_timeout(
  File "/usr/local/lib/python3.9/site-packages/prefect/utilities/executors.py", line 468, in run_task_with_timeout
    return task.run(*args, **kwargs)  # type: ignore
  File "/Users/vascoleitao/Document/flows/ups_ftp_finance/flow.py", line 47, in fetch_invoice_files
    UPS_USERNAME = AWSParametersManager.run('ftp_user')
  File "/usr/local/lib/python3.9/site-packages/prefect/utilities/tasks.py", line 455, in method
    kwargs.setdefault(attr, getattr(self, attr))
AttributeError: 'str' object has no attribute 'parameter_name'

Kevin_Kho @Kevin_Kho: Is fetch_invoice_files a task?
Ah I see. Can you try:

UPS_USERNAME = AWSParametersManager().run('ftp_user')

@Vasco_Leitão: Yeah, it was that, though I needed to reference the arg in the run method.

UPS_USERNAME = AWSParametersManager().run(parameter_name='ftp_user')

However, I’m getting unexpected values. I should get the clear value of the parameter, right?

Kevin_Kho @Kevin_Kho: What are you getting?

@Vasco_Leitão: These two parameters are SecureString

print(UPS_USERNAME)
print(UPS_PW)

AQICAHjJcwEWLpsKGepQ9HpDWyYbuEDjdo5wDqcdcbs0xLhB4gGKDfIHfHZUA5CspbLvk+NKAAAAazBpBgkqhkiG9w0BBwagXDBaAgEAMFUGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQMG79ax+1uE+LuAb7AAgEQgCiEqBR/t74O9x2qFLEJNAg

AQICAHjJcwEWLpsKGepQ9HpDWyYbuEDjdo5wDqcdcbs0xLhB4gFrFkanm+V4F1cko5UBGctcAAAAaTBnBgkqhkiG9w0BBwagWjBYAgEAMFMGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQMTNnIXGejIWXNzUOYAgEQgCa

Kevin_Kho @Kevin_Kho: I haven’t used it, but the source is pretty straightforward. Do you know how you’d do it in normal python?

GitHub: prefect/parameter_store_manager.py at 40e2e2bd1c1de5651bc7fb484c823e43cfc69877 · PrefectHQ/prefect

@Vasco_Leitão: Yeah. I’m going to try it out with the boto3 SSM client and see if the problem still holds. Thank you for the swift response! If I get more issues on this, I’ll update the thread.
I think I understood what’s the issue here. SecureString parameters require decryption to be accessed. The boto3 SSM client has the option to decrypt the value (docs), while AWSParameterManager still has not.

Kevin_Kho @Kevin_Kho: Ah you can make your own task then (override the run method). Pretty sure we’d welcome a PR to make this configurable also if you’re interested :slightly_smiling_face: