How to use Prefect with awswrangler?

Setup

First, install:

pip install awswrangler prefect-aws

Then, register the block:

prefect block register -m prefect_aws.credentials

Create the credentials block, either from the UI or via code:

from prefect_aws.credentials import AwsCredentials

aws = AwsCredentials(
    aws_access_key_id="xxx",
    aws_secret_access_key="xxx",
)
aws.save("prod", overwrite=True)

Usage

Here is a usage example retrieving the boto3 session from the credentials block and passing it to one of many operations you can do with awswrangler:

import awswrangler as wr
from prefect_aws.s3 import AwsCredentials

creds = AwsCredentials.load("prod")
session = creds.get_boto3_session()

wr.s3.to_parquet(boto3_session=session, ...)

Benefits

  1. Your credentials are stored securely in one place
  2. You can easily rotate them in one place
  3. No need for injecting environment variables or hardcoding credentials