This example adds boto3 and botocore loggers so that debug logs from those libraries are displayed in the Prefect UI. Those extra log messages allow to debug issues such as bad requests made to some AWS APIs.
First, set those two environment variables on the machine from which you register your flow (e.g. your laptop):
export PREFECT__LOGGING__EXTRA_LOGGERS="['boto3', 'botocore']"
export PREFECT__LOGGING__LEVEL="DEBUG"
Now, when you use boto3 in your flow code and set the level to DEBUG explicitly, all of boto3 debug logs such as API response headers become available in the logs. Here is an example:
import boto3
import logging
import pandas as pd
import subprocess
import prefect
from prefect import task, Flow
from prefect.storage import Local
from prefect.run_configs import LocalRun
@task
def get_s3_data():
logging.getLogger("boto3").setLevel(logging.DEBUG)
logging.getLogger("botocore").setLevel(logging.DEBUG)
prefect_logger = prefect.context.get("logger")
s3 = boto3.client("s3", region_name="us-east-1")
obj = s3.get_object(Bucket="nyc-tlc", Key="trip data/yellow_tripdata_2021-07.csv",)
df = pd.read_csv(obj["Body"], nrows=5, low_memory=False)
prefect_logger.info("Read df: %s", df.head(5))
with Flow(
"log_test",
storage=Local(add_default_labels=False),
run_config=LocalRun(
labels=["logtest"],
env={
"PREFECT__LOGGING__EXTRA_LOGGERS": "['boto3', 'botocore']",
"PREFECT__LOGGING__LEVEL": "DEBUG",
},
),
) as flow:
get_s3_data()
if __name__ == "__main__":
flow_id = flow.register("xyz")
subprocess.run(f"prefect run --id {flow_id}", shell=True)
subprocess.run(
"prefect agent local start --label logtest --no-hostname-label", shell=True
)
When you run the command above, this will register your flow, trigger a flow run and spin up a local agent. When you execute it, you should see the detailed logs from boto3.
In this example, we are using a public dataset, so you should be able to run it without changing anything