How to set up a custom logger to stream logs to Prefect Cloud backend? (extra loggers)

How to set up a custom logger to stream logs to the Prefect Cloud backend? (extra loggers)

Many libraries (e.g. boto3) are set up to emit their own logs. Similarly, you may have your own custom logger within your custom modules.

Environment variable

In order to make those extra loggers stream logs to the Prefect backend (making those logs available in the Prefect UI), add the following environment variable:

export PREFECT__LOGGING__EXTRA_LOGGERS="['boto3', 'custom_lib']"

Note that you can also add this environment variable directly to your run configuration:

DockerRun(env={"PREFECT__LOGGING__EXTRA_LOGGERS": "['custom_lib']"})

config.toml

Alternatively, add the extra_loggers to your config.toml:

[logging]
extra_loggers = "['boto3', 'custom_lib']"

Source: Logging | Prefect Docs

Questions about it in the Slack community

Giving a full end to end example here:

#test.py
import logging
import sys

logger = logging.getLogger("custom")
logger.setLevel('DEBUG')

def run():
    logger.info('This is an info message')
    logger.debug('This is an debug message')

and the flow:

# flow.py
from prefect import Flow, task
from test import run

@task
def abc():
    run()
    return

with Flow("...") as flow:
    abc()

flow.run()

and my config.toml

[logging]
# Extra loggers for Prefect log configuration
extra_loggers = "['custom']"

and my logs:

(prefect) kevinkho@Kevins-MacBook-Pro sources % python query.py
[2022-02-24 14:51:57-0500] INFO - prefect.FlowRunner | Beginning Flow run for '...'
[2022-02-24 14:51:57-0500] INFO - prefect.TaskRunner | Task 'abc': Starting task run...
[2022-02-24 14:51:57-0500] INFO - custom | This is an info message
[2022-02-24 14:51:57-0500] DEBUG - custom | This is an debug message
[2022-02-24 14:51:57-0500] INFO - prefect.TaskRunner | Task 'abc': Finished task run for task with final state: 'Success'
[2022-02-24 14:51:57-0500] INFO - prefect.FlowRunner | Flow run SUCCESS: all reference tasks succeeded
2 Likes