How to set up a custom logger to stream logs to the Prefect Cloud backend? (extra loggers)
Many libraries (e.g. boto3) are set up to emit their own logs. Similarly, you may have your own custom logger within your custom modules.
Environment variable
In order to make those extra loggers stream logs to the Prefect backend (making those logs available in the Prefect UI), add the following environment variable:
#test.py
import logging
import sys
logger = logging.getLogger("custom")
logger.setLevel('DEBUG')
def run():
logger.info('This is an info message')
logger.debug('This is an debug message')
and the flow:
# flow.py
from prefect import Flow, task
from test import run
@task
def abc():
run()
return
with Flow("...") as flow:
abc()
flow.run()
and my config.toml
[logging]
# Extra loggers for Prefect log configuration
extra_loggers = "['custom']"
and my logs:
(prefect) kevinkho@Kevins-MacBook-Pro sources % python query.py
[2022-02-24 14:51:57-0500] INFO - prefect.FlowRunner | Beginning Flow run for '...'
[2022-02-24 14:51:57-0500] INFO - prefect.TaskRunner | Task 'abc': Starting task run...
[2022-02-24 14:51:57-0500] INFO - custom | This is an info message
[2022-02-24 14:51:57-0500] DEBUG - custom | This is an debug message
[2022-02-24 14:51:57-0500] INFO - prefect.TaskRunner | Task 'abc': Finished task run for task with final state: 'Success'
[2022-02-24 14:51:57-0500] INFO - prefect.FlowRunner | Flow run SUCCESS: all reference tasks succeeded