How to set up a custom logger to stream logs to Prefect Cloud backend? (extra loggers)

Archive
, , , , , , ,
2

Giving a full end to end example here:

#test.py
import logging
import sys

logger = logging.getLogger("custom")
logger.setLevel('DEBUG')

def run():
    logger.info('This is an info message')
    logger.debug('This is an debug message')

and the flow:

# flow.py
from prefect import Flow, task
from test import run

@task
def abc():
    run()
    return

with Flow("...") as flow:
    abc()

flow.run()

and my config.toml

[logging]
# Extra loggers for Prefect log configuration
extra_loggers = "['custom']"

and my logs:

(prefect) kevinkho@Kevins-MacBook-Pro sources % python query.py
[2022-02-24 14:51:57-0500] INFO - prefect.FlowRunner | Beginning Flow run for '...'
[2022-02-24 14:51:57-0500] INFO - prefect.TaskRunner | Task 'abc': Starting task run...
[2022-02-24 14:51:57-0500] INFO - custom | This is an info message
[2022-02-24 14:51:57-0500] DEBUG - custom | This is an debug message
[2022-02-24 14:51:57-0500] INFO - prefect.TaskRunner | Task 'abc': Finished task run for task with final state: 'Success'
[2022-02-24 14:51:57-0500] INFO - prefect.FlowRunner | Flow run SUCCESS: all reference tasks succeeded
2 Likes