How to log messages in imported classes

Archive
,
1

So I’m quite new with Prefect. I’m getting to know 2.0 and and I have some flows that I’m working on. I have quite a lot of code in functions or classes that I would want to import from external files and use withing flows and tasks. Most of that code is used for data manipulation and occasionally I need to print out certain things. I just cannot seem to be able to figure out how to get that to show in the logs.

As an example I might have a file called helpers.py:

import logging
logging.basicConfig(level=logging.INFO)

def say_hi():
    logging.info('Hi!')

Then in the flow file I would use it like this:

from prefect import flow, get_run_logger
from helpers import say_hi

@flow(name="log-example-flow")
def logger_flow():
    logger = get_run_logger()
    logger.info("Flow level message. Works fine")
    say_hi()

How to get that “Hi!” shown? I would be happy to just replace the standard logger with prefect logger but that doesn’t seem to be possible outside of a flow or task context.

I tried to look into adding extra loggers but didn’t exactly understand if and how it would work in this case.

I noticed that in Prefect 1.0 it seems to be possible to forward the output of stdout to a logger by setting log_stdout=True on the task. Something like that would be totally fine for me if I could just print instead of using the logger but apparently that is not possible in 2.0.

Any guidance on how to achieve this? I hope the explanation was clear enough.

1 Like
2

Extra loggers is the way to go here, if this doesn’t work for you or you struggle to configure it, LMK

here is an example:

prefect config set PREFECT_LOGGING_EXTRA_LOGGERS="logger1,logger2"

in your case, this would be root logger

3

Thanks for the tip. Is there any documentation on how this works? I tried to look for but didn’t really find much.

So I tried setting this configuration with the profile that is active where the agent is run:

prefect config set PREFECT_LOGGING_EXTRA_LOGGERS="rootlogger,loguru,root,root logger"

Also tried those one by one.

What else should I do?

Logs from loguru are shown in the prefect agent logs but still not in prefect cloud.

1 Like
4

loguru is a bit… special - this link is for v1, but it should still be valid for Prefect 2:

A standard Python logger set as an extra logger should work - if it doesn’t work for you with a standard Python logger, please open a bug report on GitHub

5

So apparently the issue was in the logging level.

Having set
prefect config set PREFECT_LOGGING_EXTRA_LOGGERS="root"

I noticed this:

> By default, the root logger is configured to output WARNING level logs to the console.

and realized I was able get a warning level message to show:
logging.warning(“warning test”)

And then setting the logging root level env variable made the info level messages visible also.

export PREFECT_LOGGING_ROOT_LEVEL=INFO

Thanks for the help!

1 Like
6

Thanks so much for the update! Super helpful and glad it’s working now

7

Bumping up this old thread, but I had similar issue as I had a codebase with a lot of functions with loguru loggers and are there some Prefect 3 related changes as I could not seem to get the logs show up in my Prefect UI with these 2 commands?

prefect config set PREFECT_LOGGING_EXTRA_LOGGERS="root"
export PREFECT_LOGGING_ROOT_LEVEL=INFO

Do I just change loguru to standard logging? Or is there an easier way to get “Prefect-native” logs? It is just a lot easier to program using the loguru_logger.info() type of lines “all over the place” without explicitly defining the Prefect logger within each function?