STDOUT not logged with log_prints=True

Help
1

Hi everyone,

I am just starting to use Prefect (version 2.10.4) and my goal is to simply have the same logging behavior as with usual Python execution when loggers print to stdout (or stderr).

It seems like there have been several discussions in the past that lead to the “log_prints” option. Now it does work for “print”, but no other use case. This can be tested simply by using stdout write directly:

@flow(
    log_prints=True
)
def test_log() -> None:
    sys.stdout.write('STDOUT is shown')
    sys.stdout.flush()
    print('PRINTS are shown')

Which when run only shows “PRINTS are shown”

Any ideas on how I can get Prefect to log all outputs to stdout? (such as other loggers)

2

Update: I tried a stupid hack to have the loggers in the package use the print function instead via adding a PrintHandler:

class PrintHandler(logging.Handler):
    """Handler to use print instead of stdout stream.
    
    This is needed for Prefect to properly capture the logs
    """
    @override
    def emit(self, record):
        print(self.format(record))

Even though in a pure Python function the handler gets called correctly, somehow the logging config is overwritten when it’s run within a Prefect context.

As a workaround, for the time being, I decided to manually overwrite the loggers from the packages I am using, which is of course not ideal - really happy to hear if I did a thinking error?

custom_package.logger = get_run_loger()
custom_package.function()
3

Hey Max! Thanks for posting. The log_prints flow decorator kwarg is working as intended in this case (see source code here). If log_prints = True, we only patch the builtin print with a Prefect logger. It’s only meant to log print statements. Feel free to open a feature request and share it here as well, we’d be happy to take a look at it!

4

@Bianca_Hoch I have been thinking further about this issue and in the end what we need is the ability to:

  • Somehow pipe the output of any subprocess to Prefect loggers
  • Have a logging config for our own packages we can easily adapt so all the logs get forwarded to Prefect (without changing the source code of course…)

There must be some best practices around these needs?

1 Like