Can I use loguru logs in Prefect flows?

[anna_geller]

At the time of writing, there is no built-in integration to incorporate loguru logs in your Prefect flow.

Best practices: We recommend using the Prefect logger as much as it’s possible - this way, your logs will be automatically captured in the Prefect’s backend. This topic explains it in more detail:

How can I add logs to my flow?

Prefect 2.0 Orion provides the get_run_logger function that returns a logging.Logger object. Retrieve the logger within the flow or task function and use it there. Here is how you can use it in your tasks, flows, and subflows: from prefect import task, get_run_logger @task def marvins_favorite_task(): logger = get_run_logger() logger.info("Hello from the Marvin's favorite task!")

An example flow with various log levels

from prefect import flow, task, get_run_logger @task def foo(): …

However, if you wish to use loguru in your custom modules and functions, you may add Prefect’s log handlers as loguru sink. Here is the syntax for Prefect <= 1.0 that you may try:

import prefect
from loguru import logger
from prefect import task, Flow
from prefect.utilities.logging import get_logger


def custom_non_task_function():
    prefect_logger = get_logger()
    stream_handler, cloud_handler = prefect_logger.handlers
    logger.add(sink=stream_handler)
    logger.add(sink=cloud_handler)
    logger.info("Logging from loguru!")


@task(log_stdout=True)
def task_printing_stdout():
    print("Logging by printing to stdout!")


@task
def logging_task():
    prefect_logger = prefect.context.get("logger")
    prefect_logger.info("Logging from Prefect root logger retrieved from context")
    custom_non_task_function()


with Flow("loguru_flow") as flow:
    hw = task_printing_stdout()
    logging_task(upstream_tasks=[hw])

This results in the following output in the Prefect Cloud UI:

Using loguru with DaskExecutor

import prefect
from loguru import logger
from prefect import task, Flow
from prefect.utilities.logging import get_logger
from prefect.executors import DaskExecutor


def custom_non_task_function():
    prefect_logger = get_logger()
    stream_handler, cloud_handler = prefect_logger.handlers
    logger.add(sink=stream_handler)
    logger.add(sink=cloud_handler)
    logger.info("Logging from loguru!")


@task(log_stdout=True)
def task_printing_stdout():
    print("Logging by printing to stdout!")


@task
def logging_task():
    prefect_logger = prefect.context.get("logger")
    prefect_logger.info("Logging from Prefect root logger retrieved from context")
    custom_non_task_function()


with Flow("loguru_flow", executor=DaskExecutor()) as flow:
    hw = task_printing_stdout()
    logging_task(upstream_tasks=[hw])

Again, the output from Prefect Cloud UI:

This behavior was tested using Prefect 0.15.13.

Note that: the logs may not be available in the UI if you are using a remote distributed Dask cluster. For more about that, see:

Why are the Dask logs not being shown in the Prefect UI?

Independent of Prefect, using Dask on a cluster does not capture worker logs and send them back to the scheduler. The LocalDaskExecutor will show the logs in the Prefect UI because the LocalDaskExecutor just uses dask while the DaskExecutor uses distributed . On the Prefect side, the native Python logger gets serialized and then sent to Dask workers. When this Prefect logger gets deserialized, it loses the configuration attached to it, thus the logs are not directed to Prefect Cloud. There are…

[anna_geller]

From Slack discussion

Confirmation that this approach works when using a KubeCluster

View in #prefect-community on Slack

@Anna_Geller: @Chris_Arderne I looked a bit into loguru logs and wrote down what I’ve found so far in this Discourse topic - not sure if this works with a remote distributed Dask cluster but using a local Dask Cluster seemed to work - LMK if you can’t reproduce.

@Chris_Arderne: Confirming that it works on a distributed KubeCluster. Only difference from what I tried before is to move the stream_handler stuff inside the called function, which is pretty obvious in hindsight!

@Anna_Geller: Amazing and thanks for confirming that! I will add that insight to the topic