Prefect and Kedro Integration Logging

Show and Tell
, ,
1

Hi , I am using Prefect 2 instead of Airflow to run Kedro pipelines. The problem I have is with logging. I want to log anything in kedro in my UI console. I have tried PREFECT_LOGGING_EXTRA_LOGGERS="[kedro]" but it has no effect, and the get_run_logger is a specific Prefect Adaptor.

1 Like
2

I am having the same issue here. I did find a work around though.

In Kedro you can have a custom logging.yml (see Project-side logging configuration). What I did to make the Kedro logs appear in Prefect was to add a new handler in the handlers list:

  prefect_handler:
    class:  prefect.logging.handlers.APILogHandler
    level: INFO
    formatter: simple

And then you just also have to modify the root handlers by adding this new prefect_handler:

root:
  handlers: [rich, info_file_handler, error_file_handler, prefect_handler]

I hope this helps!