How to set a custom logging formatter

Show and Tell
1

Prefect log formatters specify the format of log messages. All logging configurations are in ~/.prefect/logging.yml. You can change the setting of the logging by typing:

prefect config set PREFECT_LOGGING_[PATH]_[TO]_[KEY]=VALUE

To change the formatter to json, you can type:

prefect config set PREFECT_LOGGING_HANDLERS_CONSOLE_FORMATTER=json

prefect config set PREFECT_LOGGING_HANDLERS_CONSOLE_TASK_RUNS_FORMATTER=json

prefect config set PREFECT_LOGGING_HANDLERS_CONSOLE_FLOW_RUNS_FORMATTER=json

Or simply change the value of handlers in ~/.prefect/logging.yml

handlers:

    # The handlers we define here will output all logs they receive by default
    # but we include the `level` so it can be overridden by environment

    console:
        level: 0
        class: logging.StreamHandler
        formatter: json

    console_flow_runs:
        level: 0
        class: logging.StreamHandler
        formatter: json

    console_task_runs:
        level: 0
        class: logging.StreamHandler
        formatter: json

    orion:
        level: 0
        class: prefect.logging.handlers.OrionHandler
2 Likes
2

As of Prefect 2.6.9, all console handlers are unified into a single console handler with Rich formatting:

handlers:

    # The handlers we define here will output all logs they receieve by default
    # but we include the `level` so it can be overridden by environment

    console:
        level: 0
        class: prefect.logging.handlers.PrefectConsoleHandler
        formatter: standard
        styles:
            log.web_url: bright_blue
            log.local_url: bright_blue

            log.info_level: cyan
            log.warning_level: yellow3
            log.error_level: red3
            log.critical_level: bright_red

            log.completed_state: green
            log.cancelled_state: yellow3
            log.failed_state: red3
            log.crashed_state: bright_red

            log.flow_run_name: magenta
            log.flow_name: bold magenta

This makes it quick and easy to make Prefect’s output look exactly the way you want (See the Rich docs for more information on available styles). If you want to get started with log customization, the easiest way is to use a custom logging.yml file like Khuyen mentioned above.

Prefect’s default logging.yml file makes a great starting point. Make a copy of it and then customize it any way you’d like!

To get Prefect to use your custom logging.yml automatically, place a copy of it in your Prefect home directory. This will vary depending on your operating system:

  • /home/<your username>/.prefect on Linux
  • /Users/<your username>/.prefect on MacOS
  • C:\Users\<your username>\.prefect on Windows

Alternatively, you can keep your logging configuration elsewhere and run the following to tell Prefect where to look for logging configuration:

prefect config set PREFECT_LOGGING_SETTINGS_PATH=/path/to/your/logging.yml

Either way, next time you run a Prefect flow, it will use your custom logging configuration!

1 Like