How do I suppress "Created task run" logs?

Archive
, ,
9

Here’s a better way, taken from Feature Request: Suppression of "task created" logs and UI log limitations · Issue #5952 · PrefectHQ/prefect · GitHub

After digging into logging more, I discovered the logging.yml file is pretty powerful, and I was able to remove “Created task run”, without dropping other logs!

Here’s my logging.yml file that I placed in ~/.prefect/logging.yml

# Prefect logging config file.
#
# Any item in this file can be overridden with an environment variable:
#    `PREFECT_LOGGING_[PATH]_[TO]_[KEY]=VALUE`
#
# Templated values can be used to insert values from the Prefect settings at runtime.

version: 1
disable_existing_loggers: False

formatters:
    simple:
        format: "%(asctime)s.%(msecs)03d | %(message)s"
        datefmt: "%H:%M:%S"

    standard:
        format: "%(asctime)s.%(msecs)03d | %(levelname)-7s | %(name)s - %(message)s"
        datefmt: "%H:%M:%S"

    flow_runs:
        format: "%(asctime)s.%(msecs)03d | %(levelname)-7s | Flow run %(flow_run_name)r - %(message)s"
        datefmt: "%H:%M:%S"

    task_runs:
        format: "%(asctime)s.%(msecs)03d | %(levelname)-7s | Task run %(task_run_name)r - %(message)s"
        datefmt: "%H:%M:%S"

    json:
        class: prefect.logging.formatters.JsonFormatter
        format: "default"

filters:
    remove_created_task_run:
        (): __main__.RemoveCreatedTaskRun

handlers:

    # The handlers we define here will output all logs they receieve by default
    # but we include the `level` so it can be overridden by environment

    console:
        level: 0
        class: logging.StreamHandler
        formatter: standard
        filters: [remove_created_task_run]

    console_flow_runs:
        level: 0
        class: logging.StreamHandler
        formatter: flow_runs
        filters: [remove_created_task_run]

    console_task_runs:
        level: 0
        class: logging.StreamHandler
        formatter: task_runs
        filters: [remove_created_task_run]

    orion:
        level: 0
        class: prefect.logging.handlers.OrionHandler

loggers:
    prefect:
        level: "${PREFECT_LOGGING_LEVEL}"
        handlers: [console]
        propagate: no

    prefect.extra:
        level: "${PREFECT_LOGGING_LEVEL}"
        handlers: [orion, console]
        propagate: no

    prefect.flow_runs:
        level: NOTSET
        handlers: [orion, console_flow_runs]
        propagate: no

    prefect.task_runs:
        level: NOTSET
        handlers: [orion, console_task_runs]
        propagate: no

    prefect.orion:
        level: "${PREFECT_LOGGING_SERVER_LEVEL}"

    uvicorn:
        level: "${PREFECT_LOGGING_SERVER_LEVEL}"
        handlers: [console]
        propagate: no

    fastapi:
        level: "${PREFECT_LOGGING_SERVER_LEVEL}"
        handlers: [console]
        propagate: no

# The root logger: any logger without propagation disabled sends to here as well
root:
    # By default, we display warning level logs from any library in the console
    # to match Python's default behavior while formatting logs nicely
    level: WARNING
    handlers: [console]

Here’s the Python code:

import logging
from prefect import flow, task, get_run_logger


class RemoveCreatedTaskRun(logging.Filter):
    def filter(self, record):
        return 'Created task run' not in record.msg


@flow
def test_flow():
    test_task()


@task
def test_task():
    logger = get_run_logger()
    logger.info("abc")


test_flow()

Here’s the output!

13:00:39.210 | INFO    | prefect.engine - Created flow run 'turquoise-leech' for flow 'test-flow'
13:00:39.211 | INFO    | Flow run 'turquoise-leech' - Using task runner 'ConcurrentTaskRunner'
13:00:39.800 | INFO    | Task run 'test_task-c60a5fe2-0' - abc
13:00:39.931 | INFO    | Task run 'test_task-c60a5fe2-0' - Finished in state Completed()
13:00:40.056 | INFO    | Flow run 'turquoise-leech' - Finished in state Completed('All states completed.')
Completed(message='All states completed.', type=COMPLETED, result=[Completed(message=None, type=COMPLETED, result=None, task_run_id=f448446f-ffd9-48a4-b2bf-b1ab4533af54)], flow_run_id=c189df09-d06d-42d4-90e3-cbe357a7f63d)

The possibilities are endless!

class RemoveFinishedInState(logging.Filter):
    def filter(self, record):
        return 'Finished in state' not in record.msg

    remove_finished_in_state:
        (): __main__.RemoveFinishedInState
     ...
     filters: [remove_created_task_run, remove_finished_in_state]

References:
Prefect Logging: Logging - Prefect 2.0
Logging YAML: logging.config — Logging configuration — Python 3.10.5 documentation
Filters: Logging in python with YAML and filter - Stack Overflow

Hope this helps!

2 Likes