Here’s a better way, taken from Feature Request: Suppression of "task created" logs and UI log limitations · Issue #5952 · PrefectHQ/prefect · GitHub
After digging into logging more, I discovered the logging.yml file is pretty powerful, and I was able to remove “Created task run”, without dropping other logs!
Here’s my logging.yml file that I placed in ~/.prefect/logging.yml
# Prefect logging config file.
#
# Any item in this file can be overridden with an environment variable:
# `PREFECT_LOGGING_[PATH]_[TO]_[KEY]=VALUE`
#
# Templated values can be used to insert values from the Prefect settings at runtime.
version: 1
disable_existing_loggers: False
formatters:
simple:
format: "%(asctime)s.%(msecs)03d | %(message)s"
datefmt: "%H:%M:%S"
standard:
format: "%(asctime)s.%(msecs)03d | %(levelname)-7s | %(name)s - %(message)s"
datefmt: "%H:%M:%S"
flow_runs:
format: "%(asctime)s.%(msecs)03d | %(levelname)-7s | Flow run %(flow_run_name)r - %(message)s"
datefmt: "%H:%M:%S"
task_runs:
format: "%(asctime)s.%(msecs)03d | %(levelname)-7s | Task run %(task_run_name)r - %(message)s"
datefmt: "%H:%M:%S"
json:
class: prefect.logging.formatters.JsonFormatter
format: "default"
filters:
remove_created_task_run:
(): __main__.RemoveCreatedTaskRun
handlers:
# The handlers we define here will output all logs they receieve by default
# but we include the `level` so it can be overridden by environment
console:
level: 0
class: logging.StreamHandler
formatter: standard
filters: [remove_created_task_run]
console_flow_runs:
level: 0
class: logging.StreamHandler
formatter: flow_runs
filters: [remove_created_task_run]
console_task_runs:
level: 0
class: logging.StreamHandler
formatter: task_runs
filters: [remove_created_task_run]
orion:
level: 0
class: prefect.logging.handlers.OrionHandler
loggers:
prefect:
level: "${PREFECT_LOGGING_LEVEL}"
handlers: [console]
propagate: no
prefect.extra:
level: "${PREFECT_LOGGING_LEVEL}"
handlers: [orion, console]
propagate: no
prefect.flow_runs:
level: NOTSET
handlers: [orion, console_flow_runs]
propagate: no
prefect.task_runs:
level: NOTSET
handlers: [orion, console_task_runs]
propagate: no
prefect.orion:
level: "${PREFECT_LOGGING_SERVER_LEVEL}"
uvicorn:
level: "${PREFECT_LOGGING_SERVER_LEVEL}"
handlers: [console]
propagate: no
fastapi:
level: "${PREFECT_LOGGING_SERVER_LEVEL}"
handlers: [console]
propagate: no
# The root logger: any logger without propagation disabled sends to here as well
root:
# By default, we display warning level logs from any library in the console
# to match Python's default behavior while formatting logs nicely
level: WARNING
handlers: [console]
Here’s the Python code:
import logging
from prefect import flow, task, get_run_logger
class RemoveCreatedTaskRun(logging.Filter):
def filter(self, record):
return 'Created task run' not in record.msg
@flow
def test_flow():
test_task()
@task
def test_task():
logger = get_run_logger()
logger.info("abc")
test_flow()
Here’s the output!
13:00:39.210 | INFO | prefect.engine - Created flow run 'turquoise-leech' for flow 'test-flow'
13:00:39.211 | INFO | Flow run 'turquoise-leech' - Using task runner 'ConcurrentTaskRunner'
13:00:39.800 | INFO | Task run 'test_task-c60a5fe2-0' - abc
13:00:39.931 | INFO | Task run 'test_task-c60a5fe2-0' - Finished in state Completed()
13:00:40.056 | INFO | Flow run 'turquoise-leech' - Finished in state Completed('All states completed.')
Completed(message='All states completed.', type=COMPLETED, result=[Completed(message=None, type=COMPLETED, result=None, task_run_id=f448446f-ffd9-48a4-b2bf-b1ab4533af54)], flow_run_id=c189df09-d06d-42d4-90e3-cbe357a7f63d)
The possibilities are endless!
class RemoveFinishedInState(logging.Filter):
def filter(self, record):
return 'Finished in state' not in record.msg
remove_finished_in_state:
(): __main__.RemoveFinishedInState
...
filters: [remove_created_task_run, remove_finished_in_state]
References:
Prefect Logging: Logging - Prefect 2.0
Logging YAML: logging.config — Logging configuration — Python 3.10.5 documentation
Filters: Logging in python with YAML and filter - Stack Overflow
Hope this helps!
