Prefect 1.0 -> Prefect 2.0 Logging migration

Archive
, , ,
1

Summary

Prefect allow you to log various elements about your flow and task runs. This can help add visibility to aid in troubleshooting and additional observation.

In Prefect 1.0, logging requires additional syntax in order to create the necessary objects to facilitate the logging schema.

In Prefect 2.0, the syntax is more simplified to feel like you are developing with native python. The logging syntax is cleaner and more applicable.

References and documentation are linked down below.

Video Link

Audience
Want to see how logging has changed in Prefect 2.0 and converting their affiliated flows.

What is Different?
In Prefect 1.0, we use prefect.context to get the information of our current run. We have scrapped this syntax to instead create a logger object based on your specified need.

In Prefect 2.0, we create a logger object with get_run_logger() to define either a task or flow logger for our current run.

How to Convert from 1.0 to 2.0
Example of a Prefect 1.0 is down below.

import prefect
from prefect import task, Flow

# PREFECT 1.0 FLOW

# little more syntactual sugar in order to use Prefect's logging system
@task
def basic_flow_log():
    logger = prefect.context.get("logger")
    logger.info("Hello cloud!")

# using your own custom print statements
@task(log_stdout=True)
def log_my_stdout():
    print("Using my own custom logging sequence!")

with Flow("logging_test") as flow:
    basic_flow_log()
    log_my_stdout()


flow.run()

We are logging custom standard out outputs and adding in our own verbiage for the logger info.

The main differences to note is the flow syntax changes thus the logging schema changes to follow a more pythonic way. Also we are not restricted to just flows, as 2.0 allow us to log within tasks and flows.

How can we Refactor to work with Prefect 2.0?

  • The import for Flow is now flow
  • Import get_run_logger
  • Replace prefect.context.get("logger") with get_run_logger()

*Keep in mind, in the below example get_run_logger() is used as we want to get the logs for the current run (task or flows), but flow_run_logger and task_run_logger can be used respectively

Prefect 2.0 Flow Example

from prefect import flow, task, get_run_logger

# get_run_logger = Get a Prefect logger for the current task run or flow run

# PREFECT 2.0 FLOW

# log within a flow
@flow(name="flow_log_example")
def logger_flow():
    logger1 = get_run_logger()
    logger1.info("INFO level custom log message from only a flow.")

# log within a task
@task
def task_logger(name="task-example-task"):
    logger2 = get_run_logger()
    logger2.info("INFO level custom log message from only a task.")

# log within a flow and a task
@flow(name="task_flow_log_example")
def flow_and_task_logger():
    task_logger()
    logger3 = get_run_logger()
    logger3.info("INFO We can log custom information on both tasks & flows!")

logger_flow()
flow_and_task_logger()

What is Staying the Same?
Customizing using the environmental variables stays the same and offers additional opportunities for formatting the log statements.

For Example:
To set locally your logging schema to what the default is for your flow run logs, set
export PREFECT_LOGGING_FORMATTERS_FLOW_RUNS_FORMAT="%(asctime)s.%(msecs)03d | %(levelname)-7s | %(flow_name)s - %(message)s"

To only get the messages of your flow runs, set
export PREFECT_LOGGING_FORMATTERS_FLOW_RUNS_FORMAT="%(asctime)s.%(msecs)03d | %(message)s"

These exports follow PREFECT_LOGGING_[PATH]_[TO]_[KEY] so you can modify not only flow run logs but task run logs as well

For example:
To only get the messages of your task runs, set
export PREFECT_LOGGING_FORMATTERS_TASK_RUNS_FORMAT="%(asctime)s.%(msecs)03d | %(message)s"

Finally, within Prefect 2.0 you can modify directly the logging.yml file that is packaged with Prefect to create your own custom settings as well.

For more information on modifying logging.yml, please refer to the additional documentation below.

Links/Documentation

2 Likes