I’m wondering if there is a way for prefect to grab the logs from the default python logging library:
Say I have the code:
import sys
import prefect
from prefect import flow, task, get_run_logger
from utilities import AN_IMPORTED_MESSAGE
import logging
logging.basicConfig(format='%(asctime)s | %(levelname)s : %(message)s',
level=logging.INFO, stream=sys.stdout)
@task
def log_task(name):
logger = get_run_logger()
logger.info("Hello %s!", name)
logger.info("Prefect Version = %s 🚀", prefect.__version__)
logger.debug(AN_IMPORTED_MESSAGE)
print("This is a print")
logging.info("This is python's default logging")
@flow(log_prints=True)
def log_flow(name: str):
log_task(name)
if __name__ == "__main__":
name = sys.argv[1]
log_flow(name)
Is there any way to capture the output from the following line?
logging.info("This is python's default logging")
Something equivalent to logging the prints when we add the log_prints=True arguments but with the logging module.
FYI The output from the task looks like this:
13:04:33.953 | INFO | Flow run 'jasmine-copperhead' - Created task run 'log_task-82fbd1c0-0' for task 'log_task'
13:04:33.954 | INFO | Flow run 'jasmine-copperhead' - Executing 'log_task-82fbd1c0-0' immediately...
13:04:33.982 | DEBUG | Task run 'log_task-82fbd1c0-0' - Beginning execution...
13:04:33.982 | INFO | Task run 'log_task-82fbd1c0-0' - Hello 123!
13:04:33.983 | INFO | Task run 'log_task-82fbd1c0-0' - Prefect Version = 2.7.0 🚀
13:04:33.983 | DEBUG | Task run 'log_task-82fbd1c0-0' - Hello from another file
13:04:33.984 | INFO | Task run 'log_task-82fbd1c0-0' - This is a print
13:04:34.001 | INFO | Task run 'log_task-82fbd1c0-0' - Finished in state Completed()
13:04:34.028 | DEBUG | prefect.task_runner.concurrent - Shutting down task runner...
1 Like
So turns out all I had to do was set my personal logger to propagate! Basically created mylogger object and made sure that it was set to propagate.
Honestly figured it out because there was ONE line here which was a bit vague but hey managed to get there in the end: Logging - Prefect 2 - Coordinating the world's dataflows
Full code:
import sys
import prefect
from prefect import flow, task, get_run_logger
from utilities import AN_IMPORTED_MESSAGE
import logging
logging.basicConfig(format='%(asctime)s | %(levelname)s : %(message)s',
level=logging.INFO, stream=sys.stdout)
mylogger = logging.getLogger("My Personal")
mylogger.propagate = True
mylogger.setLevel(logging.DEBUG)
@task
def log_task(name):
logger = get_run_logger()
logger.info("Hello %s!", name)
logger.info("Prefect Version = %s 🚀", prefect.__version__)
logger.debug(AN_IMPORTED_MESSAGE)
print("This is a print")
mylogger.info("This is python's default logging")
@flow(log_prints=False)
def log_flow(name: str):
log_task(name)
if __name__ == "__main__":
name = sys.argv[1]
log_flow(name)
It seems I celebrated prematurely 
In Orion UI, the logs seems to be missing, but in the console it’s all there? Not sure if this is a bug or not…
As you can see below, the UI is missing the 'My Personal - This is python's default logging' line
16:37:29.344 | DEBUG | Flow run 'cheerful-petrel' - Beginning execution...
16:37:29.367 | INFO | Flow run 'cheerful-petrel' - Created task run 'log_task-82fbd1c0-0' for task 'log_task'
16:37:29.367 | INFO | Flow run 'cheerful-petrel' - Executing 'log_task-82fbd1c0-0' immediately...
16:37:29.396 | DEBUG | Task run 'log_task-82fbd1c0-0' - Beginning execution...
16:37:29.397 | INFO | Task run 'log_task-82fbd1c0-0' - Hello 1234!
16:37:29.397 | INFO | Task run 'log_task-82fbd1c0-0' - Prefect Version = 2.7.0 🚀
16:37:29.398 | DEBUG | Task run 'log_task-82fbd1c0-0' - Hello from another file
This is a print
16:37:29.398 | INFO | My Personal - This is python's default logging
16:37:29.415 | INFO | Task run 'log_task-82fbd1c0-0' - Finished in state Completed()
16:37:29.436 | DEBUG | prefect.task_runner.concurrent - Shutting down task runner...
16:37:29.436 | INFO | Flow run 'cheerful-petrel' - Finished in state Completed('All states completed.')
1 Like
Great progress. You would need to add that custom logger into your settings as PREFECT_EXTRA_LOGGERS and if you need more examples, check out the topics tagged with logging or extra loggers, or the logging documentation. This setting should be everything you need
