Avoid traceback encountered during a flow run

Archive
1

I run a prefect server/agent locally with the binaries provided by pip.

from prefect import flow, get_run_logger, task

@task
def my_task():
    logger = get_run_logger()
    logger.info("task_start")
    raise Exception("my_task failed")

@flow()
def my_flow():
    my_task()

This is all the logs from the agent:

10:05:01.959 | INFO    | Flow run 'space-tortoise' - Created task run 'my_task-c06c9343-0' for task 'my_task'
10:05:01.960 | INFO    | Flow run 'space-tortoise' - Executing 'my_task-c06c9343-0' immediately...
10:05:02.025 | INFO    | Task run 'my_task-c06c9343-0' - task_start
10:05:02.025 | ERROR   | Task run 'my_task-c06c9343-0' - Encountered exception during execution:
Traceback (most recent call last):
  File "/home/vmerat/dev/pca-automation/vpca-automation/lib/python3.8/site-packages/prefect/engine.py", line 1221, in orchestrate_task_run
    result = await run_sync(task.fn, *args, **kwargs)
  File "/home/vmerat/dev/pca-automation/vpca-automation/lib/python3.8/site-packages/prefect/utilities/asyncutils.py", line 68, in run_sync_in_worker_thread
    return await anyio.to_thread.run_sync(call, cancellable=True)
  File "/home/vmerat/dev/pca-automation/vpca-automation/lib/python3.8/site-packages/anyio/to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/home/vmerat/dev/pca-automation/vpca-automation/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "/home/vmerat/dev/pca-automation/vpca-automation/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "/tmp/tmp8lefznnwprefect/mine/v2/flow44.py", line 8, in my_task
    raise Exception("my_task failed")
Exception: my_task failed
10:05:02.069 | ERROR   | Task run 'my_task-c06c9343-0' - Finished in state Failed('Task run encountered an exception.')
10:05:02.070 | ERROR   | Flow run 'space-tortoise' - Encountered exception during execution:
Traceback (most recent call last):
  File "/home/vmerat/dev/pca-automation/vpca-automation/lib/python3.8/site-packages/prefect/engine.py", line 596, in orchestrate_flow_run
    result = await run_sync(flow_call)
  File "/home/vmerat/dev/pca-automation/vpca-automation/lib/python3.8/site-packages/prefect/utilities/asyncutils.py", line 68, in run_sync_in_worker_thread
    return await anyio.to_thread.run_sync(call, cancellable=True)
  File "/home/vmerat/dev/pca-automation/vpca-automation/lib/python3.8/site-packages/anyio/to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/home/vmerat/dev/pca-automation/vpca-automation/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "/home/vmerat/dev/pca-automation/vpca-automation/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "/tmp/tmp8lefznnwprefect/mine/v2/flow44.py", line 13, in my_flow
    my_task()
  File "/home/vmerat/dev/pca-automation/vpca-automation/lib/python3.8/site-packages/prefect/tasks.py", line 295, in __call__
    return enter_task_run_engine(
  File "/home/vmerat/dev/pca-automation/vpca-automation/lib/python3.8/site-packages/prefect/engine.py", line 736, in enter_task_run_engine
    return run_async_from_worker_thread(begin_run)
  File "/home/vmerat/dev/pca-automation/vpca-automation/lib/python3.8/site-packages/prefect/utilities/asyncutils.py", line 148, in run_async_from_worker_thread
    return anyio.from_thread.run(call)
  File "/home/vmerat/dev/pca-automation/vpca-automation/lib/python3.8/site-packages/anyio/from_thread.py", line 49, in run
    return asynclib.run_async_from_thread(func, *args)
  File "/home/vmerat/dev/pca-automation/vpca-automation/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 970, in run_async_from_thread
    return f.result()
  File "/usr/lib/python3.8/concurrent/futures/_base.py", line 444, in result
    return self.__get_result()
  File "/usr/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/home/vmerat/dev/pca-automation/vpca-automation/lib/python3.8/site-packages/prefect/engine.py", line 874, in get_task_call_return_value
    return await future._result()
  File "/home/vmerat/dev/pca-automation/vpca-automation/lib/python3.8/site-packages/prefect/futures.py", line 237, in _result
    return final_state.result(raise_on_failure=raise_on_failure)
  File "/home/vmerat/dev/pca-automation/vpca-automation/lib/python3.8/site-packages/prefect/orion/schemas/states.py", line 145, in result
    raise data
  File "/home/vmerat/dev/pca-automation/vpca-automation/lib/python3.8/site-packages/prefect/engine.py", line 1221, in orchestrate_task_run
    result = await run_sync(task.fn, *args, **kwargs)
  File "/home/vmerat/dev/pca-automation/vpca-automation/lib/python3.8/site-packages/prefect/utilities/asyncutils.py", line 68, in run_sync_in_worker_thread
    return await anyio.to_thread.run_sync(call, cancellable=True)
  File "/home/vmerat/dev/pca-automation/vpca-automation/lib/python3.8/site-packages/anyio/to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/home/vmerat/dev/pca-automation/vpca-automation/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "/home/vmerat/dev/pca-automation/vpca-automation/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "/tmp/tmp8lefznnwprefect/mine/v2/flow44.py", line 8, in my_task
    raise Exception("my_task failed")
Exception: my_task failed
10:05:02.115 | ERROR   | Flow run 'space-tortoise' - Finished in state Failed('Flow run encountered an exception.')
10:05:03.029 | INFO    | prefect.infrastructure.process - Process 'space-tortoise' exited cleanly.

Getting the traceback of my task is not a problem, it concerns my task failure.
However, the huge traceback after Flow run 'space-tortoise' - Encountered exception during execution: is not relevant.

I saw in the function orchestrate_flow_run() in engine.py that when a flow raises an exception that is not a timeout, a log is made with exc_info=True. In order to debug prefect, this is a really good thing, but it there a reason to make it the default behavior ?
In other terms, could the boolean given to exc_info be a option in conf ?

Or maybe raising an exception is not the good approach, but it’s a very pythonic way to deal with errors.

2 Likes
2

In general, you shouldn’t worry about trapping exceptions unless you want to. If you would add raise_on_failure=False, then the exception is trapped, but you can still use it downstream because it’s stored in the result object.

from prefect import flow, task


@task
def my_task():
    raise ValueError()


@flow
def my_flow():
    state = my_task(return_state=True)

    maybe_result = state.result(raise_on_failure=False)
    if isinstance(maybe_result, ValueError):
        print("Oh no! The task failed. Falling back to '1'.")
        result = 1
    else:
        result = maybe_result

    return result + 1


flow_result = my_flow()
assert flow_result == 2

So there are many ways you can handle/work with exceptions/failure without changes to the current implementation of orchestrate_flow_run() in engine.py