How to continue flow to completion even if individual tasks raise exceptions (Prefect 2.0)

Help
1

Hi all. I am in the process of migrating an ETL app from Prefect 1.0 to Prefct 2.0. I’m running prefect==2.7.1 locally. I’ve made an initial migration and it appears to be working fine, except there is one behaviour I’d like to change. My ETL consists of a number of tasks, all of them independent from each other (yet they run sequentially, per default settings, which is ok by me).

The desired behavior is that if one task fails, all subsequent tasks continue, without halting the flow; this happened by default in Prefect 1.0 without me touching anything, but it’s not happening by default in Prefect 2.0. That is, whenever one of my tasks raises an exception, the entire flow stops. How can I replicate the behavior I had in Prefect 1.0? (which for my use case suited me better).

Example code:

from prefect import flow, task


@task
def fail_fast():
    raise Exception("Error raised for testing")


@task
def ask_question():
    print("What is your favorite number?")


@flow
def run_entry_point():
    fail_fast()
    # fail_fast().submit()  # .submit() or not makes no difference
    ask_question()
    return 42


if __name__ == "__main__":
    print(run_entry_point())

I’ve looked into this thread: How can I handle errors, allowing my flow runs to continue until complete and not raise upon failure? and quite a few others, but I just can’t manage to find what “knob” to tweak to make it happen.

Thanks in advance.

1 Like
2

Hi, check out this discourse article here. Another option would be to use try/except blocks.

3

Hi, as stated in my OP, I had already tried the thread you pointed me to; the code there doesn’t work as advertised. I ended up writing my own try/except logic.

In terms of feedback, I would have preferred some out-of-the-box solution as a @flow param setting that I could just set to True or False, especially considering I already had the behavior as default in Prefect 1.0

Thanks James.