Executing tasks in parallel

Help
1

I am trying to execute 4 independent tasks, each running a notebook with papermill. Whatever I do, tasks are executed sequentially.

from prefect import flow, task
from prefect_dask.task_runners import DaskTaskRunner
from prefect.task_runners import ConcurrentTaskRunner
import papermill as pm

@task(name="Sea ice plot monthly")
def sea_ice_plot_monthly(settings, hemisphere, month):
    settings["hemisphere"] = hemisphere
    settings["month"] = month
    pm.execute_notebook(
        "ice_plot_month.ipynb",
        f"ice_plot_month_{hemisphere}_{month}.ipynb",
        parameters=settings,
    )


@task(name="Sea ice plot monthly2")
def sea_ice_plot_monthly2(settings, hemisphere, month):
    settings["hemisphere"] = hemisphere
    settings["month"] = month
    pm.execute_notebook(
        "ice_plot_month.ipynb",
        f"ice_plot_month_{hemisphere}_{month}.ipynb",
        parameters=settings,
    )

@flow(name="My Example Flow", task_runner=ConcurrentTaskRunner())
def my_flow(settings):
    a = sea_ice_plot_monthly(settings, "n", 3)
    b = sea_ice_plot_monthly2(settings, "n", 9)
    c = sea_ice_plot_monthly(settings, "s", 3)
    d = sea_ice_plot_monthly2(settings, "s", 9)

    return a, b, c, d

if __name__ == "__main__":

    status = my_flow(settings)

I’ve tried DaskTaskRunner with the same effect. Is it me missing a concept of how prefect should operate in this case (running 4 tasks in the same time)?

2

To make sure that the tasks within your flow can run concurrently or in parallel, add .submit() to your task run. This method will return a PrefectFuture instead of a Python object.
.submit().

.map() will iterate and submit and is another option for concurrent/parallel execution.

3

Thanks a lot for the answer! It indeed move me one step forward, but now I am getting:

Finished in state Failed('Task run encountered an exception: RuntimeError: This event loop is already running\n')

Looks like prefect is not really playing well with papermill :frowning: Again, thanks for pointing to the right place in the docs! :slight_smile: