How can I run tasks in parallel?

Prefect 2.0

Orion allows your tasks to run in parallel, as long as you attach a DaskTaskRunner. Under the hood, it deploys a local Dask cluster and parallelizes work across local threads and processes.

from prefect.task_runners import DaskTaskRunner

@flow(task_runner=DaskTaskRunner())

Similarly, when you attach a RayTaskRunner(), a local Ray cluster will be created:

from prefect.task_runners import RayTaskRunner

@flow(task_runner=RayTaskRunner())

Lastly, when you don’t attach any task runner explicitly, Prefect 2.0 uses a ConcurrentTaskRunner() by default, running your tasks concurrently.

Prefect 1.0

LocalDaskExecutor parallelizes task run execution across local threads and processes.

from prefect.executors import LocalDaskExecutor

with Flow("parallel_task_runs", executor=LocalDaskExecutor()) as flow:

Important note!

Prefect can only parallelize actual tasks decorated with @task. Given that Prefect 2.0 allows you to run arbitrary Python code in your flows, this distinction is important. Check this topic for more details:

and this Github issue:

2 Likes