Orion allows your tasks to run in parallel, as long as you attach a
DaskTaskRunner. Under the hood, it deploys a local Dask cluster and parallelizes work across local threads and processes.
from prefect.task_runners import DaskTaskRunner @flow(task_runner=DaskTaskRunner())
Similarly, when you attach a
RayTaskRunner(), a local Ray cluster will be created:
from prefect.task_runners import RayTaskRunner @flow(task_runner=RayTaskRunner())
Lastly, when you don’t attach any task runner explicitly, Prefect 2.0 uses a
ConcurrentTaskRunner() by default, running your tasks concurrently.
LocalDaskExecutor parallelizes task run execution across local threads and processes.
from prefect.executors import LocalDaskExecutor with Flow("parallel_task_runs", executor=LocalDaskExecutor()) as flow:
Prefect can only parallelize actual tasks decorated with
@task. Given that Prefect 2.0 allows you to run arbitrary Python code in your flows, this distinction is important. Check this topic for more details:
and this Github issue: