How can I parallelize execution across 8 CPU cores?

[anna_geller]

Prefect 2.0

You can add a Dask annotation specifying the desired number of threads or processes:

import dask
from prefect import flow
from prefect_dask import DaskTaskRunner

@flow(task_runner=DaskTaskRunner())
def my_flow():
    with dask.annotate(resources={'process': 8}):

Prefect 1.0

You can specify the desired number of threads or processes on your LocalDaskExecutor.

flow.executor = LocalDaskExecutor(scheduler="processes", num_workers=8)

Additional helpful docs

Parallelism within a Prefect flow | Prefect Docs

Executors | Prefect Docs

Related topics:

How can I configure my flow to run with Dask?

Additionally, if you want to use Dask dataframes in your flow, check out this issue:

What is the difference between a DaskExecutor and a LocalDaskExecutor?

DaskExecutor vs LocalDaskExecutor in general In general, the main difference between those two is the choice of scheduler. The LocalDaskExecutor is configurable to use either threads or processes as a scheduler. In contrast, the DaskExecutor uses the Dask Distributed scheduler. This way, LocalDaskExecutor can effectively provide parallelism and help with multithreading/multiprocessing out of the box in a very lightweight way, but at the same time, it can only run on a single machine and is limi…