How can I parallelize execution across 8 CPU cores?

Prefect 2.0

You can add a Dask annotation specifying the desired number of threads or processes:

import dask
from prefect import flow
from prefect.task_runners import DaskTaskRunner

@flow(task_runner=DaskTaskRunner())
def my_flow():
    with dask.annotate(resources={'process': 8}):

Prefect 1.0

You can specify the desired number of threads or processes on your LocalDaskExecutor.

flow.executor = LocalDaskExecutor(scheduler="processes", num_workers=8)

Additional helpful docs

Related topics:

1 Like