Topics tagged parallel-processing

Topic	Replies	Views	Activity
How can I run multiple subflows (or child flows) in parallel? Workflow Orchestration migration-guide , prefect-1-0 , prefect-2-0 , flow-of-flows , parallel-processing , create_flow_run , orchestrator-pattern , subflows , async , mapping , local-dask-executor	13	2534	January 3, 2023
Edgecase Error: "Prefect Task run TASK_RUN_ID already finished" Workflow Orchestration prefect-2-0 , parallel-processing , ray , ray-task-runner	10	352	December 1, 2022
How can I map over a list of inputs? Workflow Orchestration migration-guide , prefect-1-0 , prefect-2-0 , dask , parallel-processing , dask-task-runner , mapping , local-dask-executor , getting-started , to-review	13	944	November 30, 2022
Avoiding locked database with larger flows -- recommendations for something better than a delay? Workflow Orchestration prefect-2-0 , dask , parallel-processing , dask-task-runner	7	487	October 27, 2022
Running numerous tasks on dask cluster with large data inputs/outputs causes timeout errors Distributed Computing prefect-2-0 , dask , parallel-processing , timeout	3	418	August 22, 2022
New Orion release 2.0a13 featuring work queues, agents, profiles, remote object storage and distributed parallel processing with Ray! Announcements prefect-2-0 , release-notes , storage , aws , s3 , gcp , task-runner , parallel-processing , agent , azure , concurrency-limits , kubernetes-flow-runner , filters , gcs , docker-flow-runner , profiles-prefect-config , work-queue , ray , ray-task-runner , azure-blob-storage	7	653	July 29, 2022
Prefect 2.0 example repository with many examples a.o. using Dask, Ray, subflows, parametrization and various packagers - by Oliver Mannion Tutorials & Resources prefect-2-0 , deployment , parameters , parallel-processing , user-contribution , code-dependencies , infrastructure-as-code , repository-structure , packaging , minio	0	317	July 12, 2022
RayTaskRunner freezes when writing data to a local directory Distributed Computing prefect-2-0 , parallel-processing , ray , ray-task-runner	4	368	July 5, 2022
How can I configure my flow to run with Dask? Distributed Computing migration-guide , prefect-1-0 , prefect-2-0 , dask , executor , task-runner , parallel-processing , dask-task-runner , getting-started	2	1012	July 5, 2022
How can I configure a specific Dask cluster class? Distributed Computing migration-guide , prefect-1-0 , prefect-2-0 , aws , dask , executor , task-runner , parallel-processing , fargate , dask-task-runner , dask-cloud-provider , docker-image	0	368	January 21, 2022
Prefect 2.0 UI doesn't seem to show all task runs running on Dask in real-time Distributed Computing prefect-2-0 , dask , parallel-processing , dask-task-runner	20	413	July 5, 2022
How can I parallelize execution across 8 CPU cores? Workflow Orchestration migration-guide , prefect-1-0 , prefect-2-0 , dask , executor , task-runner , parallel-processing , dask-task-runner , local-dask-executor , infrastructure , multithreading , multiprocessing	0	683	January 31, 2022
Why is my function not running in parallel even with DaskTaskRunner / RayTaskRunner? Distributed Computing prefect-2-0 , dask , task-runner , parallel-processing , dask-task-runner , dask-annotations	14	583	July 5, 2022
How can I run tasks in parallel? Workflow Orchestration migration-guide , prefect-1-0 , prefect-2-0 , dask , executor , task-runner , parallel-processing , dask-task-runner , infrastructure , multithreading , multiprocessing , basics	0	946	January 31, 2022
How to use Dask without mapping in Prefect 1.0: using Dask worker_client to call client.submit() inside a Prefect task Distributed Computing prefect-1-0 , resource-manager , dask , parallel-processing , checkpointing , mapping , resource-clean-up , dask-dataframes	13	532	May 12, 2022
How to GROUP BY in Prefect 1.0 or 2.0? Usage Questions prefect-1-0 , dask , parallel-processing	3	282	April 29, 2022
Is there any best practices in case I want to run a library that has its own parallelism? Distributed Computing prefect-1-0 , prefect-2-0 , dask , parallel-processing	0	216	April 2, 2022
What is the difference between a DaskExecutor and a LocalDaskExecutor? Distributed Computing prefect-1-0 , local-development , dask , parallel-processing , local-dask-executor , dask-executor , kube-cluster	0	1132	February 25, 2022
Tips for Debugging apply_map Workflow Orchestration prefect-1-0 , dask , parallel-processing , conditional-logic , control-flow , mapping , dask-executor , apply_map	0	202	March 10, 2022
Are there any guidelines for using a temporary vs. static Dask cluster? How to set one on Kubernetes? Distributed Computing prefect-1-0 , dask , parallel-processing , dask-executor , kube-cluster	1	497	March 10, 2022
We want to create a mapped task that retries N times and then proceeds to the downstream task if "any_successful" - Is it possible? Workflow Orchestration prefect-1-0 , parallel-processing , state-dependencies , mapping , local-dask-executor , slack-notifications , notifications , failure	0	484	March 6, 2022
Introducing Radar - by Bill Palombi Tutorials & Resources prefect-2-0 , blog-post , parallel-processing , ui , data-dependencies , radar-view , flow-visualization	0	389	March 4, 2022
Scale your Prefect + Dask Workflows to the Cloud - by Richard Pelgrim Distributed Computing prefect-1-0 , resource-manager , blog-post , dask , executor , parallel-processing , user-contribution , mapping , coiled , dask-executor	0	270	February 25, 2022
Can Prefect run multiple (distinct) mapped tasks at a time? Distributed Computing prefect-1-0 , dask , parallel-processing , mapping , local-dask-executor	0	679	February 20, 2022
How to build conditional logic within mapped tasks? Workflow Orchestration prefect-1-0 , dag-flow-structure , parallel-processing , conditional-logic , control-flow , mapping , apply_map	0	507	February 16, 2022
Introducing Prefect-ML: Orchestrate a Distributed Hyperparameter Grid Search on Dask - by Kevin Kho Tutorials & Resources prefect-1-0 , dask , parallel-processing , mapping , dask-executor , data-science-workflows , artifacts-api	0	282	February 16, 2022
New Orion release 2.0a10 including concurrency limits, ConcurrentTaskRunner, CLI refinements and improved database migrations Announcements prefect-2-0 , release-notes , cli , parameters , task-runner , parallel-processing , concurrency-limits , async , database-migrations	0	292	February 9, 2022
Scaling out with Prefect Tutorials & Resources prefect-1-0 , blog-post , dask , vertex , gcp , executor , parallel-processing , local-dask-executor	0	225	January 21, 2022

How can I run multiple subflows (or child flows) in parallel?

migration-guide , prefect-1-0 , prefect-2-0 , flow-of-flows , parallel-processing , create_flow_run , orchestrator-pattern , subflows , async , mapping , local-dask-executor

13

2534

January 3, 2023

Edgecase Error: "Prefect Task run TASK_RUN_ID already finished"

Workflow Orchestration

prefect-2-0 , parallel-processing , ray , ray-task-runner

10

352

December 1, 2022

How can I map over a list of inputs?

Workflow Orchestration

migration-guide , prefect-1-0 , prefect-2-0 , dask , parallel-processing , dask-task-runner , mapping , local-dask-executor , getting-started , to-review

13

944

November 30, 2022

Avoiding locked database with larger flows -- recommendations for something better than a delay?

Workflow Orchestration

prefect-2-0 , dask , parallel-processing , dask-task-runner

7

487

October 27, 2022

Running numerous tasks on dask cluster with large data inputs/outputs causes timeout errors

Distributed Computing

prefect-2-0 , dask , parallel-processing , timeout

3

418

August 22, 2022

New Orion release 2.0a13 featuring work queues, agents, profiles, remote object storage and distributed parallel processing with Ray!

Announcements

prefect-2-0 , release-notes , storage , aws , s3 , gcp , task-runner , parallel-processing , agent , azure , concurrency-limits , kubernetes-flow-runner , filters , gcs , docker-flow-runner , profiles-prefect-config , work-queue , ray , ray-task-runner , azure-blob-storage

7

653

July 29, 2022

Prefect 2.0 example repository with many examples a.o. using Dask, Ray, subflows, parametrization and various packagers - by Oliver Mannion

Tutorials & Resources

prefect-2-0 , deployment , parameters , parallel-processing , user-contribution , code-dependencies , infrastructure-as-code , repository-structure , packaging , minio

0

317

July 12, 2022

RayTaskRunner freezes when writing data to a local directory

Distributed Computing

prefect-2-0 , parallel-processing , ray , ray-task-runner

4

368

July 5, 2022

How can I configure my flow to run with Dask?

Distributed Computing

migration-guide , prefect-1-0 , prefect-2-0 , dask , executor , task-runner , parallel-processing , dask-task-runner , getting-started

2

1012

July 5, 2022

How can I configure a specific Dask cluster class?

Distributed Computing

migration-guide , prefect-1-0 , prefect-2-0 , aws , dask , executor , task-runner , parallel-processing , fargate , dask-task-runner , dask-cloud-provider , docker-image

0

368

January 21, 2022

Prefect 2.0 UI doesn't seem to show all task runs running on Dask in real-time

Distributed Computing

prefect-2-0 , dask , parallel-processing , dask-task-runner

20

413

July 5, 2022

How can I parallelize execution across 8 CPU cores?

Workflow Orchestration

migration-guide , prefect-1-0 , prefect-2-0 , dask , executor , task-runner , parallel-processing , dask-task-runner , local-dask-executor , infrastructure , multithreading , multiprocessing

0

683

January 31, 2022

Why is my function not running in parallel even with DaskTaskRunner / RayTaskRunner?

Distributed Computing

prefect-2-0 , dask , task-runner , parallel-processing , dask-task-runner , dask-annotations

14

583

July 5, 2022

How can I run tasks in parallel?

Workflow Orchestration

migration-guide , prefect-1-0 , prefect-2-0 , dask , executor , task-runner , parallel-processing , dask-task-runner , infrastructure , multithreading , multiprocessing , basics

0

946

January 31, 2022

How to use Dask without mapping in Prefect 1.0: using Dask worker_client to call client.submit() inside a Prefect task

Distributed Computing

prefect-1-0 , resource-manager , dask , parallel-processing , checkpointing , mapping , resource-clean-up , dask-dataframes