Latest Distributed Computing topics

Topic	Replies	Views	Activity
About the Distributed Computing category	0	253	January 21, 2022
Error trying to run Dask job on AKS prefect-2-0 , kubernetes , dask , troubleshooting	1	22	March 29, 2023
MissingResult: State data is missing. (DaskTaskRunner) prefect-2-0	5	243	February 17, 2023
Tasks are not running in parallel when using DaskTaskRunner() and returning state on the task.submit() call prefect-2-0 , dask-task-runner	3	125	January 12, 2023
Error when using local DaskTaskRunner - Failed to serialize cannot pickle 'SSLContext' object prefect-2-0	1	123	January 8, 2023
Using Dask Dataframes in prefect 2 tasks prefect-2-0 , dask-dataframes	1	368	September 30, 2022
Running numerous tasks on dask cluster with large data inputs/outputs causes timeout errors prefect-2-0 , dask , parallel-processing , timeout	3	420	August 22, 2022
After prefect>=2.0b9 to run in parallel, calling the submit method is required! prefect-2-0 , dask , dask-task-runner , ray , ray-task-runner	0	214	July 20, 2022
ConcurrentTaskRunner is blocking. Why and how to debug?	2	146	July 7, 2022
RayTaskRunner freezes when writing data to a local directory prefect-2-0 , parallel-processing , ray , ray-task-runner	4	368	July 5, 2022
How can I configure my flow to run with Dask? migration-guide , prefect-1-0 , prefect-2-0 , dask , executor , task-runner , parallel-processing , dask-task-runner , getting-started	2	1015	July 5, 2022
How can I configure a specific Dask cluster class? migration-guide , prefect-1-0 , prefect-2-0 , aws , dask , executor , task-runner , parallel-processing , fargate , dask-task-runner , dask-cloud-provider , docker-image	0	370	January 21, 2022
Prefect 2.0 UI doesn't seem to show all task runs running on Dask in real-time prefect-2-0 , dask , parallel-processing , dask-task-runner	20	415	July 5, 2022
Why is my function not running in parallel even with DaskTaskRunner / RayTaskRunner? prefect-2-0 , dask , task-runner , parallel-processing , dask-task-runner , dask-annotations	14	590	July 5, 2022
How can I force memory to be released at the end of a dask task? prefect-2-0 , dask , dask-task-runner , cpu-memory-resource-allocation	4	546	July 5, 2022
PicklingError error when using a DaskTaskRunner prefect-2-0 , pickle , dask-task-runner	1	297	July 5, 2022
Why are the Dask logs not being shown in the Prefect UI? prefect-1-0 , prefect-2-0 , logging , kubernetes , dask , ui , dask-task-runner	3	928	June 21, 2022
How to use Dask without mapping in Prefect 1.0: using Dask worker_client to call client.submit() inside a Prefect task prefect-1-0 , resource-manager , dask , parallel-processing , checkpointing , mapping , resource-clean-up , dask-dataframes	13	533	May 12, 2022
How to use Prefect together with an EMR cluster on AWS? prefect-1-0 , dag-flow-structure , aws , resource-manager , emr , spark	0	519	February 7, 2022
Why are my mapped tasks running out of expected order when using multiple levels of mapped tasks? prefect-1-0 , state-dependencies , mapping , local-dask-executor , upstream_tasks , unmapped	0	194	April 12, 2022
How can I change the number of Dask workers in a DaskExecutor based on a custom parameter value? prefect-1-0 , parameters , dask , executor , dynamicism	1	383	April 12, 2022
Is there any best practices in case I want to run a library that has its own parallelism? prefect-1-0 , prefect-2-0 , dask , parallel-processing	0	219	April 2, 2022
How to pass custom worker resource annotations to Dask scheduler when using DaskExecutor? prefect-1-0 , dask , dask-executor , cpu-memory-resource-allocation	0	314	April 1, 2022
What is the difference between a DaskExecutor and a LocalDaskExecutor? prefect-1-0 , local-development , dask , parallel-processing , local-dask-executor , dask-executor , kube-cluster	0	1147	February 25, 2022
How does Prefect send work to a DaskExecutor and handle memory? prefect-1-0 , dask , dask-executor , oom-error	0	393	March 10, 2022
Are there any guidelines for using a temporary vs. static Dask cluster? How to set one on Kubernetes? prefect-1-0 , dask , parallel-processing , dask-executor , kube-cluster	1	501	March 10, 2022
Scale your Prefect + Dask Workflows to the Cloud - by Richard Pelgrim prefect-1-0 , resource-manager , blog-post , dask , executor , parallel-processing , user-contribution , mapping , coiled , dask-executor	0	270	February 25, 2022
How does Prefect Cloud Task Version Locking work when using Dask executor? prefect-1-0 , cloud-1-0 , dask , dask-executor , version-locking	0	306	February 21, 2022
Can Prefect run multiple (distinct) mapped tasks at a time? prefect-1-0 , dask , parallel-processing , mapping , local-dask-executor	0	689	February 20, 2022
How to allocate more memory or more worker nodes on a per flow run basis? prefect-1-0 , storage , run_config , kubernetes , dask , executor , kubernetes-run , coiled , dask-executor , memory_request , cpu_request	0	356	January 31, 2022

About the Distributed Computing category

0

253

January 21, 2022

Error trying to run Dask job on AKS

1

22

March 29, 2023

MissingResult: State data is missing. (DaskTaskRunner)

prefect-2-0

5

243

February 17, 2023

Tasks are not running in parallel when using DaskTaskRunner() and returning state on the task.submit() call

prefect-2-0 , dask-task-runner

3

125

January 12, 2023

Error when using local DaskTaskRunner - Failed to serialize cannot pickle 'SSLContext' object

prefect-2-0

1

123

January 8, 2023

Using Dask Dataframes in prefect 2 tasks

prefect-2-0 , dask-dataframes

1

368

September 30, 2022

Running numerous tasks on dask cluster with large data inputs/outputs causes timeout errors

prefect-2-0 , dask , parallel-processing , timeout

3

420

August 22, 2022

After prefect>=2.0b9 to run in parallel, calling the submit method is required!

prefect-2-0 , dask , dask-task-runner , ray , ray-task-runner

0

214

July 20, 2022

ConcurrentTaskRunner is blocking. Why and how to debug?

2

146

July 7, 2022

RayTaskRunner freezes when writing data to a local directory

prefect-2-0 , parallel-processing , ray , ray-task-runner

4

368

July 5, 2022

How can I configure my flow to run with Dask?

migration-guide , prefect-1-0 , prefect-2-0 , dask , executor , task-runner , parallel-processing , dask-task-runner , getting-started

2

1015

July 5, 2022

How can I configure a specific Dask cluster class?

migration-guide , prefect-1-0 , prefect-2-0 , aws , dask , executor , task-runner , parallel-processing , fargate , dask-task-runner , dask-cloud-provider , docker-image

0

370

January 21, 2022

Prefect 2.0 UI doesn't seem to show all task runs running on Dask in real-time

prefect-2-0 , dask , parallel-processing , dask-task-runner

20

415

July 5, 2022

Why is my function not running in parallel even with DaskTaskRunner / RayTaskRunner?

prefect-2-0 , dask , task-runner , parallel-processing , dask-task-runner , dask-annotations

14

590

July 5, 2022

How can I force memory to be released at the end of a dask task?

prefect-2-0 , dask , dask-task-runner , cpu-memory-resource-allocation

4

546

July 5, 2022

PicklingError error when using a DaskTaskRunner

prefect-2-0 , pickle , dask-task-runner

1

297

July 5, 2022

Why are the Dask logs not being shown in the Prefect UI?

prefect-1-0 , prefect-2-0 , logging , kubernetes , dask , ui , dask-task-runner

3

928

June 21, 2022

How to use Dask without mapping in Prefect 1.0: using Dask worker_client to call client.submit() inside a Prefect task

prefect-1-0 , resource-manager , dask , parallel-processing , checkpointing , mapping , resource-clean-up , dask-dataframes

13

533

May 12, 2022

How to use Prefect together with an EMR cluster on AWS?

prefect-1-0 , dag-flow-structure , aws , resource-manager , emr , spark

0

519

February 7, 2022

Why are my mapped tasks running out of expected order when using multiple levels of mapped tasks?

prefect-1-0 , state-dependencies , mapping , local-dask-executor , upstream_tasks , unmapped

0

194

April 12, 2022

How can I change the number of Dask workers in a DaskExecutor based on a custom parameter value?

prefect-1-0 , parameters , dask , executor , dynamicism

1

383

April 12, 2022

Is there any best practices in case I want to run a library that has its own parallelism?

prefect-1-0 , prefect-2-0 , dask , parallel-processing

0

219

April 2, 2022

How to pass custom worker resource annotations to Dask scheduler when using DaskExecutor?

prefect-1-0 , dask , dask-executor , cpu-memory-resource-allocation

0

314

April 1, 2022

What is the difference between a DaskExecutor and a LocalDaskExecutor?

prefect-1-0 , local-development , dask , parallel-processing , local-dask-executor , dask-executor , kube-cluster

0

1147

February 25, 2022

How does Prefect send work to a DaskExecutor and handle memory?

prefect-1-0 , dask , dask-executor , oom-error

0

393

March 10, 2022

Are there any guidelines for using a temporary vs. static Dask cluster? How to set one on Kubernetes?

prefect-1-0 , dask , parallel-processing , dask-executor , kube-cluster

1

501

March 10, 2022

Scale your Prefect + Dask Workflows to the Cloud - by Richard Pelgrim

prefect-1-0 , resource-manager , blog-post , dask , executor , parallel-processing , user-contribution , mapping , coiled , dask-executor

0

270

February 25, 2022

How does Prefect Cloud Task Version Locking work when using Dask executor?

prefect-1-0 , cloud-1-0 , dask , dask-executor , version-locking

0

306

February 21, 2022

Can Prefect run multiple (distinct) mapped tasks at a time?

prefect-1-0 , dask , parallel-processing , mapping , local-dask-executor

0

689

February 20, 2022

How to allocate more memory or more worker nodes on a per flow run basis?

prefect-1-0 , storage , run_config , kubernetes , dask , executor , kubernetes-run , coiled , dask-executor , memory_request , cpu_request

0

356

January 31, 2022

Distributed Computing

Don't Panic

Read our Privacy Policy