How to scale out Prefect 2.0 flows with Dask on Kubernetes using KubeCluster

This Discourse topic is waiting for a contributor! Add your content as a comment, and we’ll review and consolidate it.

Thanks for helping us grow this knowledge base! :heart:

Explanation

Using DaskTaskRunner with KubeCluster creates an on-demand Kubernetes cluster. This will spin up a pod for the flow run and additional task run pods created by Dask.

Resources you may use: