Is there a limit to how much data can be passed between tasks in prefect?

Archive
1

I am sure this has been asked before but I couldn’t find it anywhere… Is there a limit (or rule of thumb) to how much data you can pass between prefect tasks?

I have always thought you weren’t supposed to pass large amounts of data (e.g. dataframes) between tasks, which I am assuming I read somewhere in the docs at one point or another. I was trying to find where I read this and couldn’t find it again. We are using prefect 1.0; not sure if this has changed in prefect 2.0.

1 Like
2

There is no limit - as much as your execution layer can handle. No data is stored on the Prefect side due to the hybrid model:

that’s fine between tasks, it’s hard between subflows though

3

Got it, thank @anna_geller !

1 Like