I’m trying to understand my prefect flow memory allocation between tasks and sub flows.
I’m reading data from a REST API (task 1), and saving a backup to my local disk (task 2), and then I run a task to check if the REST API result is not null (task 3), then I do a transformation (task 4) and I finally load to a database (task 5).
I load my entire result object between tasks. And I am finding that my memory usage is quite high.
Does Prefect store the parsed variables in memory as a duplicate? Or is this something typical of Python in general?
The strange thing is that my task 3 does not parse the result object, but rather just the length of the object, so it should be a really quick function to run, but it takes really long to run too. So I’m not sure what caching or parsing is going on.
Is there a way to reduce the memory or processor requirements to speed up that task? Is there some documentation I’m missing?