Currently, I developing and running many flows that process large amounts of data. When I run these flows, they store a large amount of data in the storage folder within the local file system, because that is the default. However, these files quickly fill up my memory, and I frequently find myself needing to manually delete them.
Is there a way to get the flows and tasks to stop persisting the data? Is it possible to have the flow clear any remaining meta data after the run is over, even when the flow fails? I have looked through the documentation on Prefect settings, but have not been able to find anything like this yet.