Hi,
we are trying to use prefect in order to monitor Flows that are dynamically composed using the class Flow. Inside a Flow we dynamically create and submit Tasks to a ConcurrentTaskRunner.
These Flows are then used as normal Callables and run in parallel. We do not use Deployments nor serve these Flows, we just run them locally, a remote prefect instance is used to query their status.
If, say, 3 Flows are run on 3 different OS threads at the same time, and some of them call the same Tasks the outcome will likely be that one among these Flows completes seamlessly. The others simply crash at some point during their execution. The log message associated is, for example:
ERROR | Task run 'B-2' - Crash detected! Execution was cancelled by the runtime environment.
I found some GitHub issues wich seam quite eloquent this or this. Anyone have some workaround in mind?
Thanks a lot