Good morning and hope to find you well.
We are orchestrating a flow through Prefect in Databricks , and we are facing a “Python Recursion Error” when feeding a dataframe from a task to a subflow. Nothe that the dataframe is read through pandas-pyspark api ( Pandas API on Spark — PySpark 3.5.0 documentation (apache.org) ).
Note also that when changing that subflow to a task , it runs perfectly. But we cannot do that because we have other tasks running under that subflow which needs to be flow otherwise we get the error of we cannot run a task inside of a task.
Would be grateful for your help and thanks in advance.