I’m encountering an issue where the parent flow crashes immediately after the subflow is completed. I am using run_deployment
to execute the subflow, and while the subflow finishes successfully, the parent flow transitions to a crashed state right after the subflow completion.
The following error is logged:
Process for flow run 'aspiring-cockatoo' exited with status code: -9; This indicates that the process exited due to a SIGKILL signal. Typically, this is either caused by manual cancellation or high memory usage causing the operating system to terminate the process.
11:28:18 AM
prefect.flow_runs.runner
Reported flow run '54e4deb7-a72d-42e5-877e-e82f6941ffbc' as crashed: Flow run process exited with non-zero status code -9.
Some additional context:
- The subflow does not return any result data.
- Resources are sufficient: CPU (16 cores), Memory (32 Gi). Based on
top
output, memory usage appears to be well within limits just before the parent flow crashes. - Here is the
top
output from the parent flow container right before the crash:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1 root 20 0 4361048 354936 89356 S 91.7 0.5 0:06.13 prefect
178 root 20 0 0 0 0 Z 0.0 0.0 0:03.76 python
303 root 20 0 18524 3436 3004 S 0.0 0.0 0:00.00 bash
313 root 20 0 36660 3180 2700 R 0.0 0.0 0:00.09 top
Below is the code I’m using to deploy the subflow within the parent flow. After logging the flow run state, the parent flow also terminates:
from prefect.deployments import run_deployment
flow_run = run_deployment(
name=name,
flow_run_name=flow_run_name,
parameters=parameter_dict,
timeout=0,
)
logger.info(f"State for flow_run: {flow_run.state}")
Given that there doesn’t appear to be any memory issues based on top
output and the subflow completes successfully without returning any result, I’m unsure why the parent flow is crashing. Any insights would be appreciated.