Crash detected but flow did not get status crashed, and flow still running: PrefectHTTPStatusError: Server error '500 Internal Server Error' for url 'http://ephemeral-prefect/api/task_runs/'

Help
1

Crash detected but flow did not get status crashed, but still running when this error have happened. This error is happend every 4-5 days.
Could you help and tell me how I can solve this problem?

Crash detected! Execution was interrupted by an unexpected exception: PrefectHTTPStatusError: Server error '500 Internal Server Error' for url 'http://ephemeral-prefect/api/task_runs/'
Response: {'exception_message': 'Internal Server Error'}
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/500

Crash details:
Traceback (most recent call last):
  File "/home/ubuntu/.local/lib/python3.10/site-packages/prefect/engine.py", line 1913, in report_flow_run_crashes
    yield
  File "/usr/lib/python3.10/contextlib.py", line 697, in __aexit__
    cb_suppress = await cb(*exc_details)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 597, in __aexit__
    raise exceptions[0]
  File "/home/ubuntu/.local/lib/python3.10/site-packages/prefect/engine.py", line 1377, in create_task_run_then_submit
    task_run = await create_task_run(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/prefect/engine.py", line 1422, in create_task_run
    task_run = await flow_run_context.client.create_task_run(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/prefect/client/orchestration.py", line 1906, in create_task_run
    response = await self._client.post(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/httpx/_client.py", line 1848, in post
    return await self.request(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/httpx/_client.py", line 1530, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/prefect/client/base.py", line 285, in send
    response.raise_for_status()
  File "/home/ubuntu/.local/lib/python3.10/site-packages/prefect/client/base.py", line 138, in raise_for_status
    raise PrefectHTTPStatusError.from_httpx_error(exc) from exc.__cause__
prefect.exceptions.PrefectHTTPStatusError: Server error '500 Internal Server Error' for url 'http://ephemeral-prefect/api/task_runs/'
Response: {'exception_message': 'Internal Server Error'}
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/500
1 Like
2

Same issue. Any resolutions? I’ve looked at DB size and network traffic as potential issues, but haven’t found anything yet.