Prefect Agent CrashLoopBackOff

Help
, , ,
1

This is happening in my Prefect agent pod.
Cloud Provider - AWS, EKS

Starting v2.8.3 agent connected to 
https://api.prefect.cloud/api/accounts/25af8892-2dd8-4e0c-ba9f-0ebfbdde89d6/work
spaces/ce71e0ef-fdf9-425a-bdf7-306d1e160a18...

  ___ ___ ___ ___ ___ ___ _____     _   ___ ___ _  _ _____
 | _ \ _ \ __| __| __/ __|_   _|   /_\ / __| __| \| |_   _|
 |  _/   / _|| _|| _| (__  | |    / _ \ (_ | _|| .` | | |
 |_| |_|_\___|_| |___\___| |_|   /_/ \_\___|___|_|\_| |_|


Agent started! Looking for work from queue(s): kubernetes-dev...
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/prefect/cli/_utilities.py", line 41, in wrapper
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 230, in coroutine_wrapper
    return run_async_in_new_loop(async_fn, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 181, in run_async_in_new_loop
    return anyio.run(partial(__fn, *args, **kwargs))
  File "/usr/local/lib/python3.10/site-packages/anyio/_core/_eventloop.py", line 70, in run
    return asynclib.run(func, *args, **backend_options)
  File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 292, in run
    return native_run(wrapper(), debug=debug)
  File "/usr/local/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 287, in wrapper
    return await func(*args)
  File "/usr/local/lib/python3.10/site-packages/prefect/cli/agent.py", line 189, in start
    async with anyio.create_task_group() as tg:
  File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 662, in __aexit__
    raise exceptions[0]
  File "/usr/local/lib/python3.10/site-packages/prefect/utilities/services.py", line 46, in critical_service_loop
    await workload()
  File "/usr/local/lib/python3.10/site-packages/prefect/agent.py", line 265, in check_for_cancelled_flow_runs
    named_cancelling_flow_runs = await self.client.read_flow_runs(
  File "/usr/local/lib/python3.10/site-packages/prefect/client/orchestration.py", line 1715, in read_flow_runs
    response = await self._client.post(f"/flow_runs/filter", json=body)
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1848, in post
    return await self.request(
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1533, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/usr/local/lib/python3.10/site-packages/prefect/client/base.py", line 253, in send
    response.raise_for_status()
  File "/usr/local/lib/python3.10/site-packages/prefect/client/base.py", line 130, in raise_for_status
    raise PrefectHTTPStatusError.from_httpx_error(exc) from exc.__cause__
prefect.exceptions.PrefectHTTPStatusError: Client error '404 Not Found' for url 'https://api.prefect.cloud/api/accounts/25af8892-2dd8-4e0c-ba9f-0ebfbdde89d6/workspaces/ce71e0ef-fdf9-425a-bdf7-306d1e160a18/flow_runs/filter'
Response: {'detail': 'Not Found'}
For more information check: https://httpstatuses.com/404
An exception occurred.