Network issues when running Prefect Agent - httpx.ReadTimeout

Archive
,
1

We recently deployed a Prefect Orion server on one of our servers, due to failure of Prefect Cloud. The connection to the server is fast, but the prefect agent still occasionally disconnects to the Prefect Orion server, raising HTTP errors. Is this normal and will it affect the flow runs?
The error message is listed below:

13:25:44.742 | ERROR   | prefect.agent -
Traceback (most recent call last):
  File "D:\Anaconda3\envs\pe_1.5\lib\site-packages\httpcore\backends\asyncio.py", line 33, in read
    return await self._stream.receive(max_bytes=max_bytes)
  File "D:\Anaconda3\envs\pe_1.5\lib\site-packages\anyio\_backends\_asyncio.py", line 1265, in receive
    await self._protocol.read_event.wait()
  File "D:\Anaconda3\envs\pe_1.5\lib\asyncio\locks.py", line 214, in wait
    await fut
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\Anaconda3\envs\pe_1.5\lib\site-packages\httpcore\_exceptions.py", line 8, in map_exceptions
    yield
  File "D:\Anaconda3\envs\pe_1.5\lib\site-packages\httpcore\backends\asyncio.py", line 31, in read
    with anyio.fail_after(timeout):
  File "D:\Anaconda3\envs\pe_1.5\lib\site-packages\anyio\_core\_tasks.py", line 118, in __exit__
    raise TimeoutError
TimeoutError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\Anaconda3\envs\pe_1.5\lib\site-packages\httpx\_transports\default.py", line 60, in map_httpcore_exceptions
    yield
  File "D:\Anaconda3\envs\pe_1.5\lib\site-packages\httpx\_transports\default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "D:\Anaconda3\envs\pe_1.5\lib\site-packages\httpcore\_async\connection_pool.py", line 253, in handle_async_request
    raise exc
  File "D:\Anaconda3\envs\pe_1.5\lib\site-packages\httpcore\_async\connection_pool.py", line 237, in handle_async_request
    response = await connection.handle_async_request(request)
  File "D:\Anaconda3\envs\pe_1.5\lib\site-packages\httpcore\_async\connection.py", line 90, in handle_async_request
    return await self._connection.handle_async_request(request)
  File "D:\Anaconda3\envs\pe_1.5\lib\site-packages\httpcore\_async\http11.py", line 105, in handle_async_request
    raise exc
  File "D:\Anaconda3\envs\pe_1.5\lib\site-packages\httpcore\_async\http11.py", line 84, in handle_async_request
    ) = await self._receive_response_headers(**kwargs)
  File "D:\Anaconda3\envs\pe_1.5\lib\site-packages\httpcore\_async\http11.py", line 148, in _receive_response_headers
    event = await self._receive_event(timeout=timeout)
  File "D:\Anaconda3\envs\pe_1.5\lib\site-packages\httpcore\_async\http11.py", line 177, in _receive_event
    data = await self._network_stream.read(
  File "D:\Anaconda3\envs\pe_1.5\lib\site-packages\httpcore\backends\asyncio.py", line 30, in read
    with map_exceptions(exc_map):
  File "D:\Anaconda3\envs\pe_1.5\lib\contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "D:\Anaconda3\envs\pe_1.5\lib\site-packages\httpcore\_exceptions.py", line 12, in map_exceptions
    raise to_exc(exc)
httpcore.ReadTimeout

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "D:\Anaconda3\envs\pe_1.5\lib\site-packages\prefect\agent.py", line 154, in get_and_submit_flow_runs
    queue_runs = await self.client.get_runs_in_work_queue(
  File "D:\Anaconda3\envs\pe_1.5\lib\site-packages\prefect\client\orion.py", line 759, in get_runs_in_work_queue
    response = await self._client.post(
  File "D:\Anaconda3\envs\pe_1.5\lib\site-packages\httpx\_client.py", line 1842, in post
    return await self.request(
  File "D:\Anaconda3\envs\pe_1.5\lib\site-packages\httpx\_client.py", line 1527, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "D:\Anaconda3\envs\pe_1.5\lib\site-packages\prefect\client\base.py", line 159, in send
    await super().send(*args, **kwargs)
  File "D:\Anaconda3\envs\pe_1.5\lib\site-packages\httpx\_client.py", line 1614, in send
    response = await self._send_handling_auth(
  File "D:\Anaconda3\envs\pe_1.5\lib\site-packages\httpx\_client.py", line 1642, in _send_handling_auth
    response = await self._send_handling_redirects(
  File "D:\Anaconda3\envs\pe_1.5\lib\site-packages\httpx\_client.py", line 1679, in _send_handling_redirects
    response = await self._send_single_request(request)
  File "D:\Anaconda3\envs\pe_1.5\lib\site-packages\httpx\_client.py", line 1716, in _send_single_request
    response = await transport.handle_async_request(request)
  File "D:\Anaconda3\envs\pe_1.5\lib\site-packages\httpx\_transports\default.py", line 352, in handle_async_request
    with map_httpcore_exceptions():
  File "D:\Anaconda3\envs\pe_1.5\lib\contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "D:\Anaconda3\envs\pe_1.5\lib\site-packages\httpx\_transports\default.py", line 77, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ReadTimeout
2

could you explain more? sorry to hear that a temporary outage affected your workflows but some failures are inevitable; we do our best to keep the uptime as high as possible

when the agent is down or fails to communicate with the server, then it’s expected that it cannot create and orchestrate flow runs

3

I have been having this issue since my transition to Prefect 2.0. It doesn’s seem to affect any functionality, but really annoying. Let me know if you find any solution :smiley:

4

I would love to, but I didn’t understand the problem

5

Hi Anna,

Pretty similar to the original post…while the local agent is running backgound on terminal, it pop up below exceptions all the time. It seems related to asyncio/http.

The flow can be schedule and executed without issue, just on terminal it shows those annoying exceptions.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File “C:\Users\XXXXX.XXXXX\AppData\Local\Programs\Python\Python39\lib\site-packages\httpcore_exceptions.py”, line 8, in map_exceptions
yield
File “C:\Users\XXXXX.XXXXX\AppData\Local\Programs\Python\Python39\lib\site-packages\httpcore\backends\asyncio.py”, line 33, in read
return await self._stream.receive(max_bytes=max_bytes)
File “C:\Users\XXXXX.XXXXX\AppData\Local\Programs\Python\Python39\lib\site-packages\anyio_backends_asyncio.py”, line 1274, in receive
raise self._protocol.exception
anyio.BrokenResourceError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “C:\Users\XXXXX.XXXXX\AppData\Local\Programs\Python\Python39\lib\site-packages\httpx_transports\default.py”, line 60, in map_httpcore_exceptions
yield
File “C:\Users\XXXXX.XXXXX\AppData\Local\Programs\Python\Python39\lib\site-packages\httpx_transports\default.py”, line 353, in handle_async_request
resp = await self._pool.handle_async_request(req)
File “C:\Users\XXXXX.XXXXX\AppData\Local\Programs\Python\Python39\lib\site-packages\httpcore_async\connection_pool.py”, line 253, in handle_async_request
raise exc
File “C:\Users\XXXXX.XXXXX\AppData\Local\Programs\Python\Python39\lib\site-packages\httpcore_async\connection_pool.py”, line 237, in handle_async_request
response = await connection.handle_async_request(request)
File “C:\Users\XXXXX.XXXXX\AppData\Local\Programs\Python\Python39\lib\site-packages\httpcore_async\connection.py”, line 90, in handle_async_request
return await self._connection.handle_async_request(request)
File “C:\Users\XXXXX.XXXXX\AppData\Local\Programs\Python\Python39\lib\site-packages\httpcore_async\http11.py”, line 105, in handle_async_request
raise exc
File “C:\Users\XXXXX.XXXXX\AppData\Local\Programs\Python\Python39\lib\site-packages\httpcore_async\http11.py”, line 84, in handle_async_request
) = await self._receive_response_headers(**kwargs)
File “C:\Users\XXXXX.XXXXX\AppData\Local\Programs\Python\Python39\lib\site-packages\httpcore_async\http11.py”, line 148, in _receive_response_headers
event = await self._receive_event(timeout=timeout)
File “C:\Users\XXXXX.XXXXX\AppData\Local\Programs\Python\Python39\lib\site-packages\httpcore_async\http11.py”, line 177, in _receive_event
data = await self._network_stream.read(
File “C:\Users\XXXXX.XXXXX\AppData\Local\Programs\Python\Python39\lib\site-packages\httpcore\backends\asyncio.py”, line 35, in read
return b""
File “C:\Users\XXXXX.XXXXX\AppData\Local\Programs\Python\Python39\lib\contextlib.py”, line 137, in exit
self.gen.throw(typ, value, traceback)
File “C:\Users\XXXXX.XXXXX\AppData\Local\Programs\Python\Python39\lib\site-packages\httpcore_exceptions.py”, line 12, in map_exceptions
raise to_exc(exc)
httpcore.ReadError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File “C:\Users\XXXXX.XXXXX\AppData\Local\Programs\Python\Python39\lib\site-packages\prefect\agent.py”, line 154, in get_and_submit_flow_runs
queue_runs = await self.client.get_runs_in_work_queue(
File “C:\Users\XXXXX.XXXXX\AppData\Local\Programs\Python\Python39\lib\site-packages\prefect\client\orion.py”, line 759, in get_runs_in_work_queue
response = await self._client.post(
File “C:\Users\XXXXX.XXXXX\AppData\Local\Programs\Python\Python39\lib\site-packages\httpx_client.py”, line 1842, in post
return await self.request(
File “C:\Users\XXXXX.XXXXX\AppData\Local\Programs\Python\Python39\lib\site-packages\httpx_client.py”, line 1527, in request
return await self.send(request, auth=auth, follow_redirects=follow_redirects)
File “C:\Users\XXXXX.XXXXX\AppData\Local\Programs\Python\Python39\lib\site-packages\prefect\client\base.py”, line 159, in send
await super().send(*args, **kwargs)
File “C:\Users\XXXXX.XXXXX\AppData\Local\Programs\Python\Python39\lib\site-packages\httpx_client.py”, line 1614, in send
response = await self._send_handling_auth(
File “C:\Users\XXXXX.XXXXX\AppData\Local\Programs\Python\Python39\lib\site-packages\httpx_client.py”, line 1642, in _send_handling_auth
response = await self._send_handling_redirects(
File “C:\Users\XXXXX.XXXXX\AppData\Local\Programs\Python\Python39\lib\site-packages\httpx_client.py”, line 1679, in _send_handling_redirects
response = await self._send_single_request(request)
File “C:\Users\XXXXX.XXXXX\AppData\Local\Programs\Python\Python39\lib\site-packages\httpx_client.py”, line 1716, in _send_single_request
response = await transport.handle_async_request(request)
File “C:\Users\XXXXX.XXXXX\AppData\Local\Programs\Python\Python39\lib\site-packages\httpx_transports\default.py”, line 353, in handle_async_request
resp = await self._pool.handle_async_request(req)
File “C:\Users\XXXXX.XXXXX\AppData\Local\Programs\Python\Python39\lib\contextlib.py”, line 137, in exit
self.gen.throw(typ, value, traceback)
File “C:\Users\XXXXX.XXXXX\AppData\Local\Programs\Python\Python39\lib\site-packages\httpx_transports\default.py”, line 77, in map_httpcore_exceptions
raise mapped_exc(message) from exc
httpx.ReadError

6

This looks like a transient network issue - do you run it on some unstable network? things like this happen. Maybe you can run it as a persistent service?

7

I actually ran it on my company’s intranet…what do you mean by persistent service?

8

Kubernetes deployment or ECS task - those two are examples how you can run your agent in a way that a failed process will get restarted automatically

9

I had the same error when running agent. Any solution to that ?

10

Solve it. For me it was a network error I made by mistake when changing the url configuration of prefect 2. Thanks!

1 Like