Prefect2 agent crashing with cloud connection

DataflowOps for Prefect 2.0
1

Hi,

I’ve been using Prefect 1 for the last 18 months running a local agent and cloud setup. This has worked perfectly with no crashes at all. Now I’m trying to migrate to Prefect2 but the agent crashes after a few hours even before any runs have been started.

In my test I’m running prefect1 and 2 side by side on the same server (different virtual envs) and the prefect1 agent runs smoothly while the prefect2 agent crashes. Either something is wrong or prefect2 is very sensitive to network issues that 1 wasn’t.

Here’s prefect1 and prefect2 agents running side by side with 1 running and 2 crashed. Love prefect but I’m nervous about my ability to migrate to prefect2 right now.

image
image986×711 134 KB

And here’s the crash info.

  ___ ___ ___ ___ ___ ___ _____     _   ___ ___ _  _ _____
 | _ \ _ \ __| __| __/ __|_   _|   /_\ / __| __| \| |_   _|
 |  _/   / _|| _|| _| (__  | |    / _ \ (_ | _|| .` | | |
 |_| |_|_\___|_| |___\___| |_|   /_/ \_\___|___|_|\_| |_|


Agent started! Looking for work from work pool 'default-agent-pool'...

Failed the last 3 attempts.  Please check your environment and configuration.
Examples of recent errors:

Traceback (most recent call last):
  File "C:\Users\Public\python_envs\prefect2\lib\site-packages\h2\connection.py", line 224, in process_input
    func, target_state = self._transitions[(self.state, input_)]
KeyError: (<ConnectionState.CLOSED: 3>, <ConnectionInputs.SEND_HEADERS: 0>)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\Public\python_envs\prefect2\lib\site-packages\httpcore\_async\http2.py", line 106, in
handle_async_request
    await self._send_request_headers(request=request, stream_id=stream_id)
  File "C:\Users\Public\python_envs\prefect2\lib\site-packages\httpcore\_async\http2.py", line 203, in
_send_request_headers
    self._h2_state.send_headers(stream_id, headers, end_stream=end_stream)
  File "C:\Users\Public\python_envs\prefect2\lib\site-packages\h2\connection.py", line 766, in send_headers
    self.state_machine.process_input(ConnectionInputs.SEND_HEADERS)
  File "C:\Users\Public\python_envs\prefect2\lib\site-packages\h2\connection.py", line 228, in process_input
    raise ProtocolError(
h2.exceptions.ProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\Public\python_envs\prefect2\lib\site-packages\httpx\_transports\default.py", line 60, in
map_httpcore_exceptions
    yield
  File "C:\Users\Public\python_envs\prefect2\lib\site-packages\httpx\_transports\default.py", line 353, in
handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "C:\Users\Public\python_envs\prefect2\lib\site-packages\httpcore\_async\connection_pool.py", line 253, in
handle_async_request
    raise exc
  File "C:\Users\Public\python_envs\prefect2\lib\site-packages\httpcore\_async\connection_pool.py", line 237, in
handle_async_request
    response = await connection.handle_async_request(request)
  File "C:\Users\Public\python_envs\prefect2\lib\site-packages\httpcore\_async\connection.py", line 90, in
handle_async_request
    return await self._connection.handle_async_request(request)
  File "C:\Users\Public\python_envs\prefect2\lib\site-packages\httpcore\_async\http2.py", line 142, in
handle_async_request
    raise LocalProtocolError(exc)  # pragma: nocover
httpcore.LocalProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Users\Public\python_envs\prefect2\lib\site-packages\prefect\utilities\services.py", line 46, in
critical_service_loop
    await workload()
  File "C:\Users\Public\python_envs\prefect2\lib\site-packages\prefect\agent.py", line 264, in
check_for_cancelled_flow_runs
    named_cancelling_flow_runs = await self.client.read_flow_runs(
  File "C:\Users\Public\python_envs\prefect2\lib\site-packages\prefect\client\orchestration.py", line 1689, in
read_flow_runs
    response = await self._client.post(f"/flow_runs/filter", json=body)
  File "C:\Users\Public\python_envs\prefect2\lib\site-packages\httpx\_client.py", line 1848, in post
    return await self.request(
  File "C:\Users\Public\python_envs\prefect2\lib\site-packages\httpx\_client.py", line 1533, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "C:\Users\Public\python_envs\prefect2\lib\site-packages\prefect\client\base.py", line 227, in send
    response = await self._send_with_retry(
  File "C:\Users\Public\python_envs\prefect2\lib\site-packages\prefect\client\base.py", line 187, in _send_with_retry
    response = await request()
  File "C:\Users\Public\python_envs\prefect2\lib\site-packages\httpx\_client.py", line 1620, in send
    response = await self._send_handling_auth(
  File "C:\Users\Public\python_envs\prefect2\lib\site-packages\httpx\_client.py", line 1648, in _send_handling_auth
    response = await self._send_handling_redirects(
  File "C:\Users\Public\python_envs\prefect2\lib\site-packages\httpx\_client.py", line 1685, in
_send_handling_redirects
    response = await self._send_single_request(request)
  File "C:\Users\Public\python_envs\prefect2\lib\site-packages\httpx\_client.py", line 1722, in _send_single_request
    response = await transport.handle_async_request(request)
  File "C:\Users\Public\python_envs\prefect2\lib\site-packages\httpx\_transports\default.py", line 352, in
handle_async_request
    with map_httpcore_exceptions():
  File "C:\Program Files\Python310\lib\contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "C:\Users\Public\python_envs\prefect2\lib\site-packages\httpx\_transports\default.py", line 77, in
map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.LocalProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED



Failed the last 3 attempts.  Please check your environment and configuration.
Examples of recent errors:

Traceback (most recent call last):
  File "C:\Users\Public\python_envs\prefect2\lib\site-packages\h2\connection.py", line 224, in process_input
    func, target_state = self._transitions[(self.state, input_)]
KeyError: (<ConnectionState.CLOSED: 3>, <ConnectionInputs.SEND_HEADERS: 0>)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\Public\python_envs\prefect2\lib\site-packages\httpcore\_async\http2.py", line 106, in
handle_async_request
    await self._send_request_headers(request=request, stream_id=stream_id)
  File "C:\Users\Public\python_envs\prefect2\lib\site-packages\httpcore\_async\http2.py", line 203, in
_send_request_headers
    self._h2_state.send_headers(stream_id, headers, end_stream=end_stream)
  File "C:\Users\Public\python_envs\prefect2\lib\site-packages\h2\connection.py", line 766, in send_headers
    self.state_machine.process_input(ConnectionInputs.SEND_HEADERS)
  File "C:\Users\Public\python_envs\prefect2\lib\site-packages\h2\connection.py", line 228, in process_input
    raise ProtocolError(
h2.exceptions.ProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\Public\python_envs\prefect2\lib\site-packages\httpx\_transports\default.py", line 60, in
map_httpcore_exceptions
    yield
  File "C:\Users\Public\python_envs\prefect2\lib\site-packages\httpx\_transports\default.py", line 353, in
handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "C:\Users\Public\python_envs\prefect2\lib\site-packages\httpcore\_async\connection_pool.py", line 253, in
handle_async_request
    raise exc
  File "C:\Users\Public\python_envs\prefect2\lib\site-packages\httpcore\_async\connection_pool.py", line 237, in
handle_async_request
    response = await connection.handle_async_request(request)
  File "C:\Users\Public\python_envs\prefect2\lib\site-packages\httpcore\_async\connection.py", line 90, in
handle_async_request
    return await self._connection.handle_async_request(request)
  File "C:\Users\Public\python_envs\prefect2\lib\site-packages\httpcore\_async\http2.py", line 142, in
handle_async_request
    raise LocalProtocolError(exc)  # pragma: nocover
httpcore.LocalProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Users\Public\python_envs\prefect2\lib\site-packages\prefect\utilities\services.py", line 46, in
critical_service_loop
    await workload()
  File "C:\Users\Public\python_envs\prefect2\lib\site-packages\prefect\agent.py", line 194, in get_and_submit_flow_runs
    responses = await self.client.get_scheduled_flow_runs_for_work_pool(
  File "C:\Users\Public\python_envs\prefect2\lib\site-packages\prefect\client\orchestration.py", line 2264, in
get_scheduled_flow_runs_for_work_pool
    response = await self._client.post(
  File "C:\Users\Public\python_envs\prefect2\lib\site-packages\httpx\_client.py", line 1848, in post
    return await self.request(
  File "C:\Users\Public\python_envs\prefect2\lib\site-packages\httpx\_client.py", line 1533, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "C:\Users\Public\python_envs\prefect2\lib\site-packages\prefect\client\base.py", line 227, in send
    response = await self._send_with_retry(
  File "C:\Users\Public\python_envs\prefect2\lib\site-packages\prefect\client\base.py", line 187, in _send_with_retry
    response = await request()
  File "C:\Users\Public\python_envs\prefect2\lib\site-packages\httpx\_client.py", line 1620, in send
    response = await self._send_handling_auth(
  File "C:\Users\Public\python_envs\prefect2\lib\site-packages\httpx\_client.py", line 1648, in _send_handling_auth
    response = await self._send_handling_redirects(
  File "C:\Users\Public\python_envs\prefect2\lib\site-packages\httpx\_client.py", line 1685, in
_send_handling_redirects
    response = await self._send_single_request(request)
  File "C:\Users\Public\python_envs\prefect2\lib\site-packages\httpx\_client.py", line 1722, in _send_single_request
    response = await transport.handle_async_request(request)
  File "C:\Users\Public\python_envs\prefect2\lib\site-packages\httpx\_transports\default.py", line 352, in
handle_async_request
    with map_httpcore_exceptions():
  File "C:\Program Files\Python310\lib\contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "C:\Users\Public\python_envs\prefect2\lib\site-packages\httpx\_transports\default.py", line 77, in
map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.LocalProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED


Agent stopped!
2

Seems related to this issue but I don’t have a solution.
Network issues when running Prefect Agent - httpx.ReadTimeout

3

You can set the following environment variable / config variable for the agent -
PREFECT_API_ENABLE_HTTP2=False

4

Thanks @Christopher_Boyd ,

I’ve added that to my config so I’ll report back if it seems stable.

5

@Christopher_Boyd The agent has been running for 24 hours now without issues so it seems to have fixed the problem. Thanks!

6

I’m still facing a similar issue, even after adding that as an environment variable. Is there another way to fix this? Running prefect v2.8.7

  ___ ___ ___ ___ ___ ___ _____     _   ___ ___ _  _ _____
 | _ \ _ \ __| __| __/ __|_   _|   /_\ / __| __| \| |_   _|
 |  _/   / _|| _|| _| (__  | |    / _ \ (_ | _|| .` | | |
 |_| |_|_\___|_| |___\___| |_|   /_/ \_\___|___|_|\_| |_|


Agent started! Looking for work from work pool 'susilceimpwssx2'...

Failed the last 3 attempts.  Please check your environment and configuration.
Examples of recent errors:

Traceback (most recent call last):
  File
"/opt/pysetup/.venv/lib/python3.9/site-packages/httpx/_transports/default.py",
line 60, in map_httpcore_exceptions
    yield
  File
"/opt/pysetup/.venv/lib/python3.9/site-packages/httpx/_transports/default.py",
line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File
"/opt/pysetup/.venv/lib/python3.9/site-packages/httpcore/_async/connection_pool.
py", line 208, in handle_async_request
    raise UnsupportedProtocol(
httpcore.UnsupportedProtocol: Request URL is missing an 'http://' or 'https://'
protocol.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File
"/opt/pysetup/.venv/lib/python3.9/site-packages/prefect/utilities/services.py",
line 46, in critical_service_loop
    await workload()
  File "/opt/pysetup/.venv/lib/python3.9/site-packages/prefect/agent.py", line
194, in get_and_submit_flow_runs
    responses = await self.client.get_scheduled_flow_runs_for_work_pool(
  File
"/opt/pysetup/.venv/lib/python3.9/site-packages/prefect/client/orchestration.py"
, line 2269, in get_scheduled_flow_runs_for_work_pool
    response = await self._client.post(
  File "/opt/pysetup/.venv/lib/python3.9/site-packages/httpx/_client.py", line
1848, in post
    return await self.request(
  File "/opt/pysetup/.venv/lib/python3.9/site-packages/httpx/_client.py", line
1533, in request
    return await self.send(request, auth=auth,
follow_redirects=follow_redirects)
  File "/opt/pysetup/.venv/lib/python3.9/site-packages/prefect/client/base.py",
line 243, in send
    response = await self._send_with_retry(
  File "/opt/pysetup/.venv/lib/python3.9/site-packages/prefect/client/base.py",
line 189, in _send_with_retry
    response = await request()
  File "/opt/pysetup/.venv/lib/python3.9/site-packages/httpx/_client.py", line
1620, in send
    response = await self._send_handling_auth(
  File "/opt/pysetup/.venv/lib/python3.9/site-packages/httpx/_client.py", line
1648, in _send_handling_auth
    response = await self._send_handling_redirects(
  File "/opt/pysetup/.venv/lib/python3.9/site-packages/httpx/_client.py", line
1685, in _send_handling_redirects
    response = await self._send_single_request(request)
  File "/opt/pysetup/.venv/lib/python3.9/site-packages/httpx/_client.py", line
1722, in _send_single_request
    response = await transport.handle_async_request(request)
  File
"/opt/pysetup/.venv/lib/python3.9/site-packages/httpx/_transports/default.py",
line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/usr/local/lib/python3.9/contextlib.py", line 137, in __exit__
    self.gen.throw(typ, value, traceback)
  File
"/opt/pysetup/.venv/lib/python3.9/site-packages/httpx/_transports/default.py",
line 77, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.UnsupportedProtocol: Request URL is missing an 'http://' or 'https://'
protocol.



Failed the last 3 attempts.  Please check your environment and configuration.
Examples of recent errors:

Traceback (most recent call last):
  File
"/opt/pysetup/.venv/lib/python3.9/site-packages/httpx/_transports/default.py",
line 60, in map_httpcore_exceptions
    yield
  File
"/opt/pysetup/.venv/lib/python3.9/site-packages/httpx/_transports/default.py",
line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File
"/opt/pysetup/.venv/lib/python3.9/site-packages/httpcore/_async/connection_pool.
py", line 208, in handle_async_request
    raise UnsupportedProtocol(
httpcore.UnsupportedProtocol: Request URL is missing an 'http://' or 'https://'
protocol.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File
"/opt/pysetup/.venv/lib/python3.9/site-packages/prefect/utilities/services.py",
line 46, in critical_service_loop
    await workload()
  File "/opt/pysetup/.venv/lib/python3.9/site-packages/prefect/agent.py", line
264, in check_for_cancelled_flow_runs
    named_cancelling_flow_runs = await self.client.read_flow_runs(
  File
"/opt/pysetup/.venv/lib/python3.9/site-packages/prefect/client/orchestration.py"
, line 1689, in read_flow_runs
    response = await self._client.post(f"/flow_runs/filter", json=body)
  File "/opt/pysetup/.venv/lib/python3.9/site-packages/httpx/_client.py", line
1848, in post
    return await self.request(
  File "/opt/pysetup/.venv/lib/python3.9/site-packages/httpx/_client.py", line
1533, in request
    return await self.send(request, auth=auth,
follow_redirects=follow_redirects)
  File "/opt/pysetup/.venv/lib/python3.9/site-packages/prefect/client/base.py",
line 243, in send
    response = await self._send_with_retry(
  File "/opt/pysetup/.venv/lib/python3.9/site-packages/prefect/client/base.py",
line 189, in _send_with_retry
    response = await request()
  File "/opt/pysetup/.venv/lib/python3.9/site-packages/httpx/_client.py", line
1620, in send
    response = await self._send_handling_auth(
  File "/opt/pysetup/.venv/lib/python3.9/site-packages/httpx/_client.py", line
1648, in _send_handling_auth
    response = await self._send_handling_redirects(
  File "/opt/pysetup/.venv/lib/python3.9/site-packages/httpx/_client.py", line
1685, in _send_handling_redirects
    response = await self._send_single_request(request)
  File "/opt/pysetup/.venv/lib/python3.9/site-packages/httpx/_client.py", line
1722, in _send_single_request
    response = await transport.handle_async_request(request)
  File
"/opt/pysetup/.venv/lib/python3.9/site-packages/httpx/_transports/default.py",
line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/usr/local/lib/python3.9/contextlib.py", line 137, in __exit__
    self.gen.throw(typ, value, traceback)
  File
"/opt/pysetup/.venv/lib/python3.9/site-packages/httpx/_transports/default.py",
line 77, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.UnsupportedProtocol: Request URL is missing an 'http://' or 'https://'
protocol.

Agent stopped!