Connection Error upon triggering deployment

I’m running a small task on a local Kubernetes cluster with Prefect.

I started a local Prefect Orion API Server with prefect orion start.

I also set the PREFECT_API_URL variable to point to the local API server (see below).

My prefect config view looks like:

➜  ~ prefect config view
PREFECT_PROFILE='default'
PREFECT_API_URL='http://127.0.0.1:4200/api' (from profile)

After running the deployment, I get this output:

➜  ~ prefect deployment run check/dbtesting2
Creating flow run for deployment 'check/dbtesting2'...
Created flow run 'omniscient-bobcat'.
└── UUID: 455cc82a-8075-470b-91f3-117c83f37f5c
└── Parameters: {}
└── Scheduled start time: 2023-02-14 16:32:48 CST (now)
└── URL:
http://127.0.0.1:4200/flow-runs/flow-run/455cc82a-8075-470b-91f3-117c83f37f5c

Which all looks good.

However, in the agent terminal window, I see several connection errors:

Agent started! Looking for work from queue(s): default...
16:32:51.833 | INFO    | prefect.agent - Submitting flow run '455cc82a-8075-470b-91f3-117c83f37f5c'
16:32:52.020 | INFO    | prefect.agent - Completed submission of flow run '455cc82a-8075-470b-91f3-117c83f37f5c'
16:32:52.088 | INFO    | prefect.infrastructure.kubernetes-job - Job 'omniscient-bobcat-g2nzf': Pod has status 'Pending'.
16:32:53.324 | INFO    | prefect.infrastructure.kubernetes-job - Job 'omniscient-bobcat-g2nzf': Pod has status 'Running'.
/usr/local/lib/python3.10/runpy.py:126: RuntimeWarning: 'prefect.engine' found in sys.modules after import of package 'prefect', but prior to execution of 'prefect.engine'; this may result in unpredictable behaviour
  warn(RuntimeWarning(msg))
22:32:55.335 | ERROR   | prefect.engine - Engine execution of flow run '455cc82a-8075-470b-91f3-117c83f37f5c' exited with unexpected exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/anyio/_core/_sockets.py", line 164, in try_connect
    stream = await asynclib.connect_tcp(remote_host, remote_port, local_address)
  File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 1691, in connect_tcp
    await get_running_loop().create_connection(
  File "/usr/local/lib/python3.10/asyncio/base_events.py", line 1076, in create_connection
    raise exceptions[0]
  File "/usr/local/lib/python3.10/asyncio/base_events.py", line 1060, in create_connection
    sock = await self._connect_sock(
  File "/usr/local/lib/python3.10/asyncio/base_events.py", line 969, in _connect_sock
    await self.sock_connect(sock, address)
  File "/usr/local/lib/python3.10/asyncio/selector_events.py", line 501, in sock_connect
    return await fut
  File "/usr/local/lib/python3.10/asyncio/selector_events.py", line 541, in _sock_connect_cb
    raise OSError(err, f'Connect call failed {address}')
ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 4200)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/httpcore/_exceptions.py", line 10, in map_exceptions
    yield
  File "/usr/local/lib/python3.10/site-packages/httpcore/backends/asyncio.py", line 111, in connect_tcp
    stream: anyio.abc.ByteStream = await anyio.connect_tcp(
  File "/usr/local/lib/python3.10/site-packages/anyio/_core/_sockets.py", line 222, in connect_tcp
    raise OSError("All connection attempts failed") from cause
OSError: All connection attempts failed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
    yield
  File "/usr/local/lib/python3.10/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/usr/local/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 253, in handle_async_request
    raise exc
  File "/usr/local/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 237, in handle_async_request
    response = await connection.handle_async_request(request)
  File "/usr/local/lib/python3.10/site-packages/httpcore/_async/connection.py", line 86, in handle_async_request
    raise exc
  File "/usr/local/lib/python3.10/site-packages/httpcore/_async/connection.py", line 63, in handle_async_request
    stream = await self._connect(request)
  File "/usr/local/lib/python3.10/site-packages/httpcore/_async/connection.py", line 111, in _connect
    stream = await self._network_backend.connect_tcp(**kwargs)
  File "/usr/local/lib/python3.10/site-packages/httpcore/backends/auto.py", line 29, in connect_tcp
    return await self._backend.connect_tcp(
  File "/usr/local/lib/python3.10/site-packages/httpcore/backends/asyncio.py", line 109, in connect_tcp
    with map_exceptions(exc_map):
  File "/usr/local/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/usr/local/lib/python3.10/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc)
httpcore.ConnectError: All connection attempts failed

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 1986, in <module>
    enter_flow_run_engine_from_subprocess(flow_run_id)
  File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 194, in enter_flow_run_engine_from_subprocess
    return anyio.run(retrieve_flow_then_begin_flow_run, flow_run_id)
  File "/usr/local/lib/python3.10/site-packages/anyio/_core/_eventloop.py", line 70, in run
    return asynclib.run(func, *args, **backend_options)
  File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 292, in run
    return native_run(wrapper(), debug=debug)
  File "/usr/local/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 287, in wrapper
    return await func(*args)
  File "/usr/local/lib/python3.10/site-packages/prefect/client/utilities.py", line 47, in with_injected_client
    return await fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 266, in retrieve_flow_then_begin_flow_run
    flow_run = await client.read_flow_run(flow_run_id)
  File "/usr/local/lib/python3.10/site-packages/prefect/client/orion.py", line 1576, in read_flow_run
    response = await self._client.get(f"/flow_runs/{flow_run_id}")
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1757, in get
    return await self.request(
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1533, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/usr/local/lib/python3.10/site-packages/prefect/client/base.py", line 229, in send
    response = await self._send_with_retry(
  File "/usr/local/lib/python3.10/site-packages/prefect/client/base.py", line 187, in _send_with_retry
    response = await request()
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1620, in send
    response = await self._send_handling_auth(
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1648, in _send_handling_auth
    response = await self._send_handling_redirects(
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1685, in _send_handling_redirects
    response = await self._send_single_request(request)
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1722, in _send_single_request
    response = await transport.handle_async_request(request)
  File "/usr/local/lib/python3.10/site-packages/httpx/_transports/default.py", line 352, in handle_async_request
    with map_httpcore_exceptions():
  File "/usr/local/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/usr/local/lib/python3.10/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ConnectError: All connection attempts failed

I’m not sure whats going on. It works fine when using Prefect Cloud, but it doesn’t work when using the local Prefect Orion API Server.

I’ve tried uninstalling and reinstalling Prefect and resetting the Orion database, but to no avail.

My Prefect version:

➜  ~ prefect version
Version:             2.8.0
API version:         0.8.4
Python version:      3.10.9
Git commit:          50f0ad0a
Built:               Thu, Feb 9, 2023 3:38 PM
OS/Arch:             darwin/arm64
Profile:             default
Server type:         hosted

I can see the Flow Runs in the API Server console but they are just in β€˜Pending’ state and never run.

Would greatly appreciate any assistance.

Hi @nss0204, the cluster you are running the agent likely does not have permission to connect to the Orion API you have at localhost.

You can check out this Prefect recipe to build the agent in Kubernetes with this helm chart or see an example of the configuration. prefect-recipes/devops/infrastructure-as-code/helm/prefect2-agent at main Β· PrefectHQ/prefect-recipes Β· GitHub

I met same error too. :smiling_face_with_tear:
I posted more details at ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 4200)

Hi @nss0204 I found the issue, I posted the workaround solution at ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 4200) - #3 by Hongbo-Miao
Hope it helps! :smiley: