I’m running a small task on a local Kubernetes cluster with Prefect.
I started a local Prefect Orion API Server with prefect orion start.
I also set the PREFECT_API_URL variable to point to the local API server (see below).
My prefect config view looks like:
➜ ~ prefect config view
PREFECT_PROFILE='default'
PREFECT_API_URL='http://127.0.0.1:4200/api' (from profile)
After running the deployment, I get this output:
➜ ~ prefect deployment run check/dbtesting2
Creating flow run for deployment 'check/dbtesting2'...
Created flow run 'omniscient-bobcat'.
└── UUID: 455cc82a-8075-470b-91f3-117c83f37f5c
└── Parameters: {}
└── Scheduled start time: 2023-02-14 16:32:48 CST (now)
└── URL:
http://127.0.0.1:4200/flow-runs/flow-run/455cc82a-8075-470b-91f3-117c83f37f5c
Which all looks good.
However, in the agent terminal window, I see several connection errors:
Agent started! Looking for work from queue(s): default...
16:32:51.833 | INFO | prefect.agent - Submitting flow run '455cc82a-8075-470b-91f3-117c83f37f5c'
16:32:52.020 | INFO | prefect.agent - Completed submission of flow run '455cc82a-8075-470b-91f3-117c83f37f5c'
16:32:52.088 | INFO | prefect.infrastructure.kubernetes-job - Job 'omniscient-bobcat-g2nzf': Pod has status 'Pending'.
16:32:53.324 | INFO | prefect.infrastructure.kubernetes-job - Job 'omniscient-bobcat-g2nzf': Pod has status 'Running'.
/usr/local/lib/python3.10/runpy.py:126: RuntimeWarning: 'prefect.engine' found in sys.modules after import of package 'prefect', but prior to execution of 'prefect.engine'; this may result in unpredictable behaviour
warn(RuntimeWarning(msg))
22:32:55.335 | ERROR | prefect.engine - Engine execution of flow run '455cc82a-8075-470b-91f3-117c83f37f5c' exited with unexpected exception
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/anyio/_core/_sockets.py", line 164, in try_connect
stream = await asynclib.connect_tcp(remote_host, remote_port, local_address)
File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 1691, in connect_tcp
await get_running_loop().create_connection(
File "/usr/local/lib/python3.10/asyncio/base_events.py", line 1076, in create_connection
raise exceptions[0]
File "/usr/local/lib/python3.10/asyncio/base_events.py", line 1060, in create_connection
sock = await self._connect_sock(
File "/usr/local/lib/python3.10/asyncio/base_events.py", line 969, in _connect_sock
await self.sock_connect(sock, address)
File "/usr/local/lib/python3.10/asyncio/selector_events.py", line 501, in sock_connect
return await fut
File "/usr/local/lib/python3.10/asyncio/selector_events.py", line 541, in _sock_connect_cb
raise OSError(err, f'Connect call failed {address}')
ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 4200)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/httpcore/_exceptions.py", line 10, in map_exceptions
yield
File "/usr/local/lib/python3.10/site-packages/httpcore/backends/asyncio.py", line 111, in connect_tcp
stream: anyio.abc.ByteStream = await anyio.connect_tcp(
File "/usr/local/lib/python3.10/site-packages/anyio/_core/_sockets.py", line 222, in connect_tcp
raise OSError("All connection attempts failed") from cause
OSError: All connection attempts failed
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
yield
File "/usr/local/lib/python3.10/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
resp = await self._pool.handle_async_request(req)
File "/usr/local/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 253, in handle_async_request
raise exc
File "/usr/local/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 237, in handle_async_request
response = await connection.handle_async_request(request)
File "/usr/local/lib/python3.10/site-packages/httpcore/_async/connection.py", line 86, in handle_async_request
raise exc
File "/usr/local/lib/python3.10/site-packages/httpcore/_async/connection.py", line 63, in handle_async_request
stream = await self._connect(request)
File "/usr/local/lib/python3.10/site-packages/httpcore/_async/connection.py", line 111, in _connect
stream = await self._network_backend.connect_tcp(**kwargs)
File "/usr/local/lib/python3.10/site-packages/httpcore/backends/auto.py", line 29, in connect_tcp
return await self._backend.connect_tcp(
File "/usr/local/lib/python3.10/site-packages/httpcore/backends/asyncio.py", line 109, in connect_tcp
with map_exceptions(exc_map):
File "/usr/local/lib/python3.10/contextlib.py", line 153, in __exit__
self.gen.throw(typ, value, traceback)
File "/usr/local/lib/python3.10/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
raise to_exc(exc)
httpcore.ConnectError: All connection attempts failed
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 1986, in <module>
enter_flow_run_engine_from_subprocess(flow_run_id)
File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 194, in enter_flow_run_engine_from_subprocess
return anyio.run(retrieve_flow_then_begin_flow_run, flow_run_id)
File "/usr/local/lib/python3.10/site-packages/anyio/_core/_eventloop.py", line 70, in run
return asynclib.run(func, *args, **backend_options)
File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 292, in run
return native_run(wrapper(), debug=debug)
File "/usr/local/lib/python3.10/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/usr/local/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
return future.result()
File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 287, in wrapper
return await func(*args)
File "/usr/local/lib/python3.10/site-packages/prefect/client/utilities.py", line 47, in with_injected_client
return await fn(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 266, in retrieve_flow_then_begin_flow_run
flow_run = await client.read_flow_run(flow_run_id)
File "/usr/local/lib/python3.10/site-packages/prefect/client/orion.py", line 1576, in read_flow_run
response = await self._client.get(f"/flow_runs/{flow_run_id}")
File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1757, in get
return await self.request(
File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1533, in request
return await self.send(request, auth=auth, follow_redirects=follow_redirects)
File "/usr/local/lib/python3.10/site-packages/prefect/client/base.py", line 229, in send
response = await self._send_with_retry(
File "/usr/local/lib/python3.10/site-packages/prefect/client/base.py", line 187, in _send_with_retry
response = await request()
File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1620, in send
response = await self._send_handling_auth(
File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1648, in _send_handling_auth
response = await self._send_handling_redirects(
File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1685, in _send_handling_redirects
response = await self._send_single_request(request)
File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1722, in _send_single_request
response = await transport.handle_async_request(request)
File "/usr/local/lib/python3.10/site-packages/httpx/_transports/default.py", line 352, in handle_async_request
with map_httpcore_exceptions():
File "/usr/local/lib/python3.10/contextlib.py", line 153, in __exit__
self.gen.throw(typ, value, traceback)
File "/usr/local/lib/python3.10/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
raise mapped_exc(message) from exc
httpx.ConnectError: All connection attempts failed
I’m not sure whats going on. It works fine when using Prefect Cloud, but it doesn’t work when using the local Prefect Orion API Server.
I’ve tried uninstalling and reinstalling Prefect and resetting the Orion database, but to no avail.
My Prefect version:
➜ ~ prefect version
Version: 2.8.0
API version: 0.8.4
Python version: 3.10.9
Git commit: 50f0ad0a
Built: Thu, Feb 9, 2023 3:38 PM
OS/Arch: darwin/arm64
Profile: default
Server type: hosted
I can see the Flow Runs in the API Server console but they are just in ‘Pending’ state and never run.
Would greatly appreciate any assistance.
