Prefect Agent in local failed to finish the workflow with Prefect Orion in Kubernetes

Hongbo-Miao · December 31, 2022, 3:20am

I am trying to run a flow together with Prefect Orion deployed by prefect-helm in a local Kubernetes created by Rancher Desktop.

Here is my minimum repo to reproduce my issue: https://github.com/Hongbo-Miao/issue-prefect

Any guide would be appreciate. Thanks!

Copied the steps below:

Steps to reproduce

In first terminal, run

# Install Prefect Orion
helm repo add prefect https://prefecthq.github.io/prefect-helm
helm install \
  prefect-orion \
  prefect/prefect-orion \
  --namespace=hm-prefect \
  --create-namespace \
  --values=prefect-orion/my-values.yaml
kubectl port-forward service/prefect-orion --namespace=hm-prefect 4200:4200

At this moment, you can open http://localhost:4200/ successfully.

In second terminal, run

# Set PREFECT_API_URL
prefect config set PREFECT_API_URL=http://localhost:4200/api

# (You can skip this step to use mine directly, or change to your Docker Hub username)
# Build hm-prefect-print-platform
docker build --file=print_platform/Dockerfile --tag=hongbomiao/hm-prefect-print-platform:latest .
docker push hongbomiao/hm-prefect-print-platform:latest

# Start hm-prefect-print-platform
python print_platform/add_kubernetes_job_block.py
prefect deployment build print_platform/main.py:print_platform \
  --name=print-platform \
  --infra-block=kubernetes-job/print-platform-kubernetes-job-block \
  --work-queue=hm-queue \
  --apply
prefect deployment run print-platform/print-platform

Now start the agent, and you will see the failure logs (these logs are streamed from pod. If you check the pod log, it would be same):

prefect agent start --work-queue=hm-queue

Starting v2.7.5 agent connected to http://localhost:4200/api...

  ___ ___ ___ ___ ___ ___ _____     _   ___ ___ _  _ _____
 | _ \ _ \ __| __| __/ __|_   _|   /_\ / __| __| \| |_   _|
 |  _/   / _|| _|| _| (__  | |    / _ \ (_ | _|| .` | | |
 |_| |_|_\___|_| |___\___| |_|   /_/ \_\___|___|_|\_| |_|


Agent started! Looking for work from queue(s): hm-queue...
18:55:33.995 | INFO    | prefect.agent - Submitting flow run '3ff9f120-533e-488f-a621-6044b5b65ec7'
18:55:34.469 | INFO    | prefect.infrastructure.kubernetes-job - Job 'ultra-kingfisher-rdlrg': Pod has status 'Pending'.
18:55:34.481 | INFO    | prefect.agent - Completed submission of flow run '3ff9f120-533e-488f-a621-6044b5b65ec7'
18:55:35.974 | INFO    | prefect.infrastructure.kubernetes-job - Job 'ultra-kingfisher-rdlrg': Pod has status 'Running'.
<frozen runpy>:128: RuntimeWarning: 'prefect.engine' found in sys.modules after import of package 'prefect', but prior to execution of 'prefect.engine'; this may result in unpredictable behaviour
02:55:46.368 | ERROR   | prefect.engine - Engine execution of flow run '3ff9f120-533e-488f-a621-6044b5b65ec7' exited with unexpected exception
anyio._backends._asyncio.ExceptionGroup: 2 exceptions were raised in the task group:
----------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/anyio/_core/_sockets.py", line 164, in try_connect
    stream = await asynclib.connect_tcp(remote_host, remote_port, local_address)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 1691, in connect_tcp
    await get_running_loop().create_connection(
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 1079, in create_connection
    raise exceptions[0]
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 1063, in create_connection
    sock = await self._connect_sock(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 967, in _connect_sock
    await self.sock_connect(sock, address)
  File "/usr/local/lib/python3.11/asyncio/selector_events.py", line 634, in sock_connect
    return await fut
           ^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/selector_events.py", line 674, in _sock_connect_cb
    raise OSError(err, f'Connect call failed {address}')
ConnectionRefusedError: [Errno 111] Connect call failed ('::1', 4200, 0, 0)
----------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/anyio/_core/_sockets.py", line 164, in try_connect
    stream = await asynclib.connect_tcp(remote_host, remote_port, local_address)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 1691, in connect_tcp
    await get_running_loop().create_connection(
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 1079, in create_connection
    raise exceptions[0]
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 1063, in create_connection
    sock = await self._connect_sock(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 967, in _connect_sock
    await self.sock_connect(sock, address)
  File "/usr/local/lib/python3.11/asyncio/selector_events.py", line 634, in sock_connect
    return await fut
           ^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/selector_events.py", line 674, in _sock_connect_cb
    raise OSError(err, f'Connect call failed {address}')
ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 4200)


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/httpcore/_exceptions.py", line 10, in map_exceptions
    yield
  File "/usr/local/lib/python3.11/site-packages/httpcore/backends/asyncio.py", line 111, in connect_tcp
    stream: anyio.abc.ByteStream = await anyio.connect_tcp(
                                   ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/anyio/_core/_sockets.py", line 222, in connect_tcp
    raise OSError("All connection attempts failed") from cause
OSError: All connection attempts failed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
    yield
  File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpcore/_async/connection_pool.py", line 253, in handle_async_request
    raise exc
  File "/usr/local/lib/python3.11/site-packages/httpcore/_async/connection_pool.py", line 237, in handle_async_request
    response = await connection.handle_async_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpcore/_async/connection.py", line 86, in handle_async_request
    raise exc
  File "/usr/local/lib/python3.11/site-packages/httpcore/_async/connection.py", line 63, in handle_async_request
    stream = await self._connect(request)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpcore/_async/connection.py", line 111, in _connect
    stream = await self._network_backend.connect_tcp(**kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpcore/backends/auto.py", line 29, in connect_tcp
    return await self._backend.connect_tcp(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpcore/backends/asyncio.py", line 109, in connect_tcp
    with map_exceptions(exc_map):
  File "/usr/local/lib/python3.11/contextlib.py", line 155, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/usr/local/lib/python3.11/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc)
httpcore.ConnectError: All connection attempts failed

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/prefect/engine.py", line 1898, in <module>
    enter_flow_run_engine_from_subprocess(flow_run_id)
  File "/usr/local/lib/python3.11/site-packages/prefect/engine.py", line 188, in enter_flow_run_engine_from_subprocess
    return anyio.run(retrieve_flow_then_begin_flow_run, flow_run_id)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/anyio/_core/_eventloop.py", line 70, in run
    return asynclib.run(func, *args, **backend_options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 292, in run
    return native_run(wrapper(), debug=debug)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 287, in wrapper
    return await func(*args)
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/client/utilities.py", line 47, in with_injected_client
    return await fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/engine.py", line 260, in retrieve_flow_then_begin_flow_run
    flow_run = await client.read_flow_run(flow_run_id)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/client/orion.py", line 1469, in read_flow_run
    response = await self._client.get(f"/flow_runs/{flow_run_id}")
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1757, in get
    return await self.request(
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1533, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/client/base.py", line 229, in send
    response = await self._send_with_retry(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/client/base.py", line 187, in _send_with_retry
    response = await request()
               ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1620, in send
    response = await self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1648, in _send_handling_auth
    response = await self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1685, in _send_handling_redirects
    response = await self._send_single_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1722, in _send_single_request
    response = await transport.handle_async_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 352, in handle_async_request
    with map_httpcore_exceptions():
  File "/usr/local/lib/python3.11/contextlib.py", line 155, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ConnectError: All connection attempts failed

Here is the pod screenshot:

anna_geller · January 2, 2023, 9:11pm

Hi, did you manage to solve it by now? This seems like a network issue, I suspect that the Orion API is not exposing the PREFECT_API_URL correctly, can you try kubectl port-forward prefect/prefect-orion --namespace=hm-prefect 4200:4200?

Great writeup!

Hongbo-Miao · January 3, 2023, 10:40pm

No, I haven’t made it.
I did have one step inside Steps to reproduce above using

kubectl port-forward service/prefect-orion --namespace=hm-prefect 4200:4200

anna_geller · January 3, 2023, 10:54pm

yes, I suggested prefect/prefect-orion

Hongbo-Miao · January 3, 2023, 10:58pm

Oh that doesn’t work. Here you are:

➜ kubectl get services --namespace=hm-prefect
NAME                          TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
prefect-orion-postgresql-hl   ClusterIP   None            <none>        5432/TCP   3d12h
prefect-orion-postgresql      ClusterIP   10.43.221.19    <none>        5432/TCP   3d12h
prefect-orion                 ClusterIP   10.43.176.145   <none>        4200/TCP   3d12h

➜ kubectl port-forward prefect/prefect-orion --namespace=hm-prefect 4200:4200
error: the server doesn't have a resource type "prefect"

➜ kubectl port-forward service/prefect-orion --namespace=hm-prefect 4200:4200
Forwarding from 127.0.0.1:4200 -> 4200
Forwarding from [::1]:4200 -> 4200

anna_geller · January 4, 2023, 10:15am

thanks for confirming that. I know it sounds silly, but can you delete all that and try again? You seem to have done everything correctly, perhaps some transient issue.

the alternative is to give Prefect Cloud a try or docker-compose:

https://discourse.prefect.io/t/running-prefect-2-with-docker-compose/1516

Hongbo-Miao · January 5, 2023, 6:08pm

I redeployed multiple times. Unfortunately, it does not work.

I also tried to add a Traefik Ingress Controller by pointing to a different port 40000. And I got same error. Here are steps:

Install prefect-orion by new PREFECT_API_URL:

helm install \
  prefect-orion \
  prefect/prefect-orion \
  --namespace=hm-prefect \
  --create-namespace \
  --values=prefect-orion/my-values.yaml

prefect-orion/my-values.yaml

orion:
  env:
    - name: PREFECT_API_URL
      value: http://localhost:40000/api
postgresql:
  auth:
    # username: prefect
    password: passw0rd

Install Traefik Ingress Controller

helm repo add traefik https://traefik.github.io/charts
helm install \
  traefik \
  traefik/traefik \
  --namespace=hm-traefik \
  --create-namespace

Install Traefik Ingress for the namespace hm-prefect

kubectl apply --filename=traefik/hm-prefect-traefik-ingress.yaml

traefik/hm-prefect-traefik-ingress.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: hm-traefik-ingress
  namespace: hm-prefect
  annotations:
    kubernetes.io/ingress.class: traefik
  labels:
    app.kubernetes.io/name: ingress
spec:
  rules:
    - http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: prefect-orion
                port:
                  number: 4200

Port forward Traefik

kubectl port-forward service/traefik --namespace=hm-traefik 40000:80

Set PREFECT_API_URL in local

prefect config set PREFECT_API_URL=http://localhost:40000/api

Re-run

# (You can skip this step to use mine directly, or change to your Docker Hub username)
# Build hm-prefect-print-platform
docker build --file=print_platform/Dockerfile --tag=hongbomiao/hm-prefect-print-platform:latest .
docker push hongbomiao/hm-prefect-print-platform:latest

# Start hm-prefect-print-platform
python print_platform/add_kubernetes_job_block.py
prefect deployment build print_platform/main.py:print_platform \
  --name=print-platform \
  --infra-block=kubernetes-job/print-platform-kubernetes-job-block \
  --work-queue=hm-queue \
  --apply
prefect deployment run print-platform/print-platform

At this moment, all still look good.

Start the agent locally, and now will see similar error:

prefect agent start --work-queue=hm-queue

Starting v2.7.5 agent connected to http://localhost:40000/api...

  ___ ___ ___ ___ ___ ___ _____     _   ___ ___ _  _ _____
 | _ \ _ \ __| __| __/ __|_   _|   /_\ / __| __| \| |_   _|
 |  _/   / _|| _|| _| (__  | |    / _ \ (_ | _|| .` | | |
 |_| |_|_\___|_| |___\___| |_|   /_/ \_\___|___|_|\_| |_|


Agent started! Looking for work from queue(s): hm-queue...
10:05:53.379 | INFO    | prefect.agent - Submitting flow run '106ad98e-ff8a-465c-b219-f587616abb22'
10:05:55.925 | INFO    | prefect.infrastructure.kubernetes-job - Job 'quirky-bullfinch-d2vkq': Pod has status 'Pending'.
10:05:55.927 | INFO    | prefect.agent - Completed submission of flow run '106ad98e-ff8a-465c-b219-f587616abb22'
10:06:08.084 | INFO    | prefect.infrastructure.kubernetes-job - Job 'quirky-bullfinch-d2vkq': Pod has status 'Running'.
<frozen runpy>:128: RuntimeWarning: 'prefect.engine' found in sys.modules after import of package 'prefect', but prior to execution of 'prefect.engine'; this may result in unpredictable behaviour
18:06:18.039 | ERROR   | prefect.engine - Engine execution of flow run '106ad98e-ff8a-465c-b219-f587616abb22' exited with unexpected exception
anyio._backends._asyncio.ExceptionGroup: 2 exceptions were raised in the task group:
----------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/anyio/_core/_sockets.py", line 164, in try_connect
    stream = await asynclib.connect_tcp(remote_host, remote_port, local_address)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 1691, in connect_tcp
    await get_running_loop().create_connection(
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 1079, in create_connection
    raise exceptions[0]
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 1063, in create_connection
    sock = await self._connect_sock(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 967, in _connect_sock
    await self.sock_connect(sock, address)
  File "/usr/local/lib/python3.11/asyncio/selector_events.py", line 634, in sock_connect
    return await fut
           ^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/selector_events.py", line 674, in _sock_connect_cb
    raise OSError(err, f'Connect call failed {address}')
ConnectionRefusedError: [Errno 111] Connect call failed ('::1', 40000, 0, 0)
----------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/anyio/_core/_sockets.py", line 164, in try_connect
    stream = await asynclib.connect_tcp(remote_host, remote_port, local_address)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 1691, in connect_tcp
    await get_running_loop().create_connection(
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 1079, in create_connection
    raise exceptions[0]
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 1063, in create_connection
    sock = await self._connect_sock(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 967, in _connect_sock
    await self.sock_connect(sock, address)
  File "/usr/local/lib/python3.11/asyncio/selector_events.py", line 634, in sock_connect
    return await fut
           ^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/selector_events.py", line 674, in _sock_connect_cb
    raise OSError(err, f'Connect call failed {address}')
ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 40000)


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/httpcore/_exceptions.py", line 10, in map_exceptions
    yield
  File "/usr/local/lib/python3.11/site-packages/httpcore/backends/asyncio.py", line 111, in connect_tcp
    stream: anyio.abc.ByteStream = await anyio.connect_tcp(
                                   ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/anyio/_core/_sockets.py", line 222, in connect_tcp
    raise OSError("All connection attempts failed") from cause
OSError: All connection attempts failed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
    yield
  File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpcore/_async/connection_pool.py", line 253, in handle_async_request
    raise exc
  File "/usr/local/lib/python3.11/site-packages/httpcore/_async/connection_pool.py", line 237, in handle_async_request
    response = await connection.handle_async_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpcore/_async/connection.py", line 86, in handle_async_request
    raise exc
  File "/usr/local/lib/python3.11/site-packages/httpcore/_async/connection.py", line 63, in handle_async_request
    stream = await self._connect(request)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpcore/_async/connection.py", line 111, in _connect
    stream = await self._network_backend.connect_tcp(**kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpcore/backends/auto.py", line 29, in connect_tcp
    return await self._backend.connect_tcp(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpcore/backends/asyncio.py", line 109, in connect_tcp
    with map_exceptions(exc_map):
  File "/usr/local/lib/python3.11/contextlib.py", line 155, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/usr/local/lib/python3.11/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc)
httpcore.ConnectError: All connection attempts failed

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/prefect/engine.py", line 1898, in <module>
    enter_flow_run_engine_from_subprocess(flow_run_id)
  File "/usr/local/lib/python3.11/site-packages/prefect/engine.py", line 188, in enter_flow_run_engine_from_subprocess
    return anyio.run(retrieve_flow_then_begin_flow_run, flow_run_id)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/anyio/_core/_eventloop.py", line 70, in run
    return asynclib.run(func, *args, **backend_options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 292, in run
    return native_run(wrapper(), debug=debug)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 287, in wrapper
    return await func(*args)
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/client/utilities.py", line 47, in with_injected_client
    return await fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/engine.py", line 260, in retrieve_flow_then_begin_flow_run
    flow_run = await client.read_flow_run(flow_run_id)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/client/orion.py", line 1469, in read_flow_run
    response = await self._client.get(f"/flow_runs/{flow_run_id}")
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1757, in get
    return await self.request(
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1533, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/client/base.py", line 229, in send
    response = await self._send_with_retry(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/client/base.py", line 187, in _send_with_retry
    response = await request()
               ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1620, in send
    response = await self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1648, in _send_handling_auth
    response = await self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1685, in _send_handling_redirects
    response = await self._send_single_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1722, in _send_single_request
    response = await transport.handle_async_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 352, in handle_async_request
    with map_httpcore_exceptions():
  File "/usr/local/lib/python3.11/contextlib.py", line 155, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ConnectError: All connection attempts failed

Hongbo-Miao · February 3, 2023, 12:57am

Got an answer from Ryan Peden in Slack that I can use PREFECT_API_URL for DNS name, and PREFECT_ORION_UI_API_URL for different API URL just for the web UI. Will try add report back.

Hongbo-Miao · February 3, 2023, 1:32pm

I finally made it work! Thanks for the help.

Here is the “fix” pull request: https://github.com/Hongbo-Miao/issue-prefect/pull/1/files

These files got updated to new ones:

print_platform/Dockerfile

FROM docker.io/prefecthq/prefect:2.7.5-python3.11

COPY ["print_platform/requirements.txt", "./"]
RUN pip install --no-cache-dir --requirement=requirements.txt

COPY ["print_platform/", "/opt/prefect/flows/print_platform/"]

prefect-orion/my-values.yaml

orion:
  env:
    - name: PREFECT_API_URL
      value: http://prefect-orion.hm-prefect.svc:4200/api
    - name: PREFECT_ORION_UI_API_URL
      value: http://localhost:4200/api

postgresql:
  auth:
    password: passw0rd

Hongbo-Miao · February 24, 2023, 1:23am

Adding another follow up, when I succeed at that time, maybe because of the Prefect agent running in the Kubernetes took the workflow job.

But if you want a Prefect agent in the local take the workflow job, here is the solution ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 4200) - #3 by Hongbo-Miao

Hope it saves some people time who ends here.

Topic		Replies	Views
Connection Error upon triggering deployment Help prefect-2-0 , kubernetes	3	2268	February 24, 2023
Deploying Prefect Agents on Kubernetes Archive prefect-2-0 , kubernetes , agent , kubernetes-agent , agent-setup , cloud-2-0	0	6162	August 2, 2022
Guide for testing out KubernetesFlowRunner locally Archive prefect-2-0 , local-development , kubernetes , kubernetes-flow-runner	0	582	July 1, 2022
Api dont start in Kubernetes Archive prefect-2-0 , kubernetes , ui	1	525	September 26, 2022
New Orion release 2.0a11 featuring the Kubernetes integration - you can now run all Orion components on Kubernetes! Announcements prefect-2-0 , release-notes , cli , flow-runner , kubernetes , infrastructure , kubernetes-flow-runner	0	950	February 15, 2022

Prefect Agent in local failed to finish the workflow with Prefect Orion in Kubernetes

Steps to reproduce

Related Topics