ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 4200)

Help
,
3

I found a solution! :smiley:

First let me clarify, my Prefect server is deployed in Kubernetes. I hope to run a Prefect agent on my computer locally to do some jobs (note it is not a Prefect agent in Kubernetes).

When the Prefect agent running in my local computer and starts to take a job from the queue, it will still spin a job pod in my Kubernetes with same log printed. I am not sure why this job pod got created even I am using local agent, but I am guessing this job pod helps preserve the job log in Kubernetes, and helps bridge the Prefect agent in local and Prefect server (Please correct me if I am wrong).

This local Prefect agent is able to communicate with the Prefect Server which I port-forwarded at http://localhost:4200/api

However, the corresponding job pod cannot talk to Prefect Server using http://localhost:4200/api, in my case, it has to be http://prefect-server.hm-prefect.svc:4200/api

But here is the issue, I am not aware of any way to provide two endpoints for Prefect Agent. So here is the solution:

  1. Using https://ngrok.com/ to get a public URL by

    ngrok http 4200

    This will give me a public URL something like https://xxxxx.ngrok.io

  2. Deploy Prefect Server to Kubernetes

    helm install \
  prefect-server \
  prefect/prefect-server \
  --namespace=hm-prefect \
  --create-namespace \
  --values=prefect-server/my-values.yaml

    prefect-server/my-values.yaml

    server:
  publicApiUrl: https://xxxxx.ngrok.io/api  # <- here has to be the public URL you got in step 1
postgresql:
  auth:
    # username: prefect
    password: passw0rd

  3. Set PREFECT_API_URL

    Below has to be the public URL you got in step 1

    prefect config set PREFECT_API_URL=https://xxxxx.ngrok.io/api

  4. Start the job

  5. Start Prefect agent in local

Now both the Prefect agent in local and the corresponding job pod got created can talk to Prefect server.

In real life when you deploy to AWS or Google Cloud, you won’t have this issue, because there is a public endpoint for Prefect server always.

Hope it helps! :smiley:

2 Likes