Docker agent cannot communicate with a Prefect Server Postgres database instance running locally on a host VM

Archive
, , , , , , ,
6

Thanks for explaining your use case and providing more details. I was able to replicate your issue.

The problem

The problem you see (as you might have guessed) is related to networking. You correctly connected your locally running Postgres database instance to your Prefect Server - that’s why everything works fine when using a local agent.

In order to use Docker agent with your Server, the Docker agent needs to be in the same Docker network, otherwise you’ll see errors such as:

  1. The urllib3.exceptions.MaxRetryError - exactly as the one you got:
urllib3.exceptions.MaxRetryError: 
HTTPConnectionPool(host='host.docker.internal', port=4200): 
Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f36cfd0cfa0>: 
Failed to establish a new connection: [Errno 101] Network is unreachable'))
  1. The requests.exceptions.ConnectionError - the same root cause, but a different exception.
requests.exceptions.ConnectionError: 
HTTPConnectionPool(host='host.docker.internal', port=4200): 
Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f3eacb1b0d0>: 
Failed to establish a new connection: [Errno 101] Network is unreachable'))

The solution

By default, when you start Prefect Server, Prefect creates a Docker network named prefect-server.

Example command to use a locally running Postgres instance (I used port 4203 because I’m using 4200 for Orion):

prefect server start --server-port 4203 --external-postgres \
--postgres-url postgresql://your_host_name@host.docker.internal:5432/postgres

This generates the following output:

Pulling hasura  ... done
Pulling graphql ... done
Pulling apollo  ... done
Pulling towel   ... done
Pulling ui      ... done
Creating network "prefect-server" with the default driver

The last line shows that the Docker network “prefect-server” has been created.

Now, to start your Docker agent, use:

prefect agent docker start --network prefect-server --label docker  --show-flow-logs

Using the following sample flow, we can confirm that the agent is now able to deploy the flow run container with the Docker agent:

from prefect import Flow, task
from prefect.storage import Docker
from prefect.run_configs import DockerRun


FLOW_NAME = "docker_pickle_docker_run_local_image"
docker_storage = Docker(image_name="community", image_tag="latest",)


@task(log_stdout=True)
def hello_world():
    text = f"hello from {FLOW_NAME}"
    print(text)
    return text


with Flow(
    FLOW_NAME,
    storage=docker_storage,
    run_config=DockerRun(image="community:latest", labels=["docker"],),
) as flow:
    hw = hello_world()

Let me know if you have any additional questions about that.