Docker agent cannot communicate with a Prefect Server Postgres database instance running locally on a host VM

[marfel]

Hi,
in my development setup, a PostgreSQL instance on my local host provides both the prefect database (via --external-postgres) and another database my Flow needs to access. The prefect server runs on the same host.

• If I start a LocalAgent and submit the Flow, it runs fine.
• If I start my custom container with docker run -it --network host my_image:latest bash and run the Flow from within the container manually, it runs fine.
• If I start a DockerAgent with prefect agent docker start --show-flow-logs --network host and change the Flow run config accordingly, I get the long error message below.

In case you are wondering: I don’t want put the Flow’s PG database into another container, because in production both databases are provided by a dedicated PG server running on a different machine. I figure switching the database from localhost to a different host in production should not be so difficult, should it?

Since I’m pretty much a Docker newbie, I would be very happy if someone could point me in the right direction here. I’m at a loss as to where to start looking for a solution.

Thanks in advance!

[2022-03-14 15:44:07,698] INFO - agent | Deploying flow run 2c27032d-a273-4e6e-a5b5-4708ac055ab4 to execution environment...                                                                                                                
[2022-03-14 15:44:07,997] INFO - agent | Completed deployment of flow run 2c27032d-a273-4e6e-a5b5-4708ac055ab4                                                                                                                              
Traceback (most recent call last):                                                                                                                                                                                                          
  File "/usr/local/lib/python3.9/site-packages/urllib3/connection.py", line 174, in _new_conn                                                                                                                                               
    conn = connection.create_connection(                                                                                                                                                                                                    
  File "/usr/local/lib/python3.9/site-packages/urllib3/util/connection.py", line 95, in create_connection                                                                                                                                   
    raise err                                                                                                                                                                                                                               
  File "/usr/local/lib/python3.9/site-packages/urllib3/util/connection.py", line 85, in create_connection                                                                                                                                   
    sock.connect(sa)                                                                                                                                                                                                                        
OSError: [Errno 101] Network is unreachable                                                                                                                                                                                                 
                                                                                                                                                                                                                                            
During handling of the above exception, another exception occurred:                                                                                                                                                                         
                                                                                                                                                                                                                                            
Traceback (most recent call last):                                                                                                                                                                                                          
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 703, in urlopen                                                                                                                                             
    httplib_response = self._make_request(                                                                                                                                                                                                  
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 398, in _make_request                                                                                                                                       
    conn.request(method, url, **httplib_request_kw)                                                                                                                                                                                         
  File "/usr/local/lib/python3.9/site-packages/urllib3/connection.py", line 239, in request                                                                                                                                                 
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)                                                                                                                                                            
  File "/usr/local/lib/python3.9/http/client.py", line 1285, in request                                                                                                                                                                     
    self._send_request(method, url, body, headers, encode_chunked)                                                                                                                                                                          
  File "/usr/local/lib/python3.9/http/client.py", line 1331, in _send_request                                                                                                                                                               
    self.endheaders(body, encode_chunked=encode_chunked)                                                                                                                                                                                    
  File "/usr/local/lib/python3.9/http/client.py", line 1280, in endheaders                                                                                                                                                                  
    self._send_output(message_body, encode_chunked=encode_chunked)                                                                                                                                                                          
  File "/usr/local/lib/python3.9/http/client.py", line 1040, in _send_output                                                                                                                                                                
    self.send(msg)                                                                                                                                                                                                                          
  File "/usr/local/lib/python3.9/http/client.py", line 980, in send                                                                                                                                                                         
    self.connect()                                                                                                                                                                                                                          
  File "/usr/local/lib/python3.9/site-packages/urllib3/connection.py", line 205, in connect                                                                                                                                                 
    conn = self._new_conn()                                                                                                                                                                                                                 
  File "/usr/local/lib/python3.9/site-packages/urllib3/connection.py", line 186, in _new_conn                                                                                                                                               
    raise NewConnectionError(                                                                                                                                                                                                               
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f36cfd0cfa0>: Failed to establish a new connection: [Errno 101] Network is unreachable                                                               
                                                                                                                                                                                                                                            
During handling of the above exception, another exception occurred:                                                                                                                                                                         
                                                                                                                                                                                                                                            
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 440, in send
    resp = conn.urlopen(
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 785, in urlopen
    retries = retries.increment(
  File "/usr/local/lib/python3.9/site-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='host.docker.internal', port=4200): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f36cfd0cfa0>: Failed to establ
ish a new connection: [Errno 101] Network is unreachable')) 

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/prefect", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx)) 
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx)) 
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/prefect/cli/execute.py", line 53, in flow_run
    result = client.graphql(query)
  File "/usr/local/lib/python3.9/site-packages/prefect/client/client.py", line 452, in graphql
    result = self.post(
  File "/usr/local/lib/python3.9/site-packages/prefect/client/client.py", line 407, in post
    response = self._request(
  File "/usr/local/lib/python3.9/site-packages/prefect/client/client.py", line 641, in _request
    response = self._send_request(
  File "/usr/local/lib/python3.9/site-packages/prefect/client/client.py", line 506, in _send_request
    response = session.post(
  File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 577, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 529, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 645, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 519, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='host.docker.internal', port=4200): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f36cfd0cfa0>: Failed to est
ablish a new connection: [Errno 101] Network is unreachable'))

[anna_geller]

To point you in the right direction, I would like to reproduce the issue first. Can you explain how did you set up the database?

1. Does it just run on the localhost of the VM?
2. Or was it deployed through a cloud service such as e.g. on AWS RDS?

If #2 is true, it would be the easiest to set up as you wouldn’t have to worry about setting up a bridge between a host VM’s local network and a Docker container’s network. Simultaneously this would make it much easier to move later to your production database.

[anna_geller]

@marfel

I figure switching the database from localhost to a different host in production should not be so difficult, should it?

Regarding that - it’s actually much easier to connect to an external database reachable via a public network interface rather than connecting to localhost as it’s quite complicated to reach a DB running locally on a VM host from a docker container running in Docker network.

You’re getting the error Network is not reachable because your DB is running on localhost while your Docker components communicate via an internal Docker network host.docker.internal. So it’s easiest to either connect to an external DB instance deployed somewhere in a public cloud (e.g. AWS RDS) or use a Postgres DB deployed as a container, as it’s the case in the default Server setting.

Does this explanation make sense to you?

[anna_geller]

If all of the above explanations didn’t help, please share:

• how did you define your Server start-up command,
• your Docker agent full startup command,
• and your flow definition (incl. your DockerRun) so that I can try to reproduce it.

Also note that the Docker agent itself shouldn’t be created within a container, but rather a local process:

Can I run a Docker agent in a container?

In general, the Docker agent is supposed to run in a local process (rather than in a docker container), and this local process is a layer between Prefect backend and a Docker Daemon. This agent polls the API for new flow runs, and if there are new scheduled runs, it then creates new flow runs and deploys those as Docker containers on the same machine as the agent. When the Docker agent is running within a container itself (rather than a local process), your flow runs end up deployed as contain…

[marfel]

Thanks Anna for your quick and comprehensive reply!

Now, let me try to clear things up some more:

1. For reasons that are out of my hands, we can not use cloud services (yet).
2. I’m working on a physical machine (my notebook), not on a VM.
3. The Postgres DBMS is running directly on this machine.
4. If its easier to connect to a Postgres instance via an IP address, I have no problems allowing access via to the (development) databases via any IP - I trust our firewall. In fact, that is already the case for Prefect, which is configured with this config.toml:

[server.database]
connection_url = "postgresql://postgres:postgres@192.168.0.89:5432/prefect"

To start the server, I use:

prefect server start --external-postgres

I’m not sure what you mean by “Docker agent full startup command”. Apart from a few volume mappings, which I didn’t deem relevant here, there is nothing else. Should there be? The command I mentioned in my post is also being run directly on my notebook.

So, with my very limited container knowledge I gather from your comments that if I docker run --network host a standalone container and execute the Flow inside it with flow.run(), it works because there are no other networks or containers involved. But if I start the DockerAgent and execute the Flow via the Prefect GUI, the Agent instanciates the specified container, attaching it to a different network? Or does it? The error message looks to me as if the image fails at trying to communicate with the Prefect backend, not with my DB?

[anna_geller]

Thanks for explaining your use case and providing more details. I was able to replicate your issue.

The problem

The problem you see (as you might have guessed) is related to networking. You correctly connected your locally running Postgres database instance to your Prefect Server - that’s why everything works fine when using a local agent.

In order to use Docker agent with your Server, the Docker agent needs to be in the same Docker network, otherwise you’ll see errors such as:

1. The urllib3.exceptions.MaxRetryError - exactly as the one you got:

urllib3.exceptions.MaxRetryError: 
HTTPConnectionPool(host='host.docker.internal', port=4200): 
Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f36cfd0cfa0>: 
Failed to establish a new connection: [Errno 101] Network is unreachable')) 

2. The requests.exceptions.ConnectionError - the same root cause, but a different exception.

requests.exceptions.ConnectionError: 
HTTPConnectionPool(host='host.docker.internal', port=4200): 
Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f3eacb1b0d0>: 
Failed to establish a new connection: [Errno 101] Network is unreachable'))

The solution

By default, when you start Prefect Server, Prefect creates a Docker network named prefect-server.

Example command to use a locally running Postgres instance (I used port 4203 because I’m using 4200 for Orion):

prefect server start --server-port 4203 --external-postgres \
--postgres-url postgresql://your_host_name@host.docker.internal:5432/postgres

This generates the following output:

Pulling hasura  ... done
Pulling graphql ... done
Pulling apollo  ... done
Pulling towel   ... done
Pulling ui      ... done
Creating network "prefect-server" with the default driver

The last line shows that the Docker network “prefect-server” has been created.

Now, to start your Docker agent, use:

prefect agent docker start --network prefect-server --label docker  --show-flow-logs

Using the following sample flow, we can confirm that the agent is now able to deploy the flow run container with the Docker agent:

from prefect import Flow, task
from prefect.storage import Docker
from prefect.run_configs import DockerRun


FLOW_NAME = "docker_pickle_docker_run_local_image"
docker_storage = Docker(image_name="community", image_tag="latest",)


@task(log_stdout=True)
def hello_world():
    text = f"hello from {FLOW_NAME}"
    print(text)
    return text


with Flow(
    FLOW_NAME,
    storage=docker_storage,
    run_config=DockerRun(image="community:latest", labels=["docker"],),
) as flow:
    hw = hello_world()

Let me know if you have any additional questions about that.

[marfel]

Thanks Anna, I got it to run now! I’m not sure if your_host_name@host.docker.internal is correct, though - isn’t that supposed to be the user@hostname?

Your example worked fine, but when adding sqlalchemy, the cloudpickler said `TypeError: cannot pickle ‘sqlalchemy.cprocessors.UnicodeResultProcessor’ object.

So, now I finally understand the difference between script storage and pickle storage. The former did the trick, after some fiddling with the Docker storage configuration. I think part of my problem was not to specify any Storage option in the first place.

Anyway, after throwing out the localhosts everywhere and replacing it with my ip address 192.168.0.89, I was able to access everything with a humble

prefect server start --external-postgres 
prefect agent docker start --network prefect-server --label docker --show-flow-logs

Thanks again for your help!!