I’m trying to run a prefect worker on a workpool based on a docker infrastructure. I don’t know why, it always gives me problems when the worker submits the flow and the container is run up. I’ve searched a lot but I didn’t find anything. Maybe is something related to the network?
This is the error, from the start of the worker to the failed submission.
Worker 'DockerWorker dd84ceb0-ef07-42da-93b7-bd43ed98f287' started!
17:08:25.346 | INFO | prefect.flow_runs.worker - Worker 'DockerWorker dd84ceb0-ef07-42da-93b7-bd43ed98f287' submitting flow run 'b25fc6e1-bb76-43f1-84f2-542fab9c6130'
17:08:25.775 | INFO | prefect.worker.docker.dockerworker dd84ceb0-ef07-42da-93b7-bd43ed98f287 - Creating Docker container 'delicate-goose'...
17:08:26.079 | INFO | prefect.worker.docker.dockerworker dd84ceb0-ef07-42da-93b7-bd43ed98f287 - Docker container 'delicate-goose' has status 'created'
17:08:27.284 | INFO | prefect.worker.docker.dockerworker dd84ceb0-ef07-42da-93b7-bd43ed98f287 - Docker container 'delicate-goose' has status 'running'
17:08:27.346 | INFO | prefect.flow_runs.worker - Completed submission of flow run 'b25fc6e1-bb76-43f1-84f2-542fab9c6130'
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/httpx/_transports/default.py", line 69, in map_httpcore_exceptions
yield
File "/usr/local/lib/python3.9/site-packages/httpx/_transports/default.py", line 373, in handle_async_request
resp = await self._pool.handle_async_request(req)
File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 216, in handle_async_request
raise exc from None
File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 196, in handle_async_request
response = await connection.handle_async_request(
File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection.py", line 99, in handle_async_request
raise exc
File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection.py", line 76, in handle_async_request
stream = await self._connect(request)
File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection.py", line 122, in _connect
stream = await self._network_backend.connect_tcp(**kwargs)
File "/usr/local/lib/python3.9/site-packages/httpcore/_backends/auto.py", line 30, in connect_tcp
return await self._backend.connect_tcp(
File "/usr/local/lib/python3.9/site-packages/httpcore/_backends/anyio.py", line 123, in connect_tcp
stream._raw_socket.setsockopt(*option) # type: ignore[attr-defined] # pragma: no cover
File "/usr/local/lib/python3.9/contextlib.py", line 137, in __exit__
self.gen.throw(typ, value, traceback)
File "/usr/local/lib/python3.9/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
raise to_exc(exc) from exc
httpcore.ConnectError: [Errno -2] Name or service not known
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/prefect/cli/_utilities.py", line 41, in wrapper
return fn(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/prefect/utilities/asyncutils.py", line 259, in coroutine_wrapper
return call()
File "/usr/local/lib/python3.9/site-packages/prefect/_internal/concurrency/calls.py", line 432, in __call__
return self.result()
File "/usr/local/lib/python3.9/site-packages/prefect/_internal/concurrency/calls.py", line 318, in result
return self.future.result(timeout=timeout)
File "/usr/local/lib/python3.9/site-packages/prefect/_internal/concurrency/calls.py", line 179, in result
return self.__get_result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
raise self._exception
File "/usr/local/lib/python3.9/site-packages/prefect/_internal/concurrency/calls.py", line 389, in _run_async
result = await coro
File "/usr/local/lib/python3.9/site-packages/prefect/cli/flow_run.py", line 281, in execute
await Runner().execute_flow_run(id)
File "/usr/local/lib/python3.9/site-packages/prefect/runner/runner.py", line 484, in execute_flow_run
flow_run = await self._client.read_flow_run(flow_run_id)
File "/usr/local/lib/python3.9/site-packages/prefect/client/orchestration.py", line 1983, in read_flow_run
response = await self._client.get(f"/flow_runs/{flow_run_id}")
File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1801, in get
return await self.request(
File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1574, in request
return await self.send(request, auth=auth, follow_redirects=follow_redirects)
File "/usr/local/lib/python3.9/site-packages/prefect/client/base.py", line 325, in send
response = await self._send_with_retry(
File "/usr/local/lib/python3.9/site-packages/prefect/client/base.py", line 249, in _send_with_retry
response = await send(request, *send_args, **send_kwargs)
File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1661, in send
response = await self._send_handling_auth(
File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1689, in _send_handling_auth
response = await self._send_handling_redirects(
File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1726, in _send_handling_redirects
response = await self._send_single_request(request)
File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1763, in _send_single_request
response = await transport.handle_async_request(request)
File "/usr/local/lib/python3.9/site-packages/httpx/_transports/default.py", line 373, in handle_async_request
resp = await self._pool.handle_async_request(req)
File "/usr/local/lib/python3.9/contextlib.py", line 137, in __exit__
self.gen.throw(typ, value, traceback)
File "/usr/local/lib/python3.9/site-packages/httpx/_transports/default.py", line 86, in map_httpcore_exceptions
raise mapped_exc(message) from exc
httpx.ConnectError: [Errno -2] Name or service not known
An exception occurred.
17:08:37.203 | INFO | prefect.worker.docker.dockerworker dd84ceb0-ef07-42da-93b7-bd43ed98f287 - Docker container 'delicate-goose' has status 'exited'
17:08:37.222 | INFO | prefect.worker.docker.dockerworker dd84ceb0-ef07-42da-93b7-bd43ed98f287 - Docker container 'delicate-goose' has status 'exited'
17:08:37.379 | INFO | prefect.flow_runs.worker - Reported flow run 'b25fc6e1-bb76-43f1-84f2-542fab9c6130' as crashed: Flow run infrastructure exited with non-zero status code 1.
This is the .yaml for the deployment, it’s contained into /src/
# Welcome to your prefect.yaml file! You can use this file for storing and managing
# configuration for deploying your flows. We recommend committing this file to source
# control along with your flow code.
# Generic metadata about this project
name: src
prefect-version: 2.18.0
# build section allows you to manage and build docker images
build:
- prefect_docker.deployments.steps.build_docker_image:
requires: prefect-docker>=0.3.1
id: build-image
dockerfile: auto
image_name: darioran/tes1_v1
tag: test_docker_infra
# push section allows you to manage if and how this project is uploaded to remote locations
push: null
# pull section allows you to provide instructions for cloning this project in remote locations
pull:
- prefect.deployments.steps.set_working_directory:
directory: /opt/prefect/src
# the deployments section allows you to provide configuration for deploying flows
deployments:
- name: testDeploy
version: null
tags: []
description: null
entrypoint: main.py:get_repo_info
parameters: {}
work_pool:
name: TestPool
work_queue_name: null
job_variables:
image: '{{ build-image.image }}'
schedules: []
I’d like to be able to run a flow with this type of infrastructure