Prefect scheduling not working with docker-compose

Archive
, ,
1

Flow.py:

from prefect import flow, task
from prefect.deployments import Deployment
from prefect.orion.schemas.schedules import IntervalSchedule, RRuleSchedule, CronSchedule
from datetime import datetime, timedelta



@task
def say_hello(name):
    print(f"hello {name}")

@task
def say_goodbye(name):
    print(f"goodbye {name}")

@flow
def greetings(names=["arthur", "trillian", "ford", "marvin"]):
    for name in names:
        say_hello(name)
        say_goodbye(name)

if __name__ == "__main__":
    deployment = Deployment.build_from_flow(
        name="get_weather_docker_example",  
        schedule = IntervalSchedule(interval=timedelta(seconds=30)),
        flow=greetings,        
        work_queue_name="demo"
    )
    deployment.apply()

docker-compose.yml

version: "3.9"
services:

  ### Prefect Database
  database:
    image: postgres:14.1-alpine
    restart: always
    environment:
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=postgres
      - POSTGRES_DB=orion
    expose:
      - 5432
    volumes: 
      - db:/var/lib/postgresql/data
    profiles: ["orion"]
  

  ### Prefect Orion API
  orion:
    image: prefecthq/prefect:2.6.8-python3.11
    restart: always
    volumes:
      - prefect:/root/.prefect
    entrypoint: ["prefect", "orion", "start"]
    environment:
      - PREFECT_ORION_API_HOST=0.0.0.0
      - PREFECT_ORION_DATABASE_CONNECTION_URL=postgresql+asyncpg://postgres:postgres@database:5432/orion
    ports:
      - 4200:4200
    depends_on:
      - database
    profiles: ["orion"]

  ## Prefect Agent
  agent:
    image: prefecthq/prefect:2.6.8-python3.11
    restart: always
    entrypoint: ["prefect", "agent", "start", "-q", "demo"]
    environment:
      - PREFECT_API_URL=http://orion:4200/api
    depends_on:
      - orion
    profiles: ["agent"]


  ### Prefect CLI
  cli:
    image: prefecthq/prefect:2.6.8-python3.11
    entrypoint: ["python", "flow.py"]
    working_dir: "/root/flows"
    volumes:
      - "./flows:/root/flows"
    environment:
      - PREFECT_API_URL=http://orion:4200/api
    profiles: ["cli"]


volumes:
  prefect:
  db:
  minio:
networks:
  default:
    name: prefect-network


flows/flow.py and docker-compose.yml are in same directory

command to run the docker-compose:

docker compose --profile orion --profile agent --profile cli up

Logs:

prefect-docker-compose-database-1  | 
prefect-docker-compose-database-1  | PostgreSQL Database directory appears to contain a database; Skipping initialization
prefect-docker-compose-database-1  | 
prefect-docker-compose-database-1  | 2022-11-25 11:54:21.293 UTC [1] LOG:  starting PostgreSQL 14.1 on x86_64-pc-linux-musl, compiled by gcc (Alpine 10.3.1_git20211027) 10.3.1 20211027, 64-bit
prefect-docker-compose-database-1  | 2022-11-25 11:54:21.293 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
prefect-docker-compose-database-1  | 2022-11-25 11:54:21.293 UTC [1] LOG:  listening on IPv6 address "::", port 5432
prefect-docker-compose-database-1  | 2022-11-25 11:54:21.296 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
prefect-docker-compose-database-1  | 2022-11-25 11:54:21.300 UTC [21] LOG:  database system was shut down at 2022-11-25 11:53:37 UTC
prefect-docker-compose-database-1  | 2022-11-25 11:54:21.305 UTC [1] LOG:  database system is ready to accept connections
prefect-docker-compose-orion-1     | 
prefect-docker-compose-orion-1     |  ___ ___ ___ ___ ___ ___ _____    ___  ___ ___ ___  _  _
prefect-docker-compose-orion-1     | | _ \ _ \ __| __| __/ __|_   _|  / _ \| _ \_ _/ _ \| \| |
prefect-docker-compose-orion-1     | |  _/   / _|| _|| _| (__  | |   | (_) |   /| | (_) | .` |
prefect-docker-compose-orion-1     | |_| |_|_\___|_| |___\___| |_|    \___/|_|_\___\___/|_|\_|
prefect-docker-compose-orion-1     | 
prefect-docker-compose-orion-1     | Configure Prefect to communicate with the server with:
prefect-docker-compose-orion-1     | 
prefect-docker-compose-orion-1     |     prefect config set PREFECT_API_URL=http://0.0.0.0:4200/api
prefect-docker-compose-orion-1     | 
prefect-docker-compose-orion-1     | View the API reference documentation at http://0.0.0.0:4200/docs
prefect-docker-compose-orion-1     | 
prefect-docker-compose-orion-1     | Check out the dashboard at http://0.0.0.0:4200
prefect-docker-compose-orion-1     | 
prefect-docker-compose-orion-1     | 
prefect-docker-compose-orion-1     | 
prefect-docker-compose-agent-1     | Starting v2.6.8 agent connected to http://orion:4200/api...
prefect-docker-compose-agent-1     | 
prefect-docker-compose-agent-1     |   ___ ___ ___ ___ ___ ___ _____     _   ___ ___ _  _ _____
prefect-docker-compose-agent-1     |  | _ \ _ \ __| __| __/ __|_   _|   /_\ / __| __| \| |_   _|
prefect-docker-compose-agent-1     |  |  _/   / _|| _|| _| (__  | |    / _ \ (_ | _|| .` | | |
prefect-docker-compose-agent-1     |  |_| |_|_\___|_| |___\___| |_|   /_/ \_\___|___|_|\_| |_|
prefect-docker-compose-agent-1     | 
prefect-docker-compose-agent-1     | 
prefect-docker-compose-agent-1     | Agent started! Looking for work from queue(s): demo...
prefect-docker-compose-orion-1     | INFO:     Started server process [7]
prefect-docker-compose-orion-1     | INFO:     Waiting for application startup.
prefect-docker-compose-orion-1     | INFO:     Application startup complete.
prefect-docker-compose-orion-1     | INFO:     Uvicorn running on http://0.0.0.0:4200 (Press CTRL+C to quit)
prefect-docker-compose-orion-1     | INFO:     172.21.0.5:55756 - "GET /deployments/name/greetings/get_weather_docker_example HTTP/1.1" 404 Not Found
prefect-docker-compose-orion-1     | INFO:     172.21.0.5:55770 - "POST /flows/ HTTP/1.1" 201 Created
prefect-docker-compose-orion-1     | INFO:     172.21.0.5:55780 - "GET /block_types/slug/process HTTP/1.1" 200 OK
prefect-docker-compose-orion-1     | INFO:     172.21.0.5:55780 - "PATCH /block_types/9003f2ea-3325-4191-a0a3-67497f162165 HTTP/1.1" 204 No Content
prefect-docker-compose-orion-1     | INFO:     172.21.0.5:55780 - "GET /block_schemas/checksum/sha256%3A47c4ac364708f4a6f27fb10b18086ad92c0a53fbbdd9ba07c030467067979f84?version=2.6.8 HTTP/1.1" 200 OK
prefect-docker-compose-orion-1     | INFO:     172.21.0.5:55780 - "POST /block_documents/ HTTP/1.1" 201 Created
prefect-docker-compose-orion-1     | INFO:     172.21.0.5:55770 - "POST /deployments/ HTTP/1.1" 201 Created
prefect-docker-compose-cli-1 exited with code 0
prefect-docker-compose-orion-1     | INFO:     172.21.0.1:46148 - "GET /assets/index.85f05129.css HTTP/1.1" 200 OK
prefect-docker-compose-orion-1     | INFO:     172.21.0.1:46132 - "GET /assets/index.f501c99b.js HTTP/1.1" 200 OK
prefect-docker-compose-orion-1     | INFO:     172.21.0.1:46132 - "GET /ui-settings HTTP/1.1" 200 OK
prefect-docker-compose-orion-1     | INFO:     172.21.0.1:46132 - "GET /health HTTP/1.1" 200 OK
prefect-docker-compose-orion-1     | INFO:     172.21.0.1:46132 - "POST /flow_runs/count HTTP/1.1" 200 OK
prefect-docker-compose-orion-1     | INFO:     172.21.0.1:46132 - "POST /flow_runs/count HTTP/1.1" 200 OK
prefect-docker-compose-orion-1     | INFO:     172.21.0.1:46162 - "POST /saved_searches/filter HTTP/1.1" 200 OK
prefect-docker-compose-orion-1     | INFO:     172.21.0.1:46148 - "POST /flow_runs/filter HTTP/1.1" 200 OK
prefect-docker-compose-orion-1     | INFO:     172.21.0.1:46132 - "POST /ui/flow_runs/history HTTP/1.1" 200 OK
prefect-docker-compose-orion-1     | INFO:     172.21.0.1:46172 - "POST /deployments/filter HTTP/1.1" 200 OK
prefect-docker-compose-orion-1     | INFO:     172.21.0.1:46162 - "POST /work_queues/filter HTTP/1.1" 200 OK
prefect-docker-compose-orion-1     | INFO:     172.21.0.1:46164 - "POST /flows/filter HTTP/1.1" 200 OK
prefect-docker-compose-orion-1     | INFO:     172.21.0.1:46148 - "GET /flows/91f4a98a-877d-4c40-93c7-41236a23d2e5 HTTP/1.1" 200 OK
prefect-docker-compose-orion-1     | INFO:     172.21.0.1:46148 - "GET /deployments/c9398a97-4ecc-4b57-8acb-0a50d5a7b328 HTTP/1.1" 200 OK
prefect-docker-compose-orion-1     | INFO:     172.21.0.1:46164 - "GET /work_queues/name/demo HTTP/1.1" 200 OK
prefect-docker-compose-orion-1     | INFO:     172.21.0.4:33890 - "GET /work_queues/name/demo HTTP/1.1" 200 OK
prefect-docker-compose-orion-1     | INFO:     172.21.0.4:33890 - "POST /work_queues/31d37abe-6aee-4674-beaf-4bc8d2cae7c4/get_runs HTTP/1.1" 200 OK
prefect-docker-compose-agent-1     | 11:54:55.107 | INFO    | prefect.agent - Submitting flow run '2f6ba8ff-364c-4ddf-974b-f49712fede9b'
prefect-docker-compose-orion-1     | INFO:     172.21.0.4:33890 - "POST /flow_runs/2f6ba8ff-364c-4ddf-974b-f49712fede9b/set_state HTTP/1.1" 201 Created
prefect-docker-compose-orion-1     | INFO:     172.21.0.4:33890 - "GET /deployments/c9398a97-4ecc-4b57-8acb-0a50d5a7b328 HTTP/1.1" 200 OK
prefect-docker-compose-orion-1     | INFO:     172.21.0.4:33890 - "GET /flows/91f4a98a-877d-4c40-93c7-41236a23d2e5 HTTP/1.1" 200 OK
prefect-docker-compose-orion-1     | INFO:     172.21.0.4:33890 - "GET /block_documents/7f22b67b-4947-46a1-b0d0-5ae260c71b62?include_secrets=true HTTP/1.1" 200 OK
prefect-docker-compose-agent-1     | 11:54:55.162 | INFO    | prefect.infrastructure.process - Opening process 'heavenly-macaw'...
prefect-docker-compose-agent-1     | 11:54:55.164 | INFO    | prefect.agent - Completed submission of flow run '2f6ba8ff-364c-4ddf-974b-f49712fede9b'
prefect-docker-compose-orion-1     | INFO:     172.21.0.4:33902 - "GET /flow_runs/2f6ba8ff-364c-4ddf-974b-f49712fede9b HTTP/1.1" 200 OK
prefect-docker-compose-orion-1     | INFO:     172.21.0.4:33902 - "GET /deployments/c9398a97-4ecc-4b57-8acb-0a50d5a7b328 HTTP/1.1" 200 OK
prefect-docker-compose-agent-1     | 11:54:56.745 | ERROR   | Flow run 'heavenly-macaw' - Flow could not be retrieved from deployment.
prefect-docker-compose-agent-1     | Traceback (most recent call last):
prefect-docker-compose-agent-1     |   File "/usr/local/lib/python3.11/site-packages/prefect/engine.py", line 255, in retrieve_flow_then_begin_flow_run
prefect-docker-compose-agent-1     |     flow = await load_flow_from_flow_run(flow_run, client=client)
prefect-docker-compose-agent-1     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
prefect-docker-compose-agent-1     |   File "/usr/local/lib/python3.11/contextlib.py", line 222, in __aexit__
prefect-docker-compose-agent-1     |     await self.gen.athrow(typ, value, traceback)
prefect-docker-compose-agent-1     |   File "/usr/local/lib/python3.11/site-packages/prefect/utilities/asyncutils.py", line 218, in asyncnullcontext
prefect-docker-compose-agent-1     |     yield
prefect-docker-compose-agent-1     |   File "/usr/local/lib/python3.11/site-packages/prefect/client/utilities.py", line 47, in with_injected_client
prefect-docker-compose-agent-1     |     return await fn(*args, **kwargs)
prefect-docker-compose-agent-1     |            ^^^^^^^^^^^^^^^^^^^^^^^^^
prefect-docker-compose-agent-1     |   File "/usr/local/lib/python3.11/site-packages/prefect/deployments.py", line 163, in load_flow_from_flow_run
prefect-docker-compose-agent-1     |     await storage_block.get_directory(from_path=deployment.path, local_path=".")
prefect-docker-compose-agent-1     |   File "/usr/local/lib/python3.11/site-packages/prefect/filesystems.py", line 144, in get_directory
prefect-docker-compose-agent-1     |     shutil.copytree(from_path, local_path, dirs_exist_ok=True)
prefect-docker-compose-agent-1     |   File "/usr/local/lib/python3.11/shutil.py", line 558, in copytree
prefect-docker-compose-agent-1     |     with os.scandir(src) as itr:
prefect-docker-compose-agent-1     |          ^^^^^^^^^^^^^^^
prefect-docker-compose-agent-1     | FileNotFoundError: [Errno 2] No such file or directory: '/root/flows'
2

looks like the path to the flow isn’t working, check this if you need an example of how Docker deployments can be used:

3

cc @ryan_peden

4

In most cases, you’d want to store your flow code on remote storage like MinIO or S3 so the agent can pull a fresh copy of it when it runs a flow run from a deployment.

But sharing your flow code between Docker Compose containers should also work here. However you will need to add a volume to the agent container like so:

  ## Prefect Agent
  agent:
    image: prefecthq/prefect:2.6.8-python3.11
    restart: always
    entrypoint: ["prefect", "agent", "start", "-q", "demo"]
    volumes:
      - "./flows:/root/flows"
    environment:
      - PREFECT_API_URL=http://orion:4200/api
    depends_on:
      - orion
    profiles: ["agent"]

This is because when you create a deployment without remote storage, Prefect looks for the code at its original path on the filesystem when the deployment was created.

The flow code resided at /root/flows/flow.py in the CLI container when you created the deployment, but since the volume containing the flow directory wasn’t attached to the agent container, the agent looked for /root/flows/flow.py but found that the /root/flows directory did not exist.

2 Likes
5

Hi Ryan,

I ran with your suggestion and the code worked. And, thanks for the explaination. I understand the issue.

After the fix, I got this error from the agent, but the flow ran anyways:

[+] Running 4/0
 ⠿ Container prefect-docker-compose-database-1  Created                                                                        0.0s
 ⠿ Container prefect-docker-compose-cli-1       Recreated                                                                      0.0s
 ⠿ Container prefect-docker-compose-orion-1     Created                                                                        0.0s
 ⠿ Container prefect-docker-compose-agent-1     Created                                                                        0.0s
Attaching to prefect-docker-compose-agent-1, prefect-docker-compose-cli-1, prefect-docker-compose-database-1, prefect-docker-compose-orion-1
canceled
prefect-docker-compose-database-1  | 
prefect-docker-compose-database-1  | PostgreSQL Database directory appears to contain a database; Skipping initialization
prefect-docker-compose-database-1  | 
prefect-docker-compose-database-1  | 2022-11-26 09:31:46.913 UTC [1] LOG:  starting PostgreSQL 14.1 on x86_64-pc-linux-musl, compiled by gcc (Alpine 10.3.1_git20211027) 10.3.1 20211027, 64-bit
prefect-docker-compose-database-1  | 2022-11-26 09:31:46.913 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
prefect-docker-compose-database-1  | 2022-11-26 09:31:46.913 UTC [1] LOG:  listening on IPv6 address "::", port 5432
prefect-docker-compose-database-1  | 2022-11-26 09:31:46.918 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
prefect-docker-compose-database-1  | 2022-11-26 09:31:46.925 UTC [21] LOG:  database system was shut down at 2022-11-26 09:31:31 UTC
prefect-docker-compose-database-1  | 2022-11-26 09:31:46.931 UTC [1] LOG:  database system is ready to accept connections
prefect-docker-compose-orion-1     | 
prefect-docker-compose-orion-1     |  ___ ___ ___ ___ ___ ___ _____    ___  ___ ___ ___  _  _
prefect-docker-compose-orion-1     | | _ \ _ \ __| __| __/ __|_   _|  / _ \| _ \_ _/ _ \| \| |
prefect-docker-compose-orion-1     | |  _/   / _|| _|| _| (__  | |   | (_) |   /| | (_) | .` |
prefect-docker-compose-orion-1     | |_| |_|_\___|_| |___\___| |_|    \___/|_|_\___\___/|_|\_|
prefect-docker-compose-orion-1     | 
prefect-docker-compose-orion-1     | Configure Prefect to communicate with the server with:
prefect-docker-compose-orion-1     | 
prefect-docker-compose-orion-1     |     prefect config set PREFECT_API_URL=http://0.0.0.0:4200/api
prefect-docker-compose-orion-1     | 
prefect-docker-compose-orion-1     | View the API reference documentation at http://0.0.0.0:4200/docs
prefect-docker-compose-orion-1     | 
prefect-docker-compose-orion-1     | Check out the dashboard at http://0.0.0.0:4200
prefect-docker-compose-orion-1     | 
prefect-docker-compose-orion-1     | 
prefect-docker-compose-orion-1     | 
prefect-docker-compose-agent-1     | Starting v2.6.8 agent connected to http://orion:4200/api...
prefect-docker-compose-agent-1     | 
prefect-docker-compose-agent-1     |   ___ ___ ___ ___ ___ ___ _____     _   ___ ___ _  _ _____
prefect-docker-compose-agent-1     |  | _ \ _ \ __| __| __/ __|_   _|   /_\ / __| __| \| |_   _|
prefect-docker-compose-agent-1     |  |  _/   / _|| _|| _| (__  | |    / _ \ (_ | _|| .` | | |
prefect-docker-compose-agent-1     |  |_| |_|_\___|_| |___\___| |_|   /_/ \_\___|___|_|\_| |_|
prefect-docker-compose-agent-1     | 
prefect-docker-compose-agent-1     | 
prefect-docker-compose-agent-1     | Agent started! Looking for work from queue(s): demo...
prefect-docker-compose-cli-1       | Traceback (most recent call last):
prefect-docker-compose-cli-1       |   File "/usr/local/lib/python3.11/site-packages/anyio/_core/_sockets.py", line 164, in try_connect
prefect-docker-compose-cli-1       |     stream = await asynclib.connect_tcp(remote_host, remote_port, local_address)
prefect-docker-compose-cli-1       |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
prefect-docker-compose-cli-1       |   File "/usr/local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 1691, in connect_tcp
prefect-docker-compose-cli-1       |     await get_running_loop().create_connection(
prefect-docker-compose-cli-1       |   File "/usr/local/lib/python3.11/asyncio/base_events.py", line 1073, in create_connection
prefect-docker-compose-cli-1       |     raise exceptions[0]
prefect-docker-compose-cli-1       |   File "/usr/local/lib/python3.11/asyncio/base_events.py", line 1058, in create_connection
prefect-docker-compose-cli-1       |     sock = await self._connect_sock(
prefect-docker-compose-cli-1       |            ^^^^^^^^^^^^^^^^^^^^^^^^^
prefect-docker-compose-cli-1       |   File "/usr/local/lib/python3.11/asyncio/base_events.py", line 964, in _connect_sock
prefect-docker-compose-cli-1       |     await self.sock_connect(sock, address)
prefect-docker-compose-cli-1       |   File "/usr/local/lib/python3.11/asyncio/selector_events.py", line 633, in sock_connect
prefect-docker-compose-cli-1       |     return await fut
prefect-docker-compose-cli-1       |            ^^^^^^^^^
prefect-docker-compose-cli-1       |   File "/usr/local/lib/python3.11/asyncio/selector_events.py", line 668, in _sock_connect_cb
prefect-docker-compose-cli-1       |     raise OSError(err, f'Connect call failed {address}')
prefect-docker-compose-cli-1       | ConnectionRefusedError: [Errno 111] Connect call failed ('172.21.0.4', 4200)
prefect-docker-compose-cli-1       | 
prefect-docker-compose-cli-1       | The above exception was the direct cause of the following exception:
prefect-docker-compose-cli-1       | 
prefect-docker-compose-cli-1       | Traceback (most recent call last):
prefect-docker-compose-cli-1       |   File "/usr/local/lib/python3.11/site-packages/httpcore/_exceptions.py", line 8, in map_exceptions
prefect-docker-compose-cli-1       |     yield
prefect-docker-compose-cli-1       |   File "/usr/local/lib/python3.11/site-packages/httpcore/backends/asyncio.py", line 109, in connect_tcp
prefect-docker-compose-cli-1       |     stream: anyio.abc.ByteStream = await anyio.connect_tcp(
prefect-docker-compose-cli-1       |                                    ^^^^^^^^^^^^^^^^^^^^^^^^
prefect-docker-compose-cli-1       |   File "/usr/local/lib/python3.11/site-packages/anyio/_core/_sockets.py", line 222, in connect_tcp
prefect-docker-compose-cli-1       |     raise OSError("All connection attempts failed") from cause
prefect-docker-compose-cli-1       | OSError: All connection attempts failed
prefect-docker-compose-cli-1       | 
prefect-docker-compose-cli-1       | During handling of the above exception, another exception occurred:
prefect-docker-compose-cli-1       | 
prefect-docker-compose-cli-1       | Traceback (most recent call last):
prefect-docker-compose-cli-1       |   File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
prefect-docker-compose-cli-1       |     yield
prefect-docker-compose-cli-1       |   File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
prefect-docker-compose-cli-1       |     resp = await self._pool.handle_async_request(req)
prefect-docker-compose-cli-1       |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
prefect-docker-compose-cli-1       |   File "/usr/local/lib/python3.11/site-packages/httpcore/_async/connection_pool.py", line 253, in handle_async_request
prefect-docker-compose-cli-1       |     raise exc
prefect-docker-compose-cli-1       |   File "/usr/local/lib/python3.11/site-packages/httpcore/_async/connection_pool.py", line 237, in handle_async_request
prefect-docker-compose-cli-1       |     response = await connection.handle_async_request(request)
prefect-docker-compose-cli-1       |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
prefect-docker-compose-cli-1       |   File "/usr/local/lib/python3.11/site-packages/httpcore/_async/connection.py", line 86, in handle_async_request
prefect-docker-compose-cli-1       |     raise exc
prefect-docker-compose-cli-1       |   File "/usr/local/lib/python3.11/site-packages/httpcore/_async/connection.py", line 63, in handle_async_request
prefect-docker-compose-cli-1       |     stream = await self._connect(request)
prefect-docker-compose-cli-1       |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
prefect-docker-compose-cli-1       |   File "/usr/local/lib/python3.11/site-packages/httpcore/_async/connection.py", line 111, in _connect
prefect-docker-compose-cli-1       |     stream = await self._network_backend.connect_tcp(**kwargs)
prefect-docker-compose-cli-1       |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
prefect-docker-compose-cli-1       |   File "/usr/local/lib/python3.11/site-packages/httpcore/backends/auto.py", line 29, in connect_tcp
prefect-docker-compose-cli-1       |     return await self._backend.connect_tcp(
prefect-docker-compose-cli-1       |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

prefect-docker-compose-cli-1       |   File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1642, in _send_handling_auth
prefect-docker-compose-cli-1       |     response = await self._send_handling_redirects(
prefect-docker-compose-cli-1       |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
prefect-docker-compose-cli-1       |   File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1679, in _send_handling_redirects
prefect-docker-compose-cli-1       |     response = await self._send_single_request(request)
prefect-docker-compose-cli-1       |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
prefect-docker-compose-cli-1       |   File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1716, in _send_single_request
prefect-docker-compose-cli-1       |     response = await transport.handle_async_request(request)
prefect-docker-compose-cli-1       |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
prefect-docker-compose-cli-1       |   File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 352, in handle_async_request
prefect-docker-compose-cli-1       |     with map_httpcore_exceptions():
prefect-docker-compose-cli-1       |   File "/usr/local/lib/python3.11/contextlib.py", line 155, in __exit__
prefect-docker-compose-cli-1       |     self.gen.throw(typ, value, traceback)
prefect-docker-compose-cli-1       |   File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
prefect-docker-compose-cli-1       |     raise mapped_exc(message) from exc
prefect-docker-compose-cli-1       | httpx.ConnectError: All connection attempts failed
prefect-docker-compose-cli-1 exited with code 1
prefect-docker-compose-orion-1     | INFO:     Started server process [7]
prefect-docker-compose-orion-1     | INFO:     Waiting for application startup.
prefect-docker-compose-orion-1     | INFO:     Application startup complete.
prefect-docker-compose-orion-1     | INFO:     Uvicorn running on http://0.0.0.0:4200 (Press CTRL+C to quit)
prefect-docker-compose-orion-1     | INFO:     172.21.0.5:41732 - "GET /work_queues/name/demo HTTP/1.1" 200 OK
prefect-docker-compose-orion-1     | INFO:     172.21.0.5:41732 - "POST /work_queues/31d37abe-6aee-4674-beaf-4bc8d2cae7c4/get_runs HTTP/1.1" 200 OK
prefect-docker-compose-agent-1     | 09:32:20.619 | INFO    | prefect.agent - Submitting flow run '2cf37a44-6e4f-44da-9769-4712ddf051bd'
 immediately...
prefect-docker-compose-orion-1     | INFO:     172.21.0.5:51972 - "GET /health HTTP/1.1" 200 OK
prefect-docker-compose-orion-1     | INFO:     172.21.0.5:51962 - "POST /task_runs/3ff4b40b-7793-42aa-9d4e-806483576d63/set_state HTTP/1.1" 201 Created
prefect-docker-compose-orion-1     | INFO:     172.21.0.5:51962 - "GET /task_runs/3ff4b40b-7793-42aa-9d4e-806483576d63 HTTP/1.1" 200 OK
prefect-docker-compose-orion-1     | INFO:     172.21.0.5:51972 - "POST /task_runs/8805c831-22f9-4680-b9e7-9ad646bc3b17/set_state HTTP/1.1" 201 Created
prefect-docker-compose-orion-1     | INFO:     172.21.0.5:51972 - "GET /task_runs/8805c831-22f9-4680-b9e7-9ad646bc3b17 HTTP/1.1" 200 OK
prefect-docker-compose-orion-1     | INFO:     172.21.0.5:51962 - "POST /task_runs/3ff4b40b-7793-42aa-9d4e-806483576d63/set_state HTTP/1.1" 201 Created
prefect-docker-compose-agent-1     | 09:32:22.969 | INFO    | Task run 'say_goodbye-0c32b902-3' - Finished in state Completed()
prefect-docker-compose-orion-1     | INFO:     172.21.0.5:51972 - "POST /task_runs/8805c831-22f9-4680-b9e7-9ad646bc3b17/set_state HTTP/1.1" 201 Created
prefect-docker-compose-agent-1     | 09:32:22.978 | INFO    | Task run 'say_goodbye-0c32b902-3' - Finished in state Completed()
prefect-docker-compose-orion-1     | INFO:     172.21.0.5:51962 - "POST /flow_runs/2cf37a44-6e4f-44da-9769-4712ddf051bd/set_state HTTP/1.1" 201 Created
prefect-docker-compose-agent-1     | 09:32:22.988 | INFO    | Flow run 'ivory-tamarin' - Finished in state Completed('All states completed.')
prefect-docker-compose-orion-1     | INFO:     172.21.0.5:51972 - "POST /flow_runs/387c9d84-780a-4eb0-a10b-b33e9c586f4c/set_state HTTP/1.1" 201 Created
prefect-docker-compose-agent-1     | 09:32:23.089 | INFO    | Flow run 'lush-taipan' - Finished in state Completed('All states completed.')
prefect-docker-compose-orion-1     | INFO:     172.21.0.5:51992 - "POST /logs/ HTTP/1.1" 201 Created
prefect-docker-compose-agent-1     | hello arthur
prefect-docker-compose-agent-1     | goodbye arthur
prefect-docker-compose-agent-1     | hello trillian
prefect-docker-compose-agent-1     | goodbye trillian
prefect-docker-compose-agent-1     | hello ford
prefect-docker-compose-agent-1     | goodbye ford
prefect-docker-compose-agent-1     | hello marvin
prefect-docker-compose-agent-1     | goodbye marvin
prefect-docker-compose-orion-1     | INFO:     172.21.0.5:52002 - "POST /logs/ HTTP/1.1" 201 Created
prefect-docker-compose-agent-1     | hello arthur
prefect-docker-compose-agent-1     | goodbye arthur
prefect-docker-compose-agent-1     | hello trillian
prefect-docker-compose-agent-1     | goodbye trillian
prefect-docker-compose-agent-1     | hello ford
prefect-docker-compose-agent-1     | goodbye ford
prefect-docker-compose-agent-1     | hello marvin
prefect-docker-compose-agent-1     | goodbye marvin
6

That happened because the script you are running in the CLI container is running before the Orion container has started, so it is trying to connect to the API but the API is not running yet.

I don’t think you need to run the CLI container here, though. You have already created the deployment so unless you want to update the deployment to change something about it, you don’t need to re-run the code in your if __name__ == "__main__": block.

Just running docker compose --profile orion --profile agent should be enough. The agent should start and run your scheduled deployment every 30 seconds.

1 Like