Logs not displayed in UI when worker is running inside container

Help
, , , ,
1

Hello,

I’m currently setting up prefect infrastructure and have the issue that logs from prefect.flow_runs and prefect.task_runs are not displayed in the UI - neither logs my application creates nor logs prefect creates such as Created task run 'prepare_data-0' for task 'prepare_data'.

Logs from prefect.flow_runs.worker, such as Worker 'ProcessWorker 123' submitting flow run '456 are, however, displayed.

circumstances under which logs are displayed correctly:

  • the worker runs as a local process directly on my machine (instead of inside a container)
  • the flows are executed “directly” (i.e. without using deployments, e.g. with python flow-code.py) - in this case, it does not matter whether I execute the code from inside a docker container or on my machine directly - both work fine

circumstance under which logs are not displayed:

  • the worker runs inside a docker container
  • and I start a deployment via prefect deployment run 'Flow-Name/Deployment-Name'

infrastructure:

  • The docker compose file I use for server and database is similar to this one.
    • postgresql image: postgres:15.3-bookworm
    • server image: prefecthq/prefect:2.13.2-python3.10
    • the environment variables are adjusted to comply with the latest version of prefect documentation
  • I use a separate docker compose file for the worker:
    • image: I build on top of prefecthq/prefect:2.13.2-python3.10 and install some dependencies into the image such as openssh-client and poetry.
    • the command specified inside the compose file used to start the worker is sh -c "prefect worker start --pool process-pool --install-policy if-not-present"

deployment steps:

  1. git clone: the code is cloned from our private gitlab repository
  2. run_shell_script: ssh-agent is started, private key is added, known hosts are updated, and the dependencies are installed via poetry

what I’ve tried so far:

  • setting log_prints=True (and the respective environment variable) in combination with print statements
  • setting all kinds of environment variables related to logging to “DEBUG”
  • starting the worker inside the same compose file as the one used for server and postgresql
  • logging via the logger retrieved via get_run_logger()
  • setting PREFECT_LOGGING_EXTRA_LOGGERS
  • (and some more, but I don’t remember everything I already tried)

The deployment itself works fine - the only issue is that logs are not displayed. Hope someone has an idea what this might be caused by.

Thanks in advance

2

Hi @rsand
I’ve got the same behavior. Did you figure it out ?