`prefect worker start` stuck in a hanging state when launched from EC2

Help
, , , ,
1

Hello,

EDIT: This was resolved, turns out to be related to a networking issue in my setup. Once resolved, the container attaches:

Attaching to prefect_worker
prefect_worker  | Discovered type 'process' for work pool 'private-stuff'.
prefect_worker  | Worker 'ProcessWorker 3f4d7103-96e0-40a3-bb5f-d2sj49al1if1' started!

I lost a bit of time trying to troubleshoot docker. Just know, there doesn’t seem to be a timeout implemented for all cases here.

Original Post:

I am attempting to host Prefect Server on an EC2 instance via Docker, and a Prefect Worker on another EC2 instance via Docker. Both on t2.micro instances.

The Prefect Server is hosted at 172.0.0.1:4200 and the Worker is at 172.0.0.2.
The server is working fine and is even reachable when I test Prefect Workers locally. This goes through https://prefect.domain-example.com.

Here is how I am testing locally (via docker-compose):

version: '3.8'

services:
  prefect_worker:
    image: prefecthq/prefect:2.14.12-python3.11
    container_name: prefect_worker
    entrypoint: ['/opt/prefect/entrypoint.sh', 'prefect', 'worker', 'start', '--pool', 'private-stuff']
    environment:
      - PREFECT_API_URL=${PREFECT_API_URL}
      - PREFECT_API_KEY=${PREFECT_API_KEY}

with the variables:

PREFECT_API_URL=https://prefect.domain-example.com/api
PREFECT_API_KEY=debugging-example-key

This works fine locally.

However, I build an EC2 machine like so:

resource "aws_instance" "worker_server" {
  ami           = var.instance_config["ami"]
  instance_type = "t2.micro"
  key_name      = var.instance_config["key_name"]

  root_block_device {
    volume_size           = 8
    volume_type           ="gp2"
    delete_on_termination = false
  }

  user_data = file("ec2/user_data.sh")
}

./ec2/user_data.sh

#!/bin/bash

# Install Docker
sudo yum update
sudo yum install docker -y

# Add the user 'ec2-user' to the 'docker' group
sudo usermod -a -G docker ec2-user
newgrp docker

# Install docker-compose
wget https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m) 
sudo mv docker-compose-$(uname -s)-$(uname -m) /usr/local/bin/docker-compose
sudo chmod -v +x /usr/local/bin/docker-compose

# Enable and start Docker
sudo systemctl enable docker.service
sudo systemctl start docker.service

Then in the EC2 instance I set the variables as such:

PREFECT_API_URL=http://172.0.0.1:4200/api
PREFECT_API_KEY=debugging-example-key

and I attempt to launch it with docker-compose up. At this point, it hangs forever:

[ec2-user@ip-172-0-0-2 application]$ docker-compose up
[+] Running 1/0
 ✔ Container prefect_worker  Created                                                                               0.0s
Attaching to prefect_worker

I also attempted getting into the container and manually launching the worker. This also hangs forever:

[ec2-user@ip-172-0-0-2 application]$ docker exec -it prefect_worker /bin/bash
root@31248c7c8660:/opt/prefect# prefect worker start --pool private-stuff

Any ideas whats going on? It’s strange that it works locally but not remotely. This makes me wonder if it’s a networking issue, but then I would expect a timeout.