Getting 403 when running tutorial code for docker based prefect 2

I configured a docker instance based on this

version: '3.5'
services:
  # --------------------------------------#
  #               Postgres                #
  # --------------------------------------#
  postgres:
    image: postgres:14
    command:
      - postgres
      - -c
      - max_connections=150
    environment:
      POSTGRES_USER: prefect
      POSTGRES_PASSWORD: test-password
      POSTGRES_DB: prefect_server
    healthcheck:
      interval: 10s
      retries: 60
      start_period: 30s
      test: pg_isready -q -d $${POSTGRES_DB} -U $${POSTGRES_USER} | grep "accepting connections" || exit 1
      timeout: 30s
    ports:
    - 5432:5432
    restart: on-failure
    volumes:
      - ./volumes/postgres:/var/lib/postgresql/data
  # --------------------------------------#
  #             Prefect Server            #
  # --------------------------------------#
  prefect-server:
    image: prefecthq/prefect:2.4.1-python3.10
    command:
      - prefect
      - orion
      - start
    ports:
      - 4200:4200
    depends_on:
      postgres:
        condition: service_started
    volumes:
      - ./volumes/prefect:/root/.prefect
      - ./flows:/flows
    environment:
      PREFECT_ORION_API_HOST: 0.0.0.0
      PREFECT_ORION_DATABASE_CONNECTION_URL: postgresql+asyncpg://prefect:test-password@postgres:5432/prefect_server
      PREFECT_ORION_ANALYTICS_ENABLED: "false"
      PREFECT_LOGGING_SERVER_LEVEL: WARNING
      PREFECT_API_URL: http://local.a10ai.com:4200/api
  # --------------------------------------#
  #             Docker Agent              #
  # --------------------------------------#
  prefect-agent:
    image: prefecthq/prefect:2.4.1-python3.10
    command:
      - prefect
      - agent
      - start
      - -q
      - 'default'
    depends_on:
      prefect-server:
        condition: service_started
    environment:
      PREFECT_API_URL: http://local.a10ai.com:4200/api
      PREFECT_LOGGING_LEVEL: DEBUG
      DOCKER_HOST: unix://var/run/docker.sock
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

  # --------------------------------------#
  #         Minio for storage             #
  # --------------------------------------#
  minio:
    image: minio/minio
    environment:
      - MINIO_ROOT_USER=blablabla
      - MINIO_ROOT_PASSWORD=blablabla
    volumes:
      - ./volumes/minio:/data
    entrypoint: /bin/bash 
    command: "-c 'mkdir -p /data/prefect-flows && sleep 10 && minio server /data --console-address :9001'"
    ports:
      - 9000:9000
      - 9001:9001

The UI works as expected but when I get the following error when trying to create a deployment

prefect.exceptions.PrefectHTTPStatusError: Client error '403 Forbidden' for url 'http://local.a10ai.com:4200/api/block_types/c7b965d8-768d-489e-8cf9-ba5ee39c7899'

Here is the code. I’m unsure what else I need to setup for this.


import os

os.environ["APP_ENV"] = "DEV"
os.environ["PREFECT_API_URL"] = "http://local.a10ai.com:4200/api/"

from prefect.settings import PREFECT_API_URL
from datetime import datetime, timedelta
from prefect.orion.schemas.schedules import IntervalSchedule
from prefect.deployments import Deployment
from prefect import task, flow, get_run_logger

print(PREFECT_API_URL.value())


@flow()
def log_flow():
    logger = get_run_logger()
    logger.info("running sample_screenshots")

deployment = Deployment.build_from_flow(
    flow=log_flow,
    name="simple-log",
    parameters={},
    infra_overrides={"env": {"PREFECT_LOGGING_LEVEL": "DEBUG"}},
    work_queue_name="test",
)

if __name__ == "__main__":
    deployment.apply()

You mean the web UI and commands in the shell? Can you create the deployment with something like prefect deployment build flows/script.py:log_flow -n simple-log -q test? Deployments - Prefect 2

1 Like

Did you try:

prefect config set PREFECT_API_URL="http://0.0.0.0:4200/api"

I get the same error when trying to apply it from the console

The python file sets the PREFECT_API_URL in the code before running the deployment for now.

if this is not working, I’d recommend taking it step by step and try to determine the root cause and share your findings

Not really possible when the literal most basic example fails. There is no smaller step. If I can’t run a basic logger output because I get a 403 which typically means some sort of permission failure I can’t continue.

Is there some sort of permission setup that has to be done with a default image to make it work? Some key I have to get from prefect? The documentation pretends like it should ‘just work’ when it clearly doesn’t. This is clearly a permission issue of some form with creating a block but no documentation I can find on giving my code those permissions for a docker image version of prefect.

Not sure why but i upgraded the docker image and it works now.

1 Like

weird, but could be some version mismatch? still, well-done and thanks for sharing

Hi.

I am facing the same error right now. The prefect server is running at abc-loadbalancer.com and I am trying to deploy flows from xyz.com. The network request is passing through I assume, since I am already getting 403, and its not like the my request is being rejected by load balancer itself.

The versions match (which is 2.13.8) in both the abc and xyz.

Can you provide me a hint?


I figured what was causing it, but it still is a very weird situation which I was unable to replicate. Following are the curl commands that shows the errors. These requests are sent when we do prefect deploy and prefect sends a POST request to prefect server requesting for available workers.

➜  ~ curl -X 'POST'  'https://prefect2.abc.net/api/work_pools/filter'   -H 'accept: application/json' -H 'Content-Type: application/json'  -d '{"offset": 0, "work_pools": null, "limit": null}'
[{"id":"5cc0cc08-0c20-4412-8819-79c2c7d784e7","created":"2023-10-24T14:18:23.490000+00:00","updated":"2023-10-24T14:18:23.490000+00:00","name":"default-agent-pool","description":null,"type":"prefect-agent","base_job_template":{},"is_paused":false,"concurrency_limit":null,"default_queue_id":"da4cfc4c-6ad5-46b4-9a8d-de619dec2f52","status":null}, ...]%

➜  ~ curl -X 'POST'  'https://prefect2.abc.net/api/work_pools/filter'   -H 'accept: application/json' -H 'Content-Type: application/json'  -d '{"limit": null, "offset": 0, "work_pools": null}'
<html>
<head><title>403 Forbidden</title></head>
<body>
<center><h1>403 Forbidden</h1></center>
</body>
</html>

The only difference in these commands are the placement of the limit key in the data body (-d argument in the curl request).

The request is initiated from this code