Scaling Prefect - what's the difference between ephemeral and hosted API

DataflowOps for Prefect 2.0
,
#1

When scaling OSS Prefect beyond the ephemeral API usage, you need to set the PREFECT_API_URL separately.

Why?

When PREFECT_API_URL is None, the API server is run locally in memory. If you then try to run a flow without having first set the PREFECT_API_URL, you will run into an issue that your custom self-hosted API backend will not be able to find that flow run ID because this run will only get created within the ephemeral API that uses in-memory SQLite database (and will be stored there instead of your persistent self-hosted API backend).

Ephemeral vs. hosted API

The ephemeral API is great for the following use cases:

  • when you are getting started with Prefect and learning how to use it
  • when you are doing a quick PoC
  • when you are running multiple ephemeral API server instances, all writing to the same Postgres backend

However, when you are self-hosting Prefect, e.g. on a Linux VM on-prem or in the public Cloud, you can’t just do: prefect orion start without explicitly setting the PREFECT_API_URL. Otherwise, you would use the ephemeral API instead of your hosted API.

Hosted API

Here is how you can configure the hosted API. First, start the Orion server with the -host flag:

prefect orion start --host 0.0.0.0

Note about the --host flag

:warning: Note that: adding --host 0.0.0.0 exposes the API server for nonlocal connection, so you need it whenever you’re hosting on another machine or in a container (can depend on Docker networking).

Then, set the PREFECT_API_URL to the hostname of your machine. E.g., if you are running the Orion server locally, this would be:

prefect config set PREFECT_API_URL="http://127.0.0.1:4200/api"

However, on an EC2 instance with a public IP of 3.227.102.13, you would need to:

  • open port 4200 for ingress traffic on the VM from 0.0.0.0/0 (all traffic) as well as all HTTP traffic
  • set the PREFECT_API_URL and PREFECT_ORION_UI_API_URL as follows:
prefect config set PREFECT_API_URL="http://3.227.102.13:4200/api"
prefect config set PREFECT_ORION_UI_API_URL="http://3.227.102.13:4200/api"

Setting these values is especially important if you want to interact with that remote API from your machine. This allows you to run your flows from your laptop, and those runs will be recorded by that remote API server backend hosted on a VM.

Adding a DB connection to Postgres

Imagine that you want this remote API to interact with your self-hosted Postgres database (recommended option, SQLite is not recommended for production usage with concurrent writes):

prefect config set PREFECT_ORION_DATABASE_CONNECTION_URL="postgresql+asyncpg://postgres:yourTopSecretPassword@localhost:5432/orion"

Self-hosting on a VM with docker-compose and Postgres

More examples

3 Likes
How to install Prefect 2.0 on a Linux VM
#2

This is really helpful to understand some conceptual things about how prefect Orion works.

I was able to run two separate Prefect environments on a single machine with these settings. Without these changes even with prefect installed in separate virtual environments, the service API gets shared between them, and ultimately the DB data and everything mixed.

  • single PREFECT_HOME location
  • single profiles.toml file
  • two distinct db: orion.db, orion_personal.db
  • two Prefect UI interfaces at different local port locations.
  • havent tested the multiple agents
prefect profile create 'personal'
prefect config set \
    PREFECT_ORION_DATABASE_CONNECTION_URL='sqlite+aiosqlite:///${PREFECT_HOME}/orion_personal.db'
prefect config set PREFECT_ORION_API_PORT='4202'
prefect config set PREFECT_API_URL='http://127.0.0.1:4202/api'
1 Like