How to run Prefect 2 on CentOS 7 or RHEL 7

CentOS 7 and Red Hat Enterprise Linux 7 are old Linux distributions that come with very old versions of Python and SQLite. Consequently, Prefect 2 will not run on a fresh install of either distro.

However, many enterprises still use CentOS 7 and RHEL 7 — so you may need to use one of them to run Prefect.

Fortunately, there are two easy ways to run Prefect Orion or a Prefect agent on CentOS 7:

  • Anaconda
  • Docker

This post will walk through the steps needed for each.

Prerequisites

To complete any of the setup methods described below, you will need:

  • A CentOS 7 server or VM
  • A user account with superuser privileges, i.e. you can run sudo.

Anaconda

Anaconda is the easiest way to run Prefect on CentOS 7. Start by finding the URL of the most recent Anaconda installer for Linux on the Anaconda distribution page. At the time this post was written, the newest installer was located at https://repo.anaconda.com/archive/Anaconda3-2022.05-Linux-x86_64.sh.

Note that you may need a commercial license to use Anaconda depending on the size of your organization. See the Anaconda Commercial FAQ for details.

In a CentOS bash terminal, you can download Anaconda by running

curl -O https://repo.anaconda.com/archive/Anaconda3-2022.05-Linux-x86_64.sh

When the download finishes, make it executable by running:

sudo chmod 755 ./Anaconda3-2022.05-Linux-x86_64.sh

Follow the instructions in the installer. It will ask you to read and accept the license agreement. It’s not too long, so consider reading the entire agreement to ensure it is acceptable.

After you accept the license, the installer will install Anaconda and a complete set of tools including Python 3.9 and an up-to-date version of SQLite.

To get access to Anaconda without rebooting, run:

source ~/.bashrc

Your bash prompt will now see (base) before your normal Bash prompt. This mean you are using Anaconda’s base environment. Create a new environment for Prefect by running:

conda create --name prefect python=3.10

You can specify a different version of Python if you application needs it as long as it meets the requiements of the Prefect version you are using.

Activate the new environment by running:

conda activate prefect

Then, install Prefect by running:

pip install prefect

Pip will install Prefect and its dependencies, and you will then be ready to run Prefect on CentOS 7! If you will be running a Prefect agent, make sure you install any pip packages needed to run your flows.

Docker

If Anaconda does not meet your needs, you can run Prefect on CentOS 7 using Docker.

Before you can proceed, you will need to install Docker Engine by following the instructions in Docker’s documentation.

Once you have Docker up and running, you can use it to run either a Prefect Orion server or a Prefect agent.

By default, Prefect uses a local SQLite database stored in ~/.prefect, which you would normally lose every time you restart your Prefect container.

Fortunately, you can use a Docker volume to ensure the database persists across container runs. The easiest way to do this is with Docker Compose.

To run Prefect Orion, put the following in a file named docker-compose.yml:

version: "3.9"
services:
  orion:
    image: prefecthq/prefect:2.2.0-python3.10
    volumes:
      - prefect:/root/.prefect
    entrypoint: ["prefect", "orion", "start"]
    environment:
      - PREFECT_ORION_API_HOST=0.0.0.0
    ports:
      - 4200:4200
volumes:
  prefect:

Then, run docker-compose up in the same directory as the docker-compose.yml file. The Orion API will start, and once it is up and running it will be accessible at port 4200 on the CentOS 7 host machine.

If needed, you can change the image property to use a different Prefect image tag from Docker Hub.

Similarly, you can run a Prefect agent with the following docker-compose.yml:

version: "3.9"
services:
  orion:
    image: prefecthq/prefect:2.2.0-python3.10
    volumes:
      - prefect:/root/.prefect
    entrypoint: ["prefect", "agent", "start", "-q", "YOUR_WORK_QUEUE"]
    environment:
      - PREFECT_API_URL=http://your-api-url
      - PREFECT_API_KEY=YOUR_PREFECT_CLOUD_API_KEY
volumes:
  prefect:

Update YOUR_WORK_QUEUE to match the name of the work queue you want the agent to connect to, and update PREFECT_API_URL to match the URL of your Prefect Orion server or Prefect Cloud API.

If you want to run multiple Prefect agents with Docker on the same host, give each one of them a different volume name.

If you are running your own Prefect Orion instance, remove the PREFECT_API_KEY property from environment. Otherwise, change it to match your Prefect Cloud API key.

Note that when you run a Prefect agent inside a Docker container, the agent won’t be able to run deployments with DockerContainer infrastructure. If you want a Prefect agent to run DockerContainer deployments, the best approach on CentOS 7 is installing Docker so the agent can use it and then running the agent using Anaconda as described above.

2 Likes