How to run a Prefect 2 agent as a systemd service on Linux

This post demonstrates how to set up a systemd service to run a Prefect 2 agent.

A systemd service is an ideal way to run a Prefect agent on a Linux VM or physical server if you want to:

  • Automatically start a Prefect agent when Linux starts.
  • Automatically restart a Prefect agent if it crashes.

Prerequisites

Here’s what you’ll need before proceeding:

  • A working Linux machine with systemd and Python 3.7 or later.
  • A superuser account, i.e. you can run sudo.
  • A Prefect Cloud account, or a Prefect Orion server running somewhere on your local network.
  • A Prefect deployment with a work queue your agent can connect to.

Adding a user

First, create a user account for the Prefect agent. While you can run the agent as root, it’s best to avoid doing so unless you are sure you need to.

In a terminal, run:

sudo useradd -m prefect
sudo passwd prefect

When prompted, enter a password for the prefect account.

Next, log in to the prefect account by running:

sudo su prefect

Installing Prefect

Now, it’s time to install Prefect. Run:

pip install prefect

This assumes you are installing Prefect globally, not in a virtual environment. Running a systemd service from a venv will work, but when you set up the service below, you’ll need to change the ExecPath to target the prefect application in the bin subdirectory of your virtual environment.

Next, you must set up Prefect so it will know which server to connect to.

If you are connecting to a local Prefect Orion server, run the following and substiture the IP address of your server:

prefect config set PREFECT_API_URL=http://your-orion-server-IP:4200

If you are connecting to Prefect Cloud, follow the instructions to obtain an API key and then run:

prefect cloud login -k YOUR_API_KEY

When prompted, choose the Prefect workspace you’d like to log in to.

Finally, run the exit command to sign out of the prefect Linux account. This switches you back to your sudo-enabled account so you will be able to run the commands in the next section.

Setting up a systemd service for the Prefect agent

Now you’re ready to set up your Prefect agent systemd service. Run the following commands:

cd /etc/systemd/system
sudo vim prefect-agent.service

Feel free to use another editor like Nano or Emacs instead of Vim if you prefer. Add the following to the file:

[Unit]
Description=Prefect Agent

[Service]
User=prefect
WorkingDirectory=/home/prefect
ExecStart=/usr/local/bin/prefect agent start -q YOUR_WORK_QUEUE_NAME
Restart=always

[Install]
WantedBy=multi-user.target

Make sure you substitute your own work queue name. When you’re done, save the file and exit. If you’re stuck in Vim, don’t worry! Just hit the escape key, type :wq!, then press the return key.

Next, run systemctl daemon-reload to make systemd aware of your new service.

Then, run systemctl enable prefect-agent to enable the service. This will ensure it runs when your system boots.

Finally, run systemctl start prefect-agent to start the service. And that’s it! You now have a systemd service that starts a Prefect agent when your system boots, and will restart the agent if it ever crashes.

4 Likes