This post demonstrates how to set up a systemd service to run a Prefect 2 worker. Systemd is a tool you can use do daemonize your Prefect 2 Worker.
A systemd service is an ideal way to run a Prefect worker on a Linux VM or physical server if you want to:
- Automatically start a Prefect worker when Linux starts.
- Automatically restart a Prefect worker if it crashes.
Prerequisites
Here’s what you’ll need before proceeding:
- A working Linux machine with systemd and Python 3.7 or later.
- A superuser account, i.e. you can run
sudo. - A Prefect Cloud account, or a local instance of Prefect Server running somewhere on your network.
- A Prefect deployment with a work pool your worker can connect to.
Adding a user
First, create a user account for the Prefect worker. While you can run the worker as root, it’s best to avoid doing so unless you are sure you need to.
In a terminal, run:
sudo useradd -m prefect
sudo passwd prefect
When prompted, enter a password for the prefect account.
Next, log in to the prefect account by running:
sudo su prefect
Installing Prefect
Now, it’s time to install Prefect. Run:
pip install prefect
This assumes you are installing Prefect globally, not in a virtual environment. Running a systemd service from a venv will work, but when you set up the service below, you’ll need to change the ExecPath to target the prefect application in the bin subdirectory of your virtual environment.
Next, you must set up Prefect so it will know which server to connect to.
If you are connecting to a local Prefect server, run the following and substiture the IP address of your server:
prefect config set PREFECT_API_URL=http://your-prefect-server-IP:4200
If you are connecting to Prefect Cloud, follow the instructions to obtain an API key and then run:
prefect cloud login -k YOUR_API_KEY
When prompted, choose the Prefect workspace you’d like to log in to.
Finally, run the exit command to sign out of the prefect Linux account. This switches you back to your sudo-enabled account so you will be able to run the commands in the next section.
Setting up a systemd service for the Prefect worker
Now you’re ready to set up your Prefect worker systemd service. Run the following commands:
cd /etc/systemd/system
sudo vim prefect-worker.service
Feel free to use another editor like Nano or Emacs instead of Vim if you prefer. Add the following to the file:
[Unit]
Description=Prefect Worker
[Service]
User=prefect
WorkingDirectory=/home/prefect
ExecStart=/usr/local/bin/prefect worker start --pool YOUR_WORK_POOL_NAME
Restart=always
[Install]
WantedBy=multi-user.target
Make sure you substitute your own work pool name. When you’re done, save the file and exit. If you’re stuck in Vim, don’t worry! Just hit the escape key, type :wq!, then press the return key.
Next, run systemctl daemon-reload to make systemd aware of your new service.
Then, run systemctl enable prefect-worker to enable the service. This will ensure it runs when your system boots.
Finally, run systemctl start prefect-worker to start the service. And that’s it! You now have a systemd service that starts a Prefect worker when your system boots, and will restart the worker if it ever crashes.
