This post demonstrates how to set up a systemd service to run a Prefect 2 worker. Systemd is a tool you can use do daemonize your Prefect 2 Worker.
A systemd service is an ideal way to run a Prefect worker on a Linux VM or physical server if you want to:
- Automatically start a Prefect worker when Linux starts.
- Automatically restart a Prefect worker if it crashes.
Here’s what you’ll need before proceeding:
- A working Linux machine with systemd and Python 3.7 or later.
- A superuser account, i.e. you can run
- A Prefect Cloud account, or a local instance of Prefect Server running somewhere on your network.
- A Prefect deployment with a work pool your worker can connect to.
First, create a user account for the Prefect worker. While you can run the worker as root, it’s best to avoid doing so unless you are sure you need to.
In a terminal, run:
sudo useradd -m prefect
sudo passwd prefect
When prompted, enter a password for the
Next, log in to the
prefect account by running:
sudo su prefect
Now, it’s time to install Prefect. Run:
pip install prefect
This assumes you are installing Prefect globally, not in a virtual environment. Running a systemd service from a venv will work, but when you set up the service below, you’ll need to change the ExecPath to target the
prefect application in the
bin subdirectory of your virtual environment.
Next, you must set up Prefect so it will know which server to connect to.
If you are connecting to a local Prefect server, run the following and substiture the IP address of your server:
prefect config set PREFECT_API_URL=http://your-prefect-server-IP:4200
If you are connecting to Prefect Cloud, follow the instructions to obtain an API key and then run:
prefect cloud login -k YOUR_API_KEY
When prompted, choose the Prefect workspace you’d like to log in to.
Finally, run the
exit command to sign out of the
prefect Linux account. This switches you back to your sudo-enabled account so you will be able to run the commands in the next section.
Now you’re ready to set up your Prefect worker systemd service. Run the following commands:
sudo vim prefect-worker.service
Feel free to use another editor like Nano or Emacs instead of Vim if you prefer. Add the following to the file:
ExecStart=/usr/local/bin/prefect worker start --pool YOUR_WORK_POOL_NAME
Make sure you substitute your own work pool name. When you’re done, save the file and exit. If you’re stuck in Vim, don’t worry! Just hit the escape key, type
:wq!, then press the return key.
systemctl daemon-reload to make systemd aware of your new service.
systemctl enable prefect-worker to enable the service. This will ensure it runs when your system boots.
systemctl start prefect-worker to start the service. And that’s it! You now have a systemd service that starts a Prefect worker when your system boots, and will restart the worker if it ever crashes.