Running Prefect 2 server on my own machine I had one deployment with zero jobs running. After leaving it up for a day, I came back to find out that the server process had completely pegged my server CPUs (this being a pretty beefy server, so it was consuming a lot of resources), with the following messages on repeat in the Prefect server output:
12:56:35.596 | WARNING | prefect.server.services.scheduler - Scheduler took 760.745296 seconds to run, which is longer than its loop interval of 60.0 seconds.
12:58:40.611 | WARNING | prefect.server.services.flowrunnotifications - FlowRunNotifications took 1064.676601 seconds to run, which is longer than its loop interval of 4 seconds.
12:59:58.621 | WARNING | prefect.server.services.failexpiredpauses - FailExpiredPauses took 1087.888235 seconds to run, which is longer than its loop interval of 5.0 seconds.
13:02:23.768 | WARNING | prefect.server.services.recentdeploymentsscheduler - RecentDeploymentsScheduler took 774.146208 seconds to run, which is longer than its loop interval of 5 seconds.
13:10:39.303 | WARNING | prefect.server.services.scheduler - Scheduler took 824.865576 seconds to run, which is longer than its loop interval of 60.0 seconds.
13:12:39.154 | WARNING | prefect.server.services.marklateruns - MarkLateRuns took 1066.243996 seconds to run, which is longer than its loop interval of 5.0 seconds.
13:17:20.455 | WARNING | prefect.server.services.recentdeploymentsscheduler - RecentDeploymentsScheduler took 878.664012 seconds to run, which is longer than its loop interval of 5 seconds.
13:18:17.087 | WARNING | prefect.server.services.failexpiredpauses - FailExpiredPauses took 1072.838626 seconds to run, which is longer than its loop interval of 5.0 seconds.
13:19:36.163 | WARNING | prefect.server.services.flowrunnotifications - FlowRunNotifications took 1230.665801 seconds to run, which is longer than its loop interval of 4 seconds.
13:25:00.938 | WARNING | prefect.server.services.cancellationcleanup - CancellationCleanup took 1934.423313 seconds to run, which is longer than its loop interval of 20.0 seconds.
13:26:15.665 | WARNING | prefect.server.services.scheduler - Scheduler took 866.2781 seconds to run, which is longer than its loop interval of 60.0 seconds.
Note that no jobs were running this whole time, just one deployment with no active flows.
ChatGPT suggests it could be a database bottleneck, I’m currently just using the default setup and have not migrated to Postgres. Thoughts?