I’ve built a flow which contains a task in which a Playwright session is started and a Chromium instance is used to visit a website. When I deploy the flow in Prefect Orion (deployed on a local computer running Ubuntu) the Chromium instance opens very slow (around 30 seconds to open) and all the actions are also executed much slower than normal (each click takes about 1 second).
I’ve tested the task without using it as part of a Prefect flow and confirmed it works at a normal speed. When using Playwright with browsers (such as firefox) I also have the same problem.
I wonder if this is behaviour normal? Are the agents running the work queue’s perhaps constrained in memory and CPU? Appreciate any tips or debugging help for this problem. Thanks!
I wonder what are you comparing against each other here? Could you share a more detailed description of what performance benchmark you are using it, what’s the end goal of this, and how did you go about implementing this?
My use case is to create a flow which will periodically crawl data from a website (an ERP system). The code itself works well when it’s not part of a Prefect flow. Currently, I’m benchmarking it by measuring the start-up time of the Playwright bundled Chromium browser (around 2 seconds without Prefect, around 10 seconds with Prefect). Another benchmark is the time that it takes for the programmed scrolling behaviour to complete. Without Prefect this will be around 0.5 seconds, while with Prefect each action will take a few seconds (around 3-5 seconds).
I’ve been running some more tests, and the problem can be isolated to my Ubuntu set-up. When running the flow in an agent on Mac or Windows it runs without any time delay (the Orion server itself is still running on the Ubuntu system). Without Prefect it runs without delay also on the Ubuntu system.
Gotcha. There might always be some performance improvements, but to explain why it takes longer than a locally running function: your Prefect flow is actually constantly talking to a backend, which is either Postgres or SQLite and this adds some extra latency.