What’s a best practice look like for setting up hundreds of concurrent REST API calls using prefect tasks?
We want to use some kind of concurrency limit to avoid overwhelming the API. However, the concurrency limits in prefect make tasks wait 30 seconds before checking for an available slot. This results in a HUGE slowdown, assuming each API call takes significantly less than 30 seconds a pop (less than 1 second each is normal).
An answer in slack suggested the use of anyio.CapacityLimiter, but we’d like to avoid migrating our pipelines into asyncio code if at all possible because of the technical hurdle.