### First check
- [X] I added a descriptive title to this issue.
- [X] I used t…he GitHub search to find a similar issue and didn't find it.
- [X] I searched the Prefect documentation for this issue.
- [X] I checked that this issue is related to Prefect and not one of its dependencies.
### Bug summary
A blocking async task can cause a Queue limit timeout which eventually crashes the flow run.
Is it possibly to increase this queue limit?
### Reproduction
```python
import asyncio
import random
import time
from prefect import flow, task, get_run_logger
async def async_func():
if random.random() > 0.3:
time.sleep(1)
await asyncio.sleep(random.randint(1, 5))
return 1
@task
async def my_task():
result = await async_func()
get_run_logger().info(f"test")
get_run_logger().info(f"hello")
return result
@flow()
async def my_flow():
tasks = []
for _ in range(500):
t = my_task()
tasks.append(t)
await asyncio.gather(*tasks, return_exceptions=True)
if __name__ == "__main__":
asyncio.run(my_flow())
```
### Error
```
10:40:31.368 | ERROR | prefect.orion - Encountered exception in request:
Traceback (most recent call last):
File "/home/test/lib/python3.9/site-packages/starlette/middleware/errors.py", line 162, in __call__
await self.app(scope, receive, _send)
File "/home/test/lib/python3.9/site-packages/starlette/exceptions.py", line 93, in __call__
raise exc
File "/home/test/lib/python3.9/site-packages/starlette/exceptions.py", line 82, in __call__
await self.app(scope, receive, sender)
File "/home/test/lib/python3.9/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in __call__
raise e
File "/home/test/lib/python3.9/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
await self.app(scope, receive, send)
File "/home/test/lib/python3.9/site-packages/starlette/routing.py", line 670, in __call__
await route.handle(scope, receive, send)
File "/home/test/lib/python3.9/site-packages/starlette/routing.py", line 266, in handle
await self.app(scope, receive, send)
File "/home/test/lib/python3.9/site-packages/starlette/routing.py", line 65, in app
response = await func(request)
File "/home/test/lib/python3.9/site-packages/prefect/orion/utilities/server.py", line 101, in handle_response_scoped_depends
response = await default_handler(request)
File "/home/test/lib/python3.9/site-packages/fastapi/routing.py", line 227, in app
raw_response = await run_endpoint_function(
File "/home/test/lib/python3.9/site-packages/fastapi/routing.py", line 160, in run_endpoint_function
return await dependant.call(**values)
File "/home/test/lib/python3.9/site-packages/prefect/orion/api/task_runs.py", line 186, in set_task_run_state
orchestration_result = await models.task_runs.set_task_run_state(
File "/home/test/lib/python3.9/site-packages/prefect/orion/models/task_runs.py", line 280, in set_task_run_state
run = await models.task_runs.read_task_run(session=session, task_run_id=task_run_id)
File "/home/test/lib/python3.9/site-packages/prefect/orion/database/dependencies.py", line 112, in async_wrapper
return await fn(*args, **kwargs)
File "/home/test/lib/python3.9/site-packages/prefect/orion/models/task_runs.py", line 98, in read_task_run
model = await session.get(db.TaskRun, task_run_id)
File "/home/test/lib/python3.9/site-packages/sqlalchemy/ext/asyncio/session.py", line 296, in get
return await greenlet_spawn(
File "/home/test/lib/python3.9/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 126, in greenlet_spawn
result = context.throw(*sys.exc_info())
File "/home/test/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 2805, in get
return self._get_impl(
File "/home/test/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 2912, in _get_impl
return db_load_fn(
File "/home/test/lib/python3.9/site-packages/sqlalchemy/orm/loading.py", line 530, in load_on_pk_identity
session.execute(
File "/home/test/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 1711, in execute
conn = self._connection_for_bind(bind)
File "/home/test/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 1552, in _connection_for_bind
return self._transaction._connection_for_bind(
File "/home/test/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 747, in _connection_for_bind
conn = bind.connect()
File "/home/test/lib/python3.9/site-packages/sqlalchemy/future/engine.py", line 406, in connect
return super(Engine, self).connect()
File "/home/test/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3234, in connect
return self._connection_cls(self, close_with_result=close_with_result)
File "/home/test/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 96, in __init__
else engine.raw_connection()
File "/home/test/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3313, in raw_connection
return self._wrap_pool_connect(self.pool.connect, _connection)
File "/home/test/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3280, in _wrap_pool_connect
return fn()
File "/home/test/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 310, in connect
return _ConnectionFairy._checkout(self)
File "/home/test/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 868, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File "/home/test/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 476, in checkout
rec = pool._do_get()
File "/home/test/lib/python3.9/site-packages/sqlalchemy/pool/impl.py", line 134, in _do_get
raise exc.TimeoutError(
sqlalchemy.exc.TimeoutError: QueuePool limit of size 5 overflow 10 reached, connection timed out, timeout 30.00 (Background on this error at: https://sqlalche.me/e/14/3o7r)
```
### Versions
```
Version: 2.0.2
API version: 0.8.0
Python version: 3.9.13
Git commit: 062ac5ac
Built: Tue, Aug 2, 2022 2:59 PM
OS/Arch: darwin/x86_64
Profile: default
Server type: ephemeral
Server:
Database: postgresql
```
### Additional context
_No response_