What is the setting PREFECT_ORION_DATABASE_CONNECTION_TIMEOUT used for and can this setting be used to mitigate my timeout errors?

Archive
,
1

I have been getting some errors such as below. This specific problem seems to be solved by increasing PREFECT_ORION_DATABASE_CONNECTION_TIMEOUT.

  • what is the purpose of this timeout and why is the default value so low?
  • how should one deal with prefect errors generally versus data errors? for example I am expecting tasks to fail due to data so do not generally retry or fail. However if prefect has an error it will skip some tasks. is it possible to automatically retry tasks where there is a prefect failure?

httpx.HTTPStatusError: Server error ‘500 Internal Server Error’ for url ‘http://host.docker.internal:4200/api/logs/’"

‘HTTPStatusError("Server error '500 Internal Server Error' for url 'http://host.docker.internal:4200/api/task_runs/74cc4d07-e3be-45f3-8f4d-c6521c6bacf4/set_state\’\nFor more information check: 500 Internal Server Error - HTTP Status Code Glossary - WebFX")’

2

Could you share more under which circumstances do you experience this error? the HTTP 500 Internal Server Error is a general error that may occur for a variety of reasons

I would need to know more about the use case and your setup to replicate and help troubleshoot in any meaningful way.

Do you still run it on WSL2 with DockerFlowRunner? Can you share the flow code and your output of prefect version?

3

Also, to answer your question directly:

  • the purpose of this variable is to protect the backend database against expensive queries
  • the reason: queries shouldn’t take longer than 1.5 seconds. That’s why 5 seconds is a rather sensible and conservative default

Could you explain more about when are you getting such errors?

4

The errors appear randomly so hard to nail down. However on the server it looks like error is postgres request. And pretty sure that timeout solves it. Will run more tests tomorrow.

I don’t have any database requests that take a long time. none should as no data is passed to the server I think. However are a lot of tasks so potentially many requests at same time. Sometimes even fails on log request.

Latest version 2 days ago. 2c I recall

1 Like
5

I have my application running in a container with prefect 2.0b3 installed and executing on local dask cluster. Then I have separate containers for orion server (also prefect 2.0b3) and postgres.

Unfortunately sometimes it works fine and sometimes not. When it fails the orion log shows a timeout error and the failure is in asyncpg. Also on one occasion I had “MarkLateRuns took 5.730745 seconds to run, which is longer than its loop interval of 5.0 seconds.”

My guess is that the database action would easily be completed in 5 seconds if it had a dedicated processor but there are lots of tasks running in parallel. Could it be that postgres requests are in a separate thread/process and has to wait its turn for CPU resource? In my case I have 8 vCPUs but set n_workers=6 to allow extra RAM/worker.

A recent orion log is below:

12:31:25.891 | ERROR   | prefect.orion - Encountered exception in request:
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 159, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 82, in __call__
    raise exc
  File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 71, in __call__
    await self.app(scope, receive, sender)
  File "/usr/local/lib/python3.7/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in __call__
    raise e
  File "/usr/local/lib/python3.7/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 656, in __call__
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 259, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 61, in app
    response = await func(request)
  File "/usr/local/lib/python3.7/site-packages/prefect/orion/utilities/server.py", line 87, in handle_response_scoped_depends
    response = await default_handler(request)
  File "/usr/local/lib/python3.7/contextlib.py", line 678, in __aexit__
    raise exc_details[1]
  File "/usr/local/lib/python3.7/contextlib.py", line 661, in __aexit__
    cb_suppress = await cb(*exc_details)
  File "/usr/local/lib/python3.7/contextlib.py", line 552, in _exit_wrapper
    return await cm_exit(cm, exc_type, exc, tb)
  File "/usr/local/lib/python3.7/contextlib.py", line 177, in __aexit__
    await self.gen.__anext__()
  File "/usr/local/lib/python3.7/site-packages/prefect/orion/api/dependencies.py", line 28, in get_session
    yield session
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/ext/asyncio/session.py", line 705, in __aexit__
    self._sync_transaction().__exit__, type_, value, traceback
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 136, in greenlet_spawn
    result = context.switch(value)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/util.py", line 237, in __exit__
    self.rollback()
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/langhelpers.py", line 72, in __exit__
    with_traceback=exc_tb,
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
    raise exception
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/util.py", line 233, in __exit__
    self.commit()
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 836, in commit
    trans.commit()
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2378, in commit
    self._do_commit()
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2568, in _do_commit
    self._connection_commit_impl()
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2539, in _connection_commit_impl
    self.connection._commit_impl()
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1017, in _commit_impl
    self._handle_dbapi_exception(e, None, None, None, None)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2047, in _handle_dbapi_exception
    util.raise_(exc_info[1], with_traceback=exc_info[2])
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
    raise exception
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1015, in _commit_impl
    self.engine.dialect.do_commit(self.connection)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 685, in do_commit
    dbapi_connection.commit()
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 741, in commit
    self._handle_exception(error)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 684, in _handle_exception
    raise error
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 739, in commit
    self.await_(self._transaction.commit())
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 76, in await_only
    return current.driver.switch(awaitable)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 129, in greenlet_spawn
    value = await result
  File "/usr/local/lib/python3.7/site-packages/asyncpg/transaction.py", line 211, in commit
    await self.__commit()
  File "/usr/local/lib/python3.7/site-packages/asyncpg/transaction.py", line 179, in __commit
    await self._connection.execute(query)
  File "/usr/local/lib/python3.7/site-packages/asyncpg/connection.py", line 318, in execute
    return await self._protocol.query(query, timeout)
  File "asyncpg/protocol/protocol.pyx", line 338, in query
concurrent.futures._base.TimeoutError
12:31:25.973 | ERROR   | uvicorn.error - Exception in ASGI application
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/uvicorn/protocols/http/h11_impl.py", line 366, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "/usr/local/lib/python3.7/site-packages/uvicorn/middleware/proxy_headers.py", line 75, in __call__
    return await self.app(scope, receive, send)
  File "/usr/local/lib/python3.7/site-packages/uvicorn/middleware/message_logger.py", line 82, in __call__
    raise exc from None
  File "/usr/local/lib/python3.7/site-packages/uvicorn/middleware/message_logger.py", line 78, in __call__
    await self.app(scope, inner_receive, inner_send)
  File "/usr/local/lib/python3.7/site-packages/fastapi/applications.py", line 261, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.7/site-packages/starlette/applications.py", line 112, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 181, in __call__
    raise exc
  File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 159, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.7/site-packages/starlette/middleware/cors.py", line 84, in __call__
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 82, in __call__
    raise exc
  File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 71, in __call__
    await self.app(scope, receive, sender)
  File "/usr/local/lib/python3.7/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in __call__
    raise e
  File "/usr/local/lib/python3.7/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 656, in __call__
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 408, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.7/site-packages/fastapi/applications.py", line 261, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.7/site-packages/starlette/applications.py", line 112, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 181, in __call__
    raise exc
  File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 159, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 82, in __call__
    raise exc
  File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 71, in __call__
    await self.app(scope, receive, sender)
  File "/usr/local/lib/python3.7/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in __call__
    raise e
  File "/usr/local/lib/python3.7/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 656, in __call__
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 259, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 61, in app
    response = await func(request)
  File "/usr/local/lib/python3.7/site-packages/prefect/orion/utilities/server.py", line 87, in handle_response_scoped_depends
    response = await default_handler(request)
  File "/usr/local/lib/python3.7/contextlib.py", line 678, in __aexit__
    raise exc_details[1]
  File "/usr/local/lib/python3.7/contextlib.py", line 661, in __aexit__
    cb_suppress = await cb(*exc_details)
  File "/usr/local/lib/python3.7/contextlib.py", line 552, in _exit_wrapper
    return await cm_exit(cm, exc_type, exc, tb)
  File "/usr/local/lib/python3.7/contextlib.py", line 177, in __aexit__
    await self.gen.__anext__()
  File "/usr/local/lib/python3.7/site-packages/prefect/orion/api/dependencies.py", line 28, in get_session
    yield session
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/ext/asyncio/session.py", line 705, in __aexit__
    self._sync_transaction().__exit__, type_, value, traceback
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 136, in greenlet_spawn
    result = context.switch(value)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/util.py", line 237, in __exit__
    self.rollback()
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/langhelpers.py", line 72, in __exit__
    with_traceback=exc_tb,
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
    raise exception
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/util.py", line 233, in __exit__
    self.commit()
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 836, in commit
    trans.commit()
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2378, in commit
    self._do_commit()
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2568, in _do_commit
    self._connection_commit_impl()
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2539, in _connection_commit_impl
    self.connection._commit_impl()
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1017, in _commit_impl
    self._handle_dbapi_exception(e, None, None, None, None)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2047, in _handle_dbapi_exception
    util.raise_(exc_info[1], with_traceback=exc_info[2])
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
    raise exception
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1015, in _commit_impl
    self.engine.dialect.do_commit(self.connection)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 685, in do_commit
    dbapi_connection.commit()
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 741, in commit
    self._handle_exception(error)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 684, in _handle_exception
    raise error
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 739, in commit
    self.await_(self._transaction.commit())
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 76, in await_only
    return current.driver.switch(awaitable)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 129, in greenlet_spawn
    value = await result
  File "/usr/local/lib/python3.7/site-packages/asyncpg/transaction.py", line 211, in commit
    await self.__commit()
  File "/usr/local/lib/python3.7/site-packages/asyncpg/transaction.py", line 179, in __commit
    await self._connection.execute(query)
  File "/usr/local/lib/python3.7/site-packages/asyncpg/connection.py", line 318, in execute
    return await self._protocol.query(query, timeout)
  File "asyncpg/protocol/protocol.pyx", line 338, in query
concurrent.futures._base.TimeoutError
12:31:31.482 | ERROR   | prefect.orion - Encountered exception in request:
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 159, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 82, in __call__
    raise exc
  File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 71, in __call__
    await self.app(scope, receive, sender)
  File "/usr/local/lib/python3.7/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in __call__
    raise e
  File "/usr/local/lib/python3.7/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 656, in __call__
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 259, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 61, in app
    response = await func(request)
  File "/usr/local/lib/python3.7/site-packages/prefect/orion/utilities/server.py", line 87, in handle_response_scoped_depends
    response = await default_handler(request)
  File "/usr/local/lib/python3.7/site-packages/fastapi/routing.py", line 228, in app
    dependant=dependant, values=values, is_coroutine=is_coroutine
  File "/usr/local/lib/python3.7/site-packages/fastapi/routing.py", line 160, in run_endpoint_function
    return await dependant.call(**values)
  File "/usr/local/lib/python3.7/site-packages/prefect/orion/api/task_runs.py", line 193, in set_task_run_state
    task_policy=task_policy,
  File "/usr/local/lib/python3.7/site-packages/prefect/orion/models/task_runs.py", line 279, in set_task_run_state
    run = await models.task_runs.read_task_run(session=session, task_run_id=task_run_id)
  File "/usr/local/lib/python3.7/site-packages/prefect/orion/database/dependencies.py", line 112, in async_wrapper
    return await fn(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/prefect/orion/models/task_runs.py", line 97, in read_task_run
    model = await session.get(db.TaskRun, task_run_id)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/ext/asyncio/session.py", line 303, in get
    identity_token=identity_token,
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 136, in greenlet_spawn
    result = context.switch(value)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 2793, in get
    execution_options=execution_options,
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 2896, in _get_impl
    load_options=load_options,
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/loading.py", line 534, in load_on_pk_identity
    bind_arguments=bind_arguments,
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 1692, in execute
    result = conn._execute_20(statement, params or {}, execution_options)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1631, in _execute_20
    return meth(self, args_10style, kwargs_10style, execution_options)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 326, in _execute_on_connection
    self, multiparams, params, execution_options
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1508, in _execute_clauseelement
    cache_hit=cache_hit,
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1863, in _execute_context
    e, statement, parameters, cursor, context
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2047, in _handle_dbapi_exception
    util.raise_(exc_info[1], with_traceback=exc_info[2])
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
    raise exception
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1820, in _execute_context
    cursor, statement, parameters, context
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 732, in do_execute
    cursor.execute(statement, parameters)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 480, in execute
    self._prepare_and_execute(operation, parameters)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 76, in await_only
    return current.driver.switch(awaitable)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 129, in greenlet_spawn
    value = await result
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 454, in _prepare_and_execute
    self._handle_exception(error)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 389, in _handle_exception
    self._adapt_connection._handle_exception(error)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 684, in _handle_exception
    raise error
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 419, in _prepare_and_execute
    operation, self._invalidate_schema_cache_asof
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 660, in _prepare
    prepared_stmt = await self._connection.prepare(operation)
  File "/usr/local/lib/python3.7/site-packages/asyncpg/connection.py", line 571, in prepare
    record_class=record_class,
  File "/usr/local/lib/python3.7/site-packages/asyncpg/connection.py", line 589, in _prepare
    record_class=record_class,
  File "/usr/local/lib/python3.7/site-packages/asyncpg/connection.py", line 414, in _get_statement
    types_with_missing_codecs, timeout)
  File "/usr/local/lib/python3.7/site-packages/asyncpg/connection.py", line 464, in _introspect_types
    ignore_custom_codec=True,
  File "/usr/local/lib/python3.7/site-packages/asyncpg/connection.py", line 1689, in __execute
    ignore_custom_codec=ignore_custom_codec,
  File "/usr/local/lib/python3.7/site-packages/asyncpg/connection.py", line 1734, in _do_execute
    result = await executor(stmt, timeout)
  File "asyncpg/protocol/protocol.pyx", line 201, in bind_execute
concurrent.futures._base.TimeoutError
12:31:31.495 | ERROR   | uvicorn.error - Exception in ASGI application
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/uvicorn/protocols/http/h11_impl.py", line 366, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "/usr/local/lib/python3.7/site-packages/uvicorn/middleware/proxy_headers.py", line 75, in __call__
    return await self.app(scope, receive, send)
  File "/usr/local/lib/python3.7/site-packages/uvicorn/middleware/message_logger.py", line 82, in __call__
    raise exc from None
  File "/usr/local/lib/python3.7/site-packages/uvicorn/middleware/message_logger.py", line 78, in __call__
    await self.app(scope, inner_receive, inner_send)
  File "/usr/local/lib/python3.7/site-packages/fastapi/applications.py", line 261, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.7/site-packages/starlette/applications.py", line 112, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 181, in __call__
    raise exc
  File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 159, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.7/site-packages/starlette/middleware/cors.py", line 84, in __call__
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 82, in __call__
    raise exc
  File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 71, in __call__
    await self.app(scope, receive, sender)
  File "/usr/local/lib/python3.7/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in __call__
    raise e
  File "/usr/local/lib/python3.7/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 656, in __call__
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 408, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.7/site-packages/fastapi/applications.py", line 261, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.7/site-packages/starlette/applications.py", line 112, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 181, in __call__
    raise exc
  File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 159, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 82, in __call__
    raise exc
  File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 71, in __call__
    await self.app(scope, receive, sender)
  File "/usr/local/lib/python3.7/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in __call__
    raise e
  File "/usr/local/lib/python3.7/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 656, in __call__
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 259, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 61, in app
    response = await func(request)
  File "/usr/local/lib/python3.7/site-packages/prefect/orion/utilities/server.py", line 87, in handle_response_scoped_depends
    response = await default_handler(request)
  File "/usr/local/lib/python3.7/site-packages/fastapi/routing.py", line 228, in app
    dependant=dependant, values=values, is_coroutine=is_coroutine
  File "/usr/local/lib/python3.7/site-packages/fastapi/routing.py", line 160, in run_endpoint_function
    return await dependant.call(**values)
  File "/usr/local/lib/python3.7/site-packages/prefect/orion/api/task_runs.py", line 193, in set_task_run_state
    task_policy=task_policy,
  File "/usr/local/lib/python3.7/site-packages/prefect/orion/models/task_runs.py", line 279, in set_task_run_state
    run = await models.task_runs.read_task_run(session=session, task_run_id=task_run_id)
  File "/usr/local/lib/python3.7/site-packages/prefect/orion/database/dependencies.py", line 112, in async_wrapper
    return await fn(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/prefect/orion/models/task_runs.py", line 97, in read_task_run
    model = await session.get(db.TaskRun, task_run_id)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/ext/asyncio/session.py", line 303, in get
    identity_token=identity_token,
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 136, in greenlet_spawn
    result = context.switch(value)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 2793, in get
    execution_options=execution_options,
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 2896, in _get_impl
    load_options=load_options,
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/loading.py", line 534, in load_on_pk_identity
    bind_arguments=bind_arguments,
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 1692, in execute
    result = conn._execute_20(statement, params or {}, execution_options)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1631, in _execute_20
    return meth(self, args_10style, kwargs_10style, execution_options)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 326, in _execute_on_connection
    self, multiparams, params, execution_options
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1508, in _execute_clauseelement
    cache_hit=cache_hit,
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1863, in _execute_context
    e, statement, parameters, cursor, context
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2047, in _handle_dbapi_exception
    util.raise_(exc_info[1], with_traceback=exc_info[2])
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
    raise exception
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1820, in _execute_context
    cursor, statement, parameters, context
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 732, in do_execute
    cursor.execute(statement, parameters)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 480, in execute
    self._prepare_and_execute(operation, parameters)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 76, in await_only
    return current.driver.switch(awaitable)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 129, in greenlet_spawn
    value = await result
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 454, in _prepare_and_execute
    self._handle_exception(error)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 389, in _handle_exception
    self._adapt_connection._handle_exception(error)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 684, in _handle_exception
    raise error
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 419, in _prepare_and_execute
    operation, self._invalidate_schema_cache_asof
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 660, in _prepare
    prepared_stmt = await self._connection.prepare(operation)
  File "/usr/local/lib/python3.7/site-packages/asyncpg/connection.py", line 571, in prepare
    record_class=record_class,
  File "/usr/local/lib/python3.7/site-packages/asyncpg/connection.py", line 589, in _prepare
    record_class=record_class,
  File "/usr/local/lib/python3.7/site-packages/asyncpg/connection.py", line 414, in _get_statement
    types_with_missing_codecs, timeout)
  File "/usr/local/lib/python3.7/site-packages/asyncpg/connection.py", line 464, in _introspect_types
    ignore_custom_codec=True,
  File "/usr/local/lib/python3.7/site-packages/asyncpg/connection.py", line 1689, in __execute
    ignore_custom_codec=ignore_custom_codec,
  File "/usr/local/lib/python3.7/site-packages/asyncpg/connection.py", line 1734, in _do_execute
    result = await executor(stmt, timeout)
  File "asyncpg/protocol/protocol.pyx", line 201, in bind_execute
concurrent.futures._base.TimeoutError
6

what do you mean by application? do you consider a Prefect flow an application?

Do you mean that you are attaching DaskTaskRunner like so?

@flow(task_runner=DaskTaskRunner())

We currently don’t fully support such containerized setup because the server host address is hardcoded.

This comment from Michael confirms that:

Perhaps it makes sense to wait and, for now, just run it all as a monolithic service running locally?

You could also switch to Prefect Cloud Beta, available here:
https://beta.prefect.io/

7

Application is my tasks and flows. Prefect is installed there.

Yes @flow(task_runner=DaskTaskRunner())

The setup does work. I have replied to the github issue with how it works for me. It seemed to be working perfectly with sqlite but when I switched to progres I found sometimes I get the 500 errors. It appears to work sometimes. Possibly the more cpu and memory being used the more likely it fails.

1 Like