What may be the reason for the error "prefect.exceptions.ObjectNotFound"?

View in #prefect-community on Slack

Xavier_Babu @Xavier_Babu: For some reason, from today onwards in our Orion server, every new execute is going as scheduled by default, not running the flow immediately. Please shed some light.
I am running it via REST APIs. If I run it using CLI, it works fine. Is it possible my scheduled.type is messed up?

Kevin_Kho @Kevin_Kho: What happens when you preview the work queue?

Xavier_Babu @Xavier_Babu: It shows “scheduled” in the UI and never runs right away.

Kevin_Kho @Kevin_Kho: Do you have an agent running?

Xavier_Babu @Xavier_Babu: yes

Kevin_Kho @Kevin_Kho: What are the tags/settings on the work queue the agent is using?

Anna_Geller @Anna_Geller: Scheduled means the agent doesn’t pick it up from the work queue - it’s likely you configured your work queue for specific tags and you forgot this tag on your DeploymentSpec - could it be?

if not, can you share your DeploymentSpec and your work queue and agent configuration?

Xavier_Babu @Xavier_Babu: ok, let me look.
Agent started! Looking for work from queue ‘9a1bc942-6c14-4054-9cb9-c77c1e99fe2b’

Kevin_Kho @Kevin_Kho: Is it working now?

Xavier_Babu @Xavier_Babu: no
still scheduling all jobs

Anna_Geller @Anna_Geller: could you answer my questions?

Xavier_Babu @Xavier_Babu: DeploymentSpec(
flow=custom_flow2,
name=“XBparent_flow2_deploy”,
tags={‘waves_q’},
parameters={‘sql’: ‘SELECT count(*) from table_name;’}
)
If I run in CLI, it runs immediately

Kevin_Kho @Kevin_Kho: We can’t help you without knowing the work queue configuration

Xavier_Babu @Xavier_Babu: Sorry, please let me know how to get the WorkQueue configuration.

Kevin_Kho @Kevin_Kho: In the UI, click on work queues on the left side and then open the one your agent is using

Xavier_Babu @Xavier_Babu: When I click the 3rd button from the top on the left hand side of the UI, it is not showing only the title WorkQueue and nothing more.

Anna_Geller @Anna_Geller: tags should be a list rather than a dictionary - the brackets are just wrong

tags=["waves_q"]

Xavier_Babu @Xavier_Babu: let me change it and get back to you. Thanks, Anna and Kevin.
is_schedule_active=True? What would it do?
Still not working. Even I do QuickRUn from the UI and it shows the state as “late”. I haven’t added any schedule in this deployment.

Anna_Geller @Anna_Geller: can you share more info? perhaps you could describe all steps you took step by step so that we can find out what went wrong

Xavier_Babu @Xavier_Babu: Is it possible to have a zoom call and share it with you? I can deploy the flow with no error and while execute/run via REST APIs, it is going in “scheduled” or “late” state instead of running. But If I run it via CLI like “prefect deployment execute”, it runs ok
For some reason, when we deploy from RESTAPI, it is not moving from Submitted to Running State. Instead it goes to Late state.
In another case when I deploy using scheduled in the DeploymentSpec I got the following exception in the agent and agent crashed.

File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ConnectError: All connection attempts failed
An exception occurred.
Agent started! Looking for work from queue '9a1bc942-6c14-4054-9cb9-c77c1e99fe2b'...
Traceback (most recent call last):
  File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/prefect/client.py", line 747, in get_runs_in_work_queue
    response = await [self._client.post](http://self._client.post)(
  File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/prefect/utilities/httpx.py", line 137, in post
    return await self.request(
  File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/prefect/utilities/httpx.py", line 80, in request
    response.raise_for_status()
  File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/httpx/_models.py", line 1510, in raise_for_status
    raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Client error '404 Not Found' for url '<http://dtord03aop02d.dc.dotomi.net:5200/api/work_queues/9a1bc942-6c14-4054-9cb9-c77c1e99fe2b/get_runs>'
For more information check: <https://httpstatuses.com/404>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/prefect/cli/base.py", line 59, in wrapper
    return fn(**args, ***kwargs)
  File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/prefect/utilities/asyncio.py", line 120, in wrapper
    return run_async_in_new_loop(async_fn, **args, ***kwargs)
  File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/prefect/utilities/asyncio.py", line 67, in run_async_in_new_loop
    return anyio.run(partial(__fn, **args, ***kwargs))
  File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/anyio/_core/_eventloop.py", line 56, in run
    return asynclib.run(func, **args, ***backend_options)
  File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 233, in run
    return native_run(wrapper(), debug=debug)
  File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/asyncio/base_events.py", line 641, in run_until_complete
    return future.result()
  File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 228, in wrapper
    return await func(*args)
  File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/prefect/cli/agent.py", line 71, in start
    await agent.get_and_submit_flow_runs()
  File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/prefect/agent.py", line 88, in get_and_submit_flow_runs
    submittable_runs = await self.client.get_runs_in_work_queue(
  File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/prefect/client.py", line 753, in get_runs_in_work_queue
    raise prefect.exceptions.ObjectNotFound(http_exc=e) from e
prefect.exceptions.ObjectNotFound
An exception occurred.

Anna_Geller @Anna_Geller: > Is it possible to have a zoom call and share it with you?
we don’t do that as part of community support, you would need paid support tier - you could reach out to cs@prefect.io

can you share the flow code? this spec is fine:

DeploymentSpec(
    flow=custom_flow2,
    name="XBparent_flow2_deploy",
    tags=["waves_q"],
    parameters={'sql': 'SELECT count(*) from table_name;'}
)

provided that you have a work queue with this tag:

prefect work-queue create somename -t waves_q

Xavier_Babu @Xavier_Babu: after we scheduled a flow, while running agent crashes with the following message.

raise ValueError(f"Unregistered flow runner {typename!r}")

ValueError: Unregistered flow runner ‘string’
An exception occurred.

Do we need to register the FlowRunner? What we have to fill-in in flowrunner config()?
I fixed the FlowRunner using Subprocess and working fine now. Thanks Anna and Kevin.

Anna_Geller @Anna_Geller: Thanks for sharing - so it was some sort of typo or misconfiguration in the flow runner? Or were you using some custom subclass of a FlowRunner?

Xavier_Babu @Xavier_Babu: Misconfiguration
But it shouldn’t crash the agent every time. Instead it should have thrown some error message to guide us.

Anna_Geller @Anna_Geller: thanks for sharing and glad it’s resolved now