View in #prefect-community on Slack
@Xavier_Babu: For some reason, from today onwards in our Orion server, every new execute is going as scheduled by default, not running the flow immediately. Please shed some light.
I am running it via REST APIs. If I run it using CLI, it works fine. Is it possible my scheduled.type is messed up?
@Kevin_Kho: What happens when you preview the work queue?
@Xavier_Babu: It shows “scheduled” in the UI and never runs right away.
@Kevin_Kho: Do you have an agent running?
@Xavier_Babu: yes
@Kevin_Kho: What are the tags/settings on the work queue the agent is using?
@Anna_Geller: Scheduled means the agent doesn’t pick it up from the work queue - it’s likely you configured your work queue for specific tags and you forgot this tag on your DeploymentSpec - could it be?
if not, can you share your DeploymentSpec and your work queue and agent configuration?
@Xavier_Babu: ok, let me look.
Agent started! Looking for work from queue ‘9a1bc942-6c14-4054-9cb9-c77c1e99fe2b’
@Kevin_Kho: Is it working now?
@Xavier_Babu: no
still scheduling all jobs
@Anna_Geller: could you answer my questions?
@Xavier_Babu: DeploymentSpec(
flow=custom_flow2,
name=“XBparent_flow2_deploy”,
tags={‘waves_q’},
parameters={‘sql’: ‘SELECT count(*) from table_name;’}
)
If I run in CLI, it runs immediately
@Kevin_Kho: We can’t help you without knowing the work queue configuration
@Xavier_Babu: Sorry, please let me know how to get the WorkQueue configuration.
@Kevin_Kho: In the UI, click on work queues on the left side and then open the one your agent is using
@Xavier_Babu: When I click the 3rd button from the top on the left hand side of the UI, it is not showing only the title WorkQueue and nothing more.
@Anna_Geller: tags should be a list rather than a dictionary - the brackets are just wrong
tags=["waves_q"]
@Xavier_Babu: let me change it and get back to you. Thanks, Anna and Kevin.
is_schedule_active=True? What would it do?
Still not working. Even I do QuickRUn from the UI and it shows the state as “late”. I haven’t added any schedule in this deployment.
@Anna_Geller: can you share more info? perhaps you could describe all steps you took step by step so that we can find out what went wrong
@Xavier_Babu: Is it possible to have a zoom call and share it with you? I can deploy the flow with no error and while execute/run via REST APIs, it is going in “scheduled” or “late” state instead of running. But If I run it via CLI like “prefect deployment execute”, it runs ok
For some reason, when we deploy from RESTAPI, it is not moving from Submitted to Running State. Instead it goes to Late state.
In another case when I deploy using scheduled in the DeploymentSpec I got the following exception in the agent and agent crashed.
File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/contextlib.py", line 153, in __exit__
self.gen.throw(typ, value, traceback)
File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
raise mapped_exc(message) from exc
httpx.ConnectError: All connection attempts failed
An exception occurred.
Agent started! Looking for work from queue '9a1bc942-6c14-4054-9cb9-c77c1e99fe2b'...
Traceback (most recent call last):
File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/prefect/client.py", line 747, in get_runs_in_work_queue
response = await [self._client.post](http://self._client.post)(
File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/prefect/utilities/httpx.py", line 137, in post
return await self.request(
File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/prefect/utilities/httpx.py", line 80, in request
response.raise_for_status()
File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/httpx/_models.py", line 1510, in raise_for_status
raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Client error '404 Not Found' for url '<http://dtord03aop02d.dc.dotomi.net:5200/api/work_queues/9a1bc942-6c14-4054-9cb9-c77c1e99fe2b/get_runs>'
For more information check: <https://httpstatuses.com/404>
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/prefect/cli/base.py", line 59, in wrapper
return fn(**args, ***kwargs)
File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/prefect/utilities/asyncio.py", line 120, in wrapper
return run_async_in_new_loop(async_fn, **args, ***kwargs)
File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/prefect/utilities/asyncio.py", line 67, in run_async_in_new_loop
return anyio.run(partial(__fn, **args, ***kwargs))
File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/anyio/_core/_eventloop.py", line 56, in run
return asynclib.run(func, **args, ***backend_options)
File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 233, in run
return native_run(wrapper(), debug=debug)
File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/asyncio/base_events.py", line 641, in run_until_complete
return future.result()
File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 228, in wrapper
return await func(*args)
File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/prefect/cli/agent.py", line 71, in start
await agent.get_and_submit_flow_runs()
File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/prefect/agent.py", line 88, in get_and_submit_flow_runs
submittable_runs = await self.client.get_runs_in_work_queue(
File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/prefect/client.py", line 753, in get_runs_in_work_queue
raise prefect.exceptions.ObjectNotFound(http_exc=e) from e
prefect.exceptions.ObjectNotFound
An exception occurred.
@Anna_Geller: > Is it possible to have a zoom call and share it with you?
we don’t do that as part of community support, you would need paid support tier - you could reach out to cs@prefect.io
can you share the flow code? this spec is fine:
DeploymentSpec(
flow=custom_flow2,
name="XBparent_flow2_deploy",
tags=["waves_q"],
parameters={'sql': 'SELECT count(*) from table_name;'}
)
provided that you have a work queue with this tag:
prefect work-queue create somename -t waves_q
@Xavier_Babu: after we scheduled a flow, while running agent crashes with the following message.
raise ValueError(f"Unregistered flow runner {typename!r}")
ValueError: Unregistered flow runner ‘string’
An exception occurred.
Do we need to register the FlowRunner? What we have to fill-in in flowrunner config()?
I fixed the FlowRunner using Subprocess and working fine now. Thanks Anna and Kevin.
@Anna_Geller: Thanks for sharing - so it was some sort of typo or misconfiguration in the flow runner? Or were you using some custom subclass of a FlowRunner?
@Xavier_Babu: Misconfiguration
But it shouldn’t crash the agent every time. Instead it should have thrown some error message to guide us.
@Anna_Geller: thanks for sharing and glad it’s resolved now