Prefect server schedule stops after 10 runs

Archive
1

I have a prefect server setup with multiple registered flows. When I schedule the flows, 10 runs will be created per flow and after running those, no more runs will get created and the flow stops. I need to turn off the schedule and turn it back on, in order to have 10 more runs.
Is this the expected behavior? Or is it a bug or I’m doing something wrong here?

Core Version: 1.2.2
API Release: Apr 15 • 12:08AM
UI Release: May 18 • 7:08PM

2

It’s true that by default, 10 runs will be created but after those runs, new 10 runs will be scheduled again. If this is not the case, you would need to provide a bit more information including your run configuration, and schedule, how you attach those to your flow, and how you register your flow.

3

Thanks for your reply.
Please consider that I have multiple flows with tasks that are taking a few minutes to finish with daily schedules and all of them are facing the same issue. Here you can find information on a simple task which I created to quickly replicate my issue.

flow.py:

import time
from prefect import task, Flow

if __name__ == '__main__':

    @task(log_stdout=True)
    def my_task() -> None:
        time.sleep(1)

    with Flow( "My Flow") as flow:
        my_task()
        
    flow.register(project_name='My Project', labels=['my_label'])

the project gets build into docker image with a custom “prefect.toml” file and the following command:

CMD ["prefect", "agent", "local", "start", "-l", "my_label"]

After running the docker container, flow gets registered with below command (inside the container):

python3 flow.py

Inside the Prefect frontend, in the flow setting “SCHEDULES” table, I’m setting a schedule to “Run every 1 minute” then turning the “SCHEDULES” radio button on.
It will successfully runs all the scheduled 10 runs and no more runs get scheduled.

The Prefect Server itself is hosted in an Air-Gaped network using official provided compose file with some small modification to match our infrastructure. I see no errors in “hasura”, “graphql” and “frontend” container logs. Only below error keeps popping up in the “apollo” container which is caused by having no internet access.

Sending telemetry to Prefect Technologies, Inc.: {"source":"prefect_server","type":"heartbeat","payload":{"id":"feb0a78a-8556-464c-a7ec-635be60fb057","prefect_server_version":"2022.04.14","api_version":"0.2.0"}}
(node:26) UnhandledPromiseRejectionWarning: FetchError: request to https://sens-o-matic.prefect.io/ failed, reason: connect ETIMEDOUT 172.217.20.83:443
    at ClientRequest.<anonymous> (/apollo/node_modules/node-fetch/lib/index.js:1491:11)
    at ClientRequest.emit (events.js:315:20)
    at TLSSocket.socketErrorListener (_http_client.js:469:9)
    at TLSSocket.emit (events.js:315:20)
    at emitErrorNT (internal/streams/destroy.js:106:8)
    at emitErrorCloseNT (internal/streams/destroy.js:74:3)
    at processTicksAndRejections (internal/process/task_queues.js:80:21)
(node:26) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 7920)
4

Thanks for sharing more info. In your flow code, you keep registering the script rather than scheduling:

Try this instead:

import time
from prefect import task, Flow
from prefect.schedules import CronSchedule

@task(log_stdout=True)
def my_task() -> None:
    time.sleep(1)

with Flow( "My Flow", schedule=CronSchedule("*/15 * * * *"), run_config=your_run_config_with_agent_label) as flow:
    my_task()

if __name__ == '__main__':
    flow.register(project_name='My Project')

If you’re just getting started, you should use Prefect 2 as support for Prefect 1 will end next year. Docs on prefect 2: docs.prefect.io

1 Like
5

Unfortunately your solution didn’t change anything.
We have had this sort of struggles with Prefect server for almost a year now. However the Prefect core is great and we are continue using it, but without a server for managing multiple hundreds of our flows, its just a fancy python “schedule”. I think it is time for us to move on from the Prefect server, which is a shame since a lot of time and effort of my team has been put to develope prefect flows, cloud hooks and etc.
These are only my thoughts and you have every right to ignore them, but because of my great love twoard Prefect I would like to share my frustrations.

6

Scheduling is entirely optional in both v1 and v2, so I don’t fully understand where this scheduler argument is coming from. It would be great if we could be constructive. What do you think should be improved? is some documentation lacking?

I encourage you to give Prefect 2 a try: docs.prefect.io. as this is much easier to use, and no flow registration is required. Feedback is always welcome, as long as this is constructive feedback that we can use to improve the product and user experience upon.