I have a task that takes 4 hours that might hang. I want to be able to check it is making progress. Is there a mechanism to notify the dashboard of % complete or work done over last 10 minutes?
1 Like
Hi @simon_mackenzie! 
Good question! The team did explore this functionality in the past but we can’t implement that for the following reasons (sorted in the order of importance):
-
We don’t know how long a task will take ahead of time. Likely even you as a workflow author couldn’t know that in advance until you run it.
-
Even if we start displaying progress bars starting from the second task run computation and display the bar based on how long the task took last time, this estimate would be highly unreliable because your task run duration can vary a lot depending on:
- the amount of data your current task run has to process (perhaps in the previous run you used 1 million rows, but the next one, due to very natural exponential growth of data, may now need to process 2 million rows)
- your task may sometimes take longer for nondeterministic reasons such as the reliability and performance of your network or the amount of compute resources available for a given task run
-
Such progress bars would be hard to implement in a consistent way across various platforms and edge cases e.g. how to do that in a performant way across millions of mapped task runs? and does anyone care about a progress bar if you have millions of tasks? you will unlikely manually click through a million of tasks to check the progress bar of each mapped child task run
-
Not many users requested that feature - it turned out that only very few use cases could benefit from that e.g. training a Deep Learning model and seeing a progress bar for each epoch.
Having said that, there are a couple of things that may still be helpful to make task run progress more visible in Prefect:
- first and foremost, you can get a sense of your task run progress by emitting specific log messages; for instance, if your task is performing some data cleaning, you may send custom logs to make it easier for you to track which specific data transformation your task is currently performing (say, applying some regular expression to clean messy string values). This also makes it easier to track when some part of that data cleaning has failed
- related to the above point, you may break down your task into smaller tasks - something we always encourage at Prefect, as it gives you more visibility into the state of your workflow run and may save you a lot of time when something goes wrong; this will likely give you much more visibility into the actual status of your workflow, more than any progress bar can do!
- you can even leverage mapping to give you a much clearer indication of which input a specific operation is applied over and gives you fine-grained visibility into their execution states
- the timeline view allows you to see how long a task run takes during the flow run and you can see tasks in progress marked blue, which gives some indication of the progress of your task runs:
- the Artifacts API allows you to build a fully custom report on task run completion so that you can view custom information (e.g. accuracy metrics of your ML model) directly from the UI. But note that this report is only available once the task run finishes
To read more about the Artifacts API: 
2 Likes
The success of tqdm I think shows that there are places where a progress bar can be useful. It is probably pointless to integrate into prefect at this point (though Jenkins is just a task-runner really and it has them), but the author might try printing tqdm to stdout and looking at the text logs of the flow.
1 Like
Agreed. I use tqdm with ascii=True