I have mapped tasks that feed into a reducing task. The mapped tasks complete in around 2 minutes and are shown as completed in the table on the server display. However the timeline display shows them as still running as long as the reducing task is running. The bar gets longer and if I hover it shows duration increasing.
When the reduce task completes the bars for those tasks shrink to their correct duration. Is this a bug or is there some reason for it?
If I understood correctly, I’d say it’s normal - it takes some time to update the task run states, and until then, the task is shown as Running.
But if you could share your flow code and some UI screenshots, it would be easier to assess what’s happening.
Also, it would be helpful if you could share the output of the command prefect diagnostics.
The blue bar cannot start until the five long green bars have finished. Those tasks finished in 3 minutes but still show as a bar. When I hover they show duration 10 minutes increasing. Once the blue task has finished the bars revert to their correct duration of 3 minutes.
The flow is as below. The running blue task is “predict”
basename_pngs = basename.map(pngs)
rotated = pngtasks.rotate.map(pngs, unmapped(csv), basename_pngs)
boxes = pngtasks.OCR.map(rotated, unmapped(csv), basename_pngs)
parsed = pngtasks.parse.map(boxes, basename_pngs)
matched = pngtasks.match.map(parsed, unmapped(csv), basename_pngs)
# merge
merged = jobtasks.merge(matched, csv)
data = jobtasks.prepare_data(merged)
preds = jobtasks.predict(data)
Thanks for explaining. The only issue I see here is that your mapped tasks accept two mapped inputs. You should only have one input to map over and all other arguments to your mapped tasks should be wrapped with unmapped.
basename_pngs = basename.map(pngs)
rotated = pngtasks.rotate.map(pngs, unmapped(csv), unmapped(basename_pngs))
boxes = pngtasks.OCR.map(rotated, unmapped(csv), unmapped(basename_pngs))
parsed = pngtasks.parse.map(boxes, unmapped(basename_pngs))
matched = pngtasks.match.map(parsed, unmapped(csv), unmapped(basename_pngs))
pngs is a list of paths. basename_pngs is the basename of those paths which is used in a target template. So I need to map both as it won’t work as a list. Are you saying it is not possible to do that?
EDIT: if only the first parameter can be mapped then why is there a need for the unmapped function?
This is needed so that Prefect knows which mapped input to iterate over, and which values are supposed to be used as constant values. This documentation page explains it best.
As with almost anything in Prefect, it certainly is possible 
the only question is what’s the right way to design a given flow. For instance, if you need to map over both of those paths, the input to your pngtasks.rotate could be a list of dictionaries rather than a list of strings (paths). This way, within your task, you can grab the right element of that dict.
[{"png": "aaa", "basename_png": "xxx"},
{"png": "bbb", "basename_png": "yyy"},
{"png": "ccc", "basename_png": "zzz"}, ...]
The docs you link to imply you can have more than one mapped input and indeed it does seem to work except for the progress bar issue. Would not work as a dictionary as I have @task(target=“path/to/{basename}”) which requires basename to be one of the arguments.
In next iteration I will move to prefect2 and use the database rather than targets anyway. Will update if the progress bar issue still happens or not.
Where do you see it? If you mean apply_map, that’s a completely different thing.
I can only see:
Sometimes, we don’t want to iterate over one of the inputs – perhaps it’s a constant value, or a list that’s required in its entirety. Prefect supplies a convenient
unmapped()annotation for this case.
Sounds like a good plan!
Sometimes, we don’t want to iterate over one of the inputs
If you had 3 inputs and don’t want to iterate over one then that implies you do want to iterate over the other two.