Hi, I am new to Prefect and I’ve been trying things out. So far, I like it very much and everything has worked smoothly. (so far I have only tried flows locally, though)
But here’s something I struggled with. Not sure if it is simply not meant to be used that way, or that I just haven’t figure it out yet, but here it is… Consider the workflow I have in mind:
- define a flow, with a bunch of tasks. All tasks are cached, using
task_input_hash, because they can take a while to complete. - run it locally
So far so good. Now what?, well, I would like to take a look at the result of the flow and all of its tasks, since they are cached anyway. Ideally, this should be just just a matter of lunching a python REPL or a Jupyter notebook, and read the saved results. How can I do that?
After reading about results, unfortunately, I haven’t figure it out. Yeah, I can add some more tasks to the flow and in there use the results of the previous tasks, or invoke the flow from other flow and use the results. But that’s not ideal. I was looking for a way to be able to just examine the results interactively, so I can take the results of task1 and take a look at a diagnostic plot, and take the results of task2 and calculate some aggregates, and so on, to iterate that way without having to mess around again with the flow.
Is this possible?, how can I do it?
For reference, when working in R, I am a heavy user of the targets package. With targets you define your pipeline, run it and targets saves the results (as Prefect’s cached tasks do). And then interactively, you can simply access saved results as tar_read(task_name) and voilà. That’s the kind of thing I was trying to achieve using Prefect.