Running a Subset of all Tasks in a Flow Ad-hoc in Prefect Orion

Archive
,
1

Hi,

In our use case we sometimes want to run a subset of all the tasks in a flow but preserve the dependency structure (i.e. in a flow run, only run a selective set of tasks and skip all other tasks.).

Our tasks do not output any returned variable, so there is no data dependency. In 1.0, we used to use custom state handler + ad-hoc run context to pass in a list of task names and let each task know whether to run or to skip. Is there a way to accomplish similar things in 2.0?

Thanks!

1 Like
2

you can do that just using if/else - check this:

1 Like
3

Thanks, Anna! I have tried having conditional logic in my flow, and it works for some of our other use cases. For this specific one, I am thinking more in line with the following situation:

  • A flow has 100 tasks with dependencies setup
  • I want to run 5 tasks in that flow, but preserve their dependency

To do this with if-else, the only way I can think of would be:

  • Let the flow take in a ā€œtask_to_runā€ list as a parameter, default to None (run all tasks)
  • Do sth like this in the flow definition:
    • if task_1.name in tasks_to_run: task_1.submit(wait_for=[ā€˜some_other_tasks]ā€™)
    • repeat for all 100 tasks

This way the partial flow functionality should be fully supported, just that it may not be as clean and concise as I would love to be. Would be great if there is a better way to do this! :smiley:

1 Like
4

no right or wrong here - the best recommendation I can give you here is: write it as if you would do it in plain Python and add decorators and Prefect features only where needed

you could really do it even as one script with a for/if/else and a single flow decorator if you donā€™t need that each of those components is a specific task for retries/caching etc

1 Like