Changing task_runner in prefect deployment

Piotr_Siejda · January 24, 2023, 3:42pm

Is there a way to change the task_runner within a prefect deployment?

Aleksandr_Liadov · January 25, 2023, 4:08pm

It seems to me that non, I tried with

deployment_main_flow = Deployment.build_from_flow(
            flow=main_flow.with_options(name=config.main_flow),
            apply=True,
        )

I succeed only to change flow name on deployment.

Piotr_Siejda · January 27, 2023, 5:17pm

Solution to my problem was posted here:

Permanent fix:

github.com/PrefectHQ/prefect

Configuring different task runner for dev and prod deployments

opened 11:44AM - 16 Mar 22 UTC

marvin-robot

from:slack v2 status:roadmap

## Opened from the [Prefect Public Slack Community](https://prefect.io/slack) *…*davzucky**: How can I setup in Orion the task runner at deployment? I want to be able to change the mode between local dev and prod. I can only see how to do that for flow runners at the deployment level **anna**: It's not possible to do that directly atm since currently the task runner must be specified on the `flow` decorator rather than on the `DeploymentSpec`. And the more I think about it, the more it actually makes sense. The problem you are describing is not "How can I override the task runner used by a flow on a `DeploymentSpec`" - this is one possible _solution_. The actual _problem_ is: "How can I use a different task runner for development and production deployments". And to solve that problem, the intended solution is to have two different deployments: one for dev, and one for prod. The entire dev vs. prod story is not yet fully established - we are working on adding e.g. GitHub storage. What would likely be a good solution to your issue (_that is currently not possible yet_) is that you may have in the end two different branches on the same repo: • the "`dev`" branch may have flow code with, say `DaskTaskRunner`, • the "`main`" or "`prod`" branch may have flow code with a different task runner e.g. `ConcurrentTaskRunner` and then you can have two different deployments - one for dev and one for prod - each of those references the flow code on a given branch. This would give a clear separation of code and environments and would make building CI/CD pipelines much easier. Having said that, it could be worth exploring adding some sort of override on the `DeploymentSpec` (exactly as you mentioned) - I will open an issue to open this up for discussion. The only problem I currently see with that is that it goes a bit against the runtime discoverability in Orion - the way I understand how deployments work is that you should be able to create a single deployment for `your_flow.py` - then, you can run your deployed flow first using say `DaskTaskRunner`. But then you can modify the flow code `your_flow.py`, referenced as `flow_location` on the `DeploymentSpec`, change the task runner to e.g. `ConcurrentTaskRunner` and you can create a flow run of this deployed flow with this new task runner without having to recreate the deployment as Orion allows for runtime discoverability and doesn't force you to preregister any DAG's metadata. Frankly, even in Prefect 1.0, Prefect doesn't store the executor (effectively the same as task runner in 2.0) information in the backend for privacy reasons because it may contain private information such as your Dask/Ray cluster address. Instead, Prefect retrieves this information at runtime from storage - this is another reason I would be leaning more towards two different versions of this flow in dev and prod branches as a solution to this problem. For now, you could certainly introduce a hack by setting some custom parameter value (that can be set on your `DeploymentSpec`) that determines which subflow to call: ``` from prefect import flow from prefect.task_runners import DaskTaskRunner, SequentialTaskRunner @flow(task_runner=DaskTaskRunner()) def dask_flow(): pass # your flow logic here - it may be some extra function @flow(task_runner=SequentialTaskRunner()) def sequential_flow(): pass # your flow logic here - it may be some extra function @flow def parent_flow(environment: str = "dev"): if environment == "dev": dask_flow() else: sequential_flow() ``` <@ULVA73B9P> open "Orion: as a user, how can I use a different task runner for development and production deployments?" Original thread can be found [here](https://prefect-community.slack.com/archives/CL09KU1K7/p1647411686456139?thread_ts=1647411686.456139&cid=CL09KU1K7).

Changing task_runner in prefect deployment

Don't Panic

Read our Privacy Policy