Deploying many flows at once

DataflowOps for Prefect 2.0
,
1

Continuing the discussion from How to deploy all flows from a project at once?:

In Prefect 1 you could point prefect register at a directory and register all flows within it at once. I am not seeing what the equivalent to that would be in Prefect 2.

Is there no way to point prefect deployment build or prefect deployment apply to a whole directory or project at once?

What I’m seeing in the examples from the above linked topic is that each deployment needs its own call to prefect deployment. So if you have 100 deployments for 100 flows, you need to make 100 calls to prefect deployment (with or without --skip-upload).

Am I misunderstanding? Is there no better way to do a mass deployment, similar to Prefect 1’s prefect register?

2

What I am thinking of doing is having a Python script that creates Deployment objects for each flow I want to deploy, and simply running that.

For example:

DEPLOYMENTS = [
    Deployment.build_from_flow(...),
    Deployment.build_from_flow(...),
    ...
]

if __name__ == '__main__':
    for deployment in DEPLOYMENTS:
        deployment.apply()

This gives me a single script to deploy all the flows I care about. It doesn’t feel right, but I’m not seeing how else to get the mass-deployment capability that was possible with Prefect 1’s register.

Perhaps @anna_geller can shed some light here.

3

HI @nchammas - each “deployment” ties together a number of things:

  • Schedule
  • Infrastructure
  • Tags
  • Flow
  • Storage

To your point, each unique deployment would need to be applied.
You can create a list of deployments to apply, or parse them out of a folder structure and apply them en masse however you would still need to specify the deployment for each.
If they are all very similar, then I expect this could work, but maybe you could describe your use case a little more here? What is the scenario where you need to build and re-register 100 deployments simultaneously?

4

My intention is to have all deployments described in code in our repo for describing and managing flows. From there, I want to be able to run a single script or command to sync the deployments described in the repo up to Prefect Cloud.

I’m looking at it the way I look at the typical infrastructure codebase. I have some declarative description of what I want, and a command to deploy that into the environment. That command may be executed by hand, but it may also be triggered by an automated build.

Are people not managing their Prefect deployments in this way? If you have a large repo of dozens of flows and matching deployments, how do you ensure that as various team members make changes those changes are deployed consistently to the environment where they will run from?

It seems most natural to want some kind of single command that gathers all flows and deployments and syncs them up to Prefect Cloud. Prefect 1’s register did something roughly along these lines, but Prefect 2 doesn’t seem to offer anything equivalent.

5

Generally for larger scale environments, or multiple environments I have seen this primarily done through CI/CD (whichever particular tool you use).

As an example, I can have 3 flows, with 3 branches (dev, test, stage).
On each merge action in their respective branch, I fire a pipeline that runs a deployment.py which updates whatever changes I might have made to the deployment, and uses the environment variables of the branch to determine what / where to apply it to (so dev branch gets dev labels, and goes to the dev worker pool for agent pick up).

If just the flow changes, and it’s source controlled, the flow doesn’t need to be re-registered, you just pull it down using your preferred blocks (github, S3, azure, etc.)

Another case is you store the code in the image. This could go in a variety of ways depending on the dynamism of your code - it could be very straightforward where you re-package the image and run the deployment without making any changes to your deployment whatsoever.

As long as your parameters don’t change, this shouldn’t be an issue at all, although if your parameters do change, you would need to update the deployment to reflect that as well.

In short - you’re right, I do not believe we have a similar feature like pointing to prefect register to a directory, however, due to the way deployments are an aggregate of storage, flow, execution, etc., it’s not really relevant that I’ve seen. You can update and use your latest flow code, without needing to re-apply an entirely new deployment if you handle the location of your code already. If you don’t, and you expect prefect to handle the storage of your code for you (like deployment.apply() pushing your code into a bucket), then you would want to run a deployment for changes, but that’s where I’d expect CI/CD to provide that functionality?

6

OK, so how would that typically work?

We have Prefect agents running on Amazon ECS. When someone pushes an update to some flow, then yes, we want CI to update the flow code we have stored on S3. But we also want CI to update the deployments on Prefect Cloud if someone pushes an update to the deployment config, like the configured work queue, the execution schedule, or any of the other possible deployment options.

So when you say:

I am wondering how, roughly, your deployment.py works. Is it similar to the example snippet I posted above?