CI/CD for Prefect 2.0

Hi everyone!

We are aware that DevOps and GitOps are important aspects for Prefect users. We want to support you in following industry best practices when building and deploying your flows. Building reliable CI/CD pipelines and figuring out the right repository structures and execution layer are negative engineering problems - the recipes in this post attempt to eliminate them.

# Prefect 2.0 and Cloud 2.0 recipes :cloud:

With all recipes in this section, start by signing up for Cloud 2.0 as described in this topic:

Then, follow one of the recipes below depending on your use case, cloud provider, platform, and your data stack overall.

Running unit tests as part of a CI/CD pipeline

CircleCI

todo

AWS Code Build process

todo

GCP Cloud Build

todo

GitLab pipelines

todo

Bitbucket pipelines

todo

Self-hosted Prefect 2.0 recipes :gear:

Creating a CI/CD pipeline for a self-hosted Orion instance is difficult since you would likely want to deploy this to private infrastructure, preventing using tools such as GitHub Actions.

You may explore:

  • custom GitHub Actions runners deployed to the same private infrastructure as your Prefect 2.0 orchestration layer and your execution layer
  • custom Gitlab or other Git server and corresponding pipelines

For the OSS Prefect 2.0 recipes, we’d love to see community contributions. If you have something worth sharing, we encourage you to create a Discourse topic or simply reply to this topic and share your approach. We don’t require your solution to be perfect or follow best practices (although this would be great) - feel free to share your approach as is - no judgement here, we are all learning along the way.

Thanks to everyone helping to build out this knowledge base about DevOps, GitOps and DataOps solutions using Prefect.