CI/CD for Prefect 2.0

[anna_geller]

Hi everyone!

We are aware that DevOps and GitOps are important aspects for Prefect users. We want to support you in following industry best practices when building and deploying your flows. Building reliable CI/CD pipelines and figuring out the right repository structures and execution layer are negative engineering problems - the recipes in this post attempt to eliminate them.

# Prefect 2.0 and Cloud 2.0 recipes :cloud:

With all recipes in this section, start by signing up for Cloud 2.0 as described in this topic:

How to get started with Prefect Cloud 2.0

To get started with Prefect Cloud 2.0, follow these steps: Sign in or register a Prefect Cloud account. Create workspaces for your account. Create an API key to authorize a local execution environment. Configure Orion settings to use Prefect Cloud. Configure storage. Run a flow and display the flow run in Prefect Cloud. This page will walk you through the process: Login from CLI After installing Prefect 2.0, you may login to your workspace using: prefect cloud login -k API_KEY -w "a…

Then, follow one of the recipes below depending on your use case, cloud provider, platform, and your data stack overall.

Simple repository with GitHub Actions workflow (based on Prefect base image) for a single flow with S3 storage

The simplest GitHub action for Cloud 2.0: name: CI on: push: branches: [ main ] jobs: build: runs-on: ubuntu-latest container: prefecthq/prefect:2.0b6-python3.9 env: API_KEY: ${{ secrets.PREFECT_API_KEY}} steps: - uses: actions/checkout@v3 - name: Login to Prefect Cloud 2.0 run: prefect cloud login --key $API_KEY --workspace 'annaprefect/prod' - name: Check Prefect version run: prefect version - name: Configure AWS Credentia…

Simple repository with GitHub Actions workflow - manually specified flows in the GHA matrix + explicitly defined S3 FileStorageBlock

GitHub Actions workflow name: CI on: push: branches: [ main ] jobs: build: runs-on: ubuntu-latest env: API_KEY: ${{ secrets.PREFECT_API_KEY}} strategy: matrix: flows: [ hello_flow.py, hello_flow_2.py ] steps: - uses: actions/checkout@v3 - name: Set up Python 3.9 uses: actions/setup-python@v3 with: python-version: 3.9 - name: Dev dependencies run: | pip install s3fs pip install -U…

Simple repository with GitHub Actions workflow for multiple flows/deployments (manually specified flows in the GHA matrix) and S3 storage

The simplest GitHub action for Cloud 2.0: name: CI on: push: branches: [ main ] jobs: build: runs-on: ubuntu-latest container: prefecthq/prefect:2.0b6-python3.9 env: API_KEY: ${{ secrets.PREFECT_API_KEY}} strategy: matrix: flows: [ hello_flow.py, hello_flow_2.py ] steps: - uses: actions/checkout@v3 - name: Login to Prefect Cloud 2.0 run: prefect cloud login --key $API_KEY --workspace 'annaprefect/prod' - name: Check Pref…

Simple repository with GitHub Actions workflow for multiple flows/deployments (automatically inferred flows in the GHA matrix) and GCS storage (with both fsspec gcsfs + GoogleCloudStorageBlock examples)

GitHub Actions workflow name: CI on: push: branches: [ 05-fsspec-gcs-dynamic-matrix ] jobs: list-flows: runs-on: ubuntu-latest outputs: matrix: ${{ steps.set-matrix.outputs.matrix }} steps: - uses: actions/checkout@v3 - id: set-matrix run: echo "::set-output name=matrix::$(ls flows/*.py | jq -R -s -c 'split("\n")[:-1]')" deploy: needs: list-flows runs-on: ubuntu-latest env: API_KEY: ${{ secrets.PREFECT_API_KEY}} strategy: …

Running unit tests as part of a CI/CD pipeline

Running unit tests as part of a CI/CD pipeline

This Discourse topic is waiting for a contributor! Add your content as a comment, and we’ll review and consolidate it. Thanks for helping us grow this knowledge base! Resources & related topics you can use

CircleCI

todo

AWS Code Build process

todo

GCP Cloud Build

todo

Cloud Build Serverless CI/CD Platform  |  Google Cloud

GitLab pipelines

todo

Bitbucket pipelines

todo

Self-hosted Prefect 2.0 recipes

Creating a CI/CD pipeline for a self-hosted Orion instance is difficult since you would likely want to deploy this to private infrastructure, preventing using tools such as GitHub Actions.

You may explore:

• custom GitHub Actions runners deployed to the same private infrastructure as your Prefect 2.0 orchestration layer and your execution layer
• custom Gitlab or other Git server and corresponding pipelines

For the OSS Prefect 2.0 recipes, we’d love to see community contributions. If you have something worth sharing, we encourage you to create a Discourse topic or simply reply to this topic and share your approach. We don’t require your solution to be perfect or follow best practices (although this would be great) - feel free to share your approach as is - no judgement here, we are all learning along the way.

Thanks to everyone helping to build out this knowledge base about DevOps, GitOps and DataOps solutions using Prefect.