Main topic: Flow Deployment Patterns & Recipes for Prefect 2.0

This list of resources will point you to any topic of your interest regarding how to build a reliable flow deployment process suitable to your use cases, your data stack, and your team.

1. Getting started with Prefect 2.0 - learn concepts, run first flows

Getting-started repository templates for Prefect 2.0

Explanation of various concepts and how they relate to each other

For each of the bullet points below :point_down: , you may create a separate topic in the DataOps category:

  • Storage for flow code vs. persisting task run result (both might be based on the same storage block)
  • Agents and work-queues relationship - why do you need both?
  • How to manage multiple work queues and agents for various use cases (ELT, data transformations, ML, reporting, process automation, etc)
  • Can I load-balance work across agents? if so, how to spread the load equally across multiple agents? Is Prefect the right tool to do that?
  • Why does Prefect 2.0 have a concept of work queues but doesn’t have any message queue (such as Celery, RabbitMQ) as part of the architecture?
  • Can I spin up agents on demand? What are the pros and cons of not running an agent 24/7?
  • How to ensure my agent and execution layer can scale according to my workflows? How to ensure elasticity and balance that with compute costs?
  • When do I have to recreate a deployment?
  • How can I configure alerts on failure?
  • Demo of a failure and how Prefect 2.0 help troubleshoot broken pipelines
  • What can you do from CLI vs. UI vs. from Python client?

2. DevOps and GitOps patterns

Managing environments with Prefect 2.0 (dev, staging, prod)

Deploying your execution layer - your Prefect 2.0 agents

Continuous Integration & Deployment

Packaging code dependencies into a Docker image and/or Python package

This will be largely covered by DockerImagePackager abstraction soon, but the recipes needed are about how to authenticate to various container registries

Unit testing

Managing credentials in Prefect 2.0

3. Common workflow patterns & use cases

Parametrization & type validation

Event-driven workflows in Prefect 2.0

Real-time streaming workflows in Prefect 2.0

4. Orchestrating Data Warehousing flows & the Modern Data Stack

5. PoC recipes based on your tech stack

How to build a Prefect 2.0 PoC on AWS

How to build a Prefect 2.0 PoC on GCP

How to build a Prefect 2.0 PoC on Azure

How to build a Prefect 2.0 PoC on a single server or VM instance

6. Concurrency and Parallelism with Async, Dask and Ray

1 Like