This list of resources will point you to any topic of your interest regarding how to build a reliable flow deployment process suitable to your use cases, your data stack, and your team.
1. Getting started with Prefect 2.0 - learn concepts, run first flows
Getting-started repository templates for Prefect 2.0
Explanation of various concepts and how they relate to each other
For each of the bullet points below , you may create a separate topic in the DataOps category:
- Storage for flow code vs. persisting task run result (both might be based on the same storage block)
- Agents and work-queues relationship - why do you need both?
- How to manage multiple work queues and agents for various use cases (ELT, data transformations, ML, reporting, process automation, etc)
- Can I load-balance work across agents? if so, how to spread the load equally across multiple agents? Is Prefect the right tool to do that?
- Why does Prefect 2.0 have a concept of work queues but doesn’t have any message queue (such as Celery, RabbitMQ) as part of the architecture?
- Can I spin up agents on demand? What are the pros and cons of not running an agent 24/7?
- How to ensure my agent and execution layer can scale according to my workflows? How to ensure elasticity and balance that with compute costs?
- When do I have to recreate a deployment?
- How can I configure alerts on failure?
- Demo of a failure and how Prefect 2.0 help troubleshoot broken pipelines
- What can you do from CLI vs. UI vs. from Python client?
2. DevOps and GitOps patterns
Managing environments with Prefect 2.0 (dev, staging, prod)
Deploying your execution layer - your Prefect 2.0 agents
Continuous Integration & Deployment
Packaging code dependencies into a Docker image and/or Python package
This will be largely covered by DockerImagePackager abstraction soon, but the recipes needed are about how to authenticate to various container registries
Managing credentials in Prefect 2.0
3. Common workflow patterns & use cases
Parametrization & type validation
Event-driven workflows in Prefect 2.0
Real-time streaming workflows in Prefect 2.0
4. Orchestrating Data Warehousing flows & the Modern Data Stack
5. PoC recipes based on your tech stack
How to build a Prefect 2.0 PoC on AWS
How to build a Prefect 2.0 PoC on GCP
How to build a Prefect 2.0 PoC on Azure
How to build a Prefect 2.0 PoC on a single server or VM instance
6. Concurrency and Parallelism with Async, Dask and Ray