The launch week announcements for today are not over yet!
Have you ever wondered how can you solve one of the following problems?
- Deploy a Prefect agent on AWS EC2, EKS, and ECS using Terraform templates
- Ensure repeatable deployments across various environments (dev, staging, production) using Infrastructure as Code
- Build a CI/CD pipeline to automatically register Prefect flows upon closed pull requests using GitHub Action workflows
- Integrate Prefect with Airbyte for EL and dbt for T (Yes, it stands for Extract - Load - Transform - a very common pattern in data engineering workflows)
All of the above points reflect extremely common use cases that nearly every new Prefect user needs to figure out on their own when getting started with Prefect. Since our mission is to eliminate negative engineering, we want to eliminate that burden and make it easy for you to implement best engineering practices when using Prefect. Therefore, we’re extremely excited to release something we’ve been working on for several months in the Customer Success Team - the Customer Success Recipes!
All of the recipes can be found in this public repository:
Shoutout to @George_Coyne, @jamiedick and @nate who put this together. This repository is a real treasure with many common flow deployment patterns and Prefect integrations. Don’t be fooled by the repository description:
Snippets and templates representing common Customer Success patterns
This extremely modest description is a complete understatement with respect to how much value it provides! Make sure to check it out and star it!
There are currently 14 recipes in there as of today, with plans to add many more over the coming weeks. Please feel free to request specific recipes via GitHub Issues. We also welcome contributions from your side if you are interested in collaborating on some of those efforts!
Here is a full list of available recipes and the roadmap ahead as of today:
- Airbyte On EC2: Deploys Airbyte on an EC2 instance in an autoscaling group.
- Delete Default VPCs: Removes the default VPC across all AWS regions.
- Basic Network: Deploy private networking infrastructure.
- Terraform State Management: Deploy infrastructure to host future Terraform state files.
- Docker Agent on EC2: Deploy 1 or more EC2 instances within an Autoscaling Group that will host the Prefect agent.
- ECS Agent on ECS: Deploy the Prefect agent on AWS Elastic Container Service as a long-running service.
- Kubernetes Agent on EKS: A terraform module to deploy an Amazon EKS cluster with the Prefect agent installed.
- Event Driven Flow: Lambda for event-driven Prefect flows
- Airbyte Orchestration: kicking off Airbyte syncs with a Prefect flow
- Using S3 Storage: using S3 to register flows as scripts
Secret Import: Import your secrets from your local
~/.prefect/config.tomlinto your active Prefect Cloud tenant.
- DBT orchestration
- AWS Batch deployments
- Orchestrating AWS batch with Prefect
- Prefect agent deployments on GCP
- CI registration to Prefect Cloud
See the open issues for a full list of proposed features (and known issues).
Thank you for jumping into Prefect 2.0b1 with such enthusiasm! Based on the feedback we’ve heard so far, the team has cut a release with some minor enhancements:
Docker flow runners can connect to local API applications on Linux without binding to
with_optionsmethod to flows allowing override of settings e.g. the task runner.
…and a few fixes as well:
- The CLI no longer displays tracebacks on a successful exit.
- Returning pandas objects from tasks does not error.
- Flows are now listed correctly in the UI dashboard.
You can learn more in the release notes.
Keep the feedback coming - it’s the fuel that drives Prefect 2.0’s development!