This Discourse topic is meant to be used as a comprehensive list of resources pointing you to the right documentation or a tutorial. Depending on a different cloud provider or on-premise infrastructure requirements, you may need a different way of deploying Prefect 2.0.
Architecture & components of Prefect 2.0
Prefect 2.0 was designed to be a lightweight product that can simultaneously be used as an ephemeral orchestration API for local development (with embedded SQLite database), as well as a product that can scale to massive workloads when running it in a distributed fashion e.g. with Cloud 2.0.
Docs
This documentation should serve as your main source about the Prefect orchestration layer explaining all components and how they work together:
Consider everything else in this Discours…
Self-hosting on a Linux VM
Installation guide
Troubleshooting tips
Python version
Make sure you are using supported Python version - you can find the minimal version here:
Postgres backend
When using a Postgres backend, make sure to use Postgres version >= 13.
SQLite backend
The main challenge when installing Prefect 2.0 with SQLite backend on Linux is finding the right SQLite version matching the environment (Python version and Linux distribution). Older versions of Linux package a version of SQLite that cannot…
Self-hosting on AWS EC2
VIDEO
These instructions are accurate as of Prefect 2.0b5
Create a VM on Cloud Provider of Choice
Open port 4200 ingress on the VM from 0.0.0.0/0 (all traffic) as well as HTTP in.
pip install prefect 2.0
Set the UI_API_URL with :
prefect config set PREFECT_ORION_UI_API_URL="http://<external-ip>:4200/api"
Start Orion with:
prefect orion start --host 0.0.0.0
From local machine, configure to hit the API with:
prefect config set PREFECT_API_URL="http://<external-ip>:4200/api"
The remote UI will…
Source
Content
Negative Engineering
What is workflow orchestration?
Introduction to Prefect 2.0
First Prefect flow and Basics
Workflow Orchestration
It’s a set of tools that schedule and monitor work that you want to accomplish. Ex: Scheduling ML models training
Example pipeline:
PostgresQL -> Parquet -> Pandas -> Sklearn -> mlflow
↳ Rest API ↳ Flask (If deploying)
Random Points of Failure can occur in the pipeline. The goal of the workflow orchest…
Docker-compose
View in #show-us-what-you-got on Slack
[Paco_Ibañez] @Paco_Ibañez: Hello! I have being exploring Orion this week and I have put together a docker-compose to run Prefect 2.0 locally. I have tested it on mac and linux. The script runs prefect server, postgres, a docker agent, and minio to store flows. The start script configures a work queue and an example flow is included. Sharing in case someone finds it useful: https://github.com/fraibacas/prefect-orion
GitHub: GitHub - fraibacas/prefect-ori…
I’m trying to spin up a local prefect orion server for testing/development.
here’s the compose file I’ve come up with:
version: "3.7"
services:
postgres-orion:
image: "postgres:11"
environment:
POSTGRES_USER: ${POSTGRES_USER}
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
POSTGRES_DB: ${POSTGRES_DB}
volumes:
- postgres-orion-volume:/var/lib/postgresql/data
networks:
- prefect-orion
restart: "always"
command:
- "postgres"
- "-c"
…
Helm chart
opened 03:10PM - 16 Mar 22 UTC
from:slack
v2
status:roadmap
## Opened from the [Prefect Public Slack Community](https://prefect.io/slack)
*… *bradley**: Hi Prefect Experts - Is it possible to not display generated tasks like `List` and `Dict` from the visual schematic displayed in the UI?
**bradley**:
**bradley**: I've tried to search the docs, but didn't really see anything.
**kevin701**: <https://discourse.prefect.io/t/how-to-avoid-tasks-such-as-list-tuple-or-dict-in-a-flow-structure/318|This> discourse page addresses that, but the short answer is not really unless you modify your tasks
**anna**: check out <https://discourse.prefect.io/t/how-to-avoid-tasks-such-as-list-tuple-or-dict-in-a-flow-structure/318>
**bradley**: You guys are awesome!
**anna**: btw those tasks are gone in Prefect 2.0 because there is no more DAG :slightly_smiling_face:
**bradley**: Prefect 2.0 is Orion, correct?
**anna**: sort of, 2.0 is more than just Orion - check out <https://discourse.prefect.io/t/the-second-launch-week-initiative-announcing-prefect-2-0/499>
**bradley**: If you were building out a new Prefect deployment would you build it on 2.0 or stay on 1.0 for now?
**anna**: It's totally up to you, you can certainly start directly with Orion, and in fact, this would be quite beneficial for us since you could provide us (incredibly valuable) feedback as an early adopter. But 2.0 is not yet fully production-ready and it may take a couple of months to bring 2.0 to that stage.
I would honestly encourage you to still explore Prefect 1.x, sign up for a free tier of Prefect Cloud and put some pipelines to production already to get a feeling of how it is to work with Prefect in general. The user experience shouldn't change - you can still run your workflows locally first by adding a couple of decorators to your workflow. Then you can build/deploy your flows - in 1.0 this step is called flow registration, in 2.0 it's creating a deployment.
Having used and understood 1.0 first will likely make you appreciate many Orion features more :slightly_smiling_face:
**scottjmccallen**: This is awesome. Thank you so much for the info. We'd definitely be interested in trying to run Prefect 2.0 - We are deploying Prefect 1.0 on Kubernetes using the Helm chart from the PrefectHQ project. <https://github.com/PrefectHQ/server/tree/master/helm/prefect-server>
Is there an updated Helm chart for Prefect 2.0?
**bradley**: Scott is one of my co-workers for context.
**anna**: I wouldn't expect a Helm chart for Prefect 2.0 any time soon due to other much higher priorities, but let me open an issue with your feature request, for now, to have it in our backlog.
For now, we do have a very simple way to deploy all Orion components to Kubernetes described here: <https://orion-docs.prefect.io/tutorials/kubernetes-flow-runner/>
<@ULVA73B9P> open "Orion: add a Helm chart for easier deployment of Prefect 2.0"
Original thread can be found [here](https://prefect-community.slack.com/archives/CL09KU1K7/p1647441819228689?thread_ts=1647441819.228689&cid=CL09KU1K7).
Cloud 2.0 as an alternative
Remember that there is an always-free tier - you can sign up and test out the platform. No trial accounts and no credit card are required.
Related resources about getting started with Prefect 2.0
Prefect 2.0 was designed to be a lightweight product that can simultaneously be used as an ephemeral orchestration API for local development (with embedded SQLite database), as well as a product that can scale to massive workloads when running it in a distributed fashion e.g. with Cloud 2.0.
Docs
This documentation should serve as your main source about the Prefect orchestration layer explaining all components and how they work together:
Consider everything else in this Discours…
Short answer: we highly encourage all new Prefect users to start with Prefect 2.0 and Cloud 2.0 .
At the time of writing, both Prefect 2.0 and Cloud 2.0 will remain under active development in beta status until approximately July 2022 , with a number of major features yet to come.
The target for the General Availability of Cloud 2.0 and stable release of Prefect 2.0 is July 2022.
Can I use Cloud 2.0 in production already?
Cloud 2.0 is a beta product, with beta product guarantees. When decid…
1 Like