How can I use Prefect with OpenShift?

Archive
, , , , , ,
1

View in #prefect-community on Slack

Gowtham_Manne @Gowtham_Manne: Hi All, General Question - Can we deploy prefect on open shift ? Is there a doc for this ? Appreciate your help!

Kevin_Kho @Kevin_Kho: I am not sure. No content but you might be able to use the helm chart. Any reason why you can’t use Cloud though? It will be a lot easier and there are 20k task runs, which is more than enough to get started with

Gowtham_Manne @Gowtham_Manne: Thanks for reply! We have our own cloud, and cannot install docker-compose in VMs. So wanted to check if we can deploy on OpenShift

@Anna_Geller: are you asking for Prefect 2.0 or 1.0? you can always use OpenShift as your execution layer and leverage Prefect Cloud as your orchestration layer

Prefect Community: Should I start with Prefect 2.0 (Orion), skipping Prefect 1.0?

if you have privacy concerns regarding Cloud-based SaaS, check this page about Prefect’s Hybrid Execution Model

Hybrid Model

Gowtham_Manne @Gowtham_Manne: Hi Anna, Thanks for the links above. We are just planning on migrating from dagster to prefect, I am new to prefect with basic knowledge. From Hybrid Model step 3 is what we are unable to do. we cannot install docker on the VM. So is there any otherway of deploying prefect on openshift ?
or containerize prefect for openshift ?

@Anna_Geller: there is a definitely a way to use Prefect on your infrastructure, but it’s hard to say immediately how this may look like

Did you decide if you:

  1. would like to use Prefect Cloud vs. do you want to use only OSS version?
  2. want to use Prefect 2.0 vs. use only Prefect 1.0 for now?
    For OpenShift, you can leverage Kubernetes integration to deploy flow runs

Gowtham_Manne @Gowtham_Manne: Is this Kubernetes with OSS version ? https://docs.prefect.io/orchestration/agents/kubernetes.html#requirements

Kubernetes Agent | Prefect Docs

Kevin_Kho @Kevin_Kho: Yeah

Gowtham_Manne @Gowtham_Manne: This is what I tried https://docs.prefect.io/orchestration/server/deploy-local.html and failed because of “pip install docker-compose”

Deploying to a single node | Prefect Docs

Kevin_Kho @Kevin_Kho: prefect server start is using docker compose, not Kubernetes. For Kubernetes, you need to use helm

Gowtham_Manne @Gowtham_Manne: Thanks Kevin!

davzucky @davzucky: Hi @Gowtham_Manne We are running all the Prefect infrastructure including Perfect Server on OC. All the containers are rootless which is a requirement for OC.
The rest is a basic Kubernetes setup. Just ensure your RBAC is right and agents have the right role attached to them.

@Anna_Geller: for the Helm chart, this README is a great description showing how to use it

GitHub: server/helm/prefect-server at master · PrefectHQ/server