How can I structure my flows (both in the repository and the UI) and set project-based permissions (RBAC)?

View in #prefect-community on Slack

Андрій_Демиденко @Андрій_Демиденко: Hi everybody,

It seems like our team is going to have tons of flows. Is there any ideas how to create a hierarchy of the flows or to arrange them somehow? Thank you.

Anna_Geller @Anna_Geller: Hard to tell without knowing your use case. It depends a lot on how you would like to structure your flows. E.g. if you would build it for a data warehousing project, you could structure your flows according to your data warehouse schemas:

dwh_project
|-- business_vault
|-- data_mart
|-- data_services
|-- monitoring
|-- operational
|-- raw_vault
`-- stage
    |-- crm
    |-- erp
    |-- flat_file
    |-- google_analytics
    |-- price_api
    |-- shipment_api
    |-- shop_system
    `-- wms

I used a similar structure as the above in my last DWH project.

But if you need a repo structure for multiple teams, you could build a structure around your teams:

teams_flows
|-- analytics_engineering
|-- automations
|-- data_engineering
|   |-- data_ingestion_flows
|   `-- data_replication_flows
|-- data_science
|   |-- batch_prediction_flows
|   |-- preprocessing_flows
|   `-- training_flows
`-- platform

It’s super hard to give any recommendations without knowing your use case
and you could also opt for a single repo per project - e.g. having all ML training flows within their own repository and have a structure based on their business logic

Андрій_Демиденко @Андрій_Демиденко: I’m looking for a tool how to separate some flows into kind of a group inside a project(it seems like we are going to have 1 repo but still tons of flows)

Anna_Geller @Anna_Geller: can you explain your use case? there is no tool that will build a structure for your repo; it’s an important decision you need to make based on your use case and the needs of your organization, not based on some tool

Андрій_Демиденко @Андрій_Демиденко: For example we have a project “working_with_client” but we would also want to have some folder “client_name” containing a group of flows related to the specific client, which we can see in the UI and if it’s possible to give some partial permissions to this (but it’s optional)

Anna_Geller @Anna_Geller: I see, so it’s not about a structure of a repo but rather a way to provide permissions based on a project, correct?

are you asking for Prefect 2.0 or 1.0?

Андрій_Демиденко @Андрій_Демиденко: 1.0

Anna_Geller @Anna_Geller: can you answer all my questions rather than only the last one? :slightly_smiling_face:

Андрій_Демиденко @Андрій_Демиденко: Yes the first is correct (I didn’t notices the question :slightly_smiling_face: )

Anna_Geller @Anna_Geller: thanks a lot! are you on Prefect Cloud or on Prefect Server?

Андрій_Демиденко @Андрій_Демиденко: Cloud

Anna_Geller @Anna_Geller: There’s a way of providing permissions based on a project, but I believe this is only for Enterprise plans. Afaik such granular permissions are not available on a self-serve Cloud tier. The only restriction you could impose on a self-serve plan is providing Read-only access - perhaps that’s enough for your use case?

Otherwise, you can send an email to sales@prefect.io to get more detailed information