How can I organize my flows based on my business logic or team structure?

Prefect 2.0

Orion provides a flexible and highly customizable way of organizing your flows, tasks, and deployments through a concept of tags. You can save dashboards that only include flows, tasks, flow runs, task runs, and deployments with certain tags.

To set tags, you can leverage the TagsContext context manager:

  • setting tags for a task:
from prefect import flow, task, tags

@task
def my_task():
    pass

@flow
def my_flow():
    with tags("project_name", "team_name"):
        my_task()  # has tags: project_name, team_name

Alternatively, you can also leverage the tags argument in a @task decorator:

@task(tags=["project_name", "team_name"])
  • setting tags for a flow:
from prefect import flow, task, tags

@task
def my_task():
    pass

@flow
def my_flow():
    my_task()

with tags("project_name", "team_name"):
    my_flow()  # has tags: project_name, team_name
  • setting tags for a deployment:
from prefect.deployments import DeploymentSpec

DeploymentSpec(
    flow_location="flow.py",
    name="flow-deployment",
    tags=["production", "data-engineering-team", "project-x"]
)

Prefect 1.0

You can deploy your flows to various projects. Each project can refer to a different topic, team, environment, or business unit in your organization. You can create a project either from the UI or using the CLI:

prefect create project "project_name"
1 Like