Can I specify deployments using Python code rather than CLI?

Show and Tell
,
1

Yes, that’s possible by using the prefect.deployments.Deployment class.

from flows.healthcheck import healthcheck
from prefect.deployments import Deployment
from prefect.filesystems import S3

deployment = Deployment.build_from_flow(
    flow=healthcheck,
    name="pythonic_s3",
    description="hi!",
    version="v1",
    work_queue_name="dev",
    tags=["myproject"],
    storage=S3.load("dev"),
    infra_overrides=dict(env={"PREFECT_LOGGING_LEVEL": "DEBUG"}), 
    output="your.yaml"
)
if __name__ == "__main__":
    deployment.apply()

Does this Python code still package my custom modules and configuration files?

Yes.

Does this mean that Python code represents the source of truth of how my deployment look like?

No. This is just one method to send deployment information to the backend. The API still remains the single source of truth and unless you would always trigger deployments from CI/CD triggered from this Python code, this could get out of sync. Think of both CLI and Python as convenience methods to properly construct the payload for an API call that will eventually create or modify a deployment.

Are there any limitations to that Python Deployment UX?

Very few, but there are some limitations of that approach:

  • it’s bit more difficult to integrate with CI/CD - you can’t easily pass your Git commit via CLI (unless you add argparse on top and effectively transform this Python process into a CLI process for invocation)

  • you have to write and maintain extra code which could be avoided when using CLI

  • it may not be immediately clear which files get packaged alongside your flow code (think of import paths) depending on directory from which you execute that Python script

    • in CLI approach, you specify the path explicitly or as an absolute path in relation to the root directory which separates between path (your root project directory), and the entrypoint (e.g. flow.py:flow_func), while the Python UX combines this information within the entrypoint and keeps the path argument as None

  • many parts of the Deployment object that need to be sent to the API can be abstracted away if you package that e.g. into a function; this allows possibly more convenience at the cost of hiding away some information and making it less explicit — as always, consider the tradeoffs and choose based on your use case.

3 Likes
2

Here’s a guide to Deploying Prefect Pipelines with Python

1 Like