How to deploy Prefect 2.0 flows to run as a local Process, Docker container or a Kubernetes job

This page will walk you through creating deployments in Prefect 2.0.

1. Create a project in your favorite code IDE

You may have a project with a similar structure:

Example project structure
├── Dockerfile
├── dbt_models
├── tests
├── deployment.yaml (generated by Prefect)
├── flows
│   ├── healthcheck.py
│   ├── hello.py
│   └── log_levels_flow.py
├── healthcheck-manifest.json (generated by Prefect)
├── prefect_dataops
│   ├── postgres_utils.py
│   └── snowflake_utils.py
└── requirements.txt.py

Your flows might be located in the flows directory, and the shared code utilities may live within the prefect_dataops module, as in the above example.

2. Write at least one flow

Example flow called healthcheck.py
import prefect
from prefect import task, flow
from prefect import get_run_logger


@task
def say_hi():
    logger = get_run_logger()
    logger.info("Hello from the Prefect 2.0 flow! 👋")


@task
def log_platform_info():
    import platform
    import sys
    from prefect.orion.api.server import ORION_API_VERSION

    logger = get_run_logger()
    logger.info("Host's network name = %s", platform.node())
    logger.info("Python version = %s", platform.python_version())
    logger.info("Platform information (instance type) = %s ", platform.platform())
    logger.info("OS/Arch = %s/%s", sys.platform, platform.machine())
    logger.info("Prefect Version = %s 🚀", prefect.__version__)
    logger.info("Prefect API Version = %s", ORION_API_VERSION)


@flow
def healthcheck():
    hi = say_hi()
    log_platform_info(wait_for=[hi])


if __name__ == "__main__":
    healthcheck()

3. Deploy your flow

Manifest generation using the prefect deployment build CLI

First, run the following command in your CLI to familiarize yourself with the available commands.

 prefect deployment build --help
It shows the usage and available options

                                                                                                                                                                                                     
 Usage: prefect deployment build [OPTIONS] ENTRYPOINT                                                                                                                                                
                                                                                                                                                                                                     
 Generate a deployment YAML from /path/to/file.py:flow_function                                                                                                                                      
                                                                                                                                                                                                     
╭─ Arguments ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ *    entrypoint      TEXT  The path to a flow entrypoint, in the form of `./path/to/file.py:flow_func_name` [default: None] [required]                                                            │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Options ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ --name           -n       TEXT                                                              The name to give the deployment. [default: None]                                                      │
│ --version        -v       TEXT                                                              A version to give the deployment. [default: None]                                                     │
│ --tag            -t       TEXT                                                              One or more optional tags to apply to the deployment. Note: tags are used only for organizational     │
│                                                                                             purposes. For delegating work to agents, use the --work-queue flag.                                   │
│                                                                                             [default: None]                                                                                       │
│ --work-queue     -q       TEXT                                                              The work queue that will handle this deployment's runs. It will be created if it doesn't already      │
│                                                                                             exist. Defaults to `None`. Note that if a work queue is not set, work will not be scheduled.          │
│                                                                                             [default: None]                                                                                       │
│ --limit          -l       INTEGER                                                           Sets the concurrency limit on the work queue that handles this deployment's runs [default: None]      │
│ --infra          -i       [docker-container|kubernetes-job|process|cloud-run-job|ecs-task]  The infrastructure type to use, prepopulated with defaults. [default: None]                           │
│ --infra-block    -ib      TEXT                                                              The slug of the infrastructure block to use as a template. [default: None]                            │
│ --override                TEXT                                                              One or more optional infrastructure overrides provided as a dot delimited path, e.g.,                 │
│                                                                                             `env.env_key=env_value`                                                                               │
│                                                                                             [default: None]                                                                                       │
│ --storage-block  -sb      TEXT                                                              The slug of a remote storage block. Use the syntax: 'block_type/block_name', where block_type must be │
│                                                                                             one of 'github', 's3', 'gcs', 'azure', 'smb'                                                          │
│                                                                                             [default: None]                                                                                       │
│ --skip-upload                                                                               A flag that, when provided, skips uploading this deployment's files to remote storage.                │
│ --cron                    TEXT                                                              A cron string that will be used to set a CronSchedule on the deployment. [default: None]              │
│ --interval                INTEGER                                                           An integer specifying an interval (in seconds) that will be used to set an IntervalSchedule on the    │
│                                                                                             deployment.                                                                                           │
│                                                                                             [default: None]                                                                                       │
│ --anchor-date             TEXT                                                              The anchor date for an interval schedule [default: None]                                              │
│ --rrule                   TEXT                                                              An RRule that will be used to set an RRuleSchedule on the deployment. [default: None]                 │
│ --timezone                TEXT                                                              Deployment schedule timezone string e.g. 'America/New_York' [default: None]                           │
│ --path                    TEXT                                                              An optional path to specify a subdirectory of remote storage to upload to, or to point to a           │
│                                                                                             subdirectory of a locally stored flow.                                                                │
│                                                                                             [default: None]                                                                                       │
│ --output         -o       TEXT                                                              An optional filename to write the deployment file to. [default: None]                                 │
│ --apply          -a                                                                         An optional flag to automatically register the resulting deployment with the API.                     │
│ --param                   TEXT                                                              An optional parameter override, values are parsed as JSON strings e.g. --param question=ultimate      │
│                                                                                             --param answer=42                                                                                     │
│                                                                                             [default: None]                                                                                       │
│ --params                  TEXT                                                              An optional parameter override in a JSON string format e.g. --params='{"question": "ultimate",        │
│                                                                                             "answer": 42}'                                                                                        │
│                                                                                             [default: None]                                                                                       │
│ --help                                                                                      Show this message and exit.                                                                           │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯


a) Generate a simple manifest file for a local file system and a local Process block

When building a deployment, you can package your project’s code (both flow code and your flow utility modules!) into your desired location using both local or a remote file system. This command:

  1. builds the manifest file for your deployment, and
  2. uploads your project files to the desired file system location.

For local development, start with the following command:

prefect deployment build flows/healthcheck.py:healthcheck -n dev -q dev

The above command will by default create two anonymous blocks:

  1. Local file system block
  2. Local Process block

b) Generate a manifest file for S3 storage and a KubernetesJob infrastructure block

Go to the UI and create an S3 and KubernetesJob blocks. Then, run the following command:

prefect deployment build flows/healthcheck.py:healthcheck -n dev -q dev -ib k8s/dev -sb s3/dev

c) Generate a manifest file for S3 storage and a DockerContainer infrastructure

Go to the UI and create an S3 and DockerContainer blocks. Then, run the following command to build and apply your deployment:

prefect deployment build flows/healthcheck.py:healthcheck -n dev -q dev -ib docker/dev -sb s3/dev -a

4. Start your agent and trigger a deployment

You can now trigger a flow run from that deployment, either from the UI or from the CLI:

prefect deployment run healthcheck/dev

When you now start your agent:

prefect agent start -q dev

The agent should pick up the run (due to the same tag dev)! :tada:

2 Likes