How to deploy Prefect 2.0 flows to run as a local Process, Docker container or a Kubernetes job

This page will walk you through creating deployments in Prefect 2.0.

1. Create a project in your favorite code IDE

You may have a project with a similar structure:

Example project structure
├── Dockerfile
├── dbt_models
├── tests
├── deployment.yaml (generated by Prefect)
├── flows
│   ├── healthcheck.py
│   ├── hello.py
│   └── log_levels_flow.py
├── healthcheck-manifest.json (generated by Prefect)
├── prefect_dataops
│   ├── postgres_utils.py
│   └── snowflake_utils.py
└── requirements.txt.py

Your flows might be located in the flows directory, and the shared code utilities may live within the prefect_dataops module, as in the above example.

2. Write at least one flow

Example flow called healthcheck.py
import prefect
from prefect import task, flow
from prefect import get_run_logger


@task
def say_hi():
    logger = get_run_logger()
    logger.info("Hello from the Prefect 2.0 flow! 👋")


@task
def log_platform_info():
    import platform
    import sys
    from prefect.orion.api.server import ORION_API_VERSION

    logger = get_run_logger()
    logger.info("Host's network name = %s", platform.node())
    logger.info("Python version = %s", platform.python_version())
    logger.info("Platform information (instance type) = %s ", platform.platform())
    logger.info("OS/Arch = %s/%s", sys.platform, platform.machine())
    logger.info("Prefect Version = %s 🚀", prefect.__version__)
    logger.info("Prefect API Version = %s", ORION_API_VERSION)


@flow
def healthcheck():
    hi = say_hi()
    log_platform_info(wait_for=[hi])


if __name__ == "__main__":
    healthcheck()

3. Deploy your flow

3.1 Manifest generation using the prefect deployment build CLI

First, run the following command in your CLI to familiarize yourself with the available commands.

 prefect deployment build --help
It shows the usage and available options
Usage: prefect deployment build [OPTIONS] PATH

Arguments:
  PATH  [required]

Options:
  --manifest-only                 Generate the manifest file only.
  -n, --name TEXT                 The name of the deployment.
  -t, --tag TEXT                  One or more optional tags to apply to the
                                  deployment.
  -i, --infra [kubernetes-job|process|docker-container]
                                  The infrastructure type to use.  [default:
                                  process]
  -ib, --infra-block TEXT         The type/name of the infrastructure block to
                                  use.
  -sb, --storage-block TEXT       The type/name of the storage block to use.
  --help                          Show this message and exit.

a) Generate a simple manifest file for a local file system and a local Process block

When building a deployment, you can package your project’s code (both flow code and your flow utility modules!) into your desired location using both local or a remote file system. This command:

  1. builds the manifest file for your deployment, and
  2. uploads your project files to the desired file system location.

For local development, start with the following command:

prefect deployment build flows/healthcheck.py:healthcheck --name dev --tag dev

The above command will by default create two anonymous blocks:

  1. Local file system block
  2. Local Process block

b) Generate a manifest file for S3 storage and a KubernetesJob infrastructure block

Go to the UI and create an S3 and KubernetesJob blocks. Then, run the following command:

prefect deployment build flows/healthcheck.py:healthcheck --name dev --tag dev --infra k8s/dev --storage-block s3/dev

c) Generate a manifest file for S3 storage and a DockerContainer infrastructure

Go to the UI and create an S3 and DockerContainer blocks. Then, run the following command:

prefect deployment build flows/healthcheck.py:healthcheck --name dev --tag dev --infra docker/dev --storage-block s3/dev

3.2 Apply a deployment

prefect deployment apply deployment.yaml

4. Start your agent and trigger a deployment

You can now trigger a flow run from that deployment, either from the UI or from the CLI:

prefect deployment run healthcheck/dev

When you now start your agent:

prefect agent start --tag dev

The agent should pick uo the run (due to the same tag dev)! :tada:

1 Like