This page will walk you through creating deployments in Prefect 2.0.
1. Create a project in your favorite code IDE
You may have a project with a similar structure:
Example project structure
├── Dockerfile
├── dbt_models
├── tests
├── deployment.yaml (generated by Prefect)
├── flows
│ ├── healthcheck.py
│ ├── hello.py
│ └── log_levels_flow.py
├── healthcheck-manifest.json (generated by Prefect)
├── prefect_dataops
│ ├── postgres_utils.py
│ └── snowflake_utils.py
└── requirements.txt.py
Your flows might be located in the flows directory, and the shared code utilities may live within the prefect_dataops module, as in the above example.
2. Write at least one flow
Example flow called healthcheck.py
import prefect
from prefect import task, flow
from prefect import get_run_logger
@task
def say_hi():
logger = get_run_logger()
logger.info("Hello from the Prefect 2.0 flow! 👋")
@task
def log_platform_info():
import platform
import sys
from prefect.orion.api.server import ORION_API_VERSION
logger = get_run_logger()
logger.info("Host's network name = %s", platform.node())
logger.info("Python version = %s", platform.python_version())
logger.info("Platform information (instance type) = %s ", platform.platform())
logger.info("OS/Arch = %s/%s", sys.platform, platform.machine())
logger.info("Prefect Version = %s 🚀", prefect.__version__)
logger.info("Prefect API Version = %s", ORION_API_VERSION)
@flow
def healthcheck():
hi = say_hi()
log_platform_info(wait_for=[hi])
if __name__ == "__main__":
healthcheck()
3. Deploy your flow
Manifest generation using the prefect deployment build CLI
First, run the following command in your CLI to familiarize yourself with the available commands.
prefect deployment build --help
It shows the usage and available options
Usage: prefect deployment build [OPTIONS] ENTRYPOINT
Generate a deployment YAML from /path/to/file.py:flow_function
╭─ Arguments ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ * entrypoint TEXT The path to a flow entrypoint, in the form of `./path/to/file.py:flow_func_name` [default: None] [required] │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Options ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ --name -n TEXT The name to give the deployment. [default: None] │
│ --version -v TEXT A version to give the deployment. [default: None] │
│ --tag -t TEXT One or more optional tags to apply to the deployment. Note: tags are used only for organizational │
│ purposes. For delegating work to agents, use the --work-queue flag. │
│ [default: None] │
│ --work-queue -q TEXT The work queue that will handle this deployment's runs. It will be created if it doesn't already │
│ exist. Defaults to `None`. Note that if a work queue is not set, work will not be scheduled. │
│ [default: None] │
│ --limit -l INTEGER Sets the concurrency limit on the work queue that handles this deployment's runs [default: None] │
│ --infra -i [docker-container|kubernetes-job|process|cloud-run-job|ecs-task] The infrastructure type to use, prepopulated with defaults. [default: None] │
│ --infra-block -ib TEXT The slug of the infrastructure block to use as a template. [default: None] │
│ --override TEXT One or more optional infrastructure overrides provided as a dot delimited path, e.g., │
│ `env.env_key=env_value` │
│ [default: None] │
│ --storage-block -sb TEXT The slug of a remote storage block. Use the syntax: 'block_type/block_name', where block_type must be │
│ one of 'github', 's3', 'gcs', 'azure', 'smb' │
│ [default: None] │
│ --skip-upload A flag that, when provided, skips uploading this deployment's files to remote storage. │
│ --cron TEXT A cron string that will be used to set a CronSchedule on the deployment. [default: None] │
│ --interval INTEGER An integer specifying an interval (in seconds) that will be used to set an IntervalSchedule on the │
│ deployment. │
│ [default: None] │
│ --anchor-date TEXT The anchor date for an interval schedule [default: None] │
│ --rrule TEXT An RRule that will be used to set an RRuleSchedule on the deployment. [default: None] │
│ --timezone TEXT Deployment schedule timezone string e.g. 'America/New_York' [default: None] │
│ --path TEXT An optional path to specify a subdirectory of remote storage to upload to, or to point to a │
│ subdirectory of a locally stored flow. │
│ [default: None] │
│ --output -o TEXT An optional filename to write the deployment file to. [default: None] │
│ --apply -a An optional flag to automatically register the resulting deployment with the API. │
│ --param TEXT An optional parameter override, values are parsed as JSON strings e.g. --param question=ultimate │
│ --param answer=42 │
│ [default: None] │
│ --params TEXT An optional parameter override in a JSON string format e.g. --params='{"question": "ultimate", │
│ "answer": 42}' │
│ [default: None] │
│ --help Show this message and exit. │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
a) Generate a simple manifest file for a local file system and a local Process block
When building a deployment, you can package your project’s code (both flow code and your flow utility modules!) into your desired location using both local or a remote file system. This command:
- builds the manifest file for your deployment, and
- uploads your project files to the desired file system location.
For local development, start with the following command:
prefect deployment build flows/healthcheck.py:healthcheck -n dev -q dev
The above command will by default create two anonymous blocks:
- Local file system block
- Local
Processblock
b) Generate a manifest file for S3 storage and a KubernetesJob infrastructure block
Go to the UI and create an S3 and KubernetesJob blocks. Then, run the following command:
prefect deployment build flows/healthcheck.py:healthcheck -n dev -q dev -ib k8s/dev -sb s3/dev
c) Generate a manifest file for S3 storage and a DockerContainer infrastructure
Go to the UI and create an S3 and DockerContainer blocks. Then, run the following command to build and apply your deployment:
prefect deployment build flows/healthcheck.py:healthcheck -n dev -q dev -ib docker/dev -sb s3/dev -a
4. Start your agent and trigger a deployment
You can now trigger a flow run from that deployment, either from the UI or from the CLI:
prefect deployment run healthcheck/dev
When you now start your agent:
prefect agent start -q dev
The agent should pick up the run (due to the same tag dev)! 