How does the "prefect deployment build" and "apply" CLI work and how can you customize it?

Archive
, ,
1

We assume that an agent is spun up with a work queue name dev in all the examples below.:

prefect agent start -q dev

What does prefect deployment build do?

The build step is needed to:

  1. Access the flow object and extract information from it, such as:
  • parameter schema
  • parameter defaults
  • flow entrypoint
  1. Upload the entire workflow directory to a configurable storage location (the default is a local file system that does not perform any movement of files - instead, it only stores the location to your flow in the manifest’s entrypoint). This upload allows for relative imports as well as static configuration files to be preserved.

CLI flags available on the prefect deployment build command

There are several CLI flag shortcuts, but you can always use the equivalent full identifiers:

  • Organizational concerns

    • -n stands for --name - the name to give this deployment.

    • -t stands for --tag - one or more optional tags to apply to the deployment to organize your deployments and flow runs based on your projects.

    • -v stands for --version - version of this deployment; you could implement custom versioning or (recommended) use the Git commit SHA set from your CI/CD.

    • -o stands for --output - the optional location for the YAML manifest.

  • Infrastructure concerns

    • If you don’t provide any infrastructure configuration, Prefect will assume that you want to run your flow code as a local process.

    • -infra stands for --infrastructure - the optional flag to specify the infrastructure type. This flag is helpful to get started because this way Prefect will create the infrastructure block for you.

      • You can use -infra to generate a default infrastructure scaffold in YAML and manually override the values in YAML if needed.

      • For more customized setup :technologist: and for production workloads, we recommend using -ib, i.e. infrastructure blocks with optional per-deployment --override arguments.

    • -ib stands for --infrastructure-block - name of an already created infrastructure block in the format: block_type/block_name

      • -ib process/dev - for a Process block with name dev

      • -ib docker-container/dev - for a DockerContainer block with name dev

      • -ib kubernetes-job/dev - for a KubernetesJob block with name dev

      • :muscle: infrastructure blocks are powerful. They allow you to have a parent infrastructure block defining a generic structure you can follow for most deployments. This pattern prevents boilerplate code and allows you to reuse the same generic configuration. :computer:

      • :gear: It makes your DevOps process easy to change (e.g. changing a Kubernetes namespace or the default image), as you can do all that from a single place. If some deployments require a slight deviation from the standard infrastructure configuration (e.g. a different Docker image :fish: with custom package dependencies) you can override those settings from the CLI without having to modify your code.

    • --override - a flag for infrastructure overrides → you can pass multiple override flags to a deployment build command to override multiple arguments. Examples:

      • --override env.PREFECT_LOGGING_LEVEL=DEBUG will override or add the environment variable to set log level to DEBUG

      • --override image=prefecthq/prefect:2-python3.9 will override the Docker image set on the block to this specific image (this is the image using the latest Prefect 2.0 version and Python 3.9)

  • Storage concerns

    • if you don’t provide any storage block configuration, Prefect will assume that the files are in the computer, i.e. that the flow code and related module dependencies exist on the local machine from which you create and run your deployment

    • -sb stands for --storage-block - name of an already created storage block in the format: block_type/block_name

      • -sb s3/dev - for an S3 block with name dev
      • -sb gcs/dev - for a GCS block with name dev
      • -sb azure/dev - for an Azure block with name dev

  • Scheduling concerns :clock10:

    • --cron - a CRON string that will be used to set a CronSchedule on the deployment, e.g. --cron "*/1 * * * *" to schedule flow runs from that deployment every minute.

    • --interval - an integer specifying an interval (in seconds) that will be used to set an IntervalSchedule on the deployment, e.g. --interval 60 to schedule flow runs from that deployment every minute.

    • --rrule - a string with an rrule allowing you to set highly customized schedules on your deployment, e.g. to schedule flow runs from that deployment every business hour, use:

      prefect deployment build ... --rrule \
'FREQ=HOURLY;BYDAY=MO,TU,WE,TH,FR;BYHOUR=9,10,11,12,13,14,15,16,17'

  • -a stands for --apply - allows you to directly apply the manifest, effectively creating deployment with a single command, while still retaining the YAML manifest that you can use as a build artifact.

What does prefect deployment apply do?

If you haven’t applied the deployment already during build, you can use the YAML file generated as part of the build step and leverage the command below to send the newly created or updated deployment information to the API backend.

prefect deployment apply your.yaml