Overview of Infrastructure Blocks in CLI-based deployments

Archive
, , ,
1

Video Explanation

Youtube Link

take this lil flow for example

from prefect import flow, get_run_logger, task
from prefect.context import get_run_context

@task
def bar():
    logger = get_run_logger()
    task_run_name = get_run_context().task_run.name
    logger.info(f"👋 from {task_run_name}!")
    
@flow
def foo():
    # This can be as complicated as you like >:)
    bar()
    
if __name__ == "__main__":
    foo()

where can I deploy and run a flow like this?

Locally as a process, on Docker, on Kubernetes, (whether Linux, Mac, Windows) and more to come!

Wherever we are, all we need is:

  • an agent process running somewhere with Prefect installed and an API url / key set
  • a runtime environment that our agent can send flow runs to, with python dependencies installed
  • some Prefect Infrastructure blocks set in our Prefect Cloud workspace
  • a Deployment that refers to these infrastructure blocks to decide where to submit flow runs

how do I create Blocks?

Either in the UI or in python like:

from prefect.infrastructure import KubernetesJob, KubernetesImagePullPolicy

k8s_job = KubernetesJob(
    namespace="dev",
    image="prefecthq/prefect:2-python3.9",
    image_pull_policy=KubernetesImagePullPolicy.IF_NOT_PRESENT,
)

k8s_job.save("test-k8s-job")

how do I create a deployment? where are its flow runs executed?

As of writing, it’s easy to create deployments based on three main types of Infrastructure block types:

  • as a local Process on a machine of your choosing
❯ prefect deployment build example.py:foo \
    --name my-process-deployment \
    -ib process/test-process \
    --apply

❯ prefect deployment build example.py:foo \
    --name my-process-deployment \
    -ib docker-container/test-container \
    -sb s3/flow-script-storage/docker-example \
    --apply

❯ prefect deployment build example.py:foo \
    --name my-process-deployment \
    -ib kubernetes-job/test-k8s-job \
    -sb s3/flow-script-storage/k8s-example \
    --apply

how can I run my flows?

3 Likes