Prefect 1.0 -> Prefect 2.0 Docker_run to DockerContainer

Archive
, , ,
1

Summary

This article is for those currently using Prefect’s 1.0 Docker flowrunner or plan on using docker to run your flows. Running flows inside a docker container has stayed mostly the same with differences in where we configure our runner and what we are able to configure in it.

Video

What’s changing?

Where as before we configured our flow runner inside the run_config object of a flow i.e:

    with Flow("example", run_config=DockerRun(image="registry/example")) as flow:
    ...

They are now set inside an infrastructure block. Infra blocks are the equivalent of the run_config object from 1.0. In order to use an infrastructure block, it must be inside a deployment. An easy way to initialize an infrastructure block inside a deployment is to set the -i or -infra flag when building a deployment, i.e.

prefect deployment build ./2.0.py:flow -n dev -i docker-container

Which creates a deployment.yml with the following in it:

infrastructure:
  type: docker-container
  env: {}
  labels: {}
  name: null
  command:
  - python
  - -m
  - prefect.engine
  image: prefecthq/prefect:2.0.0-python3.9
  image_pull_policy: null
  networks: []
  network_mode: null
  auto_remove: false
  volumes: []
  stream_output: true

This will create an infra block of type Docker Container in which you can configure all the settings from 1.0s docker flow runner. Finally, for Orion there is one Agent, so there is no need to set up your own Docker agent. This means that streaming of flow logs, network, and volumes which were configured on the agent can now be configured on the flow runner.

3 Likes