Can I run a Docker agent in a container?

In general, the Docker agent is supposed to run in a local process (rather than in a docker container), and this local process is a layer between Prefect backend and a Docker Daemon.

This agent polls the API for new flow runs, and if there are new scheduled runs, it then creates new flow runs and deploys those as Docker containers on the same machine as the agent.

When the Docker agent is running within a container itself (rather than a local process), your flow runs end up deployed as containers, but not as individual containers, but rather within the agent container. You effectively have a single agent container spinning up new containers within itself (docker in docker), which may have many unintended consequences such as issues with scale and resource utilization.

If you want more environment isolation for this agent process, you can run it within a virtual environment.

And if you have a strict requirement that every process must run in a container, consider using the KubernetesAgent instead.

Related Slack discussion

View in #prefect-community on Slack

Elio @Elio: Hi, we are using prefect Local Agent wrapped in a docker container, we would like to switch to Docker Agent. Does someone knows if it’s possible to setup a Docker Agent with Docker IN Docker (dind) ? Thanks !

Kevin_Kho @Kevin_Kho: You could by mounting the sock:

prefect agent docker start --volume //var/run/docker.sock:/var/run/docker.sock

but this is normally not a good pattern. Why do you need to do this?

Elio @Elio: Because I don’t want to install a python package on my environment and I prefer to run it in a docker container

Anna_Geller @Anna_Geller: I guess what Kevin meant is that docker-in-docker is sometimes hard to manage. Are you on Prefect Cloud or Server? on Prefect 1.0 or 2.0?

if the problem you try to solve is managing code dependencies across flow runs, then it’s enough if flow runs are packaged as containers, the agent itself can run as a subprocess. The Prefect agent is a lightweight process that doesn’t require any custom dependencies other than Prefect and it can deploy flow runs with custom dependencies as docker containers

Elio @Elio: We are using Prefect Cloud, we are actually using prefect 1 and we can migrate to 2.0

I’m not trying to solve managing code dependencies, we just don’t want anything installed outside Docker. So using Docker Agent, we don’t want to pip install on our servers and would like to use it in it’s own container.

Anna_Geller @Anna_Geller: > we just don’t want anything installed outside Docker
As our CTO Chris White often says: “Never say just” :slightly_smiling_face: I understand the sentiment toward containerization but you need to keep in mind that there is no one-size-fits-all way of packaging dependencies.

For Prefect Docker agent specifically, it needs to run as an independent process on your machine in order to communicate with your Docker client to spin up Docker containers for a flow run. Think of it this way: you can’t install a Docker client (e.g. Docker Desktop) as a Docker container, would you agree? It must be installed and running in order to spin up dockerized workloads. You can think of the Prefect Docker agent as a similar process that needs direct access to your system in order to spin up containerized workloads.

To minimize any headaches with dependency management, many Prefect users who use a Docker agent spin up the agent process within a virtual environment.