View in #prefect-community on Slack
@Madhup_Sukoon: Do you happen to know if there are any arm-based prefect docker images?
@Anna_Geller: for an ARM-based agent you can’t use the Prefect’s default image because it’s AMD-based. You’re right that it’s likely the issue now. You would need to build a custom image or change to use Docker storage instead with:
flow.storage = Docker(..., build_kwargs={"platform": "linux/arm64"}) # or: "linux/amd64"})
We don’t have separate images for different architectures atm. You can check that on Docker
but even if you build your own image or change to Docker Storage, I would recommend still starting this a simple hello world flow before you use it with custom dependencies - it’s easier to do it step by step
@Madhup_Sukoon: Yup, I’ll try starting from a simple flow and checking if things work. Can you tell me if there’s a way for me to build the Prefect image from scratch? (I can build it for arm and test)
Or if it would be possible for you guys to build an arm version?
@Anna_Geller: Sure, let me help you. It’s quite simple.
You need a Dockerfile. Given your packages you may use this:
FROM prefecthq/prefect:0.15.13-python3.8 # choose version
RUN pip install requests pyyaml
Then to build this image, run this command from the same directory where your dockerfile is:
docker build --platform linux/arm64 -t your_image .
Then, once your image is built, in your DockerRun pass the image name:
flow.run_config = DockerRun(image="your_image:latest")
This flag does all you need to build arm image: --platform linux/arm64
@Madhup_Sukoon: Thanks, I’ll try this.
Also many thanks for your patience!
After a while discussion continues since this hasn’t solved the issue yet
@Madhup_Sukoon: Sorry, the image is on dockerhub
Just added --show-flow-logs and noticed this
standard_init_linux.go:228: exec user process caused: exec format error
@Kevin_Kho: That is most likely related to the image incompatibility I think
@Madhup_Sukoon:
[2022-02-01 21:06:19,944] DEBUG - agent | Sleeping flow run poller for 1.0 seconds...
[2022-02-01 21:06:20,944] DEBUG - agent | Querying for ready flow runs...
[2022-02-01 21:06:21,290] DEBUG - agent | No ready flow runs found.
[2022-02-01 21:06:21,290] DEBUG - agent | Sleeping flow run poller for 2.0 seconds...
[2022-02-01 21:06:22,534] DEBUG - agent | {'status': 'Pulling from <http://docker.io/me/prefect|docker.io/me/prefect>', 'id': 'latest-python3.9-arm'}
[2022-02-01 21:06:22,534] DEBUG - agent | {'status': 'Digest: sha256:aa9d2263f25beafa320633949fcba81b5d51e4cb2b0a0b78e67d5d1af2a272e0'}
[2022-02-01 21:06:22,534] DEBUG - agent | {'status': 'Status: Image is up to date for me/prefect:latest-python3.9-arm'}
[2022-02-01 21:06:22,535] INFO - agent | Successfully pulled image me/prefect:latest-python3.9-arm
[2022-02-01 21:06:22,535] DEBUG - agent | Creating Docker container me/prefect:latest-python3.9-arm
/usr/local/lib/python3.6/site-packages/prefect/agent/docker/agent.py:423: UserWarning: `host.docker.internal` could not be automatically resolved to your local host. This feature is not supported on Docker Engine v19.03.11-ol, upgrade to v20.10.0+ if you encounter issues.
"`host.docker.internal` could not be automatically resolved to your "
[2022-02-01 21:06:22,565] DEBUG - agent | Starting Docker container with ID f193912f48643eed4c3437f6cf2f0ca29a8cf50ad4600741878edb869c0d0754 and name 'cuddly-waxbill'
[2022-02-01 21:06:22,853] DEBUG - agent | Docker container f193912f48643eed4c3437f6cf2f0ca29a8cf50ad4600741878edb869c0d0754 started
[2022-02-01 21:06:22,854] INFO - agent | Completed deployment of flow run 9badb74e-4c9f-4f20-902c-ea32969a57de
standard_init_linux.go:228: exec user process caused: exec format error
[2022-02-01 21:06:23,290] DEBUG - agent | Querying for ready flow runs...
[2022-02-01 21:06:23,644] DEBUG - agent | No ready flow runs found.
[2022-02-01 21:06:23,645] DEBUG - agent | Sleeping flow run poller for 4.0 seconds...
@Kevin_Kho: Did you build with the --platform like Anna suggested?
@Madhup_Sukoon: yes
@Kevin_Kho: Let me look at some stuff
Ok, so it looks like even if you build it on the ARM platform, you need to specify it on the container creation side (when the Flow is spun up). The problem is that the docker BuildKit CLI lets you do this, but docker-py, which Prefect uses under the hood does not. You can read the details here
This means you need some kind of hack like installing buildkit on the agent and loading the image already so that Prefect doesn’t try to download it. Ultimately, I don’t think the pieces from the underlying tools are there to support this yet (specifically docker-py)
DockerStorage not running on different architecture (x86 vs M1) · Issue #4918 · PrefectHQ/prefect · GitHub