Can I use Ubuntu as a base image for Prefect flows rather than using one of the official Prefect base images?

[anna_geller]

View in #prefect-community on Slack

@Aqib_Fayyaz: Can i use docker-storage for flow with ubtuntu 20.04 as a base image not prefect in Dockerfile?

@Matthias: You sure can! As long as the proper dependencies and entrypoint are set that it. Note however that installing python in a “raw” ubuntu is not as straightforward as you might think (been there done that). So why would you want to do that?

@Aqib_Fayyaz: I need to install spark and when using ubuntu as base image while registering flow it fails health checks and does not register the flow

@Matthias: But you can use a Python image, install JDK and pip install spark. It should install all it’s dependencies
Maybe you can even start from Prefect’s base image for that?

@Anna_Geller: To add to Matthias’s excellent answer, if you are running Spark just within a Docker image, you are not taking advantage of Spark right? Where is your Spark cluster running? Usually, you should be able to just run a “Spark submit” command with e.g. ShellTask to submit the job to your cluster

Also, we’ve already had this conversation about Spark and packaging it into a Docker image, let me check if I can find it

The linked conversation is here

How to package Prefect and Spark into the same Docker image?

Also, the official Prefect base images:
https://hub.docker.com/r/prefecthq/prefect