How to package a common task library into a Docker image and use it in my run configuration

Archive
, , ,
1

I’m going through the docs about installing dependencies to docker containers and in the following code snippet it says that my_task_library needs to be installed. However I don’t understand where I need to copy it into the docker container?

from prefect import Flow, task
from prefect.storage import GitHub

import numpy
from my_task_library import task_1

@task
def task_2(x):
    return numpy.sum(x)

with Flow("example") as flow:
    x = task_1()
    y = task_2(x)

flow.storage = GitHub(repo="my_username/my_repo", path="path/to/flow.py")

Normally I would do something like this in my Dockerfile

COPY my_task_library /app/my_task_library
COPY main.py /app/

since the my_task_library is in the same directory in the container as the main entrypoint it gets picked up by python.

Can I do something similar with prefect? If I’m using GitHub storage, where does prefect execute the flow from within the container so that these common tasks get picked up by python’s module system?

1 Like
2

To execute your flow run as a docker container, you would need to use DockerRun run configuration and a Docker agent. For various configurations of storage and run configurations, check out this repository:

And for more background on this error, check this Discourse topic:

This post may also help:

LMK, if you still have any questions after reading this, happy to explain more.