View in #prefect-server on Slack
@Ben_Welsh: I have a private Python package bundled up and stored in Google Artifact Registry (as opposed to an open-source package on PyPI). I’d like to include it in my flow, which uses a Docker storage instance in production. I know that I can use the python_dependencies kwarg to the class to include open-source packages. But how do I get a private package from Google Artifact Registry included as well? Is there an established pattern for this?
@Kevin_Kho: So this is happening because you are using the DockerStorage as the interface to build the image. If it’s too limiting, you can supply your own Dockerfile or image and just handle it yourself.
But for this, I think there is a chance we can get it to work by pointing your pip to the Artifact Registry. Each one of the dependencies are just pip installed here by adding commands to the container
GitHub: prefect/docker.py at 23f541b15de35d95a533cf706bd9cdb7d672bc64 · PrefectHQ/prefect
So this will work if doiing
pip install some_library" magically worked. I’ll look a bit
You can also add extra commands to the Docker build if ever we can’t pip install it
I think you need to add your own pip install command with the extra things like this
Google Cloud: Managing Python packages | Artifact Registry documentation | Google Cloud
@Anna_Geller: Adding --index-url should do the trick - you can add it to your Dockerfile like so:
RUN pip install --index-url https://LOCATION-python.pkg.dev/PROJECT/REPOSITORY/simple/ PACKAGE
But when you register your flow and build the image, your terminal must be authenticated with Artifact Registry Reader permissions.
You could also just add the index URL at the top of your requirements.txt like so:
--index-url https://LOCATION-python.pkg.dev/PROJECT/REPOSITORY/simple>
--extra-index-url https://pypi.org/simple
# PyPi dependencies
pandas==1.4.1
# your custom private packages
custom==0.1
If you are using JFrog Artifactory, it would be (it assumes those environment variables being set in your environment):
--index-url https://${ARTIFACTORY_USER}:${ARTIFACTORY_PASSWORD}@hostname.jfrog.io/artifactory/api/pypi/pypi/simple
--extra-index-url https://pypi.org/simple
# PyPi dependencies
pandas==1.4.1
# your custom private packages
custom==0.1