How to set GCP_CREDENTIALS or GOOGLE_APPLICATION_CREDENTIALS to use GCP modules or GCS storage with Prefect flows?

[anna_geller]

View in #prefect-community on Slack

@Taylor_Harless: I’m struggling to implement the GCP_CREDENTIALS default secret feature in a local Prefect flow.
• I’ve set an environmental variable PREFECT__CONTEXT__SECRETS_GCP_CREDENTIALS pointing to a full local path for the GCP credentials .json downloaded from the GCP project to which I’d like to upload some files.
• I’ve run the following:

from prefect import Flow
from prefect.tasks.gcp.storage import GCSUpload

with Flow("google-cloud-test") as flow:

    GCSUpload(bucket="test-upload", create_bucket=True)(
        data="test-file.csv", credentials="GCP_CREDENTIALS"
    )

flow.run()

and received an error: AttributeError: 'str' object has no attribute 'keys'. I’ve incorporated the feedback from a similar error discussion recently, but can’t figure out what the issue is. Any help is much appreciated.

@Kevin_Kho: Hi @Taylor_Harless, I think Google convention and Prefect convention are different here. Google convention for this is an env variable named GOOGLE_APPLICATION_CREDENTIALS that points to the json. The GCP_CREDENTIALS though contains the service account information. It is fed into line 29 here. If the credentials aren’t there, it goes to line 33 where it uses the default GOOGLE_APPLICATION_CREDENTIALS

GitHub: prefect/gcp.py at 96183e95ad069c927799a6a272bc3d9a9cce3197 · PrefectHQ/prefect

@Taylor_Harless: Hi Kevin, thanks for the quick response. Yes, I can see that those are two different conventions. I’ve tried defining the GCP_CREDENTIALS env variable with the contents of the json, but it’s too large/long to fully add. I’ll could try cherry picking a few of the key/value pairs from the .json to add to the GCP_CREDENTIALS variable to see if that solves it. Any ideas on other methods I should try?

@Kevin_Kho: Man I swear their docs had an example somewhere but I can’t fine the page. What error do you get when you do that? You could have just left the GOOGLE_APPLICATION_CREDENTIALS env variable too and that would have worked as long as your execution environment has the file
I would say fiddle with what makes the Credentials.from_service_account_info happy. You can test this outside of Prefect by just importing and creating the connection. And then whatever works there will work has the secret value

@Taylor_Harless: I get the same error AttributeError when I leave: 1. remove the credentials attribute definition, or 2. update the PREFECT__CONTEXT__SECRETS_GCP_CREDENTIALS env variable to contain the partial dictionary inside the GCP credentials json. I appreciate all the good feedback, and on a weekend no less - I’ll look at the Credentials class you mentioned at some point tomorrow and report back here.

@Kevin_Kho: I can try this more tonight

@Anna_Geller: What’s your Prefect setup @Taylor_Harless ? Do you use Prefect Cloud or Server? What agent type and flow storage do you use?
Since I don’t know which agent you use, let me explain it for a Local and Docker agents.

Docstring in this example explains how you can set GCP credentials. The easiest is to use the Google convention, as Kevin explained. To set this up, create a service account JSON file as explained here, then in your execution environment (e.g. on your agent) set this env variable:

export GOOGLE_APPLICATION_CREDENTIALS="/Users/yourname/path/to/your/repo/packaging-prefect-flows/gcs_sa.json"

But note that this variable is just a path to the JSON file, rather than its content with the actual secret.

For uploading things to GCS, you can set this up directly on your agent as follows:
• for a Docker agent - the command below will make sure to mount the credentials JSON file to each flow run container and set the environment variable GOOGLE_APPLICATION_CREDENTIALS to ensure that GCP modules such as GCSUpload can find where this mounted file is located within your flow run Docker container (i.e. your execution environment)

prefect agent docker start --label AGENT_LABEL --volume /Users/anna/repos/packaging-prefect-flows/gcs_sa.json:/opt/prefect/gcs_sa.json --env GOOGLE_APPLICATION_CREDENTIALS="/opt/prefect/gcs_sa.json"
        

• for a Local agent, you can do the same but you don’t have to mount a volume since everything runs locally on the same VM

prefect agent local start --label AGENT_LABEL --env GOOGLE_APPLICATION_CREDENTIALS="/Users/anna/repos/packaging-prefect-flows/gcs_sa.json"

Adjust the path to where you stored your JSON file and you should be good to go.