How to set Secrets (e.g. GITHUB_ACCESS_TOKEN) on Server?

Archive
, , , , , , , , ,
1

The Prefect Server uses local Secrets. Therefore, the local Secret e.g. the Secret for your storage access token must be set:

  • either within the ~/.prefect/config.toml
  • or as an environment variable.

We generally recommend setting it via an environment variable, but here are several options how you can configure it.

1) config.toml

You can add your Secret within the configuration file as follows:

[context.secrets]
GITHUB_ACCESS_TOKEN = "your_token"

2) Environment variable set on agent startup

You can set the environment variable directly when you start your agent.

prefect agent local start --env PREFECT__CONTEXT__SECRETS__GITHUB_ACCESS_TOKEN=xxx

It works the same way regardless of the agent type. For instance, if you use a Kubernetes agent configured via a YAML file, you can set environment variables as follows to set a custom Secret:

    spec:
      containers:
      - args:
        - prefect agent kubernetes start
        command:
        - /bin/bash
        - -c
        env:
        - name: PREFECT__CLOUD__USE_LOCAL_SECRETS
          value: true
        - name: PREFECT__CONTEXT__SECRETS__GITHUB_ACCESS_TOKEN
          value: xxx
        - name: PREFECT__CLOUD__AGENT__AUTH_TOKEN
          value: ''
        - name: PREFECT__CLOUD__API
          value: "http://some_ip:4200/graphql" # paste your GraphQL Server endpoint here
        - name: PREFECT__BACKEND
          value: server

3) Environment variable set in your execution environment

You could set the environment variable on your machine before starting the agent:

export PREFECT__CONTEXT__SECRETS__GITHUB_ACCESS_TOKEN=xxx

4) Environment variable set in the run config

Alternatively, you can supply it to your run configuration’s metadata (which gets stored in the backend):

flow.run_config = UniversalRun(env={"PREFECT__CONTEXT__SECRETS__GITHUB_ACCESS_TOKEN": "xxx"})

5) Add it as an environment variable in your Kubernetes job template

If you use Kubernetes agent (e.g. deployed together with Server Helm Chart), you can set it on the job template for flow runs, for instance by using Kubernetes Secrets set as environment variables. This will load the Kubernetes Secret into the flow run pod as an environment variable.

Using this approach, you don’t have to set it on the agent, since the flow isn’t pulled from Storage until the flow run pod starts. Here is an example job template with Storage secrets:

# src/prefect/agent/kubernetes/job_template.yaml
apiVersion: batch/v1
kind: Job
spec:
  template:
    spec:
      containers:
      - name: flow
        image: prefecthq/prefect:0.15.12-python3.8
        env:
          - name: GITHUB_ACCESS_TOKEN
            valueFrom:
              secretKeyRef:
                name: GITHUB_ACCESS_TOKEN
                key: xxx
      restartPolicy: Never

How to ensure that your backend uses local Secrets?

:point_right: Note that: you may need to set this extra environment variable to declare on your backend that you are on the Prefect Server and you are therefore using local Secrets:

export PREFECT__CLOUD__USE_LOCAL_SECRETS=true
2

This does not seem to work for v2.

prefect agent local start --env PREFECT__CONTEXT__SECRETS__GITHUB_ACCESS_TOKEN=xxx
yields errors:
No such command 'local'.
and
No such option: --env

is there a way to load the gh token either in the gh storage definition or the deployment?

For example, the S3 block has built-in options for passing the the key/secret docs.

Another possibility could using some type of Deployment.infra_overrides env variable, perhaps?

3

Correct, that’s why this topic is labeled with the tag prefect-1-0. In Prefect 2 you can leverage Blocks to store secrets, check out: