Issues using GitLab storage with KubernetesAgent and Prefect Server configured with Helm chart - GitlabGetError('404 Project Not Found') or File or Directory Not Found

Archive
, , , ,
1

View in #prefect-community on Slack

Maria_Lee @Maria_Lee: I’m running into an issue trying to use GitLab storage to reference flows stored in my gitlab repo. This is how I have it in my script:

from prefect.storage import GitLab
storage = GitLab(repo="org/repo",
                 path="/hello/hello_k8s.py",
                 ref="prefect")

And when I try to run the flow, I get 404 error file not found but the file is there. Is there a way to print out what path prefect runner is using?

Kevin_Kho @Kevin_Kho: The call is pretty easy to replicate if you check the code here. But I have an example one sec.

GitHub: prefect/gitlab.py at master · PrefectHQ/prefect

@Anna_Geller: you also have a typo - is it a copy-paste issue? it should be GitLab not Gitlab

Kevin_Kho @Kevin_Kho: I believe this works. Can you try removing your first slash in the path?

GitLab: myflow.py · main · Kevin Kho / prefect-gitlab

Maria_Lee @Maria_Lee: @Kevin_Kho I tried removing the first slash in the path but still doesn’t work. @Anna_Geller sorry it was a typo here. I had it GitLab.

Kevin_Kho @Kevin_Kho: So to test this easily, you can do:

storage = GitLab(repo="org/repo",
                 path="/hello/hello_k8s.py",
                 ref="prefect")

storage.get_flow()

to see if it works. does it work with something from the main branch if you can test that?
Could you show me your traceback?

Maria_Lee @Maria_Lee: do I also have to add flow name to storage.get_flow()?

Kevin_Kho @Kevin_Kho: Ah yes
Actually you might need to build it let me make a working code snippet

Maria_Lee @Maria_Lee: I’m getting following error when i try to do storage.get_flow(<flowname>)

...envs/prefect_env/lib/python3.8/site-packages/prefect/storage/gitlab.py" line 83, in get_flow
    raise ValueError("Flow is not contained in this Storage")
  ValueError: Flow is not contained in this Storage

Kevin_Kho @Kevin_Kho: Here is a working example:

from prefect import Flow, task
import prefect
from prefect.storage.gitlab import GitLab
from prefect.run_configs import KubernetesRun

@task
def abc(x):
    return x+1

with Flow("gh storage") as flow:
    abc(1)

storage = GitLab(
repo="kvnkho/prefect-gitlab", 
path="myflow.py",
ref="main")

storage.add_flow(flow)
storage.get_flow("gh storage")

Is your 404 Commit Not Found?

Maria_Lee @Maria_Lee: no i get this

Failed to load and execute flow run: GitlabGetError('404 Project Not Found')

i’ve also tried main branch but same error message

@Anna_Geller: check this https://discourse.prefect.io/t/i-got-an-error-failed-to-load-and-execute-flows-environment-gitlabgeterror-404-project-not-found-how-to-fix-it/112

Prefect Community: I got an error: Failed to load and execute Flow’s environment: GitlabGetError(‘404 Project Not Found’) - how to fix it?

this seems to be a permission issue

Kevin_Kho @Kevin_Kho: If it’s project, it can’t even find the repo. If it’s commit, it can’t find the branch

Maria_Lee @Maria_Lee: For the kubernetes deployment, do I configure the secret in config.toml?

Kevin_Kho @Kevin_Kho: better if on the agent. when you do prefect agent kubernetes install, you can add --env PREFECT___CONTEXT___SECRETS__MYSECRET="VALUE" and it adds it to the manifest. or you can add it in the job template. I am not confident it gets added with the config.toml for KubernetesRun. I think it would work though but the env is more sure

Maria_Lee @Maria_Lee: this would be the case when i’m running the server?? What if I have the k8s remotely set up with Helm? do I store it as k8s secrets in the namespace? or do I update values.yaml file and upgrade?

@Anna_Geller: here are several ways you can do it - using an environment variable is the main way of approaching this https://discourse.prefect.io/t/how-to-set-secrets-e-g-github-access-token-on-server/70

Prefect Community: How to set Secrets (e.g. GITHUB_ACCESS_TOKEN) on Server?

Kevin_Kho @Kevin_Kho: Yeah everything in the config.toml can be specified with the env variable. For Server, you only have local secrets so you can define it with the env variable on the agent like mentioned here . And then the agent will pass this env variable to the Flow run.

Secrets | Prefect Docs

Follow Anna. Her link is way more detailed

Maria_Lee @Maria_Lee: I’ve tried all the five steps in Anna’s link but they all didn’t work out (the setup i have is with K8s with Helm Chart). I’ve also tried out creating secret with kubectl and tried to pass it from there but that also didn’t work. I’m not sure how else to check:disappointed_relieved:

Kevin_Kho @Kevin_Kho: is that a private GitLab or public?

Maria_Lee @Maria_Lee: private
this is the error message i get

Failed to load and execute flow run: ValueError('Local Secret "GITLAB-ACCESS-TOKEN" was not found.')

Kevin_Kho @Kevin_Kho: Oh this hits the public one though. Did you specify the host argument?

Maria_Lee @Maria_Lee: yes i did

Kevin_Kho @Kevin_Kho: Ah ok how did you specify the secret?

Maria_Lee @Maria_Lee: i tried specifying in 1) config.toml like below

[context.secrets]
GITHUB_ACCESS_TOKEN = "xxx"

didn’t work.

  1. I tried agent startup env setup
prefect agent kubernetes install --env PREFECT__CONTEXT__SECRETS__GITLAB-ACCESS-TOKEN="xxx"

didn’t work.

  1. tried setting up on my execution environment
export PREFECT__CONTEXT__SECRETS__GITLAB-ACCESS-TOKEN="xxx"

didn’t work.

  1. Actually haven’t tried this: Environment variable set in the run config because I didn’t want to save token to the script

  2. trie to add it as an environment variable in your Kubernetes job template and also stored as kubernetes secret by

kubectl create secret generic gitlab-access-token --namespace prefect --from-literal=username="mygitlabusername" --from-literal=token="xxx"

and tried to to run kubectl apply -f job_template.yaml

apiVersion: batch/v1
kind: Job
metadata:
  name: prefect-server-agent
spec:
  template:
    spec:
      containers:
        - name: flow
          image: prefecthq/prefect:0.15.12-python3.8
          env:
            - name: PREFECT__CONTEXT__SECRETS__GITLAB-ACCESS-TOKEN
              valueFrom:
                secretKeyRef:
                  name: gitlab-access-token
                  key: xxx
      restartPolicy: Never

which also didn’t work…

Prefect Community: How to set Secrets (e.g. GITHUB_ACCESS_TOKEN) on Server?

Kevin_Kho @Kevin_Kho: You re-redeployed the agent after number 2? 3 will not work because the context is made at the start of the flow so putting it in the execution environment is too late

Maria_Lee @Maria_Lee: when you say re-deplyed, do you mean prefect agent kubernetes start? When I do that, I get

[2022-03-31 01:32:47+0200] WARNING - prefect.kubernetes | Service host/port is not set. Using out of cluster configuration option.
[2022-03-31 01:32:47+0200] WARNING - prefect.kubernetes | Service host/port is not set. Using out of cluster configuration option.

and then i think it’s just listening for flow runs to be submitted to public prefect cloud. I’d like to run it on my own k8 cluster server

Kevin_Kho @Kevin_Kho: How did you get the previous agent? So the step in number 2 just creates the manifest but you still have to apply it to your cluster. The instructions are here . So you need to make a new agent that will have the secret and then when the Flows get picked up by the new agent, the will get the local secret. So you do the install command and then pass it to kubectl apply to spin up the agent. In order to point it to your server, you need the env vars:

PREFECT__BACKEND="server"
PREFECT__SERVER__ENDPOINT=<your-endpoint>

I think you must have done this for the other agent?

Kubernetes Agent | Prefect Docs

Number 5 also requires an agent re-deployment because you are passing it upon agent start here and then it will be applied to flows. You can use this --job-template flag along with the install command

Kubernetes Agent | Prefect Docs

Maria_Lee @Maria_Lee: Do you mean as a local environment? or pass it as environment variable to prefect agent install?

prefect agent kubernetes install --env PREFECT__CONTEXT__SECRETS__GITLAB-ACCESS-TOKEN="xxx" PREFECT__BACKEND="server" PREFECT__SERVER__ENDPOINT=<my prefect server endpoint> | kubectl apply --namespace=prefect -f -

above command returns arguments PREFECT__BACKEND and PREFECT__SERVER__ENDPOINT don’t exist

and when i execute it after setting it in the local environment, it creates a prefect-agent pod which keeps crashing.
I get my previous agents from helm install prefect-server prefecthq/prefect-server --namespace prefect -f values.yaml . This command spawns prefect-server-agent, prefect-server-apollo, prefect-server-create-tenant-job, prefect-server-graphql, prefect-server-hasura, prefect-server-postgresql, prefect-server-towel, and prefect-server-ui. And I’ve updated ~/.prefect/config.toml to point it to my host server

Kevin_Kho @Kevin_Kho: Yes like the command you mentioned. You need to do --env for each env variable you are setting. When you say execute on local environment, do you mean you do:

export PREFECT__CONTEXT__SECRETS__GITLAB-ACCESS-TOKEN="xxx"

and then run the

kubectl apply --namespace=prefect -f -

command after?

What is the error on the crash?

I see on the helm install. You can add the env variable for the secret on the agent in the values.yaml file as well I believe. Although I am personally less experienced with that.

Maria_Lee @Maria_Lee: This is the logs i get from pods crash


Traceback (most recent call last):
  File "/usr/local/bin/prefect", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/prefect/cli/agent.py", line 305, in start
    start_agent(KubernetesAgent, image_pull_secrets=image_pull_secrets, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/prefect/cli/agent.py", line 128, in start_agent
    agent.start()
  File "/usr/local/lib/python3.7/site-packages/prefect/agent/agent.py", line 186, in start
    self._setup_api_connection()
  File "/usr/local/lib/python3.7/site-packages/prefect/agent/agent.py", line 870, in _setup_api_connection
    self.client.attach_headers({"X-PREFECT-AGENT-ID": self._register_agent()})
  File "/usr/local/lib/python3.7/site-packages/prefect/agent/agent.py", line 831, in _register_agent
    agent_config_id=self.agent_config_id,
  File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 1782, in register_agent
    tenant_id=self.tenant_id,
  File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 273, in tenant_id
    self._tenant_id = self._get_default_server_tenant()
  File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 185, in _get_default_server_tenant
    response = self.graphql({"query": {"tenant": {"id"}}})
  File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 458, in graphql
    retry_on_api_error=retry_on_api_error,
  File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 414, in post
    retry_on_api_error=retry_on_api_error,
  File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 654, in _request
    raise ClientError("Malformed response received from API.") from exc
prefect.exceptions.ClientError: Malformed response received from API.

So I’ve updated value.yaml for helm and created an agent with the variable PREFECT__CONTEXT__SECRETS__GITLAB-ACCESS-TOKEN. When I check in the pod, the env variable exists, but i still get an error that local secret with the name is not found

@Anna_Geller: it should be underscores:

PREFECT__CONTEXT__SECRETS__GITLAB_ACCESS_TOKEN

Maria_Lee @Maria_Lee: I’ve tried both

Failed to load and execute flow run: ValueError('Local Secret "GITLAB_ACCESS_TOKEN" was not found.')

@Anna_Geller: how exactly did you set it on your helm chart?
when using env variables, you simultaneously need this Gitlab secret + export PREFECT__CLOUD__USE_LOCAL_SECRETS=true

Maria_Lee @Maria_Lee: This is a snippet of my value.yaml for helm install/upgrade

agent:
  # enabled determines if the Prefect Kubernetes agent is deployed
  enabled: true

  # prefectLabels defines what scheduling labels (not K8s labels) should
  # be associated with the agent
  prefectLabels: []

  # jobTemplateFilePath defines which template to use for the agent's jobs. Defaults
  # to an empty string, which will use the default template.
  # reference: <https://docs.prefect.io/orchestration/agents/kubernetes.html#custom-job-template>
  jobTemplateFilePath: ""

  # image configures the container image for the agent deployment
  image:
    name: prefecthq/prefect
    tag: null
    pullPolicy: Always
    pullSecrets: []
   

  labels: {}
  annotations: {}
  podAnnotations: {}
  replicas: 1
  strategy: {}
  podSecurityContext: {}
  securityContext: {}
  env: #[]
    - name: PREFECT__CONTEXT__SECRETS__GITLAB_ACCESS_TOKEN 
      value: "xxx"
    - name: PREFECT__CLOUD__USE_LOCAL_SECRETS
      value: true
  nodeSelector: {}
  tolerations: []
  affinity: {}

after helm install runs successfully, i was able to check that both PREFECT__CONTEXT__SECRETS__GITLAB_ACCESS_TOKEN and PREFECT__CLOUD__USE_LOCAL_SECRETS exists in the agent pod locally as env variables. but the run still fails on this agent with the same local secret not found.

@Anna_Geller: Nice work checking that and ensuring that the agent has that environment variable. It looks like it works well, but it doesn’t propagate to the Kubernetes flow run pod. To fix this, you could add a Kubernetes job template to your KubernetesRun run configuration in your flow.
This could be your job template:

job_template = """
apiVersion: batch/v1
kind: Job
spec:
  template:
    spec:
      containers:
      - name: flow
        image: prefecthq/prefect:latest-python3.8
        env:
          - name: PREFECT__CLOUD__USE_LOCAL_SECRETS
            value: true
          - name: GITHUB_ACCESS_TOKEN
            valueFrom:
              secretKeyRef:
                name: GITLAB_ACCESS_TOKEN
                key: xxx
      restartPolicy: Never
"""

and then to apply that on your KubernetesRun:

flow.run_config = KubernetesRun(job_template=job_template)

Maria_Lee @Maria_Lee: That worked for now. Thank you @Kevin_Khoand @Anna_Geller. So to summarize, I’ve updated my values.yaml for helm chart in the agent section to pass env variables, and then updated my hello_k8s.py script with job_template variable defined (the only thing i changed in Anna’s example above was setting key as GITLAB_ACCESS_TOKEN and not the actual access token string). That solved the “local secret xyz not found” issue.

Unfortunately, my hello flow is still not working… onto the next error message(below). I noticed that my agent is trying to create job containers but they are failing. So I checked logs in the agent. :

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/prefect/agent/kubernetes/agent.py", line 413, in heartbeat
    self.manage_jobs()
  File "/usr/local/lib/python3.7/site-packages/prefect/agent/kubernetes/agent.py", line 193, in manage_jobs
    f"Job {job.name!r} is for flow run {flow_run_id!r} "
AttributeError: 'V1Job' object has no attribute 'name'
ERROR:agent:Error while managing existing k8s jobs
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/prefect/agent/kubernetes/agent.py", line 190, in manage_jobs
    flow_run_state = self.client.get_flow_run_state(flow_run_id)
  File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 1340, in get_flow_run_state
    raise ObjectNotFoundError(f"Flow run {flow_run_id!r} not found.")
prefect.exceptions.ObjectNotFoundError: Flow run "xxx" not found.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/prefect/agent/kubernetes/agent.py", line 413, in heartbeat
    self.manage_jobs()
  File "/usr/local/lib/python3.7/site-packages/prefect/agent/kubernetes/agent.py", line 193, in manage_jobs
    f"Job {job.name!r} is for flow run {flow_run_id!r} "
AttributeError: 'V1Job' object has no attribute 'name'

Where could this be coming from?
I’m getting endless repeat log messages on Prefect UI like this:

03:51:02
DEBUG
k8s-infra
Event: 'Failed' on pod 'prefect-job-123-123'
	Message: Error: couldn't find key gitlab-access-token in Secret prefect/gitlab-access-token
03:51:14
DEBUG
k8s-infra
Event: 'Pulled' on pod 'prefect-job-123-123'
	Message: Container image "prefecthq/prefect:latest-python3.8" already present on machine

Kevin_Kho @Kevin_Kho: Seems like there is an open issue for the AtteibuteError you have. I think this might be because of the latest version of kubernetes. Maybe you can try downgrading your library a version?

GitHub: Kubernetes Agent causes nested exception during error handling while monitoring Jobs · Issue #5427 · PrefectHQ/prefect

Am pretty confused why it still can’t find the secret. Anna may have better ideas than me but could you try adding it to storage?

flow.storage = GitLab(..., secrets=["GITLAB_ACCESS_TOKEN"]

though I don’t think you should even have to do this, but this will explicitly tell Prefect to load a secret into the context

Maria_Lee @Maria_Lee: isn’t the error coming from prefect? I’ve also tried adding flow.storage = GitLab(..., secrets=["GITLAB_ACCESS_TOKEN"]) , no changes…

Kevin_Kho @Kevin_Kho: The first traceback with the attribute error is more like Prefect calls a method/attribute that doesnt exist in a newer version of the package. I am personally pretty confused since it seems like we went through all the options. The RunConfig should be super sure already of it getting into the pod. I looked at this again, and I think

Message: Error: couldn't find key gitlab-access-token in Secret prefect/gitlab-access-token

might be an error from the job template trying to fetch it from the secretKeyRef

          - name: GITHUB_ACCESS_TOKEN
            valueFrom:
              secretKeyRef:
                name: GITLAB_ACCESS_TOKEN
                key: xxx

Not super sure but the message doesn’t look like a log from the Prefect library. Not exactly sure myself though. Anna would know more

Maria_Lee @Maria_Lee: That took me a while! I finally got the hello example to work. I didn’t need to pass “secret=” argument in the GitLab() call in my prefect script. This time it was just getting the right key names in the job_template

job_template = """
apiVersion: batch/v1
kind: Job
spec:
  template:
    spec:
      containers:
      - name: flow
        image: prefecthq/prefect:latest-python3.8
        env:
          - name: PREFECT__CLOUD__USE_LOCAL_SECRETS
            value: true
          - name: PREFECT__CONTEXT__SECRETS__GITLAB_ACCESS_TOKEN
            valueFrom:
              secretKeyRef:
                name: GITLAB_ACCESS_TOKEN
                key: #this should be the key name that you defined in the kubernetes secret
      restartPolicy: Never
"""

Thank you @Kevin_Kho @Anna_Geller for your help and quick responses!

@Anna_Geller: This makes so much sense since GITLAB_ACCESS_TOKEN is the default secret name Prefect is looking for when using this storage. Thanks so much for sharing, and great work solving the issue!