View in #prefect-community on Slack
@Leon_Kozlowski: For a k8s agent, should all of the pods created for a flow runs inherit the Environment:* from the agent? I am having issues persisting a service account and role ARN that give flows privileges to hit aws resources (details in thread)
When I describe my agent pod I am seeing the correct service account and role ARN with privs:
Environment:
AWS_DEFAULT_REGION: us-west-1
AWS_REGION: us-west-1
AWS_ROLE_ARN: arn:aws:iam::<ID>:role/<ROLE_NAME>
AWS_WEB_IDENTITY_TOKEN_FILE: <TOKEN_LOCATION>
But when a job pod is created I no longer see those values and I am getting an AccessDenied error
botocore.exceptions.ClientError: An error occurred (AccessDeniedException) when calling the Query operation: User <USER> is not authorized to perform: dynamodb:Query
When the role listed in the agent is authorized to preform the query
I tried including the env var SERVICE_ACCOUNT_NAME in the run config in the UI for a quick test and still got the same error
@Kevin_Kho: This is beyond me. Will have to ask around.
@Leon_Kozlowski: Thanks @Kevin_Kho
@Jamie_Dick: Hey Leon - just to confirm, your AWS role allows the mentioned dynamodb:query action?
And if you describe one of your job pods, you are no longer seeing those env values present?
@Leon_Kozlowski: Hi Jamie - yes, the role I created has dynamodb:query and dynamodb:scan for the table and index that I’m hitting
When I describe my agent pod I can see the role ARN for the correct role + a mount for the serviceaccount
and when I show the yaml kubectl get deployment <NAME> -o yaml - I can see the serviceAccount and serviceAccountName as they should be
then when I describe an in progress job for a flow run, I no longer see the role in Environment: or the mount
@Jamie_Dick: are you describing the job itself or the pod hosting the job?
i would assume the job pods would inherit configuration from the agent pod but let me double check on this behavior
@Leon_Kozlowski: I was describing the pod sorry
Like this: kubectl describe pod prefect-job-XXXXXX-XXXXX
@Jamie_Dick: ah got it. ok let me look into this
can you share your kubernetes run config here?
@Leon_Kozlowski actually i think this is all you need - https://docs.prefect.io/orchestration/agents/kubernetes.html#service-account
Kubernetes Agent | Prefect Docs
you can either specify the name of the service account in your run config service_account_name = arn:aws:iam::<ID>:role/<ROLE_NAME> or you can define it when you whereever you are starting the prefect agent
@Leon_Kozlowski: it should be set to the arn? or the service account name
@Jamie_Dick: yeah the SA, sorry about that!
@Leon_Kozlowski: Can this be done in the job_template?
Was able to get it working by setting serviceAccountName in my job_template.yaml - thanks @Jamie_Dick
@Mariia_Kerimova: Hi! I think you could add annotation to your kubernetes service account like this:
https://docs.aws.amazon.com/eks/latest/userguide/specify-service-account-role.html
Make sure to annotate correct kubernetes service account (the one that used by your prefect job).
@Anna_Geller: I was reading on IAM roles for Service Accounts, but what Maria shared seems to be equivalent because it also results in the same IAM role annotation - in both cases, you would need to create OIDC provider for your EKS cluster (need to be done only once) if you want to use IAM roles for SA.
IAM roles for service accounts - Amazon EKS
@Leon_Kozlowski: Thanks all