Kubernetes Job giving 401 Unauthorized error in AKS (azure kubernetes service)

Help
, , , ,
1

I am trying to run prefect flow in azure kubernetes service (AKS) using prefect kubenetes job infrastructure. I am able to create kubenetes block and also KubernetesClusterConfig block for context.
But I am getting below 401 Unauthorized error. It seems when prefect server is trying to create infrastructure using configuration it is getting authorized, and k8 cluster is not allowing to create new job.
Error I get on agent after submitting my job through prefect server:

HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}

My question is how to do we handle AKS authorization to create kubernetes object.

# Define cluster configuration
cluster_config = KubernetesClusterConfig.from_file(
    Path(f"{Path(__file__).parent.resolve()}/context.yaml")
)
cluster_config .save("cluster-config", overwrite=True)

# Define the K8 job manifest
k8job= KubernetesJob(
    cluster_config=KubernetesClusterConfig.load("cluster-config"),
    finished_job_ttl=3600,
    image=os.environ["PREFECT_JOB_IMAGE"],
    image_pull_policy=KubernetesImagePullPolicy.IF_NOT_PRESENT,
    namespace="my-namespace",
    pod_watch_timeout_seconds=120,
    name="prefect-job",
    stream_output=True,
)

k8job.save("prefect-k8-job", overwrite=True)


k8_job_deployment = Deployment.build_from_flow(
    flow=my_flow,
    name="K8 Job",
    work_queue_name=WORK_QUEUE,
    infrastructure=KubernetesJob.load("prefect-k8-job"),
    parameters={
        "destination_folder": os.path.join(CURRENT_DIR, PREFETCH_DIR_NAME),
      },
  )

k8_job_deployment .apply()