Originally asked in Stack Overflow. Here is a copy. 
As Prefect work pools and workers are now generally available after 2.11.0. I am trying to switch from Prefect Agent to Prefect Worker.
I deployed Prefect Server by
helm upgrade \
prefect-server \
prefect-server \
--install \
--repo=https://prefecthq.github.io/prefect-helm \
--namespace=hm-prefect \
--create-namespace \
--values=prefect-server-values.yaml
prefect-server-values.yaml:
server:
image:
repository: docker.io/prefecthq/prefect
prefectTag: 2.11.0-python3.11-kubernetes
publicApiUrl: https://prefect.mydomain.com/api
helm upgrade \
prefect-worker \
prefect-worker \
--install \
--repo=https://prefecthq.github.io/prefect-helm \
--namespace=hm-prefect \
--create-namespace \
--values=prefect-worker-values.yaml
prefect-worker-values.yaml:
worker:
image:
repository: docker.io/prefecthq/prefect
prefectTag: 2.11.0-python3.11-kubernetes
apiConfig: server
config:
workPool: hm-kubernetes-pool
serverApiConfig:
apiUrl: http://prefect-server.hm-prefect.svc:4200/api
➜ helm list -n hm-prefect
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
prefect-server hm-prefect 1 2023-07-31 17:07:50.401888 -0700 PDT deployed prefect-server-2023.07.27 2.11.1
prefect-worker hm-prefect 1 2023-07-31 17:18:57.586027 -0700 PDT deployed prefect-worker-2023.07.27 2.11.1
➜ kubectl get deployment -n hm-prefect
NAME READY UP-TO-DATE AVAILABLE AGE
prefect-server 1/1 1 1 17m
prefect-worker 1/1 1 1 6m2s
And I can see Prefect Worker in the UI:
Then I generated YAML file by:
➜ prefect deployment build src/main.py:print_platform --name=print-platform --infra-block=kubernetes-job/print-platform-kubernetes-job-block --apply --pool=hm-kubernetes-pool
Found flow 'print-platform'
Deployment YAML created at '/Users/hongbo-miao/Clouds/Git/hongbomiao.com/hm-prefect/workflows/print-platform/print_platform-deployment.yaml'.
Deployment storage None does not have upload capabilities; no files uploaded. Pass --skip-upload to suppress this warning.
Deployment 'print-platform/print-platform' successfully created with id '7f7603ca-697c-4dca-9bcb-28a889165fe8'.
Here is the generated file print_platform-deployment.yaml content:
###
### A complete description of a Prefect Deployment for flow 'print-platform'
###
name: print-platform
description: null
version: e4da5dae95465f73a0e3e0bece1555bb
# The work queue that will handle this deployment's runs
work_queue_name: default
work_pool_name: hm-kubernetes-pool
tags: []
parameters: {}
schedule: null
is_schedule_active: true
infra_overrides: {}
###
### DO NOT EDIT BELOW THIS LINE
###
flow_name: print-platform
manifest_path: null
infrastructure:
type: kubernetes-job
env: {}
labels: {}
name: null
command: null
image: ghcr.io/hongbo-miao/hm-prefect-print-platform:latest
namespace: hm-prefect
service_account_name: null
image_pull_policy: Always
cluster_config: null
job:
apiVersion: batch/v1
kind: Job
metadata:
labels: {}
spec:
template:
spec:
parallelism: 1
completions: 1
restartPolicy: Never
containers:
- name: prefect-job
env: []
customizations: []
job_watch_timeout_seconds: null
pod_watch_timeout_seconds: 60
stream_output: true
finished_job_ttl: null
_block_document_id: 1f5b585c-581d-4ca4-adfa-c69dc5319941
_block_document_name: print-platform-kubernetes-job-block
_is_anonymous: false
block_type_slug: kubernetes-job
_block_type_slug: kubernetes-job
storage: null
path: /opt/prefect/flows
entrypoint: src/main.py:print_platform
parameter_openapi_schema:
title: Parameters
type: object
properties: {}
required: null
definitions: null
timestamp: '2023-08-01T00:32:45.975410+00:00'
triggers: []
Next, I try to run by
➜ prefect deployment run print-platform/print-platform
Creating flow run for deployment 'print-platform/print-platform'...
Created flow run 'onyx-fennec'.
└── UUID: 065326e7-1d3e-455a-86fb-b15d553af5bd
└── Parameters: {}
└── Scheduled start time: 2023-07-31 17:32:50 PDT (now)
└── URL: https://prefect.mydomain.com/flow-runs/flow-run/065326e7-1d3e-455a-86fb-b15d553af5bd
However, this gives me error:
Worker 'KubernetesWorker 180550e0-fe47-4a0d-998d-b772d53e14b0' submitting flow run '065326e7-1d3e-455a-86fb-b15d553af5bd'
Creating Kubernetes job...
Failed to submit flow run '065326e7-1d3e-455a-86fb-b15d553af5bd' to infrastructure.
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/prefect_kubernetes/worker.py", line 628, in _create_job
job = batch_client.create_namespaced_job(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/kubernetes/client/api/batch_v1_api.py", line 210, in create_namespaced_job
return self.create_namespaced_job_with_http_info(namespace, body, **kwargs) # noqa: E501
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/kubernetes/client/api/batch_v1_api.py", line 309, in create_namespaced_job_with_http_info
return self.api_client.call_api(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api
return self.__call_api(resource_path, method,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
response_data = self.request(
^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 391, in request
return self.rest_client.POST(url,
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/kubernetes/client/rest.py", line 276, in POST
return self.request("POST", url,
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/kubernetes/client/rest.py", line 235, in request
raise ApiException(http_resp=r)
kubernetes.client.exceptions.ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': '7871421d-254d-4d72-9a30-a7ff3306822b', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': 'e5d21bfa-f8ff-4689-965a-2c8efc99569b', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'f86dde2c-b36e-4c12-a44c-31e36a8ecf05', 'Date': 'Tue, 01 Aug 2023 00:32:51 GMT', 'Content-Length': '321'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"jobs.batch is forbidden: User \"system:serviceaccount:hm-prefect:prefect-worker\" cannot create resource \"jobs\" in API group \"batch\" in the namespace \"default\"","reason":"Forbidden","details":{"group":"batch","kind":"jobs"},"code":403}
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/prefect/workers/base.py", line 834, in _submit_run_and_capture_errors
result = await self.run(
^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/prefect_kubernetes/worker.py", line 506, in run
job = await run_sync_in_worker_thread(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/prefect/utilities/asyncutils.py", line 91, in run_sync_in_worker_thread
return await anyio.to_thread.run_sync(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/anyio/to_thread.py", line 33, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread
return await future
^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 807, in run
result = context.run(func, *args)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/prefect_kubernetes/worker.py", line 637, in _create_job
message += ": " + exc.body["message"]
~~~~~~~~^^^^^^^^^^^
TypeError: string indices must be integers, not 'str'
Completed submission of flow run '065326e7-1d3e-455a-86fb-b15d553af5bd'
Reported flow run '065326e7-1d3e-455a-86fb-b15d553af5bd' as crashed: Flow run could not be submitted to infrastructure
Inside seems this line is the issue
{“kind”:“Status”,“apiVersion”:“v1”,“metadata”:{},“status”:“Failure”,“message”:“jobs.batch is forbidden: User "system:serviceaccount:hm-prefect:prefect-worker" cannot create resource "jobs" in API group "batch" in the namespace "default"”,“reason”:“Forbidden”,“details”:{“group”:“batch”,“kind”:“jobs”},“code”:403}
I am not sure why it tries to create job in the namespace default instead of hm-prefect. Any ideas? Thanks!
