You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "yovio-rca (via GitHub)" <gi...@apache.org> on 2025/11/26 17:51:57 UTC
[I] Latest KubernetesPodOperator (Ver: 10.10.0) get 403 error (Forbidden to list K8s Events) [airflow]
yovio-rca opened a new issue, #58736:
URL: https://github.com/apache/airflow/issues/58736
### Apache Airflow Provider(s)
cncf-kubernetes
### Versions of Apache Airflow Providers
apache-airflow-providers-amazon==9.17.0
apache-airflow-providers-apprise==2.2.0
apache-airflow-providers-celery==3.13.1
**apache-airflow-providers-cncf-kubernetes==10.10.0**
apache-airflow-providers-common-compat==1.9.0
apache-airflow-providers-common-io==1.6.0
apache-airflow-providers-common-sql==1.27.1
apache-airflow-providers-docker==4.4.5
apache-airflow-providers-fab==1.5.3
apache-airflow-providers-ftp==3.13.0
apache-airflow-providers-github==2.9.4
apache-airflow-providers-grpc==3.8.3
apache-airflow-providers-http==5.3.0
apache-airflow-providers-imap==3.9.0
apache-airflow-providers-microsoft-winrm==3.12.0
apache-airflow-providers-mongo==5.2.3
apache-airflow-providers-mysql==6.3.5
apache-airflow-providers-opsgenie==5.9.3
apache-airflow-providers-postgres==6.4.1
apache-airflow-providers-redis==4.3.3
apache-airflow-providers-samba==4.11.1
apache-airflow-providers-sftp==5.4.2
apache-airflow-providers-smtp==2.1.0
apache-airflow-providers-sqlite==4.1.0
apache-airflow-providers-ssh==4.1.6
apache-airflow-providers-standard==1.9.2
### Apache Airflow version
2.11.0
### Operating System
Debian GNU/Linux 12 (bookworm)
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
_No response_
### What happened
Due to New feature in version 10.10.0: KubernetesPodTriggerer reads pod logs instead of KubernetesPodOperator (#57531)
Now airflow triggerer will read logs from airflow, but I suspect part of log read also listing Kubernetes events.
Meanwhile airflow-airflow-pod-log-reader-role created by helm chart does have list events permission.
Earlier version works well because log read was handled by KubernetesPodOperator which run in worker that uses airflow-airflow-pod-launcher-role and has list events permission.
### What you think should happen instead
We should fix helm chart template for airflow-airflow-pod-log-reader-role
### How to reproduce
Create a dag with a task using KubernetesPodOperator, make it defferable=true.
We will get error like:
kubernetes_asyncio.client.exceptions.ApiException: (403)
Reason: Forbidden
HTTP response headers: <CIMultiDictProxy('Audit-Id': '63f99090-5c09-45e5-9716-76cb8cb9e006', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': '92570511-6de3-4f0c-b903-ef06f04261e2', 'X-Kubernetes-Pf-Prioritylevel-Uid': '271a661f-e33f-4d2d-8831-5979edf49286', 'Date': 'Wed, 26 Nov 2025 16:40:58 GMT', 'Content-Length': '299')>
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"events is forbidden: User \"system:serviceaccount:airflow:airflow-triggerer\" cannot list resource \"events\" in API group \"\" in the namespace \"ra-batch\"","reason":"Forbidden","details":{"kind":"events"},"code":403}
### Anything else
Full stack of the error:
[2025-11-26, 10:42:08 CST] {taskinstance.py:3336} ERROR - Task failed with exception
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/models/taskinstance.py", line 776, in _execute_task
result = _execute_callable(context=context, **execute_callable_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/models/taskinstance.py", line 742, in _execute_callable
return ExecutionCallableRunner(
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/utils/operator_helpers.py", line 252, in run
return self.func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/models/baseoperator.py", line 1816, in resume_execution
return execute_callable(context)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/cncf/kubernetes/operators/pod.py", line 933, in trigger_reentry
raise AirflowException(message)
airflow.exceptions.AirflowException: Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/cncf/kubernetes/triggers/pod.py", line 158, in run
state = await self._wait_for_pod_start()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/cncf/kubernetes/triggers/pod.py", line 251, in _wait_for_pod_start
await asyncio.gather(pod_start_task, events_task)
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/cncf/kubernetes/utils/pod_manager.py", line 1013, in watch_pod_events
await watch_pod_events(pod_manager=self, pod=pod, check_interval=check_interval)
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/cncf/kubernetes/utils/pod_manager.py", line 122, in watch_pod_events
events = await pod_manager.read_pod_events(pod)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/asyncio/__init__.py", line 189, in async_wrapped
return await copy(fn, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/asyncio/__init__.py", line 111, in __call__
do = await self.iter(retry_state=retry_state)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/asyncio/__init__.py", line 153, in iter
result = await action(retry_state)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/_utils.py", line 99, in inner
return call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 420, in exc_check
raise retry_exc.reraise()
^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 187, in reraise
raise self.last_attempt.result()
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 449, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/asyncio/__init__.py", line 114, in __call__
result = await fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/cncf/kubernetes/utils/pod_manager.py", line 1006, in read_pod_events
return await self._hook.get_pod_events(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/cncf/kubernetes/hooks/kubernetes.py", line 950, in get_pod_events
events: CoreV1EventList = await v1_api.list_namespaced_event(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/kubernetes_asyncio/client/api_client.py", line 192, in __call_api
raise e
File "/home/airflow/.local/lib/python3.12/site-packages/kubernetes_asyncio/client/api_client.py", line 185, in __call_api
response_data = await self.request(
^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/kubernetes_asyncio/client/rest.py", line 212, in GET
return (await self.request("GET", url,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/kubernetes_asyncio/client/rest.py", line 206, in request
raise ApiException(http_resp=r)
kubernetes_asyncio.client.exceptions.ApiException: (403)
Reason: Forbidden
HTTP response headers: <CIMultiDictProxy('Audit-Id': '63f99090-5c09-45e5-9716-76cb8cb9e006', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': '92570511-6de3-4f0c-b903-ef06f04261e2', 'X-Kubernetes-Pf-Prioritylevel-Uid': '271a661f-e33f-4d2d-8831-5979edf49286', 'Date': 'Wed, 26 Nov 2025 16:40:58 GMT', 'Content-Length': '299')>
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"events is forbidden: User \"system:serviceaccount:airflow:airflow-triggerer\" cannot list resource \"events\" in API group \"\" in the namespace \"ra-batch\"","reason":"Forbidden","details":{"kind":"events"},"code":403}
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Latest KubernetesPodOperator (Ver: 10.10.0) get 403 error (Forbidden to list K8s Events) [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal closed issue #58736: Latest KubernetesPodOperator (Ver: 10.10.0) get 403 error (Forbidden to list K8s Events)
URL: https://github.com/apache/airflow/issues/58736
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Latest KubernetesPodOperator (Ver: 10.10.0) get 403 error (Forbidden to list K8s Events) [airflow]
Posted by "shivainu1413 (via GitHub)" <gi...@apache.org>.
shivainu1413 commented on issue #58736:
URL: https://github.com/apache/airflow/issues/58736#issuecomment-3588005354
Hi, I’m new to contributing to Airflow.
If this issue is still open, I’d like to work on it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Latest KubernetesPodOperator (Ver: 10.10.0) get 403 error (Forbidden to list K8s Events) [airflow]
Posted by "yovio-rca (via GitHub)" <gi...@apache.org>.
yovio-rca commented on issue #58736:
URL: https://github.com/apache/airflow/issues/58736#issuecomment-3589683087
Actually there is already a PR to fix this issue that already been merged but not released: https://github.com/apache/airflow/pull/56872
The author of the PR that causes this issue (https://github.com/apache/airflow/pull/57531) has mentioned that the above PR need to be release first (Helm chart change). However what actually happened was The solution merge but not release yet, and the PR that depends on it released.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Latest KubernetesPodOperator (Ver: 10.10.0) get 403 error (Forbidden to list K8s Events) [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #58736:
URL: https://github.com/apache/airflow/issues/58736#issuecomment-4369280434
> Actually there is already a PR to fix this issue that already been merged but not released: https://github.com/apache/airflow/pull/56872
Closing as fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org