You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "victor-perchwell (via GitHub)" <gi...@apache.org> on 2023/02/03 20:03:38 UTC
[GitHub] [airflow] victor-perchwell opened a new issue, #29365: Airflow Wokers Trying to Create Pods in Default Namespace
victor-perchwell opened a new issue, #29365:
URL: https://github.com/apache/airflow/issues/29365
### Official Helm Chart version
1.7.0 (latest released)
### Apache Airflow version
2.4.1
### Kubernetes Version
v1.26.1
### Helm Chart configuration
_No response_
### Docker Image customizations
_No response_
### What happened
After installing the official airflow helm chart and trying to run a DAG, the airflow workers attempt to create pods in the default namespace instead of the airflow namespace, and they don't have permission for that.
Here's the output from the worker logs:
```
[2023-02-03, 18:31:30 UTC] {taskinstance.py:1383} INFO - Executing <Task(KubernetesPodOperator): passing-task> on 2023-02-03 18:31:24.873405+00:00
[2023-02-03, 18:31:30 UTC] {standard_task_runner.py:54} INFO - Started process 78 to run task
[2023-02-03, 18:31:30 UTC] {standard_task_runner.py:82} INFO - Running: ['airflow', 'tasks', 'run', 'kube_sample', 'passing-task', 'manual__2023-02-03T18:31:24.873405+00:00', '--job-id', '4', '--raw', '--subdir', 'DAGS_FOLDER/kube_sample_dag.py', '--cfg-path', '/tmp/tmp_yzq00jr']
[2023-02-03, 18:31:30 UTC] {standard_task_runner.py:83} INFO - Job 4: Subtask passing-task
[2023-02-03, 18:31:30 UTC] {dagbag.py:525} INFO - Filling up the DagBag from /opt/airflow/dags/kube_sample_dag.py
[2023-02-03, 18:31:31 UTC] {task_command.py:384} INFO - Running <TaskInstance: kube_sample.passing-task manual__2023-02-03T18:31:24.873405+00:00 [running]> on host airflow-worker-0.airflow-worker.airflow.svc.cluster.local
[2023-02-03, 18:31:32 UTC] {taskinstance.py:1592} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_EMAIL=airflow@example.com
AIRFLOW_CTX_DAG_OWNER=airflow
AIRFLOW_CTX_DAG_ID=kube_sample
AIRFLOW_CTX_TASK_ID=passing-task
AIRFLOW_CTX_EXECUTION_DATE=2023-02-03T18:31:24.873405+00:00
AIRFLOW_CTX_TRY_NUMBER=1
AIRFLOW_CTX_DAG_RUN_ID=manual__2023-02-03T18:31:24.873405+00:00
[2023-02-03, 18:31:32 UTC] {kubernetes_pod.py:587} INFO - Creating pod passing-test-6055b8067ab345bdb6d46939603268fd with labels: {'dag_id': 'kube_sample', 'task_id': 'passing-task', 'run_id': 'manual__2023-02-03T183124.8734050000-88eab1eb0', 'kubernetes_pod_operator': 'True', 'try_number': '1'}
[2023-02-03, 18:31:32 UTC] {taskinstance.py:1851} ERROR - Task failed with exception
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 419, in execute
context=context,
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 387, in get_or_create_pod
pod = self.find_pod(self.namespace or pod_request_obj.metadata.namespace, context=context)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 371, in find_pod
label_selector=label_selector,
File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/api/core_v1_api.py", line 15697, in list_namespaced_pod
return self.list_namespaced_pod_with_http_info(namespace, **kwargs) # noqa: E501
File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/api/core_v1_api.py", line 15826, in list_namespaced_pod_with_http_info
collection_formats=collection_formats)
File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 353, in call_api
_preload_content, _request_timeout, _host)
File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 184, in __call_api
_request_timeout=_request_timeout)
File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 377, in request
headers=headers)
File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/rest.py", line 244, in GET
query_params=query_params)
File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/rest.py", line 234, in request
raise ApiException(http_resp=r)
kubernetes.client.exceptions.ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': '6c8ef6d6-b713-4080-8872-22377eeb9143', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': 'ec584bbc-619e-4370-ae0c-98b9e6e4b4d1', 'X-Kubernetes-Pf-Prioritylevel-Uid': '9cc2b786-09db-4bfc-85ec-4155174d6446', 'Date': 'Fri, 03 Feb 2023 18:31:32 GMT', 'Content-Length': '289'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods is forbidden: User \"system:serviceaccount:airflow:airflow-worker\" cannot list resource \"pods\" in API group \"\" in the namespace \"default\"","reason":"Forbidden","details":{"kind":"pods"},"code":403}
```
### What you think should happen instead
The airflow worker should instead attempt to create pods in the airflow namespace.
### How to reproduce
Install the official helm chart listed here: https://airflow.apache.org/docs/helm-chart/stable/index.html
Attempt to execute a DAG via the UI.
### Anything else
I checked role bindings and they appear to exists:
```
~ ❯ kubectl get rolebindings,clusterrolebindings --all-namespaces -o wide | grep airflow 14:04:07
airflow rolebinding.rbac.authorization.k8s.io/airflow-pod-launcher-rolebinding Role/airflow-pod-launcher-role 36m airflow/airflow-worker
airflow rolebinding.rbac.authorization.k8s.io/airflow-pod-log-reader-rolebinding Role/airflow-pod-log-reader-role 36m airflow/airflow-webserver, airflow/airflow-triggerer
```
Checking via kubectl:
```
~ ❯ kubectl auth can-i get pods --as system:serviceaccount:airflow:airflow-webserver --namespace airflow
Yes
~ ❯ kubectl auth can-i get pods --as system:serviceaccount:airflow:airflow-webserver --namespace airflow
No
```
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] L022937 commented on issue #29365: Airflow Workers Trying to Create Pods in Default Namespace
Posted by "L022937 (via GitHub)" <gi...@apache.org>.
L022937 commented on issue #29365:
URL: https://github.com/apache/airflow/issues/29365#issuecomment-1416797708
This could be due to multi_namespace_mode having true
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #29365: Airflow Wokers Trying to Create Pods in Default Namespace
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #29365:
URL: https://github.com/apache/airflow/issues/29365#issuecomment-1416344158
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] victor-perchwell commented on issue #29365: Airflow Workers Trying to Create Pods in Default Namespace
Posted by "victor-perchwell (via GitHub)" <gi...@apache.org>.
victor-perchwell commented on issue #29365:
URL: https://github.com/apache/airflow/issues/29365#issuecomment-1421364042
> what do you get on running this? use your scheduler pod name
>
> ` kubectl exec -c scheduler [scheduler_pod_name] -- /bin/bash -c "airflow config get-value kubernetes namespace"`
Here is the response with and without specifying the airflow namespace:
```
~ ❯ kubectl exec -c scheduler airflow-scheduler-bb55bf4cc-95lm2 --namespace airflow -- /bin/bash -c "airflow config get-value kubernetes namespace"
airflow
~ ❯ kubectl exec -c scheduler airflow-scheduler-bb55bf4cc-95lm2 -- /bin/bash -c "airflow config get-value kubernetes namespace" 14:58:20
Error from server (NotFound): pods "airflow-scheduler-bb55bf4cc-95lm2" not found
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] arjunanan6 commented on issue #29365: Airflow Workers Trying to Create Pods in Default Namespace
Posted by "arjunanan6 (via GitHub)" <gi...@apache.org>.
arjunanan6 commented on issue #29365:
URL: https://github.com/apache/airflow/issues/29365#issuecomment-1427383627
multiNamespaceMode is set to false by default in the helm parameters. Since you're using the KubernetesPodOperator, does it make any difference at all if you specify the namespace within your task? Take a look at [this](https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/_api/airflow/providers/cncf/kubernetes/operators/kubernetes_pod/index.html#airflow.providers.cncf.kubernetes.operators.kubernetes_pod.KubernetesPodOperator).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] victor-perchwell commented on issue #29365: Airflow Workers Trying to Create Pods in Default Namespace
Posted by "victor-perchwell (via GitHub)" <gi...@apache.org>.
victor-perchwell commented on issue #29365:
URL: https://github.com/apache/airflow/issues/29365#issuecomment-1421367143
> This could be due to multi_namespace_mode having true
It looks like multi_namespace_mode is set to false in values.yaml:
```
multi_namespace_mode: '{{ ternary "True" "False" .Values.multiNamespaceMode }}'
# Whether Airflow can launch workers and/or pods in multiple namespaces
# If true, it creates ClusterRole/ClusterRolebinding (with access to entire cluster)
multiNamespaceMode: false
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] snjypl commented on issue #29365: Airflow Workers Trying to Create Pods in Default Namespace
Posted by "snjypl (via GitHub)" <gi...@apache.org>.
snjypl commented on issue #29365:
URL: https://github.com/apache/airflow/issues/29365#issuecomment-1416437531
what do you get on running this? use your scheduler pod name
` kubectl exec -c scheduler [scheduler_pod_name] -- /bin/bash -c "airflow config get-value kubernetes namespace"`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] justabaka commented on issue #29365: Airflow Workers Trying to Create Pods in Default Namespace
Posted by "justabaka (via GitHub)" <gi...@apache.org>.
justabaka commented on issue #29365:
URL: https://github.com/apache/airflow/issues/29365#issuecomment-1435988327
I have a similar issue but in my case the namespace name is a correct one. No signs of the proper RBAC for the scheduler SA though:
`pods is forbidden: User \"system:serviceaccount:airflow:airflow-scheduler\" cannot list resource \"pods\" in API group \"\" in the namespace \"airflow\"`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #29365: Airflow Workers Trying to Create Pods in Default Namespace
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #29365: Airflow Workers Trying to Create Pods in Default Namespace
URL: https://github.com/apache/airflow/issues/29365
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org