You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/05/20 04:42:36 UTC
[GitHub] [airflow] yogyang opened a new issue #15960: K8s executor failed to manual re-run the task
yogyang opened a new issue #15960:
URL: https://github.com/apache/airflow/issues/15960
<!--
IMPORTANT!!!
PLEASE CHECK "SIMILAR TO X EXISTING ISSUES" OPTION IF VISIBLE
NEXT TO "SUBMIT NEW ISSUE" BUTTON!!!
PLEASE CHECK IF THIS ISSUE HAS BEEN REPORTED PREVIOUSLY USING SEARCH!!!
Please complete the next sections or the issue will be closed.
These questions are the first thing we need to know to understand the context.
-->
**Apache Airflow version**: 2.0.2
**Kubernetes version (if you are using kubernetes)** (use `kubectl version`):
```
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.7", GitCommit:"1dd5338295409edcfff11505e7bb246f0d325d15", GitTreeState:"clean", BuildDate:"2021-01-13T13:23:52Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.4", GitCommit:"8d8aa39598534325ad77120c120a22b3a990b5ea", GitTreeState:"clean", BuildDate:"2020-03-12T20:55:23Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}
```
**Environment**:
- **Cloud provider or hardware configuration**: rancher ec2
- **OS** (e.g. from /etc/os-release): Debian GNU/Linux 10 (buster)
- **Kernel** (e.g. `uname -a`): Linux airflow-webserver-57ccd474bc-8rfg5 5.4.0-1045-aws #47-Ubuntu SMP Tue Apr 13 07:02:25 UTC 2021 x86_64 GNU/Linux
- **Install tools**:
- **Others**: K8s executor mode
**What happened**:
click task -> run ignore state -> Task would not be executed
webserver gives error:
```
WARNING - ApiException when attempting to run task, re-queueing. Message: pods is forbidden: User "system:serviceaccount:production-airflow:airflow-webserver" cannot create resource "pods" in API group "" in the namespace "production-airflow"
[2021-05-20 04:28:07,532] {kubernetes_executor.py:275} INFO - Kubernetes job is (TaskInstanceKey xxxxx)
[2021-05-20 04:28:07,534] {pod_launcher.py:86} ERROR - Exception when attempting to create Namespaced Pod:
"metadata": {
"annotations": {
"dag_id": "xxxx",
"task_id": "xxxx",
"execution_date": "2021-05-20T03:00:00+00:00",
"try_number": "3",
"ad.datadoghq.com/tags": "{ \"type\": \"job\",\"task\": \"xx.xx\" }"
},
"labels": {
"airflow-worker": "manual",
"dag_id": "xxx",
"task_id": "xxx",
"execution_date": "2021-05-20T03_00_00_plus_00_00",
"try_number": "3",
"airflow_version": "2.0.2",
"kubernetes_executor": "True"
},
"name": "xxxx.b2bd66df4f3b44198d7f23cdcaae07d1",
"namespace": "production-airflow"
},
"spec": {...}
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/kubernetes/pod_launcher.py", line 82, in run_pod_async
body=sanitized_pod, namespace=pod.metadata.namespace, **kwargs
File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/api/core_v1_api.py", line 6174, in create_namespaced_pod
(data) = self.create_namespaced_pod_with_http_info(namespace, body, **kwargs) # noqa: E501
File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/api/core_v1_api.py", line 6265, in create_namespaced_pod_with_http_info
collection_formats=collection_formats)
File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 345, in call_api
_preload_content, _request_timeout)
File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 176, in __call_api
_request_timeout=_request_timeout)
File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 388, in request
body=body)
File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/rest.py", line 278, in POST
body=body)
File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/rest.py", line 231, in request
raise ApiException(http_resp=r)
kubernetes.client.rest.ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'Date': 'Thu, 20 May 2021 04:28:07 GMT', 'Content-Length': '316'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods is forbidden: User \"system:serviceaccount:production-airflow:airflow-webserver\" cannot create resource \"pods\" in API group \"\" in the namespace \"production-airflow\"","reason":"Forbidden","details":{"kind":"pods"},"code":403}
```
<!-- (please include exact error messages if you can) -->
**What you expected to happen**:
Task can be ran
<!-- What do you think went wrong? -->
**How to reproduce it**:
set up a K8s airflow, manually trigger task run by task -> run (Ignore state)
<!---
As minimally and precisely as possible. Keep in mind we do not have access to your cluster or dags.
If you are using kubernetes, please attempt to recreate the issue using minikube or kind.
## Install minikube/kind
- Minikube https://minikube.sigs.k8s.io/docs/start/
- Kind https://kind.sigs.k8s.io/docs/user/quick-start/
If this is a UI bug, please provide a screenshot of the bug or a link to a youtube video of the bug in action
You can include images using the .md style of
To record a screencast, mac users can use QuickTime and then create an unlisted youtube video with the resulting .mov file.
--->
**Anything else we need to know**:
<!--
How often does this problem occur? Once? Every time etc?
Any relevant logs to include? Put them here in side a detail tag:
<details><summary>x.log</summary> lots of stuff </details>
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] yogyang commented on issue #15960: K8s executor failed to manual re-run the task
Posted by GitBox <gi...@apache.org>.
yogyang commented on issue #15960:
URL: https://github.com/apache/airflow/issues/15960#issuecomment-973904755
upgrade to 2.1.2, seems not this problem now. close it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] yogyang commented on issue #15960: K8s executor failed to manual re-run the task
Posted by GitBox <gi...@apache.org>.
yogyang commented on issue #15960:
URL: https://github.com/apache/airflow/issues/15960#issuecomment-844710173
add rolebinding to serviceaccount:airflow-webserver would result in another problem: the pod airlfow-webserver created is not using pod templated defined in the `podTemplate` file
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] yogyang edited a comment on issue #15960: K8s executor failed to manual re-run the task
Posted by GitBox <gi...@apache.org>.
yogyang edited a comment on issue #15960:
URL: https://github.com/apache/airflow/issues/15960#issuecomment-844710173
add rolebinding to serviceaccount:airflow-webserver would result in another problem: the pod airlfow-webserver created is not using pod templated defined in the `pod_template_file` file
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] yogyang closed issue #15960: K8s executor failed to manual re-run the task
Posted by GitBox <gi...@apache.org>.
yogyang closed issue #15960:
URL: https://github.com/apache/airflow/issues/15960
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] yogyang edited a comment on issue #15960: K8s executor failed to manual re-run the task
Posted by GitBox <gi...@apache.org>.
yogyang edited a comment on issue #15960:
URL: https://github.com/apache/airflow/issues/15960#issuecomment-844710173
add rolebinding to serviceaccount:airflow-webserver would result in another problem: the pod airflow-webserver created is not using pod templated defined in the `pod_template_file` file, as the helm did not mount the pod_template_file for web-server pod by default.
Need to set webserver extratVolumeMout to mount the pod_template_file
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] yogyang edited a comment on issue #15960: K8s executor failed to manual re-run the task
Posted by GitBox <gi...@apache.org>.
yogyang edited a comment on issue #15960:
URL: https://github.com/apache/airflow/issues/15960#issuecomment-844710173
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org