You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/07/08 15:04:18 UTC
[GitHub] [airflow] duc-luong-tnsl opened a new issue #16888: Bug on KubernetesJobWatcher in Scheduler
duc-luong-tnsl opened a new issue #16888:
URL: https://github.com/apache/airflow/issues/16888
I got this issue on my scheduler with Airflow 2.1.0 run with Kubernetes.
The scheduler does not die but often gets an error log.
```
Process KubernetesJobWatcher-8:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/executors/kubernetes_executor.py", line 102, in run
self.resource_version = self._run(
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/executors/kubernetes_executor.py", line 152, in _run
'dag_id': annotations['dag_id'],
KeyError: 'dag_id'
```
IMPORTANT!!!
**Apache Airflow version**: 2.1.0
**Environment**:
- **Cloud provider or hardware configuration**:
- **OS** (e.g. from /etc/os-release): Ubuntu
- **Kernel** (e.g. `uname -a`): Linux airflow-v2-scheduler-66c999b4b9-nqr4p 4.19.112+ #1 SMP Fri Sep 4 12:00:04 PDT 2020 x86_64 GNU/Linux
- **Install tools**: Docker
- **Others**:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on issue #16888: Bug on KubernetesJobWatcher in Scheduler
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on issue #16888:
URL: https://github.com/apache/airflow/issues/16888#issuecomment-876779262
@duc-luong-tnsl, we watch pods with this label:
https://github.com/apache/airflow/blob/2.1.0/airflow/executors/kubernetes_executor.py#L129
In your scheduler log, right before the traceback, what pod name was logged? Does it looks like a kubernetesexecutor worker (e.g. dagidtaskid.random)?
https://github.com/apache/airflow/blob/2.1.0/airflow/executors/kubernetes_executor.py#L147
The only time I've seen this previously is when incorrect labels were added via `pod_mutation_hook` causing KPO pods to be considered as workers in the worker adoption logic, which then lead to this traceback. See https://github.com/astronomer/airflow-chart/pull/231.
Do you have a `pod_mutation_hook` that sets `kubernetes_executor`? Or happen to use the https://github.com/astronomer/airflow-chart chart?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] closed issue #16888: Bug on KubernetesJobWatcher in Scheduler
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed issue #16888:
URL: https://github.com/apache/airflow/issues/16888
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #16888: Bug on KubernetesJobWatcher in Scheduler
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #16888:
URL: https://github.com/apache/airflow/issues/16888#issuecomment-876515601
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] closed issue #16888: Bug on KubernetesJobWatcher in Scheduler
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed issue #16888:
URL: https://github.com/apache/airflow/issues/16888
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on issue #16888: Bug on KubernetesJobWatcher in Scheduler
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #16888:
URL: https://github.com/apache/airflow/issues/16888#issuecomment-905055628
This issue has been automatically marked as stale because it has been open for 30 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on issue #16888: Bug on KubernetesJobWatcher in Scheduler
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #16888:
URL: https://github.com/apache/airflow/issues/16888#issuecomment-909748839
This issue has been closed because it has not received response from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on issue #16888: Bug on KubernetesJobWatcher in Scheduler
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #16888:
URL: https://github.com/apache/airflow/issues/16888#issuecomment-909748839
This issue has been closed because it has not received response from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org