You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/02/03 02:09:37 UTC
[GitHub] [airflow] wolfier opened a new issue #21284: Container status for KubernetesPodOperator / KubernetesExecutor
wolfier opened a new issue #21284:
URL: https://github.com/apache/airflow/issues/21284
### Description
When a container fails, the reason is not presented is a clear manner.
For example, when the base container fails due to OOM, I would like to this some version of this printed in the task logs.
```
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: OOMKilled
Exit Code: 137
Started: Thu, 06 Jan 2022 12:03:22 -0500
Finished: Thu, 06 Jan 2022 12:03:46 -0500
Ready: False
Restart Count: 208
```
### Use case/motivation
It's difficult to determine why a pod fails when the logs only says it has an event of Failed or if the worker pod received a SIGTERM.
```
[2022-01-25, 20:13:24 UTC] {pod_launcher.py:236} INFO - Event: optimize-lh.0d0e03a1a72a4b0b8035cb49d0bd1de3 had an event of type Failed
[2022-01-25, 20:13:24 UTC] {pod_launcher.py:236} INFO - Event: optimize-lh.0d0e03a1a72a4b0b8035cb49d0bd1de3 had an event of type Failed
[2022-01-25, 20:13:25 UTC] {taskinstance.py:1267} INFO - Marking task as FAILED. dag_id=bin_optimizer, task_id=optimize_lh, execution_date=20220125T190000, start_date=20220125T201144, end_date=20220125T201325
[2022-01-25, 20:13:25 UTC] {local_task_job.py:154} INFO - Task exited with return code 1
[2022-01-25, 20:13:25 UTC] {local_task_job.py:264} INFO - 0 downstream tasks scheduled from follow-on schedule check
```
### Related issues
_No response_
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] raphaelauv commented on issue #21284: Container status for KubernetesPodOperator / KubernetesExecutor
Posted by GitBox <gi...@apache.org>.
raphaelauv commented on issue #21284:
URL: https://github.com/apache/airflow/issues/21284#issuecomment-1030712507
With the version 3.0.0 of the cncf provider it's perfectly clear
```log
'status': {'conditions': None,
'container_statuses': None,
'host_ip': None,
'init_container_statuses': None,
'message': 'Pod ephemeral local storage usage exceeds the total '
'limit of containers 1G. ',
'nominated_node_name': None,
'phase': 'Failed',
'pod_ip': None,
'qos_class': None,
'reason': 'Evicted',
'start_time': datetime.datetime(2022, 2, 5, 10, 33, 46, tzinfo=tzlocal())}}
[2022-02-05 10:54:07,601] {taskinstance.py:1511} INFO - Marking task as FAILED. dag_id=XXXXXXXXXXXX, task_id=XXXXXXXXXXXX, execution_date=20220204T000000, start_date=20220205T103341, end_date=20220205T105407
[2022-02-05 10:54:07,779] {local_task_job.py:151} INFO - Task exited with return code 1
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] wolfier commented on issue #21284: Container status for KubernetesPodOperator / KubernetesExecutor
Posted by GitBox <gi...@apache.org>.
wolfier commented on issue #21284:
URL: https://github.com/apache/airflow/issues/21284#issuecomment-1028537456
I believe it should be an easy additional (at least for the KubernetesPodOperator) since the information is part of the V1Pod class, `V1Pod.status.container_statuses[].state.terminated.reason`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] wolfier edited a comment on issue #21284: Container status for KubernetesPodOperator / KubernetesExecutor
Posted by GitBox <gi...@apache.org>.
wolfier edited a comment on issue #21284:
URL: https://github.com/apache/airflow/issues/21284#issuecomment-1028537456
I believe it should be an easy additional (at least for the KubernetesPodOperator) since the information is part of the V1Pod class, `V1Pod.status.container_statuses[].state.terminated.reason`.
The change would most likely be in the [cleanup section](https://github.com/apache/airflow/blob/main/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py#L398).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] wolfier edited a comment on issue #21284: Container status for KubernetesPodOperator / KubernetesExecutor
Posted by GitBox <gi...@apache.org>.
wolfier edited a comment on issue #21284:
URL: https://github.com/apache/airflow/issues/21284#issuecomment-1028537456
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org