You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/09/30 09:11:09 UTC
[GitHub] [airflow] yamrzou opened a new issue #11201: Timeouts in KubernetesPodOperator
yamrzou opened a new issue #11201:
URL: https://github.com/apache/airflow/issues/11201
I have a number of long running KubernetesPodOperator tasks, which fail after around two hours with the following error.
```
[2020-09-30 05:44:27,065] {pod_launcher.py:173} INFO - Event: run-processing-ec9d835e49604024aa90b5867b1797a0 had an event of type Running
[2020-09-30 05:44:27,065] {pod_launcher.py:166} INFO - Pod run-processing-ec9d835e49604024aa90b5867b1797a0 has state running
[2020-09-30 05:44:29,152] {pod_launcher.py:173} INFO - Event: run-processing-ec9d835e49604024aa90b5867b1797a0 had an event of type Running
[2020-09-30 05:44:29,153] {pod_launcher.py:166} INFO - Pod run-processing-ec9d835e49604024aa90b5867b1797a0 has state running
[2020-09-30 05:44:29,923] {timeout.py:42} ERROR - Process timed out, PID: 12846
[2020-09-30 05:44:30,060] {taskinstance.py:1150} ERROR - Pod Launching failed: Pod Launching failed: Timeout, PID: 12846
Traceback (most recent call last):
File "/opt/bitnami/airflow/venv/lib/python3.6/site-packages/airflow/contrib/operators/kubernetes_pod_operator.py", line 403, in create_new_pod_for_operator
final_state, result = launcher.monitor_pod(pod=pod, get_logs=self.get_logs)
File "/opt/bitnami/airflow/venv/lib/python3.6/site-packages/airflow/kubernetes/pod_launcher.py", line 167, in monitor_pod
time.sleep(2)
File "/opt/bitnami/airflow/venv/lib/python3.6/site-packages/airflow/utils/timeout.py", line 43, in handle_timeout
raise AirflowTaskTimeout(self.error_message)
airflow.exceptions.AirflowTaskTimeout: Timeout, PID: 12846
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/bitnami/airflow/venv/lib/python3.6/site-packages/airflow/contrib/operators/kubernetes_pod_operator.py", line 284, in execute
final_state, _, result = self.create_new_pod_for_operator(labels, launcher)
File "/opt/bitnami/airflow/venv/lib/python3.6/site-packages/airflow/contrib/operators/kubernetes_pod_operator.py", line 408, in create_new_pod_for_operator
raise AirflowException('Pod Launching failed: {error}'.format(error=ex))
airflow.exceptions.AirflowException: Pod Launching failed: Timeout, PID: 12846
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/bitnami/airflow/venv/lib/python3.6/site-packages/airflow/models/taskinstance.py", line 979, in _run_raw_task
result = task_copy.execute(context=context)
File "/opt/bitnami/airflow/venv/lib/python3.6/site-packages/airflow/contrib/operators/kubernetes_pod_operator.py", line 290, in execute
raise AirflowException('Pod Launching failed: {error}'.format(error=ex))
airflow.exceptions.AirflowException: Pod Launching failed: Pod Launching failed: Timeout, PID: 12846
[2020-09-30 05:44:30,062] {taskinstance.py:1194} INFO - Marking task as FAILED.
```
Airflow Version: 1.10.12
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] yamrzou commented on issue #11201: Timeouts in KubernetesPodOperator
Posted by GitBox <gi...@apache.org>.
yamrzou commented on issue #11201:
URL: https://github.com/apache/airflow/issues/11201#issuecomment-744316800
Closing, it turns out this was due to an `execution_timeout` present in the dag. A more explicit message with the cause would have been more helpful than the current `ERROR - Process timed out`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] yamrzou closed issue #11201: Timeouts in KubernetesPodOperator
Posted by GitBox <gi...@apache.org>.
yamrzou closed issue #11201:
URL: https://github.com/apache/airflow/issues/11201
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org