You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/04/22 18:05:05 UTC
[GitHub] [airflow] MatthewRBruce commented on pull request #15490: Fix unsuccessful KubernetesPod final_state call when `is_delete_operator_pod=True`
MatthewRBruce commented on pull request #15490:
URL: https://github.com/apache/airflow/pull/15490#issuecomment-825070171
If anyone's interesting here the logs for a run pre-change:
```
[2021-04-22 14:28:55,336] {pod_launcher.py:177} INFO - Event: hello-world-pod-fail.0c8adcbf2e15483d8511404d771eed12 had an event of type Pending
[2021-04-22 14:28:55,336] {pod_launcher.py:113} WARNING - Pod not yet started: hello-world-pod-fail.0c8adcbf2e15483d8511404d771eed12
[2021-04-22 14:28:56,344] {pod_launcher.py:177} INFO - Event: hello-world-pod-fail.0c8adcbf2e15483d8511404d771eed12 had an event of type Pending
[2021-04-22 14:28:56,345] {pod_launcher.py:113} WARNING - Pod not yet started: hello-world-pod-fail.0c8adcbf2e15483d8511404d771eed12
[2021-04-22 14:28:57,354] {pod_launcher.py:177} INFO - Event: hello-world-pod-fail.0c8adcbf2e15483d8511404d771eed12 had an event of type Pending
[2021-04-22 14:28:57,354] {pod_launcher.py:113} WARNING - Pod not yet started: hello-world-pod-fail.0c8adcbf2e15483d8511404d771eed12
[2021-04-22 14:28:58,363] {pod_launcher.py:177} INFO - Event: hello-world-pod-fail.0c8adcbf2e15483d8511404d771eed12 had an event of type Failed
[2021-04-22 14:28:58,364] {pod_launcher.py:287} ERROR - Event with job id hello-world-pod-fail.0c8adcbf2e15483d8511404d771eed12 Failed
[2021-04-22 14:28:59,397] {pod_launcher.py:177} INFO - Event: hello-world-pod-fail.0c8adcbf2e15483d8511404d771eed12 had an event of type Failed
[2021-04-22 14:28:59,397] {pod_launcher.py:287} ERROR - Event with job id hello-world-pod-fail.0c8adcbf2e15483d8511404d771eed12 Failed
[2021-04-22 14:28:59,405] {pod_launcher.py:177} INFO - Event: hello-world-pod-fail.0c8adcbf2e15483d8511404d771eed12 had an event of type Failed
[2021-04-22 14:28:59,405] {pod_launcher.py:287} ERROR - Event with job id hello-world-pod-fail.0c8adcbf2e15483d8511404d771eed12 Failed
[2021-04-22 14:28:59,430] {taskinstance.py:1482} ERROR - Task failed with exception
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1138, in _run_raw_task
self._prepare_and_execute_task_with_callbacks(context, task)
File "/usr/local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1311, in _prepare_and_execute_task_with_callbacks
result = self._execute_task(context, task_copy)
File "/usr/local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1341, in _execute_task
result = task_copy.execute(context=context)
File "/usr/local/lib/python3.8/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 341, in execute
status = self.client.read_namespaced_pod(self.pod.metadata.name, self.namespace)
File "/usr/local/lib/python3.8/site-packages/kubernetes/client/apis/core_v1_api.py", line 18446, in read_namespaced_pod
(data) = self.read_namespaced_pod_with_http_info(name, namespace, **kwargs)
File "/usr/local/lib/python3.8/site-packages/kubernetes/client/apis/core_v1_api.py", line 18524, in read_namespaced_pod_with_http_info
return self.api_client.call_api('/api/v1/namespaces/{namespace}/pods/{name}', 'GET',
File "/usr/local/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 330, in call_api
return self.__call_api(resource_path, method,
File "/usr/local/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 163, in __call_api
response_data = self.request(method, url,
File "/usr/local/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 351, in request
return self.rest_client.GET(url,
File "/usr/local/lib/python3.8/site-packages/kubernetes/client/rest.py", line 227, in GET
return self.request("GET", url,
File "/usr/local/lib/python3.8/site-packages/kubernetes/client/rest.py", line 222, in request
raise ApiException(http_resp=r)
kubernetes.client.rest.ApiException: (404)
Reason: Not Found
```
and post change:
```
[2021-04-22 15:24:16,494] {pod_launcher.py:177} INFO - Event: hello-world-pod-fail.2f52f1aa0a2c447ab187626bec759e55 had an event of type Pending
[2021-04-22 15:24:16,494] {pod_launcher.py:113} WARNING - Pod not yet started: hello-world-pod-fail.2f52f1aa0a2c447ab187626bec759e55
[2021-04-22 15:24:17,503] {pod_launcher.py:177} INFO - Event: hello-world-pod-fail.2f52f1aa0a2c447ab187626bec759e55 had an event of type Pending
[2021-04-22 15:24:17,503] {pod_launcher.py:113} WARNING - Pod not yet started: hello-world-pod-fail.2f52f1aa0a2c447ab187626bec759e55
[2021-04-22 15:24:18,510] {pod_launcher.py:177} INFO - Event: hello-world-pod-fail.2f52f1aa0a2c447ab187626bec759e55 had an event of type Pending
[2021-04-22 15:24:18,511] {pod_launcher.py:113} WARNING - Pod not yet started: hello-world-pod-fail.2f52f1aa0a2c447ab187626bec759e55
[2021-04-22 15:24:19,520] {pod_launcher.py:177} INFO - Event: hello-world-pod-fail.2f52f1aa0a2c447ab187626bec759e55 had an event of type Failed
[2021-04-22 15:24:19,520] {pod_launcher.py:287} ERROR - Event with job id hello-world-pod-fail.2f52f1aa0a2c447ab187626bec759e55 Failed
[2021-04-22 15:24:20,554] {pod_launcher.py:177} INFO - Event: hello-world-pod-fail.2f52f1aa0a2c447ab187626bec759e55 had an event of type Failed
[2021-04-22 15:24:20,554] {pod_launcher.py:287} ERROR - Event with job id hello-world-pod-fail.2f52f1aa0a2c447ab187626bec759e55 Failed
[2021-04-22 15:24:20,563] {pod_launcher.py:177} INFO - Event: hello-world-pod-fail.2f52f1aa0a2c447ab187626bec759e55 had an event of type Failed
[2021-04-22 15:24:20,563] {pod_launcher.py:287} ERROR - Event with job id hello-world-pod-fail.2f52f1aa0a2c447ab187626bec759e55 Failed
[2021-04-22 15:24:20,598] {taskinstance.py:1482} ERROR - Task failed with exception
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 347, in execute
raise AirflowException(f'Pod {self.pod.metadata.name} returned a failure: {status}')
airflow.exceptions.AirflowException: Pod hello-world-pod-fail.2f52f1aa0a2c447ab187626bec759e55 returned a failure: failed
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1138, in _run_raw_task
self._prepare_and_execute_task_with_callbacks(context, task)
File "/usr/local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1311, in _prepare_and_execute_task_with_callbacks
result = self._execute_task(context, task_copy)
File "/usr/local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1341, in _execute_task
result = task_copy.execute(context=context)
File "/usr/local/lib/python3.8/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 350, in execute
raise AirflowException(f'Pod Launching failed: {ex}')
airflow.exceptions.AirflowException: Pod Launching failed: Pod hello-world-pod-fail.2f52f1aa0a2c447ab187626bec759e55 returned a failure: failed
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org