You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/03/03 00:30:40 UTC
[GitHub] [airflow] ziliangpeng edited a comment on issue #13129: Reattach to kubernetes pod only if it's running
ziliangpeng edited a comment on issue #13129:
URL: https://github.com/apache/airflow/issues/13129#issuecomment-789324654
I got a similar issue where it tries to re-attach to a pod while it's initializing..
```
[2021-03-02 21:22:00,796] {taskinstance.py:1455} ERROR - (400)
Reason: Bad Request
HTTP response headers: HTTPHeaderDict({'Audit-Id': '23926992-dc19-4a0c-8d4b-8f17cdc91bd2', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Date': 'Tue, 02 Mar 2021 21:22:00 GMT', 'Content-Length': '280'})
HTTP response body: b'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"container \\"base\\" in pod \\"afj-prem-range-image-test-job-utils-20210302102909-dump-da7wf1s.0cf16bbe75a0401786514f8629164efe\\" is waiting to start: ContainerCreating","reason":"BadRequest","code":400}\n'
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1112, in _run_raw_task
self._prepare_and_execute_task_with_callbacks(context, task)
File "/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1285, in _prepare_and_execute_task_with_callbacks
result = self._execute_task(context, task_copy)
File "/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1315, in _execute_task
result = task_copy.execute(context=context)
File "/usr/local/lib/python3.7/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 335, in execute
labels, try_numbers_match, launcher, pod_list.items[0]
File "/usr/local/lib/python3.7/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 374, in handle_pod_overlap
final_state, result = self.monitor_launched_pod(launcher, pod)
File "/usr/local/lib/python3.7/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 513, in monitor_launched_pod
(final_state, result) = launcher.monitor_pod(pod, get_logs=self.get_logs)
File "/usr/local/lib/python3.7/site-packages/airflow/kubernetes/pod_launcher.py", line 132, in monitor_pod
logs = self.read_pod_logs(pod, timestamps=True, since_seconds=read_logs_since_sec)
File "/usr/local/lib/python3.7/site-packages/tenacity/__init__.py", line 329, in wrapped_f
return self.call(f, *args, **kw)
File "/usr/local/lib/python3.7/site-packages/tenacity/__init__.py", line 409, in call
do = self.iter(retry_state=retry_state)
File "/usr/local/lib/python3.7/site-packages/tenacity/__init__.py", line 368, in iter
raise retry_exc.reraise()
File "/usr/local/lib/python3.7/site-packages/tenacity/__init__.py", line 186, in reraise
raise self.last_attempt.result()
File "/usr/local/lib/python3.7/concurrent/futures/_base.py", line 428, in result
return self.__get_result()
File "/usr/local/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
raise self._exception
File "/usr/local/lib/python3.7/site-packages/tenacity/__init__.py", line 412, in call
result = fn(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/airflow/kubernetes/pod_launcher.py", line 222, in read_pod_logs
**additional_kwargs,
File "/usr/local/lib/python3.7/site-packages/kubernetes/client/api/core_v1_api.py", line 19199, in read_namespaced_pod_log
(data) = self.read_namespaced_pod_log_with_http_info(name, namespace, **kwargs) # noqa: E501
File "/usr/local/lib/python3.7/site-packages/kubernetes/client/api/core_v1_api.py", line 19305, in read_namespaced_pod_log_with_http_info
collection_formats=collection_formats)
File "/usr/local/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 345, in call_api
_preload_content, _request_timeout)
File "/usr/local/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 176, in __call_api
_request_timeout=_request_timeout)
File "/usr/local/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 366, in request
headers=headers)
File "/usr/local/lib/python3.7/site-packages/kubernetes/client/rest.py", line 241, in GET
query_params=query_params)
File "/usr/local/lib/python3.7/site-packages/kubernetes/client/rest.py", line 231, in request
raise ApiException(http_resp=r)
kubernetes.client.rest.ApiException: (400)
Reason: Bad Request
HTTP response headers: HTTPHeaderDict({'Audit-Id': '23926992-dc19-4a0c-8d4b-8f17cdc91bd2', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Date': 'Tue, 02 Mar 2021 21:22:00 GMT', 'Content-Length': '280'})
HTTP response body: b'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"container \\"base\\" in pod \\"afj-prem-range-image-test-job-utils-20210302102909-dump-da7wf1s.0cf16bbe75a0401786514f8629164efe\\" is waiting to start: ContainerCreating","reason":"BadRequest","code":400}\n'
```
A fix to this would be nice.
@rmanvar-indeed are you still working on a fix?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org