You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "marcdjohossou (via GitHub)" <gi...@apache.org> on 2024/08/02 15:03:20 UTC
Re: [I] 404 Pod not found can be seen often [airflow]
marcdjohossou commented on issue #37090:
URL: https://github.com/apache/airflow/issues/37090#issuecomment-2265601842
Hello,
We've started to see this issue after moving from Airflow 2.3.4 (apache-airflow-providers-cncf-kubernetes==5.0.0) to Airflow 2.5.3 (apache-airflow-providers-cncf-kubernetes==7.10.0)
Last seen on 2024-08-02
Sample log
--------------------------------------------
[2024-08-02, 13:33:09 UTC] {taskinstance.py:1778} ERROR - Task failed with exception
Traceback (most recent call last):
File "/opt/python3.8/lib/python3.8/site-packages/airflow/providers/cncf/kubernetes/operators/pod.py", line 613, in execute_sync
self.pod_manager.fetch_requested_container_logs(
File "/opt/python3.8/lib/python3.8/site-packages/airflow/providers/cncf/kubernetes/utils/pod_manager.py", line 537, in fetch_requested_container_logs
status = self.fetch_container_logs(pod=pod, container_name=c, follow=follow_logs)
File "/opt/python3.8/lib/python3.8/site-packages/airflow/providers/cncf/kubernetes/utils/pod_manager.py", line 463, in fetch_container_logs
if not self.container_is_running(pod, container_name=container_name):
File "/opt/python3.8/lib/python3.8/site-packages/airflow/providers/cncf/kubernetes/utils/pod_manager.py", line 595, in container_is_running
remote_pod = self.read_pod(pod)
File "/opt/python3.8/lib/python3.8/site-packages/tenacity/__init__.py", line 289, in wrapped_f
return self(f, *args, **kw)
File "/opt/python3.8/lib/python3.8/site-packages/tenacity/__init__.py", line 379, in __call__
do = self.iter(retry_state=retry_state)
File "/opt/python3.8/lib/python3.8/site-packages/tenacity/__init__.py", line 325, in iter
raise retry_exc.reraise()
File "/opt/python3.8/lib/python3.8/site-packages/tenacity/__init__.py", line 158, in reraise
raise self.last_attempt.result()
File "/opt/python3.8/lib/python3.8/concurrent/futures/_base.py", line 437, in result
return self.__get_result()
File "/opt/python3.8/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/opt/python3.8/lib/python3.8/site-packages/tenacity/__init__.py", line 382, in __call__
result = fn(*args, **kwargs)
File "/opt/python3.8/lib/python3.8/site-packages/airflow/providers/cncf/kubernetes/utils/pod_manager.py", line 668, in read_pod
return self._client.read_namespaced_pod(pod.metadata.name, pod.metadata.namespace)
File "/opt/python3.8/lib/python3.8/site-packages/kubernetes/client/api/core_v1_api.py", line 23483, in read_namespaced_pod
return self.read_namespaced_pod_with_http_info(name, namespace, **kwargs) # noqa: E501
File "/opt/python3.8/lib/python3.8/site-packages/kubernetes/client/api/core_v1_api.py", line 23570, in read_namespaced_pod_with_http_info
return self.api_client.call_api(
File "/opt/python3.8/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 348, in call_api
return self.__call_api(resource_path, method,
File "/opt/python3.8/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
response_data = self.request(
File "/opt/python3.8/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 373, in request
return self.rest_client.GET(url,
File "/opt/python3.8/lib/python3.8/site-packages/kubernetes/client/rest.py", line 240, in GET
return self.request("GET", url,
File "/opt/python3.8/lib/python3.8/site-packages/kubernetes/client/rest.py", line 234, in request
raise ApiException(http_resp=r)
kubernetes.client.exceptions.ApiException: (404)
Reason: Not Found
HTTP response headers: HTTPHeaderDict({'Audit-Id': '198ceff6-56c9-4388-bec9-66d259dae756', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '77910e62-75bc-4bc0-a56a-46b99420f62d', 'X-Kubernetes-Pf-Prioritylevel-Uid': '7079822d-2a41-4448-88c8-b6faf19a3948', 'Date': 'Fri, 02 Aug 2024 13:33:06 GMT', 'Content-Length': '228'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods \"run-dataflow-pipeline-i8f3gl7i\" not found","reason":"NotFound","details":{"name":"run-dataflow-pipeline-i8f3gl7i","kind":"pods"},"code":404}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org