[GitHub] [airflow] hsnprsd commented on issue #12136: KubernetesPodOperator breaks with active log-collection for long running tasks

[GitBox <gi...@apache.org>]

hsnprsd commented on issue #12136:
URL: https://github.com/apache/airflow/issues/12136#issuecomment-864388380


   We are having the same issue. Here is the log:
   ```
   [2021-06-19 09:12:08,519] {taskinstance.py:1482} ERROR - Task failed with exception
   Traceback (most recent call last):
     File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/response.py", line 696, in _update_chunk_length
       self.chunk_left = int(line, 16)
   ValueError: invalid literal for int() with base 16: b''
   
   During handling of the above exception, another exception occurred:
   
   Traceback (most recent call last):
     File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/response.py", line 436, in _error_catcher
       yield
     File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/response.py", line 763, in read_chunked
       self._update_chunk_length()
     File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/response.py", line 700, in _update_chunk_length
       raise httplib.IncompleteRead(line)
   http.client.IncompleteRead: IncompleteRead(0 bytes read)
   
   During handling of the above exception, another exception occurred:
   
   Traceback (most recent call last):
     File "/home/airflow/.local/lib/python3.6/site-packages/airflow/models/taskinstance.py", line 1138, in _run_raw_task
       self._prepare_and_execute_task_with_callbacks(context, task)
     File "/home/airflow/.local/lib/python3.6/site-packages/airflow/models/taskinstance.py", line 1311, in _prepare_and_execute_task_with_callbacks
       result = self._execute_task(context, task_copy)
     File "/home/airflow/.local/lib/python3.6/site-packages/airflow/models/taskinstance.py", line 1336, in _execute_task
       result = task_copy.execute(context=context)
     File "/home/airflow/.local/lib/python3.6/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 361, in execute
       final_state, _, result = self.create_new_pod_for_operator(labels, launcher)
     File "/home/airflow/.local/lib/python3.6/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 508, in create_new_pod_for_operator
       final_state, result = launcher.monitor_pod(pod=self.pod, get_logs=self.get_logs)
     File "/home/airflow/.local/lib/python3.6/site-packages/airflow/kubernetes/pod_launcher.py", line 133, in monitor_pod
       for line in logs:
     File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/response.py", line 807, in __iter__
       for chunk in self.stream(decode_content=True):
     File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/response.py", line 571, in stream
       for line in self.read_chunked(amt, decode_content=decode_content):
     File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/response.py", line 792, in read_chunked
       self._original_response.close()
     File "/usr/local/lib/python3.6/contextlib.py", line 99, in __exit__
       self.gen.throw(type, value, traceback)
     File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/response.py", line 454, in _error_catcher
       raise ProtocolError("Connection broken: %r" % e, e)
   urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org