[GitHub] [airflow] michalmisiewicz commented on a change in pull request #11325: KubernetesPodOperator should retry log tailing in case of interruption

[GitBox <gi...@apache.org>]

michalmisiewicz commented on a change in pull request #11325:
URL: https://github.com/apache/airflow/pull/11325#discussion_r501820421



##########
File path: airflow/kubernetes/pod_launcher.py
##########
@@ -124,9 +125,21 @@ def monitor_pod(self, pod: V1Pod, get_logs: bool) -> Tuple[State, Optional[str]]
         :return:  Tuple[State, Optional[str]]
         """
         if get_logs:
-            logs = self.read_pod_logs(pod)
-            for line in logs:
-                self.log.info(line)
+            read_logs_since_sec = None
+            while True:
+                logs = self.read_pod_logs(pod, since_seconds=read_logs_since_sec)
+                for line in logs:
+                    self.log.info(line)
+                curr_time = dt.now()
+                time.sleep(1)
+
+                if not self.base_container_is_running(pod):
+                    break
+
+                self.log.warning('Pod %s log read interrupted', pod.metadata.name)
+                delta = dt.now() - curr_time
+                # Prefer logs duplication rather than loss
+                read_logs_since_sec = math.ceil(delta.total_seconds())

Review comment:
       Good questions. `list_namespaced_pod` has option to prepend timestamp to log line. Example
   ```
   2020-10-08T14:16:17.793417674Z 172.32.2.36 - - [08/Oct/2020:14:16:17 +0000] "GET /health HTTP/1.1" 200 187 "-" "kube-probe/1.17
   ```
   We can enable this option, and split log on timestamp and message in `monitor_pod`. Timestamp is returned in RFC3339Nano format so it should be easily parsable using `pendulum`.  `Pendulum` should handle possible timezone difference between Kubernetes API and pods.  
   If you are ok with new design I can refactor the code.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org