[GitHub] [airflow] ephraimbuddy commented on a change in pull request #15336: Fail task when containers inside a pod fails

[GitBox <gi...@apache.org>]

ephraimbuddy commented on a change in pull request #15336:
URL: https://github.com/apache/airflow/pull/15336#discussion_r612022098



##########
File path: tests/executors/test_kubernetes_executor.py
##########
@@ -507,3 +506,113 @@ def test_process_status_catchall(self):
 
         self._run()
         self.watcher.watcher_queue.put.assert_not_called()
+
+    def test_container_status_of_terminating_fails_pod(self):
+        self.pod.status.phase = "Pending"
+        self.pod.status.container_statuses = [
+            k8s.V1ContainerStatus(
+                container_id=None,
+                image="apache/airflow:2.0.1-python3.8",
+                image_id="",
+                name="base",
+                ready="false",
+                restart_count=0,
+                state=k8s.V1ContainerState(
+                    terminated=k8s.V1ContainerStateTerminated(
+                        reason="Terminating", exit_code=1

Review comment:
       > Have you seen, or can you recreate a `phase=Pending` and `state.terminated` pod? I don't see how it is possible to have both.
   > 
   > I've tried a few scenarios with both init containers and sidecars and every case has resulted in the watcher marking it as failed (though maybe not immediately, because `phase=Running`) - however the TI still gets marked as success.
   > 
   > Said another way, I think there are bugs around here, but I don't think looking at stuff in `phase=Pending` will help?
   
   The state.terminated is of the container inside the pod. It happens. This is how you can reproduce it:
   1. Checkout this PR,
   2. Go to values.yaml and set `worker_container_repository: apache/airflow`, `worker_container_tag: 2.0.1-python3.8`
   3. Use breeze to start the cluster: `./breeze kind-cluster start`,  `./breeze kind-cluster deploy`
   4. Monitor the pods in another terminal with k9s `./breeze kind-cluster k9s`
   5. Check the scheduler logs, it will print the 'event' object at each watcher run. Inspect the object and you'll see the pod phase and container state. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org