Re: [I] Kubernetes Executor leaving pods in a "Running" state after a task is completed [airflow]

["ahipp13 (via GitHub)" <gi...@apache.org>]

ahipp13 commented on issue #40957:
URL: https://github.com/apache/airflow/issues/40957#issuecomment-2263146206

   Another update for anybody that sees this, tried to look at pod and scheduler logs combined in our Splunk instance and this is what I found:
   
   For a task that completes and closes normally, the "kubernetes_executor_utils" will log that it has succeeded, like so:
   
   ```
   2024-08-01T08:21:01.528695415-05:00 stdout F [2024-08-01T13:21:01.528+0000] {taskinstance.py:1206} INFO - Marking task as SUCCESS. dag_id=DAG_ID, task_id=delete_data, run_id=scheduled__2024-08-01T13:10:00+00:00, execution_date=20240801T131000, start_date=20240801T132059, end_date=20240801T132101
   
   2024-08-01T08:21:01.587171193-05:00 stdout F [2024-08-01T13:21:01.586+0000] {local_task_job_runner.py:240} INFO - Task exited with return code 0
   
   2024-08-01T08:21:01.611141571-05:00 stdout F [2024-08-01T13:21:01.610+0000] {taskinstance.py:3503} INFO - 1 downstream tasks scheduled from follow-on schedule check
   
   2024-08-01T08:21:01.612627302-05:00 stdout F [2024-08-01T13:21:01.612+0000] {local_task_job_runner.py:222} INFO - ::endgroup::
   
   2024-08-01T08:21:01.703212706-05:00 stdout F [2024-08-01T13:21:01.701+0000] {base.py:84} INFO - Using connection ID 'CONN' for task execution.
   
   2024-08-01T08:21:01.924805315-05:00 stdout F [2024-08-01T13:21:01.923+0000] {get_token_mixin.py:104} INFO - ClientSecretCredential.get_token succeeded
   
   2024-08-01T08:21:02.968331016-05:00 stdout F [2024-08-01T13:21:02.967+0000] {kubernetes_executor_utils.py:299} INFO - Event: POD_NAME is Running, annotations: <omitted>
   
   2024-08-01T08:21:04.525216578-05:00 stdout F [2024-08-01T13:21:04.524+0000] {kubernetes_executor_utils.py:284} INFO - Event: POD_NAME Succeeded, annotations: <omitted>
   ```
   
   For a task that stays in a "Running" state, the kubernetes_executor_utils does not say it succeed and keeps saying it is "Running" even though the task gets marked as succeeded and returns with an exit code of 0:
   
   ```
   2024-07-31T23:50:25.538069838-05:00 stdout F [2024-08-01T04:50:25.537+0000] {base.py:294} INFO - Success criteria met. Exiting.
   
   2024-07-31T23:50:25.553145453-05:00 stdout F [2024-08-01T04:50:25.552+0000] {taskinstance.py:1206} INFO - Marking task as SUCCESS. dag_id=secplat_dev_delete_data_alerts, task_id=http_sensor_check, run_id=scheduled__2024-08-01T04:40:00+00:00, execution_date=20240801T044000, start_date=20240801T045025, end_date=20240801T045025
   
   2024-07-31T23:50:25.578659388-05:00 stdout F [2024-08-01T04:50:25.577+0000] {local_task_job_runner.py:240} INFO - Task exited with return code 0
   
   2024-07-31T23:50:25.592799182-05:00 stdout F [2024-08-01T04:50:25.592+0000] {local_task_job_runner.py:222} INFO - ::endgroup::
   
   2024-07-31T23:50:25.683516859-05:00 stdout F [2024-08-01T04:50:25.681+0000] {base.py:84} INFO - Using connection ID 'edw_az_conn' for task execution.
   
   2024-07-31T23:50:25.985346073-05:00 stdout F [2024-08-01T04:50:25.984+0000] {get_token_mixin.py:104} INFO - ClientSecretCredential.get_token succeeded
   
   2024-07-31T23:52:30.827037969-05:00 stdout F [2024-08-01T04:52:30.826+0000] {kubernetes_executor_utils.py:299} INFO - Event: secplat-dev-delete-data-alerts-http-sensor-check-179jgg3i is Running, annotations: <omitted>
   (This message goes on and on and on)
   ```
   
   So my question is, why randomly does the kubernetes_executor_utils not mark the task as succeeded? @eladkal Any tips on more debugging steps I can take? Really just don't get why this happens randomly. If I only have 1 scheduler up I wouldn't think it is a database problem


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org