You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/09/29 05:42:43 UTC

[GitHub] [airflow] bhavaniravi opened a new issue #11190: KubernetesPodOperator set's task failure on `startup_timeout` but doesn't kill the pod

bhavaniravi opened a new issue #11190:
URL: https://github.com/apache/airflow/issues/11190


   
   **Apache Airflow version**:
   1.10.
   
   **Kubernetes version (if you are using kubernetes)** (use `kubectl version`): 1.16
   
   **Environment**:
   
   - **Cloud provider or hardware configuration**: AWS
   - **OS** (e.g. from /etc/os-release): 
   - **Kernel** (e.g. `uname -a`):
   - **Install tools**:
   - **Others**:
   
   **What happened**:
   
   We are an airflow set up in production on a Kubernetes cluster. We found that some of the tasks using `KubernetesPodOperator` are failing abruptly. 
   
   On checking the task logs on airflow it said `Pod Took too long to start` along with the pod id
   
   On checking the pod ID in rancher, we found that the pod had ran and successfully completed the task. 
   
   I spent some time digging the airflow code, the `run_pod` function 
   1. Raises and exception
   2. Sets the task state as a failure
   
   But it doesn't do anything about the pod in pending state. 
   
   ```
   def run_pod(self, pod, startup_timeout=120, get_logs=True):
        ...
        if delta.total_seconds() >= startup_timeout:
              raise AirflowException("Pod took too long to start")
   ```
   
   **What you expected to happen**:
   
   With the KubernetesPodOperator if the task is set to Failure due to any event, the corresponding task pod should be shut down
   
   **How to reproduce it**:
   
   1. Have the pod hang in pending state for more than 120 seconds
   2. Reduce the startup_timeout to a lower number 
   
   
   **Anything else we need to know**:
   Random
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] eladkal commented on issue #11190: KubernetesPodOperator set's task failure on `startup_timeout` but doesn't kill the pod

Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #11190:
URL: https://github.com/apache/airflow/issues/11190#issuecomment-921657125


   hi @bhavaniravi can you please check against latest airflow version + latest kubernetes provider?
   This may have already been fixed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] bhavaniravi closed issue #11190: KubernetesPodOperator set's task failure on `startup_timeout` but doesn't kill the pod

Posted by GitBox <gi...@apache.org>.
bhavaniravi closed issue #11190:
URL: https://github.com/apache/airflow/issues/11190


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] bhavaniravi commented on issue #11190: KubernetesPodOperator set's task failure on `startup_timeout` but doesn't kill the pod

Posted by GitBox <gi...@apache.org>.
bhavaniravi commented on issue #11190:
URL: https://github.com/apache/airflow/issues/11190#issuecomment-921681268


   Yes, it looks like this has been handled well in the latest versions. Closing this issue


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org