You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/03/16 18:19:42 UTC

[GitHub] [airflow] tnyz opened a new issue #22318: KubernetesPodOperator xcom sidecar stuck in running

tnyz opened a new issue #22318:
URL: https://github.com/apache/airflow/issues/22318


   ### Apache Airflow version
   
   2.2.4 (latest released)
   
   ### What happened
   
   When the main container errors and failed to write a return.json file, the xcom sidecar hangs and doesn't exit properly with an empty return.json.
   
   This is a problem because we want to suppress the following error, as the reason the pod failed should not be that xcom failed.
   ```
   [2022-03-16, 17:08:07 UTC] {pod_manager.py:342} INFO - Running command... cat /airflow/xcom/return.json
   
   [2022-03-16, 17:08:07 UTC] {pod_manager.py:349} INFO - stderr from command: cat: can't open '/airflow/xcom/return.json': No such file or directory
   
   [2022-03-16, 17:08:07 UTC] {pod_manager.py:342} INFO - Running command... kill -s SIGINT 1
   
   [2022-03-16, 17:08:08 UTC] {kubernetes_pod.py:417} INFO - Deleting pod: test.20882a4c607d418d94e87231214d34c0
   [2022-03-16, 17:08:08 UTC] {taskinstance.py:1718} ERROR - Task failed with exception
   Traceback (most recent call last):
     File "/usr/local/lib/python3.8/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 385, in execute
       result = self.extract_xcom(pod=self.pod)
     File "/usr/local/lib/python3.8/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 360, in extract_xcom
       result = self.pod_manager.extract_xcom(pod)
     File "/usr/local/lib/python3.8/site-packages/airflow/providers/cncf/kubernetes/utils/pod_manager.py", line 337, in extract_xcom
       raise AirflowException(f'Failed to extract xcom from pod: {pod.metadata.name}')
   airflow.exceptions.AirflowException: Failed to extract xcom from pod: test.20882a4c607d418d94e87231214d34c0
   ```
   and have the KubernetesPodOperator exit gracefully
   
   ### What you think should happen instead
   
   sidecar should exit with an empty xcom return value
   
   ### How to reproduce
   
   KubernetesPodOperator with command `mkdir -p /airflow/xcom;touch /airflow/xcom/return.json; cat a >> /airflow/xcom/return.json`
   
   ### Operating System
   
   -
   
   ### Versions of Apache Airflow Providers
   
   _No response_
   
   ### Deployment
   
   Other Docker-based deployment
   
   ### Deployment details
   
   _No response_
   
   ### Anything else
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on issue #22318: KubernetesPodOperator xcom sidecar stuck in running

Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #22318:
URL: https://github.com/apache/airflow/issues/22318#issuecomment-1069412224


   Thanks for opening your first issue here! Be sure to follow the issue template!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] eladkal commented on issue #22318: KubernetesPodOperator xcom sidecar stuck in running

Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #22318:
URL: https://github.com/apache/airflow/issues/22318#issuecomment-1073118442


   @tnyz assigned to you.
   Feel free to open PR


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org