You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/02/09 05:33:06 UTC

[GitHub] [hudi] Prasheel3001 opened a new issue #4771: [Issue]: Hudi spill path not getting cleared when containers getting killed abruptly.

Prasheel3001 opened a new issue #4771:
URL: https://github.com/apache/hudi/issues/4771


   **_Tips before filing an issue_**
   
   - Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)? 
   
   - Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.
   
   - If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.
   
   **Describe the problem you faced**
   When yarn kills the containers abruptly for any reason while hudi stage is in progress then the spill path created by hudi on the disk is not cleaned and as a result of which the nodes on the cluster start running out of space. We need to clear the spill path manually to free out disk space.
   
   **To Reproduce**
   
   Steps to reproduce the behavior:
   Problem would be reproducible if hudi stage fails with containers getting killed abruptly.
   
   **Expected behavior**
   
   Hudi should clear the spill path even if the containers are killed abruptly by yarn for any reason.
   
   **Environment Description**
   
   * Hudi version : 0.7.0
   
   * Spark version : 2.4.6
   
   * Hive version :
   
   * Hadoop version :
   
   * Storage (HDFS/S3/GCS..) : HDFS
   
   * Running on Docker? (yes/no) : no
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan closed issue #4771: [Issue]: Hudi spill path not getting cleared when containers getting killed abruptly.

Posted by GitBox <gi...@apache.org>.
nsivabalan closed issue #4771:
URL: https://github.com/apache/hudi/issues/4771


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan closed issue #4771: [Issue]: Hudi spill path not getting cleared when containers getting killed abruptly.

Posted by GitBox <gi...@apache.org>.
nsivabalan closed issue #4771:
URL: https://github.com/apache/hudi/issues/4771


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan commented on issue #4771: [Issue]: Hudi spill path not getting cleared when containers getting killed abruptly.

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #4771:
URL: https://github.com/apache/hudi/issues/4771#issuecomment-1039522697


   thanks for reporting. have filed a tracking [jira](https://issues.apache.org/jira/browse/HUDI-3425). Let us know if you are interested to work on it. 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan commented on issue #4771: [Issue]: Hudi spill path not getting cleared when containers getting killed abruptly.

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #4771:
URL: https://github.com/apache/hudi/issues/4771#issuecomment-1039522697


   thanks for reporting. have filed a tracking [jira](https://issues.apache.org/jira/browse/HUDI-3425). Let us know if you are interested to work on it. 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org