[jira] [Commented] (HUDI-3425) Clean up spill path created by Hudi during uneventful shutdown

["sivabalan narayanan (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HUDI-3425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17605545#comment-17605545 ] 

sivabalan narayanan commented on HUDI-3425:
-------------------------------------------

I went through the code using ExternalSpillableMap. 

 

We do create with `deleteOnExit()`. And also, we do add a shutdown hook as well. So, closing the issue as not valid. If user reports similar issue again, we can look into it. 

> Clean up spill path created by Hudi during uneventful shutdown
> --------------------------------------------------------------
>
>                 Key: HUDI-3425
>                 URL: https://issues.apache.org/jira/browse/HUDI-3425
>             Project: Apache Hudi
>          Issue Type: Improvement
>          Components: compaction
>            Reporter: sivabalan narayanan
>            Assignee: sivabalan narayanan
>            Priority: Critical
>             Fix For: 0.12.1
>
>
> h1. Hudi spill path not getting cleared when containers getting killed abruptly. 
>  
> When yarn kills the containers abruptly for any reason while hudi stage is in progress then the spill path created by hudi on the disk is not cleaned and as a result of which the nodes on the cluster start running out of space. We need to clear the spill path manually to free out disk space.
>  
> Ref issue: https://github.com/apache/hudi/issues/4771



--
This message was sent by Atlassian Jira
(v8.20.10#820010)