You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@storm.apache.org by "Aaron Gresch (JIRA)" <ji...@apache.org> on 2019/07/25 21:01:00 UTC

[jira] [Created] (STORM-3476) LocalizedResourceRetentionSet cleanup causing excessive load on Hadoop namenode

Aaron Gresch created STORM-3476:
-----------------------------------

             Summary: LocalizedResourceRetentionSet cleanup causing excessive load on Hadoop namenode
                 Key: STORM-3476
                 URL: https://issues.apache.org/jira/browse/STORM-3476
             Project: Apache Storm
          Issue Type: Improvement
            Reporter: Aaron Gresch
            Assignee: Aaron Gresch


One of our local dev Hadoop devs noticed our storm user was by far creating the heaviest load on our production Hadoop cluster.  Looking at one of the heaviest supervisor nodes, and comparing debug logs to the Hadoop audit log, it looks like LocalizedResourceRetentionSet cleanup was constantly doing opens and never deleting any files.

 

The frequency can be addressed by supervisor.localizer.cleanup.interval.ms, but even so, it seems we will continually look for files to delete even when the target size is acceptable, resulting in unnecessary calls to Hadoop.

 

 



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)