You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@storm.apache.org by "Ethan Li (JIRA)" <ji...@apache.org> on 2019/08/10 22:13:00 UTC
[jira] [Reopened] (STORM-3476) LocalizedResourceRetentionSet
cleanup causing excessive load on Hadoop namenode
[ https://issues.apache.org/jira/browse/STORM-3476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ethan Li reopened STORM-3476:
-----------------------------
> LocalizedResourceRetentionSet cleanup causing excessive load on Hadoop namenode
> -------------------------------------------------------------------------------
>
> Key: STORM-3476
> URL: https://issues.apache.org/jira/browse/STORM-3476
> Project: Apache Storm
> Issue Type: Improvement
> Affects Versions: 2.0.0
> Reporter: Aaron Gresch
> Assignee: Aaron Gresch
> Priority: Major
> Labels: pull-request-available
> Fix For: 2.0.1
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> One of our local dev Hadoop devs noticed our storm user was by far creating the heaviest load on our production Hadoop cluster. Looking at one of the heaviest supervisor nodes, and comparing debug logs to the Hadoop audit log, it looks like LocalizedResourceRetentionSet cleanup was constantly doing opens and never deleting any files.
>
> The frequency can be addressed by supervisor.localizer.cleanup.interval.ms, but even so, it seems we will continually look for files to delete even when the target size is acceptable, resulting in unnecessary calls to Hadoop.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)