You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Stephen O'Donnell (Jira)" <ji...@apache.org> on 2021/04/26 11:59:00 UTC
[jira] [Resolved] (HDFS-15621) Datanode DirectoryScanner uses
excessive memory
[ https://issues.apache.org/jira/browse/HDFS-15621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Stephen O'Donnell resolved HDFS-15621.
--------------------------------------
Resolution: Fixed
> Datanode DirectoryScanner uses excessive memory
> -----------------------------------------------
>
> Key: HDFS-15621
> URL: https://issues.apache.org/jira/browse/HDFS-15621
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
> Affects Versions: 3.4.0
> Reporter: Stephen O'Donnell
> Assignee: Stephen O'Donnell
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.3.1, 3.4.0, 3.1.5, 3.2.3
>
> Attachments: Screenshot 2020-10-09 at 14.11.36.png, Screenshot 2020-10-09 at 15.20.56.png
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> We generally work a rule of 1GB heap on a datanode per 1M blocks. For nodes with a lot of blocks, this can mean a lot of heap.
> We recently captured a heapdump of a DN with about 22M blocks and found only about 1.5GB was occupied by the ReplicaMap. Another 9GB of the heap is taken by the DirectoryScanner ScanInfo objects. Most of this memory was alloated to strings.
> Checking the strings in question, we can see two strings per scanInfo, looking like:
> {code}
> /current/BP-671271071-10.163.205.13-1552020401842/current/finalized/subdir28/subdir17/blk_1180438785
> _106716708.meta
> {code}
> I will update a screen shot from MAT showing this.
> For the first string especially, the part "/current/BP-671271071-10.163.205.13-1552020401842/current/finalized/" will be the same for every block in the block pool as the scanner is only concerned about finalized blocks.
> We can probably also store just the subdir indexes "28" and "27" rather than "subdir28/subdir17" and then construct the path when it is requested via the getter.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org