[jira] [Updated] (HDFS-15907) Reduce Memory Overhead of AclFeature by avoiding AtomicInteger

["Shilun Fan (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HDFS-15907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Shilun Fan updated HDFS-15907:
------------------------------
    Affects Version/s: 3.3.1
                       3.4.0

> Reduce Memory Overhead of AclFeature by avoiding AtomicInteger
> --------------------------------------------------------------
>
>                 Key: HDFS-15907
>                 URL: https://issues.apache.org/jira/browse/HDFS-15907
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: namenode
>    Affects Versions: 3.3.1, 3.4.0
>            Reporter: Stephen O'Donnell
>            Assignee: Stephen O'Donnell
>            Priority: Major
>             Fix For: 3.3.1, 3.4.0
>
>         Attachments: HDFS-15907.001.patch
>
>
> In HDFS-15792 we made some changes to the AclFeature and ReferenceCountedMap classes to address a rare bug when loading the FSImage in parallel.
> One change we made was to replace an int inside AclFeature with an AtomicInteger to avoid synchronising the methods in AclFeature.
> Discussing this change with [~weichiu], he pointed out that while the AclFeature cache is intended to reduce the count of AclFeature objects, on a large cluster, it is possible for there to be many millions of AclFeature objects.
> Previously, the int will have taken 4 bytes of heap.
> By moving to a AtomicInteger, we probably have an overhead of:
>  4 bytes (or 8 if the heap is over 32GB) for a reference to the atomic long object
>  12 byte overhead for the java object
>  4 bytes inside the atomic long to store an int.
>  
> So the total heap overhead has gone from 4 bytes to 20 bytes just to use an AtomicInteger.
> Therefore I think it makes sense to remove the AtomicInteger and just synchronise the methods of AclFeature where the value is incremented / decremented / retrieved.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org