[jira] [Created] (HDFS-15907) Reduce Memory Overhead of AclFeature by avoiding AtomicInteger

["Stephen O'Donnell (Jira)" <ji...@apache.org>]

Stephen O'Donnell created HDFS-15907:
----------------------------------------

             Summary: Reduce Memory Overhead of AclFeature by avoiding AtomicInteger
                 Key: HDFS-15907
                 URL: https://issues.apache.org/jira/browse/HDFS-15907
             Project: Hadoop HDFS
          Issue Type: Improvement
          Components: namenode
            Reporter: Stephen O'Donnell
            Assignee: Stephen O'Donnell


In HDFS-15792 we made some changes to the AclFeature and ReferenceCountedMap classes to address a rare bug when loading the FSImage in parallel.

One change we made was to replace an int inside AclFeature with an AtomicInteger to avoid synchronising the methods in AclFeature.

Discussing this change with [~weichiu], he pointed out that while the AclFeature cache is intended to reduce the count of AclFeature objects, on a large cluster, it is possible for there to be many millions of AclFeature objects.

Previously, the int will have taken 4 bytes of heap.

By moving to a AtomicInteger, we probably have an overhead of:

 4 bytes (or 8 if the heap is over 32GB) for a reference to the atomic long object
 12 byte overhead for the java object
 4 bytes inside the atomic long to store an int.
 
So the total heap overhead has gone from 4 bytes to 20 bytes just to use an AtomicInteger.

Therefore I think it makes sense to remove the AtomicInteger and just synchronise the methods of AclFeature where the value is incremented / decremented / retrieved.





--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org