[jira] [Resolved] (HDFS-5620) NameNode: implement Global ACL Set as a memory optimization.

["Chris Nauroth (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HDFS-5620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris Nauroth resolved HDFS-5620.
---------------------------------

    Resolution: Won't Fix

I'm resolving this issue as won't fix.  [~wheat9] was right to question whether or not we really need this optimization.

Now that ACLs implementation is almost done, I can get a more accurate measurement of the memory impact using jmap.  I'm including a plot that estimates memory consumption by ACLs in a presumed 50 GB NameNode, trying to store up to 100 million inodes.  I plotted 3 different usage scenarios:
# max usage: All inodes have an ACL, and each one has the maximum ACL entries.
# mid usage: Half the inodes have an ACL, and each ACL has 10 entries.
# low usage: 10% of inodes have an ACL, and each ACL has 2 entries.

For each scenario, I plotted the unoptimized memory consumption and also the optimized memory consumption, assuming that only 20% of the ACLs are unique.  Each scenario uses a different line color.  The unoptimized version uses a solid line, and the optimized version uses a dashed line.  I'm also attaching the GnuPlot script I wrote to generate this.

The plot demonstrates that ACL usage really needs to get quite high (very large number of inodes with a very high proportion of them having ACLs) before this memory optimization starts to provide a benefit.

I am +1 for skipping this for now.  We can always resurrect this patch if we observe a need for it based on real world usage of ACLs.  I'm also going to put this information into a new revision of the design doc on HDFS-4685.

BTW, if we do resurrect this patch, then I probably wouldn't commit the exact HDFS-5620.1.patch that I put together quickly as a prototype.  The Guava interner has a spinlock inside of it, which we don't really need, because this would only be accessed under the namesystem write lock anyway.

> NameNode: implement Global ACL Set as a memory optimization.
> ------------------------------------------------------------
>
>                 Key: HDFS-5620
>                 URL: https://issues.apache.org/jira/browse/HDFS-5620
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: namenode
>    Affects Versions: HDFS ACLs (HDFS-4685)
>            Reporter: Chris Nauroth
>            Assignee: Chris Nauroth
>         Attachments: HDFS-5620.1.patch, aclRamSizingEstimates, aclRamSizingEstimates.png
>
>
> The {{AclManager}} can maintain a Global ACL Set to store all distinct ACLs in use by the file system.  All inodes that have the same ACL entries can share the same ACL instance.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)