[jira] [Comment Edited] (HDFS-8900) Optimize XAttr memory footprint.

["Yi Liu (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDFS-8900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14704529#comment-14704529 ] 

Yi Liu edited comment on HDFS-8900 at 8/20/15 1:46 PM:
-------------------------------------------------------

This optimization:
-   For each XAttr, can save: 12bytes (object overhead) + 2bytes (compact), memory alignment of value
-   For each XAttrFeature, can save: (ImmutableList object size) + num * (XAttr saved).

So for each XAttrFeature (associated with an INodeFile or INodeDirectory), we can *save* (bytes): (ImmutableList object size) + n * (14 + memory alignment of value).
-  *If n == 1*,  (ImmutableList object size) = 12 + 4 (bytes). Totally *30* bytes.
-  *if n > 1*, ImmutableList  is RegularImmutableList, so (ImmutableList object size) = 12 + 4 + 4 + 12 + n * 4 + object alignment.  Totally *36 + n \* (18 + memory alignment of value) bytes*.

We can see the improvement is significant especially there is more than one XAttr in XAttrFeature.


The patch also makes code logic more simper, and has no impact on NN performance.

Besides, the xattrMaxSize is used to restrict user/trust namespace xattrs, I add a hard limit for max size. It's reasonable and make Xattr more controllable. 


was (Author: hitliuyi):
This optimization:
-   For each XAttr, can save: 12bytes (object overhead) + 2bytes (compact), memory alignment of value
-   For each XAttrFeature, can save: 12 bytes (object overhead) + num * XAttr saved.

So for each XAttrFeature (associated with an INodeFile or INodeDirectory), we can save (bytes): 12 + n * (14 + memory alignment of value in each XAttr) 


The patch also makes code logic more simper, and has no impact on NN performance.

Besides, the xattrMaxSize is used to restrict user/trust namespace xattrs, I add a hard limit for max size. It's reasonable and make Xattr more controllable. 

> Optimize XAttr memory footprint.
> --------------------------------
>
>                 Key: HDFS-8900
>                 URL: https://issues.apache.org/jira/browse/HDFS-8900
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: namenode
>            Reporter: Yi Liu
>            Assignee: Yi Liu
>         Attachments: HDFS-8900.001.patch
>
>
> {code}
> private final ImmutableList<XAttr> xAttrs;
> {code}
> Currently we use above in XAttrFeature, it's not efficient from memory point of view, since {{ImmutableList}} and {{XAttr}} have object memory overhead, and each object has memory alignment. 
> We can use a {{byte[]}} in XAttrFeature and do some compact in {{XAttr}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)