[jira] [Commented] (HDFS-6709) Implement off-heap data structures for NameNode and other HDFS memory optimization

["Kai Zheng (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDFS-6709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14070021#comment-14070021 ] 

Kai Zheng commented on HDFS-6709:
---------------------------------

Hi Colin,

I learned the patch. The Slab stuffs look great. A question, look at the following codes I doubt if we need the path to call unsafe.getByte(). Either allocated from direct buffer or heap buffer, ByteBuffer interface can be used to get/set byte from/to the buffer. Note it would be good to avoid using Unsafe if possible. Please clarify if I misunderstand anything here, thanks.

{code}
+  byte getByte(int offset) {
+    if (base != 0) {
+      return NativeIO.getUnsafe().getByte(null, base + offset);
+    } else {
+      buf.position(offset);
+      return buf.get();
+    }
+  }
{code}


> Implement off-heap data structures for NameNode and other HDFS memory optimization
> ----------------------------------------------------------------------------------
>
>                 Key: HDFS-6709
>                 URL: https://issues.apache.org/jira/browse/HDFS-6709
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: namenode
>            Reporter: Colin Patrick McCabe
>            Assignee: Colin Patrick McCabe
>         Attachments: HDFS-6709.001.patch
>
>
> We should investigate implementing off-heap data structures for NameNode and other HDFS memory optimization.  These data structures could reduce latency by avoiding the long GC times that occur with large Java heaps.  We could also avoid per-object memory overheads and control memory layout a little bit better.  This also would allow us to use the JVM's "compressed oops" optimization even with really large namespaces, if we could get the Java heap below 32 GB for those cases.  This would provide another performance and memory efficiency boost.



--
This message was sent by Atlassian JIRA
(v6.2#6252)