[jira] [Commented] (HBASE-4956) Control direct memory buffer consumption by HBaseClient

["Bob Copeland (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-4956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13267797#comment-13267797 ] 

Bob Copeland commented on HBASE-4956:
-------------------------------------

I have this bug on our servers (0.90.4-cdh3).  I think Johnathan's analysis was spot on, but it matters much more for read path since readFully() allocates a direct buffer as large as the target read.

Here's an easy way to reproduce:

1) create a ~10 meg row in some table
2) edit bin/hbase script to have max direct memory size < 10m, e.g.:
{code}
+JAVA_HEAP_MAX="-Xmx1000m -XX:MaxDirectMemorySize=1m"
{code}
3) now have any client try to read this row.  If read side were chunking with < 1m pieces, it should still be able to read.

Obviously this example is contrived, but we hit the problem with around 1-2000 threads (we have large rows in our DB).  Here's what our stack trace looks like:
{code}
java.lang.OutOfMemoryError: Direct buffer memory
        at java.nio.Bits.reserveMemory(Bits.java:633)
        at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:98)
        at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:288)
        at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:155)
        at sun.nio.ch.IOUtil.read(IOUtil.java:174)
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:243)
        at org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:55)
        at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
        at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
        at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
        at java.io.FilterInputStream.read(FilterInputStream.java:116)
        at org.apache.hadoop.hbase.ipc.HBaseClient$Connection$PingInputStream.read(HBaseClient.java:299)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
        at java.io.DataInputStream.readFully(DataInputStream.java:178)
        at org.apache.hadoop.hbase.client.Result.readFields(Result.java:504)
        at org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:521)
        at org.apache.hadoop.hbase.io.HbaseObjectWritable.readFields(HbaseObjectWritable.java:259)
        at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:554)
        at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:477)
{code}

The following works around the problem by only doing 8k allocations:

{code}
diff --git a/src/main/java/org/apache/hadoop/hbase/client/Result.java b/src/main/java/org/apache/hadoop/hbase/client/Result.java
index 8a0c1a9..378ca88 100644
--- a/src/main/java/org/apache/hadoop/hbase/client/Result.java
+++ b/src/main/java/org/apache/hadoop/hbase/client/Result.java
@@ -502,10 +502,18 @@ public class Result implements Writable, WritableWithSize {
       return;
     }
     byte [] raw = new byte[totalBuffer];
-    in.readFully(raw, 0, totalBuffer);
+    readChunked(in, raw, 0, totalBuffer);
     bytes = new ImmutableBytesWritable(raw, 0, totalBuffer);
   }
 
+  private void readChunked(final DataInput in, byte[] dest, int ofs, int len)
+  throws IOException {
+    int maxread = 8192;
+
+    for (; ofs < len; ofs += maxread)
+      in.readFully(dest, ofs, Math.min(len - ofs, maxread));
+  }
+
   //Create KeyValue[] when needed
   private void readFields() {
     if (bytes == null) {
{code}

                
> Control direct memory buffer consumption by HBaseClient
> -------------------------------------------------------
>
>                 Key: HBASE-4956
>                 URL: https://issues.apache.org/jira/browse/HBASE-4956
>             Project: HBase
>          Issue Type: New Feature
>            Reporter: Ted Yu
>
> As Jonathan explained here https://groups.google.com/group/asynchbase/browse_thread/thread/c45bc7ba788b2357?pli=1 , standard hbase client inadvertently consumes large amount of direct memory.
> We should consider using netty for NIO-related tasks.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira