You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Ted Yu (Created) (JIRA)" <ji...@apache.org> on 2011/12/05 19:50:40 UTC
[jira] [Created] (HBASE-4956) Control direct memory buffer
consumption by HBaseClient
Control direct memory buffer consumption by HBaseClient
-------------------------------------------------------
Key: HBASE-4956
URL: https://issues.apache.org/jira/browse/HBASE-4956
Project: HBase
Issue Type: New Feature
Reporter: Ted Yu
As Jonathan explained here https://groups.google.com/group/asynchbase/browse_thread/thread/c45bc7ba788b2357?pli=1 , standard hbase client inadvertently consumes large amount of direct memory.
We should consider using netty for NIO-related tasks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4956) Control direct memory buffer
consumption by HBaseClient
Posted by "Zhihong Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13418655#comment-13418655 ]
Zhihong Ted Yu commented on HBASE-4956:
---------------------------------------
Integrated to 0.94 since this JIRA is marked against 0.94.1 release.
> Control direct memory buffer consumption by HBaseClient
> -------------------------------------------------------
>
> Key: HBASE-4956
> URL: https://issues.apache.org/jira/browse/HBASE-4956
> Project: HBase
> Issue Type: New Feature
> Reporter: Ted Yu
> Fix For: 0.96.0, 0.94.1
>
> Attachments: 4956.txt, thread_get.rb
>
>
> As Jonathan explained here https://groups.google.com/group/asynchbase/browse_thread/thread/c45bc7ba788b2357?pli=1 , standard hbase client inadvertently consumes large amount of direct memory.
> We should consider using netty for NIO-related tasks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4956) Control direct memory buffer
consumption by HBaseClient
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13418383#comment-13418383 ]
Hadoop QA commented on HBASE-4956:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12537187/4956.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.
+1 javadoc. The javadoc tool did not generate any warning messages.
-1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings).
-1 findbugs. The patch appears to introduce 12 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2410//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2410//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2410//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2410//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2410//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2410//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2410//console
This message is automatically generated.
> Control direct memory buffer consumption by HBaseClient
> -------------------------------------------------------
>
> Key: HBASE-4956
> URL: https://issues.apache.org/jira/browse/HBASE-4956
> Project: HBase
> Issue Type: New Feature
> Reporter: Ted Yu
> Fix For: 0.96.0, 0.94.1
>
> Attachments: 4956.txt, thread_get.rb
>
>
> As Jonathan explained here https://groups.google.com/group/asynchbase/browse_thread/thread/c45bc7ba788b2357?pli=1 , standard hbase client inadvertently consumes large amount of direct memory.
> We should consider using netty for NIO-related tasks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4956) Control direct memory buffer
consumption by HBaseClient
Posted by "Lars Hofhansl (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13173917#comment-13173917 ]
Lars Hofhansl commented on HBASE-4956:
--------------------------------------
Just checked the HTable code in trunk... Puts (even single put operations) already go through a thread from the HTable's thread pool. A single get request is issuing a request directly, but get(List<Get>) is also using the pool.
So limiting the number of threads in a single thread pool and using the new HTable constructor from HBASE-4805, this can be controlled. Need to make one is using only get(List<Get>) in addition to a relatively small global threadpool.
At Salesforce we'll shoot for pool with ~100 threads and a waiting queue of about 100 as well (for very large clusters that may be too small though).
> Control direct memory buffer consumption by HBaseClient
> -------------------------------------------------------
>
> Key: HBASE-4956
> URL: https://issues.apache.org/jira/browse/HBASE-4956
> Project: HBase
> Issue Type: New Feature
> Reporter: Ted Yu
>
> As Jonathan explained here https://groups.google.com/group/asynchbase/browse_thread/thread/c45bc7ba788b2357?pli=1 , standard hbase client inadvertently consumes large amount of direct memory.
> We should consider using netty for NIO-related tasks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4956) Control direct memory buffer
consumption by HBaseClient
Posted by "Bob Copeland (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bob Copeland updated HBASE-4956:
--------------------------------
Attachment: thread_get.rb
Test script for reproducing the error (with max direct memory limits in place) or benchmarking different chunk sizes. Although it can do threading, I have mostly passed "1" to do the tests in a single thread.
> Control direct memory buffer consumption by HBaseClient
> -------------------------------------------------------
>
> Key: HBASE-4956
> URL: https://issues.apache.org/jira/browse/HBASE-4956
> Project: HBase
> Issue Type: New Feature
> Reporter: Ted Yu
> Attachments: thread_get.rb
>
>
> As Jonathan explained here https://groups.google.com/group/asynchbase/browse_thread/thread/c45bc7ba788b2357?pli=1 , standard hbase client inadvertently consumes large amount of direct memory.
> We should consider using netty for NIO-related tasks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4956) Control direct memory buffer
consumption by HBaseClient
Posted by "Bob Copeland (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13418330#comment-13418330 ]
Bob Copeland commented on HBASE-4956:
-------------------------------------
Just a follow-up -- we've been running with this patch (8k window size) for a couple of months now, and have no more OOMs, and no noticeable performance or functionality regressions.
> Control direct memory buffer consumption by HBaseClient
> -------------------------------------------------------
>
> Key: HBASE-4956
> URL: https://issues.apache.org/jira/browse/HBASE-4956
> Project: HBase
> Issue Type: New Feature
> Reporter: Ted Yu
> Attachments: thread_get.rb
>
>
> As Jonathan explained here https://groups.google.com/group/asynchbase/browse_thread/thread/c45bc7ba788b2357?pli=1 , standard hbase client inadvertently consumes large amount of direct memory.
> We should consider using netty for NIO-related tasks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4956) Control direct memory buffer
consumption by HBaseClient
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13418680#comment-13418680 ]
Hudson commented on HBASE-4956:
-------------------------------
Integrated in HBase-TRUNK #3153 (See [https://builds.apache.org/job/HBase-TRUNK/3153/])
HBASE-4956 Control direct memory buffer consumption by HBaseClient (Bob Copeland) (Revision 1363526)
Result = FAILURE
tedyu :
Files :
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/Result.java
> Control direct memory buffer consumption by HBaseClient
> -------------------------------------------------------
>
> Key: HBASE-4956
> URL: https://issues.apache.org/jira/browse/HBASE-4956
> Project: HBase
> Issue Type: New Feature
> Reporter: Ted Yu
> Fix For: 0.96.0, 0.94.1
>
> Attachments: 4956.txt, thread_get.rb
>
>
> As Jonathan explained here https://groups.google.com/group/asynchbase/browse_thread/thread/c45bc7ba788b2357?pli=1 , standard hbase client inadvertently consumes large amount of direct memory.
> We should consider using netty for NIO-related tasks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4956) Control direct memory buffer
consumption by HBaseClient
Posted by "Zhihong Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13164798#comment-13164798 ]
Zhihong Yu commented on HBASE-4956:
-----------------------------------
Since the proposal involves asynchronous communication, we should devise new API which can be used to validate the reduction in use of direct memory buffer.
> Control direct memory buffer consumption by HBaseClient
> -------------------------------------------------------
>
> Key: HBASE-4956
> URL: https://issues.apache.org/jira/browse/HBASE-4956
> Project: HBase
> Issue Type: New Feature
> Reporter: Ted Yu
>
> As Jonathan explained here https://groups.google.com/group/asynchbase/browse_thread/thread/c45bc7ba788b2357?pli=1 , standard hbase client inadvertently consumes large amount of direct memory.
> We should consider using netty for NIO-related tasks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4956) Control direct memory buffer
consumption by HBaseClient
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13418657#comment-13418657 ]
Lars Hofhansl commented on HBASE-4956:
--------------------------------------
Thanks Ted.
> Control direct memory buffer consumption by HBaseClient
> -------------------------------------------------------
>
> Key: HBASE-4956
> URL: https://issues.apache.org/jira/browse/HBASE-4956
> Project: HBase
> Issue Type: New Feature
> Reporter: Ted Yu
> Fix For: 0.96.0, 0.94.1
>
> Attachments: 4956.txt, thread_get.rb
>
>
> As Jonathan explained here https://groups.google.com/group/asynchbase/browse_thread/thread/c45bc7ba788b2357?pli=1 , standard hbase client inadvertently consumes large amount of direct memory.
> We should consider using netty for NIO-related tasks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4956) Control direct memory buffer
consumption by HBaseClient
Posted by "Bob Copeland (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13418394#comment-13418394 ]
Bob Copeland commented on HBASE-4956:
-------------------------------------
Yeah, buffer size doesn't really matter to our workload so out of laziness I stuck with 8k, but a larger one should be OK. Worth noting that 8k is better than the full length (i.e. unpatched) in my tests, presumably because of the additional allocation overhead.
> Control direct memory buffer consumption by HBaseClient
> -------------------------------------------------------
>
> Key: HBASE-4956
> URL: https://issues.apache.org/jira/browse/HBASE-4956
> Project: HBase
> Issue Type: New Feature
> Reporter: Ted Yu
> Fix For: 0.96.0, 0.94.1
>
> Attachments: 4956.txt, thread_get.rb
>
>
> As Jonathan explained here https://groups.google.com/group/asynchbase/browse_thread/thread/c45bc7ba788b2357?pli=1 , standard hbase client inadvertently consumes large amount of direct memory.
> We should consider using netty for NIO-related tasks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4956) Control direct memory buffer
consumption by HBaseClient
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13267849#comment-13267849 ]
Lars Hofhansl commented on HBASE-4956:
--------------------------------------
Yeah, we should quantify this. 8k seems like a very reasonable chunk size, though (probably anything between 1k and 64k would be OK :) ).
@Bob: Any chance you can whip up a quick performance test for this?
> Control direct memory buffer consumption by HBaseClient
> -------------------------------------------------------
>
> Key: HBASE-4956
> URL: https://issues.apache.org/jira/browse/HBASE-4956
> Project: HBase
> Issue Type: New Feature
> Reporter: Ted Yu
>
> As Jonathan explained here https://groups.google.com/group/asynchbase/browse_thread/thread/c45bc7ba788b2357?pli=1 , standard hbase client inadvertently consumes large amount of direct memory.
> We should consider using netty for NIO-related tasks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4956) Control direct memory buffer
consumption by HBaseClient
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13174884#comment-13174884 ]
stack commented on HBASE-4956:
------------------------------
@Lars So supposition is that we bound the number of direct buffers client allocates by bounding the number of threads in the thread pool (and have all accesses go via the HTable thread pool)? If so, sounds good. What happens if as many HTable instances as there are application-level threads and say the application spawns lots of threads? The application should then go via HTablePool? Or rather, you are suggesting that application uses new HBASE-4805 api and pass common executor to all HTable instances?
> Control direct memory buffer consumption by HBaseClient
> -------------------------------------------------------
>
> Key: HBASE-4956
> URL: https://issues.apache.org/jira/browse/HBASE-4956
> Project: HBase
> Issue Type: New Feature
> Reporter: Ted Yu
>
> As Jonathan explained here https://groups.google.com/group/asynchbase/browse_thread/thread/c45bc7ba788b2357?pli=1 , standard hbase client inadvertently consumes large amount of direct memory.
> We should consider using netty for NIO-related tasks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4956) Control direct memory buffer
consumption by HBaseClient
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13182908#comment-13182908 ]
stack commented on HBASE-4956:
------------------------------
Hmmm... Looking in code, we allocate a BufferedOutputStream using default buffer size:
{code}
this.out = new DataOutputStream
(new BufferedOutputStream(NetUtils.getOutputStream(socket)));
{code}
... so, it seems that yeah, at any one time, you'd think the maximum allocation 8k as you say Lars.
Jonathan says 3 of these buffers allocated when reading. Presume similar writing (easy to check I suppose. I havent'). Thats 6*8k per thread since these are thread local.... which still don't seem like that much. You'd need lots of threads to run into trouble.
We need to reproduce.
> Control direct memory buffer consumption by HBaseClient
> -------------------------------------------------------
>
> Key: HBASE-4956
> URL: https://issues.apache.org/jira/browse/HBASE-4956
> Project: HBase
> Issue Type: New Feature
> Reporter: Ted Yu
>
> As Jonathan explained here https://groups.google.com/group/asynchbase/browse_thread/thread/c45bc7ba788b2357?pli=1 , standard hbase client inadvertently consumes large amount of direct memory.
> We should consider using netty for NIO-related tasks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4956) Control direct memory buffer
consumption by HBaseClient
Posted by "Zhihong Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13418646#comment-13418646 ]
Zhihong Ted Yu commented on HBASE-4956:
---------------------------------------
Patch integrated to trunk.
Thanks for the patch, Bob.
Thanks for the review, Lars.
> Control direct memory buffer consumption by HBaseClient
> -------------------------------------------------------
>
> Key: HBASE-4956
> URL: https://issues.apache.org/jira/browse/HBASE-4956
> Project: HBase
> Issue Type: New Feature
> Reporter: Ted Yu
> Fix For: 0.96.0, 0.94.1
>
> Attachments: 4956.txt, thread_get.rb
>
>
> As Jonathan explained here https://groups.google.com/group/asynchbase/browse_thread/thread/c45bc7ba788b2357?pli=1 , standard hbase client inadvertently consumes large amount of direct memory.
> We should consider using netty for NIO-related tasks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4956) Control direct memory buffer
consumption by HBaseClient
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13428750#comment-13428750 ]
Hudson commented on HBASE-4956:
-------------------------------
Integrated in HBase-0.94-security-on-Hadoop-23 #6 (See [https://builds.apache.org/job/HBase-0.94-security-on-Hadoop-23/6/])
HBASE-4956 Control direct memory buffer consumption by HBaseClient (Bob Copeland) (Revision 1363533)
Result = FAILURE
tedyu :
Files :
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/Result.java
> Control direct memory buffer consumption by HBaseClient
> -------------------------------------------------------
>
> Key: HBASE-4956
> URL: https://issues.apache.org/jira/browse/HBASE-4956
> Project: HBase
> Issue Type: New Feature
> Reporter: Ted Yu
> Assignee: Bob Copeland
> Fix For: 0.96.0, 0.94.1
>
> Attachments: 4956.txt, thread_get.rb
>
>
> As Jonathan explained here https://groups.google.com/group/asynchbase/browse_thread/thread/c45bc7ba788b2357?pli=1 , standard hbase client inadvertently consumes large amount of direct memory.
> We should consider using netty for NIO-related tasks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4956) Control direct memory buffer
consumption by HBaseClient
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13418797#comment-13418797 ]
Hudson commented on HBASE-4956:
-------------------------------
Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #100 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/100/])
HBASE-4956 Control direct memory buffer consumption by HBaseClient (Bob Copeland) (Revision 1363526)
Result = FAILURE
tedyu :
Files :
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/Result.java
> Control direct memory buffer consumption by HBaseClient
> -------------------------------------------------------
>
> Key: HBASE-4956
> URL: https://issues.apache.org/jira/browse/HBASE-4956
> Project: HBase
> Issue Type: New Feature
> Reporter: Ted Yu
> Assignee: Bob Copeland
> Fix For: 0.96.0, 0.94.1
>
> Attachments: 4956.txt, thread_get.rb
>
>
> As Jonathan explained here https://groups.google.com/group/asynchbase/browse_thread/thread/c45bc7ba788b2357?pli=1 , standard hbase client inadvertently consumes large amount of direct memory.
> We should consider using netty for NIO-related tasks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4956) Control direct memory buffer
consumption by HBaseClient
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13267832#comment-13267832 ]
Lars Hofhansl commented on HBASE-4956:
--------------------------------------
This looks like a good approach. I'd be OK committing this (after passing tests, etc).
> Control direct memory buffer consumption by HBaseClient
> -------------------------------------------------------
>
> Key: HBASE-4956
> URL: https://issues.apache.org/jira/browse/HBASE-4956
> Project: HBase
> Issue Type: New Feature
> Reporter: Ted Yu
>
> As Jonathan explained here https://groups.google.com/group/asynchbase/browse_thread/thread/c45bc7ba788b2357?pli=1 , standard hbase client inadvertently consumes large amount of direct memory.
> We should consider using netty for NIO-related tasks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4956) Control direct memory buffer
consumption by HBaseClient
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-4956:
---------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Control direct memory buffer consumption by HBaseClient
> -------------------------------------------------------
>
> Key: HBASE-4956
> URL: https://issues.apache.org/jira/browse/HBASE-4956
> Project: HBase
> Issue Type: New Feature
> Reporter: Ted Yu
> Assignee: Bob Copeland
> Fix For: 0.96.0, 0.94.1
>
> Attachments: 4956.txt, thread_get.rb
>
>
> As Jonathan explained here https://groups.google.com/group/asynchbase/browse_thread/thread/c45bc7ba788b2357?pli=1 , standard hbase client inadvertently consumes large amount of direct memory.
> We should consider using netty for NIO-related tasks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4956) Control direct memory buffer
consumption by HBaseClient
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419652#comment-13419652 ]
Hudson commented on HBASE-4956:
-------------------------------
Integrated in HBase-0.94-security #44 (See [https://builds.apache.org/job/HBase-0.94-security/44/])
HBASE-4956 Control direct memory buffer consumption by HBaseClient (Bob Copeland) (Revision 1363533)
Result = FAILURE
tedyu :
Files :
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/Result.java
> Control direct memory buffer consumption by HBaseClient
> -------------------------------------------------------
>
> Key: HBASE-4956
> URL: https://issues.apache.org/jira/browse/HBASE-4956
> Project: HBase
> Issue Type: New Feature
> Reporter: Ted Yu
> Assignee: Bob Copeland
> Fix For: 0.96.0, 0.94.1
>
> Attachments: 4956.txt, thread_get.rb
>
>
> As Jonathan explained here https://groups.google.com/group/asynchbase/browse_thread/thread/c45bc7ba788b2357?pli=1 , standard hbase client inadvertently consumes large amount of direct memory.
> We should consider using netty for NIO-related tasks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4956) Control direct memory buffer
consumption by HBaseClient
Posted by "Lars Hofhansl (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13181872#comment-13181872 ]
Lars Hofhansl commented on HBASE-4956:
--------------------------------------
@Stack and Ted: Yes, if the application uses the API introduced in HBASE-4805 the number of threads doing network ops is limited to a single thread pools (that can be sized accordingly).
Still not ideal. The regionserver side of things does chunking (8k chunks by default), so even if a lot of buffers are cached for a lot of threads it won't add to much memory - even for 1000 threads each caching 3 buffer you'd only use 24mb, and since they are the same size they are reusable). Maybe the client do the same.
> Control direct memory buffer consumption by HBaseClient
> -------------------------------------------------------
>
> Key: HBASE-4956
> URL: https://issues.apache.org/jira/browse/HBASE-4956
> Project: HBase
> Issue Type: New Feature
> Reporter: Ted Yu
>
> As Jonathan explained here https://groups.google.com/group/asynchbase/browse_thread/thread/c45bc7ba788b2357?pli=1 , standard hbase client inadvertently consumes large amount of direct memory.
> We should consider using netty for NIO-related tasks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4956) Control direct memory buffer
consumption by HBaseClient
Posted by "Bob Copeland (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13267797#comment-13267797 ]
Bob Copeland commented on HBASE-4956:
-------------------------------------
I have this bug on our servers (0.90.4-cdh3). I think Johnathan's analysis was spot on, but it matters much more for read path since readFully() allocates a direct buffer as large as the target read.
Here's an easy way to reproduce:
1) create a ~10 meg row in some table
2) edit bin/hbase script to have max direct memory size < 10m, e.g.:
{code}
+JAVA_HEAP_MAX="-Xmx1000m -XX:MaxDirectMemorySize=1m"
{code}
3) now have any client try to read this row. If read side were chunking with < 1m pieces, it should still be able to read.
Obviously this example is contrived, but we hit the problem with around 1-2000 threads (we have large rows in our DB). Here's what our stack trace looks like:
{code}
java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Bits.java:633)
at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:98)
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:288)
at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:155)
at sun.nio.ch.IOUtil.read(IOUtil.java:174)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:243)
at org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:55)
at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at java.io.FilterInputStream.read(FilterInputStream.java:116)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection$PingInputStream.read(HBaseClient.java:299)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
at java.io.DataInputStream.readFully(DataInputStream.java:178)
at org.apache.hadoop.hbase.client.Result.readFields(Result.java:504)
at org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:521)
at org.apache.hadoop.hbase.io.HbaseObjectWritable.readFields(HbaseObjectWritable.java:259)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:554)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:477)
{code}
The following works around the problem by only doing 8k allocations:
{code}
diff --git a/src/main/java/org/apache/hadoop/hbase/client/Result.java b/src/main/java/org/apache/hadoop/hbase/client/Result.java
index 8a0c1a9..378ca88 100644
--- a/src/main/java/org/apache/hadoop/hbase/client/Result.java
+++ b/src/main/java/org/apache/hadoop/hbase/client/Result.java
@@ -502,10 +502,18 @@ public class Result implements Writable, WritableWithSize {
return;
}
byte [] raw = new byte[totalBuffer];
- in.readFully(raw, 0, totalBuffer);
+ readChunked(in, raw, 0, totalBuffer);
bytes = new ImmutableBytesWritable(raw, 0, totalBuffer);
}
+ private void readChunked(final DataInput in, byte[] dest, int ofs, int len)
+ throws IOException {
+ int maxread = 8192;
+
+ for (; ofs < len; ofs += maxread)
+ in.readFully(dest, ofs, Math.min(len - ofs, maxread));
+ }
+
//Create KeyValue[] when needed
private void readFields() {
if (bytes == null) {
{code}
> Control direct memory buffer consumption by HBaseClient
> -------------------------------------------------------
>
> Key: HBASE-4956
> URL: https://issues.apache.org/jira/browse/HBASE-4956
> Project: HBase
> Issue Type: New Feature
> Reporter: Ted Yu
>
> As Jonathan explained here https://groups.google.com/group/asynchbase/browse_thread/thread/c45bc7ba788b2357?pli=1 , standard hbase client inadvertently consumes large amount of direct memory.
> We should consider using netty for NIO-related tasks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4956) Control direct memory buffer
consumption by HBaseClient
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13267838#comment-13267838 ]
Lars Hofhansl commented on HBASE-4956:
--------------------------------------
Potentially every call to readFully over the wire is suspect. Just did a quick check. It's used in Put/Append/KeyValue/Bytes/etc/etc.
> Control direct memory buffer consumption by HBaseClient
> -------------------------------------------------------
>
> Key: HBASE-4956
> URL: https://issues.apache.org/jira/browse/HBASE-4956
> Project: HBase
> Issue Type: New Feature
> Reporter: Ted Yu
>
> As Jonathan explained here https://groups.google.com/group/asynchbase/browse_thread/thread/c45bc7ba788b2357?pli=1 , standard hbase client inadvertently consumes large amount of direct memory.
> We should consider using netty for NIO-related tasks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4956) Control direct memory buffer
consumption by HBaseClient
Posted by "Zhihong Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zhihong Ted Yu updated HBASE-4956:
----------------------------------
Status: Patch Available (was: Open)
> Control direct memory buffer consumption by HBaseClient
> -------------------------------------------------------
>
> Key: HBASE-4956
> URL: https://issues.apache.org/jira/browse/HBASE-4956
> Project: HBase
> Issue Type: New Feature
> Reporter: Ted Yu
> Attachments: 4956.txt, thread_get.rb
>
>
> As Jonathan explained here https://groups.google.com/group/asynchbase/browse_thread/thread/c45bc7ba788b2357?pli=1 , standard hbase client inadvertently consumes large amount of direct memory.
> We should consider using netty for NIO-related tasks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4956) Control direct memory buffer
consumption by HBaseClient
Posted by "Zhihong Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zhihong Ted Yu updated HBASE-4956:
----------------------------------
Attachment: 4956.txt
Patch from Bob.
> Control direct memory buffer consumption by HBaseClient
> -------------------------------------------------------
>
> Key: HBASE-4956
> URL: https://issues.apache.org/jira/browse/HBASE-4956
> Project: HBase
> Issue Type: New Feature
> Reporter: Ted Yu
> Attachments: 4956.txt, thread_get.rb
>
>
> As Jonathan explained here https://groups.google.com/group/asynchbase/browse_thread/thread/c45bc7ba788b2357?pli=1 , standard hbase client inadvertently consumes large amount of direct memory.
> We should consider using netty for NIO-related tasks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4956) Control direct memory buffer
consumption by HBaseClient
Posted by "Lars Hofhansl (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13164199#comment-13164199 ]
Lars Hofhansl commented on HBASE-4956:
--------------------------------------
We might not have to go all the way to use netty (although that would be nice).
If we find that it is possible to avoid calling HBaseClient.Connection.sendParam from the client thread, but have the call actually be made from the Connection thread (after the callable representing the operation was queued), we have limited the number of threads that will have cached DirectBuffer on their behalf.
Stack suggested the only reason for the direct call might be to pass errors back to the client. We could hand the client a Deferred or Future that will eventually hold any encountered exception, the client could (and would be default to keep the current synchronous behavior) also wait on that object.
> Control direct memory buffer consumption by HBaseClient
> -------------------------------------------------------
>
> Key: HBASE-4956
> URL: https://issues.apache.org/jira/browse/HBASE-4956
> Project: HBase
> Issue Type: New Feature
> Reporter: Ted Yu
>
> As Jonathan explained here https://groups.google.com/group/asynchbase/browse_thread/thread/c45bc7ba788b2357?pli=1 , standard hbase client inadvertently consumes large amount of direct memory.
> We should consider using netty for NIO-related tasks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-4956) Control direct memory buffer
consumption by HBaseClient
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl reassigned HBASE-4956:
------------------------------------
Assignee: Bob Copeland
Added you as contributor Bob.
> Control direct memory buffer consumption by HBaseClient
> -------------------------------------------------------
>
> Key: HBASE-4956
> URL: https://issues.apache.org/jira/browse/HBASE-4956
> Project: HBase
> Issue Type: New Feature
> Reporter: Ted Yu
> Assignee: Bob Copeland
> Fix For: 0.96.0, 0.94.1
>
> Attachments: 4956.txt, thread_get.rb
>
>
> As Jonathan explained here https://groups.google.com/group/asynchbase/browse_thread/thread/c45bc7ba788b2357?pli=1 , standard hbase client inadvertently consumes large amount of direct memory.
> We should consider using netty for NIO-related tasks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (HBASE-4956) Control direct memory
buffer consumption by HBaseClient
Posted by "Lars Hofhansl (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13164199#comment-13164199 ]
Lars Hofhansl edited comment on HBASE-4956 at 12/7/11 7:18 AM:
---------------------------------------------------------------
We might not have to go all the way to use netty (although that would be nice).
If we find that it is possible to avoid calling HBaseClient.Connection.sendParam from the client thread, but have the call actually be made from the Connection thread (after the callable representing the operation was queued), we have limited the number of threads that will have cached DirectBuffer on their behalf.
Stack suggested the only reason for the direct call might be to pass errors back to the client. We could hand the client a Deferred or Future that will eventually hold any encountered exception, the client could (and would by default to keep the current synchronous behavior) also wait on that object.
was (Author: lhofhansl):
We might not have to go all the way to use netty (although that would be nice).
If we find that it is possible to avoid calling HBaseClient.Connection.sendParam from the client thread, but have the call actually be made from the Connection thread (after the callable representing the operation was queued), we have limited the number of threads that will have cached DirectBuffer on their behalf.
Stack suggested the only reason for the direct call might be to pass errors back to the client. We could hand the client a Deferred or Future that will eventually hold any encountered exception, the client could (and would be default to keep the current synchronous behavior) also wait on that object.
> Control direct memory buffer consumption by HBaseClient
> -------------------------------------------------------
>
> Key: HBASE-4956
> URL: https://issues.apache.org/jira/browse/HBASE-4956
> Project: HBase
> Issue Type: New Feature
> Reporter: Ted Yu
>
> As Jonathan explained here https://groups.google.com/group/asynchbase/browse_thread/thread/c45bc7ba788b2357?pli=1 , standard hbase client inadvertently consumes large amount of direct memory.
> We should consider using netty for NIO-related tasks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4956) Control direct memory buffer
consumption by HBaseClient
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-4956:
---------------------------------
Fix Version/s: 0.94.1
0.96.0
> Control direct memory buffer consumption by HBaseClient
> -------------------------------------------------------
>
> Key: HBASE-4956
> URL: https://issues.apache.org/jira/browse/HBASE-4956
> Project: HBase
> Issue Type: New Feature
> Reporter: Ted Yu
> Fix For: 0.96.0, 0.94.1
>
> Attachments: 4956.txt, thread_get.rb
>
>
> As Jonathan explained here https://groups.google.com/group/asynchbase/browse_thread/thread/c45bc7ba788b2357?pli=1 , standard hbase client inadvertently consumes large amount of direct memory.
> We should consider using netty for NIO-related tasks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4956) Control direct memory buffer
consumption by HBaseClient
Posted by "Lars Hofhansl (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13182851#comment-13182851 ]
Lars Hofhansl commented on HBASE-4956:
--------------------------------------
Looking at the stacktrace that Jonathan posted to the asynchhabse group:
{code}
java.nio.Buffer.<init>(Buffer.java:172)
java.nio.ByteBuffer.<init>(ByteBuffer.java:259)
java.nio.ByteBuffer.<init>(ByteBuffer.java:267)
java.nio.MappedByteBuffer.<init>(MappedByteBuffer.java:64)
java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:97)
java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:288)
sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:155)
sun.nio.ch.IOUtil.write(IOUtil.java:37)
sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:55)
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
--> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
--> java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
java.io.DataOutputStream.flush(DataOutputStream.java:106)
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.sendParam(HBaseClient.java:518)
org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:751)
org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
$Proxy11.getProtocolVersion(<Unknown Source>:Unknown line)
{code}
Since the writes go through a BufferOutputStream they are already limited to 8k chunks (default buffer size).
So I do not really know what the problem is... In order to make this any kind of problem you need have buffers cached for 1m threads (to get to 8GB).
Anyway with the optional global ExecutorService of HBASE-4805 this should no longer be a problem.
> Control direct memory buffer consumption by HBaseClient
> -------------------------------------------------------
>
> Key: HBASE-4956
> URL: https://issues.apache.org/jira/browse/HBASE-4956
> Project: HBase
> Issue Type: New Feature
> Reporter: Ted Yu
>
> As Jonathan explained here https://groups.google.com/group/asynchbase/browse_thread/thread/c45bc7ba788b2357?pli=1 , standard hbase client inadvertently consumes large amount of direct memory.
> We should consider using netty for NIO-related tasks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4956) Control direct memory buffer
consumption by HBaseClient
Posted by "Bob Copeland (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13268435#comment-13268435 ]
Bob Copeland commented on HBASE-4956:
-------------------------------------
With regard to readFully() being everywhere -- perhaps it would be less intrusive to subclass DataInput, or to modify one of the read functions further down in the stack like org.apache.hadoop.net.SocketInputStream.read() to do the chunking. You guys know the code better than I.
These numbers are for standalone mode. I can't easily do a test on our distributed cluster but I don't think it would change the numbers much. I'll upload my test script and then anyone who wants can repeat the experiment on their setup.
Each trial consists of a run of "hbase org.jruby.Main thread_get.rb 1", which does 100 gets of a ~10 MB row, 3 times in a row, computing avg & stddev of wall-clock time. Maxread variable is changed for every run.
{code}
size avg stddev
~~~~ ~~~~~~ ~~~~~~
1 33.690 .10
1k 3.072 .18
2k 3.395 .19
4k 3.493 .16
8k 3.306 .19
16k 2.918 .21
32k 2.835 .06
64k 2.689 .08
128k 3.048 .03
len 3.328 .08
{code}
> Control direct memory buffer consumption by HBaseClient
> -------------------------------------------------------
>
> Key: HBASE-4956
> URL: https://issues.apache.org/jira/browse/HBASE-4956
> Project: HBase
> Issue Type: New Feature
> Reporter: Ted Yu
>
> As Jonathan explained here https://groups.google.com/group/asynchbase/browse_thread/thread/c45bc7ba788b2357?pli=1 , standard hbase client inadvertently consumes large amount of direct memory.
> We should consider using netty for NIO-related tasks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4956) Control direct memory buffer
consumption by HBaseClient
Posted by "Zhihong Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13174892#comment-13174892 ]
Zhihong Yu commented on HBASE-4956:
-----------------------------------
Currently lily-project uses a hack to share ExecutorService among all HTable instances.
Once HBASE-5084 is integrated, client application would have full control over reusing ExecutorService across all HTable instances.
> Control direct memory buffer consumption by HBaseClient
> -------------------------------------------------------
>
> Key: HBASE-4956
> URL: https://issues.apache.org/jira/browse/HBASE-4956
> Project: HBase
> Issue Type: New Feature
> Reporter: Ted Yu
>
> As Jonathan explained here https://groups.google.com/group/asynchbase/browse_thread/thread/c45bc7ba788b2357?pli=1 , standard hbase client inadvertently consumes large amount of direct memory.
> We should consider using netty for NIO-related tasks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4956) Control direct memory buffer
consumption by HBaseClient
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13267839#comment-13267839 ]
Todd Lipcon commented on HBASE-4956:
------------------------------------
I'd think we should make the max read configurable. Otherwise we should do some benchmarks to show that this doesn't cause a regression. My worry is that this would translate to a bunch more syscalls, no?
> Control direct memory buffer consumption by HBaseClient
> -------------------------------------------------------
>
> Key: HBASE-4956
> URL: https://issues.apache.org/jira/browse/HBASE-4956
> Project: HBase
> Issue Type: New Feature
> Reporter: Ted Yu
>
> As Jonathan explained here https://groups.google.com/group/asynchbase/browse_thread/thread/c45bc7ba788b2357?pli=1 , standard hbase client inadvertently consumes large amount of direct memory.
> We should consider using netty for NIO-related tasks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4956) Control direct memory buffer
consumption by HBaseClient
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13418705#comment-13418705 ]
Hudson commented on HBASE-4956:
-------------------------------
Integrated in HBase-0.94 #341 (See [https://builds.apache.org/job/HBase-0.94/341/])
HBASE-4956 Control direct memory buffer consumption by HBaseClient (Bob Copeland) (Revision 1363533)
Result = SUCCESS
tedyu :
Files :
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/Result.java
> Control direct memory buffer consumption by HBaseClient
> -------------------------------------------------------
>
> Key: HBASE-4956
> URL: https://issues.apache.org/jira/browse/HBASE-4956
> Project: HBase
> Issue Type: New Feature
> Reporter: Ted Yu
> Assignee: Bob Copeland
> Fix For: 0.96.0, 0.94.1
>
> Attachments: 4956.txt, thread_get.rb
>
>
> As Jonathan explained here https://groups.google.com/group/asynchbase/browse_thread/thread/c45bc7ba788b2357?pli=1 , standard hbase client inadvertently consumes large amount of direct memory.
> We should consider using netty for NIO-related tasks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4956) Control direct memory buffer
consumption by HBaseClient
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13418384#comment-13418384 ]
Lars Hofhansl commented on HBASE-4956:
--------------------------------------
Thanks Bob! I think we should commit this. To 0.94 and 0.96 at least.
If there're no objections I'll do that.
>From your analysis it looks like 32k would be the sweet spot, but since you ran it in production with an 8k window, let's commit this.
> Control direct memory buffer consumption by HBaseClient
> -------------------------------------------------------
>
> Key: HBASE-4956
> URL: https://issues.apache.org/jira/browse/HBASE-4956
> Project: HBase
> Issue Type: New Feature
> Reporter: Ted Yu
> Fix For: 0.96.0, 0.94.1
>
> Attachments: 4956.txt, thread_get.rb
>
>
> As Jonathan explained here https://groups.google.com/group/asynchbase/browse_thread/thread/c45bc7ba788b2357?pli=1 , standard hbase client inadvertently consumes large amount of direct memory.
> We should consider using netty for NIO-related tasks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (HBASE-4956) Control direct memory buffer
consumption by HBaseClient
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl closed HBASE-4956.
--------------------------------
> Control direct memory buffer consumption by HBaseClient
> -------------------------------------------------------
>
> Key: HBASE-4956
> URL: https://issues.apache.org/jira/browse/HBASE-4956
> Project: HBase
> Issue Type: New Feature
> Reporter: Ted Yu
> Assignee: Bob Copeland
> Fix For: 0.94.1, 0.96.0
>
> Attachments: 4956.txt, thread_get.rb
>
>
> As Jonathan explained here https://groups.google.com/group/asynchbase/browse_thread/thread/c45bc7ba788b2357?pli=1 , standard hbase client inadvertently consumes large amount of direct memory.
> We should consider using netty for NIO-related tasks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira