You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Jean-Daniel Cryans (Created) (JIRA)" <ji...@apache.org> on 2012/01/13 03:03:39 UTC
[jira] [Created] (HBASE-5190) Limit the IPC queue size based on
calls' payload size
Limit the IPC queue size based on calls' payload size
-----------------------------------------------------
Key: HBASE-5190
URL: https://issues.apache.org/jira/browse/HBASE-5190
Project: HBase
Issue Type: Improvement
Affects Versions: 0.90.5
Reporter: Jean-Daniel Cryans
Fix For: 0.94.0
Currently we limit the number of calls in the IPC queue only on their count. It used to be really high and was dropped down recently to num_handlers * 10 (so 100 by default) because it was easy to OOME yourself when huge calls were being queued. It's still possible to hit this problem if you use really big values and/or a lot of handlers, so the idea is that we should take into account the payload size. I can see 3 solutions:
- Do the accounting outside of the queue itself for all calls coming in and out and when a call doesn't fit, throw a retryable exception.
- Same accounting but instead block the call when it comes in until space is made available.
- Add a new parameter for the maximum size (in bytes) of a Call and then set the size the IPC queue (in terms of the number of items) so that it could only contain as many items as some predefined maximum size (in bytes) for the whole queue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5190) Limit the IPC queue size based on
calls' payload size
Posted by "Jean-Daniel Cryans (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans updated HBASE-5190:
--------------------------------------
Attachment: HBASE-5190-v3.patch
New patch. I added some code (that bends over backward) to support the old ipc.server.max.queue.size which I'm planning to only add to 0.94, trunk will just read from the new config.
bq. Should we use high_scale_lib for the callQueueSize ?
Added that.
bq. Please wrap the log with braces.
Seems superfluous, anyways I changed that too.
bq. Would giving some details about queue capacity help client make better decision ?
I fixed the comment a bit. I don't want to be too chatty in the log, we can add more in the book.
bq. Should the above comment be added to 0.90.7 ?
So I cleared that out. We could add a warning in 0.90 and 0.92 yeah.
> Limit the IPC queue size based on calls' payload size
> -----------------------------------------------------
>
> Key: HBASE-5190
> URL: https://issues.apache.org/jira/browse/HBASE-5190
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.5
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.94.0
>
> Attachments: HBASE-5190-v2.patch, HBASE-5190-v3.patch, HBASE-5190.patch
>
>
> Currently we limit the number of calls in the IPC queue only on their count. It used to be really high and was dropped down recently to num_handlers * 10 (so 100 by default) because it was easy to OOME yourself when huge calls were being queued. It's still possible to hit this problem if you use really big values and/or a lot of handlers, so the idea is that we should take into account the payload size. I can see 3 solutions:
> - Do the accounting outside of the queue itself for all calls coming in and out and when a call doesn't fit, throw a retryable exception.
> - Same accounting but instead block the call when it comes in until space is made available.
> - Add a new parameter for the maximum size (in bytes) of a Call and then set the size the IPC queue (in terms of the number of items) so that it could only contain as many items as some predefined maximum size (in bytes) for the whole queue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-5190) Limit the IPC queue size based on
calls' payload size
Posted by "Jean-Daniel Cryans (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans resolved HBASE-5190.
---------------------------------------
Resolution: Fixed
Fix Version/s: 0.96.0
Release Note: Each region server can only have 1GB worth of calls's payload in flight and will refuse new calls when above that limit (those get automatically retried on the client side). Also, ipc.server.max.queue.size is now called ipc.server.max.callqueue.length
Hadoop Flags: Incompatible change,Reviewed
Committed to 0.94 and trunk, thanks for the reviews guys.
> Limit the IPC queue size based on calls' payload size
> -----------------------------------------------------
>
> Key: HBASE-5190
> URL: https://issues.apache.org/jira/browse/HBASE-5190
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.5
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.94.0, 0.96.0
>
> Attachments: HBASE-5190-v2.patch, HBASE-5190-v3.patch, HBASE-5190.patch
>
>
> Currently we limit the number of calls in the IPC queue only on their count. It used to be really high and was dropped down recently to num_handlers * 10 (so 100 by default) because it was easy to OOME yourself when huge calls were being queued. It's still possible to hit this problem if you use really big values and/or a lot of handlers, so the idea is that we should take into account the payload size. I can see 3 solutions:
> - Do the accounting outside of the queue itself for all calls coming in and out and when a call doesn't fit, throw a retryable exception.
> - Same accounting but instead block the call when it comes in until space is made available.
> - Add a new parameter for the maximum size (in bytes) of a Call and then set the size the IPC queue (in terms of the number of items) so that it could only contain as many items as some predefined maximum size (in bytes) for the whole queue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5190) Limit the IPC queue size based on
calls' payload size
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13236810#comment-13236810 ]
stack commented on HBASE-5190:
------------------------------
@JD Can do in another issue.
> Limit the IPC queue size based on calls' payload size
> -----------------------------------------------------
>
> Key: HBASE-5190
> URL: https://issues.apache.org/jira/browse/HBASE-5190
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.5
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.94.0
>
> Attachments: HBASE-5190-v2.patch, HBASE-5190-v3.patch, HBASE-5190.patch
>
>
> Currently we limit the number of calls in the IPC queue only on their count. It used to be really high and was dropped down recently to num_handlers * 10 (so 100 by default) because it was easy to OOME yourself when huge calls were being queued. It's still possible to hit this problem if you use really big values and/or a lot of handlers, so the idea is that we should take into account the payload size. I can see 3 solutions:
> - Do the accounting outside of the queue itself for all calls coming in and out and when a call doesn't fit, throw a retryable exception.
> - Same accounting but instead block the call when it comes in until space is made available.
> - Add a new parameter for the maximum size (in bytes) of a Call and then set the size the IPC queue (in terms of the number of items) so that it could only contain as many items as some predefined maximum size (in bytes) for the whole queue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5190) Limit the IPC queue size based on
calls' payload size
Posted by "Jean-Daniel Cryans (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13235852#comment-13235852 ]
Jean-Daniel Cryans commented on HBASE-5190:
-------------------------------------------
The current patch works, I've tested it extensively through massive imports. Current concerns:
- I haven't done a performance comparison, like is it going to slow down traffic because of additional checks? Most of my testing was done so that I'm hitting the limit all the time, so that does definitely slow down my throughput but it's expected :)
- The exception "Call queue already full" doesn't make it to the client, what happens is that it's being printed server-side and the client gets an EOF. That's bad.
- What default should we use? In my testing I saw that 100MB might be too small, but ideally that needs to scale with the amount of memory.
I don't mind finishing this for 0.94 if there's demand/motivation for it.
> Limit the IPC queue size based on calls' payload size
> -----------------------------------------------------
>
> Key: HBASE-5190
> URL: https://issues.apache.org/jira/browse/HBASE-5190
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.5
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.94.0
>
> Attachments: HBASE-5190.patch
>
>
> Currently we limit the number of calls in the IPC queue only on their count. It used to be really high and was dropped down recently to num_handlers * 10 (so 100 by default) because it was easy to OOME yourself when huge calls were being queued. It's still possible to hit this problem if you use really big values and/or a lot of handlers, so the idea is that we should take into account the payload size. I can see 3 solutions:
> - Do the accounting outside of the queue itself for all calls coming in and out and when a call doesn't fit, throw a retryable exception.
> - Same accounting but instead block the call when it comes in until space is made available.
> - Add a new parameter for the maximum size (in bytes) of a Call and then set the size the IPC queue (in terms of the number of items) so that it could only contain as many items as some predefined maximum size (in bytes) for the whole queue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5190) Limit the IPC queue size based on
calls' payload size
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13240892#comment-13240892 ]
Hudson commented on HBASE-5190:
-------------------------------
Integrated in HBase-TRUNK #2697 (See [https://builds.apache.org/job/HBase-TRUNK/2697/])
HBASE-5190 Limit the IPC queue size based on calls' payload size
(Ted's addendum) (Revision 1305468)
Result = FAILURE
jdcryans :
Files :
* /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java
> Limit the IPC queue size based on calls' payload size
> -----------------------------------------------------
>
> Key: HBASE-5190
> URL: https://issues.apache.org/jira/browse/HBASE-5190
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.5
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.94.0, 0.96.0
>
> Attachments: 5190.addendum, HBASE-5190-v2.patch, HBASE-5190-v3.patch, HBASE-5190.patch
>
>
> Currently we limit the number of calls in the IPC queue only on their count. It used to be really high and was dropped down recently to num_handlers * 10 (so 100 by default) because it was easy to OOME yourself when huge calls were being queued. It's still possible to hit this problem if you use really big values and/or a lot of handlers, so the idea is that we should take into account the payload size. I can see 3 solutions:
> - Do the accounting outside of the queue itself for all calls coming in and out and when a call doesn't fit, throw a retryable exception.
> - Same accounting but instead block the call when it comes in until space is made available.
> - Add a new parameter for the maximum size (in bytes) of a Call and then set the size the IPC queue (in terms of the number of items) so that it could only contain as many items as some predefined maximum size (in bytes) for the whole queue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5190) Limit the IPC queue size based on
calls' payload size
Posted by "Lars Hofhansl (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13234536#comment-13234536 ]
Lars Hofhansl commented on HBASE-5190:
--------------------------------------
Wanna finish this for 0.94, J-D? Or should bump to 0.96?
> Limit the IPC queue size based on calls' payload size
> -----------------------------------------------------
>
> Key: HBASE-5190
> URL: https://issues.apache.org/jira/browse/HBASE-5190
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.5
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.94.0
>
> Attachments: HBASE-5190.patch
>
>
> Currently we limit the number of calls in the IPC queue only on their count. It used to be really high and was dropped down recently to num_handlers * 10 (so 100 by default) because it was easy to OOME yourself when huge calls were being queued. It's still possible to hit this problem if you use really big values and/or a lot of handlers, so the idea is that we should take into account the payload size. I can see 3 solutions:
> - Do the accounting outside of the queue itself for all calls coming in and out and when a call doesn't fit, throw a retryable exception.
> - Same accounting but instead block the call when it comes in until space is made available.
> - Add a new parameter for the maximum size (in bytes) of a Call and then set the size the IPC queue (in terms of the number of items) so that it could only contain as many items as some predefined maximum size (in bytes) for the whole queue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5190) Limit the IPC queue size based on
calls' payload size
Posted by "Jean-Daniel Cryans (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans updated HBASE-5190:
--------------------------------------
Attachment: HBASE-5190.patch
This patch is what I've been testing, it's not polished yet. I implemented the first solution.
DEFAULT_MAX_QUEUE_SIZE_PER_HANDLER gets renamed to DEFAULT_MAX_CALLQUEUE_LENGTH_PER_HANDLER in order to add DEFAULT_MAX_CALLQUEUE_SIZE which is the max size for all calls that are in flight (now that I write this I think I need to change that name too hehe).
An AtomicLong is responsible to keep track of the total size. It could be a bottleneck, I haven't done any checks in that regard, but if it becomes one it could be lessen by not taking into account small calls (like <32KB).
We keep into account the call size from the moment it's inserted into the queue up to when the call is done. I considered doing that after the call is taken off the queue but then it's still easy to OOME yourself if many big calls come at the same time and handlers are free.
When a call is rejected it's being retried on the client side.
> Limit the IPC queue size based on calls' payload size
> -----------------------------------------------------
>
> Key: HBASE-5190
> URL: https://issues.apache.org/jira/browse/HBASE-5190
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.5
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.94.0
>
> Attachments: HBASE-5190.patch
>
>
> Currently we limit the number of calls in the IPC queue only on their count. It used to be really high and was dropped down recently to num_handlers * 10 (so 100 by default) because it was easy to OOME yourself when huge calls were being queued. It's still possible to hit this problem if you use really big values and/or a lot of handlers, so the idea is that we should take into account the payload size. I can see 3 solutions:
> - Do the accounting outside of the queue itself for all calls coming in and out and when a call doesn't fit, throw a retryable exception.
> - Same accounting but instead block the call when it comes in until space is made available.
> - Add a new parameter for the maximum size (in bytes) of a Call and then set the size the IPC queue (in terms of the number of items) so that it could only contain as many items as some predefined maximum size (in bytes) for the whole queue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5190) Limit the IPC queue size based on
calls' payload size
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13238697#comment-13238697 ]
Hudson commented on HBASE-5190:
-------------------------------
Integrated in HBase-0.94 #56 (See [https://builds.apache.org/job/HBase-0.94/56/])
HBASE-5190 Limit the IPC queue size based on calls' payload size
(Ted's addendum) (Revision 1305469)
Result = FAILURE
jdcryans :
Files :
* /hbase/branches/0.94/security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java
> Limit the IPC queue size based on calls' payload size
> -----------------------------------------------------
>
> Key: HBASE-5190
> URL: https://issues.apache.org/jira/browse/HBASE-5190
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.5
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.94.0, 0.96.0
>
> Attachments: 5190.addendum, HBASE-5190-v2.patch, HBASE-5190-v3.patch, HBASE-5190.patch
>
>
> Currently we limit the number of calls in the IPC queue only on their count. It used to be really high and was dropped down recently to num_handlers * 10 (so 100 by default) because it was easy to OOME yourself when huge calls were being queued. It's still possible to hit this problem if you use really big values and/or a lot of handlers, so the idea is that we should take into account the payload size. I can see 3 solutions:
> - Do the accounting outside of the queue itself for all calls coming in and out and when a call doesn't fit, throw a retryable exception.
> - Same accounting but instead block the call when it comes in until space is made available.
> - Add a new parameter for the maximum size (in bytes) of a Call and then set the size the IPC queue (in terms of the number of items) so that it could only contain as many items as some predefined maximum size (in bytes) for the whole queue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5190) Limit the IPC queue size based on
calls' payload size
Posted by "Lars Hofhansl (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13236205#comment-13236205 ]
Lars Hofhansl commented on HBASE-5190:
--------------------------------------
+1 on v3
> Limit the IPC queue size based on calls' payload size
> -----------------------------------------------------
>
> Key: HBASE-5190
> URL: https://issues.apache.org/jira/browse/HBASE-5190
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.5
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.94.0
>
> Attachments: HBASE-5190-v2.patch, HBASE-5190-v3.patch, HBASE-5190.patch
>
>
> Currently we limit the number of calls in the IPC queue only on their count. It used to be really high and was dropped down recently to num_handlers * 10 (so 100 by default) because it was easy to OOME yourself when huge calls were being queued. It's still possible to hit this problem if you use really big values and/or a lot of handlers, so the idea is that we should take into account the payload size. I can see 3 solutions:
> - Do the accounting outside of the queue itself for all calls coming in and out and when a call doesn't fit, throw a retryable exception.
> - Same accounting but instead block the call when it comes in until space is made available.
> - Add a new parameter for the maximum size (in bytes) of a Call and then set the size the IPC queue (in terms of the number of items) so that it could only contain as many items as some predefined maximum size (in bytes) for the whole queue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5190) Limit the IPC queue size based on
calls' payload size
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13239211#comment-13239211 ]
Hudson commented on HBASE-5190:
-------------------------------
Integrated in HBase-TRUNK-security #151 (See [https://builds.apache.org/job/HBase-TRUNK-security/151/])
HBASE-5190 Limit the IPC queue size based on calls' payload size
(Ted's addendum) (Revision 1305468)
Result = FAILURE
jdcryans :
Files :
* /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java
> Limit the IPC queue size based on calls' payload size
> -----------------------------------------------------
>
> Key: HBASE-5190
> URL: https://issues.apache.org/jira/browse/HBASE-5190
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.5
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.94.0, 0.96.0
>
> Attachments: 5190.addendum, HBASE-5190-v2.patch, HBASE-5190-v3.patch, HBASE-5190.patch
>
>
> Currently we limit the number of calls in the IPC queue only on their count. It used to be really high and was dropped down recently to num_handlers * 10 (so 100 by default) because it was easy to OOME yourself when huge calls were being queued. It's still possible to hit this problem if you use really big values and/or a lot of handlers, so the idea is that we should take into account the payload size. I can see 3 solutions:
> - Do the accounting outside of the queue itself for all calls coming in and out and when a call doesn't fit, throw a retryable exception.
> - Same accounting but instead block the call when it comes in until space is made available.
> - Add a new parameter for the maximum size (in bytes) of a Call and then set the size the IPC queue (in terms of the number of items) so that it could only contain as many items as some predefined maximum size (in bytes) for the whole queue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HBASE-5190) Limit the IPC queue size based on
calls' payload size
Posted by "Ted Yu (Reopened) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ted Yu reopened HBASE-5190:
---------------------------
The new parameter to Call ctor causes secure HBase build to fail.
> Limit the IPC queue size based on calls' payload size
> -----------------------------------------------------
>
> Key: HBASE-5190
> URL: https://issues.apache.org/jira/browse/HBASE-5190
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.5
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.94.0, 0.96.0
>
> Attachments: HBASE-5190-v2.patch, HBASE-5190-v3.patch, HBASE-5190.patch
>
>
> Currently we limit the number of calls in the IPC queue only on their count. It used to be really high and was dropped down recently to num_handlers * 10 (so 100 by default) because it was easy to OOME yourself when huge calls were being queued. It's still possible to hit this problem if you use really big values and/or a lot of handlers, so the idea is that we should take into account the payload size. I can see 3 solutions:
> - Do the accounting outside of the queue itself for all calls coming in and out and when a call doesn't fit, throw a retryable exception.
> - Same accounting but instead block the call when it comes in until space is made available.
> - Add a new parameter for the maximum size (in bytes) of a Call and then set the size the IPC queue (in terms of the number of items) so that it could only contain as many items as some predefined maximum size (in bytes) for the whole queue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5190) Limit the IPC queue size based on
calls' payload size
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13237529#comment-13237529 ]
Hudson commented on HBASE-5190:
-------------------------------
Integrated in HBase-TRUNK #2694 (See [https://builds.apache.org/job/HBase-TRUNK/2694/])
HBASE-5190 Limit the IPC queue size based on calls' payload size (Revision 1304634)
Result = SUCCESS
jdcryans :
Files :
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java
> Limit the IPC queue size based on calls' payload size
> -----------------------------------------------------
>
> Key: HBASE-5190
> URL: https://issues.apache.org/jira/browse/HBASE-5190
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.5
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.94.0, 0.96.0
>
> Attachments: HBASE-5190-v2.patch, HBASE-5190-v3.patch, HBASE-5190.patch
>
>
> Currently we limit the number of calls in the IPC queue only on their count. It used to be really high and was dropped down recently to num_handlers * 10 (so 100 by default) because it was easy to OOME yourself when huge calls were being queued. It's still possible to hit this problem if you use really big values and/or a lot of handlers, so the idea is that we should take into account the payload size. I can see 3 solutions:
> - Do the accounting outside of the queue itself for all calls coming in and out and when a call doesn't fit, throw a retryable exception.
> - Same accounting but instead block the call when it comes in until space is made available.
> - Add a new parameter for the maximum size (in bytes) of a Call and then set the size the IPC queue (in terms of the number of items) so that it could only contain as many items as some predefined maximum size (in bytes) for the whole queue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5190) Limit the IPC queue size based on
calls' payload size
Posted by "Jean-Daniel Cryans (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13235909#comment-13235909 ]
Jean-Daniel Cryans commented on HBASE-5190:
-------------------------------------------
bq. 100MB is per RegionServer, right? Does seem a bit small. Maybe 1G?
Might be a good default, those with the default heap will definitely not get any help here though.
bq. I take it that maybe this is something to consider for 0.96. Agreed?
The more I think about it, the more I want this in 0.94 because it can really give us a better understanding of those issues we see on the mailing list.
> Limit the IPC queue size based on calls' payload size
> -----------------------------------------------------
>
> Key: HBASE-5190
> URL: https://issues.apache.org/jira/browse/HBASE-5190
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.5
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.94.0
>
> Attachments: HBASE-5190.patch
>
>
> Currently we limit the number of calls in the IPC queue only on their count. It used to be really high and was dropped down recently to num_handlers * 10 (so 100 by default) because it was easy to OOME yourself when huge calls were being queued. It's still possible to hit this problem if you use really big values and/or a lot of handlers, so the idea is that we should take into account the payload size. I can see 3 solutions:
> - Do the accounting outside of the queue itself for all calls coming in and out and when a call doesn't fit, throw a retryable exception.
> - Same accounting but instead block the call when it comes in until space is made available.
> - Add a new parameter for the maximum size (in bytes) of a Call and then set the size the IPC queue (in terms of the number of items) so that it could only contain as many items as some predefined maximum size (in bytes) for the whole queue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-5190) Limit the IPC queue size based on
calls' payload size
Posted by "Jean-Daniel Cryans (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans resolved HBASE-5190.
---------------------------------------
Resolution: Fixed
Committed the addendum, thanks for looking at this Ted. Also, sorry I forgot about security.
> Limit the IPC queue size based on calls' payload size
> -----------------------------------------------------
>
> Key: HBASE-5190
> URL: https://issues.apache.org/jira/browse/HBASE-5190
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.5
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.94.0, 0.96.0
>
> Attachments: 5190.addendum, HBASE-5190-v2.patch, HBASE-5190-v3.patch, HBASE-5190.patch
>
>
> Currently we limit the number of calls in the IPC queue only on their count. It used to be really high and was dropped down recently to num_handlers * 10 (so 100 by default) because it was easy to OOME yourself when huge calls were being queued. It's still possible to hit this problem if you use really big values and/or a lot of handlers, so the idea is that we should take into account the payload size. I can see 3 solutions:
> - Do the accounting outside of the queue itself for all calls coming in and out and when a call doesn't fit, throw a retryable exception.
> - Same accounting but instead block the call when it comes in until space is made available.
> - Add a new parameter for the maximum size (in bytes) of a Call and then set the size the IPC queue (in terms of the number of items) so that it could only contain as many items as some predefined maximum size (in bytes) for the whole queue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5190) Limit the IPC queue size based on
calls' payload size
Posted by "Lars Hofhansl (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13235970#comment-13235970 ]
Lars Hofhansl commented on HBASE-5190:
--------------------------------------
Fair enough. I think we don't need to worry about a slow down here.
So that leaves passing the exception on to the client.
How is the exception handled currently when we get over the queue length, if the client in that case also gets an EOF then we can live with that.
> Limit the IPC queue size based on calls' payload size
> -----------------------------------------------------
>
> Key: HBASE-5190
> URL: https://issues.apache.org/jira/browse/HBASE-5190
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.5
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.94.0
>
> Attachments: HBASE-5190.patch
>
>
> Currently we limit the number of calls in the IPC queue only on their count. It used to be really high and was dropped down recently to num_handlers * 10 (so 100 by default) because it was easy to OOME yourself when huge calls were being queued. It's still possible to hit this problem if you use really big values and/or a lot of handlers, so the idea is that we should take into account the payload size. I can see 3 solutions:
> - Do the accounting outside of the queue itself for all calls coming in and out and when a call doesn't fit, throw a retryable exception.
> - Same accounting but instead block the call when it comes in until space is made available.
> - Add a new parameter for the maximum size (in bytes) of a Call and then set the size the IPC queue (in terms of the number of items) so that it could only contain as many items as some predefined maximum size (in bytes) for the whole queue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5190) Limit the IPC queue size based on
calls' payload size
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13237307#comment-13237307 ]
Hudson commented on HBASE-5190:
-------------------------------
Integrated in HBase-0.94 #49 (See [https://builds.apache.org/job/HBase-0.94/49/])
HBASE-5190 Limit the IPC queue size based on calls' payload size (Revision 1304635)
Result = SUCCESS
jdcryans :
Files :
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java
> Limit the IPC queue size based on calls' payload size
> -----------------------------------------------------
>
> Key: HBASE-5190
> URL: https://issues.apache.org/jira/browse/HBASE-5190
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.5
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.94.0, 0.96.0
>
> Attachments: HBASE-5190-v2.patch, HBASE-5190-v3.patch, HBASE-5190.patch
>
>
> Currently we limit the number of calls in the IPC queue only on their count. It used to be really high and was dropped down recently to num_handlers * 10 (so 100 by default) because it was easy to OOME yourself when huge calls were being queued. It's still possible to hit this problem if you use really big values and/or a lot of handlers, so the idea is that we should take into account the payload size. I can see 3 solutions:
> - Do the accounting outside of the queue itself for all calls coming in and out and when a call doesn't fit, throw a retryable exception.
> - Same accounting but instead block the call when it comes in until space is made available.
> - Add a new parameter for the maximum size (in bytes) of a Call and then set the size the IPC queue (in terms of the number of items) so that it could only contain as many items as some predefined maximum size (in bytes) for the whole queue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5190) Limit the IPC queue size based on
calls' payload size
Posted by "Jean-Daniel Cryans (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13236037#comment-13236037 ]
Jean-Daniel Cryans commented on HBASE-5190:
-------------------------------------------
Instead of throwing the IOE it's better if I put it in the try block just below in the code as it already handles the setting up of a response. I just need to change the messaging there and refactor some bits.
> Limit the IPC queue size based on calls' payload size
> -----------------------------------------------------
>
> Key: HBASE-5190
> URL: https://issues.apache.org/jira/browse/HBASE-5190
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.5
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.94.0
>
> Attachments: HBASE-5190.patch
>
>
> Currently we limit the number of calls in the IPC queue only on their count. It used to be really high and was dropped down recently to num_handlers * 10 (so 100 by default) because it was easy to OOME yourself when huge calls were being queued. It's still possible to hit this problem if you use really big values and/or a lot of handlers, so the idea is that we should take into account the payload size. I can see 3 solutions:
> - Do the accounting outside of the queue itself for all calls coming in and out and when a call doesn't fit, throw a retryable exception.
> - Same accounting but instead block the call when it comes in until space is made available.
> - Add a new parameter for the maximum size (in bytes) of a Call and then set the size the IPC queue (in terms of the number of items) so that it could only contain as many items as some predefined maximum size (in bytes) for the whole queue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5190) Limit the IPC queue size based on
calls' payload size
Posted by "Lars Hofhansl (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13236855#comment-13236855 ]
Lars Hofhansl commented on HBASE-5190:
--------------------------------------
@J-D: Oh yeah forgot that. Fine then. :)
Still +1
> Limit the IPC queue size based on calls' payload size
> -----------------------------------------------------
>
> Key: HBASE-5190
> URL: https://issues.apache.org/jira/browse/HBASE-5190
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.5
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.94.0
>
> Attachments: HBASE-5190-v2.patch, HBASE-5190-v3.patch, HBASE-5190.patch
>
>
> Currently we limit the number of calls in the IPC queue only on their count. It used to be really high and was dropped down recently to num_handlers * 10 (so 100 by default) because it was easy to OOME yourself when huge calls were being queued. It's still possible to hit this problem if you use really big values and/or a lot of handlers, so the idea is that we should take into account the payload size. I can see 3 solutions:
> - Do the accounting outside of the queue itself for all calls coming in and out and when a call doesn't fit, throw a retryable exception.
> - Same accounting but instead block the call when it comes in until space is made available.
> - Add a new parameter for the maximum size (in bytes) of a Call and then set the size the IPC queue (in terms of the number of items) so that it could only contain as many items as some predefined maximum size (in bytes) for the whole queue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5190) Limit the IPC queue size based on
calls' payload size
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13238014#comment-13238014 ]
stack commented on HBASE-5190:
------------------------------
Addendum looks good to me. +1
> Limit the IPC queue size based on calls' payload size
> -----------------------------------------------------
>
> Key: HBASE-5190
> URL: https://issues.apache.org/jira/browse/HBASE-5190
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.5
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.94.0, 0.96.0
>
> Attachments: 5190.addendum, HBASE-5190-v2.patch, HBASE-5190-v3.patch, HBASE-5190.patch
>
>
> Currently we limit the number of calls in the IPC queue only on their count. It used to be really high and was dropped down recently to num_handlers * 10 (so 100 by default) because it was easy to OOME yourself when huge calls were being queued. It's still possible to hit this problem if you use really big values and/or a lot of handlers, so the idea is that we should take into account the payload size. I can see 3 solutions:
> - Do the accounting outside of the queue itself for all calls coming in and out and when a call doesn't fit, throw a retryable exception.
> - Same accounting but instead block the call when it comes in until space is made available.
> - Add a new parameter for the maximum size (in bytes) of a Call and then set the size the IPC queue (in terms of the number of items) so that it could only contain as many items as some predefined maximum size (in bytes) for the whole queue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5190) Limit the IPC queue size based on
calls' payload size
Posted by "Jean-Daniel Cryans (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13237146#comment-13237146 ]
Jean-Daniel Cryans commented on HBASE-5190:
-------------------------------------------
Yep, right after I finishing running medium tests on trunk.
> Limit the IPC queue size based on calls' payload size
> -----------------------------------------------------
>
> Key: HBASE-5190
> URL: https://issues.apache.org/jira/browse/HBASE-5190
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.5
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.94.0
>
> Attachments: HBASE-5190-v2.patch, HBASE-5190-v3.patch, HBASE-5190.patch
>
>
> Currently we limit the number of calls in the IPC queue only on their count. It used to be really high and was dropped down recently to num_handlers * 10 (so 100 by default) because it was easy to OOME yourself when huge calls were being queued. It's still possible to hit this problem if you use really big values and/or a lot of handlers, so the idea is that we should take into account the payload size. I can see 3 solutions:
> - Do the accounting outside of the queue itself for all calls coming in and out and when a call doesn't fit, throw a retryable exception.
> - Same accounting but instead block the call when it comes in until space is made available.
> - Add a new parameter for the maximum size (in bytes) of a Call and then set the size the IPC queue (in terms of the number of items) so that it could only contain as many items as some predefined maximum size (in bytes) for the whole queue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5190) Limit the IPC queue size based on
calls' payload size
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13236115#comment-13236115 ]
stack commented on HBASE-5190:
------------------------------
+1 after chatting w/ J-D about what happens when we fill the queue (we send client IOE, retries, fills network w/ more back and forths but it should act like a pushback on the client...)
> Limit the IPC queue size based on calls' payload size
> -----------------------------------------------------
>
> Key: HBASE-5190
> URL: https://issues.apache.org/jira/browse/HBASE-5190
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.5
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.94.0
>
> Attachments: HBASE-5190-v2.patch, HBASE-5190.patch
>
>
> Currently we limit the number of calls in the IPC queue only on their count. It used to be really high and was dropped down recently to num_handlers * 10 (so 100 by default) because it was easy to OOME yourself when huge calls were being queued. It's still possible to hit this problem if you use really big values and/or a lot of handlers, so the idea is that we should take into account the payload size. I can see 3 solutions:
> - Do the accounting outside of the queue itself for all calls coming in and out and when a call doesn't fit, throw a retryable exception.
> - Same accounting but instead block the call when it comes in until space is made available.
> - Add a new parameter for the maximum size (in bytes) of a Call and then set the size the IPC queue (in terms of the number of items) so that it could only contain as many items as some predefined maximum size (in bytes) for the whole queue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5190) Limit the IPC queue size based on
calls' payload size
Posted by "Zhihong Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13236110#comment-13236110 ]
Zhihong Yu commented on HBASE-5190:
-----------------------------------
Should we use high_scale_lib for the callQueueSize ?
{code}
if (LOG.isDebugEnabled())
- LOG.debug(" got call #" + id + ", " + buf.length + " bytes");
+ LOG.debug(" got call #" + id + ", " + callSize + " bytes");
{code}
Please wrap the log with braces.
{code}
+ IOException.class.getName(),
+ "Call queue is full, retry later");
{code}
Would giving some details about queue capacity help client make better decision ?
{code}
+ // TODO add a deprecated comment for ipc.server.max.queue.size
{code}
Should the above comment be added to 0.90.7 ?
> Limit the IPC queue size based on calls' payload size
> -----------------------------------------------------
>
> Key: HBASE-5190
> URL: https://issues.apache.org/jira/browse/HBASE-5190
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.5
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.94.0
>
> Attachments: HBASE-5190-v2.patch, HBASE-5190.patch
>
>
> Currently we limit the number of calls in the IPC queue only on their count. It used to be really high and was dropped down recently to num_handlers * 10 (so 100 by default) because it was easy to OOME yourself when huge calls were being queued. It's still possible to hit this problem if you use really big values and/or a lot of handlers, so the idea is that we should take into account the payload size. I can see 3 solutions:
> - Do the accounting outside of the queue itself for all calls coming in and out and when a call doesn't fit, throw a retryable exception.
> - Same accounting but instead block the call when it comes in until space is made available.
> - Add a new parameter for the maximum size (in bytes) of a Call and then set the size the IPC queue (in terms of the number of items) so that it could only contain as many items as some predefined maximum size (in bytes) for the whole queue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5190) Limit the IPC queue size based on
calls' payload size
Posted by "Jean-Daniel Cryans (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13236790#comment-13236790 ]
Jean-Daniel Cryans commented on HBASE-5190:
-------------------------------------------
It's not exactly 10, it's num_handlers * 10 so by default 100. Is that better?
> Limit the IPC queue size based on calls' payload size
> -----------------------------------------------------
>
> Key: HBASE-5190
> URL: https://issues.apache.org/jira/browse/HBASE-5190
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.5
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.94.0
>
> Attachments: HBASE-5190-v2.patch, HBASE-5190-v3.patch, HBASE-5190.patch
>
>
> Currently we limit the number of calls in the IPC queue only on their count. It used to be really high and was dropped down recently to num_handlers * 10 (so 100 by default) because it was easy to OOME yourself when huge calls were being queued. It's still possible to hit this problem if you use really big values and/or a lot of handlers, so the idea is that we should take into account the payload size. I can see 3 solutions:
> - Do the accounting outside of the queue itself for all calls coming in and out and when a call doesn't fit, throw a retryable exception.
> - Same accounting but instead block the call when it comes in until space is made available.
> - Add a new parameter for the maximum size (in bytes) of a Call and then set the size the IPC queue (in terms of the number of items) so that it could only contain as many items as some predefined maximum size (in bytes) for the whole queue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5190) Limit the IPC queue size based on
calls' payload size
Posted by "Lars Hofhansl (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13235895#comment-13235895 ]
Lars Hofhansl commented on HBASE-5190:
--------------------------------------
I doubt you'll see a slowdown from this. Then again AtomicLong needs a memory barrier as far as I know, so could potentially add a noticeable slowdown. Hmm...
100MB is per RegionServer, right? Does seem a bit small. Maybe 1G?
We really need a streaming API for large rows, but that's a different story.
I take it that maybe this is something to consider for 0.96. Agreed?
> Limit the IPC queue size based on calls' payload size
> -----------------------------------------------------
>
> Key: HBASE-5190
> URL: https://issues.apache.org/jira/browse/HBASE-5190
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.5
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.94.0
>
> Attachments: HBASE-5190.patch
>
>
> Currently we limit the number of calls in the IPC queue only on their count. It used to be really high and was dropped down recently to num_handlers * 10 (so 100 by default) because it was easy to OOME yourself when huge calls were being queued. It's still possible to hit this problem if you use really big values and/or a lot of handlers, so the idea is that we should take into account the payload size. I can see 3 solutions:
> - Do the accounting outside of the queue itself for all calls coming in and out and when a call doesn't fit, throw a retryable exception.
> - Same accounting but instead block the call when it comes in until space is made available.
> - Add a new parameter for the maximum size (in bytes) of a Call and then set the size the IPC queue (in terms of the number of items) so that it could only contain as many items as some predefined maximum size (in bytes) for the whole queue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5190) Limit the IPC queue size based on
calls' payload size
Posted by "Jean-Daniel Cryans (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13185909#comment-13185909 ]
Jean-Daniel Cryans commented on HBASE-5190:
-------------------------------------------
About #3.
I like the elegance of that solution since we don't have to keep track of the calls in flight but I see 2 big issues:
- if you set a max call size you need to keep both clients and servers in sink and also decide who's going to do the check.
- if you plan for big calls by default, you may end up with a tiny size for the queue. For example, let's say you cap calls at 10% of the heap and set their max individual size at 10MB, it means that you can only allow 10 items in the queue (and you don't account listeners).
> Limit the IPC queue size based on calls' payload size
> -----------------------------------------------------
>
> Key: HBASE-5190
> URL: https://issues.apache.org/jira/browse/HBASE-5190
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.5
> Reporter: Jean-Daniel Cryans
> Fix For: 0.94.0
>
>
> Currently we limit the number of calls in the IPC queue only on their count. It used to be really high and was dropped down recently to num_handlers * 10 (so 100 by default) because it was easy to OOME yourself when huge calls were being queued. It's still possible to hit this problem if you use really big values and/or a lot of handlers, so the idea is that we should take into account the payload size. I can see 3 solutions:
> - Do the accounting outside of the queue itself for all calls coming in and out and when a call doesn't fit, throw a retryable exception.
> - Same accounting but instead block the call when it comes in until space is made available.
> - Add a new parameter for the maximum size (in bytes) of a Call and then set the size the IPC queue (in terms of the number of items) so that it could only contain as many items as some predefined maximum size (in bytes) for the whole queue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5190) Limit the IPC queue size based on
calls' payload size
Posted by "Ted Yu (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ted Yu updated HBASE-5190:
--------------------------
Attachment: 5190.addendum
Suggested addendum.
@J-D:
Please take a look.
> Limit the IPC queue size based on calls' payload size
> -----------------------------------------------------
>
> Key: HBASE-5190
> URL: https://issues.apache.org/jira/browse/HBASE-5190
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.5
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.94.0, 0.96.0
>
> Attachments: 5190.addendum, HBASE-5190-v2.patch, HBASE-5190-v3.patch, HBASE-5190.patch
>
>
> Currently we limit the number of calls in the IPC queue only on their count. It used to be really high and was dropped down recently to num_handlers * 10 (so 100 by default) because it was easy to OOME yourself when huge calls were being queued. It's still possible to hit this problem if you use really big values and/or a lot of handlers, so the idea is that we should take into account the payload size. I can see 3 solutions:
> - Do the accounting outside of the queue itself for all calls coming in and out and when a call doesn't fit, throw a retryable exception.
> - Same accounting but instead block the call when it comes in until space is made available.
> - Add a new parameter for the maximum size (in bytes) of a Call and then set the size the IPC queue (in terms of the number of items) so that it could only contain as many items as some predefined maximum size (in bytes) for the whole queue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5190) Limit the IPC queue size based on
calls' payload size
Posted by "Jean-Daniel Cryans (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13185943#comment-13185943 ]
Jean-Daniel Cryans commented on HBASE-5190:
-------------------------------------------
About #1.
The issue with this is that by throwing the call back to the client you generate a lot more IO and CPU (serializing, deserializing) and there's the possibility the of starving those clients that have bigger calls.
> Limit the IPC queue size based on calls' payload size
> -----------------------------------------------------
>
> Key: HBASE-5190
> URL: https://issues.apache.org/jira/browse/HBASE-5190
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.5
> Reporter: Jean-Daniel Cryans
> Fix For: 0.94.0
>
>
> Currently we limit the number of calls in the IPC queue only on their count. It used to be really high and was dropped down recently to num_handlers * 10 (so 100 by default) because it was easy to OOME yourself when huge calls were being queued. It's still possible to hit this problem if you use really big values and/or a lot of handlers, so the idea is that we should take into account the payload size. I can see 3 solutions:
> - Do the accounting outside of the queue itself for all calls coming in and out and when a call doesn't fit, throw a retryable exception.
> - Same accounting but instead block the call when it comes in until space is made available.
> - Add a new parameter for the maximum size (in bytes) of a Call and then set the size the IPC queue (in terms of the number of items) so that it could only contain as many items as some predefined maximum size (in bytes) for the whole queue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5190) Limit the IPC queue size based on
calls' payload size
Posted by "Todd Lipcon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13186047#comment-13186047 ]
Todd Lipcon commented on HBASE-5190:
------------------------------------
I haven't looked at the patch yet (sorry) but it seems from the discussion that this is trying to do the limiting at queue level. If instead, we did the limiting where we read from the network socket, then we'd have an automatic backpressure mechanism - clients would block on send, and the memory usage would be pushed to the client side TCP buffers rather than the HBase heap.
The downside is, of course, that META requests and such can get blocked behind others. But that can be solved other ways as well (eg separate TCP ports for meta regions or somesuch)
> Limit the IPC queue size based on calls' payload size
> -----------------------------------------------------
>
> Key: HBASE-5190
> URL: https://issues.apache.org/jira/browse/HBASE-5190
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.5
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.94.0
>
> Attachments: HBASE-5190.patch
>
>
> Currently we limit the number of calls in the IPC queue only on their count. It used to be really high and was dropped down recently to num_handlers * 10 (so 100 by default) because it was easy to OOME yourself when huge calls were being queued. It's still possible to hit this problem if you use really big values and/or a lot of handlers, so the idea is that we should take into account the payload size. I can see 3 solutions:
> - Do the accounting outside of the queue itself for all calls coming in and out and when a call doesn't fit, throw a retryable exception.
> - Same accounting but instead block the call when it comes in until space is made available.
> - Add a new parameter for the maximum size (in bytes) of a Call and then set the size the IPC queue (in terms of the number of items) so that it could only contain as many items as some predefined maximum size (in bytes) for the whole queue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5190) Limit the IPC queue size based on
calls' payload size
Posted by "Lars Hofhansl (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13236227#comment-13236227 ]
Lars Hofhansl commented on HBASE-5190:
--------------------------------------
Actually, can we new increase the default queue length?
For small requests 10 is awfully small.
> Limit the IPC queue size based on calls' payload size
> -----------------------------------------------------
>
> Key: HBASE-5190
> URL: https://issues.apache.org/jira/browse/HBASE-5190
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.5
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.94.0
>
> Attachments: HBASE-5190-v2.patch, HBASE-5190-v3.patch, HBASE-5190.patch
>
>
> Currently we limit the number of calls in the IPC queue only on their count. It used to be really high and was dropped down recently to num_handlers * 10 (so 100 by default) because it was easy to OOME yourself when huge calls were being queued. It's still possible to hit this problem if you use really big values and/or a lot of handlers, so the idea is that we should take into account the payload size. I can see 3 solutions:
> - Do the accounting outside of the queue itself for all calls coming in and out and when a call doesn't fit, throw a retryable exception.
> - Same accounting but instead block the call when it comes in until space is made available.
> - Add a new parameter for the maximum size (in bytes) of a Call and then set the size the IPC queue (in terms of the number of items) so that it could only contain as many items as some predefined maximum size (in bytes) for the whole queue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5190) Limit the IPC queue size based on
calls' payload size
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13236354#comment-13236354 ]
stack commented on HBASE-5190:
------------------------------
bq. Actually, can we new increase the default queue length?
This is a good suggestion
> Limit the IPC queue size based on calls' payload size
> -----------------------------------------------------
>
> Key: HBASE-5190
> URL: https://issues.apache.org/jira/browse/HBASE-5190
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.5
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.94.0
>
> Attachments: HBASE-5190-v2.patch, HBASE-5190-v3.patch, HBASE-5190.patch
>
>
> Currently we limit the number of calls in the IPC queue only on their count. It used to be really high and was dropped down recently to num_handlers * 10 (so 100 by default) because it was easy to OOME yourself when huge calls were being queued. It's still possible to hit this problem if you use really big values and/or a lot of handlers, so the idea is that we should take into account the payload size. I can see 3 solutions:
> - Do the accounting outside of the queue itself for all calls coming in and out and when a call doesn't fit, throw a retryable exception.
> - Same accounting but instead block the call when it comes in until space is made available.
> - Add a new parameter for the maximum size (in bytes) of a Call and then set the size the IPC queue (in terms of the number of items) so that it could only contain as many items as some predefined maximum size (in bytes) for the whole queue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5190) Limit the IPC queue size based on
calls' payload size
Posted by "Jean-Daniel Cryans (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans updated HBASE-5190:
--------------------------------------
Attachment: HBASE-5190-v2.patch
New patch: maximum queue size is bumped to 1GB, the exception is now sent back to the client.
> Limit the IPC queue size based on calls' payload size
> -----------------------------------------------------
>
> Key: HBASE-5190
> URL: https://issues.apache.org/jira/browse/HBASE-5190
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.5
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.94.0
>
> Attachments: HBASE-5190-v2.patch, HBASE-5190.patch
>
>
> Currently we limit the number of calls in the IPC queue only on their count. It used to be really high and was dropped down recently to num_handlers * 10 (so 100 by default) because it was easy to OOME yourself when huge calls were being queued. It's still possible to hit this problem if you use really big values and/or a lot of handlers, so the idea is that we should take into account the payload size. I can see 3 solutions:
> - Do the accounting outside of the queue itself for all calls coming in and out and when a call doesn't fit, throw a retryable exception.
> - Same accounting but instead block the call when it comes in until space is made available.
> - Add a new parameter for the maximum size (in bytes) of a Call and then set the size the IPC queue (in terms of the number of items) so that it could only contain as many items as some predefined maximum size (in bytes) for the whole queue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5190) Limit the IPC queue size based on
calls' payload size
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13185898#comment-13185898 ]
stack commented on HBASE-5190:
------------------------------
Talking w/ J-D on backchannel, #2 is bad (blocking is what we currently do too so its bad toooooo).
If we block, its likely client can timeout. If it times out, it will redo the request adding a new payload to the serverside ipc queue. Now queue has same load twice (bad if ICV). Server doesn't know client timed out till it goes to reply and the client is no longer there so it can double-count or double enter value.
> Limit the IPC queue size based on calls' payload size
> -----------------------------------------------------
>
> Key: HBASE-5190
> URL: https://issues.apache.org/jira/browse/HBASE-5190
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.5
> Reporter: Jean-Daniel Cryans
> Fix For: 0.94.0
>
>
> Currently we limit the number of calls in the IPC queue only on their count. It used to be really high and was dropped down recently to num_handlers * 10 (so 100 by default) because it was easy to OOME yourself when huge calls were being queued. It's still possible to hit this problem if you use really big values and/or a lot of handlers, so the idea is that we should take into account the payload size. I can see 3 solutions:
> - Do the accounting outside of the queue itself for all calls coming in and out and when a call doesn't fit, throw a retryable exception.
> - Same accounting but instead block the call when it comes in until space is made available.
> - Add a new parameter for the maximum size (in bytes) of a Call and then set the size the IPC queue (in terms of the number of items) so that it could only contain as many items as some predefined maximum size (in bytes) for the whole queue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5190) Limit the IPC queue size based on
calls' payload size
Posted by "Zhihong Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13186052#comment-13186052 ]
Zhihong Yu commented on HBASE-5190:
-----------------------------------
{code}
- this.priorityCallQueue = new LinkedBlockingQueue<Call>(maxQueueSize); // TODO hack on size
+ this.priorityCallQueue = new LinkedBlockingQueue<Call>(maxQueueLength); // TODO hack on size
{code}
The TODO above can be dropped.
Todd's comment makes sense. But since it requires more work, we can checkin this patch for now.
> Limit the IPC queue size based on calls' payload size
> -----------------------------------------------------
>
> Key: HBASE-5190
> URL: https://issues.apache.org/jira/browse/HBASE-5190
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.5
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.94.0
>
> Attachments: HBASE-5190.patch
>
>
> Currently we limit the number of calls in the IPC queue only on their count. It used to be really high and was dropped down recently to num_handlers * 10 (so 100 by default) because it was easy to OOME yourself when huge calls were being queued. It's still possible to hit this problem if you use really big values and/or a lot of handlers, so the idea is that we should take into account the payload size. I can see 3 solutions:
> - Do the accounting outside of the queue itself for all calls coming in and out and when a call doesn't fit, throw a retryable exception.
> - Same accounting but instead block the call when it comes in until space is made available.
> - Add a new parameter for the maximum size (in bytes) of a Call and then set the size the IPC queue (in terms of the number of items) so that it could only contain as many items as some predefined maximum size (in bytes) for the whole queue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5190) Limit the IPC queue size based on
calls' payload size
Posted by "Jean-Daniel Cryans (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13186064#comment-13186064 ]
Jean-Daniel Cryans commented on HBASE-5190:
-------------------------------------------
bq. The TODO above can be dropped
My understanding of that todo is that it shouldn't be setting as many PRI as there's normal handlers, so it still stands.
bq. Todd's comment makes sense
I agree.
bq. But since it requires more work, we can checkin this patch for now.
I disagree, I'll investigate his idea and I'm in no hurry to check this in since it's targeted for 0.94
> Limit the IPC queue size based on calls' payload size
> -----------------------------------------------------
>
> Key: HBASE-5190
> URL: https://issues.apache.org/jira/browse/HBASE-5190
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.5
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.94.0
>
> Attachments: HBASE-5190.patch
>
>
> Currently we limit the number of calls in the IPC queue only on their count. It used to be really high and was dropped down recently to num_handlers * 10 (so 100 by default) because it was easy to OOME yourself when huge calls were being queued. It's still possible to hit this problem if you use really big values and/or a lot of handlers, so the idea is that we should take into account the payload size. I can see 3 solutions:
> - Do the accounting outside of the queue itself for all calls coming in and out and when a call doesn't fit, throw a retryable exception.
> - Same accounting but instead block the call when it comes in until space is made available.
> - Add a new parameter for the maximum size (in bytes) of a Call and then set the size the IPC queue (in terms of the number of items) so that it could only contain as many items as some predefined maximum size (in bytes) for the whole queue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-5190) Limit the IPC queue size based on
calls' payload size
Posted by "Jean-Daniel Cryans (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans reassigned HBASE-5190:
-----------------------------------------
Assignee: Jean-Daniel Cryans
> Limit the IPC queue size based on calls' payload size
> -----------------------------------------------------
>
> Key: HBASE-5190
> URL: https://issues.apache.org/jira/browse/HBASE-5190
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.5
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.94.0
>
>
> Currently we limit the number of calls in the IPC queue only on their count. It used to be really high and was dropped down recently to num_handlers * 10 (so 100 by default) because it was easy to OOME yourself when huge calls were being queued. It's still possible to hit this problem if you use really big values and/or a lot of handlers, so the idea is that we should take into account the payload size. I can see 3 solutions:
> - Do the accounting outside of the queue itself for all calls coming in and out and when a call doesn't fit, throw a retryable exception.
> - Same accounting but instead block the call when it comes in until space is made available.
> - Add a new parameter for the maximum size (in bytes) of a Call and then set the size the IPC queue (in terms of the number of items) so that it could only contain as many items as some predefined maximum size (in bytes) for the whole queue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5190) Limit the IPC queue size based on
calls' payload size
Posted by "Lars Hofhansl (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13237124#comment-13237124 ]
Lars Hofhansl commented on HBASE-5190:
--------------------------------------
Wanna commit, J-D?
> Limit the IPC queue size based on calls' payload size
> -----------------------------------------------------
>
> Key: HBASE-5190
> URL: https://issues.apache.org/jira/browse/HBASE-5190
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.5
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.94.0
>
> Attachments: HBASE-5190-v2.patch, HBASE-5190-v3.patch, HBASE-5190.patch
>
>
> Currently we limit the number of calls in the IPC queue only on their count. It used to be really high and was dropped down recently to num_handlers * 10 (so 100 by default) because it was easy to OOME yourself when huge calls were being queued. It's still possible to hit this problem if you use really big values and/or a lot of handlers, so the idea is that we should take into account the payload size. I can see 3 solutions:
> - Do the accounting outside of the queue itself for all calls coming in and out and when a call doesn't fit, throw a retryable exception.
> - Same accounting but instead block the call when it comes in until space is made available.
> - Add a new parameter for the maximum size (in bytes) of a Call and then set the size the IPC queue (in terms of the number of items) so that it could only contain as many items as some predefined maximum size (in bytes) for the whole queue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5190) Limit the IPC queue size based on
calls' payload size
Posted by "Lars Hofhansl (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13237977#comment-13237977 ]
Lars Hofhansl commented on HBASE-5190:
--------------------------------------
+1 on addendum
> Limit the IPC queue size based on calls' payload size
> -----------------------------------------------------
>
> Key: HBASE-5190
> URL: https://issues.apache.org/jira/browse/HBASE-5190
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.5
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.94.0, 0.96.0
>
> Attachments: 5190.addendum, HBASE-5190-v2.patch, HBASE-5190-v3.patch, HBASE-5190.patch
>
>
> Currently we limit the number of calls in the IPC queue only on their count. It used to be really high and was dropped down recently to num_handlers * 10 (so 100 by default) because it was easy to OOME yourself when huge calls were being queued. It's still possible to hit this problem if you use really big values and/or a lot of handlers, so the idea is that we should take into account the payload size. I can see 3 solutions:
> - Do the accounting outside of the queue itself for all calls coming in and out and when a call doesn't fit, throw a retryable exception.
> - Same accounting but instead block the call when it comes in until space is made available.
> - Add a new parameter for the maximum size (in bytes) of a Call and then set the size the IPC queue (in terms of the number of items) so that it could only contain as many items as some predefined maximum size (in bytes) for the whole queue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5190) Limit the IPC queue size based on
calls' payload size
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13238999#comment-13238999 ]
Hudson commented on HBASE-5190:
-------------------------------
Integrated in HBase-0.94-security #3 (See [https://builds.apache.org/job/HBase-0.94-security/3/])
HBASE-5190 Limit the IPC queue size based on calls' payload size
(Ted's addendum) (Revision 1305469)
Result = ABORTED
jdcryans :
Files :
* /hbase/branches/0.94/security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java
> Limit the IPC queue size based on calls' payload size
> -----------------------------------------------------
>
> Key: HBASE-5190
> URL: https://issues.apache.org/jira/browse/HBASE-5190
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.5
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.94.0, 0.96.0
>
> Attachments: 5190.addendum, HBASE-5190-v2.patch, HBASE-5190-v3.patch, HBASE-5190.patch
>
>
> Currently we limit the number of calls in the IPC queue only on their count. It used to be really high and was dropped down recently to num_handlers * 10 (so 100 by default) because it was easy to OOME yourself when huge calls were being queued. It's still possible to hit this problem if you use really big values and/or a lot of handlers, so the idea is that we should take into account the payload size. I can see 3 solutions:
> - Do the accounting outside of the queue itself for all calls coming in and out and when a call doesn't fit, throw a retryable exception.
> - Same accounting but instead block the call when it comes in until space is made available.
> - Add a new parameter for the maximum size (in bytes) of a Call and then set the size the IPC queue (in terms of the number of items) so that it could only contain as many items as some predefined maximum size (in bytes) for the whole queue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5190) Limit the IPC queue size based on
calls' payload size
Posted by "Zhihong Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13186043#comment-13186043 ]
Zhihong Yu commented on HBASE-5190:
-----------------------------------
Can we utilize Cliff Click Counter to reduce contention ?
> Limit the IPC queue size based on calls' payload size
> -----------------------------------------------------
>
> Key: HBASE-5190
> URL: https://issues.apache.org/jira/browse/HBASE-5190
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.5
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.94.0
>
> Attachments: HBASE-5190.patch
>
>
> Currently we limit the number of calls in the IPC queue only on their count. It used to be really high and was dropped down recently to num_handlers * 10 (so 100 by default) because it was easy to OOME yourself when huge calls were being queued. It's still possible to hit this problem if you use really big values and/or a lot of handlers, so the idea is that we should take into account the payload size. I can see 3 solutions:
> - Do the accounting outside of the queue itself for all calls coming in and out and when a call doesn't fit, throw a retryable exception.
> - Same accounting but instead block the call when it comes in until space is made available.
> - Add a new parameter for the maximum size (in bytes) of a Call and then set the size the IPC queue (in terms of the number of items) so that it could only contain as many items as some predefined maximum size (in bytes) for the whole queue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira