You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Kannan Muthukkaruppan (JIRA)" <ji...@apache.org> on 2012/10/24 20:08:12 UTC
[jira] [Comment Edited] (HBASE-6728) [89-fb] prevent OOM
possibility due to per connection responseQueue being unbounded
[ https://issues.apache.org/jira/browse/HBASE-6728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13483438#comment-13483438 ]
Kannan Muthukkaruppan edited comment on HBASE-6728 at 10/24/12 6:06 PM:
------------------------------------------------------------------------
Lars wrote: <<<Will this work right if I set scanner caching to 1000 and then deal with 2mb rows? In that case every response will be 2g, and it would always block and never make any progress, right?>>
Yes, we considered that, and that's the reason for not using a simple counting semaphore that's initialized to the max size. We want the implementation to allow one request to exceed the queue size. We set the default at 1G, but we can exceed the limit by 1 requests' size amount.
>From http://svn.apache.org/viewvc/hbase/branches/0.89-fb/src/main/java/org/apache/hadoop/hbase/util/SizeBasedThrottler.java?view=markup&pathrev=1385058 :
{code}
15 * This implementation allows you to set the value of internal
16 * counter to be greater than threshold. It happens
17 * when internal counter is lower than threshold and
18 * increase method is called with parameter 'delta' big enough
19 * so that sum of delta and internal counter is greater than
20 * threshold. This is not a bug, this is a feature.
21 * It solves some problems:
22 * - thread calling increase with big parameter will not be
23 * starved by other threads calling increase with small
24 * arguments.
25 * - thread calling increase with argument greater than
26 * threshold won't deadlock. This is useful when throttling
27 * queues - you can submit object that is bigger than limit.
28 *
29 * This implementation introduces small costs in terms of
30 * synchronization (no synchronization in most cases at all), but is
31 * vulnerable to races. For details see documentation of
32 * increase method.
{code}
was (Author: kannanm):
Lars wrote: <<<Will this work right if I set scanner caching to 1000 and then deal with 2mb rows? In that case every response will be 2g, and it would always block and never make any progress, right?>>
Yes, we considered that, and that's the reason for not using a simple counting semaphore that's initialize to the max size. We want the implementation to allow one request to exceed the queue size. We set the default at 1G, but we can exceed the limit by 1 requests' size amount.
>From http://svn.apache.org/viewvc/hbase/branches/0.89-fb/src/main/java/org/apache/hadoop/hbase/util/SizeBasedThrottler.java?view=markup&pathrev=1385058 :
{code}
15 * This implementation allows you to set the value of internal
16 * counter to be greater than threshold. It happens
17 * when internal counter is lower than threshold and
18 * increase method is called with parameter 'delta' big enough
19 * so that sum of delta and internal counter is greater than
20 * threshold. This is not a bug, this is a feature.
21 * It solves some problems:
22 * - thread calling increase with big parameter will not be
23 * starved by other threads calling increase with small
24 * arguments.
25 * - thread calling increase with argument greater than
26 * threshold won't deadlock. This is useful when throttling
27 * queues - you can submit object that is bigger than limit.
28 *
29 * This implementation introduces small costs in terms of
30 * synchronization (no synchronization in most cases at all), but is
31 * vulnerable to races. For details see documentation of
32 * increase method.
{code}
> [89-fb] prevent OOM possibility due to per connection responseQueue being unbounded
> -----------------------------------------------------------------------------------
>
> Key: HBASE-6728
> URL: https://issues.apache.org/jira/browse/HBASE-6728
> Project: HBase
> Issue Type: Bug
> Reporter: Kannan Muthukkaruppan
> Assignee: Michal Gregorczyk
> Fix For: 0.94.3
>
> Attachments: 6728.94, 6728-trunk.txt
>
>
> The per connection responseQueue is an unbounded queue. The request handler threads today try to send the response in line, but if things start to backup, the response is sent via a per connection responder thread. This intermediate queue, because it has no bounds, can be another source of OOMs.
> [Have not looked at this issue in trunk. So it may or may not be applicable there.]
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira