[jira] [Created] (PHOENIX-2982) Keep the number of active handlers balanced across salt buckets

["Junegunn Choi (JIRA)" <ji...@apache.org>]

Junegunn Choi created PHOENIX-2982:
--------------------------------------

             Summary: Keep the number of active handlers balanced across salt buckets
                 Key: PHOENIX-2982
                 URL: https://issues.apache.org/jira/browse/PHOENIX-2982
             Project: Phoenix
          Issue Type: Improvement
            Reporter: Junegunn Choi
            Assignee: Junegunn Choi


I'd like to discuss the idea of keeping the numbers of active handlers balanced across the salt buckets during parallel scan by exposing the counters to JobManager queue.

h4. Background

I was testing Phoenix on a 10-node test cluster. When I was running a few full-scan queries simultaneously on a table whose {{SALT_BUCKETS}} is 100, I noticed small queries such as {{SELECT * FROM T LIMIT 100}} are occasionally blocked for up to tens of seconds due to the exhaustion of regionserver handler threads.

{{hbase.regionserver.handler.count}} was set to 100, which is much larger than the default 30, so I didn't expect this to happen ({{phoenix.query.threadPoolSize}} = 128 / 10 nodes * 4 queries =~ 50 < 100), but from periodic thread dumps, I could observe that the numbers of active handlers across the regionservers are often skewed during the execution.

{noformat}
# Obtained by periodic thread dumps
# (key = regionserver ID, value = number of active handlers)
17:23:48: {1=>6,  2=>3,  3=>27, 4=>12, 5=>13, 6=>23, 7=>23, 8=>5,  9=>5,  10=>10}
17:24:18: {1=>8,  2=>6,  3=>26, 4=>3,  5=>13, 6=>41, 7=>11, 8=>5,  9=>8,  10=>5}
17:24:48: {1=>15, 3=>30, 4=>3,  5=>8,  6=>22, 7=>11, 8=>16, 9=>16, 10=>7}
17:25:18: {1=>6,  2=>12, 3=>37, 4=>6,  5=>4,  6=>2,  7=>21, 8=>10, 9=>24, 10=>5}
17:25:48: {1=>4,  2=>9,  3=>48, 4=>14, 5=>2,  6=>7,  7=>18, 8=>16, 9=>2,  10=>8}
{noformat}

Although {{ParallelIterators.submitWork}} shuffles the parallel scan tasks before submitting them to {{ThreadPoolExecutor}}, there's currently no mechanism to prevent the skew from happening during runtime.

h4. Suggestion

Maintain "active" counter for each salt bucket, and expose the numbers to JobManager queue via specialized {{Producer}} implementation so that it can choose a scan for the least loaded bucket.

By doing so we can prevent the handler exhaustion problem described above and can expect more consistent utilization of the resources.

h4. Evaluation

I measured the response times of a full-scan query with and without the patch. The plan for the query is given as follows:

{noformat}
> explain select count(*) from image;
+---------------------------------------------------------------------------------------------+
|                                            PLAN                                             |
+---------------------------------------------------------------------------------------------+
| CLIENT 2700-CHUNK 1395337221 ROWS 817889379319 BYTES PARALLEL 100-WAY FULL SCAN OVER IMAGE  |
|     SERVER FILTER BY FIRST KEY ONLY                                                         |
|     SERVER AGGREGATE INTO SINGLE ROW                                                        |
+---------------------------------------------------------------------------------------------+
{noformat}

Result:

- Without patch: 260 sec
- With patch: 220 sec

And with the patch, CPU utilizations of the regionservers are very stable during the run.

I'd like to hear what you think about the approach. Please let me know if there's any concerns.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)