You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org> on 2008/09/17 23:21:44 UTC
[jira] Commented: (HBASE-887) Fix a hotspot in scanners
[ https://issues.apache.org/jira/browse/HBASE-887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12631953#action_12631953 ]
Jean-Daniel Cryans commented on HBASE-887:
------------------------------------------
Some tests I did with a modified PE. I ran the PE with a number from 10 to 3000 and incremented by 10. The garbage collector was called after each run. Here are the results:
{quote}
08/09/17 16:48:54 INFO hbase.ManyPerformanceEvaluation: 10 42725
08/09/17 16:49:31 INFO hbase.ManyPerformanceEvaluation: 20 36744
08/09/17 16:50:06 INFO hbase.ManyPerformanceEvaluation: 30 34928
08/09/17 16:50:40 INFO hbase.ManyPerformanceEvaluation: 40 34150
08/09/17 16:51:14 INFO hbase.ManyPerformanceEvaluation: 50 33664
08/09/17 16:51:47 INFO hbase.ManyPerformanceEvaluation: 60 33570
08/09/17 16:52:21 INFO hbase.ManyPerformanceEvaluation: 70 33142
08/09/17 16:52:54 INFO hbase.ManyPerformanceEvaluation: 80 32901
08/09/17 16:53:26 INFO hbase.ManyPerformanceEvaluation: 90 32864
08/09/17 16:53:59 INFO hbase.ManyPerformanceEvaluation: 100 32776
08/09/17 16:54:32 INFO hbase.ManyPerformanceEvaluation: 110 32725
08/09/17 16:55:05 INFO hbase.ManyPerformanceEvaluation: 120 32747
08/09/17 16:55:38 INFO hbase.ManyPerformanceEvaluation: 130 32698
08/09/17 16:56:10 INFO hbase.ManyPerformanceEvaluation: 140 32693
08/09/17 16:56:43 INFO hbase.ManyPerformanceEvaluation: 150 32643
08/09/17 16:57:16 INFO hbase.ManyPerformanceEvaluation: 160 32686
08/09/17 16:57:48 INFO hbase.ManyPerformanceEvaluation: 170 32665
08/09/17 16:58:21 INFO hbase.ManyPerformanceEvaluation: 180 32448
08/09/17 16:58:53 INFO hbase.ManyPerformanceEvaluation: 190 32004
08/09/17 16:59:25 INFO hbase.ManyPerformanceEvaluation: 200 31985
08/09/17 16:59:57 INFO hbase.ManyPerformanceEvaluation: 210 31964
08/09/17 17:00:29 INFO hbase.ManyPerformanceEvaluation: 220 31983
08/09/17 17:01:01 INFO hbase.ManyPerformanceEvaluation: 230 31778
08/09/17 17:01:32 INFO hbase.ManyPerformanceEvaluation: 240 31766
08/09/17 17:02:04 INFO hbase.ManyPerformanceEvaluation: 250 31886
08/09/17 17:02:36 INFO hbase.ManyPerformanceEvaluation: 260 31773
08/09/17 17:03:08 INFO hbase.ManyPerformanceEvaluation: 270 31709
08/09/17 17:03:40 INFO hbase.ManyPerformanceEvaluation: 280 31669
08/09/17 17:04:11 INFO hbase.ManyPerformanceEvaluation: 290 31647
08/09/17 17:04:43 INFO hbase.ManyPerformanceEvaluation: 300 31634
08/09/17 17:05:15 INFO hbase.ManyPerformanceEvaluation: 310 31674
08/09/17 17:05:47 INFO hbase.ManyPerformanceEvaluation: 320 31935
08/09/17 17:06:18 INFO hbase.ManyPerformanceEvaluation: 330 31802
08/09/17 17:06:50 INFO hbase.ManyPerformanceEvaluation: 340 31571
08/09/17 17:07:21 INFO hbase.ManyPerformanceEvaluation: 350 31484
08/09/17 17:07:53 INFO hbase.ManyPerformanceEvaluation: 360 31713
08/09/17 17:08:25 INFO hbase.ManyPerformanceEvaluation: 370 31512
08/09/17 17:08:56 INFO hbase.ManyPerformanceEvaluation: 380 31456
08/09/17 17:09:28 INFO hbase.ManyPerformanceEvaluation: 390 32055
08/09/17 17:10:01 INFO hbase.ManyPerformanceEvaluation: 400 32260
08/09/17 17:10:33 INFO hbase.ManyPerformanceEvaluation: 410 32167
08/09/17 17:11:05 INFO hbase.ManyPerformanceEvaluation: 420 31962
08/09/17 17:11:37 INFO hbase.ManyPerformanceEvaluation: 430 32003
08/09/17 17:12:09 INFO hbase.ManyPerformanceEvaluation: 440 32054
08/09/17 17:12:41 INFO hbase.ManyPerformanceEvaluation: 450 32105
08/09/17 17:13:13 INFO hbase.ManyPerformanceEvaluation: 460 32126
08/09/17 17:13:45 INFO hbase.ManyPerformanceEvaluation: 470 31961
08/09/17 17:14:17 INFO hbase.ManyPerformanceEvaluation: 480 31968
...
{quote}
It seems that 2000 was an overkill. The gain is impressive between 1 and 10 then around 300 we pretty much reduced the RPC call at it's smallest value.
> Fix a hotspot in scanners
> -------------------------
>
> Key: HBASE-887
> URL: https://issues.apache.org/jira/browse/HBASE-887
> Project: Hadoop HBase
> Issue Type: Improvement
> Components: client, regionserver
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.19.0
>
> Attachments: hbase-887-v1.patch
>
>
> When scanning, we do a lot of RPCs and this has a huge performance hit. I propose that we add a way to fetch more rows during next() and put them in cache. This should be configurable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.