You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "stack (JIRA)" <ji...@apache.org> on 2011/08/10 22:18:27 UTC
[jira] [Created] (HBASE-4188) Make in-memory table scanning faster,
part2 (continuation of hbase-1938)
Make in-memory table scanning faster, part2 (continuation of hbase-1938)
------------------------------------------------------------------------
Key: HBASE-4188
URL: https://issues.apache.org/jira/browse/HBASE-4188
Project: HBase
Issue Type: Improvement
Components: performance
Reporter: stack
Assignee: nkeywal
Priority: Critical
Fix For: 0.92.0
This issue is a continuation of hbase-1938 work (That issue is closed). This issue is about getting the last patch posted by nkeywal over in hbase-1938 applied (assigned nkeywal since he's done the work).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4188) Make in-memory table scanning
faster, part2 (continuation of hbase-1938)
Posted by "nkeywal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13098948#comment-13098948 ]
nkeywal commented on HBASE-4188:
--------------------------------
See HBASE-4195. It's the same test.
> Make in-memory table scanning faster, part2 (continuation of hbase-1938)
> ------------------------------------------------------------------------
>
> Key: HBASE-4188
> URL: https://issues.apache.org/jira/browse/HBASE-4188
> Project: HBase
> Issue Type: Improvement
> Components: performance
> Reporter: stack
> Assignee: nkeywal
> Priority: Critical
> Fix For: 0.92.0
>
> Attachments: 4188-mssp.txt
>
>
> This issue is a continuation of HBASE-1938 work (That issue is closed). This issue is about getting the last patch posted by nkeywal over in HBASE-1938 applied (assigned nkeywal since he's done the work).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4188) Make in-memory table scanning
faster, part2 (continuation of hbase-1938)
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13090047#comment-13090047 ]
stack commented on HBASE-4188:
------------------------------
Hmm... so I still see fails. Out of 42 runs, three failed which is less than previous....
> Make in-memory table scanning faster, part2 (continuation of hbase-1938)
> ------------------------------------------------------------------------
>
> Key: HBASE-4188
> URL: https://issues.apache.org/jira/browse/HBASE-4188
> Project: HBase
> Issue Type: Improvement
> Components: performance
> Reporter: stack
> Assignee: nkeywal
> Priority: Critical
> Fix For: 0.92.0
>
> Attachments: 4188-mssp.txt
>
>
> This issue is a continuation of hbase-1938 work (That issue is closed). This issue is about getting the last patch posted by nkeywal over in hbase-1938 applied (assigned nkeywal since he's done the work).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4188) Make in-memory table scanning
faster, part2 (continuation of hbase-1938)
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13090758#comment-13090758 ]
stack commented on HBASE-4188:
------------------------------
I changed the int flushInterval to 3 and now I get no failures. Trying 5.
> Make in-memory table scanning faster, part2 (continuation of hbase-1938)
> ------------------------------------------------------------------------
>
> Key: HBASE-4188
> URL: https://issues.apache.org/jira/browse/HBASE-4188
> Project: HBase
> Issue Type: Improvement
> Components: performance
> Reporter: stack
> Assignee: nkeywal
> Priority: Critical
> Fix For: 0.92.0
>
> Attachments: 4188-mssp.txt
>
>
> This issue is a continuation of hbase-1938 work (That issue is closed). This issue is about getting the last patch posted by nkeywal over in hbase-1938 applied (assigned nkeywal since he's done the work).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4188) Make in-memory table scanning
faster, part2 (continuation of hbase-1938)
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13090787#comment-13090787 ]
stack commented on HBASE-4188:
------------------------------
Same for 5. No errors. Putting it back to 10.
> Make in-memory table scanning faster, part2 (continuation of hbase-1938)
> ------------------------------------------------------------------------
>
> Key: HBASE-4188
> URL: https://issues.apache.org/jira/browse/HBASE-4188
> Project: HBase
> Issue Type: Improvement
> Components: performance
> Reporter: stack
> Assignee: nkeywal
> Priority: Critical
> Fix For: 0.92.0
>
> Attachments: 4188-mssp.txt
>
>
> This issue is a continuation of hbase-1938 work (That issue is closed). This issue is about getting the last patch posted by nkeywal over in hbase-1938 applied (assigned nkeywal since he's done the work).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4188) Make in-memory table scanning
faster, part2 (continuation of hbase-1938)
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13089821#comment-13089821 ]
stack commented on HBASE-4188:
------------------------------
So, it looks like timing difference seen by Andrew is no longer present: with patch loads happen faster and though the timings have wide variance, after a few runs, with patch we seem to faster.
Here is a run of MemStoreScanPerformance before I applied above patch:
{code}
stack-7:0.90 Stack$ ./bin/hbase org.apache.hadoop.hbase.regionserver.MemStoreScanPerformance
JUnit version 4.8.1
.Loaded in 2479 ms
Scan with size 50000: 177 ms
Scan with size 50000: 185 ms
Scan with size 50000: 234 ms
Scan with size 50000: 86 ms
Scan with size 50000: 115 ms
Scan with size 50000: 92 ms
Scan with size 50000: 53 ms
Scan with size 50000: 58 ms
Scan with size 50000: 69 ms
Scan with size 50000: 61 ms
Loaded in 3134 ms
Scan with size 75000: 77 ms
Scan with size 75000: 87 ms
Scan with size 75000: 88 ms
Scan with size 75000: 84 ms
Scan with size 75000: 71 ms
Scan with size 75000: 110 ms
Scan with size 75000: 109 ms
Scan with size 75000: 100 ms
Scan with size 75000: 114 ms
Scan with size 75000: 111 ms
Loaded in 4748 ms
Scan with size 100000: 101 ms
Scan with size 100000: 85 ms
Scan with size 100000: 75 ms
Scan with size 100000: 91 ms
Scan with size 100000: 106 ms
Scan with size 100000: 84 ms
Scan with size 100000: 100 ms
Scan with size 100000: 91 ms
Scan with size 100000: 102 ms
Scan with size 100000: 81 ms
Loaded in 5904 ms
Scan with size 125000: 138 ms
Scan with size 125000: 175 ms
Scan with size 125000: 163 ms
Scan with size 125000: 145 ms
Scan with size 125000: 150 ms
Scan with size 125000: 122 ms
Scan with size 125000: 112 ms
Scan with size 125000: 118 ms
Scan with size 125000: 146 ms
Scan with size 125000: 136 ms
Loaded in 6125 ms
Scan with size 150000: 126 ms
Scan with size 150000: 106 ms
Scan with size 150000: 117 ms
Scan with size 150000: 119 ms
Scan with size 150000: 118 ms
Scan with size 150000: 118 ms
Scan with size 150000: 126 ms
Scan with size 150000: 122 ms
Scan with size 150000: 127 ms
Scan with size 150000: 117 ms
Total load time: 22390 ms (i.e:22 seconds)
Total scan time: 5597 ms (i.e:5 seconds)
Rows scanned per seconds: 893335
Rows loaded per seconds: 223313
Time: 28.787
OK (1 test)
{code}
Here is afterward:
{code}
stack-7:0.90 Stack$ ./bin/hbase org.apache.hadoop.hbase.regionserver.MemStoreScanPerformance
JUnit version 4.8.1
.Loaded in 2293 ms
Scan with size 50000: 147 ms
Scan with size 50000: 195 ms
Scan with size 50000: 164 ms
Scan with size 50000: 88 ms
Scan with size 50000: 98 ms
Scan with size 50000: 80 ms
Scan with size 50000: 86 ms
Scan with size 50000: 57 ms
Scan with size 50000: 61 ms
Scan with size 50000: 48 ms
Loaded in 2447 ms
Scan with size 75000: 54 ms
Scan with size 75000: 94 ms
Scan with size 75000: 60 ms
Scan with size 75000: 61 ms
Scan with size 75000: 75 ms
Scan with size 75000: 59 ms
Scan with size 75000: 69 ms
Scan with size 75000: 71 ms
Scan with size 75000: 83 ms
Scan with size 75000: 57 ms
Loaded in 3381 ms
Scan with size 100000: 81 ms
Scan with size 100000: 84 ms
Scan with size 100000: 84 ms
Scan with size 100000: 85 ms
Scan with size 100000: 83 ms
Scan with size 100000: 85 ms
Scan with size 100000: 108 ms
Scan with size 100000: 78 ms
Scan with size 100000: 85 ms
Scan with size 100000: 75 ms
Loaded in 4395 ms
Scan with size 125000: 102 ms
Scan with size 125000: 104 ms
Scan with size 125000: 106 ms
Scan with size 125000: 88 ms
Scan with size 125000: 109 ms
Scan with size 125000: 99 ms
Scan with size 125000: 99 ms
Scan with size 125000: 110 ms
Scan with size 125000: 105 ms
Scan with size 125000: 105 ms
Loaded in 6552 ms
Scan with size 150000: 110 ms
Scan with size 150000: 112 ms
Scan with size 150000: 107 ms
Scan with size 150000: 126 ms
Scan with size 150000: 114 ms
Scan with size 150000: 118 ms
Scan with size 150000: 131 ms
Scan with size 150000: 110 ms
Scan with size 150000: 113 ms
Scan with size 150000: 117 ms
Total load time: 19068 ms (i.e:19 seconds)
Total scan time: 4731 ms (i.e:4 seconds)
Rows scanned per seconds: 1056859
Rows loaded per seconds: 262219
Time: 24.52
OK (1 test)
stack-7:0.90 Stack$
{code}
> Make in-memory table scanning faster, part2 (continuation of hbase-1938)
> ------------------------------------------------------------------------
>
> Key: HBASE-4188
> URL: https://issues.apache.org/jira/browse/HBASE-4188
> Project: HBase
> Issue Type: Improvement
> Components: performance
> Reporter: stack
> Assignee: nkeywal
> Priority: Critical
> Fix For: 0.92.0
>
>
> This issue is a continuation of hbase-1938 work (That issue is closed). This issue is about getting the last patch posted by nkeywal over in hbase-1938 applied (assigned nkeywal since he's done the work).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4188) Make in-memory table scanning
faster, part2 (continuation of hbase-1938)
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13089796#comment-13089796 ]
stack commented on HBASE-4188:
------------------------------
Here is the patch we are talking about: https://issues.apache.org/jira/secure/attachment/12489104/20110802_MemStore.patch
Patch looks good. Running 50 TestHRegions on 0.90 in a row....
> Make in-memory table scanning faster, part2 (continuation of hbase-1938)
> ------------------------------------------------------------------------
>
> Key: HBASE-4188
> URL: https://issues.apache.org/jira/browse/HBASE-4188
> Project: HBase
> Issue Type: Improvement
> Components: performance
> Reporter: stack
> Assignee: nkeywal
> Priority: Critical
> Fix For: 0.92.0
>
>
> This issue is a continuation of hbase-1938 work (That issue is closed). This issue is about getting the last patch posted by nkeywal over in hbase-1938 applied (assigned nkeywal since he's done the work).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Work started] (HBASE-4188) Make in-memory table scanning
faster, part2 (continuation of hbase-1938)
Posted by "nkeywal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Work on HBASE-4188 started by nkeywal.
> Make in-memory table scanning faster, part2 (continuation of hbase-1938)
> ------------------------------------------------------------------------
>
> Key: HBASE-4188
> URL: https://issues.apache.org/jira/browse/HBASE-4188
> Project: HBase
> Issue Type: Improvement
> Components: performance
> Reporter: stack
> Assignee: nkeywal
> Priority: Critical
> Fix For: 0.92.0
>
> Attachments: 4188-mssp.txt
>
>
> This issue is a continuation of HBASE-1938 work (That issue is closed). This issue is about getting the last patch posted by nkeywal over in HBASE-1938 applied (assigned nkeywal since he's done the work).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4188) Make in-memory table scanning
faster, part2 (continuation of hbase-1938)
Posted by "nkeywal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13090625#comment-13090625 ]
nkeywal commented on HBASE-4188:
--------------------------------
Yes, I would expect them to be the scenario with consistency between read
and flush mentioned in the bug
HBASE-4195<https://issues.apache.org/jira/browse/HBASE-4195>(and
already managed in
HBASE-2856 <https://issues.apache.org/jira/browse/HBASE-2856>). You can
check this by changing "int flushInterval = 10;" in the method
"testWritesWhileGetting", for example by setting it to 2 or 3. This should
increase the number of failure. As well, the MemStoreTS of the KV in error
will be 0 (it comes from a file). This error (or at least the error I
analyzed) comes from the fact that the flush is finished before the end of
the Store#next, and not directly to the MemStore.
A xarning as well the patch in
HBASE-4195<https://issues.apache.org/jira/browse/HBASE-4195>will
conflict with this one, because they both touch the "seek" & "reseek"
function (even if it is actually different parts that are modified). Nothing
complex, but I won't be able to do the followup on this until end of next
week.
> Make in-memory table scanning faster, part2 (continuation of hbase-1938)
> ------------------------------------------------------------------------
>
> Key: HBASE-4188
> URL: https://issues.apache.org/jira/browse/HBASE-4188
> Project: HBase
> Issue Type: Improvement
> Components: performance
> Reporter: stack
> Assignee: nkeywal
> Priority: Critical
> Fix For: 0.92.0
>
> Attachments: 4188-mssp.txt
>
>
> This issue is a continuation of hbase-1938 work (That issue is closed). This issue is about getting the last patch posted by nkeywal over in hbase-1938 applied (assigned nkeywal since he's done the work).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-4188) Make in-memory table scanning
faster, part2 (continuation of hbase-1938)
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack resolved HBASE-4188.
--------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Resolving as fixed by hbase-4195 application.
> Make in-memory table scanning faster, part2 (continuation of hbase-1938)
> ------------------------------------------------------------------------
>
> Key: HBASE-4188
> URL: https://issues.apache.org/jira/browse/HBASE-4188
> Project: HBase
> Issue Type: Improvement
> Components: performance
> Reporter: stack
> Assignee: nkeywal
> Priority: Critical
> Fix For: 0.92.0
>
> Attachments: 4188-mssp.txt
>
>
> This issue is a continuation of HBASE-1938 work (That issue is closed). This issue is about getting the last patch posted by nkeywal over in HBASE-1938 applied (assigned nkeywal since he's done the work).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4188) Make in-memory table scanning faster,
part2 (continuation of hbase-1938)
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ted Yu updated HBASE-4188:
--------------------------
Description: This issue is a continuation of HBASE-1938 work (That issue is closed). This issue is about getting the last patch posted by nkeywal over in HBASE-1938 applied (assigned nkeywal since he's done the work). (was: This issue is a continuation of hbase-1938 work (That issue is closed). This issue is about getting the last patch posted by nkeywal over in hbase-1938 applied (assigned nkeywal since he's done the work).)
> Make in-memory table scanning faster, part2 (continuation of hbase-1938)
> ------------------------------------------------------------------------
>
> Key: HBASE-4188
> URL: https://issues.apache.org/jira/browse/HBASE-4188
> Project: HBase
> Issue Type: Improvement
> Components: performance
> Reporter: stack
> Assignee: nkeywal
> Priority: Critical
> Fix For: 0.92.0
>
> Attachments: 4188-mssp.txt
>
>
> This issue is a continuation of HBASE-1938 work (That issue is closed). This issue is about getting the last patch posted by nkeywal over in HBASE-1938 applied (assigned nkeywal since he's done the work).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4188) Make in-memory table scanning
faster, part2 (continuation of hbase-1938)
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13090558#comment-13090558 ]
stack commented on HBASE-4188:
------------------------------
Did more. 8 of 60 runs failed. All were testWritesWhileGetting failures.
> Make in-memory table scanning faster, part2 (continuation of hbase-1938)
> ------------------------------------------------------------------------
>
> Key: HBASE-4188
> URL: https://issues.apache.org/jira/browse/HBASE-4188
> Project: HBase
> Issue Type: Improvement
> Components: performance
> Reporter: stack
> Assignee: nkeywal
> Priority: Critical
> Fix For: 0.92.0
>
> Attachments: 4188-mssp.txt
>
>
> This issue is a continuation of hbase-1938 work (That issue is closed). This issue is about getting the last patch posted by nkeywal over in hbase-1938 applied (assigned nkeywal since he's done the work).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4188) Make in-memory table scanning
faster, part2 (continuation of hbase-1938)
Posted by "nkeywal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13106634#comment-13106634 ]
nkeywal commented on HBASE-4188:
--------------------------------
Patch for this jira bug is included in HBASE-4195.
> Make in-memory table scanning faster, part2 (continuation of hbase-1938)
> ------------------------------------------------------------------------
>
> Key: HBASE-4188
> URL: https://issues.apache.org/jira/browse/HBASE-4188
> Project: HBase
> Issue Type: Improvement
> Components: performance
> Reporter: stack
> Assignee: nkeywal
> Priority: Critical
> Fix For: 0.92.0
>
> Attachments: 4188-mssp.txt
>
>
> This issue is a continuation of HBASE-1938 work (That issue is closed). This issue is about getting the last patch posted by nkeywal over in HBASE-1938 applied (assigned nkeywal since he's done the work).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4188) Make in-memory table scanning faster,
part2 (continuation of hbase-1938)
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-4188:
-------------------------
Attachment: 4188-mssp.txt
Here is MemStoreScanPerformance with apache license. We should add this when we commit this patch.
> Make in-memory table scanning faster, part2 (continuation of hbase-1938)
> ------------------------------------------------------------------------
>
> Key: HBASE-4188
> URL: https://issues.apache.org/jira/browse/HBASE-4188
> Project: HBase
> Issue Type: Improvement
> Components: performance
> Reporter: stack
> Assignee: nkeywal
> Priority: Critical
> Fix For: 0.92.0
>
> Attachments: 4188-mssp.txt
>
>
> This issue is a continuation of hbase-1938 work (That issue is closed). This issue is about getting the last patch posted by nkeywal over in hbase-1938 applied (assigned nkeywal since he's done the work).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira