You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Kannan Muthukkaruppan (JIRA)" <ji...@apache.org> on 2011/09/19 09:44:09 UTC
[jira] [Created] (HBASE-4434) Don't do HFile Scanner next() unless
the next KV is needed:
Don't do HFile Scanner next() unless the next KV is needed:
-----------------------------------------------------------
Key: HBASE-4434
URL: https://issues.apache.org/jira/browse/HBASE-4434
Project: HBase
Issue Type: Bug
Reporter: Kannan Muthukkaruppan
Assignee: Kannan Muthukkaruppan
When a seek/reseek is done on StoreFileScanner, in addition to setting the current KV, it also does a HFileScanner level next() ahead of time even if the next KV is never actually required. This inefficiency can potentially result in additional disk seeks and sub-optimal use of the block cache (primarily for cases where the KVs are large and each occupies an HFile block of its own).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4434) seek optimization: don't do eager
HFile Scanner next() unless the next KV is needed
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113701#comment-13113701 ]
jiraposter@reviews.apache.org commented on HBASE-4434:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2034/#review2056
-----------------------------------------------------------
Ship it!
- Michael
On 2011-09-23 05:13:36, Kannan Muthukkaruppan wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/2034/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-09-23 05:13:36)
bq.
bq.
bq. Review request for Michael Stack, Jonathan Gray and Mikhail Bautin.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. The increase in blocks read in one sub-test case is due to HBASE-4466, which should be fixed soon. It is primarily an accounting issue, not an extra cache miss.
bq.
bq.
bq. This addresses bug HBASE-4434.
bq. https://issues.apache.org/jira/browse/HBASE-4434
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java 1174514
bq. http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksRead.java 1174514
bq.
bq. Diff: https://reviews.apache.org/r/2034/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Ran TestBlocksRead. Will run the entire suite as well. On our 89 based internal branch, the tests passed without any issues.
bq.
bq.
bq. Thanks,
bq.
bq. Kannan
bq.
bq.
> seek optimization: don't do eager HFile Scanner next() unless the next KV is needed
> -----------------------------------------------------------------------------------
>
> Key: HBASE-4434
> URL: https://issues.apache.org/jira/browse/HBASE-4434
> Project: HBase
> Issue Type: Improvement
> Reporter: Kannan Muthukkaruppan
> Assignee: Kannan Muthukkaruppan
>
> When a seek/reseek is done on StoreFileScanner, in addition to setting the current KV, it also does a HFileScanner level next() ahead of time even if the next KV is never actually required. This inefficiency can potentially result in additional disk seeks and sub-optimal use of the block cache (primarily for cases where the KVs are large and each occupies an HFile block of its own).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4434) Don't do HFile Scanner next()
unless the next KV is needed:
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113218#comment-13113218 ]
jiraposter@reviews.apache.org commented on HBASE-4434:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2034/#review2036
-----------------------------------------------------------
Ship it!
+1 if all unit tests passing. easy win! nice work
- Jonathan
On 2011-09-23 05:13:36, Kannan Muthukkaruppan wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/2034/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-09-23 05:13:36)
bq.
bq.
bq. Review request for Michael Stack, Jonathan Gray and Mikhail Bautin.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. The increase in blocks read in one sub-test case is due to HBASE-4466, which should be fixed soon. It is primarily an accounting issue, not an extra cache miss.
bq.
bq.
bq. This addresses bug HBASE-4434.
bq. https://issues.apache.org/jira/browse/HBASE-4434
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java 1174514
bq. http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksRead.java 1174514
bq.
bq. Diff: https://reviews.apache.org/r/2034/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Ran TestBlocksRead. Will run the entire suite as well. On our 89 based internal branch, the tests passed without any issues.
bq.
bq.
bq. Thanks,
bq.
bq. Kannan
bq.
bq.
> Don't do HFile Scanner next() unless the next KV is needed:
> -----------------------------------------------------------
>
> Key: HBASE-4434
> URL: https://issues.apache.org/jira/browse/HBASE-4434
> Project: HBase
> Issue Type: Improvement
> Reporter: Kannan Muthukkaruppan
> Assignee: Kannan Muthukkaruppan
>
> When a seek/reseek is done on StoreFileScanner, in addition to setting the current KV, it also does a HFileScanner level next() ahead of time even if the next KV is never actually required. This inefficiency can potentially result in additional disk seeks and sub-optimal use of the block cache (primarily for cases where the KVs are large and each occupies an HFile block of its own).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4434) seek optimization: don't do eager
HFile Scanner next() unless the next KV is needed
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113903#comment-13113903 ]
Hudson commented on HBASE-4434:
-------------------------------
Integrated in HBase-TRUNK #2247 (See [https://builds.apache.org/job/HBase-TRUNK/2247/])
HBASE-4434 seek optimization: don't do eager HFile Scanner next() unless the next KV is needed
stack :
Files :
* /hbase/trunk/CHANGES.txt
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksRead.java
> seek optimization: don't do eager HFile Scanner next() unless the next KV is needed
> -----------------------------------------------------------------------------------
>
> Key: HBASE-4434
> URL: https://issues.apache.org/jira/browse/HBASE-4434
> Project: HBase
> Issue Type: Improvement
> Reporter: Kannan Muthukkaruppan
> Assignee: Kannan Muthukkaruppan
> Fix For: 0.92.0
>
> Attachments: HBASE-4434.txt
>
>
> When a seek/reseek is done on StoreFileScanner, in addition to setting the current KV, it also does a HFileScanner level next() ahead of time even if the next KV is never actually required. This inefficiency can potentially result in additional disk seeks and sub-optimal use of the block cache (primarily for cases where the KVs are large and each occupies an HFile block of its own).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4434) seek optimization: don't do eager
HFile Scanner next() unless the next KV is needed
Posted by "Kannan Muthukkaruppan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kannan Muthukkaruppan updated HBASE-4434:
-----------------------------------------
Summary: seek optimization: don't do eager HFile Scanner next() unless the next KV is needed (was: Don't do HFile Scanner next() unless the next KV is needed:)
> seek optimization: don't do eager HFile Scanner next() unless the next KV is needed
> -----------------------------------------------------------------------------------
>
> Key: HBASE-4434
> URL: https://issues.apache.org/jira/browse/HBASE-4434
> Project: HBase
> Issue Type: Improvement
> Reporter: Kannan Muthukkaruppan
> Assignee: Kannan Muthukkaruppan
>
> When a seek/reseek is done on StoreFileScanner, in addition to setting the current KV, it also does a HFileScanner level next() ahead of time even if the next KV is never actually required. This inefficiency can potentially result in additional disk seeks and sub-optimal use of the block cache (primarily for cases where the KVs are large and each occupies an HFile block of its own).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-4434) seek optimization: don't do eager
HFile Scanner next() unless the next KV is needed
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack resolved HBASE-4434.
--------------------------
Resolution: Fixed
Fix Version/s: 0.92.0
Hadoop Flags: [Reviewed]
Committed to 0.92 branch and to TRUNK. Thanks for the sweet patch Kannan.
> seek optimization: don't do eager HFile Scanner next() unless the next KV is needed
> -----------------------------------------------------------------------------------
>
> Key: HBASE-4434
> URL: https://issues.apache.org/jira/browse/HBASE-4434
> Project: HBase
> Issue Type: Improvement
> Reporter: Kannan Muthukkaruppan
> Assignee: Kannan Muthukkaruppan
> Fix For: 0.92.0
>
> Attachments: HBASE-4434.txt
>
>
> When a seek/reseek is done on StoreFileScanner, in addition to setting the current KV, it also does a HFileScanner level next() ahead of time even if the next KV is never actually required. This inefficiency can potentially result in additional disk seeks and sub-optimal use of the block cache (primarily for cases where the KVs are large and each occupies an HFile block of its own).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4434) seek optimization: don't do eager
HFile Scanner next() unless the next KV is needed
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-4434:
-------------------------
Attachment: HBASE-4434.txt
Copied from rb.
> seek optimization: don't do eager HFile Scanner next() unless the next KV is needed
> -----------------------------------------------------------------------------------
>
> Key: HBASE-4434
> URL: https://issues.apache.org/jira/browse/HBASE-4434
> Project: HBase
> Issue Type: Improvement
> Reporter: Kannan Muthukkaruppan
> Assignee: Kannan Muthukkaruppan
> Attachments: HBASE-4434.txt
>
>
> When a seek/reseek is done on StoreFileScanner, in addition to setting the current KV, it also does a HFileScanner level next() ahead of time even if the next KV is never actually required. This inefficiency can potentially result in additional disk seeks and sub-optimal use of the block cache (primarily for cases where the KVs are large and each occupies an HFile block of its own).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4434) Don't do HFile Scanner next() unless
the next KV is needed:
Posted by "Kannan Muthukkaruppan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kannan Muthukkaruppan updated HBASE-4434:
-----------------------------------------
Issue Type: Improvement (was: Bug)
Bug -> Improvement
> Don't do HFile Scanner next() unless the next KV is needed:
> -----------------------------------------------------------
>
> Key: HBASE-4434
> URL: https://issues.apache.org/jira/browse/HBASE-4434
> Project: HBase
> Issue Type: Improvement
> Reporter: Kannan Muthukkaruppan
> Assignee: Kannan Muthukkaruppan
>
> When a seek/reseek is done on StoreFileScanner, in addition to setting the current KV, it also does a HFileScanner level next() ahead of time even if the next KV is never actually required. This inefficiency can potentially result in additional disk seeks and sub-optimal use of the block cache (primarily for cases where the KVs are large and each occupies an HFile block of its own).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4434) Don't do HFile Scanner next()
unless the next KV is needed:
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113160#comment-13113160 ]
jiraposter@reviews.apache.org commented on HBASE-4434:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2034/
-----------------------------------------------------------
Review request for Michael Stack, Jonathan Gray and Mikhail Bautin.
Summary
-------
The increase in blocks read in one sub-test case is due to HBASE-4466, which should be fixed soon. It is primarily an accounting issue, not an extra cache miss.
This addresses bug HBASE-4434.
https://issues.apache.org/jira/browse/HBASE-4434
Diffs
-----
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java 1174514
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksRead.java 1174514
Diff: https://reviews.apache.org/r/2034/diff
Testing
-------
Ran TestBlocksRead. Will run the entire suite as well. On our 89 based internal branch, the tests passed without any issues.
Thanks,
Kannan
> Don't do HFile Scanner next() unless the next KV is needed:
> -----------------------------------------------------------
>
> Key: HBASE-4434
> URL: https://issues.apache.org/jira/browse/HBASE-4434
> Project: HBase
> Issue Type: Improvement
> Reporter: Kannan Muthukkaruppan
> Assignee: Kannan Muthukkaruppan
>
> When a seek/reseek is done on StoreFileScanner, in addition to setting the current KV, it also does a HFileScanner level next() ahead of time even if the next KV is never actually required. This inefficiency can potentially result in additional disk seeks and sub-optimal use of the block cache (primarily for cases where the KVs are large and each occupies an HFile block of its own).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4434) seek optimization: don't do eager
HFile Scanner next() unless the next KV is needed
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113877#comment-13113877 ]
Hudson commented on HBASE-4434:
-------------------------------
Integrated in HBase-0.92 #18 (See [https://builds.apache.org/job/HBase-0.92/18/])
HBASE-4434: Don't do HFile Scanner next() unless the next KV is needed
stack :
Files :
* /hbase/branches/0.92/CHANGES.txt
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksRead.java
> seek optimization: don't do eager HFile Scanner next() unless the next KV is needed
> -----------------------------------------------------------------------------------
>
> Key: HBASE-4434
> URL: https://issues.apache.org/jira/browse/HBASE-4434
> Project: HBase
> Issue Type: Improvement
> Reporter: Kannan Muthukkaruppan
> Assignee: Kannan Muthukkaruppan
> Fix For: 0.92.0
>
> Attachments: HBASE-4434.txt
>
>
> When a seek/reseek is done on StoreFileScanner, in addition to setting the current KV, it also does a HFileScanner level next() ahead of time even if the next KV is never actually required. This inefficiency can potentially result in additional disk seeks and sub-optimal use of the block cache (primarily for cases where the KVs are large and each occupies an HFile block of its own).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4434) Don't do HFile Scanner next()
unless the next KV is needed:
Posted by "Kannan Muthukkaruppan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113569#comment-13113569 ]
Kannan Muthukkaruppan commented on HBASE-4434:
----------------------------------------------
Jonathan: Thanks for the review!
Unit tests ran fine, but for some completely unrelated/env related failures.
org.apache.hadoop.hbase.avro.TestAvroServer: Unresolved compilation problem:
testThreeRSAbort(org.apache.hadoop.hbase.master.TestDistributedLogSplitting): test timed out after 300000 milliseconds
testOrphanLogCreation(org.apache.hadoop.hbase.master.TestDistributedLogSplitting): test timed out after 300000 milliseconds
testStatusTemplateNoTables(org.apache.hadoop.hbase.master.TestMasterStatusServlet): Unresolved compilation problems:
Can someone commit? (I will submit patch for HBASE-4433 after the commit since that patch will update the TestBlocksRead again).
> Don't do HFile Scanner next() unless the next KV is needed:
> -----------------------------------------------------------
>
> Key: HBASE-4434
> URL: https://issues.apache.org/jira/browse/HBASE-4434
> Project: HBase
> Issue Type: Improvement
> Reporter: Kannan Muthukkaruppan
> Assignee: Kannan Muthukkaruppan
>
> When a seek/reseek is done on StoreFileScanner, in addition to setting the current KV, it also does a HFileScanner level next() ahead of time even if the next KV is never actually required. This inefficiency can potentially result in additional disk seeks and sub-optimal use of the block cache (primarily for cases where the KVs are large and each occupies an HFile block of its own).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira