You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Kannan Muthukkaruppan (JIRA)" <ji...@apache.org> on 2011/09/19 09:44:09 UTC

[jira] [Created] (HBASE-4434) Don't do HFile Scanner next() unless the next KV is needed:

Don't do HFile Scanner next() unless the next KV is needed:
-----------------------------------------------------------

                 Key: HBASE-4434
                 URL: https://issues.apache.org/jira/browse/HBASE-4434
             Project: HBase
          Issue Type: Bug
            Reporter: Kannan Muthukkaruppan
            Assignee: Kannan Muthukkaruppan


When a seek/reseek is done on StoreFileScanner, in addition to setting the current KV, it also does a HFileScanner level next() ahead of time even if the next KV is never actually required. This inefficiency can potentially result in additional disk seeks and sub-optimal use of the block cache (primarily for cases where the KVs are large and each occupies an HFile block of its own).




--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HBASE-4434) seek optimization: don't do eager HFile Scanner next() unless the next KV is needed

Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-4434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113701#comment-13113701 ] 

jiraposter@reviews.apache.org commented on HBASE-4434:
------------------------------------------------------


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2034/#review2056
-----------------------------------------------------------

Ship it!


- Michael


On 2011-09-23 05:13:36, Kannan Muthukkaruppan wrote:
bq.  
bq.  -----------------------------------------------------------
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2034/
bq.  -----------------------------------------------------------
bq.  
bq.  (Updated 2011-09-23 05:13:36)
bq.  
bq.  
bq.  Review request for Michael Stack, Jonathan Gray and Mikhail Bautin.
bq.  
bq.  
bq.  Summary
bq.  -------
bq.  
bq.  The increase in blocks read in one sub-test case is due to HBASE-4466, which should be fixed soon. It is primarily an accounting issue, not an extra cache miss.
bq.  
bq.  
bq.  This addresses bug HBASE-4434.
bq.      https://issues.apache.org/jira/browse/HBASE-4434
bq.  
bq.  
bq.  Diffs
bq.  -----
bq.  
bq.    http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java 1174514 
bq.    http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksRead.java 1174514 
bq.  
bq.  Diff: https://reviews.apache.org/r/2034/diff
bq.  
bq.  
bq.  Testing
bq.  -------
bq.  
bq.  Ran TestBlocksRead. Will run the entire suite as well. On our 89 based internal branch, the tests passed without any issues.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Kannan
bq.  
bq.



> seek optimization: don't do eager HFile Scanner next() unless the next KV is needed
> -----------------------------------------------------------------------------------
>
>                 Key: HBASE-4434
>                 URL: https://issues.apache.org/jira/browse/HBASE-4434
>             Project: HBase
>          Issue Type: Improvement
>            Reporter: Kannan Muthukkaruppan
>            Assignee: Kannan Muthukkaruppan
>
> When a seek/reseek is done on StoreFileScanner, in addition to setting the current KV, it also does a HFileScanner level next() ahead of time even if the next KV is never actually required. This inefficiency can potentially result in additional disk seeks and sub-optimal use of the block cache (primarily for cases where the KVs are large and each occupies an HFile block of its own).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HBASE-4434) Don't do HFile Scanner next() unless the next KV is needed:

Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-4434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113218#comment-13113218 ] 

jiraposter@reviews.apache.org commented on HBASE-4434:
------------------------------------------------------


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2034/#review2036
-----------------------------------------------------------

Ship it!


+1 if all unit tests passing.  easy win!  nice work

- Jonathan


On 2011-09-23 05:13:36, Kannan Muthukkaruppan wrote:
bq.  
bq.  -----------------------------------------------------------
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2034/
bq.  -----------------------------------------------------------
bq.  
bq.  (Updated 2011-09-23 05:13:36)
bq.  
bq.  
bq.  Review request for Michael Stack, Jonathan Gray and Mikhail Bautin.
bq.  
bq.  
bq.  Summary
bq.  -------
bq.  
bq.  The increase in blocks read in one sub-test case is due to HBASE-4466, which should be fixed soon. It is primarily an accounting issue, not an extra cache miss.
bq.  
bq.  
bq.  This addresses bug HBASE-4434.
bq.      https://issues.apache.org/jira/browse/HBASE-4434
bq.  
bq.  
bq.  Diffs
bq.  -----
bq.  
bq.    http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java 1174514 
bq.    http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksRead.java 1174514 
bq.  
bq.  Diff: https://reviews.apache.org/r/2034/diff
bq.  
bq.  
bq.  Testing
bq.  -------
bq.  
bq.  Ran TestBlocksRead. Will run the entire suite as well. On our 89 based internal branch, the tests passed without any issues.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Kannan
bq.  
bq.



> Don't do HFile Scanner next() unless the next KV is needed:
> -----------------------------------------------------------
>
>                 Key: HBASE-4434
>                 URL: https://issues.apache.org/jira/browse/HBASE-4434
>             Project: HBase
>          Issue Type: Improvement
>            Reporter: Kannan Muthukkaruppan
>            Assignee: Kannan Muthukkaruppan
>
> When a seek/reseek is done on StoreFileScanner, in addition to setting the current KV, it also does a HFileScanner level next() ahead of time even if the next KV is never actually required. This inefficiency can potentially result in additional disk seeks and sub-optimal use of the block cache (primarily for cases where the KVs are large and each occupies an HFile block of its own).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HBASE-4434) seek optimization: don't do eager HFile Scanner next() unless the next KV is needed

Posted by "Hudson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-4434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113903#comment-13113903 ] 

Hudson commented on HBASE-4434:
-------------------------------

Integrated in HBase-TRUNK #2247 (See [https://builds.apache.org/job/HBase-TRUNK/2247/])
    HBASE-4434 seek optimization: don't do eager HFile Scanner next() unless the next KV is needed

stack : 
Files : 
* /hbase/trunk/CHANGES.txt
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksRead.java


> seek optimization: don't do eager HFile Scanner next() unless the next KV is needed
> -----------------------------------------------------------------------------------
>
>                 Key: HBASE-4434
>                 URL: https://issues.apache.org/jira/browse/HBASE-4434
>             Project: HBase
>          Issue Type: Improvement
>            Reporter: Kannan Muthukkaruppan
>            Assignee: Kannan Muthukkaruppan
>             Fix For: 0.92.0
>
>         Attachments: HBASE-4434.txt
>
>
> When a seek/reseek is done on StoreFileScanner, in addition to setting the current KV, it also does a HFileScanner level next() ahead of time even if the next KV is never actually required. This inefficiency can potentially result in additional disk seeks and sub-optimal use of the block cache (primarily for cases where the KVs are large and each occupies an HFile block of its own).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (HBASE-4434) seek optimization: don't do eager HFile Scanner next() unless the next KV is needed

Posted by "Kannan Muthukkaruppan (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-4434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kannan Muthukkaruppan updated HBASE-4434:
-----------------------------------------

    Summary: seek optimization: don't do eager HFile Scanner next() unless the next KV is needed  (was: Don't do HFile Scanner next() unless the next KV is needed:)

> seek optimization: don't do eager HFile Scanner next() unless the next KV is needed
> -----------------------------------------------------------------------------------
>
>                 Key: HBASE-4434
>                 URL: https://issues.apache.org/jira/browse/HBASE-4434
>             Project: HBase
>          Issue Type: Improvement
>            Reporter: Kannan Muthukkaruppan
>            Assignee: Kannan Muthukkaruppan
>
> When a seek/reseek is done on StoreFileScanner, in addition to setting the current KV, it also does a HFileScanner level next() ahead of time even if the next KV is never actually required. This inefficiency can potentially result in additional disk seeks and sub-optimal use of the block cache (primarily for cases where the KVs are large and each occupies an HFile block of its own).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Resolved] (HBASE-4434) seek optimization: don't do eager HFile Scanner next() unless the next KV is needed

Posted by "stack (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-4434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

stack resolved HBASE-4434.
--------------------------

       Resolution: Fixed
    Fix Version/s: 0.92.0
     Hadoop Flags: [Reviewed]

Committed to 0.92 branch and to TRUNK.  Thanks for the sweet patch Kannan.

> seek optimization: don't do eager HFile Scanner next() unless the next KV is needed
> -----------------------------------------------------------------------------------
>
>                 Key: HBASE-4434
>                 URL: https://issues.apache.org/jira/browse/HBASE-4434
>             Project: HBase
>          Issue Type: Improvement
>            Reporter: Kannan Muthukkaruppan
>            Assignee: Kannan Muthukkaruppan
>             Fix For: 0.92.0
>
>         Attachments: HBASE-4434.txt
>
>
> When a seek/reseek is done on StoreFileScanner, in addition to setting the current KV, it also does a HFileScanner level next() ahead of time even if the next KV is never actually required. This inefficiency can potentially result in additional disk seeks and sub-optimal use of the block cache (primarily for cases where the KVs are large and each occupies an HFile block of its own).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (HBASE-4434) seek optimization: don't do eager HFile Scanner next() unless the next KV is needed

Posted by "stack (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-4434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

stack updated HBASE-4434:
-------------------------

    Attachment: HBASE-4434.txt

Copied from rb.

> seek optimization: don't do eager HFile Scanner next() unless the next KV is needed
> -----------------------------------------------------------------------------------
>
>                 Key: HBASE-4434
>                 URL: https://issues.apache.org/jira/browse/HBASE-4434
>             Project: HBase
>          Issue Type: Improvement
>            Reporter: Kannan Muthukkaruppan
>            Assignee: Kannan Muthukkaruppan
>         Attachments: HBASE-4434.txt
>
>
> When a seek/reseek is done on StoreFileScanner, in addition to setting the current KV, it also does a HFileScanner level next() ahead of time even if the next KV is never actually required. This inefficiency can potentially result in additional disk seeks and sub-optimal use of the block cache (primarily for cases where the KVs are large and each occupies an HFile block of its own).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (HBASE-4434) Don't do HFile Scanner next() unless the next KV is needed:

Posted by "Kannan Muthukkaruppan (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-4434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kannan Muthukkaruppan updated HBASE-4434:
-----------------------------------------

    Issue Type: Improvement  (was: Bug)

Bug -> Improvement

> Don't do HFile Scanner next() unless the next KV is needed:
> -----------------------------------------------------------
>
>                 Key: HBASE-4434
>                 URL: https://issues.apache.org/jira/browse/HBASE-4434
>             Project: HBase
>          Issue Type: Improvement
>            Reporter: Kannan Muthukkaruppan
>            Assignee: Kannan Muthukkaruppan
>
> When a seek/reseek is done on StoreFileScanner, in addition to setting the current KV, it also does a HFileScanner level next() ahead of time even if the next KV is never actually required. This inefficiency can potentially result in additional disk seeks and sub-optimal use of the block cache (primarily for cases where the KVs are large and each occupies an HFile block of its own).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HBASE-4434) Don't do HFile Scanner next() unless the next KV is needed:

Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-4434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113160#comment-13113160 ] 

jiraposter@reviews.apache.org commented on HBASE-4434:
------------------------------------------------------


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2034/
-----------------------------------------------------------

Review request for Michael Stack, Jonathan Gray and Mikhail Bautin.


Summary
-------

The increase in blocks read in one sub-test case is due to HBASE-4466, which should be fixed soon. It is primarily an accounting issue, not an extra cache miss.


This addresses bug HBASE-4434.
    https://issues.apache.org/jira/browse/HBASE-4434


Diffs
-----

  http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java 1174514 
  http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksRead.java 1174514 

Diff: https://reviews.apache.org/r/2034/diff


Testing
-------

Ran TestBlocksRead. Will run the entire suite as well. On our 89 based internal branch, the tests passed without any issues.


Thanks,

Kannan



> Don't do HFile Scanner next() unless the next KV is needed:
> -----------------------------------------------------------
>
>                 Key: HBASE-4434
>                 URL: https://issues.apache.org/jira/browse/HBASE-4434
>             Project: HBase
>          Issue Type: Improvement
>            Reporter: Kannan Muthukkaruppan
>            Assignee: Kannan Muthukkaruppan
>
> When a seek/reseek is done on StoreFileScanner, in addition to setting the current KV, it also does a HFileScanner level next() ahead of time even if the next KV is never actually required. This inefficiency can potentially result in additional disk seeks and sub-optimal use of the block cache (primarily for cases where the KVs are large and each occupies an HFile block of its own).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HBASE-4434) seek optimization: don't do eager HFile Scanner next() unless the next KV is needed

Posted by "Hudson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-4434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113877#comment-13113877 ] 

Hudson commented on HBASE-4434:
-------------------------------

Integrated in HBase-0.92 #18 (See [https://builds.apache.org/job/HBase-0.92/18/])
    HBASE-4434: Don't do HFile Scanner next() unless the next KV is needed

stack : 
Files : 
* /hbase/branches/0.92/CHANGES.txt
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksRead.java


> seek optimization: don't do eager HFile Scanner next() unless the next KV is needed
> -----------------------------------------------------------------------------------
>
>                 Key: HBASE-4434
>                 URL: https://issues.apache.org/jira/browse/HBASE-4434
>             Project: HBase
>          Issue Type: Improvement
>            Reporter: Kannan Muthukkaruppan
>            Assignee: Kannan Muthukkaruppan
>             Fix For: 0.92.0
>
>         Attachments: HBASE-4434.txt
>
>
> When a seek/reseek is done on StoreFileScanner, in addition to setting the current KV, it also does a HFileScanner level next() ahead of time even if the next KV is never actually required. This inefficiency can potentially result in additional disk seeks and sub-optimal use of the block cache (primarily for cases where the KVs are large and each occupies an HFile block of its own).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HBASE-4434) Don't do HFile Scanner next() unless the next KV is needed:

Posted by "Kannan Muthukkaruppan (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-4434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113569#comment-13113569 ] 

Kannan Muthukkaruppan commented on HBASE-4434:
----------------------------------------------

Jonathan: Thanks for the review!

Unit tests ran fine, but for some completely unrelated/env related failures. 
 org.apache.hadoop.hbase.avro.TestAvroServer: Unresolved compilation problem:
  testThreeRSAbort(org.apache.hadoop.hbase.master.TestDistributedLogSplitting): test timed out after 300000 milliseconds
  testOrphanLogCreation(org.apache.hadoop.hbase.master.TestDistributedLogSplitting): test timed out after 300000 milliseconds
  testStatusTemplateNoTables(org.apache.hadoop.hbase.master.TestMasterStatusServlet): Unresolved compilation problems:

Can someone commit? (I will submit patch for HBASE-4433 after the commit since that patch will update the TestBlocksRead again).

> Don't do HFile Scanner next() unless the next KV is needed:
> -----------------------------------------------------------
>
>                 Key: HBASE-4434
>                 URL: https://issues.apache.org/jira/browse/HBASE-4434
>             Project: HBase
>          Issue Type: Improvement
>            Reporter: Kannan Muthukkaruppan
>            Assignee: Kannan Muthukkaruppan
>
> When a seek/reseek is done on StoreFileScanner, in addition to setting the current KV, it also does a HFileScanner level next() ahead of time even if the next KV is never actually required. This inefficiency can potentially result in additional disk seeks and sub-optimal use of the block cache (primarily for cases where the KVs are large and each occupies an HFile block of its own).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira