You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Zhihong Yu (JIRA)" <ji...@apache.org> on 2012/05/17 20:04:06 UTC
[jira] [Created] (HBASE-6032) Port HFileBlockIndex improvement from
HBASE-5987
Zhihong Yu created HBASE-6032:
---------------------------------
Summary: Port HFileBlockIndex improvement from HBASE-5987
Key: HBASE-6032
URL: https://issues.apache.org/jira/browse/HBASE-6032
Project: HBase
Issue Type: Task
Reporter: Zhihong Yu
Excerpt from HBASE-5987:
First, we propose to lookahead for one more block index so that the HFileScanner would know the start key value of next data block. So if the target key value for the scan(reSeekTo) is "smaller" than that start kv of next data block, it means the target key value has a very high possibility in the current data block (if not in current data block, then the start kv of next data block should be returned. +Indexing on the start key has some defects here+) and it shall NOT query the HFileBlockIndex in this case. On the contrary, if the target key value is "bigger", then it shall query the HFileBlockIndex. This improvement shall help to reduce the hotness of HFileBlockIndex and avoid some unnecessary IdLock Contention or Index Block Cache lookup.
This JIRA is to port the fix to HBase trunk, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6032) Port HFileBlockIndex improvement
from HBASE-5987
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13284576#comment-13284576 ]
Hudson commented on HBASE-6032:
-------------------------------
Integrated in HBase-TRUNK #2941 (See [https://builds.apache.org/job/HBase-TRUNK/2941/])
HBASE-6032 Port HFileBlockIndex improvement from HBASE-5987 (Revision 1343413)
Result = FAILURE
tedyu :
Files :
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/HConstants.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/BlockWithScanInfo.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestCase.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestReseekTo.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestSeekTo.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksScanned.java
> Port HFileBlockIndex improvement from HBASE-5987
> ------------------------------------------------
>
> Key: HBASE-6032
> URL: https://issues.apache.org/jira/browse/HBASE-6032
> Project: HBase
> Issue Type: Task
> Reporter: Zhihong Yu
> Assignee: Zhihong Yu
> Fix For: 0.96.0
>
> Attachments: 6032-ports-5987-v2.txt, 6032-ports-5987.txt
>
>
> Excerpt from HBASE-5987:
> First, we propose to lookahead for one more block index so that the HFileScanner would know the start key value of next data block. So if the target key value for the scan(reSeekTo) is "smaller" than that start kv of next data block, it means the target key value has a very high possibility in the current data block (if not in current data block, then the start kv of next data block should be returned. +Indexing on the start key has some defects here+) and it shall NOT query the HFileBlockIndex in this case. On the contrary, if the target key value is "bigger", then it shall query the HFileBlockIndex. This improvement shall help to reduce the hotness of HFileBlockIndex and avoid some unnecessary IdLock Contention or Index Block Cache lookup.
> This JIRA is to port the fix to HBase trunk, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6032) Port HFileBlockIndex improvement
from HBASE-5987
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478678#comment-13478678 ]
Hudson commented on HBASE-6032:
-------------------------------
Integrated in HBase-0.94 #539 (See [https://builds.apache.org/job/HBase-0.94/539/])
HBASE-6032 Port HFileBlockIndex improvement from HBASE-5987 (Liyin, Ted, Stack) (Revision 1399513)
Result = FAILURE
larsh :
Files :
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/HConstants.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/HBaseTestCase.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/io/hfile/TestReseekTo.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/io/hfile/TestSeekTo.java
> Port HFileBlockIndex improvement from HBASE-5987
> ------------------------------------------------
>
> Key: HBASE-6032
> URL: https://issues.apache.org/jira/browse/HBASE-6032
> Project: HBase
> Issue Type: Task
> Reporter: Ted Yu
> Assignee: Ted Yu
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 6032.094.txt, 6032-ports-5987.txt, 6032-ports-5987-v2.txt, 6032v3.txt
>
>
> Excerpt from HBASE-5987:
> First, we propose to lookahead for one more block index so that the HFileScanner would know the start key value of next data block. So if the target key value for the scan(reSeekTo) is "smaller" than that start kv of next data block, it means the target key value has a very high possibility in the current data block (if not in current data block, then the start kv of next data block should be returned. +Indexing on the start key has some defects here+) and it shall NOT query the HFileBlockIndex in this case. On the contrary, if the target key value is "bigger", then it shall query the HFileBlockIndex. This improvement shall help to reduce the hotness of HFileBlockIndex and avoid some unnecessary IdLock Contention or Index Block Cache lookup.
> This JIRA is to port the fix to HBase trunk, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6032) Port HFileBlockIndex improvement
from HBASE-5987
Posted by "Anoop Sam John (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13477624#comment-13477624 ]
Anoop Sam John commented on HBASE-6032:
---------------------------------------
+1 for having this for 0.94 version..
In fact I was trying to make a port and test and as per that raise a new issue for porting..
> Port HFileBlockIndex improvement from HBASE-5987
> ------------------------------------------------
>
> Key: HBASE-6032
> URL: https://issues.apache.org/jira/browse/HBASE-6032
> Project: HBase
> Issue Type: Task
> Reporter: Ted Yu
> Assignee: Ted Yu
> Fix For: 0.96.0
>
> Attachments: 6032-ports-5987.txt, 6032-ports-5987-v2.txt
>
>
> Excerpt from HBASE-5987:
> First, we propose to lookahead for one more block index so that the HFileScanner would know the start key value of next data block. So if the target key value for the scan(reSeekTo) is "smaller" than that start kv of next data block, it means the target key value has a very high possibility in the current data block (if not in current data block, then the start kv of next data block should be returned. +Indexing on the start key has some defects here+) and it shall NOT query the HFileBlockIndex in this case. On the contrary, if the target key value is "bigger", then it shall query the HFileBlockIndex. This improvement shall help to reduce the hotness of HFileBlockIndex and avoid some unnecessary IdLock Contention or Index Block Cache lookup.
> This JIRA is to port the fix to HBase trunk, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6032) Port HFileBlockIndex improvement from
HBASE-5987
Posted by "Zhihong Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zhihong Yu updated HBASE-6032:
------------------------------
Status: Open (was: Patch Available)
> Port HFileBlockIndex improvement from HBASE-5987
> ------------------------------------------------
>
> Key: HBASE-6032
> URL: https://issues.apache.org/jira/browse/HBASE-6032
> Project: HBase
> Issue Type: Task
> Reporter: Zhihong Yu
> Assignee: Zhihong Yu
> Fix For: 0.96.0
>
> Attachments: 6032-ports-5987-v2.txt, 6032-ports-5987.txt
>
>
> Excerpt from HBASE-5987:
> First, we propose to lookahead for one more block index so that the HFileScanner would know the start key value of next data block. So if the target key value for the scan(reSeekTo) is "smaller" than that start kv of next data block, it means the target key value has a very high possibility in the current data block (if not in current data block, then the start kv of next data block should be returned. +Indexing on the start key has some defects here+) and it shall NOT query the HFileBlockIndex in this case. On the contrary, if the target key value is "bigger", then it shall query the HFileBlockIndex. This improvement shall help to reduce the hotness of HFileBlockIndex and avoid some unnecessary IdLock Contention or Index Block Cache lookup.
> This JIRA is to port the fix to HBase trunk, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6032) Port HFileBlockIndex improvement from
HBASE-5987
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-6032:
-------------------------
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
This was committed a good while back.
> Port HFileBlockIndex improvement from HBASE-5987
> ------------------------------------------------
>
> Key: HBASE-6032
> URL: https://issues.apache.org/jira/browse/HBASE-6032
> Project: HBase
> Issue Type: Task
> Reporter: Ted Yu
> Assignee: Ted Yu
> Fix For: 0.96.0
>
> Attachments: 6032-ports-5987.txt, 6032-ports-5987-v2.txt
>
>
> Excerpt from HBASE-5987:
> First, we propose to lookahead for one more block index so that the HFileScanner would know the start key value of next data block. So if the target key value for the scan(reSeekTo) is "smaller" than that start kv of next data block, it means the target key value has a very high possibility in the current data block (if not in current data block, then the start kv of next data block should be returned. +Indexing on the start key has some defects here+) and it shall NOT query the HFileBlockIndex in this case. On the contrary, if the target key value is "bigger", then it shall query the HFileBlockIndex. This improvement shall help to reduce the hotness of HFileBlockIndex and avoid some unnecessary IdLock Contention or Index Block Cache lookup.
> This JIRA is to port the fix to HBase trunk, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6032) Port HFileBlockIndex improvement
from HBASE-5987
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13477610#comment-13477610 ]
Lars Hofhansl commented on HBASE-6032:
--------------------------------------
How come we missed this for 0.94?
This looks like an important performance improvement.
> Port HFileBlockIndex improvement from HBASE-5987
> ------------------------------------------------
>
> Key: HBASE-6032
> URL: https://issues.apache.org/jira/browse/HBASE-6032
> Project: HBase
> Issue Type: Task
> Reporter: Ted Yu
> Assignee: Ted Yu
> Fix For: 0.96.0
>
> Attachments: 6032-ports-5987.txt, 6032-ports-5987-v2.txt
>
>
> Excerpt from HBASE-5987:
> First, we propose to lookahead for one more block index so that the HFileScanner would know the start key value of next data block. So if the target key value for the scan(reSeekTo) is "smaller" than that start kv of next data block, it means the target key value has a very high possibility in the current data block (if not in current data block, then the start kv of next data block should be returned. +Indexing on the start key has some defects here+) and it shall NOT query the HFileBlockIndex in this case. On the contrary, if the target key value is "bigger", then it shall query the HFileBlockIndex. This improvement shall help to reduce the hotness of HFileBlockIndex and avoid some unnecessary IdLock Contention or Index Block Cache lookup.
> This JIRA is to port the fix to HBase trunk, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6032) Port HFileBlockIndex improvement
from HBASE-5987
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13286102#comment-13286102 ]
stack commented on HBASE-6032:
------------------------------
@Liyin Yeah. We should port them over. One day!
> Port HFileBlockIndex improvement from HBASE-5987
> ------------------------------------------------
>
> Key: HBASE-6032
> URL: https://issues.apache.org/jira/browse/HBASE-6032
> Project: HBase
> Issue Type: Task
> Reporter: Zhihong Yu
> Assignee: Zhihong Yu
> Fix For: 0.96.0
>
> Attachments: 6032-ports-5987-v2.txt, 6032-ports-5987.txt
>
>
> Excerpt from HBASE-5987:
> First, we propose to lookahead for one more block index so that the HFileScanner would know the start key value of next data block. So if the target key value for the scan(reSeekTo) is "smaller" than that start kv of next data block, it means the target key value has a very high possibility in the current data block (if not in current data block, then the start kv of next data block should be returned. +Indexing on the start key has some defects here+) and it shall NOT query the HFileBlockIndex in this case. On the contrary, if the target key value is "bigger", then it shall query the HFileBlockIndex. This improvement shall help to reduce the hotness of HFileBlockIndex and avoid some unnecessary IdLock Contention or Index Block Cache lookup.
> This JIRA is to port the fix to HBase trunk, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6032) Port HFileBlockIndex improvement
from HBASE-5987
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13479182#comment-13479182 ]
Lars Hofhansl commented on HBASE-6032:
--------------------------------------
Ran queueFailover manually. Passes locally.
> Port HFileBlockIndex improvement from HBASE-5987
> ------------------------------------------------
>
> Key: HBASE-6032
> URL: https://issues.apache.org/jira/browse/HBASE-6032
> Project: HBase
> Issue Type: Task
> Reporter: Ted Yu
> Assignee: Ted Yu
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 6032.094.txt, 6032-ports-5987.txt, 6032-ports-5987-v2.txt, 6032v3.txt
>
>
> Excerpt from HBASE-5987:
> First, we propose to lookahead for one more block index so that the HFileScanner would know the start key value of next data block. So if the target key value for the scan(reSeekTo) is "smaller" than that start kv of next data block, it means the target key value has a very high possibility in the current data block (if not in current data block, then the start kv of next data block should be returned. +Indexing on the start key has some defects here+) and it shall NOT query the HFileBlockIndex in this case. On the contrary, if the target key value is "bigger", then it shall query the HFileBlockIndex. This improvement shall help to reduce the hotness of HFileBlockIndex and avoid some unnecessary IdLock Contention or Index Block Cache lookup.
> This JIRA is to port the fix to HBase trunk, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6032) Port HFileBlockIndex improvement from
HBASE-5987
Posted by "Zhihong Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zhihong Yu updated HBASE-6032:
------------------------------
Attachment: 6032-ports-5987.txt
> Port HFileBlockIndex improvement from HBASE-5987
> ------------------------------------------------
>
> Key: HBASE-6032
> URL: https://issues.apache.org/jira/browse/HBASE-6032
> Project: HBase
> Issue Type: Task
> Reporter: Zhihong Yu
> Attachments: 6032-ports-5987.txt
>
>
> Excerpt from HBASE-5987:
> First, we propose to lookahead for one more block index so that the HFileScanner would know the start key value of next data block. So if the target key value for the scan(reSeekTo) is "smaller" than that start kv of next data block, it means the target key value has a very high possibility in the current data block (if not in current data block, then the start kv of next data block should be returned. +Indexing on the start key has some defects here+) and it shall NOT query the HFileBlockIndex in this case. On the contrary, if the target key value is "bigger", then it shall query the HFileBlockIndex. This improvement shall help to reduce the hotness of HFileBlockIndex and avoid some unnecessary IdLock Contention or Index Block Cache lookup.
> This JIRA is to port the fix to HBase trunk, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6032) Port HFileBlockIndex improvement from
HBASE-5987
Posted by "Zhihong Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zhihong Yu updated HBASE-6032:
------------------------------
Fix Version/s: 0.96.0
> Port HFileBlockIndex improvement from HBASE-5987
> ------------------------------------------------
>
> Key: HBASE-6032
> URL: https://issues.apache.org/jira/browse/HBASE-6032
> Project: HBase
> Issue Type: Task
> Reporter: Zhihong Yu
> Fix For: 0.96.0
>
> Attachments: 6032-ports-5987.txt
>
>
> Excerpt from HBASE-5987:
> First, we propose to lookahead for one more block index so that the HFileScanner would know the start key value of next data block. So if the target key value for the scan(reSeekTo) is "smaller" than that start kv of next data block, it means the target key value has a very high possibility in the current data block (if not in current data block, then the start kv of next data block should be returned. +Indexing on the start key has some defects here+) and it shall NOT query the HFileBlockIndex in this case. On the contrary, if the target key value is "bigger", then it shall query the HFileBlockIndex. This improvement shall help to reduce the hotness of HFileBlockIndex and avoid some unnecessary IdLock Contention or Index Block Cache lookup.
> This JIRA is to port the fix to HBase trunk, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6032) Port HFileBlockIndex improvement from
HBASE-5987
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-6032:
---------------------------------
Fix Version/s: 0.94.3
Committed to 0.94 as well.
> Port HFileBlockIndex improvement from HBASE-5987
> ------------------------------------------------
>
> Key: HBASE-6032
> URL: https://issues.apache.org/jira/browse/HBASE-6032
> Project: HBase
> Issue Type: Task
> Reporter: Ted Yu
> Assignee: Ted Yu
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 6032.094.txt, 6032-ports-5987.txt, 6032-ports-5987-v2.txt, 6032v3.txt
>
>
> Excerpt from HBASE-5987:
> First, we propose to lookahead for one more block index so that the HFileScanner would know the start key value of next data block. So if the target key value for the scan(reSeekTo) is "smaller" than that start kv of next data block, it means the target key value has a very high possibility in the current data block (if not in current data block, then the start kv of next data block should be returned. +Indexing on the start key has some defects here+) and it shall NOT query the HFileBlockIndex in this case. On the contrary, if the target key value is "bigger", then it shall query the HFileBlockIndex. This improvement shall help to reduce the hotness of HFileBlockIndex and avoid some unnecessary IdLock Contention or Index Block Cache lookup.
> This JIRA is to port the fix to HBase trunk, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6032) Port HFileBlockIndex improvement
from HBASE-5987
Posted by "Zhihong Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13284567#comment-13284567 ]
Zhihong Yu commented on HBASE-6032:
-----------------------------------
Integrated patch v2 to trunk.
> Port HFileBlockIndex improvement from HBASE-5987
> ------------------------------------------------
>
> Key: HBASE-6032
> URL: https://issues.apache.org/jira/browse/HBASE-6032
> Project: HBase
> Issue Type: Task
> Reporter: Zhihong Yu
> Assignee: Zhihong Yu
> Fix For: 0.96.0
>
> Attachments: 6032-ports-5987-v2.txt, 6032-ports-5987.txt
>
>
> Excerpt from HBASE-5987:
> First, we propose to lookahead for one more block index so that the HFileScanner would know the start key value of next data block. So if the target key value for the scan(reSeekTo) is "smaller" than that start kv of next data block, it means the target key value has a very high possibility in the current data block (if not in current data block, then the start kv of next data block should be returned. +Indexing on the start key has some defects here+) and it shall NOT query the HFileBlockIndex in this case. On the contrary, if the target key value is "bigger", then it shall query the HFileBlockIndex. This improvement shall help to reduce the hotness of HFileBlockIndex and avoid some unnecessary IdLock Contention or Index Block Cache lookup.
> This JIRA is to port the fix to HBase trunk, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6032) Port HFileBlockIndex improvement
from HBASE-5987
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13286092#comment-13286092 ]
stack commented on HBASE-6032:
------------------------------
Thanks for forward porting Ted.
Weird this patch uses TestHBaseCase and does all this fixup in it -- its a deprecated class -- and that it makes data members static.
Otherwise, patch is great on cursory review.
> Port HFileBlockIndex improvement from HBASE-5987
> ------------------------------------------------
>
> Key: HBASE-6032
> URL: https://issues.apache.org/jira/browse/HBASE-6032
> Project: HBase
> Issue Type: Task
> Reporter: Zhihong Yu
> Assignee: Zhihong Yu
> Fix For: 0.96.0
>
> Attachments: 6032-ports-5987-v2.txt, 6032-ports-5987.txt
>
>
> Excerpt from HBASE-5987:
> First, we propose to lookahead for one more block index so that the HFileScanner would know the start key value of next data block. So if the target key value for the scan(reSeekTo) is "smaller" than that start kv of next data block, it means the target key value has a very high possibility in the current data block (if not in current data block, then the start kv of next data block should be returned. +Indexing on the start key has some defects here+) and it shall NOT query the HFileBlockIndex in this case. On the contrary, if the target key value is "bigger", then it shall query the HFileBlockIndex. This improvement shall help to reduce the hotness of HFileBlockIndex and avoid some unnecessary IdLock Contention or Index Block Cache lookup.
> This JIRA is to port the fix to HBase trunk, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6032) Port HFileBlockIndex improvement
from HBASE-5987
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13284580#comment-13284580 ]
Hudson commented on HBASE-6032:
-------------------------------
Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #30 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/30/])
HBASE-6032 Port HFileBlockIndex improvement from HBASE-5987 (Revision 1343413)
Result = FAILURE
tedyu :
Files :
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/HConstants.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/BlockWithScanInfo.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestCase.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestReseekTo.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestSeekTo.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksScanned.java
> Port HFileBlockIndex improvement from HBASE-5987
> ------------------------------------------------
>
> Key: HBASE-6032
> URL: https://issues.apache.org/jira/browse/HBASE-6032
> Project: HBase
> Issue Type: Task
> Reporter: Zhihong Yu
> Assignee: Zhihong Yu
> Fix For: 0.96.0
>
> Attachments: 6032-ports-5987-v2.txt, 6032-ports-5987.txt
>
>
> Excerpt from HBASE-5987:
> First, we propose to lookahead for one more block index so that the HFileScanner would know the start key value of next data block. So if the target key value for the scan(reSeekTo) is "smaller" than that start kv of next data block, it means the target key value has a very high possibility in the current data block (if not in current data block, then the start kv of next data block should be returned. +Indexing on the start key has some defects here+) and it shall NOT query the HFileBlockIndex in this case. On the contrary, if the target key value is "bigger", then it shall query the HFileBlockIndex. This improvement shall help to reduce the hotness of HFileBlockIndex and avoid some unnecessary IdLock Contention or Index Block Cache lookup.
> This JIRA is to port the fix to HBase trunk, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6032) Port HFileBlockIndex improvement
from HBASE-5987
Posted by "Liyin Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13286099#comment-13286099 ]
Liyin Tang commented on HBASE-6032:
-----------------------------------
Thanks Ted for porting HBASE-5987 to trunk !
Some of the static functions in TestHBaseCase are still quite useful. It allows the unit test play around in the region level.
Maybe we shall move these useful code to HBaseTestingUtility :)
> Port HFileBlockIndex improvement from HBASE-5987
> ------------------------------------------------
>
> Key: HBASE-6032
> URL: https://issues.apache.org/jira/browse/HBASE-6032
> Project: HBase
> Issue Type: Task
> Reporter: Zhihong Yu
> Assignee: Zhihong Yu
> Fix For: 0.96.0
>
> Attachments: 6032-ports-5987-v2.txt, 6032-ports-5987.txt
>
>
> Excerpt from HBASE-5987:
> First, we propose to lookahead for one more block index so that the HFileScanner would know the start key value of next data block. So if the target key value for the scan(reSeekTo) is "smaller" than that start kv of next data block, it means the target key value has a very high possibility in the current data block (if not in current data block, then the start kv of next data block should be returned. +Indexing on the start key has some defects here+) and it shall NOT query the HFileBlockIndex in this case. On the contrary, if the target key value is "bigger", then it shall query the HFileBlockIndex. This improvement shall help to reduce the hotness of HFileBlockIndex and avoid some unnecessary IdLock Contention or Index Block Cache lookup.
> This JIRA is to port the fix to HBase trunk, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6032) Port HFileBlockIndex improvement
from HBASE-5987
Posted by "Zhihong Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13284270#comment-13284270 ]
Zhihong Yu commented on HBASE-6032:
-----------------------------------
I plan to integrate patch v2 tomorrow if there is no objection.
> Port HFileBlockIndex improvement from HBASE-5987
> ------------------------------------------------
>
> Key: HBASE-6032
> URL: https://issues.apache.org/jira/browse/HBASE-6032
> Project: HBase
> Issue Type: Task
> Reporter: Zhihong Yu
> Assignee: Zhihong Yu
> Fix For: 0.96.0
>
> Attachments: 6032-ports-5987-v2.txt, 6032-ports-5987.txt
>
>
> Excerpt from HBASE-5987:
> First, we propose to lookahead for one more block index so that the HFileScanner would know the start key value of next data block. So if the target key value for the scan(reSeekTo) is "smaller" than that start kv of next data block, it means the target key value has a very high possibility in the current data block (if not in current data block, then the start kv of next data block should be returned. +Indexing on the start key has some defects here+) and it shall NOT query the HFileBlockIndex in this case. On the contrary, if the target key value is "bigger", then it shall query the HFileBlockIndex. This improvement shall help to reduce the hotness of HFileBlockIndex and avoid some unnecessary IdLock Contention or Index Block Cache lookup.
> This JIRA is to port the fix to HBase trunk, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6032) Port HFileBlockIndex improvement from
HBASE-5987
Posted by "Zhihong Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zhihong Yu updated HBASE-6032:
------------------------------
Attachment: 6032-ports-5987-v2.txt
Rebase for trunk.
> Port HFileBlockIndex improvement from HBASE-5987
> ------------------------------------------------
>
> Key: HBASE-6032
> URL: https://issues.apache.org/jira/browse/HBASE-6032
> Project: HBase
> Issue Type: Task
> Reporter: Zhihong Yu
> Assignee: Zhihong Yu
> Fix For: 0.96.0
>
> Attachments: 6032-ports-5987-v2.txt, 6032-ports-5987.txt
>
>
> Excerpt from HBASE-5987:
> First, we propose to lookahead for one more block index so that the HFileScanner would know the start key value of next data block. So if the target key value for the scan(reSeekTo) is "smaller" than that start kv of next data block, it means the target key value has a very high possibility in the current data block (if not in current data block, then the start kv of next data block should be returned. +Indexing on the start key has some defects here+) and it shall NOT query the HFileBlockIndex in this case. On the contrary, if the target key value is "bigger", then it shall query the HFileBlockIndex. This improvement shall help to reduce the hotness of HFileBlockIndex and avoid some unnecessary IdLock Contention or Index Block Cache lookup.
> This JIRA is to port the fix to HBase trunk, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6032) Port HFileBlockIndex improvement
from HBASE-5987
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13283098#comment-13283098 ]
Hadoop QA commented on HBASE-6032:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12529563/6032-ports-5987.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 15 new or modified tests.
+1 hadoop23. The patch compiles against the hadoop 0.23.x profile.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 36 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1992//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1992//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1992//console
This message is automatically generated.
> Port HFileBlockIndex improvement from HBASE-5987
> ------------------------------------------------
>
> Key: HBASE-6032
> URL: https://issues.apache.org/jira/browse/HBASE-6032
> Project: HBase
> Issue Type: Task
> Reporter: Zhihong Yu
> Attachments: 6032-ports-5987.txt
>
>
> Excerpt from HBASE-5987:
> First, we propose to lookahead for one more block index so that the HFileScanner would know the start key value of next data block. So if the target key value for the scan(reSeekTo) is "smaller" than that start kv of next data block, it means the target key value has a very high possibility in the current data block (if not in current data block, then the start kv of next data block should be returned. +Indexing on the start key has some defects here+) and it shall NOT query the HFileBlockIndex in this case. On the contrary, if the target key value is "bigger", then it shall query the HFileBlockIndex. This improvement shall help to reduce the hotness of HFileBlockIndex and avoid some unnecessary IdLock Contention or Index Block Cache lookup.
> This JIRA is to port the fix to HBase trunk, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6032) Port HFileBlockIndex improvement
from HBASE-5987
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478252#comment-13478252 ]
Lars Hofhansl commented on HBASE-6032:
--------------------------------------
I ran the relevant tests. And also used with some other test cases I have. (The set was with a local HDFS, so I didn't observe any performance benefits)
> Port HFileBlockIndex improvement from HBASE-5987
> ------------------------------------------------
>
> Key: HBASE-6032
> URL: https://issues.apache.org/jira/browse/HBASE-6032
> Project: HBase
> Issue Type: Task
> Reporter: Ted Yu
> Assignee: Ted Yu
> Fix For: 0.96.0
>
> Attachments: 6032.094.txt, 6032-ports-5987.txt, 6032-ports-5987-v2.txt, 6032v3.txt
>
>
> Excerpt from HBASE-5987:
> First, we propose to lookahead for one more block index so that the HFileScanner would know the start key value of next data block. So if the target key value for the scan(reSeekTo) is "smaller" than that start kv of next data block, it means the target key value has a very high possibility in the current data block (if not in current data block, then the start kv of next data block should be returned. +Indexing on the start key has some defects here+) and it shall NOT query the HFileBlockIndex in this case. On the contrary, if the target key value is "bigger", then it shall query the HFileBlockIndex. This improvement shall help to reduce the hotness of HFileBlockIndex and avoid some unnecessary IdLock Contention or Index Block Cache lookup.
> This JIRA is to port the fix to HBase trunk, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6032) Port HFileBlockIndex improvement
from HBASE-5987
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478077#comment-13478077 ]
Lars Hofhansl commented on HBASE-6032:
--------------------------------------
Awesome. That was quick. Thanks Stack!
I'll test out a bit today and then commit if all is good.
> Port HFileBlockIndex improvement from HBASE-5987
> ------------------------------------------------
>
> Key: HBASE-6032
> URL: https://issues.apache.org/jira/browse/HBASE-6032
> Project: HBase
> Issue Type: Task
> Reporter: Ted Yu
> Assignee: Ted Yu
> Fix For: 0.96.0
>
> Attachments: 6032.094.txt, 6032-ports-5987.txt, 6032-ports-5987-v2.txt, 6032v3.txt
>
>
> Excerpt from HBASE-5987:
> First, we propose to lookahead for one more block index so that the HFileScanner would know the start key value of next data block. So if the target key value for the scan(reSeekTo) is "smaller" than that start kv of next data block, it means the target key value has a very high possibility in the current data block (if not in current data block, then the start kv of next data block should be returned. +Indexing on the start key has some defects here+) and it shall NOT query the HFileBlockIndex in this case. On the contrary, if the target key value is "bigger", then it shall query the HFileBlockIndex. This improvement shall help to reduce the hotness of HFileBlockIndex and avoid some unnecessary IdLock Contention or Index Block Cache lookup.
> This JIRA is to port the fix to HBase trunk, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6032) Port HFileBlockIndex improvement
from HBASE-5987
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13284269#comment-13284269 ]
Hadoop QA commented on HBASE-6032:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12529917/6032-ports-5987-v2.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 15 new or modified tests.
+1 hadoop23. The patch compiles against the hadoop 0.23.x profile.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2021//testReport/
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2021//console
This message is automatically generated.
> Port HFileBlockIndex improvement from HBASE-5987
> ------------------------------------------------
>
> Key: HBASE-6032
> URL: https://issues.apache.org/jira/browse/HBASE-6032
> Project: HBase
> Issue Type: Task
> Reporter: Zhihong Yu
> Assignee: Zhihong Yu
> Fix For: 0.96.0
>
> Attachments: 6032-ports-5987-v2.txt, 6032-ports-5987.txt
>
>
> Excerpt from HBASE-5987:
> First, we propose to lookahead for one more block index so that the HFileScanner would know the start key value of next data block. So if the target key value for the scan(reSeekTo) is "smaller" than that start kv of next data block, it means the target key value has a very high possibility in the current data block (if not in current data block, then the start kv of next data block should be returned. +Indexing on the start key has some defects here+) and it shall NOT query the HFileBlockIndex in this case. On the contrary, if the target key value is "bigger", then it shall query the HFileBlockIndex. This improvement shall help to reduce the hotness of HFileBlockIndex and avoid some unnecessary IdLock Contention or Index Block Cache lookup.
> This JIRA is to port the fix to HBase trunk, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6032) Port HFileBlockIndex improvement
from HBASE-5987
Posted by "Zhihong Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13283122#comment-13283122 ]
Zhihong Yu commented on HBASE-6032:
-----------------------------------
Can someone review the port please ?
> Port HFileBlockIndex improvement from HBASE-5987
> ------------------------------------------------
>
> Key: HBASE-6032
> URL: https://issues.apache.org/jira/browse/HBASE-6032
> Project: HBase
> Issue Type: Task
> Reporter: Zhihong Yu
> Assignee: Zhihong Yu
> Fix For: 0.96.0
>
> Attachments: 6032-ports-5987.txt
>
>
> Excerpt from HBASE-5987:
> First, we propose to lookahead for one more block index so that the HFileScanner would know the start key value of next data block. So if the target key value for the scan(reSeekTo) is "smaller" than that start kv of next data block, it means the target key value has a very high possibility in the current data block (if not in current data block, then the start kv of next data block should be returned. +Indexing on the start key has some defects here+) and it shall NOT query the HFileBlockIndex in this case. On the contrary, if the target key value is "bigger", then it shall query the HFileBlockIndex. This improvement shall help to reduce the hotness of HFileBlockIndex and avoid some unnecessary IdLock Contention or Index Block Cache lookup.
> This JIRA is to port the fix to HBase trunk, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-6032) Port HFileBlockIndex improvement
from HBASE-5987
Posted by "Zhihong Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zhihong Yu reassigned HBASE-6032:
---------------------------------
Assignee: Zhihong Yu
> Port HFileBlockIndex improvement from HBASE-5987
> ------------------------------------------------
>
> Key: HBASE-6032
> URL: https://issues.apache.org/jira/browse/HBASE-6032
> Project: HBase
> Issue Type: Task
> Reporter: Zhihong Yu
> Assignee: Zhihong Yu
> Fix For: 0.96.0
>
> Attachments: 6032-ports-5987.txt
>
>
> Excerpt from HBASE-5987:
> First, we propose to lookahead for one more block index so that the HFileScanner would know the start key value of next data block. So if the target key value for the scan(reSeekTo) is "smaller" than that start kv of next data block, it means the target key value has a very high possibility in the current data block (if not in current data block, then the start kv of next data block should be returned. +Indexing on the start key has some defects here+) and it shall NOT query the HFileBlockIndex in this case. On the contrary, if the target key value is "bigger", then it shall query the HFileBlockIndex. This improvement shall help to reduce the hotness of HFileBlockIndex and avoid some unnecessary IdLock Contention or Index Block Cache lookup.
> This JIRA is to port the fix to HBase trunk, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6032) Port HFileBlockIndex improvement
from HBASE-5987
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478777#comment-13478777 ]
Hudson commented on HBASE-6032:
-------------------------------
Integrated in HBase-0.94 #540 (See [https://builds.apache.org/job/HBase-0.94/540/])
HBASE-6032 Addendum, forgot to add two files (Revision 1399521)
Result = FAILURE
larsh :
Files :
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/BlockWithScanInfo.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksScanned.java
> Port HFileBlockIndex improvement from HBASE-5987
> ------------------------------------------------
>
> Key: HBASE-6032
> URL: https://issues.apache.org/jira/browse/HBASE-6032
> Project: HBase
> Issue Type: Task
> Reporter: Ted Yu
> Assignee: Ted Yu
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 6032.094.txt, 6032-ports-5987.txt, 6032-ports-5987-v2.txt, 6032v3.txt
>
>
> Excerpt from HBASE-5987:
> First, we propose to lookahead for one more block index so that the HFileScanner would know the start key value of next data block. So if the target key value for the scan(reSeekTo) is "smaller" than that start kv of next data block, it means the target key value has a very high possibility in the current data block (if not in current data block, then the start kv of next data block should be returned. +Indexing on the start key has some defects here+) and it shall NOT query the HFileBlockIndex in this case. On the contrary, if the target key value is "bigger", then it shall query the HFileBlockIndex. This improvement shall help to reduce the hotness of HFileBlockIndex and avoid some unnecessary IdLock Contention or Index Block Cache lookup.
> This JIRA is to port the fix to HBase trunk, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6032) Port HFileBlockIndex improvement
from HBASE-5987
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478690#comment-13478690 ]
Lars Hofhansl commented on HBASE-6032:
--------------------------------------
Whoops. Forgot to "svn add" the two new files. Done through an addendum.
> Port HFileBlockIndex improvement from HBASE-5987
> ------------------------------------------------
>
> Key: HBASE-6032
> URL: https://issues.apache.org/jira/browse/HBASE-6032
> Project: HBase
> Issue Type: Task
> Reporter: Ted Yu
> Assignee: Ted Yu
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 6032.094.txt, 6032-ports-5987.txt, 6032-ports-5987-v2.txt, 6032v3.txt
>
>
> Excerpt from HBASE-5987:
> First, we propose to lookahead for one more block index so that the HFileScanner would know the start key value of next data block. So if the target key value for the scan(reSeekTo) is "smaller" than that start kv of next data block, it means the target key value has a very high possibility in the current data block (if not in current data block, then the start kv of next data block should be returned. +Indexing on the start key has some defects here+) and it shall NOT query the HFileBlockIndex in this case. On the contrary, if the target key value is "bigger", then it shall query the HFileBlockIndex. This improvement shall help to reduce the hotness of HFileBlockIndex and avoid some unnecessary IdLock Contention or Index Block Cache lookup.
> This JIRA is to port the fix to HBase trunk, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6032) Port HFileBlockIndex improvement from
HBASE-5987
Posted by "Zhihong Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zhihong Yu updated HBASE-6032:
------------------------------
Status: Patch Available (was: Open)
> Port HFileBlockIndex improvement from HBASE-5987
> ------------------------------------------------
>
> Key: HBASE-6032
> URL: https://issues.apache.org/jira/browse/HBASE-6032
> Project: HBase
> Issue Type: Task
> Reporter: Zhihong Yu
> Attachments: 6032-ports-5987.txt
>
>
> Excerpt from HBASE-5987:
> First, we propose to lookahead for one more block index so that the HFileScanner would know the start key value of next data block. So if the target key value for the scan(reSeekTo) is "smaller" than that start kv of next data block, it means the target key value has a very high possibility in the current data block (if not in current data block, then the start kv of next data block should be returned. +Indexing on the start key has some defects here+) and it shall NOT query the HFileBlockIndex in this case. On the contrary, if the target key value is "bigger", then it shall query the HFileBlockIndex. This improvement shall help to reduce the hotness of HFileBlockIndex and avoid some unnecessary IdLock Contention or Index Block Cache lookup.
> This JIRA is to port the fix to HBase trunk, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6032) Port HFileBlockIndex improvement
from HBASE-5987
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13490376#comment-13490376 ]
Hudson commented on HBASE-6032:
-------------------------------
Integrated in HBase-0.94-security-on-Hadoop-23 #9 (See [https://builds.apache.org/job/HBase-0.94-security-on-Hadoop-23/9/])
HBASE-6032 Addendum, forgot to add two files (Revision 1399521)
HBASE-6032 Port HFileBlockIndex improvement from HBASE-5987 (Liyin, Ted, Stack) (Revision 1399513)
Result = FAILURE
larsh :
Files :
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/BlockWithScanInfo.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksScanned.java
larsh :
Files :
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/HConstants.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/HBaseTestCase.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/io/hfile/TestReseekTo.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/io/hfile/TestSeekTo.java
> Port HFileBlockIndex improvement from HBASE-5987
> ------------------------------------------------
>
> Key: HBASE-6032
> URL: https://issues.apache.org/jira/browse/HBASE-6032
> Project: HBase
> Issue Type: Task
> Reporter: Ted Yu
> Assignee: Ted Yu
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 6032.094.txt, 6032-ports-5987.txt, 6032-ports-5987-v2.txt, 6032v3.txt
>
>
> Excerpt from HBASE-5987:
> First, we propose to lookahead for one more block index so that the HFileScanner would know the start key value of next data block. So if the target key value for the scan(reSeekTo) is "smaller" than that start kv of next data block, it means the target key value has a very high possibility in the current data block (if not in current data block, then the start kv of next data block should be returned. +Indexing on the start key has some defects here+) and it shall NOT query the HFileBlockIndex in this case. On the contrary, if the target key value is "bigger", then it shall query the HFileBlockIndex. This improvement shall help to reduce the hotness of HFileBlockIndex and avoid some unnecessary IdLock Contention or Index Block Cache lookup.
> This JIRA is to port the fix to HBase trunk, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6032) Port HFileBlockIndex improvement
from HBASE-5987
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13477638#comment-13477638 ]
stack commented on HBASE-6032:
------------------------------
I've not run the tests but +1 on commit if all tests run (nice tests included w/ this patch)
> Port HFileBlockIndex improvement from HBASE-5987
> ------------------------------------------------
>
> Key: HBASE-6032
> URL: https://issues.apache.org/jira/browse/HBASE-6032
> Project: HBase
> Issue Type: Task
> Reporter: Ted Yu
> Assignee: Ted Yu
> Fix For: 0.96.0
>
> Attachments: 6032.094.txt, 6032-ports-5987.txt, 6032-ports-5987-v2.txt, 6032v3.txt
>
>
> Excerpt from HBASE-5987:
> First, we propose to lookahead for one more block index so that the HFileScanner would know the start key value of next data block. So if the target key value for the scan(reSeekTo) is "smaller" than that start kv of next data block, it means the target key value has a very high possibility in the current data block (if not in current data block, then the start kv of next data block should be returned. +Indexing on the start key has some defects here+) and it shall NOT query the HFileBlockIndex in this case. On the contrary, if the target key value is "bigger", then it shall query the HFileBlockIndex. This improvement shall help to reduce the hotness of HFileBlockIndex and avoid some unnecessary IdLock Contention or Index Block Cache lookup.
> This JIRA is to port the fix to HBase trunk, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6032) Port HFileBlockIndex improvement from
HBASE-5987
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-6032:
-------------------------
Attachment: 6032.094.txt
6032v3.txt
Version of patch that will apply to 0.94
> Port HFileBlockIndex improvement from HBASE-5987
> ------------------------------------------------
>
> Key: HBASE-6032
> URL: https://issues.apache.org/jira/browse/HBASE-6032
> Project: HBase
> Issue Type: Task
> Reporter: Ted Yu
> Assignee: Ted Yu
> Fix For: 0.96.0
>
> Attachments: 6032.094.txt, 6032-ports-5987.txt, 6032-ports-5987-v2.txt, 6032v3.txt
>
>
> Excerpt from HBASE-5987:
> First, we propose to lookahead for one more block index so that the HFileScanner would know the start key value of next data block. So if the target key value for the scan(reSeekTo) is "smaller" than that start kv of next data block, it means the target key value has a very high possibility in the current data block (if not in current data block, then the start kv of next data block should be returned. +Indexing on the start key has some defects here+) and it shall NOT query the HFileBlockIndex in this case. On the contrary, if the target key value is "bigger", then it shall query the HFileBlockIndex. This improvement shall help to reduce the hotness of HFileBlockIndex and avoid some unnecessary IdLock Contention or Index Block Cache lookup.
> This JIRA is to port the fix to HBase trunk, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6032) Port HFileBlockIndex improvement from
HBASE-5987
Posted by "Zhihong Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zhihong Yu updated HBASE-6032:
------------------------------
Status: Patch Available (was: Open)
> Port HFileBlockIndex improvement from HBASE-5987
> ------------------------------------------------
>
> Key: HBASE-6032
> URL: https://issues.apache.org/jira/browse/HBASE-6032
> Project: HBase
> Issue Type: Task
> Reporter: Zhihong Yu
> Assignee: Zhihong Yu
> Fix For: 0.96.0
>
> Attachments: 6032-ports-5987-v2.txt, 6032-ports-5987.txt
>
>
> Excerpt from HBASE-5987:
> First, we propose to lookahead for one more block index so that the HFileScanner would know the start key value of next data block. So if the target key value for the scan(reSeekTo) is "smaller" than that start kv of next data block, it means the target key value has a very high possibility in the current data block (if not in current data block, then the start kv of next data block should be returned. +Indexing on the start key has some defects here+) and it shall NOT query the HFileBlockIndex in this case. On the contrary, if the target key value is "bigger", then it shall query the HFileBlockIndex. This improvement shall help to reduce the hotness of HFileBlockIndex and avoid some unnecessary IdLock Contention or Index Block Cache lookup.
> This JIRA is to port the fix to HBase trunk, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6032) Port HFileBlockIndex improvement
from HBASE-5987
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13477612#comment-13477612 ]
ramkrishna.s.vasudevan commented on HBASE-6032:
-----------------------------------------------
We need this for 0.94 i think as per the changes in HBASE-6577.
> Port HFileBlockIndex improvement from HBASE-5987
> ------------------------------------------------
>
> Key: HBASE-6032
> URL: https://issues.apache.org/jira/browse/HBASE-6032
> Project: HBase
> Issue Type: Task
> Reporter: Ted Yu
> Assignee: Ted Yu
> Fix For: 0.96.0
>
> Attachments: 6032-ports-5987.txt, 6032-ports-5987-v2.txt
>
>
> Excerpt from HBASE-5987:
> First, we propose to lookahead for one more block index so that the HFileScanner would know the start key value of next data block. So if the target key value for the scan(reSeekTo) is "smaller" than that start kv of next data block, it means the target key value has a very high possibility in the current data block (if not in current data block, then the start kv of next data block should be returned. +Indexing on the start key has some defects here+) and it shall NOT query the HFileBlockIndex in this case. On the contrary, if the target key value is "bigger", then it shall query the HFileBlockIndex. This improvement shall help to reduce the hotness of HFileBlockIndex and avoid some unnecessary IdLock Contention or Index Block Cache lookup.
> This JIRA is to port the fix to HBase trunk, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira