You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Nicolas Spiegelberg (JIRA)" <ji...@apache.org> on 2010/11/13 05:42:13 UTC
[jira] Created: (HBASE-3232) Fix KeyOnlyFilter + Add Value Length
Fix KeyOnlyFilter + Add Value Length
------------------------------------
Key: HBASE-3232
URL: https://issues.apache.org/jira/browse/HBASE-3232
Project: HBase
Issue Type: Bug
Reporter: Nicolas Spiegelberg
Assignee: Nicolas Spiegelberg
Priority: Blocker
Fix For: 0.90.0
HBASE-3211 altered filter code to mutate KeyValues. What could go wrong? Well, your scan could mess up because the KVHeap compare functions don't work properly. If we're going to soft mutate KVs in filter code, we also need to soft copy the KV before filtering. This was found while adding the ability to have KeyOnlyFilter have the option to return the Value's length. This is useful for grouping your reduce tasks into equal-sized blocks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] [Commented] (HBASE-3232) Fix KeyOnlyFilter + Add Value
Length
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13148950#comment-13148950 ]
Hudson commented on HBASE-3232:
-------------------------------
Integrated in HBase-0.92 #125 (See [https://builds.apache.org/job/HBase-0.92/125/])
HBASE-3433 Remove the KV copy of every KV in Scan; introduced by HBASE-3232
larsh :
Files :
* /hbase/branches/0.92/CHANGES.txt
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/KeyValue.java
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/filter/Filter.java
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/filter/FilterBase.java
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/filter/FilterList.java
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/filter/KeyOnlyFilter.java
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/filter/SkipFilter.java
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/filter/WhileMatchFilter.java
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/TestKeyValue.java
> Fix KeyOnlyFilter + Add Value Length
> ------------------------------------
>
> Key: HBASE-3232
> URL: https://issues.apache.org/jira/browse/HBASE-3232
> Project: HBase
> Issue Type: Bug
> Reporter: Nicolas Spiegelberg
> Assignee: Nicolas Spiegelberg
> Priority: Blocker
> Fix For: 0.90.0
>
>
> HBASE-3211 altered filter code to mutate KeyValues. What could go wrong? Well, your scan could mess up because the KVHeap compare functions don't work properly. If we're going to soft mutate KVs in filter code, we also need to soft copy the KV before filtering. This was found while adding the ability to have KeyOnlyFilter have the option to return the Value's length. This is useful for grouping your reduce tasks into equal-sized blocks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HBASE-3232) Fix KeyOnlyFilter + Add Value Length
Posted by "HBase Review Board (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12931967#action_12931967 ]
HBase Review Board commented on HBASE-3232:
-------------------------------------------
Message from: "Ryan Rawson" <ry...@gmail.com>
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/1213/#review1922
-----------------------------------------------------------
looks great, i just committed it
- Ryan
> Fix KeyOnlyFilter + Add Value Length
> ------------------------------------
>
> Key: HBASE-3232
> URL: https://issues.apache.org/jira/browse/HBASE-3232
> Project: HBase
> Issue Type: Bug
> Reporter: Nicolas Spiegelberg
> Assignee: Nicolas Spiegelberg
> Priority: Blocker
> Fix For: 0.90.0
>
>
> HBASE-3211 altered filter code to mutate KeyValues. What could go wrong? Well, your scan could mess up because the KVHeap compare functions don't work properly. If we're going to soft mutate KVs in filter code, we also need to soft copy the KV before filtering. This was found while adding the ability to have KeyOnlyFilter have the option to return the Value's length. This is useful for grouping your reduce tasks into equal-sized blocks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3232) Fix KeyOnlyFilter + Add Value Length
Posted by "HBase Review Board (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12931630#action_12931630 ]
HBase Review Board commented on HBASE-3232:
-------------------------------------------
Message from: "Nicolas" <ns...@facebook.com>
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/1213/
-----------------------------------------------------------
Review request for hbase.
Summary
-------
HBASE-3211 altered filter code to mutate KeyValues. What could go wrong? Well, your scan could mess up because the KVHeap compare functions don't work properly. If we're going to soft mutate KVs in filter code, we also need to soft copy the KV before filtering. This was found while adding the ability to have KeyOnlyFilter have the option to return the Value's length. This is useful for grouping your reduce tasks into equal-sized blocks.
This addresses bug HBASE-3232.
http://issues.apache.org/jira/browse/HBASE-3232
Diffs
-----
trunk/src/main/java/org/apache/hadoop/hbase/KeyValue.java 1034646
trunk/src/main/java/org/apache/hadoop/hbase/filter/KeyOnlyFilter.java 1034646
trunk/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java 1034646
trunk/src/test/java/org/apache/hadoop/hbase/TestKeyValue.java 1034646
trunk/src/test/java/org/apache/hadoop/hbase/filter/TestFilter.java 1034646
Diff: http://review.cloudera.org/r/1213/diff
Testing
-------
mvn clean test
Thanks,
Nicolas
> Fix KeyOnlyFilter + Add Value Length
> ------------------------------------
>
> Key: HBASE-3232
> URL: https://issues.apache.org/jira/browse/HBASE-3232
> Project: HBase
> Issue Type: Bug
> Reporter: Nicolas Spiegelberg
> Assignee: Nicolas Spiegelberg
> Priority: Blocker
> Fix For: 0.90.0
>
>
> HBASE-3211 altered filter code to mutate KeyValues. What could go wrong? Well, your scan could mess up because the KVHeap compare functions don't work properly. If we're going to soft mutate KVs in filter code, we also need to soft copy the KV before filtering. This was found while adding the ability to have KeyOnlyFilter have the option to return the Value's length. This is useful for grouping your reduce tasks into equal-sized blocks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3232) Fix KeyOnlyFilter + Add Value Length
Posted by "HBase Review Board (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12931962#action_12931962 ]
HBase Review Board commented on HBASE-3232:
-------------------------------------------
Message from: "Nicolas" <ns...@facebook.com>
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/1213/
-----------------------------------------------------------
(Updated 2010-11-14 18:28:50.196433)
Review request for hbase.
Changes
-------
Because I didn't implement write/readFields for KeyOnlyFilter when I added the param, client -> server serialization didn't work and the default value of false was always used. Fixed + added associated unit test
Summary
-------
HBASE-3211 altered filter code to mutate KeyValues. What could go wrong? Well, your scan could mess up because the KVHeap compare functions don't work properly. If we're going to soft mutate KVs in filter code, we also need to soft copy the KV before filtering. This was found while adding the ability to have KeyOnlyFilter have the option to return the Value's length. This is useful for grouping your reduce tasks into equal-sized blocks.
This addresses bug HBASE-3232.
http://issues.apache.org/jira/browse/HBASE-3232
Diffs (updated)
-----
trunk/src/main/java/org/apache/hadoop/hbase/KeyValue.java 1034646
trunk/src/main/java/org/apache/hadoop/hbase/filter/KeyOnlyFilter.java 1034646
trunk/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java 1034646
trunk/src/test/java/org/apache/hadoop/hbase/TestKeyValue.java 1034646
trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java 1034646
trunk/src/test/java/org/apache/hadoop/hbase/filter/TestFilter.java 1034646
Diff: http://review.cloudera.org/r/1213/diff
Testing
-------
mvn clean test
Thanks,
Nicolas
> Fix KeyOnlyFilter + Add Value Length
> ------------------------------------
>
> Key: HBASE-3232
> URL: https://issues.apache.org/jira/browse/HBASE-3232
> Project: HBase
> Issue Type: Bug
> Reporter: Nicolas Spiegelberg
> Assignee: Nicolas Spiegelberg
> Priority: Blocker
> Fix For: 0.90.0
>
>
> HBASE-3211 altered filter code to mutate KeyValues. What could go wrong? Well, your scan could mess up because the KVHeap compare functions don't work properly. If we're going to soft mutate KVs in filter code, we also need to soft copy the KV before filtering. This was found while adding the ability to have KeyOnlyFilter have the option to return the Value's length. This is useful for grouping your reduce tasks into equal-sized blocks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] [Commented] (HBASE-3232) Fix KeyOnlyFilter + Add Value
Length
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13148982#comment-13148982 ]
Hudson commented on HBASE-3232:
-------------------------------
Integrated in HBase-TRUNK #2430 (See [https://builds.apache.org/job/HBase-TRUNK/2430/])
HBASE-3433 Remove the KV copy of every KV in Scan; introduced by HBASE-3232
larsh :
Files :
* /hbase/trunk/CHANGES.txt
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/KeyValue.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/Filter.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/FilterBase.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/FilterList.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/KeyOnlyFilter.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/SkipFilter.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/WhileMatchFilter.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/TestKeyValue.java
> Fix KeyOnlyFilter + Add Value Length
> ------------------------------------
>
> Key: HBASE-3232
> URL: https://issues.apache.org/jira/browse/HBASE-3232
> Project: HBase
> Issue Type: Bug
> Reporter: Nicolas Spiegelberg
> Assignee: Nicolas Spiegelberg
> Priority: Blocker
> Fix For: 0.90.0
>
>
> HBASE-3211 altered filter code to mutate KeyValues. What could go wrong? Well, your scan could mess up because the KVHeap compare functions don't work properly. If we're going to soft mutate KVs in filter code, we also need to soft copy the KV before filtering. This was found while adding the ability to have KeyOnlyFilter have the option to return the Value's length. This is useful for grouping your reduce tasks into equal-sized blocks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HBASE-3232) Fix KeyOnlyFilter + Add Value Length
Posted by "ryan rawson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12931624#action_12931624 ]
ryan rawson commented on HBASE-3232:
------------------------------------
This feels like something that should be done in co processors
> Fix KeyOnlyFilter + Add Value Length
> ------------------------------------
>
> Key: HBASE-3232
> URL: https://issues.apache.org/jira/browse/HBASE-3232
> Project: HBase
> Issue Type: Bug
> Reporter: Nicolas Spiegelberg
> Assignee: Nicolas Spiegelberg
> Priority: Blocker
> Fix For: 0.90.0
>
>
> HBASE-3211 altered filter code to mutate KeyValues. What could go wrong? Well, your scan could mess up because the KVHeap compare functions don't work properly. If we're going to soft mutate KVs in filter code, we also need to soft copy the KV before filtering. This was found while adding the ability to have KeyOnlyFilter have the option to return the Value's length. This is useful for grouping your reduce tasks into equal-sized blocks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3232) Fix KeyOnlyFilter + Add Value Length
Posted by "Nicolas Spiegelberg (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12931625#action_12931625 ]
Nicolas Spiegelberg commented on HBASE-3232:
--------------------------------------------
I agree. This was a lightweight solution necessary for our existing deployment to scale in 0.90.
> Fix KeyOnlyFilter + Add Value Length
> ------------------------------------
>
> Key: HBASE-3232
> URL: https://issues.apache.org/jira/browse/HBASE-3232
> Project: HBase
> Issue Type: Bug
> Reporter: Nicolas Spiegelberg
> Assignee: Nicolas Spiegelberg
> Priority: Blocker
> Fix For: 0.90.0
>
>
> HBASE-3211 altered filter code to mutate KeyValues. What could go wrong? Well, your scan could mess up because the KVHeap compare functions don't work properly. If we're going to soft mutate KVs in filter code, we also need to soft copy the KV before filtering. This was found while adding the ability to have KeyOnlyFilter have the option to return the Value's length. This is useful for grouping your reduce tasks into equal-sized blocks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HBASE-3232) Fix KeyOnlyFilter + Add Value Length
Posted by "ryan rawson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ryan rawson resolved HBASE-3232.
--------------------------------
Resolution: Fixed
committed to trunk + 0.90
> Fix KeyOnlyFilter + Add Value Length
> ------------------------------------
>
> Key: HBASE-3232
> URL: https://issues.apache.org/jira/browse/HBASE-3232
> Project: HBase
> Issue Type: Bug
> Reporter: Nicolas Spiegelberg
> Assignee: Nicolas Spiegelberg
> Priority: Blocker
> Fix For: 0.90.0
>
>
> HBASE-3211 altered filter code to mutate KeyValues. What could go wrong? Well, your scan could mess up because the KVHeap compare functions don't work properly. If we're going to soft mutate KVs in filter code, we also need to soft copy the KV before filtering. This was found while adding the ability to have KeyOnlyFilter have the option to return the Value's length. This is useful for grouping your reduce tasks into equal-sized blocks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.