You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "stack (JIRA)" <ji...@apache.org> on 2009/09/10 00:05:58 UTC
[jira] Created: (HBASE-1821) Filtering by SingleColumnValueFilter
bug
Filtering by SingleColumnValueFilter bug
----------------------------------------
Key: HBASE-1821
URL: https://issues.apache.org/jira/browse/HBASE-1821
Project: Hadoop HBase
Issue Type: Bug
Reporter: stack
Fix For: 0.20.1
See up in hbase-user mailing list under title 'Filtering by SingleColumnValueFilter'. Adrian describes roughly how he fixed the issue (And what he wanted the filter to do).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1821) Filtering by SingleColumnValueFilter
bug
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Gray updated HBASE-1821:
---------------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
Committed to branch and trunk.
> Filtering by SingleColumnValueFilter bug
> ----------------------------------------
>
> Key: HBASE-1821
> URL: https://issues.apache.org/jira/browse/HBASE-1821
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: Jonathan Gray
> Fix For: 0.20.1, 0.21.0
>
> Attachments: 1821-SCVF.patch, HBASE-1821-v2.patch, HBASE-1821-v3.patch, HBASE-1821-v5.patch, HBASE-1821.patch
>
>
> See up in hbase-user mailing list under title 'Filtering by SingleColumnValueFilter'. Adrian describes roughly how he fixed the issue (And what he wanted the filter to do).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1821) Filtering by SingleColumnValueFilter
bug
Posted by "Clint Morgan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12753885#action_12753885 ]
Clint Morgan commented on HBASE-1821:
-------------------------------------
I just found this.
In the ValueFilter that I ported for 0.20 we had a filterIfColumnMissing property that I used for this. Looks like JG wipped it when fixing 1790 between rc2 and rc3. Wish I would have caught it then because I would have -1 RC3...
Sometimes I want missing columns to cause the row to be filtered, sometimes not...
I'll try out this patch and see if it can work for us.
> Filtering by SingleColumnValueFilter bug
> ----------------------------------------
>
> Key: HBASE-1821
> URL: https://issues.apache.org/jira/browse/HBASE-1821
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: Jonathan Gray
> Fix For: 0.20.1, 0.21.0
>
> Attachments: HBASE-1821-v2.patch, HBASE-1821-v3.patch, HBASE-1821.patch
>
>
> See up in hbase-user mailing list under title 'Filtering by SingleColumnValueFilter'. Adrian describes roughly how he fixed the issue (And what he wanted the filter to do).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1821) Filtering by SingleColumnValueFilter
bug
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12754460#action_12754460 ]
Jonathan Gray commented on HBASE-1821:
--------------------------------------
New patch coming. SkipFilter is really not going to work with this. I think we should leave it in because it could be applicable to other cases.
Behavior of SCVF should be that if the column is found, then the row being emitted or not is based on whether or not the comparison passed or failed. Is that what you expect Clint?
> Filtering by SingleColumnValueFilter bug
> ----------------------------------------
>
> Key: HBASE-1821
> URL: https://issues.apache.org/jira/browse/HBASE-1821
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: Jonathan Gray
> Fix For: 0.20.1, 0.21.0
>
> Attachments: 1821-SCVF.patch, HBASE-1821-v2.patch, HBASE-1821-v3.patch, HBASE-1821.patch
>
>
> See up in hbase-user mailing list under title 'Filtering by SingleColumnValueFilter'. Adrian describes roughly how he fixed the issue (And what he wanted the filter to do).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1821) Filtering by SingleColumnValueFilter
bug
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12754447#action_12754447 ]
Jonathan Gray commented on HBASE-1821:
--------------------------------------
To further explain... SkipFilter can be used to say "if the test for the single column fails, skip the entire row"... what it doesn't say is, "if nothing fails but the column isn't found, skip or pass through the entire row"
> Filtering by SingleColumnValueFilter bug
> ----------------------------------------
>
> Key: HBASE-1821
> URL: https://issues.apache.org/jira/browse/HBASE-1821
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: Jonathan Gray
> Fix For: 0.20.1, 0.21.0
>
> Attachments: 1821-SCVF.patch, HBASE-1821-v2.patch, HBASE-1821-v3.patch, HBASE-1821.patch
>
>
> See up in hbase-user mailing list under title 'Filtering by SingleColumnValueFilter'. Adrian describes roughly how he fixed the issue (And what he wanted the filter to do).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1821) Filtering by SingleColumnValueFilter
bug
Posted by "Clint Morgan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12755115#action_12755115 ]
Clint Morgan commented on HBASE-1821:
-------------------------------------
Yeah, this works for me. Thanks!
> Filtering by SingleColumnValueFilter bug
> ----------------------------------------
>
> Key: HBASE-1821
> URL: https://issues.apache.org/jira/browse/HBASE-1821
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: Jonathan Gray
> Fix For: 0.20.1, 0.21.0
>
> Attachments: 1821-SCVF.patch, HBASE-1821-v2.patch, HBASE-1821-v3.patch, HBASE-1821-v5.patch, HBASE-1821.patch
>
>
> See up in hbase-user mailing list under title 'Filtering by SingleColumnValueFilter'. Adrian describes roughly how he fixed the issue (And what he wanted the filter to do).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1821) Filtering by SingleColumnValueFilter
bug
Posted by "Olariu Adrian (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12753678#action_12753678 ]
Olariu Adrian commented on HBASE-1821:
--------------------------------------
I also added this:
public ReturnCode filterKeyValue(KeyValue keyValue) {
+ if (matchFound) {
+ return ReturnCode.INCLUDE;
+ }
if (!keyValue.matchingColumn(this.columnFamily, this.columnQualifier)) {
If a match on the row was already been found don't check for column match anymore.
This can be efficient when filtering rows whit very large number of columns and the column searched for it's on the first ones.
> Filtering by SingleColumnValueFilter bug
> ----------------------------------------
>
> Key: HBASE-1821
> URL: https://issues.apache.org/jira/browse/HBASE-1821
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: Jonathan Gray
> Fix For: 0.20.1
>
> Attachments: HBASE-1821.patch
>
>
> See up in hbase-user mailing list under title 'Filtering by SingleColumnValueFilter'. Adrian describes roughly how he fixed the issue (And what he wanted the filter to do).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1821) Filtering by SingleColumnValueFilter
bug
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12754273#action_12754273 ]
Jonathan Gray commented on HBASE-1821:
--------------------------------------
I will try to write up a couple more unit tests so we can clearly define expected/actual behavior.
> Filtering by SingleColumnValueFilter bug
> ----------------------------------------
>
> Key: HBASE-1821
> URL: https://issues.apache.org/jira/browse/HBASE-1821
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: Jonathan Gray
> Fix For: 0.20.1, 0.21.0
>
> Attachments: 1821-SCVF.patch, HBASE-1821-v2.patch, HBASE-1821-v3.patch, HBASE-1821.patch
>
>
> See up in hbase-user mailing list under title 'Filtering by SingleColumnValueFilter'. Adrian describes roughly how he fixed the issue (And what he wanted the filter to do).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1821) Filtering by SingleColumnValueFilter
bug
Posted by "Olariu Adrian (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Olariu Adrian updated HBASE-1821:
---------------------------------
Attachment: HBASE-1821.patch
Problem description:
Filter filter1 = new SingleColumnValueFilter(Bytes.toBytes("F1"), Bytes.toBytes("c1"), CompareOp.EQUAL, Bytes.toBytes("1"));
Filter filter2 = new SingleColumnValueFilter(Bytes.toBytes("F1"), Bytes.toBytes("c2"), CompareOp.EQUAL, Bytes.toBytes("1"));
List<Filter> rowFilters = new ArrayList<Filter>();
rowFilters.add(filter1);rowFilters.add(filter2);
Filter filter = new FilterList(FilterList.Operator.MUST_PASS_ALL, rowFilters);
s.addColumn(Bytes.toBytes("F1"), Bytes.toBytes("c1"));
s.addColumn(Bytes.toBytes("F1"), Bytes.toBytes("c2"));
s.setFilter(filter);
This scan returns all rows the contains F1:c1 whit val "1" or contains F1:c2 whit val "1" or contains both. It should return only the rows that match both: filter1 and filter2.
Solution:
if the filter noesn't find inside a row a Column-Value match than the FilterRow method should return true.
Patch attached.
> Filtering by SingleColumnValueFilter bug
> ----------------------------------------
>
> Key: HBASE-1821
> URL: https://issues.apache.org/jira/browse/HBASE-1821
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: Jonathan Gray
> Fix For: 0.20.1
>
> Attachments: HBASE-1821.patch
>
>
> See up in hbase-user mailing list under title 'Filtering by SingleColumnValueFilter'. Adrian describes roughly how he fixed the issue (And what he wanted the filter to do).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1821) Filtering by SingleColumnValueFilter
bug
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12753889#action_12753889 ]
Jonathan Gray commented on HBASE-1821:
--------------------------------------
@Clint the behavior you had in the existing ValueFilter is not specific to ValueFilter (or not SingleColumnValueFilter). I thought it to be generically useful.
I created SkipFilter instead. So you can wrap SCVF with a SkipFilter, and you will get the behavior you are looking for.
Let me know if something still doesn't work for you. Don't worry I wasn't going to drop your behavior completely! :)
> Filtering by SingleColumnValueFilter bug
> ----------------------------------------
>
> Key: HBASE-1821
> URL: https://issues.apache.org/jira/browse/HBASE-1821
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: Jonathan Gray
> Fix For: 0.20.1, 0.21.0
>
> Attachments: HBASE-1821-v2.patch, HBASE-1821-v3.patch, HBASE-1821.patch
>
>
> See up in hbase-user mailing list under title 'Filtering by SingleColumnValueFilter'. Adrian describes roughly how he fixed the issue (And what he wanted the filter to do).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1821) Filtering by SingleColumnValueFilter
bug
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Gray updated HBASE-1821:
---------------------------------
Attachment: HBASE-1821-v2.patch
Patch is influenced from Olariu's patch. Thanks.
Building on it, we need to track both whether we hit the column at all, as well as whether we matched it successfully or not.
Adds a test to TestFilter that defines the behavior and shows how to achieve what was originally wanted.
One thing... If you want to prevent rows from showing up that don't have the column at all (rather than passed/failed the value check), you need to wrap them in a SkipFilter. This is explained and defined in the test as well.
Also had to make two edits to the existing TestSCVF. Once the filter hits the column and matches, it will always return true for further KVs. This is from the optimization Olariu explained which is part of my patch as well.
Tests pass. Please review.
> Filtering by SingleColumnValueFilter bug
> ----------------------------------------
>
> Key: HBASE-1821
> URL: https://issues.apache.org/jira/browse/HBASE-1821
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: Jonathan Gray
> Fix For: 0.20.1
>
> Attachments: HBASE-1821-v2.patch, HBASE-1821.patch
>
>
> See up in hbase-user mailing list under title 'Filtering by SingleColumnValueFilter'. Adrian describes roughly how he fixed the issue (And what he wanted the filter to do).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1821) Filtering by SingleColumnValueFilter
bug
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12755161#action_12755161 ]
stack commented on HBASE-1821:
------------------------------
I like all the tests. Commit I'd say. Clint says it works for him.
> Filtering by SingleColumnValueFilter bug
> ----------------------------------------
>
> Key: HBASE-1821
> URL: https://issues.apache.org/jira/browse/HBASE-1821
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: Jonathan Gray
> Fix For: 0.20.1, 0.21.0
>
> Attachments: 1821-SCVF.patch, HBASE-1821-v2.patch, HBASE-1821-v3.patch, HBASE-1821-v5.patch, HBASE-1821.patch
>
>
> See up in hbase-user mailing list under title 'Filtering by SingleColumnValueFilter'. Adrian describes roughly how he fixed the issue (And what he wanted the filter to do).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1821) Filtering by SingleColumnValueFilter
bug
Posted by "Clint Morgan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Clint Morgan updated HBASE-1821:
--------------------------------
Attachment: 1821-SCVF.patch
Ok, I see it. But I can't make that SkipFilter business work as I want it.
I want to let the whole row through if the column is not there. But this does not seem possible with SkipFilter. I have not dug to deep, but just wrapping with a SF does not work. Instead I added back a filterIfColValMissing to SCVF. That works for me.
Am I just being dense and missing something?
Attached is that patch I used for SCVF (on top of yours). Sorry for the lack of unit tests; it fixed several of our tests...
BTW, I really like all the other changes to filters. Thanks for cleaning it up!
> Filtering by SingleColumnValueFilter bug
> ----------------------------------------
>
> Key: HBASE-1821
> URL: https://issues.apache.org/jira/browse/HBASE-1821
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: Jonathan Gray
> Fix For: 0.20.1, 0.21.0
>
> Attachments: 1821-SCVF.patch, HBASE-1821-v2.patch, HBASE-1821-v3.patch, HBASE-1821.patch
>
>
> See up in hbase-user mailing list under title 'Filtering by SingleColumnValueFilter'. Adrian describes roughly how he fixed the issue (And what he wanted the filter to do).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1821) Filtering by SingleColumnValueFilter
bug
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Gray updated HBASE-1821:
---------------------------------
Attachment: HBASE-1821-v5.patch
Adds a bunch of unit tests that define and verify behavior.
Adds/modifies class comments in SCVF to explain how it behaves.
All tests pass.
Please review and comment, especially Clint so I can make sure to get what you need in here.
> Filtering by SingleColumnValueFilter bug
> ----------------------------------------
>
> Key: HBASE-1821
> URL: https://issues.apache.org/jira/browse/HBASE-1821
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: Jonathan Gray
> Fix For: 0.20.1, 0.21.0
>
> Attachments: 1821-SCVF.patch, HBASE-1821-v2.patch, HBASE-1821-v3.patch, HBASE-1821-v5.patch, HBASE-1821.patch
>
>
> See up in hbase-user mailing list under title 'Filtering by SingleColumnValueFilter'. Adrian describes roughly how he fixed the issue (And what he wanted the filter to do).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1821) Filtering by SingleColumnValueFilter
bug
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Gray updated HBASE-1821:
---------------------------------
Fix Version/s: 0.21.0
Status: Patch Available (was: Open)
> Filtering by SingleColumnValueFilter bug
> ----------------------------------------
>
> Key: HBASE-1821
> URL: https://issues.apache.org/jira/browse/HBASE-1821
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: Jonathan Gray
> Fix For: 0.20.1, 0.21.0
>
> Attachments: HBASE-1821-v2.patch, HBASE-1821-v3.patch, HBASE-1821.patch
>
>
> See up in hbase-user mailing list under title 'Filtering by SingleColumnValueFilter'. Adrian describes roughly how he fixed the issue (And what he wanted the filter to do).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1821) Filtering by SingleColumnValueFilter
bug
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12754446#action_12754446 ]
Jonathan Gray commented on HBASE-1821:
--------------------------------------
Thought on this more. I think you're right, Clint. There's no way for a wrapping class to specify "do not let row through if column not found", or vice versa depending on default behavior of SCVF.
I will merge our patches and try to get some more tests finished. Early next week.
> Filtering by SingleColumnValueFilter bug
> ----------------------------------------
>
> Key: HBASE-1821
> URL: https://issues.apache.org/jira/browse/HBASE-1821
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: Jonathan Gray
> Fix For: 0.20.1, 0.21.0
>
> Attachments: 1821-SCVF.patch, HBASE-1821-v2.patch, HBASE-1821-v3.patch, HBASE-1821.patch
>
>
> See up in hbase-user mailing list under title 'Filtering by SingleColumnValueFilter'. Adrian describes roughly how he fixed the issue (And what he wanted the filter to do).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HBASE-1821) Filtering by SingleColumnValueFilter
bug
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Gray reassigned HBASE-1821:
------------------------------------
Assignee: Jonathan Gray
Assigning to myself so I look at this tomorrow.
> Filtering by SingleColumnValueFilter bug
> ----------------------------------------
>
> Key: HBASE-1821
> URL: https://issues.apache.org/jira/browse/HBASE-1821
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: Jonathan Gray
> Fix For: 0.20.1
>
>
> See up in hbase-user mailing list under title 'Filtering by SingleColumnValueFilter'. Adrian describes roughly how he fixed the issue (And what he wanted the filter to do).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1821) Filtering by SingleColumnValueFilter
bug
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Gray updated HBASE-1821:
---------------------------------
Attachment: HBASE-1821-v3.patch
Adds another optimization. Once we find the column, if we didn't match it, all keys should return NEXT_ROW.
Updated testSCVF unit test to reflect and test new behavior.
> Filtering by SingleColumnValueFilter bug
> ----------------------------------------
>
> Key: HBASE-1821
> URL: https://issues.apache.org/jira/browse/HBASE-1821
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: Jonathan Gray
> Fix For: 0.20.1, 0.21.0
>
> Attachments: HBASE-1821-v2.patch, HBASE-1821-v3.patch, HBASE-1821.patch
>
>
> See up in hbase-user mailing list under title 'Filtering by SingleColumnValueFilter'. Adrian describes roughly how he fixed the issue (And what he wanted the filter to do).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.