You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Niels Basjes <Ni...@basjes.nl> on 2014/02/03 14:27:33 UTC
Trouble writing custom filter for use in FilterList
Hi,
I'm trying to write a custom filter that is to be used (in my use case) in
a FilterList.
Because I ran into something I do not understand I reduced the code to the
absolute minimal and posted it here:
https://github.com/nielsbasjes/HBase-filter-problem
What I now have (just to show my problem) is a filter (classname
= AlwaysNextColFilter) that only implements
@Override
public ReturnCode filterKeyValue(KeyValue v) {
return ReturnCode.NEXT_COL;
}
The expected behavior is that this filter should indicate to everything
that is offered that it should be filtered out.
I then:
utility = new HBaseTestingUtility();
utility.startMiniCluster();
and I put some rows in there.
Put put = new Put("Row AA".getBytes());
put.add(colFamBytes, "Col A".getBytes(), "Foo".getBytes());
table.put(put);
put = new Put("Row BB".getBytes());
put.add(colFamBytes, "Col B".getBytes(), "FooFoo".getBytes());
table.put(put);
put = new Put("Row CC".getBytes());
put.add(colFamBytes, "Col C".getBytes(), "Bar".getBytes());
table.put(put);
put = new Put("Row DD".getBytes());
put.add(colFamBytes, "Col D".getBytes(), "BarBar".getBytes());
table.put(put);
Now I create a scan (to scan the entire table ... of 4 rows) and I set the
filters.
1) With this I get an empty result set. * (= Good/As I expect it)*
s.setFilter(new *AlwaysNextColFilter()*);
2) With this I get an empty result set. * (= Good/As I expect it)*
FilterList flist = new
FilterList(FilterList.Operator.MUST_PASS_ONE);
flist.addFilter(new *AlwaysNextColFilter()*);
s.setFilter(flist);
3) With this I get only the rows starting with "Row B". * (= Good/As I
expect it)*
FilterList flist = new
FilterList(FilterList.Operator.MUST_PASS_ONE);
flist.addFilter(new *PrefixFilter("Row B".getBytes())*);
s.setFilter(flist);
4) With this I get only the rows starting with "Row A" and "Row B". *(=
NOT as I expect it)*
FilterList flist = new
FilterList(FilterList.Operator.MUST_PASS_ONE);
flist.addFilter(new *AlwaysNextColFilter()*);
flist.addFilter(new *PrefixFilter("Row B".getBytes())*);
s.setFilter(flist);
In 4) I expected to get ONLY the rows starting with "Row B" because these
are the only ones that "PASS" at least one of the provided filters.
Did I misunderstand the way this should work ... or is this bug ( in
FilterList? )?
--
Best regards / Met vriendelijke groeten,
Niels Basjes
Re: Trouble writing custom filter for use in FilterList
Posted by Ted Yu <yu...@gmail.com>.
I logged HBASE-10485 for this issue and attached patch for 0.94 there.
Cheers
On Fri, Feb 7, 2014 at 1:07 PM, Ted Yu <yu...@gmail.com> wrote:
> I got the following test failure from the unit test published at
> https://github.com/nielsbasjes/HBase-filter-problem/ :
>
> Failed tests:
> testFilterListTwoFiltersMustPassOne(org.apache.hadoop.hbase.filter.TestFilterListAdditional):
> The rowid of this row does not start with "Row C": keyvalues={Row AA/F:Col
> A1/1391806814810/Put/vlen=3/ts=0, Row AA/F:Col
> A2/1391806814810/Put/vlen=3/ts=0}
>
> Let me debug.
>
> FYI
>
>
> On Mon, Feb 3, 2014 at 5:56 AM, Niels Basjes <Ni...@basjes.nl> wrote:
>
>> Additional info:
>> If I replace the NEXT_COL in the filter into either SKIP or NEXT_ROW the
>> result remains the same.
>>
>>
>> On Mon, Feb 3, 2014 at 2:27 PM, Niels Basjes <Ni...@basjes.nl> wrote:
>>
>> > Hi,
>> >
>> > I'm trying to write a custom filter that is to be used (in my use case)
>> in
>> > a FilterList.
>> > Because I ran into something I do not understand I reduced the code to
>> the
>> > absolute minimal and posted it here:
>> > https://github.com/nielsbasjes/HBase-filter-problem
>> >
>> > What I now have (just to show my problem) is a filter (classname
>> > = AlwaysNextColFilter) that only implements
>> >
>> > @Override
>> > public ReturnCode filterKeyValue(KeyValue v) {
>> > return ReturnCode.NEXT_COL;
>> > }
>> >
>> >
>> > The expected behavior is that this filter should indicate to everything
>> > that is offered that it should be filtered out.
>> >
>> > I then:
>> > utility = new HBaseTestingUtility();
>> > utility.startMiniCluster();
>> >
>> > and I put some rows in there.
>> >
>> > Put put = new Put("Row AA".getBytes());
>> > put.add(colFamBytes, "Col A".getBytes(), "Foo".getBytes());
>> > table.put(put);
>> >
>> > put = new Put("Row BB".getBytes());
>> > put.add(colFamBytes, "Col B".getBytes(), "FooFoo".getBytes());
>> > table.put(put);
>> >
>> > put = new Put("Row CC".getBytes());
>> > put.add(colFamBytes, "Col C".getBytes(), "Bar".getBytes());
>> > table.put(put);
>> >
>> > put = new Put("Row DD".getBytes());
>> > put.add(colFamBytes, "Col D".getBytes(), "BarBar".getBytes());
>> > table.put(put);
>> >
>> > Now I create a scan (to scan the entire table ... of 4 rows) and I set
>> the
>> > filters.
>> >
>> > 1) With this I get an empty result set. * (= Good/As I expect it)*
>> > s.setFilter(new *AlwaysNextColFilter()*);
>> >
>> > 2) With this I get an empty result set. * (= Good/As I expect it)*
>> > FilterList flist = new
>> > FilterList(FilterList.Operator.MUST_PASS_ONE);
>> > flist.addFilter(new *AlwaysNextColFilter()*);
>> > s.setFilter(flist);
>> >
>> > 3) With this I get only the rows starting with "Row B". * (= Good/As I
>> > expect it)*
>> > FilterList flist = new
>> > FilterList(FilterList.Operator.MUST_PASS_ONE);
>> > flist.addFilter(new *PrefixFilter("Row B".getBytes())*);
>> > s.setFilter(flist);
>> >
>> > 4) With this I get only the rows starting with "Row A" and "Row B". *(=
>> > NOT as I expect it)*
>> > FilterList flist = new
>> > FilterList(FilterList.Operator.MUST_PASS_ONE);
>> > flist.addFilter(new *AlwaysNextColFilter()*);
>> > flist.addFilter(new *PrefixFilter("Row B".getBytes())*);
>> > s.setFilter(flist);
>> >
>> > In 4) I expected to get ONLY the rows starting with "Row B" because
>> these
>> > are the only ones that "PASS" at least one of the provided filters.
>> >
>> > Did I misunderstand the way this should work ... or is this bug ( in
>> > FilterList? )?
>> >
>> > --
>> > Best regards / Met vriendelijke groeten,
>> >
>> > Niels Basjes
>> >
>>
>>
>>
>> --
>> Best regards / Met vriendelijke groeten,
>>
>> Niels Basjes
>>
>
>
Re: Trouble writing custom filter for use in FilterList
Posted by Ted Yu <yu...@gmail.com>.
I got the following test failure from the unit test published at
https://github.com/nielsbasjes/HBase-filter-problem/ :
Failed tests:
testFilterListTwoFiltersMustPassOne(org.apache.hadoop.hbase.filter.TestFilterListAdditional):
The rowid of this row does not start with "Row C": keyvalues={Row AA/F:Col
A1/1391806814810/Put/vlen=3/ts=0, Row AA/F:Col
A2/1391806814810/Put/vlen=3/ts=0}
Let me debug.
FYI
On Mon, Feb 3, 2014 at 5:56 AM, Niels Basjes <Ni...@basjes.nl> wrote:
> Additional info:
> If I replace the NEXT_COL in the filter into either SKIP or NEXT_ROW the
> result remains the same.
>
>
> On Mon, Feb 3, 2014 at 2:27 PM, Niels Basjes <Ni...@basjes.nl> wrote:
>
> > Hi,
> >
> > I'm trying to write a custom filter that is to be used (in my use case)
> in
> > a FilterList.
> > Because I ran into something I do not understand I reduced the code to
> the
> > absolute minimal and posted it here:
> > https://github.com/nielsbasjes/HBase-filter-problem
> >
> > What I now have (just to show my problem) is a filter (classname
> > = AlwaysNextColFilter) that only implements
> >
> > @Override
> > public ReturnCode filterKeyValue(KeyValue v) {
> > return ReturnCode.NEXT_COL;
> > }
> >
> >
> > The expected behavior is that this filter should indicate to everything
> > that is offered that it should be filtered out.
> >
> > I then:
> > utility = new HBaseTestingUtility();
> > utility.startMiniCluster();
> >
> > and I put some rows in there.
> >
> > Put put = new Put("Row AA".getBytes());
> > put.add(colFamBytes, "Col A".getBytes(), "Foo".getBytes());
> > table.put(put);
> >
> > put = new Put("Row BB".getBytes());
> > put.add(colFamBytes, "Col B".getBytes(), "FooFoo".getBytes());
> > table.put(put);
> >
> > put = new Put("Row CC".getBytes());
> > put.add(colFamBytes, "Col C".getBytes(), "Bar".getBytes());
> > table.put(put);
> >
> > put = new Put("Row DD".getBytes());
> > put.add(colFamBytes, "Col D".getBytes(), "BarBar".getBytes());
> > table.put(put);
> >
> > Now I create a scan (to scan the entire table ... of 4 rows) and I set
> the
> > filters.
> >
> > 1) With this I get an empty result set. * (= Good/As I expect it)*
> > s.setFilter(new *AlwaysNextColFilter()*);
> >
> > 2) With this I get an empty result set. * (= Good/As I expect it)*
> > FilterList flist = new
> > FilterList(FilterList.Operator.MUST_PASS_ONE);
> > flist.addFilter(new *AlwaysNextColFilter()*);
> > s.setFilter(flist);
> >
> > 3) With this I get only the rows starting with "Row B". * (= Good/As I
> > expect it)*
> > FilterList flist = new
> > FilterList(FilterList.Operator.MUST_PASS_ONE);
> > flist.addFilter(new *PrefixFilter("Row B".getBytes())*);
> > s.setFilter(flist);
> >
> > 4) With this I get only the rows starting with "Row A" and "Row B". *(=
> > NOT as I expect it)*
> > FilterList flist = new
> > FilterList(FilterList.Operator.MUST_PASS_ONE);
> > flist.addFilter(new *AlwaysNextColFilter()*);
> > flist.addFilter(new *PrefixFilter("Row B".getBytes())*);
> > s.setFilter(flist);
> >
> > In 4) I expected to get ONLY the rows starting with "Row B" because these
> > are the only ones that "PASS" at least one of the provided filters.
> >
> > Did I misunderstand the way this should work ... or is this bug ( in
> > FilterList? )?
> >
> > --
> > Best regards / Met vriendelijke groeten,
> >
> > Niels Basjes
> >
>
>
>
> --
> Best regards / Met vriendelijke groeten,
>
> Niels Basjes
>
Re: Trouble writing custom filter for use in FilterList
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
Super. Thanks Ted.
2014-02-24 9:28 GMT-05:00 Ted Yu <yu...@gmail.com>:
> Niels' test is an end to end test.
> The issue is covered by a new test in TestFilterList.
>
> Cheers
>
> On Feb 24, 2014, at 6:25 AM, Jean-Marc Spaggiari <je...@spaggiari.org>
> wrote:
>
> > Hi Ted,
> >
> > Any reason why this was not failing of Jenkins too? Can we add
> > https://github.com/nielsbasjes/HBase-filter-problem/ to the test suite?
> >
> > JM
> >
> >
> > 2014-02-19 6:51 GMT-05:00 Ted Yu <yu...@gmail.com>:
> >
> >> Hi, Niels:
> >> This issue has been fixed in HBASE-10485 and would be included in the
> >> upcoming 0.94.17 release.
> >>
> >> Cheers
> >>
> >> On Feb 3, 2014, at 7:56 AM, Niels Basjes <Ni...@basjes.nl> wrote:
> >>
> >>> Additional info:
> >>> If I replace the NEXT_COL in the filter into either SKIP or NEXT_ROW
> the
> >>> result remains the same.
> >>>
> >>>
> >>> On Mon, Feb 3, 2014 at 2:27 PM, Niels Basjes <Ni...@basjes.nl> wrote:
> >>>
> >>>> Hi,
> >>>>
> >>>> I'm trying to write a custom filter that is to be used (in my use
> case)
> >> in
> >>>> a FilterList.
> >>>> Because I ran into something I do not understand I reduced the code to
> >> the
> >>>> absolute minimal and posted it here:
> >>>> https://github.com/nielsbasjes/HBase-filter-problem
> >>>>
> >>>> What I now have (just to show my problem) is a filter (classname
> >>>> = AlwaysNextColFilter) that only implements
> >>>>
> >>>> @Override
> >>>> public ReturnCode filterKeyValue(KeyValue v) {
> >>>> return ReturnCode.NEXT_COL;
> >>>> }
> >>>>
> >>>>
> >>>> The expected behavior is that this filter should indicate to
> everything
> >>>> that is offered that it should be filtered out.
> >>>>
> >>>> I then:
> >>>> utility = new HBaseTestingUtility();
> >>>> utility.startMiniCluster();
> >>>>
> >>>> and I put some rows in there.
> >>>>
> >>>> Put put = new Put("Row AA".getBytes());
> >>>> put.add(colFamBytes, "Col A".getBytes(), "Foo".getBytes());
> >>>> table.put(put);
> >>>>
> >>>> put = new Put("Row BB".getBytes());
> >>>> put.add(colFamBytes, "Col B".getBytes(), "FooFoo".getBytes());
> >>>> table.put(put);
> >>>>
> >>>> put = new Put("Row CC".getBytes());
> >>>> put.add(colFamBytes, "Col C".getBytes(), "Bar".getBytes());
> >>>> table.put(put);
> >>>>
> >>>> put = new Put("Row DD".getBytes());
> >>>> put.add(colFamBytes, "Col D".getBytes(), "BarBar".getBytes());
> >>>> table.put(put);
> >>>>
> >>>> Now I create a scan (to scan the entire table ... of 4 rows) and I set
> >> the
> >>>> filters.
> >>>>
> >>>> 1) With this I get an empty result set. * (= Good/As I expect it)*
> >>>> s.setFilter(new *AlwaysNextColFilter()*);
> >>>>
> >>>> 2) With this I get an empty result set. * (= Good/As I expect it)*
> >>>> FilterList flist = new
> >>>> FilterList(FilterList.Operator.MUST_PASS_ONE);
> >>>> flist.addFilter(new *AlwaysNextColFilter()*);
> >>>> s.setFilter(flist);
> >>>>
> >>>> 3) With this I get only the rows starting with "Row B". * (= Good/As I
> >>>> expect it)*
> >>>> FilterList flist = new
> >>>> FilterList(FilterList.Operator.MUST_PASS_ONE);
> >>>> flist.addFilter(new *PrefixFilter("Row B".getBytes())*);
> >>>> s.setFilter(flist);
> >>>>
> >>>> 4) With this I get only the rows starting with "Row A" and "Row B".
> *(=
> >>>> NOT as I expect it)*
> >>>> FilterList flist = new
> >>>> FilterList(FilterList.Operator.MUST_PASS_ONE);
> >>>> flist.addFilter(new *AlwaysNextColFilter()*);
> >>>> flist.addFilter(new *PrefixFilter("Row B".getBytes())*);
> >>>> s.setFilter(flist);
> >>>>
> >>>> In 4) I expected to get ONLY the rows starting with "Row B" because
> >> these
> >>>> are the only ones that "PASS" at least one of the provided filters.
> >>>>
> >>>> Did I misunderstand the way this should work ... or is this bug ( in
> >>>> FilterList? )?
> >>>>
> >>>> --
> >>>> Best regards / Met vriendelijke groeten,
> >>>>
> >>>> Niels Basjes
> >>>
> >>>
> >>>
> >>> --
> >>> Best regards / Met vriendelijke groeten,
> >>>
> >>> Niels Basjes
> >>
>
Re: Trouble writing custom filter for use in FilterList
Posted by Ted Yu <yu...@gmail.com>.
Niels' test is an end to end test.
The issue is covered by a new test in TestFilterList.
Cheers
On Feb 24, 2014, at 6:25 AM, Jean-Marc Spaggiari <je...@spaggiari.org> wrote:
> Hi Ted,
>
> Any reason why this was not failing of Jenkins too? Can we add
> https://github.com/nielsbasjes/HBase-filter-problem/ to the test suite?
>
> JM
>
>
> 2014-02-19 6:51 GMT-05:00 Ted Yu <yu...@gmail.com>:
>
>> Hi, Niels:
>> This issue has been fixed in HBASE-10485 and would be included in the
>> upcoming 0.94.17 release.
>>
>> Cheers
>>
>> On Feb 3, 2014, at 7:56 AM, Niels Basjes <Ni...@basjes.nl> wrote:
>>
>>> Additional info:
>>> If I replace the NEXT_COL in the filter into either SKIP or NEXT_ROW the
>>> result remains the same.
>>>
>>>
>>> On Mon, Feb 3, 2014 at 2:27 PM, Niels Basjes <Ni...@basjes.nl> wrote:
>>>
>>>> Hi,
>>>>
>>>> I'm trying to write a custom filter that is to be used (in my use case)
>> in
>>>> a FilterList.
>>>> Because I ran into something I do not understand I reduced the code to
>> the
>>>> absolute minimal and posted it here:
>>>> https://github.com/nielsbasjes/HBase-filter-problem
>>>>
>>>> What I now have (just to show my problem) is a filter (classname
>>>> = AlwaysNextColFilter) that only implements
>>>>
>>>> @Override
>>>> public ReturnCode filterKeyValue(KeyValue v) {
>>>> return ReturnCode.NEXT_COL;
>>>> }
>>>>
>>>>
>>>> The expected behavior is that this filter should indicate to everything
>>>> that is offered that it should be filtered out.
>>>>
>>>> I then:
>>>> utility = new HBaseTestingUtility();
>>>> utility.startMiniCluster();
>>>>
>>>> and I put some rows in there.
>>>>
>>>> Put put = new Put("Row AA".getBytes());
>>>> put.add(colFamBytes, "Col A".getBytes(), "Foo".getBytes());
>>>> table.put(put);
>>>>
>>>> put = new Put("Row BB".getBytes());
>>>> put.add(colFamBytes, "Col B".getBytes(), "FooFoo".getBytes());
>>>> table.put(put);
>>>>
>>>> put = new Put("Row CC".getBytes());
>>>> put.add(colFamBytes, "Col C".getBytes(), "Bar".getBytes());
>>>> table.put(put);
>>>>
>>>> put = new Put("Row DD".getBytes());
>>>> put.add(colFamBytes, "Col D".getBytes(), "BarBar".getBytes());
>>>> table.put(put);
>>>>
>>>> Now I create a scan (to scan the entire table ... of 4 rows) and I set
>> the
>>>> filters.
>>>>
>>>> 1) With this I get an empty result set. * (= Good/As I expect it)*
>>>> s.setFilter(new *AlwaysNextColFilter()*);
>>>>
>>>> 2) With this I get an empty result set. * (= Good/As I expect it)*
>>>> FilterList flist = new
>>>> FilterList(FilterList.Operator.MUST_PASS_ONE);
>>>> flist.addFilter(new *AlwaysNextColFilter()*);
>>>> s.setFilter(flist);
>>>>
>>>> 3) With this I get only the rows starting with "Row B". * (= Good/As I
>>>> expect it)*
>>>> FilterList flist = new
>>>> FilterList(FilterList.Operator.MUST_PASS_ONE);
>>>> flist.addFilter(new *PrefixFilter("Row B".getBytes())*);
>>>> s.setFilter(flist);
>>>>
>>>> 4) With this I get only the rows starting with "Row A" and "Row B". *(=
>>>> NOT as I expect it)*
>>>> FilterList flist = new
>>>> FilterList(FilterList.Operator.MUST_PASS_ONE);
>>>> flist.addFilter(new *AlwaysNextColFilter()*);
>>>> flist.addFilter(new *PrefixFilter("Row B".getBytes())*);
>>>> s.setFilter(flist);
>>>>
>>>> In 4) I expected to get ONLY the rows starting with "Row B" because
>> these
>>>> are the only ones that "PASS" at least one of the provided filters.
>>>>
>>>> Did I misunderstand the way this should work ... or is this bug ( in
>>>> FilterList? )?
>>>>
>>>> --
>>>> Best regards / Met vriendelijke groeten,
>>>>
>>>> Niels Basjes
>>>
>>>
>>>
>>> --
>>> Best regards / Met vriendelijke groeten,
>>>
>>> Niels Basjes
>>
Re: Trouble writing custom filter for use in FilterList
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
Hi Ted,
Any reason why this was not failing of Jenkins too? Can we add
https://github.com/nielsbasjes/HBase-filter-problem/ to the test suite?
JM
2014-02-19 6:51 GMT-05:00 Ted Yu <yu...@gmail.com>:
> Hi, Niels:
> This issue has been fixed in HBASE-10485 and would be included in the
> upcoming 0.94.17 release.
>
> Cheers
>
> On Feb 3, 2014, at 7:56 AM, Niels Basjes <Ni...@basjes.nl> wrote:
>
> > Additional info:
> > If I replace the NEXT_COL in the filter into either SKIP or NEXT_ROW the
> > result remains the same.
> >
> >
> > On Mon, Feb 3, 2014 at 2:27 PM, Niels Basjes <Ni...@basjes.nl> wrote:
> >
> >> Hi,
> >>
> >> I'm trying to write a custom filter that is to be used (in my use case)
> in
> >> a FilterList.
> >> Because I ran into something I do not understand I reduced the code to
> the
> >> absolute minimal and posted it here:
> >> https://github.com/nielsbasjes/HBase-filter-problem
> >>
> >> What I now have (just to show my problem) is a filter (classname
> >> = AlwaysNextColFilter) that only implements
> >>
> >> @Override
> >> public ReturnCode filterKeyValue(KeyValue v) {
> >> return ReturnCode.NEXT_COL;
> >> }
> >>
> >>
> >> The expected behavior is that this filter should indicate to everything
> >> that is offered that it should be filtered out.
> >>
> >> I then:
> >> utility = new HBaseTestingUtility();
> >> utility.startMiniCluster();
> >>
> >> and I put some rows in there.
> >>
> >> Put put = new Put("Row AA".getBytes());
> >> put.add(colFamBytes, "Col A".getBytes(), "Foo".getBytes());
> >> table.put(put);
> >>
> >> put = new Put("Row BB".getBytes());
> >> put.add(colFamBytes, "Col B".getBytes(), "FooFoo".getBytes());
> >> table.put(put);
> >>
> >> put = new Put("Row CC".getBytes());
> >> put.add(colFamBytes, "Col C".getBytes(), "Bar".getBytes());
> >> table.put(put);
> >>
> >> put = new Put("Row DD".getBytes());
> >> put.add(colFamBytes, "Col D".getBytes(), "BarBar".getBytes());
> >> table.put(put);
> >>
> >> Now I create a scan (to scan the entire table ... of 4 rows) and I set
> the
> >> filters.
> >>
> >> 1) With this I get an empty result set. * (= Good/As I expect it)*
> >> s.setFilter(new *AlwaysNextColFilter()*);
> >>
> >> 2) With this I get an empty result set. * (= Good/As I expect it)*
> >> FilterList flist = new
> >> FilterList(FilterList.Operator.MUST_PASS_ONE);
> >> flist.addFilter(new *AlwaysNextColFilter()*);
> >> s.setFilter(flist);
> >>
> >> 3) With this I get only the rows starting with "Row B". * (= Good/As I
> >> expect it)*
> >> FilterList flist = new
> >> FilterList(FilterList.Operator.MUST_PASS_ONE);
> >> flist.addFilter(new *PrefixFilter("Row B".getBytes())*);
> >> s.setFilter(flist);
> >>
> >> 4) With this I get only the rows starting with "Row A" and "Row B". *(=
> >> NOT as I expect it)*
> >> FilterList flist = new
> >> FilterList(FilterList.Operator.MUST_PASS_ONE);
> >> flist.addFilter(new *AlwaysNextColFilter()*);
> >> flist.addFilter(new *PrefixFilter("Row B".getBytes())*);
> >> s.setFilter(flist);
> >>
> >> In 4) I expected to get ONLY the rows starting with "Row B" because
> these
> >> are the only ones that "PASS" at least one of the provided filters.
> >>
> >> Did I misunderstand the way this should work ... or is this bug ( in
> >> FilterList? )?
> >>
> >> --
> >> Best regards / Met vriendelijke groeten,
> >>
> >> Niels Basjes
> >
> >
> >
> > --
> > Best regards / Met vriendelijke groeten,
> >
> > Niels Basjes
>
Re: Trouble writing custom filter for use in FilterList
Posted by Ted Yu <yu...@gmail.com>.
Hi, Niels:
This issue has been fixed in HBASE-10485 and would be included in the upcoming 0.94.17 release.
Cheers
On Feb 3, 2014, at 7:56 AM, Niels Basjes <Ni...@basjes.nl> wrote:
> Additional info:
> If I replace the NEXT_COL in the filter into either SKIP or NEXT_ROW the
> result remains the same.
>
>
> On Mon, Feb 3, 2014 at 2:27 PM, Niels Basjes <Ni...@basjes.nl> wrote:
>
>> Hi,
>>
>> I'm trying to write a custom filter that is to be used (in my use case) in
>> a FilterList.
>> Because I ran into something I do not understand I reduced the code to the
>> absolute minimal and posted it here:
>> https://github.com/nielsbasjes/HBase-filter-problem
>>
>> What I now have (just to show my problem) is a filter (classname
>> = AlwaysNextColFilter) that only implements
>>
>> @Override
>> public ReturnCode filterKeyValue(KeyValue v) {
>> return ReturnCode.NEXT_COL;
>> }
>>
>>
>> The expected behavior is that this filter should indicate to everything
>> that is offered that it should be filtered out.
>>
>> I then:
>> utility = new HBaseTestingUtility();
>> utility.startMiniCluster();
>>
>> and I put some rows in there.
>>
>> Put put = new Put("Row AA".getBytes());
>> put.add(colFamBytes, "Col A".getBytes(), "Foo".getBytes());
>> table.put(put);
>>
>> put = new Put("Row BB".getBytes());
>> put.add(colFamBytes, "Col B".getBytes(), "FooFoo".getBytes());
>> table.put(put);
>>
>> put = new Put("Row CC".getBytes());
>> put.add(colFamBytes, "Col C".getBytes(), "Bar".getBytes());
>> table.put(put);
>>
>> put = new Put("Row DD".getBytes());
>> put.add(colFamBytes, "Col D".getBytes(), "BarBar".getBytes());
>> table.put(put);
>>
>> Now I create a scan (to scan the entire table ... of 4 rows) and I set the
>> filters.
>>
>> 1) With this I get an empty result set. * (= Good/As I expect it)*
>> s.setFilter(new *AlwaysNextColFilter()*);
>>
>> 2) With this I get an empty result set. * (= Good/As I expect it)*
>> FilterList flist = new
>> FilterList(FilterList.Operator.MUST_PASS_ONE);
>> flist.addFilter(new *AlwaysNextColFilter()*);
>> s.setFilter(flist);
>>
>> 3) With this I get only the rows starting with "Row B". * (= Good/As I
>> expect it)*
>> FilterList flist = new
>> FilterList(FilterList.Operator.MUST_PASS_ONE);
>> flist.addFilter(new *PrefixFilter("Row B".getBytes())*);
>> s.setFilter(flist);
>>
>> 4) With this I get only the rows starting with "Row A" and "Row B". *(=
>> NOT as I expect it)*
>> FilterList flist = new
>> FilterList(FilterList.Operator.MUST_PASS_ONE);
>> flist.addFilter(new *AlwaysNextColFilter()*);
>> flist.addFilter(new *PrefixFilter("Row B".getBytes())*);
>> s.setFilter(flist);
>>
>> In 4) I expected to get ONLY the rows starting with "Row B" because these
>> are the only ones that "PASS" at least one of the provided filters.
>>
>> Did I misunderstand the way this should work ... or is this bug ( in
>> FilterList? )?
>>
>> --
>> Best regards / Met vriendelijke groeten,
>>
>> Niels Basjes
>
>
>
> --
> Best regards / Met vriendelijke groeten,
>
> Niels Basjes
Re: Trouble writing custom filter for use in FilterList
Posted by Niels Basjes <Ni...@basjes.nl>.
Additional info:
If I replace the NEXT_COL in the filter into either SKIP or NEXT_ROW the
result remains the same.
On Mon, Feb 3, 2014 at 2:27 PM, Niels Basjes <Ni...@basjes.nl> wrote:
> Hi,
>
> I'm trying to write a custom filter that is to be used (in my use case) in
> a FilterList.
> Because I ran into something I do not understand I reduced the code to the
> absolute minimal and posted it here:
> https://github.com/nielsbasjes/HBase-filter-problem
>
> What I now have (just to show my problem) is a filter (classname
> = AlwaysNextColFilter) that only implements
>
> @Override
> public ReturnCode filterKeyValue(KeyValue v) {
> return ReturnCode.NEXT_COL;
> }
>
>
> The expected behavior is that this filter should indicate to everything
> that is offered that it should be filtered out.
>
> I then:
> utility = new HBaseTestingUtility();
> utility.startMiniCluster();
>
> and I put some rows in there.
>
> Put put = new Put("Row AA".getBytes());
> put.add(colFamBytes, "Col A".getBytes(), "Foo".getBytes());
> table.put(put);
>
> put = new Put("Row BB".getBytes());
> put.add(colFamBytes, "Col B".getBytes(), "FooFoo".getBytes());
> table.put(put);
>
> put = new Put("Row CC".getBytes());
> put.add(colFamBytes, "Col C".getBytes(), "Bar".getBytes());
> table.put(put);
>
> put = new Put("Row DD".getBytes());
> put.add(colFamBytes, "Col D".getBytes(), "BarBar".getBytes());
> table.put(put);
>
> Now I create a scan (to scan the entire table ... of 4 rows) and I set the
> filters.
>
> 1) With this I get an empty result set. * (= Good/As I expect it)*
> s.setFilter(new *AlwaysNextColFilter()*);
>
> 2) With this I get an empty result set. * (= Good/As I expect it)*
> FilterList flist = new
> FilterList(FilterList.Operator.MUST_PASS_ONE);
> flist.addFilter(new *AlwaysNextColFilter()*);
> s.setFilter(flist);
>
> 3) With this I get only the rows starting with "Row B". * (= Good/As I
> expect it)*
> FilterList flist = new
> FilterList(FilterList.Operator.MUST_PASS_ONE);
> flist.addFilter(new *PrefixFilter("Row B".getBytes())*);
> s.setFilter(flist);
>
> 4) With this I get only the rows starting with "Row A" and "Row B". *(=
> NOT as I expect it)*
> FilterList flist = new
> FilterList(FilterList.Operator.MUST_PASS_ONE);
> flist.addFilter(new *AlwaysNextColFilter()*);
> flist.addFilter(new *PrefixFilter("Row B".getBytes())*);
> s.setFilter(flist);
>
> In 4) I expected to get ONLY the rows starting with "Row B" because these
> are the only ones that "PASS" at least one of the provided filters.
>
> Did I misunderstand the way this should work ... or is this bug ( in
> FilterList? )?
>
> --
> Best regards / Met vriendelijke groeten,
>
> Niels Basjes
>
--
Best regards / Met vriendelijke groeten,
Niels Basjes