You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Premal Shah <pr...@gmail.com> on 2013/07/25 08:13:30 UTC
FuzzyRowFilter with ColumnRangeFilter
Hi,
I was playing around with using FuzzyRowFilter for my use case which looks
like this
rk - 18 bytes (2 bytes + 8 bytes + 8 bytes)
bytes 2-9 are fuzzy
the fuzzy mask looks like this (0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0,
0, 0, 0 ,0)
the start and stop key control the scan range
Running a scanner with just the FuzzyRowFilter works fine. However, I need
to limit the columns returned. So I have a ColumnRangeFilter too. Both the
filters are wrapped in a FilterList
The combination of both, returns no rows from hbase with MUST_PASS_ALL set.
Hbase seems to be aborting when the row does not match Fuzzy
If MUST_PASS_ONE is set, then it returns the columns from the rows that
don't pass Fuzzy.
How do you go about using the FilterList with both filters and return the
required rows only?
--
Regards,
Premal Shah.
Re: FuzzyRowFilter with ColumnRangeFilter
Posted by Ted Yu <yu...@gmail.com>.
In HBASE-6618, FuzzyRowFilter with range support is added.
Does that serve your use case so that you don't need to use FilterList ?
Cheers
On Jul 24, 2013, at 11:13 PM, Premal Shah <pr...@gmail.com> wrote:
> Hi,
> I was playing around with using FuzzyRowFilter for my use case which looks
> like this
>
> rk - 18 bytes (2 bytes + 8 bytes + 8 bytes)
> bytes 2-9 are fuzzy
> the fuzzy mask looks like this (0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0,
> 0, 0, 0 ,0)
> the start and stop key control the scan range
>
> Running a scanner with just the FuzzyRowFilter works fine. However, I need
> to limit the columns returned. So I have a ColumnRangeFilter too. Both the
> filters are wrapped in a FilterList
>
> The combination of both, returns no rows from hbase with MUST_PASS_ALL set.
> Hbase seems to be aborting when the row does not match Fuzzy
> If MUST_PASS_ONE is set, then it returns the columns from the rows that
> don't pass Fuzzy.
>
> How do you go about using the FilterList with both filters and return the
> required rows only?
>
> --
> Regards,
> Premal Shah.
Re: FuzzyRowFilter with ColumnRangeFilter
Posted by Premal Shah <pr...@gmail.com>.
The umask for set to 0002 instead of 0022. After updating it, the tests ran
fine.
Viral, who has opened another thread with title "FilterList: possible bug
in getNextKeyHint" will submit the test and the patch
Thanx.
On Mon, Jul 29, 2013 at 10:40 AM, Ted Yu <yu...@gmail.com> wrote:
> To run a single test, you should use the following command:
>
> mvn test -PrunAllTests -DfailIfNoTests=false -Dtest=xxx
>
> I ran TestColumnRangeFilter using tip of 0.94 code base and it passed.
> Did you use tip of 0.94 ?
>
> Cheers
>
> On Mon, Jul 29, 2013 at 10:32 AM, Premal Shah <premal.j.shah@gmail.com
> >wrote:
>
> > Questions about unit tests:
> > 1) I ran this to execute all tests in the filter package - mvn test
> > -Dtest=org.apache.hadoop.hbase.filter.*
> > The ColumnRangeFilter test fails with this error
> >
> >
> >
> -------------------------------------------------------------------------------
> > Test set: org.apache.hadoop.hbase.filter.TestColumnRangeFilter
> >
> >
> -------------------------------------------------------------------------------
> > Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 3.136 sec
> > <<< FAILURE!
> > org.apache.hadoop.hbase.filter.TestColumnRangeFilter Time elapsed: 0 sec
> > <<< ERROR!
> > java.lang.NullPointerException
> > at
> >
> >
> org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:422)
> > at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:280)
> > at
> >
> >
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster(HBaseTestingUtility.java:449)
> > at
> >
> >
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:617)
> > at
> >
> >
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:573)
> > at
> >
> >
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:542)
> > at
> >
> >
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter.setUpBeforeClass(TestColumnRangeFilter.java:135)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> >
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
> > at
> >
> >
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> > at
> >
> >
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
> > at
> >
> >
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27)
> > at
> >
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
> > at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
> > at org.junit.runners.Suite.runChild(Suite.java:128)
> > at org.junit.runners.Suite.runChild(Suite.java:24)
> > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
> > at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
> > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> > at
> >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > at
> >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > at java.lang.Thread.run(Thread.java:662)
> >
> > Is this a permissions issue?
> >
> > 2) I tried to run a single test like this *mvn test
> > -Dtest=org.apache.hadoop.hbase.filter.TestPrefixFilter* and this *mvn
> test
> > -Dtest=TestPrefixFilter*
> >
> > But that fails with this error
> >
> > Failed to execute goal
> > org.apache.maven.plugins:maven-surefire-plugin:2.12-TRUNK-HBASE-2:test
> > (secondPartTestsExecution) on project hbase: No tests were executed!
> (Set
> > -DfailIfNoTests=false to ignore this error.) -> [Help 1]
> >
> > What am I doing wrong here?
> >
> >
> >
> > On Thu, Jul 25, 2013 at 1:54 PM, Premal Shah <premal.j.shah@gmail.com
> > >wrote:
> >
> > > Hi Ted,
> > > I'm using 0.94.6.
> > >
> > > I'll setup a unix test.
> > >
> > >
> > > On Thu, Jul 25, 2013 at 1:50 AM, Ted Yu <yu...@gmail.com> wrote:
> > >
> > >> What HBase release are you using ?
> > >>
> > >> Can you put the scenario below in a unit test ?
> > >>
> > >> Thanks
> > >>
> > >> On Jul 24, 2013, at 11:13 PM, Premal Shah <pr...@gmail.com>
> > >> wrote:
> > >>
> > >> > Hi,
> > >> > I was playing around with using FuzzyRowFilter for my use case which
> > >> looks
> > >> > like this
> > >> >
> > >> > rk - 18 bytes (2 bytes + 8 bytes + 8 bytes)
> > >> > bytes 2-9 are fuzzy
> > >> > the fuzzy mask looks like this (0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0,
> 0,
> > >> 0,
> > >> > 0, 0, 0 ,0)
> > >> > the start and stop key control the scan range
> > >> >
> > >> > Running a scanner with just the FuzzyRowFilter works fine. However,
> I
> > >> need
> > >> > to limit the columns returned. So I have a ColumnRangeFilter too.
> Both
> > >> the
> > >> > filters are wrapped in a FilterList
> > >> >
> > >> > The combination of both, returns no rows from hbase with
> MUST_PASS_ALL
> > >> set.
> > >> > Hbase seems to be aborting when the row does not match Fuzzy
> > >> > If MUST_PASS_ONE is set, then it returns the columns from the rows
> > that
> > >> > don't pass Fuzzy.
> > >> >
> > >> > How do you go about using the FilterList with both filters and
> return
> > >> the
> > >> > required rows only?
> > >> >
> > >> > --
> > >> > Regards,
> > >> > Premal Shah.
> > >>
> > >
> > >
> > >
> > > --
> > > Regards,
> > > Premal Shah.
> > >
> >
> >
> >
> > --
> > Regards,
> > Premal Shah.
> >
>
--
Regards,
Premal Shah.
Re: FuzzyRowFilter with ColumnRangeFilter
Posted by Ted Yu <yu...@gmail.com>.
To run a single test, you should use the following command:
mvn test -PrunAllTests -DfailIfNoTests=false -Dtest=xxx
I ran TestColumnRangeFilter using tip of 0.94 code base and it passed.
Did you use tip of 0.94 ?
Cheers
On Mon, Jul 29, 2013 at 10:32 AM, Premal Shah <pr...@gmail.com>wrote:
> Questions about unit tests:
> 1) I ran this to execute all tests in the filter package - mvn test
> -Dtest=org.apache.hadoop.hbase.filter.*
> The ColumnRangeFilter test fails with this error
>
>
> -------------------------------------------------------------------------------
> Test set: org.apache.hadoop.hbase.filter.TestColumnRangeFilter
>
> -------------------------------------------------------------------------------
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 3.136 sec
> <<< FAILURE!
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter Time elapsed: 0 sec
> <<< ERROR!
> java.lang.NullPointerException
> at
>
> org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:422)
> at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:280)
> at
>
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster(HBaseTestingUtility.java:449)
> at
>
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:617)
> at
>
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:573)
> at
>
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:542)
> at
>
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter.setUpBeforeClass(TestColumnRangeFilter.java:135)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
>
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
> at
>
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at
>
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
> at
>
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
> at org.junit.runners.Suite.runChild(Suite.java:128)
> at org.junit.runners.Suite.runChild(Suite.java:24)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> Is this a permissions issue?
>
> 2) I tried to run a single test like this *mvn test
> -Dtest=org.apache.hadoop.hbase.filter.TestPrefixFilter* and this *mvn test
> -Dtest=TestPrefixFilter*
>
> But that fails with this error
>
> Failed to execute goal
> org.apache.maven.plugins:maven-surefire-plugin:2.12-TRUNK-HBASE-2:test
> (secondPartTestsExecution) on project hbase: No tests were executed! (Set
> -DfailIfNoTests=false to ignore this error.) -> [Help 1]
>
> What am I doing wrong here?
>
>
>
> On Thu, Jul 25, 2013 at 1:54 PM, Premal Shah <premal.j.shah@gmail.com
> >wrote:
>
> > Hi Ted,
> > I'm using 0.94.6.
> >
> > I'll setup a unix test.
> >
> >
> > On Thu, Jul 25, 2013 at 1:50 AM, Ted Yu <yu...@gmail.com> wrote:
> >
> >> What HBase release are you using ?
> >>
> >> Can you put the scenario below in a unit test ?
> >>
> >> Thanks
> >>
> >> On Jul 24, 2013, at 11:13 PM, Premal Shah <pr...@gmail.com>
> >> wrote:
> >>
> >> > Hi,
> >> > I was playing around with using FuzzyRowFilter for my use case which
> >> looks
> >> > like this
> >> >
> >> > rk - 18 bytes (2 bytes + 8 bytes + 8 bytes)
> >> > bytes 2-9 are fuzzy
> >> > the fuzzy mask looks like this (0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0,
> >> 0,
> >> > 0, 0, 0 ,0)
> >> > the start and stop key control the scan range
> >> >
> >> > Running a scanner with just the FuzzyRowFilter works fine. However, I
> >> need
> >> > to limit the columns returned. So I have a ColumnRangeFilter too. Both
> >> the
> >> > filters are wrapped in a FilterList
> >> >
> >> > The combination of both, returns no rows from hbase with MUST_PASS_ALL
> >> set.
> >> > Hbase seems to be aborting when the row does not match Fuzzy
> >> > If MUST_PASS_ONE is set, then it returns the columns from the rows
> that
> >> > don't pass Fuzzy.
> >> >
> >> > How do you go about using the FilterList with both filters and return
> >> the
> >> > required rows only?
> >> >
> >> > --
> >> > Regards,
> >> > Premal Shah.
> >>
> >
> >
> >
> > --
> > Regards,
> > Premal Shah.
> >
>
>
>
> --
> Regards,
> Premal Shah.
>
Re: FuzzyRowFilter with ColumnRangeFilter
Posted by Premal Shah <pr...@gmail.com>.
Questions about unit tests:
1) I ran this to execute all tests in the filter package - mvn test
-Dtest=org.apache.hadoop.hbase.filter.*
The ColumnRangeFilter test fails with this error
-------------------------------------------------------------------------------
Test set: org.apache.hadoop.hbase.filter.TestColumnRangeFilter
-------------------------------------------------------------------------------
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 3.136 sec
<<< FAILURE!
org.apache.hadoop.hbase.filter.TestColumnRangeFilter Time elapsed: 0 sec
<<< ERROR!
java.lang.NullPointerException
at
org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:422)
at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:280)
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster(HBaseTestingUtility.java:449)
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:617)
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:573)
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:542)
at
org.apache.hadoop.hbase.filter.TestColumnRangeFilter.setUpBeforeClass(TestColumnRangeFilter.java:135)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at org.junit.runners.Suite.runChild(Suite.java:128)
at org.junit.runners.Suite.runChild(Suite.java:24)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Is this a permissions issue?
2) I tried to run a single test like this *mvn test
-Dtest=org.apache.hadoop.hbase.filter.TestPrefixFilter* and this *mvn test
-Dtest=TestPrefixFilter*
But that fails with this error
Failed to execute goal
org.apache.maven.plugins:maven-surefire-plugin:2.12-TRUNK-HBASE-2:test
(secondPartTestsExecution) on project hbase: No tests were executed! (Set
-DfailIfNoTests=false to ignore this error.) -> [Help 1]
What am I doing wrong here?
On Thu, Jul 25, 2013 at 1:54 PM, Premal Shah <pr...@gmail.com>wrote:
> Hi Ted,
> I'm using 0.94.6.
>
> I'll setup a unix test.
>
>
> On Thu, Jul 25, 2013 at 1:50 AM, Ted Yu <yu...@gmail.com> wrote:
>
>> What HBase release are you using ?
>>
>> Can you put the scenario below in a unit test ?
>>
>> Thanks
>>
>> On Jul 24, 2013, at 11:13 PM, Premal Shah <pr...@gmail.com>
>> wrote:
>>
>> > Hi,
>> > I was playing around with using FuzzyRowFilter for my use case which
>> looks
>> > like this
>> >
>> > rk - 18 bytes (2 bytes + 8 bytes + 8 bytes)
>> > bytes 2-9 are fuzzy
>> > the fuzzy mask looks like this (0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0,
>> 0,
>> > 0, 0, 0 ,0)
>> > the start and stop key control the scan range
>> >
>> > Running a scanner with just the FuzzyRowFilter works fine. However, I
>> need
>> > to limit the columns returned. So I have a ColumnRangeFilter too. Both
>> the
>> > filters are wrapped in a FilterList
>> >
>> > The combination of both, returns no rows from hbase with MUST_PASS_ALL
>> set.
>> > Hbase seems to be aborting when the row does not match Fuzzy
>> > If MUST_PASS_ONE is set, then it returns the columns from the rows that
>> > don't pass Fuzzy.
>> >
>> > How do you go about using the FilterList with both filters and return
>> the
>> > required rows only?
>> >
>> > --
>> > Regards,
>> > Premal Shah.
>>
>
>
>
> --
> Regards,
> Premal Shah.
>
--
Regards,
Premal Shah.
Re: FuzzyRowFilter with ColumnRangeFilter
Posted by Premal Shah <pr...@gmail.com>.
Hi Ted,
I'm using 0.94.6.
I'll setup a unix test.
On Thu, Jul 25, 2013 at 1:50 AM, Ted Yu <yu...@gmail.com> wrote:
> What HBase release are you using ?
>
> Can you put the scenario below in a unit test ?
>
> Thanks
>
> On Jul 24, 2013, at 11:13 PM, Premal Shah <pr...@gmail.com> wrote:
>
> > Hi,
> > I was playing around with using FuzzyRowFilter for my use case which
> looks
> > like this
> >
> > rk - 18 bytes (2 bytes + 8 bytes + 8 bytes)
> > bytes 2-9 are fuzzy
> > the fuzzy mask looks like this (0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0,
> > 0, 0, 0 ,0)
> > the start and stop key control the scan range
> >
> > Running a scanner with just the FuzzyRowFilter works fine. However, I
> need
> > to limit the columns returned. So I have a ColumnRangeFilter too. Both
> the
> > filters are wrapped in a FilterList
> >
> > The combination of both, returns no rows from hbase with MUST_PASS_ALL
> set.
> > Hbase seems to be aborting when the row does not match Fuzzy
> > If MUST_PASS_ONE is set, then it returns the columns from the rows that
> > don't pass Fuzzy.
> >
> > How do you go about using the FilterList with both filters and return the
> > required rows only?
> >
> > --
> > Regards,
> > Premal Shah.
>
--
Regards,
Premal Shah.
Re: FuzzyRowFilter with ColumnRangeFilter
Posted by Ted Yu <yu...@gmail.com>.
What HBase release are you using ?
Can you put the scenario below in a unit test ?
Thanks
On Jul 24, 2013, at 11:13 PM, Premal Shah <pr...@gmail.com> wrote:
> Hi,
> I was playing around with using FuzzyRowFilter for my use case which looks
> like this
>
> rk - 18 bytes (2 bytes + 8 bytes + 8 bytes)
> bytes 2-9 are fuzzy
> the fuzzy mask looks like this (0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0,
> 0, 0, 0 ,0)
> the start and stop key control the scan range
>
> Running a scanner with just the FuzzyRowFilter works fine. However, I need
> to limit the columns returned. So I have a ColumnRangeFilter too. Both the
> filters are wrapped in a FilterList
>
> The combination of both, returns no rows from hbase with MUST_PASS_ALL set.
> Hbase seems to be aborting when the row does not match Fuzzy
> If MUST_PASS_ONE is set, then it returns the columns from the rows that
> don't pass Fuzzy.
>
> How do you go about using the FilterList with both filters and return the
> required rows only?
>
> --
> Regards,
> Premal Shah.