You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by "leiwangouc@gmail.com" <le...@gmail.com> on 2014/01/17 08:58:56 UTC
How to quickly count the rows that meet several conditions using hbase coprocessor
Hi,
I know that hbase copocessor provides a quick way to count the rows of a table.
But how can i count the rows that meet several conditions.
Take this for example.
I have a hbase table with one column family, several columns. I want to caculate the number of rows that meet 3 conditions:
has column1
has column2
has column3 and the value of column3 satisfy a regular expression
Thans,
Lei
leiwangouc@gmail.com
Re: Re: How to quickly count the rows that meet several conditions using hbase coprocessor
Posted by "leiwangouc@gmail.com" <le...@gmail.com>.
Hi Ted,
I have tried. However, this will not return the expected result for 2 QualifierFilter.
For example,
String filterString = "QualifierFilter(=, 'binary:tags') AND QualifierFilter(=, 'binary:googleid')" return 0. Although there are rows that have both column 'tags' and column 'googleid'.
Thanks,
Lei
leiwangouc@gmail.com
From: Ted Yu
Date: 2014-01-18 12:00
To: user@hbase.apache.org
Subject: Re: Re: How to quickly count the rows that meet several conditions using hbase coprocessor
Please take a look at TestParseFilter#testCompoundFilter2
You can construct compound filter which involves more than
one QualifierFilter.
Cheers
On Fri, Jan 17, 2014 at 7:41 PM, leiwangouc@gmail.com
<le...@gmail.com>wrote:
> Hi Ted,
>
> According to the initial email, I need two QualifierFilter and one
> SingleColumnValueFilter. But apply 2 QualifierFilter on a scan will not
> work as described
> http://stackoverflow.com/questions/13379350/how-to-apply-several-qualifierfilter-to-a-row-in-hbaseso i transfer them to SingleColumnValueFilter.
>
> Thanks,
> Lei
>
>
>
>
> leiwangouc@gmail.com
>
> From: Ted Yu
> Date: 2014-01-18 11:06
> To: user@hbase.apache.org
> Subject: Re: Re: How to quickly count the rows that meet several
> conditions using hbase coprocessor
> Copying the last reply from the link you gave:
>
> @Henrik I don't know how much data you have but I'm afraid you are right.
> Another option would be to implement a custom filter which takes the
> qualifier list you are looking for
>
> It was acknowledged that the proposed transformation may not give the best
> performance.
>
> From your initial email:
> bq. has column3 and the value of column3 satisfy a regular expression
>
> Why do you need 2 SingleColumnValueFilter's if you only want to check the
> value of column3 ?
>
> Cheers
>
>
> On Fri, Jan 17, 2014 at 6:47 PM, leiwangouc@gmail.com
> <le...@gmail.com>wrote:
>
> > Hi Ted,
> >
> > Haven't tried the performance without using coprocessor.
> >
> > Actually I transfered two 2 QualifierFilter to 2 SingleColumnValueFilter
> > according to the describtion
> >
> http://stackoverflow.com/questions/13379350/how-to-apply-several-qualifierfilter-to-a-row-in-hbaseand then pass the scan to to AggregationClient#rowCount.
> >
> > Is there anything special in regard to SingleColumnValueFilter that
> > results the slowness?
> >
> > Thanks,
> > Lei
> >
> >
> >
> >
> > leiwangouc@gmail.com
> >
> > From: Ted Yu
> > Date: 2014-01-18 09:49
> > To: user@hbase.apache.org
> > CC: user
> > Subject: Re: How to quickly count the rows that meet several conditions
> > using hbase coprocessor
> > Do you see the same slowness scanning regions with FilterList outside
> > coprocessor ?
> >
> > Thanks
> >
> > On Jan 17, 2014, at 5:24 PM, "leiwangouc@gmail.com" <
> leiwangouc@gmail.com>
> > wrote:
> >
> > > Hi,
> > >
> > > I have tried.
> > > For a talbe with about 600 million rowkey, just pass a single
> > QualifierFilter, it can get the result quickly.
> > > But when i add the SingleColumnValueFilter with FilterList, it becoumes
> > very slow and i can't stand it.
> > >
> > > I think i can write my own custumed aggregation client. Is there any
> > example or user guide about how to write custumed aggregation client
> using
> > coprocessor?
> > >
> > > Thanks,
> > > Lei
> > >
> > >
> > >
> > >
> > > leiwangouc@gmail.com
> > >
> > > From: Ted Yu
> > > Date: 2014-01-17 18:03
> > > To: user@hbase.apache.org
> > > CC: user
> > > Subject: Re: How to quickly count the rows that meet several conditions
> > using hbase coprocessor
> > > Take a look at
> >
> http://hbase.apache.org/0.94/apidocs/org/apache/hadoop/hbase/client/coprocessor/AggregationClient.html#rowCount(byte[],%20org.apache.hadoop.hbase.coprocessor.ColumnInterpreter,%20org.apache.hadoop.hbase.client.Scan)
> > >
> > > You can pass custom filter through Scan parameter.
> > >
> > > Cheers
> > >
> > > On Jan 16, 2014, at 11:58 PM, "leiwangouc@gmail.com" <
> > leiwangouc@gmail.com> wrote:
> > >
> > >> Hi,
> > >>
> > >> I know that hbase copocessor provides a quick way to count the rows of
> > a table.
> > >> But how can i count the rows that meet several conditions.
> > >>
> > >> Take this for example.
> > >> I have a hbase table with one column family, several columns. I want
> to
> > caculate the number of rows that meet 3 conditions:
> > >> has column1
> > >> has column2
> > >> has column3 and the value of column3 satisfy a regular expression
> > >>
> > >> Thans,
> > >> Lei
> > >>
> > >>
> > >>
> > >> leiwangouc@gmail.com
> >
>
Re: Re: How to quickly count the rows that meet several conditions
using hbase coprocessor
Posted by Ted Yu <yu...@gmail.com>.
Please take a look at TestParseFilter#testCompoundFilter2
You can construct compound filter which involves more than
one QualifierFilter.
Cheers
On Fri, Jan 17, 2014 at 7:41 PM, leiwangouc@gmail.com
<le...@gmail.com>wrote:
> Hi Ted,
>
> According to the initial email, I need two QualifierFilter and one
> SingleColumnValueFilter. But apply 2 QualifierFilter on a scan will not
> work as described
> http://stackoverflow.com/questions/13379350/how-to-apply-several-qualifierfilter-to-a-row-in-hbaseso i transfer them to SingleColumnValueFilter.
>
> Thanks,
> Lei
>
>
>
>
> leiwangouc@gmail.com
>
> From: Ted Yu
> Date: 2014-01-18 11:06
> To: user@hbase.apache.org
> Subject: Re: Re: How to quickly count the rows that meet several
> conditions using hbase coprocessor
> Copying the last reply from the link you gave:
>
> @Henrik I don't know how much data you have but I'm afraid you are right.
> Another option would be to implement a custom filter which takes the
> qualifier list you are looking for
>
> It was acknowledged that the proposed transformation may not give the best
> performance.
>
> From your initial email:
> bq. has column3 and the value of column3 satisfy a regular expression
>
> Why do you need 2 SingleColumnValueFilter's if you only want to check the
> value of column3 ?
>
> Cheers
>
>
> On Fri, Jan 17, 2014 at 6:47 PM, leiwangouc@gmail.com
> <le...@gmail.com>wrote:
>
> > Hi Ted,
> >
> > Haven't tried the performance without using coprocessor.
> >
> > Actually I transfered two 2 QualifierFilter to 2 SingleColumnValueFilter
> > according to the describtion
> >
> http://stackoverflow.com/questions/13379350/how-to-apply-several-qualifierfilter-to-a-row-in-hbaseand then pass the scan to to AggregationClient#rowCount.
> >
> > Is there anything special in regard to SingleColumnValueFilter that
> > results the slowness?
> >
> > Thanks,
> > Lei
> >
> >
> >
> >
> > leiwangouc@gmail.com
> >
> > From: Ted Yu
> > Date: 2014-01-18 09:49
> > To: user@hbase.apache.org
> > CC: user
> > Subject: Re: How to quickly count the rows that meet several conditions
> > using hbase coprocessor
> > Do you see the same slowness scanning regions with FilterList outside
> > coprocessor ?
> >
> > Thanks
> >
> > On Jan 17, 2014, at 5:24 PM, "leiwangouc@gmail.com" <
> leiwangouc@gmail.com>
> > wrote:
> >
> > > Hi,
> > >
> > > I have tried.
> > > For a talbe with about 600 million rowkey, just pass a single
> > QualifierFilter, it can get the result quickly.
> > > But when i add the SingleColumnValueFilter with FilterList, it becoumes
> > very slow and i can't stand it.
> > >
> > > I think i can write my own custumed aggregation client. Is there any
> > example or user guide about how to write custumed aggregation client
> using
> > coprocessor?
> > >
> > > Thanks,
> > > Lei
> > >
> > >
> > >
> > >
> > > leiwangouc@gmail.com
> > >
> > > From: Ted Yu
> > > Date: 2014-01-17 18:03
> > > To: user@hbase.apache.org
> > > CC: user
> > > Subject: Re: How to quickly count the rows that meet several conditions
> > using hbase coprocessor
> > > Take a look at
> >
> http://hbase.apache.org/0.94/apidocs/org/apache/hadoop/hbase/client/coprocessor/AggregationClient.html#rowCount(byte[],%20org.apache.hadoop.hbase.coprocessor.ColumnInterpreter,%20org.apache.hadoop.hbase.client.Scan)
> > >
> > > You can pass custom filter through Scan parameter.
> > >
> > > Cheers
> > >
> > > On Jan 16, 2014, at 11:58 PM, "leiwangouc@gmail.com" <
> > leiwangouc@gmail.com> wrote:
> > >
> > >> Hi,
> > >>
> > >> I know that hbase copocessor provides a quick way to count the rows of
> > a table.
> > >> But how can i count the rows that meet several conditions.
> > >>
> > >> Take this for example.
> > >> I have a hbase table with one column family, several columns. I want
> to
> > caculate the number of rows that meet 3 conditions:
> > >> has column1
> > >> has column2
> > >> has column3 and the value of column3 satisfy a regular expression
> > >>
> > >> Thans,
> > >> Lei
> > >>
> > >>
> > >>
> > >> leiwangouc@gmail.com
> >
>
Re: Re: How to quickly count the rows that meet several conditions using hbase coprocessor
Posted by "leiwangouc@gmail.com" <le...@gmail.com>.
Hi Ted,
According to the initial email, I need two QualifierFilter and one SingleColumnValueFilter. But apply 2 QualifierFilter on a scan will not work as described http://stackoverflow.com/questions/13379350/how-to-apply-several-qualifierfilter-to-a-row-in-hbase so i transfer them to SingleColumnValueFilter.
Thanks,
Lei
leiwangouc@gmail.com
From: Ted Yu
Date: 2014-01-18 11:06
To: user@hbase.apache.org
Subject: Re: Re: How to quickly count the rows that meet several conditions using hbase coprocessor
Copying the last reply from the link you gave:
@Henrik I don't know how much data you have but I'm afraid you are right.
Another option would be to implement a custom filter which takes the
qualifier list you are looking for
It was acknowledged that the proposed transformation may not give the best
performance.
From your initial email:
bq. has column3 and the value of column3 satisfy a regular expression
Why do you need 2 SingleColumnValueFilter's if you only want to check the
value of column3 ?
Cheers
On Fri, Jan 17, 2014 at 6:47 PM, leiwangouc@gmail.com
<le...@gmail.com>wrote:
> Hi Ted,
>
> Haven't tried the performance without using coprocessor.
>
> Actually I transfered two 2 QualifierFilter to 2 SingleColumnValueFilter
> according to the describtion
> http://stackoverflow.com/questions/13379350/how-to-apply-several-qualifierfilter-to-a-row-in-hbase and then pass the scan to to AggregationClient#rowCount.
>
> Is there anything special in regard to SingleColumnValueFilter that
> results the slowness?
>
> Thanks,
> Lei
>
>
>
>
> leiwangouc@gmail.com
>
> From: Ted Yu
> Date: 2014-01-18 09:49
> To: user@hbase.apache.org
> CC: user
> Subject: Re: How to quickly count the rows that meet several conditions
> using hbase coprocessor
> Do you see the same slowness scanning regions with FilterList outside
> coprocessor ?
>
> Thanks
>
> On Jan 17, 2014, at 5:24 PM, "leiwangouc@gmail.com" <le...@gmail.com>
> wrote:
>
> > Hi,
> >
> > I have tried.
> > For a talbe with about 600 million rowkey, just pass a single
> QualifierFilter, it can get the result quickly.
> > But when i add the SingleColumnValueFilter with FilterList, it becoumes
> very slow and i can't stand it.
> >
> > I think i can write my own custumed aggregation client. Is there any
> example or user guide about how to write custumed aggregation client using
> coprocessor?
> >
> > Thanks,
> > Lei
> >
> >
> >
> >
> > leiwangouc@gmail.com
> >
> > From: Ted Yu
> > Date: 2014-01-17 18:03
> > To: user@hbase.apache.org
> > CC: user
> > Subject: Re: How to quickly count the rows that meet several conditions
> using hbase coprocessor
> > Take a look at
> http://hbase.apache.org/0.94/apidocs/org/apache/hadoop/hbase/client/coprocessor/AggregationClient.html#rowCount(byte[],%20org.apache.hadoop.hbase.coprocessor.ColumnInterpreter,%20org.apache.hadoop.hbase.client.Scan)
> >
> > You can pass custom filter through Scan parameter.
> >
> > Cheers
> >
> > On Jan 16, 2014, at 11:58 PM, "leiwangouc@gmail.com" <
> leiwangouc@gmail.com> wrote:
> >
> >> Hi,
> >>
> >> I know that hbase copocessor provides a quick way to count the rows of
> a table.
> >> But how can i count the rows that meet several conditions.
> >>
> >> Take this for example.
> >> I have a hbase table with one column family, several columns. I want to
> caculate the number of rows that meet 3 conditions:
> >> has column1
> >> has column2
> >> has column3 and the value of column3 satisfy a regular expression
> >>
> >> Thans,
> >> Lei
> >>
> >>
> >>
> >> leiwangouc@gmail.com
>
Re: Re: How to quickly count the rows that meet several conditions
using hbase coprocessor
Posted by Ted Yu <yu...@gmail.com>.
Copying the last reply from the link you gave:
@Henrik I don't know how much data you have but I'm afraid you are right.
Another option would be to implement a custom filter which takes the
qualifier list you are looking for
It was acknowledged that the proposed transformation may not give the best
performance.
>From your initial email:
bq. has column3 and the value of column3 satisfy a regular expression
Why do you need 2 SingleColumnValueFilter's if you only want to check the
value of column3 ?
Cheers
On Fri, Jan 17, 2014 at 6:47 PM, leiwangouc@gmail.com
<le...@gmail.com>wrote:
> Hi Ted,
>
> Haven't tried the performance without using coprocessor.
>
> Actually I transfered two 2 QualifierFilter to 2 SingleColumnValueFilter
> according to the describtion
> http://stackoverflow.com/questions/13379350/how-to-apply-several-qualifierfilter-to-a-row-in-hbase and then pass the scan to to AggregationClient#rowCount.
>
> Is there anything special in regard to SingleColumnValueFilter that
> results the slowness?
>
> Thanks,
> Lei
>
>
>
>
> leiwangouc@gmail.com
>
> From: Ted Yu
> Date: 2014-01-18 09:49
> To: user@hbase.apache.org
> CC: user
> Subject: Re: How to quickly count the rows that meet several conditions
> using hbase coprocessor
> Do you see the same slowness scanning regions with FilterList outside
> coprocessor ?
>
> Thanks
>
> On Jan 17, 2014, at 5:24 PM, "leiwangouc@gmail.com" <le...@gmail.com>
> wrote:
>
> > Hi,
> >
> > I have tried.
> > For a talbe with about 600 million rowkey, just pass a single
> QualifierFilter, it can get the result quickly.
> > But when i add the SingleColumnValueFilter with FilterList, it becoumes
> very slow and i can't stand it.
> >
> > I think i can write my own custumed aggregation client. Is there any
> example or user guide about how to write custumed aggregation client using
> coprocessor?
> >
> > Thanks,
> > Lei
> >
> >
> >
> >
> > leiwangouc@gmail.com
> >
> > From: Ted Yu
> > Date: 2014-01-17 18:03
> > To: user@hbase.apache.org
> > CC: user
> > Subject: Re: How to quickly count the rows that meet several conditions
> using hbase coprocessor
> > Take a look at
> http://hbase.apache.org/0.94/apidocs/org/apache/hadoop/hbase/client/coprocessor/AggregationClient.html#rowCount(byte[],%20org.apache.hadoop.hbase.coprocessor.ColumnInterpreter,%20org.apache.hadoop.hbase.client.Scan)
> >
> > You can pass custom filter through Scan parameter.
> >
> > Cheers
> >
> > On Jan 16, 2014, at 11:58 PM, "leiwangouc@gmail.com" <
> leiwangouc@gmail.com> wrote:
> >
> >> Hi,
> >>
> >> I know that hbase copocessor provides a quick way to count the rows of
> a table.
> >> But how can i count the rows that meet several conditions.
> >>
> >> Take this for example.
> >> I have a hbase table with one column family, several columns. I want to
> caculate the number of rows that meet 3 conditions:
> >> has column1
> >> has column2
> >> has column3 and the value of column3 satisfy a regular expression
> >>
> >> Thans,
> >> Lei
> >>
> >>
> >>
> >> leiwangouc@gmail.com
>
Re: Re: How to quickly count the rows that meet several conditions using hbase coprocessor
Posted by "leiwangouc@gmail.com" <le...@gmail.com>.
Hi Ted,
Haven't tried the performance without using coprocessor.
Actually I transfered two 2 QualifierFilter to 2 SingleColumnValueFilter according to the describtion http://stackoverflow.com/questions/13379350/how-to-apply-several-qualifierfilter-to-a-row-in-hbase and then pass the scan to to AggregationClient#rowCount.
Is there anything special in regard to SingleColumnValueFilter that results the slowness?
Thanks,
Lei
leiwangouc@gmail.com
From: Ted Yu
Date: 2014-01-18 09:49
To: user@hbase.apache.org
CC: user
Subject: Re: How to quickly count the rows that meet several conditions using hbase coprocessor
Do you see the same slowness scanning regions with FilterList outside coprocessor ?
Thanks
On Jan 17, 2014, at 5:24 PM, "leiwangouc@gmail.com" <le...@gmail.com> wrote:
> Hi,
>
> I have tried.
> For a talbe with about 600 million rowkey, just pass a single QualifierFilter, it can get the result quickly.
> But when i add the SingleColumnValueFilter with FilterList, it becoumes very slow and i can't stand it.
>
> I think i can write my own custumed aggregation client. Is there any example or user guide about how to write custumed aggregation client using coprocessor?
>
> Thanks,
> Lei
>
>
>
>
> leiwangouc@gmail.com
>
> From: Ted Yu
> Date: 2014-01-17 18:03
> To: user@hbase.apache.org
> CC: user
> Subject: Re: How to quickly count the rows that meet several conditions using hbase coprocessor
> Take a look at http://hbase.apache.org/0.94/apidocs/org/apache/hadoop/hbase/client/coprocessor/AggregationClient.html#rowCount(byte[],%20org.apache.hadoop.hbase.coprocessor.ColumnInterpreter,%20org.apache.hadoop.hbase.client.Scan)
>
> You can pass custom filter through Scan parameter.
>
> Cheers
>
> On Jan 16, 2014, at 11:58 PM, "leiwangouc@gmail.com" <le...@gmail.com> wrote:
>
>> Hi,
>>
>> I know that hbase copocessor provides a quick way to count the rows of a table.
>> But how can i count the rows that meet several conditions.
>>
>> Take this for example.
>> I have a hbase table with one column family, several columns. I want to caculate the number of rows that meet 3 conditions:
>> has column1
>> has column2
>> has column3 and the value of column3 satisfy a regular expression
>>
>> Thans,
>> Lei
>>
>>
>>
>> leiwangouc@gmail.com
Re: How to quickly count the rows that meet several conditions using hbase coprocessor
Posted by Ted Yu <yu...@gmail.com>.
Do you see the same slowness scanning regions with FilterList outside coprocessor ?
Thanks
On Jan 17, 2014, at 5:24 PM, "leiwangouc@gmail.com" <le...@gmail.com> wrote:
> Hi,
>
> I have tried.
> For a talbe with about 600 million rowkey, just pass a single QualifierFilter, it can get the result quickly.
> But when i add the SingleColumnValueFilter with FilterList, it becoumes very slow and i can't stand it.
>
> I think i can write my own custumed aggregation client. Is there any example or user guide about how to write custumed aggregation client using coprocessor?
>
> Thanks,
> Lei
>
>
>
>
> leiwangouc@gmail.com
>
> From: Ted Yu
> Date: 2014-01-17 18:03
> To: user@hbase.apache.org
> CC: user
> Subject: Re: How to quickly count the rows that meet several conditions using hbase coprocessor
> Take a look at http://hbase.apache.org/0.94/apidocs/org/apache/hadoop/hbase/client/coprocessor/AggregationClient.html#rowCount(byte[],%20org.apache.hadoop.hbase.coprocessor.ColumnInterpreter,%20org.apache.hadoop.hbase.client.Scan)
>
> You can pass custom filter through Scan parameter.
>
> Cheers
>
> On Jan 16, 2014, at 11:58 PM, "leiwangouc@gmail.com" <le...@gmail.com> wrote:
>
>> Hi,
>>
>> I know that hbase copocessor provides a quick way to count the rows of a table.
>> But how can i count the rows that meet several conditions.
>>
>> Take this for example.
>> I have a hbase table with one column family, several columns. I want to caculate the number of rows that meet 3 conditions:
>> has column1
>> has column2
>> has column3 and the value of column3 satisfy a regular expression
>>
>> Thans,
>> Lei
>>
>>
>>
>> leiwangouc@gmail.com
Re: Re: How to quickly count the rows that meet several conditions using hbase coprocessor
Posted by lars hofhansl <la...@apache.org>.
The real fix is in the parent (HBASE-9428), though.
-- Lars
________________________________
From: Ted Yu <yu...@gmail.com>
To: "user@hbase.apache.org" <us...@hbase.apache.org>
Sent: Sunday, January 19, 2014 9:22 PM
Subject: Re: Re: How to quickly count the rows that meet several conditions using hbase coprocessor
bq. The HBase version 0.94.6-cdh4.3.1
That explains it :-)
See HBASE-9711
Please upgrade your HBase.
Cheers
On Sun, Jan 19, 2014 at 9:08 PM, leiwangouc@gmail.com
<le...@gmail.com>wrote:
>
> It makes no difference even i change it to a single character "A".
>
> Thanks,
> Lei
>
>
>
>
> leiwangouc@gmail.com
>
> From: Ted Yu
> Date: 2014-01-18 14:28
> To: user@hbase.apache.org
> CC: user; lars hofhansl
> Subject: Re: How to quickly count the rows that meet several conditions
> using hbase coprocessor
> Can you use other string for fake value ?
> DOESNOTEXIST is a bit long. Shouldn't be difficult to come up with a
> single character string that doesn't appear in the first two columns.
>
> Cheers
>
> On Jan 17, 2014, at 8:34 PM, "leiwangouc@gmail.com" <le...@gmail.com>
> wrote:
>
> > Hi Lars,
> >
> > public class AggregationCountForMultiFilter {
> >
> > private static final byte[] TABLE_NAME = Bytes.toBytes("userdigest");
> > private static final byte[] CF = Bytes.toBytes("cf");
> > private static final byte[] FAKE_VLAUE = Bytes.toBytes("DOESNOTEXIST");
> >
> > public static void main(String[] args) {
> >
> > Configuration conf = new Configuration();
> > Configuration configuration = HBaseConfiguration.create(conf);
> > AggregationClient aggregationClient = new
> AggregationClient(configuration);
> >
> > byte[] colA = Bytes.toBytes("tags");
> > byte[] colB = Bytes.toBytes("googleid");
> > byte[] colC = Bytes.toBytes("createtime");
> >
> > List<Filter> filters = new ArrayList<Filter>();
> >
> > SingleColumnValueFilter filter1 = new SingleColumnValueFilter(CF, colA,
> CompareOp.NOT_EQUAL, FAKE_VLAUE);
> > filter1.setFilterIfMissing(true);
> > filters.add(filter1);
> >
> > SingleColumnValueFilter filter2 = new SingleColumnValueFilter(CF, colB,
> CompareOp.NOT_EQUAL, FAKE_VLAUE);
> > filter2.setFilterIfMissing(true);
> > filters.add(filter2);
> >
> > SingleColumnValueFilter filter3 = new SingleColumnValueFilter(CF, colC,
> CompareOp.EQUAL, new RegexStringComparator("^2014-01-15"));
> > filter3.setFilterIfMissing(true);
> > filters.add(filter3);
> >
> > FilterList filterList = new
> FilterList(FilterList.Operator.MUST_PASS_ALL, filters);
> >
> > Scan scan = new Scan();
> > scan.addFamily(CF);
> > scan.setFilter(filterList);
> >
> > long rowCount = 0;
> > try {
> > rowCount = aggregationClient.rowCount(TABLE_NAME, null, scan);
> > } catch (Throwable e) {
> > e.printStackTrace();
> > }
> > System.out.println("rowCount: " + rowCount);
> > }
> > }
> > }
> >
> > The HBase version 0.94.6-cdh4.3.1
> >
> > Thanks,
> > Lei
> >
> >
> >
> > leiwangouc@gmail.com
> >
> > From: lars hofhansl
> > Date: 2014-01-18 11:18
> > To: user@hbase.apache.org
> > Subject: Re: Re: How to quickly count the rows that meet several
> conditions using hbase coprocessor
> > Offhand there is no reason for that.
> > If you send some sample code that can seed the data and then run the
> filter that shows the problem, I'll offer to do some profiling.
> >
> > Which version of HBase are you using?
> >
> > -- Lars
> >
> >
> > ________________________________
> > From: "leiwangouc@gmail.com" <le...@gmail.com>
> > To: user <us...@hbase.apache.org>
> > Cc: user <us...@hbase.apache.org>
> > Sent: Friday, January 17, 2014 5:24 PM
> > Subject: Re: Re: How to quickly count the rows that meet several
> conditions using hbase coprocessor
> >
> > Hi,
> >
> > I have tried.
> > For a talbe with about 600 million rowkey, just pass a single
> QualifierFilter, it can get the result quickly.
> > But when i add the SingleColumnValueFilter with FilterList, it becoumes
> very slow and i can't stand it.
> >
> > I think i can write my own custumed aggregation client. Is there any
> example or user guide about how to write custumed aggregation client using
> coprocessor?
> >
> > Thanks,
> > Lei
> >
> >
> >
> >
> > leiwangouc@gmail.com
> >
> > From: Ted Yu
> > Date: 2014-01-17 18:03
> > To: user@hbase.apache.org
> > CC: user
> > Subject: Re: How to quickly count the rows that meet several conditions
> using hbase coprocessor
> > Take a look at
> http://hbase.apache.org/0.94/apidocs/org/apache/hadoop/hbase/client/coprocessor/AggregationClient.html#rowCount(byte[],%20org.apache.hadoop.hbase.coprocessor.ColumnInterpreter,%20org.apache.hadoop.hbase.client.Scan)
> >
> > You can pass custom filter through Scan parameter.
> >
> > Cheers
> >
> > On Jan 16, 2014, at 11:58 PM, "leiwangouc@gmail.com" <
> leiwangouc@gmail.com> wrote:
> >
> >> Hi,
> >>
> >> I know that hbase copocessor provides a quick way to count the rows of
> a table.
> >> But how can i count the rows that meet several conditions.
> >>
> >> Take this for example.
> >> I have a hbase table with one column family, several columns. I want to
> caculate the number of rows that meet 3 conditions:
> >> has column1
> >> has column2
> >> has column3 and the value of column3 satisfy a regular expression
> >>
> >> Thans,
> >> Lei
> >>
> >>
> >>
> >> leiwangouc@gmail.com
>
Re: Re: How to quickly count the rows that meet several conditions
using hbase coprocessor
Posted by Ted Yu <yu...@gmail.com>.
bq. The HBase version 0.94.6-cdh4.3.1
That explains it :-)
See HBASE-9711
Please upgrade your HBase.
Cheers
On Sun, Jan 19, 2014 at 9:08 PM, leiwangouc@gmail.com
<le...@gmail.com>wrote:
>
> It makes no difference even i change it to a single character "A".
>
> Thanks,
> Lei
>
>
>
>
> leiwangouc@gmail.com
>
> From: Ted Yu
> Date: 2014-01-18 14:28
> To: user@hbase.apache.org
> CC: user; lars hofhansl
> Subject: Re: How to quickly count the rows that meet several conditions
> using hbase coprocessor
> Can you use other string for fake value ?
> DOESNOTEXIST is a bit long. Shouldn't be difficult to come up with a
> single character string that doesn't appear in the first two columns.
>
> Cheers
>
> On Jan 17, 2014, at 8:34 PM, "leiwangouc@gmail.com" <le...@gmail.com>
> wrote:
>
> > Hi Lars,
> >
> > public class AggregationCountForMultiFilter {
> >
> > private static final byte[] TABLE_NAME = Bytes.toBytes("userdigest");
> > private static final byte[] CF = Bytes.toBytes("cf");
> > private static final byte[] FAKE_VLAUE = Bytes.toBytes("DOESNOTEXIST");
> >
> > public static void main(String[] args) {
> >
> > Configuration conf = new Configuration();
> > Configuration configuration = HBaseConfiguration.create(conf);
> > AggregationClient aggregationClient = new
> AggregationClient(configuration);
> >
> > byte[] colA = Bytes.toBytes("tags");
> > byte[] colB = Bytes.toBytes("googleid");
> > byte[] colC = Bytes.toBytes("createtime");
> >
> > List<Filter> filters = new ArrayList<Filter>();
> >
> > SingleColumnValueFilter filter1 = new SingleColumnValueFilter(CF, colA,
> CompareOp.NOT_EQUAL, FAKE_VLAUE);
> > filter1.setFilterIfMissing(true);
> > filters.add(filter1);
> >
> > SingleColumnValueFilter filter2 = new SingleColumnValueFilter(CF, colB,
> CompareOp.NOT_EQUAL, FAKE_VLAUE);
> > filter2.setFilterIfMissing(true);
> > filters.add(filter2);
> >
> > SingleColumnValueFilter filter3 = new SingleColumnValueFilter(CF, colC,
> CompareOp.EQUAL, new RegexStringComparator("^2014-01-15"));
> > filter3.setFilterIfMissing(true);
> > filters.add(filter3);
> >
> > FilterList filterList = new
> FilterList(FilterList.Operator.MUST_PASS_ALL, filters);
> >
> > Scan scan = new Scan();
> > scan.addFamily(CF);
> > scan.setFilter(filterList);
> >
> > long rowCount = 0;
> > try {
> > rowCount = aggregationClient.rowCount(TABLE_NAME, null, scan);
> > } catch (Throwable e) {
> > e.printStackTrace();
> > }
> > System.out.println("rowCount: " + rowCount);
> > }
> > }
> > }
> >
> > The HBase version 0.94.6-cdh4.3.1
> >
> > Thanks,
> > Lei
> >
> >
> >
> > leiwangouc@gmail.com
> >
> > From: lars hofhansl
> > Date: 2014-01-18 11:18
> > To: user@hbase.apache.org
> > Subject: Re: Re: How to quickly count the rows that meet several
> conditions using hbase coprocessor
> > Offhand there is no reason for that.
> > If you send some sample code that can seed the data and then run the
> filter that shows the problem, I'll offer to do some profiling.
> >
> > Which version of HBase are you using?
> >
> > -- Lars
> >
> >
> > ________________________________
> > From: "leiwangouc@gmail.com" <le...@gmail.com>
> > To: user <us...@hbase.apache.org>
> > Cc: user <us...@hbase.apache.org>
> > Sent: Friday, January 17, 2014 5:24 PM
> > Subject: Re: Re: How to quickly count the rows that meet several
> conditions using hbase coprocessor
> >
> > Hi,
> >
> > I have tried.
> > For a talbe with about 600 million rowkey, just pass a single
> QualifierFilter, it can get the result quickly.
> > But when i add the SingleColumnValueFilter with FilterList, it becoumes
> very slow and i can't stand it.
> >
> > I think i can write my own custumed aggregation client. Is there any
> example or user guide about how to write custumed aggregation client using
> coprocessor?
> >
> > Thanks,
> > Lei
> >
> >
> >
> >
> > leiwangouc@gmail.com
> >
> > From: Ted Yu
> > Date: 2014-01-17 18:03
> > To: user@hbase.apache.org
> > CC: user
> > Subject: Re: How to quickly count the rows that meet several conditions
> using hbase coprocessor
> > Take a look at
> http://hbase.apache.org/0.94/apidocs/org/apache/hadoop/hbase/client/coprocessor/AggregationClient.html#rowCount(byte[],%20org.apache.hadoop.hbase.coprocessor.ColumnInterpreter,%20org.apache.hadoop.hbase.client.Scan)
> >
> > You can pass custom filter through Scan parameter.
> >
> > Cheers
> >
> > On Jan 16, 2014, at 11:58 PM, "leiwangouc@gmail.com" <
> leiwangouc@gmail.com> wrote:
> >
> >> Hi,
> >>
> >> I know that hbase copocessor provides a quick way to count the rows of
> a table.
> >> But how can i count the rows that meet several conditions.
> >>
> >> Take this for example.
> >> I have a hbase table with one column family, several columns. I want to
> caculate the number of rows that meet 3 conditions:
> >> has column1
> >> has column2
> >> has column3 and the value of column3 satisfy a regular expression
> >>
> >> Thans,
> >> Lei
> >>
> >>
> >>
> >> leiwangouc@gmail.com
>
Re: Re: How to quickly count the rows that meet several conditions using hbase coprocessor
Posted by "leiwangouc@gmail.com" <le...@gmail.com>.
It makes no difference even i change it to a single character "A".
Thanks,
Lei
leiwangouc@gmail.com
From: Ted Yu
Date: 2014-01-18 14:28
To: user@hbase.apache.org
CC: user; lars hofhansl
Subject: Re: How to quickly count the rows that meet several conditions using hbase coprocessor
Can you use other string for fake value ?
DOESNOTEXIST is a bit long. Shouldn't be difficult to come up with a single character string that doesn't appear in the first two columns.
Cheers
On Jan 17, 2014, at 8:34 PM, "leiwangouc@gmail.com" <le...@gmail.com> wrote:
> Hi Lars,
>
> public class AggregationCountForMultiFilter {
>
> private static final byte[] TABLE_NAME = Bytes.toBytes("userdigest");
> private static final byte[] CF = Bytes.toBytes("cf");
> private static final byte[] FAKE_VLAUE = Bytes.toBytes("DOESNOTEXIST");
>
> public static void main(String[] args) {
>
> Configuration conf = new Configuration();
> Configuration configuration = HBaseConfiguration.create(conf);
> AggregationClient aggregationClient = new AggregationClient(configuration);
>
> byte[] colA = Bytes.toBytes("tags");
> byte[] colB = Bytes.toBytes("googleid");
> byte[] colC = Bytes.toBytes("createtime");
>
> List<Filter> filters = new ArrayList<Filter>();
>
> SingleColumnValueFilter filter1 = new SingleColumnValueFilter(CF, colA, CompareOp.NOT_EQUAL, FAKE_VLAUE);
> filter1.setFilterIfMissing(true);
> filters.add(filter1);
>
> SingleColumnValueFilter filter2 = new SingleColumnValueFilter(CF, colB, CompareOp.NOT_EQUAL, FAKE_VLAUE);
> filter2.setFilterIfMissing(true);
> filters.add(filter2);
>
> SingleColumnValueFilter filter3 = new SingleColumnValueFilter(CF, colC, CompareOp.EQUAL, new RegexStringComparator("^2014-01-15"));
> filter3.setFilterIfMissing(true);
> filters.add(filter3);
>
> FilterList filterList = new FilterList(FilterList.Operator.MUST_PASS_ALL, filters);
>
> Scan scan = new Scan();
> scan.addFamily(CF);
> scan.setFilter(filterList);
>
> long rowCount = 0;
> try {
> rowCount = aggregationClient.rowCount(TABLE_NAME, null, scan);
> } catch (Throwable e) {
> e.printStackTrace();
> }
> System.out.println("rowCount: " + rowCount);
> }
> }
> }
>
> The HBase version 0.94.6-cdh4.3.1
>
> Thanks,
> Lei
>
>
>
> leiwangouc@gmail.com
>
> From: lars hofhansl
> Date: 2014-01-18 11:18
> To: user@hbase.apache.org
> Subject: Re: Re: How to quickly count the rows that meet several conditions using hbase coprocessor
> Offhand there is no reason for that.
> If you send some sample code that can seed the data and then run the filter that shows the problem, I'll offer to do some profiling.
>
> Which version of HBase are you using?
>
> -- Lars
>
>
> ________________________________
> From: "leiwangouc@gmail.com" <le...@gmail.com>
> To: user <us...@hbase.apache.org>
> Cc: user <us...@hbase.apache.org>
> Sent: Friday, January 17, 2014 5:24 PM
> Subject: Re: Re: How to quickly count the rows that meet several conditions using hbase coprocessor
>
> Hi,
>
> I have tried.
> For a talbe with about 600 million rowkey, just pass a single QualifierFilter, it can get the result quickly.
> But when i add the SingleColumnValueFilter with FilterList, it becoumes very slow and i can't stand it.
>
> I think i can write my own custumed aggregation client. Is there any example or user guide about how to write custumed aggregation client using coprocessor?
>
> Thanks,
> Lei
>
>
>
>
> leiwangouc@gmail.com
>
> From: Ted Yu
> Date: 2014-01-17 18:03
> To: user@hbase.apache.org
> CC: user
> Subject: Re: How to quickly count the rows that meet several conditions using hbase coprocessor
> Take a look at http://hbase.apache.org/0.94/apidocs/org/apache/hadoop/hbase/client/coprocessor/AggregationClient.html#rowCount(byte[],%20org.apache.hadoop.hbase.coprocessor.ColumnInterpreter,%20org.apache.hadoop.hbase.client.Scan)
>
> You can pass custom filter through Scan parameter.
>
> Cheers
>
> On Jan 16, 2014, at 11:58 PM, "leiwangouc@gmail.com" <le...@gmail.com> wrote:
>
>> Hi,
>>
>> I know that hbase copocessor provides a quick way to count the rows of a table.
>> But how can i count the rows that meet several conditions.
>>
>> Take this for example.
>> I have a hbase table with one column family, several columns. I want to caculate the number of rows that meet 3 conditions:
>> has column1
>> has column2
>> has column3 and the value of column3 satisfy a regular expression
>>
>> Thans,
>> Lei
>>
>>
>>
>> leiwangouc@gmail.com
Re: How to quickly count the rows that meet several conditions using hbase coprocessor
Posted by Ted Yu <yu...@gmail.com>.
Can you use other string for fake value ?
DOESNOTEXIST is a bit long. Shouldn't be difficult to come up with a single character string that doesn't appear in the first two columns.
Cheers
On Jan 17, 2014, at 8:34 PM, "leiwangouc@gmail.com" <le...@gmail.com> wrote:
> Hi Lars,
>
> public class AggregationCountForMultiFilter {
>
> private static final byte[] TABLE_NAME = Bytes.toBytes("userdigest");
> private static final byte[] CF = Bytes.toBytes("cf");
> private static final byte[] FAKE_VLAUE = Bytes.toBytes("DOESNOTEXIST");
>
> public static void main(String[] args) {
>
> Configuration conf = new Configuration();
> Configuration configuration = HBaseConfiguration.create(conf);
> AggregationClient aggregationClient = new AggregationClient(configuration);
>
> byte[] colA = Bytes.toBytes("tags");
> byte[] colB = Bytes.toBytes("googleid");
> byte[] colC = Bytes.toBytes("createtime");
>
> List<Filter> filters = new ArrayList<Filter>();
>
> SingleColumnValueFilter filter1 = new SingleColumnValueFilter(CF, colA, CompareOp.NOT_EQUAL, FAKE_VLAUE);
> filter1.setFilterIfMissing(true);
> filters.add(filter1);
>
> SingleColumnValueFilter filter2 = new SingleColumnValueFilter(CF, colB, CompareOp.NOT_EQUAL, FAKE_VLAUE);
> filter2.setFilterIfMissing(true);
> filters.add(filter2);
>
> SingleColumnValueFilter filter3 = new SingleColumnValueFilter(CF, colC, CompareOp.EQUAL, new RegexStringComparator("^2014-01-15"));
> filter3.setFilterIfMissing(true);
> filters.add(filter3);
>
> FilterList filterList = new FilterList(FilterList.Operator.MUST_PASS_ALL, filters);
>
> Scan scan = new Scan();
> scan.addFamily(CF);
> scan.setFilter(filterList);
>
> long rowCount = 0;
> try {
> rowCount = aggregationClient.rowCount(TABLE_NAME, null, scan);
> } catch (Throwable e) {
> e.printStackTrace();
> }
> System.out.println("rowCount: " + rowCount);
> }
> }
> }
>
> The HBase version 0.94.6-cdh4.3.1
>
> Thanks,
> Lei
>
>
>
> leiwangouc@gmail.com
>
> From: lars hofhansl
> Date: 2014-01-18 11:18
> To: user@hbase.apache.org
> Subject: Re: Re: How to quickly count the rows that meet several conditions using hbase coprocessor
> Offhand there is no reason for that.
> If you send some sample code that can seed the data and then run the filter that shows the problem, I'll offer to do some profiling.
>
> Which version of HBase are you using?
>
> -- Lars
>
>
> ________________________________
> From: "leiwangouc@gmail.com" <le...@gmail.com>
> To: user <us...@hbase.apache.org>
> Cc: user <us...@hbase.apache.org>
> Sent: Friday, January 17, 2014 5:24 PM
> Subject: Re: Re: How to quickly count the rows that meet several conditions using hbase coprocessor
>
> Hi,
>
> I have tried.
> For a talbe with about 600 million rowkey, just pass a single QualifierFilter, it can get the result quickly.
> But when i add the SingleColumnValueFilter with FilterList, it becoumes very slow and i can't stand it.
>
> I think i can write my own custumed aggregation client. Is there any example or user guide about how to write custumed aggregation client using coprocessor?
>
> Thanks,
> Lei
>
>
>
>
> leiwangouc@gmail.com
>
> From: Ted Yu
> Date: 2014-01-17 18:03
> To: user@hbase.apache.org
> CC: user
> Subject: Re: How to quickly count the rows that meet several conditions using hbase coprocessor
> Take a look at http://hbase.apache.org/0.94/apidocs/org/apache/hadoop/hbase/client/coprocessor/AggregationClient.html#rowCount(byte[],%20org.apache.hadoop.hbase.coprocessor.ColumnInterpreter,%20org.apache.hadoop.hbase.client.Scan)
>
> You can pass custom filter through Scan parameter.
>
> Cheers
>
> On Jan 16, 2014, at 11:58 PM, "leiwangouc@gmail.com" <le...@gmail.com> wrote:
>
>> Hi,
>>
>> I know that hbase copocessor provides a quick way to count the rows of a table.
>> But how can i count the rows that meet several conditions.
>>
>> Take this for example.
>> I have a hbase table with one column family, several columns. I want to caculate the number of rows that meet 3 conditions:
>> has column1
>> has column2
>> has column3 and the value of column3 satisfy a regular expression
>>
>> Thans,
>> Lei
>>
>>
>>
>> leiwangouc@gmail.com
Re: Re: How to quickly count the rows that meet several conditions using hbase coprocessor
Posted by "leiwangouc@gmail.com" <le...@gmail.com>.
Hi Lars,
public class AggregationCountForMultiFilter {
private static final byte[] TABLE_NAME = Bytes.toBytes("userdigest");
private static final byte[] CF = Bytes.toBytes("cf");
private static final byte[] FAKE_VLAUE = Bytes.toBytes("DOESNOTEXIST");
public static void main(String[] args) {
Configuration conf = new Configuration();
Configuration configuration = HBaseConfiguration.create(conf);
AggregationClient aggregationClient = new AggregationClient(configuration);
byte[] colA = Bytes.toBytes("tags");
byte[] colB = Bytes.toBytes("googleid");
byte[] colC = Bytes.toBytes("createtime");
List<Filter> filters = new ArrayList<Filter>();
SingleColumnValueFilter filter1 = new SingleColumnValueFilter(CF, colA, CompareOp.NOT_EQUAL, FAKE_VLAUE);
filter1.setFilterIfMissing(true);
filters.add(filter1);
SingleColumnValueFilter filter2 = new SingleColumnValueFilter(CF, colB, CompareOp.NOT_EQUAL, FAKE_VLAUE);
filter2.setFilterIfMissing(true);
filters.add(filter2);
SingleColumnValueFilter filter3 = new SingleColumnValueFilter(CF, colC, CompareOp.EQUAL, new RegexStringComparator("^2014-01-15"));
filter3.setFilterIfMissing(true);
filters.add(filter3);
FilterList filterList = new FilterList(FilterList.Operator.MUST_PASS_ALL, filters);
Scan scan = new Scan();
scan.addFamily(CF);
scan.setFilter(filterList);
long rowCount = 0;
try {
rowCount = aggregationClient.rowCount(TABLE_NAME, null, scan);
} catch (Throwable e) {
e.printStackTrace();
}
System.out.println("rowCount: " + rowCount);
}
}
}
The HBase version 0.94.6-cdh4.3.1
Thanks,
Lei
leiwangouc@gmail.com
From: lars hofhansl
Date: 2014-01-18 11:18
To: user@hbase.apache.org
Subject: Re: Re: How to quickly count the rows that meet several conditions using hbase coprocessor
Offhand there is no reason for that.
If you send some sample code that can seed the data and then run the filter that shows the problem, I'll offer to do some profiling.
Which version of HBase are you using?
-- Lars
________________________________
From: "leiwangouc@gmail.com" <le...@gmail.com>
To: user <us...@hbase.apache.org>
Cc: user <us...@hbase.apache.org>
Sent: Friday, January 17, 2014 5:24 PM
Subject: Re: Re: How to quickly count the rows that meet several conditions using hbase coprocessor
Hi,
I have tried.
For a talbe with about 600 million rowkey, just pass a single QualifierFilter, it can get the result quickly.
But when i add the SingleColumnValueFilter with FilterList, it becoumes very slow and i can't stand it.
I think i can write my own custumed aggregation client. Is there any example or user guide about how to write custumed aggregation client using coprocessor?
Thanks,
Lei
leiwangouc@gmail.com
From: Ted Yu
Date: 2014-01-17 18:03
To: user@hbase.apache.org
CC: user
Subject: Re: How to quickly count the rows that meet several conditions using hbase coprocessor
Take a look at http://hbase.apache.org/0.94/apidocs/org/apache/hadoop/hbase/client/coprocessor/AggregationClient.html#rowCount(byte[],%20org.apache.hadoop.hbase.coprocessor.ColumnInterpreter,%20org.apache.hadoop.hbase.client.Scan)
You can pass custom filter through Scan parameter.
Cheers
On Jan 16, 2014, at 11:58 PM, "leiwangouc@gmail.com" <le...@gmail.com> wrote:
> Hi,
>
> I know that hbase copocessor provides a quick way to count the rows of a table.
> But how can i count the rows that meet several conditions.
>
> Take this for example.
> I have a hbase table with one column family, several columns. I want to caculate the number of rows that meet 3 conditions:
> has column1
> has column2
> has column3 and the value of column3 satisfy a regular expression
>
> Thans,
> Lei
>
>
>
> leiwangouc@gmail.com
Re: Re: How to quickly count the rows that meet several conditions using hbase coprocessor
Posted by lars hofhansl <la...@apache.org>.
Offhand there is no reason for that.
If you send some sample code that can seed the data and then run the filter that shows the problem, I'll offer to do some profiling.
Which version of HBase are you using?
-- Lars
________________________________
From: "leiwangouc@gmail.com" <le...@gmail.com>
To: user <us...@hbase.apache.org>
Cc: user <us...@hbase.apache.org>
Sent: Friday, January 17, 2014 5:24 PM
Subject: Re: Re: How to quickly count the rows that meet several conditions using hbase coprocessor
Hi,
I have tried.
For a talbe with about 600 million rowkey, just pass a single QualifierFilter, it can get the result quickly.
But when i add the SingleColumnValueFilter with FilterList, it becoumes very slow and i can't stand it.
I think i can write my own custumed aggregation client. Is there any example or user guide about how to write custumed aggregation client using coprocessor?
Thanks,
Lei
leiwangouc@gmail.com
From: Ted Yu
Date: 2014-01-17 18:03
To: user@hbase.apache.org
CC: user
Subject: Re: How to quickly count the rows that meet several conditions using hbase coprocessor
Take a look at http://hbase.apache.org/0.94/apidocs/org/apache/hadoop/hbase/client/coprocessor/AggregationClient.html#rowCount(byte[],%20org.apache.hadoop.hbase.coprocessor.ColumnInterpreter,%20org.apache.hadoop.hbase.client.Scan)
You can pass custom filter through Scan parameter.
Cheers
On Jan 16, 2014, at 11:58 PM, "leiwangouc@gmail.com" <le...@gmail.com> wrote:
> Hi,
>
> I know that hbase copocessor provides a quick way to count the rows of a table.
> But how can i count the rows that meet several conditions.
>
> Take this for example.
> I have a hbase table with one column family, several columns. I want to caculate the number of rows that meet 3 conditions:
> has column1
> has column2
> has column3 and the value of column3 satisfy a regular expression
>
> Thans,
> Lei
>
>
>
> leiwangouc@gmail.com
Re: Re: How to quickly count the rows that meet several conditions using hbase coprocessor
Posted by "leiwangouc@gmail.com" <le...@gmail.com>.
Hi,
I have tried.
For a talbe with about 600 million rowkey, just pass a single QualifierFilter, it can get the result quickly.
But when i add the SingleColumnValueFilter with FilterList, it becoumes very slow and i can't stand it.
I think i can write my own custumed aggregation client. Is there any example or user guide about how to write custumed aggregation client using coprocessor?
Thanks,
Lei
leiwangouc@gmail.com
From: Ted Yu
Date: 2014-01-17 18:03
To: user@hbase.apache.org
CC: user
Subject: Re: How to quickly count the rows that meet several conditions using hbase coprocessor
Take a look at http://hbase.apache.org/0.94/apidocs/org/apache/hadoop/hbase/client/coprocessor/AggregationClient.html#rowCount(byte[],%20org.apache.hadoop.hbase.coprocessor.ColumnInterpreter,%20org.apache.hadoop.hbase.client.Scan)
You can pass custom filter through Scan parameter.
Cheers
On Jan 16, 2014, at 11:58 PM, "leiwangouc@gmail.com" <le...@gmail.com> wrote:
> Hi,
>
> I know that hbase copocessor provides a quick way to count the rows of a table.
> But how can i count the rows that meet several conditions.
>
> Take this for example.
> I have a hbase table with one column family, several columns. I want to caculate the number of rows that meet 3 conditions:
> has column1
> has column2
> has column3 and the value of column3 satisfy a regular expression
>
> Thans,
> Lei
>
>
>
> leiwangouc@gmail.com
Re: How to quickly count the rows that meet several conditions using hbase coprocessor
Posted by Ted Yu <yu...@gmail.com>.
Take a look at http://hbase.apache.org/0.94/apidocs/org/apache/hadoop/hbase/client/coprocessor/AggregationClient.html#rowCount(byte[],%20org.apache.hadoop.hbase.coprocessor.ColumnInterpreter,%20org.apache.hadoop.hbase.client.Scan)
You can pass custom filter through Scan parameter.
Cheers
On Jan 16, 2014, at 11:58 PM, "leiwangouc@gmail.com" <le...@gmail.com> wrote:
> Hi,
>
> I know that hbase copocessor provides a quick way to count the rows of a table.
> But how can i count the rows that meet several conditions.
>
> Take this for example.
> I have a hbase table with one column family, several columns. I want to caculate the number of rows that meet 3 conditions:
> has column1
> has column2
> has column3 and the value of column3 satisfy a regular expression
>
> Thans,
> Lei
>
>
>
> leiwangouc@gmail.com