You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by "leiwangouc@gmail.com" <le...@gmail.com> on 2014/06/10 13:38:43 UTC
HBase coprocessor AggregationClient take too long time
A simple java class to calculate the number of rows for a table.
public class RowCount {
private static final byte[] TABLE_NAME = Bytes.toBytes("userdigest");
private static final byte[] CF = Bytes.toBytes("cf");
public static void main(String[] args) throws Throwable {
Configuration conf = new Configuration();
conf.set("hbase.rpc.timeout", "1000000");
conf.set("zookeeper.session.timeout", "300000");
conf.setLong("hbase.client.scanner.caching", 10000);
Configuration configuration = HBaseConfiguration.create(conf);
AggregationClient aggregationClient = new AggregationClient(configuration);
Scan scan = new Scan();
scan.addFamily(CF);
long rowCount = aggregationClient.rowCount(TABLE_NAME, null, scan);
System.out.println("row count is " + rowCount);
}
}
It didn't finish even after 2 hours.
Any insight on this?
The total number of rows for the table is about 2G(2,000,000,000).
Thanks,
Lei
leiwangouc@gmail.com
Re: Re: HBase coprocessor AggregationClient take too long time
Posted by Ted Yu <yu...@gmail.com>.
See the example
from src/test/java/org/apache/hadoop/hbase/coprocessor/TestAggregateProtocol.java:
final ColumnInterpreter<Long, Long> ci = new LongColumnInterpreter();
long rowCount = aClient.rowCount(TEST_TABLE, ci,
scan);
Cheers
On Tue, Jun 10, 2014 at 7:23 PM, leiwangouc@gmail.com <le...@gmail.com>
wrote:
>
> There're 1632 regions for the table.
> hbase release: 0.94.6-cdh4.3.1
> No region server error logs found during this period.
> There was other jobs runing during this period.
>
> Thanks,
> Lei
>
>
>
>
> leiwangouc@gmail.com
>
> From: Ted Yu
> Date: 2014-06-10 20:44
> To: user@hbase.apache.org
> CC: user
> Subject: Re: HBase coprocessor AggregationClient take too long time
> How many regions does this table have ?
> What release of hbase are you using ?
>
> Do you observe exceptions in region server logs during this period ?
> Were there other jobs running around this period ?
>
> Cheers
>
> On Jun 10, 2014, at 4:38 AM, "leiwangouc@gmail.com" <le...@gmail.com>
> wrote:
>
> >
> > A simple java class to calculate the number of rows for a table.
> >
> > public class RowCount {
> > private static final byte[] TABLE_NAME = Bytes.toBytes("userdigest");
> > private static final byte[] CF = Bytes.toBytes("cf");
> >
> > public static void main(String[] args) throws Throwable {
> >
> > Configuration conf = new Configuration();
> > conf.set("hbase.rpc.timeout", "1000000");
> > conf.set("zookeeper.session.timeout", "300000");
> > conf.setLong("hbase.client.scanner.caching", 10000);
> > Configuration configuration = HBaseConfiguration.create(conf);
> >
> > AggregationClient aggregationClient = new
> AggregationClient(configuration);
> > Scan scan = new Scan();
> > scan.addFamily(CF);
> > long rowCount = aggregationClient.rowCount(TABLE_NAME, null, scan);
> > System.out.println("row count is " + rowCount);
> > }
> > }
> >
> > It didn't finish even after 2 hours.
> > Any insight on this?
> >
> > The total number of rows for the table is about 2G(2,000,000,000).
> >
> > Thanks,
> > Lei
> >
> >
> >
> > leiwangouc@gmail.com
>
Re: Re: HBase coprocessor AggregationClient take too long time
Posted by "leiwangouc@gmail.com" <le...@gmail.com>.
There're 1632 regions for the table.
hbase release: 0.94.6-cdh4.3.1
No region server error logs found during this period.
There was other jobs runing during this period.
Thanks,
Lei
leiwangouc@gmail.com
From: Ted Yu
Date: 2014-06-10 20:44
To: user@hbase.apache.org
CC: user
Subject: Re: HBase coprocessor AggregationClient take too long time
How many regions does this table have ?
What release of hbase are you using ?
Do you observe exceptions in region server logs during this period ?
Were there other jobs running around this period ?
Cheers
On Jun 10, 2014, at 4:38 AM, "leiwangouc@gmail.com" <le...@gmail.com> wrote:
>
> A simple java class to calculate the number of rows for a table.
>
> public class RowCount {
> private static final byte[] TABLE_NAME = Bytes.toBytes("userdigest");
> private static final byte[] CF = Bytes.toBytes("cf");
>
> public static void main(String[] args) throws Throwable {
>
> Configuration conf = new Configuration();
> conf.set("hbase.rpc.timeout", "1000000");
> conf.set("zookeeper.session.timeout", "300000");
> conf.setLong("hbase.client.scanner.caching", 10000);
> Configuration configuration = HBaseConfiguration.create(conf);
>
> AggregationClient aggregationClient = new AggregationClient(configuration);
> Scan scan = new Scan();
> scan.addFamily(CF);
> long rowCount = aggregationClient.rowCount(TABLE_NAME, null, scan);
> System.out.println("row count is " + rowCount);
> }
> }
>
> It didn't finish even after 2 hours.
> Any insight on this?
>
> The total number of rows for the table is about 2G(2,000,000,000).
>
> Thanks,
> Lei
>
>
>
> leiwangouc@gmail.com
Re: HBase coprocessor AggregationClient take too long time
Posted by Ted Yu <yu...@gmail.com>.
How many regions does this table have ?
What release of hbase are you using ?
Do you observe exceptions in region server logs during this period ?
Were there other jobs running around this period ?
Cheers
On Jun 10, 2014, at 4:38 AM, "leiwangouc@gmail.com" <le...@gmail.com> wrote:
>
> A simple java class to calculate the number of rows for a table.
>
> public class RowCount {
> private static final byte[] TABLE_NAME = Bytes.toBytes("userdigest");
> private static final byte[] CF = Bytes.toBytes("cf");
>
> public static void main(String[] args) throws Throwable {
>
> Configuration conf = new Configuration();
> conf.set("hbase.rpc.timeout", "1000000");
> conf.set("zookeeper.session.timeout", "300000");
> conf.setLong("hbase.client.scanner.caching", 10000);
> Configuration configuration = HBaseConfiguration.create(conf);
>
> AggregationClient aggregationClient = new AggregationClient(configuration);
> Scan scan = new Scan();
> scan.addFamily(CF);
> long rowCount = aggregationClient.rowCount(TABLE_NAME, null, scan);
> System.out.println("row count is " + rowCount);
> }
> }
>
> It didn't finish even after 2 hours.
> Any insight on this?
>
> The total number of rows for the table is about 2G(2,000,000,000).
>
> Thanks,
> Lei
>
>
>
> leiwangouc@gmail.com