You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by tgh <gu...@ia.ac.cn> on 2012/12/27 03:21:10 UTC
how to use API to statistic how many message has been store in the table in hbase
Hi
I try to use hbaseAPI to store data, and I want to get the number of
the message stored in the table in HBase,
how to use API to statistic how many message has been store in the
table in hbase
What API can do this,
Could you help me
Thank you
---------------------
Tian Guanhua
答复: which API is to get table meta data in hbase
Posted by tgh <gu...@ia.ac.cn>.
Thank you for your reply
I try to use AggregationClient API , but as you know ,I want to know how
many rows there, scanner will scan whole table one by one row, is it,
And It is not slow for a big table , is it,
And you say make a record when we insert message in the table, it is a good
idea, but we have store the record there, and want to statistic the number
of record ,
And there is some speed up method for the scan,
Could you help me
Thank you
---------------------
Tian Guanhua
-----邮件原件-----
发件人: user-return-32787-guanhua.tian=ia.ac.cn@hbase.apache.org
[mailto:user-return-32787-guanhua.tian=ia.ac.cn@hbase.apache.org] 代表 Anoop
Sam John
发送时间: 2012年12月27日 16:23
收件人: user@hbase.apache.org
主题: RE: which API is to get table meta data in hbase
> But I say, there need some meta data which record how many row
number in the give table, I say , hbase has this meta data, is it,
And which API is to get it, and how to use API,
There is no such meta data for a table.
You can check whether you can do this work on your own using co processors.
There were some discussion on this in the mailing list. Sorry I am not able
find the links for that now. You can create another table and store the
stats there in the second table.
But see whether you can make use of the AggregationClient API. U have some
column for which every row will have a not null value??
-Anoop-
________________________________________
From: tgh [guanhua.tian@ia.ac.cn]
Sent: Thursday, December 27, 2012 1:45 PM
To: user@hbase.apache.org
Subject: which API is to get table meta data in hbase
Hi
I try to use API of hbase, and I store a big table in hbase,
But I have not find how to get the row number of this big table,
I say ,
AggregationClient#rowCount() is to statistic the Number of given
Family and given col in a table, and it scan the related region to satistic
the Number, that is, some colume can be null, is it,
But I say, there need some meta data which record how many row
number in the give table, I say , hbase has this meta data, is it,
And which API is to get it, and how to use API,
Could you help me
Thank you
---------------------
Tian Guanhua=
RE: which API is to get table meta data in hbase
Posted by Anoop Sam John <an...@huawei.com>.
> But I say, there need some meta data which record how many row
number in the give table, I say , hbase has this meta data, is it,
And which API is to get it, and how to use API,
There is no such meta data for a table.
You can check whether you can do this work on your own using co processors. There were some discussion on this in the mailing list. Sorry I am not able find the links for that now. You can create another table and store the stats there in the second table.
But see whether you can make use of the AggregationClient API. U have some column for which every row will have a not null value??
-Anoop-
________________________________________
From: tgh [guanhua.tian@ia.ac.cn]
Sent: Thursday, December 27, 2012 1:45 PM
To: user@hbase.apache.org
Subject: which API is to get table meta data in hbase
Hi
I try to use API of hbase, and I store a big table in hbase,
But I have not find how to get the row number of this big table,
I say ,
AggregationClient#rowCount() is to statistic the Number of given
Family and given col in a table, and it scan the related region to satistic
the Number, that is, some colume can be null, is it,
But I say, there need some meta data which record how many row
number in the give table, I say , hbase has this meta data, is it,
And which API is to get it, and how to use API,
Could you help me
Thank you
---------------------
Tian Guanhua
which API is to get table meta data in hbase
Posted by tgh <gu...@ia.ac.cn>.
Hi
I try to use API of hbase, and I store a big table in hbase,
But I have not find how to get the row number of this big table,
I say ,
AggregationClient#rowCount() is to statistic the Number of given
Family and given col in a table, and it scan the related region to satistic
the Number, that is, some colume can be null, is it,
But I say, there need some meta data which record how many row
number in the give table, I say , hbase has this meta data, is it,
And which API is to get it, and how to use API,
Could you help me
Thank you
---------------------
Tian Guanhua
答复: how to use API to statistic how many message has been store in the table in hbase
Posted by tgh <gu...@ia.ac.cn>.
Meanwhile , I see , the AggregationClient#rowCount() is to statistic the
Number of given Family and given col in a table, and it scan the related
region to satistic the Number, that is, some colume can be null, is it,
But If I want to know the row number of table, no matter have data there, I
just want to know the row number of the whole table, and it is a large
table,
And which API should I use, and how to use it,
Could you help me
Thank you
---------------------
Tian Guanhua
-----邮件原件-----
发件人: user-return-32780-guanhua.tian=ia.ac.cn@hbase.apache.org
[mailto:user-return-32780-guanhua.tian=ia.ac.cn@hbase.apache.org] 代表 Anoop
Sam John
发送时间: 2012年12月27日 13:31
收件人: user@hbase.apache.org
主题: RE: how to use API to statistic how many message has been store in the
table in hbase
So you want to know the no# of rows in a table?
Have a look at AggregationClient#rowCount()
-Anoop-
________________________________________
From: tgh [guanhua.tian@ia.ac.cn]
Sent: Thursday, December 27, 2012 7:51 AM
To: user@hbase.apache.org
Subject: how to use API to statistic how many message has been store in the
table in hbase
Hi
I try to use hbaseAPI to store data, and I want to get the number of
the message stored in the table in HBase,
how to use API to statistic how many message has been store in the
table in hbase
What API can do this,
Could you help me
Thank you
---------------------
Tian Guanhua=
答复: how to use API to statistic how many message has been store in the table in hbase
Posted by tgh <gu...@ia.ac.cn>.
Thank you for your reply
And I AggregationClient#rowCount()
But I am not familiar with these API ,and could you help me
Thank you
---------------------------
Tian Guanhua
----------table schema -----------
Name 'sms'
byte[] family1 = Bytes.toBytes("smsdata");
byte[] family2 = Bytes.toBytes("smsdata");
byte[] col1 = Bytes.toBytes("smsdata");
byte[] col2 = Bytes.toBytes("dndata");
--------- code --------------
package com.pattek.sms.hb;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.KeyValue;
import org.apache.hadoop.hbase.client.Get;
import org.apache.hadoop.hbase.client.Scan;
import org.apache.hadoop.hbase.client.HTable;
import org.apache.hadoop.hbase.client.Put;
import org.apache.hadoop.hbase.client.Result;
import org.apache.hadoop.hbase.client.ResultScanner;
import org.apache.hadoop.hbase.util.Bytes;
import org.apache.hadoop.hbase.client.coprocessor.AggregationClient;
import org.apache.log4j.Logger;
import org.apache.log4j.PropertyConfigurator;
import com.pattek.sms.SmsBase;
public class HbOpr1 extends SmsBase {
public static void main(String[] args){
// TODO Auto-generated method stub
Configuration hbase_config = new Configuration();
hbase_config.set("hbase.zookeeper.quorum",
"192.168.76.233");
HTable table = null;
//System.out.println("hello");
try {
// log.info(String.format("Connect To HBase %s",
args));
table = new
HTable(HBaseConfiguration.create(hbase_config), "sms");
System.out.println("Connect To HBase Completely");
} catch (Exception e1) {
e1.printStackTrace();
System.out.println("wrong1");
//SetError();
}
byte[] TABLE_NAME = Bytes.toBytes("sms");
byte[] CF = Bytes.toBytes("smsdata");
//byte[] CF1 = Bytes.toBytes("smsdata");
Configuration configuration =
HBaseConfiguration.create();
configuration.setLong("hbase.client.scanner.caching",
1000);
AggregationClient aggregationClient = new
AggregationClient( configuration);
Scan scan = new Scan();
byte[] startrow =
Bytes.toBytes("20121221040000000000009651550307");
byte[] endrow =
Bytes.toBytes("20121221040000000000009651550308");
scan.setStartRow(startrow);
scan.setStopRow(endrow);
//scan.addColumn(CF, Bytes.toBytes("smsdata"));
scan.addFamily(CF);
scan.addColumn(CF, Bytes.toBytes("smsdata"));
//System.out.println("**************");
long startime=System.currentTimeMillis();
try{
System.out.println("row count is " +
aggregationClient.rowCount(TABLE_NAME, null, scan));
} catch (Throwable e3) {
e3.printStackTrace();
System.out.println("wrong3");
}
long endtime=System.currentTimeMillis();
System.out.println("\rElapsed
time:"+(endtime-startime)+" mini seconds ");
}
}
-------- code --------------
------------ERROR ----------------
Jul 11, 2012 9:49:32 AM org.apache.zookeeper.Environment logEnv
INFO: Client environment:zookeeper.version=3.4.5-1392090, built on
09/30/2012 17:52 GMT
Jul 11, 2012 9:49:32 AM org.apache.zookeeper.Environment logEnv
INFO: Client environment:host.name=fnode2
Jul 11, 2012 9:49:32 AM org.apache.zookeeper.Environment logEnv
INFO: Client environment:java.version=1.7.0_09
Jul 11, 2012 9:49:32 AM org.apache.zookeeper.Environment logEnv
INFO: Client environment:java.vendor=Oracle Corporation
Jul 11, 2012 9:49:32 AM org.apache.zookeeper.Environment logEnv
INFO: Client environment:java.home=/home/jdk1.7.0_09/jre
Jul 11, 2012 9:49:32 AM org.apache.zookeeper.Environment logEnv
INFO: Client
environment:java.class.path=/home/tianguanhua1/smslx/bin:/home/tianguanhua1/
smslx/lib/apache-solr-core-4.0.0.jar:/home/tianguanhua1/smslx/lib/apache-sol
r-solrj-4.0.0.jar:/home/tianguanhua1/smslx/lib/mchange-commons-java-0.2.3.ja
r:/home/tianguanhua1/smslx/lib/c3p0-0.9.2-pre5.jar:/home/tianguanhua1/smslx/
lib/c3p0-oracle-thin-extras-0.9.2-pre5.jar:/home/tianguanhua1/smslx/lib/comm
ons-beanutils-1.7.0.jar:/home/tianguanhua1/smslx/lib/commons-beanutils-core-
1.8.0.jar:/home/tianguanhua1/smslx/lib/commons-cli-1.2.jar:/home/tianguanhua
1/smslx/lib/commons-codec-1.4.jar:/home/tianguanhua1/smslx/lib/commons-colle
ctions-3.2.1.jar:/home/tianguanhua1/smslx/lib/commons-configuration-1.6.jar:
/home/tianguanhua1/smslx/lib/commons-daemon-1.0.1.jar:/home/tianguanhua1/sms
lx/lib/commons-dbutils-1.5.jar:/home/tianguanhua1/smslx/lib/commons-digester
-1.8.jar:/home/tianguanhua1/smslx/lib/commons-el-1.0.jar:/home/tianguanhua1/
smslx/lib/commons-httpclient-3.0.1.jar:/home/tianguanhua1/smslx/lib/commons-
io-2.1.jar:/home/tianguanhua1/smslx/lib/commons-lang-2.4.jar:/home/tianguanh
ua1/smslx/lib/commons-logging-1.1.1.jar:/home/tianguanhua1/smslx/lib/commons
-logging-api-1.0.4.jar:/home/tianguanhua1/smslx/lib/commons-math-2.1.jar:/ho
me/tianguanhua1/smslx/lib/commons-net-3.1.jar:/home/tianguanhua1/smslx/lib/c
ommons-pool-1.6.jar:/home/tianguanhua1/smslx/lib/guava-r09-jarjar.jar:/home/
tianguanhua1/smslx/lib/hadoop-core-1.1.1.jar:/home/tianguanhua1/smslx/lib/hb
ase-0.94.3.jar:/home/tianguanhua1/smslx/lib/httpclient-4.1.3.jar:/home/tiang
uanhua1/smslx/lib/httpcore-4.1.4.jar:/home/tianguanhua1/smslx/lib/httpmime-4
.1.3.jar:/home/tianguanhua1/smslx/lib/jcl-over-slf4j-1.6.4.jar:/home/tiangua
nhua1/smslx/lib/jna.jar:/home/tianguanhua1/smslx/lib/junit-4.11-beta-1.jar:/
home/tianguanhua1/smslx/lib/log4j-1.2.14.jar:/home/tianguanhua1/smslx/lib/lo
g4j-over-slf4j-1.6.4.jar:/home/tianguanhua1/smslx/lib/lucene-analyzers-commo
n-4.0.0.jar:/home/tianguanhua1/smslx/lib/lucene-core-4.0.0.jar:/home/tiangua
nhua1/smslx/lib/lucene-queryparser-4.0.0.jar:/home/tianguanhua1/smslx/lib/my
sql-connector-java-5.1.12-bin.jar:/home/tianguanhua1/smslx/lib/mysql-connect
or-java-5.1.22-bin.jar:/home/tianguanhua1/smslx/lib/ojdbc14.jar:/home/tiangu
anhua1/smslx/lib/protobuf-java-2.4.1.jar:/home/tianguanhua1/smslx/lib/slf4j-
api-1.6.4.jar:/home/tianguanhua1/smslx/lib/slf4j-jdk14-1.6.4.jar:/home/tiang
uanhua1/smslx/lib/zookeeper-3.4.5.jar:/home/liuxin/jdk1.7.0_09/jre/lib/alt-r
t.jar:/home/liuxin/jdk1.7.0_09/jre/lib/charsets.jar:/home/liuxin/jdk1.7.0_09
/jre/lib/deploy.jar:/home/liuxin/jdk1.7.0_09/jre/lib/javaws.jar:/home/liuxin
/jdk1.7.0_09/jre/lib/jce.jar:/home/liuxin/jdk1.7.0_09/jre/lib/jfr.jar:/home/
liuxin/jdk1.7.0_09/jre/lib/jfxrt.jar:/home/liuxin/jdk1.7.0_09/jre/lib/jsse.j
ar:/home/liuxin/jdk1.7.0_09/jre/lib/management-agent.jar:/home/liuxin/jdk1.7
.0_09/jre/lib/plugin.jar:/home/liuxin/jdk1.7.0_09/jre/lib/resources.jar:/hom
e/liuxin/jdk1.7.0_09/jre/lib/rt.jar
Jul 11, 2012 9:49:32 AM org.apache.zookeeper.Environment logEnv
INFO: Client
environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64
:/lib:/usr/lib
Jul 11, 2012 9:49:32 AM org.apache.zookeeper.Environment logEnv
INFO: Client environment:java.io.tmpdir=/tmp
Jul 11, 2012 9:49:32 AM org.apache.zookeeper.Environment logEnv
INFO: Client environment:java.compiler=<NA>
Jul 11, 2012 9:49:32 AM org.apache.zookeeper.Environment logEnv
INFO: Client environment:os.name=Linux
Jul 11, 2012 9:49:32 AM org.apache.zookeeper.Environment logEnv
INFO: Client environment:os.arch=amd64
Jul 11, 2012 9:49:32 AM org.apache.zookeeper.Environment logEnv
INFO: Client environment:os.version=2.6.32-279.el6.x86_64
Jul 11, 2012 9:49:32 AM org.apache.zookeeper.Environment logEnv
INFO: Client environment:user.name=root
Jul 11, 2012 9:49:32 AM org.apache.zookeeper.Environment logEnv
INFO: Client environment:user.home=/root
Jul 11, 2012 9:49:32 AM org.apache.zookeeper.Environment logEnv
INFO: Client environment:user.dir=/home/tianguanhua1/smslx
Jul 11, 2012 9:49:32 AM org.apache.zookeeper.ZooKeeper <init>
INFO: Initiating client connection, connectString=192.168.76.233:2181
sessionTimeout=180000 watcher=hconnection
Jul 11, 2012 9:49:32 AM
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper <init>
INFO: The identifier of this process is 37575@fnode2
Jul 11, 2012 9:49:32 AM org.apache.zookeeper.ClientCnxn$SendThread
logStartConnect
INFO: Opening socket connection to server blade1/192.168.76.233:2181. Will
not attempt to authenticate using SASL (unknown error)
Jul 11, 2012 9:49:32 AM org.apache.zookeeper.ClientCnxn$SendThread
primeConnection
INFO: Socket connection established to blade1/192.168.76.233:2181,
initiating session
Jul 11, 2012 9:49:32 AM org.apache.zookeeper.ClientCnxn$SendThread
onConnected
INFO: Session establishment complete on server blade1/192.168.76.233:2181,
sessionid = 0x139ae6ded420116, negotiated timeout = 40000
Connect To HBase Completely
Jul 11, 2012 9:49:33 AM org.apache.zookeeper.ZooKeeper <init>
INFO: Initiating client connection, connectString=localhost:2181
sessionTimeout=180000 watcher=hconnection
Jul 11, 2012 9:49:33 AM
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper <init>
INFO: The identifier of this process is 37575@fnode2
Jul 11, 2012 9:49:33 AM org.apache.zookeeper.ClientCnxn$SendThread
logStartConnect
INFO: Opening socket connection to server localhost/127.0.0.1:2181. Will not
attempt to authenticate using SASL (unknown error)
Jul 11, 2012 9:49:33 AM org.apache.zookeeper.ClientCnxn$SendThread run
WARNING: Session 0x0 for server null, unexpected error, closing socket
connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:692)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.jav
a:350)
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
-----邮件原件-----
发件人: user-return-32780-guanhua.tian=ia.ac.cn@hbase.apache.org
[mailto:user-return-32780-guanhua.tian=ia.ac.cn@hbase.apache.org] 代表 Anoop
Sam John
发送时间: 2012年12月27日 13:31
收件人: user@hbase.apache.org
主题: RE: how to use API to statistic how many message has been store in the
table in hbase
So you want to know the no# of rows in a table?
Have a look at AggregationClient#rowCount()
-Anoop-
________________________________________
From: tgh [guanhua.tian@ia.ac.cn]
Sent: Thursday, December 27, 2012 7:51 AM
To: user@hbase.apache.org
Subject: how to use API to statistic how many message has been store in the
table in hbase
Hi
I try to use hbaseAPI to store data, and I want to get the number of
the message stored in the table in HBase,
how to use API to statistic how many message has been store in the
table in hbase
What API can do this,
Could you help me
Thank you
---------------------
Tian Guanhua=
RE: how to use API to statistic how many message has been store in
the table in hbase
Posted by Anoop Sam John <an...@huawei.com>.
So you want to know the no# of rows in a table?
Have a look at AggregationClient#rowCount()
-Anoop-
________________________________________
From: tgh [guanhua.tian@ia.ac.cn]
Sent: Thursday, December 27, 2012 7:51 AM
To: user@hbase.apache.org
Subject: how to use API to statistic how many message has been store in the table in hbase
Hi
I try to use hbaseAPI to store data, and I want to get the number of
the message stored in the table in HBase,
how to use API to statistic how many message has been store in the
table in hbase
What API can do this,
Could you help me
Thank you
---------------------
Tian Guanhua