You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by "Buttler, David" <bu...@llnl.gov> on 2010/08/09 19:02:52 UTC
hbase error caused by DFS timeout
Hi all,
I seem to get this error far too frequently:
2010-08-09 09:54:03,685 INFO org.apache.hadoop.hbase.regionserver.Store: Started compaction of 1 file(s) in annotations of doc,293817e024ed1d54a11e9e7c9b836dd837badbbc,1281372823189, hasReferences=true, into /hbase/doc/compaction.dir/237345967, seqid=1182913218
2010-08-09 09:54:03,784 WARN org.apache.hadoop.hdfs.DFSClient: DFSOutputStream ResponseProcessor exception for block blk_-4556852958383799371_431518java.net.SocketTimeoutException: 6000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/10.220.5.35:49924 remote=/10.220.5.14:50010]
at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at java.io.DataInputStream.readFully(DataInputStream.java:178)
at java.io.DataInputStream.readLong(DataInputStream.java:399)
at org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.readFields(DataTransferProtocol.java:119)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2424)
This basically is taking one of clusters down consistently. Is there an obvious thing I can do about this?
I have seen this across three different clusters with radically different hardware, leading me to believe that I have misconfigured something in either hbase or hdfs
Any ideas of where to look?
Thanks,
Dave
RE: hbase error caused by DFS timeout
Posted by "Buttler, David" <bu...@llnl.gov>.
Thanks Ryan.
I actually already had this in the hbase-site.xml, but somehow missed putting it in the core-site.xml.
The cluster seems to be much happier.
Dave
-----Original Message-----
From: Ryan Rawson [mailto:ryanobjc@gmail.com]
Sent: Monday, August 09, 2010 12:01 PM
To: user@hbase.apache.org
Subject: Re: hbase error caused by DFS timeout
Try this config:
<property>
<name>dfs.datanode.socket.write.timeout</name>
<value>0</value>
</property>
in both hbase-site.xml and core-site.xml in the hadoop configs.
-ryan
On Mon, Aug 9, 2010 at 10:02 AM, Buttler, David <bu...@llnl.gov> wrote:
> Hi all,
> I seem to get this error far too frequently:
>
> 2010-08-09 09:54:03,685 INFO org.apache.hadoop.hbase.regionserver.Store: Started compaction of 1 file(s) in annotations of doc,293817e024ed1d54a11e9e7c9b836dd837badbbc,1281372823189, hasReferences=true, into /hbase/doc/compaction.dir/237345967, seqid=1182913218
> 2010-08-09 09:54:03,784 WARN org.apache.hadoop.hdfs.DFSClient: DFSOutputStream ResponseProcessor exception for block blk_-4556852958383799371_431518java.net.SocketTimeoutException: 6000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/10.220.5.35:49924 remote=/10.220.5.14:50010]
> at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
> at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
> at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
> at java.io.DataInputStream.readFully(DataInputStream.java:178)
> at java.io.DataInputStream.readLong(DataInputStream.java:399)
> at org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.readFields(DataTransferProtocol.java:119)
> at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2424)
>
> This basically is taking one of clusters down consistently. Is there an obvious thing I can do about this?
> I have seen this across three different clusters with radically different hardware, leading me to believe that I have misconfigured something in either hbase or hdfs
>
> Any ideas of where to look?
>
> Thanks,
> Dave
>
RE: how many regions a regionserver can support
Posted by Jonathan Gray <jg...@facebook.com>.
Again, the read/write load has much more to do with cluster sizing than the dataset (total capacity aside).
To give you an idea of how widely it varies, I had a client who put several hundred GBs of data onto a single node setup of HBase. I've also seen clusters of 20-100 nodes with only 10s of GBs on it (very high concurrent write load). Recently I've been playing with a 100 node cluster with about 20TB of data on it (before replication).
Each of these clusters had very different load profiles. And node count is not the only important metric. That one node cluster was a pair of 2TB disks while these 100 node clusters are packed with 12 1TB disks per node.
JG
> -----Original Message-----
> From: Jinsong Hu [mailto:jinsong_hu@hotmail.com]
> Sent: Wednesday, September 01, 2010 11:22 AM
> To: user@hbase.apache.org
> Subject: Re: how many regions a regionserver can support
>
> I did a testing with 6 regionserver cluster with a key design that
> spread
> the incoming data to all regions.
> I noticed after pumping data for 3-4 days for about 3 TB data, one of
> the
> regionserver shuts down because
> of channel IO error. on a 3 regionserver cluster and same key design,
> the
> regionservers shuts down after only
> 45G data insertion.
>
> I notice that if the key is designed so that it doesn't spread to all
> regions, but only to small portion of regions and that
> portion of regions spread approximately evenly among all regionservers,
> then
> the HDFS size becomes the limit of
> the total number of regions that can be supported and I don't run into
> this
> IO issue.
>
> Can any body show us the actual example of the hbase data size and
> cluster
> size ?
>
> Jimmy.
>
> --------------------------------------------------
> From: "Jonathan Gray" <jg...@facebook.com>
> Sent: Friday, August 27, 2010 10:55 AM
> To: <us...@hbase.apache.org>
> Subject: RE: how many regions a regionserver can support
>
> > There is no fixed limit, it has much more to do with the read/write
> load
> > than the actual dataset size.
> >
> > HBase is usually fine having very densely packed RegionServers, if
> much of
> > the data is rarely accessed. If you have extremely high numbers of
> > regions per server and you are writing to all of these regions, or
> even
> > reading from all of them, you could have issues. Though storage
> capacity
> > needs to be considered, capacity planning often has much more to do
> with
> > how much memory you need to support the read/write load you expect.
> Reads
> > mostly from a performance POV but for writes, there are some
> important
> > considerations related to the number of regions per server (and thus
> data
> > density and determining your max region size).
> >
> > In any case, you should probably increase your max size to 1GB or so
> and
> > can go higher if necessary.
> >
> > JG
> >
> >> -----Original Message-----
> >> From: Jinsong Hu [mailto:jinsong_hu@hotmail.com]
> >> Sent: Friday, August 27, 2010 10:03 AM
> >> To: user@hbase.apache.org
> >> Subject: how many regions a regionserver can support
> >>
> >> Hi, There :
> >> Does anybody know how many region a regionserver can support ? I
> >> have
> >> regionservers with 8G ram and 1.5T disk and 4 core CPU.
> >> I searched http://www.facebook.com/note.php?note_id=142473677002 and
> >> they
> >> say google target is 100 regions of 200M for each
> >> regionserver.
> >> In my case, I have 2700 regions spread to 6 regionservers. each
> >> region is
> >> set to default size of 256M . and it seems it is still running fine.
> I
> >> am
> >> running CDH3. I just wonder what is the upper limit so that I can
> do
> >> capacity planning. Does anybody know this ?
> >>
> >> Jimmy.
> >
> >
Re: how many regions a regionserver can support
Posted by Scott Whitecross <sw...@gmail.com>.
"be sureto compress your data and set the split size bigger than the default
of 256MB or you'll end up with too many regions."
How many regions are to many? I have a decent sized cluster (~30 nodes) and
started inserting new data, and noticed that after a day, I went from 30
regions on each server to 60. That is using the default region size. I
haven't tested increasing the region file sizes, as I'm concerned about
performance scanning data.
On Wed, Sep 1, 2010 at 2:35 PM, Jean-Daniel Cryans <jd...@apache.org>wrote:
> Is that really a good test? Unless you are planning to write about 1TB
> of new data per day into HBase I don't see how you are testing
> capacity, you're more likely testing how HBase can sustain a constant
> import of a lot of data. Regarding that, I'd be interested in knowing
> exactly the circumstances of the region server failure.
>
> Regarding real life example, one of our cluster has about 2.5TB of
> LZOed data (not sure about the raw size) according to dfs -du, on 20
> nodes (FWIW). When trying to reach high density on your nodes, be sure
> to compress your data and set the split size bigger than the default
> of 256MB or you'll end up with too many regions.
>
> J-D
>
> On Wed, Sep 1, 2010 at 11:21 AM, Jinsong Hu <ji...@hotmail.com>
> wrote:
> > I did a testing with 6 regionserver cluster with a key design that spread
> > the incoming data to all regions.
> > I noticed after pumping data for 3-4 days for about 3 TB data, one of the
> > regionserver shuts down because
> > of channel IO error. on a 3 regionserver cluster and same key design,
> the
> > regionservers shuts down after only
> > 45G data insertion.
> >
> > I notice that if the key is designed so that it doesn't spread to all
> > regions, but only to small portion of regions and that
> > portion of regions spread approximately evenly among all regionservers,
> then
> > the HDFS size becomes the limit of
> > the total number of regions that can be supported and I don't run into
> this
> > IO issue.
> >
> > Can any body show us the actual example of the hbase data size and
> cluster
> > size ?
> >
> > Jimmy.
> >
> > --------------------------------------------------
> > From: "Jonathan Gray" <jg...@facebook.com>
> > Sent: Friday, August 27, 2010 10:55 AM
> > To: <us...@hbase.apache.org>
> > Subject: RE: how many regions a regionserver can support
> >
> >> There is no fixed limit, it has much more to do with the read/write load
> >> than the actual dataset size.
> >>
> >> HBase is usually fine having very densely packed RegionServers, if much
> of
> >> the data is rarely accessed. If you have extremely high numbers of
> regions
> >> per server and you are writing to all of these regions, or even reading
> from
> >> all of them, you could have issues. Though storage capacity needs to be
> >> considered, capacity planning often has much more to do with how much
> memory
> >> you need to support the read/write load you expect. Reads mostly from a
> >> performance POV but for writes, there are some important considerations
> >> related to the number of regions per server (and thus data density and
> >> determining your max region size).
> >>
> >> In any case, you should probably increase your max size to 1GB or so and
> >> can go higher if necessary.
> >>
> >> JG
> >>
> >>> -----Original Message-----
> >>> From: Jinsong Hu [mailto:jinsong_hu@hotmail.com]
> >>> Sent: Friday, August 27, 2010 10:03 AM
> >>> To: user@hbase.apache.org
> >>> Subject: how many regions a regionserver can support
> >>>
> >>> Hi, There :
> >>> Does anybody know how many region a regionserver can support ? I
> >>> have
> >>> regionservers with 8G ram and 1.5T disk and 4 core CPU.
> >>> I searched http://www.facebook.com/note.php?note_id=142473677002 and
> >>> they
> >>> say google target is 100 regions of 200M for each
> >>> regionserver.
> >>> In my case, I have 2700 regions spread to 6 regionservers. each
> >>> region is
> >>> set to default size of 256M . and it seems it is still running fine. I
> >>> am
> >>> running CDH3. I just wonder what is the upper limit so that I can do
> >>> capacity planning. Does anybody know this ?
> >>>
> >>> Jimmy.
> >>
> >>
> >
>
Re: how many regions a regionserver can support
Posted by Jinsong Hu <ji...@hotmail.com>.
Yes, I am indeed testing the sustained rate. the channel I/O exception shows
the I/O killed the regionserver.
the data node side shows:
2010-08-28 23:46:27,854 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Ex
ception in receiveBlock for block blk_7209586757797236713_2442298
java.io.Interr
uptedIOException: Interruped while waiting for IO on channel
java.nio.channels.S
ocketChannel[connected local=/10.110.24.89:50010
remote=/10.110.24.89:42524]. 0
millis timeout left.
the regionserver side shows:
2010-08-28 23:47:13,148 WARN org.apache.hadoop.hdfs.DFSClient:
DFSOutputStream R
esponseProcessor exception for block
blk_7209586757797236713_2442298java.io.EOF
Exception
I agree that if the insertion rate is slower, we will support more data in
hbase. In this case,
I do want to stress test the hbase and see what is the limit. Our
application continuously collects
data from network and insert to hbase, and I want to see what happens during
the extreme cases.
it looks channel I/O doesn't become bottleneck under such stress test.
dfs -dus shows we have 1.17 TB of data when one of the regionserver crashed.
the data
is gzip compressed as I found that gzip compression actually gives better
writing rate.
I may test larger region size later. Previous test with 2 GB also cause lots
of I/O and
finally hbase regionserver crashed too.
Jimmy.
--------------------------------------------------
From: "Jean-Daniel Cryans" <jd...@apache.org>
Sent: Wednesday, September 01, 2010 11:35 AM
To: <us...@hbase.apache.org>
Subject: Re: how many regions a regionserver can support
> Is that really a good test? Unless you are planning to write about 1TB
> of new data per day into HBase I don't see how you are testing
> capacity, you're more likely testing how HBase can sustain a constant
> import of a lot of data. Regarding that, I'd be interested in knowing
> exactly the circumstances of the region server failure.
>
> Regarding real life example, one of our cluster has about 2.5TB of
> LZOed data (not sure about the raw size) according to dfs -du, on 20
> nodes (FWIW). When trying to reach high density on your nodes, be sure
> to compress your data and set the split size bigger than the default
> of 256MB or you'll end up with too many regions.
>
> J-D
>
> On Wed, Sep 1, 2010 at 11:21 AM, Jinsong Hu <ji...@hotmail.com>
> wrote:
>> I did a testing with 6 regionserver cluster with a key design that spread
>> the incoming data to all regions.
>> I noticed after pumping data for 3-4 days for about 3 TB data, one of the
>> regionserver shuts down because
>> of channel IO error. on a 3 regionserver cluster and same key design,
>> the
>> regionservers shuts down after only
>> 45G data insertion.
>>
>> I notice that if the key is designed so that it doesn't spread to all
>> regions, but only to small portion of regions and that
>> portion of regions spread approximately evenly among all regionservers,
>> then
>> the HDFS size becomes the limit of
>> the total number of regions that can be supported and I don't run into
>> this
>> IO issue.
>>
>> Can any body show us the actual example of the hbase data size and
>> cluster
>> size ?
>>
>> Jimmy.
>>
>> --------------------------------------------------
>> From: "Jonathan Gray" <jg...@facebook.com>
>> Sent: Friday, August 27, 2010 10:55 AM
>> To: <us...@hbase.apache.org>
>> Subject: RE: how many regions a regionserver can support
>>
>>> There is no fixed limit, it has much more to do with the read/write load
>>> than the actual dataset size.
>>>
>>> HBase is usually fine having very densely packed RegionServers, if much
>>> of
>>> the data is rarely accessed. If you have extremely high numbers of
>>> regions
>>> per server and you are writing to all of these regions, or even reading
>>> from
>>> all of them, you could have issues. Though storage capacity needs to be
>>> considered, capacity planning often has much more to do with how much
>>> memory
>>> you need to support the read/write load you expect. Reads mostly from a
>>> performance POV but for writes, there are some important considerations
>>> related to the number of regions per server (and thus data density and
>>> determining your max region size).
>>>
>>> In any case, you should probably increase your max size to 1GB or so and
>>> can go higher if necessary.
>>>
>>> JG
>>>
>>>> -----Original Message-----
>>>> From: Jinsong Hu [mailto:jinsong_hu@hotmail.com]
>>>> Sent: Friday, August 27, 2010 10:03 AM
>>>> To: user@hbase.apache.org
>>>> Subject: how many regions a regionserver can support
>>>>
>>>> Hi, There :
>>>> Does anybody know how many region a regionserver can support ? I
>>>> have
>>>> regionservers with 8G ram and 1.5T disk and 4 core CPU.
>>>> I searched http://www.facebook.com/note.php?note_id=142473677002 and
>>>> they
>>>> say google target is 100 regions of 200M for each
>>>> regionserver.
>>>> In my case, I have 2700 regions spread to 6 regionservers. each
>>>> region is
>>>> set to default size of 256M . and it seems it is still running fine. I
>>>> am
>>>> running CDH3. I just wonder what is the upper limit so that I can do
>>>> capacity planning. Does anybody know this ?
>>>>
>>>> Jimmy.
>>>
>>>
>>
>
Re: how many regions a regionserver can support
Posted by Jean-Daniel Cryans <jd...@apache.org>.
Is that really a good test? Unless you are planning to write about 1TB
of new data per day into HBase I don't see how you are testing
capacity, you're more likely testing how HBase can sustain a constant
import of a lot of data. Regarding that, I'd be interested in knowing
exactly the circumstances of the region server failure.
Regarding real life example, one of our cluster has about 2.5TB of
LZOed data (not sure about the raw size) according to dfs -du, on 20
nodes (FWIW). When trying to reach high density on your nodes, be sure
to compress your data and set the split size bigger than the default
of 256MB or you'll end up with too many regions.
J-D
On Wed, Sep 1, 2010 at 11:21 AM, Jinsong Hu <ji...@hotmail.com> wrote:
> I did a testing with 6 regionserver cluster with a key design that spread
> the incoming data to all regions.
> I noticed after pumping data for 3-4 days for about 3 TB data, one of the
> regionserver shuts down because
> of channel IO error. on a 3 regionserver cluster and same key design, the
> regionservers shuts down after only
> 45G data insertion.
>
> I notice that if the key is designed so that it doesn't spread to all
> regions, but only to small portion of regions and that
> portion of regions spread approximately evenly among all regionservers, then
> the HDFS size becomes the limit of
> the total number of regions that can be supported and I don't run into this
> IO issue.
>
> Can any body show us the actual example of the hbase data size and cluster
> size ?
>
> Jimmy.
>
> --------------------------------------------------
> From: "Jonathan Gray" <jg...@facebook.com>
> Sent: Friday, August 27, 2010 10:55 AM
> To: <us...@hbase.apache.org>
> Subject: RE: how many regions a regionserver can support
>
>> There is no fixed limit, it has much more to do with the read/write load
>> than the actual dataset size.
>>
>> HBase is usually fine having very densely packed RegionServers, if much of
>> the data is rarely accessed. If you have extremely high numbers of regions
>> per server and you are writing to all of these regions, or even reading from
>> all of them, you could have issues. Though storage capacity needs to be
>> considered, capacity planning often has much more to do with how much memory
>> you need to support the read/write load you expect. Reads mostly from a
>> performance POV but for writes, there are some important considerations
>> related to the number of regions per server (and thus data density and
>> determining your max region size).
>>
>> In any case, you should probably increase your max size to 1GB or so and
>> can go higher if necessary.
>>
>> JG
>>
>>> -----Original Message-----
>>> From: Jinsong Hu [mailto:jinsong_hu@hotmail.com]
>>> Sent: Friday, August 27, 2010 10:03 AM
>>> To: user@hbase.apache.org
>>> Subject: how many regions a regionserver can support
>>>
>>> Hi, There :
>>> Does anybody know how many region a regionserver can support ? I
>>> have
>>> regionservers with 8G ram and 1.5T disk and 4 core CPU.
>>> I searched http://www.facebook.com/note.php?note_id=142473677002 and
>>> they
>>> say google target is 100 regions of 200M for each
>>> regionserver.
>>> In my case, I have 2700 regions spread to 6 regionservers. each
>>> region is
>>> set to default size of 256M . and it seems it is still running fine. I
>>> am
>>> running CDH3. I just wonder what is the upper limit so that I can do
>>> capacity planning. Does anybody know this ?
>>>
>>> Jimmy.
>>
>>
>
Re: how many regions a regionserver can support
Posted by Jinsong Hu <ji...@hotmail.com>.
I did a testing with 6 regionserver cluster with a key design that spread
the incoming data to all regions.
I noticed after pumping data for 3-4 days for about 3 TB data, one of the
regionserver shuts down because
of channel IO error. on a 3 regionserver cluster and same key design, the
regionservers shuts down after only
45G data insertion.
I notice that if the key is designed so that it doesn't spread to all
regions, but only to small portion of regions and that
portion of regions spread approximately evenly among all regionservers, then
the HDFS size becomes the limit of
the total number of regions that can be supported and I don't run into this
IO issue.
Can any body show us the actual example of the hbase data size and cluster
size ?
Jimmy.
--------------------------------------------------
From: "Jonathan Gray" <jg...@facebook.com>
Sent: Friday, August 27, 2010 10:55 AM
To: <us...@hbase.apache.org>
Subject: RE: how many regions a regionserver can support
> There is no fixed limit, it has much more to do with the read/write load
> than the actual dataset size.
>
> HBase is usually fine having very densely packed RegionServers, if much of
> the data is rarely accessed. If you have extremely high numbers of
> regions per server and you are writing to all of these regions, or even
> reading from all of them, you could have issues. Though storage capacity
> needs to be considered, capacity planning often has much more to do with
> how much memory you need to support the read/write load you expect. Reads
> mostly from a performance POV but for writes, there are some important
> considerations related to the number of regions per server (and thus data
> density and determining your max region size).
>
> In any case, you should probably increase your max size to 1GB or so and
> can go higher if necessary.
>
> JG
>
>> -----Original Message-----
>> From: Jinsong Hu [mailto:jinsong_hu@hotmail.com]
>> Sent: Friday, August 27, 2010 10:03 AM
>> To: user@hbase.apache.org
>> Subject: how many regions a regionserver can support
>>
>> Hi, There :
>> Does anybody know how many region a regionserver can support ? I
>> have
>> regionservers with 8G ram and 1.5T disk and 4 core CPU.
>> I searched http://www.facebook.com/note.php?note_id=142473677002 and
>> they
>> say google target is 100 regions of 200M for each
>> regionserver.
>> In my case, I have 2700 regions spread to 6 regionservers. each
>> region is
>> set to default size of 256M . and it seems it is still running fine. I
>> am
>> running CDH3. I just wonder what is the upper limit so that I can do
>> capacity planning. Does anybody know this ?
>>
>> Jimmy.
>
>
Re: truncate large table suggestion
Posted by Jinsong Hu <ji...@hotmail.com>.
unfortunately. I tried flush the table and disable, and then drop, and it
doesn't work.
I even wrote a utility to remove all records from the large table and then
do so,
and it doesn't work either. strangely. I looked at the web UI, and still see
many regions
even the number of rows in the table is 0.
the hbase that I use is CDH3 's latest version, which is hbase 0.89 version
released
in June 2010.
Jinsong
--------------------------------------------------
From: "Jean-Daniel Cryans" <jd...@apache.org>
Sent: Wednesday, September 01, 2010 11:36 AM
To: <us...@hbase.apache.org>
Subject: Re: truncate large table suggestion
> One trick is to pre- force flush the table. Also try out the new 0.89,
> it has 2 fixes regarding a race condition between the BaseScanner and
> the closing of regions. The release candidate is here
> http://people.apache.org/~jdcryans/hbase-0.89.20100830-candidate-1
>
> J-D
>
> On Wed, Sep 1, 2010 at 11:28 AM, Jinsong Hu <ji...@hotmail.com>
> wrote:
>> Hi, Team:
>> I have noticed that the truncate/drop table with large amount of data
>> fails
>> and actually corrupt the hbase. In the worse case, we can't even
>> create the table with the same name any more and I was forced to dump the
>> whole hbase records and recreate all tables again.
>> I noticed there is a rename command and I tried that one, it turns out
>> it
>> actually works. One problem it causes is that the hbase master run out of
>> memory after I run the command. but after restart it, I am able to
>> recreate
>> the table since the old table is renamed.
>> I then disable the renamed table and drop it, but it is not really
>> successful. the table disappears from the list command, but if I recreate
>> a
>> table with the new name, the shell says it exist already. the
>> regionserver
>> shows that they still have regions with the renamed table.
>> In the end, I have to write a utility to remove data from the renamed
>> table, I will try to see if this trick works although it is expensive.
>>
>>
>> Can the disable/truncate or drop for large table be fixed ?
>>
>> Jimmy.
>>
>
Re: truncate large table suggestion
Posted by Jean-Daniel Cryans <jd...@apache.org>.
That version doesn't have the fixes I referred to, and disabling large
tables will likely hit the race condition.
J-D
On Wed, Sep 1, 2010 at 2:47 PM, Jinsong Hu <ji...@hotmail.com> wrote:
> unfortunately. I tried flush the table and disable, and then drop, and it
> doesn't work.
> I even wrote a utility to remove all records from the large table and then
> do so,
> and it doesn't work either. strangely. I looked at the web UI, and still see
> many regions
> even the number of rows in the table is 0.
>
> the hbase that I use is CDH3 's latest version, which is hbase 0.89 version
> released
> in June 2010.
>
> Jinsong
>
>
> --------------------------------------------------
> From: "Jean-Daniel Cryans" <jd...@apache.org>
> Sent: Wednesday, September 01, 2010 11:36 AM
> To: <us...@hbase.apache.org>
> Subject: Re: truncate large table suggestion
>
>> One trick is to pre- force flush the table. Also try out the new 0.89,
>> it has 2 fixes regarding a race condition between the BaseScanner and
>> the closing of regions. The release candidate is here
>> http://people.apache.org/~jdcryans/hbase-0.89.20100830-candidate-1
>>
>> J-D
>>
>> On Wed, Sep 1, 2010 at 11:28 AM, Jinsong Hu <ji...@hotmail.com>
>> wrote:
>>>
>>> Hi, Team:
>>> I have noticed that the truncate/drop table with large amount of data
>>> fails
>>> and actually corrupt the hbase. In the worse case, we can't even
>>> create the table with the same name any more and I was forced to dump the
>>> whole hbase records and recreate all tables again.
>>> I noticed there is a rename command and I tried that one, it turns out
>>> it
>>> actually works. One problem it causes is that the hbase master run out of
>>> memory after I run the command. but after restart it, I am able to
>>> recreate
>>> the table since the old table is renamed.
>>> I then disable the renamed table and drop it, but it is not really
>>> successful. the table disappears from the list command, but if I recreate
>>> a
>>> table with the new name, the shell says it exist already. the
>>> regionserver
>>> shows that they still have regions with the renamed table.
>>> In the end, I have to write a utility to remove data from the renamed
>>> table, I will try to see if this trick works although it is expensive.
>>>
>>>
>>> Can the disable/truncate or drop for large table be fixed ?
>>>
>>> Jimmy.
>>>
>>
>
Re: truncate large table suggestion
Posted by Jean-Daniel Cryans <jd...@apache.org>.
One trick is to pre- force flush the table. Also try out the new 0.89,
it has 2 fixes regarding a race condition between the BaseScanner and
the closing of regions. The release candidate is here
http://people.apache.org/~jdcryans/hbase-0.89.20100830-candidate-1
J-D
On Wed, Sep 1, 2010 at 11:28 AM, Jinsong Hu <ji...@hotmail.com> wrote:
> Hi, Team:
> I have noticed that the truncate/drop table with large amount of data fails
> and actually corrupt the hbase. In the worse case, we can't even
> create the table with the same name any more and I was forced to dump the
> whole hbase records and recreate all tables again.
> I noticed there is a rename command and I tried that one, it turns out it
> actually works. One problem it causes is that the hbase master run out of
> memory after I run the command. but after restart it, I am able to recreate
> the table since the old table is renamed.
> I then disable the renamed table and drop it, but it is not really
> successful. the table disappears from the list command, but if I recreate a
> table with the new name, the shell says it exist already. the regionserver
> shows that they still have regions with the renamed table.
> In the end, I have to write a utility to remove data from the renamed
> table, I will try to see if this trick works although it is expensive.
>
>
> Can the disable/truncate or drop for large table be fixed ?
>
> Jimmy.
>
truncate large table suggestion
Posted by Jinsong Hu <ji...@hotmail.com>.
Hi, Team:
I have noticed that the truncate/drop table with large amount of data
fails and actually corrupt the hbase. In the worse case, we can't even
create the table with the same name any more and I was forced to dump the
whole hbase records and recreate all tables again.
I noticed there is a rename command and I tried that one, it turns out it
actually works. One problem it causes is that the hbase master run out of
memory after I run the command. but after restart it, I am able to recreate
the table since the old table is renamed.
I then disable the renamed table and drop it, but it is not really
successful. the table disappears from the list command, but if I recreate a
table with the new name, the shell says it exist already. the regionserver
shows that they still have regions with the renamed table.
In the end, I have to write a utility to remove data from the renamed
table, I will try to see if this trick works although it is expensive.
Can the disable/truncate or drop for large table be fixed ?
Jimmy.
Re: how many regions a regionserver can support
Posted by Scott Whitecross <sw...@gmail.com>.
Can you explain this a bit more? I thought one benefit to small regions was
increased performance pulling blocks? Is there an upper limit to the number
of regions per region server that is recommended?
On Sat, Aug 28, 2010 at 4:05 PM, Ryan Rawson <ry...@gmail.com> wrote:
> The only downside to having so many regions on a regionserver is
> opening and reassigning them is not as fast as you'd like.
>
> In the future with increased parallelism and other things we will be
> able to speed it up.
>
>
>
> On Fri, Aug 27, 2010 at 10:55 AM, Jonathan Gray <jg...@facebook.com>
> wrote:
> > There is no fixed limit, it has much more to do with the read/write load
> than the actual dataset size.
> >
> > HBase is usually fine having very densely packed RegionServers, if much
> of the data is rarely accessed. If you have extremely high numbers of
> regions per server and you are writing to all of these regions, or even
> reading from all of them, you could have issues. Though storage capacity
> needs to be considered, capacity planning often has much more to do with how
> much memory you need to support the read/write load you expect. Reads
> mostly from a performance POV but for writes, there are some important
> considerations related to the number of regions per server (and thus data
> density and determining your max region size).
> >
> > In any case, you should probably increase your max size to 1GB or so and
> can go higher if necessary.
> >
> > JG
> >
> >> -----Original Message-----
> >> From: Jinsong Hu [mailto:jinsong_hu@hotmail.com]
> >> Sent: Friday, August 27, 2010 10:03 AM
> >> To: user@hbase.apache.org
> >> Subject: how many regions a regionserver can support
> >>
> >> Hi, There :
> >> Does anybody know how many region a regionserver can support ? I
> >> have
> >> regionservers with 8G ram and 1.5T disk and 4 core CPU.
> >> I searched http://www.facebook.com/note.php?note_id=142473677002 and
> >> they
> >> say google target is 100 regions of 200M for each
> >> regionserver.
> >> In my case, I have 2700 regions spread to 6 regionservers. each
> >> region is
> >> set to default size of 256M . and it seems it is still running fine. I
> >> am
> >> running CDH3. I just wonder what is the upper limit so that I can do
> >> capacity planning. Does anybody know this ?
> >>
> >> Jimmy.
> >
> >
>
Re: how many regions a regionserver can support
Posted by Ryan Rawson <ry...@gmail.com>.
The only downside to having so many regions on a regionserver is
opening and reassigning them is not as fast as you'd like.
In the future with increased parallelism and other things we will be
able to speed it up.
On Fri, Aug 27, 2010 at 10:55 AM, Jonathan Gray <jg...@facebook.com> wrote:
> There is no fixed limit, it has much more to do with the read/write load than the actual dataset size.
>
> HBase is usually fine having very densely packed RegionServers, if much of the data is rarely accessed. If you have extremely high numbers of regions per server and you are writing to all of these regions, or even reading from all of them, you could have issues. Though storage capacity needs to be considered, capacity planning often has much more to do with how much memory you need to support the read/write load you expect. Reads mostly from a performance POV but for writes, there are some important considerations related to the number of regions per server (and thus data density and determining your max region size).
>
> In any case, you should probably increase your max size to 1GB or so and can go higher if necessary.
>
> JG
>
>> -----Original Message-----
>> From: Jinsong Hu [mailto:jinsong_hu@hotmail.com]
>> Sent: Friday, August 27, 2010 10:03 AM
>> To: user@hbase.apache.org
>> Subject: how many regions a regionserver can support
>>
>> Hi, There :
>> Does anybody know how many region a regionserver can support ? I
>> have
>> regionservers with 8G ram and 1.5T disk and 4 core CPU.
>> I searched http://www.facebook.com/note.php?note_id=142473677002 and
>> they
>> say google target is 100 regions of 200M for each
>> regionserver.
>> In my case, I have 2700 regions spread to 6 regionservers. each
>> region is
>> set to default size of 256M . and it seems it is still running fine. I
>> am
>> running CDH3. I just wonder what is the upper limit so that I can do
>> capacity planning. Does anybody know this ?
>>
>> Jimmy.
>
>
RE: how many regions a regionserver can support
Posted by Jonathan Gray <jg...@facebook.com>.
There is no fixed limit, it has much more to do with the read/write load than the actual dataset size.
HBase is usually fine having very densely packed RegionServers, if much of the data is rarely accessed. If you have extremely high numbers of regions per server and you are writing to all of these regions, or even reading from all of them, you could have issues. Though storage capacity needs to be considered, capacity planning often has much more to do with how much memory you need to support the read/write load you expect. Reads mostly from a performance POV but for writes, there are some important considerations related to the number of regions per server (and thus data density and determining your max region size).
In any case, you should probably increase your max size to 1GB or so and can go higher if necessary.
JG
> -----Original Message-----
> From: Jinsong Hu [mailto:jinsong_hu@hotmail.com]
> Sent: Friday, August 27, 2010 10:03 AM
> To: user@hbase.apache.org
> Subject: how many regions a regionserver can support
>
> Hi, There :
> Does anybody know how many region a regionserver can support ? I
> have
> regionservers with 8G ram and 1.5T disk and 4 core CPU.
> I searched http://www.facebook.com/note.php?note_id=142473677002 and
> they
> say google target is 100 regions of 200M for each
> regionserver.
> In my case, I have 2700 regions spread to 6 regionservers. each
> region is
> set to default size of 256M . and it seems it is still running fine. I
> am
> running CDH3. I just wonder what is the upper limit so that I can do
> capacity planning. Does anybody know this ?
>
> Jimmy.
how many regions a regionserver can support
Posted by Jinsong Hu <ji...@hotmail.com>.
Hi, There :
Does anybody know how many region a regionserver can support ? I have
regionservers with 8G ram and 1.5T disk and 4 core CPU.
I searched http://www.facebook.com/note.php?note_id=142473677002 and they
say google target is 100 regions of 200M for each
regionserver.
In my case, I have 2700 regions spread to 6 regionservers. each region is
set to default size of 256M . and it seems it is still running fine. I am
running CDH3. I just wonder what is the upper limit so that I can do
capacity planning. Does anybody know this ?
Jimmy.
Re: CDH3 has 2 versions of hbase executable, one work, another doesn't
Posted by Todd Lipcon <to...@cloudera.com>.
Hi Jimmy,
This is a known bug in our packaging:
https://issues.cloudera.org/browse/DISTRO-23
We'll resolve it in our next beta.
Thanks
-Todd
On Fri, Sep 3, 2010 at 4:31 PM, Jinsong Hu <ji...@hotmail.com> wrote:
> I noticed that CDH3 has 2 executable
>
> /usr/bin/hbase
>
> /usr/lib/hbase/bin/hbase
>
> I compared them and they are different. it turns out that I run
>
> /usr/bin/hbase shell
>
> and then list table, it works, but if I run
>
> /usr/lib/hbase/bin/hbase shell
>
> and list tables, it freezes. In the next distribution, please make
> /usr/lib/hbase/bin/hbase
> a softlink to /usr/bin/hbase , or the otherway. and make sure the
> executable works.
>
> Jimmy.
>
--
Todd Lipcon
Software Engineer, Cloudera
Re: CDH3 has 2 versions of hbase executable, one work, another doesn't
Posted by Edward Capriolo <ed...@gmail.com>.
Jimmy.
Many people across the forums seem to be confusing cdh with the
upstream hadoop/hive/hbase projects. In this case the problem is in
the cdh packaging and not in upstream hbase.
The proper place to take up packaging issues is with the packager.
When you are new to software this might be confusing.
To be clear hadoop, hive, and hbase are open source products that come
in a tar.gz. if you notice an issue in the core functionality that
would be appropriate for xxxx-user@apache , but if you are running
into a problem with a derivative package this may not be the place to
get that resolved.
On Friday, September 3, 2010, Jinsong Hu <ji...@hotmail.com> wrote:
> I noticed that CDH3 has 2 executable
>
> /usr/bin/hbase
>
> /usr/lib/hbase/bin/hbase
>
> I compared them and they are different. it turns out that I run
>
> /usr/bin/hbase shell
>
> and then list table, it works, but if I run
>
> /usr/lib/hbase/bin/hbase shell
>
> and list tables, it freezes. In the next distribution, please make /usr/lib/hbase/bin/hbase
> a softlink to /usr/bin/hbase , or the otherway. and make sure the executable works.
>
> Jimmy.
>
CDH3 has 2 versions of hbase executable, one work, another doesn't
Posted by Jinsong Hu <ji...@hotmail.com>.
I noticed that CDH3 has 2 executable
/usr/bin/hbase
/usr/lib/hbase/bin/hbase
I compared them and they are different. it turns out that I run
/usr/bin/hbase shell
and then list table, it works, but if I run
/usr/lib/hbase/bin/hbase shell
and list tables, it freezes. In the next distribution, please make
/usr/lib/hbase/bin/hbase
a softlink to /usr/bin/hbase , or the otherway. and make sure the executable
works.
Jimmy.
Re: thrift for hbase in CDH3 broken ?
Posted by Jinsong Hu <ji...@hotmail.com>.
which hbase-thrift version do you use ? I used hbase-0.89 version and my
perl version is 5.8.8.
and the hbase-thrift version I used is the hbase-thrift-0.1.tar.gz from
http://code.google.com/p/hbase-thrift/
I tried again and it just doesn't work, complaining
getRow failed: unknown result at
/usr/lib/perl5/site_perl/5.8.8/Hbase/Hbase.pm l
ine 5898.
I suspect it is the problem of Hbase.pm version that is not compatible with
hbase-0.89 .
the http://code.google.com/p/hbase-thrift/ mention the required hbase
version is 0.18, which
is too old.
Jimmy
--------------------------------------------------
From: "Igor Ranitovic" <ir...@gmail.com>
Sent: Wednesday, September 08, 2010 3:58 PM
To: <us...@hbase.apache.org>
Subject: Re: thrift for hbase in CDH3 broken ?
>> There is no firewall. As you can see, on the same client machine, I am
>> able to get the ruby version of the code to work.
>> This confirms that the thrift server is not the problem. Basically I am
>> just trying to fetch the same row of data
>> as that of the ruby program.
>>
>> I am not running thrift server on the same regionserver. I am running the
>> thrift server on a standalone machine
>> that is configured to point to the zookeeper for the hbase cluster.
>>
>> since the ruby version of the client code works, I would assume that the
>> thrift server is not the problem.
>> I also tried java version and it doesn't work either. in the previous
>> post somebody asked why I use java.
>> The reason is because I want to test and see if the thrift server works.
>> I never managed to get java working,
>> even until now.
>>
>> Have you gotten the perl version to work ? Have you been able to read a
>> row of data using perl ?
>
> It works for me. I run cdh3 on a single test node in pseudo-distributed
> mode. You mentioned that you get an exception when fetching a row. What is
> the exception? Try
>
> my $row;
> eval {
> $row = $client->getRow('table', 'row');
> };
>
> if ($@) {
> print Dumper($@);
> }
>
> i.
>
>> Jimmy.
>>
>> --------------------------------------------------
>> From: "Igor Ranitovic" <ir...@gmail.com>
>> Sent: Tuesday, September 07, 2010 8:18 PM
>> To: <us...@hbase.apache.org>
>> Subject: Re: thrift for hbase in CDH3 broken ?
>>
>>> Jinsong Hu wrote:
>>>> I tried, this doesn't work. I noticed
>>>> $transport->open();
>>>> is missing in this code. so I added it.
>>>
>>> Yup. Sorry about that. Copy and paste error :(
>>>
>>>> following code first successfully print all tables, then in the line
>>>> getRow(), it throws exception, even with ruby client, the row data is
>>>> there
>>>>
>>>>
>>>>
>>>> $transport->open();
>>>>
>>>>
>>>> my @names=$client->getTableNames();
>>>>
>>>> print Dumper(@names);
>>>> print "\n";
>>>>
>>>> my $row = $client->getRow('table12345', "key123");
>>>>
>>>> print Dumper($row);
>>>> print "\n";
>>>>
>>>>
>>>>
>>>> $transport->close();
>>>
>>>
>>> So you can scan META table on the master, but can fetch a row from a RS.
>>> Are there any firewalls in place ? Are you running thrift servers on the
>>> same nodes as region servers? What kind of exception do you get?
>>>
>>> i.
>>>
>>>> --------------------------------------------------
>>>> From: "Igor Ranitovic" <ir...@gmail.com>
>>>> Sent: Friday, September 03, 2010 11:45 AM
>>>> To: <us...@hbase.apache.org>
>>>> Subject: Re: thrift for hbase in CDH3 broken ?
>>>>
>>>>> Not should what the test code is...would this test your setup?
>>>>>
>>>>>
>>>>> #!/usr/bin/env perl
>>>>>
>>>>> use strict;
>>>>> use warnings;
>>>>>
>>>>> use Thrift::BinaryProtocol;
>>>>> use Thrift::BufferedTransport;
>>>>> use Thrift::Socket;
>>>>> use Hbase::Hbase;
>>>>> use Data::Dumper;
>>>>>
>>>>> my $sock = Thrift::Socket->new('127.0.0.1', '9090');
>>>>> $sock->setRecvTimeout(60000);
>>>>> my $transport = Thrift::BufferedTransport->new($sock);
>>>>> my $protocol = Thrift::BinaryProtocol->new($transport);
>>>>> my $client = Hbase::HbaseClient->new($protocol);
>>>>>
>>>>> my $row = $client->getRow('table_test', 'row_123');
>>>>> print Dumper($row);
>>>>>
>>>>> $transport->close();
>>>>>
>>>>>
>>>>> BTW, I am not sure why you would want to use java to talk to the HBase
>>>>> via the thirft server.
>>>>>
>>>>> i.
>>>>>
>>>>>
>>>>> Jinsong Hu wrote:
>>>>>> by the way, does anybody have a perl version of the test code ?
>>>>>>
>>>>>> Jimmy
>>>>>>
>>>>>> --------------------------------------------------
>>>>>> From: "Jinsong Hu" <ji...@hotmail.com>
>>>>>> Sent: Friday, September 03, 2010 11:17 AM
>>>>>> To: <us...@hbase.apache.org>
>>>>>> Subject: Re: thrift for hbase in CDH3 broken ?
>>>>>>
>>>>>>>
>>>>>>> I tried your code and indeed it works. but the java version doesn't
>>>>>>> work. so it looks like it is a bug
>>>>>>> of the java library supplied by the thrift-0.2.0 version.
>>>>>>>
>>>>>>> Jimmy.
>>>>>>> --------------------------------------------------
>>>>>>> From: "Alexey Kovyrin" <al...@kovyrin.net>
>>>>>>> Sent: Friday, September 03, 2010 12:31 AM
>>>>>>> To: <us...@hbase.apache.org>
>>>>>>> Subject: Re: thrift for hbase in CDH3 broken ?
>>>>>>>
>>>>>>>> yes, Centos 5.5 + CDH3b2
>>>>>>>>
>>>>>>>> On Fri, Sep 3, 2010 at 3:26 AM, Jinsong Hu <ji...@hotmail.com>
>>>>>>>> wrote:
>>>>>>>>> are you using CDH3 distribution ?
>>>>>>>>>
>>>>>>>>> Jinsong
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --------------------------------------------------
>>>>>>>>> From: "Alexey Kovyrin" <al...@kovyrin.net>
>>>>>>>>> Sent: Friday, September 03, 2010 12:04 AM
>>>>>>>>> To: <us...@hbase.apache.org>
>>>>>>>>> Subject: Re: thrift for hbase in CDH3 broken ?
>>>>>>>>>
>>>>>>>>>> http://github.com/kovyrin/hbase-thrift-client-examples - just
>>>>>>>>>> wrote
>>>>>>>>>> this example and tested it in our cluster, works as expected.
>>>>>>>>>> For this to work you'd need to install rubygems and thrift gem
>>>>>>>>>> (gem
>>>>>>>>>> install thrift).
>>>>>>>>>>
>>>>>>>>>> On Fri, Sep 3, 2010 at 12:01 AM, Jinsong Hu
>>>>>>>>>> <ji...@hotmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Can you send me some ruby test code and so I can try against the
>>>>>>>>>>> latest
>>>>>>>>>>> CDH3
>>>>>>>>>>> ?
>>>>>>>>>>>
>>>>>>>>>>> Jimmy.
>>>>>>>>>>>
>>>>>>>>>>> --------------------------------------------------
>>>>>>>>>>> From: "Alexey Kovyrin" <al...@kovyrin.net>
>>>>>>>>>>> Sent: Thursday, September 02, 2010 8:15 PM
>>>>>>>>>>> To: <us...@hbase.apache.org>
>>>>>>>>>>> Subject: Re: thrift for hbase in CDH3 broken ?
>>>>>>>>>>>
>>>>>>>>>>>> We use it in Scribd.com. All clients are ruby web apps.
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Sep 2, 2010 at 10:49 PM, Todd Lipcon
>>>>>>>>>>>> <to...@cloudera.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Sep 2, 2010 at 5:35 PM, Jinsong Hu
>>>>>>>>>>>>> <ji...@hotmail.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Yes, I confirmed that it is indeed thrift server.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> and the fact that the API
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> List<byte[]> tableNamesList=client.getTableNames();
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> for (byte [] name : tableNamesList)
>>>>>>>>>>>>>>>> {
>>>>>>>>>>>>>>>> System.out.println(new String(name));
>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> successfully printed all table names shows that it is indeed
>>>>>>>>>>>>>> thrift
>>>>>>>>>>>>>> server.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> if it is hue, it won't print the table names.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Ah, sorry, I missed that in your original message. Not sure
>>>>>>>>>>>>>> what's up,
>>>>>>>>>>>>>> then
>>>>>>>>>>>>>
>>>>>>>>>>>>> - we don't have any changes in CDH that would affect this.
>>>>>>>>>>>>> Anyone here
>>>>>>>>>>>>> used
>>>>>>>>>>>>> thrift on 0.89.20100621?
>>>>>>>>>>>>>
>>>>>>>>>>>>> -Todd
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Jimmy.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --------------------------------------------------
>>>>>>>>>>>>>> From: "Todd Lipcon" <to...@cloudera.com>
>>>>>>>>>>>>>> Sent: Thursday, September 02, 2010 5:18 PM
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> To: <us...@hbase.apache.org>
>>>>>>>>>>>>>> Subject: Re: thrift for hbase in CDH3 broken ?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Jinsong,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Are you sure that the port you're connecting to is indeed
>>>>>>>>>>>>>>> the thrift
>>>>>>>>>>>>>>> server?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Unfortunately both the HBase thrift server and the Hue
>>>>>>>>>>>>>>> namenode
>>>>>>>>>>>>>>> plugin
>>>>>>>>>>>>>>> listen on port 9090, so you might be having an issue where
>>>>>>>>>>>>>>> your HBase
>>>>>>>>>>>>>>> client
>>>>>>>>>>>>>>> is trying to connect to the Namenode server instead of
>>>>>>>>>>>>>>> HBase.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> You can verify the ports using a command like
>>>>>>>>>>>>>>> "/sbin/fuser -n tcp
>>>>>>>>>>>>>>> 9090"
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> see which pid has it open, then cross reference against sudo
>>>>>>>>>>>>>>> jps.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>> -Todd
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Thu, Sep 2, 2010 at 4:40 PM, Jinsong Hu
>>>>>>>>>>>>>>> <ji...@hotmail.com>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi, There,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I am trying to test and see if thrift for hbase works. I
>>>>>>>>>>>>>>>> followed
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> example from
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> http://www.workhabit.com/labs/centos-55-and-thriftscribe
>>>>>>>>>>>>>>>> http://incubator.apache.org/thrift/
>>>>>>>>>>>>>>>> http://wiki.apache.org/hadoop/Hbase/ThriftApi
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> and wrote test code: I found that client.getTableNames();
>>>>>>>>>>>>>>>> returns all table names successfully, but the scanner never
>>>>>>>>>>>>>>>> returned
>>>>>>>>>>>>>>>> any record. and it even throws exception:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> org.apache.thrift.TApplicationException: get failed:
>>>>>>>>>>>>>>>> unknown result
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> org.apache.hadoop.hbase.thrift.generated.Hbase$Client.recv_get(Hbase.java:785)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> org.apache.hadoop.hbase.thrift.generated.Hbase$Client.get(Hbase.java:750)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> org.apache.hadoop.hbase.thrift.HbaseThriftTest.testThriftAPI(HbaseThriftTest.java:73)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> org.apache.hadoop.hbase.thrift.HbaseThriftTest.main(HbaseThriftTest.java:128)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I checked the syntax and the table and confirmed that the
>>>>>>>>>>>>>>>> table does
>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>> record and it should
>>>>>>>>>>>>>>>> return result, but it doesn't. Can anybody tell me what is
>>>>>>>>>>>>>>>> wrong ?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> At this time, I am suspecting the thrift server shipped
>>>>>>>>>>>>>>>> with CDH3
>>>>>>>>>>>>>>>> may
>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>> problem. if there anyway
>>>>>>>>>>>>>>>> I can get this test code working ?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I have to use thrift-0.2.0-incubating.tar.gz , as I found
>>>>>>>>>>>>>>>> that the
>>>>>>>>>>>>>>>> code
>>>>>>>>>>>>>>>> generated from hbase-thrift project
>>>>>>>>>>>>>>>> doesn't even compile with 0.3.0 and 0.4.0 thrift.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Jimmy.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> public void testThriftAPI()
>>>>>>>>>>>>>>>> {
>>>>>>>>>>>>>>>> // Make socket
>>>>>>>>>>>>>>>> TSocket transport = new TSocket("10.20.12.13", 9090);
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> // Wrap in a protocol
>>>>>>>>>>>>>>>> TBinaryProtocol protocol = new
>>>>>>>>>>>>>>>> TBinaryProtocol(transport);
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hbase.Client client = new Hbase.Client(protocol);
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> try {
>>>>>>>>>>>>>>>> transport.open();
>>>>>>>>>>>>>>>> List<byte[]> tableNamesList=client.getTableNames();
>>>>>>>>>>>>>>>> for (byte [] name : tableNamesList)
>>>>>>>>>>>>>>>> {
>>>>>>>>>>>>>>>> System.out.println(new String(name));
>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> String tableName="HEARTBEAT_CLUSTER";
>>>>>>>>>>>>>>>> String startRow="";
>>>>>>>>>>>>>>>> List<byte[]> columns= new ArrayList<byte[]>();
>>>>>>>>>>>>>>>> columns.add("fields:time_format".getBytes());
>>>>>>>>>>>>>>>> columns.add("fields:customer_id".getBytes());
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> int aScannerId=0;
>>>>>>>>>>>>>>>> try {
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> TCell cell=client.get(tableName.getBytes(),
>>>>>>>>>>>>>>>> "2010-08\tproduction-2".getBytes(),
>>>>>>>>>>>>>>>> "fields:customer_id".getBytes());
>>>>>>>>>>>>>>>> if (cell != null)
>>>>>>>>>>>>>>>> {
>>>>>>>>>>>>>>>> String value= new String( cell.value);
>>>>>>>>>>>>>>>> System.out.println(value);
>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> aScannerId=client.scannerOpen(tableName.getBytes(),
>>>>>>>>>>>>>>>> startRow.getBytes(), columns);
>>>>>>>>>>>>>>>> TRowResult result=null;
>>>>>>>>>>>>>>>> for (int i=0; i< 10; i++)
>>>>>>>>>>>>>>>> {
>>>>>>>>>>>>>>>> try {
>>>>>>>>>>>>>>>> result=client.scannerGet(aScannerId);
>>>>>>>>>>>>>>>> } catch (Exception ex1)
>>>>>>>>>>>>>>>> {
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>> if (result ==null) break;
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> String
>>>>>>>>>>>>>>>> timeFormat=result.getFieldValue(0).toString();
>>>>>>>>>>>>>>>> String
>>>>>>>>>>>>>>>> customerId=result.getFieldValue(1).toString();
>>>>>>>>>>>>>>>> System.out.println(timeFormat + "\t" +
>>>>>>>>>>>>>>>> customerId);
>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>> } catch (Exception ex)
>>>>>>>>>>>>>>>> {
>>>>>>>>>>>>>>>> ex.printStackTrace();
>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>> finally {
>>>>>>>>>>>>>>>> if (aScannerId >0 )
>>>>>>>>>>>>>>>> client.scannerClose(aScannerId);
>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>> } catch (Exception exp)
>>>>>>>>>>>>>>>> {
>>>>>>>>>>>>>>>> exp.printStackTrace();
>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>> finally{
>>>>>>>>>>>>>>>> transport.close();
>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> Todd Lipcon
>>>>>>>>>>>>>>> Software Engineer, Cloudera
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Todd Lipcon
>>>>>>>>>>>>> Software Engineer, Cloudera
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Alexey Kovyrin
>>>>>>>>>>>> http://kovyrin.net/
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Alexey Kovyrin
>>>>>>>>>> http://kovyrin.net/
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Alexey Kovyrin
>>>>>>>> http://kovyrin.net/
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>
>
>
Re: thrift for hbase in CDH3 broken ?
Posted by Igor Ranitovic <ir...@gmail.com>.
> There is no firewall. As you can see, on the same client machine, I am
> able to get the ruby version of the code to work.
> This confirms that the thrift server is not the problem. Basically I am
> just trying to fetch the same row of data
> as that of the ruby program.
>
> I am not running thrift server on the same regionserver. I am running
> the thrift server on a standalone machine
> that is configured to point to the zookeeper for the hbase cluster.
>
> since the ruby version of the client code works, I would assume that
> the thrift server is not the problem.
> I also tried java version and it doesn't work either. in the previous
> post somebody asked why I use java.
> The reason is because I want to test and see if the thrift server works.
> I never managed to get java working,
> even until now.
>
> Have you gotten the perl version to work ? Have you been able to read a
> row of data using perl ?
It works for me. I run cdh3 on a single test node in pseudo-distributed
mode. You mentioned that you get an exception when fetching a row. What
is the exception? Try
my $row;
eval {
$row = $client->getRow('table', 'row');
};
if ($@) {
print Dumper($@);
}
i.
> Jimmy.
>
> --------------------------------------------------
> From: "Igor Ranitovic" <ir...@gmail.com>
> Sent: Tuesday, September 07, 2010 8:18 PM
> To: <us...@hbase.apache.org>
> Subject: Re: thrift for hbase in CDH3 broken ?
>
>> Jinsong Hu wrote:
>>> I tried, this doesn't work. I noticed
>>> $transport->open();
>>> is missing in this code. so I added it.
>>
>> Yup. Sorry about that. Copy and paste error :(
>>
>>> following code first successfully print all tables, then in the line
>>> getRow(), it throws exception, even with ruby client, the row data is
>>> there
>>>
>>>
>>>
>>> $transport->open();
>>>
>>>
>>> my @names=$client->getTableNames();
>>>
>>> print Dumper(@names);
>>> print "\n";
>>>
>>> my $row = $client->getRow('table12345', "key123");
>>>
>>> print Dumper($row);
>>> print "\n";
>>>
>>>
>>>
>>> $transport->close();
>>
>>
>> So you can scan META table on the master, but can fetch a row from a RS.
>> Are there any firewalls in place ? Are you running thrift servers on
>> the same nodes as region servers? What kind of exception do you get?
>>
>> i.
>>
>>> --------------------------------------------------
>>> From: "Igor Ranitovic" <ir...@gmail.com>
>>> Sent: Friday, September 03, 2010 11:45 AM
>>> To: <us...@hbase.apache.org>
>>> Subject: Re: thrift for hbase in CDH3 broken ?
>>>
>>>> Not should what the test code is...would this test your setup?
>>>>
>>>>
>>>> #!/usr/bin/env perl
>>>>
>>>> use strict;
>>>> use warnings;
>>>>
>>>> use Thrift::BinaryProtocol;
>>>> use Thrift::BufferedTransport;
>>>> use Thrift::Socket;
>>>> use Hbase::Hbase;
>>>> use Data::Dumper;
>>>>
>>>> my $sock = Thrift::Socket->new('127.0.0.1', '9090');
>>>> $sock->setRecvTimeout(60000);
>>>> my $transport = Thrift::BufferedTransport->new($sock);
>>>> my $protocol = Thrift::BinaryProtocol->new($transport);
>>>> my $client = Hbase::HbaseClient->new($protocol);
>>>>
>>>> my $row = $client->getRow('table_test', 'row_123');
>>>> print Dumper($row);
>>>>
>>>> $transport->close();
>>>>
>>>>
>>>> BTW, I am not sure why you would want to use java to talk to the
>>>> HBase via the thirft server.
>>>>
>>>> i.
>>>>
>>>>
>>>> Jinsong Hu wrote:
>>>>> by the way, does anybody have a perl version of the test code ?
>>>>>
>>>>> Jimmy
>>>>>
>>>>> --------------------------------------------------
>>>>> From: "Jinsong Hu" <ji...@hotmail.com>
>>>>> Sent: Friday, September 03, 2010 11:17 AM
>>>>> To: <us...@hbase.apache.org>
>>>>> Subject: Re: thrift for hbase in CDH3 broken ?
>>>>>
>>>>>>
>>>>>> I tried your code and indeed it works. but the java version
>>>>>> doesn't work. so it looks like it is a bug
>>>>>> of the java library supplied by the thrift-0.2.0 version.
>>>>>>
>>>>>> Jimmy.
>>>>>> --------------------------------------------------
>>>>>> From: "Alexey Kovyrin" <al...@kovyrin.net>
>>>>>> Sent: Friday, September 03, 2010 12:31 AM
>>>>>> To: <us...@hbase.apache.org>
>>>>>> Subject: Re: thrift for hbase in CDH3 broken ?
>>>>>>
>>>>>>> yes, Centos 5.5 + CDH3b2
>>>>>>>
>>>>>>> On Fri, Sep 3, 2010 at 3:26 AM, Jinsong Hu
>>>>>>> <ji...@hotmail.com> wrote:
>>>>>>>> are you using CDH3 distribution ?
>>>>>>>>
>>>>>>>> Jinsong
>>>>>>>>
>>>>>>>>
>>>>>>>> --------------------------------------------------
>>>>>>>> From: "Alexey Kovyrin" <al...@kovyrin.net>
>>>>>>>> Sent: Friday, September 03, 2010 12:04 AM
>>>>>>>> To: <us...@hbase.apache.org>
>>>>>>>> Subject: Re: thrift for hbase in CDH3 broken ?
>>>>>>>>
>>>>>>>>> http://github.com/kovyrin/hbase-thrift-client-examples - just
>>>>>>>>> wrote
>>>>>>>>> this example and tested it in our cluster, works as expected.
>>>>>>>>> For this to work you'd need to install rubygems and thrift gem
>>>>>>>>> (gem
>>>>>>>>> install thrift).
>>>>>>>>>
>>>>>>>>> On Fri, Sep 3, 2010 at 12:01 AM, Jinsong Hu
>>>>>>>>> <ji...@hotmail.com>
>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Can you send me some ruby test code and so I can try against
>>>>>>>>>> the latest
>>>>>>>>>> CDH3
>>>>>>>>>> ?
>>>>>>>>>>
>>>>>>>>>> Jimmy.
>>>>>>>>>>
>>>>>>>>>> --------------------------------------------------
>>>>>>>>>> From: "Alexey Kovyrin" <al...@kovyrin.net>
>>>>>>>>>> Sent: Thursday, September 02, 2010 8:15 PM
>>>>>>>>>> To: <us...@hbase.apache.org>
>>>>>>>>>> Subject: Re: thrift for hbase in CDH3 broken ?
>>>>>>>>>>
>>>>>>>>>>> We use it in Scribd.com. All clients are ruby web apps.
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Sep 2, 2010 at 10:49 PM, Todd Lipcon
>>>>>>>>>>> <to...@cloudera.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Sep 2, 2010 at 5:35 PM, Jinsong Hu
>>>>>>>>>>>> <ji...@hotmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Yes, I confirmed that it is indeed thrift server.
>>>>>>>>>>>>>
>>>>>>>>>>>>> and the fact that the API
>>>>>>>>>>>>>
>>>>>>>>>>>>> List<byte[]> tableNamesList=client.getTableNames();
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> for (byte [] name : tableNamesList)
>>>>>>>>>>>>>>> {
>>>>>>>>>>>>>>> System.out.println(new String(name));
>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>> successfully printed all table names shows that it is
>>>>>>>>>>>>> indeed thrift
>>>>>>>>>>>>> server.
>>>>>>>>>>>>>
>>>>>>>>>>>>> if it is hue, it won't print the table names.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Ah, sorry, I missed that in your original message. Not sure
>>>>>>>>>>>>> what's up,
>>>>>>>>>>>>> then
>>>>>>>>>>>>
>>>>>>>>>>>> - we don't have any changes in CDH that would affect this.
>>>>>>>>>>>> Anyone here
>>>>>>>>>>>> used
>>>>>>>>>>>> thrift on 0.89.20100621?
>>>>>>>>>>>>
>>>>>>>>>>>> -Todd
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> Jimmy.
>>>>>>>>>>>>>
>>>>>>>>>>>>> --------------------------------------------------
>>>>>>>>>>>>> From: "Todd Lipcon" <to...@cloudera.com>
>>>>>>>>>>>>> Sent: Thursday, September 02, 2010 5:18 PM
>>>>>>>>>>>>>
>>>>>>>>>>>>> To: <us...@hbase.apache.org>
>>>>>>>>>>>>> Subject: Re: thrift for hbase in CDH3 broken ?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Jinsong,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Are you sure that the port you're connecting to is indeed
>>>>>>>>>>>>>> the thrift
>>>>>>>>>>>>>> server?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Unfortunately both the HBase thrift server and the Hue
>>>>>>>>>>>>>> namenode
>>>>>>>>>>>>>> plugin
>>>>>>>>>>>>>> listen on port 9090, so you might be having an issue where
>>>>>>>>>>>>>> your HBase
>>>>>>>>>>>>>> client
>>>>>>>>>>>>>> is trying to connect to the Namenode server instead of HBase.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> You can verify the ports using a command like "/sbin/fuser
>>>>>>>>>>>>>> -n tcp
>>>>>>>>>>>>>> 9090"
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> see which pid has it open, then cross reference against
>>>>>>>>>>>>>> sudo jps.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>> -Todd
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Thu, Sep 2, 2010 at 4:40 PM, Jinsong Hu
>>>>>>>>>>>>>> <ji...@hotmail.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi, There,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I am trying to test and see if thrift for hbase works. I
>>>>>>>>>>>>>>> followed
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> example from
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> http://www.workhabit.com/labs/centos-55-and-thriftscribe
>>>>>>>>>>>>>>> http://incubator.apache.org/thrift/
>>>>>>>>>>>>>>> http://wiki.apache.org/hadoop/Hbase/ThriftApi
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> and wrote test code: I found that client.getTableNames();
>>>>>>>>>>>>>>> returns all table names successfully, but the scanner
>>>>>>>>>>>>>>> never returned
>>>>>>>>>>>>>>> any record. and it even throws exception:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> org.apache.thrift.TApplicationException: get failed:
>>>>>>>>>>>>>>> unknown result
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> org.apache.hadoop.hbase.thrift.generated.Hbase$Client.recv_get(Hbase.java:785)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> org.apache.hadoop.hbase.thrift.generated.Hbase$Client.get(Hbase.java:750)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> org.apache.hadoop.hbase.thrift.HbaseThriftTest.testThriftAPI(HbaseThriftTest.java:73)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> org.apache.hadoop.hbase.thrift.HbaseThriftTest.main(HbaseThriftTest.java:128)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I checked the syntax and the table and confirmed that the
>>>>>>>>>>>>>>> table does
>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>> record and it should
>>>>>>>>>>>>>>> return result, but it doesn't. Can anybody tell me what
>>>>>>>>>>>>>>> is wrong ?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> At this time, I am suspecting the thrift server shipped
>>>>>>>>>>>>>>> with CDH3
>>>>>>>>>>>>>>> may
>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>> problem. if there anyway
>>>>>>>>>>>>>>> I can get this test code working ?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I have to use thrift-0.2.0-incubating.tar.gz , as I found
>>>>>>>>>>>>>>> that the
>>>>>>>>>>>>>>> code
>>>>>>>>>>>>>>> generated from hbase-thrift project
>>>>>>>>>>>>>>> doesn't even compile with 0.3.0 and 0.4.0 thrift.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Jimmy.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> public void testThriftAPI()
>>>>>>>>>>>>>>> {
>>>>>>>>>>>>>>> // Make socket
>>>>>>>>>>>>>>> TSocket transport = new TSocket("10.20.12.13", 9090);
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> // Wrap in a protocol
>>>>>>>>>>>>>>> TBinaryProtocol protocol = new
>>>>>>>>>>>>>>> TBinaryProtocol(transport);
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hbase.Client client = new Hbase.Client(protocol);
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> try {
>>>>>>>>>>>>>>> transport.open();
>>>>>>>>>>>>>>> List<byte[]> tableNamesList=client.getTableNames();
>>>>>>>>>>>>>>> for (byte [] name : tableNamesList)
>>>>>>>>>>>>>>> {
>>>>>>>>>>>>>>> System.out.println(new String(name));
>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> String tableName="HEARTBEAT_CLUSTER";
>>>>>>>>>>>>>>> String startRow="";
>>>>>>>>>>>>>>> List<byte[]> columns= new ArrayList<byte[]>();
>>>>>>>>>>>>>>> columns.add("fields:time_format".getBytes());
>>>>>>>>>>>>>>> columns.add("fields:customer_id".getBytes());
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> int aScannerId=0;
>>>>>>>>>>>>>>> try {
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> TCell cell=client.get(tableName.getBytes(),
>>>>>>>>>>>>>>> "2010-08\tproduction-2".getBytes(),
>>>>>>>>>>>>>>> "fields:customer_id".getBytes());
>>>>>>>>>>>>>>> if (cell != null)
>>>>>>>>>>>>>>> {
>>>>>>>>>>>>>>> String value= new String( cell.value);
>>>>>>>>>>>>>>> System.out.println(value);
>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> aScannerId=client.scannerOpen(tableName.getBytes(),
>>>>>>>>>>>>>>> startRow.getBytes(), columns);
>>>>>>>>>>>>>>> TRowResult result=null;
>>>>>>>>>>>>>>> for (int i=0; i< 10; i++)
>>>>>>>>>>>>>>> {
>>>>>>>>>>>>>>> try {
>>>>>>>>>>>>>>> result=client.scannerGet(aScannerId);
>>>>>>>>>>>>>>> } catch (Exception ex1)
>>>>>>>>>>>>>>> {
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>> if (result ==null) break;
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> String
>>>>>>>>>>>>>>> timeFormat=result.getFieldValue(0).toString();
>>>>>>>>>>>>>>> String
>>>>>>>>>>>>>>> customerId=result.getFieldValue(1).toString();
>>>>>>>>>>>>>>> System.out.println(timeFormat + "\t" +
>>>>>>>>>>>>>>> customerId);
>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>> } catch (Exception ex)
>>>>>>>>>>>>>>> {
>>>>>>>>>>>>>>> ex.printStackTrace();
>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>> finally {
>>>>>>>>>>>>>>> if (aScannerId >0 )
>>>>>>>>>>>>>>> client.scannerClose(aScannerId);
>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>> } catch (Exception exp)
>>>>>>>>>>>>>>> {
>>>>>>>>>>>>>>> exp.printStackTrace();
>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>> finally{
>>>>>>>>>>>>>>> transport.close();
>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Todd Lipcon
>>>>>>>>>>>>>> Software Engineer, Cloudera
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Todd Lipcon
>>>>>>>>>>>> Software Engineer, Cloudera
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Alexey Kovyrin
>>>>>>>>>>> http://kovyrin.net/
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Alexey Kovyrin
>>>>>>>>> http://kovyrin.net/
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Alexey Kovyrin
>>>>>>> http://kovyrin.net/
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>>
>
Re: thrift for hbase in CDH3 broken ?
Posted by Jinsong Hu <ji...@hotmail.com>.
There is no firewall. As you can see, on the same client machine, I am able
to get the ruby version of the code to work.
This confirms that the thrift server is not the problem. Basically I am just
trying to fetch the same row of data
as that of the ruby program.
I am not running thrift server on the same regionserver. I am running the
thrift server on a standalone machine
that is configured to point to the zookeeper for the hbase cluster.
since the ruby version of the client code works, I would assume that the
thrift server is not the problem.
I also tried java version and it doesn't work either. in the previous post
somebody asked why I use java.
The reason is because I want to test and see if the thrift server works. I
never managed to get java working,
even until now.
Have you gotten the perl version to work ? Have you been able to read a row
of data using perl ?
Jimmy.
--------------------------------------------------
From: "Igor Ranitovic" <ir...@gmail.com>
Sent: Tuesday, September 07, 2010 8:18 PM
To: <us...@hbase.apache.org>
Subject: Re: thrift for hbase in CDH3 broken ?
> Jinsong Hu wrote:
>> I tried, this doesn't work. I noticed
>> $transport->open();
>> is missing in this code. so I added it.
>
> Yup. Sorry about that. Copy and paste error :(
>
>> following code first successfully print all tables, then in the line
>> getRow(), it throws exception, even with ruby client, the row data is
>> there
>>
>>
>>
>> $transport->open();
>>
>>
>> my @names=$client->getTableNames();
>>
>> print Dumper(@names);
>> print "\n";
>>
>> my $row = $client->getRow('table12345', "key123");
>>
>> print Dumper($row);
>> print "\n";
>>
>>
>>
>> $transport->close();
>
>
> So you can scan META table on the master, but can fetch a row from a RS.
> Are there any firewalls in place ? Are you running thrift servers on the
> same nodes as region servers? What kind of exception do you get?
>
> i.
>
>> --------------------------------------------------
>> From: "Igor Ranitovic" <ir...@gmail.com>
>> Sent: Friday, September 03, 2010 11:45 AM
>> To: <us...@hbase.apache.org>
>> Subject: Re: thrift for hbase in CDH3 broken ?
>>
>>> Not should what the test code is...would this test your setup?
>>>
>>>
>>> #!/usr/bin/env perl
>>>
>>> use strict;
>>> use warnings;
>>>
>>> use Thrift::BinaryProtocol;
>>> use Thrift::BufferedTransport;
>>> use Thrift::Socket;
>>> use Hbase::Hbase;
>>> use Data::Dumper;
>>>
>>> my $sock = Thrift::Socket->new('127.0.0.1', '9090');
>>> $sock->setRecvTimeout(60000);
>>> my $transport = Thrift::BufferedTransport->new($sock);
>>> my $protocol = Thrift::BinaryProtocol->new($transport);
>>> my $client = Hbase::HbaseClient->new($protocol);
>>>
>>> my $row = $client->getRow('table_test', 'row_123');
>>> print Dumper($row);
>>>
>>> $transport->close();
>>>
>>>
>>> BTW, I am not sure why you would want to use java to talk to the HBase
>>> via the thirft server.
>>>
>>> i.
>>>
>>>
>>> Jinsong Hu wrote:
>>>> by the way, does anybody have a perl version of the test code ?
>>>>
>>>> Jimmy
>>>>
>>>> --------------------------------------------------
>>>> From: "Jinsong Hu" <ji...@hotmail.com>
>>>> Sent: Friday, September 03, 2010 11:17 AM
>>>> To: <us...@hbase.apache.org>
>>>> Subject: Re: thrift for hbase in CDH3 broken ?
>>>>
>>>>>
>>>>> I tried your code and indeed it works. but the java version doesn't
>>>>> work. so it looks like it is a bug
>>>>> of the java library supplied by the thrift-0.2.0 version.
>>>>>
>>>>> Jimmy.
>>>>> --------------------------------------------------
>>>>> From: "Alexey Kovyrin" <al...@kovyrin.net>
>>>>> Sent: Friday, September 03, 2010 12:31 AM
>>>>> To: <us...@hbase.apache.org>
>>>>> Subject: Re: thrift for hbase in CDH3 broken ?
>>>>>
>>>>>> yes, Centos 5.5 + CDH3b2
>>>>>>
>>>>>> On Fri, Sep 3, 2010 at 3:26 AM, Jinsong Hu <ji...@hotmail.com>
>>>>>> wrote:
>>>>>>> are you using CDH3 distribution ?
>>>>>>>
>>>>>>> Jinsong
>>>>>>>
>>>>>>>
>>>>>>> --------------------------------------------------
>>>>>>> From: "Alexey Kovyrin" <al...@kovyrin.net>
>>>>>>> Sent: Friday, September 03, 2010 12:04 AM
>>>>>>> To: <us...@hbase.apache.org>
>>>>>>> Subject: Re: thrift for hbase in CDH3 broken ?
>>>>>>>
>>>>>>>> http://github.com/kovyrin/hbase-thrift-client-examples - just wrote
>>>>>>>> this example and tested it in our cluster, works as expected.
>>>>>>>> For this to work you'd need to install rubygems and thrift gem (gem
>>>>>>>> install thrift).
>>>>>>>>
>>>>>>>> On Fri, Sep 3, 2010 at 12:01 AM, Jinsong Hu
>>>>>>>> <ji...@hotmail.com>
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Can you send me some ruby test code and so I can try against the
>>>>>>>>> latest
>>>>>>>>> CDH3
>>>>>>>>> ?
>>>>>>>>>
>>>>>>>>> Jimmy.
>>>>>>>>>
>>>>>>>>> --------------------------------------------------
>>>>>>>>> From: "Alexey Kovyrin" <al...@kovyrin.net>
>>>>>>>>> Sent: Thursday, September 02, 2010 8:15 PM
>>>>>>>>> To: <us...@hbase.apache.org>
>>>>>>>>> Subject: Re: thrift for hbase in CDH3 broken ?
>>>>>>>>>
>>>>>>>>>> We use it in Scribd.com. All clients are ruby web apps.
>>>>>>>>>>
>>>>>>>>>> On Thu, Sep 2, 2010 at 10:49 PM, Todd Lipcon <to...@cloudera.com>
>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Sep 2, 2010 at 5:35 PM, Jinsong Hu
>>>>>>>>>>> <ji...@hotmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Yes, I confirmed that it is indeed thrift server.
>>>>>>>>>>>>
>>>>>>>>>>>> and the fact that the API
>>>>>>>>>>>>
>>>>>>>>>>>> List<byte[]> tableNamesList=client.getTableNames();
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> for (byte [] name : tableNamesList)
>>>>>>>>>>>>>> {
>>>>>>>>>>>>>> System.out.println(new String(name));
>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>> successfully printed all table names shows that it is indeed
>>>>>>>>>>>> thrift
>>>>>>>>>>>> server.
>>>>>>>>>>>>
>>>>>>>>>>>> if it is hue, it won't print the table names.
>>>>>>>>>>>>
>>>>>>>>>>>> Ah, sorry, I missed that in your original message. Not sure
>>>>>>>>>>>> what's up,
>>>>>>>>>>>> then
>>>>>>>>>>>
>>>>>>>>>>> - we don't have any changes in CDH that would affect this.
>>>>>>>>>>> Anyone here
>>>>>>>>>>> used
>>>>>>>>>>> thrift on 0.89.20100621?
>>>>>>>>>>>
>>>>>>>>>>> -Todd
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> Jimmy.
>>>>>>>>>>>>
>>>>>>>>>>>> --------------------------------------------------
>>>>>>>>>>>> From: "Todd Lipcon" <to...@cloudera.com>
>>>>>>>>>>>> Sent: Thursday, September 02, 2010 5:18 PM
>>>>>>>>>>>>
>>>>>>>>>>>> To: <us...@hbase.apache.org>
>>>>>>>>>>>> Subject: Re: thrift for hbase in CDH3 broken ?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Hi Jinsong,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Are you sure that the port you're connecting to is indeed the
>>>>>>>>>>>>> thrift
>>>>>>>>>>>>> server?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Unfortunately both the HBase thrift server and the Hue
>>>>>>>>>>>>> namenode
>>>>>>>>>>>>> plugin
>>>>>>>>>>>>> listen on port 9090, so you might be having an issue where
>>>>>>>>>>>>> your HBase
>>>>>>>>>>>>> client
>>>>>>>>>>>>> is trying to connect to the Namenode server instead of HBase.
>>>>>>>>>>>>>
>>>>>>>>>>>>> You can verify the ports using a command like "/sbin/fuser -n
>>>>>>>>>>>>> tcp
>>>>>>>>>>>>> 9090"
>>>>>>>>>>>>> to
>>>>>>>>>>>>> see which pid has it open, then cross reference against sudo
>>>>>>>>>>>>> jps.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>> -Todd
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Sep 2, 2010 at 4:40 PM, Jinsong Hu
>>>>>>>>>>>>> <ji...@hotmail.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi, There,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I am trying to test and see if thrift for hbase works. I
>>>>>>>>>>>>>> followed
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> example from
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> http://www.workhabit.com/labs/centos-55-and-thriftscribe
>>>>>>>>>>>>>> http://incubator.apache.org/thrift/
>>>>>>>>>>>>>> http://wiki.apache.org/hadoop/Hbase/ThriftApi
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> and wrote test code: I found that client.getTableNames();
>>>>>>>>>>>>>> returns all table names successfully, but the scanner never
>>>>>>>>>>>>>> returned
>>>>>>>>>>>>>> any record. and it even throws exception:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> org.apache.thrift.TApplicationException: get failed: unknown
>>>>>>>>>>>>>> result
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> org.apache.hadoop.hbase.thrift.generated.Hbase$Client.recv_get(Hbase.java:785)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> org.apache.hadoop.hbase.thrift.generated.Hbase$Client.get(Hbase.java:750)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> org.apache.hadoop.hbase.thrift.HbaseThriftTest.testThriftAPI(HbaseThriftTest.java:73)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> org.apache.hadoop.hbase.thrift.HbaseThriftTest.main(HbaseThriftTest.java:128)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I checked the syntax and the table and confirmed that the
>>>>>>>>>>>>>> table does
>>>>>>>>>>>>>> have
>>>>>>>>>>>>>> record and it should
>>>>>>>>>>>>>> return result, but it doesn't. Can anybody tell me what is
>>>>>>>>>>>>>> wrong ?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> At this time, I am suspecting the thrift server shipped with
>>>>>>>>>>>>>> CDH3
>>>>>>>>>>>>>> may
>>>>>>>>>>>>>> have
>>>>>>>>>>>>>> problem. if there anyway
>>>>>>>>>>>>>> I can get this test code working ?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I have to use thrift-0.2.0-incubating.tar.gz , as I found
>>>>>>>>>>>>>> that the
>>>>>>>>>>>>>> code
>>>>>>>>>>>>>> generated from hbase-thrift project
>>>>>>>>>>>>>> doesn't even compile with 0.3.0 and 0.4.0 thrift.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Jimmy.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> public void testThriftAPI()
>>>>>>>>>>>>>> {
>>>>>>>>>>>>>> // Make socket
>>>>>>>>>>>>>> TSocket transport = new TSocket("10.20.12.13", 9090);
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> // Wrap in a protocol
>>>>>>>>>>>>>> TBinaryProtocol protocol = new
>>>>>>>>>>>>>> TBinaryProtocol(transport);
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hbase.Client client = new Hbase.Client(protocol);
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> try {
>>>>>>>>>>>>>> transport.open();
>>>>>>>>>>>>>> List<byte[]> tableNamesList=client.getTableNames();
>>>>>>>>>>>>>> for (byte [] name : tableNamesList)
>>>>>>>>>>>>>> {
>>>>>>>>>>>>>> System.out.println(new String(name));
>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> String tableName="HEARTBEAT_CLUSTER";
>>>>>>>>>>>>>> String startRow="";
>>>>>>>>>>>>>> List<byte[]> columns= new ArrayList<byte[]>();
>>>>>>>>>>>>>> columns.add("fields:time_format".getBytes());
>>>>>>>>>>>>>> columns.add("fields:customer_id".getBytes());
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> int aScannerId=0;
>>>>>>>>>>>>>> try {
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> TCell cell=client.get(tableName.getBytes(),
>>>>>>>>>>>>>> "2010-08\tproduction-2".getBytes(),
>>>>>>>>>>>>>> "fields:customer_id".getBytes());
>>>>>>>>>>>>>> if (cell != null)
>>>>>>>>>>>>>> {
>>>>>>>>>>>>>> String value= new String( cell.value);
>>>>>>>>>>>>>> System.out.println(value);
>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> aScannerId=client.scannerOpen(tableName.getBytes(),
>>>>>>>>>>>>>> startRow.getBytes(), columns);
>>>>>>>>>>>>>> TRowResult result=null;
>>>>>>>>>>>>>> for (int i=0; i< 10; i++)
>>>>>>>>>>>>>> {
>>>>>>>>>>>>>> try {
>>>>>>>>>>>>>> result=client.scannerGet(aScannerId);
>>>>>>>>>>>>>> } catch (Exception ex1)
>>>>>>>>>>>>>> {
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> }
>>>>>>>>>>>>>> if (result ==null) break;
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> String
>>>>>>>>>>>>>> timeFormat=result.getFieldValue(0).toString();
>>>>>>>>>>>>>> String
>>>>>>>>>>>>>> customerId=result.getFieldValue(1).toString();
>>>>>>>>>>>>>> System.out.println(timeFormat + "\t" +
>>>>>>>>>>>>>> customerId);
>>>>>>>>>>>>>> }
>>>>>>>>>>>>>> } catch (Exception ex)
>>>>>>>>>>>>>> {
>>>>>>>>>>>>>> ex.printStackTrace();
>>>>>>>>>>>>>> }
>>>>>>>>>>>>>> finally {
>>>>>>>>>>>>>> if (aScannerId >0 )
>>>>>>>>>>>>>> client.scannerClose(aScannerId);
>>>>>>>>>>>>>> }
>>>>>>>>>>>>>> } catch (Exception exp)
>>>>>>>>>>>>>> {
>>>>>>>>>>>>>> exp.printStackTrace();
>>>>>>>>>>>>>> }
>>>>>>>>>>>>>> finally{
>>>>>>>>>>>>>> transport.close();
>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Todd Lipcon
>>>>>>>>>>>>> Software Engineer, Cloudera
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Todd Lipcon
>>>>>>>>>>> Software Engineer, Cloudera
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Alexey Kovyrin
>>>>>>>>>> http://kovyrin.net/
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Alexey Kovyrin
>>>>>>>> http://kovyrin.net/
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Alexey Kovyrin
>>>>>> http://kovyrin.net/
>>>>>>
>>>>>
>>>>
>>>
>>>
>>
>
>
Re: thrift for hbase in CDH3 broken ?
Posted by Igor Ranitovic <ir...@gmail.com>.
Jinsong Hu wrote:
> I tried, this doesn't work. I noticed
> $transport->open();
> is missing in this code. so I added it.
Yup. Sorry about that. Copy and paste error :(
> following code first successfully print all tables, then in the line
> getRow(), it throws exception, even with ruby client, the row data is there
>
>
>
> $transport->open();
>
>
> my @names=$client->getTableNames();
>
> print Dumper(@names);
> print "\n";
>
> my $row = $client->getRow('table12345', "key123");
>
> print Dumper($row);
> print "\n";
>
>
>
> $transport->close();
So you can scan META table on the master, but can fetch a row from a RS.
Are there any firewalls in place ? Are you running thrift servers on the
same nodes as region servers? What kind of exception do you get?
i.
> --------------------------------------------------
> From: "Igor Ranitovic" <ir...@gmail.com>
> Sent: Friday, September 03, 2010 11:45 AM
> To: <us...@hbase.apache.org>
> Subject: Re: thrift for hbase in CDH3 broken ?
>
>> Not should what the test code is...would this test your setup?
>>
>>
>> #!/usr/bin/env perl
>>
>> use strict;
>> use warnings;
>>
>> use Thrift::BinaryProtocol;
>> use Thrift::BufferedTransport;
>> use Thrift::Socket;
>> use Hbase::Hbase;
>> use Data::Dumper;
>>
>> my $sock = Thrift::Socket->new('127.0.0.1', '9090');
>> $sock->setRecvTimeout(60000);
>> my $transport = Thrift::BufferedTransport->new($sock);
>> my $protocol = Thrift::BinaryProtocol->new($transport);
>> my $client = Hbase::HbaseClient->new($protocol);
>>
>> my $row = $client->getRow('table_test', 'row_123');
>> print Dumper($row);
>>
>> $transport->close();
>>
>>
>> BTW, I am not sure why you would want to use java to talk to the HBase
>> via the thirft server.
>>
>> i.
>>
>>
>> Jinsong Hu wrote:
>>> by the way, does anybody have a perl version of the test code ?
>>>
>>> Jimmy
>>>
>>> --------------------------------------------------
>>> From: "Jinsong Hu" <ji...@hotmail.com>
>>> Sent: Friday, September 03, 2010 11:17 AM
>>> To: <us...@hbase.apache.org>
>>> Subject: Re: thrift for hbase in CDH3 broken ?
>>>
>>>>
>>>> I tried your code and indeed it works. but the java version doesn't
>>>> work. so it looks like it is a bug
>>>> of the java library supplied by the thrift-0.2.0 version.
>>>>
>>>> Jimmy.
>>>> --------------------------------------------------
>>>> From: "Alexey Kovyrin" <al...@kovyrin.net>
>>>> Sent: Friday, September 03, 2010 12:31 AM
>>>> To: <us...@hbase.apache.org>
>>>> Subject: Re: thrift for hbase in CDH3 broken ?
>>>>
>>>>> yes, Centos 5.5 + CDH3b2
>>>>>
>>>>> On Fri, Sep 3, 2010 at 3:26 AM, Jinsong Hu <ji...@hotmail.com>
>>>>> wrote:
>>>>>> are you using CDH3 distribution ?
>>>>>>
>>>>>> Jinsong
>>>>>>
>>>>>>
>>>>>> --------------------------------------------------
>>>>>> From: "Alexey Kovyrin" <al...@kovyrin.net>
>>>>>> Sent: Friday, September 03, 2010 12:04 AM
>>>>>> To: <us...@hbase.apache.org>
>>>>>> Subject: Re: thrift for hbase in CDH3 broken ?
>>>>>>
>>>>>>> http://github.com/kovyrin/hbase-thrift-client-examples - just wrote
>>>>>>> this example and tested it in our cluster, works as expected.
>>>>>>> For this to work you'd need to install rubygems and thrift gem (gem
>>>>>>> install thrift).
>>>>>>>
>>>>>>> On Fri, Sep 3, 2010 at 12:01 AM, Jinsong Hu <ji...@hotmail.com>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Can you send me some ruby test code and so I can try against the
>>>>>>>> latest
>>>>>>>> CDH3
>>>>>>>> ?
>>>>>>>>
>>>>>>>> Jimmy.
>>>>>>>>
>>>>>>>> --------------------------------------------------
>>>>>>>> From: "Alexey Kovyrin" <al...@kovyrin.net>
>>>>>>>> Sent: Thursday, September 02, 2010 8:15 PM
>>>>>>>> To: <us...@hbase.apache.org>
>>>>>>>> Subject: Re: thrift for hbase in CDH3 broken ?
>>>>>>>>
>>>>>>>>> We use it in Scribd.com. All clients are ruby web apps.
>>>>>>>>>
>>>>>>>>> On Thu, Sep 2, 2010 at 10:49 PM, Todd Lipcon
>>>>>>>>> <to...@cloudera.com> wrote:
>>>>>>>>>>
>>>>>>>>>> On Thu, Sep 2, 2010 at 5:35 PM, Jinsong Hu
>>>>>>>>>> <ji...@hotmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Yes, I confirmed that it is indeed thrift server.
>>>>>>>>>>>
>>>>>>>>>>> and the fact that the API
>>>>>>>>>>>
>>>>>>>>>>> List<byte[]> tableNamesList=client.getTableNames();
>>>>>>>>>>>>>
>>>>>>>>>>>>> for (byte [] name : tableNamesList)
>>>>>>>>>>>>> {
>>>>>>>>>>>>> System.out.println(new String(name));
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> successfully printed all table names shows that it is indeed
>>>>>>>>>>> thrift
>>>>>>>>>>> server.
>>>>>>>>>>>
>>>>>>>>>>> if it is hue, it won't print the table names.
>>>>>>>>>>>
>>>>>>>>>>> Ah, sorry, I missed that in your original message. Not sure
>>>>>>>>>>> what's up,
>>>>>>>>>>> then
>>>>>>>>>>
>>>>>>>>>> - we don't have any changes in CDH that would affect this.
>>>>>>>>>> Anyone here
>>>>>>>>>> used
>>>>>>>>>> thrift on 0.89.20100621?
>>>>>>>>>>
>>>>>>>>>> -Todd
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Jimmy.
>>>>>>>>>>>
>>>>>>>>>>> --------------------------------------------------
>>>>>>>>>>> From: "Todd Lipcon" <to...@cloudera.com>
>>>>>>>>>>> Sent: Thursday, September 02, 2010 5:18 PM
>>>>>>>>>>>
>>>>>>>>>>> To: <us...@hbase.apache.org>
>>>>>>>>>>> Subject: Re: thrift for hbase in CDH3 broken ?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Hi Jinsong,
>>>>>>>>>>>>
>>>>>>>>>>>> Are you sure that the port you're connecting to is indeed
>>>>>>>>>>>> the thrift
>>>>>>>>>>>> server?
>>>>>>>>>>>>
>>>>>>>>>>>> Unfortunately both the HBase thrift server and the Hue namenode
>>>>>>>>>>>> plugin
>>>>>>>>>>>> listen on port 9090, so you might be having an issue where
>>>>>>>>>>>> your HBase
>>>>>>>>>>>> client
>>>>>>>>>>>> is trying to connect to the Namenode server instead of HBase.
>>>>>>>>>>>>
>>>>>>>>>>>> You can verify the ports using a command like "/sbin/fuser
>>>>>>>>>>>> -n tcp
>>>>>>>>>>>> 9090"
>>>>>>>>>>>> to
>>>>>>>>>>>> see which pid has it open, then cross reference against sudo
>>>>>>>>>>>> jps.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks
>>>>>>>>>>>> -Todd
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Sep 2, 2010 at 4:40 PM, Jinsong Hu
>>>>>>>>>>>> <ji...@hotmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Hi, There,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I am trying to test and see if thrift for hbase works. I
>>>>>>>>>>>>> followed
>>>>>>>>>>>>> the
>>>>>>>>>>>>> example from
>>>>>>>>>>>>>
>>>>>>>>>>>>> http://www.workhabit.com/labs/centos-55-and-thriftscribe
>>>>>>>>>>>>> http://incubator.apache.org/thrift/
>>>>>>>>>>>>> http://wiki.apache.org/hadoop/Hbase/ThriftApi
>>>>>>>>>>>>>
>>>>>>>>>>>>> and wrote test code: I found that client.getTableNames();
>>>>>>>>>>>>> returns all table names successfully, but the scanner never
>>>>>>>>>>>>> returned
>>>>>>>>>>>>> any record. and it even throws exception:
>>>>>>>>>>>>>
>>>>>>>>>>>>> org.apache.thrift.TApplicationException: get failed:
>>>>>>>>>>>>> unknown result
>>>>>>>>>>>>> at
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> org.apache.hadoop.hbase.thrift.generated.Hbase$Client.recv_get(Hbase.java:785)
>>>>>>>>>>>>>
>>>>>>>>>>>>> at
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> org.apache.hadoop.hbase.thrift.generated.Hbase$Client.get(Hbase.java:750)
>>>>>>>>>>>>>
>>>>>>>>>>>>> at
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> org.apache.hadoop.hbase.thrift.HbaseThriftTest.testThriftAPI(HbaseThriftTest.java:73)
>>>>>>>>>>>>>
>>>>>>>>>>>>> at
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> org.apache.hadoop.hbase.thrift.HbaseThriftTest.main(HbaseThriftTest.java:128)
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> I checked the syntax and the table and confirmed that the
>>>>>>>>>>>>> table does
>>>>>>>>>>>>> have
>>>>>>>>>>>>> record and it should
>>>>>>>>>>>>> return result, but it doesn't. Can anybody tell me what is
>>>>>>>>>>>>> wrong ?
>>>>>>>>>>>>>
>>>>>>>>>>>>> At this time, I am suspecting the thrift server shipped
>>>>>>>>>>>>> with CDH3
>>>>>>>>>>>>> may
>>>>>>>>>>>>> have
>>>>>>>>>>>>> problem. if there anyway
>>>>>>>>>>>>> I can get this test code working ?
>>>>>>>>>>>>>
>>>>>>>>>>>>> I have to use thrift-0.2.0-incubating.tar.gz , as I found
>>>>>>>>>>>>> that the
>>>>>>>>>>>>> code
>>>>>>>>>>>>> generated from hbase-thrift project
>>>>>>>>>>>>> doesn't even compile with 0.3.0 and 0.4.0 thrift.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Jimmy.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> public void testThriftAPI()
>>>>>>>>>>>>> {
>>>>>>>>>>>>> // Make socket
>>>>>>>>>>>>> TSocket transport = new TSocket("10.20.12.13", 9090);
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> // Wrap in a protocol
>>>>>>>>>>>>> TBinaryProtocol protocol = new TBinaryProtocol(transport);
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hbase.Client client = new Hbase.Client(protocol);
>>>>>>>>>>>>>
>>>>>>>>>>>>> try {
>>>>>>>>>>>>> transport.open();
>>>>>>>>>>>>> List<byte[]> tableNamesList=client.getTableNames();
>>>>>>>>>>>>> for (byte [] name : tableNamesList)
>>>>>>>>>>>>> {
>>>>>>>>>>>>> System.out.println(new String(name));
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>> String tableName="HEARTBEAT_CLUSTER";
>>>>>>>>>>>>> String startRow="";
>>>>>>>>>>>>> List<byte[]> columns= new ArrayList<byte[]>();
>>>>>>>>>>>>> columns.add("fields:time_format".getBytes());
>>>>>>>>>>>>> columns.add("fields:customer_id".getBytes());
>>>>>>>>>>>>>
>>>>>>>>>>>>> int aScannerId=0;
>>>>>>>>>>>>> try {
>>>>>>>>>>>>>
>>>>>>>>>>>>> TCell cell=client.get(tableName.getBytes(),
>>>>>>>>>>>>> "2010-08\tproduction-2".getBytes(),
>>>>>>>>>>>>> "fields:customer_id".getBytes());
>>>>>>>>>>>>> if (cell != null)
>>>>>>>>>>>>> {
>>>>>>>>>>>>> String value= new String( cell.value);
>>>>>>>>>>>>> System.out.println(value);
>>>>>>>>>>>>> }
>>>>>>>>>>>>> aScannerId=client.scannerOpen(tableName.getBytes(),
>>>>>>>>>>>>> startRow.getBytes(), columns);
>>>>>>>>>>>>> TRowResult result=null;
>>>>>>>>>>>>> for (int i=0; i< 10; i++)
>>>>>>>>>>>>> {
>>>>>>>>>>>>> try {
>>>>>>>>>>>>> result=client.scannerGet(aScannerId);
>>>>>>>>>>>>> } catch (Exception ex1)
>>>>>>>>>>>>> {
>>>>>>>>>>>>>
>>>>>>>>>>>>> }
>>>>>>>>>>>>> if (result ==null) break;
>>>>>>>>>>>>>
>>>>>>>>>>>>> String
>>>>>>>>>>>>> timeFormat=result.getFieldValue(0).toString();
>>>>>>>>>>>>> String
>>>>>>>>>>>>> customerId=result.getFieldValue(1).toString();
>>>>>>>>>>>>> System.out.println(timeFormat + "\t" +
>>>>>>>>>>>>> customerId);
>>>>>>>>>>>>> }
>>>>>>>>>>>>> } catch (Exception ex)
>>>>>>>>>>>>> {
>>>>>>>>>>>>> ex.printStackTrace();
>>>>>>>>>>>>> }
>>>>>>>>>>>>> finally {
>>>>>>>>>>>>> if (aScannerId >0 )
>>>>>>>>>>>>> client.scannerClose(aScannerId);
>>>>>>>>>>>>> }
>>>>>>>>>>>>> } catch (Exception exp)
>>>>>>>>>>>>> {
>>>>>>>>>>>>> exp.printStackTrace();
>>>>>>>>>>>>> }
>>>>>>>>>>>>> finally{
>>>>>>>>>>>>> transport.close();
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Todd Lipcon
>>>>>>>>>>>> Software Engineer, Cloudera
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Todd Lipcon
>>>>>>>>>> Software Engineer, Cloudera
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Alexey Kovyrin
>>>>>>>>> http://kovyrin.net/
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Alexey Kovyrin
>>>>>>> http://kovyrin.net/
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Alexey Kovyrin
>>>>> http://kovyrin.net/
>>>>>
>>>>
>>>
>>
>>
>
Re: thrift for hbase in CDH3 broken ?
Posted by Jinsong Hu <ji...@hotmail.com>.
anybody has a fix for the perl version of the hbase thrift API yet ?
Jimmy.
--------------------------------------------------
From: "Jinsong Hu" <ji...@hotmail.com>
Sent: Friday, September 03, 2010 12:16 PM
To: <us...@hbase.apache.org>
Subject: Re: thrift for hbase in CDH3 broken ?
> I tried, this doesn't work. I noticed
> $transport->open();
> is missing in this code. so I added it.
>
> following code first successfully print all tables, then in the line
> getRow(), it throws exception, even with ruby client, the row data is
> there
>
>
>
> $transport->open();
>
>
> my @names=$client->getTableNames();
>
> print Dumper(@names);
> print "\n";
>
> my $row = $client->getRow('table12345', "key123");
>
> print Dumper($row);
> print "\n";
>
>
>
> $transport->close();
>
>
>
>
> --------------------------------------------------
> From: "Igor Ranitovic" <ir...@gmail.com>
> Sent: Friday, September 03, 2010 11:45 AM
> To: <us...@hbase.apache.org>
> Subject: Re: thrift for hbase in CDH3 broken ?
>
>> Not should what the test code is...would this test your setup?
>>
>>
>> #!/usr/bin/env perl
>>
>> use strict;
>> use warnings;
>>
>> use Thrift::BinaryProtocol;
>> use Thrift::BufferedTransport;
>> use Thrift::Socket;
>> use Hbase::Hbase;
>> use Data::Dumper;
>>
>> my $sock = Thrift::Socket->new('127.0.0.1', '9090');
>> $sock->setRecvTimeout(60000);
>> my $transport = Thrift::BufferedTransport->new($sock);
>> my $protocol = Thrift::BinaryProtocol->new($transport);
>> my $client = Hbase::HbaseClient->new($protocol);
>>
>> my $row = $client->getRow('table_test', 'row_123');
>> print Dumper($row);
>>
>> $transport->close();
>>
>>
>> BTW, I am not sure why you would want to use java to talk to the HBase
>> via the thirft server.
>>
>> i.
>>
>>
>> Jinsong Hu wrote:
>>> by the way, does anybody have a perl version of the test code ?
>>>
>>> Jimmy
>>>
>>> --------------------------------------------------
>>> From: "Jinsong Hu" <ji...@hotmail.com>
>>> Sent: Friday, September 03, 2010 11:17 AM
>>> To: <us...@hbase.apache.org>
>>> Subject: Re: thrift for hbase in CDH3 broken ?
>>>
>>>>
>>>> I tried your code and indeed it works. but the java version doesn't
>>>> work. so it looks like it is a bug
>>>> of the java library supplied by the thrift-0.2.0 version.
>>>>
>>>> Jimmy.
>>>> --------------------------------------------------
>>>> From: "Alexey Kovyrin" <al...@kovyrin.net>
>>>> Sent: Friday, September 03, 2010 12:31 AM
>>>> To: <us...@hbase.apache.org>
>>>> Subject: Re: thrift for hbase in CDH3 broken ?
>>>>
>>>>> yes, Centos 5.5 + CDH3b2
>>>>>
>>>>> On Fri, Sep 3, 2010 at 3:26 AM, Jinsong Hu <ji...@hotmail.com>
>>>>> wrote:
>>>>>> are you using CDH3 distribution ?
>>>>>>
>>>>>> Jinsong
>>>>>>
>>>>>>
>>>>>> --------------------------------------------------
>>>>>> From: "Alexey Kovyrin" <al...@kovyrin.net>
>>>>>> Sent: Friday, September 03, 2010 12:04 AM
>>>>>> To: <us...@hbase.apache.org>
>>>>>> Subject: Re: thrift for hbase in CDH3 broken ?
>>>>>>
>>>>>>> http://github.com/kovyrin/hbase-thrift-client-examples - just wrote
>>>>>>> this example and tested it in our cluster, works as expected.
>>>>>>> For this to work you'd need to install rubygems and thrift gem (gem
>>>>>>> install thrift).
>>>>>>>
>>>>>>> On Fri, Sep 3, 2010 at 12:01 AM, Jinsong Hu <ji...@hotmail.com>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Can you send me some ruby test code and so I can try against the
>>>>>>>> latest
>>>>>>>> CDH3
>>>>>>>> ?
>>>>>>>>
>>>>>>>> Jimmy.
>>>>>>>>
>>>>>>>> --------------------------------------------------
>>>>>>>> From: "Alexey Kovyrin" <al...@kovyrin.net>
>>>>>>>> Sent: Thursday, September 02, 2010 8:15 PM
>>>>>>>> To: <us...@hbase.apache.org>
>>>>>>>> Subject: Re: thrift for hbase in CDH3 broken ?
>>>>>>>>
>>>>>>>>> We use it in Scribd.com. All clients are ruby web apps.
>>>>>>>>>
>>>>>>>>> On Thu, Sep 2, 2010 at 10:49 PM, Todd Lipcon <to...@cloudera.com>
>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> On Thu, Sep 2, 2010 at 5:35 PM, Jinsong Hu
>>>>>>>>>> <ji...@hotmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Yes, I confirmed that it is indeed thrift server.
>>>>>>>>>>>
>>>>>>>>>>> and the fact that the API
>>>>>>>>>>>
>>>>>>>>>>> List<byte[]> tableNamesList=client.getTableNames();
>>>>>>>>>>>>>
>>>>>>>>>>>>> for (byte [] name : tableNamesList)
>>>>>>>>>>>>> {
>>>>>>>>>>>>> System.out.println(new String(name));
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> successfully printed all table names shows that it is indeed
>>>>>>>>>>> thrift
>>>>>>>>>>> server.
>>>>>>>>>>>
>>>>>>>>>>> if it is hue, it won't print the table names.
>>>>>>>>>>>
>>>>>>>>>>> Ah, sorry, I missed that in your original message. Not sure
>>>>>>>>>>> what's up,
>>>>>>>>>>> then
>>>>>>>>>>
>>>>>>>>>> - we don't have any changes in CDH that would affect this. Anyone
>>>>>>>>>> here
>>>>>>>>>> used
>>>>>>>>>> thrift on 0.89.20100621?
>>>>>>>>>>
>>>>>>>>>> -Todd
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Jimmy.
>>>>>>>>>>>
>>>>>>>>>>> --------------------------------------------------
>>>>>>>>>>> From: "Todd Lipcon" <to...@cloudera.com>
>>>>>>>>>>> Sent: Thursday, September 02, 2010 5:18 PM
>>>>>>>>>>>
>>>>>>>>>>> To: <us...@hbase.apache.org>
>>>>>>>>>>> Subject: Re: thrift for hbase in CDH3 broken ?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Hi Jinsong,
>>>>>>>>>>>>
>>>>>>>>>>>> Are you sure that the port you're connecting to is indeed the
>>>>>>>>>>>> thrift
>>>>>>>>>>>> server?
>>>>>>>>>>>>
>>>>>>>>>>>> Unfortunately both the HBase thrift server and the Hue namenode
>>>>>>>>>>>> plugin
>>>>>>>>>>>> listen on port 9090, so you might be having an issue where your
>>>>>>>>>>>> HBase
>>>>>>>>>>>> client
>>>>>>>>>>>> is trying to connect to the Namenode server instead of HBase.
>>>>>>>>>>>>
>>>>>>>>>>>> You can verify the ports using a command like "/sbin/fuser -n
>>>>>>>>>>>> tcp
>>>>>>>>>>>> 9090"
>>>>>>>>>>>> to
>>>>>>>>>>>> see which pid has it open, then cross reference against sudo
>>>>>>>>>>>> jps.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks
>>>>>>>>>>>> -Todd
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Sep 2, 2010 at 4:40 PM, Jinsong Hu
>>>>>>>>>>>> <ji...@hotmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Hi, There,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I am trying to test and see if thrift for hbase works. I
>>>>>>>>>>>>> followed
>>>>>>>>>>>>> the
>>>>>>>>>>>>> example from
>>>>>>>>>>>>>
>>>>>>>>>>>>> http://www.workhabit.com/labs/centos-55-and-thriftscribe
>>>>>>>>>>>>> http://incubator.apache.org/thrift/
>>>>>>>>>>>>> http://wiki.apache.org/hadoop/Hbase/ThriftApi
>>>>>>>>>>>>>
>>>>>>>>>>>>> and wrote test code: I found that client.getTableNames();
>>>>>>>>>>>>> returns all table names successfully, but the scanner never
>>>>>>>>>>>>> returned
>>>>>>>>>>>>> any record. and it even throws exception:
>>>>>>>>>>>>>
>>>>>>>>>>>>> org.apache.thrift.TApplicationException: get failed: unknown
>>>>>>>>>>>>> result
>>>>>>>>>>>>> at
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> org.apache.hadoop.hbase.thrift.generated.Hbase$Client.recv_get(Hbase.java:785)
>>>>>>>>>>>>> at
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> org.apache.hadoop.hbase.thrift.generated.Hbase$Client.get(Hbase.java:750)
>>>>>>>>>>>>> at
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> org.apache.hadoop.hbase.thrift.HbaseThriftTest.testThriftAPI(HbaseThriftTest.java:73)
>>>>>>>>>>>>> at
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> org.apache.hadoop.hbase.thrift.HbaseThriftTest.main(HbaseThriftTest.java:128)
>>>>>>>>>>>>>
>>>>>>>>>>>>> I checked the syntax and the table and confirmed that the
>>>>>>>>>>>>> table does
>>>>>>>>>>>>> have
>>>>>>>>>>>>> record and it should
>>>>>>>>>>>>> return result, but it doesn't. Can anybody tell me what is
>>>>>>>>>>>>> wrong ?
>>>>>>>>>>>>>
>>>>>>>>>>>>> At this time, I am suspecting the thrift server shipped with
>>>>>>>>>>>>> CDH3
>>>>>>>>>>>>> may
>>>>>>>>>>>>> have
>>>>>>>>>>>>> problem. if there anyway
>>>>>>>>>>>>> I can get this test code working ?
>>>>>>>>>>>>>
>>>>>>>>>>>>> I have to use thrift-0.2.0-incubating.tar.gz , as I found that
>>>>>>>>>>>>> the
>>>>>>>>>>>>> code
>>>>>>>>>>>>> generated from hbase-thrift project
>>>>>>>>>>>>> doesn't even compile with 0.3.0 and 0.4.0 thrift.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Jimmy.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> public void testThriftAPI()
>>>>>>>>>>>>> {
>>>>>>>>>>>>> // Make socket
>>>>>>>>>>>>> TSocket transport = new TSocket("10.20.12.13", 9090);
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> // Wrap in a protocol
>>>>>>>>>>>>> TBinaryProtocol protocol = new TBinaryProtocol(transport);
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hbase.Client client = new Hbase.Client(protocol);
>>>>>>>>>>>>>
>>>>>>>>>>>>> try {
>>>>>>>>>>>>> transport.open();
>>>>>>>>>>>>> List<byte[]> tableNamesList=client.getTableNames();
>>>>>>>>>>>>> for (byte [] name : tableNamesList)
>>>>>>>>>>>>> {
>>>>>>>>>>>>> System.out.println(new String(name));
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>> String tableName="HEARTBEAT_CLUSTER";
>>>>>>>>>>>>> String startRow="";
>>>>>>>>>>>>> List<byte[]> columns= new ArrayList<byte[]>();
>>>>>>>>>>>>> columns.add("fields:time_format".getBytes());
>>>>>>>>>>>>> columns.add("fields:customer_id".getBytes());
>>>>>>>>>>>>>
>>>>>>>>>>>>> int aScannerId=0;
>>>>>>>>>>>>> try {
>>>>>>>>>>>>>
>>>>>>>>>>>>> TCell cell=client.get(tableName.getBytes(),
>>>>>>>>>>>>> "2010-08\tproduction-2".getBytes(),
>>>>>>>>>>>>> "fields:customer_id".getBytes());
>>>>>>>>>>>>> if (cell != null)
>>>>>>>>>>>>> {
>>>>>>>>>>>>> String value= new String( cell.value);
>>>>>>>>>>>>> System.out.println(value);
>>>>>>>>>>>>> }
>>>>>>>>>>>>> aScannerId=client.scannerOpen(tableName.getBytes(),
>>>>>>>>>>>>> startRow.getBytes(), columns);
>>>>>>>>>>>>> TRowResult result=null;
>>>>>>>>>>>>> for (int i=0; i< 10; i++)
>>>>>>>>>>>>> {
>>>>>>>>>>>>> try {
>>>>>>>>>>>>> result=client.scannerGet(aScannerId);
>>>>>>>>>>>>> } catch (Exception ex1)
>>>>>>>>>>>>> {
>>>>>>>>>>>>>
>>>>>>>>>>>>> }
>>>>>>>>>>>>> if (result ==null) break;
>>>>>>>>>>>>>
>>>>>>>>>>>>> String
>>>>>>>>>>>>> timeFormat=result.getFieldValue(0).toString();
>>>>>>>>>>>>> String
>>>>>>>>>>>>> customerId=result.getFieldValue(1).toString();
>>>>>>>>>>>>> System.out.println(timeFormat + "\t" +
>>>>>>>>>>>>> customerId);
>>>>>>>>>>>>> }
>>>>>>>>>>>>> } catch (Exception ex)
>>>>>>>>>>>>> {
>>>>>>>>>>>>> ex.printStackTrace();
>>>>>>>>>>>>> }
>>>>>>>>>>>>> finally {
>>>>>>>>>>>>> if (aScannerId >0 )
>>>>>>>>>>>>> client.scannerClose(aScannerId);
>>>>>>>>>>>>> }
>>>>>>>>>>>>> } catch (Exception exp)
>>>>>>>>>>>>> {
>>>>>>>>>>>>> exp.printStackTrace();
>>>>>>>>>>>>> }
>>>>>>>>>>>>> finally{
>>>>>>>>>>>>> transport.close();
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Todd Lipcon
>>>>>>>>>>>> Software Engineer, Cloudera
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Todd Lipcon
>>>>>>>>>> Software Engineer, Cloudera
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Alexey Kovyrin
>>>>>>>>> http://kovyrin.net/
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Alexey Kovyrin
>>>>>>> http://kovyrin.net/
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Alexey Kovyrin
>>>>> http://kovyrin.net/
>>>>>
>>>>
>>>
>>
>>
>
Re: thrift for hbase in CDH3 broken ?
Posted by Jinsong Hu <ji...@hotmail.com>.
I tried, this doesn't work. I noticed
$transport->open();
is missing in this code. so I added it.
following code first successfully print all tables, then in the line
getRow(), it throws exception, even with ruby client, the row data is there
$transport->open();
my @names=$client->getTableNames();
print Dumper(@names);
print "\n";
my $row = $client->getRow('table12345', "key123");
print Dumper($row);
print "\n";
$transport->close();
--------------------------------------------------
From: "Igor Ranitovic" <ir...@gmail.com>
Sent: Friday, September 03, 2010 11:45 AM
To: <us...@hbase.apache.org>
Subject: Re: thrift for hbase in CDH3 broken ?
> Not should what the test code is...would this test your setup?
>
>
> #!/usr/bin/env perl
>
> use strict;
> use warnings;
>
> use Thrift::BinaryProtocol;
> use Thrift::BufferedTransport;
> use Thrift::Socket;
> use Hbase::Hbase;
> use Data::Dumper;
>
> my $sock = Thrift::Socket->new('127.0.0.1', '9090');
> $sock->setRecvTimeout(60000);
> my $transport = Thrift::BufferedTransport->new($sock);
> my $protocol = Thrift::BinaryProtocol->new($transport);
> my $client = Hbase::HbaseClient->new($protocol);
>
> my $row = $client->getRow('table_test', 'row_123');
> print Dumper($row);
>
> $transport->close();
>
>
> BTW, I am not sure why you would want to use java to talk to the HBase via
> the thirft server.
>
> i.
>
>
> Jinsong Hu wrote:
>> by the way, does anybody have a perl version of the test code ?
>>
>> Jimmy
>>
>> --------------------------------------------------
>> From: "Jinsong Hu" <ji...@hotmail.com>
>> Sent: Friday, September 03, 2010 11:17 AM
>> To: <us...@hbase.apache.org>
>> Subject: Re: thrift for hbase in CDH3 broken ?
>>
>>>
>>> I tried your code and indeed it works. but the java version doesn't
>>> work. so it looks like it is a bug
>>> of the java library supplied by the thrift-0.2.0 version.
>>>
>>> Jimmy.
>>> --------------------------------------------------
>>> From: "Alexey Kovyrin" <al...@kovyrin.net>
>>> Sent: Friday, September 03, 2010 12:31 AM
>>> To: <us...@hbase.apache.org>
>>> Subject: Re: thrift for hbase in CDH3 broken ?
>>>
>>>> yes, Centos 5.5 + CDH3b2
>>>>
>>>> On Fri, Sep 3, 2010 at 3:26 AM, Jinsong Hu <ji...@hotmail.com>
>>>> wrote:
>>>>> are you using CDH3 distribution ?
>>>>>
>>>>> Jinsong
>>>>>
>>>>>
>>>>> --------------------------------------------------
>>>>> From: "Alexey Kovyrin" <al...@kovyrin.net>
>>>>> Sent: Friday, September 03, 2010 12:04 AM
>>>>> To: <us...@hbase.apache.org>
>>>>> Subject: Re: thrift for hbase in CDH3 broken ?
>>>>>
>>>>>> http://github.com/kovyrin/hbase-thrift-client-examples - just wrote
>>>>>> this example and tested it in our cluster, works as expected.
>>>>>> For this to work you'd need to install rubygems and thrift gem (gem
>>>>>> install thrift).
>>>>>>
>>>>>> On Fri, Sep 3, 2010 at 12:01 AM, Jinsong Hu <ji...@hotmail.com>
>>>>>> wrote:
>>>>>>>
>>>>>>> Can you send me some ruby test code and so I can try against the
>>>>>>> latest
>>>>>>> CDH3
>>>>>>> ?
>>>>>>>
>>>>>>> Jimmy.
>>>>>>>
>>>>>>> --------------------------------------------------
>>>>>>> From: "Alexey Kovyrin" <al...@kovyrin.net>
>>>>>>> Sent: Thursday, September 02, 2010 8:15 PM
>>>>>>> To: <us...@hbase.apache.org>
>>>>>>> Subject: Re: thrift for hbase in CDH3 broken ?
>>>>>>>
>>>>>>>> We use it in Scribd.com. All clients are ruby web apps.
>>>>>>>>
>>>>>>>> On Thu, Sep 2, 2010 at 10:49 PM, Todd Lipcon <to...@cloudera.com>
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> On Thu, Sep 2, 2010 at 5:35 PM, Jinsong Hu
>>>>>>>>> <ji...@hotmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Yes, I confirmed that it is indeed thrift server.
>>>>>>>>>>
>>>>>>>>>> and the fact that the API
>>>>>>>>>>
>>>>>>>>>> List<byte[]> tableNamesList=client.getTableNames();
>>>>>>>>>>>>
>>>>>>>>>>>> for (byte [] name : tableNamesList)
>>>>>>>>>>>> {
>>>>>>>>>>>> System.out.println(new String(name));
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> successfully printed all table names shows that it is indeed
>>>>>>>>>> thrift
>>>>>>>>>> server.
>>>>>>>>>>
>>>>>>>>>> if it is hue, it won't print the table names.
>>>>>>>>>>
>>>>>>>>>> Ah, sorry, I missed that in your original message. Not sure
>>>>>>>>>> what's up,
>>>>>>>>>> then
>>>>>>>>>
>>>>>>>>> - we don't have any changes in CDH that would affect this. Anyone
>>>>>>>>> here
>>>>>>>>> used
>>>>>>>>> thrift on 0.89.20100621?
>>>>>>>>>
>>>>>>>>> -Todd
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Jimmy.
>>>>>>>>>>
>>>>>>>>>> --------------------------------------------------
>>>>>>>>>> From: "Todd Lipcon" <to...@cloudera.com>
>>>>>>>>>> Sent: Thursday, September 02, 2010 5:18 PM
>>>>>>>>>>
>>>>>>>>>> To: <us...@hbase.apache.org>
>>>>>>>>>> Subject: Re: thrift for hbase in CDH3 broken ?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Hi Jinsong,
>>>>>>>>>>>
>>>>>>>>>>> Are you sure that the port you're connecting to is indeed the
>>>>>>>>>>> thrift
>>>>>>>>>>> server?
>>>>>>>>>>>
>>>>>>>>>>> Unfortunately both the HBase thrift server and the Hue namenode
>>>>>>>>>>> plugin
>>>>>>>>>>> listen on port 9090, so you might be having an issue where your
>>>>>>>>>>> HBase
>>>>>>>>>>> client
>>>>>>>>>>> is trying to connect to the Namenode server instead of HBase.
>>>>>>>>>>>
>>>>>>>>>>> You can verify the ports using a command like "/sbin/fuser -n
>>>>>>>>>>> tcp
>>>>>>>>>>> 9090"
>>>>>>>>>>> to
>>>>>>>>>>> see which pid has it open, then cross reference against sudo
>>>>>>>>>>> jps.
>>>>>>>>>>>
>>>>>>>>>>> Thanks
>>>>>>>>>>> -Todd
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Sep 2, 2010 at 4:40 PM, Jinsong Hu
>>>>>>>>>>> <ji...@hotmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi, There,
>>>>>>>>>>>>
>>>>>>>>>>>> I am trying to test and see if thrift for hbase works. I
>>>>>>>>>>>> followed
>>>>>>>>>>>> the
>>>>>>>>>>>> example from
>>>>>>>>>>>>
>>>>>>>>>>>> http://www.workhabit.com/labs/centos-55-and-thriftscribe
>>>>>>>>>>>> http://incubator.apache.org/thrift/
>>>>>>>>>>>> http://wiki.apache.org/hadoop/Hbase/ThriftApi
>>>>>>>>>>>>
>>>>>>>>>>>> and wrote test code: I found that client.getTableNames();
>>>>>>>>>>>> returns all table names successfully, but the scanner never
>>>>>>>>>>>> returned
>>>>>>>>>>>> any record. and it even throws exception:
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.thrift.TApplicationException: get failed: unknown
>>>>>>>>>>>> result
>>>>>>>>>>>> at
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.hadoop.hbase.thrift.generated.Hbase$Client.recv_get(Hbase.java:785)
>>>>>>>>>>>> at
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.hadoop.hbase.thrift.generated.Hbase$Client.get(Hbase.java:750)
>>>>>>>>>>>> at
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.hadoop.hbase.thrift.HbaseThriftTest.testThriftAPI(HbaseThriftTest.java:73)
>>>>>>>>>>>> at
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.hadoop.hbase.thrift.HbaseThriftTest.main(HbaseThriftTest.java:128)
>>>>>>>>>>>>
>>>>>>>>>>>> I checked the syntax and the table and confirmed that the table
>>>>>>>>>>>> does
>>>>>>>>>>>> have
>>>>>>>>>>>> record and it should
>>>>>>>>>>>> return result, but it doesn't. Can anybody tell me what is
>>>>>>>>>>>> wrong ?
>>>>>>>>>>>>
>>>>>>>>>>>> At this time, I am suspecting the thrift server shipped with
>>>>>>>>>>>> CDH3
>>>>>>>>>>>> may
>>>>>>>>>>>> have
>>>>>>>>>>>> problem. if there anyway
>>>>>>>>>>>> I can get this test code working ?
>>>>>>>>>>>>
>>>>>>>>>>>> I have to use thrift-0.2.0-incubating.tar.gz , as I found that
>>>>>>>>>>>> the
>>>>>>>>>>>> code
>>>>>>>>>>>> generated from hbase-thrift project
>>>>>>>>>>>> doesn't even compile with 0.3.0 and 0.4.0 thrift.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Jimmy.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> public void testThriftAPI()
>>>>>>>>>>>> {
>>>>>>>>>>>> // Make socket
>>>>>>>>>>>> TSocket transport = new TSocket("10.20.12.13", 9090);
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> // Wrap in a protocol
>>>>>>>>>>>> TBinaryProtocol protocol = new TBinaryProtocol(transport);
>>>>>>>>>>>>
>>>>>>>>>>>> Hbase.Client client = new Hbase.Client(protocol);
>>>>>>>>>>>>
>>>>>>>>>>>> try {
>>>>>>>>>>>> transport.open();
>>>>>>>>>>>> List<byte[]> tableNamesList=client.getTableNames();
>>>>>>>>>>>> for (byte [] name : tableNamesList)
>>>>>>>>>>>> {
>>>>>>>>>>>> System.out.println(new String(name));
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>> String tableName="HEARTBEAT_CLUSTER";
>>>>>>>>>>>> String startRow="";
>>>>>>>>>>>> List<byte[]> columns= new ArrayList<byte[]>();
>>>>>>>>>>>> columns.add("fields:time_format".getBytes());
>>>>>>>>>>>> columns.add("fields:customer_id".getBytes());
>>>>>>>>>>>>
>>>>>>>>>>>> int aScannerId=0;
>>>>>>>>>>>> try {
>>>>>>>>>>>>
>>>>>>>>>>>> TCell cell=client.get(tableName.getBytes(),
>>>>>>>>>>>> "2010-08\tproduction-2".getBytes(),
>>>>>>>>>>>> "fields:customer_id".getBytes());
>>>>>>>>>>>> if (cell != null)
>>>>>>>>>>>> {
>>>>>>>>>>>> String value= new String( cell.value);
>>>>>>>>>>>> System.out.println(value);
>>>>>>>>>>>> }
>>>>>>>>>>>> aScannerId=client.scannerOpen(tableName.getBytes(),
>>>>>>>>>>>> startRow.getBytes(), columns);
>>>>>>>>>>>> TRowResult result=null;
>>>>>>>>>>>> for (int i=0; i< 10; i++)
>>>>>>>>>>>> {
>>>>>>>>>>>> try {
>>>>>>>>>>>> result=client.scannerGet(aScannerId);
>>>>>>>>>>>> } catch (Exception ex1)
>>>>>>>>>>>> {
>>>>>>>>>>>>
>>>>>>>>>>>> }
>>>>>>>>>>>> if (result ==null) break;
>>>>>>>>>>>>
>>>>>>>>>>>> String
>>>>>>>>>>>> timeFormat=result.getFieldValue(0).toString();
>>>>>>>>>>>> String
>>>>>>>>>>>> customerId=result.getFieldValue(1).toString();
>>>>>>>>>>>> System.out.println(timeFormat + "\t" +
>>>>>>>>>>>> customerId);
>>>>>>>>>>>> }
>>>>>>>>>>>> } catch (Exception ex)
>>>>>>>>>>>> {
>>>>>>>>>>>> ex.printStackTrace();
>>>>>>>>>>>> }
>>>>>>>>>>>> finally {
>>>>>>>>>>>> if (aScannerId >0 ) client.scannerClose(aScannerId);
>>>>>>>>>>>> }
>>>>>>>>>>>> } catch (Exception exp)
>>>>>>>>>>>> {
>>>>>>>>>>>> exp.printStackTrace();
>>>>>>>>>>>> }
>>>>>>>>>>>> finally{
>>>>>>>>>>>> transport.close();
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Todd Lipcon
>>>>>>>>>>> Software Engineer, Cloudera
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Todd Lipcon
>>>>>>>>> Software Engineer, Cloudera
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Alexey Kovyrin
>>>>>>>> http://kovyrin.net/
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Alexey Kovyrin
>>>>>> http://kovyrin.net/
>>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Alexey Kovyrin
>>>> http://kovyrin.net/
>>>>
>>>
>>
>
>
Re: thrift for hbase in CDH3 broken ?
Posted by Igor Ranitovic <ir...@gmail.com>.
Not should what the test code is...would this test your setup?
#!/usr/bin/env perl
use strict;
use warnings;
use Thrift::BinaryProtocol;
use Thrift::BufferedTransport;
use Thrift::Socket;
use Hbase::Hbase;
use Data::Dumper;
my $sock = Thrift::Socket->new('127.0.0.1', '9090');
$sock->setRecvTimeout(60000);
my $transport = Thrift::BufferedTransport->new($sock);
my $protocol = Thrift::BinaryProtocol->new($transport);
my $client = Hbase::HbaseClient->new($protocol);
my $row = $client->getRow('table_test', 'row_123');
print Dumper($row);
$transport->close();
BTW, I am not sure why you would want to use java to talk to the HBase
via the thirft server.
i.
Jinsong Hu wrote:
> by the way, does anybody have a perl version of the test code ?
>
> Jimmy
>
> --------------------------------------------------
> From: "Jinsong Hu" <ji...@hotmail.com>
> Sent: Friday, September 03, 2010 11:17 AM
> To: <us...@hbase.apache.org>
> Subject: Re: thrift for hbase in CDH3 broken ?
>
>>
>> I tried your code and indeed it works. but the java version doesn't
>> work. so it looks like it is a bug
>> of the java library supplied by the thrift-0.2.0 version.
>>
>> Jimmy.
>> --------------------------------------------------
>> From: "Alexey Kovyrin" <al...@kovyrin.net>
>> Sent: Friday, September 03, 2010 12:31 AM
>> To: <us...@hbase.apache.org>
>> Subject: Re: thrift for hbase in CDH3 broken ?
>>
>>> yes, Centos 5.5 + CDH3b2
>>>
>>> On Fri, Sep 3, 2010 at 3:26 AM, Jinsong Hu <ji...@hotmail.com>
>>> wrote:
>>>> are you using CDH3 distribution ?
>>>>
>>>> Jinsong
>>>>
>>>>
>>>> --------------------------------------------------
>>>> From: "Alexey Kovyrin" <al...@kovyrin.net>
>>>> Sent: Friday, September 03, 2010 12:04 AM
>>>> To: <us...@hbase.apache.org>
>>>> Subject: Re: thrift for hbase in CDH3 broken ?
>>>>
>>>>> http://github.com/kovyrin/hbase-thrift-client-examples - just wrote
>>>>> this example and tested it in our cluster, works as expected.
>>>>> For this to work you'd need to install rubygems and thrift gem (gem
>>>>> install thrift).
>>>>>
>>>>> On Fri, Sep 3, 2010 at 12:01 AM, Jinsong Hu <ji...@hotmail.com>
>>>>> wrote:
>>>>>>
>>>>>> Can you send me some ruby test code and so I can try against the
>>>>>> latest
>>>>>> CDH3
>>>>>> ?
>>>>>>
>>>>>> Jimmy.
>>>>>>
>>>>>> --------------------------------------------------
>>>>>> From: "Alexey Kovyrin" <al...@kovyrin.net>
>>>>>> Sent: Thursday, September 02, 2010 8:15 PM
>>>>>> To: <us...@hbase.apache.org>
>>>>>> Subject: Re: thrift for hbase in CDH3 broken ?
>>>>>>
>>>>>>> We use it in Scribd.com. All clients are ruby web apps.
>>>>>>>
>>>>>>> On Thu, Sep 2, 2010 at 10:49 PM, Todd Lipcon <to...@cloudera.com>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> On Thu, Sep 2, 2010 at 5:35 PM, Jinsong Hu <ji...@hotmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Yes, I confirmed that it is indeed thrift server.
>>>>>>>>>
>>>>>>>>> and the fact that the API
>>>>>>>>>
>>>>>>>>> List<byte[]> tableNamesList=client.getTableNames();
>>>>>>>>>>>
>>>>>>>>>>> for (byte [] name : tableNamesList)
>>>>>>>>>>> {
>>>>>>>>>>> System.out.println(new String(name));
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> successfully printed all table names shows that it is indeed
>>>>>>>>> thrift
>>>>>>>>> server.
>>>>>>>>>
>>>>>>>>> if it is hue, it won't print the table names.
>>>>>>>>>
>>>>>>>>> Ah, sorry, I missed that in your original message. Not sure
>>>>>>>>> what's up,
>>>>>>>>> then
>>>>>>>>
>>>>>>>> - we don't have any changes in CDH that would affect this.
>>>>>>>> Anyone here
>>>>>>>> used
>>>>>>>> thrift on 0.89.20100621?
>>>>>>>>
>>>>>>>> -Todd
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> Jimmy.
>>>>>>>>>
>>>>>>>>> --------------------------------------------------
>>>>>>>>> From: "Todd Lipcon" <to...@cloudera.com>
>>>>>>>>> Sent: Thursday, September 02, 2010 5:18 PM
>>>>>>>>>
>>>>>>>>> To: <us...@hbase.apache.org>
>>>>>>>>> Subject: Re: thrift for hbase in CDH3 broken ?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hi Jinsong,
>>>>>>>>>>
>>>>>>>>>> Are you sure that the port you're connecting to is indeed the
>>>>>>>>>> thrift
>>>>>>>>>> server?
>>>>>>>>>>
>>>>>>>>>> Unfortunately both the HBase thrift server and the Hue namenode
>>>>>>>>>> plugin
>>>>>>>>>> listen on port 9090, so you might be having an issue where
>>>>>>>>>> your HBase
>>>>>>>>>> client
>>>>>>>>>> is trying to connect to the Namenode server instead of HBase.
>>>>>>>>>>
>>>>>>>>>> You can verify the ports using a command like "/sbin/fuser -n tcp
>>>>>>>>>> 9090"
>>>>>>>>>> to
>>>>>>>>>> see which pid has it open, then cross reference against sudo jps.
>>>>>>>>>>
>>>>>>>>>> Thanks
>>>>>>>>>> -Todd
>>>>>>>>>>
>>>>>>>>>> On Thu, Sep 2, 2010 at 4:40 PM, Jinsong Hu
>>>>>>>>>> <ji...@hotmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Hi, There,
>>>>>>>>>>>
>>>>>>>>>>> I am trying to test and see if thrift for hbase works. I
>>>>>>>>>>> followed
>>>>>>>>>>> the
>>>>>>>>>>> example from
>>>>>>>>>>>
>>>>>>>>>>> http://www.workhabit.com/labs/centos-55-and-thriftscribe
>>>>>>>>>>> http://incubator.apache.org/thrift/
>>>>>>>>>>> http://wiki.apache.org/hadoop/Hbase/ThriftApi
>>>>>>>>>>>
>>>>>>>>>>> and wrote test code: I found that client.getTableNames();
>>>>>>>>>>> returns all table names successfully, but the scanner never
>>>>>>>>>>> returned
>>>>>>>>>>> any record. and it even throws exception:
>>>>>>>>>>>
>>>>>>>>>>> org.apache.thrift.TApplicationException: get failed: unknown
>>>>>>>>>>> result
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> org.apache.hadoop.hbase.thrift.generated.Hbase$Client.recv_get(Hbase.java:785)
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> org.apache.hadoop.hbase.thrift.generated.Hbase$Client.get(Hbase.java:750)
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> org.apache.hadoop.hbase.thrift.HbaseThriftTest.testThriftAPI(HbaseThriftTest.java:73)
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> org.apache.hadoop.hbase.thrift.HbaseThriftTest.main(HbaseThriftTest.java:128)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I checked the syntax and the table and confirmed that the
>>>>>>>>>>> table does
>>>>>>>>>>> have
>>>>>>>>>>> record and it should
>>>>>>>>>>> return result, but it doesn't. Can anybody tell me what is
>>>>>>>>>>> wrong ?
>>>>>>>>>>>
>>>>>>>>>>> At this time, I am suspecting the thrift server shipped with
>>>>>>>>>>> CDH3
>>>>>>>>>>> may
>>>>>>>>>>> have
>>>>>>>>>>> problem. if there anyway
>>>>>>>>>>> I can get this test code working ?
>>>>>>>>>>>
>>>>>>>>>>> I have to use thrift-0.2.0-incubating.tar.gz , as I found
>>>>>>>>>>> that the
>>>>>>>>>>> code
>>>>>>>>>>> generated from hbase-thrift project
>>>>>>>>>>> doesn't even compile with 0.3.0 and 0.4.0 thrift.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Jimmy.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> public void testThriftAPI()
>>>>>>>>>>> {
>>>>>>>>>>> // Make socket
>>>>>>>>>>> TSocket transport = new TSocket("10.20.12.13", 9090);
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> // Wrap in a protocol
>>>>>>>>>>> TBinaryProtocol protocol = new TBinaryProtocol(transport);
>>>>>>>>>>>
>>>>>>>>>>> Hbase.Client client = new Hbase.Client(protocol);
>>>>>>>>>>>
>>>>>>>>>>> try {
>>>>>>>>>>> transport.open();
>>>>>>>>>>> List<byte[]> tableNamesList=client.getTableNames();
>>>>>>>>>>> for (byte [] name : tableNamesList)
>>>>>>>>>>> {
>>>>>>>>>>> System.out.println(new String(name));
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> String tableName="HEARTBEAT_CLUSTER";
>>>>>>>>>>> String startRow="";
>>>>>>>>>>> List<byte[]> columns= new ArrayList<byte[]>();
>>>>>>>>>>> columns.add("fields:time_format".getBytes());
>>>>>>>>>>> columns.add("fields:customer_id".getBytes());
>>>>>>>>>>>
>>>>>>>>>>> int aScannerId=0;
>>>>>>>>>>> try {
>>>>>>>>>>>
>>>>>>>>>>> TCell cell=client.get(tableName.getBytes(),
>>>>>>>>>>> "2010-08\tproduction-2".getBytes(),
>>>>>>>>>>> "fields:customer_id".getBytes());
>>>>>>>>>>> if (cell != null)
>>>>>>>>>>> {
>>>>>>>>>>> String value= new String( cell.value);
>>>>>>>>>>> System.out.println(value);
>>>>>>>>>>> }
>>>>>>>>>>> aScannerId=client.scannerOpen(tableName.getBytes(),
>>>>>>>>>>> startRow.getBytes(), columns);
>>>>>>>>>>> TRowResult result=null;
>>>>>>>>>>> for (int i=0; i< 10; i++)
>>>>>>>>>>> {
>>>>>>>>>>> try {
>>>>>>>>>>> result=client.scannerGet(aScannerId);
>>>>>>>>>>> } catch (Exception ex1)
>>>>>>>>>>> {
>>>>>>>>>>>
>>>>>>>>>>> }
>>>>>>>>>>> if (result ==null) break;
>>>>>>>>>>>
>>>>>>>>>>> String
>>>>>>>>>>> timeFormat=result.getFieldValue(0).toString();
>>>>>>>>>>> String
>>>>>>>>>>> customerId=result.getFieldValue(1).toString();
>>>>>>>>>>> System.out.println(timeFormat + "\t" +
>>>>>>>>>>> customerId);
>>>>>>>>>>> }
>>>>>>>>>>> } catch (Exception ex)
>>>>>>>>>>> {
>>>>>>>>>>> ex.printStackTrace();
>>>>>>>>>>> }
>>>>>>>>>>> finally {
>>>>>>>>>>> if (aScannerId >0 ) client.scannerClose(aScannerId);
>>>>>>>>>>> }
>>>>>>>>>>> } catch (Exception exp)
>>>>>>>>>>> {
>>>>>>>>>>> exp.printStackTrace();
>>>>>>>>>>> }
>>>>>>>>>>> finally{
>>>>>>>>>>> transport.close();
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Todd Lipcon
>>>>>>>>>> Software Engineer, Cloudera
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Todd Lipcon
>>>>>>>> Software Engineer, Cloudera
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Alexey Kovyrin
>>>>>>> http://kovyrin.net/
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Alexey Kovyrin
>>>>> http://kovyrin.net/
>>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Alexey Kovyrin
>>> http://kovyrin.net/
>>>
>>
>
Re: thrift for hbase in CDH3 broken ?
Posted by Jinsong Hu <ji...@hotmail.com>.
by the way, does anybody have a perl version of the test code ?
Jimmy
--------------------------------------------------
From: "Jinsong Hu" <ji...@hotmail.com>
Sent: Friday, September 03, 2010 11:17 AM
To: <us...@hbase.apache.org>
Subject: Re: thrift for hbase in CDH3 broken ?
>
> I tried your code and indeed it works. but the java version doesn't work.
> so it looks like it is a bug
> of the java library supplied by the thrift-0.2.0 version.
>
> Jimmy.
> --------------------------------------------------
> From: "Alexey Kovyrin" <al...@kovyrin.net>
> Sent: Friday, September 03, 2010 12:31 AM
> To: <us...@hbase.apache.org>
> Subject: Re: thrift for hbase in CDH3 broken ?
>
>> yes, Centos 5.5 + CDH3b2
>>
>> On Fri, Sep 3, 2010 at 3:26 AM, Jinsong Hu <ji...@hotmail.com>
>> wrote:
>>> are you using CDH3 distribution ?
>>>
>>> Jinsong
>>>
>>>
>>> --------------------------------------------------
>>> From: "Alexey Kovyrin" <al...@kovyrin.net>
>>> Sent: Friday, September 03, 2010 12:04 AM
>>> To: <us...@hbase.apache.org>
>>> Subject: Re: thrift for hbase in CDH3 broken ?
>>>
>>>> http://github.com/kovyrin/hbase-thrift-client-examples - just wrote
>>>> this example and tested it in our cluster, works as expected.
>>>> For this to work you'd need to install rubygems and thrift gem (gem
>>>> install thrift).
>>>>
>>>> On Fri, Sep 3, 2010 at 12:01 AM, Jinsong Hu <ji...@hotmail.com>
>>>> wrote:
>>>>>
>>>>> Can you send me some ruby test code and so I can try against the
>>>>> latest
>>>>> CDH3
>>>>> ?
>>>>>
>>>>> Jimmy.
>>>>>
>>>>> --------------------------------------------------
>>>>> From: "Alexey Kovyrin" <al...@kovyrin.net>
>>>>> Sent: Thursday, September 02, 2010 8:15 PM
>>>>> To: <us...@hbase.apache.org>
>>>>> Subject: Re: thrift for hbase in CDH3 broken ?
>>>>>
>>>>>> We use it in Scribd.com. All clients are ruby web apps.
>>>>>>
>>>>>> On Thu, Sep 2, 2010 at 10:49 PM, Todd Lipcon <to...@cloudera.com>
>>>>>> wrote:
>>>>>>>
>>>>>>> On Thu, Sep 2, 2010 at 5:35 PM, Jinsong Hu <ji...@hotmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Yes, I confirmed that it is indeed thrift server.
>>>>>>>>
>>>>>>>> and the fact that the API
>>>>>>>>
>>>>>>>> List<byte[]> tableNamesList=client.getTableNames();
>>>>>>>>>>
>>>>>>>>>> for (byte [] name : tableNamesList)
>>>>>>>>>> {
>>>>>>>>>> System.out.println(new String(name));
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>
>>>>>>>> successfully printed all table names shows that it is indeed thrift
>>>>>>>> server.
>>>>>>>>
>>>>>>>> if it is hue, it won't print the table names.
>>>>>>>>
>>>>>>>> Ah, sorry, I missed that in your original message. Not sure what's
>>>>>>>> up,
>>>>>>>> then
>>>>>>>
>>>>>>> - we don't have any changes in CDH that would affect this. Anyone
>>>>>>> here
>>>>>>> used
>>>>>>> thrift on 0.89.20100621?
>>>>>>>
>>>>>>> -Todd
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> Jimmy.
>>>>>>>>
>>>>>>>> --------------------------------------------------
>>>>>>>> From: "Todd Lipcon" <to...@cloudera.com>
>>>>>>>> Sent: Thursday, September 02, 2010 5:18 PM
>>>>>>>>
>>>>>>>> To: <us...@hbase.apache.org>
>>>>>>>> Subject: Re: thrift for hbase in CDH3 broken ?
>>>>>>>>
>>>>>>>>
>>>>>>>> Hi Jinsong,
>>>>>>>>>
>>>>>>>>> Are you sure that the port you're connecting to is indeed the
>>>>>>>>> thrift
>>>>>>>>> server?
>>>>>>>>>
>>>>>>>>> Unfortunately both the HBase thrift server and the Hue namenode
>>>>>>>>> plugin
>>>>>>>>> listen on port 9090, so you might be having an issue where your
>>>>>>>>> HBase
>>>>>>>>> client
>>>>>>>>> is trying to connect to the Namenode server instead of HBase.
>>>>>>>>>
>>>>>>>>> You can verify the ports using a command like "/sbin/fuser -n tcp
>>>>>>>>> 9090"
>>>>>>>>> to
>>>>>>>>> see which pid has it open, then cross reference against sudo jps.
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>> -Todd
>>>>>>>>>
>>>>>>>>> On Thu, Sep 2, 2010 at 4:40 PM, Jinsong Hu
>>>>>>>>> <ji...@hotmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Hi, There,
>>>>>>>>>>
>>>>>>>>>> I am trying to test and see if thrift for hbase works. I
>>>>>>>>>> followed
>>>>>>>>>> the
>>>>>>>>>> example from
>>>>>>>>>>
>>>>>>>>>> http://www.workhabit.com/labs/centos-55-and-thriftscribe
>>>>>>>>>> http://incubator.apache.org/thrift/
>>>>>>>>>> http://wiki.apache.org/hadoop/Hbase/ThriftApi
>>>>>>>>>>
>>>>>>>>>> and wrote test code: I found that client.getTableNames();
>>>>>>>>>> returns all table names successfully, but the scanner never
>>>>>>>>>> returned
>>>>>>>>>> any record. and it even throws exception:
>>>>>>>>>>
>>>>>>>>>> org.apache.thrift.TApplicationException: get failed: unknown
>>>>>>>>>> result
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.apache.hadoop.hbase.thrift.generated.Hbase$Client.recv_get(Hbase.java:785)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.apache.hadoop.hbase.thrift.generated.Hbase$Client.get(Hbase.java:750)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.apache.hadoop.hbase.thrift.HbaseThriftTest.testThriftAPI(HbaseThriftTest.java:73)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.apache.hadoop.hbase.thrift.HbaseThriftTest.main(HbaseThriftTest.java:128)
>>>>>>>>>>
>>>>>>>>>> I checked the syntax and the table and confirmed that the table
>>>>>>>>>> does
>>>>>>>>>> have
>>>>>>>>>> record and it should
>>>>>>>>>> return result, but it doesn't. Can anybody tell me what is wrong
>>>>>>>>>> ?
>>>>>>>>>>
>>>>>>>>>> At this time, I am suspecting the thrift server shipped with CDH3
>>>>>>>>>> may
>>>>>>>>>> have
>>>>>>>>>> problem. if there anyway
>>>>>>>>>> I can get this test code working ?
>>>>>>>>>>
>>>>>>>>>> I have to use thrift-0.2.0-incubating.tar.gz , as I found that
>>>>>>>>>> the
>>>>>>>>>> code
>>>>>>>>>> generated from hbase-thrift project
>>>>>>>>>> doesn't even compile with 0.3.0 and 0.4.0 thrift.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Jimmy.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> public void testThriftAPI()
>>>>>>>>>> {
>>>>>>>>>> // Make socket
>>>>>>>>>> TSocket transport = new TSocket("10.20.12.13", 9090);
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> // Wrap in a protocol
>>>>>>>>>> TBinaryProtocol protocol = new TBinaryProtocol(transport);
>>>>>>>>>>
>>>>>>>>>> Hbase.Client client = new Hbase.Client(protocol);
>>>>>>>>>>
>>>>>>>>>> try {
>>>>>>>>>> transport.open();
>>>>>>>>>> List<byte[]> tableNamesList=client.getTableNames();
>>>>>>>>>> for (byte [] name : tableNamesList)
>>>>>>>>>> {
>>>>>>>>>> System.out.println(new String(name));
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> String tableName="HEARTBEAT_CLUSTER";
>>>>>>>>>> String startRow="";
>>>>>>>>>> List<byte[]> columns= new ArrayList<byte[]>();
>>>>>>>>>> columns.add("fields:time_format".getBytes());
>>>>>>>>>> columns.add("fields:customer_id".getBytes());
>>>>>>>>>>
>>>>>>>>>> int aScannerId=0;
>>>>>>>>>> try {
>>>>>>>>>>
>>>>>>>>>> TCell cell=client.get(tableName.getBytes(),
>>>>>>>>>> "2010-08\tproduction-2".getBytes(),
>>>>>>>>>> "fields:customer_id".getBytes());
>>>>>>>>>> if (cell != null)
>>>>>>>>>> {
>>>>>>>>>> String value= new String( cell.value);
>>>>>>>>>> System.out.println(value);
>>>>>>>>>> }
>>>>>>>>>> aScannerId=client.scannerOpen(tableName.getBytes(),
>>>>>>>>>> startRow.getBytes(), columns);
>>>>>>>>>> TRowResult result=null;
>>>>>>>>>> for (int i=0; i< 10; i++)
>>>>>>>>>> {
>>>>>>>>>> try {
>>>>>>>>>> result=client.scannerGet(aScannerId);
>>>>>>>>>> } catch (Exception ex1)
>>>>>>>>>> {
>>>>>>>>>>
>>>>>>>>>> }
>>>>>>>>>> if (result ==null) break;
>>>>>>>>>>
>>>>>>>>>> String
>>>>>>>>>> timeFormat=result.getFieldValue(0).toString();
>>>>>>>>>> String
>>>>>>>>>> customerId=result.getFieldValue(1).toString();
>>>>>>>>>> System.out.println(timeFormat + "\t" +
>>>>>>>>>> customerId);
>>>>>>>>>> }
>>>>>>>>>> } catch (Exception ex)
>>>>>>>>>> {
>>>>>>>>>> ex.printStackTrace();
>>>>>>>>>> }
>>>>>>>>>> finally {
>>>>>>>>>> if (aScannerId >0 ) client.scannerClose(aScannerId);
>>>>>>>>>> }
>>>>>>>>>> } catch (Exception exp)
>>>>>>>>>> {
>>>>>>>>>> exp.printStackTrace();
>>>>>>>>>> }
>>>>>>>>>> finally{
>>>>>>>>>> transport.close();
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Todd Lipcon
>>>>>>>>> Software Engineer, Cloudera
>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Todd Lipcon
>>>>>>> Software Engineer, Cloudera
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Alexey Kovyrin
>>>>>> http://kovyrin.net/
>>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Alexey Kovyrin
>>>> http://kovyrin.net/
>>>>
>>>
>>
>>
>>
>> --
>> Alexey Kovyrin
>> http://kovyrin.net/
>>
>
Re: thrift for hbase in CDH3 broken ?
Posted by Jinsong Hu <ji...@hotmail.com>.
I tried your code and indeed it works. but the java version doesn't work. so
it looks like it is a bug
of the java library supplied by the thrift-0.2.0 version.
Jimmy.
--------------------------------------------------
From: "Alexey Kovyrin" <al...@kovyrin.net>
Sent: Friday, September 03, 2010 12:31 AM
To: <us...@hbase.apache.org>
Subject: Re: thrift for hbase in CDH3 broken ?
> yes, Centos 5.5 + CDH3b2
>
> On Fri, Sep 3, 2010 at 3:26 AM, Jinsong Hu <ji...@hotmail.com> wrote:
>> are you using CDH3 distribution ?
>>
>> Jinsong
>>
>>
>> --------------------------------------------------
>> From: "Alexey Kovyrin" <al...@kovyrin.net>
>> Sent: Friday, September 03, 2010 12:04 AM
>> To: <us...@hbase.apache.org>
>> Subject: Re: thrift for hbase in CDH3 broken ?
>>
>>> http://github.com/kovyrin/hbase-thrift-client-examples - just wrote
>>> this example and tested it in our cluster, works as expected.
>>> For this to work you'd need to install rubygems and thrift gem (gem
>>> install thrift).
>>>
>>> On Fri, Sep 3, 2010 at 12:01 AM, Jinsong Hu <ji...@hotmail.com>
>>> wrote:
>>>>
>>>> Can you send me some ruby test code and so I can try against the latest
>>>> CDH3
>>>> ?
>>>>
>>>> Jimmy.
>>>>
>>>> --------------------------------------------------
>>>> From: "Alexey Kovyrin" <al...@kovyrin.net>
>>>> Sent: Thursday, September 02, 2010 8:15 PM
>>>> To: <us...@hbase.apache.org>
>>>> Subject: Re: thrift for hbase in CDH3 broken ?
>>>>
>>>>> We use it in Scribd.com. All clients are ruby web apps.
>>>>>
>>>>> On Thu, Sep 2, 2010 at 10:49 PM, Todd Lipcon <to...@cloudera.com>
>>>>> wrote:
>>>>>>
>>>>>> On Thu, Sep 2, 2010 at 5:35 PM, Jinsong Hu <ji...@hotmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Yes, I confirmed that it is indeed thrift server.
>>>>>>>
>>>>>>> and the fact that the API
>>>>>>>
>>>>>>> List<byte[]> tableNamesList=client.getTableNames();
>>>>>>>>>
>>>>>>>>> for (byte [] name : tableNamesList)
>>>>>>>>> {
>>>>>>>>> System.out.println(new String(name));
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>
>>>>>>> successfully printed all table names shows that it is indeed thrift
>>>>>>> server.
>>>>>>>
>>>>>>> if it is hue, it won't print the table names.
>>>>>>>
>>>>>>> Ah, sorry, I missed that in your original message. Not sure what's
>>>>>>> up,
>>>>>>> then
>>>>>>
>>>>>> - we don't have any changes in CDH that would affect this. Anyone
>>>>>> here
>>>>>> used
>>>>>> thrift on 0.89.20100621?
>>>>>>
>>>>>> -Todd
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Jimmy.
>>>>>>>
>>>>>>> --------------------------------------------------
>>>>>>> From: "Todd Lipcon" <to...@cloudera.com>
>>>>>>> Sent: Thursday, September 02, 2010 5:18 PM
>>>>>>>
>>>>>>> To: <us...@hbase.apache.org>
>>>>>>> Subject: Re: thrift for hbase in CDH3 broken ?
>>>>>>>
>>>>>>>
>>>>>>> Hi Jinsong,
>>>>>>>>
>>>>>>>> Are you sure that the port you're connecting to is indeed the
>>>>>>>> thrift
>>>>>>>> server?
>>>>>>>>
>>>>>>>> Unfortunately both the HBase thrift server and the Hue namenode
>>>>>>>> plugin
>>>>>>>> listen on port 9090, so you might be having an issue where your
>>>>>>>> HBase
>>>>>>>> client
>>>>>>>> is trying to connect to the Namenode server instead of HBase.
>>>>>>>>
>>>>>>>> You can verify the ports using a command like "/sbin/fuser -n tcp
>>>>>>>> 9090"
>>>>>>>> to
>>>>>>>> see which pid has it open, then cross reference against sudo jps.
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>> -Todd
>>>>>>>>
>>>>>>>> On Thu, Sep 2, 2010 at 4:40 PM, Jinsong Hu <ji...@hotmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Hi, There,
>>>>>>>>>
>>>>>>>>> I am trying to test and see if thrift for hbase works. I followed
>>>>>>>>> the
>>>>>>>>> example from
>>>>>>>>>
>>>>>>>>> http://www.workhabit.com/labs/centos-55-and-thriftscribe
>>>>>>>>> http://incubator.apache.org/thrift/
>>>>>>>>> http://wiki.apache.org/hadoop/Hbase/ThriftApi
>>>>>>>>>
>>>>>>>>> and wrote test code: I found that client.getTableNames();
>>>>>>>>> returns all table names successfully, but the scanner never
>>>>>>>>> returned
>>>>>>>>> any record. and it even throws exception:
>>>>>>>>>
>>>>>>>>> org.apache.thrift.TApplicationException: get failed: unknown
>>>>>>>>> result
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.apache.hadoop.hbase.thrift.generated.Hbase$Client.recv_get(Hbase.java:785)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.apache.hadoop.hbase.thrift.generated.Hbase$Client.get(Hbase.java:750)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.apache.hadoop.hbase.thrift.HbaseThriftTest.testThriftAPI(HbaseThriftTest.java:73)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.apache.hadoop.hbase.thrift.HbaseThriftTest.main(HbaseThriftTest.java:128)
>>>>>>>>>
>>>>>>>>> I checked the syntax and the table and confirmed that the table
>>>>>>>>> does
>>>>>>>>> have
>>>>>>>>> record and it should
>>>>>>>>> return result, but it doesn't. Can anybody tell me what is wrong ?
>>>>>>>>>
>>>>>>>>> At this time, I am suspecting the thrift server shipped with CDH3
>>>>>>>>> may
>>>>>>>>> have
>>>>>>>>> problem. if there anyway
>>>>>>>>> I can get this test code working ?
>>>>>>>>>
>>>>>>>>> I have to use thrift-0.2.0-incubating.tar.gz , as I found that the
>>>>>>>>> code
>>>>>>>>> generated from hbase-thrift project
>>>>>>>>> doesn't even compile with 0.3.0 and 0.4.0 thrift.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Jimmy.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> public void testThriftAPI()
>>>>>>>>> {
>>>>>>>>> // Make socket
>>>>>>>>> TSocket transport = new TSocket("10.20.12.13", 9090);
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> // Wrap in a protocol
>>>>>>>>> TBinaryProtocol protocol = new TBinaryProtocol(transport);
>>>>>>>>>
>>>>>>>>> Hbase.Client client = new Hbase.Client(protocol);
>>>>>>>>>
>>>>>>>>> try {
>>>>>>>>> transport.open();
>>>>>>>>> List<byte[]> tableNamesList=client.getTableNames();
>>>>>>>>> for (byte [] name : tableNamesList)
>>>>>>>>> {
>>>>>>>>> System.out.println(new String(name));
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> String tableName="HEARTBEAT_CLUSTER";
>>>>>>>>> String startRow="";
>>>>>>>>> List<byte[]> columns= new ArrayList<byte[]>();
>>>>>>>>> columns.add("fields:time_format".getBytes());
>>>>>>>>> columns.add("fields:customer_id".getBytes());
>>>>>>>>>
>>>>>>>>> int aScannerId=0;
>>>>>>>>> try {
>>>>>>>>>
>>>>>>>>> TCell cell=client.get(tableName.getBytes(),
>>>>>>>>> "2010-08\tproduction-2".getBytes(),
>>>>>>>>> "fields:customer_id".getBytes());
>>>>>>>>> if (cell != null)
>>>>>>>>> {
>>>>>>>>> String value= new String( cell.value);
>>>>>>>>> System.out.println(value);
>>>>>>>>> }
>>>>>>>>> aScannerId=client.scannerOpen(tableName.getBytes(),
>>>>>>>>> startRow.getBytes(), columns);
>>>>>>>>> TRowResult result=null;
>>>>>>>>> for (int i=0; i< 10; i++)
>>>>>>>>> {
>>>>>>>>> try {
>>>>>>>>> result=client.scannerGet(aScannerId);
>>>>>>>>> } catch (Exception ex1)
>>>>>>>>> {
>>>>>>>>>
>>>>>>>>> }
>>>>>>>>> if (result ==null) break;
>>>>>>>>>
>>>>>>>>> String
>>>>>>>>> timeFormat=result.getFieldValue(0).toString();
>>>>>>>>> String
>>>>>>>>> customerId=result.getFieldValue(1).toString();
>>>>>>>>> System.out.println(timeFormat + "\t" + customerId);
>>>>>>>>> }
>>>>>>>>> } catch (Exception ex)
>>>>>>>>> {
>>>>>>>>> ex.printStackTrace();
>>>>>>>>> }
>>>>>>>>> finally {
>>>>>>>>> if (aScannerId >0 ) client.scannerClose(aScannerId);
>>>>>>>>> }
>>>>>>>>> } catch (Exception exp)
>>>>>>>>> {
>>>>>>>>> exp.printStackTrace();
>>>>>>>>> }
>>>>>>>>> finally{
>>>>>>>>> transport.close();
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Todd Lipcon
>>>>>>>> Software Engineer, Cloudera
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Todd Lipcon
>>>>>> Software Engineer, Cloudera
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Alexey Kovyrin
>>>>> http://kovyrin.net/
>>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Alexey Kovyrin
>>> http://kovyrin.net/
>>>
>>
>
>
>
> --
> Alexey Kovyrin
> http://kovyrin.net/
>
Re: thrift for hbase in CDH3 broken ?
Posted by Alexey Kovyrin <al...@kovyrin.net>.
yes, Centos 5.5 + CDH3b2
On Fri, Sep 3, 2010 at 3:26 AM, Jinsong Hu <ji...@hotmail.com> wrote:
> are you using CDH3 distribution ?
>
> Jinsong
>
>
> --------------------------------------------------
> From: "Alexey Kovyrin" <al...@kovyrin.net>
> Sent: Friday, September 03, 2010 12:04 AM
> To: <us...@hbase.apache.org>
> Subject: Re: thrift for hbase in CDH3 broken ?
>
>> http://github.com/kovyrin/hbase-thrift-client-examples - just wrote
>> this example and tested it in our cluster, works as expected.
>> For this to work you'd need to install rubygems and thrift gem (gem
>> install thrift).
>>
>> On Fri, Sep 3, 2010 at 12:01 AM, Jinsong Hu <ji...@hotmail.com>
>> wrote:
>>>
>>> Can you send me some ruby test code and so I can try against the latest
>>> CDH3
>>> ?
>>>
>>> Jimmy.
>>>
>>> --------------------------------------------------
>>> From: "Alexey Kovyrin" <al...@kovyrin.net>
>>> Sent: Thursday, September 02, 2010 8:15 PM
>>> To: <us...@hbase.apache.org>
>>> Subject: Re: thrift for hbase in CDH3 broken ?
>>>
>>>> We use it in Scribd.com. All clients are ruby web apps.
>>>>
>>>> On Thu, Sep 2, 2010 at 10:49 PM, Todd Lipcon <to...@cloudera.com> wrote:
>>>>>
>>>>> On Thu, Sep 2, 2010 at 5:35 PM, Jinsong Hu <ji...@hotmail.com>
>>>>> wrote:
>>>>>
>>>>>> Yes, I confirmed that it is indeed thrift server.
>>>>>>
>>>>>> and the fact that the API
>>>>>>
>>>>>> List<byte[]> tableNamesList=client.getTableNames();
>>>>>>>>
>>>>>>>> for (byte [] name : tableNamesList)
>>>>>>>> {
>>>>>>>> System.out.println(new String(name));
>>>>>>>> }
>>>>>>>>
>>>>>>>
>>>>>> successfully printed all table names shows that it is indeed thrift
>>>>>> server.
>>>>>>
>>>>>> if it is hue, it won't print the table names.
>>>>>>
>>>>>> Ah, sorry, I missed that in your original message. Not sure what's up,
>>>>>> then
>>>>>
>>>>> - we don't have any changes in CDH that would affect this. Anyone here
>>>>> used
>>>>> thrift on 0.89.20100621?
>>>>>
>>>>> -Todd
>>>>>
>>>>>
>>>>>
>>>>>> Jimmy.
>>>>>>
>>>>>> --------------------------------------------------
>>>>>> From: "Todd Lipcon" <to...@cloudera.com>
>>>>>> Sent: Thursday, September 02, 2010 5:18 PM
>>>>>>
>>>>>> To: <us...@hbase.apache.org>
>>>>>> Subject: Re: thrift for hbase in CDH3 broken ?
>>>>>>
>>>>>>
>>>>>> Hi Jinsong,
>>>>>>>
>>>>>>> Are you sure that the port you're connecting to is indeed the thrift
>>>>>>> server?
>>>>>>>
>>>>>>> Unfortunately both the HBase thrift server and the Hue namenode
>>>>>>> plugin
>>>>>>> listen on port 9090, so you might be having an issue where your HBase
>>>>>>> client
>>>>>>> is trying to connect to the Namenode server instead of HBase.
>>>>>>>
>>>>>>> You can verify the ports using a command like "/sbin/fuser -n tcp
>>>>>>> 9090"
>>>>>>> to
>>>>>>> see which pid has it open, then cross reference against sudo jps.
>>>>>>>
>>>>>>> Thanks
>>>>>>> -Todd
>>>>>>>
>>>>>>> On Thu, Sep 2, 2010 at 4:40 PM, Jinsong Hu <ji...@hotmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> Hi, There,
>>>>>>>>
>>>>>>>> I am trying to test and see if thrift for hbase works. I followed
>>>>>>>> the
>>>>>>>> example from
>>>>>>>>
>>>>>>>> http://www.workhabit.com/labs/centos-55-and-thriftscribe
>>>>>>>> http://incubator.apache.org/thrift/
>>>>>>>> http://wiki.apache.org/hadoop/Hbase/ThriftApi
>>>>>>>>
>>>>>>>> and wrote test code: I found that client.getTableNames();
>>>>>>>> returns all table names successfully, but the scanner never returned
>>>>>>>> any record. and it even throws exception:
>>>>>>>>
>>>>>>>> org.apache.thrift.TApplicationException: get failed: unknown result
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.hadoop.hbase.thrift.generated.Hbase$Client.recv_get(Hbase.java:785)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.hadoop.hbase.thrift.generated.Hbase$Client.get(Hbase.java:750)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.hadoop.hbase.thrift.HbaseThriftTest.testThriftAPI(HbaseThriftTest.java:73)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.hadoop.hbase.thrift.HbaseThriftTest.main(HbaseThriftTest.java:128)
>>>>>>>>
>>>>>>>> I checked the syntax and the table and confirmed that the table does
>>>>>>>> have
>>>>>>>> record and it should
>>>>>>>> return result, but it doesn't. Can anybody tell me what is wrong ?
>>>>>>>>
>>>>>>>> At this time, I am suspecting the thrift server shipped with CDH3
>>>>>>>> may
>>>>>>>> have
>>>>>>>> problem. if there anyway
>>>>>>>> I can get this test code working ?
>>>>>>>>
>>>>>>>> I have to use thrift-0.2.0-incubating.tar.gz , as I found that the
>>>>>>>> code
>>>>>>>> generated from hbase-thrift project
>>>>>>>> doesn't even compile with 0.3.0 and 0.4.0 thrift.
>>>>>>>>
>>>>>>>>
>>>>>>>> Jimmy.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> public void testThriftAPI()
>>>>>>>> {
>>>>>>>> // Make socket
>>>>>>>> TSocket transport = new TSocket("10.20.12.13", 9090);
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> // Wrap in a protocol
>>>>>>>> TBinaryProtocol protocol = new TBinaryProtocol(transport);
>>>>>>>>
>>>>>>>> Hbase.Client client = new Hbase.Client(protocol);
>>>>>>>>
>>>>>>>> try {
>>>>>>>> transport.open();
>>>>>>>> List<byte[]> tableNamesList=client.getTableNames();
>>>>>>>> for (byte [] name : tableNamesList)
>>>>>>>> {
>>>>>>>> System.out.println(new String(name));
>>>>>>>> }
>>>>>>>>
>>>>>>>> String tableName="HEARTBEAT_CLUSTER";
>>>>>>>> String startRow="";
>>>>>>>> List<byte[]> columns= new ArrayList<byte[]>();
>>>>>>>> columns.add("fields:time_format".getBytes());
>>>>>>>> columns.add("fields:customer_id".getBytes());
>>>>>>>>
>>>>>>>> int aScannerId=0;
>>>>>>>> try {
>>>>>>>>
>>>>>>>> TCell cell=client.get(tableName.getBytes(),
>>>>>>>> "2010-08\tproduction-2".getBytes(),
>>>>>>>> "fields:customer_id".getBytes());
>>>>>>>> if (cell != null)
>>>>>>>> {
>>>>>>>> String value= new String( cell.value);
>>>>>>>> System.out.println(value);
>>>>>>>> }
>>>>>>>> aScannerId=client.scannerOpen(tableName.getBytes(),
>>>>>>>> startRow.getBytes(), columns);
>>>>>>>> TRowResult result=null;
>>>>>>>> for (int i=0; i< 10; i++)
>>>>>>>> {
>>>>>>>> try {
>>>>>>>> result=client.scannerGet(aScannerId);
>>>>>>>> } catch (Exception ex1)
>>>>>>>> {
>>>>>>>>
>>>>>>>> }
>>>>>>>> if (result ==null) break;
>>>>>>>>
>>>>>>>> String timeFormat=result.getFieldValue(0).toString();
>>>>>>>> String customerId=result.getFieldValue(1).toString();
>>>>>>>> System.out.println(timeFormat + "\t" + customerId);
>>>>>>>> }
>>>>>>>> } catch (Exception ex)
>>>>>>>> {
>>>>>>>> ex.printStackTrace();
>>>>>>>> }
>>>>>>>> finally {
>>>>>>>> if (aScannerId >0 ) client.scannerClose(aScannerId);
>>>>>>>> }
>>>>>>>> } catch (Exception exp)
>>>>>>>> {
>>>>>>>> exp.printStackTrace();
>>>>>>>> }
>>>>>>>> finally{
>>>>>>>> transport.close();
>>>>>>>> }
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Todd Lipcon
>>>>>>> Software Engineer, Cloudera
>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Todd Lipcon
>>>>> Software Engineer, Cloudera
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Alexey Kovyrin
>>>> http://kovyrin.net/
>>>>
>>>
>>
>>
>>
>> --
>> Alexey Kovyrin
>> http://kovyrin.net/
>>
>
--
Alexey Kovyrin
http://kovyrin.net/
Re: thrift for hbase in CDH3 broken ?
Posted by Jinsong Hu <ji...@hotmail.com>.
are you using CDH3 distribution ?
Jinsong
--------------------------------------------------
From: "Alexey Kovyrin" <al...@kovyrin.net>
Sent: Friday, September 03, 2010 12:04 AM
To: <us...@hbase.apache.org>
Subject: Re: thrift for hbase in CDH3 broken ?
> http://github.com/kovyrin/hbase-thrift-client-examples - just wrote
> this example and tested it in our cluster, works as expected.
> For this to work you'd need to install rubygems and thrift gem (gem
> install thrift).
>
> On Fri, Sep 3, 2010 at 12:01 AM, Jinsong Hu <ji...@hotmail.com>
> wrote:
>> Can you send me some ruby test code and so I can try against the latest
>> CDH3
>> ?
>>
>> Jimmy.
>>
>> --------------------------------------------------
>> From: "Alexey Kovyrin" <al...@kovyrin.net>
>> Sent: Thursday, September 02, 2010 8:15 PM
>> To: <us...@hbase.apache.org>
>> Subject: Re: thrift for hbase in CDH3 broken ?
>>
>>> We use it in Scribd.com. All clients are ruby web apps.
>>>
>>> On Thu, Sep 2, 2010 at 10:49 PM, Todd Lipcon <to...@cloudera.com> wrote:
>>>>
>>>> On Thu, Sep 2, 2010 at 5:35 PM, Jinsong Hu <ji...@hotmail.com>
>>>> wrote:
>>>>
>>>>> Yes, I confirmed that it is indeed thrift server.
>>>>>
>>>>> and the fact that the API
>>>>>
>>>>> List<byte[]> tableNamesList=client.getTableNames();
>>>>>>>
>>>>>>> for (byte [] name : tableNamesList)
>>>>>>> {
>>>>>>> System.out.println(new String(name));
>>>>>>> }
>>>>>>>
>>>>>>
>>>>> successfully printed all table names shows that it is indeed thrift
>>>>> server.
>>>>>
>>>>> if it is hue, it won't print the table names.
>>>>>
>>>>> Ah, sorry, I missed that in your original message. Not sure what's up,
>>>>> then
>>>>
>>>> - we don't have any changes in CDH that would affect this. Anyone here
>>>> used
>>>> thrift on 0.89.20100621?
>>>>
>>>> -Todd
>>>>
>>>>
>>>>
>>>>> Jimmy.
>>>>>
>>>>> --------------------------------------------------
>>>>> From: "Todd Lipcon" <to...@cloudera.com>
>>>>> Sent: Thursday, September 02, 2010 5:18 PM
>>>>>
>>>>> To: <us...@hbase.apache.org>
>>>>> Subject: Re: thrift for hbase in CDH3 broken ?
>>>>>
>>>>>
>>>>> Hi Jinsong,
>>>>>>
>>>>>> Are you sure that the port you're connecting to is indeed the thrift
>>>>>> server?
>>>>>>
>>>>>> Unfortunately both the HBase thrift server and the Hue namenode
>>>>>> plugin
>>>>>> listen on port 9090, so you might be having an issue where your HBase
>>>>>> client
>>>>>> is trying to connect to the Namenode server instead of HBase.
>>>>>>
>>>>>> You can verify the ports using a command like "/sbin/fuser -n tcp
>>>>>> 9090"
>>>>>> to
>>>>>> see which pid has it open, then cross reference against sudo jps.
>>>>>>
>>>>>> Thanks
>>>>>> -Todd
>>>>>>
>>>>>> On Thu, Sep 2, 2010 at 4:40 PM, Jinsong Hu <ji...@hotmail.com>
>>>>>> wrote:
>>>>>>
>>>>>> Hi, There,
>>>>>>>
>>>>>>> I am trying to test and see if thrift for hbase works. I followed
>>>>>>> the
>>>>>>> example from
>>>>>>>
>>>>>>> http://www.workhabit.com/labs/centos-55-and-thriftscribe
>>>>>>> http://incubator.apache.org/thrift/
>>>>>>> http://wiki.apache.org/hadoop/Hbase/ThriftApi
>>>>>>>
>>>>>>> and wrote test code: I found that client.getTableNames();
>>>>>>> returns all table names successfully, but the scanner never returned
>>>>>>> any record. and it even throws exception:
>>>>>>>
>>>>>>> org.apache.thrift.TApplicationException: get failed: unknown result
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.apache.hadoop.hbase.thrift.generated.Hbase$Client.recv_get(Hbase.java:785)
>>>>>>> at
>>>>>>>
>>>>>>> org.apache.hadoop.hbase.thrift.generated.Hbase$Client.get(Hbase.java:750)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.apache.hadoop.hbase.thrift.HbaseThriftTest.testThriftAPI(HbaseThriftTest.java:73)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.apache.hadoop.hbase.thrift.HbaseThriftTest.main(HbaseThriftTest.java:128)
>>>>>>>
>>>>>>> I checked the syntax and the table and confirmed that the table does
>>>>>>> have
>>>>>>> record and it should
>>>>>>> return result, but it doesn't. Can anybody tell me what is wrong ?
>>>>>>>
>>>>>>> At this time, I am suspecting the thrift server shipped with CDH3
>>>>>>> may
>>>>>>> have
>>>>>>> problem. if there anyway
>>>>>>> I can get this test code working ?
>>>>>>>
>>>>>>> I have to use thrift-0.2.0-incubating.tar.gz , as I found that the
>>>>>>> code
>>>>>>> generated from hbase-thrift project
>>>>>>> doesn't even compile with 0.3.0 and 0.4.0 thrift.
>>>>>>>
>>>>>>>
>>>>>>> Jimmy.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> public void testThriftAPI()
>>>>>>> {
>>>>>>> // Make socket
>>>>>>> TSocket transport = new TSocket("10.20.12.13", 9090);
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> // Wrap in a protocol
>>>>>>> TBinaryProtocol protocol = new TBinaryProtocol(transport);
>>>>>>>
>>>>>>> Hbase.Client client = new Hbase.Client(protocol);
>>>>>>>
>>>>>>> try {
>>>>>>> transport.open();
>>>>>>> List<byte[]> tableNamesList=client.getTableNames();
>>>>>>> for (byte [] name : tableNamesList)
>>>>>>> {
>>>>>>> System.out.println(new String(name));
>>>>>>> }
>>>>>>>
>>>>>>> String tableName="HEARTBEAT_CLUSTER";
>>>>>>> String startRow="";
>>>>>>> List<byte[]> columns= new ArrayList<byte[]>();
>>>>>>> columns.add("fields:time_format".getBytes());
>>>>>>> columns.add("fields:customer_id".getBytes());
>>>>>>>
>>>>>>> int aScannerId=0;
>>>>>>> try {
>>>>>>>
>>>>>>> TCell cell=client.get(tableName.getBytes(),
>>>>>>> "2010-08\tproduction-2".getBytes(),
>>>>>>> "fields:customer_id".getBytes());
>>>>>>> if (cell != null)
>>>>>>> {
>>>>>>> String value= new String( cell.value);
>>>>>>> System.out.println(value);
>>>>>>> }
>>>>>>> aScannerId=client.scannerOpen(tableName.getBytes(),
>>>>>>> startRow.getBytes(), columns);
>>>>>>> TRowResult result=null;
>>>>>>> for (int i=0; i< 10; i++)
>>>>>>> {
>>>>>>> try {
>>>>>>> result=client.scannerGet(aScannerId);
>>>>>>> } catch (Exception ex1)
>>>>>>> {
>>>>>>>
>>>>>>> }
>>>>>>> if (result ==null) break;
>>>>>>>
>>>>>>> String
>>>>>>> timeFormat=result.getFieldValue(0).toString();
>>>>>>> String
>>>>>>> customerId=result.getFieldValue(1).toString();
>>>>>>> System.out.println(timeFormat + "\t" + customerId);
>>>>>>> }
>>>>>>> } catch (Exception ex)
>>>>>>> {
>>>>>>> ex.printStackTrace();
>>>>>>> }
>>>>>>> finally {
>>>>>>> if (aScannerId >0 ) client.scannerClose(aScannerId);
>>>>>>> }
>>>>>>> } catch (Exception exp)
>>>>>>> {
>>>>>>> exp.printStackTrace();
>>>>>>> }
>>>>>>> finally{
>>>>>>> transport.close();
>>>>>>> }
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Todd Lipcon
>>>>>> Software Engineer, Cloudera
>>>>>>
>>>>>>
>>>>
>>>>
>>>> --
>>>> Todd Lipcon
>>>> Software Engineer, Cloudera
>>>>
>>>
>>>
>>>
>>> --
>>> Alexey Kovyrin
>>> http://kovyrin.net/
>>>
>>
>
>
>
> --
> Alexey Kovyrin
> http://kovyrin.net/
>
Re: thrift for hbase in CDH3 broken ?
Posted by Alexey Kovyrin <al...@kovyrin.net>.
http://github.com/kovyrin/hbase-thrift-client-examples - just wrote
this example and tested it in our cluster, works as expected.
For this to work you'd need to install rubygems and thrift gem (gem
install thrift).
On Fri, Sep 3, 2010 at 12:01 AM, Jinsong Hu <ji...@hotmail.com> wrote:
> Can you send me some ruby test code and so I can try against the latest CDH3
> ?
>
> Jimmy.
>
> --------------------------------------------------
> From: "Alexey Kovyrin" <al...@kovyrin.net>
> Sent: Thursday, September 02, 2010 8:15 PM
> To: <us...@hbase.apache.org>
> Subject: Re: thrift for hbase in CDH3 broken ?
>
>> We use it in Scribd.com. All clients are ruby web apps.
>>
>> On Thu, Sep 2, 2010 at 10:49 PM, Todd Lipcon <to...@cloudera.com> wrote:
>>>
>>> On Thu, Sep 2, 2010 at 5:35 PM, Jinsong Hu <ji...@hotmail.com>
>>> wrote:
>>>
>>>> Yes, I confirmed that it is indeed thrift server.
>>>>
>>>> and the fact that the API
>>>>
>>>> List<byte[]> tableNamesList=client.getTableNames();
>>>>>>
>>>>>> for (byte [] name : tableNamesList)
>>>>>> {
>>>>>> System.out.println(new String(name));
>>>>>> }
>>>>>>
>>>>>
>>>> successfully printed all table names shows that it is indeed thrift
>>>> server.
>>>>
>>>> if it is hue, it won't print the table names.
>>>>
>>>> Ah, sorry, I missed that in your original message. Not sure what's up,
>>>> then
>>>
>>> - we don't have any changes in CDH that would affect this. Anyone here
>>> used
>>> thrift on 0.89.20100621?
>>>
>>> -Todd
>>>
>>>
>>>
>>>> Jimmy.
>>>>
>>>> --------------------------------------------------
>>>> From: "Todd Lipcon" <to...@cloudera.com>
>>>> Sent: Thursday, September 02, 2010 5:18 PM
>>>>
>>>> To: <us...@hbase.apache.org>
>>>> Subject: Re: thrift for hbase in CDH3 broken ?
>>>>
>>>>
>>>> Hi Jinsong,
>>>>>
>>>>> Are you sure that the port you're connecting to is indeed the thrift
>>>>> server?
>>>>>
>>>>> Unfortunately both the HBase thrift server and the Hue namenode plugin
>>>>> listen on port 9090, so you might be having an issue where your HBase
>>>>> client
>>>>> is trying to connect to the Namenode server instead of HBase.
>>>>>
>>>>> You can verify the ports using a command like "/sbin/fuser -n tcp 9090"
>>>>> to
>>>>> see which pid has it open, then cross reference against sudo jps.
>>>>>
>>>>> Thanks
>>>>> -Todd
>>>>>
>>>>> On Thu, Sep 2, 2010 at 4:40 PM, Jinsong Hu <ji...@hotmail.com>
>>>>> wrote:
>>>>>
>>>>> Hi, There,
>>>>>>
>>>>>> I am trying to test and see if thrift for hbase works. I followed the
>>>>>> example from
>>>>>>
>>>>>> http://www.workhabit.com/labs/centos-55-and-thriftscribe
>>>>>> http://incubator.apache.org/thrift/
>>>>>> http://wiki.apache.org/hadoop/Hbase/ThriftApi
>>>>>>
>>>>>> and wrote test code: I found that client.getTableNames();
>>>>>> returns all table names successfully, but the scanner never returned
>>>>>> any record. and it even throws exception:
>>>>>>
>>>>>> org.apache.thrift.TApplicationException: get failed: unknown result
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.hbase.thrift.generated.Hbase$Client.recv_get(Hbase.java:785)
>>>>>> at
>>>>>>
>>>>>> org.apache.hadoop.hbase.thrift.generated.Hbase$Client.get(Hbase.java:750)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.hbase.thrift.HbaseThriftTest.testThriftAPI(HbaseThriftTest.java:73)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.hbase.thrift.HbaseThriftTest.main(HbaseThriftTest.java:128)
>>>>>>
>>>>>> I checked the syntax and the table and confirmed that the table does
>>>>>> have
>>>>>> record and it should
>>>>>> return result, but it doesn't. Can anybody tell me what is wrong ?
>>>>>>
>>>>>> At this time, I am suspecting the thrift server shipped with CDH3 may
>>>>>> have
>>>>>> problem. if there anyway
>>>>>> I can get this test code working ?
>>>>>>
>>>>>> I have to use thrift-0.2.0-incubating.tar.gz , as I found that the
>>>>>> code
>>>>>> generated from hbase-thrift project
>>>>>> doesn't even compile with 0.3.0 and 0.4.0 thrift.
>>>>>>
>>>>>>
>>>>>> Jimmy.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> public void testThriftAPI()
>>>>>> {
>>>>>> // Make socket
>>>>>> TSocket transport = new TSocket("10.20.12.13", 9090);
>>>>>>
>>>>>>
>>>>>>
>>>>>> // Wrap in a protocol
>>>>>> TBinaryProtocol protocol = new TBinaryProtocol(transport);
>>>>>>
>>>>>> Hbase.Client client = new Hbase.Client(protocol);
>>>>>>
>>>>>> try {
>>>>>> transport.open();
>>>>>> List<byte[]> tableNamesList=client.getTableNames();
>>>>>> for (byte [] name : tableNamesList)
>>>>>> {
>>>>>> System.out.println(new String(name));
>>>>>> }
>>>>>>
>>>>>> String tableName="HEARTBEAT_CLUSTER";
>>>>>> String startRow="";
>>>>>> List<byte[]> columns= new ArrayList<byte[]>();
>>>>>> columns.add("fields:time_format".getBytes());
>>>>>> columns.add("fields:customer_id".getBytes());
>>>>>>
>>>>>> int aScannerId=0;
>>>>>> try {
>>>>>>
>>>>>> TCell cell=client.get(tableName.getBytes(),
>>>>>> "2010-08\tproduction-2".getBytes(), "fields:customer_id".getBytes());
>>>>>> if (cell != null)
>>>>>> {
>>>>>> String value= new String( cell.value);
>>>>>> System.out.println(value);
>>>>>> }
>>>>>> aScannerId=client.scannerOpen(tableName.getBytes(),
>>>>>> startRow.getBytes(), columns);
>>>>>> TRowResult result=null;
>>>>>> for (int i=0; i< 10; i++)
>>>>>> {
>>>>>> try {
>>>>>> result=client.scannerGet(aScannerId);
>>>>>> } catch (Exception ex1)
>>>>>> {
>>>>>>
>>>>>> }
>>>>>> if (result ==null) break;
>>>>>>
>>>>>> String timeFormat=result.getFieldValue(0).toString();
>>>>>> String customerId=result.getFieldValue(1).toString();
>>>>>> System.out.println(timeFormat + "\t" + customerId);
>>>>>> }
>>>>>> } catch (Exception ex)
>>>>>> {
>>>>>> ex.printStackTrace();
>>>>>> }
>>>>>> finally {
>>>>>> if (aScannerId >0 ) client.scannerClose(aScannerId);
>>>>>> }
>>>>>> } catch (Exception exp)
>>>>>> {
>>>>>> exp.printStackTrace();
>>>>>> }
>>>>>> finally{
>>>>>> transport.close();
>>>>>> }
>>>>>>
>>>>>> }
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Todd Lipcon
>>>>> Software Engineer, Cloudera
>>>>>
>>>>>
>>>
>>>
>>> --
>>> Todd Lipcon
>>> Software Engineer, Cloudera
>>>
>>
>>
>>
>> --
>> Alexey Kovyrin
>> http://kovyrin.net/
>>
>
--
Alexey Kovyrin
http://kovyrin.net/
Re: thrift for hbase in CDH3 broken ?
Posted by Jinsong Hu <ji...@hotmail.com>.
Can you send me some ruby test code and so I can try against the latest CDH3
?
Jimmy.
--------------------------------------------------
From: "Alexey Kovyrin" <al...@kovyrin.net>
Sent: Thursday, September 02, 2010 8:15 PM
To: <us...@hbase.apache.org>
Subject: Re: thrift for hbase in CDH3 broken ?
> We use it in Scribd.com. All clients are ruby web apps.
>
> On Thu, Sep 2, 2010 at 10:49 PM, Todd Lipcon <to...@cloudera.com> wrote:
>> On Thu, Sep 2, 2010 at 5:35 PM, Jinsong Hu <ji...@hotmail.com>
>> wrote:
>>
>>> Yes, I confirmed that it is indeed thrift server.
>>>
>>> and the fact that the API
>>>
>>> List<byte[]> tableNamesList=client.getTableNames();
>>>>> for (byte [] name : tableNamesList)
>>>>> {
>>>>> System.out.println(new String(name));
>>>>> }
>>>>>
>>>>
>>> successfully printed all table names shows that it is indeed thrift
>>> server.
>>>
>>> if it is hue, it won't print the table names.
>>>
>>> Ah, sorry, I missed that in your original message. Not sure what's up,
>>> then
>> - we don't have any changes in CDH that would affect this. Anyone here
>> used
>> thrift on 0.89.20100621?
>>
>> -Todd
>>
>>
>>
>>> Jimmy.
>>>
>>> --------------------------------------------------
>>> From: "Todd Lipcon" <to...@cloudera.com>
>>> Sent: Thursday, September 02, 2010 5:18 PM
>>>
>>> To: <us...@hbase.apache.org>
>>> Subject: Re: thrift for hbase in CDH3 broken ?
>>>
>>>
>>> Hi Jinsong,
>>>>
>>>> Are you sure that the port you're connecting to is indeed the thrift
>>>> server?
>>>>
>>>> Unfortunately both the HBase thrift server and the Hue namenode plugin
>>>> listen on port 9090, so you might be having an issue where your HBase
>>>> client
>>>> is trying to connect to the Namenode server instead of HBase.
>>>>
>>>> You can verify the ports using a command like "/sbin/fuser -n tcp 9090"
>>>> to
>>>> see which pid has it open, then cross reference against sudo jps.
>>>>
>>>> Thanks
>>>> -Todd
>>>>
>>>> On Thu, Sep 2, 2010 at 4:40 PM, Jinsong Hu <ji...@hotmail.com>
>>>> wrote:
>>>>
>>>> Hi, There,
>>>>> I am trying to test and see if thrift for hbase works. I followed the
>>>>> example from
>>>>>
>>>>> http://www.workhabit.com/labs/centos-55-and-thriftscribe
>>>>> http://incubator.apache.org/thrift/
>>>>> http://wiki.apache.org/hadoop/Hbase/ThriftApi
>>>>>
>>>>> and wrote test code: I found that client.getTableNames();
>>>>> returns all table names successfully, but the scanner never returned
>>>>> any record. and it even throws exception:
>>>>>
>>>>> org.apache.thrift.TApplicationException: get failed: unknown result
>>>>> at
>>>>>
>>>>> org.apache.hadoop.hbase.thrift.generated.Hbase$Client.recv_get(Hbase.java:785)
>>>>> at
>>>>> org.apache.hadoop.hbase.thrift.generated.Hbase$Client.get(Hbase.java:750)
>>>>> at
>>>>>
>>>>> org.apache.hadoop.hbase.thrift.HbaseThriftTest.testThriftAPI(HbaseThriftTest.java:73)
>>>>> at
>>>>>
>>>>> org.apache.hadoop.hbase.thrift.HbaseThriftTest.main(HbaseThriftTest.java:128)
>>>>>
>>>>> I checked the syntax and the table and confirmed that the table does
>>>>> have
>>>>> record and it should
>>>>> return result, but it doesn't. Can anybody tell me what is wrong ?
>>>>>
>>>>> At this time, I am suspecting the thrift server shipped with CDH3 may
>>>>> have
>>>>> problem. if there anyway
>>>>> I can get this test code working ?
>>>>>
>>>>> I have to use thrift-0.2.0-incubating.tar.gz , as I found that the
>>>>> code
>>>>> generated from hbase-thrift project
>>>>> doesn't even compile with 0.3.0 and 0.4.0 thrift.
>>>>>
>>>>>
>>>>> Jimmy.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> public void testThriftAPI()
>>>>> {
>>>>> // Make socket
>>>>> TSocket transport = new TSocket("10.20.12.13", 9090);
>>>>>
>>>>>
>>>>>
>>>>> // Wrap in a protocol
>>>>> TBinaryProtocol protocol = new TBinaryProtocol(transport);
>>>>>
>>>>> Hbase.Client client = new Hbase.Client(protocol);
>>>>>
>>>>> try {
>>>>> transport.open();
>>>>> List<byte[]> tableNamesList=client.getTableNames();
>>>>> for (byte [] name : tableNamesList)
>>>>> {
>>>>> System.out.println(new String(name));
>>>>> }
>>>>>
>>>>> String tableName="HEARTBEAT_CLUSTER";
>>>>> String startRow="";
>>>>> List<byte[]> columns= new ArrayList<byte[]>();
>>>>> columns.add("fields:time_format".getBytes());
>>>>> columns.add("fields:customer_id".getBytes());
>>>>>
>>>>> int aScannerId=0;
>>>>> try {
>>>>>
>>>>> TCell cell=client.get(tableName.getBytes(),
>>>>> "2010-08\tproduction-2".getBytes(), "fields:customer_id".getBytes());
>>>>> if (cell != null)
>>>>> {
>>>>> String value= new String( cell.value);
>>>>> System.out.println(value);
>>>>> }
>>>>> aScannerId=client.scannerOpen(tableName.getBytes(),
>>>>> startRow.getBytes(), columns);
>>>>> TRowResult result=null;
>>>>> for (int i=0; i< 10; i++)
>>>>> {
>>>>> try {
>>>>> result=client.scannerGet(aScannerId);
>>>>> } catch (Exception ex1)
>>>>> {
>>>>>
>>>>> }
>>>>> if (result ==null) break;
>>>>>
>>>>> String timeFormat=result.getFieldValue(0).toString();
>>>>> String customerId=result.getFieldValue(1).toString();
>>>>> System.out.println(timeFormat + "\t" + customerId);
>>>>> }
>>>>> } catch (Exception ex)
>>>>> {
>>>>> ex.printStackTrace();
>>>>> }
>>>>> finally {
>>>>> if (aScannerId >0 ) client.scannerClose(aScannerId);
>>>>> }
>>>>> } catch (Exception exp)
>>>>> {
>>>>> exp.printStackTrace();
>>>>> }
>>>>> finally{
>>>>> transport.close();
>>>>> }
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Todd Lipcon
>>>> Software Engineer, Cloudera
>>>>
>>>>
>>
>>
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera
>>
>
>
>
> --
> Alexey Kovyrin
> http://kovyrin.net/
>
Re: thrift for hbase in CDH3 broken ?
Posted by Alexey Kovyrin <al...@kovyrin.net>.
We use it in Scribd.com. All clients are ruby web apps.
On Thu, Sep 2, 2010 at 10:49 PM, Todd Lipcon <to...@cloudera.com> wrote:
> On Thu, Sep 2, 2010 at 5:35 PM, Jinsong Hu <ji...@hotmail.com> wrote:
>
>> Yes, I confirmed that it is indeed thrift server.
>>
>> and the fact that the API
>>
>> List<byte[]> tableNamesList=client.getTableNames();
>>>> for (byte [] name : tableNamesList)
>>>> {
>>>> System.out.println(new String(name));
>>>> }
>>>>
>>>
>> successfully printed all table names shows that it is indeed thrift server.
>>
>> if it is hue, it won't print the table names.
>>
>> Ah, sorry, I missed that in your original message. Not sure what's up, then
> - we don't have any changes in CDH that would affect this. Anyone here used
> thrift on 0.89.20100621?
>
> -Todd
>
>
>
>> Jimmy.
>>
>> --------------------------------------------------
>> From: "Todd Lipcon" <to...@cloudera.com>
>> Sent: Thursday, September 02, 2010 5:18 PM
>>
>> To: <us...@hbase.apache.org>
>> Subject: Re: thrift for hbase in CDH3 broken ?
>>
>>
>> Hi Jinsong,
>>>
>>> Are you sure that the port you're connecting to is indeed the thrift
>>> server?
>>>
>>> Unfortunately both the HBase thrift server and the Hue namenode plugin
>>> listen on port 9090, so you might be having an issue where your HBase
>>> client
>>> is trying to connect to the Namenode server instead of HBase.
>>>
>>> You can verify the ports using a command like "/sbin/fuser -n tcp 9090" to
>>> see which pid has it open, then cross reference against sudo jps.
>>>
>>> Thanks
>>> -Todd
>>>
>>> On Thu, Sep 2, 2010 at 4:40 PM, Jinsong Hu <ji...@hotmail.com>
>>> wrote:
>>>
>>> Hi, There,
>>>> I am trying to test and see if thrift for hbase works. I followed the
>>>> example from
>>>>
>>>> http://www.workhabit.com/labs/centos-55-and-thriftscribe
>>>> http://incubator.apache.org/thrift/
>>>> http://wiki.apache.org/hadoop/Hbase/ThriftApi
>>>>
>>>> and wrote test code: I found that client.getTableNames();
>>>> returns all table names successfully, but the scanner never returned
>>>> any record. and it even throws exception:
>>>>
>>>> org.apache.thrift.TApplicationException: get failed: unknown result
>>>> at
>>>>
>>>> org.apache.hadoop.hbase.thrift.generated.Hbase$Client.recv_get(Hbase.java:785)
>>>> at
>>>> org.apache.hadoop.hbase.thrift.generated.Hbase$Client.get(Hbase.java:750)
>>>> at
>>>>
>>>> org.apache.hadoop.hbase.thrift.HbaseThriftTest.testThriftAPI(HbaseThriftTest.java:73)
>>>> at
>>>>
>>>> org.apache.hadoop.hbase.thrift.HbaseThriftTest.main(HbaseThriftTest.java:128)
>>>>
>>>> I checked the syntax and the table and confirmed that the table does have
>>>> record and it should
>>>> return result, but it doesn't. Can anybody tell me what is wrong ?
>>>>
>>>> At this time, I am suspecting the thrift server shipped with CDH3 may
>>>> have
>>>> problem. if there anyway
>>>> I can get this test code working ?
>>>>
>>>> I have to use thrift-0.2.0-incubating.tar.gz , as I found that the code
>>>> generated from hbase-thrift project
>>>> doesn't even compile with 0.3.0 and 0.4.0 thrift.
>>>>
>>>>
>>>> Jimmy.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> public void testThriftAPI()
>>>> {
>>>> // Make socket
>>>> TSocket transport = new TSocket("10.20.12.13", 9090);
>>>>
>>>>
>>>>
>>>> // Wrap in a protocol
>>>> TBinaryProtocol protocol = new TBinaryProtocol(transport);
>>>>
>>>> Hbase.Client client = new Hbase.Client(protocol);
>>>>
>>>> try {
>>>> transport.open();
>>>> List<byte[]> tableNamesList=client.getTableNames();
>>>> for (byte [] name : tableNamesList)
>>>> {
>>>> System.out.println(new String(name));
>>>> }
>>>>
>>>> String tableName="HEARTBEAT_CLUSTER";
>>>> String startRow="";
>>>> List<byte[]> columns= new ArrayList<byte[]>();
>>>> columns.add("fields:time_format".getBytes());
>>>> columns.add("fields:customer_id".getBytes());
>>>>
>>>> int aScannerId=0;
>>>> try {
>>>>
>>>> TCell cell=client.get(tableName.getBytes(),
>>>> "2010-08\tproduction-2".getBytes(), "fields:customer_id".getBytes());
>>>> if (cell != null)
>>>> {
>>>> String value= new String( cell.value);
>>>> System.out.println(value);
>>>> }
>>>> aScannerId=client.scannerOpen(tableName.getBytes(),
>>>> startRow.getBytes(), columns);
>>>> TRowResult result=null;
>>>> for (int i=0; i< 10; i++)
>>>> {
>>>> try {
>>>> result=client.scannerGet(aScannerId);
>>>> } catch (Exception ex1)
>>>> {
>>>>
>>>> }
>>>> if (result ==null) break;
>>>>
>>>> String timeFormat=result.getFieldValue(0).toString();
>>>> String customerId=result.getFieldValue(1).toString();
>>>> System.out.println(timeFormat + "\t" + customerId);
>>>> }
>>>> } catch (Exception ex)
>>>> {
>>>> ex.printStackTrace();
>>>> }
>>>> finally {
>>>> if (aScannerId >0 ) client.scannerClose(aScannerId);
>>>> }
>>>> } catch (Exception exp)
>>>> {
>>>> exp.printStackTrace();
>>>> }
>>>> finally{
>>>> transport.close();
>>>> }
>>>>
>>>> }
>>>>
>>>>
>>>
>>>
>>> --
>>> Todd Lipcon
>>> Software Engineer, Cloudera
>>>
>>>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>
--
Alexey Kovyrin
http://kovyrin.net/
Re: thrift for hbase in CDH3 broken ?
Posted by Todd Lipcon <to...@cloudera.com>.
On Thu, Sep 2, 2010 at 5:35 PM, Jinsong Hu <ji...@hotmail.com> wrote:
> Yes, I confirmed that it is indeed thrift server.
>
> and the fact that the API
>
> List<byte[]> tableNamesList=client.getTableNames();
>>> for (byte [] name : tableNamesList)
>>> {
>>> System.out.println(new String(name));
>>> }
>>>
>>
> successfully printed all table names shows that it is indeed thrift server.
>
> if it is hue, it won't print the table names.
>
> Ah, sorry, I missed that in your original message. Not sure what's up, then
- we don't have any changes in CDH that would affect this. Anyone here used
thrift on 0.89.20100621?
-Todd
> Jimmy.
>
> --------------------------------------------------
> From: "Todd Lipcon" <to...@cloudera.com>
> Sent: Thursday, September 02, 2010 5:18 PM
>
> To: <us...@hbase.apache.org>
> Subject: Re: thrift for hbase in CDH3 broken ?
>
>
> Hi Jinsong,
>>
>> Are you sure that the port you're connecting to is indeed the thrift
>> server?
>>
>> Unfortunately both the HBase thrift server and the Hue namenode plugin
>> listen on port 9090, so you might be having an issue where your HBase
>> client
>> is trying to connect to the Namenode server instead of HBase.
>>
>> You can verify the ports using a command like "/sbin/fuser -n tcp 9090" to
>> see which pid has it open, then cross reference against sudo jps.
>>
>> Thanks
>> -Todd
>>
>> On Thu, Sep 2, 2010 at 4:40 PM, Jinsong Hu <ji...@hotmail.com>
>> wrote:
>>
>> Hi, There,
>>> I am trying to test and see if thrift for hbase works. I followed the
>>> example from
>>>
>>> http://www.workhabit.com/labs/centos-55-and-thriftscribe
>>> http://incubator.apache.org/thrift/
>>> http://wiki.apache.org/hadoop/Hbase/ThriftApi
>>>
>>> and wrote test code: I found that client.getTableNames();
>>> returns all table names successfully, but the scanner never returned
>>> any record. and it even throws exception:
>>>
>>> org.apache.thrift.TApplicationException: get failed: unknown result
>>> at
>>>
>>> org.apache.hadoop.hbase.thrift.generated.Hbase$Client.recv_get(Hbase.java:785)
>>> at
>>> org.apache.hadoop.hbase.thrift.generated.Hbase$Client.get(Hbase.java:750)
>>> at
>>>
>>> org.apache.hadoop.hbase.thrift.HbaseThriftTest.testThriftAPI(HbaseThriftTest.java:73)
>>> at
>>>
>>> org.apache.hadoop.hbase.thrift.HbaseThriftTest.main(HbaseThriftTest.java:128)
>>>
>>> I checked the syntax and the table and confirmed that the table does have
>>> record and it should
>>> return result, but it doesn't. Can anybody tell me what is wrong ?
>>>
>>> At this time, I am suspecting the thrift server shipped with CDH3 may
>>> have
>>> problem. if there anyway
>>> I can get this test code working ?
>>>
>>> I have to use thrift-0.2.0-incubating.tar.gz , as I found that the code
>>> generated from hbase-thrift project
>>> doesn't even compile with 0.3.0 and 0.4.0 thrift.
>>>
>>>
>>> Jimmy.
>>>
>>>
>>>
>>>
>>>
>>>
>>> public void testThriftAPI()
>>> {
>>> // Make socket
>>> TSocket transport = new TSocket("10.20.12.13", 9090);
>>>
>>>
>>>
>>> // Wrap in a protocol
>>> TBinaryProtocol protocol = new TBinaryProtocol(transport);
>>>
>>> Hbase.Client client = new Hbase.Client(protocol);
>>>
>>> try {
>>> transport.open();
>>> List<byte[]> tableNamesList=client.getTableNames();
>>> for (byte [] name : tableNamesList)
>>> {
>>> System.out.println(new String(name));
>>> }
>>>
>>> String tableName="HEARTBEAT_CLUSTER";
>>> String startRow="";
>>> List<byte[]> columns= new ArrayList<byte[]>();
>>> columns.add("fields:time_format".getBytes());
>>> columns.add("fields:customer_id".getBytes());
>>>
>>> int aScannerId=0;
>>> try {
>>>
>>> TCell cell=client.get(tableName.getBytes(),
>>> "2010-08\tproduction-2".getBytes(), "fields:customer_id".getBytes());
>>> if (cell != null)
>>> {
>>> String value= new String( cell.value);
>>> System.out.println(value);
>>> }
>>> aScannerId=client.scannerOpen(tableName.getBytes(),
>>> startRow.getBytes(), columns);
>>> TRowResult result=null;
>>> for (int i=0; i< 10; i++)
>>> {
>>> try {
>>> result=client.scannerGet(aScannerId);
>>> } catch (Exception ex1)
>>> {
>>>
>>> }
>>> if (result ==null) break;
>>>
>>> String timeFormat=result.getFieldValue(0).toString();
>>> String customerId=result.getFieldValue(1).toString();
>>> System.out.println(timeFormat + "\t" + customerId);
>>> }
>>> } catch (Exception ex)
>>> {
>>> ex.printStackTrace();
>>> }
>>> finally {
>>> if (aScannerId >0 ) client.scannerClose(aScannerId);
>>> }
>>> } catch (Exception exp)
>>> {
>>> exp.printStackTrace();
>>> }
>>> finally{
>>> transport.close();
>>> }
>>>
>>> }
>>>
>>>
>>
>>
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera
>>
>>
--
Todd Lipcon
Software Engineer, Cloudera
Re: thrift for hbase in CDH3 broken ?
Posted by Jinsong Hu <ji...@hotmail.com>.
Yes, I confirmed that it is indeed thrift server.
and the fact that the API
>> List<byte[]> tableNamesList=client.getTableNames();
>> for (byte [] name : tableNamesList)
>> {
>> System.out.println(new String(name));
>> }
successfully printed all table names shows that it is indeed thrift server.
if it is hue, it won't print the table names.
Jimmy.
--------------------------------------------------
From: "Todd Lipcon" <to...@cloudera.com>
Sent: Thursday, September 02, 2010 5:18 PM
To: <us...@hbase.apache.org>
Subject: Re: thrift for hbase in CDH3 broken ?
> Hi Jinsong,
>
> Are you sure that the port you're connecting to is indeed the thrift
> server?
>
> Unfortunately both the HBase thrift server and the Hue namenode plugin
> listen on port 9090, so you might be having an issue where your HBase
> client
> is trying to connect to the Namenode server instead of HBase.
>
> You can verify the ports using a command like "/sbin/fuser -n tcp 9090" to
> see which pid has it open, then cross reference against sudo jps.
>
> Thanks
> -Todd
>
> On Thu, Sep 2, 2010 at 4:40 PM, Jinsong Hu <ji...@hotmail.com> wrote:
>
>> Hi, There,
>> I am trying to test and see if thrift for hbase works. I followed the
>> example from
>>
>> http://www.workhabit.com/labs/centos-55-and-thriftscribe
>> http://incubator.apache.org/thrift/
>> http://wiki.apache.org/hadoop/Hbase/ThriftApi
>>
>> and wrote test code: I found that client.getTableNames();
>> returns all table names successfully, but the scanner never returned
>> any record. and it even throws exception:
>>
>> org.apache.thrift.TApplicationException: get failed: unknown result
>> at
>> org.apache.hadoop.hbase.thrift.generated.Hbase$Client.recv_get(Hbase.java:785)
>> at
>> org.apache.hadoop.hbase.thrift.generated.Hbase$Client.get(Hbase.java:750)
>> at
>> org.apache.hadoop.hbase.thrift.HbaseThriftTest.testThriftAPI(HbaseThriftTest.java:73)
>> at
>> org.apache.hadoop.hbase.thrift.HbaseThriftTest.main(HbaseThriftTest.java:128)
>>
>> I checked the syntax and the table and confirmed that the table does have
>> record and it should
>> return result, but it doesn't. Can anybody tell me what is wrong ?
>>
>> At this time, I am suspecting the thrift server shipped with CDH3 may
>> have
>> problem. if there anyway
>> I can get this test code working ?
>>
>> I have to use thrift-0.2.0-incubating.tar.gz , as I found that the code
>> generated from hbase-thrift project
>> doesn't even compile with 0.3.0 and 0.4.0 thrift.
>>
>>
>> Jimmy.
>>
>>
>>
>>
>>
>>
>> public void testThriftAPI()
>> {
>> // Make socket
>> TSocket transport = new TSocket("10.20.12.13", 9090);
>>
>>
>>
>> // Wrap in a protocol
>> TBinaryProtocol protocol = new TBinaryProtocol(transport);
>>
>> Hbase.Client client = new Hbase.Client(protocol);
>>
>> try {
>> transport.open();
>> List<byte[]> tableNamesList=client.getTableNames();
>> for (byte [] name : tableNamesList)
>> {
>> System.out.println(new String(name));
>> }
>>
>> String tableName="HEARTBEAT_CLUSTER";
>> String startRow="";
>> List<byte[]> columns= new ArrayList<byte[]>();
>> columns.add("fields:time_format".getBytes());
>> columns.add("fields:customer_id".getBytes());
>>
>> int aScannerId=0;
>> try {
>>
>> TCell cell=client.get(tableName.getBytes(),
>> "2010-08\tproduction-2".getBytes(), "fields:customer_id".getBytes());
>> if (cell != null)
>> {
>> String value= new String( cell.value);
>> System.out.println(value);
>> }
>> aScannerId=client.scannerOpen(tableName.getBytes(),
>> startRow.getBytes(), columns);
>> TRowResult result=null;
>> for (int i=0; i< 10; i++)
>> {
>> try {
>> result=client.scannerGet(aScannerId);
>> } catch (Exception ex1)
>> {
>>
>> }
>> if (result ==null) break;
>>
>> String timeFormat=result.getFieldValue(0).toString();
>> String customerId=result.getFieldValue(1).toString();
>> System.out.println(timeFormat + "\t" + customerId);
>> }
>> } catch (Exception ex)
>> {
>> ex.printStackTrace();
>> }
>> finally {
>> if (aScannerId >0 ) client.scannerClose(aScannerId);
>> }
>> } catch (Exception exp)
>> {
>> exp.printStackTrace();
>> }
>> finally{
>> transport.close();
>> }
>>
>> }
>>
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>
Re: thrift for hbase in CDH3 broken ?
Posted by Todd Lipcon <to...@cloudera.com>.
Hi Jinsong,
Are you sure that the port you're connecting to is indeed the thrift server?
Unfortunately both the HBase thrift server and the Hue namenode plugin
listen on port 9090, so you might be having an issue where your HBase client
is trying to connect to the Namenode server instead of HBase.
You can verify the ports using a command like "/sbin/fuser -n tcp 9090" to
see which pid has it open, then cross reference against sudo jps.
Thanks
-Todd
On Thu, Sep 2, 2010 at 4:40 PM, Jinsong Hu <ji...@hotmail.com> wrote:
> Hi, There,
> I am trying to test and see if thrift for hbase works. I followed the
> example from
>
> http://www.workhabit.com/labs/centos-55-and-thriftscribe
> http://incubator.apache.org/thrift/
> http://wiki.apache.org/hadoop/Hbase/ThriftApi
>
> and wrote test code: I found that client.getTableNames();
> returns all table names successfully, but the scanner never returned
> any record. and it even throws exception:
>
> org.apache.thrift.TApplicationException: get failed: unknown result
> at
> org.apache.hadoop.hbase.thrift.generated.Hbase$Client.recv_get(Hbase.java:785)
> at
> org.apache.hadoop.hbase.thrift.generated.Hbase$Client.get(Hbase.java:750)
> at
> org.apache.hadoop.hbase.thrift.HbaseThriftTest.testThriftAPI(HbaseThriftTest.java:73)
> at
> org.apache.hadoop.hbase.thrift.HbaseThriftTest.main(HbaseThriftTest.java:128)
>
> I checked the syntax and the table and confirmed that the table does have
> record and it should
> return result, but it doesn't. Can anybody tell me what is wrong ?
>
> At this time, I am suspecting the thrift server shipped with CDH3 may have
> problem. if there anyway
> I can get this test code working ?
>
> I have to use thrift-0.2.0-incubating.tar.gz , as I found that the code
> generated from hbase-thrift project
> doesn't even compile with 0.3.0 and 0.4.0 thrift.
>
>
> Jimmy.
>
>
>
>
>
>
> public void testThriftAPI()
> {
> // Make socket
> TSocket transport = new TSocket("10.20.12.13", 9090);
>
>
>
> // Wrap in a protocol
> TBinaryProtocol protocol = new TBinaryProtocol(transport);
>
> Hbase.Client client = new Hbase.Client(protocol);
>
> try {
> transport.open();
> List<byte[]> tableNamesList=client.getTableNames();
> for (byte [] name : tableNamesList)
> {
> System.out.println(new String(name));
> }
>
> String tableName="HEARTBEAT_CLUSTER";
> String startRow="";
> List<byte[]> columns= new ArrayList<byte[]>();
> columns.add("fields:time_format".getBytes());
> columns.add("fields:customer_id".getBytes());
>
> int aScannerId=0;
> try {
>
> TCell cell=client.get(tableName.getBytes(),
> "2010-08\tproduction-2".getBytes(), "fields:customer_id".getBytes());
> if (cell != null)
> {
> String value= new String( cell.value);
> System.out.println(value);
> }
> aScannerId=client.scannerOpen(tableName.getBytes(),
> startRow.getBytes(), columns);
> TRowResult result=null;
> for (int i=0; i< 10; i++)
> {
> try {
> result=client.scannerGet(aScannerId);
> } catch (Exception ex1)
> {
>
> }
> if (result ==null) break;
>
> String timeFormat=result.getFieldValue(0).toString();
> String customerId=result.getFieldValue(1).toString();
> System.out.println(timeFormat + "\t" + customerId);
> }
> } catch (Exception ex)
> {
> ex.printStackTrace();
> }
> finally {
> if (aScannerId >0 ) client.scannerClose(aScannerId);
> }
> } catch (Exception exp)
> {
> exp.printStackTrace();
> }
> finally{
> transport.close();
> }
>
> }
>
--
Todd Lipcon
Software Engineer, Cloudera
thrift for hbase in CDH3 broken ?
Posted by Jinsong Hu <ji...@hotmail.com>.
Hi, There,
I am trying to test and see if thrift for hbase works. I followed the
example from
http://www.workhabit.com/labs/centos-55-and-thriftscribe
http://incubator.apache.org/thrift/
http://wiki.apache.org/hadoop/Hbase/ThriftApi
and wrote test code: I found that client.getTableNames();
returns all table names successfully, but the scanner never returned
any record. and it even throws exception:
org.apache.thrift.TApplicationException: get failed: unknown result
at
org.apache.hadoop.hbase.thrift.generated.Hbase$Client.recv_get(Hbase.java:785)
at
org.apache.hadoop.hbase.thrift.generated.Hbase$Client.get(Hbase.java:750)
at
org.apache.hadoop.hbase.thrift.HbaseThriftTest.testThriftAPI(HbaseThriftTest.java:73)
at
org.apache.hadoop.hbase.thrift.HbaseThriftTest.main(HbaseThriftTest.java:128)
I checked the syntax and the table and confirmed that the table does have
record and it should
return result, but it doesn't. Can anybody tell me what is wrong ?
At this time, I am suspecting the thrift server shipped with CDH3 may have
problem. if there anyway
I can get this test code working ?
I have to use thrift-0.2.0-incubating.tar.gz , as I found that the code
generated from hbase-thrift project
doesn't even compile with 0.3.0 and 0.4.0 thrift.
Jimmy.
public void testThriftAPI()
{
// Make socket
TSocket transport = new TSocket("10.20.12.13", 9090);
// Wrap in a protocol
TBinaryProtocol protocol = new TBinaryProtocol(transport);
Hbase.Client client = new Hbase.Client(protocol);
try {
transport.open();
List<byte[]> tableNamesList=client.getTableNames();
for (byte [] name : tableNamesList)
{
System.out.println(new String(name));
}
String tableName="HEARTBEAT_CLUSTER";
String startRow="";
List<byte[]> columns= new ArrayList<byte[]>();
columns.add("fields:time_format".getBytes());
columns.add("fields:customer_id".getBytes());
int aScannerId=0;
try {
TCell cell=client.get(tableName.getBytes(),
"2010-08\tproduction-2".getBytes(), "fields:customer_id".getBytes());
if (cell != null)
{
String value= new String( cell.value);
System.out.println(value);
}
aScannerId=client.scannerOpen(tableName.getBytes(),
startRow.getBytes(), columns);
TRowResult result=null;
for (int i=0; i< 10; i++)
{
try {
result=client.scannerGet(aScannerId);
} catch (Exception ex1)
{
}
if (result ==null) break;
String timeFormat=result.getFieldValue(0).toString();
String customerId=result.getFieldValue(1).toString();
System.out.println(timeFormat + "\t" + customerId);
}
} catch (Exception ex)
{
ex.printStackTrace();
}
finally {
if (aScannerId >0 ) client.scannerClose(aScannerId);
}
} catch (Exception exp)
{
exp.printStackTrace();
}
finally{
transport.close();
}
}
Re: memory usage too high in regionserver
Posted by Ted Yu <yu...@gmail.com>.
In HBase client, you can utilize HeapSize through Put:
public class Put implements HeapSize, Writable, Row, Comparable<Row> {
Cheers
On Wed, Aug 25, 2010 at 3:13 PM, Ryan Rawson <ry...@gmail.com> wrote:
> Hey,
>
> It's all over the place. HeapSize is one interface. There is also
> stuff in HRegion, Store and Memstore that tracks memory used. On top
> of it the LruBlockCache does more stuff as well.
>
> -ryan
>
> On Wed, Aug 25, 2010 at 10:59 AM, Alex Baranau <al...@gmail.com>
> wrote:
> > Hello guys,
> >
> >> HBase is a self memory tuning app
> >
> > Is there any documentation on that or if you could point to some sources
> I
> > will be very thankful (even if sources are particular classes, please
> name
> > at least few to look at).
> >
> > Thank you in advance,
> > Alex Baranau.
> >
> > On Tue, Aug 10, 2010 at 1:25 AM, Ryan Rawson <ry...@gmail.com> wrote:
> >
> >> Hey,
> >>
> >> HBase is a self memory tuning app, unlike most Java apps you may have
> >> experience with. We use as much memory as possible for memstore and
> >> block cache, but we also flush memory to keep overall usage below
> >> maximums.
> >>
> >> Enjoy,
> >> -ryan
> >>
> >> On Mon, Aug 9, 2010 at 3:22 PM, Jinsong Hu <ji...@hotmail.com>
> wrote:
> >> > you are right. now it shows up as
> >> >
> >> > request=180.5, regions=60, stores=61, storefiles=67,
> >> storefileIndexSize=143,
> >> > memstoreSize=20, compactionQueueSize=0, usedHeap=1417, maxHeap=6127,
> >> > blockCacheSize=1061944288, blockCacheFree=223059360,
> >> blockCacheCount=16239,
> >> > blockCacheHitRatio=14
> >> >
> >> > still 60 regions, and the usedHeap is only 1.4G. I already put all the
> >> > memory tunning suggested in the hbase-env.sh to do incremental GC.
> >> > looks the jvm still have drastic GC even with the incremental GC
> turned
> >> on.
> >> > now the GC log shows:
> >> >
> >> > 1462948.533: [GC 1462948.533: [ParNew: 138376K->1441K(153344K),
> 0.0037940
> >> > secs]
> >> > 1446591K->1309805K(6274432K) icms_dc=0 , 0.0038990 secs] [Times:
> >> user=0.03
> >> > sys=0
> >> > .00, real=0.00 secs]
> >> > 1463005.844: [GC 1463005.845: [ParNew: 137716K->17024K(153344K),
> >> 0.0139630
> >> > secs]
> >> > 1446081K->1327172K(6274432K) icms_dc=0 , 0.0140740 secs] [Times:
> >> user=0.11
> >> > sys=
> >> > 0.00, real=0.02 secs]
> >> >
> >> > which is reasonable.
> >> >
> >> > Jimmy.
> >> >
> >> > --------------------------------------------------
> >> > From: "Ryan Rawson" <ry...@gmail.com>
> >> > Sent: Monday, August 09, 2010 3:07 PM
> >> > To: <us...@hbase.apache.org>
> >> > Subject: Re: memory usage too high in regionserver
> >> >
> >> >> Unfortunately 'usedHeap' includes the unfound garbage...
> >> >>
> >> >> Perhaps you can post more of your GC log.
> >> >>
> >> >> On Mon, Aug 9, 2010 at 3:02 PM, Jinsong Hu <ji...@hotmail.com>
> >> wrote:
> >> >>>
> >> >>> Hi, There:
> >> >>>
> >> >>> I am using cloudera cdh3 regionserver and today, I noticed that one
> of
> >> >>> the
> >> >>> regionserver memory usage is very high:
> >> >>>
> >> >>> request=8.8, regions=60, stores=61, storefiles=67,
> >> >>> storefileIndexSize=142,
> >> >>> memstoreSize=58, compactionQueueSize=0, usedHeap=5869, maxHeap=6127,
> >> >>> blockCacheSize=1061944288, blockCacheFree=223059360,
> >> >>> blockCacheCount=16239,
> >> >>> blockCacheHitRatio=14
> >> >>>
> >> >>> this shows that only 60 regions calused close to 6G of memory usage.
> >> This
> >> >>> is
> >> >>> very high.
> >> >>>
> >> >>> the gc-hbase.log shows:
> >> >>>
> >> >>> 1461575.642: [GC 1461575.643: [ParNew: 143538K->17024K(153344K),
> >> >>> 0.0487050
> >> >>> secs]
> >> >>> 5948479K->5846957K(6274432K) icms_dc=0 , 0.0488210 secs] [Times:
> >> >>> user=0.33
> >> >>> sys=
> >> >>> 0.00, real=0.05 secs]
> >> >>> 1461693.726: [GC 1461693.726: [ParNew: 153344K->17024K(153344K),
> >> >>> 0.0454440
> >> >>> secs]
> >> >>> 5983277K->5875331K(6274432K) icms_dc=0 , 0.0455680 secs] [Times:
> >> >>> user=0.31
> >> >>> sys=
> >> >>> 0.01, real=0.05 secs]
> >> >>>
> >> >>>
> >> >>> This shows that the regionserver is indeed using 6G of memory. very
> >> close
> >> >>> to
> >> >>> the setting of -Xmx6G limit.
> >> >>> and very soon the regionserver will be OOME.
> >> >>>
> >> >>>
> >> >>> if the above holds true, that means a regionserver can only support
> >> >>> around
> >> >>> 60 regions. I configured the region size to be default 256M.
> >> >>>
> >> >>> I checked the load on the server and the load is very light. cpu and
> >> disk
> >> >>> usage is very low. What can I do to drastically increase the number
> of
> >> >>> regions
> >> >>> supported per regionserver in hbase ?
> >> >>>
> >> >>>
> >> >>> Jimmy
> >> >>>
> >> >>
> >> >
> >>
> >
>
Re: memory usage too high in regionserver
Posted by Ryan Rawson <ry...@gmail.com>.
Hey,
It's all over the place. HeapSize is one interface. There is also
stuff in HRegion, Store and Memstore that tracks memory used. On top
of it the LruBlockCache does more stuff as well.
-ryan
On Wed, Aug 25, 2010 at 10:59 AM, Alex Baranau <al...@gmail.com> wrote:
> Hello guys,
>
>> HBase is a self memory tuning app
>
> Is there any documentation on that or if you could point to some sources I
> will be very thankful (even if sources are particular classes, please name
> at least few to look at).
>
> Thank you in advance,
> Alex Baranau.
>
> On Tue, Aug 10, 2010 at 1:25 AM, Ryan Rawson <ry...@gmail.com> wrote:
>
>> Hey,
>>
>> HBase is a self memory tuning app, unlike most Java apps you may have
>> experience with. We use as much memory as possible for memstore and
>> block cache, but we also flush memory to keep overall usage below
>> maximums.
>>
>> Enjoy,
>> -ryan
>>
>> On Mon, Aug 9, 2010 at 3:22 PM, Jinsong Hu <ji...@hotmail.com> wrote:
>> > you are right. now it shows up as
>> >
>> > request=180.5, regions=60, stores=61, storefiles=67,
>> storefileIndexSize=143,
>> > memstoreSize=20, compactionQueueSize=0, usedHeap=1417, maxHeap=6127,
>> > blockCacheSize=1061944288, blockCacheFree=223059360,
>> blockCacheCount=16239,
>> > blockCacheHitRatio=14
>> >
>> > still 60 regions, and the usedHeap is only 1.4G. I already put all the
>> > memory tunning suggested in the hbase-env.sh to do incremental GC.
>> > looks the jvm still have drastic GC even with the incremental GC turned
>> on.
>> > now the GC log shows:
>> >
>> > 1462948.533: [GC 1462948.533: [ParNew: 138376K->1441K(153344K), 0.0037940
>> > secs]
>> > 1446591K->1309805K(6274432K) icms_dc=0 , 0.0038990 secs] [Times:
>> user=0.03
>> > sys=0
>> > .00, real=0.00 secs]
>> > 1463005.844: [GC 1463005.845: [ParNew: 137716K->17024K(153344K),
>> 0.0139630
>> > secs]
>> > 1446081K->1327172K(6274432K) icms_dc=0 , 0.0140740 secs] [Times:
>> user=0.11
>> > sys=
>> > 0.00, real=0.02 secs]
>> >
>> > which is reasonable.
>> >
>> > Jimmy.
>> >
>> > --------------------------------------------------
>> > From: "Ryan Rawson" <ry...@gmail.com>
>> > Sent: Monday, August 09, 2010 3:07 PM
>> > To: <us...@hbase.apache.org>
>> > Subject: Re: memory usage too high in regionserver
>> >
>> >> Unfortunately 'usedHeap' includes the unfound garbage...
>> >>
>> >> Perhaps you can post more of your GC log.
>> >>
>> >> On Mon, Aug 9, 2010 at 3:02 PM, Jinsong Hu <ji...@hotmail.com>
>> wrote:
>> >>>
>> >>> Hi, There:
>> >>>
>> >>> I am using cloudera cdh3 regionserver and today, I noticed that one of
>> >>> the
>> >>> regionserver memory usage is very high:
>> >>>
>> >>> request=8.8, regions=60, stores=61, storefiles=67,
>> >>> storefileIndexSize=142,
>> >>> memstoreSize=58, compactionQueueSize=0, usedHeap=5869, maxHeap=6127,
>> >>> blockCacheSize=1061944288, blockCacheFree=223059360,
>> >>> blockCacheCount=16239,
>> >>> blockCacheHitRatio=14
>> >>>
>> >>> this shows that only 60 regions calused close to 6G of memory usage.
>> This
>> >>> is
>> >>> very high.
>> >>>
>> >>> the gc-hbase.log shows:
>> >>>
>> >>> 1461575.642: [GC 1461575.643: [ParNew: 143538K->17024K(153344K),
>> >>> 0.0487050
>> >>> secs]
>> >>> 5948479K->5846957K(6274432K) icms_dc=0 , 0.0488210 secs] [Times:
>> >>> user=0.33
>> >>> sys=
>> >>> 0.00, real=0.05 secs]
>> >>> 1461693.726: [GC 1461693.726: [ParNew: 153344K->17024K(153344K),
>> >>> 0.0454440
>> >>> secs]
>> >>> 5983277K->5875331K(6274432K) icms_dc=0 , 0.0455680 secs] [Times:
>> >>> user=0.31
>> >>> sys=
>> >>> 0.01, real=0.05 secs]
>> >>>
>> >>>
>> >>> This shows that the regionserver is indeed using 6G of memory. very
>> close
>> >>> to
>> >>> the setting of -Xmx6G limit.
>> >>> and very soon the regionserver will be OOME.
>> >>>
>> >>>
>> >>> if the above holds true, that means a regionserver can only support
>> >>> around
>> >>> 60 regions. I configured the region size to be default 256M.
>> >>>
>> >>> I checked the load on the server and the load is very light. cpu and
>> disk
>> >>> usage is very low. What can I do to drastically increase the number of
>> >>> regions
>> >>> supported per regionserver in hbase ?
>> >>>
>> >>>
>> >>> Jimmy
>> >>>
>> >>
>> >
>>
>
Re: memory usage too high in regionserver
Posted by Alex Baranau <al...@gmail.com>.
Hello guys,
> HBase is a self memory tuning app
Is there any documentation on that or if you could point to some sources I
will be very thankful (even if sources are particular classes, please name
at least few to look at).
Thank you in advance,
Alex Baranau.
On Tue, Aug 10, 2010 at 1:25 AM, Ryan Rawson <ry...@gmail.com> wrote:
> Hey,
>
> HBase is a self memory tuning app, unlike most Java apps you may have
> experience with. We use as much memory as possible for memstore and
> block cache, but we also flush memory to keep overall usage below
> maximums.
>
> Enjoy,
> -ryan
>
> On Mon, Aug 9, 2010 at 3:22 PM, Jinsong Hu <ji...@hotmail.com> wrote:
> > you are right. now it shows up as
> >
> > request=180.5, regions=60, stores=61, storefiles=67,
> storefileIndexSize=143,
> > memstoreSize=20, compactionQueueSize=0, usedHeap=1417, maxHeap=6127,
> > blockCacheSize=1061944288, blockCacheFree=223059360,
> blockCacheCount=16239,
> > blockCacheHitRatio=14
> >
> > still 60 regions, and the usedHeap is only 1.4G. I already put all the
> > memory tunning suggested in the hbase-env.sh to do incremental GC.
> > looks the jvm still have drastic GC even with the incremental GC turned
> on.
> > now the GC log shows:
> >
> > 1462948.533: [GC 1462948.533: [ParNew: 138376K->1441K(153344K), 0.0037940
> > secs]
> > 1446591K->1309805K(6274432K) icms_dc=0 , 0.0038990 secs] [Times:
> user=0.03
> > sys=0
> > .00, real=0.00 secs]
> > 1463005.844: [GC 1463005.845: [ParNew: 137716K->17024K(153344K),
> 0.0139630
> > secs]
> > 1446081K->1327172K(6274432K) icms_dc=0 , 0.0140740 secs] [Times:
> user=0.11
> > sys=
> > 0.00, real=0.02 secs]
> >
> > which is reasonable.
> >
> > Jimmy.
> >
> > --------------------------------------------------
> > From: "Ryan Rawson" <ry...@gmail.com>
> > Sent: Monday, August 09, 2010 3:07 PM
> > To: <us...@hbase.apache.org>
> > Subject: Re: memory usage too high in regionserver
> >
> >> Unfortunately 'usedHeap' includes the unfound garbage...
> >>
> >> Perhaps you can post more of your GC log.
> >>
> >> On Mon, Aug 9, 2010 at 3:02 PM, Jinsong Hu <ji...@hotmail.com>
> wrote:
> >>>
> >>> Hi, There:
> >>>
> >>> I am using cloudera cdh3 regionserver and today, I noticed that one of
> >>> the
> >>> regionserver memory usage is very high:
> >>>
> >>> request=8.8, regions=60, stores=61, storefiles=67,
> >>> storefileIndexSize=142,
> >>> memstoreSize=58, compactionQueueSize=0, usedHeap=5869, maxHeap=6127,
> >>> blockCacheSize=1061944288, blockCacheFree=223059360,
> >>> blockCacheCount=16239,
> >>> blockCacheHitRatio=14
> >>>
> >>> this shows that only 60 regions calused close to 6G of memory usage.
> This
> >>> is
> >>> very high.
> >>>
> >>> the gc-hbase.log shows:
> >>>
> >>> 1461575.642: [GC 1461575.643: [ParNew: 143538K->17024K(153344K),
> >>> 0.0487050
> >>> secs]
> >>> 5948479K->5846957K(6274432K) icms_dc=0 , 0.0488210 secs] [Times:
> >>> user=0.33
> >>> sys=
> >>> 0.00, real=0.05 secs]
> >>> 1461693.726: [GC 1461693.726: [ParNew: 153344K->17024K(153344K),
> >>> 0.0454440
> >>> secs]
> >>> 5983277K->5875331K(6274432K) icms_dc=0 , 0.0455680 secs] [Times:
> >>> user=0.31
> >>> sys=
> >>> 0.01, real=0.05 secs]
> >>>
> >>>
> >>> This shows that the regionserver is indeed using 6G of memory. very
> close
> >>> to
> >>> the setting of -Xmx6G limit.
> >>> and very soon the regionserver will be OOME.
> >>>
> >>>
> >>> if the above holds true, that means a regionserver can only support
> >>> around
> >>> 60 regions. I configured the region size to be default 256M.
> >>>
> >>> I checked the load on the server and the load is very light. cpu and
> disk
> >>> usage is very low. What can I do to drastically increase the number of
> >>> regions
> >>> supported per regionserver in hbase ?
> >>>
> >>>
> >>> Jimmy
> >>>
> >>
> >
>
Re: memory usage too high in regionserver
Posted by Ryan Rawson <ry...@gmail.com>.
Hey,
HBase is a self memory tuning app, unlike most Java apps you may have
experience with. We use as much memory as possible for memstore and
block cache, but we also flush memory to keep overall usage below
maximums.
Enjoy,
-ryan
On Mon, Aug 9, 2010 at 3:22 PM, Jinsong Hu <ji...@hotmail.com> wrote:
> you are right. now it shows up as
>
> request=180.5, regions=60, stores=61, storefiles=67, storefileIndexSize=143,
> memstoreSize=20, compactionQueueSize=0, usedHeap=1417, maxHeap=6127,
> blockCacheSize=1061944288, blockCacheFree=223059360, blockCacheCount=16239,
> blockCacheHitRatio=14
>
> still 60 regions, and the usedHeap is only 1.4G. I already put all the
> memory tunning suggested in the hbase-env.sh to do incremental GC.
> looks the jvm still have drastic GC even with the incremental GC turned on.
> now the GC log shows:
>
> 1462948.533: [GC 1462948.533: [ParNew: 138376K->1441K(153344K), 0.0037940
> secs]
> 1446591K->1309805K(6274432K) icms_dc=0 , 0.0038990 secs] [Times: user=0.03
> sys=0
> .00, real=0.00 secs]
> 1463005.844: [GC 1463005.845: [ParNew: 137716K->17024K(153344K), 0.0139630
> secs]
> 1446081K->1327172K(6274432K) icms_dc=0 , 0.0140740 secs] [Times: user=0.11
> sys=
> 0.00, real=0.02 secs]
>
> which is reasonable.
>
> Jimmy.
>
> --------------------------------------------------
> From: "Ryan Rawson" <ry...@gmail.com>
> Sent: Monday, August 09, 2010 3:07 PM
> To: <us...@hbase.apache.org>
> Subject: Re: memory usage too high in regionserver
>
>> Unfortunately 'usedHeap' includes the unfound garbage...
>>
>> Perhaps you can post more of your GC log.
>>
>> On Mon, Aug 9, 2010 at 3:02 PM, Jinsong Hu <ji...@hotmail.com> wrote:
>>>
>>> Hi, There:
>>>
>>> I am using cloudera cdh3 regionserver and today, I noticed that one of
>>> the
>>> regionserver memory usage is very high:
>>>
>>> request=8.8, regions=60, stores=61, storefiles=67,
>>> storefileIndexSize=142,
>>> memstoreSize=58, compactionQueueSize=0, usedHeap=5869, maxHeap=6127,
>>> blockCacheSize=1061944288, blockCacheFree=223059360,
>>> blockCacheCount=16239,
>>> blockCacheHitRatio=14
>>>
>>> this shows that only 60 regions calused close to 6G of memory usage. This
>>> is
>>> very high.
>>>
>>> the gc-hbase.log shows:
>>>
>>> 1461575.642: [GC 1461575.643: [ParNew: 143538K->17024K(153344K),
>>> 0.0487050
>>> secs]
>>> 5948479K->5846957K(6274432K) icms_dc=0 , 0.0488210 secs] [Times:
>>> user=0.33
>>> sys=
>>> 0.00, real=0.05 secs]
>>> 1461693.726: [GC 1461693.726: [ParNew: 153344K->17024K(153344K),
>>> 0.0454440
>>> secs]
>>> 5983277K->5875331K(6274432K) icms_dc=0 , 0.0455680 secs] [Times:
>>> user=0.31
>>> sys=
>>> 0.01, real=0.05 secs]
>>>
>>>
>>> This shows that the regionserver is indeed using 6G of memory. very close
>>> to
>>> the setting of -Xmx6G limit.
>>> and very soon the regionserver will be OOME.
>>>
>>>
>>> if the above holds true, that means a regionserver can only support
>>> around
>>> 60 regions. I configured the region size to be default 256M.
>>>
>>> I checked the load on the server and the load is very light. cpu and disk
>>> usage is very low. What can I do to drastically increase the number of
>>> regions
>>> supported per regionserver in hbase ?
>>>
>>>
>>> Jimmy
>>>
>>
>
Re: memory usage too high in regionserver
Posted by Jinsong Hu <ji...@hotmail.com>.
you are right. now it shows up as
request=180.5, regions=60, stores=61, storefiles=67, storefileIndexSize=143,
memstoreSize=20, compactionQueueSize=0, usedHeap=1417, maxHeap=6127,
blockCacheSize=1061944288, blockCacheFree=223059360, blockCacheCount=16239,
blockCacheHitRatio=14
still 60 regions, and the usedHeap is only 1.4G. I already put all the
memory tunning suggested in the hbase-env.sh to do incremental GC.
looks the jvm still have drastic GC even with the incremental GC turned on.
now the GC log shows:
1462948.533: [GC 1462948.533: [ParNew: 138376K->1441K(153344K), 0.0037940
secs]
1446591K->1309805K(6274432K) icms_dc=0 , 0.0038990 secs] [Times: user=0.03
sys=0
.00, real=0.00 secs]
1463005.844: [GC 1463005.845: [ParNew: 137716K->17024K(153344K), 0.0139630
secs]
1446081K->1327172K(6274432K) icms_dc=0 , 0.0140740 secs] [Times: user=0.11
sys=
0.00, real=0.02 secs]
which is reasonable.
Jimmy.
--------------------------------------------------
From: "Ryan Rawson" <ry...@gmail.com>
Sent: Monday, August 09, 2010 3:07 PM
To: <us...@hbase.apache.org>
Subject: Re: memory usage too high in regionserver
> Unfortunately 'usedHeap' includes the unfound garbage...
>
> Perhaps you can post more of your GC log.
>
> On Mon, Aug 9, 2010 at 3:02 PM, Jinsong Hu <ji...@hotmail.com> wrote:
>> Hi, There:
>>
>> I am using cloudera cdh3 regionserver and today, I noticed that one of
>> the
>> regionserver memory usage is very high:
>>
>> request=8.8, regions=60, stores=61, storefiles=67,
>> storefileIndexSize=142,
>> memstoreSize=58, compactionQueueSize=0, usedHeap=5869, maxHeap=6127,
>> blockCacheSize=1061944288, blockCacheFree=223059360,
>> blockCacheCount=16239,
>> blockCacheHitRatio=14
>>
>> this shows that only 60 regions calused close to 6G of memory usage. This
>> is
>> very high.
>>
>> the gc-hbase.log shows:
>>
>> 1461575.642: [GC 1461575.643: [ParNew: 143538K->17024K(153344K),
>> 0.0487050
>> secs]
>> 5948479K->5846957K(6274432K) icms_dc=0 , 0.0488210 secs] [Times:
>> user=0.33
>> sys=
>> 0.00, real=0.05 secs]
>> 1461693.726: [GC 1461693.726: [ParNew: 153344K->17024K(153344K),
>> 0.0454440
>> secs]
>> 5983277K->5875331K(6274432K) icms_dc=0 , 0.0455680 secs] [Times:
>> user=0.31
>> sys=
>> 0.01, real=0.05 secs]
>>
>>
>> This shows that the regionserver is indeed using 6G of memory. very close
>> to
>> the setting of -Xmx6G limit.
>> and very soon the regionserver will be OOME.
>>
>>
>> if the above holds true, that means a regionserver can only support
>> around
>> 60 regions. I configured the region size to be default 256M.
>>
>> I checked the load on the server and the load is very light. cpu and disk
>> usage is very low. What can I do to drastically increase the number of
>> regions
>> supported per regionserver in hbase ?
>>
>>
>> Jimmy
>>
>
Re: memory usage too high in regionserver
Posted by Ryan Rawson <ry...@gmail.com>.
Unfortunately 'usedHeap' includes the unfound garbage...
Perhaps you can post more of your GC log.
On Mon, Aug 9, 2010 at 3:02 PM, Jinsong Hu <ji...@hotmail.com> wrote:
> Hi, There:
>
> I am using cloudera cdh3 regionserver and today, I noticed that one of the
> regionserver memory usage is very high:
>
> request=8.8, regions=60, stores=61, storefiles=67, storefileIndexSize=142,
> memstoreSize=58, compactionQueueSize=0, usedHeap=5869, maxHeap=6127,
> blockCacheSize=1061944288, blockCacheFree=223059360, blockCacheCount=16239,
> blockCacheHitRatio=14
>
> this shows that only 60 regions calused close to 6G of memory usage. This is
> very high.
>
> the gc-hbase.log shows:
>
> 1461575.642: [GC 1461575.643: [ParNew: 143538K->17024K(153344K), 0.0487050
> secs]
> 5948479K->5846957K(6274432K) icms_dc=0 , 0.0488210 secs] [Times: user=0.33
> sys=
> 0.00, real=0.05 secs]
> 1461693.726: [GC 1461693.726: [ParNew: 153344K->17024K(153344K), 0.0454440
> secs]
> 5983277K->5875331K(6274432K) icms_dc=0 , 0.0455680 secs] [Times: user=0.31
> sys=
> 0.01, real=0.05 secs]
>
>
> This shows that the regionserver is indeed using 6G of memory. very close to
> the setting of -Xmx6G limit.
> and very soon the regionserver will be OOME.
>
>
> if the above holds true, that means a regionserver can only support around
> 60 regions. I configured the region size to be default 256M.
>
> I checked the load on the server and the load is very light. cpu and disk
> usage is very low. What can I do to drastically increase the number of
> regions
> supported per regionserver in hbase ?
>
>
> Jimmy
>
memory usage too high in regionserver
Posted by Jinsong Hu <ji...@hotmail.com>.
Hi, There:
I am using cloudera cdh3 regionserver and today, I noticed that one of the
regionserver memory usage is very high:
request=8.8, regions=60, stores=61, storefiles=67, storefileIndexSize=142,
memstoreSize=58, compactionQueueSize=0, usedHeap=5869, maxHeap=6127,
blockCacheSize=1061944288, blockCacheFree=223059360, blockCacheCount=16239,
blockCacheHitRatio=14
this shows that only 60 regions calused close to 6G of memory usage. This is
very high.
the gc-hbase.log shows:
1461575.642: [GC 1461575.643: [ParNew: 143538K->17024K(153344K), 0.0487050
secs]
5948479K->5846957K(6274432K) icms_dc=0 , 0.0488210 secs] [Times: user=0.33
sys=
0.00, real=0.05 secs]
1461693.726: [GC 1461693.726: [ParNew: 153344K->17024K(153344K), 0.0454440
secs]
5983277K->5875331K(6274432K) icms_dc=0 , 0.0455680 secs] [Times: user=0.31
sys=
0.01, real=0.05 secs]
This shows that the regionserver is indeed using 6G of memory. very close to
the setting of -Xmx6G limit.
and very soon the regionserver will be OOME.
if the above holds true, that means a regionserver can only support around
60 regions. I configured the region size to be default 256M.
I checked the load on the server and the load is very light. cpu and disk
usage is very low. What can I do to drastically increase the number of
regions
supported per regionserver in hbase ?
Jimmy
Re: hbase error caused by DFS timeout
Posted by Ryan Rawson <ry...@gmail.com>.
Try this config:
<property>
<name>dfs.datanode.socket.write.timeout</name>
<value>0</value>
</property>
in both hbase-site.xml and core-site.xml in the hadoop configs.
-ryan
On Mon, Aug 9, 2010 at 10:02 AM, Buttler, David <bu...@llnl.gov> wrote:
> Hi all,
> I seem to get this error far too frequently:
>
> 2010-08-09 09:54:03,685 INFO org.apache.hadoop.hbase.regionserver.Store: Started compaction of 1 file(s) in annotations of doc,293817e024ed1d54a11e9e7c9b836dd837badbbc,1281372823189, hasReferences=true, into /hbase/doc/compaction.dir/237345967, seqid=1182913218
> 2010-08-09 09:54:03,784 WARN org.apache.hadoop.hdfs.DFSClient: DFSOutputStream ResponseProcessor exception for block blk_-4556852958383799371_431518java.net.SocketTimeoutException: 6000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/10.220.5.35:49924 remote=/10.220.5.14:50010]
> at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
> at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
> at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
> at java.io.DataInputStream.readFully(DataInputStream.java:178)
> at java.io.DataInputStream.readLong(DataInputStream.java:399)
> at org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.readFields(DataTransferProtocol.java:119)
> at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2424)
>
> This basically is taking one of clusters down consistently. Is there an obvious thing I can do about this?
> I have seen this across three different clusters with radically different hardware, leading me to believe that I have misconfigured something in either hbase or hdfs
>
> Any ideas of where to look?
>
> Thanks,
> Dave
>