You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Adam Silberstein <si...@yahoo-inc.com> on 2010/01/13 20:25:31 UTC
Java client problem w/ zookeeper
Hi,
I set up a new Hbase cluster and am seeing an error I haven't seen before.
When I start up Hbase I have this NodeExists error in the zookeeper log:
org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode =
NodeExists
at
org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProcess
or.java:245)
at
org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.ja
va:114)
2010-01-13 11:10:47,503 WARN
org.apache.zookeeper.server.PrepRequestProcessor: Got exception when
processing sessionid:0x1262918396b0002 type:create cxid:0xc
zxid:0xfffffffffffffffe txntype:unknown n/a
org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode =
NodeExists
at
org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProcess
or.java:245)
at
org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.ja
va:114)
I've read that may be harmless. But then when I run the java client, it
cannot do any operations. And I see a similar NodeExists error in the
zookeeper log.
2010-01-13 11:11:24,661 INFO org.apache.zookeeper.server.NIOServerCnxn:
Connected to /203.104.21.210:36928 lastZxid 0
2010-01-13 11:11:24,661 INFO org.apache.zookeeper.server.NIOServerCnxn:
Creating new session 0x1262918396b0004
2010-01-13 11:11:24,664 INFO org.apache.zookeeper.server.NIOServerCnxn:
Finished init of 0x1262918396b0004 valid:true
2010-01-13 11:11:24,973 WARN
org.apache.zookeeper.server.PrepRequestProcessor: Got exception when
processing sessionid:0x1262918396b0004 type:create cxid:0x2
zxid:0xfffffffffffffffe txntype:unknown n/a
org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode =
NodeExists
at
org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProcess
or.java:245)
at
org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.ja
va:114)
2010-01-13 11:11:33,088 WARN org.apache.zookeeper.server.NIOServerCnxn:
Exception causing close of session 0x1262918396b0004 due to
java.io.IOException: Read error
2010-01-13 11:11:33,089 INFO org.apache.zookeeper.server.NIOServerCnxn:
closing session:0x1262918396b0004 NIOServerCnxn:
java.nio.channels.SocketChannel[connected local=/203.104.21.210:2181
remote=/203.104.21.210:36928]
2010-01-13 11:11:42,709 INFO org.apache.zookeeper.server.NIOServerCnxn:
Connected to /203.104.21.212:54837 lastZxid 0
2010-01-13 11:11:42,709 INFO org.apache.zookeeper.server.NIOServerCnxn:
Creating new session 0x1262918396b0005
2010-01-13 11:11:42,711 INFO org.apache.zookeeper.server.NIOServerCnxn:
Finished init of 0x1262918396b0005 valid:true
2010-01-13 11:11:42,711 WARN
org.apache.zookeeper.server.PrepRequestProcessor: Got exception when
processing sessionid:0x1262918396b0005 type:create cxid:0x1
zxid:0xfffffffffffffffe txntype:unknown n/a
org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode =
NodeExists
at
org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProcess
or.java:245)
at
org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.ja
va:114)
2010-01-13 11:12:06,002 INFO org.apache.zookeeper.server.SessionTrackerImpl:
Expiring session 0x1262918396b0004
2010-01-13 11:12:06,002 INFO org.apache.zookeeper.server.ZooKeeperServer:
Expiring session 0x1262918396b0004
2010-01-13 11:12:06,002 INFO
org.apache.zookeeper.server.PrepRequestProcessor: Processed session
termination request for id: 0x1262918396b0004
I've tried wiping the whole setup, including the zookeeper files that Hbase
puts in /tmp and formatting HDFS. But I get the same problem again. I'm
not seeing any exceptions on the client-side.
Thanks,
Adam
Re: Java client problem w/ zookeeper
Posted by Adam Silberstein <si...@yahoo-inc.com>.
Yes, I do have that set. In my case, the quorum is just a single machine,
that is also the Hbase master. And I'm pretty sure that the client is
reaching it, because as soon as I start my client, I see a bunch of the
"NodeExists" exceptions in the zookeeper log.
-Adam
On 1/13/10 1:07 PM, "Jean-Daniel Cryans" <jd...@apache.org> wrote:
> Oh, so HBase is working fine but you have a remote client connectivity
> problem. You gave the good hbase.zookeeper.quorum to HTable?
>
> J-D
>
> On Wed, Jan 13, 2010 at 1:04 PM, Adam Silberstein
> <si...@yahoo-inc.com> wrote:
>> No, nothing at the master, other than a bunch of NodeExists exceptions in
>> the zookeeper log.
>>
>> Something else I've found is that I can add records to the table through the
>> shell, without problems. It's only from the java client, on a separate box,
>> that I'm having problems.
>>
>> Thanks,
>> Adam
>>
>>
>> On 1/13/10 12:08 PM, "Jean-Daniel Cryans" <jd...@apache.org> wrote:
>>
>>> It means the root region likely wasn't assigned, anything in the
>>> master about it?
>>>
>>> J-D
>>>
>>> On Wed, Jan 13, 2010 at 11:43 AM, Adam Silberstein
>>> <si...@yahoo-inc.com> wrote:
>>>> I have version 0.20.2. The client requests hang.
>>>> And, I looked in the region server logs...here is the likely culprit:
>>>>
>>>> java.io.IOException: org.apache.zookeeper.KeeperException$NoNodeException:
>>>> KeeperErrorCode = NoNode for /hbase/root-region-server
>>>> at
>>>>
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKe>>>>
e
>>>> perWrapper.java:332)
>>>> at
>>>>
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddress(ZooKeeperWra>>>>
p
>>>> per.java:318)
>>>> at
>>>>
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readRootRegionLocation(Z>>>>
o
>>>> oKeeperWrapper.java:231)
>>>> at
>>>>
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:4>>>>
4
>>>> 2)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> Caused by: org.apache.zookeeper.KeeperException$NoNodeException:
>>>> KeeperErrorCode = NoNode for /hbase/root-region-server
>>>> at
>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
>>>> at
>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
>>>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:892)
>>>> at
>>>>
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKe>>>>
e
>>>> perWrapper.java:328)
>>>> ... 4 more
>>>> java.io.IOException: org.apache.zookeeper.KeeperException$NoNodeException:
>>>> KeeperErrorCode = NoNode for /hbase/root-region-server
>>>> at
>>>>
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKe>>>>
e
>>>> perWrapper.java:332)
>>>> at
>>>>
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddress(ZooKeeperWra>>>>
p
>>>> per.java:318)
>>>> at
>>>>
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readRootRegionLocation(Z>>>>
o
>>>> oKeeperWrapper.java:231)
>>>> at
>>>>
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:4>>>>
4
>>>> 2)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> Caused by: org.apache.zookeeper.KeeperException$NoNodeException:
>>>> KeeperErrorCode = NoNode for /hbase/root-region-server
>>>> at
>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
>>>> at
>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
>>>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:892)
>>>> at
>>>>
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKe>>>>
e
>>>> perWrapper.java:328)
>>>> ... 4 more
>>>>
>>>> Is there zk state on the region servers? Maybe when wiping the system, I
>>>> should have wiped something on them.
>>>>
>>>> Thanks,
>>>> Adam
>>>>
>>>> On 1/13/10 11:33 AM, "Jean-Daniel Cryans" <jd...@apache.org> wrote:
>>>>
>>>>> Those node exists are usually harmless, I would like the see what's on
>>>>> the HBase region server and master side.
>>>>>
>>>>> You say it cannot do any operation but you don't see exceptions
>>>>> client-side? It just hangs? Which version is it?
>>>>>
>>>>> Thx,
>>>>>
>>>>> J-D
>>>>>
>>>>> On Wed, Jan 13, 2010 at 11:25 AM, Adam Silberstein
>>>>> <si...@yahoo-inc.com> wrote:
>>>>>> Hi,
>>>>>> I set up a new Hbase cluster and am seeing an error I haven't seen
>>>>>> before.
>>>>>> When I start up Hbase I have this NodeExists error in the zookeeper log:
>>>>>>
>>>>>> org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode
>>>>>> =
>>>>>> NodeExists
>>>>>> at
>>>>>>
>> org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProces>>
>> >>
>> s
>>>>>> or.java:245)
>>>>>> at
>>>>>>
>> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.j>>
>> >>
>> a
>>>>>> va:114)
>>>>>> 2010-01-13 11:10:47,503 WARN
>>>>>> org.apache.zookeeper.server.PrepRequestProcessor: Got exception when
>>>>>> processing sessionid:0x1262918396b0002 type:create cxid:0xc
>>>>>> zxid:0xfffffffffffffffe txntype:unknown n/a
>>>>>> org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode
>>>>>> =
>>>>>> NodeExists
>>>>>> at
>>>>>>
>> org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProces>>
>> >>
>> s
>>>>>> or.java:245)
>>>>>> at
>>>>>>
>> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.j>>
>> >>
>> a
>>>>>> va:114)
>>>>>>
>>>>>> I've read that may be harmless. But then when I run the java client, it
>>>>>> cannot do any operations. And I see a similar NodeExists error in the
>>>>>> zookeeper log.
>>>>>>
>>>>>>
>>>>>> 2010-01-13 11:11:24,661 INFO org.apache.zookeeper.server.NIOServerCnxn:
>>>>>> Connected to /203.104.21.210:36928 lastZxid 0
>>>>>> 2010-01-13 11:11:24,661 INFO org.apache.zookeeper.server.NIOServerCnxn:
>>>>>> Creating new session 0x1262918396b0004
>>>>>> 2010-01-13 11:11:24,664 INFO org.apache.zookeeper.server.NIOServerCnxn:
>>>>>> Finished init of 0x1262918396b0004 valid:true
>>>>>> 2010-01-13 11:11:24,973 WARN
>>>>>> org.apache.zookeeper.server.PrepRequestProcessor: Got exception when
>>>>>> processing sessionid:0x1262918396b0004 type:create cxid:0x2
>>>>>> zxid:0xfffffffffffffffe txntype:unknown n/a
>>>>>> org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode
>>>>>> =
>>>>>> NodeExists
>>>>>> at
>>>>>>
>> org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProces>>
>> >>
>> s
>>>>>> or.java:245)
>>>>>> at
>>>>>>
>> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.j>>
>> >>
>> a
>>>>>> va:114)
>>>>>> 2010-01-13 11:11:33,088 WARN org.apache.zookeeper.server.NIOServerCnxn:
>>>>>> Exception causing close of session 0x1262918396b0004 due to
>>>>>> java.io.IOException: Read error
>>>>>> 2010-01-13 11:11:33,089 INFO org.apache.zookeeper.server.NIOServerCnxn:
>>>>>> closing session:0x1262918396b0004 NIOServerCnxn:
>>>>>> java.nio.channels.SocketChannel[connected local=/203.104.21.210:2181
>>>>>> remote=/203.104.21.210:36928]
>>>>>> 2010-01-13 11:11:42,709 INFO org.apache.zookeeper.server.NIOServerCnxn:
>>>>>> Connected to /203.104.21.212:54837 lastZxid 0
>>>>>> 2010-01-13 11:11:42,709 INFO org.apache.zookeeper.server.NIOServerCnxn:
>>>>>> Creating new session 0x1262918396b0005
>>>>>> 2010-01-13 11:11:42,711 INFO org.apache.zookeeper.server.NIOServerCnxn:
>>>>>> Finished init of 0x1262918396b0005 valid:true
>>>>>> 2010-01-13 11:11:42,711 WARN
>>>>>> org.apache.zookeeper.server.PrepRequestProcessor: Got exception when
>>>>>> processing sessionid:0x1262918396b0005 type:create cxid:0x1
>>>>>> zxid:0xfffffffffffffffe txntype:unknown n/a
>>>>>> org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode
>>>>>> =
>>>>>> NodeExists
>>>>>> at
>>>>>>
>> org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProces>>
>> >>
>> s
>>>>>> or.java:245)
>>>>>> at
>>>>>>
>> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.j>>
>> >>
>> a
>>>>>> va:114)
>>>>>> 2010-01-13 11:12:06,002 INFO
>>>>>> org.apache.zookeeper.server.SessionTrackerImpl:
>>>>>> Expiring session 0x1262918396b0004
>>>>>> 2010-01-13 11:12:06,002 INFO org.apache.zookeeper.server.ZooKeeperServer:
>>>>>> Expiring session 0x1262918396b0004
>>>>>> 2010-01-13 11:12:06,002 INFO
>>>>>> org.apache.zookeeper.server.PrepRequestProcessor: Processed session
>>>>>> termination request for id: 0x1262918396b0004
>>>>>>
>>>>>> I've tried wiping the whole setup, including the zookeeper files that
>>>>>> Hbase
>>>>>> puts in /tmp and formatting HDFS. But I get the same problem again. I'm
>>>>>> not seeing any exceptions on the client-side.
>>>>>>
>>>>>> Thanks,
>>>>>> Adam
>>>>>>
>>>>>>
>>>>
>>>>
>>
>>
Re: Java client problem w/ zookeeper
Posted by Jean-Daniel Cryans <jd...@apache.org>.
Oh, so HBase is working fine but you have a remote client connectivity
problem. You gave the good hbase.zookeeper.quorum to HTable?
J-D
On Wed, Jan 13, 2010 at 1:04 PM, Adam Silberstein
<si...@yahoo-inc.com> wrote:
> No, nothing at the master, other than a bunch of NodeExists exceptions in
> the zookeeper log.
>
> Something else I've found is that I can add records to the table through the
> shell, without problems. It's only from the java client, on a separate box,
> that I'm having problems.
>
> Thanks,
> Adam
>
>
> On 1/13/10 12:08 PM, "Jean-Daniel Cryans" <jd...@apache.org> wrote:
>
>> It means the root region likely wasn't assigned, anything in the
>> master about it?
>>
>> J-D
>>
>> On Wed, Jan 13, 2010 at 11:43 AM, Adam Silberstein
>> <si...@yahoo-inc.com> wrote:
>>> I have version 0.20.2. The client requests hang.
>>> And, I looked in the region server logs...here is the likely culprit:
>>>
>>> java.io.IOException: org.apache.zookeeper.KeeperException$NoNodeException:
>>> KeeperErrorCode = NoNode for /hbase/root-region-server
>>> at
>>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKee
>>> perWrapper.java:332)
>>> at
>>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddress(ZooKeeperWrap
>>> per.java:318)
>>> at
>>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readRootRegionLocation(Zo
>>> oKeeperWrapper.java:231)
>>> at
>>> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:44
>>> 2)
>>> at java.lang.Thread.run(Thread.java:619)
>>> Caused by: org.apache.zookeeper.KeeperException$NoNodeException:
>>> KeeperErrorCode = NoNode for /hbase/root-region-server
>>> at
>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
>>> at
>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
>>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:892)
>>> at
>>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKee
>>> perWrapper.java:328)
>>> ... 4 more
>>> java.io.IOException: org.apache.zookeeper.KeeperException$NoNodeException:
>>> KeeperErrorCode = NoNode for /hbase/root-region-server
>>> at
>>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKee
>>> perWrapper.java:332)
>>> at
>>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddress(ZooKeeperWrap
>>> per.java:318)
>>> at
>>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readRootRegionLocation(Zo
>>> oKeeperWrapper.java:231)
>>> at
>>> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:44
>>> 2)
>>> at java.lang.Thread.run(Thread.java:619)
>>> Caused by: org.apache.zookeeper.KeeperException$NoNodeException:
>>> KeeperErrorCode = NoNode for /hbase/root-region-server
>>> at
>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
>>> at
>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
>>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:892)
>>> at
>>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKee
>>> perWrapper.java:328)
>>> ... 4 more
>>>
>>> Is there zk state on the region servers? Maybe when wiping the system, I
>>> should have wiped something on them.
>>>
>>> Thanks,
>>> Adam
>>>
>>> On 1/13/10 11:33 AM, "Jean-Daniel Cryans" <jd...@apache.org> wrote:
>>>
>>>> Those node exists are usually harmless, I would like the see what's on
>>>> the HBase region server and master side.
>>>>
>>>> You say it cannot do any operation but you don't see exceptions
>>>> client-side? It just hangs? Which version is it?
>>>>
>>>> Thx,
>>>>
>>>> J-D
>>>>
>>>> On Wed, Jan 13, 2010 at 11:25 AM, Adam Silberstein
>>>> <si...@yahoo-inc.com> wrote:
>>>>> Hi,
>>>>> I set up a new Hbase cluster and am seeing an error I haven't seen before.
>>>>> When I start up Hbase I have this NodeExists error in the zookeeper log:
>>>>>
>>>>> org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode =
>>>>> NodeExists
>>>>> at
>>>>>
> org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProces>>>>
> s
>>>>> or.java:245)
>>>>> at
>>>>>
> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.j>>>>
> a
>>>>> va:114)
>>>>> 2010-01-13 11:10:47,503 WARN
>>>>> org.apache.zookeeper.server.PrepRequestProcessor: Got exception when
>>>>> processing sessionid:0x1262918396b0002 type:create cxid:0xc
>>>>> zxid:0xfffffffffffffffe txntype:unknown n/a
>>>>> org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode =
>>>>> NodeExists
>>>>> at
>>>>>
> org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProces>>>>
> s
>>>>> or.java:245)
>>>>> at
>>>>>
> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.j>>>>
> a
>>>>> va:114)
>>>>>
>>>>> I've read that may be harmless. But then when I run the java client, it
>>>>> cannot do any operations. And I see a similar NodeExists error in the
>>>>> zookeeper log.
>>>>>
>>>>>
>>>>> 2010-01-13 11:11:24,661 INFO org.apache.zookeeper.server.NIOServerCnxn:
>>>>> Connected to /203.104.21.210:36928 lastZxid 0
>>>>> 2010-01-13 11:11:24,661 INFO org.apache.zookeeper.server.NIOServerCnxn:
>>>>> Creating new session 0x1262918396b0004
>>>>> 2010-01-13 11:11:24,664 INFO org.apache.zookeeper.server.NIOServerCnxn:
>>>>> Finished init of 0x1262918396b0004 valid:true
>>>>> 2010-01-13 11:11:24,973 WARN
>>>>> org.apache.zookeeper.server.PrepRequestProcessor: Got exception when
>>>>> processing sessionid:0x1262918396b0004 type:create cxid:0x2
>>>>> zxid:0xfffffffffffffffe txntype:unknown n/a
>>>>> org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode =
>>>>> NodeExists
>>>>> at
>>>>>
> org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProces>>>>
> s
>>>>> or.java:245)
>>>>> at
>>>>>
> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.j>>>>
> a
>>>>> va:114)
>>>>> 2010-01-13 11:11:33,088 WARN org.apache.zookeeper.server.NIOServerCnxn:
>>>>> Exception causing close of session 0x1262918396b0004 due to
>>>>> java.io.IOException: Read error
>>>>> 2010-01-13 11:11:33,089 INFO org.apache.zookeeper.server.NIOServerCnxn:
>>>>> closing session:0x1262918396b0004 NIOServerCnxn:
>>>>> java.nio.channels.SocketChannel[connected local=/203.104.21.210:2181
>>>>> remote=/203.104.21.210:36928]
>>>>> 2010-01-13 11:11:42,709 INFO org.apache.zookeeper.server.NIOServerCnxn:
>>>>> Connected to /203.104.21.212:54837 lastZxid 0
>>>>> 2010-01-13 11:11:42,709 INFO org.apache.zookeeper.server.NIOServerCnxn:
>>>>> Creating new session 0x1262918396b0005
>>>>> 2010-01-13 11:11:42,711 INFO org.apache.zookeeper.server.NIOServerCnxn:
>>>>> Finished init of 0x1262918396b0005 valid:true
>>>>> 2010-01-13 11:11:42,711 WARN
>>>>> org.apache.zookeeper.server.PrepRequestProcessor: Got exception when
>>>>> processing sessionid:0x1262918396b0005 type:create cxid:0x1
>>>>> zxid:0xfffffffffffffffe txntype:unknown n/a
>>>>> org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode =
>>>>> NodeExists
>>>>> at
>>>>>
> org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProces>>>>
> s
>>>>> or.java:245)
>>>>> at
>>>>>
> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.j>>>>
> a
>>>>> va:114)
>>>>> 2010-01-13 11:12:06,002 INFO
>>>>> org.apache.zookeeper.server.SessionTrackerImpl:
>>>>> Expiring session 0x1262918396b0004
>>>>> 2010-01-13 11:12:06,002 INFO org.apache.zookeeper.server.ZooKeeperServer:
>>>>> Expiring session 0x1262918396b0004
>>>>> 2010-01-13 11:12:06,002 INFO
>>>>> org.apache.zookeeper.server.PrepRequestProcessor: Processed session
>>>>> termination request for id: 0x1262918396b0004
>>>>>
>>>>> I've tried wiping the whole setup, including the zookeeper files that Hbase
>>>>> puts in /tmp and formatting HDFS. But I get the same problem again. I'm
>>>>> not seeing any exceptions on the client-side.
>>>>>
>>>>> Thanks,
>>>>> Adam
>>>>>
>>>>>
>>>
>>>
>
>
Re: Java client problem w/ zookeeper
Posted by Adam Silberstein <si...@yahoo-inc.com>.
No, nothing at the master, other than a bunch of NodeExists exceptions in
the zookeeper log.
Something else I've found is that I can add records to the table through the
shell, without problems. It's only from the java client, on a separate box,
that I'm having problems.
Thanks,
Adam
On 1/13/10 12:08 PM, "Jean-Daniel Cryans" <jd...@apache.org> wrote:
> It means the root region likely wasn't assigned, anything in the
> master about it?
>
> J-D
>
> On Wed, Jan 13, 2010 at 11:43 AM, Adam Silberstein
> <si...@yahoo-inc.com> wrote:
>> I have version 0.20.2. The client requests hang.
>> And, I looked in the region server logs...here is the likely culprit:
>>
>> java.io.IOException: org.apache.zookeeper.KeeperException$NoNodeException:
>> KeeperErrorCode = NoNode for /hbase/root-region-server
>> at
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKee
>> perWrapper.java:332)
>> at
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddress(ZooKeeperWrap
>> per.java:318)
>> at
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readRootRegionLocation(Zo
>> oKeeperWrapper.java:231)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:44
>> 2)
>> at java.lang.Thread.run(Thread.java:619)
>> Caused by: org.apache.zookeeper.KeeperException$NoNodeException:
>> KeeperErrorCode = NoNode for /hbase/root-region-server
>> at
>> org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
>> at
>> org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:892)
>> at
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKee
>> perWrapper.java:328)
>> ... 4 more
>> java.io.IOException: org.apache.zookeeper.KeeperException$NoNodeException:
>> KeeperErrorCode = NoNode for /hbase/root-region-server
>> at
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKee
>> perWrapper.java:332)
>> at
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddress(ZooKeeperWrap
>> per.java:318)
>> at
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readRootRegionLocation(Zo
>> oKeeperWrapper.java:231)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:44
>> 2)
>> at java.lang.Thread.run(Thread.java:619)
>> Caused by: org.apache.zookeeper.KeeperException$NoNodeException:
>> KeeperErrorCode = NoNode for /hbase/root-region-server
>> at
>> org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
>> at
>> org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:892)
>> at
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKee
>> perWrapper.java:328)
>> ... 4 more
>>
>> Is there zk state on the region servers? Maybe when wiping the system, I
>> should have wiped something on them.
>>
>> Thanks,
>> Adam
>>
>> On 1/13/10 11:33 AM, "Jean-Daniel Cryans" <jd...@apache.org> wrote:
>>
>>> Those node exists are usually harmless, I would like the see what's on
>>> the HBase region server and master side.
>>>
>>> You say it cannot do any operation but you don't see exceptions
>>> client-side? It just hangs? Which version is it?
>>>
>>> Thx,
>>>
>>> J-D
>>>
>>> On Wed, Jan 13, 2010 at 11:25 AM, Adam Silberstein
>>> <si...@yahoo-inc.com> wrote:
>>>> Hi,
>>>> I set up a new Hbase cluster and am seeing an error I haven't seen before.
>>>> When I start up Hbase I have this NodeExists error in the zookeeper log:
>>>>
>>>> org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode =
>>>> NodeExists
>>>> at
>>>>
org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProces>>>>
s
>>>> or.java:245)
>>>> at
>>>>
org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.j>>>>
a
>>>> va:114)
>>>> 2010-01-13 11:10:47,503 WARN
>>>> org.apache.zookeeper.server.PrepRequestProcessor: Got exception when
>>>> processing sessionid:0x1262918396b0002 type:create cxid:0xc
>>>> zxid:0xfffffffffffffffe txntype:unknown n/a
>>>> org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode =
>>>> NodeExists
>>>> at
>>>>
org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProces>>>>
s
>>>> or.java:245)
>>>> at
>>>>
org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.j>>>>
a
>>>> va:114)
>>>>
>>>> I've read that may be harmless. But then when I run the java client, it
>>>> cannot do any operations. And I see a similar NodeExists error in the
>>>> zookeeper log.
>>>>
>>>>
>>>> 2010-01-13 11:11:24,661 INFO org.apache.zookeeper.server.NIOServerCnxn:
>>>> Connected to /203.104.21.210:36928 lastZxid 0
>>>> 2010-01-13 11:11:24,661 INFO org.apache.zookeeper.server.NIOServerCnxn:
>>>> Creating new session 0x1262918396b0004
>>>> 2010-01-13 11:11:24,664 INFO org.apache.zookeeper.server.NIOServerCnxn:
>>>> Finished init of 0x1262918396b0004 valid:true
>>>> 2010-01-13 11:11:24,973 WARN
>>>> org.apache.zookeeper.server.PrepRequestProcessor: Got exception when
>>>> processing sessionid:0x1262918396b0004 type:create cxid:0x2
>>>> zxid:0xfffffffffffffffe txntype:unknown n/a
>>>> org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode =
>>>> NodeExists
>>>> at
>>>>
org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProces>>>>
s
>>>> or.java:245)
>>>> at
>>>>
org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.j>>>>
a
>>>> va:114)
>>>> 2010-01-13 11:11:33,088 WARN org.apache.zookeeper.server.NIOServerCnxn:
>>>> Exception causing close of session 0x1262918396b0004 due to
>>>> java.io.IOException: Read error
>>>> 2010-01-13 11:11:33,089 INFO org.apache.zookeeper.server.NIOServerCnxn:
>>>> closing session:0x1262918396b0004 NIOServerCnxn:
>>>> java.nio.channels.SocketChannel[connected local=/203.104.21.210:2181
>>>> remote=/203.104.21.210:36928]
>>>> 2010-01-13 11:11:42,709 INFO org.apache.zookeeper.server.NIOServerCnxn:
>>>> Connected to /203.104.21.212:54837 lastZxid 0
>>>> 2010-01-13 11:11:42,709 INFO org.apache.zookeeper.server.NIOServerCnxn:
>>>> Creating new session 0x1262918396b0005
>>>> 2010-01-13 11:11:42,711 INFO org.apache.zookeeper.server.NIOServerCnxn:
>>>> Finished init of 0x1262918396b0005 valid:true
>>>> 2010-01-13 11:11:42,711 WARN
>>>> org.apache.zookeeper.server.PrepRequestProcessor: Got exception when
>>>> processing sessionid:0x1262918396b0005 type:create cxid:0x1
>>>> zxid:0xfffffffffffffffe txntype:unknown n/a
>>>> org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode =
>>>> NodeExists
>>>> at
>>>>
org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProces>>>>
s
>>>> or.java:245)
>>>> at
>>>>
org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.j>>>>
a
>>>> va:114)
>>>> 2010-01-13 11:12:06,002 INFO
>>>> org.apache.zookeeper.server.SessionTrackerImpl:
>>>> Expiring session 0x1262918396b0004
>>>> 2010-01-13 11:12:06,002 INFO org.apache.zookeeper.server.ZooKeeperServer:
>>>> Expiring session 0x1262918396b0004
>>>> 2010-01-13 11:12:06,002 INFO
>>>> org.apache.zookeeper.server.PrepRequestProcessor: Processed session
>>>> termination request for id: 0x1262918396b0004
>>>>
>>>> I've tried wiping the whole setup, including the zookeeper files that Hbase
>>>> puts in /tmp and formatting HDFS. But I get the same problem again. I'm
>>>> not seeing any exceptions on the client-side.
>>>>
>>>> Thanks,
>>>> Adam
>>>>
>>>>
>>
>>
Re: Java client problem w/ zookeeper
Posted by Jean-Daniel Cryans <jd...@apache.org>.
It means the root region likely wasn't assigned, anything in the
master about it?
J-D
On Wed, Jan 13, 2010 at 11:43 AM, Adam Silberstein
<si...@yahoo-inc.com> wrote:
> I have version 0.20.2. The client requests hang.
> And, I looked in the region server logs...here is the likely culprit:
>
> java.io.IOException: org.apache.zookeeper.KeeperException$NoNodeException:
> KeeperErrorCode = NoNode for /hbase/root-region-server
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKee
> perWrapper.java:332)
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddress(ZooKeeperWrap
> per.java:318)
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readRootRegionLocation(Zo
> oKeeperWrapper.java:231)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:44
> 2)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: org.apache.zookeeper.KeeperException$NoNodeException:
> KeeperErrorCode = NoNode for /hbase/root-region-server
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:892)
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKee
> perWrapper.java:328)
> ... 4 more
> java.io.IOException: org.apache.zookeeper.KeeperException$NoNodeException:
> KeeperErrorCode = NoNode for /hbase/root-region-server
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKee
> perWrapper.java:332)
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddress(ZooKeeperWrap
> per.java:318)
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readRootRegionLocation(Zo
> oKeeperWrapper.java:231)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:44
> 2)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: org.apache.zookeeper.KeeperException$NoNodeException:
> KeeperErrorCode = NoNode for /hbase/root-region-server
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:892)
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKee
> perWrapper.java:328)
> ... 4 more
>
> Is there zk state on the region servers? Maybe when wiping the system, I
> should have wiped something on them.
>
> Thanks,
> Adam
>
> On 1/13/10 11:33 AM, "Jean-Daniel Cryans" <jd...@apache.org> wrote:
>
>> Those node exists are usually harmless, I would like the see what's on
>> the HBase region server and master side.
>>
>> You say it cannot do any operation but you don't see exceptions
>> client-side? It just hangs? Which version is it?
>>
>> Thx,
>>
>> J-D
>>
>> On Wed, Jan 13, 2010 at 11:25 AM, Adam Silberstein
>> <si...@yahoo-inc.com> wrote:
>>> Hi,
>>> I set up a new Hbase cluster and am seeing an error I haven't seen before.
>>> When I start up Hbase I have this NodeExists error in the zookeeper log:
>>>
>>> org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode =
>>> NodeExists
>>> at
>>> org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProcess
>>> or.java:245)
>>> at
>>> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.ja
>>> va:114)
>>> 2010-01-13 11:10:47,503 WARN
>>> org.apache.zookeeper.server.PrepRequestProcessor: Got exception when
>>> processing sessionid:0x1262918396b0002 type:create cxid:0xc
>>> zxid:0xfffffffffffffffe txntype:unknown n/a
>>> org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode =
>>> NodeExists
>>> at
>>> org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProcess
>>> or.java:245)
>>> at
>>> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.ja
>>> va:114)
>>>
>>> I've read that may be harmless. But then when I run the java client, it
>>> cannot do any operations. And I see a similar NodeExists error in the
>>> zookeeper log.
>>>
>>>
>>> 2010-01-13 11:11:24,661 INFO org.apache.zookeeper.server.NIOServerCnxn:
>>> Connected to /203.104.21.210:36928 lastZxid 0
>>> 2010-01-13 11:11:24,661 INFO org.apache.zookeeper.server.NIOServerCnxn:
>>> Creating new session 0x1262918396b0004
>>> 2010-01-13 11:11:24,664 INFO org.apache.zookeeper.server.NIOServerCnxn:
>>> Finished init of 0x1262918396b0004 valid:true
>>> 2010-01-13 11:11:24,973 WARN
>>> org.apache.zookeeper.server.PrepRequestProcessor: Got exception when
>>> processing sessionid:0x1262918396b0004 type:create cxid:0x2
>>> zxid:0xfffffffffffffffe txntype:unknown n/a
>>> org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode =
>>> NodeExists
>>> at
>>> org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProcess
>>> or.java:245)
>>> at
>>> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.ja
>>> va:114)
>>> 2010-01-13 11:11:33,088 WARN org.apache.zookeeper.server.NIOServerCnxn:
>>> Exception causing close of session 0x1262918396b0004 due to
>>> java.io.IOException: Read error
>>> 2010-01-13 11:11:33,089 INFO org.apache.zookeeper.server.NIOServerCnxn:
>>> closing session:0x1262918396b0004 NIOServerCnxn:
>>> java.nio.channels.SocketChannel[connected local=/203.104.21.210:2181
>>> remote=/203.104.21.210:36928]
>>> 2010-01-13 11:11:42,709 INFO org.apache.zookeeper.server.NIOServerCnxn:
>>> Connected to /203.104.21.212:54837 lastZxid 0
>>> 2010-01-13 11:11:42,709 INFO org.apache.zookeeper.server.NIOServerCnxn:
>>> Creating new session 0x1262918396b0005
>>> 2010-01-13 11:11:42,711 INFO org.apache.zookeeper.server.NIOServerCnxn:
>>> Finished init of 0x1262918396b0005 valid:true
>>> 2010-01-13 11:11:42,711 WARN
>>> org.apache.zookeeper.server.PrepRequestProcessor: Got exception when
>>> processing sessionid:0x1262918396b0005 type:create cxid:0x1
>>> zxid:0xfffffffffffffffe txntype:unknown n/a
>>> org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode =
>>> NodeExists
>>> at
>>> org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProcess
>>> or.java:245)
>>> at
>>> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.ja
>>> va:114)
>>> 2010-01-13 11:12:06,002 INFO org.apache.zookeeper.server.SessionTrackerImpl:
>>> Expiring session 0x1262918396b0004
>>> 2010-01-13 11:12:06,002 INFO org.apache.zookeeper.server.ZooKeeperServer:
>>> Expiring session 0x1262918396b0004
>>> 2010-01-13 11:12:06,002 INFO
>>> org.apache.zookeeper.server.PrepRequestProcessor: Processed session
>>> termination request for id: 0x1262918396b0004
>>>
>>> I've tried wiping the whole setup, including the zookeeper files that Hbase
>>> puts in /tmp and formatting HDFS. But I get the same problem again. I'm
>>> not seeing any exceptions on the client-side.
>>>
>>> Thanks,
>>> Adam
>>>
>>>
>
>
Re: Java client problem w/ zookeeper
Posted by Adam Silberstein <si...@yahoo-inc.com>.
I have version 0.20.2. The client requests hang.
And, I looked in the region server logs...here is the likely culprit:
java.io.IOException: org.apache.zookeeper.KeeperException$NoNodeException:
KeeperErrorCode = NoNode for /hbase/root-region-server
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKee
perWrapper.java:332)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddress(ZooKeeperWrap
per.java:318)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readRootRegionLocation(Zo
oKeeperWrapper.java:231)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:44
2)
at java.lang.Thread.run(Thread.java:619)
Caused by: org.apache.zookeeper.KeeperException$NoNodeException:
KeeperErrorCode = NoNode for /hbase/root-region-server
at
org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
at
org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:892)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKee
perWrapper.java:328)
... 4 more
java.io.IOException: org.apache.zookeeper.KeeperException$NoNodeException:
KeeperErrorCode = NoNode for /hbase/root-region-server
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKee
perWrapper.java:332)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddress(ZooKeeperWrap
per.java:318)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readRootRegionLocation(Zo
oKeeperWrapper.java:231)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:44
2)
at java.lang.Thread.run(Thread.java:619)
Caused by: org.apache.zookeeper.KeeperException$NoNodeException:
KeeperErrorCode = NoNode for /hbase/root-region-server
at
org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
at
org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:892)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKee
perWrapper.java:328)
... 4 more
Is there zk state on the region servers? Maybe when wiping the system, I
should have wiped something on them.
Thanks,
Adam
On 1/13/10 11:33 AM, "Jean-Daniel Cryans" <jd...@apache.org> wrote:
> Those node exists are usually harmless, I would like the see what's on
> the HBase region server and master side.
>
> You say it cannot do any operation but you don't see exceptions
> client-side? It just hangs? Which version is it?
>
> Thx,
>
> J-D
>
> On Wed, Jan 13, 2010 at 11:25 AM, Adam Silberstein
> <si...@yahoo-inc.com> wrote:
>> Hi,
>> I set up a new Hbase cluster and am seeing an error I haven't seen before.
>> When I start up Hbase I have this NodeExists error in the zookeeper log:
>>
>> org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode =
>> NodeExists
>> at
>> org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProcess
>> or.java:245)
>> at
>> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.ja
>> va:114)
>> 2010-01-13 11:10:47,503 WARN
>> org.apache.zookeeper.server.PrepRequestProcessor: Got exception when
>> processing sessionid:0x1262918396b0002 type:create cxid:0xc
>> zxid:0xfffffffffffffffe txntype:unknown n/a
>> org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode =
>> NodeExists
>> at
>> org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProcess
>> or.java:245)
>> at
>> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.ja
>> va:114)
>>
>> I've read that may be harmless. But then when I run the java client, it
>> cannot do any operations. And I see a similar NodeExists error in the
>> zookeeper log.
>>
>>
>> 2010-01-13 11:11:24,661 INFO org.apache.zookeeper.server.NIOServerCnxn:
>> Connected to /203.104.21.210:36928 lastZxid 0
>> 2010-01-13 11:11:24,661 INFO org.apache.zookeeper.server.NIOServerCnxn:
>> Creating new session 0x1262918396b0004
>> 2010-01-13 11:11:24,664 INFO org.apache.zookeeper.server.NIOServerCnxn:
>> Finished init of 0x1262918396b0004 valid:true
>> 2010-01-13 11:11:24,973 WARN
>> org.apache.zookeeper.server.PrepRequestProcessor: Got exception when
>> processing sessionid:0x1262918396b0004 type:create cxid:0x2
>> zxid:0xfffffffffffffffe txntype:unknown n/a
>> org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode =
>> NodeExists
>> at
>> org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProcess
>> or.java:245)
>> at
>> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.ja
>> va:114)
>> 2010-01-13 11:11:33,088 WARN org.apache.zookeeper.server.NIOServerCnxn:
>> Exception causing close of session 0x1262918396b0004 due to
>> java.io.IOException: Read error
>> 2010-01-13 11:11:33,089 INFO org.apache.zookeeper.server.NIOServerCnxn:
>> closing session:0x1262918396b0004 NIOServerCnxn:
>> java.nio.channels.SocketChannel[connected local=/203.104.21.210:2181
>> remote=/203.104.21.210:36928]
>> 2010-01-13 11:11:42,709 INFO org.apache.zookeeper.server.NIOServerCnxn:
>> Connected to /203.104.21.212:54837 lastZxid 0
>> 2010-01-13 11:11:42,709 INFO org.apache.zookeeper.server.NIOServerCnxn:
>> Creating new session 0x1262918396b0005
>> 2010-01-13 11:11:42,711 INFO org.apache.zookeeper.server.NIOServerCnxn:
>> Finished init of 0x1262918396b0005 valid:true
>> 2010-01-13 11:11:42,711 WARN
>> org.apache.zookeeper.server.PrepRequestProcessor: Got exception when
>> processing sessionid:0x1262918396b0005 type:create cxid:0x1
>> zxid:0xfffffffffffffffe txntype:unknown n/a
>> org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode =
>> NodeExists
>> at
>> org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProcess
>> or.java:245)
>> at
>> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.ja
>> va:114)
>> 2010-01-13 11:12:06,002 INFO org.apache.zookeeper.server.SessionTrackerImpl:
>> Expiring session 0x1262918396b0004
>> 2010-01-13 11:12:06,002 INFO org.apache.zookeeper.server.ZooKeeperServer:
>> Expiring session 0x1262918396b0004
>> 2010-01-13 11:12:06,002 INFO
>> org.apache.zookeeper.server.PrepRequestProcessor: Processed session
>> termination request for id: 0x1262918396b0004
>>
>> I've tried wiping the whole setup, including the zookeeper files that Hbase
>> puts in /tmp and formatting HDFS. But I get the same problem again. I'm
>> not seeing any exceptions on the client-side.
>>
>> Thanks,
>> Adam
>>
>>
Re: Java client problem w/ zookeeper
Posted by Jean-Daniel Cryans <jd...@apache.org>.
Those node exists are usually harmless, I would like the see what's on
the HBase region server and master side.
You say it cannot do any operation but you don't see exceptions
client-side? It just hangs? Which version is it?
Thx,
J-D
On Wed, Jan 13, 2010 at 11:25 AM, Adam Silberstein
<si...@yahoo-inc.com> wrote:
> Hi,
> I set up a new Hbase cluster and am seeing an error I haven't seen before.
> When I start up Hbase I have this NodeExists error in the zookeeper log:
>
> org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode =
> NodeExists
> at
> org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProcess
> or.java:245)
> at
> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.ja
> va:114)
> 2010-01-13 11:10:47,503 WARN
> org.apache.zookeeper.server.PrepRequestProcessor: Got exception when
> processing sessionid:0x1262918396b0002 type:create cxid:0xc
> zxid:0xfffffffffffffffe txntype:unknown n/a
> org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode =
> NodeExists
> at
> org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProcess
> or.java:245)
> at
> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.ja
> va:114)
>
> I've read that may be harmless. But then when I run the java client, it
> cannot do any operations. And I see a similar NodeExists error in the
> zookeeper log.
>
>
> 2010-01-13 11:11:24,661 INFO org.apache.zookeeper.server.NIOServerCnxn:
> Connected to /203.104.21.210:36928 lastZxid 0
> 2010-01-13 11:11:24,661 INFO org.apache.zookeeper.server.NIOServerCnxn:
> Creating new session 0x1262918396b0004
> 2010-01-13 11:11:24,664 INFO org.apache.zookeeper.server.NIOServerCnxn:
> Finished init of 0x1262918396b0004 valid:true
> 2010-01-13 11:11:24,973 WARN
> org.apache.zookeeper.server.PrepRequestProcessor: Got exception when
> processing sessionid:0x1262918396b0004 type:create cxid:0x2
> zxid:0xfffffffffffffffe txntype:unknown n/a
> org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode =
> NodeExists
> at
> org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProcess
> or.java:245)
> at
> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.ja
> va:114)
> 2010-01-13 11:11:33,088 WARN org.apache.zookeeper.server.NIOServerCnxn:
> Exception causing close of session 0x1262918396b0004 due to
> java.io.IOException: Read error
> 2010-01-13 11:11:33,089 INFO org.apache.zookeeper.server.NIOServerCnxn:
> closing session:0x1262918396b0004 NIOServerCnxn:
> java.nio.channels.SocketChannel[connected local=/203.104.21.210:2181
> remote=/203.104.21.210:36928]
> 2010-01-13 11:11:42,709 INFO org.apache.zookeeper.server.NIOServerCnxn:
> Connected to /203.104.21.212:54837 lastZxid 0
> 2010-01-13 11:11:42,709 INFO org.apache.zookeeper.server.NIOServerCnxn:
> Creating new session 0x1262918396b0005
> 2010-01-13 11:11:42,711 INFO org.apache.zookeeper.server.NIOServerCnxn:
> Finished init of 0x1262918396b0005 valid:true
> 2010-01-13 11:11:42,711 WARN
> org.apache.zookeeper.server.PrepRequestProcessor: Got exception when
> processing sessionid:0x1262918396b0005 type:create cxid:0x1
> zxid:0xfffffffffffffffe txntype:unknown n/a
> org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode =
> NodeExists
> at
> org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProcess
> or.java:245)
> at
> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.ja
> va:114)
> 2010-01-13 11:12:06,002 INFO org.apache.zookeeper.server.SessionTrackerImpl:
> Expiring session 0x1262918396b0004
> 2010-01-13 11:12:06,002 INFO org.apache.zookeeper.server.ZooKeeperServer:
> Expiring session 0x1262918396b0004
> 2010-01-13 11:12:06,002 INFO
> org.apache.zookeeper.server.PrepRequestProcessor: Processed session
> termination request for id: 0x1262918396b0004
>
> I've tried wiping the whole setup, including the zookeeper files that Hbase
> puts in /tmp and formatting HDFS. But I get the same problem again. I'm
> not seeing any exceptions on the client-side.
>
> Thanks,
> Adam
>
>
Re: Java client problem w/ zookeeper
Posted by stack <st...@duboce.net>.
On Fri, Jan 15, 2010 at 10:18 PM, Adam Silberstein
<si...@yahoo-inc.com>wrote:
> I tried turning up logging in the regionservers from INFO to
> DEBUG, but on restart, I see no extra debug messages. To make the change,
> I
> just changed hbase.root.logger in hbase/conf/log4j.properties, but no
> effect. Am I missing something?
>
Here is how to enable DEBUG: http://wiki.apache.org/hadoop/Hbase/FAQ#A5
>
> Let me know if there are other threads I should pay attention to.
Adam: May I have the thread dumps and logs from the regionservers at DEBUG
level? I'm a little stumped. It seems as though one or more of your
servers has locked up. It could be a deadlock or it could be something like
a write barrier coming down and not ever being raised (You'll see messages
in the regionserver log on the barrier). If you post logs somewhere I can
pull, I'll check 'em out.
St.Ack
Re: Java client problem w/ zookeeper
Posted by Patrick Hunt <ph...@apache.org>.
fyi, zk's maxclientcnxns controls not the total number of connections to
a zk server, but rather the total number of connections from a single
host to a zk server:
http://hadoop.apache.org/zookeeper/docs/current/zookeeperAdmin.html#sc_advancedConfiguration
"Limits the number of concurrent connections (at the socket level) that
a single client, identified by IP address, may make to a single member
of the ZooKeeper ensemble. This is used to prevent certain classes of
DoS attacks, including file descriptor exhaustion. Setting this to 0
removes the limit on concurrent connections."
Patrick
Andrew Purtell wrote:
> Have you done something like:
>
> <property>
> <name>hbase.regionserver.handler.count</name>
> <value>100</value>
> </property>
> <property>
> <name>hbase.zookeeper.property.maxClientCnxns</name>
> <value>100</value>
> </property>
>
> ?
>
> - Andy
>
> On Wed, Jan 13, 2010 at 10:48 PM, Adam Silberstein wrote:
>
> [...]
>> And then, 5 seconds into it, I start seeing these:
>>
>> 10/01/13 22:33:27 DEBUGclient.HConnectionManager$TableServers: Reloading
>> region usertable,user263042644,1263431524192 location because regionserver
>> didn't accept updates; tries=0 of max=10, waiting=2000ms
>>
>> But the region servers are all up, and none logged any messages during the
>> client run.
>
>
>
>
Re: Java client problem w/ zookeeper
Posted by Andrew Purtell <ap...@apache.org>.
Have you done something like:
<property>
<name>hbase.regionserver.handler.count</name>
<value>100</value>
</property>
<property>
<name>hbase.zookeeper.property.maxClientCnxns</name>
<value>100</value>
</property>
?
- Andy
On Wed, Jan 13, 2010 at 10:48 PM, Adam Silberstein wrote:
[...]
> And then, 5 seconds into it, I start seeing these:
>
> 10/01/13 22:33:27 DEBUGclient.HConnectionManager$TableServers: Reloading
> region usertable,user263042644,1263431524192 location because regionserver
> didn't accept updates; tries=0 of max=10, waiting=2000ms
>
> But the region servers are all up, and none logged any messages during the
> client run.
Re: Java client problem w/ zookeeper
Posted by Adam Silberstein <si...@yahoo-inc.com>.
I don't see the RetriesExhaustedException at the client. But the pattern
that I see is that each client thread makes a certain number of requests
(none of those requests actually succeed), before one of its requests gets
blocked and hangs. That number is the same every time. But, nothing
particularly useful in the client log. I also see nothing in the region
server log. I tried turning up logging in the regionservers from INFO to
DEBUG, but on restart, I see no extra debug messages. To make the change, I
just changed hbase.root.logger in hbase/conf/log4j.properties, but no
effect. Am I missing something?
This problem happens even after an Hbase restart.
I've tried different number of client threads. For small numbers (1-5 e.g.)
I get stuck. I tried some very high numbers like 200, and that always makes
progress, although it slows way down. So it seems that many client threads
are getting stuck.
I haven't been able to figure out which region servers have the problem. I
only have 3, though, so I did a thread dump on each. I looked at the
handlers and they all look like this:
"IPC Server handler 94 on 60020" daemon prio=10 tid=0x08550c00 nid=0x6014
waiting on condition [0x6b1cd000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x78382718> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
AbstractQueuedSynchronizer.java:1925)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901)
Let me know if there are other threads I should pay attention to.
Finally, as for version, I have the official 0.20.2 release, which I
downloaded perhaps 2 months ago.
Thanks,
Adam
On 1/14/10 10:51 PM, "stack" <st...@duboce.net> wrote:
> On Wed, Jan 13, 2010 at 10:48 PM, Adam Silberstein
> <si...@yahoo-inc.com>wrote:
>
>> ....
>> And then, 5 seconds into it, I start seeing these:
>>
>> 10/01/13 22:33:27 DEBUGclient.HConnectionManager$TableServers: Reloading
>> region usertable,user263042644,1263431524192 location because regionserver
>> didn't accept updates; tries=0 of max=10, waiting=2000ms
>>
>> But the region servers are all up, and none logged any messages during the
>> client run.
>>
>
> This sounds like a lockup Adam or a close out.. like the client can't get
> into the server because all its handlers are occupied. Does it got
> to RetriesExhaustedException? If so, whats the log message. Would be
> interesting to see if any of the edits in the batch are making it on to the
> server. Can you figure which server its going to? Thread dump it? Your
> hbase is up to date? There was a deadlock in hbase there for about a day
> about a week or so back. Maybe you picked it up?
>
> St.Ack
Re: Java client problem w/ zookeeper
Posted by stack <st...@duboce.net>.
On Wed, Jan 13, 2010 at 10:48 PM, Adam Silberstein
<si...@yahoo-inc.com>wrote:
> ....
> And then, 5 seconds into it, I start seeing these:
>
> 10/01/13 22:33:27 DEBUGclient.HConnectionManager$TableServers: Reloading
> region usertable,user263042644,1263431524192 location because regionserver
> didn't accept updates; tries=0 of max=10, waiting=2000ms
>
> But the region servers are all up, and none logged any messages during the
> client run.
>
This sounds like a lockup Adam or a close out.. like the client can't get
into the server because all its handlers are occupied. Does it got
to RetriesExhaustedException? If so, whats the log message. Would be
interesting to see if any of the edits in the batch are making it on to the
server. Can you figure which server its going to? Thread dump it? Your
hbase is up to date? There was a deadlock in hbase there for about a day
about a week or so back. Maybe you picked it up?
St.Ack
Re: Java client problem w/ zookeeper
Posted by Adam Silberstein <si...@yahoo-inc.com>.
OK, got the log4j stuff squared away. I don't see any ZK errors after the
start, so perhaps that's not it. Here's a sample of the Hbase Client
messages I'm seeing:
0/01/13 22:33:23 DEBUGipc.HbaseRPC: Call: getClosestRowBefore 11
10/01/13 22:33:23 DEBUGclient.HConnectionManager$TableServers: Cached
location address: 66.196.83.87:60020, regioninfo: REGION => {NAME =>
'usertable,user1029086433,1263433456604', STARTKEY => 'user1029086433',
ENDKEY => 'user1036327488', ENCODED => 266348591, TABLE => {{NAME =>
'usertable', FAMILIES => [{NAME => 'values', VERSIONS => '3', COMPRESSION =>
'NONE', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false',
BLOCKCACHE => 'true'}]}}
10/01/13 22:33:23 DEBUGipc.HBaseClient: IPC Client (47) connection to
/66.196.83.87:60020 from an unknown user sending #13
10/01/13 22:33:23 DEBUGipc.HBaseClient: IPC Client (47) connection to
/66.196.83.87:60020 from an unknown user got value #13
10/01/13 22:33:23 DEBUGipc.HbaseRPC: Call: put 9
10/01/13 22:33:23 DEBUGipc.HBaseClient: IPC Client (47) connection to
/66.196.83.85:60020 from an unknown user sending #14
10/01/13 22:33:23 DEBUGipc.HBaseClient: IPC Client (47) connection to
/66.196.83.85:60020 from an unknown user got value #14
And then, 5 seconds into it, I start seeing these:
10/01/13 22:33:27 DEBUGclient.HConnectionManager$TableServers: Reloading
region usertable,user263042644,1263431524192 location because regionserver
didn't accept updates; tries=0 of max=10, waiting=2000ms
But the region servers are all up, and none logged any messages during the
client run.
Any thoughts?
Thanks,
Adam
On 1/13/10 9:40 PM, "stack" <st...@duboce.net> wrote:
> On Wed, Jan 13, 2010 at 7:40 PM, Adam Silberstein
> <si...@yahoo-inc.com>wrote:
>
>> Hi stack,
>> Having some trouble trying this. I created a log4j.properties file and
>> tried different combinations of these, with the goal of seeing if I could
>> first adjust the log level at all from current INFO:
>> log4j.logger.org.apache.zookeeper=ERROR,stdout
>> zookeeper.root.logger=ERROR, stdout
>> log4j.rootLogger=ERROR, stdout
>>
>>
> Are you thinking that ERROR is finer-grained than DEBUG by chance? If so,
> http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/Level.html.
> ERROR doesn't show DEBUG.
>
> It looks like the shell is overriding the log4j setting if thats of any
> help. If I disable that facility, it seems the shell does DEBUG level zk
> logging.
>
> Index: bin/hirb.rb
> ===================================================================
> --- bin/hirb.rb (revision 899058)
> +++ bin/hirb.rb (working copy)
> @@ -36,7 +36,7 @@
> format = 'console'
> format_width = 110
> script2run = nil
> -logLevel = org.apache.log4j.Level::ERROR
> +#logLevel = org.apache.log4j.Level::ERROR
> for arg in ARGV
> if arg =~ /^--format=(.+)/i
> format = $1
> @@ -78,7 +78,7 @@
>
> # Set logging level to avoid verboseness
> logger = org.apache.log4j.Logger.getLogger("org.apache.zookeeper")
> -logger.setLevel(logLevel);
> +#logger.setLevel(logLevel);
> logger = org.apache.log4j.Logger.getLogger("org.apache.hadoop.hbase")
> logger.setLevel(logLevel);
> # Require HBase now after setting log levels
>
>
>
>> But no effect. Probably a dumb log4j error, but it's blocking me.
>>
>>
> If in log4j, I set zk to debug level:
>
> pynchon:0.20 stack$ svn diff conf/
> Index: conf/log4j.properties
> ===================================================================
> --- conf/log4j.properties (revision 898656)
> +++ conf/log4j.properties (working copy)
> @@ -40,7 +40,7 @@
>
> # Custom Logging levels
>
> -log4j.logger.org.apache.zookeeper=INFO
> +log4j.logger.org.apache.zookeeper=DEBUG
> #log4j.logger.org.apache.hadoop.fs.FSNamesystem=DEBUG
> log4j.logger.org.apache.hadoop.hbase=DEBUG
> #log4j.logger.org.apache.hadoop.dfs=DEBUG
>
> ... and run for example the PerformanceEvaluation client:
>
> $ ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation sequentialWrite
> 1
>
> ... it seems to report zk at DEBUG level.
>
>
>
>> Also, I moved to yet another Hbase deployment. This time I was able to
>> insert about 20 million records from my client. Then I encountered the
>> same
>> problem. I can insert with the shell, but not from my client.
>
>
> For sure, your client is finding the same conf and the hbase shell? I
> wonder why it was working, then stopped. If you look in the zk cluster
> member logs, anything happen at the 20M mark?
>
>
>> The only
>> Hbase-side thing I see are the NodeExistsException: KeeperErrorCode =
>> NodeExists in the zookeeper log, and there is one for each of my threads.
>>
>> Also, I tried restarting Hbase/Hadoop, making sure the everything was idle
>> before I started again, but no luck.
>>
>> And if you restart the zk cluster?
>
> St.Ack
>
>
>> Thanks,
>> Adam
>>
>>
>>
>> On 1/13/10 2:54 PM, "stack" <st...@duboce.net> wrote:
>>
>>> I wonder what the 'node' is that its complaining about? It doesn't say.
>> I
>>> wonder if you did DEBUG for zk if you'd learn more? Could enable it on
>> the
>>> problematic client only?
>>>
>>>
>>> St.Ack
>>>
>>> On Wed, Jan 13, 2010 at 11:25 AM, Adam Silberstein
>>> <si...@yahoo-inc.com>wrote:
>>>
>>>> Hi,
>>>> I set up a new Hbase cluster and am seeing an error I haven't seen
>> before.
>>>> When I start up Hbase I have this NodeExists error in the zookeeper log:
>>>>
>>>> org.apache.zookeeper.KeeperException$NodeExistsException:
>> KeeperErrorCode =
>>>> NodeExists
>>>> at
>>>>
>>>>
>> org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProcess
>>>> or.java:245)
>>>> at
>>>>
>>>>
>> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.ja
>>>> va:114)
>>>> 2010-01-13 11:10:47,503 WARN
>>>> org.apache.zookeeper.server.PrepRequestProcessor: Got exception when
>>>> processing sessionid:0x1262918396b0002 type:create cxid:0xc
>>>> zxid:0xfffffffffffffffe txntype:unknown n/a
>>>> org.apache.zookeeper.KeeperException$NodeExistsException:
>> KeeperErrorCode =
>>>> NodeExists
>>>> at
>>>>
>>>>
>> org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProcess
>>>> or.java:245)
>>>> at
>>>>
>>>>
>> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.ja
>>>> va:114)
>>>>
>>>> I've read that may be harmless. But then when I run the java client, it
>>>> cannot do any operations. And I see a similar NodeExists error in the
>>>> zookeeper log.
>>>>
>>>>
>>>> 2010-01-13 11:11:24,661 INFO org.apache.zookeeper.server.NIOServerCnxn:
>>>> Connected to /203.104.21.210:36928 lastZxid 0
>>>> 2010-01-13 11:11:24,661 INFO org.apache.zookeeper.server.NIOServerCnxn:
>>>> Creating new session 0x1262918396b0004
>>>> 2010-01-13 11:11:24,664 INFO org.apache.zookeeper.server.NIOServerCnxn:
>>>> Finished init of 0x1262918396b0004 valid:true
>>>> 2010-01-13 11:11:24,973 WARN
>>>> org.apache.zookeeper.server.PrepRequestProcessor: Got exception when
>>>> processing sessionid:0x1262918396b0004 type:create cxid:0x2
>>>> zxid:0xfffffffffffffffe txntype:unknown n/a
>>>> org.apache.zookeeper.KeeperException$NodeExistsException:
>> KeeperErrorCode =
>>>> NodeExists
>>>> at
>>>>
>>>>
>> org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProcess
>>>> or.java:245)
>>>> at
>>>>
>>>>
>> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.ja
>>>> va:114)
>>>> 2010-01-13 11:11:33,088 WARN org.apache.zookeeper.server.NIOServerCnxn:
>>>> Exception causing close of session 0x1262918396b0004 due to
>>>> java.io.IOException: Read error
>>>> 2010-01-13 11:11:33,089 INFO org.apache.zookeeper.server.NIOServerCnxn:
>>>> closing session:0x1262918396b0004 NIOServerCnxn:
>>>> java.nio.channels.SocketChannel[connected local=/203.104.21.210:2181
>>>> remote=/203.104.21.210:36928]
>>>> 2010-01-13 11:11:42,709 INFO org.apache.zookeeper.server.NIOServerCnxn:
>>>> Connected to /203.104.21.212:54837 lastZxid 0
>>>> 2010-01-13 11:11:42,709 INFO org.apache.zookeeper.server.NIOServerCnxn:
>>>> Creating new session 0x1262918396b0005
>>>> 2010-01-13 11:11:42,711 INFO org.apache.zookeeper.server.NIOServerCnxn:
>>>> Finished init of 0x1262918396b0005 valid:true
>>>> 2010-01-13 11:11:42,711 WARN
>>>> org.apache.zookeeper.server.PrepRequestProcessor: Got exception when
>>>> processing sessionid:0x1262918396b0005 type:create cxid:0x1
>>>> zxid:0xfffffffffffffffe txntype:unknown n/a
>>>> org.apache.zookeeper.KeeperException$NodeExistsException:
>> KeeperErrorCode =
>>>> NodeExists
>>>> at
>>>>
>>>>
>> org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProcess
>>>> or.java:245)
>>>> at
>>>>
>>>>
>> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.ja
>>>> va:114)
>>>> 2010-01-13 11:12:06,002 INFO
>>>> org.apache.zookeeper.server.SessionTrackerImpl:
>>>> Expiring session 0x1262918396b0004
>>>> 2010-01-13 11:12:06,002 INFO
>> org.apache.zookeeper.server.ZooKeeperServer:
>>>> Expiring session 0x1262918396b0004
>>>> 2010-01-13 11:12:06,002 INFO
>>>> org.apache.zookeeper.server.PrepRequestProcessor: Processed session
>>>> termination request for id: 0x1262918396b0004
>>>>
>>>> I've tried wiping the whole setup, including the zookeeper files that
>> Hbase
>>>> puts in /tmp and formatting HDFS. But I get the same problem again.
>> I'm
>>>> not seeing any exceptions on the client-side.
>>>>
>>>> Thanks,
>>>> Adam
>>>>
>>>>
>>
>>
Re: Java client problem w/ zookeeper
Posted by stack <st...@duboce.net>.
On Wed, Jan 13, 2010 at 7:40 PM, Adam Silberstein <si...@yahoo-inc.com>wrote:
> Hi stack,
> Having some trouble trying this. I created a log4j.properties file and
> tried different combinations of these, with the goal of seeing if I could
> first adjust the log level at all from current INFO:
> log4j.logger.org.apache.zookeeper=ERROR,stdout
> zookeeper.root.logger=ERROR, stdout
> log4j.rootLogger=ERROR, stdout
>
>
Are you thinking that ERROR is finer-grained than DEBUG by chance? If so,
http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/Level.html.
ERROR doesn't show DEBUG.
It looks like the shell is overriding the log4j setting if thats of any
help. If I disable that facility, it seems the shell does DEBUG level zk
logging.
Index: bin/hirb.rb
===================================================================
--- bin/hirb.rb (revision 899058)
+++ bin/hirb.rb (working copy)
@@ -36,7 +36,7 @@
format = 'console'
format_width = 110
script2run = nil
-logLevel = org.apache.log4j.Level::ERROR
+#logLevel = org.apache.log4j.Level::ERROR
for arg in ARGV
if arg =~ /^--format=(.+)/i
format = $1
@@ -78,7 +78,7 @@
# Set logging level to avoid verboseness
logger = org.apache.log4j.Logger.getLogger("org.apache.zookeeper")
-logger.setLevel(logLevel);
+#logger.setLevel(logLevel);
logger = org.apache.log4j.Logger.getLogger("org.apache.hadoop.hbase")
logger.setLevel(logLevel);
# Require HBase now after setting log levels
> But no effect. Probably a dumb log4j error, but it's blocking me.
>
>
If in log4j, I set zk to debug level:
pynchon:0.20 stack$ svn diff conf/
Index: conf/log4j.properties
===================================================================
--- conf/log4j.properties (revision 898656)
+++ conf/log4j.properties (working copy)
@@ -40,7 +40,7 @@
# Custom Logging levels
-log4j.logger.org.apache.zookeeper=INFO
+log4j.logger.org.apache.zookeeper=DEBUG
#log4j.logger.org.apache.hadoop.fs.FSNamesystem=DEBUG
log4j.logger.org.apache.hadoop.hbase=DEBUG
#log4j.logger.org.apache.hadoop.dfs=DEBUG
... and run for example the PerformanceEvaluation client:
$ ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation sequentialWrite
1
... it seems to report zk at DEBUG level.
> Also, I moved to yet another Hbase deployment. This time I was able to
> insert about 20 million records from my client. Then I encountered the
> same
> problem. I can insert with the shell, but not from my client.
For sure, your client is finding the same conf and the hbase shell? I
wonder why it was working, then stopped. If you look in the zk cluster
member logs, anything happen at the 20M mark?
> The only
> Hbase-side thing I see are the NodeExistsException: KeeperErrorCode =
> NodeExists in the zookeeper log, and there is one for each of my threads.
>
> Also, I tried restarting Hbase/Hadoop, making sure the everything was idle
> before I started again, but no luck.
>
> And if you restart the zk cluster?
St.Ack
> Thanks,
> Adam
>
>
>
> On 1/13/10 2:54 PM, "stack" <st...@duboce.net> wrote:
>
> > I wonder what the 'node' is that its complaining about? It doesn't say.
> I
> > wonder if you did DEBUG for zk if you'd learn more? Could enable it on
> the
> > problematic client only?
> >
> >
> > St.Ack
> >
> > On Wed, Jan 13, 2010 at 11:25 AM, Adam Silberstein
> > <si...@yahoo-inc.com>wrote:
> >
> >> Hi,
> >> I set up a new Hbase cluster and am seeing an error I haven't seen
> before.
> >> When I start up Hbase I have this NodeExists error in the zookeeper log:
> >>
> >> org.apache.zookeeper.KeeperException$NodeExistsException:
> KeeperErrorCode =
> >> NodeExists
> >> at
> >>
> >>
> org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProcess
> >> or.java:245)
> >> at
> >>
> >>
> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.ja
> >> va:114)
> >> 2010-01-13 11:10:47,503 WARN
> >> org.apache.zookeeper.server.PrepRequestProcessor: Got exception when
> >> processing sessionid:0x1262918396b0002 type:create cxid:0xc
> >> zxid:0xfffffffffffffffe txntype:unknown n/a
> >> org.apache.zookeeper.KeeperException$NodeExistsException:
> KeeperErrorCode =
> >> NodeExists
> >> at
> >>
> >>
> org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProcess
> >> or.java:245)
> >> at
> >>
> >>
> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.ja
> >> va:114)
> >>
> >> I've read that may be harmless. But then when I run the java client, it
> >> cannot do any operations. And I see a similar NodeExists error in the
> >> zookeeper log.
> >>
> >>
> >> 2010-01-13 11:11:24,661 INFO org.apache.zookeeper.server.NIOServerCnxn:
> >> Connected to /203.104.21.210:36928 lastZxid 0
> >> 2010-01-13 11:11:24,661 INFO org.apache.zookeeper.server.NIOServerCnxn:
> >> Creating new session 0x1262918396b0004
> >> 2010-01-13 11:11:24,664 INFO org.apache.zookeeper.server.NIOServerCnxn:
> >> Finished init of 0x1262918396b0004 valid:true
> >> 2010-01-13 11:11:24,973 WARN
> >> org.apache.zookeeper.server.PrepRequestProcessor: Got exception when
> >> processing sessionid:0x1262918396b0004 type:create cxid:0x2
> >> zxid:0xfffffffffffffffe txntype:unknown n/a
> >> org.apache.zookeeper.KeeperException$NodeExistsException:
> KeeperErrorCode =
> >> NodeExists
> >> at
> >>
> >>
> org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProcess
> >> or.java:245)
> >> at
> >>
> >>
> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.ja
> >> va:114)
> >> 2010-01-13 11:11:33,088 WARN org.apache.zookeeper.server.NIOServerCnxn:
> >> Exception causing close of session 0x1262918396b0004 due to
> >> java.io.IOException: Read error
> >> 2010-01-13 11:11:33,089 INFO org.apache.zookeeper.server.NIOServerCnxn:
> >> closing session:0x1262918396b0004 NIOServerCnxn:
> >> java.nio.channels.SocketChannel[connected local=/203.104.21.210:2181
> >> remote=/203.104.21.210:36928]
> >> 2010-01-13 11:11:42,709 INFO org.apache.zookeeper.server.NIOServerCnxn:
> >> Connected to /203.104.21.212:54837 lastZxid 0
> >> 2010-01-13 11:11:42,709 INFO org.apache.zookeeper.server.NIOServerCnxn:
> >> Creating new session 0x1262918396b0005
> >> 2010-01-13 11:11:42,711 INFO org.apache.zookeeper.server.NIOServerCnxn:
> >> Finished init of 0x1262918396b0005 valid:true
> >> 2010-01-13 11:11:42,711 WARN
> >> org.apache.zookeeper.server.PrepRequestProcessor: Got exception when
> >> processing sessionid:0x1262918396b0005 type:create cxid:0x1
> >> zxid:0xfffffffffffffffe txntype:unknown n/a
> >> org.apache.zookeeper.KeeperException$NodeExistsException:
> KeeperErrorCode =
> >> NodeExists
> >> at
> >>
> >>
> org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProcess
> >> or.java:245)
> >> at
> >>
> >>
> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.ja
> >> va:114)
> >> 2010-01-13 11:12:06,002 INFO
> >> org.apache.zookeeper.server.SessionTrackerImpl:
> >> Expiring session 0x1262918396b0004
> >> 2010-01-13 11:12:06,002 INFO
> org.apache.zookeeper.server.ZooKeeperServer:
> >> Expiring session 0x1262918396b0004
> >> 2010-01-13 11:12:06,002 INFO
> >> org.apache.zookeeper.server.PrepRequestProcessor: Processed session
> >> termination request for id: 0x1262918396b0004
> >>
> >> I've tried wiping the whole setup, including the zookeeper files that
> Hbase
> >> puts in /tmp and formatting HDFS. But I get the same problem again.
> I'm
> >> not seeing any exceptions on the client-side.
> >>
> >> Thanks,
> >> Adam
> >>
> >>
>
>
Re: Java client problem w/ zookeeper
Posted by Adam Silberstein <si...@yahoo-inc.com>.
Hi stack,
Having some trouble trying this. I created a log4j.properties file and
tried different combinations of these, with the goal of seeing if I could
first adjust the log level at all from current INFO:
log4j.logger.org.apache.zookeeper=ERROR,stdout
zookeeper.root.logger=ERROR, stdout
log4j.rootLogger=ERROR, stdout
But no effect. Probably a dumb log4j error, but it's blocking me.
Also, I moved to yet another Hbase deployment. This time I was able to
insert about 20 million records from my client. Then I encountered the same
problem. I can insert with the shell, but not from my client. The only
Hbase-side thing I see are the NodeExistsException: KeeperErrorCode =
NodeExists in the zookeeper log, and there is one for each of my threads.
Also, I tried restarting Hbase/Hadoop, making sure the everything was idle
before I started again, but no luck.
Thanks,
Adam
On 1/13/10 2:54 PM, "stack" <st...@duboce.net> wrote:
> I wonder what the 'node' is that its complaining about? It doesn't say. I
> wonder if you did DEBUG for zk if you'd learn more? Could enable it on the
> problematic client only?
>
>
> St.Ack
>
> On Wed, Jan 13, 2010 at 11:25 AM, Adam Silberstein
> <si...@yahoo-inc.com>wrote:
>
>> Hi,
>> I set up a new Hbase cluster and am seeing an error I haven't seen before.
>> When I start up Hbase I have this NodeExists error in the zookeeper log:
>>
>> org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode =
>> NodeExists
>> at
>>
>> org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProcess
>> or.java:245)
>> at
>>
>> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.ja
>> va:114)
>> 2010-01-13 11:10:47,503 WARN
>> org.apache.zookeeper.server.PrepRequestProcessor: Got exception when
>> processing sessionid:0x1262918396b0002 type:create cxid:0xc
>> zxid:0xfffffffffffffffe txntype:unknown n/a
>> org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode =
>> NodeExists
>> at
>>
>> org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProcess
>> or.java:245)
>> at
>>
>> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.ja
>> va:114)
>>
>> I've read that may be harmless. But then when I run the java client, it
>> cannot do any operations. And I see a similar NodeExists error in the
>> zookeeper log.
>>
>>
>> 2010-01-13 11:11:24,661 INFO org.apache.zookeeper.server.NIOServerCnxn:
>> Connected to /203.104.21.210:36928 lastZxid 0
>> 2010-01-13 11:11:24,661 INFO org.apache.zookeeper.server.NIOServerCnxn:
>> Creating new session 0x1262918396b0004
>> 2010-01-13 11:11:24,664 INFO org.apache.zookeeper.server.NIOServerCnxn:
>> Finished init of 0x1262918396b0004 valid:true
>> 2010-01-13 11:11:24,973 WARN
>> org.apache.zookeeper.server.PrepRequestProcessor: Got exception when
>> processing sessionid:0x1262918396b0004 type:create cxid:0x2
>> zxid:0xfffffffffffffffe txntype:unknown n/a
>> org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode =
>> NodeExists
>> at
>>
>> org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProcess
>> or.java:245)
>> at
>>
>> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.ja
>> va:114)
>> 2010-01-13 11:11:33,088 WARN org.apache.zookeeper.server.NIOServerCnxn:
>> Exception causing close of session 0x1262918396b0004 due to
>> java.io.IOException: Read error
>> 2010-01-13 11:11:33,089 INFO org.apache.zookeeper.server.NIOServerCnxn:
>> closing session:0x1262918396b0004 NIOServerCnxn:
>> java.nio.channels.SocketChannel[connected local=/203.104.21.210:2181
>> remote=/203.104.21.210:36928]
>> 2010-01-13 11:11:42,709 INFO org.apache.zookeeper.server.NIOServerCnxn:
>> Connected to /203.104.21.212:54837 lastZxid 0
>> 2010-01-13 11:11:42,709 INFO org.apache.zookeeper.server.NIOServerCnxn:
>> Creating new session 0x1262918396b0005
>> 2010-01-13 11:11:42,711 INFO org.apache.zookeeper.server.NIOServerCnxn:
>> Finished init of 0x1262918396b0005 valid:true
>> 2010-01-13 11:11:42,711 WARN
>> org.apache.zookeeper.server.PrepRequestProcessor: Got exception when
>> processing sessionid:0x1262918396b0005 type:create cxid:0x1
>> zxid:0xfffffffffffffffe txntype:unknown n/a
>> org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode =
>> NodeExists
>> at
>>
>> org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProcess
>> or.java:245)
>> at
>>
>> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.ja
>> va:114)
>> 2010-01-13 11:12:06,002 INFO
>> org.apache.zookeeper.server.SessionTrackerImpl:
>> Expiring session 0x1262918396b0004
>> 2010-01-13 11:12:06,002 INFO org.apache.zookeeper.server.ZooKeeperServer:
>> Expiring session 0x1262918396b0004
>> 2010-01-13 11:12:06,002 INFO
>> org.apache.zookeeper.server.PrepRequestProcessor: Processed session
>> termination request for id: 0x1262918396b0004
>>
>> I've tried wiping the whole setup, including the zookeeper files that Hbase
>> puts in /tmp and formatting HDFS. But I get the same problem again. I'm
>> not seeing any exceptions on the client-side.
>>
>> Thanks,
>> Adam
>>
>>
Re: Java client problem w/ zookeeper
Posted by stack <st...@duboce.net>.
I wonder what the 'node' is that its complaining about? It doesn't say. I
wonder if you did DEBUG for zk if you'd learn more? Could enable it on the
problematic client only?
St.Ack
On Wed, Jan 13, 2010 at 11:25 AM, Adam Silberstein
<si...@yahoo-inc.com>wrote:
> Hi,
> I set up a new Hbase cluster and am seeing an error I haven't seen before.
> When I start up Hbase I have this NodeExists error in the zookeeper log:
>
> org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode =
> NodeExists
> at
>
> org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProcess
> or.java:245)
> at
>
> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.ja
> va:114)
> 2010-01-13 11:10:47,503 WARN
> org.apache.zookeeper.server.PrepRequestProcessor: Got exception when
> processing sessionid:0x1262918396b0002 type:create cxid:0xc
> zxid:0xfffffffffffffffe txntype:unknown n/a
> org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode =
> NodeExists
> at
>
> org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProcess
> or.java:245)
> at
>
> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.ja
> va:114)
>
> I've read that may be harmless. But then when I run the java client, it
> cannot do any operations. And I see a similar NodeExists error in the
> zookeeper log.
>
>
> 2010-01-13 11:11:24,661 INFO org.apache.zookeeper.server.NIOServerCnxn:
> Connected to /203.104.21.210:36928 lastZxid 0
> 2010-01-13 11:11:24,661 INFO org.apache.zookeeper.server.NIOServerCnxn:
> Creating new session 0x1262918396b0004
> 2010-01-13 11:11:24,664 INFO org.apache.zookeeper.server.NIOServerCnxn:
> Finished init of 0x1262918396b0004 valid:true
> 2010-01-13 11:11:24,973 WARN
> org.apache.zookeeper.server.PrepRequestProcessor: Got exception when
> processing sessionid:0x1262918396b0004 type:create cxid:0x2
> zxid:0xfffffffffffffffe txntype:unknown n/a
> org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode =
> NodeExists
> at
>
> org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProcess
> or.java:245)
> at
>
> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.ja
> va:114)
> 2010-01-13 11:11:33,088 WARN org.apache.zookeeper.server.NIOServerCnxn:
> Exception causing close of session 0x1262918396b0004 due to
> java.io.IOException: Read error
> 2010-01-13 11:11:33,089 INFO org.apache.zookeeper.server.NIOServerCnxn:
> closing session:0x1262918396b0004 NIOServerCnxn:
> java.nio.channels.SocketChannel[connected local=/203.104.21.210:2181
> remote=/203.104.21.210:36928]
> 2010-01-13 11:11:42,709 INFO org.apache.zookeeper.server.NIOServerCnxn:
> Connected to /203.104.21.212:54837 lastZxid 0
> 2010-01-13 11:11:42,709 INFO org.apache.zookeeper.server.NIOServerCnxn:
> Creating new session 0x1262918396b0005
> 2010-01-13 11:11:42,711 INFO org.apache.zookeeper.server.NIOServerCnxn:
> Finished init of 0x1262918396b0005 valid:true
> 2010-01-13 11:11:42,711 WARN
> org.apache.zookeeper.server.PrepRequestProcessor: Got exception when
> processing sessionid:0x1262918396b0005 type:create cxid:0x1
> zxid:0xfffffffffffffffe txntype:unknown n/a
> org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode =
> NodeExists
> at
>
> org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProcess
> or.java:245)
> at
>
> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.ja
> va:114)
> 2010-01-13 11:12:06,002 INFO
> org.apache.zookeeper.server.SessionTrackerImpl:
> Expiring session 0x1262918396b0004
> 2010-01-13 11:12:06,002 INFO org.apache.zookeeper.server.ZooKeeperServer:
> Expiring session 0x1262918396b0004
> 2010-01-13 11:12:06,002 INFO
> org.apache.zookeeper.server.PrepRequestProcessor: Processed session
> termination request for id: 0x1262918396b0004
>
> I've tried wiping the whole setup, including the zookeeper files that Hbase
> puts in /tmp and formatting HDFS. But I get the same problem again. I'm
> not seeing any exceptions on the client-side.
>
> Thanks,
> Adam
>
>