You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Zhenyu Zhong <zh...@gmail.com> on 2009/11/11 18:02:11 UTC
regionserver disconnection
Hi,
I have been monitoring the system resources such as memory Heap usage for
Regionserver, load average, network bandwidth, etc.
I experienced a disconnection last night. When I checked the memory Heap for
Regionserver, it was 2336 which is way below the max Heap 4096. I don't
think it is Regionserver problem. The requests that the regionserver got was
about 126 which doesn't seem to be high. The load average is kind of high
>100. (but I saw disconnections while the load average is around 30). The
bandwidth is not a problem.
However, I found error messages from the zookeeper. Please see the following
error messages. There are some
BufferUnderflowException error, IOException error,etc.
May I ask what might be the causes that generate these error messages?
Many thanks
zhenyu
2009-11-11 04:29:00,810 WARN org.apache.zookeeper.server.Request: Ignoring
exception during toString
java.nio.BufferUnderflowException
at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
at java.nio.ByteBuffer.get(ByteBuffer.java:675)
at org.apache.zookeeper.server.Request.toString(Request.java:199)
at java.lang.String.valueOf(String.java:2827)
at java.lang.StringBuilder.append(StringBuilder.java:115)
at
org.apache.zookeeper.server.quorum.CommitProcessor.processRequest(CommitProcessor.java:167)
at
org.apache.zookeeper.server.quorum.FollowerRequestProcessor.run(FollowerRequestProcessor.java:68)
2009-11-11 04:29:00,810 WARN org.apache.zookeeper.server.NIOServerCnxn:
Exception causing close of session 0x424dfd908c50009
due to java.io.IOException: Read error
2009-11-11 04:29:00,810 DEBUG org.apache.zookeeper.server.NIOServerCnxn:
IOException stack trace
java.io.IOException: Read error
at
org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:372)
at
org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:221)
2009-11-11 04:29:00,810 DEBUG
org.apache.zookeeper.server.quorum.CommitProcessor: Processing request::
sessionid:0x424dfd908c50009 type:setWatches cxid:0xfffffffffffffff8
zxid:0xfffffffffffffffe txntype:unknown n/a
2009-11-11 04:29:00,810 WARN org.apache.zookeeper.server.Request: Ignoring
exception during toString
java.nio.BufferUnderflowException
at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
at java.nio.ByteBuffer.get(ByteBuffer.java:675)
at org.apache.zookeeper.server.Request.toString(Request.java:199)
at java.lang.String.valueOf(String.java:2827)
at java.lang.StringBuilder.append(StringBuilder.java:115)
at
org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:74)
at
org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)
2009-11-11 04:29:00,810 DEBUG
org.apache.zookeeper.server.FinalRequestProcessor: Processing request::
sessionid:0x424dfd908c50009 type:setWatches cxid:0xfffffffffffffff8
zxid:0xfffffffffffffffe txntype:unknown n/a
2009-11-11 04:29:00,810 WARN org.apache.zookeeper.server.Request: Ignoring
exception during toString
java.nio.BufferUnderflowException
at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
at java.nio.ByteBuffer.get(ByteBuffer.java:675)
at org.apache.zookeeper.server.Request.toString(Request.java:199)
at
org.apache.log4j.or.DefaultRenderer.doRender(DefaultRenderer.java:36)
at
org.apache.log4j.or.RendererMap.findAndRender(RendererMap.java:80)
at
org.apache.log4j.spi.LoggingEvent.getRenderedMessage(LoggingEvent.java:362)
at
org.apache.log4j.helpers.PatternParser$BasicPatternConverter.convert(PatternParser.java:403)
at
org.apache.log4j.helpers.PatternConverter.format(PatternConverter.java:65)
at org.apache.log4j.PatternLayout.format(PatternLayout.java:502)
at
org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:302)
at
org.apache.log4j.DailyRollingFileAppender.subAppend(DailyRollingFileAppender.java:359)
at org.apache.log4j.WriterAppender.append(WriterAppender.java:160)
at
org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
at
org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
at org.apache.log4j.Category.callAppenders(Category.java:206)
at org.apache.log4j.Category.forcedLog(Category.java:391)
at org.apache.log4j.Category.debug(Category.java:260)
at
org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:147)
at
org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)
2009-11-11 04:29:00,810 WARN org.apache.zookeeper.server.Request: Ignoring
exception during toString
sessionid:0x424dfd908c50009 type:setWatches cxid:0xfffffffffffffff8
zxid:0xfffffffffffffffe txntype:unknown n/a
2009-11-11 04:29:00,810 INFO org.apache.zookeeper.server.NIOServerCnxn:
closing session:0x424dfd908c50009 NIOServerCnxn:
java.nio.channels.SocketChannel[connected local=/192.168.100.131:2181remote=/
192.168.100.131:35500]
2009-11-11 04:29:00,810 DEBUG org.apache.zookeeper.server.NIOServerCnxn:
ignoring exception during input shutdown
java.net.SocketException: Transport endpoint is not connected
at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
at
sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
at
org.apache.zookeeper.server.NIOServerCnxn.close(NIOServerCnxn.java:848)
at
org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:498)
at
org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:221)
Re: regionserver disconnection
Posted by Jean-Daniel Cryans <jd...@apache.org>.
That would make sense.
The master log shows us that the times fit, the region server had some
kind of pause for 1 minute and lost its session. It still looks like
the config of 10 minutes didn't get to the region servers.
J-D
On Thu, Nov 12, 2009 at 11:50 AM, Zhenyu Zhong <zh...@gmail.com> wrote:
> Stack,
>
> I am very appreciated for your great help and insightful comments.
>
> In my cluster, half of the cluster have 2 disks 400GB each per machine, and
> half of the cluster have 6 disks per machine. Maybe we should run zookeeper
> on the machines with 2 disks and RS on machines with 6 disks?
>
> BTW, the 10 minutes zookeeper.session.timeout has been set during the
> experiment.
>
> thanks
> zhenyu
>
> On Thu, Nov 12, 2009 at 2:08 PM, stack <st...@duboce.net> wrote:
>
>> On Thu, Nov 12, 2009 at 8:40 AM, Zhenyu Zhong <zhongresearch@gmail.com
>> >wrote:
>>
>>
>> > Though I experienced 2 regionserver disconnection this morning, it looks
>> > better from the regionserver log. (please see the following log)
>> >
>> >
>> > http://pastebin.com/m496dbfae
>> >
>> >
>> > I checked diskIO for one of the regionserver(192.168.100.116) during the
>> > time it disconnected.
>> >
>> > Time: 03:04:58 AM
>> > Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
>> > sda 105.31 5458.83 19503.64 9043873239 32312473676
>> > sda1 2.90 6.64 99.25 10993934 164433464
>> > sda2 1.72 23.76 30.25 39365817 50124008
>> > sda3 0.30 0.38 3.58 624930 5923000
>> > sda4 100.39 5428.06 19370.56 8992888270 32091993204
>> >
>> >
>> Is this high for you? 20k blocks/second would seem to be high but its one
>> disk only and its not being shared by zk anymore so shouldn't matter?
>>
>>
>> > I also checked the zookeeper quorum server that the regionserver tried to
>> > connect according to the log. However, I don't see 192.168.100.116 in the
>> > client list of the zookeeper quorum server that the regionserver tried to
>> > connect.
>> > Would that be a problem?
>> >
>> >
>> Is that because the ephemeral node for the regionserver had evaporated?
>> Lost
>> its lease w/ zk by the time you went to look?
>>
>>
>>
>>
>> > Thu Nov 12 15:04:52 UTC 2009
>> > Zookeeper version: 3.2.1-808558, built on 08/27/2009 18:48 GMT
>> > Clients:
>> > /192.168.100.127:43045[1](queued=0,recved=26,sent=0)
>> > /192.168.100.131:39091[1](queued=0,recved=964,sent=0)
>> > /192.168.100.124:35961[1](queued=0,recved=3266,sent=0)
>> > /192.168.100.123:47935[1](queued=0,recved=233,sent=0)
>> > /192.168.100.125:46931[1](queued=0,recved=2,sent=0)
>> > /192.168.100.118:54924[1](queued=0,recved=295,sent=0)
>> > /192.168.100.118:41390[1](queued=0,recved=2290,sent=0)
>> > /192.168.100.136:42243[1](queued=0,recved=0,sent=0)
>> >
>> > Latency min/avg/max: 0/17/6333
>> > Received: 47111
>> > Sent: 0
>> > Outstanding: 0
>> > Zxid: 0x77000083f4
>> > Mode: leader
>> > Node count: 23
>> >
>>
>>
>> That 6 second maximum latency is pretty bad but should be well within the
>> zk
>> session timeout.
>>
>> So, problem is likely on the zk client side of the session; i.e. in the RS.
>> You could enable GC logging as J-D suggested to see if you have any big
>> pauses, pauses > zk session timeout.
>>
>> When the RS went down, it didn't look too heavily loaded:
>>
>>
>> 1. 2009-11-12 15:04:52,830 INFO
>> org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics:
>> request=1.5166667, regions=322, stores=657, storefiles=631,
>> storefileIndexSize=61, memstoreSize=1472, usedHeap=2819, maxHeap=4079,
>> blockCacheSize=658110960, blockCacheFree=197395984, blockCacheCount=9903,
>> blockCacheHitRatio=99
>>
>> Its serving a few reads? The number of store files seems fine. Not too
>> much memory used.
>>
>> Looking at the logs, I see the Lease Still Held exception. This happens
>> when the RS does its regular report to the master but the master thinks the
>> RS has since restarted. It'll think this probably because it noticed that
>> the RS's znode in zk had gone away and it considered the RS dead.
>>
>> Looking too at your logs I see this gap in the zk pinging:
>>
>>
>> 1. 2009-11-12 15:03:39,325 DEBUG org.apache.zookeeper.ClientCnxn: Got
>> ping response for sessionid:0x224e55436ad0004 after 0ms
>> 2. 2009-11-12 15:03:43,113 DEBUG org.apache.zookeeper.ClientCnxn: Got
>> ping response for sessionid:0x24e55436a0007d after 0ms
>>
>> Where in the lines above it, its reporting about every ten seconds, here
>> there is a big gap.
>>
>> Do you have ganglia or something that will let you look more into what was
>> happening on this machine around this time? Is the machine OK? It looks
>> lightly loaded and you have your cluster nicely laid out. Something odd is
>> going on. What about things like the write speed to disk? In the past
>> strange issues have been explained by incorrectly set BIOS which made disks
>> run at 1/100th of their proper speed.
>>
>> St.Ack
>>
>>
>>
>>
>>
>>
>>
>> >
>> > Best,
>> > zhenyu
>> >
>> >
>> > On Wed, Nov 11, 2009 at 3:58 PM, Zhenyu Zhong <zhongresearch@gmail.com
>> > >wrote:
>> >
>> > > Stack
>> > >
>> > > I am very appreciated for your comments.
>> > > I will use the zookeeper monitoring script on my cluster and let it run
>> > > overnight to see the result.
>> > > I will follow up the thread when I get anything.
>> > >
>> > > thanks
>> > > zhenyu
>> > >
>> > >
>> > > On Wed, Nov 11, 2009 at 3:52 PM, stack <st...@duboce.net> wrote:
>> > >
>> > >> I see these in your log too:
>> > >>
>> > >>
>> > >> 1. 2009-11-11 04:27:19,018 DEBUG org.apache.zookeeper.ClientCnxn:
>> Got
>> > >> ping response for sessionid:0x424dfd908c50009 after 4544ms
>> > >> 2. 2009-11-11 04:27:19,018 DEBUG org.apache.zookeeper.ClientCnxn:
>> Got
>> > >> ping response for sessionid:0x24dfd90c810002 after 4548ms
>> > >> 3. 2009-11-11 04:27:43,960 DEBUG org.apache.zookeeper.ClientCnxn:
>> Got
>> > >> ping response for sessionid:0x424dfd908c50009 after 9030ms
>> > >> 4. 2009-11-11 04:27:43,960 DEBUG org.apache.zookeeper.ClientCnxn:
>> Got
>> > >> ping response for sessionid:0x24dfd90c810002 after 9030ms
>> > >> 5. 2009-11-11 04:27:48,830 DEBUG org.apache.zookeeper.ClientCnxn:
>> Got
>> > >> ping response for sessionid:0x424dfd908c50009 after 0ms
>> > >> 6. 2009-11-11 04:27:48,830 DEBUG org.apache.zookeeper.ClientCnxn:
>> Got
>> > >> ping response for sessionid:0x24dfd90c810002 after 0ms
>> > >>
>> > >> Which would indicate lag talking to zk too.
>> > >>
>> > >> Appending HLog also seems to get steadily worse.
>> > >>
>> > >> I also see these:
>> > >>
>> > >>
>> > >> 1. 2009-11-11 04:27:01,233 WARN org.apache.zookeeper.ClientCnxn:
>> > >> Exception closing session 0x24dfd90c810002 to
>> > >> sun.nio.ch.SelectionKeyImpl@5e16a7e3
>> > >> 2. java.io.IOException: Read error rc = -1
>> > >> java.nio.DirectByteBuffer[pos=0 lim=4 cap=4]
>> > >> 3. at
>> > >>
>> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
>> > >> 4. at
>> > >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
>> > >>
>> > >>
>> > >> Were all 5 quorum members participating? Check by doing "echo stat|nc
>> > >> QUORUM_HOST port" for each member of quorum.... the below might help:
>> > >>
>> > >> members="1,2,3,4,5"
>> > >> while true; do date; for i in $members; do echo $i; echo stat | nc $i
>> > >> 2181;
>> > >> done; sleep 10; done
>> > >>
>> > >> We need to improve our zk_dump. It just reports quorum members... not
>> > if
>> > >> they are actually participating or not. The above will tell you if
>> dead
>> > >> or
>> > >> alive.
>> > >>
>> > >> Please report back after checking above and following the Jean-Daniel
>> > >> prescription.
>> > >>
>> > >> Thanks,
>> > >> St.Ack
>> > >>
>> > >>
>> > >>
>> > >> On Wed, Nov 11, 2009 at 12:29 PM, Zhenyu Zhong <
>> zhongresearch@gmail.com
>> > >> >wrote:
>> > >>
>> > >> > Thanks a lot for the help.
>> > >> >
>> > >> > I understand that the change of disk configurations should be done
>> > >> > progressively such that HDFS can balance and recover from a disk
>> going
>> > >> > away.
>> > >> >
>> > >> > The data we are trying to import into HBase is about 200million
>> rows,
>> > >> each
>> > >> > row is about hundreds of bytes. Basically, we read in a text file
>> from
>> > >> > HDFS,
>> > >> > then for each row in the text file, we fetch data from HBase table,
>> > we
>> > >> > compare the column value and update the latest value back into
>> HBase.
>> > I
>> > >> was
>> > >> > suggested to use CheckAndPut. Anyway this is the common usecase on
>> my
>> > >> > side.
>> > >> >
>> > >> >
>> > >> > Many thanks
>> > >> > zhenyu
>> > >> >
>> > >> >
>> > >> >
>> > >> > On Wed, Nov 11, 2009 at 3:03 PM, Jean-Daniel Cryans <
>> > >> jdcryans@apache.org
>> > >> > >wrote:
>> > >> >
>> > >> > > 6 disks, 19 machines and you see those kind of issues? I'm quite
>> > >> > > surprised and I want to learn more about what you trying to to.
>> > >> > >
>> > >> > > But first, if hbase is managing ZK for you then you can change the
>> > >> > > value of hbase.zookeeper.property.dataDir so that it points to its
>> > own
>> > >> > > mount. Make sure you also change dfs.data.dir in
>> conf/hdfs-site.xml
>> > >> > > (hadoop stuff) so that it stops pointing there (you should have 6
>> > >> > > values there at the moment if you configured it correctly). Doing
>> > this
>> > >> > > could result in data loss if you do it all at once so do the
>> change
>> > >> > > progressively. This also means that you may have to keep a
>> seperate
>> > >> > > HDFS configuration for those particular nodes. Please ask more
>> > >> > > questions if it's not very clear.
>> > >> > >
>> > >> > > WRT to dedicating nodes to Zookeeeper, it makes sense only if you
>> > have
>> > >> > > a lot of nodes but in your case, unless there's also CPU
>> contention,
>> > >> > > you're good with only a dedicated disk.
>> > >> > >
>> > >> > > Now for what you are trying to do, is this a massive import that
>> > goes
>> > >> > > over many hours? How big is each row? I'd love to get a lot of
>> > details
>> > >> > > to help you optimize your cluster for your particular problem.
>> > >> > >
>> > >> > > Thx,
>> > >> > >
>> > >> > > J-D
>> > >> > >
>> > >> > > On Wed, Nov 11, 2009 at 11:51 AM, Zhenyu Zhong <
>> > >> zhongresearch@gmail.com>
>> > >> > > wrote:
>> > >> > > > J-D,
>> > >> > > >
>> > >> > > > Thanks a lot for the insightful comments.
>> > >> > > > We are running 5 zookeeper quorums now. On these 5 quorum
>> servers,
>> > >> we
>> > >> > > also
>> > >> > > > run datanode, tasktracker and regionserver.
>> > >> > > > Now we have 6 disks per machine.
>> > >> > > > If it is the disk IO saturation, is there anyway to configure
>> > >> zookeeper
>> > >> > > to
>> > >> > > > write to a seperate disk which is not the same one used by HDFS?
>> > >> > > > Or shall we run zookeeper quorums on different nodes other than
>> > >> HDFS?
>> > >> > > >
>> > >> > > > Are HLogs generated by Regionservers? Would it alleviate the
>> issue
>> > >> if
>> > >> > we
>> > >> > > use
>> > >> > > > the first 14 for regionserver, and the rest of 5 for
>> zookeeper?(In
>> > >> that
>> > >> > > > sense, the node running zookeeper quorum won't run regionserver)
>> > >> > > >
>> > >> > > > I am very appreciated for your comments.
>> > >> > > >
>> > >> > > > Best,
>> > >> > > > zhenyu
>> > >> > > >
>> > >> > > >
>> > >> > > >
>> > >> > > > On Wed, Nov 11, 2009 at 2:30 PM, Jean-Daniel Cryans <
>> > >> > jdcryans@apache.org
>> > >> > > >wrote:
>> > >> > > >
>> > >> > > >> The problem seems to be that the IO is saturated. When you see
>> > >> this:
>> > >> > > >>
>> > >> > > >> #
>> > >> > > >> 2009-11-11 04:28:10,308 WARN
>> > >> > > >> org.apache.hadoop.hbase.regionserver.HLog: IPC Server handler 9
>> > on
>> > >> > > >> 60021 took 6819ms appending an edit to hlog; editcount=370
>> > >> > > >> #
>> > >> > > >> 2009-11-11 04:28:10,308 WARN
>> > >> > > >> org.apache.hadoop.hbase.regionserver.HLog: IPC Server handler 2
>> > on
>> > >> > > >> 60021 took 4412ms appending an edit to hlog; editcount=371
>> > >> > > >>
>> > >> > > >> It's a sign that HDFS is slow. I guess that the zookeeper
>> quorum
>> > >> > > >> members are on those same nodes using the same disks? If so,
>> > >> there's
>> > >> > > >> the problem because Zookeeper needs to be able to commit
>> > >> transactions
>> > >> > > >> very fast and if your disks are already hogged then they have
>> to
>> > >> wait
>> > >> > > >> and you can get disconnections. Do you still have 19 quorum
>> > >> members?
>> > >> > > >> How many disks do you have per machine?
>> > >> > > >>
>> > >> > > >> Thx
>> > >> > > >>
>> > >> > > >> J-D
>> > >> > > >>
>> > >> > > >> On Wed, Nov 11, 2009 at 11:22 AM, Zhenyu Zhong <
>> > >> > zhongresearch@gmail.com
>> > >> > > >
>> > >> > > >> wrote:
>> > >> > > >> > Please take a look at the following regionserver log.
>> > >> > > >> > The error started to come out from 4:24am UTC to 4:29am UTC.
>> > >> > > >> >
>> > >> > > >> > http://pastebin.com/m3717d064
>> > >> > > >> >
>> > >> > > >> > Thanks in advance.
>> > >> > > >> >
>> > >> > > >> > Best,
>> > >> > > >> > zhenyu
>> > >> > > >> >
>> > >> > > >> >
>> > >> > > >> > On Wed, Nov 11, 2009 at 2:01 PM, Jean-Daniel Cryans <
>> > >> > > jdcryans@apache.org
>> > >> > > >> >wrote:
>> > >> > > >> >
>> > >> > > >> >> Could you paste a bigger log trace (interested to see at
>> least
>> > >> all
>> > >> > > 300
>> > >> > > >> >> lines before that) in a pastebin?
>> > >> > > >> >>
>> > >> > > >> >> Thx,
>> > >> > > >> >>
>> > >> > > >> >> J-D
>> > >> > > >> >>
>> > >> > > >> >> On Wed, Nov 11, 2009 at 10:56 AM, Zhenyu Zhong <
>> > >> > > zhongresearch@gmail.com
>> > >> > > >> >
>> > >> > > >> >> wrote:
>> > >> > > >> >> > J-D,
>> > >> > > >> >> >
>> > >> > > >> >> > Thanks for the reply.
>> > >> > > >> >> >
>> > >> > > >> >> > The problem I have is one or more of the regionservers
>> > >> disconnect
>> > >> > > from
>> > >> > > >> >> the
>> > >> > > >> >> > cluster.
>> > >> > > >> >> >
>> > >> > > >> >> > I checked the resource usage including memory Heap usage
>> for
>> > >> > > >> >> regionserver,
>> > >> > > >> >> > and found nothing suspicious. The regionserver heap usage
>> is
>> > >> way
>> > >> > > below
>> > >> > > >> >> the
>> > >> > > >> >> > maxHeap defined in jvm.
>> > >> > > >> >> >
>> > >> > > >> >> > So I checked zookeeper log, then I found the following
>> > >> > > >> >> > BufferUnderflowException, eventually it leads to
>> > >> SocketException:
>> > >> > > >> >> Transport
>> > >> > > >> >> > endpoint is not connected.
>> > >> > > >> >> >
>> > >> > > >> >> >
>> > >> > > >> >> > I also checked the regionserver log. It looks like the
>> > >> > regionserver
>> > >> > > >> tried
>> > >> > > >> >> to
>> > >> > > >> >> > connect to other zookeeper quorum server. It connects
>> > >> > successfully,
>> > >> > > >> but
>> > >> > > >> >> then
>> > >> > > >> >> > it complained about SocketException: transport endpoint is
>> > not
>> > >> > > >> connected.
>> > >> > > >> >> >
>> > >> > > >> >> > So both zookeeper log and regionserver log told me that
>> the
>> > >> > > transport
>> > >> > > >> >> > endpoint is not connected. I don't know why.
>> > >> > > >> >> >
>> > >> > > >> >> > I would be very appreciated if you could point me to the
>> > right
>> > >> > > >> direction.
>> > >> > > >> >> >
>> > >> > > >> >> > Best,
>> > >> > > >> >> >
>> > >> > > >> >> >
>> > >> > > >> >> > Regionserver log:
>> > >> > > >> >> > 2009-11-11 04:24:53,718 INFO
>> > org.apache.zookeeper.ClientCnxn:
>> > >> > > >> Attempting
>> > >> > > >> >> > connection to server superpyxis0018/192.168.100.132:2181
>> > >> > > >> >> > 2009-11-11 04:24:53,718 INFO
>> > org.apache.zookeeper.ClientCnxn:
>> > >> > > Priming
>> > >> > > >> >> > connection to java.nio.channels.SocketChannel[connected
>> > >> local=/
>> > >> > > >> >> > 192.168.100.131:43065 remote=superpyxis0018/
>> > >> 192.168.100.132:2181
>> > >> > ]
>> > >> > > >> >> > 2009-11-11 04:24:53,718 INFO
>> > org.apache.zookeeper.ClientCnxn:
>> > >> > > Server
>> > >> > > >> >> > connection successful
>> > >> > > >> >> > 2009-11-11 04:24:53,768 INFO
>> > >> > > >> >> > org.apache.hadoop.hbase.regionserver.HRegionServer: Got
>> > >> ZooKeeper
>> > >> > > >> event,
>> > >> > > >> >> > state: Disconnected, type: None, path: null
>> > >> > > >> >> > 2009-11-11 04:24:54,118 INFO
>> > org.apache.zookeeper.ClientCnxn:
>> > >> > > >> Attempting
>> > >> > > >> >> > connection to server superpyxis0017/192.168.100.131:2181
>> > >> > > >> >> > 2009-11-11 04:24:54,118 INFO
>> > org.apache.zookeeper.ClientCnxn:
>> > >> > > Priming
>> > >> > > >> >> > connection to java.nio.channels.SocketChannel[connected
>> > >> local=/
>> > >> > > >> >> > 192.168.100.131:46903 remote=superpyxis0017/
>> > >> 192.168.100.131:2181
>> > >> > ]
>> > >> > > >> >> > 2009-11-11 04:24:54,118 INFO
>> > org.apache.zookeeper.ClientCnxn:
>> > >> > > Server
>> > >> > > >> >> > connection successful
>> > >> > > >> >> > 2009-11-11 04:24:54,118 WARN
>> > org.apache.zookeeper.ClientCnxn:
>> > >> > > >> Exception
>> > >> > > >> >> > closing session 0x24dfd90c810002 to
>> > >> > > >> sun.nio.ch.SelectionKeyImpl@2dd78ccf
>> > >> > > >> >> > java.io.IOException: Read error rc = -1
>> > >> > > >> java.nio.DirectByteBuffer[pos=0
>> > >> > > >> >> > lim=4 cap=4]
>> > >> > > >> >> > at
>> > >> > > >> >> >
>> > >> > >
>> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
>> > >> > > >> >> > at
>> > >> > > >> >> >
>> > >> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
>> > >> > > >> >> > 2009-11-11 04:24:54,118 WARN
>> > org.apache.zookeeper.ClientCnxn:
>> > >> > > Ignoring
>> > >> > > >> >> > exception during shutdown input
>> > >> > > >> >> > java.net.SocketException: Transport endpoint is not
>> > connected
>> > >> > > >> >> > at sun.nio.ch.SocketChannelImpl.shutdown(Native
>> > Method)
>> > >> > > >> >> > at
>> > >> > > >> >> >
>> > >> > >
>> > sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
>> > >> > > >> >> > at
>> > >> > > >> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>> > >> > > >> >> > at
>> > >> > > >> >> >
>> > >> > > >>
>> > >> >
>> > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
>> > >> > > >> >> > at
>> > >> > > >> >> >
>> > >> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
>> > >> > > >> >> >
>> > >> > > >> >> >
>> > >> > > >> >> >
>> > >> > > >> >> >
>> > >> > > >> >> >
>> > >> > > >> >> > On Wed, Nov 11, 2009 at 1:40 PM, Jean-Daniel Cryans <
>> > >> > > >> jdcryans@apache.org
>> > >> > > >> >> >wrote:
>> > >> > > >> >> >
>> > >> > > >> >> >> I'm really not sure exactly what the situation is with
>> the
>> > >> > > >> >> >> java.nio.BufferUnderflowException, but what do you mean
>> by
>> > >> > > >> >> >> "regionserver disconnection"? Could you give more
>> details?
>> > >> > > >> >> >>
>> > >> > > >> >> >> Thx,
>> > >> > > >> >> >>
>> > >> > > >> >> >> J-D
>> > >> > > >> >> >>
>> > >> > > >> >> >> On Wed, Nov 11, 2009 at 10:34 AM, Zhenyu Zhong <
>> > >> > > >> zhongresearch@gmail.com
>> > >> > > >> >> >
>> > >> > > >> >> >> wrote:
>> > >> > > >> >> >> > I searched around and found HBASE-1645 "HBase could be
>> a
>> > >> > little
>> > >> > > >> more
>> > >> > > >> >> >> dainty
>> > >> > > >> >> >> > when zk is pulled out from under it"
>> > >> > > >> >> >> >
>> > >> > > >> >> >> > I am wondering whether my problem is related to this
>> > >> > HBase-1645.
>> > >> > > Or
>> > >> > > >> >> >> probably
>> > >> > > >> >> >> > has nothing to do with it?
>> > >> > > >> >> >> >
>> > >> > > >> >> >> > thanks
>> > >> > > >> >> >> >
>> > >> > > >> >> >> >
>> > >> > > >> >> >> > On Wed, Nov 11, 2009 at 12:02 PM, Zhenyu Zhong <
>> > >> > > >> >> zhongresearch@gmail.com
>> > >> > > >> >> >> >wrote:
>> > >> > > >> >> >> >
>> > >> > > >> >> >> >> Hi,
>> > >> > > >> >> >> >>
>> > >> > > >> >> >> >> I have been monitoring the system resources such as
>> > memory
>> > >> > Heap
>> > >> > > >> usage
>> > >> > > >> >> >> for
>> > >> > > >> >> >> >> Regionserver, load average, network bandwidth, etc.
>> > >> > > >> >> >> >>
>> > >> > > >> >> >> >> I experienced a disconnection last night. When I
>> checked
>> > >> the
>> > >> > > >> memory
>> > >> > > >> >> Heap
>> > >> > > >> >> >> >> for Regionserver, it was 2336 which is way below the
>> max
>> > >> Heap
>> > >> > > >> 4096. I
>> > >> > > >> >> >> don't
>> > >> > > >> >> >> >> think it is Regionserver problem. The requests that
>> the
>> > >> > > >> regionserver
>> > >> > > >> >> got
>> > >> > > >> >> >> was
>> > >> > > >> >> >> >> about 126 which doesn't seem to be high. The load
>> > average
>> > >> is
>> > >> > > kind
>> > >> > > >> of
>> > >> > > >> >> >> high
>> > >> > > >> >> >> >> >100. (but I saw disconnections while the load average
>> > is
>> > >> > > around
>> > >> > > >> 30).
>> > >> > > >> >> >> The
>> > >> > > >> >> >> >> bandwidth is not a problem.
>> > >> > > >> >> >> >>
>> > >> > > >> >> >> >> However, I found error messages from the zookeeper.
>> > Please
>> > >> > see
>> > >> > > the
>> > >> > > >> >> >> >> following error messages. There are some
>> > >> > > >> >> >> >> BufferUnderflowException error, IOException error,etc.
>> > >> > > >> >> >> >> May I ask what might be the causes that generate these
>> > >> error
>> > >> > > >> >> messages?
>> > >> > > >> >> >> >>
>> > >> > > >> >> >> >> Many thanks
>> > >> > > >> >> >> >> zhenyu
>> > >> > > >> >> >> >>
>> > >> > > >> >> >> >>
>> > >> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
>> > >> > > org.apache.zookeeper.server.Request:
>> > >> > > >> >> >> Ignoring
>> > >> > > >> >> >> >> exception during toString
>> > >> > > >> >> >> >> java.nio.BufferUnderflowException
>> > >> > > >> >> >> >> at
>> > >> > java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
>> > >> > > >> >> >> >> at
>> java.nio.ByteBuffer.get(ByteBuffer.java:675)
>> > >> > > >> >> >> >> at
>> > >> > > >> >> >>
>> > >> org.apache.zookeeper.server.Request.toString(Request.java:199)
>> > >> > > >> >> >> >> at java.lang.String.valueOf(String.java:2827)
>> > >> > > >> >> >> >> at
>> > >> > > java.lang.StringBuilder.append(StringBuilder.java:115)
>> > >> > > >> >> >> >> at
>> > >> > > >> >> >> >>
>> > >> > > >> >> >>
>> > >> > > >> >>
>> > >> > > >>
>> > >> > >
>> > >> >
>> > >>
>> >
>> org.apache.zookeeper.server.quorum.CommitProcessor.processRequest(CommitProcessor.java:167)
>> > >> > > >> >> >> >> at
>> > >> > > >> >> >> >>
>> > >> > > >> >> >>
>> > >> > > >> >>
>> > >> > > >>
>> > >> > >
>> > >> >
>> > >>
>> >
>> org.apache.zookeeper.server.quorum.FollowerRequestProcessor.run(FollowerRequestProcessor.java:68)
>> > >> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
>> > >> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
>> > >> > > >> >> >> >> Exception causing close of session 0x424dfd908c50009
>> > >> > > >> >> >> >> due to java.io.IOException: Read error
>> > >> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
>> > >> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
>> > >> > > >> >> >> >> IOException stack trace
>> > >> > > >> >> >> >> java.io.IOException: Read error
>> > >> > > >> >> >> >> at
>> > >> > > >> >> >> >>
>> > >> > > >> >>
>> > >> > >
>> > org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:372)
>> > >> > > >> >> >> >> at
>> > >> > > >> >> >> >>
>> > >> > > >> >> >>
>> > >> > > >> >>
>> > >> > > >>
>> > >> > >
>> > >> >
>> > >>
>> >
>> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:221)
>> > >> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
>> > >> > > >> >> >> >> org.apache.zookeeper.server.quorum.CommitProcessor:
>> > >> > Processing
>> > >> > > >> >> request::
>> > >> > > >> >> >> >> sessionid:0x424dfd908c50009 type:setWatches
>> > >> > > >> cxid:0xfffffffffffffff8
>> > >> > > >> >> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
>> > >> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
>> > >> > > org.apache.zookeeper.server.Request:
>> > >> > > >> >> >> Ignoring
>> > >> > > >> >> >> >> exception during toString
>> > >> > > >> >> >> >> java.nio.BufferUnderflowException
>> > >> > > >> >> >> >> at
>> > >> > java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
>> > >> > > >> >> >> >> at
>> java.nio.ByteBuffer.get(ByteBuffer.java:675)
>> > >> > > >> >> >> >> at
>> > >> > > >> >> >>
>> > >> org.apache.zookeeper.server.Request.toString(Request.java:199)
>> > >> > > >> >> >> >> at java.lang.String.valueOf(String.java:2827)
>> > >> > > >> >> >> >> at
>> > >> > > java.lang.StringBuilder.append(StringBuilder.java:115)
>> > >> > > >> >> >> >> at
>> > >> > > >> >> >> >>
>> > >> > > >> >> >>
>> > >> > > >> >>
>> > >> > > >>
>> > >> > >
>> > >> >
>> > >>
>> >
>> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:74)
>> > >> > > >> >> >> >> at
>> > >> > > >> >> >> >>
>> > >> > > >> >> >>
>> > >> > > >> >>
>> > >> > > >>
>> > >> > >
>> > >> >
>> > >>
>> >
>> org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)
>> > >> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
>> > >> > > >> >> >> >> org.apache.zookeeper.server.FinalRequestProcessor:
>> > >> Processing
>> > >> > > >> >> request::
>> > >> > > >> >> >> >> sessionid:0x424dfd908c50009 type:setWatches
>> > >> > > >> cxid:0xfffffffffffffff8
>> > >> > > >> >> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
>> > >> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
>> > >> > > org.apache.zookeeper.server.Request:
>> > >> > > >> >> >> Ignoring
>> > >> > > >> >> >> >> exception during toString
>> > >> > > >> >> >> >> java.nio.BufferUnderflowException
>> > >> > > >> >> >> >> at
>> > >> > java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
>> > >> > > >> >> >> >> at
>> java.nio.ByteBuffer.get(ByteBuffer.java:675)
>> > >> > > >> >> >> >> at
>> > >> > > >> >> >>
>> > >> org.apache.zookeeper.server.Request.toString(Request.java:199)
>> > >> > > >> >> >> >> at
>> > >> > > >> >> >> >>
>> > >> > > >>
>> > >> org.apache.log4j.or.DefaultRenderer.doRender(DefaultRenderer.java:36)
>> > >> > > >> >> >> >> at
>> > >> > > >> >> >> >>
>> > >> > > org.apache.log4j.or.RendererMap.findAndRender(RendererMap.java:80)
>> > >> > > >> >> >> >> at
>> > >> > > >> >> >> >>
>> > >> > > >> >> >>
>> > >> > > >> >>
>> > >> > > >>
>> > >> > >
>> > >> >
>> > >>
>> >
>> org.apache.log4j.spi.LoggingEvent.getRenderedMessage(LoggingEvent.java:362)
>> > >> > > >> >> >> >> at
>> > >> > > >> >> >> >>
>> > >> > > >> >> >>
>> > >> > > >> >>
>> > >> > > >>
>> > >> > >
>> > >> >
>> > >>
>> >
>> org.apache.log4j.helpers.PatternParser$BasicPatternConverter.convert(PatternParser.java:403)
>> > >> > > >> >> >> >> at
>> > >> > > >> >> >> >>
>> > >> > > >> >> >>
>> > >> > > >> >>
>> > >> > > >>
>> > >> > >
>> > >> >
>> > >>
>> >
>> org.apache.log4j.helpers.PatternConverter.format(PatternConverter.java:65)
>> > >> > > >> >> >> >> at
>> > >> > > >> >>
>> org.apache.log4j.PatternLayout.format(PatternLayout.java:502)
>> > >> > > >> >> >> >> at
>> > >> > > >> >> >> >>
>> > >> > > org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:302)
>> > >> > > >> >> >> >> at
>> > >> > > >> >> >> >>
>> > >> > > >> >> >>
>> > >> > > >> >>
>> > >> > > >>
>> > >> > >
>> > >> >
>> > >>
>> >
>> org.apache.log4j.DailyRollingFileAppender.subAppend(DailyRollingFileAppender.java:359)
>> > >> > > >> >> >> >> at
>> > >> > > >> >> >>
>> > >> org.apache.log4j.WriterAppender.append(WriterAppender.java:160)
>> > >> > > >> >> >> >> at
>> > >> > > >> >> >> >>
>> > >> > > >>
>> > >> org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
>> > >> > > >> >> >> >> at
>> > >> > > >> >> >> >>
>> > >> > > >> >> >>
>> > >> > > >> >>
>> > >> > > >>
>> > >> > >
>> > >> >
>> > >>
>> >
>> org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
>> > >> > > >> >> >> >> at
>> > >> > > >> org.apache.log4j.Category.callAppenders(Category.java:206)
>> > >> > > >> >> >> >> at
>> > >> > > org.apache.log4j.Category.forcedLog(Category.java:391)
>> > >> > > >> >> >> >> at
>> > >> org.apache.log4j.Category.debug(Category.java:260)
>> > >> > > >> >> >> >> at
>> > >> > > >> >> >> >>
>> > >> > > >> >> >>
>> > >> > > >> >>
>> > >> > > >>
>> > >> > >
>> > >> >
>> > >>
>> >
>> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:147)
>> > >> > > >> >> >> >> at
>> > >> > > >> >> >> >>
>> > >> > > >> >> >>
>> > >> > > >> >>
>> > >> > > >>
>> > >> > >
>> > >> >
>> > >>
>> >
>> org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)
>> > >> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
>> > >> > > org.apache.zookeeper.server.Request:
>> > >> > > >> >> >> Ignoring
>> > >> > > >> >> >> >> exception during toString
>> > >> > > >> >> >> >> sessionid:0x424dfd908c50009 type:setWatches
>> > >> > > >> cxid:0xfffffffffffffff8
>> > >> > > >> >> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
>> > >> > > >> >> >> >> 2009-11-11 04:29:00,810 INFO
>> > >> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
>> > >> > > >> >> >> >> closing session:0x424dfd908c50009 NIOServerCnxn:
>> > >> > > >> >> >> >> java.nio.channels.SocketChannel[connected local=/
>> > >> > > >> >> 192.168.100.131:2181
>> > >> > > >> >> >> remote=/
>> > >> > > >> >> >> >> 192.168.100.131:35500]
>> > >> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
>> > >> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
>> > >> > > >> >> >> >> ignoring exception during input shutdown
>> > >> > > >> >> >> >> java.net.SocketException: Transport endpoint is not
>> > >> connected
>> > >> > > >> >> >> >> at
>> sun.nio.ch.SocketChannelImpl.shutdown(Native
>> > >> > Method)
>> > >> > > >> >> >> >> at
>> > >> > > >> >> >> >>
>> > >> > > >> >>
>> > >> > >
>> > sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
>> > >> > > >> >> >> >> at
>> > >> > > >> >> >>
>> > >> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>> > >> > > >> >> >> >> at
>> > >> > > >> >> >> >>
>> > >> > > >> >>
>> > >> > >
>> > >>
>> org.apache.zookeeper.server.NIOServerCnxn.close(NIOServerCnxn.java:848)
>> > >> > > >> >> >> >> at
>> > >> > > >> >> >> >>
>> > >> > > >> >>
>> > >> > >
>> > org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:498)
>> > >> > > >> >> >> >> at
>> > >> > > >> >> >> >>
>> > >> > > >> >> >>
>> > >> > > >> >>
>> > >> > > >>
>> > >> > >
>> > >> >
>> > >>
>> >
>> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:221)
>> > >> > > >> >> >> >>
>> > >> > > >> >> >> >
>> > >> > > >> >> >>
>> > >> > > >> >> >
>> > >> > > >> >>
>> > >> > > >> >
>> > >> > > >>
>> > >> > > >
>> > >> > >
>> > >> >
>> > >>
>> > >
>> > >
>> >
>>
>
Re: regionserver disconnection
Posted by stack <st...@duboce.net>.
On Fri, Nov 13, 2009 at 10:01 PM, Zhenyu Zhong <zh...@gmail.com>wrote:
>
> Now my system has a clean start. However, it seems that the
> zookeeper.session.timeout value doesn't take into effect. I still found
> around 60 seconds pauses from the disconnected regionserver. I really don't
> know why regionserver only times out after 60 seconds instead of 10 minutes
> which I set for zookeeper.session.timeout.
>
> Usually, its because your hbase-site.xml changes are not on the CLASSPATH
when the process starts up.
> Is there any other timeout value coming into play before the actual session
> times out?
>
Not that I know of. You can grep the hbase-default.xml for 60. You could
also just add a print out of value of session timeout to the hbase daemons
just to make sure its reading the config. you expect.
St.Ack
Re: regionserver disconnection
Posted by Patrick Hunt <ph...@apache.org>.
I guess I don't get out much. ;-)
Patrick
Ryan Rawson wrote:
> I have given mini-talks about this at HUG and generally ranted about
> GC to many people, talking about fragmenting vs compacting GC, and how
> CMS-mode failures can pause you for a long time, the more Xmx the
> worse it is.
>
> Generally in the hadoop community, real time is not something that is
> really done, thus we are at the "forefront" of high memory high
> performance low latency Java. I cheerlead G1 as often as I can.
>
> -ryan
>
> On Mon, Nov 30, 2009 at 3:17 PM, Patrick Hunt <ph...@apache.org> wrote:
>> FYI: https://issues.apache.org/jira/browse/ZOOKEEPER-601
>>
>> Stack/JD/HBaseTeam - have you considered detailing/presenting Sun JVM GC
>> limitations you've faced as part of HBase effort? I find this really
>> fascinating (4min gc pauses even with "realtime" gc turned on), I doubt many
>> people realize things can get this bad, and I think it would be a great
>> service to the community if you could detail some of your findings (say at
>> the next BAHUG?) You might even tag team it with someone from the Hadoop
>> MR/HDFS team, I hear they face similar issues in the name node.
>>
>> Regards,
>>
>> Patrick
>>
>> Zhenyu Zhong wrote:
>>> Tons of thanks to Stack and J-D as well as the Zookeeper fellows.
>>>
>>> Previously we experienced RS session timeouts even we set the
>>> zookeeper.session.timeout to 10 minutes.
>>> It turned out that the zookeeper has both lower bound and upper bound for
>>> the session timeout.
>>> It actually relates to the tickerTime, so the session timeout should be no
>>> less than 2 times tickertime and no greater than 20 times tickerTime. By
>>> default the tickerTime is 2 sec, so the maximum timeout is 40 seconds by
>>> default no matter how large the value is set for
>>> zookeeper.session.timeout.
>>>
>>> Due to some unknown reason, (probably system heavy load, GC pauses, etc),
>>> 40 seconds is not long enough for us, so we have to extend that value.
>>>
>>> One way to do that is to up the hbase.zookeeper.property.tickTime. Now
>>> with
>>> 4000ms set on that value, we haven't seen the RS session TIMEOUT so far.
>>>
>>> Many thanks to Stack, J-D and Zookeeper fellows.
>>>
>>> I will keep you posted if the session timeout comes back again or there is
>>> anything else causes the RS disconnections.
>>>
>>> Best,
>>> zhenyu
>>>
>>>
>>>
>>>
>>> On Thu, Nov 19, 2009 at 1:37 PM, stack <st...@duboce.net> wrote:
>>>
>>>> From the zk fellas, try running your zk cluster at INFO level rather than
>>>> at
>>>> DEBUG. One supposition is that the shear amount of logging is causing
>>>> distress. They are still looking into your logs...
>>>>
>>>> Thanks for your patience Zhenyu.
>>>> St.Ack
>>>>
>>>>
>>>> On Thu, Nov 19, 2009 at 8:57 AM, Zhenyu Zhong <zhongresearch@gmail.com
>>>>> wrote:
>>>>> Yes, I believe so. And I don't know why the leader gave up.
>>>>> For this case, the regionserver got TIMEOUT warning message at
>>>>> 2009-11-18
>>>>> 12:42:01,482
>>>>>
>>>>>
>>>>> Best,
>>>>> zhenyu
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Nov 19, 2009 at 11:43 AM, stack <st...@duboce.net> wrote:
>>>>>
>>>>>> So, the time at which we see
>>>>>>
>>>>>> 2009-11-18 12:41:56,782 ERROR
>>>>>> org.apache.zookeeper.server.quorum.FollowerHandler: Unexpected
>>>> exception
>>>>>> causing shutdown while sock still open
>>>>>>
>>>>>> ... corresponds to the time at which the regionserver loses is session?
>>>>>>
>>>>>> St.Ack
>>>>>>
>>>>>> On Thu, Nov 19, 2009 at 8:38 AM, Zhenyu Zhong <zhongresearch@gmail.com
>>>>>>> wrote:
>>>>>>> Here is the zookeeper leader log. It had an unexpected exception. I
>>>>>> don't
>>>>>>> know why.
>>>>>>>
>>>>>>> http://pastebin.com/m6594ccc
>>>>>>>
>>>>>>> During that time, I don't think the system load is high. Memory usage
>>>>>>> should
>>>>>>> be normal.
>>>>>>>
>>>>>>> Best,
>>>>>>> zhenyu
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Nov 19, 2009 at 11:09 AM, stack <st...@duboce.net> wrote:
>>>>>>>
>>>>>>>> What is in the other zk logs? Why did the leader go away? Memory
>>>>>> issue
>>>>>>> or
>>>>>>>> something? Why'd it shutdown? Thanks for pastebin'ing this stuff.
>>>>>>>> St.Ack
>>>>>>>>
>>>>>>>> On Thu, Nov 19, 2009 at 7:53 AM, Zhenyu Zhong <
>>>>> zhongresearch@gmail.com
>>>>>>>>> wrote:
>>>>>>>>> After digging more on the zookeeper log, I found that there
>>>> seemed
>>>>> to
>>>>>>> be
>>>>>>>> a
>>>>>>>>> socketconnection timeout. Also some warnings indicate that
>>>>> zookeeper
>>>>>>>> server
>>>>>>>>> is not running. Probably I need to understand the mechanism of
>>>>>>> zookeeper
>>>>>>>>> first.
>>>>>>>>>
>>>>>>>>> Please find the zookeeper log around that time.
>>>>>>>>>
>>>>>>>>> http://pastebin.com/m37bb4ad1
>>>>>>>>>
>>>>>>>>> Many thanks
>>>>>>>>> zhenyu
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Nov 19, 2009 at 10:31 AM, Zhenyu Zhong <
>>>>>>> zhongresearch@gmail.com
>>>>>>>>>> wrote:
>>>>>>>>>> Stack,
>>>>>>>>>>
>>>>>>>>>> I am very appreciated. Let me dig into the zookeeper log more.
>>>>>>>>>>
>>>>>>>>>> FYI, I saw the sleeper complained about the delay for the first
>>>>>> time.
>>>>>>>> It
>>>>>>>>>> looks like there are different reasons for the RS
>>>> disconnections.
>>>>>>>>>> 2009-11-19 08:08:32,708 DEBUG org.apache.zookeeper.ClientCnxn:
>>>>> Got
>>>>>>> ping
>>>>>>>>>> response for sessionid:0x42508716bfc0001 after 0ms
>>>>>>>>>> 2009-11-19 08:08:37,595 DEBUG org.apache.zookeeper.ClientCnxn:
>>>>> Got
>>>>>>> ping
>>>>>>>>>> response for sessionid:0x42508716bfc0082 after 0ms
>>>>>>>>>> 2009-11-19 08:08:46,073 DEBUG org.apache.zookeeper.ClientCnxn:
>>>>> Got
>>>>>>> ping
>>>>>>>>>> response for sessionid:0x42508716bfc0001 after 0ms
>>>>>>>>>> 2009-11-19 08:09:39,880 WARN org.apache.zookeeper.ClientCnxn:
>>>>>>> Exception
>>>>>>>>>> closing session 0x42508716bfc0001 to
>>>>>>>> sun.nio.ch.SelectionKeyImpl@4f3a608f
>>>>>>>>>> java.io.IOException: TIMED OUT
>>>>>>>>>> at
>>>>>>>>>>
>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:906)
>>>>>>>>>> 2009-11-19 08:09:39,880 WARN org.apache.zookeeper.ClientCnxn:
>>>>>>> Exception
>>>>>>>>>> closing session 0x42508716bfc0082 to
>>>>>>>> sun.nio.ch.SelectionKeyImpl@14c8875
>>>>>>>>>> java.io.IOException: TIMED OUT
>>>>>>>>>> at
>>>>>>>>>>
>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:906)
>>>>>>>>>> 2009-11-19 08:09:39,880 WARN
>>>>> org.apache.hadoop.hbase.util.Sleeper:
>>>>>> We
>>>>>>>>> slept
>>>>>>>>>> 53556ms, ten times longer than scheduled: 3000
>>>>>>>>>> 2009-11-19 08:09:39,980 WARN
>>>>>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Attempt=1
>>>>>>>>>> org.apache.hadoop.hbase.Leases$LeaseStillHeldException
>>>>>>>>>> at
>>>>>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>>>>>>>>>> Method)
>>>>>>>>>> at
>>>>>>>>>>
>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>>>>>>>>>> at
>>>>>>>>>>
>>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>>>>>>>>>> at
>>>>>>>>> java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>>>>>>>>>> at
>>>>>>>>>>
>>>> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
>>>>>>>>>> at
>>>>>>>>>>
>>>> org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
>>>>>>>>>> at
>>>>>>>>>>
>>>> org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
>>>>>>>>>> at
>>>>>>>>>>
>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:571)
>>>>>>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>>>>>>> 2009-11-19 08:09:39,980 INFO
>>>>>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Got
>>>> ZooKeeper
>>>>>>>> event,
>>>>>>>>>> state: Disconnected, type: None, path: null
>>>>>>>>>> 2009-11-19 08:09:40,530 INFO org.apache.zookeeper.ClientCnxn:
>>>>>>>> Attempting
>>>>>>>>>> connection to server superpyxis0005/192.168.100.119:2181
>>>>>>>>>> 2009-11-19 08:09:40,530 INFO org.apache.zookeeper.ClientCnxn:
>>>>>> Priming
>>>>>>>>>> connection to java.nio.channels.SocketChannel[connected local=/
>>>>>>>>>> 192.168.100.122:40659 remote=superpyxis0005/
>>>> 192.168.100.119:2181
>>>>> ]
>>>>>>>>>>
>>>>>>>>>> Best,
>>>>>>>>>> zhenyu
>>>>>>>>>>
>>>>>>>>>> On Wed, Nov 18, 2009 at 6:52 PM, stack <st...@duboce.net>
>>>> wrote:
>>>>>>>>>>> Patrick Hunt, one of the zk lads, answered with the following:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> According to the following snippet (I cut to just session
>>>>>>>>>>> 0x425035c48210002)
>>>>>>>>>>> the client is getting pings for this session id regularly
>>>> every
>>>>> 6
>>>>>>>>> seconds
>>>>>>>>>>> or
>>>>>>>>>>> so, then after 41,016 there's a 20 second gap (corresponds to
>>>> a
>>>>> 30
>>>>>>>>> second
>>>>>>>>>>> timeout being set during session creation) at which point we
>>>>>>>> disconnect
>>>>>>>>>>> from
>>>>>>>>>>> the server.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> 2009-11-18 12:41:34,788 DEBUG
>>>> org.apache.zookeeper.ClientCnxn:
>>>>>> Got
>>>>>>>>> ping
>>>>>>>>>>>> response for sessionid:0x425035c48210002 after 0ms
>>>>>>>>>>>> 2009-11-18 12:42:01,482 WARN
>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>>>> Exception
>>>>>>>>>>>> closing session 0x425035c48210002 to
>>>>>>>>>>> sun.nio.ch.SelectionKeyImpl@421690ab
>>>>>>>>>>>> java.io.IOException: TIMED OUT
>>>>>>>>>>> the client notifies hbase of the disco, and attempts to
>>>>> reconnect
>>>>>> to
>>>>>>> a
>>>>>>>>>>> server
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Got
>>>>>> ZooKeeper
>>>>>>>>> event,
>>>>>>>>>>>> state: Disconnected, type: None, path: null
>>>>>>>>>>>> 2009-11-18 12:42:01,782 INFO
>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>>>> Attempting
>>>>>>>>>>>> connection to server superpyxis0003/192.168.100.117:2181
>>>>>>>>>>> but the server won't allow the session be be established for
>>>>> some
>>>>>>>> reason
>>>>>>>>>>> (is
>>>>>>>>>>> that the right server:port that's it's trying to connect to?)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> 2009-11-18 12:42:02,182 INFO
>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>> Server
>>>>>>>>>>>> connection successful
>>>>>>>>>>>> 2009-11-18 12:42:02,182 WARN
>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>>>> Exception
>>>>>>>>>>>> closing session 0x425035c48210002 to
>>>>>>>>>>> sun.nio.ch.SelectionKeyImpl@5c07076b
>>>>>>>>>>>> java.io.IOException: Read error rc = -1
>>>>>>>>> java.nio.DirectByteBuffer[pos=0
>>>>>>>>>>>> lim=4 cap=4]
>>>>>>>>>>> The "read error rc = -1" means that the client read from the
>>>>>> socket
>>>>>>>> and
>>>>>>>>>>> got
>>>>>>>>>>> EOS. But it's not a session expiration afaict (and should not
>>>> be
>>>>>> at
>>>>>>>> this
>>>>>>>>>>> point in time).
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> You need to look at the ZK server logs for this period in time
>>>>>> (for
>>>>>>>> this
>>>>>>>>>>> server superpyxis0003:2181, and superpyxis0001:2181, the
>>>> client
>>>>>>> tries
>>>>>>>> to
>>>>>>>>>>> connect to both of these). There should be some indication
>>>> there
>>>>>> why
>>>>>>>> the
>>>>>>>>>>> server is closing down the connection and not allowing the
>>>>> session
>>>>>>> to
>>>>>>>> be
>>>>>>>>>>> re-established.
>>>>>>>>>>>
>>>>>>>>>>> One thought - by default we limit clients to a max of 10
>>>>>> connections
>>>>>>>>> from
>>>>>>>>>>> a
>>>>>>>>>>> particular host to the server. If you exceed this limit the zk
>>>>>>> server
>>>>>>>>>>> (it's
>>>>>>>>>>> per server though, not per cluster, so perhaps this is not it)
>>>>>> will
>>>>>>>> not
>>>>>>>>>>> allow any additional connections from the client host (you
>>>>> can't
>>>>>>>> exceed
>>>>>>>>>>> 10
>>>>>>>>>>> on a server). Could this be it? i.e. the user is running a
>>>>> number
>>>>>> of
>>>>>>>> ZK
>>>>>>>>>>> clients (say hbase processes) from the same host? Regardless,
>>>>> get
>>>>>>> the
>>>>>>>>>>> server
>>>>>>>>>>> log for this time period and it should be more clear.
>>>>>>>>>>>
>>>>>>>>>>> Patrick
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Nov 18, 2009 at 11:18 AM, stack <st...@duboce.net>
>>>>> wrote:
>>>>>>>>>>>> And running stat on each of your zookeeper nodes tells you
>>>>> what
>>>>>>>> about
>>>>>>>>>>>> averages for connect-times?
>>>>>>>>>>>>
>>>>>>>>>>>> Its odd that there is a ping just 20 seconds beforehand and
>>>>> you
>>>>>>> have
>>>>>>>>> set
>>>>>>>>>>>> the zk session timeout at 60 seconds (or ten minutes?). Let
>>>>> me
>>>>>>> ask
>>>>>>>>> the
>>>>>>>>>>> zk
>>>>>>>>>>>> lads about it.
>>>>>>>>>>>>
>>>>>>>>>>>> If you look in the GC log for around this time, do you see
>>>>>>> anything?
>>>>>>>>>>>> St.Ack
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Nov 18, 2009 at 7:32 AM, Zhenyu Zhong <
>>>>>>>>> zhongresearch@gmail.com
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> After running more experiment, it seems that the
>>>>> disconnection
>>>>>> is
>>>>>>>> not
>>>>>>>>>>>>> related to the heavy load job, because the load average is
>>>>> low,
>>>>>>> the
>>>>>>>>>>> diskio
>>>>>>>>>>>>> is normal, memory Heap was not reached, also the virtual
>>>>> memory
>>>>>>>> stats
>>>>>>>>>>>>> shows
>>>>>>>>>>>>> that no swappings. However, the disconnection still
>>>> happens.
>>>>>>>>>>>>> It looks like this time it pauses for 20 seconds. No idea
>>>> why
>>>>>>>>>>> regionserver
>>>>>>>>>>>>> disconnected.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Any other suggestions please?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2009-11-18 12:41:00,956 DEBUG
>>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>> Got
>>>>>>>>> ping
>>>>>>>>>>>>> response for sessionid:0x125035c495d02bb after 0ms
>>>>>>>>>>>>> 2009-11-18 12:41:08,074 DEBUG
>>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>> Got
>>>>>>>>> ping
>>>>>>>>>>>>> response for sessionid:0x425035c48210002 after 0ms
>>>>>>>>>>>>> 2009-11-18 12:41:14,302 DEBUG
>>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>> Got
>>>>>>>>> ping
>>>>>>>>>>>>> response for sessionid:0x125035c495d02bb after 0ms
>>>>>>>>>>>>> 2009-11-18 12:41:21,433 DEBUG
>>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>> Got
>>>>>>>>> ping
>>>>>>>>>>>>> response for sessionid:0x425035c48210002 after 0ms
>>>>>>>>>>>>> 2009-11-18 12:41:27,656 DEBUG
>>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>> Got
>>>>>>>>> ping
>>>>>>>>>>>>> response for sessionid:0x125035c495d02bb after 0ms
>>>>>>>>>>>>> 2009-11-18 12:41:34,788 DEBUG
>>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>> Got
>>>>>>>>> ping
>>>>>>>>>>>>> response for sessionid:0x425035c48210002 after 0ms
>>>>>>>>>>>>> 2009-11-18 12:41:41,016 DEBUG
>>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>> Got
>>>>>>>>> ping
>>>>>>>>>>>>> response for sessionid:0x125035c495d02bb after 1ms
>>>>>>>>>>>>> 2009-11-18 12:42:01,482 WARN
>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>>>> Exception
>>>>>>>>>>>>> closing session 0x425035c48210002 to
>>>>>>>>>>> sun.nio.ch.SelectionKeyImpl@421690ab
>>>>>>>>>>>>> java.io.IOException: TIMED OUT
>>>>>>>>>>>>> at
>>>>>>>>>>>>>
>>>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:906)
>>>>>>>>>>>>> 2009-11-18 12:42:01,582 INFO
>>>>>>>>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Got
>>>>>> ZooKeeper
>>>>>>>>>>> event,
>>>>>>>>>>>>> state: Disconnected, type: None, path: null
>>>>>>>>>>>>> 2009-11-18 12:42:01,782 INFO
>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>>>>>> Attempting
>>>>>>>>>>>>> connection to server superpyxis0003/192.168.100.117:2181
>>>>>>>>>>>>> 2009-11-18 12:42:01,782 INFO
>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>>> Priming
>>>>>>>>>>>>> connection to java.nio.channels.SocketChannel[connected
>>>>> local=/
>>>>>>>>>>>>> 192.168.100.132:46610 remote=superpyxis0003/
>>>>>> 192.168.100.117:2181
>>>>>>> ]
>>>>>>>>>>>>> 2009-11-18 12:42:01,782 INFO
>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>>> Server
>>>>>>>>>>>>> connection successful
>>>>>>>>>>>>> 2009-11-18 12:42:01,782 WARN
>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>>>> Exception
>>>>>>>>>>>>> closing session 0x425035c48210002 to
>>>>>>>>>>> sun.nio.ch.SelectionKeyImpl@56dc6fac
>>>>>>>>>>>>> java.io.IOException: Read error rc = -1
>>>>>>>>> java.nio.DirectByteBuffer[pos=0
>>>>>>>>>>>>> lim=4 cap=4]
>>>>>>>>>>>>> at
>>>>>>>>>>>>>
>>>> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
>>>>>>>>>>>>> at
>>>>>>>>>>>>>
>>>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
>>>>>>>>>>>>> 2009-11-18 12:42:01,782 WARN
>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>>>> Ignoring
>>>>>>>>>>>>> exception during shutdown input
>>>>>>>>>>>>> java.net.SocketException: Transport endpoint is not
>>>> connected
>>>>>>>>>>>>> at sun.nio.ch.SocketChannelImpl.shutdown(Native
>>>>> Method)
>>>>>>>>>>>>> at
>>>>>>>>>>>>>
>>>>>> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
>>>>>>>>>>>>> at
>>>>>>>>>>> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>>>>>>>>>>>>> at
>>>>>>>>>>>>>
>>>>>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
>>>>>>>>>>>>> at
>>>>>>>>>>>>>
>>>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
>>>>>>>>>>>>> 2009-11-18 12:42:01,782 WARN
>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>>>> Ignoring
>>>>>>>>>>>>> exception during shutdown output
>>>>>>>>>>>>> java.net.SocketException: Transport endpoint is not
>>>> connected
>>>>>>>>>>>>> at sun.nio.ch.SocketChannelImpl.shutdown(Native
>>>>> Method)
>>>>>>>>>>>>> at
>>>>>>>>>>>>>
>>>>>> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
>>>>>>>>>>>>> at
>>>> sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
>>>>>>>>>>>>> at
>>>>>>>>>>>>>
>>>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
>>>>>>>>>>>>> at
>>>>>>>>>>>>>
>>>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
>>>>>>>>>>>>> 2009-11-18 12:42:02,182 INFO
>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>>>>>> Attempting
>>>>>>>>>>>>> connection to server superpyxis0001/192.168.100.115:2181
>>>>>>>>>>>>> 2009-11-18 12:42:02,182 INFO
>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>>> Priming
>>>>>>>>>>>>> connection to java.nio.channels.SocketChannel[connected
>>>>> local=/
>>>>>>>>>>>>> 192.168.100.132:36197 remote=superpyxis0001/
>>>>>> 192.168.100.115:2181
>>>>>>> ]
>>>>>>>>>>>>> 2009-11-18 12:42:02,182 INFO
>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>>> Server
>>>>>>>>>>>>> connection successful
>>>>>>>>>>>>> 2009-11-18 12:42:02,182 WARN
>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>>>> Exception
>>>>>>>>>>>>> closing session 0x425035c48210002 to
>>>>>>>>>>> sun.nio.ch.SelectionKeyImpl@5c07076b
>>>>>>>>>>>>> java.io.IOException: Read error rc = -1
>>>>>>>>> java.nio.DirectByteBuffer[pos=0
>>>>>>>>>>>>> lim=4 cap=4]
>>>>>>>>>>>>> at
>>>>>>>>>>>>>
>>>> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
>>>>>>>>>>>>> at
>>>>>>>>>>>>>
>>>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
>>>>>>>>>>>>> 2009-11-18 12:42:02,182 WARN
>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>>>> Ignoring
>>>>>>>>>>>>> exception during shutdown input
>>>>>>>>>>>>> java.net.SocketException: Transport endpoint is not
>>>> connected
>>>>>>>>>>>>> at sun.nio.ch.SocketChannelImpl.shutdown(Native
>> ...
>>
>> [Message clipped]
Re: regionserver disconnection
Posted by Ryan Rawson <ry...@gmail.com>.
I have given mini-talks about this at HUG and generally ranted about
GC to many people, talking about fragmenting vs compacting GC, and how
CMS-mode failures can pause you for a long time, the more Xmx the
worse it is.
Generally in the hadoop community, real time is not something that is
really done, thus we are at the "forefront" of high memory high
performance low latency Java. I cheerlead G1 as often as I can.
-ryan
On Mon, Nov 30, 2009 at 3:17 PM, Patrick Hunt <ph...@apache.org> wrote:
> FYI: https://issues.apache.org/jira/browse/ZOOKEEPER-601
>
> Stack/JD/HBaseTeam - have you considered detailing/presenting Sun JVM GC
> limitations you've faced as part of HBase effort? I find this really
> fascinating (4min gc pauses even with "realtime" gc turned on), I doubt many
> people realize things can get this bad, and I think it would be a great
> service to the community if you could detail some of your findings (say at
> the next BAHUG?) You might even tag team it with someone from the Hadoop
> MR/HDFS team, I hear they face similar issues in the name node.
>
> Regards,
>
> Patrick
>
> Zhenyu Zhong wrote:
>>
>> Tons of thanks to Stack and J-D as well as the Zookeeper fellows.
>>
>> Previously we experienced RS session timeouts even we set the
>> zookeeper.session.timeout to 10 minutes.
>> It turned out that the zookeeper has both lower bound and upper bound for
>> the session timeout.
>> It actually relates to the tickerTime, so the session timeout should be no
>> less than 2 times tickertime and no greater than 20 times tickerTime. By
>> default the tickerTime is 2 sec, so the maximum timeout is 40 seconds by
>> default no matter how large the value is set for
>> zookeeper.session.timeout.
>>
>> Due to some unknown reason, (probably system heavy load, GC pauses, etc),
>> 40 seconds is not long enough for us, so we have to extend that value.
>>
>> One way to do that is to up the hbase.zookeeper.property.tickTime. Now
>> with
>> 4000ms set on that value, we haven't seen the RS session TIMEOUT so far.
>>
>> Many thanks to Stack, J-D and Zookeeper fellows.
>>
>> I will keep you posted if the session timeout comes back again or there is
>> anything else causes the RS disconnections.
>>
>> Best,
>> zhenyu
>>
>>
>>
>>
>> On Thu, Nov 19, 2009 at 1:37 PM, stack <st...@duboce.net> wrote:
>>
>>> From the zk fellas, try running your zk cluster at INFO level rather than
>>> at
>>> DEBUG. One supposition is that the shear amount of logging is causing
>>> distress. They are still looking into your logs...
>>>
>>> Thanks for your patience Zhenyu.
>>> St.Ack
>>>
>>>
>>> On Thu, Nov 19, 2009 at 8:57 AM, Zhenyu Zhong <zhongresearch@gmail.com
>>>>
>>>> wrote:
>>>> Yes, I believe so. And I don't know why the leader gave up.
>>>> For this case, the regionserver got TIMEOUT warning message at
>>>> 2009-11-18
>>>> 12:42:01,482
>>>>
>>>>
>>>> Best,
>>>> zhenyu
>>>>
>>>>
>>>>
>>>>
>>>> On Thu, Nov 19, 2009 at 11:43 AM, stack <st...@duboce.net> wrote:
>>>>
>>>>> So, the time at which we see
>>>>>
>>>>> 2009-11-18 12:41:56,782 ERROR
>>>>> org.apache.zookeeper.server.quorum.FollowerHandler: Unexpected
>>>
>>> exception
>>>>>
>>>>> causing shutdown while sock still open
>>>>>
>>>>> ... corresponds to the time at which the regionserver loses is session?
>>>>>
>>>>> St.Ack
>>>>>
>>>>> On Thu, Nov 19, 2009 at 8:38 AM, Zhenyu Zhong <zhongresearch@gmail.com
>>>>>>
>>>>>> wrote:
>>>>>> Here is the zookeeper leader log. It had an unexpected exception. I
>>>>>
>>>>> don't
>>>>>>
>>>>>> know why.
>>>>>>
>>>>>> http://pastebin.com/m6594ccc
>>>>>>
>>>>>> During that time, I don't think the system load is high. Memory usage
>>>>>> should
>>>>>> be normal.
>>>>>>
>>>>>> Best,
>>>>>> zhenyu
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Nov 19, 2009 at 11:09 AM, stack <st...@duboce.net> wrote:
>>>>>>
>>>>>>> What is in the other zk logs? Why did the leader go away? Memory
>>>>>
>>>>> issue
>>>>>>
>>>>>> or
>>>>>>>
>>>>>>> something? Why'd it shutdown? Thanks for pastebin'ing this stuff.
>>>>>>> St.Ack
>>>>>>>
>>>>>>> On Thu, Nov 19, 2009 at 7:53 AM, Zhenyu Zhong <
>>>>
>>>> zhongresearch@gmail.com
>>>>>>>>
>>>>>>>> wrote:
>>>>>>>> After digging more on the zookeeper log, I found that there
>>>
>>> seemed
>>>>
>>>> to
>>>>>>
>>>>>> be
>>>>>>>
>>>>>>> a
>>>>>>>>
>>>>>>>> socketconnection timeout. Also some warnings indicate that
>>>>
>>>> zookeeper
>>>>>>>
>>>>>>> server
>>>>>>>>
>>>>>>>> is not running. Probably I need to understand the mechanism of
>>>>>>
>>>>>> zookeeper
>>>>>>>>
>>>>>>>> first.
>>>>>>>>
>>>>>>>> Please find the zookeeper log around that time.
>>>>>>>>
>>>>>>>> http://pastebin.com/m37bb4ad1
>>>>>>>>
>>>>>>>> Many thanks
>>>>>>>> zhenyu
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Nov 19, 2009 at 10:31 AM, Zhenyu Zhong <
>>>>>>
>>>>>> zhongresearch@gmail.com
>>>>>>>>>
>>>>>>>>> wrote:
>>>>>>>>> Stack,
>>>>>>>>>
>>>>>>>>> I am very appreciated. Let me dig into the zookeeper log more.
>>>>>>>>>
>>>>>>>>> FYI, I saw the sleeper complained about the delay for the first
>>>>>
>>>>> time.
>>>>>>>
>>>>>>> It
>>>>>>>>>
>>>>>>>>> looks like there are different reasons for the RS
>>>
>>> disconnections.
>>>>>>>>>
>>>>>>>>> 2009-11-19 08:08:32,708 DEBUG org.apache.zookeeper.ClientCnxn:
>>>>
>>>> Got
>>>>>>
>>>>>> ping
>>>>>>>>>
>>>>>>>>> response for sessionid:0x42508716bfc0001 after 0ms
>>>>>>>>> 2009-11-19 08:08:37,595 DEBUG org.apache.zookeeper.ClientCnxn:
>>>>
>>>> Got
>>>>>>
>>>>>> ping
>>>>>>>>>
>>>>>>>>> response for sessionid:0x42508716bfc0082 after 0ms
>>>>>>>>> 2009-11-19 08:08:46,073 DEBUG org.apache.zookeeper.ClientCnxn:
>>>>
>>>> Got
>>>>>>
>>>>>> ping
>>>>>>>>>
>>>>>>>>> response for sessionid:0x42508716bfc0001 after 0ms
>>>>>>>>> 2009-11-19 08:09:39,880 WARN org.apache.zookeeper.ClientCnxn:
>>>>>>
>>>>>> Exception
>>>>>>>>>
>>>>>>>>> closing session 0x42508716bfc0001 to
>>>>>>>
>>>>>>> sun.nio.ch.SelectionKeyImpl@4f3a608f
>>>>>>>>>
>>>>>>>>> java.io.IOException: TIMED OUT
>>>>>>>>> at
>>>>>>>>>
>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:906)
>>>>>>>>>
>>>>>>>>> 2009-11-19 08:09:39,880 WARN org.apache.zookeeper.ClientCnxn:
>>>>>>
>>>>>> Exception
>>>>>>>>>
>>>>>>>>> closing session 0x42508716bfc0082 to
>>>>>>>
>>>>>>> sun.nio.ch.SelectionKeyImpl@14c8875
>>>>>>>>>
>>>>>>>>> java.io.IOException: TIMED OUT
>>>>>>>>> at
>>>>>>>>>
>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:906)
>>>>>>>>>
>>>>>>>>> 2009-11-19 08:09:39,880 WARN
>>>>
>>>> org.apache.hadoop.hbase.util.Sleeper:
>>>>>
>>>>> We
>>>>>>>>
>>>>>>>> slept
>>>>>>>>>
>>>>>>>>> 53556ms, ten times longer than scheduled: 3000
>>>>>>>>> 2009-11-19 08:09:39,980 WARN
>>>>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Attempt=1
>>>>>>>>> org.apache.hadoop.hbase.Leases$LeaseStillHeldException
>>>>>>>>> at
>>>>>>>
>>>>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>>>>>>>>>
>>>>>>>>> Method)
>>>>>>>>> at
>>>>>>>>>
>>>
>>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>>
>>>
>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>
>>>>>>>> java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>>
>>>
>>> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>>
>>>
>>> org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>>
>>>
>>> org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>>
>>>
>>> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:571)
>>>>>>>>>
>>>>>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>>>>>> 2009-11-19 08:09:39,980 INFO
>>>>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Got
>>>
>>> ZooKeeper
>>>>>>>
>>>>>>> event,
>>>>>>>>>
>>>>>>>>> state: Disconnected, type: None, path: null
>>>>>>>>> 2009-11-19 08:09:40,530 INFO org.apache.zookeeper.ClientCnxn:
>>>>>>>
>>>>>>> Attempting
>>>>>>>>>
>>>>>>>>> connection to server superpyxis0005/192.168.100.119:2181
>>>>>>>>> 2009-11-19 08:09:40,530 INFO org.apache.zookeeper.ClientCnxn:
>>>>>
>>>>> Priming
>>>>>>>>>
>>>>>>>>> connection to java.nio.channels.SocketChannel[connected local=/
>>>>>>>>> 192.168.100.122:40659 remote=superpyxis0005/
>>>
>>> 192.168.100.119:2181
>>>>
>>>> ]
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>> zhenyu
>>>>>>>>>
>>>>>>>>> On Wed, Nov 18, 2009 at 6:52 PM, stack <st...@duboce.net>
>>>
>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Patrick Hunt, one of the zk lads, answered with the following:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> According to the following snippet (I cut to just session
>>>>>>>>>> 0x425035c48210002)
>>>>>>>>>> the client is getting pings for this session id regularly
>>>
>>> every
>>>>
>>>> 6
>>>>>>>>
>>>>>>>> seconds
>>>>>>>>>>
>>>>>>>>>> or
>>>>>>>>>> so, then after 41,016 there's a 20 second gap (corresponds to
>>>
>>> a
>>>>
>>>> 30
>>>>>>>>
>>>>>>>> second
>>>>>>>>>>
>>>>>>>>>> timeout being set during session creation) at which point we
>>>>>>>
>>>>>>> disconnect
>>>>>>>>>>
>>>>>>>>>> from
>>>>>>>>>> the server.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> 2009-11-18 12:41:34,788 DEBUG
>>>
>>> org.apache.zookeeper.ClientCnxn:
>>>>>
>>>>> Got
>>>>>>>>
>>>>>>>> ping
>>>>>>>>>>>
>>>>>>>>>>> response for sessionid:0x425035c48210002 after 0ms
>>>>>>>>>>> 2009-11-18 12:42:01,482 WARN
>>>
>>> org.apache.zookeeper.ClientCnxn:
>>>>>>>>
>>>>>>>> Exception
>>>>>>>>>>>
>>>>>>>>>>> closing session 0x425035c48210002 to
>>>>>>>>>>
>>>>>>>>>> sun.nio.ch.SelectionKeyImpl@421690ab
>>>>>>>>>>>
>>>>>>>>>>> java.io.IOException: TIMED OUT
>>>>>>>>>>
>>>>>>>>>> the client notifies hbase of the disco, and attempts to
>>>>
>>>> reconnect
>>>>>
>>>>> to
>>>>>>
>>>>>> a
>>>>>>>>>>
>>>>>>>>>> server
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Got
>>>>>
>>>>> ZooKeeper
>>>>>>>>
>>>>>>>> event,
>>>>>>>>>>>
>>>>>>>>>>> state: Disconnected, type: None, path: null
>>>>>>>>>>> 2009-11-18 12:42:01,782 INFO
>>>
>>> org.apache.zookeeper.ClientCnxn:
>>>>>>>>
>>>>>>>> Attempting
>>>>>>>>>>>
>>>>>>>>>>> connection to server superpyxis0003/192.168.100.117:2181
>>>>>>>>>>
>>>>>>>>>> but the server won't allow the session be be established for
>>>>
>>>> some
>>>>>>>
>>>>>>> reason
>>>>>>>>>>
>>>>>>>>>> (is
>>>>>>>>>> that the right server:port that's it's trying to connect to?)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> 2009-11-18 12:42:02,182 INFO
>>>
>>> org.apache.zookeeper.ClientCnxn:
>>>>>>
>>>>>> Server
>>>>>>>>>>>
>>>>>>>>>>> connection successful
>>>>>>>>>>> 2009-11-18 12:42:02,182 WARN
>>>
>>> org.apache.zookeeper.ClientCnxn:
>>>>>>>>
>>>>>>>> Exception
>>>>>>>>>>>
>>>>>>>>>>> closing session 0x425035c48210002 to
>>>>>>>>>>
>>>>>>>>>> sun.nio.ch.SelectionKeyImpl@5c07076b
>>>>>>>>>>>
>>>>>>>>>>> java.io.IOException: Read error rc = -1
>>>>>>>>
>>>>>>>> java.nio.DirectByteBuffer[pos=0
>>>>>>>>>>>
>>>>>>>>>>> lim=4 cap=4]
>>>>>>>>>>
>>>>>>>>>> The "read error rc = -1" means that the client read from the
>>>>>
>>>>> socket
>>>>>>>
>>>>>>> and
>>>>>>>>>>
>>>>>>>>>> got
>>>>>>>>>> EOS. But it's not a session expiration afaict (and should not
>>>
>>> be
>>>>>
>>>>> at
>>>>>>>
>>>>>>> this
>>>>>>>>>>
>>>>>>>>>> point in time).
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> You need to look at the ZK server logs for this period in time
>>>>>
>>>>> (for
>>>>>>>
>>>>>>> this
>>>>>>>>>>
>>>>>>>>>> server superpyxis0003:2181, and superpyxis0001:2181, the
>>>
>>> client
>>>>>>
>>>>>> tries
>>>>>>>
>>>>>>> to
>>>>>>>>>>
>>>>>>>>>> connect to both of these). There should be some indication
>>>
>>> there
>>>>>
>>>>> why
>>>>>>>
>>>>>>> the
>>>>>>>>>>
>>>>>>>>>> server is closing down the connection and not allowing the
>>>>
>>>> session
>>>>>>
>>>>>> to
>>>>>>>
>>>>>>> be
>>>>>>>>>>
>>>>>>>>>> re-established.
>>>>>>>>>>
>>>>>>>>>> One thought - by default we limit clients to a max of 10
>>>>>
>>>>> connections
>>>>>>>>
>>>>>>>> from
>>>>>>>>>>
>>>>>>>>>> a
>>>>>>>>>> particular host to the server. If you exceed this limit the zk
>>>>>>
>>>>>> server
>>>>>>>>>>
>>>>>>>>>> (it's
>>>>>>>>>> per server though, not per cluster, so perhaps this is not it)
>>>>>
>>>>> will
>>>>>>>
>>>>>>> not
>>>>>>>>>>
>>>>>>>>>> allow any additional connections from the client host (you
>>>>
>>>> can't
>>>>>>>
>>>>>>> exceed
>>>>>>>>>>
>>>>>>>>>> 10
>>>>>>>>>> on a server). Could this be it? i.e. the user is running a
>>>>
>>>> number
>>>>>
>>>>> of
>>>>>>>
>>>>>>> ZK
>>>>>>>>>>
>>>>>>>>>> clients (say hbase processes) from the same host? Regardless,
>>>>
>>>> get
>>>>>>
>>>>>> the
>>>>>>>>>>
>>>>>>>>>> server
>>>>>>>>>> log for this time period and it should be more clear.
>>>>>>>>>>
>>>>>>>>>> Patrick
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, Nov 18, 2009 at 11:18 AM, stack <st...@duboce.net>
>>>>
>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> And running stat on each of your zookeeper nodes tells you
>>>>
>>>> what
>>>>>>>
>>>>>>> about
>>>>>>>>>>>
>>>>>>>>>>> averages for connect-times?
>>>>>>>>>>>
>>>>>>>>>>> Its odd that there is a ping just 20 seconds beforehand and
>>>>
>>>> you
>>>>>>
>>>>>> have
>>>>>>>>
>>>>>>>> set
>>>>>>>>>>>
>>>>>>>>>>> the zk session timeout at 60 seconds (or ten minutes?). Let
>>>>
>>>> me
>>>>>>
>>>>>> ask
>>>>>>>>
>>>>>>>> the
>>>>>>>>>>
>>>>>>>>>> zk
>>>>>>>>>>>
>>>>>>>>>>> lads about it.
>>>>>>>>>>>
>>>>>>>>>>> If you look in the GC log for around this time, do you see
>>>>>>
>>>>>> anything?
>>>>>>>>>>>
>>>>>>>>>>> St.Ack
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Nov 18, 2009 at 7:32 AM, Zhenyu Zhong <
>>>>>>>>
>>>>>>>> zhongresearch@gmail.com
>>>>>>>>>>>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> After running more experiment, it seems that the
>>>>
>>>> disconnection
>>>>>
>>>>> is
>>>>>>>
>>>>>>> not
>>>>>>>>>>>>
>>>>>>>>>>>> related to the heavy load job, because the load average is
>>>>
>>>> low,
>>>>>>
>>>>>> the
>>>>>>>>>>
>>>>>>>>>> diskio
>>>>>>>>>>>>
>>>>>>>>>>>> is normal, memory Heap was not reached, also the virtual
>>>>
>>>> memory
>>>>>>>
>>>>>>> stats
>>>>>>>>>>>>
>>>>>>>>>>>> shows
>>>>>>>>>>>> that no swappings. However, the disconnection still
>>>
>>> happens.
>>>>>>>>>>>>
>>>>>>>>>>>> It looks like this time it pauses for 20 seconds. No idea
>>>
>>> why
>>>>>>>>>>
>>>>>>>>>> regionserver
>>>>>>>>>>>>
>>>>>>>>>>>> disconnected.
>>>>>>>>>>>>
>>>>>>>>>>>> Any other suggestions please?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> 2009-11-18 12:41:00,956 DEBUG
>>>>
>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>
>>>>>> Got
>>>>>>>>
>>>>>>>> ping
>>>>>>>>>>>>
>>>>>>>>>>>> response for sessionid:0x125035c495d02bb after 0ms
>>>>>>>>>>>> 2009-11-18 12:41:08,074 DEBUG
>>>>
>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>
>>>>>> Got
>>>>>>>>
>>>>>>>> ping
>>>>>>>>>>>>
>>>>>>>>>>>> response for sessionid:0x425035c48210002 after 0ms
>>>>>>>>>>>> 2009-11-18 12:41:14,302 DEBUG
>>>>
>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>
>>>>>> Got
>>>>>>>>
>>>>>>>> ping
>>>>>>>>>>>>
>>>>>>>>>>>> response for sessionid:0x125035c495d02bb after 0ms
>>>>>>>>>>>> 2009-11-18 12:41:21,433 DEBUG
>>>>
>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>
>>>>>> Got
>>>>>>>>
>>>>>>>> ping
>>>>>>>>>>>>
>>>>>>>>>>>> response for sessionid:0x425035c48210002 after 0ms
>>>>>>>>>>>> 2009-11-18 12:41:27,656 DEBUG
>>>>
>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>
>>>>>> Got
>>>>>>>>
>>>>>>>> ping
>>>>>>>>>>>>
>>>>>>>>>>>> response for sessionid:0x125035c495d02bb after 0ms
>>>>>>>>>>>> 2009-11-18 12:41:34,788 DEBUG
>>>>
>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>
>>>>>> Got
>>>>>>>>
>>>>>>>> ping
>>>>>>>>>>>>
>>>>>>>>>>>> response for sessionid:0x425035c48210002 after 0ms
>>>>>>>>>>>> 2009-11-18 12:41:41,016 DEBUG
>>>>
>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>
>>>>>> Got
>>>>>>>>
>>>>>>>> ping
>>>>>>>>>>>>
>>>>>>>>>>>> response for sessionid:0x125035c495d02bb after 1ms
>>>>>>>>>>>> 2009-11-18 12:42:01,482 WARN
>>>
>>> org.apache.zookeeper.ClientCnxn:
>>>>>>>>
>>>>>>>> Exception
>>>>>>>>>>>>
>>>>>>>>>>>> closing session 0x425035c48210002 to
>>>>>>>>>>
>>>>>>>>>> sun.nio.ch.SelectionKeyImpl@421690ab
>>>>>>>>>>>>
>>>>>>>>>>>> java.io.IOException: TIMED OUT
>>>>>>>>>>>> at
>>>>>>>>>>>>
>>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:906)
>>>>>>>>>>>>
>>>>>>>>>>>> 2009-11-18 12:42:01,582 INFO
>>>>>>>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Got
>>>>>
>>>>> ZooKeeper
>>>>>>>>>>
>>>>>>>>>> event,
>>>>>>>>>>>>
>>>>>>>>>>>> state: Disconnected, type: None, path: null
>>>>>>>>>>>> 2009-11-18 12:42:01,782 INFO
>>>
>>> org.apache.zookeeper.ClientCnxn:
>>>>>>>>>>
>>>>>>>>>> Attempting
>>>>>>>>>>>>
>>>>>>>>>>>> connection to server superpyxis0003/192.168.100.117:2181
>>>>>>>>>>>> 2009-11-18 12:42:01,782 INFO
>>>
>>> org.apache.zookeeper.ClientCnxn:
>>>>>>>
>>>>>>> Priming
>>>>>>>>>>>>
>>>>>>>>>>>> connection to java.nio.channels.SocketChannel[connected
>>>>
>>>> local=/
>>>>>>>>>>>>
>>>>>>>>>>>> 192.168.100.132:46610 remote=superpyxis0003/
>>>>>
>>>>> 192.168.100.117:2181
>>>>>>
>>>>>> ]
>>>>>>>>>>>>
>>>>>>>>>>>> 2009-11-18 12:42:01,782 INFO
>>>
>>> org.apache.zookeeper.ClientCnxn:
>>>>>>>
>>>>>>> Server
>>>>>>>>>>>>
>>>>>>>>>>>> connection successful
>>>>>>>>>>>> 2009-11-18 12:42:01,782 WARN
>>>
>>> org.apache.zookeeper.ClientCnxn:
>>>>>>>>
>>>>>>>> Exception
>>>>>>>>>>>>
>>>>>>>>>>>> closing session 0x425035c48210002 to
>>>>>>>>>>
>>>>>>>>>> sun.nio.ch.SelectionKeyImpl@56dc6fac
>>>>>>>>>>>>
>>>>>>>>>>>> java.io.IOException: Read error rc = -1
>>>>>>>>
>>>>>>>> java.nio.DirectByteBuffer[pos=0
>>>>>>>>>>>>
>>>>>>>>>>>> lim=4 cap=4]
>>>>>>>>>>>> at
>>>>>>>>>>>>
>>> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
>>>>>>>>>>>>
>>>>>>>>>>>> at
>>>>>>>>>>>>
>>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
>>>>>>>>>>>>
>>>>>>>>>>>> 2009-11-18 12:42:01,782 WARN
>>>
>>> org.apache.zookeeper.ClientCnxn:
>>>>>>>>
>>>>>>>> Ignoring
>>>>>>>>>>>>
>>>>>>>>>>>> exception during shutdown input
>>>>>>>>>>>> java.net.SocketException: Transport endpoint is not
>>>
>>> connected
>>>>>>>>>>>>
>>>>>>>>>>>> at sun.nio.ch.SocketChannelImpl.shutdown(Native
>>>>
>>>> Method)
>>>>>>>>>>>>
>>>>>>>>>>>> at
>>>>>>>>>>>>
>>>>> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
>>>>>>>>>>>>
>>>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>>>>>>>>>>>>
>>>>>>>>>>>> at
>>>>>>>>>>>>
>>>>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
>>>>>>>>>>>>
>>>>>>>>>>>> at
>>>>>>>>>>>>
>>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
>>>>>>>>>>>>
>>>>>>>>>>>> 2009-11-18 12:42:01,782 WARN
>>>
>>> org.apache.zookeeper.ClientCnxn:
>>>>>>>>
>>>>>>>> Ignoring
>>>>>>>>>>>>
>>>>>>>>>>>> exception during shutdown output
>>>>>>>>>>>> java.net.SocketException: Transport endpoint is not
>>>
>>> connected
>>>>>>>>>>>>
>>>>>>>>>>>> at sun.nio.ch.SocketChannelImpl.shutdown(Native
>>>>
>>>> Method)
>>>>>>>>>>>>
>>>>>>>>>>>> at
>>>>>>>>>>>>
>>>>> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
>>>>>>>>>>>>
>>>>>>>>>>>> at
>>>
>>> sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
>>>>>>>>>>>>
>>>>>>>>>>>> at
>>>>>>>>>>>>
>>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
>>>>>>>>>>>>
>>>>>>>>>>>> at
>>>>>>>>>>>>
>>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
>>>>>>>>>>>>
>>>>>>>>>>>> 2009-11-18 12:42:02,182 INFO
>>>
>>> org.apache.zookeeper.ClientCnxn:
>>>>>>>>>>
>>>>>>>>>> Attempting
>>>>>>>>>>>>
>>>>>>>>>>>> connection to server superpyxis0001/192.168.100.115:2181
>>>>>>>>>>>> 2009-11-18 12:42:02,182 INFO
>>>
>>> org.apache.zookeeper.ClientCnxn:
>>>>>>>
>>>>>>> Priming
>>>>>>>>>>>>
>>>>>>>>>>>> connection to java.nio.channels.SocketChannel[connected
>>>>
>>>> local=/
>>>>>>>>>>>>
>>>>>>>>>>>> 192.168.100.132:36197 remote=superpyxis0001/
>>>>>
>>>>> 192.168.100.115:2181
>>>>>>
>>>>>> ]
>>>>>>>>>>>>
>>>>>>>>>>>> 2009-11-18 12:42:02,182 INFO
>>>
>>> org.apache.zookeeper.ClientCnxn:
>>>>>>>
>>>>>>> Server
>>>>>>>>>>>>
>>>>>>>>>>>> connection successful
>>>>>>>>>>>> 2009-11-18 12:42:02,182 WARN
>>>
>>> org.apache.zookeeper.ClientCnxn:
>>>>>>>>
>>>>>>>> Exception
>>>>>>>>>>>>
>>>>>>>>>>>> closing session 0x425035c48210002 to
>>>>>>>>>>
>>>>>>>>>> sun.nio.ch.SelectionKeyImpl@5c07076b
>>>>>>>>>>>>
>>>>>>>>>>>> java.io.IOException: Read error rc = -1
>>>>>>>>
>>>>>>>> java.nio.DirectByteBuffer[pos=0
>>>>>>>>>>>>
>>>>>>>>>>>> lim=4 cap=4]
>>>>>>>>>>>> at
>>>>>>>>>>>>
>>> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
>>>>>>>>>>>>
>>>>>>>>>>>> at
>>>>>>>>>>>>
>>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
>>>>>>>>>>>>
>>>>>>>>>>>> 2009-11-18 12:42:02,182 WARN
>>>
>>> org.apache.zookeeper.ClientCnxn:
>>>>>>>>
>>>>>>>> Ignoring
>>>>>>>>>>>>
>>>>>>>>>>>> exception during shutdown input
>>>>>>>>>>>> java.net.SocketException: Transport endpoint is not
>>>
>>> connected
>>>>>>>>>>>>
>>>>>>>>>>>> at sun.nio.ch.SocketChannelImpl.shutdown(Native
>
> ...
>
> [Message clipped]
Re: regionserver disconnection
Posted by Patrick Hunt <ph...@apache.org>.
What pages do you have that I could look at re the gc tuning you mention?
Having a concrete set of GC issues/details that we can point people to
would be very helpful IMO - both to help users but also to highlight the
issue from the perspective that Sun really needs to do a better job
here. If we are pushing the boundaries then perhaps Sun might be
interested to work more closely with us (pipe dream?). My experience
talking with non-java folks these days (c/c++, usually where I justify
ZK in java) is that the poor GC performance is one of the few argument
strongholds they have left. I'd love to help eliminate that, but today
they have a point.
I will put you in contact with someone on the HDFS team re NN gc.
Patrick
stack wrote:
> I suppose up to this I thought it a given for any java application that
> wants to do realtime whether a webserver or search application but yeah, we
> should do more to highlight the import of GC tuning especially when failure
> to do so can be relatively catastrophic (A RegionServer self-shutting itself
> down). Ryan in particular has been doing a bunch of talking up of the topic
> (He did our performance tuning wiki page too). We could start up a list of
> use cases and the tunings that helped alleviate GC woes for a particular
> cluster profile and loading (So we'd have something to present at BAHUG? Do
> you know who we might talk to regards pauses in the MR/HDFS team Patrick?
> We were introduced to the NameNode Tuner once... we should talk to him
> again). It does seem to be a problem where one tuning does not suit all
> deploys.
>
> Regards Zhenyu's case, there is still work to do IMO. What I saw in his
> logs was a failed promotion from parnew, something that could be helped
> starting CMS collection earlier (among other things). Hes also still on an
> older version of the JVM. While things are not timing out at the moment,
> IMO its still 'broke' if it has such long pauses (Zhenyu, in your GC logs,
> are you seeing 4 minutes pause?). Ryan would argue these are inevitable
> with CMS -- but at least in the one case that I saw some twiddling would
> seem to help.
>
> Thanks Patrick,
> St.Ack
>
Re: regionserver disconnection
Posted by Zhenyu Zhong <zh...@gmail.com>.
The java vm version I am running on is still 1.6.0_11. We are scheduling an
upgrade soon.
The command we use to start the RegionServer is:
/usr/local/jdk1.6.0_11/bin/java -Xmx4096m -XX:+HeapDumpOnOutOfMemoryError
-XX:+UseConcMarkSweepGC -XX:+AggressiveOpts -XX:+UseFastAccessorMethods
-XX:+PrintTenuringDistribution -XX:MaxTenuringThreshold=6
-XX:SurvivorRatio=6 -XX:CMSInitiatingOccupancyFraction=60
-XX:+UseCMSInitiatingOccupancyOnly -XX:+CMSParallelRemarkEnabled
-XX:+DisableExplicitGC -verbose:gc -XX:+PrintGCDetails
-XX:+PrintGCTimeStamps -Xloggc:/data/hbase_logs/gc-hbase.log
-XX:ErrorFile=/data/hbase_logs/java_err.log -Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.password.file=/code/hbase-0.20.1/conf/jmxremote.password
-Dcom.sun.management.jmxremote.access.file=/code/hbase-0.20.1/conf/jmxremote.access
-Dcom.sun.management.jmxremote.port=10102 -Dhbase.log.dir=/data/hbase_logs
-Dhbase.log.file=hbase-hadoop-admin-regionserver-superpyxis0006.log
-Dhbase.home.dir=/code/hbase-0.20.1 -Dhbase.id.str=hadoop-admin
-Dhbase.root.logger=INFO,DRFA
-Djava.library.path=/code/hbase-0.20.1/lib/native/Linux-amd64-64 -classpath
/code/hbase-0.20.1/bin/../conf:/usr/local/jdk1.6.0_11/lib/tools.jar:/code/hbase-0.20.1:/code/hbase-0.20.1/hbase-0.20.1.jar:/code/hbase-0.20.1/hbase-0.20.1-test.jar:/code/hbase-0.20.1/lib/AgileJSON-2009-03-30.jar:/code/hbase-0.20.1/lib/commons-cli-2.0-SNAPSHOT.jar:/code/hbase-0.20.1/lib/commons-el-from-jetty-5.1.4.jar:/code/hbase-0.20.1/lib/commons-httpclient-3.0.1.jar:/code/hbase-0.20.1/lib/commons-logging-1.0.4.jar:/code/hbase-0.20.1/lib/commons-logging-api-1.0.4.jar:/code/hbase-0.20.1/lib/commons-math-1.1.jar:/code/hbase-0.20.1/lib/hadoop-0.20.1-hdfs127-core.jar:/code/hbase-0.20.1/lib/hadoop-0.20.1-test.jar:/code/hbase-0.20.1/lib/jasper-compiler-5.5.12.jar:/code/hbase-0.20.1/lib/jasper-runtime-5.5.12.jar:/code/hbase-0.20.1/lib/jetty-6.1.14.jar:/code/hbase-0.20.1/lib/jetty-util-6.1.14.jar:/code/hbase-0.20.1/lib/jruby-complete-1.2.0.jar:/code/hbase-0.20.1/lib/json.jar:/code/hbase-0.20.1/lib/junit-3.8.1.jar:/code/hbase-0.20.1/lib/libthrift-r771587.jar:/code/hbase-0.20.1/lib/log4j-1.2.15.jar:/code/hbase-0.20.1/lib/lucene-core-2.2.0.jar:/code/hbase-0.20.1/lib/servlet-api-2.5-6.1.14.jar:/code/hbase-0.20.1/lib/xmlenc-0.52.jar:/code/hbase-0.20.1/lib/zookeeper-3.2.1.jar:/code/hbase-0.20.1/lib/jsp-2.1/jsp-2.1.jar:/code/hbase-0.20.1/lib/jsp-2.1/jsp-api-2.1.jar:/code/hadoop-0.20.1/conf
org.apache.hadoop.hbase.regionserver.HRegionServer start
Please feel free to comment.
Best,
zhenyu
On Tue, Dec 1, 2009 at 12:22 PM, Patrick Hunt <ph...@apache.org> wrote:
> Interesting, remind me, what is your current status of:
>
> java vm version?
>
> options you are providing to the JVM on startup (-XX -Xmx and the like - if
> you could provide the exact command line you use to start the jvm that would
> be nice to see)
>
> FYI: I've seen issues with use of incremental gc prior to 1.6.0_17 (in
> particular jvm crashes), I haven't seen any issues with this latest version
> (yet). This was on 64bit linux.
>
> Patrick
>
>
> Zhenyu Zhong wrote:
>
>> So far I have been using gchisto to view the gc-log.
>> In my last RS disconnection, I saw a total GC about 457 seconds. But
>> individually, the max is 1340 ms, min is 0.527ms, avg is 48ms.
>>
>> The RS disconnection might be due to other reasons. I think J-D has been
>> digging that.
>>
>> thanks
>> zhenyu
>>
>>
>>
>> On Mon, Nov 30, 2009 at 9:22 PM, stack <st...@duboce.net> wrote:
>>
>> I suppose up to this I thought it a given for any java application that
>>> wants to do realtime whether a webserver or search application but yeah,
>>> we
>>> should do more to highlight the import of GC tuning especially when
>>> failure
>>> to do so can be relatively catastrophic (A RegionServer self-shutting
>>> itself
>>> down). Ryan in particular has been doing a bunch of talking up of the
>>> topic
>>> (He did our performance tuning wiki page too). We could start up a list
>>> of
>>> use cases and the tunings that helped alleviate GC woes for a particular
>>> cluster profile and loading (So we'd have something to present at BAHUG?
>>> Do
>>> you know who we might talk to regards pauses in the MR/HDFS team Patrick?
>>> We were introduced to the NameNode Tuner once... we should talk to him
>>> again). It does seem to be a problem where one tuning does not suit all
>>> deploys.
>>>
>>> Regards Zhenyu's case, there is still work to do IMO. What I saw in his
>>> logs was a failed promotion from parnew, something that could be helped
>>> starting CMS collection earlier (among other things). Hes also still on
>>> an
>>> older version of the JVM. While things are not timing out at the
>>> moment,
>>> IMO its still 'broke' if it has such long pauses (Zhenyu, in your GC
>>> logs,
>>> are you seeing 4 minutes pause?). Ryan would argue these are inevitable
>>> with CMS -- but at least in the one case that I saw some twiddling would
>>> seem to help.
>>>
>>> Thanks Patrick,
>>> St.Ack
>>>
>>>
>>
Re: regionserver disconnection
Posted by Patrick Hunt <ph...@apache.org>.
Interesting, remind me, what is your current status of:
java vm version?
options you are providing to the JVM on startup (-XX -Xmx and the like -
if you could provide the exact command line you use to start the jvm
that would be nice to see)
FYI: I've seen issues with use of incremental gc prior to 1.6.0_17 (in
particular jvm crashes), I haven't seen any issues with this latest
version (yet). This was on 64bit linux.
Patrick
Zhenyu Zhong wrote:
> So far I have been using gchisto to view the gc-log.
> In my last RS disconnection, I saw a total GC about 457 seconds. But
> individually, the max is 1340 ms, min is 0.527ms, avg is 48ms.
>
> The RS disconnection might be due to other reasons. I think J-D has been
> digging that.
>
> thanks
> zhenyu
>
>
>
> On Mon, Nov 30, 2009 at 9:22 PM, stack <st...@duboce.net> wrote:
>
>> I suppose up to this I thought it a given for any java application that
>> wants to do realtime whether a webserver or search application but yeah, we
>> should do more to highlight the import of GC tuning especially when failure
>> to do so can be relatively catastrophic (A RegionServer self-shutting
>> itself
>> down). Ryan in particular has been doing a bunch of talking up of the
>> topic
>> (He did our performance tuning wiki page too). We could start up a list
>> of
>> use cases and the tunings that helped alleviate GC woes for a particular
>> cluster profile and loading (So we'd have something to present at BAHUG?
>> Do
>> you know who we might talk to regards pauses in the MR/HDFS team Patrick?
>> We were introduced to the NameNode Tuner once... we should talk to him
>> again). It does seem to be a problem where one tuning does not suit all
>> deploys.
>>
>> Regards Zhenyu's case, there is still work to do IMO. What I saw in his
>> logs was a failed promotion from parnew, something that could be helped
>> starting CMS collection earlier (among other things). Hes also still on an
>> older version of the JVM. While things are not timing out at the moment,
>> IMO its still 'broke' if it has such long pauses (Zhenyu, in your GC logs,
>> are you seeing 4 minutes pause?). Ryan would argue these are inevitable
>> with CMS -- but at least in the one case that I saw some twiddling would
>> seem to help.
>>
>> Thanks Patrick,
>> St.Ack
>>
>
Re: regionserver disconnection
Posted by Jean-Daniel Cryans <jd...@apache.org>.
The issue you saw is now HBASE-2022.
J-D
On Tue, Dec 1, 2009 at 7:50 AM, Zhenyu Zhong <zh...@gmail.com> wrote:
> So far I have been using gchisto to view the gc-log.
> In my last RS disconnection, I saw a total GC about 457 seconds. But
> individually, the max is 1340 ms, min is 0.527ms, avg is 48ms.
>
> The RS disconnection might be due to other reasons. I think J-D has been
> digging that.
>
> thanks
> zhenyu
>
>
>
> On Mon, Nov 30, 2009 at 9:22 PM, stack <st...@duboce.net> wrote:
>
>> I suppose up to this I thought it a given for any java application that
>> wants to do realtime whether a webserver or search application but yeah, we
>> should do more to highlight the import of GC tuning especially when failure
>> to do so can be relatively catastrophic (A RegionServer self-shutting
>> itself
>> down). Ryan in particular has been doing a bunch of talking up of the
>> topic
>> (He did our performance tuning wiki page too). We could start up a list
>> of
>> use cases and the tunings that helped alleviate GC woes for a particular
>> cluster profile and loading (So we'd have something to present at BAHUG?
>> Do
>> you know who we might talk to regards pauses in the MR/HDFS team Patrick?
>> We were introduced to the NameNode Tuner once... we should talk to him
>> again). It does seem to be a problem where one tuning does not suit all
>> deploys.
>>
>> Regards Zhenyu's case, there is still work to do IMO. What I saw in his
>> logs was a failed promotion from parnew, something that could be helped
>> starting CMS collection earlier (among other things). Hes also still on an
>> older version of the JVM. While things are not timing out at the moment,
>> IMO its still 'broke' if it has such long pauses (Zhenyu, in your GC logs,
>> are you seeing 4 minutes pause?). Ryan would argue these are inevitable
>> with CMS -- but at least in the one case that I saw some twiddling would
>> seem to help.
>>
>> Thanks Patrick,
>> St.Ack
>>
>
Re: regionserver disconnection
Posted by Zhenyu Zhong <zh...@gmail.com>.
So far I have been using gchisto to view the gc-log.
In my last RS disconnection, I saw a total GC about 457 seconds. But
individually, the max is 1340 ms, min is 0.527ms, avg is 48ms.
The RS disconnection might be due to other reasons. I think J-D has been
digging that.
thanks
zhenyu
On Mon, Nov 30, 2009 at 9:22 PM, stack <st...@duboce.net> wrote:
> I suppose up to this I thought it a given for any java application that
> wants to do realtime whether a webserver or search application but yeah, we
> should do more to highlight the import of GC tuning especially when failure
> to do so can be relatively catastrophic (A RegionServer self-shutting
> itself
> down). Ryan in particular has been doing a bunch of talking up of the
> topic
> (He did our performance tuning wiki page too). We could start up a list
> of
> use cases and the tunings that helped alleviate GC woes for a particular
> cluster profile and loading (So we'd have something to present at BAHUG?
> Do
> you know who we might talk to regards pauses in the MR/HDFS team Patrick?
> We were introduced to the NameNode Tuner once... we should talk to him
> again). It does seem to be a problem where one tuning does not suit all
> deploys.
>
> Regards Zhenyu's case, there is still work to do IMO. What I saw in his
> logs was a failed promotion from parnew, something that could be helped
> starting CMS collection earlier (among other things). Hes also still on an
> older version of the JVM. While things are not timing out at the moment,
> IMO its still 'broke' if it has such long pauses (Zhenyu, in your GC logs,
> are you seeing 4 minutes pause?). Ryan would argue these are inevitable
> with CMS -- but at least in the one case that I saw some twiddling would
> seem to help.
>
> Thanks Patrick,
> St.Ack
>
Re: regionserver disconnection
Posted by stack <st...@duboce.net>.
I suppose up to this I thought it a given for any java application that
wants to do realtime whether a webserver or search application but yeah, we
should do more to highlight the import of GC tuning especially when failure
to do so can be relatively catastrophic (A RegionServer self-shutting itself
down). Ryan in particular has been doing a bunch of talking up of the topic
(He did our performance tuning wiki page too). We could start up a list of
use cases and the tunings that helped alleviate GC woes for a particular
cluster profile and loading (So we'd have something to present at BAHUG? Do
you know who we might talk to regards pauses in the MR/HDFS team Patrick?
We were introduced to the NameNode Tuner once... we should talk to him
again). It does seem to be a problem where one tuning does not suit all
deploys.
Regards Zhenyu's case, there is still work to do IMO. What I saw in his
logs was a failed promotion from parnew, something that could be helped
starting CMS collection earlier (among other things). Hes also still on an
older version of the JVM. While things are not timing out at the moment,
IMO its still 'broke' if it has such long pauses (Zhenyu, in your GC logs,
are you seeing 4 minutes pause?). Ryan would argue these are inevitable
with CMS -- but at least in the one case that I saw some twiddling would
seem to help.
Thanks Patrick,
St.Ack
Re: regionserver disconnection
Posted by Patrick Hunt <ph...@apache.org>.
FYI: https://issues.apache.org/jira/browse/ZOOKEEPER-601
Stack/JD/HBaseTeam - have you considered detailing/presenting Sun JVM GC
limitations you've faced as part of HBase effort? I find this really
fascinating (4min gc pauses even with "realtime" gc turned on), I doubt
many people realize things can get this bad, and I think it would be a
great service to the community if you could detail some of your findings
(say at the next BAHUG?) You might even tag team it with someone from
the Hadoop MR/HDFS team, I hear they face similar issues in the name node.
Regards,
Patrick
Zhenyu Zhong wrote:
> Tons of thanks to Stack and J-D as well as the Zookeeper fellows.
>
> Previously we experienced RS session timeouts even we set the
> zookeeper.session.timeout to 10 minutes.
> It turned out that the zookeeper has both lower bound and upper bound for
> the session timeout.
> It actually relates to the tickerTime, so the session timeout should be no
> less than 2 times tickertime and no greater than 20 times tickerTime. By
> default the tickerTime is 2 sec, so the maximum timeout is 40 seconds by
> default no matter how large the value is set for zookeeper.session.timeout.
>
> Due to some unknown reason, (probably system heavy load, GC pauses, etc),
> 40 seconds is not long enough for us, so we have to extend that value.
>
> One way to do that is to up the hbase.zookeeper.property.tickTime. Now with
> 4000ms set on that value, we haven't seen the RS session TIMEOUT so far.
>
> Many thanks to Stack, J-D and Zookeeper fellows.
>
> I will keep you posted if the session timeout comes back again or there is
> anything else causes the RS disconnections.
>
> Best,
> zhenyu
>
>
>
>
> On Thu, Nov 19, 2009 at 1:37 PM, stack <st...@duboce.net> wrote:
>
>> From the zk fellas, try running your zk cluster at INFO level rather than
>> at
>> DEBUG. One supposition is that the shear amount of logging is causing
>> distress. They are still looking into your logs...
>>
>> Thanks for your patience Zhenyu.
>> St.Ack
>>
>>
>> On Thu, Nov 19, 2009 at 8:57 AM, Zhenyu Zhong <zhongresearch@gmail.com
>>> wrote:
>>> Yes, I believe so. And I don't know why the leader gave up.
>>> For this case, the regionserver got TIMEOUT warning message at 2009-11-18
>>> 12:42:01,482
>>>
>>>
>>> Best,
>>> zhenyu
>>>
>>>
>>>
>>>
>>> On Thu, Nov 19, 2009 at 11:43 AM, stack <st...@duboce.net> wrote:
>>>
>>>> So, the time at which we see
>>>>
>>>> 2009-11-18 12:41:56,782 ERROR
>>>> org.apache.zookeeper.server.quorum.FollowerHandler: Unexpected
>> exception
>>>> causing shutdown while sock still open
>>>>
>>>> ... corresponds to the time at which the regionserver loses is session?
>>>>
>>>> St.Ack
>>>>
>>>> On Thu, Nov 19, 2009 at 8:38 AM, Zhenyu Zhong <zhongresearch@gmail.com
>>>>> wrote:
>>>>> Here is the zookeeper leader log. It had an unexpected exception. I
>>>> don't
>>>>> know why.
>>>>>
>>>>> http://pastebin.com/m6594ccc
>>>>>
>>>>> During that time, I don't think the system load is high. Memory usage
>>>>> should
>>>>> be normal.
>>>>>
>>>>> Best,
>>>>> zhenyu
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Nov 19, 2009 at 11:09 AM, stack <st...@duboce.net> wrote:
>>>>>
>>>>>> What is in the other zk logs? Why did the leader go away? Memory
>>>> issue
>>>>> or
>>>>>> something? Why'd it shutdown? Thanks for pastebin'ing this stuff.
>>>>>> St.Ack
>>>>>>
>>>>>> On Thu, Nov 19, 2009 at 7:53 AM, Zhenyu Zhong <
>>> zhongresearch@gmail.com
>>>>>>> wrote:
>>>>>>> After digging more on the zookeeper log, I found that there
>> seemed
>>> to
>>>>> be
>>>>>> a
>>>>>>> socketconnection timeout. Also some warnings indicate that
>>> zookeeper
>>>>>> server
>>>>>>> is not running. Probably I need to understand the mechanism of
>>>>> zookeeper
>>>>>>> first.
>>>>>>>
>>>>>>> Please find the zookeeper log around that time.
>>>>>>>
>>>>>>> http://pastebin.com/m37bb4ad1
>>>>>>>
>>>>>>> Many thanks
>>>>>>> zhenyu
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Nov 19, 2009 at 10:31 AM, Zhenyu Zhong <
>>>>> zhongresearch@gmail.com
>>>>>>>> wrote:
>>>>>>>> Stack,
>>>>>>>>
>>>>>>>> I am very appreciated. Let me dig into the zookeeper log more.
>>>>>>>>
>>>>>>>> FYI, I saw the sleeper complained about the delay for the first
>>>> time.
>>>>>> It
>>>>>>>> looks like there are different reasons for the RS
>> disconnections.
>>>>>>>> 2009-11-19 08:08:32,708 DEBUG org.apache.zookeeper.ClientCnxn:
>>> Got
>>>>> ping
>>>>>>>> response for sessionid:0x42508716bfc0001 after 0ms
>>>>>>>> 2009-11-19 08:08:37,595 DEBUG org.apache.zookeeper.ClientCnxn:
>>> Got
>>>>> ping
>>>>>>>> response for sessionid:0x42508716bfc0082 after 0ms
>>>>>>>> 2009-11-19 08:08:46,073 DEBUG org.apache.zookeeper.ClientCnxn:
>>> Got
>>>>> ping
>>>>>>>> response for sessionid:0x42508716bfc0001 after 0ms
>>>>>>>> 2009-11-19 08:09:39,880 WARN org.apache.zookeeper.ClientCnxn:
>>>>> Exception
>>>>>>>> closing session 0x42508716bfc0001 to
>>>>>> sun.nio.ch.SelectionKeyImpl@4f3a608f
>>>>>>>> java.io.IOException: TIMED OUT
>>>>>>>> at
>>>>>>>>
>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:906)
>>>>>>>> 2009-11-19 08:09:39,880 WARN org.apache.zookeeper.ClientCnxn:
>>>>> Exception
>>>>>>>> closing session 0x42508716bfc0082 to
>>>>>> sun.nio.ch.SelectionKeyImpl@14c8875
>>>>>>>> java.io.IOException: TIMED OUT
>>>>>>>> at
>>>>>>>>
>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:906)
>>>>>>>> 2009-11-19 08:09:39,880 WARN
>>> org.apache.hadoop.hbase.util.Sleeper:
>>>> We
>>>>>>> slept
>>>>>>>> 53556ms, ten times longer than scheduled: 3000
>>>>>>>> 2009-11-19 08:09:39,980 WARN
>>>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Attempt=1
>>>>>>>> org.apache.hadoop.hbase.Leases$LeaseStillHeldException
>>>>>>>> at
>>>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>>>>>>>> Method)
>>>>>>>> at
>>>>>>>>
>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>>>>>>>> at
>>>>>>>>
>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>>>>>>>> at
>>>>>>> java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>>>>>>>> at
>>>>>>>>
>> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
>>>>>>>> at
>>>>>>>>
>> org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
>>>>>>>> at
>>>>>>>>
>> org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
>>>>>>>> at
>>>>>>>>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:571)
>>>>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>>>>> 2009-11-19 08:09:39,980 INFO
>>>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Got
>> ZooKeeper
>>>>>> event,
>>>>>>>> state: Disconnected, type: None, path: null
>>>>>>>> 2009-11-19 08:09:40,530 INFO org.apache.zookeeper.ClientCnxn:
>>>>>> Attempting
>>>>>>>> connection to server superpyxis0005/192.168.100.119:2181
>>>>>>>> 2009-11-19 08:09:40,530 INFO org.apache.zookeeper.ClientCnxn:
>>>> Priming
>>>>>>>> connection to java.nio.channels.SocketChannel[connected local=/
>>>>>>>> 192.168.100.122:40659 remote=superpyxis0005/
>> 192.168.100.119:2181
>>> ]
>>>>>>>>
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> zhenyu
>>>>>>>>
>>>>>>>> On Wed, Nov 18, 2009 at 6:52 PM, stack <st...@duboce.net>
>> wrote:
>>>>>>>>> Patrick Hunt, one of the zk lads, answered with the following:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> According to the following snippet (I cut to just session
>>>>>>>>> 0x425035c48210002)
>>>>>>>>> the client is getting pings for this session id regularly
>> every
>>> 6
>>>>>>> seconds
>>>>>>>>> or
>>>>>>>>> so, then after 41,016 there's a 20 second gap (corresponds to
>> a
>>> 30
>>>>>>> second
>>>>>>>>> timeout being set during session creation) at which point we
>>>>>> disconnect
>>>>>>>>> from
>>>>>>>>> the server.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> 2009-11-18 12:41:34,788 DEBUG
>> org.apache.zookeeper.ClientCnxn:
>>>> Got
>>>>>>> ping
>>>>>>>>>> response for sessionid:0x425035c48210002 after 0ms
>>>>>>>>>> 2009-11-18 12:42:01,482 WARN
>> org.apache.zookeeper.ClientCnxn:
>>>>>>> Exception
>>>>>>>>>> closing session 0x425035c48210002 to
>>>>>>>>> sun.nio.ch.SelectionKeyImpl@421690ab
>>>>>>>>>> java.io.IOException: TIMED OUT
>>>>>>>>> the client notifies hbase of the disco, and attempts to
>>> reconnect
>>>> to
>>>>> a
>>>>>>>>> server
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Got
>>>> ZooKeeper
>>>>>>> event,
>>>>>>>>>> state: Disconnected, type: None, path: null
>>>>>>>>>> 2009-11-18 12:42:01,782 INFO
>> org.apache.zookeeper.ClientCnxn:
>>>>>>> Attempting
>>>>>>>>>> connection to server superpyxis0003/192.168.100.117:2181
>>>>>>>>> but the server won't allow the session be be established for
>>> some
>>>>>> reason
>>>>>>>>> (is
>>>>>>>>> that the right server:port that's it's trying to connect to?)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> 2009-11-18 12:42:02,182 INFO
>> org.apache.zookeeper.ClientCnxn:
>>>>> Server
>>>>>>>>>> connection successful
>>>>>>>>>> 2009-11-18 12:42:02,182 WARN
>> org.apache.zookeeper.ClientCnxn:
>>>>>>> Exception
>>>>>>>>>> closing session 0x425035c48210002 to
>>>>>>>>> sun.nio.ch.SelectionKeyImpl@5c07076b
>>>>>>>>>> java.io.IOException: Read error rc = -1
>>>>>>> java.nio.DirectByteBuffer[pos=0
>>>>>>>>>> lim=4 cap=4]
>>>>>>>>> The "read error rc = -1" means that the client read from the
>>>> socket
>>>>>> and
>>>>>>>>> got
>>>>>>>>> EOS. But it's not a session expiration afaict (and should not
>> be
>>>> at
>>>>>> this
>>>>>>>>> point in time).
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> You need to look at the ZK server logs for this period in time
>>>> (for
>>>>>> this
>>>>>>>>> server superpyxis0003:2181, and superpyxis0001:2181, the
>> client
>>>>> tries
>>>>>> to
>>>>>>>>> connect to both of these). There should be some indication
>> there
>>>> why
>>>>>> the
>>>>>>>>> server is closing down the connection and not allowing the
>>> session
>>>>> to
>>>>>> be
>>>>>>>>> re-established.
>>>>>>>>>
>>>>>>>>> One thought - by default we limit clients to a max of 10
>>>> connections
>>>>>>> from
>>>>>>>>> a
>>>>>>>>> particular host to the server. If you exceed this limit the zk
>>>>> server
>>>>>>>>> (it's
>>>>>>>>> per server though, not per cluster, so perhaps this is not it)
>>>> will
>>>>>> not
>>>>>>>>> allow any additional connections from the client host (you
>>> can't
>>>>>> exceed
>>>>>>>>> 10
>>>>>>>>> on a server). Could this be it? i.e. the user is running a
>>> number
>>>> of
>>>>>> ZK
>>>>>>>>> clients (say hbase processes) from the same host? Regardless,
>>> get
>>>>> the
>>>>>>>>> server
>>>>>>>>> log for this time period and it should be more clear.
>>>>>>>>>
>>>>>>>>> Patrick
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Nov 18, 2009 at 11:18 AM, stack <st...@duboce.net>
>>> wrote:
>>>>>>>>>> And running stat on each of your zookeeper nodes tells you
>>> what
>>>>>> about
>>>>>>>>>> averages for connect-times?
>>>>>>>>>>
>>>>>>>>>> Its odd that there is a ping just 20 seconds beforehand and
>>> you
>>>>> have
>>>>>>> set
>>>>>>>>>> the zk session timeout at 60 seconds (or ten minutes?). Let
>>> me
>>>>> ask
>>>>>>> the
>>>>>>>>> zk
>>>>>>>>>> lads about it.
>>>>>>>>>>
>>>>>>>>>> If you look in the GC log for around this time, do you see
>>>>> anything?
>>>>>>>>>> St.Ack
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, Nov 18, 2009 at 7:32 AM, Zhenyu Zhong <
>>>>>>> zhongresearch@gmail.com
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> After running more experiment, it seems that the
>>> disconnection
>>>> is
>>>>>> not
>>>>>>>>>>> related to the heavy load job, because the load average is
>>> low,
>>>>> the
>>>>>>>>> diskio
>>>>>>>>>>> is normal, memory Heap was not reached, also the virtual
>>> memory
>>>>>> stats
>>>>>>>>>>> shows
>>>>>>>>>>> that no swappings. However, the disconnection still
>> happens.
>>>>>>>>>>> It looks like this time it pauses for 20 seconds. No idea
>> why
>>>>>>>>> regionserver
>>>>>>>>>>> disconnected.
>>>>>>>>>>>
>>>>>>>>>>> Any other suggestions please?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 2009-11-18 12:41:00,956 DEBUG
>>> org.apache.zookeeper.ClientCnxn:
>>>>> Got
>>>>>>> ping
>>>>>>>>>>> response for sessionid:0x125035c495d02bb after 0ms
>>>>>>>>>>> 2009-11-18 12:41:08,074 DEBUG
>>> org.apache.zookeeper.ClientCnxn:
>>>>> Got
>>>>>>> ping
>>>>>>>>>>> response for sessionid:0x425035c48210002 after 0ms
>>>>>>>>>>> 2009-11-18 12:41:14,302 DEBUG
>>> org.apache.zookeeper.ClientCnxn:
>>>>> Got
>>>>>>> ping
>>>>>>>>>>> response for sessionid:0x125035c495d02bb after 0ms
>>>>>>>>>>> 2009-11-18 12:41:21,433 DEBUG
>>> org.apache.zookeeper.ClientCnxn:
>>>>> Got
>>>>>>> ping
>>>>>>>>>>> response for sessionid:0x425035c48210002 after 0ms
>>>>>>>>>>> 2009-11-18 12:41:27,656 DEBUG
>>> org.apache.zookeeper.ClientCnxn:
>>>>> Got
>>>>>>> ping
>>>>>>>>>>> response for sessionid:0x125035c495d02bb after 0ms
>>>>>>>>>>> 2009-11-18 12:41:34,788 DEBUG
>>> org.apache.zookeeper.ClientCnxn:
>>>>> Got
>>>>>>> ping
>>>>>>>>>>> response for sessionid:0x425035c48210002 after 0ms
>>>>>>>>>>> 2009-11-18 12:41:41,016 DEBUG
>>> org.apache.zookeeper.ClientCnxn:
>>>>> Got
>>>>>>> ping
>>>>>>>>>>> response for sessionid:0x125035c495d02bb after 1ms
>>>>>>>>>>> 2009-11-18 12:42:01,482 WARN
>> org.apache.zookeeper.ClientCnxn:
>>>>>>> Exception
>>>>>>>>>>> closing session 0x425035c48210002 to
>>>>>>>>> sun.nio.ch.SelectionKeyImpl@421690ab
>>>>>>>>>>> java.io.IOException: TIMED OUT
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:906)
>>>>>>>>>>> 2009-11-18 12:42:01,582 INFO
>>>>>>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Got
>>>> ZooKeeper
>>>>>>>>> event,
>>>>>>>>>>> state: Disconnected, type: None, path: null
>>>>>>>>>>> 2009-11-18 12:42:01,782 INFO
>> org.apache.zookeeper.ClientCnxn:
>>>>>>>>> Attempting
>>>>>>>>>>> connection to server superpyxis0003/192.168.100.117:2181
>>>>>>>>>>> 2009-11-18 12:42:01,782 INFO
>> org.apache.zookeeper.ClientCnxn:
>>>>>> Priming
>>>>>>>>>>> connection to java.nio.channels.SocketChannel[connected
>>> local=/
>>>>>>>>>>> 192.168.100.132:46610 remote=superpyxis0003/
>>>> 192.168.100.117:2181
>>>>> ]
>>>>>>>>>>> 2009-11-18 12:42:01,782 INFO
>> org.apache.zookeeper.ClientCnxn:
>>>>>> Server
>>>>>>>>>>> connection successful
>>>>>>>>>>> 2009-11-18 12:42:01,782 WARN
>> org.apache.zookeeper.ClientCnxn:
>>>>>>> Exception
>>>>>>>>>>> closing session 0x425035c48210002 to
>>>>>>>>> sun.nio.ch.SelectionKeyImpl@56dc6fac
>>>>>>>>>>> java.io.IOException: Read error rc = -1
>>>>>>> java.nio.DirectByteBuffer[pos=0
>>>>>>>>>>> lim=4 cap=4]
>>>>>>>>>>> at
>>>>>>>>>>>
>> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
>>>>>>>>>>> 2009-11-18 12:42:01,782 WARN
>> org.apache.zookeeper.ClientCnxn:
>>>>>>> Ignoring
>>>>>>>>>>> exception during shutdown input
>>>>>>>>>>> java.net.SocketException: Transport endpoint is not
>> connected
>>>>>>>>>>> at sun.nio.ch.SocketChannelImpl.shutdown(Native
>>> Method)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
>>>>>>>>>>> at
>>>>>>>>> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
>>>>>>>>>>> 2009-11-18 12:42:01,782 WARN
>> org.apache.zookeeper.ClientCnxn:
>>>>>>> Ignoring
>>>>>>>>>>> exception during shutdown output
>>>>>>>>>>> java.net.SocketException: Transport endpoint is not
>> connected
>>>>>>>>>>> at sun.nio.ch.SocketChannelImpl.shutdown(Native
>>> Method)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
>>>>>>>>>>> at
>> sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
>>>>>>>>>>> at
>>>>>>>>>>>
>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
>>>>>>>>>>> 2009-11-18 12:42:02,182 INFO
>> org.apache.zookeeper.ClientCnxn:
>>>>>>>>> Attempting
>>>>>>>>>>> connection to server superpyxis0001/192.168.100.115:2181
>>>>>>>>>>> 2009-11-18 12:42:02,182 INFO
>> org.apache.zookeeper.ClientCnxn:
>>>>>> Priming
>>>>>>>>>>> connection to java.nio.channels.SocketChannel[connected
>>> local=/
>>>>>>>>>>> 192.168.100.132:36197 remote=superpyxis0001/
>>>> 192.168.100.115:2181
>>>>> ]
>>>>>>>>>>> 2009-11-18 12:42:02,182 INFO
>> org.apache.zookeeper.ClientCnxn:
>>>>>> Server
>>>>>>>>>>> connection successful
>>>>>>>>>>> 2009-11-18 12:42:02,182 WARN
>> org.apache.zookeeper.ClientCnxn:
>>>>>>> Exception
>>>>>>>>>>> closing session 0x425035c48210002 to
>>>>>>>>> sun.nio.ch.SelectionKeyImpl@5c07076b
>>>>>>>>>>> java.io.IOException: Read error rc = -1
>>>>>>> java.nio.DirectByteBuffer[pos=0
>>>>>>>>>>> lim=4 cap=4]
>>>>>>>>>>> at
>>>>>>>>>>>
>> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
>>>>>>>>>>> 2009-11-18 12:42:02,182 WARN
>> org.apache.zookeeper.ClientCnxn:
>>>>>>> Ignoring
>>>>>>>>>>> exception during shutdown input
>>>>>>>>>>> java.net.SocketException: Transport endpoint is not
>> connected
>>>>>>>>>>> at sun.nio.ch.SocketChannelImpl.shutdown(Native
>>> Method)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
>>>>>>>>>>> at
>>>>>>>>> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
>>>>>>>>>>> 2009-11-18 12:42:02,182 WARN
>> org.apache.zookeeper.ClientCnxn:
>>>>>>> Ignoring
>>>>>>>>>>> exception during shutdown output
>>>>>>>>>>> java.net.SocketException: Transport endpoint is not
>> connected
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Nov 16, 2009 at 4:16 PM, Zhenyu Zhong <
>>>>>>> zhongresearch@gmail.com
>>>>>>>>>>>> wrote:
>>>>>>>>>>>> Here is the diskIO and CPU around the time we had RS
>>>>>> disconnection
>>>>>>> on
>>>>>>>>>>> one
>>>>>>>>>>>> machine that runs RegionServer. It doesn't seem to be
>> high.
>>>>>> Similar
>>>>>>>>> disk
>>>>>>>>>>> and
>>>>>>>>>>>> cpu usage have been seen before and the HBase was running
>>>> fine.
>>>>>>>>>>>>
>>>>>>>>>>>> So far I haven't found why my 10 minutes session timeout
>>>>> doesn't
>>>>>>>>> apply.
>>>>>>>>>>>> Still digging.
>>>>>>>>>>>>
>>>>>>>>>>>> Many thanks!
>>>>>>>>>>>>
>>>>>>>>>>>> Time: 03:01:08 PM
>>>>>>>>>>>> avg-cpu: %user %nice %system %iowait %steal %idle
>>>>>>>>>>>> 3.82 0.00 0.89 3.97 0.00 91.32
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Device: tps Blk_read/s Blk_wrtn/s
>> Blk_read
>>>>>>>>> Blk_wrtn
>>>>>>>>>>>> sda 24.30 1605.39 6679.35
>> 6238623388
>>>>>>>>> 25956251720
>>>>>>>>>>>> sda1 0.51 1.94 15.71
>> 7547265
>>>>>>>>> 61067904
>>>>>>>>>>>> sda2 0.13 2.11 5.20
>> 8202079
>>>>>>>>> 20189232
>>>>>>>>>>>> sda3 0.13 0.25 1.63
>> 987846
>>>>>>>>> 6323336
>>>>>>>>>>>> sda4 0.00 0.00 0.00
>> 4
>>>>>>>>> 0
>>>>>>>>>>>> sda5 1.78 0.06 24.14
>> 220010
>>>>>>>>> 93817208
>>>>>>>>>>>> sda6 0.38 0.42 8.41
>> 1630584
>>>>>>>>> 32688152
>>>>>>>>>>>> sda7 21.38 1600.61 6624.26
>> 6220035272
>>>>>>>>> 25742165888
>>>>>>>>>>>> sdb 4.52 767.33 565.91
>> 2981868690
>>>>>>>>> 2199132380
>>>>>>>>>>>> sdb1 4.52 767.33 565.91
>> 2981866755
>>>>>>>>> 2199132372
>>>>>>>>>>>> sdc 4.42 742.95 563.00
>> 2887151482
>>>>>>>>> 2187823092
>>>>>>>>>>>> sdc1 4.42 742.95 563.00
>> 2887149547
>>>>>>>>> 2187823084
>>>>>>>>>>>> sdd 4.49 750.78 557.25
>> 2917554074
>>>>>>>>> 2165513500
>>>>>>>>>>>> sdd1 4.49 750.78 557.25
>> 2917552139
>>>>>>>>> 2165513492
>>>>>>>>>>>> sde 4.52 758.51 569.46
>> 2947593394
>>>>>>>>> 2212964236
>>>>>>>>>>>> sde1 4.52 758.51 569.46
>> 2947591459
>>>>>>>>> 2212964228
>>>>>>>>>>>> sdf 4.51 747.22 571.78
>> 2903740266
>>>>>>>>> 2221972652
>>>>>>>>>>>> sdf1 4.51 747.22 571.78
>> 2903738331
>>>>>>>>> 2221972644
>>>>>>>>>>>> Best,
>>>>>>>>>>>> zhenyu
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Nov 16, 2009 at 4:07 PM, stack <stack@duboce.net
>>>>> wrote:
>>>>>>>>>>>>> On Mon, Nov 16, 2009 at 12:05 PM, Zhenyu Zhong <
>>>>>>>>>>> zhongresearch@gmail.com
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> I just realized that there was a MapReduce job running
>>>>> during
>>>>>>> the
>>>>>>>>>>> time
>>>>>>>>>>>>> the
>>>>>>>>>>>>>> regionserver disconnected from the zookeeper.
>>>>>>>>>>>>>> That MapReduce Job was processing 500GB data and took
>>>> about
>>>>> 8
>>>>>>>>> minutes
>>>>>>>>>>> to
>>>>>>>>>>>>>> finish. It launched over 2000 map tasks.
>>>>>>>>>>>>>
>>>>>>>>>>>>> There was a tasktracker running on the RegionServer that
>>>>>>>>> disconnected?
>>>>>>>>>>> Is
>>>>>>>>>>>>> the map i/o or cpu heavy? Do you think it could have
>>> stole
>>>>> life
>>>>>>>>> from
>>>>>>>>>>> the
>>>>>>>>>>>>> datanode/regionserver?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> I doubt that this introduced
>>>>>>>>>>>>>> resource contention between DataNode and RegionServer
>>>> Node.
>>>>>>>>>>>>>> Also during the time that MapReduce job ran, I saw a
>> few
>>>>>> errors
>>>>>>>>>>>>> indicating
>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java.io.IOException: Could not obtain block:
>>>>>>>>>>>>>> blk_7661247556283580226_4450462
>>>>>> file=/data/xxx.txt.200910141159
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1787)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1615)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>
>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1742)
>>>>>>>>>>>>>> at
>>>>>> java.io.DataInputStream.read(DataInputStream.java:83)
>>>>>>>>>>>>>> at
>>>>> org.apache.hadoop.util.LineReader.readLine(LineReader.java:134)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>
>>> org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:136)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>
>> org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:40)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>
>> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:192)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>
>> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:176)
>>>>>>>>>>>>>> at
>>>>>>>>> org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
>>>>>>>>>>>>>> at
>>>>> org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
>>>>>>>>>>>>>> at
>>>>>> org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
>>>>>>>>>>>>>> at
>>>>> org.apache.hadoop.mapred.Child.main(Child.java:170)
>>>>>>>>>>>>> This is interesting. Your map task is running into
>>>> hdfs-127.
>>>>>>> Yeah,
>>>>>>>>>>> you
>>>>>>>>>>>>> should patch your cluster if you want to get rid of
>> these
>>>>>>> (hdfs-127
>>>>>>>>> has
>>>>>>>>>>>>> been
>>>>>>>>>>>>> applied to 0.21 hadoop and will be in the next hadoop
>>>> release
>>>>> on
>>>>>>>>> 0.20.x
>>>>>>>>>>>>> branch, hadoop 0.20.2).
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Possibly it is related to HDFS-127, but I don't see
>> any
>>>>>>> datanodes
>>>>>>>>>>> went
>>>>>>>>>>>>>> down. Does that affect Regionserver? Shall we apply
>> the
>>>>> patch?
>>>>>>>>>>>>>>
>>>>>>>>>>>>> Read up on hdfs-127 for explaination of whats going on.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Now I start to keep tracking of the virtual memory
>> stats
>>>> to
>>>>>> see
>>>>>>> if
>>>>>>>>>>> the
>>>>>>>>>>>>> same
>>>>>>>>>>>>>> issue happens tomorrow around the same time period.
>>>>>>>>>>>>>>
>>>>>>>>>>>>> Excellent. Thanks for doing this.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> And I highly suspect that this particular MapReduce
>> job
>>>>> hurts
>>>>>>>>> HBase
>>>>>>>>>>>>>> Regionserver.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> PS. I also use gcviewer to parse the GC-log, I only
>> see
>>>>> around
>>>>>>> 30
>>>>>>>>>>>>> seconds
>>>>>>>>>>>>>> pauses maximum.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Did you figure why your ten minute session is not
>>> applying?
>>>>>>>>>>>>> St.Ack
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks a lot.
>>>>>>>>>>>>>> zhenyu
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Sun, Nov 15, 2009 at 12:52 AM, Zhenyu Zhong <
>>>>>>>>>>> zhongresearch@gmail.com
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>> J-D,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thank you very much for your comments.
>>>>>>>>>>>>>>> My company block the IRC port, so I have trouble to
>>>>> connect
>>>>>> to
>>>>>>>>> IRC
>>>>>>>>>>>>>> channel.
>>>>>>>>>>>>>>> I have been trying to ask the IT to open the IRC
>> port
>>>> for
>>>>>> me,
>>>>>>> it
>>>>>>>>>>> might
>>>>>>>>>>>>>> take
>>>>>>>>>>>>>>> a while.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>> zhenyu
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Sat, Nov 14, 2009 at 2:21 PM, Jean-Daniel Cryans
>> <
>>>>>>>>>>>>> jdcryans@apache.org
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The error you are getting is a disconnection from a
>>>>>> zookeeper
>>>>>>>>>>> server
>>>>>>>>>>>>>>>> and is very generic.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> ZK-86 is still opened and the last comment refers
>> to
>>>>> ZK-111
>>>>>>>>> saying
>>>>>>>>>>>>>>>> that the bug (in unit tests) was probably fixed in
>>>>> release
>>>>>>>>> 3.0.0
>>>>>>>>>>> last
>>>>>>>>>>>>>>>> year.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> To figure the hang you have, you can try to jstack
>>> the
>>>>>>> process
>>>>>>>>> pid
>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>> see exactly what's holding the RS from shutting
>> down.
>>>>>>>>>>>>>>>> Would it be possible for you to drop by the IRC
>>>> channel?
>>>>>> This
>>>>>>>>> way
>>>>>>>>>>> we
>>>>>>>>>>>>>>>> can debug this a at much faster pace.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thx!
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> J-D
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Sat, Nov 14, 2009 at 9:57 AM, Zhenyu Zhong <
>>>>>>>>>>>>> zhongresearch@gmail.com>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>> I found this.
>>>>>>>>>>>>>>>>>
>> http://issues.apache.org/jira/browse/ZOOKEEPER-86
>>>>>>>>>>>>>>>>> It looks like the same error I had. Is it a
>>> zookeeper
>>>>>> bug?
>>>>>>>>> When
>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>> HBase
>>>>>>>>>>>>>>>>> take the zookeeper version 3.3.0?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> thanks
>>>>>>>>>>>>>>>>> zhenyu
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Sat, Nov 14, 2009 at 11:31 AM, Zhenyu Zhong <
>>>>>>>>>>>>>> zhongresearch@gmail.com
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Now I really doubt about the zookeeper, from the
>>> log
>>>> I
>>>>>> saw
>>>>>>>>>>> errors
>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>> IOException Read Error, while the zookeeper
>> client
>>>>>>>>>>> (regionserver )
>>>>>>>>>>>>>>>> tried to
>>>>>>>>>>>>>>>>>> read. But it got disconneted status from
>>> zookeeper.
>>>>>>>>>>>>>>>>>> I don't see any load on any zookeeper quorum
>>>> servers.
>>>>>>> DiskIO
>>>>>>>>> is
>>>>>>>>>>>>> fine.
>>>>>>>>>>>>>>>>>> Also when the regionserver decides to exit due
>> to
>>>> the
>>>>>>>>>>> disconnect
>>>>>>>>>>>>>> status
>>>>>>>>>>>>>>>>>> from zookeeper, sometimes the regionserver hangs
>>>>> during
>>>>>>> the
>>>>>>>>>>>>> exiting.
>>>>>>>>>>>>>> We
>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>> still see the HRegionServer process even we
>> don't
>>>> see
>>>>> it
>>>>>>> in
>>>>>>>>> the
>>>>>>>>>>>>>> Master
>>>>>>>>>>>>>>>> web
>>>>>>>>>>>>>>>>>> interface.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I also notice that there is an zookeeper.retries
>>> and
>>>>>>>>>>>>> zookeeper.pauses
>>>>>>>>>>>>>>>>>> settings in hbase-default.xml, would that matter
>> ?
>>>>>>>>>>>>>>>>>> thanks
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> You may see the IOException error in the
>> following
>>>> log
>>>>>>> when
>>>>>>>>> the
>>>>>>>>>>>>>>>>>> Regionserver lost connection to the zookeeper.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> 2009-11-14 15:58:31,993 INFO
>>>>>>>>>>>>>> org.apache.hadoop.hbase.regionserver.HLog:
>>>>>>>>>>>>>>>>>> Roll
>>>>>>>>>>>>>>>>>>
>> /hbase/.logs/superpyxis0008.scur.colo,60021,1258179101617/hlog.dat.1258179106921,
>>>>>>>>>>>>>>>>>> entries=1062964, calcsize=255014936,
>>>>> filesize=139566093.
>>>>>>> New
>>>>>>>>>>> hlog
>> /hbase/.logs/superpyxis0008.scur.colo,60021,1258179101617/hlog.dat.1258214311993
>>>>>>>>>>>>>>>>>> 2009-11-14 15:58:39,295 DEBUG
>>>>>>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>>>>>> Got
>>>>>>>>>>>>>> ping
>>>>>>>>>>>>>>>>>> response for sessionid:0x424f0ed01ea00a6 after
>> 0ms
>>>>>>>>>>>>>>>>>> 2009-11-14 15:58:52,648 DEBUG
>>>>>>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>>>>>> Got
>>>>>>>>>>>>>> ping
>>>>>>>>>>>>>>>>>> response for sessionid:0x424f0ed01ea00a6 after
>> 0ms
>>>>>>>>>>>>>>>>>> 2009-11-14 15:59:06,007 DEBUG
>>>>>>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>>>>>> Got
>>>>>>>>>>>>>> ping
>>>>>>>>>>>>>>>>>> response for sessionid:0x424f0ed01ea00a6 after
>> 0ms
>>>>>>>>>>>>>>>>>> 2009-11-14 15:59:19,365 DEBUG
>>>>>>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>>>>>> Got
>>>>>>>>>>>>>> ping
>>>>>>>>>>>>>>>>>> response for sessionid:0x424f0ed01ea00a6 after
>> 0ms
>>>>>>>>>>>>>>>>>> 2009-11-14 15:59:32,867 DEBUG
>>>>>>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>>>>>> Got
>>>>>>>>>>>>>> ping
>>>>>>>>>>>>>>>>>> response for sessionid:0x424f0ed01ea00a6 after
>>> 186ms
>>>>>>>>>>>>>>>>>> 2009-11-14 15:59:46,070 DEBUG
>>>>>>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>>>>>> Got
>>>>>>>>>>>>>> ping
>>>>>>>>>>>>>>>>>> response for sessionid:0x424f0ed01ea00a6 after
>> 0ms
>>>>>>>>>>>>>>>>>> 2009-11-14 15:59:59,423 DEBUG
>>>>>>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>>>>>> Got
>>>>>>>>>>>>>> ping
>>>>>>>>>>>>>>>>>> response for sessionid:0x424f0ed01ea00a6 after
>> 0ms
>>>>>>>>>>>>>>>>>> 2009-11-14 16:00:12,775 DEBUG
>>>>>>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>>>>>> Got
>>>>>>>>>>>>>> ping
>>>>>>>>>>>>>>>>>> response for sessionid:0x424f0ed01ea00a6 after
>> 0ms
>>>>>>>>>>>>>>>>>> 2009-11-14 16:00:28,141 DEBUG
>>>>>>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>>>>>> Got
>>>>>>>>>>>>>> ping
>>>>>>>>>>>>>>>>>> response for sessionid:0x424f0ed01ea00a6 after
>>>> 2010ms
>>>>>>>>>>>>>>>>>> 2009-11-14 16:01:10,378 WARN
>>>>>>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>>>>>>>>> Exception
>>>>>>>>>>>>>>>>>> closing session 0x424f0ed01ea00a6 to
>>>>>>>>>>>>>>>> sun.nio.ch.SelectionKeyImpl@39fbb2d6
>>>>>>>>>>>>>>>>>> java.io.IOException: Read error rc = -1
>>>>>>>>>>>>>> java.nio.DirectByteBuffer[pos=0
>>>>>>>>>>>>>>>>>> lim=4 cap=4]
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>
>>> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>
>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
>>>>>>>>>>>>>>>>>> 2009-11-14 16:01:10,478 INFO
>>>>>>>>>>>>>>>>>>
>>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>>>>> Got
>>>>>>>>>>> ZooKeeper
>>>>>>>>>>>>>>>> event,
>>>>>>>>>>>>>>>>>> state: Disconnected, type: None, path: null
>>>>>>>>>>>>>>>>>> 2009-11-14 16:01:11,333 WARN
>>>>>>>>>>>>>>>>>>
>>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>>>>>>>>> Attempt=1
>>>>>>>>>>>>>>>>>>
>>>> org.apache.hadoop.hbase.Leases$LeaseStillHeldException
>>>>>>>>>>>>>>>>>> at
>>>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>>>>>>>>>>>>>>>>>> Method)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>
>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>
>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>>>>>>>>>>>>>>>>>> at
>>>>>>> java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>
>> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>
>> org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>
>> org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:571)
>>>>>>>>>>>>>>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>>>>>>>>>>>>>>> 2009-11-14 16:01:11,433 INFO
>>>>>>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>>>>>>>>>>> Attempting
>>>>>>>>>>>>>>>>>> connection to server superpyxis0001/
>>>>>> 192.168.100.115:2181
>>>>>>>>>>>>>>>>>> 2009-11-14 16:01:11,433 INFO
>>>>>>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>>>>>>>> Priming
>>>>>>>>>>>>>>>>>> connection to
>>>>> java.nio.channels.SocketChannel[connected
>>>>>>>>> local=/
>>>>>>>>>>>>>>>>>> 192.168.100.122:59575 remote=superpyxis0001/
>>>>>>>>>>> 192.168.100.115:2181]
>>>>>>>>>>>>>>>>>> 2009-11-14 16:01:11,433 INFO
>>>>>>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>>>>>>>> Server
>>>>>>>>>>>>>>>>>> connection successful
>>>>>>>>>>>>>>>>>> 2009-11-14 16:01:11,433 INFO
>>>>>>>>>>>>>>>>>>
>>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>>>>> Got
>>>>>>>>>>> ZooKeeper
>>>>>>>>>>>>>>>> event,
>>>>>>>>>>>>>>>>>> state: Expired, type: None, path: null
>>>>>>>>>>>>>>>>>> 2009-11-14 16:01:11,433 WARN
>>>>>>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>>>>>>>>> Exception
>>>>>>>>>>>>>>>>>> closing session 0x424f0ed01ea00a6 to
>>>>>>>>>>>>>>>> sun.nio.ch.SelectionKeyImpl@6b93d343
>>>>>>>>>>>>>>>>>> java.io.IOException: Session Expired
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>
>> org.apache.zookeeper.ClientCnxn$SendThread.readConnectResult(ClientCnxn.java:589)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>
>>> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:709)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>
>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
>>>>>>>>>>>>>>>>>> 2009-11-14 16:01:11,433 ERROR
>>>>>>>>>>>>>>>>>>
>>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>>>>>>>>> ZooKeeper
>>>>>>>>>>>>> session
>>>>>>>>>>>>>>>>>> expired
>>>>>>>>>>>>>>>>>> 2009-11-14 16:01:13,513 INFO
>>>>>>>>> org.apache.hadoop.ipc.HBaseServer:
>>>>>>>>>>>>> IPC
>>>>>>>>>>>>>>>> Server
>>>>>>>>>>>>>>>>>> handler 3 on 60021, call put([B@73deb204,
>>>>>>>>>>>>>>>>>> [Lorg.apache.hadoop.hbase.client.Put;@2179600a)
>>>> from
>>>>>>>>>>>>>>>> 192.168.100.132:40728:
>>>>>>>>>>>>>>>>>> error: java.io.IOException: Server not running,
>>>>> aborting
>>>>>>>>>>>>>>>>>> java.io.IOException: Server not running,
>> aborting
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Sat, Nov 14, 2009 at 1:01 AM, Zhenyu Zhong <
>>>>>>>>>>>>>> zhongresearch@gmail.com
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>> Stack,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks a lot!
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I found out that the reason why HBase doesn't
>>> take
>>>>> the
>>>>>>>>> system
>>>>>>>>>>>>> file
>>>>>>>>>>>>>>>>>>> descriptor value I set before. I started the
>>> HBase
>>>>>> using
>>>>>>>>> root
>>>>>>>>>>>>>> instead
>>>>>>>>>>>>>>>> of the
>>>>>>>>>>>>>>>>>>> normal hadoop user, while my system configures
>>>> hadoop
>>>>>>> with
>>>>>>>>>>> higher
>>>>>>>>>>>>>> file
>>>>>>>>>>>>>>>>>>> descriptor value but configures root with 1024
>>>>> default
>>>>>>>>> value.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Now my system has a clean start. However, it
>>> seems
>>>>> that
>>>>>>> the
>>>>>>>>>>>>>>>>>>> zookeeper.session.timeout value doesn't take
>> into
>>>>>> effect.
>>>>>>> I
>>>>>>>>>>> still
>>>>>>>>>>>>>>>> found
>>>>>>>>>>>>>>>>>>> around 60 seconds pauses from the disconnected
>>>>>>>>> regionserver. I
>>>>>>>>>>>>>> really
>>>>>>>>>>>>>>>> don't
>>>>>>>>>>>>>>>>>>> know why regionserver only times out after 60
>>>> seconds
>>>>>>>>> instead
>>>>>>>>>>> of
>>>>>>>>>>>>> 10
>>>>>>>>>>>>>>>> minutes
>>>>>>>>>>>>>>>>>>> which I set for zookeeper.session.timeout.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Is there any other timeout value coming into
>> play
>>>>>> before
>>>>>>>>> the
>>>>>>>>>>>>> actual
>>>>>>>>>>>>>>>>>>> session times out?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> zhenyu
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Fri, Nov 13, 2009 at 7:08 PM, stack <
>>>>>> stack@duboce.net
>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>> Ok. Lack of file descriptors manifests in all
>>>> kinds
>>>>>> of
>>>>>>>>> weird
>>>>>>>>>>>>> ways.
>>>>>>>>>>>>>>>>>>>> Hopefully thats it. If not, lets keep
>> digging.
>>>>>>>>>>>>>>>>>>>> St.Ack
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Fri, Nov 13, 2009 at 3:44 PM, Zhenyu Zhong
>> <
>>>>>>>>>>>>>>>> zhongresearch@gmail.com
>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>> Stack,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> You are right, the master started with
>> ulimit
>>> -n
>>>>>> 1024.
>>>>>>>>> It
>>>>>>>>>>>>> doesn't
>>>>>>>>>>>>>>>> take
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> system value.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Regarding to the too many open files, it
>> looks
>>>>> like
>>>>>>> the
>>>>>>>>>>> same
>>>>>>>>>>>>> as
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> one
>>>>>>>>>>>>>>>>>>>> J-D
>>>>>>>>>>>>>>>>>>>>> put up. But I will get the Master start with
>>>>> higher
>>>>>>>>> value
>>>>>>>>>>>>> first
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>> see
>>>>>>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>>>>>> this kind of error goes away.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> thanks a lot!
>>>>>>>>>>>>>>>>>>>>> zhenyu
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Fri, Nov 13, 2009 at 6:02 PM, stack <
>>>>>>>>> stack@duboce.net>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>> Does it say
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> ulimit -n 32768
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> ...as the second line in your log file on
>>>> start
>>>>> of
>>>>>>> the
>>>>>>>>>>>>> master?
>>>>>>>>>>>>>>>>>>>>>> Is the stack trace that complains about
>> too
>>>> many
>>>>>>> open
>>>>>>>>>>> files
>>>>>>>>>>>>>> same
>>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> one
>>>>>>>>>>>>>>>>>>>>>> in the blog post J-D put up?
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> St.Ack
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Fri, Nov 13, 2009 at 1:37 PM, Zhenyu
>>> Zhong
>>>> <
>>>>>>>>>>>>>>>>>>>> zhongresearch@gmail.com
>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>> The ulimit file descriptors was set to
>>>>>> fs.file-max
>>>>>>>>>>>>> =1578334,
>>>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> limits.conf the value was set to 32768.
>>>>>>>>>>>>>>>>>>>>>>> So these are way higher than the open
>>>>>> descriptors
>>>>>>>>> for
>>>>>>>>>>> the
>>>>>>>>>>>>>>>> running
>>>>>>>>>>>>>>>>>>>>>>> processes.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> thanks
>>>>>>>>>>>>>>>>>>>>>>> zhenyu
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Fri, Nov 13, 2009 at 4:33 PM, Stack <
>>>>>>>>>>>>> saint.ack@gmail.com>
>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>> You upped the ulimit file descriptors
>> as
>>>> per
>>>>>> the
>>>>>>>>>>> getting
>>>>>>>>>>>>>>>> started
>>>>>>>>>>>>>>>>>>>> doc?
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> On Nov 13, 2009, at 1:26 PM, Zhenyu
>>> Zhong
>>>> <
>>>>>>>>>>>>>>>>>>>> zhongresearch@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>> Thanks a lot.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Bad news is my kernel is still
>> 2.6.26.
>>>>>>>>>>>>>>>>>>>>>>>>> But it was not a problem before.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Very strange.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> zhenyu
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Nov 13, 2009 at 4:16 PM,
>>>>> Jean-Daniel
>>>>>>>>> Cryans
>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>> jdcryans@apache.org
>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> Looks like
>>>>>>>>>>>>>>>>>>>>>>>>>>
>> http://pero.blogs.aprilmayjune.org/2009/01/22/hadoop-and-linux-kernel-2627-epoll-limits/
>>>>>>>>>>>>>>>>>>>>>>>>>> J-D
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Nov 13, 2009 at 1:12 PM,
>>> Zhenyu
>>>>>> Zhong
>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>> zhongresearch@gmail.com
>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> After I re-organize the cluster,
>> the
>>>>>>> experiment
>>>>>>>>>>> ran
>>>>>>>>>>>>> into
>>>>>>>>>>>>>>>>>>>> problem
>>>>>>>>>>>>>>>>>>>>>>> faster
>>>>>>>>>>>>>>>>>>>>>>>>>> than
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> before.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Basically the changes are to use
>>>> machines
>>>>>>> with
>>>>>>>>>>> less
>>>>>>>>>>>>>>>> resources
>>>>>>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>>>>>>>>>>>> zookeeper
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> quorums and machines with more
>>>> resources
>>>>> as
>>>>>>>>>>>>>> regionserver.
>>>>>>>>>>>>>>>>>>>>>>>>>>> From the log, I still see the pause
>>>>> around
>>>>>> 1
>>>>>>>>>>> minute.
>>>>>>>>>>>>>>>>>>>>>>>>>>> I enabled the GC logging, please
>> see
>>>>>>>>>>>>>>>>>>>>>>>>>>> http://pastebin.com/m1d4ce0f1
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> for details.
>>>>>>>>>>>>>>>>>>>>>>>>>>> In general I don't see many pauses
>> in
>>>> the
>>>>>> GC.
>>>>>>>>>>>>>>>>>>>>>>>>>>> What is more interesting, 1 hour
>>> after
>>>>> the
>>>>>>> 1st
>>>>>>>>>>>>>>>> regionserver
>>>>>>>>>>>>>>>>>>>>>>>>>>> disconnected,
>>>>>>>>>>>>>>>>>>>>>>>>>>> the master log complained about too
>>>> many
>>>>>> open
>>>>>>>>>>> files.
>>>>>>>>>>>>>> This
>>>>>>>>>>>>>>>>>>>> didn't
>>>>>>>>>>>>>>>>>>>>>>> happen
>>>>>>>>>>>>>>>>>>>>>>>>>>> before.
>>>>>>>>>>>>>>>>>>>>>>>>>>> I checked the system OS setup as
>> well
>>>> as
>>>>>> the
>>>>>>>>>>>>>> limits.conf.
>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>>>>>>>>>> checked
>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> running processes. The total open
>>> files
>>>>>> don't
>>>>>>>>>>> reach
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> limit.
>>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>>> am
>>>>>>>>>>>>>>>>>>>>>>>>>> confused
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> a bit.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Please see the following master
>> log.
>>>>>>>>>>>>>>>>>>>>>>>>>>> 2009-11-13 20:06:02,114 INFO
>>> org.apache.hadoop.hbase.master.BaseScanner:
>>>>>>>>>>>>>>>>>>>>>>>>>>> RegionManager.metaScanner scan of
>>> 4658
>>>>>> row(s)
>>>>>>>>> of
>>>>>>>>>>> meta
>>>>>>>>>>>>>>>> region
>>>>>>>>>>>>>>>>>>>>>> {server:
>>>>>>>>>>>>>>>>>>>>>>>>>>> 192.168.100.128:60021, regionname:
>>>>>>> .META.,,1,
>>>>>>>>>>>>> startKey:
>>>>>>>>>>>>>>>> <>}
>>>>>>>>>>>>>>>>>>>>>> complete
>>>>>>>>>>>>>>>>>>>>>>>>>>> 2009-11-13 20:06:02,114 INFO
>>> org.apache.hadoop.hbase.master.BaseScanner:
>>>>>>>>>>>>>>>>>>>>>>>>>> All
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> 1 .META. region(s) scanned
>>>>>>>>>>>>>>>>>>>>>>>>>>> 2009-11-13 20:06:07,677 DEBUG
>>>>>>>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>>>>>>>>>>>>>>> Got
>>>>>>>>>>>>>>>>>>>>>>> ping
>>>>>>>>>>>>>>>>>>>>>>>>>>> response for
>>>> sessionid:0x424eebf1c10004c
>>>>>>> after
>>>>>>>>> 3ms
>>>>>>>>>>>>>>>>>>>>>>>>>>> 2009-11-13 20:06:08,178 INFO
>>>>>>>>>>>>>>>> org.apache.hadoop.hdfs.DFSClient:
>>>>>>>>>>>>>>>>>>>>>>> Exception
>>>>>>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> createBlockOutputStream
>>>>>> java.io.IOException:
>>>>>>>>> Bad
>>>>>>>>>>>>> connect
>>>>>>>>>>>>>>>> ack
>>>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>>>>>>>>> firstBadLink 192.168.100.123:50010
>>>>>>>>>>>>>>>>>>>>>>>>>>> 2009-11-13 20:06:08,178 INFO
>>>>>>>>>>>>>>>> org.apache.hadoop.hdfs.DFSClient:
>>>>>>>>>>>>>>>>>>>>>>>>>>> Abandoning
>>>>>>>>>>>>>>>>>>>>>>>>>>> block
>>> blk_-2808245019291145247_5478039
>>>>>>>>>>>>>>>>>>>>>>>>>>> 2009-11-13 20:06:09,682 INFO
>>>>>>>>>>>>>>>> org.apache.hadoop.hdfs.DFSClient:
>>>>>>>>>>>>>>>>>>>>>>> Exception
>>>>>>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> createBlockOutputStream
>>>>>> java.io.EOFException
>>>>>>>>>>>>>>>>>>>>>>>>>>> 2009-11-13 20:06:09,682 INFO
>>>>>>>>>>>>>>>> org.apache.hadoop.hdfs.DFSClient:
>>>>>>>>>>>>>>>>>>>>>>>>>>> Abandoning
>>>>>>>>>>>>>>>>>>>>>>>>>>> block
>> blk_1074853606841896259_5478048
>>>>>>>>>>>>>>>>>>>>>>>>>>> 2009-11-13 20:06:10,334 DEBUG
>>>>>>>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>>>>>>>>>>>>>>> Got
>>>>>>>>>>>>>>>>>>>>>>> ping
>>>>>>>>>>>>>>>>>>>>>>>>>>> response for
>>> sessionid:0x24eebf1043003c
>>>>>> after
>>>>>>>>> 1ms
>>>>>>>>>>>>>>>>>>>>>>>>>>> 2009-11-13 20:06:21,018 DEBUG
>>>>>>>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>>>>>>>>>>>>>>> Got
>>>>>>>>>>>>>>>>>>>>>>> ping
>>>>>>>>>>>>>>>>>>>>>>>>>>> response for
>>>> sessionid:0x424eebf1c10004c
>>>>>>> after
>>>>>>>>> 0ms
>>>>>>>>>>>>>>>>>>>>>>>>>>> 2009-11-13 20:06:23,674 DEBUG
>>>>>>>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>>>>>>>>>>>>>>> Got
>>>>>>>>>>>>>>>>>>>>>>> ping
>>>>>>>>>>>>>>>>>>>>>>>>>>> response for
>>> sessionid:0x24eebf1043003c
>>>>>> after
>>>>>>>>> 0ms
>>>>>>>>>>>>>>>>>>>>>>>>>>> 2009-11-13 20:06:24,828 INFO
>>>>>>>>>>>>>>>> org.apache.hadoop.hdfs.DFSClient:
>>>>>>>>>>>>>>>>>>>>>>> Exception
>>>>>>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> createBlockOutputStream
>>>>>> java.io.IOException:
>>>>>>>>> Bad
>>>>>>>>>>>>> connect
>>>>>>>>>>>>>>>> ack
>>>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>>>>>>>>> firstBadLink 192.168.100.123:50010
>>>>>>>>>>>>>>>>>>>>>>>>>>> 2009-11-13 20:06:24,828 INFO
>>>>>>>>>>>>>>>> org.apache.hadoop.hdfs.DFSClient:
>>>>>>>>>>>>>>>>>>>>>>>>>>> Abandoning
>>>>>>>>>>>>>>>>>>>>>>>>>>> block
>>> blk_-6642544517082142289_5478063
>>>>>>>>>>>>>>>>>>>>>>>>>>> 2009-11-13 20:06:24,828 INFO
>>>>>>>>>>>>>>>> org.apache.hadoop.hdfs.DFSClient:
>>>>>>>>>>>>>>>>>>>>>>> Exception
>>>>>>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> createBlockOutputStream
>>>>>>>>> java.net.SocketException:
>>>>>>>>>>> Too
>>>>>>>>>>>>>> many
>>>>>>>>>>>>>>>>>>>> open
>>>>>>>>>>>>>>>>>>>>>> files
>>>>>>>>>>>>>>>>>>>>>>>>>>> 2009-11-13 20:06:24,828 INFO
>>>>>>>>>>>>>>>> org.apache.hadoop.hdfs.DFSClient:
>>>>>>>>>>>>>>>>>>>>>>>>>>> Abandoning
>>>>>>>>>>>>>>>>>>>>>>>>>>> block
>> blk_2057511041109796090_5478063
>>>>>>>>>>>>>>>>>>>>>>>>>>> 2009-11-13 20:06:24,928 INFO
>>>>>>>>>>>>>>>> org.apache.hadoop.hdfs.DFSClient:
>>>>>>>>>>>>>>>>>>>>>>> Exception
>>>>>>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> createBlockOutputStream
>>>>>>>>> java.net.SocketException:
>>>>>>>>>>> Too
>>>>>>>>>>>>>> many
>>>>>>>>>>>>>>>>>>>> open
>>>>>>>>>>>>>>>>>>>>>> files
>>>>>>>>>>>>>>>>>>>>>>>>>>> 2009-11-13 20:06:24,928 INFO
>>>>>>>>>>>>>>>> org.apache.hadoop.hdfs.DFSClient:
>>>>>>>>>>>>>>>>>>>>>>>>>>> Abandoning
>>>>>>>>>>>>>>>>>>>>>>>>>>> block
>> blk_8219260302213892894_5478064
>>>>>>>>>>>>>>>>>>>>>>>>>>> 2009-11-13 20:06:30,855 INFO
>>>>>>>>>>>>>>>> org.apache.hadoop.hdfs.DFSClient:
>>>>>>>>>>>>>>>>>>>>>>> Exception
>>>>>>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> createBlockOutputStream
>>>>>>>>> java.net.SocketException:
>>>>>>>>>>> Too
>>>>>>>>>>>>>> many
>>>>>>>>>>>>>>>>>>>> open
>>>>>>>>>>>>>>>>>>>>>> files
>>>>>>>>>>>>>>>>>>>>>>>>>>> 2009-11-13 20:06:30,855 INFO
>>>>>>>>>>>>>>>> org.apache.hadoop.hdfs.DFSClient:
>>>>>>>>>>>>>>>>>>>>>>>>>>> Abandoning
>>>>>>>>>>>>>>>>>>>>>>>>>>> block
>> blk_1669205542853067709_5478235
>>>>>>>>>>>>>>>>>>>>>>>>>>> 2009-11-13 20:06:30,905 INFO
>>>>>>>>>>>>>>>> org.apache.hadoop.hdfs.DFSClient:
>>>>>>>>>>>>>>>>>>>>>>> Exception
>>>>>>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> createBlockOutputStream
>>>>>>>>> java.net.SocketException:
>>>>>>>>>>> Too
>>>>>>>>>>>>>> many
>>>>>>>>>>>>>>>>>>>> open
>>>>>>>>>>>>>>>>>>>>>> files
>>>>>>>>>>>>>>>>>>>>>>>>>>> 2009-11-13 20:06:30,905 INFO
>>>>>>>>>>>>>>>> org.apache.hadoop.hdfs.DFSClient:
>>>>>>>>>>>>>>>>>>>>>>>>>>> Abandoning
>>>>>>>>>>>>>>>>>>>>>>>>>>> block
>> blk_9128897691346270351_5478237
>>>>>>>>>>>>>>>>>>>>>>>>>>> 2009-11-13 20:06:30,955 INFO
>>>>>>>>>>>>>>>> org.apache.hadoop.hdfs.DFSClient:
>>>>>>>>>>>>>>>>>>>>>>> Exception
>>>>>>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> createBlockOutputStream
>>>>>>>>> java.net.SocketException:
>>>>>>>>>>> Too
>>>>>>>>>>>>>> many
>>>>>>>>>>>>>>>>>>>> open
>>>>>>>>>>>>>>>>>>>>>> files
>>>>>>>>>>>>>>>>>>>>>>>>>>> 2009-11-13 20:06:30,955 INFO
>>>>>>>>>>>>>>>> org.apache.hadoop.hdfs.DFSClient:
>>>>>>>>>>>>>>>>>>>>>>>>>>> Abandoning
>>>>>>>>>>>>>>>>>>>>>>>>>>> block
>> blk_1116845144864123018_5478240
>>>>>>>>>>>>>>>>>>>>>>>>>>> 2009-11-13 20:06:34,372 DEBUG
>>>>>>>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>>>>>>>>>>>>>>> Got
>>>>>>>>>>>>>>>>>>>>>>> ping
>>>>>>>>>>>>>>>>>>>>>>>>>>> response for
>>>> sessionid:0x424eebf1c10004c
>>>>>>> after
>>>>>>>>> 0ms
>>>>>>>>>>>>>>>>>>>>>>>>>>> 2009-11-13 20:06:37,034 DEBUG
>>>>>>>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>>>>>>>>>>>>>>> Got
>>>>>>>>>>>>>>>>>>>>>>> ping
>>>>>>>>>>>>>>>>>>>>>>>>>>> response for
>>> sessionid:0x24eebf1043003c
>>>>>> after
>>>>>>>>> 0ms
>>>>>>>>>>>>>>>>>>>>>>>>>>> 2009-11-13 20:06:37,235 WARN
>>>>>>>>>>>>>>>> org.apache.hadoop.hdfs.DFSClient:
>>>>>>>>>>>>>>>>>>>>>>>>>> DataStreamer
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Exception: java.io.IOException: Too
>>>> many
>>>>>> open
>>>>>>>>>>> files
>>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>> sun.nio.ch.IOUtil.initPipe(Native
>>>>>>>>> Method)
>>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>> sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:49)
>>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:18)
>>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.get(SocketIOWithTimeout.java:407)
>>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:322)
>>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
>>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
>>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
>>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>> java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
>>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>> java.io.DataOutputStream.write(DataOutputStream.java:90)
>>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2290)
>>>>>>>>>>>>>>>>>>>>>>>>>>> 2009-11-13 20:06:37,235 WARN
>>>>>>>>>>>>>>>> org.apache.hadoop.hdfs.DFSClient:
>>>>>>>>>>>>>>>>>>>>> Error
>>>>>>>>>>>>>>>>>>>>>>>>>>> Recovery for block
>>>>>>>>> blk_8148813491785406356_5478475
>>>>>>>>>>>>> bad
>>>>>>>>>>>>>>>>>>>> datanode[0]
>>>>>>>>>>>>>>>>>>>>>>>>>>> 192.168.100.123:50010
>>>>>>>>>>>>>>>>>>>>>>>>>>> 2009-11-13 20:06:37,235 WARN
>>>>>>>>>>>>>>>> org.apache.hadoop.hdfs.DFSClient:
>>>>>>>>>>>>>>>>>>>>> Error
>>>>>>>>>>>>>>>>>>>>>>>>>>> Recovery for block
>>>>>>>>> blk_8148813491785406356_5478475
>>>>>>>>>>> in
>>>>>>>>>>>>>>>> pipeline
>>>>>>>>>>>>>>>>>>>>>>>>>>> 192.168.100.123:50010,
>>>>>> 192.168.100.134:50010
>>>>>>> ,
>>>>>>>>>>>>>>>>>>>>> 192.168.100.122:50010
>>>>>>>>>>>>>>>>>>>>>> :
>>>>>>>>>>>>>>>>>>>>>>>>>>> bad
>>>>>>>>>>>>>>>>>>>>>>>>>>> datanode 192.168.100.123:50010
>>>>>>>>>>>>>>>>>>>>>>>>>>> 2009-11-13 20:06:37,436 INFO
>>>>>>>>>>>>>>>> org.apache.hadoop.hdfs.DFSClient:
>>>>>>>>>>>>>>>>>>>>>>> Exception
>>>>>>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> createBlockOutputStream
>>>>>>>>> java.net.SocketException:
>>>>>>>>>>> Too
>>>>>>>>>>>>>> many
>>>>>>>>>>>>>>>>>>>> open
>>>>>>>>>>>>>>>>>>>>>> files
>>>>>>>>>>>>>>>>>>>>>>>>>>> 2009-11-13 20:06:37,436 INFO
>>>>>>>>>>>>>>>> org.apache.hadoop.hdfs.DFSClient:
>>>>>>>>>>>>>>>>>>>>>>>>>>> Abandoning
>>>>>>>>>>>>>>>>>>>>>>>>>>> block
>> blk_2119727700857186236_5478498
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Nov 12, 2009 at 4:21 PM,
>>> Zhenyu
>>>>>> Zhong
>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>> zhongresearch@gmail.com
>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Will do.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> thanks
>>>>>>>>>>>>>>>>>>>>>>>>>>>> zhenyu
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Nov 12, 2009 at 3:33 PM,
>>> stack
>>>> <
>>>>>>>>>>>>>> stack@duboce.net
>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Enable GC logging too on this
>> next
>>>> run
>>>>>> (see
>>>>>>>>>>>>>>>> hbase-env.sh).
>>>>>>>>>>>>>>>>>>>> Lets
>>>>>>>>>>>>>>>>>>>>>> try
>>>>>>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>>> get
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to the bottom of whats going on.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> St.Ack
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Nov 12, 2009 at 12:29 PM,
>>>>> Zhenyu
>>>>>>>>> Zhong <
>>>>>>>>>>>>>>>>>>>>>>>>>>>> zhongresearch@gmail.com
>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I can switch the boxes that run
>>>>>> zookeeper
>>>>>>>>> with
>>>>>>>>>>> the
>>>>>>>>>>>>>> ones
>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>> run
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> regionservers.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I will see the results later.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> FYI. The node does have the 10
>>>> minutes
>>>>>>>>>>>>>>>>>>>>> zookeeper.session.timeout
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> value
>>>>>>>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> place.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> thanks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> zhenyu
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Nov 12, 2009 at 3:21 PM,
>>>> stack
>>>>> <
>>>>>>>>>>>>>>>> stack@duboce.net>
>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Nov 12, 2009 at 11:50
>> AM,
>>>>>> Zhenyu
>>>>>>>>> Zhong
>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> zhongresearch@gmail.com
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> In my cluster, half of the
>>> cluster
>>>>>> have
>>>>>>> 2
>>>>>>>>>>> disks
>>>>>>>>>>>>>> 400GB
>>>>>>>>>>>>>>>>>>>> each
>>>>>>>>>>>>>>>>>>>>> per
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> machine,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> half of the cluster have 6
>> disks
>>>> per
>>>>>>>>> machine.
>>>>>>>>>>>>>> Maybe
>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>>> should
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> run
>>>>>>>>>>>>>>>>>>>>>>>>>>> zookeeper
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on the machines with 2 disks
>> and
>>>> RS
>>>>> on
>>>>>>>>>>> machines
>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>> 6
>>>>>>>>>>>>>>>>>>>> disks?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> That would make most sense
>> only
>>> in
>>>>> the
>>>>>>>>> below,
>>>>>>>>>>> it
>>>>>>>>>>>>>>>> looks
>>>>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> RS
>>>>>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> had
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> issue had 4 disks?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> BTW, the 10 minutes
>>>>>>>>> zookeeper.session.timeout
>>>>>>>>>>>>> has
>>>>>>>>>>>>>>>> been
>>>>>>>>>>>>>>>>>>>> set
>>>>>>>>>>>>>>>>>>>>>> during
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> experiment.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> And for sure this node had it
>> in
>>>>> place?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> St.Ack
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> thanks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> zhenyu
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Nov 12, 2009 at 2:08
>> PM,
>>>>> stack
>>>>>> <
>>>>>>>>>>>>>>>> stack@duboce.net
>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Nov 12, 2009 at 8:40
>>> AM,
>>>>>> Zhenyu
>>>>>>>>>>> Zhong <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> zhongresearch@gmail.com
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Though I experienced 2
>>>>> regionserver
>>>>>>>>>>>>> disconnection
>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> morning,
>>>>>>>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> looks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> better from the regionserver
>>> log.
>>>>>>> (please
>>>>>>>>>>> see
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> following
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> log)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> http://pastebin.com/m496dbfae
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I checked diskIO for one of
>>> the
>>>>>>>>>>>>>>>>>>>>> regionserver(192.168.100.116)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> during
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> time it disconnected.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Time: 03:04:58 AM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Device: tps
>>>>> Blk_read/s
>>>>>>>>>>>>> Blk_wrtn/s
>>>>>>>>>>>>>>>>>>>> Blk_read
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Blk_wrtn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> sda 105.31
>>>> 5458.83
>>>>>>>>>>> 19503.64
>>>>>>>>>>>>>>>>>>>> 9043873239
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 32312473676
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> sda1 2.90
>>>> 6.64
>>>>>>>>>>> 99.25
>>>>>>>>>>>>>>>>>>>> 10993934
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 164433464
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> sda2 1.72
>>>> 23.76
>>>>>>>>>>> 30.25
>>>>>>>>>>>>>>>>>>>> 39365817
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 50124008
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> sda3 0.30
>>>> 0.38
>>>>>>>>>>> 3.58
>>>>>>>>>>>>>>>>>>>> 624930
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 5923000
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> sda4 100.39
>>>> 5428.06
>>>>>>>>>>> 19370.56
>>>>>>>>>>>>>>>>>>>> 8992888270
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 32091993204
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Is this high for you? 20k
>>>>>>>>> blocks/second
>>>>>>>>>>>>> would
>>>>>>>>>>>>>>>> seem to
>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>> high
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>>>>>>>>>>>> its
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> one
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> disk only and its not being
>>>> shared
>>>>> by
>>>>>>> zk
>>>>>>>>>>>>> anymore
>>>>>>>>>>>>>> so
>>>>>>>>>>>>>>>>>>>>> shouldn't
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> matter?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I also checked the zookeeper
>>>>> quorum
>>>>>>>>> server
>>>>>>>>>>>>> that
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> regionserver
>>>>>>>>>>>>>>>>>>>>>>>>>>> tried
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> connect according to the log.
>>>>>> However,
>>>>>>> I
>>>>>>>>>>> don't
>>>>>>>>>>>>> see
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 192.168.100.116
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> client list of the zookeeper
>>>> quorum
>>>>>>>>> server
>>>>>>>>>>> that
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> regionserver
>>>>>>>>>>>>>>>>>>>>>>>>>>> tried
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> connect.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Would that be a problem?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Is that because the
>> ephemeral
>>>>> node
>>>>>>> for
>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> regionserver
>>>>>>>>>>>>>>>>>>>>> had
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> evaporated?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Lost
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> its lease w/ zk by the time
>> you
>>>>> went
>>>>>> to
>>>>>>>>>>> look?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thu Nov 12 15:04:52 UTC 2009
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Zookeeper version:
>>> 3.2.1-808558,
>>>>>> built
>>>>>>>>> on
>>>>>>>>>>>>>>>> 08/27/2009
>>>>>>>>>>>>>>>>>>>> 18:48
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> GMT
>>>>>>>>>>>>>>>>>>>>>>>>>>> Clients:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> /192.168.100.127:43045
>>>>>>>>>>>>>>>> [1](queued=0,recved=26,sent=0)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> /192.168.100.131:39091
>>>>>>>>>>>>>>>> [1](queued=0,recved=964,sent=0)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> /192.168.100.124:35961
>>>>>>>>>>>>>>>> [1](queued=0,recved=3266,sent=0)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> /192.168.100.123:47935
>>>>>>>>>>>>>>>> [1](queued=0,recved=233,sent=0)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> /192.168.100.125:46931
>>>>>>>>>>>>>>>> [1](queued=0,recved=2,sent=0)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> /192.168.100.118:54924
>>>>>>>>>>>>>>>> [1](queued=0,recved=295,sent=0)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> /192.168.100.118:41390
>>>>>>>>>>>>>>>> [1](queued=0,recved=2290,sent=0)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> /192.168.100.136:42243
>>>>>>>>>>>>>>>> [1](queued=0,recved=0,sent=0)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Latency min/avg/max:
>> 0/17/6333
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Received: 47111
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sent: 0
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Outstanding: 0
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Zxid: 0x77000083f4
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Mode: leader
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Node count: 23
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> That 6 second maximum latency
>>> is
>>>>>> pretty
>>>>>>>>> bad
>>>>>>>>>>> but
>>>>>>>>>>>>>>>> should
>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>> well
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> within
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> zk
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> session timeout.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> So, problem is likely on the
>> zk
>>>>>> client
>>>>>>>>> side
>>>>>>>>>>> of
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> session;
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> i.e.
>>>>>>>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> RS.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> You could enable GC logging
>> as
>>>> J-D
>>>>>>>>> suggested
>>>>>>>>>>> to
>>>>>>>>>>>>>> see
>>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>>>>> you
>>>>>>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> any
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> big
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> pauses, pauses > zk session
>>>> timeout.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> When the RS went down, it
>>> didn't
>>>>> look
>>>>>>> too
>>>>>>>>>>>>> heavily
>>>>>>>>>>>>>>>>>>>> loaded:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. 2009-11-12 15:04:52,830
>>> INFO
>>>>>>>>>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>>>>>>>>>>>>>>>>>>>> Dump of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> metrics:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> request=1.5166667,
>> regions=322,
>>>>>>>>> stores=657,
>>>>>>>>>>>>>>>>>>>> storefiles=631,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> storefileIndexSize=61,
>>>>>>>>> memstoreSize=1472,
>>>>>>>>>>>>>>>>>>>> usedHeap=2819,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> maxHeap=4079,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> blockCacheSize=658110960,
>>>>>>>>>>>>>> blockCacheFree=197395984,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> blockCacheCount=9903,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> blockCacheHitRatio=99
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Its serving a few reads? The
>>>>> number
>>>>>> of
>>>>>>>>>>> store
>>>>>>>>>>>>>> files
>>>>>>>>>>>>>>>>>>>> seems
>>>>>>>>>>>>>>>>>>>>>> fine.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Not
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> too
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> much memory used.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Looking at the logs, I see
>> the
>>>>> Lease
>>>>>>>>> Still
>>>>>>>>>>> Held
>>>>>>>>>>>>>>>>>>>> exception.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This
>>>>>>>>>>>>>>>>>>>>>>>>>>> happens
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> when the RS does its regular
>>>> report
>>>>> to
>>>>>>> the
>>>>>>>>>>>>> master
>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> master
>>>>>>>>>>>>>>>>>>>>>>>>>>> thinks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> RS has since restarted.
>> It'll
>>>>> think
>>>>>>> this
>>>>>>>>>>>>> probably
>>>>>>>>>>>>>>>>>>>> because
>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> noticed
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the RS's znode in zk had gone
>>>> away
>>>>>> and
>>>>>>> it
>>>>>>>>>>>>>> considered
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> RS
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dead.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Looking too at your logs I
>> see
>>>>> this
>>>>>>> gap
>>>>>>>>> in
>>>>>>>>>>> the
>>>>>>>>>>>>> zk
>>>>>>>>>>>>>>>>>>>> pinging:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. 2009-11-12 15:03:39,325
>>> DEBUG
>> org.apache.zookeeper.ClientCnxn:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Got
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ping response for
>>>>>>>>> sessionid:0x224e55436ad0004
>>>>>>>>>>>>> after
>>>>>>>>>>>>>>>> 0ms
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. 2009-11-12 15:03:43,113
>>> DEBUG
>> org.apache.zookeeper.ClientCnxn:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Got
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ping response for
>>>>>>>>> sessionid:0x24e55436a0007d
>>>>>>>>>>>>> after
>>>>>>>>>>>>>>>> 0ms
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Where in the lines above it,
>>> its
>>>>>>>>> reporting
>>>>>>>>>>>>> about
>>>>>>>>>>>>>>>> every
>>>>>>>>>>>>>>>>>>>> ten
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> seconds,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> here
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> there is a big gap.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Do you have ganglia or
>>> something
>>>>> that
>>>>>>>>> will
>>>>>>>>>>> let
>>>>>>>>>>>>> you
>>>>>>>>>>>>>>>> look
>>>>>>>>>>>>>>>>>>>> more
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> into
>>>>>>>>>>>>>>>>>>>>>>>>>>> what
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> was
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> happening on this machine
>>> around
>>>>> this
>>>>>>>>> time?
>>>>>>>>>>> Is
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> machine
>>>>>>>>>>>>>>>>>>>>>> OK?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> looks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> lightly loaded and you have
>> your
>>>>>> cluster
>>>>>>>>>>> nicely
>>>>>>>>>>>>>> laid
>>>>>>>>>>>>>>>> out.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Something
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> odd
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> going on. What about things
>>> like
>>>>> the
>>>>>>>>> write
>>>>>>>>>>>>> speed
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>> disk?
>>>>>>>>>>>>>>>>>>>>> In
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>> past
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> strange issues have been
>>> explained
>>>> by
>>>>>>>>>>> incorrectly
>>>>>>>>>>>>>> set
>>>>>>>>>>>>>>>> BIOS
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> which
>>>>>>>>>>>>>>>>>>>>>>>>>>> made
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> disks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> run at 1/100th of their
>> proper
>>>>> speed.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> St.Ack
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> zhenyu
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Nov 11, 2009 at 3:58
>>> PM,
>>>>>>> Zhenyu
>>>>>>>>>>> Zhong
>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> zhongresearch@gmail.com
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Stack
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I am very appreciated for
>>> your
>>>>>>>>> comments.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I will use the zookeeper
>>>>> monitoring
>>>>>>>>> script
>>>>>>>>>>> on
>>>>>>>>>>>>> my
>>>>>>>>>>>>>>>>>>>> cluster
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>>> let
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> run
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> overnight to see the result.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I will follow up the thread
>>>> when
>>>>> I
>>>>>>> get
>>>>>>>>>>>>> anything.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> thanks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> zhenyu
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Nov 11, 2009 at
>> 3:52
>>>> PM,
>>>>>>> stack
>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>> stack@duboce.net>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I see these in your log
>> too:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. 2009-11-11
>> 04:27:19,018
>>>>> DEBUG
>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Got
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ping response for
>>>>>>>>>>> sessionid:0x424dfd908c50009
>>>>>>>>>>>>>>>> after
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 4544ms
>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. 2009-11-11 04:27:19,018 DEBUG
>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Got
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ping response for
>>>>>>>>>>> sessionid:0x24dfd90c810002
>>>>>>>>>>>>>> after
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 4548ms
>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. 2009-11-11 04:27:43,960 DEBUG
>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Got
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ping response for
>>>>>>>>>>> sessionid:0x424dfd908c50009
>>>>>>>>>>>>>>>> after
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 9030ms
>>>>>>>>>>>>>>>>>>>>>>>>>>> 4. 2009-11-11 04:27:43,960 DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> org.apache.zookeeper.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>
Re: regionserver disconnection
Posted by Zhenyu Zhong <zh...@gmail.com>.
Tons of thanks to Stack and J-D as well as the Zookeeper fellows.
Previously we experienced RS session timeouts even we set the
zookeeper.session.timeout to 10 minutes.
It turned out that the zookeeper has both lower bound and upper bound for
the session timeout.
It actually relates to the tickerTime, so the session timeout should be no
less than 2 times tickertime and no greater than 20 times tickerTime. By
default the tickerTime is 2 sec, so the maximum timeout is 40 seconds by
default no matter how large the value is set for zookeeper.session.timeout.
Due to some unknown reason, (probably system heavy load, GC pauses, etc),
40 seconds is not long enough for us, so we have to extend that value.
One way to do that is to up the hbase.zookeeper.property.tickTime. Now with
4000ms set on that value, we haven't seen the RS session TIMEOUT so far.
Many thanks to Stack, J-D and Zookeeper fellows.
I will keep you posted if the session timeout comes back again or there is
anything else causes the RS disconnections.
Best,
zhenyu
On Thu, Nov 19, 2009 at 1:37 PM, stack <st...@duboce.net> wrote:
> From the zk fellas, try running your zk cluster at INFO level rather than
> at
> DEBUG. One supposition is that the shear amount of logging is causing
> distress. They are still looking into your logs...
>
> Thanks for your patience Zhenyu.
> St.Ack
>
>
> On Thu, Nov 19, 2009 at 8:57 AM, Zhenyu Zhong <zhongresearch@gmail.com
> >wrote:
>
> > Yes, I believe so. And I don't know why the leader gave up.
> > For this case, the regionserver got TIMEOUT warning message at 2009-11-18
> > 12:42:01,482
> >
> >
> > Best,
> > zhenyu
> >
> >
> >
> >
> > On Thu, Nov 19, 2009 at 11:43 AM, stack <st...@duboce.net> wrote:
> >
> > > So, the time at which we see
> > >
> > > 2009-11-18 12:41:56,782 ERROR
> > > org.apache.zookeeper.server.quorum.FollowerHandler: Unexpected
> exception
> > > causing shutdown while sock still open
> > >
> > > ... corresponds to the time at which the regionserver loses is session?
> > >
> > > St.Ack
> > >
> > > On Thu, Nov 19, 2009 at 8:38 AM, Zhenyu Zhong <zhongresearch@gmail.com
> > > >wrote:
> > >
> > > > Here is the zookeeper leader log. It had an unexpected exception. I
> > > don't
> > > > know why.
> > > >
> > > > http://pastebin.com/m6594ccc
> > > >
> > > > During that time, I don't think the system load is high. Memory usage
> > > > should
> > > > be normal.
> > > >
> > > > Best,
> > > > zhenyu
> > > >
> > > >
> > > >
> > > > On Thu, Nov 19, 2009 at 11:09 AM, stack <st...@duboce.net> wrote:
> > > >
> > > > > What is in the other zk logs? Why did the leader go away? Memory
> > > issue
> > > > or
> > > > > something? Why'd it shutdown? Thanks for pastebin'ing this stuff.
> > > > > St.Ack
> > > > >
> > > > > On Thu, Nov 19, 2009 at 7:53 AM, Zhenyu Zhong <
> > zhongresearch@gmail.com
> > > > > >wrote:
> > > > >
> > > > > > After digging more on the zookeeper log, I found that there
> seemed
> > to
> > > > be
> > > > > a
> > > > > > socketconnection timeout. Also some warnings indicate that
> > zookeeper
> > > > > server
> > > > > > is not running. Probably I need to understand the mechanism of
> > > > zookeeper
> > > > > > first.
> > > > > >
> > > > > > Please find the zookeeper log around that time.
> > > > > >
> > > > > > http://pastebin.com/m37bb4ad1
> > > > > >
> > > > > > Many thanks
> > > > > > zhenyu
> > > > > >
> > > > > >
> > > > > > On Thu, Nov 19, 2009 at 10:31 AM, Zhenyu Zhong <
> > > > zhongresearch@gmail.com
> > > > > > >wrote:
> > > > > >
> > > > > > > Stack,
> > > > > > >
> > > > > > > I am very appreciated. Let me dig into the zookeeper log more.
> > > > > > >
> > > > > > > FYI, I saw the sleeper complained about the delay for the first
> > > time.
> > > > > It
> > > > > > > looks like there are different reasons for the RS
> disconnections.
> > > > > > >
> > > > > > > 2009-11-19 08:08:32,708 DEBUG org.apache.zookeeper.ClientCnxn:
> > Got
> > > > ping
> > > > > > > response for sessionid:0x42508716bfc0001 after 0ms
> > > > > > > 2009-11-19 08:08:37,595 DEBUG org.apache.zookeeper.ClientCnxn:
> > Got
> > > > ping
> > > > > > > response for sessionid:0x42508716bfc0082 after 0ms
> > > > > > > 2009-11-19 08:08:46,073 DEBUG org.apache.zookeeper.ClientCnxn:
> > Got
> > > > ping
> > > > > > > response for sessionid:0x42508716bfc0001 after 0ms
> > > > > > > 2009-11-19 08:09:39,880 WARN org.apache.zookeeper.ClientCnxn:
> > > > Exception
> > > > > > > closing session 0x42508716bfc0001 to
> > > > > sun.nio.ch.SelectionKeyImpl@4f3a608f
> > > > > > > java.io.IOException: TIMED OUT
> > > > > > > at
> > > > > > >
> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:906)
> > > > > > > 2009-11-19 08:09:39,880 WARN org.apache.zookeeper.ClientCnxn:
> > > > Exception
> > > > > > > closing session 0x42508716bfc0082 to
> > > > > sun.nio.ch.SelectionKeyImpl@14c8875
> > > > > > > java.io.IOException: TIMED OUT
> > > > > > > at
> > > > > > >
> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:906)
> > > > > > > 2009-11-19 08:09:39,880 WARN
> > org.apache.hadoop.hbase.util.Sleeper:
> > > We
> > > > > > slept
> > > > > > > 53556ms, ten times longer than scheduled: 3000
> > > > > > > 2009-11-19 08:09:39,980 WARN
> > > > > > > org.apache.hadoop.hbase.regionserver.HRegionServer: Attempt=1
> > > > > > > org.apache.hadoop.hbase.Leases$LeaseStillHeldException
> > > > > > > at
> > > > > sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> > > > > > > Method)
> > > > > > > at
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> > > > > > > at
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> > > > > > > at
> > > > > > java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> > > > > > > at
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
> > > > > > > at
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
> > > > > > > at
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
> > > > > > > at
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:571)
> > > > > > > at java.lang.Thread.run(Thread.java:619)
> > > > > > > 2009-11-19 08:09:39,980 INFO
> > > > > > > org.apache.hadoop.hbase.regionserver.HRegionServer: Got
> ZooKeeper
> > > > > event,
> > > > > > > state: Disconnected, type: None, path: null
> > > > > > > 2009-11-19 08:09:40,530 INFO org.apache.zookeeper.ClientCnxn:
> > > > > Attempting
> > > > > > > connection to server superpyxis0005/192.168.100.119:2181
> > > > > > > 2009-11-19 08:09:40,530 INFO org.apache.zookeeper.ClientCnxn:
> > > Priming
> > > > > > > connection to java.nio.channels.SocketChannel[connected local=/
> > > > > > > 192.168.100.122:40659 remote=superpyxis0005/
> 192.168.100.119:2181
> > ]
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Best,
> > > > > > > zhenyu
> > > > > > >
> > > > > > > On Wed, Nov 18, 2009 at 6:52 PM, stack <st...@duboce.net>
> wrote:
> > > > > > >
> > > > > > >> Patrick Hunt, one of the zk lads, answered with the following:
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >> According to the following snippet (I cut to just session
> > > > > > >> 0x425035c48210002)
> > > > > > >> the client is getting pings for this session id regularly
> every
> > 6
> > > > > > seconds
> > > > > > >> or
> > > > > > >> so, then after 41,016 there's a 20 second gap (corresponds to
> a
> > 30
> > > > > > second
> > > > > > >> timeout being set during session creation) at which point we
> > > > > disconnect
> > > > > > >> from
> > > > > > >> the server.
> > > > > > >>
> > > > > > >>
> > > > > > >> > 2009-11-18 12:41:34,788 DEBUG
> org.apache.zookeeper.ClientCnxn:
> > > Got
> > > > > > ping
> > > > > > >> > response for sessionid:0x425035c48210002 after 0ms
> > > > > > >> > 2009-11-18 12:42:01,482 WARN
> org.apache.zookeeper.ClientCnxn:
> > > > > > Exception
> > > > > > >> > closing session 0x425035c48210002 to
> > > > > > >> sun.nio.ch.SelectionKeyImpl@421690ab
> > > > > > >> > java.io.IOException: TIMED OUT
> > > > > > >>
> > > > > > >> the client notifies hbase of the disco, and attempts to
> > reconnect
> > > to
> > > > a
> > > > > > >> server
> > > > > > >>
> > > > > > >>
> > > > > > >> > org.apache.hadoop.hbase.regionserver.HRegionServer: Got
> > > ZooKeeper
> > > > > > event,
> > > > > > >> > state: Disconnected, type: None, path: null
> > > > > > >> > 2009-11-18 12:42:01,782 INFO
> org.apache.zookeeper.ClientCnxn:
> > > > > > Attempting
> > > > > > >> > connection to server superpyxis0003/192.168.100.117:2181
> > > > > > >>
> > > > > > >> but the server won't allow the session be be established for
> > some
> > > > > reason
> > > > > > >> (is
> > > > > > >> that the right server:port that's it's trying to connect to?)
> > > > > > >>
> > > > > > >>
> > > > > > >> > 2009-11-18 12:42:02,182 INFO
> org.apache.zookeeper.ClientCnxn:
> > > > Server
> > > > > > >> > connection successful
> > > > > > >> > 2009-11-18 12:42:02,182 WARN
> org.apache.zookeeper.ClientCnxn:
> > > > > > Exception
> > > > > > >> > closing session 0x425035c48210002 to
> > > > > > >> sun.nio.ch.SelectionKeyImpl@5c07076b
> > > > > > >> > java.io.IOException: Read error rc = -1
> > > > > > java.nio.DirectByteBuffer[pos=0
> > > > > > >> > lim=4 cap=4]
> > > > > > >>
> > > > > > >> The "read error rc = -1" means that the client read from the
> > > socket
> > > > > and
> > > > > > >> got
> > > > > > >> EOS. But it's not a session expiration afaict (and should not
> be
> > > at
> > > > > this
> > > > > > >> point in time).
> > > > > > >>
> > > > > > >>
> > > > > > >> You need to look at the ZK server logs for this period in time
> > > (for
> > > > > this
> > > > > > >> server superpyxis0003:2181, and superpyxis0001:2181, the
> client
> > > > tries
> > > > > to
> > > > > > >> connect to both of these). There should be some indication
> there
> > > why
> > > > > the
> > > > > > >> server is closing down the connection and not allowing the
> > session
> > > > to
> > > > > be
> > > > > > >> re-established.
> > > > > > >>
> > > > > > >> One thought - by default we limit clients to a max of 10
> > > connections
> > > > > > from
> > > > > > >> a
> > > > > > >> particular host to the server. If you exceed this limit the zk
> > > > server
> > > > > > >> (it's
> > > > > > >> per server though, not per cluster, so perhaps this is not it)
> > > will
> > > > > not
> > > > > > >> allow any additional connections from the client host (you
> > can't
> > > > > exceed
> > > > > > >> 10
> > > > > > >> on a server). Could this be it? i.e. the user is running a
> > number
> > > of
> > > > > ZK
> > > > > > >> clients (say hbase processes) from the same host? Regardless,
> > get
> > > > the
> > > > > > >> server
> > > > > > >> log for this time period and it should be more clear.
> > > > > > >>
> > > > > > >> Patrick
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >> On Wed, Nov 18, 2009 at 11:18 AM, stack <st...@duboce.net>
> > wrote:
> > > > > > >>
> > > > > > >> > And running stat on each of your zookeeper nodes tells you
> > what
> > > > > about
> > > > > > >> > averages for connect-times?
> > > > > > >> >
> > > > > > >> > Its odd that there is a ping just 20 seconds beforehand and
> > you
> > > > have
> > > > > > set
> > > > > > >> > the zk session timeout at 60 seconds (or ten minutes?). Let
> > me
> > > > ask
> > > > > > the
> > > > > > >> zk
> > > > > > >> > lads about it.
> > > > > > >> >
> > > > > > >> > If you look in the GC log for around this time, do you see
> > > > anything?
> > > > > > >> >
> > > > > > >> > St.Ack
> > > > > > >> >
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > On Wed, Nov 18, 2009 at 7:32 AM, Zhenyu Zhong <
> > > > > > zhongresearch@gmail.com
> > > > > > >> >wrote:
> > > > > > >> >
> > > > > > >> >> After running more experiment, it seems that the
> > disconnection
> > > is
> > > > > not
> > > > > > >> >> related to the heavy load job, because the load average is
> > low,
> > > > the
> > > > > > >> diskio
> > > > > > >> >> is normal, memory Heap was not reached, also the virtual
> > memory
> > > > > stats
> > > > > > >> >> shows
> > > > > > >> >> that no swappings. However, the disconnection still
> happens.
> > > > > > >> >>
> > > > > > >> >> It looks like this time it pauses for 20 seconds. No idea
> why
> > > > > > >> regionserver
> > > > > > >> >> disconnected.
> > > > > > >> >>
> > > > > > >> >> Any other suggestions please?
> > > > > > >> >>
> > > > > > >> >>
> > > > > > >> >> 2009-11-18 12:41:00,956 DEBUG
> > org.apache.zookeeper.ClientCnxn:
> > > > Got
> > > > > > ping
> > > > > > >> >> response for sessionid:0x125035c495d02bb after 0ms
> > > > > > >> >> 2009-11-18 12:41:08,074 DEBUG
> > org.apache.zookeeper.ClientCnxn:
> > > > Got
> > > > > > ping
> > > > > > >> >> response for sessionid:0x425035c48210002 after 0ms
> > > > > > >> >> 2009-11-18 12:41:14,302 DEBUG
> > org.apache.zookeeper.ClientCnxn:
> > > > Got
> > > > > > ping
> > > > > > >> >> response for sessionid:0x125035c495d02bb after 0ms
> > > > > > >> >> 2009-11-18 12:41:21,433 DEBUG
> > org.apache.zookeeper.ClientCnxn:
> > > > Got
> > > > > > ping
> > > > > > >> >> response for sessionid:0x425035c48210002 after 0ms
> > > > > > >> >> 2009-11-18 12:41:27,656 DEBUG
> > org.apache.zookeeper.ClientCnxn:
> > > > Got
> > > > > > ping
> > > > > > >> >> response for sessionid:0x125035c495d02bb after 0ms
> > > > > > >> >> 2009-11-18 12:41:34,788 DEBUG
> > org.apache.zookeeper.ClientCnxn:
> > > > Got
> > > > > > ping
> > > > > > >> >> response for sessionid:0x425035c48210002 after 0ms
> > > > > > >> >> 2009-11-18 12:41:41,016 DEBUG
> > org.apache.zookeeper.ClientCnxn:
> > > > Got
> > > > > > ping
> > > > > > >> >> response for sessionid:0x125035c495d02bb after 1ms
> > > > > > >> >> 2009-11-18 12:42:01,482 WARN
> org.apache.zookeeper.ClientCnxn:
> > > > > > Exception
> > > > > > >> >> closing session 0x425035c48210002 to
> > > > > > >> sun.nio.ch.SelectionKeyImpl@421690ab
> > > > > > >> >> java.io.IOException: TIMED OUT
> > > > > > >> >> at
> > > > > > >> >>
> > > > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:906)
> > > > > > >> >> 2009-11-18 12:42:01,582 INFO
> > > > > > >> >> org.apache.hadoop.hbase.regionserver.HRegionServer: Got
> > > ZooKeeper
> > > > > > >> event,
> > > > > > >> >> state: Disconnected, type: None, path: null
> > > > > > >> >> 2009-11-18 12:42:01,782 INFO
> org.apache.zookeeper.ClientCnxn:
> > > > > > >> Attempting
> > > > > > >> >> connection to server superpyxis0003/192.168.100.117:2181
> > > > > > >> >> 2009-11-18 12:42:01,782 INFO
> org.apache.zookeeper.ClientCnxn:
> > > > > Priming
> > > > > > >> >> connection to java.nio.channels.SocketChannel[connected
> > local=/
> > > > > > >> >> 192.168.100.132:46610 remote=superpyxis0003/
> > > 192.168.100.117:2181
> > > > ]
> > > > > > >> >> 2009-11-18 12:42:01,782 INFO
> org.apache.zookeeper.ClientCnxn:
> > > > > Server
> > > > > > >> >> connection successful
> > > > > > >> >> 2009-11-18 12:42:01,782 WARN
> org.apache.zookeeper.ClientCnxn:
> > > > > > Exception
> > > > > > >> >> closing session 0x425035c48210002 to
> > > > > > >> sun.nio.ch.SelectionKeyImpl@56dc6fac
> > > > > > >> >> java.io.IOException: Read error rc = -1
> > > > > > java.nio.DirectByteBuffer[pos=0
> > > > > > >> >> lim=4 cap=4]
> > > > > > >> >> at
> > > > > > >> >>
> > > > >
> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
> > > > > > >> >> at
> > > > > > >> >>
> > > > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
> > > > > > >> >> 2009-11-18 12:42:01,782 WARN
> org.apache.zookeeper.ClientCnxn:
> > > > > > Ignoring
> > > > > > >> >> exception during shutdown input
> > > > > > >> >> java.net.SocketException: Transport endpoint is not
> connected
> > > > > > >> >> at sun.nio.ch.SocketChannelImpl.shutdown(Native
> > Method)
> > > > > > >> >> at
> > > > > > >> >>
> > > > > >
> > > sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
> > > > > > >> >> at
> > > > > > >> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> > > > > > >> >> at
> > > > > > >> >>
> > > > > >
> > > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> > > > > > >> >> at
> > > > > > >> >>
> > > > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> > > > > > >> >> 2009-11-18 12:42:01,782 WARN
> org.apache.zookeeper.ClientCnxn:
> > > > > > Ignoring
> > > > > > >> >> exception during shutdown output
> > > > > > >> >> java.net.SocketException: Transport endpoint is not
> connected
> > > > > > >> >> at sun.nio.ch.SocketChannelImpl.shutdown(Native
> > Method)
> > > > > > >> >> at
> > > > > > >> >>
> > > > > >
> > > sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
> > > > > > >> >> at
> > > > > > >>
> sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> > > > > > >> >> at
> > > > > > >> >>
> > > > > > >>
> > > > >
> > >
> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> > > > > > >> >> at
> > > > > > >> >>
> > > > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> > > > > > >> >> 2009-11-18 12:42:02,182 INFO
> org.apache.zookeeper.ClientCnxn:
> > > > > > >> Attempting
> > > > > > >> >> connection to server superpyxis0001/192.168.100.115:2181
> > > > > > >> >> 2009-11-18 12:42:02,182 INFO
> org.apache.zookeeper.ClientCnxn:
> > > > > Priming
> > > > > > >> >> connection to java.nio.channels.SocketChannel[connected
> > local=/
> > > > > > >> >> 192.168.100.132:36197 remote=superpyxis0001/
> > > 192.168.100.115:2181
> > > > ]
> > > > > > >> >> 2009-11-18 12:42:02,182 INFO
> org.apache.zookeeper.ClientCnxn:
> > > > > Server
> > > > > > >> >> connection successful
> > > > > > >> >> 2009-11-18 12:42:02,182 WARN
> org.apache.zookeeper.ClientCnxn:
> > > > > > Exception
> > > > > > >> >> closing session 0x425035c48210002 to
> > > > > > >> sun.nio.ch.SelectionKeyImpl@5c07076b
> > > > > > >> >> java.io.IOException: Read error rc = -1
> > > > > > java.nio.DirectByteBuffer[pos=0
> > > > > > >> >> lim=4 cap=4]
> > > > > > >> >> at
> > > > > > >> >>
> > > > >
> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
> > > > > > >> >> at
> > > > > > >> >>
> > > > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
> > > > > > >> >> 2009-11-18 12:42:02,182 WARN
> org.apache.zookeeper.ClientCnxn:
> > > > > > Ignoring
> > > > > > >> >> exception during shutdown input
> > > > > > >> >> java.net.SocketException: Transport endpoint is not
> connected
> > > > > > >> >> at sun.nio.ch.SocketChannelImpl.shutdown(Native
> > Method)
> > > > > > >> >> at
> > > > > > >> >>
> > > > > >
> > > sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
> > > > > > >> >> at
> > > > > > >> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> > > > > > >> >> at
> > > > > > >> >>
> > > > > >
> > > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> > > > > > >> >> at
> > > > > > >> >>
> > > > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> > > > > > >> >> 2009-11-18 12:42:02,182 WARN
> org.apache.zookeeper.ClientCnxn:
> > > > > > Ignoring
> > > > > > >> >> exception during shutdown output
> > > > > > >> >> java.net.SocketException: Transport endpoint is not
> connected
> > > > > > >> >>
> > > > > > >> >>
> > > > > > >> >> On Mon, Nov 16, 2009 at 4:16 PM, Zhenyu Zhong <
> > > > > > zhongresearch@gmail.com
> > > > > > >> >> >wrote:
> > > > > > >> >>
> > > > > > >> >> > Here is the diskIO and CPU around the time we had RS
> > > > > disconnection
> > > > > > on
> > > > > > >> >> one
> > > > > > >> >> > machine that runs RegionServer. It doesn't seem to be
> high.
> > > > > Similar
> > > > > > >> disk
> > > > > > >> >> and
> > > > > > >> >> > cpu usage have been seen before and the HBase was running
> > > fine.
> > > > > > >> >> >
> > > > > > >> >> >
> > > > > > >> >> > So far I haven't found why my 10 minutes session timeout
> > > > doesn't
> > > > > > >> apply.
> > > > > > >> >> > Still digging.
> > > > > > >> >> >
> > > > > > >> >> > Many thanks!
> > > > > > >> >> >
> > > > > > >> >> > Time: 03:01:08 PM
> > > > > > >> >> > avg-cpu: %user %nice %system %iowait %steal %idle
> > > > > > >> >> > 3.82 0.00 0.89 3.97 0.00 91.32
> > > > > > >> >> >
> > > > > > >> >> >
> > > > > > >> >> > Device: tps Blk_read/s Blk_wrtn/s
> Blk_read
> > > > > > >> Blk_wrtn
> > > > > > >> >> > sda 24.30 1605.39 6679.35
> 6238623388
> > > > > > >> 25956251720
> > > > > > >> >> > sda1 0.51 1.94 15.71
> 7547265
> > > > > > >> 61067904
> > > > > > >> >> > sda2 0.13 2.11 5.20
> 8202079
> > > > > > >> 20189232
> > > > > > >> >> > sda3 0.13 0.25 1.63
> 987846
> > > > > > >> 6323336
> > > > > > >> >> > sda4 0.00 0.00 0.00
> 4
> > > > > > >> 0
> > > > > > >> >> > sda5 1.78 0.06 24.14
> 220010
> > > > > > >> 93817208
> > > > > > >> >> > sda6 0.38 0.42 8.41
> 1630584
> > > > > > >> 32688152
> > > > > > >> >> > sda7 21.38 1600.61 6624.26
> 6220035272
> > > > > > >> 25742165888
> > > > > > >> >> > sdb 4.52 767.33 565.91
> 2981868690
> > > > > > >> 2199132380
> > > > > > >> >> > sdb1 4.52 767.33 565.91
> 2981866755
> > > > > > >> 2199132372
> > > > > > >> >> > sdc 4.42 742.95 563.00
> 2887151482
> > > > > > >> 2187823092
> > > > > > >> >> > sdc1 4.42 742.95 563.00
> 2887149547
> > > > > > >> 2187823084
> > > > > > >> >> > sdd 4.49 750.78 557.25
> 2917554074
> > > > > > >> 2165513500
> > > > > > >> >> > sdd1 4.49 750.78 557.25
> 2917552139
> > > > > > >> 2165513492
> > > > > > >> >> > sde 4.52 758.51 569.46
> 2947593394
> > > > > > >> 2212964236
> > > > > > >> >> > sde1 4.52 758.51 569.46
> 2947591459
> > > > > > >> 2212964228
> > > > > > >> >> > sdf 4.51 747.22 571.78
> 2903740266
> > > > > > >> 2221972652
> > > > > > >> >> > sdf1 4.51 747.22 571.78
> 2903738331
> > > > > > >> 2221972644
> > > > > > >> >> >
> > > > > > >> >> > Best,
> > > > > > >> >> > zhenyu
> > > > > > >> >> >
> > > > > > >> >> >
> > > > > > >> >> >
> > > > > > >> >> > On Mon, Nov 16, 2009 at 4:07 PM, stack <stack@duboce.net
> >
> > > > wrote:
> > > > > > >> >> >
> > > > > > >> >> >> On Mon, Nov 16, 2009 at 12:05 PM, Zhenyu Zhong <
> > > > > > >> >> zhongresearch@gmail.com
> > > > > > >> >> >> >wrote:
> > > > > > >> >> >>
> > > > > > >> >> >> > I just realized that there was a MapReduce job running
> > > > during
> > > > > > the
> > > > > > >> >> time
> > > > > > >> >> >> the
> > > > > > >> >> >> > regionserver disconnected from the zookeeper.
> > > > > > >> >> >> > That MapReduce Job was processing 500GB data and took
> > > about
> > > > 8
> > > > > > >> minutes
> > > > > > >> >> to
> > > > > > >> >> >> > finish. It launched over 2000 map tasks.
> > > > > > >> >> >>
> > > > > > >> >> >>
> > > > > > >> >> >> There was a tasktracker running on the RegionServer that
> > > > > > >> disconnected?
> > > > > > >> >> Is
> > > > > > >> >> >> the map i/o or cpu heavy? Do you think it could have
> > stole
> > > > life
> > > > > > >> from
> > > > > > >> >> the
> > > > > > >> >> >> datanode/regionserver?
> > > > > > >> >> >>
> > > > > > >> >> >>
> > > > > > >> >> >>
> > > > > > >> >> >> > I doubt that this introduced
> > > > > > >> >> >> > resource contention between DataNode and RegionServer
> > > Node.
> > > > > > >> >> >> > Also during the time that MapReduce job ran, I saw a
> few
> > > > > errors
> > > > > > >> >> >> indicating
> > > > > > >> >> >> > that
> > > > > > >> >> >> >
> > > > > > >> >> >> > java.io.IOException: Could not obtain block:
> > > > > > >> >> >> > blk_7661247556283580226_4450462
> > > > > file=/data/xxx.txt.200910141159
> > > > > > >> >> >> > at
> > > > > > >> >> >> >
> > > > > > >> >> >>
> > > > > > >> >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1787)
> > > > > > >> >> >> > at
> > > > > > >> >> >> >
> > > > > > >> >> >>
> > > > > > >> >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1615)
> > > > > > >> >> >> > at
> > > > > > >> >> >> >
> > > > > > >> >> >>
> > > > > > >> >>
> > > > > > >>
> > > > > >
> > > >
> > org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1742)
> > > > > > >> >> >> > at
> > > > > java.io.DataInputStream.read(DataInputStream.java:83)
> > > > > > >> >> >> > at
> > > > > > >> >> >>
> > > > org.apache.hadoop.util.LineReader.readLine(LineReader.java:134)
> > > > > > >> >> >> > at
> > > > > > >> >> >> >
> > > > > > >> >> >>
> > > > > > >> >>
> > > > > > >>
> > > > > >
> > > >
> > org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:136)
> > > > > > >> >> >> > at
> > > > > > >> >> >> >
> > > > > > >> >>
> > > > > > >>
> > > > >
> > >
> org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:40)
> > > > > > >> >> >> > at
> > > > > > >> >> >> >
> > > > > > >> >> >>
> > > > > > >> >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:192)
> > > > > > >> >> >> > at
> > > > > > >> >> >> >
> > > > > > >> >> >>
> > > > > > >> >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:176)
> > > > > > >> >> >> > at
> > > > > > >> org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
> > > > > > >> >> >> > at
> > > > > > >> >> >>
> > > > org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
> > > > > > >> >> >> > at
> > > > > org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
> > > > > > >> >> >> > at
> > > > org.apache.hadoop.mapred.Child.main(Child.java:170)
> > > > > > >> >> >> >
> > > > > > >> >> >>
> > > > > > >> >> >> This is interesting. Your map task is running into
> > > hdfs-127.
> > > > > > Yeah,
> > > > > > >> >> you
> > > > > > >> >> >> should patch your cluster if you want to get rid of
> these
> > > > > > (hdfs-127
> > > > > > >> has
> > > > > > >> >> >> been
> > > > > > >> >> >> applied to 0.21 hadoop and will be in the next hadoop
> > > release
> > > > on
> > > > > > >> 0.20.x
> > > > > > >> >> >> branch, hadoop 0.20.2).
> > > > > > >> >> >>
> > > > > > >> >> >>
> > > > > > >> >> >>
> > > > > > >> >> >> >
> > > > > > >> >> >> > Possibly it is related to HDFS-127, but I don't see
> any
> > > > > > datanodes
> > > > > > >> >> went
> > > > > > >> >> >> > down. Does that affect Regionserver? Shall we apply
> the
> > > > patch?
> > > > > > >> >> >> >
> > > > > > >> >> >> >
> > > > > > >> >> >> Read up on hdfs-127 for explaination of whats going on.
> > > > > > >> >> >>
> > > > > > >> >> >>
> > > > > > >> >> >>
> > > > > > >> >> >> > Now I start to keep tracking of the virtual memory
> stats
> > > to
> > > > > see
> > > > > > if
> > > > > > >> >> the
> > > > > > >> >> >> same
> > > > > > >> >> >> > issue happens tomorrow around the same time period.
> > > > > > >> >> >> >
> > > > > > >> >> >>
> > > > > > >> >> >> Excellent. Thanks for doing this.
> > > > > > >> >> >>
> > > > > > >> >> >>
> > > > > > >> >> >>
> > > > > > >> >> >> > And I highly suspect that this particular MapReduce
> job
> > > > hurts
> > > > > > >> HBase
> > > > > > >> >> >> > Regionserver.
> > > > > > >> >> >> >
> > > > > > >> >> >> > PS. I also use gcviewer to parse the GC-log, I only
> see
> > > > around
> > > > > > 30
> > > > > > >> >> >> seconds
> > > > > > >> >> >> > pauses maximum.
> > > > > > >> >> >> >
> > > > > > >> >> >>
> > > > > > >> >> >>
> > > > > > >> >> >> Did you figure why your ten minute session is not
> > applying?
> > > > > > >> >> >>
> > > > > > >> >> >> St.Ack
> > > > > > >> >> >>
> > > > > > >> >> >>
> > > > > > >> >> >>
> > > > > > >> >> >> >
> > > > > > >> >> >> > Thanks a lot.
> > > > > > >> >> >> > zhenyu
> > > > > > >> >> >> >
> > > > > > >> >> >> >
> > > > > > >> >> >> >
> > > > > > >> >> >> > On Sun, Nov 15, 2009 at 12:52 AM, Zhenyu Zhong <
> > > > > > >> >> zhongresearch@gmail.com
> > > > > > >> >> >> > >wrote:
> > > > > > >> >> >> >
> > > > > > >> >> >> > > J-D,
> > > > > > >> >> >> > >
> > > > > > >> >> >> > > Thank you very much for your comments.
> > > > > > >> >> >> > > My company block the IRC port, so I have trouble to
> > > > connect
> > > > > to
> > > > > > >> IRC
> > > > > > >> >> >> > channel.
> > > > > > >> >> >> > > I have been trying to ask the IT to open the IRC
> port
> > > for
> > > > > me,
> > > > > > it
> > > > > > >> >> might
> > > > > > >> >> >> > take
> > > > > > >> >> >> > > a while.
> > > > > > >> >> >> > >
> > > > > > >> >> >> > > Best,
> > > > > > >> >> >> > > zhenyu
> > > > > > >> >> >> > >
> > > > > > >> >> >> > >
> > > > > > >> >> >> > > On Sat, Nov 14, 2009 at 2:21 PM, Jean-Daniel Cryans
> <
> > > > > > >> >> >> jdcryans@apache.org
> > > > > > >> >> >> > >wrote:
> > > > > > >> >> >> > >
> > > > > > >> >> >> > >> The error you are getting is a disconnection from a
> > > > > zookeeper
> > > > > > >> >> server
> > > > > > >> >> >> > >> and is very generic.
> > > > > > >> >> >> > >>
> > > > > > >> >> >> > >> ZK-86 is still opened and the last comment refers
> to
> > > > ZK-111
> > > > > > >> saying
> > > > > > >> >> >> > >> that the bug (in unit tests) was probably fixed in
> > > > release
> > > > > > >> 3.0.0
> > > > > > >> >> last
> > > > > > >> >> >> > >> year.
> > > > > > >> >> >> > >>
> > > > > > >> >> >> > >> To figure the hang you have, you can try to jstack
> > the
> > > > > > process
> > > > > > >> pid
> > > > > > >> >> >> and
> > > > > > >> >> >> > >> see exactly what's holding the RS from shutting
> down.
> > > > > > >> >> >> > >>
> > > > > > >> >> >> > >> Would it be possible for you to drop by the IRC
> > > channel?
> > > > > This
> > > > > > >> way
> > > > > > >> >> we
> > > > > > >> >> >> > >> can debug this a at much faster pace.
> > > > > > >> >> >> > >>
> > > > > > >> >> >> > >> Thx!
> > > > > > >> >> >> > >>
> > > > > > >> >> >> > >> J-D
> > > > > > >> >> >> > >>
> > > > > > >> >> >> > >> On Sat, Nov 14, 2009 at 9:57 AM, Zhenyu Zhong <
> > > > > > >> >> >> zhongresearch@gmail.com>
> > > > > > >> >> >> > >> wrote:
> > > > > > >> >> >> > >> > I found this.
> > > > > > >> >> >> > >> >
> http://issues.apache.org/jira/browse/ZOOKEEPER-86
> > > > > > >> >> >> > >> >
> > > > > > >> >> >> > >> > It looks like the same error I had. Is it a
> > zookeeper
> > > > > bug?
> > > > > > >> When
> > > > > > >> >> >> will
> > > > > > >> >> >> > >> HBase
> > > > > > >> >> >> > >> > take the zookeeper version 3.3.0?
> > > > > > >> >> >> > >> >
> > > > > > >> >> >> > >> > thanks
> > > > > > >> >> >> > >> > zhenyu
> > > > > > >> >> >> > >> >
> > > > > > >> >> >> > >> > On Sat, Nov 14, 2009 at 11:31 AM, Zhenyu Zhong <
> > > > > > >> >> >> > zhongresearch@gmail.com
> > > > > > >> >> >> > >> >wrote:
> > > > > > >> >> >> > >> >
> > > > > > >> >> >> > >> >> Now I really doubt about the zookeeper, from the
> > log
> > > I
> > > > > saw
> > > > > > >> >> errors
> > > > > > >> >> >> > like
> > > > > > >> >> >> > >> >> IOException Read Error, while the zookeeper
> client
> > > > > > >> >> (regionserver )
> > > > > > >> >> >> > >> tried to
> > > > > > >> >> >> > >> >> read. But it got disconneted status from
> > zookeeper.
> > > > > > >> >> >> > >> >>
> > > > > > >> >> >> > >> >> I don't see any load on any zookeeper quorum
> > > servers.
> > > > > > DiskIO
> > > > > > >> is
> > > > > > >> >> >> fine.
> > > > > > >> >> >> > >> >>
> > > > > > >> >> >> > >> >> Also when the regionserver decides to exit due
> to
> > > the
> > > > > > >> >> disconnect
> > > > > > >> >> >> > status
> > > > > > >> >> >> > >> >> from zookeeper, sometimes the regionserver hangs
> > > > during
> > > > > > the
> > > > > > >> >> >> exiting.
> > > > > > >> >> >> > We
> > > > > > >> >> >> > >> can
> > > > > > >> >> >> > >> >> still see the HRegionServer process even we
> don't
> > > see
> > > > it
> > > > > > in
> > > > > > >> the
> > > > > > >> >> >> > Master
> > > > > > >> >> >> > >> web
> > > > > > >> >> >> > >> >> interface.
> > > > > > >> >> >> > >> >>
> > > > > > >> >> >> > >> >> I also notice that there is an zookeeper.retries
> > and
> > > > > > >> >> >> zookeeper.pauses
> > > > > > >> >> >> > >> >> settings in hbase-default.xml, would that matter
> ?
> > > > > > >> >> >> > >> >>
> > > > > > >> >> >> > >> >> thanks
> > > > > > >> >> >> > >> >>
> > > > > > >> >> >> > >> >> You may see the IOException error in the
> following
> > > log
> > > > > > when
> > > > > > >> the
> > > > > > >> >> >> > >> >> Regionserver lost connection to the zookeeper.
> > > > > > >> >> >> > >> >>
> > > > > > >> >> >> > >> >> 2009-11-14 15:58:31,993 INFO
> > > > > > >> >> >> > org.apache.hadoop.hbase.regionserver.HLog:
> > > > > > >> >> >> > >> >> Roll
> > > > > > >> >> >> > >> >>
> > > > > > >> >> >> > >>
> > > > > > >> >> >> >
> > > > > > >> >> >>
> > > > > > >> >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> /hbase/.logs/superpyxis0008.scur.colo,60021,1258179101617/hlog.dat.1258179106921,
> > > > > > >> >> >> > >> >> entries=1062964, calcsize=255014936,
> > > > filesize=139566093.
> > > > > > New
> > > > > > >> >> hlog
> > > > > > >> >> >> > >> >>
> > > > > > >> >> >> > >>
> > > > > > >> >> >> >
> > > > > > >> >> >>
> > > > > > >> >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> /hbase/.logs/superpyxis0008.scur.colo,60021,1258179101617/hlog.dat.1258214311993
> > > > > > >> >> >> > >> >> 2009-11-14 15:58:39,295 DEBUG
> > > > > > >> org.apache.zookeeper.ClientCnxn:
> > > > > > >> >> Got
> > > > > > >> >> >> > ping
> > > > > > >> >> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after
> 0ms
> > > > > > >> >> >> > >> >> 2009-11-14 15:58:52,648 DEBUG
> > > > > > >> org.apache.zookeeper.ClientCnxn:
> > > > > > >> >> Got
> > > > > > >> >> >> > ping
> > > > > > >> >> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after
> 0ms
> > > > > > >> >> >> > >> >> 2009-11-14 15:59:06,007 DEBUG
> > > > > > >> org.apache.zookeeper.ClientCnxn:
> > > > > > >> >> Got
> > > > > > >> >> >> > ping
> > > > > > >> >> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after
> 0ms
> > > > > > >> >> >> > >> >> 2009-11-14 15:59:19,365 DEBUG
> > > > > > >> org.apache.zookeeper.ClientCnxn:
> > > > > > >> >> Got
> > > > > > >> >> >> > ping
> > > > > > >> >> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after
> 0ms
> > > > > > >> >> >> > >> >> 2009-11-14 15:59:32,867 DEBUG
> > > > > > >> org.apache.zookeeper.ClientCnxn:
> > > > > > >> >> Got
> > > > > > >> >> >> > ping
> > > > > > >> >> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after
> > 186ms
> > > > > > >> >> >> > >> >> 2009-11-14 15:59:46,070 DEBUG
> > > > > > >> org.apache.zookeeper.ClientCnxn:
> > > > > > >> >> Got
> > > > > > >> >> >> > ping
> > > > > > >> >> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after
> 0ms
> > > > > > >> >> >> > >> >> 2009-11-14 15:59:59,423 DEBUG
> > > > > > >> org.apache.zookeeper.ClientCnxn:
> > > > > > >> >> Got
> > > > > > >> >> >> > ping
> > > > > > >> >> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after
> 0ms
> > > > > > >> >> >> > >> >> 2009-11-14 16:00:12,775 DEBUG
> > > > > > >> org.apache.zookeeper.ClientCnxn:
> > > > > > >> >> Got
> > > > > > >> >> >> > ping
> > > > > > >> >> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after
> 0ms
> > > > > > >> >> >> > >> >> 2009-11-14 16:00:28,141 DEBUG
> > > > > > >> org.apache.zookeeper.ClientCnxn:
> > > > > > >> >> Got
> > > > > > >> >> >> > ping
> > > > > > >> >> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after
> > > 2010ms
> > > > > > >> >> >> > >> >> 2009-11-14 16:01:10,378 WARN
> > > > > > >> org.apache.zookeeper.ClientCnxn:
> > > > > > >> >> >> > Exception
> > > > > > >> >> >> > >> >> closing session 0x424f0ed01ea00a6 to
> > > > > > >> >> >> > >> sun.nio.ch.SelectionKeyImpl@39fbb2d6
> > > > > > >> >> >> > >> >>
> > > > > > >> >> >> > >> >> java.io.IOException: Read error rc = -1
> > > > > > >> >> >> > java.nio.DirectByteBuffer[pos=0
> > > > > > >> >> >> > >> >> lim=4 cap=4]
> > > > > > >> >> >> > >> >> at
> > > > > > >> >> >> > >> >>
> > > > > > >> >> >>
> > > > > >
> > org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
> > > > > > >> >> >> > >> >> at
> > > > > > >> >> >> > >> >>
> > > > > > >> >> >>
> > > > > >
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
> > > > > > >> >> >> > >> >> 2009-11-14 16:01:10,478 INFO
> > > > > > >> >> >> > >> >>
> > org.apache.hadoop.hbase.regionserver.HRegionServer:
> > > > Got
> > > > > > >> >> ZooKeeper
> > > > > > >> >> >> > >> event,
> > > > > > >> >> >> > >> >> state: Disconnected, type: None, path: null
> > > > > > >> >> >> > >> >> 2009-11-14 16:01:11,333 WARN
> > > > > > >> >> >> > >> >>
> > org.apache.hadoop.hbase.regionserver.HRegionServer:
> > > > > > >> Attempt=1
> > > > > > >> >> >> > >> >>
> > > > > > >> >> >> > >> >>
> > > org.apache.hadoop.hbase.Leases$LeaseStillHeldException
> > > > > > >> >> >> > >> >> at
> > > > > > >> >> >> > >>
> > > > > sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> > > > > > >> >> >> > >> >> Method)
> > > > > > >> >> >> > >> >> at
> > > > > > >> >> >> > >> >>
> > > > > > >> >> >> > >>
> > > > > > >> >> >> >
> > > > > > >> >> >>
> > > > > > >> >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> > > > > > >> >> >> > >> >> at
> > > > > > >> >> >> > >> >>
> > > > > > >> >> >> > >>
> > > > > > >> >> >> >
> > > > > > >> >> >>
> > > > > > >> >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> > > > > > >> >> >> > >> >> at
> > > > > > >> >> >> > >>
> > > > > > java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> > > > > > >> >> >> > >> >> at
> > > > > > >> >> >> > >> >>
> > > > > > >> >> >> > >>
> > > > > > >> >> >> >
> > > > > > >> >> >>
> > > > > > >> >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
> > > > > > >> >> >> > >> >> at
> > > > > > >> >> >> > >> >>
> > > > > > >> >> >> > >>
> > > > > > >> >> >> >
> > > > > > >> >> >>
> > > > > > >> >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
> > > > > > >> >> >> > >> >> at
> > > > > > >> >> >> > >> >>
> > > > > > >> >> >> > >>
> > > > > > >> >> >> >
> > > > > > >> >> >>
> > > > > > >> >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
> > > > > > >> >> >> > >> >> at
> > > > > > >> >> >> > >> >>
> > > > > > >> >> >> > >>
> > > > > > >> >> >> >
> > > > > > >> >> >>
> > > > > > >> >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:571)
> > > > > > >> >> >> > >> >> at java.lang.Thread.run(Thread.java:619)
> > > > > > >> >> >> > >> >> 2009-11-14 16:01:11,433 INFO
> > > > > > >> org.apache.zookeeper.ClientCnxn:
> > > > > > >> >> >> > >> Attempting
> > > > > > >> >> >> > >> >> connection to server superpyxis0001/
> > > > > 192.168.100.115:2181
> > > > > > >> >> >> > >> >> 2009-11-14 16:01:11,433 INFO
> > > > > > >> org.apache.zookeeper.ClientCnxn:
> > > > > > >> >> >> Priming
> > > > > > >> >> >> > >> >> connection to
> > > > java.nio.channels.SocketChannel[connected
> > > > > > >> local=/
> > > > > > >> >> >> > >> >> 192.168.100.122:59575 remote=superpyxis0001/
> > > > > > >> >> 192.168.100.115:2181]
> > > > > > >> >> >> > >> >> 2009-11-14 16:01:11,433 INFO
> > > > > > >> org.apache.zookeeper.ClientCnxn:
> > > > > > >> >> >> Server
> > > > > > >> >> >> > >> >> connection successful
> > > > > > >> >> >> > >> >> 2009-11-14 16:01:11,433 INFO
> > > > > > >> >> >> > >> >>
> > org.apache.hadoop.hbase.regionserver.HRegionServer:
> > > > Got
> > > > > > >> >> ZooKeeper
> > > > > > >> >> >> > >> event,
> > > > > > >> >> >> > >> >> state: Expired, type: None, path: null
> > > > > > >> >> >> > >> >> 2009-11-14 16:01:11,433 WARN
> > > > > > >> org.apache.zookeeper.ClientCnxn:
> > > > > > >> >> >> > Exception
> > > > > > >> >> >> > >> >> closing session 0x424f0ed01ea00a6 to
> > > > > > >> >> >> > >> sun.nio.ch.SelectionKeyImpl@6b93d343
> > > > > > >> >> >> > >> >> java.io.IOException: Session Expired
> > > > > > >> >> >> > >> >> at
> > > > > > >> >> >> > >> >>
> > > > > > >> >> >> > >>
> > > > > > >> >> >> >
> > > > > > >> >> >>
> > > > > > >> >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.zookeeper.ClientCnxn$SendThread.readConnectResult(ClientCnxn.java:589)
> > > > > > >> >> >> > >> >> at
> > > > > > >> >> >> > >> >>
> > > > > > >> >> >>
> > > > > >
> > org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:709)
> > > > > > >> >> >> > >> >>
> > > > > > >> >> >> > >> >> at
> > > > > > >> >> >> > >> >>
> > > > > > >> >> >>
> > > > > >
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
> > > > > > >> >> >> > >> >> 2009-11-14 16:01:11,433 ERROR
> > > > > > >> >> >> > >> >>
> > org.apache.hadoop.hbase.regionserver.HRegionServer:
> > > > > > >> ZooKeeper
> > > > > > >> >> >> session
> > > > > > >> >> >> > >> >> expired
> > > > > > >> >> >> > >> >> 2009-11-14 16:01:13,513 INFO
> > > > > > >> org.apache.hadoop.ipc.HBaseServer:
> > > > > > >> >> >> IPC
> > > > > > >> >> >> > >> Server
> > > > > > >> >> >> > >> >> handler 3 on 60021, call put([B@73deb204,
> > > > > > >> >> >> > >> >> [Lorg.apache.hadoop.hbase.client.Put;@2179600a)
> > > from
> > > > > > >> >> >> > >> 192.168.100.132:40728:
> > > > > > >> >> >> > >> >> error: java.io.IOException: Server not running,
> > > > aborting
> > > > > > >> >> >> > >> >> java.io.IOException: Server not running,
> aborting
> > > > > > >> >> >> > >> >>
> > > > > > >> >> >> > >> >>
> > > > > > >> >> >> > >> >>
> > > > > > >> >> >> > >> >>
> > > > > > >> >> >> > >> >>
> > > > > > >> >> >> > >> >> On Sat, Nov 14, 2009 at 1:01 AM, Zhenyu Zhong <
> > > > > > >> >> >> > zhongresearch@gmail.com
> > > > > > >> >> >> > >> >wrote:
> > > > > > >> >> >> > >> >>
> > > > > > >> >> >> > >> >>> Stack,
> > > > > > >> >> >> > >> >>>
> > > > > > >> >> >> > >> >>> Thanks a lot!
> > > > > > >> >> >> > >> >>>
> > > > > > >> >> >> > >> >>> I found out that the reason why HBase doesn't
> > take
> > > > the
> > > > > > >> system
> > > > > > >> >> >> file
> > > > > > >> >> >> > >> >>> descriptor value I set before. I started the
> > HBase
> > > > > using
> > > > > > >> root
> > > > > > >> >> >> > instead
> > > > > > >> >> >> > >> of the
> > > > > > >> >> >> > >> >>> normal hadoop user, while my system configures
> > > hadoop
> > > > > > with
> > > > > > >> >> higher
> > > > > > >> >> >> > file
> > > > > > >> >> >> > >> >>> descriptor value but configures root with 1024
> > > > default
> > > > > > >> value.
> > > > > > >> >> >> > >> >>>
> > > > > > >> >> >> > >> >>>
> > > > > > >> >> >> > >> >>> Now my system has a clean start. However, it
> > seems
> > > > that
> > > > > > the
> > > > > > >> >> >> > >> >>> zookeeper.session.timeout value doesn't take
> into
> > > > > effect.
> > > > > > I
> > > > > > >> >> still
> > > > > > >> >> >> > >> found
> > > > > > >> >> >> > >> >>> around 60 seconds pauses from the disconnected
> > > > > > >> regionserver. I
> > > > > > >> >> >> > really
> > > > > > >> >> >> > >> don't
> > > > > > >> >> >> > >> >>> know why regionserver only times out after 60
> > > seconds
> > > > > > >> instead
> > > > > > >> >> of
> > > > > > >> >> >> 10
> > > > > > >> >> >> > >> minutes
> > > > > > >> >> >> > >> >>> which I set for zookeeper.session.timeout.
> > > > > > >> >> >> > >> >>>
> > > > > > >> >> >> > >> >>> Is there any other timeout value coming into
> play
> > > > > before
> > > > > > >> the
> > > > > > >> >> >> actual
> > > > > > >> >> >> > >> >>> session times out?
> > > > > > >> >> >> > >> >>>
> > > > > > >> >> >> > >> >>> zhenyu
> > > > > > >> >> >> > >> >>>
> > > > > > >> >> >> > >> >>>
> > > > > > >> >> >> > >> >>>
> > > > > > >> >> >> > >> >>> On Fri, Nov 13, 2009 at 7:08 PM, stack <
> > > > > stack@duboce.net
> > > > > > >
> > > > > > >> >> wrote:
> > > > > > >> >> >> > >> >>>
> > > > > > >> >> >> > >> >>>> Ok. Lack of file descriptors manifests in all
> > > kinds
> > > > > of
> > > > > > >> weird
> > > > > > >> >> >> ways.
> > > > > > >> >> >> > >> >>>> Hopefully thats it. If not, lets keep
> digging.
> > > > > > >> >> >> > >> >>>> St.Ack
> > > > > > >> >> >> > >> >>>>
> > > > > > >> >> >> > >> >>>> On Fri, Nov 13, 2009 at 3:44 PM, Zhenyu Zhong
> <
> > > > > > >> >> >> > >> zhongresearch@gmail.com
> > > > > > >> >> >> > >> >>>> >wrote:
> > > > > > >> >> >> > >> >>>>
> > > > > > >> >> >> > >> >>>> > Stack,
> > > > > > >> >> >> > >> >>>> >
> > > > > > >> >> >> > >> >>>> > You are right, the master started with
> ulimit
> > -n
> > > > > 1024.
> > > > > > >> It
> > > > > > >> >> >> doesn't
> > > > > > >> >> >> > >> take
> > > > > > >> >> >> > >> >>>> the
> > > > > > >> >> >> > >> >>>> > system value.
> > > > > > >> >> >> > >> >>>> >
> > > > > > >> >> >> > >> >>>> > Regarding to the too many open files, it
> looks
> > > > like
> > > > > > the
> > > > > > >> >> same
> > > > > > >> >> >> as
> > > > > > >> >> >> > the
> > > > > > >> >> >> > >> one
> > > > > > >> >> >> > >> >>>> J-D
> > > > > > >> >> >> > >> >>>> > put up. But I will get the Master start with
> > > > higher
> > > > > > >> value
> > > > > > >> >> >> first
> > > > > > >> >> >> > and
> > > > > > >> >> >> > >> see
> > > > > > >> >> >> > >> >>>> if
> > > > > > >> >> >> > >> >>>> > this kind of error goes away.
> > > > > > >> >> >> > >> >>>> >
> > > > > > >> >> >> > >> >>>> > thanks a lot!
> > > > > > >> >> >> > >> >>>> > zhenyu
> > > > > > >> >> >> > >> >>>> >
> > > > > > >> >> >> > >> >>>> > On Fri, Nov 13, 2009 at 6:02 PM, stack <
> > > > > > >> stack@duboce.net>
> > > > > > >> >> >> wrote:
> > > > > > >> >> >> > >> >>>> >
> > > > > > >> >> >> > >> >>>> > > Does it say
> > > > > > >> >> >> > >> >>>> > >
> > > > > > >> >> >> > >> >>>> > > ulimit -n 32768
> > > > > > >> >> >> > >> >>>> > >
> > > > > > >> >> >> > >> >>>> > > ...as the second line in your log file on
> > > start
> > > > of
> > > > > > the
> > > > > > >> >> >> master?
> > > > > > >> >> >> > >> >>>> > >
> > > > > > >> >> >> > >> >>>> > > Is the stack trace that complains about
> too
> > > many
> > > > > > open
> > > > > > >> >> files
> > > > > > >> >> >> > same
> > > > > > >> >> >> > >> as
> > > > > > >> >> >> > >> >>>> the
> > > > > > >> >> >> > >> >>>> > one
> > > > > > >> >> >> > >> >>>> > > in the blog post J-D put up?
> > > > > > >> >> >> > >> >>>> > >
> > > > > > >> >> >> > >> >>>> > > St.Ack
> > > > > > >> >> >> > >> >>>> > >
> > > > > > >> >> >> > >> >>>> > >
> > > > > > >> >> >> > >> >>>> > > On Fri, Nov 13, 2009 at 1:37 PM, Zhenyu
> > Zhong
> > > <
> > > > > > >> >> >> > >> >>>> zhongresearch@gmail.com
> > > > > > >> >> >> > >> >>>> > > >wrote:
> > > > > > >> >> >> > >> >>>> > >
> > > > > > >> >> >> > >> >>>> > > > The ulimit file descriptors was set to
> > > > > fs.file-max
> > > > > > >> >> >> =1578334,
> > > > > > >> >> >> > >> also
> > > > > > >> >> >> > >> >>>> in
> > > > > > >> >> >> > >> >>>> > the
> > > > > > >> >> >> > >> >>>> > > > limits.conf the value was set to 32768.
> > > > > > >> >> >> > >> >>>> > > > So these are way higher than the open
> > > > > descriptors
> > > > > > >> for
> > > > > > >> >> the
> > > > > > >> >> >> > >> running
> > > > > > >> >> >> > >> >>>> > > > processes.
> > > > > > >> >> >> > >> >>>> > > >
> > > > > > >> >> >> > >> >>>> > > > thanks
> > > > > > >> >> >> > >> >>>> > > > zhenyu
> > > > > > >> >> >> > >> >>>> > > >
> > > > > > >> >> >> > >> >>>> > > >
> > > > > > >> >> >> > >> >>>> > > > On Fri, Nov 13, 2009 at 4:33 PM, Stack <
> > > > > > >> >> >> saint.ack@gmail.com>
> > > > > > >> >> >> > >> >>>> wrote:
> > > > > > >> >> >> > >> >>>> > > >
> > > > > > >> >> >> > >> >>>> > > > > You upped the ulimit file descriptors
> as
> > > per
> > > > > the
> > > > > > >> >> getting
> > > > > > >> >> >> > >> started
> > > > > > >> >> >> > >> >>>> doc?
> > > > > > >> >> >> > >> >>>> > > > >
> > > > > > >> >> >> > >> >>>> > > > >
> > > > > > >> >> >> > >> >>>> > > > >
> > > > > > >> >> >> > >> >>>> > > > > On Nov 13, 2009, at 1:26 PM, Zhenyu
> > Zhong
> > > <
> > > > > > >> >> >> > >> >>>> zhongresearch@gmail.com>
> > > > > > >> >> >> > >> >>>> > > > wrote:
> > > > > > >> >> >> > >> >>>> > > > >
> > > > > > >> >> >> > >> >>>> > > > > Thanks a lot.
> > > > > > >> >> >> > >> >>>> > > > >>
> > > > > > >> >> >> > >> >>>> > > > >>
> > > > > > >> >> >> > >> >>>> > > > >> Bad news is my kernel is still
> 2.6.26.
> > > > > > >> >> >> > >> >>>> > > > >> But it was not a problem before.
> > > > > > >> >> >> > >> >>>> > > > >>
> > > > > > >> >> >> > >> >>>> > > > >> Very strange.
> > > > > > >> >> >> > >> >>>> > > > >>
> > > > > > >> >> >> > >> >>>> > > > >> zhenyu
> > > > > > >> >> >> > >> >>>> > > > >>
> > > > > > >> >> >> > >> >>>> > > > >> On Fri, Nov 13, 2009 at 4:16 PM,
> > > > Jean-Daniel
> > > > > > >> Cryans
> > > > > > >> >> <
> > > > > > >> >> >> > >> >>>> > > > jdcryans@apache.org
> > > > > > >> >> >> > >> >>>> > > > >> >wrote:
> > > > > > >> >> >> > >> >>>> > > > >>
> > > > > > >> >> >> > >> >>>> > > > >> Looks like
> > > > > > >> >> >> > >> >>>> > > > >>>
> > > > > > >> >> >> > >> >>>> > > > >>>
> > > > > > >> >> >> > >> >>>> > > >
> > > > > > >> >> >> > >> >>>> > >
> > > > > > >> >> >> > >> >>>> >
> > > > > > >> >> >> > >> >>>>
> > > > > > >> >> >> > >>
> > > > > > >> >> >> >
> > > > > > >> >> >>
> > > > > > >> >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> http://pero.blogs.aprilmayjune.org/2009/01/22/hadoop-and-linux-kernel-2627-epoll-limits/
> > > > > > >> >> >> > >> >>>> > > > >>>
> > > > > > >> >> >> > >> >>>> > > > >>> J-D
> > > > > > >> >> >> > >> >>>> > > > >>>
> > > > > > >> >> >> > >> >>>> > > > >>> On Fri, Nov 13, 2009 at 1:12 PM,
> > Zhenyu
> > > > > Zhong
> > > > > > <
> > > > > > >> >> >> > >> >>>> > > zhongresearch@gmail.com
> > > > > > >> >> >> > >> >>>> > > > >
> > > > > > >> >> >> > >> >>>> > > > >>> wrote:
> > > > > > >> >> >> > >> >>>> > > > >>>
> > > > > > >> >> >> > >> >>>> > > > >>>> Hi,
> > > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > > >> >> >> > >> >>>> > > > >>>> After I re-organize the cluster,
> the
> > > > > > experiment
> > > > > > >> >> ran
> > > > > > >> >> >> into
> > > > > > >> >> >> > >> >>>> problem
> > > > > > >> >> >> > >> >>>> > > > faster
> > > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > > >> >> >> > >> >>>> > > > >>> than
> > > > > > >> >> >> > >> >>>> > > > >>>
> > > > > > >> >> >> > >> >>>> > > > >>>> before.
> > > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > > >> >> >> > >> >>>> > > > >>>> Basically the changes are to use
> > > machines
> > > > > > with
> > > > > > >> >> less
> > > > > > >> >> >> > >> resources
> > > > > > >> >> >> > >> >>>> as
> > > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > > >> >> >> > >> >>>> > > > >>> zookeeper
> > > > > > >> >> >> > >> >>>> > > > >>>
> > > > > > >> >> >> > >> >>>> > > > >>>> quorums and machines with more
> > > resources
> > > > as
> > > > > > >> >> >> > regionserver.
> > > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > > >> >> >> > >> >>>> > > > >>>> From the log, I still see the pause
> > > > around
> > > > > 1
> > > > > > >> >> minute.
> > > > > > >> >> >> > >> >>>> > > > >>>> I enabled the GC logging, please
> see
> > > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > > >> >> >> > >> >>>> > > > >>>> http://pastebin.com/m1d4ce0f1
> > > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > > >> >> >> > >> >>>> > > > >>>> for details.
> > > > > > >> >> >> > >> >>>> > > > >>>> In general I don't see many pauses
> in
> > > the
> > > > > GC.
> > > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > > >> >> >> > >> >>>> > > > >>>> What is more interesting, 1 hour
> > after
> > > > the
> > > > > > 1st
> > > > > > >> >> >> > >> regionserver
> > > > > > >> >> >> > >> >>>> > > > >>>> disconnected,
> > > > > > >> >> >> > >> >>>> > > > >>>> the master log complained about too
> > > many
> > > > > open
> > > > > > >> >> files.
> > > > > > >> >> >> > This
> > > > > > >> >> >> > >> >>>> didn't
> > > > > > >> >> >> > >> >>>> > > > happen
> > > > > > >> >> >> > >> >>>> > > > >>>> before.
> > > > > > >> >> >> > >> >>>> > > > >>>> I checked the system OS setup as
> well
> > > as
> > > > > the
> > > > > > >> >> >> > limits.conf.
> > > > > > >> >> >> > >> I
> > > > > > >> >> >> > >> >>>> also
> > > > > > >> >> >> > >> >>>> > > > checked
> > > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > > >> >> >> > >> >>>> > > > >>> the
> > > > > > >> >> >> > >> >>>> > > > >>>
> > > > > > >> >> >> > >> >>>> > > > >>>> running processes. The total open
> > files
> > > > > don't
> > > > > > >> >> reach
> > > > > > >> >> >> the
> > > > > > >> >> >> > >> limit.
> > > > > > >> >> >> > >> >>>> I
> > > > > > >> >> >> > >> >>>> > am
> > > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > > >> >> >> > >> >>>> > > > >>> confused
> > > > > > >> >> >> > >> >>>> > > > >>>
> > > > > > >> >> >> > >> >>>> > > > >>>> a bit.
> > > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > > >> >> >> > >> >>>> > > > >>>> Please see the following master
> log.
> > > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:02,114 INFO
> > > > > > >> >> >> > >> >>>> > > >
> > org.apache.hadoop.hbase.master.BaseScanner:
> > > > > > >> >> >> > >> >>>> > > > >>>> RegionManager.metaScanner scan of
> > 4658
> > > > > row(s)
> > > > > > >> of
> > > > > > >> >> meta
> > > > > > >> >> >> > >> region
> > > > > > >> >> >> > >> >>>> > > {server:
> > > > > > >> >> >> > >> >>>> > > > >>>> 192.168.100.128:60021, regionname:
> > > > > > .META.,,1,
> > > > > > >> >> >> startKey:
> > > > > > >> >> >> > >> <>}
> > > > > > >> >> >> > >> >>>> > > complete
> > > > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:02,114 INFO
> > > > > > >> >> >> > >> >>>> > > >
> > org.apache.hadoop.hbase.master.BaseScanner:
> > > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > > >> >> >> > >> >>>> > > > >>> All
> > > > > > >> >> >> > >> >>>> > > > >>>
> > > > > > >> >> >> > >> >>>> > > > >>>> 1 .META. region(s) scanned
> > > > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:07,677 DEBUG
> > > > > > >> >> >> > >> org.apache.zookeeper.ClientCnxn:
> > > > > > >> >> >> > >> >>>> Got
> > > > > > >> >> >> > >> >>>> > > > ping
> > > > > > >> >> >> > >> >>>> > > > >>>> response for
> > > sessionid:0x424eebf1c10004c
> > > > > > after
> > > > > > >> 3ms
> > > > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:08,178 INFO
> > > > > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > > > > >> >> >> > >> >>>> > > > Exception
> > > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > > >> >> >> > >> >>>> > > > >>> in
> > > > > > >> >> >> > >> >>>> > > > >>>
> > > > > > >> >> >> > >> >>>> > > > >>>> createBlockOutputStream
> > > > > java.io.IOException:
> > > > > > >> Bad
> > > > > > >> >> >> connect
> > > > > > >> >> >> > >> ack
> > > > > > >> >> >> > >> >>>> with
> > > > > > >> >> >> > >> >>>> > > > >>>> firstBadLink 192.168.100.123:50010
> > > > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:08,178 INFO
> > > > > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > > > > >> >> >> > >> >>>> > > > >>>> Abandoning
> > > > > > >> >> >> > >> >>>> > > > >>>> block
> > blk_-2808245019291145247_5478039
> > > > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:09,682 INFO
> > > > > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > > > > >> >> >> > >> >>>> > > > Exception
> > > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > > >> >> >> > >> >>>> > > > >>> in
> > > > > > >> >> >> > >> >>>> > > > >>>
> > > > > > >> >> >> > >> >>>> > > > >>>> createBlockOutputStream
> > > > > java.io.EOFException
> > > > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:09,682 INFO
> > > > > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > > > > >> >> >> > >> >>>> > > > >>>> Abandoning
> > > > > > >> >> >> > >> >>>> > > > >>>> block
> blk_1074853606841896259_5478048
> > > > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:10,334 DEBUG
> > > > > > >> >> >> > >> org.apache.zookeeper.ClientCnxn:
> > > > > > >> >> >> > >> >>>> Got
> > > > > > >> >> >> > >> >>>> > > > ping
> > > > > > >> >> >> > >> >>>> > > > >>>> response for
> > sessionid:0x24eebf1043003c
> > > > > after
> > > > > > >> 1ms
> > > > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:21,018 DEBUG
> > > > > > >> >> >> > >> org.apache.zookeeper.ClientCnxn:
> > > > > > >> >> >> > >> >>>> Got
> > > > > > >> >> >> > >> >>>> > > > ping
> > > > > > >> >> >> > >> >>>> > > > >>>> response for
> > > sessionid:0x424eebf1c10004c
> > > > > > after
> > > > > > >> 0ms
> > > > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:23,674 DEBUG
> > > > > > >> >> >> > >> org.apache.zookeeper.ClientCnxn:
> > > > > > >> >> >> > >> >>>> Got
> > > > > > >> >> >> > >> >>>> > > > ping
> > > > > > >> >> >> > >> >>>> > > > >>>> response for
> > sessionid:0x24eebf1043003c
> > > > > after
> > > > > > >> 0ms
> > > > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:24,828 INFO
> > > > > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > > > > >> >> >> > >> >>>> > > > Exception
> > > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > > >> >> >> > >> >>>> > > > >>> in
> > > > > > >> >> >> > >> >>>> > > > >>>
> > > > > > >> >> >> > >> >>>> > > > >>>> createBlockOutputStream
> > > > > java.io.IOException:
> > > > > > >> Bad
> > > > > > >> >> >> connect
> > > > > > >> >> >> > >> ack
> > > > > > >> >> >> > >> >>>> with
> > > > > > >> >> >> > >> >>>> > > > >>>> firstBadLink 192.168.100.123:50010
> > > > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:24,828 INFO
> > > > > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > > > > >> >> >> > >> >>>> > > > >>>> Abandoning
> > > > > > >> >> >> > >> >>>> > > > >>>> block
> > blk_-6642544517082142289_5478063
> > > > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:24,828 INFO
> > > > > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > > > > >> >> >> > >> >>>> > > > Exception
> > > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > > >> >> >> > >> >>>> > > > >>> in
> > > > > > >> >> >> > >> >>>> > > > >>>
> > > > > > >> >> >> > >> >>>> > > > >>>> createBlockOutputStream
> > > > > > >> java.net.SocketException:
> > > > > > >> >> Too
> > > > > > >> >> >> > many
> > > > > > >> >> >> > >> >>>> open
> > > > > > >> >> >> > >> >>>> > > files
> > > > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:24,828 INFO
> > > > > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > > > > >> >> >> > >> >>>> > > > >>>> Abandoning
> > > > > > >> >> >> > >> >>>> > > > >>>> block
> blk_2057511041109796090_5478063
> > > > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:24,928 INFO
> > > > > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > > > > >> >> >> > >> >>>> > > > Exception
> > > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > > >> >> >> > >> >>>> > > > >>> in
> > > > > > >> >> >> > >> >>>> > > > >>>
> > > > > > >> >> >> > >> >>>> > > > >>>> createBlockOutputStream
> > > > > > >> java.net.SocketException:
> > > > > > >> >> Too
> > > > > > >> >> >> > many
> > > > > > >> >> >> > >> >>>> open
> > > > > > >> >> >> > >> >>>> > > files
> > > > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:24,928 INFO
> > > > > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > > > > >> >> >> > >> >>>> > > > >>>> Abandoning
> > > > > > >> >> >> > >> >>>> > > > >>>> block
> blk_8219260302213892894_5478064
> > > > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:30,855 INFO
> > > > > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > > > > >> >> >> > >> >>>> > > > Exception
> > > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > > >> >> >> > >> >>>> > > > >>> in
> > > > > > >> >> >> > >> >>>> > > > >>>
> > > > > > >> >> >> > >> >>>> > > > >>>> createBlockOutputStream
> > > > > > >> java.net.SocketException:
> > > > > > >> >> Too
> > > > > > >> >> >> > many
> > > > > > >> >> >> > >> >>>> open
> > > > > > >> >> >> > >> >>>> > > files
> > > > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:30,855 INFO
> > > > > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > > > > >> >> >> > >> >>>> > > > >>>> Abandoning
> > > > > > >> >> >> > >> >>>> > > > >>>> block
> blk_1669205542853067709_5478235
> > > > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:30,905 INFO
> > > > > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > > > > >> >> >> > >> >>>> > > > Exception
> > > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > > >> >> >> > >> >>>> > > > >>> in
> > > > > > >> >> >> > >> >>>> > > > >>>
> > > > > > >> >> >> > >> >>>> > > > >>>> createBlockOutputStream
> > > > > > >> java.net.SocketException:
> > > > > > >> >> Too
> > > > > > >> >> >> > many
> > > > > > >> >> >> > >> >>>> open
> > > > > > >> >> >> > >> >>>> > > files
> > > > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:30,905 INFO
> > > > > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > > > > >> >> >> > >> >>>> > > > >>>> Abandoning
> > > > > > >> >> >> > >> >>>> > > > >>>> block
> blk_9128897691346270351_5478237
> > > > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:30,955 INFO
> > > > > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > > > > >> >> >> > >> >>>> > > > Exception
> > > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > > >> >> >> > >> >>>> > > > >>> in
> > > > > > >> >> >> > >> >>>> > > > >>>
> > > > > > >> >> >> > >> >>>> > > > >>>> createBlockOutputStream
> > > > > > >> java.net.SocketException:
> > > > > > >> >> Too
> > > > > > >> >> >> > many
> > > > > > >> >> >> > >> >>>> open
> > > > > > >> >> >> > >> >>>> > > files
> > > > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:30,955 INFO
> > > > > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > > > > >> >> >> > >> >>>> > > > >>>> Abandoning
> > > > > > >> >> >> > >> >>>> > > > >>>> block
> blk_1116845144864123018_5478240
> > > > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:34,372 DEBUG
> > > > > > >> >> >> > >> org.apache.zookeeper.ClientCnxn:
> > > > > > >> >> >> > >> >>>> Got
> > > > > > >> >> >> > >> >>>> > > > ping
> > > > > > >> >> >> > >> >>>> > > > >>>> response for
> > > sessionid:0x424eebf1c10004c
> > > > > > after
> > > > > > >> 0ms
> > > > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:37,034 DEBUG
> > > > > > >> >> >> > >> org.apache.zookeeper.ClientCnxn:
> > > > > > >> >> >> > >> >>>> Got
> > > > > > >> >> >> > >> >>>> > > > ping
> > > > > > >> >> >> > >> >>>> > > > >>>> response for
> > sessionid:0x24eebf1043003c
> > > > > after
> > > > > > >> 0ms
> > > > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:37,235 WARN
> > > > > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > > >> >> >> > >> >>>> > > > >>> DataStreamer
> > > > > > >> >> >> > >> >>>> > > > >>>
> > > > > > >> >> >> > >> >>>> > > > >>>> Exception: java.io.IOException: Too
> > > many
> > > > > open
> > > > > > >> >> files
> > > > > > >> >> >> > >> >>>> > > > >>>> at
> > > sun.nio.ch.IOUtil.initPipe(Native
> > > > > > >> Method)
> > > > > > >> >> >> > >> >>>> > > > >>>> at
> > > > > > >> >> >> > >> >>>> > >
> > > > > > >> >> >>
> > > sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:49)
> > > > > > >> >> >> > >> >>>> > > > >>>> at
> > > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > > >> >> >> > >> >>>> > > >
> > > > > > >> >> >> > >> >>>> > >
> > > > > > >> >> >> > >> >>>> >
> > > > > > >> >> >> > >> >>>>
> > > > > > >> >> >> > >>
> > > > > > >> >> >> >
> > > > > > >> >> >>
> > > > > > >> >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:18)
> > > > > > >> >> >> > >> >>>> > > > >>>
> > > > > > >> >> >> > >> >>>> > > > >>>> at
> > > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > > >> >> >> > >> >>>> > > >
> > > > > > >> >> >> > >> >>>> > >
> > > > > > >> >> >> > >> >>>> >
> > > > > > >> >> >> > >> >>>>
> > > > > > >> >> >> > >>
> > > > > > >> >> >> >
> > > > > > >> >> >>
> > > > > > >> >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.get(SocketIOWithTimeout.java:407)
> > > > > > >> >> >> > >> >>>> > > > >>>
> > > > > > >> >> >> > >> >>>> > > > >>>> at
> > > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > > >> >> >> > >> >>>> > > >
> > > > > > >> >> >> > >> >>>> > >
> > > > > > >> >> >> > >> >>>> >
> > > > > > >> >> >> > >> >>>>
> > > > > > >> >> >> > >>
> > > > > > >> >> >> >
> > > > > > >> >> >>
> > > > > > >> >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:322)
> > > > > > >> >> >> > >> >>>> > > > >>>
> > > > > > >> >> >> > >> >>>> > > > >>>> at
> > > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > > >> >> >> > >> >>>> > > >
> > > > > > >> >> >> > >> >>>> > >
> > > > > > >> >> >> > >> >>>> >
> > > > > > >> >> >> > >> >>>>
> > > > > > >> >> >> > >>
> > > > > > >> >> >> >
> > > > > > >> >> >>
> > > > > > >> >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
> > > > > > >> >> >> > >> >>>> > > > >>>
> > > > > > >> >> >> > >> >>>> > > > >>>> at
> > > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > > >> >> >> > >> >>>> > > >
> > > > > > >> >> >> > >> >>>> > >
> > > > > > >> >> >> > >> >>>> >
> > > > > > >> >> >> > >> >>>>
> > > > > > >> >> >> > >>
> > > > > > >> >> >> >
> > > > > > >> >> >>
> > > > > > >> >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
> > > > > > >> >> >> > >> >>>> > > > >>>
> > > > > > >> >> >> > >> >>>> > > > >>>> at
> > > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > > >> >> >> > >> >>>> > > >
> > > > > > >> >> >> > >> >>>> > >
> > > > > > >> >> >> > >> >>>> >
> > > > > > >> >> >> > >> >>>>
> > > > > > >> >> >> > >>
> > > > > > >> >> >> >
> > > > > > >> >> >>
> > > > > > >> >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
> > > > > > >> >> >> > >> >>>> > > > >>>
> > > > > > >> >> >> > >> >>>> > > > >>>> at
> > > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > > >> >> >> > >> >>>> > > > >>>
> > > > > > >> >> >> > >> >>>>
> > > > > > >> >> >>
> > > > > java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
> > > > > > >> >> >> > >> >>>> > > > >>>
> > > > > > >> >> >> > >> >>>> > > > >>>> at
> > > > > > >> >> >> > >> >>>>
> > > > > java.io.DataOutputStream.write(DataOutputStream.java:90)
> > > > > > >> >> >> > >> >>>> > > > >>>> at
> > > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > > >> >> >> > >> >>>> > > >
> > > > > > >> >> >> > >> >>>> > >
> > > > > > >> >> >> > >> >>>> >
> > > > > > >> >> >> > >> >>>>
> > > > > > >> >> >> > >>
> > > > > > >> >> >> >
> > > > > > >> >> >>
> > > > > > >> >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2290)
> > > > > > >> >> >> > >> >>>> > > > >>>
> > > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:37,235 WARN
> > > > > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > > > > >> >> >> > >> >>>> > Error
> > > > > > >> >> >> > >> >>>> > > > >>>> Recovery for block
> > > > > > >> blk_8148813491785406356_5478475
> > > > > > >> >> >> bad
> > > > > > >> >> >> > >> >>>> datanode[0]
> > > > > > >> >> >> > >> >>>> > > > >>>> 192.168.100.123:50010
> > > > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:37,235 WARN
> > > > > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > > > > >> >> >> > >> >>>> > Error
> > > > > > >> >> >> > >> >>>> > > > >>>> Recovery for block
> > > > > > >> blk_8148813491785406356_5478475
> > > > > > >> >> in
> > > > > > >> >> >> > >> pipeline
> > > > > > >> >> >> > >> >>>> > > > >>>> 192.168.100.123:50010,
> > > > > 192.168.100.134:50010
> > > > > > ,
> > > > > > >> >> >> > >> >>>> > 192.168.100.122:50010
> > > > > > >> >> >> > >> >>>> > > :
> > > > > > >> >> >> > >> >>>> > > > >>>> bad
> > > > > > >> >> >> > >> >>>> > > > >>>> datanode 192.168.100.123:50010
> > > > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:37,436 INFO
> > > > > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > > > > >> >> >> > >> >>>> > > > Exception
> > > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > > >> >> >> > >> >>>> > > > >>> in
> > > > > > >> >> >> > >> >>>> > > > >>>
> > > > > > >> >> >> > >> >>>> > > > >>>> createBlockOutputStream
> > > > > > >> java.net.SocketException:
> > > > > > >> >> Too
> > > > > > >> >> >> > many
> > > > > > >> >> >> > >> >>>> open
> > > > > > >> >> >> > >> >>>> > > files
> > > > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:37,436 INFO
> > > > > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > > > > >> >> >> > >> >>>> > > > >>>> Abandoning
> > > > > > >> >> >> > >> >>>> > > > >>>> block
> blk_2119727700857186236_5478498
> > > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > > >> >> >> > >> >>>> > > > >>>> On Thu, Nov 12, 2009 at 4:21 PM,
> > Zhenyu
> > > > > Zhong
> > > > > > <
> > > > > > >> >> >> > >> >>>> > > > zhongresearch@gmail.com
> > > > > > >> >> >> > >> >>>> > > > >>>> wrote:
> > > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > > >> >> >> > >> >>>> > > > >>>> Will do.
> > > > > > >> >> >> > >> >>>> > > > >>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>> thanks
> > > > > > >> >> >> > >> >>>> > > > >>>>> zhenyu
> > > > > > >> >> >> > >> >>>> > > > >>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>> On Thu, Nov 12, 2009 at 3:33 PM,
> > stack
> > > <
> > > > > > >> >> >> > stack@duboce.net
> > > > > > >> >> >> > >> >
> > > > > > >> >> >> > >> >>>> wrote:
> > > > > > >> >> >> > >> >>>> > > > >>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>> Enable GC logging too on this
> next
> > > run
> > > > > (see
> > > > > > >> >> >> > >> hbase-env.sh).
> > > > > > >> >> >> > >> >>>> Lets
> > > > > > >> >> >> > >> >>>> > > try
> > > > > > >> >> >> > >> >>>> > > > >>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>> and
> > > > > > >> >> >> > >> >>>> > > > >>>
> > > > > > >> >> >> > >> >>>> > > > >>>> get
> > > > > > >> >> >> > >> >>>> > > > >>>>>> to the bottom of whats going on.
> > > > > > >> >> >> > >> >>>> > > > >>>>>> Thanks,
> > > > > > >> >> >> > >> >>>> > > > >>>>>> St.Ack
> > > > > > >> >> >> > >> >>>> > > > >>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>> On Thu, Nov 12, 2009 at 12:29 PM,
> > > > Zhenyu
> > > > > > >> Zhong <
> > > > > > >> >> >> > >> >>>> > > > >>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>> zhongresearch@gmail.com
> > > > > > >> >> >> > >> >>>> > > > >>>
> > > > > > >> >> >> > >> >>>> > > > >>>> wrote:
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>> I can switch the boxes that run
> > > > > zookeeper
> > > > > > >> with
> > > > > > >> >> the
> > > > > > >> >> >> > ones
> > > > > > >> >> >> > >> >>>> that
> > > > > > >> >> >> > >> >>>> > run
> > > > > > >> >> >> > >> >>>> > > > >>>>>>> regionservers.
> > > > > > >> >> >> > >> >>>> > > > >>>>>>> I will see the results later.
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>> FYI. The node does have the 10
> > > minutes
> > > > > > >> >> >> > >> >>>> > zookeeper.session.timeout
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>> value
> > > > > > >> >> >> > >> >>>> > > > >>>
> > > > > > >> >> >> > >> >>>> > > > >>>> in
> > > > > > >> >> >> > >> >>>> > > > >>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>> place.
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>> thanks
> > > > > > >> >> >> > >> >>>> > > > >>>>>>> zhenyu
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>> On Thu, Nov 12, 2009 at 3:21 PM,
> > > stack
> > > > <
> > > > > > >> >> >> > >> stack@duboce.net>
> > > > > > >> >> >> > >> >>>> > wrote:
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>> On Thu, Nov 12, 2009 at 11:50
> AM,
> > > > > Zhenyu
> > > > > > >> Zhong
> > > > > > >> >> <
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>> zhongresearch@gmail.com
> > > > > > >> >> >> > >> >>>> > > > >>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>> wrote:
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>> In my cluster, half of the
> > cluster
> > > > > have
> > > > > > 2
> > > > > > >> >> disks
> > > > > > >> >> >> > 400GB
> > > > > > >> >> >> > >> >>>> each
> > > > > > >> >> >> > >> >>>> > per
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>> machine,
> > > > > > >> >> >> > >> >>>> > > > >>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>> and
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>> half of the cluster have 6
> disks
> > > per
> > > > > > >> machine.
> > > > > > >> >> >> > Maybe
> > > > > > >> >> >> > >> we
> > > > > > >> >> >> > >> >>>> > should
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>> run
> > > > > > >> >> >> > >> >>>> > > > >>>
> > > > > > >> >> >> > >> >>>> > > > >>>> zookeeper
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>> on the machines with 2 disks
> and
> > > RS
> > > > on
> > > > > > >> >> machines
> > > > > > >> >> >> > with
> > > > > > >> >> >> > >> 6
> > > > > > >> >> >> > >> >>>> disks?
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>> That would make most sense
> only
> > in
> > > > the
> > > > > > >> below,
> > > > > > >> >> it
> > > > > > >> >> >> > >> looks
> > > > > > >> >> >> > >> >>>> like
> > > > > > >> >> >> > >> >>>> > the
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>> RS
> > > > > > >> >> >> > >> >>>> > > > >>>
> > > > > > >> >> >> > >> >>>> > > > >>>> that
> > > > > > >> >> >> > >> >>>> > > > >>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>> had
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>> issue had 4 disks?
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>> BTW, the 10 minutes
> > > > > > >> zookeeper.session.timeout
> > > > > > >> >> >> has
> > > > > > >> >> >> > >> been
> > > > > > >> >> >> > >> >>>> set
> > > > > > >> >> >> > >> >>>> > > during
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>> the
> > > > > > >> >> >> > >> >>>> > > > >>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>> experiment.
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>> And for sure this node had it
> in
> > > > place?
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>> St.Ack
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>> thanks
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>> zhenyu
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>> On Thu, Nov 12, 2009 at 2:08
> PM,
> > > > stack
> > > > > <
> > > > > > >> >> >> > >> stack@duboce.net
> > > > > > >> >> >> > >> >>>> >
> > > > > > >> >> >> > >> >>>> > > wrote:
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>> On Thu, Nov 12, 2009 at 8:40
> > AM,
> > > > > Zhenyu
> > > > > > >> >> Zhong <
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>> zhongresearch@gmail.com
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>> wrote:
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> Though I experienced 2
> > > > regionserver
> > > > > > >> >> >> disconnection
> > > > > > >> >> >> > >> this
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> morning,
> > > > > > >> >> >> > >> >>>> > > > >>>
> > > > > > >> >> >> > >> >>>> > > > >>>> it
> > > > > > >> >> >> > >> >>>> > > > >>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>> looks
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> better from the regionserver
> > log.
> > > > > > (please
> > > > > > >> >> see
> > > > > > >> >> >> the
> > > > > > >> >> >> > >> >>>> following
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> log)
> > > > > > >> >> >> > >> >>>> > > > >>>
> > > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> http://pastebin.com/m496dbfae
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> I checked diskIO for one of
> > the
> > > > > > >> >> >> > >> >>>> > regionserver(192.168.100.116)
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> during
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>> the
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> time it disconnected.
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> Time: 03:04:58 AM
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> Device: tps
> > > > Blk_read/s
> > > > > > >> >> >> Blk_wrtn/s
> > > > > > >> >> >> > >> >>>> Blk_read
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> Blk_wrtn
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>> sda 105.31
> > > 5458.83
> > > > > > >> >> 19503.64
> > > > > > >> >> >> > >> >>>> 9043873239
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> 32312473676
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>> sda1 2.90
> > > 6.64
> > > > > > >> >> 99.25
> > > > > > >> >> >> > >> >>>> 10993934
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> 164433464
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>> sda2 1.72
> > > 23.76
> > > > > > >> >> 30.25
> > > > > > >> >> >> > >> >>>> 39365817
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> 50124008
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>> sda3 0.30
> > > 0.38
> > > > > > >> >> 3.58
> > > > > > >> >> >> > >> >>>> 624930
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> 5923000
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>> sda4 100.39
> > > 5428.06
> > > > > > >> >> 19370.56
> > > > > > >> >> >> > >> >>>> 8992888270
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> 32091993204
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> Is this high for you? 20k
> > > > > > >> blocks/second
> > > > > > >> >> >> would
> > > > > > >> >> >> > >> seem to
> > > > > > >> >> >> > >> >>>> be
> > > > > > >> >> >> > >> >>>> > > high
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>> but
> > > > > > >> >> >> > >> >>>> > > > >>>
> > > > > > >> >> >> > >> >>>> > > > >>>> its
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>> one
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> disk only and its not being
> > > shared
> > > > by
> > > > > > zk
> > > > > > >> >> >> anymore
> > > > > > >> >> >> > so
> > > > > > >> >> >> > >> >>>> > shouldn't
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>> matter?
> > > > > > >> >> >> > >> >>>> > > > >>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> I also checked the zookeeper
> > > > quorum
> > > > > > >> server
> > > > > > >> >> >> that
> > > > > > >> >> >> > the
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> regionserver
> > > > > > >> >> >> > >> >>>> > > > >>>
> > > > > > >> >> >> > >> >>>> > > > >>>> tried
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>> to
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> connect according to the log.
> > > > > However,
> > > > > > I
> > > > > > >> >> don't
> > > > > > >> >> >> see
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> 192.168.100.116
> > > > > > >> >> >> > >> >>>> > > > >>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>> in
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>> the
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> client list of the zookeeper
> > > quorum
> > > > > > >> server
> > > > > > >> >> that
> > > > > > >> >> >> > the
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> regionserver
> > > > > > >> >> >> > >> >>>> > > > >>>
> > > > > > >> >> >> > >> >>>> > > > >>>> tried
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>> to
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> connect.
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> Would that be a problem?
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> Is that because the
> ephemeral
> > > > node
> > > > > > for
> > > > > > >> the
> > > > > > >> >> >> > >> >>>> regionserver
> > > > > > >> >> >> > >> >>>> > had
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>> evaporated?
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>> Lost
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> its lease w/ zk by the time
> you
> > > > went
> > > > > to
> > > > > > >> >> look?
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> Thu Nov 12 15:04:52 UTC 2009
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> Zookeeper version:
> > 3.2.1-808558,
> > > > > built
> > > > > > >> on
> > > > > > >> >> >> > >> 08/27/2009
> > > > > > >> >> >> > >> >>>> 18:48
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> GMT
> > > > > > >> >> >> > >> >>>> > > > >>>
> > > > > > >> >> >> > >> >>>> > > > >>>> Clients:
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.127:43045
> > > > > > >> >> >> > >> [1](queued=0,recved=26,sent=0)
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.131:39091
> > > > > > >> >> >> > >> [1](queued=0,recved=964,sent=0)
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.124:35961
> > > > > > >> >> >> > >> [1](queued=0,recved=3266,sent=0)
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.123:47935
> > > > > > >> >> >> > >> [1](queued=0,recved=233,sent=0)
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.125:46931
> > > > > > >> >> >> > >> [1](queued=0,recved=2,sent=0)
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.118:54924
> > > > > > >> >> >> > >> [1](queued=0,recved=295,sent=0)
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.118:41390
> > > > > > >> >> >> > >> [1](queued=0,recved=2290,sent=0)
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.136:42243
> > > > > > >> >> >> > >> [1](queued=0,recved=0,sent=0)
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> Latency min/avg/max:
> 0/17/6333
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> Received: 47111
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> Sent: 0
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> Outstanding: 0
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> Zxid: 0x77000083f4
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> Mode: leader
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> Node count: 23
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> That 6 second maximum latency
> > is
> > > > > pretty
> > > > > > >> bad
> > > > > > >> >> but
> > > > > > >> >> >> > >> should
> > > > > > >> >> >> > >> >>>> be
> > > > > > >> >> >> > >> >>>> > well
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>> within
> > > > > > >> >> >> > >> >>>> > > > >>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>> the
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>> zk
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> session timeout.
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> So, problem is likely on the
> zk
> > > > > client
> > > > > > >> side
> > > > > > >> >> of
> > > > > > >> >> >> the
> > > > > > >> >> >> > >> >>>> session;
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>> i.e.
> > > > > > >> >> >> > >> >>>> > > > >>>
> > > > > > >> >> >> > >> >>>> > > > >>>> in
> > > > > > >> >> >> > >> >>>> > > > >>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>> the
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>> RS.
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> You could enable GC logging
> as
> > > J-D
> > > > > > >> suggested
> > > > > > >> >> to
> > > > > > >> >> >> > see
> > > > > > >> >> >> > >> if
> > > > > > >> >> >> > >> >>>> you
> > > > > > >> >> >> > >> >>>> > > have
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>> any
> > > > > > >> >> >> > >> >>>> > > > >>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>> big
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>> pauses, pauses > zk session
> > > timeout.
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> When the RS went down, it
> > didn't
> > > > look
> > > > > > too
> > > > > > >> >> >> heavily
> > > > > > >> >> >> > >> >>>> loaded:
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> 1. 2009-11-12 15:04:52,830
> > INFO
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > > >> >> >> > org.apache.hadoop.hbase.regionserver.HRegionServer:
> > > > > > >> >> >> > >> >>>> Dump of
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>> metrics:
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>> request=1.5166667,
> regions=322,
> > > > > > >> stores=657,
> > > > > > >> >> >> > >> >>>> storefiles=631,
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> storefileIndexSize=61,
> > > > > > >> memstoreSize=1472,
> > > > > > >> >> >> > >> >>>> usedHeap=2819,
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>> maxHeap=4079,
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>> blockCacheSize=658110960,
> > > > > > >> >> >> > blockCacheFree=197395984,
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>> blockCacheCount=9903,
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> blockCacheHitRatio=99
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> Its serving a few reads? The
> > > > number
> > > > > of
> > > > > > >> >> store
> > > > > > >> >> >> > files
> > > > > > >> >> >> > >> >>>> seems
> > > > > > >> >> >> > >> >>>> > > fine.
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>> Not
> > > > > > >> >> >> > >> >>>> > > > >>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>> too
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>> much memory used.
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> Looking at the logs, I see
> the
> > > > Lease
> > > > > > >> Still
> > > > > > >> >> Held
> > > > > > >> >> >> > >> >>>> exception.
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>> This
> > > > > > >> >> >> > >> >>>> > > > >>>
> > > > > > >> >> >> > >> >>>> > > > >>>> happens
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>> when the RS does its regular
> > > report
> > > > to
> > > > > > the
> > > > > > >> >> >> master
> > > > > > >> >> >> > but
> > > > > > >> >> >> > >> the
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>> master
> > > > > > >> >> >> > >> >>>> > > > >>>
> > > > > > >> >> >> > >> >>>> > > > >>>> thinks
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>> the
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> RS has since restarted.
> It'll
> > > > think
> > > > > > this
> > > > > > >> >> >> probably
> > > > > > >> >> >> > >> >>>> because
> > > > > > >> >> >> > >> >>>> > it
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>> noticed
> > > > > > >> >> >> > >> >>>> > > > >>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>> that
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> the RS's znode in zk had gone
> > > away
> > > > > and
> > > > > > it
> > > > > > >> >> >> > considered
> > > > > > >> >> >> > >> the
> > > > > > >> >> >> > >> >>>> RS
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>> dead.
> > > > > > >> >> >> > >> >>>> > > > >>>
> > > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> Looking too at your logs I
> see
> > > > this
> > > > > > gap
> > > > > > >> in
> > > > > > >> >> the
> > > > > > >> >> >> zk
> > > > > > >> >> >> > >> >>>> pinging:
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> 1. 2009-11-12 15:03:39,325
> > DEBUG
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> org.apache.zookeeper.ClientCnxn:
> > > > > > >> >> >> > >> >>>> > > > >>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>> Got
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>> ping response for
> > > > > > >> sessionid:0x224e55436ad0004
> > > > > > >> >> >> after
> > > > > > >> >> >> > >> 0ms
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> 2. 2009-11-12 15:03:43,113
> > DEBUG
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> org.apache.zookeeper.ClientCnxn:
> > > > > > >> >> >> > >> >>>> > > > >>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>> Got
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>> ping response for
> > > > > > >> sessionid:0x24e55436a0007d
> > > > > > >> >> >> after
> > > > > > >> >> >> > >> 0ms
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> Where in the lines above it,
> > its
> > > > > > >> reporting
> > > > > > >> >> >> about
> > > > > > >> >> >> > >> every
> > > > > > >> >> >> > >> >>>> ten
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>> seconds,
> > > > > > >> >> >> > >> >>>> > > > >>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>> here
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>> there is a big gap.
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> Do you have ganglia or
> > something
> > > > that
> > > > > > >> will
> > > > > > >> >> let
> > > > > > >> >> >> you
> > > > > > >> >> >> > >> look
> > > > > > >> >> >> > >> >>>> more
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>> into
> > > > > > >> >> >> > >> >>>> > > > >>>
> > > > > > >> >> >> > >> >>>> > > > >>>> what
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>> was
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> happening on this machine
> > around
> > > > this
> > > > > > >> time?
> > > > > > >> >> Is
> > > > > > >> >> >> > the
> > > > > > >> >> >> > >> >>>> machine
> > > > > > >> >> >> > >> >>>> > > OK?
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>> It
> > > > > > >> >> >> > >> >>>> > > > >>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>> looks
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>> lightly loaded and you have
> your
> > > > > cluster
> > > > > > >> >> nicely
> > > > > > >> >> >> > laid
> > > > > > >> >> >> > >> out.
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>> Something
> > > > > > >> >> >> > >> >>>> > > > >>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>> odd
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>> is
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> going on. What about things
> > like
> > > > the
> > > > > > >> write
> > > > > > >> >> >> speed
> > > > > > >> >> >> > to
> > > > > > >> >> >> > >> >>>> disk?
> > > > > > >> >> >> > >> >>>> > In
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>> the
> > > > > > >> >> >> > >> >>>> > > > >>>
> > > > > > >> >> >> > >> >>>> > > > >>>> past
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>> strange issues have been
> > explained
> > > by
> > > > > > >> >> incorrectly
> > > > > > >> >> >> > set
> > > > > > >> >> >> > >> BIOS
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>> which
> > > > > > >> >> >> > >> >>>> > > > >>>
> > > > > > >> >> >> > >> >>>> > > > >>>> made
> > > > > > >> >> >> > >> >>>> > > > >>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>> disks
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> run at 1/100th of their
> proper
> > > > speed.
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> St.Ack
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> Best,
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> zhenyu
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> On Wed, Nov 11, 2009 at 3:58
> > PM,
> > > > > > Zhenyu
> > > > > > >> >> Zhong
> > > > > > >> >> >> <
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> zhongresearch@gmail.com
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>> wrote:
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> Stack
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>> I am very appreciated for
> > your
> > > > > > >> comments.
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>> I will use the zookeeper
> > > > monitoring
> > > > > > >> script
> > > > > > >> >> on
> > > > > > >> >> >> my
> > > > > > >> >> >> > >> >>>> cluster
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> and
> > > > > > >> >> >> > >> >>>> > > > >>>
> > > > > > >> >> >> > >> >>>> > > > >>>> let
> > > > > > >> >> >> > >> >>>> > > > >>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>> it
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>> run
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> overnight to see the result.
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>> I will follow up the thread
> > > when
> > > > I
> > > > > > get
> > > > > > >> >> >> anything.
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>> thanks
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>> zhenyu
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>> On Wed, Nov 11, 2009 at
> 3:52
> > > PM,
> > > > > > stack
> > > > > > >> <
> > > > > > >> >> >> > >> >>>> stack@duboce.net>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> wrote:
> > > > > > >> >> >> > >> >>>> > > > >>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>> I see these in your log
> too:
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>> 1. 2009-11-11
> 04:27:19,018
> > > > DEBUG
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>
> > > org.apache.zookeeper.ClientCnxn:
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>> Got
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> ping response for
> > > > > > >> >> sessionid:0x424dfd908c50009
> > > > > > >> >> >> > >> after
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>> 4544ms
> > > > > > >> >> >> > >> >>>> > > > >>>
> > > > > > >> >> >> > >> >>>> > > > >>>> 2. 2009-11-11 04:27:19,018 DEBUG
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>
> > > org.apache.zookeeper.ClientCnxn:
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>> Got
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> ping response for
> > > > > > >> >> sessionid:0x24dfd90c810002
> > > > > > >> >> >> > after
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>> 4548ms
> > > > > > >> >> >> > >> >>>> > > > >>>
> > > > > > >> >> >> > >> >>>> > > > >>>> 3. 2009-11-11 04:27:43,960 DEBUG
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>
> > > org.apache.zookeeper.ClientCnxn:
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>> Got
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> ping response for
> > > > > > >> >> sessionid:0x424dfd908c50009
> > > > > > >> >> >> > >> after
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>> 9030ms
> > > > > > >> >> >> > >> >>>> > > > >>>
> > > > > > >> >> >> > >> >>>> > > > >>>> 4. 2009-11-11 04:27:43,960 DEBUG
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>> org.apache.zookeeper.
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > > > >> >> >> > >> >>>> > > > >>>>>>>
> > > > > > >> >> >> > >> >>>> > > >
> > > > > > >> >> >> > >> >>>> > >
> > > > > > >> >> >> > >> >>>> >
> > > > > > >> >> >> > >> >>>>
> > > > > > >> >> >> > >> >>>
> > > > > > >> >> >> > >> >>>
> > > > > > >> >> >> > >> >>
> > > > > > >> >> >> > >> >
> > > > > > >> >> >> > >>
> > > > > > >> >> >> > >
> > > > > > >> >> >> > >
> > > > > > >> >> >> >
> > > > > > >> >> >>
> > > > > > >> >> >
> > > > > > >> >> >
> > > > > > >> >>
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: regionserver disconnection
Posted by stack <st...@duboce.net>.
>From the zk fellas, try running your zk cluster at INFO level rather than at
DEBUG. One supposition is that the shear amount of logging is causing
distress. They are still looking into your logs...
Thanks for your patience Zhenyu.
St.Ack
On Thu, Nov 19, 2009 at 8:57 AM, Zhenyu Zhong <zh...@gmail.com>wrote:
> Yes, I believe so. And I don't know why the leader gave up.
> For this case, the regionserver got TIMEOUT warning message at 2009-11-18
> 12:42:01,482
>
>
> Best,
> zhenyu
>
>
>
>
> On Thu, Nov 19, 2009 at 11:43 AM, stack <st...@duboce.net> wrote:
>
> > So, the time at which we see
> >
> > 2009-11-18 12:41:56,782 ERROR
> > org.apache.zookeeper.server.quorum.FollowerHandler: Unexpected exception
> > causing shutdown while sock still open
> >
> > ... corresponds to the time at which the regionserver loses is session?
> >
> > St.Ack
> >
> > On Thu, Nov 19, 2009 at 8:38 AM, Zhenyu Zhong <zhongresearch@gmail.com
> > >wrote:
> >
> > > Here is the zookeeper leader log. It had an unexpected exception. I
> > don't
> > > know why.
> > >
> > > http://pastebin.com/m6594ccc
> > >
> > > During that time, I don't think the system load is high. Memory usage
> > > should
> > > be normal.
> > >
> > > Best,
> > > zhenyu
> > >
> > >
> > >
> > > On Thu, Nov 19, 2009 at 11:09 AM, stack <st...@duboce.net> wrote:
> > >
> > > > What is in the other zk logs? Why did the leader go away? Memory
> > issue
> > > or
> > > > something? Why'd it shutdown? Thanks for pastebin'ing this stuff.
> > > > St.Ack
> > > >
> > > > On Thu, Nov 19, 2009 at 7:53 AM, Zhenyu Zhong <
> zhongresearch@gmail.com
> > > > >wrote:
> > > >
> > > > > After digging more on the zookeeper log, I found that there seemed
> to
> > > be
> > > > a
> > > > > socketconnection timeout. Also some warnings indicate that
> zookeeper
> > > > server
> > > > > is not running. Probably I need to understand the mechanism of
> > > zookeeper
> > > > > first.
> > > > >
> > > > > Please find the zookeeper log around that time.
> > > > >
> > > > > http://pastebin.com/m37bb4ad1
> > > > >
> > > > > Many thanks
> > > > > zhenyu
> > > > >
> > > > >
> > > > > On Thu, Nov 19, 2009 at 10:31 AM, Zhenyu Zhong <
> > > zhongresearch@gmail.com
> > > > > >wrote:
> > > > >
> > > > > > Stack,
> > > > > >
> > > > > > I am very appreciated. Let me dig into the zookeeper log more.
> > > > > >
> > > > > > FYI, I saw the sleeper complained about the delay for the first
> > time.
> > > > It
> > > > > > looks like there are different reasons for the RS disconnections.
> > > > > >
> > > > > > 2009-11-19 08:08:32,708 DEBUG org.apache.zookeeper.ClientCnxn:
> Got
> > > ping
> > > > > > response for sessionid:0x42508716bfc0001 after 0ms
> > > > > > 2009-11-19 08:08:37,595 DEBUG org.apache.zookeeper.ClientCnxn:
> Got
> > > ping
> > > > > > response for sessionid:0x42508716bfc0082 after 0ms
> > > > > > 2009-11-19 08:08:46,073 DEBUG org.apache.zookeeper.ClientCnxn:
> Got
> > > ping
> > > > > > response for sessionid:0x42508716bfc0001 after 0ms
> > > > > > 2009-11-19 08:09:39,880 WARN org.apache.zookeeper.ClientCnxn:
> > > Exception
> > > > > > closing session 0x42508716bfc0001 to
> > > > sun.nio.ch.SelectionKeyImpl@4f3a608f
> > > > > > java.io.IOException: TIMED OUT
> > > > > > at
> > > > > >
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:906)
> > > > > > 2009-11-19 08:09:39,880 WARN org.apache.zookeeper.ClientCnxn:
> > > Exception
> > > > > > closing session 0x42508716bfc0082 to
> > > > sun.nio.ch.SelectionKeyImpl@14c8875
> > > > > > java.io.IOException: TIMED OUT
> > > > > > at
> > > > > >
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:906)
> > > > > > 2009-11-19 08:09:39,880 WARN
> org.apache.hadoop.hbase.util.Sleeper:
> > We
> > > > > slept
> > > > > > 53556ms, ten times longer than scheduled: 3000
> > > > > > 2009-11-19 08:09:39,980 WARN
> > > > > > org.apache.hadoop.hbase.regionserver.HRegionServer: Attempt=1
> > > > > > org.apache.hadoop.hbase.Leases$LeaseStillHeldException
> > > > > > at
> > > > sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> > > > > > Method)
> > > > > > at
> > > > > >
> > > > >
> > > >
> > >
> >
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> > > > > > at
> > > > > >
> > > > >
> > > >
> > >
> >
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> > > > > > at
> > > > > java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> > > > > > at
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
> > > > > > at
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
> > > > > > at
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
> > > > > > at
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:571)
> > > > > > at java.lang.Thread.run(Thread.java:619)
> > > > > > 2009-11-19 08:09:39,980 INFO
> > > > > > org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper
> > > > event,
> > > > > > state: Disconnected, type: None, path: null
> > > > > > 2009-11-19 08:09:40,530 INFO org.apache.zookeeper.ClientCnxn:
> > > > Attempting
> > > > > > connection to server superpyxis0005/192.168.100.119:2181
> > > > > > 2009-11-19 08:09:40,530 INFO org.apache.zookeeper.ClientCnxn:
> > Priming
> > > > > > connection to java.nio.channels.SocketChannel[connected local=/
> > > > > > 192.168.100.122:40659 remote=superpyxis0005/192.168.100.119:2181
> ]
> > > > > >
> > > > > >
> > > > > >
> > > > > > Best,
> > > > > > zhenyu
> > > > > >
> > > > > > On Wed, Nov 18, 2009 at 6:52 PM, stack <st...@duboce.net> wrote:
> > > > > >
> > > > > >> Patrick Hunt, one of the zk lads, answered with the following:
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> According to the following snippet (I cut to just session
> > > > > >> 0x425035c48210002)
> > > > > >> the client is getting pings for this session id regularly every
> 6
> > > > > seconds
> > > > > >> or
> > > > > >> so, then after 41,016 there's a 20 second gap (corresponds to a
> 30
> > > > > second
> > > > > >> timeout being set during session creation) at which point we
> > > > disconnect
> > > > > >> from
> > > > > >> the server.
> > > > > >>
> > > > > >>
> > > > > >> > 2009-11-18 12:41:34,788 DEBUG org.apache.zookeeper.ClientCnxn:
> > Got
> > > > > ping
> > > > > >> > response for sessionid:0x425035c48210002 after 0ms
> > > > > >> > 2009-11-18 12:42:01,482 WARN org.apache.zookeeper.ClientCnxn:
> > > > > Exception
> > > > > >> > closing session 0x425035c48210002 to
> > > > > >> sun.nio.ch.SelectionKeyImpl@421690ab
> > > > > >> > java.io.IOException: TIMED OUT
> > > > > >>
> > > > > >> the client notifies hbase of the disco, and attempts to
> reconnect
> > to
> > > a
> > > > > >> server
> > > > > >>
> > > > > >>
> > > > > >> > org.apache.hadoop.hbase.regionserver.HRegionServer: Got
> > ZooKeeper
> > > > > event,
> > > > > >> > state: Disconnected, type: None, path: null
> > > > > >> > 2009-11-18 12:42:01,782 INFO org.apache.zookeeper.ClientCnxn:
> > > > > Attempting
> > > > > >> > connection to server superpyxis0003/192.168.100.117:2181
> > > > > >>
> > > > > >> but the server won't allow the session be be established for
> some
> > > > reason
> > > > > >> (is
> > > > > >> that the right server:port that's it's trying to connect to?)
> > > > > >>
> > > > > >>
> > > > > >> > 2009-11-18 12:42:02,182 INFO org.apache.zookeeper.ClientCnxn:
> > > Server
> > > > > >> > connection successful
> > > > > >> > 2009-11-18 12:42:02,182 WARN org.apache.zookeeper.ClientCnxn:
> > > > > Exception
> > > > > >> > closing session 0x425035c48210002 to
> > > > > >> sun.nio.ch.SelectionKeyImpl@5c07076b
> > > > > >> > java.io.IOException: Read error rc = -1
> > > > > java.nio.DirectByteBuffer[pos=0
> > > > > >> > lim=4 cap=4]
> > > > > >>
> > > > > >> The "read error rc = -1" means that the client read from the
> > socket
> > > > and
> > > > > >> got
> > > > > >> EOS. But it's not a session expiration afaict (and should not be
> > at
> > > > this
> > > > > >> point in time).
> > > > > >>
> > > > > >>
> > > > > >> You need to look at the ZK server logs for this period in time
> > (for
> > > > this
> > > > > >> server superpyxis0003:2181, and superpyxis0001:2181, the client
> > > tries
> > > > to
> > > > > >> connect to both of these). There should be some indication there
> > why
> > > > the
> > > > > >> server is closing down the connection and not allowing the
> session
> > > to
> > > > be
> > > > > >> re-established.
> > > > > >>
> > > > > >> One thought - by default we limit clients to a max of 10
> > connections
> > > > > from
> > > > > >> a
> > > > > >> particular host to the server. If you exceed this limit the zk
> > > server
> > > > > >> (it's
> > > > > >> per server though, not per cluster, so perhaps this is not it)
> > will
> > > > not
> > > > > >> allow any additional connections from the client host (you
> can't
> > > > exceed
> > > > > >> 10
> > > > > >> on a server). Could this be it? i.e. the user is running a
> number
> > of
> > > > ZK
> > > > > >> clients (say hbase processes) from the same host? Regardless,
> get
> > > the
> > > > > >> server
> > > > > >> log for this time period and it should be more clear.
> > > > > >>
> > > > > >> Patrick
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> On Wed, Nov 18, 2009 at 11:18 AM, stack <st...@duboce.net>
> wrote:
> > > > > >>
> > > > > >> > And running stat on each of your zookeeper nodes tells you
> what
> > > > about
> > > > > >> > averages for connect-times?
> > > > > >> >
> > > > > >> > Its odd that there is a ping just 20 seconds beforehand and
> you
> > > have
> > > > > set
> > > > > >> > the zk session timeout at 60 seconds (or ten minutes?). Let
> me
> > > ask
> > > > > the
> > > > > >> zk
> > > > > >> > lads about it.
> > > > > >> >
> > > > > >> > If you look in the GC log for around this time, do you see
> > > anything?
> > > > > >> >
> > > > > >> > St.Ack
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> > On Wed, Nov 18, 2009 at 7:32 AM, Zhenyu Zhong <
> > > > > zhongresearch@gmail.com
> > > > > >> >wrote:
> > > > > >> >
> > > > > >> >> After running more experiment, it seems that the
> disconnection
> > is
> > > > not
> > > > > >> >> related to the heavy load job, because the load average is
> low,
> > > the
> > > > > >> diskio
> > > > > >> >> is normal, memory Heap was not reached, also the virtual
> memory
> > > > stats
> > > > > >> >> shows
> > > > > >> >> that no swappings. However, the disconnection still happens.
> > > > > >> >>
> > > > > >> >> It looks like this time it pauses for 20 seconds. No idea why
> > > > > >> regionserver
> > > > > >> >> disconnected.
> > > > > >> >>
> > > > > >> >> Any other suggestions please?
> > > > > >> >>
> > > > > >> >>
> > > > > >> >> 2009-11-18 12:41:00,956 DEBUG
> org.apache.zookeeper.ClientCnxn:
> > > Got
> > > > > ping
> > > > > >> >> response for sessionid:0x125035c495d02bb after 0ms
> > > > > >> >> 2009-11-18 12:41:08,074 DEBUG
> org.apache.zookeeper.ClientCnxn:
> > > Got
> > > > > ping
> > > > > >> >> response for sessionid:0x425035c48210002 after 0ms
> > > > > >> >> 2009-11-18 12:41:14,302 DEBUG
> org.apache.zookeeper.ClientCnxn:
> > > Got
> > > > > ping
> > > > > >> >> response for sessionid:0x125035c495d02bb after 0ms
> > > > > >> >> 2009-11-18 12:41:21,433 DEBUG
> org.apache.zookeeper.ClientCnxn:
> > > Got
> > > > > ping
> > > > > >> >> response for sessionid:0x425035c48210002 after 0ms
> > > > > >> >> 2009-11-18 12:41:27,656 DEBUG
> org.apache.zookeeper.ClientCnxn:
> > > Got
> > > > > ping
> > > > > >> >> response for sessionid:0x125035c495d02bb after 0ms
> > > > > >> >> 2009-11-18 12:41:34,788 DEBUG
> org.apache.zookeeper.ClientCnxn:
> > > Got
> > > > > ping
> > > > > >> >> response for sessionid:0x425035c48210002 after 0ms
> > > > > >> >> 2009-11-18 12:41:41,016 DEBUG
> org.apache.zookeeper.ClientCnxn:
> > > Got
> > > > > ping
> > > > > >> >> response for sessionid:0x125035c495d02bb after 1ms
> > > > > >> >> 2009-11-18 12:42:01,482 WARN org.apache.zookeeper.ClientCnxn:
> > > > > Exception
> > > > > >> >> closing session 0x425035c48210002 to
> > > > > >> sun.nio.ch.SelectionKeyImpl@421690ab
> > > > > >> >> java.io.IOException: TIMED OUT
> > > > > >> >> at
> > > > > >> >>
> > > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:906)
> > > > > >> >> 2009-11-18 12:42:01,582 INFO
> > > > > >> >> org.apache.hadoop.hbase.regionserver.HRegionServer: Got
> > ZooKeeper
> > > > > >> event,
> > > > > >> >> state: Disconnected, type: None, path: null
> > > > > >> >> 2009-11-18 12:42:01,782 INFO org.apache.zookeeper.ClientCnxn:
> > > > > >> Attempting
> > > > > >> >> connection to server superpyxis0003/192.168.100.117:2181
> > > > > >> >> 2009-11-18 12:42:01,782 INFO org.apache.zookeeper.ClientCnxn:
> > > > Priming
> > > > > >> >> connection to java.nio.channels.SocketChannel[connected
> local=/
> > > > > >> >> 192.168.100.132:46610 remote=superpyxis0003/
> > 192.168.100.117:2181
> > > ]
> > > > > >> >> 2009-11-18 12:42:01,782 INFO org.apache.zookeeper.ClientCnxn:
> > > > Server
> > > > > >> >> connection successful
> > > > > >> >> 2009-11-18 12:42:01,782 WARN org.apache.zookeeper.ClientCnxn:
> > > > > Exception
> > > > > >> >> closing session 0x425035c48210002 to
> > > > > >> sun.nio.ch.SelectionKeyImpl@56dc6fac
> > > > > >> >> java.io.IOException: Read error rc = -1
> > > > > java.nio.DirectByteBuffer[pos=0
> > > > > >> >> lim=4 cap=4]
> > > > > >> >> at
> > > > > >> >>
> > > > org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
> > > > > >> >> at
> > > > > >> >>
> > > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
> > > > > >> >> 2009-11-18 12:42:01,782 WARN org.apache.zookeeper.ClientCnxn:
> > > > > Ignoring
> > > > > >> >> exception during shutdown input
> > > > > >> >> java.net.SocketException: Transport endpoint is not connected
> > > > > >> >> at sun.nio.ch.SocketChannelImpl.shutdown(Native
> Method)
> > > > > >> >> at
> > > > > >> >>
> > > > >
> > sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
> > > > > >> >> at
> > > > > >> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> > > > > >> >> at
> > > > > >> >>
> > > > >
> > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> > > > > >> >> at
> > > > > >> >>
> > > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> > > > > >> >> 2009-11-18 12:42:01,782 WARN org.apache.zookeeper.ClientCnxn:
> > > > > Ignoring
> > > > > >> >> exception during shutdown output
> > > > > >> >> java.net.SocketException: Transport endpoint is not connected
> > > > > >> >> at sun.nio.ch.SocketChannelImpl.shutdown(Native
> Method)
> > > > > >> >> at
> > > > > >> >>
> > > > >
> > sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
> > > > > >> >> at
> > > > > >> sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> > > > > >> >> at
> > > > > >> >>
> > > > > >>
> > > >
> > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> > > > > >> >> at
> > > > > >> >>
> > > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> > > > > >> >> 2009-11-18 12:42:02,182 INFO org.apache.zookeeper.ClientCnxn:
> > > > > >> Attempting
> > > > > >> >> connection to server superpyxis0001/192.168.100.115:2181
> > > > > >> >> 2009-11-18 12:42:02,182 INFO org.apache.zookeeper.ClientCnxn:
> > > > Priming
> > > > > >> >> connection to java.nio.channels.SocketChannel[connected
> local=/
> > > > > >> >> 192.168.100.132:36197 remote=superpyxis0001/
> > 192.168.100.115:2181
> > > ]
> > > > > >> >> 2009-11-18 12:42:02,182 INFO org.apache.zookeeper.ClientCnxn:
> > > > Server
> > > > > >> >> connection successful
> > > > > >> >> 2009-11-18 12:42:02,182 WARN org.apache.zookeeper.ClientCnxn:
> > > > > Exception
> > > > > >> >> closing session 0x425035c48210002 to
> > > > > >> sun.nio.ch.SelectionKeyImpl@5c07076b
> > > > > >> >> java.io.IOException: Read error rc = -1
> > > > > java.nio.DirectByteBuffer[pos=0
> > > > > >> >> lim=4 cap=4]
> > > > > >> >> at
> > > > > >> >>
> > > > org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
> > > > > >> >> at
> > > > > >> >>
> > > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
> > > > > >> >> 2009-11-18 12:42:02,182 WARN org.apache.zookeeper.ClientCnxn:
> > > > > Ignoring
> > > > > >> >> exception during shutdown input
> > > > > >> >> java.net.SocketException: Transport endpoint is not connected
> > > > > >> >> at sun.nio.ch.SocketChannelImpl.shutdown(Native
> Method)
> > > > > >> >> at
> > > > > >> >>
> > > > >
> > sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
> > > > > >> >> at
> > > > > >> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> > > > > >> >> at
> > > > > >> >>
> > > > >
> > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> > > > > >> >> at
> > > > > >> >>
> > > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> > > > > >> >> 2009-11-18 12:42:02,182 WARN org.apache.zookeeper.ClientCnxn:
> > > > > Ignoring
> > > > > >> >> exception during shutdown output
> > > > > >> >> java.net.SocketException: Transport endpoint is not connected
> > > > > >> >>
> > > > > >> >>
> > > > > >> >> On Mon, Nov 16, 2009 at 4:16 PM, Zhenyu Zhong <
> > > > > zhongresearch@gmail.com
> > > > > >> >> >wrote:
> > > > > >> >>
> > > > > >> >> > Here is the diskIO and CPU around the time we had RS
> > > > disconnection
> > > > > on
> > > > > >> >> one
> > > > > >> >> > machine that runs RegionServer. It doesn't seem to be high.
> > > > Similar
> > > > > >> disk
> > > > > >> >> and
> > > > > >> >> > cpu usage have been seen before and the HBase was running
> > fine.
> > > > > >> >> >
> > > > > >> >> >
> > > > > >> >> > So far I haven't found why my 10 minutes session timeout
> > > doesn't
> > > > > >> apply.
> > > > > >> >> > Still digging.
> > > > > >> >> >
> > > > > >> >> > Many thanks!
> > > > > >> >> >
> > > > > >> >> > Time: 03:01:08 PM
> > > > > >> >> > avg-cpu: %user %nice %system %iowait %steal %idle
> > > > > >> >> > 3.82 0.00 0.89 3.97 0.00 91.32
> > > > > >> >> >
> > > > > >> >> >
> > > > > >> >> > Device: tps Blk_read/s Blk_wrtn/s Blk_read
> > > > > >> Blk_wrtn
> > > > > >> >> > sda 24.30 1605.39 6679.35 6238623388
> > > > > >> 25956251720
> > > > > >> >> > sda1 0.51 1.94 15.71 7547265
> > > > > >> 61067904
> > > > > >> >> > sda2 0.13 2.11 5.20 8202079
> > > > > >> 20189232
> > > > > >> >> > sda3 0.13 0.25 1.63 987846
> > > > > >> 6323336
> > > > > >> >> > sda4 0.00 0.00 0.00 4
> > > > > >> 0
> > > > > >> >> > sda5 1.78 0.06 24.14 220010
> > > > > >> 93817208
> > > > > >> >> > sda6 0.38 0.42 8.41 1630584
> > > > > >> 32688152
> > > > > >> >> > sda7 21.38 1600.61 6624.26 6220035272
> > > > > >> 25742165888
> > > > > >> >> > sdb 4.52 767.33 565.91 2981868690
> > > > > >> 2199132380
> > > > > >> >> > sdb1 4.52 767.33 565.91 2981866755
> > > > > >> 2199132372
> > > > > >> >> > sdc 4.42 742.95 563.00 2887151482
> > > > > >> 2187823092
> > > > > >> >> > sdc1 4.42 742.95 563.00 2887149547
> > > > > >> 2187823084
> > > > > >> >> > sdd 4.49 750.78 557.25 2917554074
> > > > > >> 2165513500
> > > > > >> >> > sdd1 4.49 750.78 557.25 2917552139
> > > > > >> 2165513492
> > > > > >> >> > sde 4.52 758.51 569.46 2947593394
> > > > > >> 2212964236
> > > > > >> >> > sde1 4.52 758.51 569.46 2947591459
> > > > > >> 2212964228
> > > > > >> >> > sdf 4.51 747.22 571.78 2903740266
> > > > > >> 2221972652
> > > > > >> >> > sdf1 4.51 747.22 571.78 2903738331
> > > > > >> 2221972644
> > > > > >> >> >
> > > > > >> >> > Best,
> > > > > >> >> > zhenyu
> > > > > >> >> >
> > > > > >> >> >
> > > > > >> >> >
> > > > > >> >> > On Mon, Nov 16, 2009 at 4:07 PM, stack <st...@duboce.net>
> > > wrote:
> > > > > >> >> >
> > > > > >> >> >> On Mon, Nov 16, 2009 at 12:05 PM, Zhenyu Zhong <
> > > > > >> >> zhongresearch@gmail.com
> > > > > >> >> >> >wrote:
> > > > > >> >> >>
> > > > > >> >> >> > I just realized that there was a MapReduce job running
> > > during
> > > > > the
> > > > > >> >> time
> > > > > >> >> >> the
> > > > > >> >> >> > regionserver disconnected from the zookeeper.
> > > > > >> >> >> > That MapReduce Job was processing 500GB data and took
> > about
> > > 8
> > > > > >> minutes
> > > > > >> >> to
> > > > > >> >> >> > finish. It launched over 2000 map tasks.
> > > > > >> >> >>
> > > > > >> >> >>
> > > > > >> >> >> There was a tasktracker running on the RegionServer that
> > > > > >> disconnected?
> > > > > >> >> Is
> > > > > >> >> >> the map i/o or cpu heavy? Do you think it could have
> stole
> > > life
> > > > > >> from
> > > > > >> >> the
> > > > > >> >> >> datanode/regionserver?
> > > > > >> >> >>
> > > > > >> >> >>
> > > > > >> >> >>
> > > > > >> >> >> > I doubt that this introduced
> > > > > >> >> >> > resource contention between DataNode and RegionServer
> > Node.
> > > > > >> >> >> > Also during the time that MapReduce job ran, I saw a few
> > > > errors
> > > > > >> >> >> indicating
> > > > > >> >> >> > that
> > > > > >> >> >> >
> > > > > >> >> >> > java.io.IOException: Could not obtain block:
> > > > > >> >> >> > blk_7661247556283580226_4450462
> > > > file=/data/xxx.txt.200910141159
> > > > > >> >> >> > at
> > > > > >> >> >> >
> > > > > >> >> >>
> > > > > >> >>
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1787)
> > > > > >> >> >> > at
> > > > > >> >> >> >
> > > > > >> >> >>
> > > > > >> >>
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1615)
> > > > > >> >> >> > at
> > > > > >> >> >> >
> > > > > >> >> >>
> > > > > >> >>
> > > > > >>
> > > > >
> > >
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1742)
> > > > > >> >> >> > at
> > > > java.io.DataInputStream.read(DataInputStream.java:83)
> > > > > >> >> >> > at
> > > > > >> >> >>
> > > org.apache.hadoop.util.LineReader.readLine(LineReader.java:134)
> > > > > >> >> >> > at
> > > > > >> >> >> >
> > > > > >> >> >>
> > > > > >> >>
> > > > > >>
> > > > >
> > >
> org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:136)
> > > > > >> >> >> > at
> > > > > >> >> >> >
> > > > > >> >>
> > > > > >>
> > > >
> > org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:40)
> > > > > >> >> >> > at
> > > > > >> >> >> >
> > > > > >> >> >>
> > > > > >> >>
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:192)
> > > > > >> >> >> > at
> > > > > >> >> >> >
> > > > > >> >> >>
> > > > > >> >>
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:176)
> > > > > >> >> >> > at
> > > > > >> org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
> > > > > >> >> >> > at
> > > > > >> >> >>
> > > org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
> > > > > >> >> >> > at
> > > > org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
> > > > > >> >> >> > at
> > > org.apache.hadoop.mapred.Child.main(Child.java:170)
> > > > > >> >> >> >
> > > > > >> >> >>
> > > > > >> >> >> This is interesting. Your map task is running into
> > hdfs-127.
> > > > > Yeah,
> > > > > >> >> you
> > > > > >> >> >> should patch your cluster if you want to get rid of these
> > > > > (hdfs-127
> > > > > >> has
> > > > > >> >> >> been
> > > > > >> >> >> applied to 0.21 hadoop and will be in the next hadoop
> > release
> > > on
> > > > > >> 0.20.x
> > > > > >> >> >> branch, hadoop 0.20.2).
> > > > > >> >> >>
> > > > > >> >> >>
> > > > > >> >> >>
> > > > > >> >> >> >
> > > > > >> >> >> > Possibly it is related to HDFS-127, but I don't see any
> > > > > datanodes
> > > > > >> >> went
> > > > > >> >> >> > down. Does that affect Regionserver? Shall we apply the
> > > patch?
> > > > > >> >> >> >
> > > > > >> >> >> >
> > > > > >> >> >> Read up on hdfs-127 for explaination of whats going on.
> > > > > >> >> >>
> > > > > >> >> >>
> > > > > >> >> >>
> > > > > >> >> >> > Now I start to keep tracking of the virtual memory stats
> > to
> > > > see
> > > > > if
> > > > > >> >> the
> > > > > >> >> >> same
> > > > > >> >> >> > issue happens tomorrow around the same time period.
> > > > > >> >> >> >
> > > > > >> >> >>
> > > > > >> >> >> Excellent. Thanks for doing this.
> > > > > >> >> >>
> > > > > >> >> >>
> > > > > >> >> >>
> > > > > >> >> >> > And I highly suspect that this particular MapReduce job
> > > hurts
> > > > > >> HBase
> > > > > >> >> >> > Regionserver.
> > > > > >> >> >> >
> > > > > >> >> >> > PS. I also use gcviewer to parse the GC-log, I only see
> > > around
> > > > > 30
> > > > > >> >> >> seconds
> > > > > >> >> >> > pauses maximum.
> > > > > >> >> >> >
> > > > > >> >> >>
> > > > > >> >> >>
> > > > > >> >> >> Did you figure why your ten minute session is not
> applying?
> > > > > >> >> >>
> > > > > >> >> >> St.Ack
> > > > > >> >> >>
> > > > > >> >> >>
> > > > > >> >> >>
> > > > > >> >> >> >
> > > > > >> >> >> > Thanks a lot.
> > > > > >> >> >> > zhenyu
> > > > > >> >> >> >
> > > > > >> >> >> >
> > > > > >> >> >> >
> > > > > >> >> >> > On Sun, Nov 15, 2009 at 12:52 AM, Zhenyu Zhong <
> > > > > >> >> zhongresearch@gmail.com
> > > > > >> >> >> > >wrote:
> > > > > >> >> >> >
> > > > > >> >> >> > > J-D,
> > > > > >> >> >> > >
> > > > > >> >> >> > > Thank you very much for your comments.
> > > > > >> >> >> > > My company block the IRC port, so I have trouble to
> > > connect
> > > > to
> > > > > >> IRC
> > > > > >> >> >> > channel.
> > > > > >> >> >> > > I have been trying to ask the IT to open the IRC port
> > for
> > > > me,
> > > > > it
> > > > > >> >> might
> > > > > >> >> >> > take
> > > > > >> >> >> > > a while.
> > > > > >> >> >> > >
> > > > > >> >> >> > > Best,
> > > > > >> >> >> > > zhenyu
> > > > > >> >> >> > >
> > > > > >> >> >> > >
> > > > > >> >> >> > > On Sat, Nov 14, 2009 at 2:21 PM, Jean-Daniel Cryans <
> > > > > >> >> >> jdcryans@apache.org
> > > > > >> >> >> > >wrote:
> > > > > >> >> >> > >
> > > > > >> >> >> > >> The error you are getting is a disconnection from a
> > > > zookeeper
> > > > > >> >> server
> > > > > >> >> >> > >> and is very generic.
> > > > > >> >> >> > >>
> > > > > >> >> >> > >> ZK-86 is still opened and the last comment refers to
> > > ZK-111
> > > > > >> saying
> > > > > >> >> >> > >> that the bug (in unit tests) was probably fixed in
> > > release
> > > > > >> 3.0.0
> > > > > >> >> last
> > > > > >> >> >> > >> year.
> > > > > >> >> >> > >>
> > > > > >> >> >> > >> To figure the hang you have, you can try to jstack
> the
> > > > > process
> > > > > >> pid
> > > > > >> >> >> and
> > > > > >> >> >> > >> see exactly what's holding the RS from shutting down.
> > > > > >> >> >> > >>
> > > > > >> >> >> > >> Would it be possible for you to drop by the IRC
> > channel?
> > > > This
> > > > > >> way
> > > > > >> >> we
> > > > > >> >> >> > >> can debug this a at much faster pace.
> > > > > >> >> >> > >>
> > > > > >> >> >> > >> Thx!
> > > > > >> >> >> > >>
> > > > > >> >> >> > >> J-D
> > > > > >> >> >> > >>
> > > > > >> >> >> > >> On Sat, Nov 14, 2009 at 9:57 AM, Zhenyu Zhong <
> > > > > >> >> >> zhongresearch@gmail.com>
> > > > > >> >> >> > >> wrote:
> > > > > >> >> >> > >> > I found this.
> > > > > >> >> >> > >> > http://issues.apache.org/jira/browse/ZOOKEEPER-86
> > > > > >> >> >> > >> >
> > > > > >> >> >> > >> > It looks like the same error I had. Is it a
> zookeeper
> > > > bug?
> > > > > >> When
> > > > > >> >> >> will
> > > > > >> >> >> > >> HBase
> > > > > >> >> >> > >> > take the zookeeper version 3.3.0?
> > > > > >> >> >> > >> >
> > > > > >> >> >> > >> > thanks
> > > > > >> >> >> > >> > zhenyu
> > > > > >> >> >> > >> >
> > > > > >> >> >> > >> > On Sat, Nov 14, 2009 at 11:31 AM, Zhenyu Zhong <
> > > > > >> >> >> > zhongresearch@gmail.com
> > > > > >> >> >> > >> >wrote:
> > > > > >> >> >> > >> >
> > > > > >> >> >> > >> >> Now I really doubt about the zookeeper, from the
> log
> > I
> > > > saw
> > > > > >> >> errors
> > > > > >> >> >> > like
> > > > > >> >> >> > >> >> IOException Read Error, while the zookeeper client
> > > > > >> >> (regionserver )
> > > > > >> >> >> > >> tried to
> > > > > >> >> >> > >> >> read. But it got disconneted status from
> zookeeper.
> > > > > >> >> >> > >> >>
> > > > > >> >> >> > >> >> I don't see any load on any zookeeper quorum
> > servers.
> > > > > DiskIO
> > > > > >> is
> > > > > >> >> >> fine.
> > > > > >> >> >> > >> >>
> > > > > >> >> >> > >> >> Also when the regionserver decides to exit due to
> > the
> > > > > >> >> disconnect
> > > > > >> >> >> > status
> > > > > >> >> >> > >> >> from zookeeper, sometimes the regionserver hangs
> > > during
> > > > > the
> > > > > >> >> >> exiting.
> > > > > >> >> >> > We
> > > > > >> >> >> > >> can
> > > > > >> >> >> > >> >> still see the HRegionServer process even we don't
> > see
> > > it
> > > > > in
> > > > > >> the
> > > > > >> >> >> > Master
> > > > > >> >> >> > >> web
> > > > > >> >> >> > >> >> interface.
> > > > > >> >> >> > >> >>
> > > > > >> >> >> > >> >> I also notice that there is an zookeeper.retries
> and
> > > > > >> >> >> zookeeper.pauses
> > > > > >> >> >> > >> >> settings in hbase-default.xml, would that matter ?
> > > > > >> >> >> > >> >>
> > > > > >> >> >> > >> >> thanks
> > > > > >> >> >> > >> >>
> > > > > >> >> >> > >> >> You may see the IOException error in the following
> > log
> > > > > when
> > > > > >> the
> > > > > >> >> >> > >> >> Regionserver lost connection to the zookeeper.
> > > > > >> >> >> > >> >>
> > > > > >> >> >> > >> >> 2009-11-14 15:58:31,993 INFO
> > > > > >> >> >> > org.apache.hadoop.hbase.regionserver.HLog:
> > > > > >> >> >> > >> >> Roll
> > > > > >> >> >> > >> >>
> > > > > >> >> >> > >>
> > > > > >> >> >> >
> > > > > >> >> >>
> > > > > >> >>
> > > > > >>
> > > > >
> > > >
> > >
> >
> /hbase/.logs/superpyxis0008.scur.colo,60021,1258179101617/hlog.dat.1258179106921,
> > > > > >> >> >> > >> >> entries=1062964, calcsize=255014936,
> > > filesize=139566093.
> > > > > New
> > > > > >> >> hlog
> > > > > >> >> >> > >> >>
> > > > > >> >> >> > >>
> > > > > >> >> >> >
> > > > > >> >> >>
> > > > > >> >>
> > > > > >>
> > > > >
> > > >
> > >
> >
> /hbase/.logs/superpyxis0008.scur.colo,60021,1258179101617/hlog.dat.1258214311993
> > > > > >> >> >> > >> >> 2009-11-14 15:58:39,295 DEBUG
> > > > > >> org.apache.zookeeper.ClientCnxn:
> > > > > >> >> Got
> > > > > >> >> >> > ping
> > > > > >> >> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
> > > > > >> >> >> > >> >> 2009-11-14 15:58:52,648 DEBUG
> > > > > >> org.apache.zookeeper.ClientCnxn:
> > > > > >> >> Got
> > > > > >> >> >> > ping
> > > > > >> >> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
> > > > > >> >> >> > >> >> 2009-11-14 15:59:06,007 DEBUG
> > > > > >> org.apache.zookeeper.ClientCnxn:
> > > > > >> >> Got
> > > > > >> >> >> > ping
> > > > > >> >> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
> > > > > >> >> >> > >> >> 2009-11-14 15:59:19,365 DEBUG
> > > > > >> org.apache.zookeeper.ClientCnxn:
> > > > > >> >> Got
> > > > > >> >> >> > ping
> > > > > >> >> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
> > > > > >> >> >> > >> >> 2009-11-14 15:59:32,867 DEBUG
> > > > > >> org.apache.zookeeper.ClientCnxn:
> > > > > >> >> Got
> > > > > >> >> >> > ping
> > > > > >> >> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after
> 186ms
> > > > > >> >> >> > >> >> 2009-11-14 15:59:46,070 DEBUG
> > > > > >> org.apache.zookeeper.ClientCnxn:
> > > > > >> >> Got
> > > > > >> >> >> > ping
> > > > > >> >> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
> > > > > >> >> >> > >> >> 2009-11-14 15:59:59,423 DEBUG
> > > > > >> org.apache.zookeeper.ClientCnxn:
> > > > > >> >> Got
> > > > > >> >> >> > ping
> > > > > >> >> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
> > > > > >> >> >> > >> >> 2009-11-14 16:00:12,775 DEBUG
> > > > > >> org.apache.zookeeper.ClientCnxn:
> > > > > >> >> Got
> > > > > >> >> >> > ping
> > > > > >> >> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
> > > > > >> >> >> > >> >> 2009-11-14 16:00:28,141 DEBUG
> > > > > >> org.apache.zookeeper.ClientCnxn:
> > > > > >> >> Got
> > > > > >> >> >> > ping
> > > > > >> >> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after
> > 2010ms
> > > > > >> >> >> > >> >> 2009-11-14 16:01:10,378 WARN
> > > > > >> org.apache.zookeeper.ClientCnxn:
> > > > > >> >> >> > Exception
> > > > > >> >> >> > >> >> closing session 0x424f0ed01ea00a6 to
> > > > > >> >> >> > >> sun.nio.ch.SelectionKeyImpl@39fbb2d6
> > > > > >> >> >> > >> >>
> > > > > >> >> >> > >> >> java.io.IOException: Read error rc = -1
> > > > > >> >> >> > java.nio.DirectByteBuffer[pos=0
> > > > > >> >> >> > >> >> lim=4 cap=4]
> > > > > >> >> >> > >> >> at
> > > > > >> >> >> > >> >>
> > > > > >> >> >>
> > > > >
> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
> > > > > >> >> >> > >> >> at
> > > > > >> >> >> > >> >>
> > > > > >> >> >>
> > > > > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
> > > > > >> >> >> > >> >> 2009-11-14 16:01:10,478 INFO
> > > > > >> >> >> > >> >>
> org.apache.hadoop.hbase.regionserver.HRegionServer:
> > > Got
> > > > > >> >> ZooKeeper
> > > > > >> >> >> > >> event,
> > > > > >> >> >> > >> >> state: Disconnected, type: None, path: null
> > > > > >> >> >> > >> >> 2009-11-14 16:01:11,333 WARN
> > > > > >> >> >> > >> >>
> org.apache.hadoop.hbase.regionserver.HRegionServer:
> > > > > >> Attempt=1
> > > > > >> >> >> > >> >>
> > > > > >> >> >> > >> >>
> > org.apache.hadoop.hbase.Leases$LeaseStillHeldException
> > > > > >> >> >> > >> >> at
> > > > > >> >> >> > >>
> > > > sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> > > > > >> >> >> > >> >> Method)
> > > > > >> >> >> > >> >> at
> > > > > >> >> >> > >> >>
> > > > > >> >> >> > >>
> > > > > >> >> >> >
> > > > > >> >> >>
> > > > > >> >>
> > > > > >>
> > > > >
> > > >
> > >
> >
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> > > > > >> >> >> > >> >> at
> > > > > >> >> >> > >> >>
> > > > > >> >> >> > >>
> > > > > >> >> >> >
> > > > > >> >> >>
> > > > > >> >>
> > > > > >>
> > > > >
> > > >
> > >
> >
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> > > > > >> >> >> > >> >> at
> > > > > >> >> >> > >>
> > > > > java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> > > > > >> >> >> > >> >> at
> > > > > >> >> >> > >> >>
> > > > > >> >> >> > >>
> > > > > >> >> >> >
> > > > > >> >> >>
> > > > > >> >>
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
> > > > > >> >> >> > >> >> at
> > > > > >> >> >> > >> >>
> > > > > >> >> >> > >>
> > > > > >> >> >> >
> > > > > >> >> >>
> > > > > >> >>
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
> > > > > >> >> >> > >> >> at
> > > > > >> >> >> > >> >>
> > > > > >> >> >> > >>
> > > > > >> >> >> >
> > > > > >> >> >>
> > > > > >> >>
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
> > > > > >> >> >> > >> >> at
> > > > > >> >> >> > >> >>
> > > > > >> >> >> > >>
> > > > > >> >> >> >
> > > > > >> >> >>
> > > > > >> >>
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:571)
> > > > > >> >> >> > >> >> at java.lang.Thread.run(Thread.java:619)
> > > > > >> >> >> > >> >> 2009-11-14 16:01:11,433 INFO
> > > > > >> org.apache.zookeeper.ClientCnxn:
> > > > > >> >> >> > >> Attempting
> > > > > >> >> >> > >> >> connection to server superpyxis0001/
> > > > 192.168.100.115:2181
> > > > > >> >> >> > >> >> 2009-11-14 16:01:11,433 INFO
> > > > > >> org.apache.zookeeper.ClientCnxn:
> > > > > >> >> >> Priming
> > > > > >> >> >> > >> >> connection to
> > > java.nio.channels.SocketChannel[connected
> > > > > >> local=/
> > > > > >> >> >> > >> >> 192.168.100.122:59575 remote=superpyxis0001/
> > > > > >> >> 192.168.100.115:2181]
> > > > > >> >> >> > >> >> 2009-11-14 16:01:11,433 INFO
> > > > > >> org.apache.zookeeper.ClientCnxn:
> > > > > >> >> >> Server
> > > > > >> >> >> > >> >> connection successful
> > > > > >> >> >> > >> >> 2009-11-14 16:01:11,433 INFO
> > > > > >> >> >> > >> >>
> org.apache.hadoop.hbase.regionserver.HRegionServer:
> > > Got
> > > > > >> >> ZooKeeper
> > > > > >> >> >> > >> event,
> > > > > >> >> >> > >> >> state: Expired, type: None, path: null
> > > > > >> >> >> > >> >> 2009-11-14 16:01:11,433 WARN
> > > > > >> org.apache.zookeeper.ClientCnxn:
> > > > > >> >> >> > Exception
> > > > > >> >> >> > >> >> closing session 0x424f0ed01ea00a6 to
> > > > > >> >> >> > >> sun.nio.ch.SelectionKeyImpl@6b93d343
> > > > > >> >> >> > >> >> java.io.IOException: Session Expired
> > > > > >> >> >> > >> >> at
> > > > > >> >> >> > >> >>
> > > > > >> >> >> > >>
> > > > > >> >> >> >
> > > > > >> >> >>
> > > > > >> >>
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.zookeeper.ClientCnxn$SendThread.readConnectResult(ClientCnxn.java:589)
> > > > > >> >> >> > >> >> at
> > > > > >> >> >> > >> >>
> > > > > >> >> >>
> > > > >
> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:709)
> > > > > >> >> >> > >> >>
> > > > > >> >> >> > >> >> at
> > > > > >> >> >> > >> >>
> > > > > >> >> >>
> > > > > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
> > > > > >> >> >> > >> >> 2009-11-14 16:01:11,433 ERROR
> > > > > >> >> >> > >> >>
> org.apache.hadoop.hbase.regionserver.HRegionServer:
> > > > > >> ZooKeeper
> > > > > >> >> >> session
> > > > > >> >> >> > >> >> expired
> > > > > >> >> >> > >> >> 2009-11-14 16:01:13,513 INFO
> > > > > >> org.apache.hadoop.ipc.HBaseServer:
> > > > > >> >> >> IPC
> > > > > >> >> >> > >> Server
> > > > > >> >> >> > >> >> handler 3 on 60021, call put([B@73deb204,
> > > > > >> >> >> > >> >> [Lorg.apache.hadoop.hbase.client.Put;@2179600a)
> > from
> > > > > >> >> >> > >> 192.168.100.132:40728:
> > > > > >> >> >> > >> >> error: java.io.IOException: Server not running,
> > > aborting
> > > > > >> >> >> > >> >> java.io.IOException: Server not running, aborting
> > > > > >> >> >> > >> >>
> > > > > >> >> >> > >> >>
> > > > > >> >> >> > >> >>
> > > > > >> >> >> > >> >>
> > > > > >> >> >> > >> >>
> > > > > >> >> >> > >> >> On Sat, Nov 14, 2009 at 1:01 AM, Zhenyu Zhong <
> > > > > >> >> >> > zhongresearch@gmail.com
> > > > > >> >> >> > >> >wrote:
> > > > > >> >> >> > >> >>
> > > > > >> >> >> > >> >>> Stack,
> > > > > >> >> >> > >> >>>
> > > > > >> >> >> > >> >>> Thanks a lot!
> > > > > >> >> >> > >> >>>
> > > > > >> >> >> > >> >>> I found out that the reason why HBase doesn't
> take
> > > the
> > > > > >> system
> > > > > >> >> >> file
> > > > > >> >> >> > >> >>> descriptor value I set before. I started the
> HBase
> > > > using
> > > > > >> root
> > > > > >> >> >> > instead
> > > > > >> >> >> > >> of the
> > > > > >> >> >> > >> >>> normal hadoop user, while my system configures
> > hadoop
> > > > > with
> > > > > >> >> higher
> > > > > >> >> >> > file
> > > > > >> >> >> > >> >>> descriptor value but configures root with 1024
> > > default
> > > > > >> value.
> > > > > >> >> >> > >> >>>
> > > > > >> >> >> > >> >>>
> > > > > >> >> >> > >> >>> Now my system has a clean start. However, it
> seems
> > > that
> > > > > the
> > > > > >> >> >> > >> >>> zookeeper.session.timeout value doesn't take into
> > > > effect.
> > > > > I
> > > > > >> >> still
> > > > > >> >> >> > >> found
> > > > > >> >> >> > >> >>> around 60 seconds pauses from the disconnected
> > > > > >> regionserver. I
> > > > > >> >> >> > really
> > > > > >> >> >> > >> don't
> > > > > >> >> >> > >> >>> know why regionserver only times out after 60
> > seconds
> > > > > >> instead
> > > > > >> >> of
> > > > > >> >> >> 10
> > > > > >> >> >> > >> minutes
> > > > > >> >> >> > >> >>> which I set for zookeeper.session.timeout.
> > > > > >> >> >> > >> >>>
> > > > > >> >> >> > >> >>> Is there any other timeout value coming into play
> > > > before
> > > > > >> the
> > > > > >> >> >> actual
> > > > > >> >> >> > >> >>> session times out?
> > > > > >> >> >> > >> >>>
> > > > > >> >> >> > >> >>> zhenyu
> > > > > >> >> >> > >> >>>
> > > > > >> >> >> > >> >>>
> > > > > >> >> >> > >> >>>
> > > > > >> >> >> > >> >>> On Fri, Nov 13, 2009 at 7:08 PM, stack <
> > > > stack@duboce.net
> > > > > >
> > > > > >> >> wrote:
> > > > > >> >> >> > >> >>>
> > > > > >> >> >> > >> >>>> Ok. Lack of file descriptors manifests in all
> > kinds
> > > > of
> > > > > >> weird
> > > > > >> >> >> ways.
> > > > > >> >> >> > >> >>>> Hopefully thats it. If not, lets keep digging.
> > > > > >> >> >> > >> >>>> St.Ack
> > > > > >> >> >> > >> >>>>
> > > > > >> >> >> > >> >>>> On Fri, Nov 13, 2009 at 3:44 PM, Zhenyu Zhong <
> > > > > >> >> >> > >> zhongresearch@gmail.com
> > > > > >> >> >> > >> >>>> >wrote:
> > > > > >> >> >> > >> >>>>
> > > > > >> >> >> > >> >>>> > Stack,
> > > > > >> >> >> > >> >>>> >
> > > > > >> >> >> > >> >>>> > You are right, the master started with ulimit
> -n
> > > > 1024.
> > > > > >> It
> > > > > >> >> >> doesn't
> > > > > >> >> >> > >> take
> > > > > >> >> >> > >> >>>> the
> > > > > >> >> >> > >> >>>> > system value.
> > > > > >> >> >> > >> >>>> >
> > > > > >> >> >> > >> >>>> > Regarding to the too many open files, it looks
> > > like
> > > > > the
> > > > > >> >> same
> > > > > >> >> >> as
> > > > > >> >> >> > the
> > > > > >> >> >> > >> one
> > > > > >> >> >> > >> >>>> J-D
> > > > > >> >> >> > >> >>>> > put up. But I will get the Master start with
> > > higher
> > > > > >> value
> > > > > >> >> >> first
> > > > > >> >> >> > and
> > > > > >> >> >> > >> see
> > > > > >> >> >> > >> >>>> if
> > > > > >> >> >> > >> >>>> > this kind of error goes away.
> > > > > >> >> >> > >> >>>> >
> > > > > >> >> >> > >> >>>> > thanks a lot!
> > > > > >> >> >> > >> >>>> > zhenyu
> > > > > >> >> >> > >> >>>> >
> > > > > >> >> >> > >> >>>> > On Fri, Nov 13, 2009 at 6:02 PM, stack <
> > > > > >> stack@duboce.net>
> > > > > >> >> >> wrote:
> > > > > >> >> >> > >> >>>> >
> > > > > >> >> >> > >> >>>> > > Does it say
> > > > > >> >> >> > >> >>>> > >
> > > > > >> >> >> > >> >>>> > > ulimit -n 32768
> > > > > >> >> >> > >> >>>> > >
> > > > > >> >> >> > >> >>>> > > ...as the second line in your log file on
> > start
> > > of
> > > > > the
> > > > > >> >> >> master?
> > > > > >> >> >> > >> >>>> > >
> > > > > >> >> >> > >> >>>> > > Is the stack trace that complains about too
> > many
> > > > > open
> > > > > >> >> files
> > > > > >> >> >> > same
> > > > > >> >> >> > >> as
> > > > > >> >> >> > >> >>>> the
> > > > > >> >> >> > >> >>>> > one
> > > > > >> >> >> > >> >>>> > > in the blog post J-D put up?
> > > > > >> >> >> > >> >>>> > >
> > > > > >> >> >> > >> >>>> > > St.Ack
> > > > > >> >> >> > >> >>>> > >
> > > > > >> >> >> > >> >>>> > >
> > > > > >> >> >> > >> >>>> > > On Fri, Nov 13, 2009 at 1:37 PM, Zhenyu
> Zhong
> > <
> > > > > >> >> >> > >> >>>> zhongresearch@gmail.com
> > > > > >> >> >> > >> >>>> > > >wrote:
> > > > > >> >> >> > >> >>>> > >
> > > > > >> >> >> > >> >>>> > > > The ulimit file descriptors was set to
> > > > fs.file-max
> > > > > >> >> >> =1578334,
> > > > > >> >> >> > >> also
> > > > > >> >> >> > >> >>>> in
> > > > > >> >> >> > >> >>>> > the
> > > > > >> >> >> > >> >>>> > > > limits.conf the value was set to 32768.
> > > > > >> >> >> > >> >>>> > > > So these are way higher than the open
> > > > descriptors
> > > > > >> for
> > > > > >> >> the
> > > > > >> >> >> > >> running
> > > > > >> >> >> > >> >>>> > > > processes.
> > > > > >> >> >> > >> >>>> > > >
> > > > > >> >> >> > >> >>>> > > > thanks
> > > > > >> >> >> > >> >>>> > > > zhenyu
> > > > > >> >> >> > >> >>>> > > >
> > > > > >> >> >> > >> >>>> > > >
> > > > > >> >> >> > >> >>>> > > > On Fri, Nov 13, 2009 at 4:33 PM, Stack <
> > > > > >> >> >> saint.ack@gmail.com>
> > > > > >> >> >> > >> >>>> wrote:
> > > > > >> >> >> > >> >>>> > > >
> > > > > >> >> >> > >> >>>> > > > > You upped the ulimit file descriptors as
> > per
> > > > the
> > > > > >> >> getting
> > > > > >> >> >> > >> started
> > > > > >> >> >> > >> >>>> doc?
> > > > > >> >> >> > >> >>>> > > > >
> > > > > >> >> >> > >> >>>> > > > >
> > > > > >> >> >> > >> >>>> > > > >
> > > > > >> >> >> > >> >>>> > > > > On Nov 13, 2009, at 1:26 PM, Zhenyu
> Zhong
> > <
> > > > > >> >> >> > >> >>>> zhongresearch@gmail.com>
> > > > > >> >> >> > >> >>>> > > > wrote:
> > > > > >> >> >> > >> >>>> > > > >
> > > > > >> >> >> > >> >>>> > > > > Thanks a lot.
> > > > > >> >> >> > >> >>>> > > > >>
> > > > > >> >> >> > >> >>>> > > > >>
> > > > > >> >> >> > >> >>>> > > > >> Bad news is my kernel is still 2.6.26.
> > > > > >> >> >> > >> >>>> > > > >> But it was not a problem before.
> > > > > >> >> >> > >> >>>> > > > >>
> > > > > >> >> >> > >> >>>> > > > >> Very strange.
> > > > > >> >> >> > >> >>>> > > > >>
> > > > > >> >> >> > >> >>>> > > > >> zhenyu
> > > > > >> >> >> > >> >>>> > > > >>
> > > > > >> >> >> > >> >>>> > > > >> On Fri, Nov 13, 2009 at 4:16 PM,
> > > Jean-Daniel
> > > > > >> Cryans
> > > > > >> >> <
> > > > > >> >> >> > >> >>>> > > > jdcryans@apache.org
> > > > > >> >> >> > >> >>>> > > > >> >wrote:
> > > > > >> >> >> > >> >>>> > > > >>
> > > > > >> >> >> > >> >>>> > > > >> Looks like
> > > > > >> >> >> > >> >>>> > > > >>>
> > > > > >> >> >> > >> >>>> > > > >>>
> > > > > >> >> >> > >> >>>> > > >
> > > > > >> >> >> > >> >>>> > >
> > > > > >> >> >> > >> >>>> >
> > > > > >> >> >> > >> >>>>
> > > > > >> >> >> > >>
> > > > > >> >> >> >
> > > > > >> >> >>
> > > > > >> >>
> > > > > >>
> > > > >
> > > >
> > >
> >
> http://pero.blogs.aprilmayjune.org/2009/01/22/hadoop-and-linux-kernel-2627-epoll-limits/
> > > > > >> >> >> > >> >>>> > > > >>>
> > > > > >> >> >> > >> >>>> > > > >>> J-D
> > > > > >> >> >> > >> >>>> > > > >>>
> > > > > >> >> >> > >> >>>> > > > >>> On Fri, Nov 13, 2009 at 1:12 PM,
> Zhenyu
> > > > Zhong
> > > > > <
> > > > > >> >> >> > >> >>>> > > zhongresearch@gmail.com
> > > > > >> >> >> > >> >>>> > > > >
> > > > > >> >> >> > >> >>>> > > > >>> wrote:
> > > > > >> >> >> > >> >>>> > > > >>>
> > > > > >> >> >> > >> >>>> > > > >>>> Hi,
> > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > >> >> >> > >> >>>> > > > >>>> After I re-organize the cluster, the
> > > > > experiment
> > > > > >> >> ran
> > > > > >> >> >> into
> > > > > >> >> >> > >> >>>> problem
> > > > > >> >> >> > >> >>>> > > > faster
> > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > >> >> >> > >> >>>> > > > >>> than
> > > > > >> >> >> > >> >>>> > > > >>>
> > > > > >> >> >> > >> >>>> > > > >>>> before.
> > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > >> >> >> > >> >>>> > > > >>>> Basically the changes are to use
> > machines
> > > > > with
> > > > > >> >> less
> > > > > >> >> >> > >> resources
> > > > > >> >> >> > >> >>>> as
> > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > >> >> >> > >> >>>> > > > >>> zookeeper
> > > > > >> >> >> > >> >>>> > > > >>>
> > > > > >> >> >> > >> >>>> > > > >>>> quorums and machines with more
> > resources
> > > as
> > > > > >> >> >> > regionserver.
> > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > >> >> >> > >> >>>> > > > >>>> From the log, I still see the pause
> > > around
> > > > 1
> > > > > >> >> minute.
> > > > > >> >> >> > >> >>>> > > > >>>> I enabled the GC logging, please see
> > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > >> >> >> > >> >>>> > > > >>>> http://pastebin.com/m1d4ce0f1
> > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > >> >> >> > >> >>>> > > > >>>> for details.
> > > > > >> >> >> > >> >>>> > > > >>>> In general I don't see many pauses in
> > the
> > > > GC.
> > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > >> >> >> > >> >>>> > > > >>>> What is more interesting, 1 hour
> after
> > > the
> > > > > 1st
> > > > > >> >> >> > >> regionserver
> > > > > >> >> >> > >> >>>> > > > >>>> disconnected,
> > > > > >> >> >> > >> >>>> > > > >>>> the master log complained about too
> > many
> > > > open
> > > > > >> >> files.
> > > > > >> >> >> > This
> > > > > >> >> >> > >> >>>> didn't
> > > > > >> >> >> > >> >>>> > > > happen
> > > > > >> >> >> > >> >>>> > > > >>>> before.
> > > > > >> >> >> > >> >>>> > > > >>>> I checked the system OS setup as well
> > as
> > > > the
> > > > > >> >> >> > limits.conf.
> > > > > >> >> >> > >> I
> > > > > >> >> >> > >> >>>> also
> > > > > >> >> >> > >> >>>> > > > checked
> > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > >> >> >> > >> >>>> > > > >>> the
> > > > > >> >> >> > >> >>>> > > > >>>
> > > > > >> >> >> > >> >>>> > > > >>>> running processes. The total open
> files
> > > > don't
> > > > > >> >> reach
> > > > > >> >> >> the
> > > > > >> >> >> > >> limit.
> > > > > >> >> >> > >> >>>> I
> > > > > >> >> >> > >> >>>> > am
> > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > >> >> >> > >> >>>> > > > >>> confused
> > > > > >> >> >> > >> >>>> > > > >>>
> > > > > >> >> >> > >> >>>> > > > >>>> a bit.
> > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > >> >> >> > >> >>>> > > > >>>> Please see the following master log.
> > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:02,114 INFO
> > > > > >> >> >> > >> >>>> > > >
> org.apache.hadoop.hbase.master.BaseScanner:
> > > > > >> >> >> > >> >>>> > > > >>>> RegionManager.metaScanner scan of
> 4658
> > > > row(s)
> > > > > >> of
> > > > > >> >> meta
> > > > > >> >> >> > >> region
> > > > > >> >> >> > >> >>>> > > {server:
> > > > > >> >> >> > >> >>>> > > > >>>> 192.168.100.128:60021, regionname:
> > > > > .META.,,1,
> > > > > >> >> >> startKey:
> > > > > >> >> >> > >> <>}
> > > > > >> >> >> > >> >>>> > > complete
> > > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:02,114 INFO
> > > > > >> >> >> > >> >>>> > > >
> org.apache.hadoop.hbase.master.BaseScanner:
> > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > >> >> >> > >> >>>> > > > >>> All
> > > > > >> >> >> > >> >>>> > > > >>>
> > > > > >> >> >> > >> >>>> > > > >>>> 1 .META. region(s) scanned
> > > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:07,677 DEBUG
> > > > > >> >> >> > >> org.apache.zookeeper.ClientCnxn:
> > > > > >> >> >> > >> >>>> Got
> > > > > >> >> >> > >> >>>> > > > ping
> > > > > >> >> >> > >> >>>> > > > >>>> response for
> > sessionid:0x424eebf1c10004c
> > > > > after
> > > > > >> 3ms
> > > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:08,178 INFO
> > > > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > > > >> >> >> > >> >>>> > > > Exception
> > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > >> >> >> > >> >>>> > > > >>> in
> > > > > >> >> >> > >> >>>> > > > >>>
> > > > > >> >> >> > >> >>>> > > > >>>> createBlockOutputStream
> > > > java.io.IOException:
> > > > > >> Bad
> > > > > >> >> >> connect
> > > > > >> >> >> > >> ack
> > > > > >> >> >> > >> >>>> with
> > > > > >> >> >> > >> >>>> > > > >>>> firstBadLink 192.168.100.123:50010
> > > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:08,178 INFO
> > > > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > > > >> >> >> > >> >>>> > > > >>>> Abandoning
> > > > > >> >> >> > >> >>>> > > > >>>> block
> blk_-2808245019291145247_5478039
> > > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:09,682 INFO
> > > > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > > > >> >> >> > >> >>>> > > > Exception
> > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > >> >> >> > >> >>>> > > > >>> in
> > > > > >> >> >> > >> >>>> > > > >>>
> > > > > >> >> >> > >> >>>> > > > >>>> createBlockOutputStream
> > > > java.io.EOFException
> > > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:09,682 INFO
> > > > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > > > >> >> >> > >> >>>> > > > >>>> Abandoning
> > > > > >> >> >> > >> >>>> > > > >>>> block blk_1074853606841896259_5478048
> > > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:10,334 DEBUG
> > > > > >> >> >> > >> org.apache.zookeeper.ClientCnxn:
> > > > > >> >> >> > >> >>>> Got
> > > > > >> >> >> > >> >>>> > > > ping
> > > > > >> >> >> > >> >>>> > > > >>>> response for
> sessionid:0x24eebf1043003c
> > > > after
> > > > > >> 1ms
> > > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:21,018 DEBUG
> > > > > >> >> >> > >> org.apache.zookeeper.ClientCnxn:
> > > > > >> >> >> > >> >>>> Got
> > > > > >> >> >> > >> >>>> > > > ping
> > > > > >> >> >> > >> >>>> > > > >>>> response for
> > sessionid:0x424eebf1c10004c
> > > > > after
> > > > > >> 0ms
> > > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:23,674 DEBUG
> > > > > >> >> >> > >> org.apache.zookeeper.ClientCnxn:
> > > > > >> >> >> > >> >>>> Got
> > > > > >> >> >> > >> >>>> > > > ping
> > > > > >> >> >> > >> >>>> > > > >>>> response for
> sessionid:0x24eebf1043003c
> > > > after
> > > > > >> 0ms
> > > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:24,828 INFO
> > > > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > > > >> >> >> > >> >>>> > > > Exception
> > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > >> >> >> > >> >>>> > > > >>> in
> > > > > >> >> >> > >> >>>> > > > >>>
> > > > > >> >> >> > >> >>>> > > > >>>> createBlockOutputStream
> > > > java.io.IOException:
> > > > > >> Bad
> > > > > >> >> >> connect
> > > > > >> >> >> > >> ack
> > > > > >> >> >> > >> >>>> with
> > > > > >> >> >> > >> >>>> > > > >>>> firstBadLink 192.168.100.123:50010
> > > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:24,828 INFO
> > > > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > > > >> >> >> > >> >>>> > > > >>>> Abandoning
> > > > > >> >> >> > >> >>>> > > > >>>> block
> blk_-6642544517082142289_5478063
> > > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:24,828 INFO
> > > > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > > > >> >> >> > >> >>>> > > > Exception
> > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > >> >> >> > >> >>>> > > > >>> in
> > > > > >> >> >> > >> >>>> > > > >>>
> > > > > >> >> >> > >> >>>> > > > >>>> createBlockOutputStream
> > > > > >> java.net.SocketException:
> > > > > >> >> Too
> > > > > >> >> >> > many
> > > > > >> >> >> > >> >>>> open
> > > > > >> >> >> > >> >>>> > > files
> > > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:24,828 INFO
> > > > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > > > >> >> >> > >> >>>> > > > >>>> Abandoning
> > > > > >> >> >> > >> >>>> > > > >>>> block blk_2057511041109796090_5478063
> > > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:24,928 INFO
> > > > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > > > >> >> >> > >> >>>> > > > Exception
> > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > >> >> >> > >> >>>> > > > >>> in
> > > > > >> >> >> > >> >>>> > > > >>>
> > > > > >> >> >> > >> >>>> > > > >>>> createBlockOutputStream
> > > > > >> java.net.SocketException:
> > > > > >> >> Too
> > > > > >> >> >> > many
> > > > > >> >> >> > >> >>>> open
> > > > > >> >> >> > >> >>>> > > files
> > > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:24,928 INFO
> > > > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > > > >> >> >> > >> >>>> > > > >>>> Abandoning
> > > > > >> >> >> > >> >>>> > > > >>>> block blk_8219260302213892894_5478064
> > > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:30,855 INFO
> > > > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > > > >> >> >> > >> >>>> > > > Exception
> > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > >> >> >> > >> >>>> > > > >>> in
> > > > > >> >> >> > >> >>>> > > > >>>
> > > > > >> >> >> > >> >>>> > > > >>>> createBlockOutputStream
> > > > > >> java.net.SocketException:
> > > > > >> >> Too
> > > > > >> >> >> > many
> > > > > >> >> >> > >> >>>> open
> > > > > >> >> >> > >> >>>> > > files
> > > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:30,855 INFO
> > > > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > > > >> >> >> > >> >>>> > > > >>>> Abandoning
> > > > > >> >> >> > >> >>>> > > > >>>> block blk_1669205542853067709_5478235
> > > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:30,905 INFO
> > > > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > > > >> >> >> > >> >>>> > > > Exception
> > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > >> >> >> > >> >>>> > > > >>> in
> > > > > >> >> >> > >> >>>> > > > >>>
> > > > > >> >> >> > >> >>>> > > > >>>> createBlockOutputStream
> > > > > >> java.net.SocketException:
> > > > > >> >> Too
> > > > > >> >> >> > many
> > > > > >> >> >> > >> >>>> open
> > > > > >> >> >> > >> >>>> > > files
> > > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:30,905 INFO
> > > > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > > > >> >> >> > >> >>>> > > > >>>> Abandoning
> > > > > >> >> >> > >> >>>> > > > >>>> block blk_9128897691346270351_5478237
> > > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:30,955 INFO
> > > > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > > > >> >> >> > >> >>>> > > > Exception
> > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > >> >> >> > >> >>>> > > > >>> in
> > > > > >> >> >> > >> >>>> > > > >>>
> > > > > >> >> >> > >> >>>> > > > >>>> createBlockOutputStream
> > > > > >> java.net.SocketException:
> > > > > >> >> Too
> > > > > >> >> >> > many
> > > > > >> >> >> > >> >>>> open
> > > > > >> >> >> > >> >>>> > > files
> > > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:30,955 INFO
> > > > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > > > >> >> >> > >> >>>> > > > >>>> Abandoning
> > > > > >> >> >> > >> >>>> > > > >>>> block blk_1116845144864123018_5478240
> > > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:34,372 DEBUG
> > > > > >> >> >> > >> org.apache.zookeeper.ClientCnxn:
> > > > > >> >> >> > >> >>>> Got
> > > > > >> >> >> > >> >>>> > > > ping
> > > > > >> >> >> > >> >>>> > > > >>>> response for
> > sessionid:0x424eebf1c10004c
> > > > > after
> > > > > >> 0ms
> > > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:37,034 DEBUG
> > > > > >> >> >> > >> org.apache.zookeeper.ClientCnxn:
> > > > > >> >> >> > >> >>>> Got
> > > > > >> >> >> > >> >>>> > > > ping
> > > > > >> >> >> > >> >>>> > > > >>>> response for
> sessionid:0x24eebf1043003c
> > > > after
> > > > > >> 0ms
> > > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:37,235 WARN
> > > > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > >> >> >> > >> >>>> > > > >>> DataStreamer
> > > > > >> >> >> > >> >>>> > > > >>>
> > > > > >> >> >> > >> >>>> > > > >>>> Exception: java.io.IOException: Too
> > many
> > > > open
> > > > > >> >> files
> > > > > >> >> >> > >> >>>> > > > >>>> at
> > sun.nio.ch.IOUtil.initPipe(Native
> > > > > >> Method)
> > > > > >> >> >> > >> >>>> > > > >>>> at
> > > > > >> >> >> > >> >>>> > >
> > > > > >> >> >>
> > sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:49)
> > > > > >> >> >> > >> >>>> > > > >>>> at
> > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > >> >> >> > >> >>>> > > >
> > > > > >> >> >> > >> >>>> > >
> > > > > >> >> >> > >> >>>> >
> > > > > >> >> >> > >> >>>>
> > > > > >> >> >> > >>
> > > > > >> >> >> >
> > > > > >> >> >>
> > > > > >> >>
> > > > > >>
> > > > >
> > > >
> > >
> >
> sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:18)
> > > > > >> >> >> > >> >>>> > > > >>>
> > > > > >> >> >> > >> >>>> > > > >>>> at
> > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > >> >> >> > >> >>>> > > >
> > > > > >> >> >> > >> >>>> > >
> > > > > >> >> >> > >> >>>> >
> > > > > >> >> >> > >> >>>>
> > > > > >> >> >> > >>
> > > > > >> >> >> >
> > > > > >> >> >>
> > > > > >> >>
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.get(SocketIOWithTimeout.java:407)
> > > > > >> >> >> > >> >>>> > > > >>>
> > > > > >> >> >> > >> >>>> > > > >>>> at
> > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > >> >> >> > >> >>>> > > >
> > > > > >> >> >> > >> >>>> > >
> > > > > >> >> >> > >> >>>> >
> > > > > >> >> >> > >> >>>>
> > > > > >> >> >> > >>
> > > > > >> >> >> >
> > > > > >> >> >>
> > > > > >> >>
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:322)
> > > > > >> >> >> > >> >>>> > > > >>>
> > > > > >> >> >> > >> >>>> > > > >>>> at
> > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > >> >> >> > >> >>>> > > >
> > > > > >> >> >> > >> >>>> > >
> > > > > >> >> >> > >> >>>> >
> > > > > >> >> >> > >> >>>>
> > > > > >> >> >> > >>
> > > > > >> >> >> >
> > > > > >> >> >>
> > > > > >> >>
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
> > > > > >> >> >> > >> >>>> > > > >>>
> > > > > >> >> >> > >> >>>> > > > >>>> at
> > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > >> >> >> > >> >>>> > > >
> > > > > >> >> >> > >> >>>> > >
> > > > > >> >> >> > >> >>>> >
> > > > > >> >> >> > >> >>>>
> > > > > >> >> >> > >>
> > > > > >> >> >> >
> > > > > >> >> >>
> > > > > >> >>
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
> > > > > >> >> >> > >> >>>> > > > >>>
> > > > > >> >> >> > >> >>>> > > > >>>> at
> > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > >> >> >> > >> >>>> > > >
> > > > > >> >> >> > >> >>>> > >
> > > > > >> >> >> > >> >>>> >
> > > > > >> >> >> > >> >>>>
> > > > > >> >> >> > >>
> > > > > >> >> >> >
> > > > > >> >> >>
> > > > > >> >>
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
> > > > > >> >> >> > >> >>>> > > > >>>
> > > > > >> >> >> > >> >>>> > > > >>>> at
> > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > >> >> >> > >> >>>> > > > >>>
> > > > > >> >> >> > >> >>>>
> > > > > >> >> >>
> > > > java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
> > > > > >> >> >> > >> >>>> > > > >>>
> > > > > >> >> >> > >> >>>> > > > >>>> at
> > > > > >> >> >> > >> >>>>
> > > > java.io.DataOutputStream.write(DataOutputStream.java:90)
> > > > > >> >> >> > >> >>>> > > > >>>> at
> > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > >> >> >> > >> >>>> > > >
> > > > > >> >> >> > >> >>>> > >
> > > > > >> >> >> > >> >>>> >
> > > > > >> >> >> > >> >>>>
> > > > > >> >> >> > >>
> > > > > >> >> >> >
> > > > > >> >> >>
> > > > > >> >>
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2290)
> > > > > >> >> >> > >> >>>> > > > >>>
> > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:37,235 WARN
> > > > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > > > >> >> >> > >> >>>> > Error
> > > > > >> >> >> > >> >>>> > > > >>>> Recovery for block
> > > > > >> blk_8148813491785406356_5478475
> > > > > >> >> >> bad
> > > > > >> >> >> > >> >>>> datanode[0]
> > > > > >> >> >> > >> >>>> > > > >>>> 192.168.100.123:50010
> > > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:37,235 WARN
> > > > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > > > >> >> >> > >> >>>> > Error
> > > > > >> >> >> > >> >>>> > > > >>>> Recovery for block
> > > > > >> blk_8148813491785406356_5478475
> > > > > >> >> in
> > > > > >> >> >> > >> pipeline
> > > > > >> >> >> > >> >>>> > > > >>>> 192.168.100.123:50010,
> > > > 192.168.100.134:50010
> > > > > ,
> > > > > >> >> >> > >> >>>> > 192.168.100.122:50010
> > > > > >> >> >> > >> >>>> > > :
> > > > > >> >> >> > >> >>>> > > > >>>> bad
> > > > > >> >> >> > >> >>>> > > > >>>> datanode 192.168.100.123:50010
> > > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:37,436 INFO
> > > > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > > > >> >> >> > >> >>>> > > > Exception
> > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > >> >> >> > >> >>>> > > > >>> in
> > > > > >> >> >> > >> >>>> > > > >>>
> > > > > >> >> >> > >> >>>> > > > >>>> createBlockOutputStream
> > > > > >> java.net.SocketException:
> > > > > >> >> Too
> > > > > >> >> >> > many
> > > > > >> >> >> > >> >>>> open
> > > > > >> >> >> > >> >>>> > > files
> > > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:37,436 INFO
> > > > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > > > >> >> >> > >> >>>> > > > >>>> Abandoning
> > > > > >> >> >> > >> >>>> > > > >>>> block blk_2119727700857186236_5478498
> > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > >> >> >> > >> >>>> > > > >>>> On Thu, Nov 12, 2009 at 4:21 PM,
> Zhenyu
> > > > Zhong
> > > > > <
> > > > > >> >> >> > >> >>>> > > > zhongresearch@gmail.com
> > > > > >> >> >> > >> >>>> > > > >>>> wrote:
> > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > >> >> >> > >> >>>> > > > >>>> Will do.
> > > > > >> >> >> > >> >>>> > > > >>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>> thanks
> > > > > >> >> >> > >> >>>> > > > >>>>> zhenyu
> > > > > >> >> >> > >> >>>> > > > >>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>> On Thu, Nov 12, 2009 at 3:33 PM,
> stack
> > <
> > > > > >> >> >> > stack@duboce.net
> > > > > >> >> >> > >> >
> > > > > >> >> >> > >> >>>> wrote:
> > > > > >> >> >> > >> >>>> > > > >>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>> Enable GC logging too on this next
> > run
> > > > (see
> > > > > >> >> >> > >> hbase-env.sh).
> > > > > >> >> >> > >> >>>> Lets
> > > > > >> >> >> > >> >>>> > > try
> > > > > >> >> >> > >> >>>> > > > >>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>> and
> > > > > >> >> >> > >> >>>> > > > >>>
> > > > > >> >> >> > >> >>>> > > > >>>> get
> > > > > >> >> >> > >> >>>> > > > >>>>>> to the bottom of whats going on.
> > > > > >> >> >> > >> >>>> > > > >>>>>> Thanks,
> > > > > >> >> >> > >> >>>> > > > >>>>>> St.Ack
> > > > > >> >> >> > >> >>>> > > > >>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>> On Thu, Nov 12, 2009 at 12:29 PM,
> > > Zhenyu
> > > > > >> Zhong <
> > > > > >> >> >> > >> >>>> > > > >>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>> zhongresearch@gmail.com
> > > > > >> >> >> > >> >>>> > > > >>>
> > > > > >> >> >> > >> >>>> > > > >>>> wrote:
> > > > > >> >> >> > >> >>>> > > > >>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>> I can switch the boxes that run
> > > > zookeeper
> > > > > >> with
> > > > > >> >> the
> > > > > >> >> >> > ones
> > > > > >> >> >> > >> >>>> that
> > > > > >> >> >> > >> >>>> > run
> > > > > >> >> >> > >> >>>> > > > >>>>>>> regionservers.
> > > > > >> >> >> > >> >>>> > > > >>>>>>> I will see the results later.
> > > > > >> >> >> > >> >>>> > > > >>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>> FYI. The node does have the 10
> > minutes
> > > > > >> >> >> > >> >>>> > zookeeper.session.timeout
> > > > > >> >> >> > >> >>>> > > > >>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>> value
> > > > > >> >> >> > >> >>>> > > > >>>
> > > > > >> >> >> > >> >>>> > > > >>>> in
> > > > > >> >> >> > >> >>>> > > > >>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>> place.
> > > > > >> >> >> > >> >>>> > > > >>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>> thanks
> > > > > >> >> >> > >> >>>> > > > >>>>>>> zhenyu
> > > > > >> >> >> > >> >>>> > > > >>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>> On Thu, Nov 12, 2009 at 3:21 PM,
> > stack
> > > <
> > > > > >> >> >> > >> stack@duboce.net>
> > > > > >> >> >> > >> >>>> > wrote:
> > > > > >> >> >> > >> >>>> > > > >>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>> On Thu, Nov 12, 2009 at 11:50 AM,
> > > > Zhenyu
> > > > > >> Zhong
> > > > > >> >> <
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>> zhongresearch@gmail.com
> > > > > >> >> >> > >> >>>> > > > >>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>> wrote:
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>> In my cluster, half of the
> cluster
> > > > have
> > > > > 2
> > > > > >> >> disks
> > > > > >> >> >> > 400GB
> > > > > >> >> >> > >> >>>> each
> > > > > >> >> >> > >> >>>> > per
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>> machine,
> > > > > >> >> >> > >> >>>> > > > >>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>> and
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>> half of the cluster have 6 disks
> > per
> > > > > >> machine.
> > > > > >> >> >> > Maybe
> > > > > >> >> >> > >> we
> > > > > >> >> >> > >> >>>> > should
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>> run
> > > > > >> >> >> > >> >>>> > > > >>>
> > > > > >> >> >> > >> >>>> > > > >>>> zookeeper
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>> on the machines with 2 disks and
> > RS
> > > on
> > > > > >> >> machines
> > > > > >> >> >> > with
> > > > > >> >> >> > >> 6
> > > > > >> >> >> > >> >>>> disks?
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>> That would make most sense only
> in
> > > the
> > > > > >> below,
> > > > > >> >> it
> > > > > >> >> >> > >> looks
> > > > > >> >> >> > >> >>>> like
> > > > > >> >> >> > >> >>>> > the
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>> RS
> > > > > >> >> >> > >> >>>> > > > >>>
> > > > > >> >> >> > >> >>>> > > > >>>> that
> > > > > >> >> >> > >> >>>> > > > >>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>> had
> > > > > >> >> >> > >> >>>> > > > >>>>>>>> issue had 4 disks?
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>> BTW, the 10 minutes
> > > > > >> zookeeper.session.timeout
> > > > > >> >> >> has
> > > > > >> >> >> > >> been
> > > > > >> >> >> > >> >>>> set
> > > > > >> >> >> > >> >>>> > > during
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>> the
> > > > > >> >> >> > >> >>>> > > > >>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>> experiment.
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>> And for sure this node had it in
> > > place?
> > > > > >> >> >> > >> >>>> > > > >>>>>>>> St.Ack
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>> thanks
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>> zhenyu
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>> On Thu, Nov 12, 2009 at 2:08 PM,
> > > stack
> > > > <
> > > > > >> >> >> > >> stack@duboce.net
> > > > > >> >> >> > >> >>>> >
> > > > > >> >> >> > >> >>>> > > wrote:
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>> On Thu, Nov 12, 2009 at 8:40
> AM,
> > > > Zhenyu
> > > > > >> >> Zhong <
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>> zhongresearch@gmail.com
> > > > > >> >> >> > >> >>>> > > > >>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>> wrote:
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> Though I experienced 2
> > > regionserver
> > > > > >> >> >> disconnection
> > > > > >> >> >> > >> this
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> morning,
> > > > > >> >> >> > >> >>>> > > > >>>
> > > > > >> >> >> > >> >>>> > > > >>>> it
> > > > > >> >> >> > >> >>>> > > > >>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>> looks
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> better from the regionserver
> log.
> > > > > (please
> > > > > >> >> see
> > > > > >> >> >> the
> > > > > >> >> >> > >> >>>> following
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> log)
> > > > > >> >> >> > >> >>>> > > > >>>
> > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> http://pastebin.com/m496dbfae
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> I checked diskIO for one of
> the
> > > > > >> >> >> > >> >>>> > regionserver(192.168.100.116)
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> during
> > > > > >> >> >> > >> >>>> > > > >>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>> the
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> time it disconnected.
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> Time: 03:04:58 AM
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> Device: tps
> > > Blk_read/s
> > > > > >> >> >> Blk_wrtn/s
> > > > > >> >> >> > >> >>>> Blk_read
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> Blk_wrtn
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>> sda 105.31
> > 5458.83
> > > > > >> >> 19503.64
> > > > > >> >> >> > >> >>>> 9043873239
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> 32312473676
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>> sda1 2.90
> > 6.64
> > > > > >> >> 99.25
> > > > > >> >> >> > >> >>>> 10993934
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> 164433464
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>> sda2 1.72
> > 23.76
> > > > > >> >> 30.25
> > > > > >> >> >> > >> >>>> 39365817
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> 50124008
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>> sda3 0.30
> > 0.38
> > > > > >> >> 3.58
> > > > > >> >> >> > >> >>>> 624930
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> 5923000
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>> sda4 100.39
> > 5428.06
> > > > > >> >> 19370.56
> > > > > >> >> >> > >> >>>> 8992888270
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> 32091993204
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> Is this high for you? 20k
> > > > > >> blocks/second
> > > > > >> >> >> would
> > > > > >> >> >> > >> seem to
> > > > > >> >> >> > >> >>>> be
> > > > > >> >> >> > >> >>>> > > high
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>> but
> > > > > >> >> >> > >> >>>> > > > >>>
> > > > > >> >> >> > >> >>>> > > > >>>> its
> > > > > >> >> >> > >> >>>> > > > >>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>> one
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> disk only and its not being
> > shared
> > > by
> > > > > zk
> > > > > >> >> >> anymore
> > > > > >> >> >> > so
> > > > > >> >> >> > >> >>>> > shouldn't
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>> matter?
> > > > > >> >> >> > >> >>>> > > > >>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> I also checked the zookeeper
> > > quorum
> > > > > >> server
> > > > > >> >> >> that
> > > > > >> >> >> > the
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> regionserver
> > > > > >> >> >> > >> >>>> > > > >>>
> > > > > >> >> >> > >> >>>> > > > >>>> tried
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>> to
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> connect according to the log.
> > > > However,
> > > > > I
> > > > > >> >> don't
> > > > > >> >> >> see
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> 192.168.100.116
> > > > > >> >> >> > >> >>>> > > > >>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>> in
> > > > > >> >> >> > >> >>>> > > > >>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>> the
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> client list of the zookeeper
> > quorum
> > > > > >> server
> > > > > >> >> that
> > > > > >> >> >> > the
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> regionserver
> > > > > >> >> >> > >> >>>> > > > >>>
> > > > > >> >> >> > >> >>>> > > > >>>> tried
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>> to
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> connect.
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> Would that be a problem?
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> Is that because the ephemeral
> > > node
> > > > > for
> > > > > >> the
> > > > > >> >> >> > >> >>>> regionserver
> > > > > >> >> >> > >> >>>> > had
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>> evaporated?
> > > > > >> >> >> > >> >>>> > > > >>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>> Lost
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> its lease w/ zk by the time you
> > > went
> > > > to
> > > > > >> >> look?
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> Thu Nov 12 15:04:52 UTC 2009
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> Zookeeper version:
> 3.2.1-808558,
> > > > built
> > > > > >> on
> > > > > >> >> >> > >> 08/27/2009
> > > > > >> >> >> > >> >>>> 18:48
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> GMT
> > > > > >> >> >> > >> >>>> > > > >>>
> > > > > >> >> >> > >> >>>> > > > >>>> Clients:
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.127:43045
> > > > > >> >> >> > >> [1](queued=0,recved=26,sent=0)
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.131:39091
> > > > > >> >> >> > >> [1](queued=0,recved=964,sent=0)
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.124:35961
> > > > > >> >> >> > >> [1](queued=0,recved=3266,sent=0)
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.123:47935
> > > > > >> >> >> > >> [1](queued=0,recved=233,sent=0)
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.125:46931
> > > > > >> >> >> > >> [1](queued=0,recved=2,sent=0)
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.118:54924
> > > > > >> >> >> > >> [1](queued=0,recved=295,sent=0)
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.118:41390
> > > > > >> >> >> > >> [1](queued=0,recved=2290,sent=0)
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.136:42243
> > > > > >> >> >> > >> [1](queued=0,recved=0,sent=0)
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> Latency min/avg/max: 0/17/6333
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> Received: 47111
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> Sent: 0
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> Outstanding: 0
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> Zxid: 0x77000083f4
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> Mode: leader
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> Node count: 23
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> That 6 second maximum latency
> is
> > > > pretty
> > > > > >> bad
> > > > > >> >> but
> > > > > >> >> >> > >> should
> > > > > >> >> >> > >> >>>> be
> > > > > >> >> >> > >> >>>> > well
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>> within
> > > > > >> >> >> > >> >>>> > > > >>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>> the
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>> zk
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> session timeout.
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> So, problem is likely on the zk
> > > > client
> > > > > >> side
> > > > > >> >> of
> > > > > >> >> >> the
> > > > > >> >> >> > >> >>>> session;
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>> i.e.
> > > > > >> >> >> > >> >>>> > > > >>>
> > > > > >> >> >> > >> >>>> > > > >>>> in
> > > > > >> >> >> > >> >>>> > > > >>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>> the
> > > > > >> >> >> > >> >>>> > > > >>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>> RS.
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> You could enable GC logging as
> > J-D
> > > > > >> suggested
> > > > > >> >> to
> > > > > >> >> >> > see
> > > > > >> >> >> > >> if
> > > > > >> >> >> > >> >>>> you
> > > > > >> >> >> > >> >>>> > > have
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>> any
> > > > > >> >> >> > >> >>>> > > > >>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>> big
> > > > > >> >> >> > >> >>>> > > > >>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>> pauses, pauses > zk session
> > timeout.
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> When the RS went down, it
> didn't
> > > look
> > > > > too
> > > > > >> >> >> heavily
> > > > > >> >> >> > >> >>>> loaded:
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> 1. 2009-11-12 15:04:52,830
> INFO
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > >> >> >> > org.apache.hadoop.hbase.regionserver.HRegionServer:
> > > > > >> >> >> > >> >>>> Dump of
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>> metrics:
> > > > > >> >> >> > >> >>>> > > > >>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>> request=1.5166667, regions=322,
> > > > > >> stores=657,
> > > > > >> >> >> > >> >>>> storefiles=631,
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> storefileIndexSize=61,
> > > > > >> memstoreSize=1472,
> > > > > >> >> >> > >> >>>> usedHeap=2819,
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>> maxHeap=4079,
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>> blockCacheSize=658110960,
> > > > > >> >> >> > blockCacheFree=197395984,
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>> blockCacheCount=9903,
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> blockCacheHitRatio=99
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> Its serving a few reads? The
> > > number
> > > > of
> > > > > >> >> store
> > > > > >> >> >> > files
> > > > > >> >> >> > >> >>>> seems
> > > > > >> >> >> > >> >>>> > > fine.
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>> Not
> > > > > >> >> >> > >> >>>> > > > >>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>> too
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>> much memory used.
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> Looking at the logs, I see the
> > > Lease
> > > > > >> Still
> > > > > >> >> Held
> > > > > >> >> >> > >> >>>> exception.
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>> This
> > > > > >> >> >> > >> >>>> > > > >>>
> > > > > >> >> >> > >> >>>> > > > >>>> happens
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>> when the RS does its regular
> > report
> > > to
> > > > > the
> > > > > >> >> >> master
> > > > > >> >> >> > but
> > > > > >> >> >> > >> the
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>> master
> > > > > >> >> >> > >> >>>> > > > >>>
> > > > > >> >> >> > >> >>>> > > > >>>> thinks
> > > > > >> >> >> > >> >>>> > > > >>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>> the
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> RS has since restarted. It'll
> > > think
> > > > > this
> > > > > >> >> >> probably
> > > > > >> >> >> > >> >>>> because
> > > > > >> >> >> > >> >>>> > it
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>> noticed
> > > > > >> >> >> > >> >>>> > > > >>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>> that
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> the RS's znode in zk had gone
> > away
> > > > and
> > > > > it
> > > > > >> >> >> > considered
> > > > > >> >> >> > >> the
> > > > > >> >> >> > >> >>>> RS
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>> dead.
> > > > > >> >> >> > >> >>>> > > > >>>
> > > > > >> >> >> > >> >>>> > > > >>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> Looking too at your logs I see
> > > this
> > > > > gap
> > > > > >> in
> > > > > >> >> the
> > > > > >> >> >> zk
> > > > > >> >> >> > >> >>>> pinging:
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> 1. 2009-11-12 15:03:39,325
> DEBUG
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>> org.apache.zookeeper.ClientCnxn:
> > > > > >> >> >> > >> >>>> > > > >>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>> Got
> > > > > >> >> >> > >> >>>> > > > >>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>> ping response for
> > > > > >> sessionid:0x224e55436ad0004
> > > > > >> >> >> after
> > > > > >> >> >> > >> 0ms
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> 2. 2009-11-12 15:03:43,113
> DEBUG
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>> org.apache.zookeeper.ClientCnxn:
> > > > > >> >> >> > >> >>>> > > > >>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>> Got
> > > > > >> >> >> > >> >>>> > > > >>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>> ping response for
> > > > > >> sessionid:0x24e55436a0007d
> > > > > >> >> >> after
> > > > > >> >> >> > >> 0ms
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> Where in the lines above it,
> its
> > > > > >> reporting
> > > > > >> >> >> about
> > > > > >> >> >> > >> every
> > > > > >> >> >> > >> >>>> ten
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>> seconds,
> > > > > >> >> >> > >> >>>> > > > >>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>> here
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>> there is a big gap.
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> Do you have ganglia or
> something
> > > that
> > > > > >> will
> > > > > >> >> let
> > > > > >> >> >> you
> > > > > >> >> >> > >> look
> > > > > >> >> >> > >> >>>> more
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>> into
> > > > > >> >> >> > >> >>>> > > > >>>
> > > > > >> >> >> > >> >>>> > > > >>>> what
> > > > > >> >> >> > >> >>>> > > > >>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>> was
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> happening on this machine
> around
> > > this
> > > > > >> time?
> > > > > >> >> Is
> > > > > >> >> >> > the
> > > > > >> >> >> > >> >>>> machine
> > > > > >> >> >> > >> >>>> > > OK?
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>> It
> > > > > >> >> >> > >> >>>> > > > >>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>> looks
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>> lightly loaded and you have your
> > > > cluster
> > > > > >> >> nicely
> > > > > >> >> >> > laid
> > > > > >> >> >> > >> out.
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>> Something
> > > > > >> >> >> > >> >>>> > > > >>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>> odd
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>> is
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> going on. What about things
> like
> > > the
> > > > > >> write
> > > > > >> >> >> speed
> > > > > >> >> >> > to
> > > > > >> >> >> > >> >>>> disk?
> > > > > >> >> >> > >> >>>> > In
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>> the
> > > > > >> >> >> > >> >>>> > > > >>>
> > > > > >> >> >> > >> >>>> > > > >>>> past
> > > > > >> >> >> > >> >>>> > > > >>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>> strange issues have been
> explained
> > by
> > > > > >> >> incorrectly
> > > > > >> >> >> > set
> > > > > >> >> >> > >> BIOS
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>> which
> > > > > >> >> >> > >> >>>> > > > >>>
> > > > > >> >> >> > >> >>>> > > > >>>> made
> > > > > >> >> >> > >> >>>> > > > >>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>> disks
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> run at 1/100th of their proper
> > > speed.
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> St.Ack
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> Best,
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> zhenyu
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> On Wed, Nov 11, 2009 at 3:58
> PM,
> > > > > Zhenyu
> > > > > >> >> Zhong
> > > > > >> >> >> <
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> zhongresearch@gmail.com
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>> wrote:
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> Stack
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>> I am very appreciated for
> your
> > > > > >> comments.
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>> I will use the zookeeper
> > > monitoring
> > > > > >> script
> > > > > >> >> on
> > > > > >> >> >> my
> > > > > >> >> >> > >> >>>> cluster
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> and
> > > > > >> >> >> > >> >>>> > > > >>>
> > > > > >> >> >> > >> >>>> > > > >>>> let
> > > > > >> >> >> > >> >>>> > > > >>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>> it
> > > > > >> >> >> > >> >>>> > > > >>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>> run
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>> overnight to see the result.
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>> I will follow up the thread
> > when
> > > I
> > > > > get
> > > > > >> >> >> anything.
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>> thanks
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>> zhenyu
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>> On Wed, Nov 11, 2009 at 3:52
> > PM,
> > > > > stack
> > > > > >> <
> > > > > >> >> >> > >> >>>> stack@duboce.net>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> wrote:
> > > > > >> >> >> > >> >>>> > > > >>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>> I see these in your log too:
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>> 1. 2009-11-11 04:27:19,018
> > > DEBUG
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>
> > org.apache.zookeeper.ClientCnxn:
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>> Got
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> ping response for
> > > > > >> >> sessionid:0x424dfd908c50009
> > > > > >> >> >> > >> after
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>> 4544ms
> > > > > >> >> >> > >> >>>> > > > >>>
> > > > > >> >> >> > >> >>>> > > > >>>> 2. 2009-11-11 04:27:19,018 DEBUG
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>
> > org.apache.zookeeper.ClientCnxn:
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>> Got
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> ping response for
> > > > > >> >> sessionid:0x24dfd90c810002
> > > > > >> >> >> > after
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>> 4548ms
> > > > > >> >> >> > >> >>>> > > > >>>
> > > > > >> >> >> > >> >>>> > > > >>>> 3. 2009-11-11 04:27:43,960 DEBUG
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>
> > org.apache.zookeeper.ClientCnxn:
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>> Got
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> ping response for
> > > > > >> >> sessionid:0x424dfd908c50009
> > > > > >> >> >> > >> after
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>> 9030ms
> > > > > >> >> >> > >> >>>> > > > >>>
> > > > > >> >> >> > >> >>>> > > > >>>> 4. 2009-11-11 04:27:43,960 DEBUG
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>> org.apache.zookeeper.
> > > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > > >> >> >> > >> >>>> > > > >>>>>>>
> > > > > >> >> >> > >> >>>> > > >
> > > > > >> >> >> > >> >>>> > >
> > > > > >> >> >> > >> >>>> >
> > > > > >> >> >> > >> >>>>
> > > > > >> >> >> > >> >>>
> > > > > >> >> >> > >> >>>
> > > > > >> >> >> > >> >>
> > > > > >> >> >> > >> >
> > > > > >> >> >> > >>
> > > > > >> >> >> > >
> > > > > >> >> >> > >
> > > > > >> >> >> >
> > > > > >> >> >>
> > > > > >> >> >
> > > > > >> >> >
> > > > > >> >>
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: regionserver disconnection
Posted by Zhenyu Zhong <zh...@gmail.com>.
Yes, I believe so. And I don't know why the leader gave up.
For this case, the regionserver got TIMEOUT warning message at 2009-11-18
12:42:01,482
Best,
zhenyu
On Thu, Nov 19, 2009 at 11:43 AM, stack <st...@duboce.net> wrote:
> So, the time at which we see
>
> 2009-11-18 12:41:56,782 ERROR
> org.apache.zookeeper.server.quorum.FollowerHandler: Unexpected exception
> causing shutdown while sock still open
>
> ... corresponds to the time at which the regionserver loses is session?
>
> St.Ack
>
> On Thu, Nov 19, 2009 at 8:38 AM, Zhenyu Zhong <zhongresearch@gmail.com
> >wrote:
>
> > Here is the zookeeper leader log. It had an unexpected exception. I
> don't
> > know why.
> >
> > http://pastebin.com/m6594ccc
> >
> > During that time, I don't think the system load is high. Memory usage
> > should
> > be normal.
> >
> > Best,
> > zhenyu
> >
> >
> >
> > On Thu, Nov 19, 2009 at 11:09 AM, stack <st...@duboce.net> wrote:
> >
> > > What is in the other zk logs? Why did the leader go away? Memory
> issue
> > or
> > > something? Why'd it shutdown? Thanks for pastebin'ing this stuff.
> > > St.Ack
> > >
> > > On Thu, Nov 19, 2009 at 7:53 AM, Zhenyu Zhong <zhongresearch@gmail.com
> > > >wrote:
> > >
> > > > After digging more on the zookeeper log, I found that there seemed to
> > be
> > > a
> > > > socketconnection timeout. Also some warnings indicate that zookeeper
> > > server
> > > > is not running. Probably I need to understand the mechanism of
> > zookeeper
> > > > first.
> > > >
> > > > Please find the zookeeper log around that time.
> > > >
> > > > http://pastebin.com/m37bb4ad1
> > > >
> > > > Many thanks
> > > > zhenyu
> > > >
> > > >
> > > > On Thu, Nov 19, 2009 at 10:31 AM, Zhenyu Zhong <
> > zhongresearch@gmail.com
> > > > >wrote:
> > > >
> > > > > Stack,
> > > > >
> > > > > I am very appreciated. Let me dig into the zookeeper log more.
> > > > >
> > > > > FYI, I saw the sleeper complained about the delay for the first
> time.
> > > It
> > > > > looks like there are different reasons for the RS disconnections.
> > > > >
> > > > > 2009-11-19 08:08:32,708 DEBUG org.apache.zookeeper.ClientCnxn: Got
> > ping
> > > > > response for sessionid:0x42508716bfc0001 after 0ms
> > > > > 2009-11-19 08:08:37,595 DEBUG org.apache.zookeeper.ClientCnxn: Got
> > ping
> > > > > response for sessionid:0x42508716bfc0082 after 0ms
> > > > > 2009-11-19 08:08:46,073 DEBUG org.apache.zookeeper.ClientCnxn: Got
> > ping
> > > > > response for sessionid:0x42508716bfc0001 after 0ms
> > > > > 2009-11-19 08:09:39,880 WARN org.apache.zookeeper.ClientCnxn:
> > Exception
> > > > > closing session 0x42508716bfc0001 to
> > > sun.nio.ch.SelectionKeyImpl@4f3a608f
> > > > > java.io.IOException: TIMED OUT
> > > > > at
> > > > > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:906)
> > > > > 2009-11-19 08:09:39,880 WARN org.apache.zookeeper.ClientCnxn:
> > Exception
> > > > > closing session 0x42508716bfc0082 to
> > > sun.nio.ch.SelectionKeyImpl@14c8875
> > > > > java.io.IOException: TIMED OUT
> > > > > at
> > > > > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:906)
> > > > > 2009-11-19 08:09:39,880 WARN org.apache.hadoop.hbase.util.Sleeper:
> We
> > > > slept
> > > > > 53556ms, ten times longer than scheduled: 3000
> > > > > 2009-11-19 08:09:39,980 WARN
> > > > > org.apache.hadoop.hbase.regionserver.HRegionServer: Attempt=1
> > > > > org.apache.hadoop.hbase.Leases$LeaseStillHeldException
> > > > > at
> > > sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> > > > > Method)
> > > > > at
> > > > >
> > > >
> > >
> >
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> > > > > at
> > > > >
> > > >
> > >
> >
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> > > > > at
> > > > java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:571)
> > > > > at java.lang.Thread.run(Thread.java:619)
> > > > > 2009-11-19 08:09:39,980 INFO
> > > > > org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper
> > > event,
> > > > > state: Disconnected, type: None, path: null
> > > > > 2009-11-19 08:09:40,530 INFO org.apache.zookeeper.ClientCnxn:
> > > Attempting
> > > > > connection to server superpyxis0005/192.168.100.119:2181
> > > > > 2009-11-19 08:09:40,530 INFO org.apache.zookeeper.ClientCnxn:
> Priming
> > > > > connection to java.nio.channels.SocketChannel[connected local=/
> > > > > 192.168.100.122:40659 remote=superpyxis0005/192.168.100.119:2181]
> > > > >
> > > > >
> > > > >
> > > > > Best,
> > > > > zhenyu
> > > > >
> > > > > On Wed, Nov 18, 2009 at 6:52 PM, stack <st...@duboce.net> wrote:
> > > > >
> > > > >> Patrick Hunt, one of the zk lads, answered with the following:
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >> According to the following snippet (I cut to just session
> > > > >> 0x425035c48210002)
> > > > >> the client is getting pings for this session id regularly every 6
> > > > seconds
> > > > >> or
> > > > >> so, then after 41,016 there's a 20 second gap (corresponds to a 30
> > > > second
> > > > >> timeout being set during session creation) at which point we
> > > disconnect
> > > > >> from
> > > > >> the server.
> > > > >>
> > > > >>
> > > > >> > 2009-11-18 12:41:34,788 DEBUG org.apache.zookeeper.ClientCnxn:
> Got
> > > > ping
> > > > >> > response for sessionid:0x425035c48210002 after 0ms
> > > > >> > 2009-11-18 12:42:01,482 WARN org.apache.zookeeper.ClientCnxn:
> > > > Exception
> > > > >> > closing session 0x425035c48210002 to
> > > > >> sun.nio.ch.SelectionKeyImpl@421690ab
> > > > >> > java.io.IOException: TIMED OUT
> > > > >>
> > > > >> the client notifies hbase of the disco, and attempts to reconnect
> to
> > a
> > > > >> server
> > > > >>
> > > > >>
> > > > >> > org.apache.hadoop.hbase.regionserver.HRegionServer: Got
> ZooKeeper
> > > > event,
> > > > >> > state: Disconnected, type: None, path: null
> > > > >> > 2009-11-18 12:42:01,782 INFO org.apache.zookeeper.ClientCnxn:
> > > > Attempting
> > > > >> > connection to server superpyxis0003/192.168.100.117:2181
> > > > >>
> > > > >> but the server won't allow the session be be established for some
> > > reason
> > > > >> (is
> > > > >> that the right server:port that's it's trying to connect to?)
> > > > >>
> > > > >>
> > > > >> > 2009-11-18 12:42:02,182 INFO org.apache.zookeeper.ClientCnxn:
> > Server
> > > > >> > connection successful
> > > > >> > 2009-11-18 12:42:02,182 WARN org.apache.zookeeper.ClientCnxn:
> > > > Exception
> > > > >> > closing session 0x425035c48210002 to
> > > > >> sun.nio.ch.SelectionKeyImpl@5c07076b
> > > > >> > java.io.IOException: Read error rc = -1
> > > > java.nio.DirectByteBuffer[pos=0
> > > > >> > lim=4 cap=4]
> > > > >>
> > > > >> The "read error rc = -1" means that the client read from the
> socket
> > > and
> > > > >> got
> > > > >> EOS. But it's not a session expiration afaict (and should not be
> at
> > > this
> > > > >> point in time).
> > > > >>
> > > > >>
> > > > >> You need to look at the ZK server logs for this period in time
> (for
> > > this
> > > > >> server superpyxis0003:2181, and superpyxis0001:2181, the client
> > tries
> > > to
> > > > >> connect to both of these). There should be some indication there
> why
> > > the
> > > > >> server is closing down the connection and not allowing the session
> > to
> > > be
> > > > >> re-established.
> > > > >>
> > > > >> One thought - by default we limit clients to a max of 10
> connections
> > > > from
> > > > >> a
> > > > >> particular host to the server. If you exceed this limit the zk
> > server
> > > > >> (it's
> > > > >> per server though, not per cluster, so perhaps this is not it)
> will
> > > not
> > > > >> allow any additional connections from the client host (you can't
> > > exceed
> > > > >> 10
> > > > >> on a server). Could this be it? i.e. the user is running a number
> of
> > > ZK
> > > > >> clients (say hbase processes) from the same host? Regardless, get
> > the
> > > > >> server
> > > > >> log for this time period and it should be more clear.
> > > > >>
> > > > >> Patrick
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >> On Wed, Nov 18, 2009 at 11:18 AM, stack <st...@duboce.net> wrote:
> > > > >>
> > > > >> > And running stat on each of your zookeeper nodes tells you what
> > > about
> > > > >> > averages for connect-times?
> > > > >> >
> > > > >> > Its odd that there is a ping just 20 seconds beforehand and you
> > have
> > > > set
> > > > >> > the zk session timeout at 60 seconds (or ten minutes?). Let me
> > ask
> > > > the
> > > > >> zk
> > > > >> > lads about it.
> > > > >> >
> > > > >> > If you look in the GC log for around this time, do you see
> > anything?
> > > > >> >
> > > > >> > St.Ack
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > On Wed, Nov 18, 2009 at 7:32 AM, Zhenyu Zhong <
> > > > zhongresearch@gmail.com
> > > > >> >wrote:
> > > > >> >
> > > > >> >> After running more experiment, it seems that the disconnection
> is
> > > not
> > > > >> >> related to the heavy load job, because the load average is low,
> > the
> > > > >> diskio
> > > > >> >> is normal, memory Heap was not reached, also the virtual memory
> > > stats
> > > > >> >> shows
> > > > >> >> that no swappings. However, the disconnection still happens.
> > > > >> >>
> > > > >> >> It looks like this time it pauses for 20 seconds. No idea why
> > > > >> regionserver
> > > > >> >> disconnected.
> > > > >> >>
> > > > >> >> Any other suggestions please?
> > > > >> >>
> > > > >> >>
> > > > >> >> 2009-11-18 12:41:00,956 DEBUG org.apache.zookeeper.ClientCnxn:
> > Got
> > > > ping
> > > > >> >> response for sessionid:0x125035c495d02bb after 0ms
> > > > >> >> 2009-11-18 12:41:08,074 DEBUG org.apache.zookeeper.ClientCnxn:
> > Got
> > > > ping
> > > > >> >> response for sessionid:0x425035c48210002 after 0ms
> > > > >> >> 2009-11-18 12:41:14,302 DEBUG org.apache.zookeeper.ClientCnxn:
> > Got
> > > > ping
> > > > >> >> response for sessionid:0x125035c495d02bb after 0ms
> > > > >> >> 2009-11-18 12:41:21,433 DEBUG org.apache.zookeeper.ClientCnxn:
> > Got
> > > > ping
> > > > >> >> response for sessionid:0x425035c48210002 after 0ms
> > > > >> >> 2009-11-18 12:41:27,656 DEBUG org.apache.zookeeper.ClientCnxn:
> > Got
> > > > ping
> > > > >> >> response for sessionid:0x125035c495d02bb after 0ms
> > > > >> >> 2009-11-18 12:41:34,788 DEBUG org.apache.zookeeper.ClientCnxn:
> > Got
> > > > ping
> > > > >> >> response for sessionid:0x425035c48210002 after 0ms
> > > > >> >> 2009-11-18 12:41:41,016 DEBUG org.apache.zookeeper.ClientCnxn:
> > Got
> > > > ping
> > > > >> >> response for sessionid:0x125035c495d02bb after 1ms
> > > > >> >> 2009-11-18 12:42:01,482 WARN org.apache.zookeeper.ClientCnxn:
> > > > Exception
> > > > >> >> closing session 0x425035c48210002 to
> > > > >> sun.nio.ch.SelectionKeyImpl@421690ab
> > > > >> >> java.io.IOException: TIMED OUT
> > > > >> >> at
> > > > >> >>
> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:906)
> > > > >> >> 2009-11-18 12:42:01,582 INFO
> > > > >> >> org.apache.hadoop.hbase.regionserver.HRegionServer: Got
> ZooKeeper
> > > > >> event,
> > > > >> >> state: Disconnected, type: None, path: null
> > > > >> >> 2009-11-18 12:42:01,782 INFO org.apache.zookeeper.ClientCnxn:
> > > > >> Attempting
> > > > >> >> connection to server superpyxis0003/192.168.100.117:2181
> > > > >> >> 2009-11-18 12:42:01,782 INFO org.apache.zookeeper.ClientCnxn:
> > > Priming
> > > > >> >> connection to java.nio.channels.SocketChannel[connected local=/
> > > > >> >> 192.168.100.132:46610 remote=superpyxis0003/
> 192.168.100.117:2181
> > ]
> > > > >> >> 2009-11-18 12:42:01,782 INFO org.apache.zookeeper.ClientCnxn:
> > > Server
> > > > >> >> connection successful
> > > > >> >> 2009-11-18 12:42:01,782 WARN org.apache.zookeeper.ClientCnxn:
> > > > Exception
> > > > >> >> closing session 0x425035c48210002 to
> > > > >> sun.nio.ch.SelectionKeyImpl@56dc6fac
> > > > >> >> java.io.IOException: Read error rc = -1
> > > > java.nio.DirectByteBuffer[pos=0
> > > > >> >> lim=4 cap=4]
> > > > >> >> at
> > > > >> >>
> > > org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
> > > > >> >> at
> > > > >> >>
> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
> > > > >> >> 2009-11-18 12:42:01,782 WARN org.apache.zookeeper.ClientCnxn:
> > > > Ignoring
> > > > >> >> exception during shutdown input
> > > > >> >> java.net.SocketException: Transport endpoint is not connected
> > > > >> >> at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
> > > > >> >> at
> > > > >> >>
> > > >
> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
> > > > >> >> at
> > > > >> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> > > > >> >> at
> > > > >> >>
> > > >
> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> > > > >> >> at
> > > > >> >>
> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> > > > >> >> 2009-11-18 12:42:01,782 WARN org.apache.zookeeper.ClientCnxn:
> > > > Ignoring
> > > > >> >> exception during shutdown output
> > > > >> >> java.net.SocketException: Transport endpoint is not connected
> > > > >> >> at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
> > > > >> >> at
> > > > >> >>
> > > >
> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
> > > > >> >> at
> > > > >> sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> > > > >> >> at
> > > > >> >>
> > > > >>
> > >
> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> > > > >> >> at
> > > > >> >>
> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> > > > >> >> 2009-11-18 12:42:02,182 INFO org.apache.zookeeper.ClientCnxn:
> > > > >> Attempting
> > > > >> >> connection to server superpyxis0001/192.168.100.115:2181
> > > > >> >> 2009-11-18 12:42:02,182 INFO org.apache.zookeeper.ClientCnxn:
> > > Priming
> > > > >> >> connection to java.nio.channels.SocketChannel[connected local=/
> > > > >> >> 192.168.100.132:36197 remote=superpyxis0001/
> 192.168.100.115:2181
> > ]
> > > > >> >> 2009-11-18 12:42:02,182 INFO org.apache.zookeeper.ClientCnxn:
> > > Server
> > > > >> >> connection successful
> > > > >> >> 2009-11-18 12:42:02,182 WARN org.apache.zookeeper.ClientCnxn:
> > > > Exception
> > > > >> >> closing session 0x425035c48210002 to
> > > > >> sun.nio.ch.SelectionKeyImpl@5c07076b
> > > > >> >> java.io.IOException: Read error rc = -1
> > > > java.nio.DirectByteBuffer[pos=0
> > > > >> >> lim=4 cap=4]
> > > > >> >> at
> > > > >> >>
> > > org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
> > > > >> >> at
> > > > >> >>
> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
> > > > >> >> 2009-11-18 12:42:02,182 WARN org.apache.zookeeper.ClientCnxn:
> > > > Ignoring
> > > > >> >> exception during shutdown input
> > > > >> >> java.net.SocketException: Transport endpoint is not connected
> > > > >> >> at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
> > > > >> >> at
> > > > >> >>
> > > >
> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
> > > > >> >> at
> > > > >> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> > > > >> >> at
> > > > >> >>
> > > >
> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> > > > >> >> at
> > > > >> >>
> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> > > > >> >> 2009-11-18 12:42:02,182 WARN org.apache.zookeeper.ClientCnxn:
> > > > Ignoring
> > > > >> >> exception during shutdown output
> > > > >> >> java.net.SocketException: Transport endpoint is not connected
> > > > >> >>
> > > > >> >>
> > > > >> >> On Mon, Nov 16, 2009 at 4:16 PM, Zhenyu Zhong <
> > > > zhongresearch@gmail.com
> > > > >> >> >wrote:
> > > > >> >>
> > > > >> >> > Here is the diskIO and CPU around the time we had RS
> > > disconnection
> > > > on
> > > > >> >> one
> > > > >> >> > machine that runs RegionServer. It doesn't seem to be high.
> > > Similar
> > > > >> disk
> > > > >> >> and
> > > > >> >> > cpu usage have been seen before and the HBase was running
> fine.
> > > > >> >> >
> > > > >> >> >
> > > > >> >> > So far I haven't found why my 10 minutes session timeout
> > doesn't
> > > > >> apply.
> > > > >> >> > Still digging.
> > > > >> >> >
> > > > >> >> > Many thanks!
> > > > >> >> >
> > > > >> >> > Time: 03:01:08 PM
> > > > >> >> > avg-cpu: %user %nice %system %iowait %steal %idle
> > > > >> >> > 3.82 0.00 0.89 3.97 0.00 91.32
> > > > >> >> >
> > > > >> >> >
> > > > >> >> > Device: tps Blk_read/s Blk_wrtn/s Blk_read
> > > > >> Blk_wrtn
> > > > >> >> > sda 24.30 1605.39 6679.35 6238623388
> > > > >> 25956251720
> > > > >> >> > sda1 0.51 1.94 15.71 7547265
> > > > >> 61067904
> > > > >> >> > sda2 0.13 2.11 5.20 8202079
> > > > >> 20189232
> > > > >> >> > sda3 0.13 0.25 1.63 987846
> > > > >> 6323336
> > > > >> >> > sda4 0.00 0.00 0.00 4
> > > > >> 0
> > > > >> >> > sda5 1.78 0.06 24.14 220010
> > > > >> 93817208
> > > > >> >> > sda6 0.38 0.42 8.41 1630584
> > > > >> 32688152
> > > > >> >> > sda7 21.38 1600.61 6624.26 6220035272
> > > > >> 25742165888
> > > > >> >> > sdb 4.52 767.33 565.91 2981868690
> > > > >> 2199132380
> > > > >> >> > sdb1 4.52 767.33 565.91 2981866755
> > > > >> 2199132372
> > > > >> >> > sdc 4.42 742.95 563.00 2887151482
> > > > >> 2187823092
> > > > >> >> > sdc1 4.42 742.95 563.00 2887149547
> > > > >> 2187823084
> > > > >> >> > sdd 4.49 750.78 557.25 2917554074
> > > > >> 2165513500
> > > > >> >> > sdd1 4.49 750.78 557.25 2917552139
> > > > >> 2165513492
> > > > >> >> > sde 4.52 758.51 569.46 2947593394
> > > > >> 2212964236
> > > > >> >> > sde1 4.52 758.51 569.46 2947591459
> > > > >> 2212964228
> > > > >> >> > sdf 4.51 747.22 571.78 2903740266
> > > > >> 2221972652
> > > > >> >> > sdf1 4.51 747.22 571.78 2903738331
> > > > >> 2221972644
> > > > >> >> >
> > > > >> >> > Best,
> > > > >> >> > zhenyu
> > > > >> >> >
> > > > >> >> >
> > > > >> >> >
> > > > >> >> > On Mon, Nov 16, 2009 at 4:07 PM, stack <st...@duboce.net>
> > wrote:
> > > > >> >> >
> > > > >> >> >> On Mon, Nov 16, 2009 at 12:05 PM, Zhenyu Zhong <
> > > > >> >> zhongresearch@gmail.com
> > > > >> >> >> >wrote:
> > > > >> >> >>
> > > > >> >> >> > I just realized that there was a MapReduce job running
> > during
> > > > the
> > > > >> >> time
> > > > >> >> >> the
> > > > >> >> >> > regionserver disconnected from the zookeeper.
> > > > >> >> >> > That MapReduce Job was processing 500GB data and took
> about
> > 8
> > > > >> minutes
> > > > >> >> to
> > > > >> >> >> > finish. It launched over 2000 map tasks.
> > > > >> >> >>
> > > > >> >> >>
> > > > >> >> >> There was a tasktracker running on the RegionServer that
> > > > >> disconnected?
> > > > >> >> Is
> > > > >> >> >> the map i/o or cpu heavy? Do you think it could have stole
> > life
> > > > >> from
> > > > >> >> the
> > > > >> >> >> datanode/regionserver?
> > > > >> >> >>
> > > > >> >> >>
> > > > >> >> >>
> > > > >> >> >> > I doubt that this introduced
> > > > >> >> >> > resource contention between DataNode and RegionServer
> Node.
> > > > >> >> >> > Also during the time that MapReduce job ran, I saw a few
> > > errors
> > > > >> >> >> indicating
> > > > >> >> >> > that
> > > > >> >> >> >
> > > > >> >> >> > java.io.IOException: Could not obtain block:
> > > > >> >> >> > blk_7661247556283580226_4450462
> > > file=/data/xxx.txt.200910141159
> > > > >> >> >> > at
> > > > >> >> >> >
> > > > >> >> >>
> > > > >> >>
> > > > >>
> > > >
> > >
> >
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1787)
> > > > >> >> >> > at
> > > > >> >> >> >
> > > > >> >> >>
> > > > >> >>
> > > > >>
> > > >
> > >
> >
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1615)
> > > > >> >> >> > at
> > > > >> >> >> >
> > > > >> >> >>
> > > > >> >>
> > > > >>
> > > >
> > org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1742)
> > > > >> >> >> > at
> > > java.io.DataInputStream.read(DataInputStream.java:83)
> > > > >> >> >> > at
> > > > >> >> >>
> > org.apache.hadoop.util.LineReader.readLine(LineReader.java:134)
> > > > >> >> >> > at
> > > > >> >> >> >
> > > > >> >> >>
> > > > >> >>
> > > > >>
> > > >
> > org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:136)
> > > > >> >> >> > at
> > > > >> >> >> >
> > > > >> >>
> > > > >>
> > >
> org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:40)
> > > > >> >> >> > at
> > > > >> >> >> >
> > > > >> >> >>
> > > > >> >>
> > > > >>
> > > >
> > >
> >
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:192)
> > > > >> >> >> > at
> > > > >> >> >> >
> > > > >> >> >>
> > > > >> >>
> > > > >>
> > > >
> > >
> >
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:176)
> > > > >> >> >> > at
> > > > >> org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
> > > > >> >> >> > at
> > > > >> >> >>
> > org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
> > > > >> >> >> > at
> > > org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
> > > > >> >> >> > at
> > org.apache.hadoop.mapred.Child.main(Child.java:170)
> > > > >> >> >> >
> > > > >> >> >>
> > > > >> >> >> This is interesting. Your map task is running into
> hdfs-127.
> > > > Yeah,
> > > > >> >> you
> > > > >> >> >> should patch your cluster if you want to get rid of these
> > > > (hdfs-127
> > > > >> has
> > > > >> >> >> been
> > > > >> >> >> applied to 0.21 hadoop and will be in the next hadoop
> release
> > on
> > > > >> 0.20.x
> > > > >> >> >> branch, hadoop 0.20.2).
> > > > >> >> >>
> > > > >> >> >>
> > > > >> >> >>
> > > > >> >> >> >
> > > > >> >> >> > Possibly it is related to HDFS-127, but I don't see any
> > > > datanodes
> > > > >> >> went
> > > > >> >> >> > down. Does that affect Regionserver? Shall we apply the
> > patch?
> > > > >> >> >> >
> > > > >> >> >> >
> > > > >> >> >> Read up on hdfs-127 for explaination of whats going on.
> > > > >> >> >>
> > > > >> >> >>
> > > > >> >> >>
> > > > >> >> >> > Now I start to keep tracking of the virtual memory stats
> to
> > > see
> > > > if
> > > > >> >> the
> > > > >> >> >> same
> > > > >> >> >> > issue happens tomorrow around the same time period.
> > > > >> >> >> >
> > > > >> >> >>
> > > > >> >> >> Excellent. Thanks for doing this.
> > > > >> >> >>
> > > > >> >> >>
> > > > >> >> >>
> > > > >> >> >> > And I highly suspect that this particular MapReduce job
> > hurts
> > > > >> HBase
> > > > >> >> >> > Regionserver.
> > > > >> >> >> >
> > > > >> >> >> > PS. I also use gcviewer to parse the GC-log, I only see
> > around
> > > > 30
> > > > >> >> >> seconds
> > > > >> >> >> > pauses maximum.
> > > > >> >> >> >
> > > > >> >> >>
> > > > >> >> >>
> > > > >> >> >> Did you figure why your ten minute session is not applying?
> > > > >> >> >>
> > > > >> >> >> St.Ack
> > > > >> >> >>
> > > > >> >> >>
> > > > >> >> >>
> > > > >> >> >> >
> > > > >> >> >> > Thanks a lot.
> > > > >> >> >> > zhenyu
> > > > >> >> >> >
> > > > >> >> >> >
> > > > >> >> >> >
> > > > >> >> >> > On Sun, Nov 15, 2009 at 12:52 AM, Zhenyu Zhong <
> > > > >> >> zhongresearch@gmail.com
> > > > >> >> >> > >wrote:
> > > > >> >> >> >
> > > > >> >> >> > > J-D,
> > > > >> >> >> > >
> > > > >> >> >> > > Thank you very much for your comments.
> > > > >> >> >> > > My company block the IRC port, so I have trouble to
> > connect
> > > to
> > > > >> IRC
> > > > >> >> >> > channel.
> > > > >> >> >> > > I have been trying to ask the IT to open the IRC port
> for
> > > me,
> > > > it
> > > > >> >> might
> > > > >> >> >> > take
> > > > >> >> >> > > a while.
> > > > >> >> >> > >
> > > > >> >> >> > > Best,
> > > > >> >> >> > > zhenyu
> > > > >> >> >> > >
> > > > >> >> >> > >
> > > > >> >> >> > > On Sat, Nov 14, 2009 at 2:21 PM, Jean-Daniel Cryans <
> > > > >> >> >> jdcryans@apache.org
> > > > >> >> >> > >wrote:
> > > > >> >> >> > >
> > > > >> >> >> > >> The error you are getting is a disconnection from a
> > > zookeeper
> > > > >> >> server
> > > > >> >> >> > >> and is very generic.
> > > > >> >> >> > >>
> > > > >> >> >> > >> ZK-86 is still opened and the last comment refers to
> > ZK-111
> > > > >> saying
> > > > >> >> >> > >> that the bug (in unit tests) was probably fixed in
> > release
> > > > >> 3.0.0
> > > > >> >> last
> > > > >> >> >> > >> year.
> > > > >> >> >> > >>
> > > > >> >> >> > >> To figure the hang you have, you can try to jstack the
> > > > process
> > > > >> pid
> > > > >> >> >> and
> > > > >> >> >> > >> see exactly what's holding the RS from shutting down.
> > > > >> >> >> > >>
> > > > >> >> >> > >> Would it be possible for you to drop by the IRC
> channel?
> > > This
> > > > >> way
> > > > >> >> we
> > > > >> >> >> > >> can debug this a at much faster pace.
> > > > >> >> >> > >>
> > > > >> >> >> > >> Thx!
> > > > >> >> >> > >>
> > > > >> >> >> > >> J-D
> > > > >> >> >> > >>
> > > > >> >> >> > >> On Sat, Nov 14, 2009 at 9:57 AM, Zhenyu Zhong <
> > > > >> >> >> zhongresearch@gmail.com>
> > > > >> >> >> > >> wrote:
> > > > >> >> >> > >> > I found this.
> > > > >> >> >> > >> > http://issues.apache.org/jira/browse/ZOOKEEPER-86
> > > > >> >> >> > >> >
> > > > >> >> >> > >> > It looks like the same error I had. Is it a zookeeper
> > > bug?
> > > > >> When
> > > > >> >> >> will
> > > > >> >> >> > >> HBase
> > > > >> >> >> > >> > take the zookeeper version 3.3.0?
> > > > >> >> >> > >> >
> > > > >> >> >> > >> > thanks
> > > > >> >> >> > >> > zhenyu
> > > > >> >> >> > >> >
> > > > >> >> >> > >> > On Sat, Nov 14, 2009 at 11:31 AM, Zhenyu Zhong <
> > > > >> >> >> > zhongresearch@gmail.com
> > > > >> >> >> > >> >wrote:
> > > > >> >> >> > >> >
> > > > >> >> >> > >> >> Now I really doubt about the zookeeper, from the log
> I
> > > saw
> > > > >> >> errors
> > > > >> >> >> > like
> > > > >> >> >> > >> >> IOException Read Error, while the zookeeper client
> > > > >> >> (regionserver )
> > > > >> >> >> > >> tried to
> > > > >> >> >> > >> >> read. But it got disconneted status from zookeeper.
> > > > >> >> >> > >> >>
> > > > >> >> >> > >> >> I don't see any load on any zookeeper quorum
> servers.
> > > > DiskIO
> > > > >> is
> > > > >> >> >> fine.
> > > > >> >> >> > >> >>
> > > > >> >> >> > >> >> Also when the regionserver decides to exit due to
> the
> > > > >> >> disconnect
> > > > >> >> >> > status
> > > > >> >> >> > >> >> from zookeeper, sometimes the regionserver hangs
> > during
> > > > the
> > > > >> >> >> exiting.
> > > > >> >> >> > We
> > > > >> >> >> > >> can
> > > > >> >> >> > >> >> still see the HRegionServer process even we don't
> see
> > it
> > > > in
> > > > >> the
> > > > >> >> >> > Master
> > > > >> >> >> > >> web
> > > > >> >> >> > >> >> interface.
> > > > >> >> >> > >> >>
> > > > >> >> >> > >> >> I also notice that there is an zookeeper.retries and
> > > > >> >> >> zookeeper.pauses
> > > > >> >> >> > >> >> settings in hbase-default.xml, would that matter ?
> > > > >> >> >> > >> >>
> > > > >> >> >> > >> >> thanks
> > > > >> >> >> > >> >>
> > > > >> >> >> > >> >> You may see the IOException error in the following
> log
> > > > when
> > > > >> the
> > > > >> >> >> > >> >> Regionserver lost connection to the zookeeper.
> > > > >> >> >> > >> >>
> > > > >> >> >> > >> >> 2009-11-14 15:58:31,993 INFO
> > > > >> >> >> > org.apache.hadoop.hbase.regionserver.HLog:
> > > > >> >> >> > >> >> Roll
> > > > >> >> >> > >> >>
> > > > >> >> >> > >>
> > > > >> >> >> >
> > > > >> >> >>
> > > > >> >>
> > > > >>
> > > >
> > >
> >
> /hbase/.logs/superpyxis0008.scur.colo,60021,1258179101617/hlog.dat.1258179106921,
> > > > >> >> >> > >> >> entries=1062964, calcsize=255014936,
> > filesize=139566093.
> > > > New
> > > > >> >> hlog
> > > > >> >> >> > >> >>
> > > > >> >> >> > >>
> > > > >> >> >> >
> > > > >> >> >>
> > > > >> >>
> > > > >>
> > > >
> > >
> >
> /hbase/.logs/superpyxis0008.scur.colo,60021,1258179101617/hlog.dat.1258214311993
> > > > >> >> >> > >> >> 2009-11-14 15:58:39,295 DEBUG
> > > > >> org.apache.zookeeper.ClientCnxn:
> > > > >> >> Got
> > > > >> >> >> > ping
> > > > >> >> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
> > > > >> >> >> > >> >> 2009-11-14 15:58:52,648 DEBUG
> > > > >> org.apache.zookeeper.ClientCnxn:
> > > > >> >> Got
> > > > >> >> >> > ping
> > > > >> >> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
> > > > >> >> >> > >> >> 2009-11-14 15:59:06,007 DEBUG
> > > > >> org.apache.zookeeper.ClientCnxn:
> > > > >> >> Got
> > > > >> >> >> > ping
> > > > >> >> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
> > > > >> >> >> > >> >> 2009-11-14 15:59:19,365 DEBUG
> > > > >> org.apache.zookeeper.ClientCnxn:
> > > > >> >> Got
> > > > >> >> >> > ping
> > > > >> >> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
> > > > >> >> >> > >> >> 2009-11-14 15:59:32,867 DEBUG
> > > > >> org.apache.zookeeper.ClientCnxn:
> > > > >> >> Got
> > > > >> >> >> > ping
> > > > >> >> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 186ms
> > > > >> >> >> > >> >> 2009-11-14 15:59:46,070 DEBUG
> > > > >> org.apache.zookeeper.ClientCnxn:
> > > > >> >> Got
> > > > >> >> >> > ping
> > > > >> >> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
> > > > >> >> >> > >> >> 2009-11-14 15:59:59,423 DEBUG
> > > > >> org.apache.zookeeper.ClientCnxn:
> > > > >> >> Got
> > > > >> >> >> > ping
> > > > >> >> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
> > > > >> >> >> > >> >> 2009-11-14 16:00:12,775 DEBUG
> > > > >> org.apache.zookeeper.ClientCnxn:
> > > > >> >> Got
> > > > >> >> >> > ping
> > > > >> >> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
> > > > >> >> >> > >> >> 2009-11-14 16:00:28,141 DEBUG
> > > > >> org.apache.zookeeper.ClientCnxn:
> > > > >> >> Got
> > > > >> >> >> > ping
> > > > >> >> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after
> 2010ms
> > > > >> >> >> > >> >> 2009-11-14 16:01:10,378 WARN
> > > > >> org.apache.zookeeper.ClientCnxn:
> > > > >> >> >> > Exception
> > > > >> >> >> > >> >> closing session 0x424f0ed01ea00a6 to
> > > > >> >> >> > >> sun.nio.ch.SelectionKeyImpl@39fbb2d6
> > > > >> >> >> > >> >>
> > > > >> >> >> > >> >> java.io.IOException: Read error rc = -1
> > > > >> >> >> > java.nio.DirectByteBuffer[pos=0
> > > > >> >> >> > >> >> lim=4 cap=4]
> > > > >> >> >> > >> >> at
> > > > >> >> >> > >> >>
> > > > >> >> >>
> > > > org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
> > > > >> >> >> > >> >> at
> > > > >> >> >> > >> >>
> > > > >> >> >>
> > > > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
> > > > >> >> >> > >> >> 2009-11-14 16:01:10,478 INFO
> > > > >> >> >> > >> >> org.apache.hadoop.hbase.regionserver.HRegionServer:
> > Got
> > > > >> >> ZooKeeper
> > > > >> >> >> > >> event,
> > > > >> >> >> > >> >> state: Disconnected, type: None, path: null
> > > > >> >> >> > >> >> 2009-11-14 16:01:11,333 WARN
> > > > >> >> >> > >> >> org.apache.hadoop.hbase.regionserver.HRegionServer:
> > > > >> Attempt=1
> > > > >> >> >> > >> >>
> > > > >> >> >> > >> >>
> org.apache.hadoop.hbase.Leases$LeaseStillHeldException
> > > > >> >> >> > >> >> at
> > > > >> >> >> > >>
> > > sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> > > > >> >> >> > >> >> Method)
> > > > >> >> >> > >> >> at
> > > > >> >> >> > >> >>
> > > > >> >> >> > >>
> > > > >> >> >> >
> > > > >> >> >>
> > > > >> >>
> > > > >>
> > > >
> > >
> >
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> > > > >> >> >> > >> >> at
> > > > >> >> >> > >> >>
> > > > >> >> >> > >>
> > > > >> >> >> >
> > > > >> >> >>
> > > > >> >>
> > > > >>
> > > >
> > >
> >
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> > > > >> >> >> > >> >> at
> > > > >> >> >> > >>
> > > > java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> > > > >> >> >> > >> >> at
> > > > >> >> >> > >> >>
> > > > >> >> >> > >>
> > > > >> >> >> >
> > > > >> >> >>
> > > > >> >>
> > > > >>
> > > >
> > >
> >
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
> > > > >> >> >> > >> >> at
> > > > >> >> >> > >> >>
> > > > >> >> >> > >>
> > > > >> >> >> >
> > > > >> >> >>
> > > > >> >>
> > > > >>
> > > >
> > >
> >
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
> > > > >> >> >> > >> >> at
> > > > >> >> >> > >> >>
> > > > >> >> >> > >>
> > > > >> >> >> >
> > > > >> >> >>
> > > > >> >>
> > > > >>
> > > >
> > >
> >
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
> > > > >> >> >> > >> >> at
> > > > >> >> >> > >> >>
> > > > >> >> >> > >>
> > > > >> >> >> >
> > > > >> >> >>
> > > > >> >>
> > > > >>
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:571)
> > > > >> >> >> > >> >> at java.lang.Thread.run(Thread.java:619)
> > > > >> >> >> > >> >> 2009-11-14 16:01:11,433 INFO
> > > > >> org.apache.zookeeper.ClientCnxn:
> > > > >> >> >> > >> Attempting
> > > > >> >> >> > >> >> connection to server superpyxis0001/
> > > 192.168.100.115:2181
> > > > >> >> >> > >> >> 2009-11-14 16:01:11,433 INFO
> > > > >> org.apache.zookeeper.ClientCnxn:
> > > > >> >> >> Priming
> > > > >> >> >> > >> >> connection to
> > java.nio.channels.SocketChannel[connected
> > > > >> local=/
> > > > >> >> >> > >> >> 192.168.100.122:59575 remote=superpyxis0001/
> > > > >> >> 192.168.100.115:2181]
> > > > >> >> >> > >> >> 2009-11-14 16:01:11,433 INFO
> > > > >> org.apache.zookeeper.ClientCnxn:
> > > > >> >> >> Server
> > > > >> >> >> > >> >> connection successful
> > > > >> >> >> > >> >> 2009-11-14 16:01:11,433 INFO
> > > > >> >> >> > >> >> org.apache.hadoop.hbase.regionserver.HRegionServer:
> > Got
> > > > >> >> ZooKeeper
> > > > >> >> >> > >> event,
> > > > >> >> >> > >> >> state: Expired, type: None, path: null
> > > > >> >> >> > >> >> 2009-11-14 16:01:11,433 WARN
> > > > >> org.apache.zookeeper.ClientCnxn:
> > > > >> >> >> > Exception
> > > > >> >> >> > >> >> closing session 0x424f0ed01ea00a6 to
> > > > >> >> >> > >> sun.nio.ch.SelectionKeyImpl@6b93d343
> > > > >> >> >> > >> >> java.io.IOException: Session Expired
> > > > >> >> >> > >> >> at
> > > > >> >> >> > >> >>
> > > > >> >> >> > >>
> > > > >> >> >> >
> > > > >> >> >>
> > > > >> >>
> > > > >>
> > > >
> > >
> >
> org.apache.zookeeper.ClientCnxn$SendThread.readConnectResult(ClientCnxn.java:589)
> > > > >> >> >> > >> >> at
> > > > >> >> >> > >> >>
> > > > >> >> >>
> > > > org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:709)
> > > > >> >> >> > >> >>
> > > > >> >> >> > >> >> at
> > > > >> >> >> > >> >>
> > > > >> >> >>
> > > > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
> > > > >> >> >> > >> >> 2009-11-14 16:01:11,433 ERROR
> > > > >> >> >> > >> >> org.apache.hadoop.hbase.regionserver.HRegionServer:
> > > > >> ZooKeeper
> > > > >> >> >> session
> > > > >> >> >> > >> >> expired
> > > > >> >> >> > >> >> 2009-11-14 16:01:13,513 INFO
> > > > >> org.apache.hadoop.ipc.HBaseServer:
> > > > >> >> >> IPC
> > > > >> >> >> > >> Server
> > > > >> >> >> > >> >> handler 3 on 60021, call put([B@73deb204,
> > > > >> >> >> > >> >> [Lorg.apache.hadoop.hbase.client.Put;@2179600a)
> from
> > > > >> >> >> > >> 192.168.100.132:40728:
> > > > >> >> >> > >> >> error: java.io.IOException: Server not running,
> > aborting
> > > > >> >> >> > >> >> java.io.IOException: Server not running, aborting
> > > > >> >> >> > >> >>
> > > > >> >> >> > >> >>
> > > > >> >> >> > >> >>
> > > > >> >> >> > >> >>
> > > > >> >> >> > >> >>
> > > > >> >> >> > >> >> On Sat, Nov 14, 2009 at 1:01 AM, Zhenyu Zhong <
> > > > >> >> >> > zhongresearch@gmail.com
> > > > >> >> >> > >> >wrote:
> > > > >> >> >> > >> >>
> > > > >> >> >> > >> >>> Stack,
> > > > >> >> >> > >> >>>
> > > > >> >> >> > >> >>> Thanks a lot!
> > > > >> >> >> > >> >>>
> > > > >> >> >> > >> >>> I found out that the reason why HBase doesn't take
> > the
> > > > >> system
> > > > >> >> >> file
> > > > >> >> >> > >> >>> descriptor value I set before. I started the HBase
> > > using
> > > > >> root
> > > > >> >> >> > instead
> > > > >> >> >> > >> of the
> > > > >> >> >> > >> >>> normal hadoop user, while my system configures
> hadoop
> > > > with
> > > > >> >> higher
> > > > >> >> >> > file
> > > > >> >> >> > >> >>> descriptor value but configures root with 1024
> > default
> > > > >> value.
> > > > >> >> >> > >> >>>
> > > > >> >> >> > >> >>>
> > > > >> >> >> > >> >>> Now my system has a clean start. However, it seems
> > that
> > > > the
> > > > >> >> >> > >> >>> zookeeper.session.timeout value doesn't take into
> > > effect.
> > > > I
> > > > >> >> still
> > > > >> >> >> > >> found
> > > > >> >> >> > >> >>> around 60 seconds pauses from the disconnected
> > > > >> regionserver. I
> > > > >> >> >> > really
> > > > >> >> >> > >> don't
> > > > >> >> >> > >> >>> know why regionserver only times out after 60
> seconds
> > > > >> instead
> > > > >> >> of
> > > > >> >> >> 10
> > > > >> >> >> > >> minutes
> > > > >> >> >> > >> >>> which I set for zookeeper.session.timeout.
> > > > >> >> >> > >> >>>
> > > > >> >> >> > >> >>> Is there any other timeout value coming into play
> > > before
> > > > >> the
> > > > >> >> >> actual
> > > > >> >> >> > >> >>> session times out?
> > > > >> >> >> > >> >>>
> > > > >> >> >> > >> >>> zhenyu
> > > > >> >> >> > >> >>>
> > > > >> >> >> > >> >>>
> > > > >> >> >> > >> >>>
> > > > >> >> >> > >> >>> On Fri, Nov 13, 2009 at 7:08 PM, stack <
> > > stack@duboce.net
> > > > >
> > > > >> >> wrote:
> > > > >> >> >> > >> >>>
> > > > >> >> >> > >> >>>> Ok. Lack of file descriptors manifests in all
> kinds
> > > of
> > > > >> weird
> > > > >> >> >> ways.
> > > > >> >> >> > >> >>>> Hopefully thats it. If not, lets keep digging.
> > > > >> >> >> > >> >>>> St.Ack
> > > > >> >> >> > >> >>>>
> > > > >> >> >> > >> >>>> On Fri, Nov 13, 2009 at 3:44 PM, Zhenyu Zhong <
> > > > >> >> >> > >> zhongresearch@gmail.com
> > > > >> >> >> > >> >>>> >wrote:
> > > > >> >> >> > >> >>>>
> > > > >> >> >> > >> >>>> > Stack,
> > > > >> >> >> > >> >>>> >
> > > > >> >> >> > >> >>>> > You are right, the master started with ulimit -n
> > > 1024.
> > > > >> It
> > > > >> >> >> doesn't
> > > > >> >> >> > >> take
> > > > >> >> >> > >> >>>> the
> > > > >> >> >> > >> >>>> > system value.
> > > > >> >> >> > >> >>>> >
> > > > >> >> >> > >> >>>> > Regarding to the too many open files, it looks
> > like
> > > > the
> > > > >> >> same
> > > > >> >> >> as
> > > > >> >> >> > the
> > > > >> >> >> > >> one
> > > > >> >> >> > >> >>>> J-D
> > > > >> >> >> > >> >>>> > put up. But I will get the Master start with
> > higher
> > > > >> value
> > > > >> >> >> first
> > > > >> >> >> > and
> > > > >> >> >> > >> see
> > > > >> >> >> > >> >>>> if
> > > > >> >> >> > >> >>>> > this kind of error goes away.
> > > > >> >> >> > >> >>>> >
> > > > >> >> >> > >> >>>> > thanks a lot!
> > > > >> >> >> > >> >>>> > zhenyu
> > > > >> >> >> > >> >>>> >
> > > > >> >> >> > >> >>>> > On Fri, Nov 13, 2009 at 6:02 PM, stack <
> > > > >> stack@duboce.net>
> > > > >> >> >> wrote:
> > > > >> >> >> > >> >>>> >
> > > > >> >> >> > >> >>>> > > Does it say
> > > > >> >> >> > >> >>>> > >
> > > > >> >> >> > >> >>>> > > ulimit -n 32768
> > > > >> >> >> > >> >>>> > >
> > > > >> >> >> > >> >>>> > > ...as the second line in your log file on
> start
> > of
> > > > the
> > > > >> >> >> master?
> > > > >> >> >> > >> >>>> > >
> > > > >> >> >> > >> >>>> > > Is the stack trace that complains about too
> many
> > > > open
> > > > >> >> files
> > > > >> >> >> > same
> > > > >> >> >> > >> as
> > > > >> >> >> > >> >>>> the
> > > > >> >> >> > >> >>>> > one
> > > > >> >> >> > >> >>>> > > in the blog post J-D put up?
> > > > >> >> >> > >> >>>> > >
> > > > >> >> >> > >> >>>> > > St.Ack
> > > > >> >> >> > >> >>>> > >
> > > > >> >> >> > >> >>>> > >
> > > > >> >> >> > >> >>>> > > On Fri, Nov 13, 2009 at 1:37 PM, Zhenyu Zhong
> <
> > > > >> >> >> > >> >>>> zhongresearch@gmail.com
> > > > >> >> >> > >> >>>> > > >wrote:
> > > > >> >> >> > >> >>>> > >
> > > > >> >> >> > >> >>>> > > > The ulimit file descriptors was set to
> > > fs.file-max
> > > > >> >> >> =1578334,
> > > > >> >> >> > >> also
> > > > >> >> >> > >> >>>> in
> > > > >> >> >> > >> >>>> > the
> > > > >> >> >> > >> >>>> > > > limits.conf the value was set to 32768.
> > > > >> >> >> > >> >>>> > > > So these are way higher than the open
> > > descriptors
> > > > >> for
> > > > >> >> the
> > > > >> >> >> > >> running
> > > > >> >> >> > >> >>>> > > > processes.
> > > > >> >> >> > >> >>>> > > >
> > > > >> >> >> > >> >>>> > > > thanks
> > > > >> >> >> > >> >>>> > > > zhenyu
> > > > >> >> >> > >> >>>> > > >
> > > > >> >> >> > >> >>>> > > >
> > > > >> >> >> > >> >>>> > > > On Fri, Nov 13, 2009 at 4:33 PM, Stack <
> > > > >> >> >> saint.ack@gmail.com>
> > > > >> >> >> > >> >>>> wrote:
> > > > >> >> >> > >> >>>> > > >
> > > > >> >> >> > >> >>>> > > > > You upped the ulimit file descriptors as
> per
> > > the
> > > > >> >> getting
> > > > >> >> >> > >> started
> > > > >> >> >> > >> >>>> doc?
> > > > >> >> >> > >> >>>> > > > >
> > > > >> >> >> > >> >>>> > > > >
> > > > >> >> >> > >> >>>> > > > >
> > > > >> >> >> > >> >>>> > > > > On Nov 13, 2009, at 1:26 PM, Zhenyu Zhong
> <
> > > > >> >> >> > >> >>>> zhongresearch@gmail.com>
> > > > >> >> >> > >> >>>> > > > wrote:
> > > > >> >> >> > >> >>>> > > > >
> > > > >> >> >> > >> >>>> > > > > Thanks a lot.
> > > > >> >> >> > >> >>>> > > > >>
> > > > >> >> >> > >> >>>> > > > >>
> > > > >> >> >> > >> >>>> > > > >> Bad news is my kernel is still 2.6.26.
> > > > >> >> >> > >> >>>> > > > >> But it was not a problem before.
> > > > >> >> >> > >> >>>> > > > >>
> > > > >> >> >> > >> >>>> > > > >> Very strange.
> > > > >> >> >> > >> >>>> > > > >>
> > > > >> >> >> > >> >>>> > > > >> zhenyu
> > > > >> >> >> > >> >>>> > > > >>
> > > > >> >> >> > >> >>>> > > > >> On Fri, Nov 13, 2009 at 4:16 PM,
> > Jean-Daniel
> > > > >> Cryans
> > > > >> >> <
> > > > >> >> >> > >> >>>> > > > jdcryans@apache.org
> > > > >> >> >> > >> >>>> > > > >> >wrote:
> > > > >> >> >> > >> >>>> > > > >>
> > > > >> >> >> > >> >>>> > > > >> Looks like
> > > > >> >> >> > >> >>>> > > > >>>
> > > > >> >> >> > >> >>>> > > > >>>
> > > > >> >> >> > >> >>>> > > >
> > > > >> >> >> > >> >>>> > >
> > > > >> >> >> > >> >>>> >
> > > > >> >> >> > >> >>>>
> > > > >> >> >> > >>
> > > > >> >> >> >
> > > > >> >> >>
> > > > >> >>
> > > > >>
> > > >
> > >
> >
> http://pero.blogs.aprilmayjune.org/2009/01/22/hadoop-and-linux-kernel-2627-epoll-limits/
> > > > >> >> >> > >> >>>> > > > >>>
> > > > >> >> >> > >> >>>> > > > >>> J-D
> > > > >> >> >> > >> >>>> > > > >>>
> > > > >> >> >> > >> >>>> > > > >>> On Fri, Nov 13, 2009 at 1:12 PM, Zhenyu
> > > Zhong
> > > > <
> > > > >> >> >> > >> >>>> > > zhongresearch@gmail.com
> > > > >> >> >> > >> >>>> > > > >
> > > > >> >> >> > >> >>>> > > > >>> wrote:
> > > > >> >> >> > >> >>>> > > > >>>
> > > > >> >> >> > >> >>>> > > > >>>> Hi,
> > > > >> >> >> > >> >>>> > > > >>>>
> > > > >> >> >> > >> >>>> > > > >>>>
> > > > >> >> >> > >> >>>> > > > >>>> After I re-organize the cluster, the
> > > > experiment
> > > > >> >> ran
> > > > >> >> >> into
> > > > >> >> >> > >> >>>> problem
> > > > >> >> >> > >> >>>> > > > faster
> > > > >> >> >> > >> >>>> > > > >>>>
> > > > >> >> >> > >> >>>> > > > >>> than
> > > > >> >> >> > >> >>>> > > > >>>
> > > > >> >> >> > >> >>>> > > > >>>> before.
> > > > >> >> >> > >> >>>> > > > >>>>
> > > > >> >> >> > >> >>>> > > > >>>> Basically the changes are to use
> machines
> > > > with
> > > > >> >> less
> > > > >> >> >> > >> resources
> > > > >> >> >> > >> >>>> as
> > > > >> >> >> > >> >>>> > > > >>>>
> > > > >> >> >> > >> >>>> > > > >>> zookeeper
> > > > >> >> >> > >> >>>> > > > >>>
> > > > >> >> >> > >> >>>> > > > >>>> quorums and machines with more
> resources
> > as
> > > > >> >> >> > regionserver.
> > > > >> >> >> > >> >>>> > > > >>>>
> > > > >> >> >> > >> >>>> > > > >>>> From the log, I still see the pause
> > around
> > > 1
> > > > >> >> minute.
> > > > >> >> >> > >> >>>> > > > >>>> I enabled the GC logging, please see
> > > > >> >> >> > >> >>>> > > > >>>>
> > > > >> >> >> > >> >>>> > > > >>>> http://pastebin.com/m1d4ce0f1
> > > > >> >> >> > >> >>>> > > > >>>>
> > > > >> >> >> > >> >>>> > > > >>>> for details.
> > > > >> >> >> > >> >>>> > > > >>>> In general I don't see many pauses in
> the
> > > GC.
> > > > >> >> >> > >> >>>> > > > >>>>
> > > > >> >> >> > >> >>>> > > > >>>> What is more interesting, 1 hour after
> > the
> > > > 1st
> > > > >> >> >> > >> regionserver
> > > > >> >> >> > >> >>>> > > > >>>> disconnected,
> > > > >> >> >> > >> >>>> > > > >>>> the master log complained about too
> many
> > > open
> > > > >> >> files.
> > > > >> >> >> > This
> > > > >> >> >> > >> >>>> didn't
> > > > >> >> >> > >> >>>> > > > happen
> > > > >> >> >> > >> >>>> > > > >>>> before.
> > > > >> >> >> > >> >>>> > > > >>>> I checked the system OS setup as well
> as
> > > the
> > > > >> >> >> > limits.conf.
> > > > >> >> >> > >> I
> > > > >> >> >> > >> >>>> also
> > > > >> >> >> > >> >>>> > > > checked
> > > > >> >> >> > >> >>>> > > > >>>>
> > > > >> >> >> > >> >>>> > > > >>> the
> > > > >> >> >> > >> >>>> > > > >>>
> > > > >> >> >> > >> >>>> > > > >>>> running processes. The total open files
> > > don't
> > > > >> >> reach
> > > > >> >> >> the
> > > > >> >> >> > >> limit.
> > > > >> >> >> > >> >>>> I
> > > > >> >> >> > >> >>>> > am
> > > > >> >> >> > >> >>>> > > > >>>>
> > > > >> >> >> > >> >>>> > > > >>> confused
> > > > >> >> >> > >> >>>> > > > >>>
> > > > >> >> >> > >> >>>> > > > >>>> a bit.
> > > > >> >> >> > >> >>>> > > > >>>>
> > > > >> >> >> > >> >>>> > > > >>>>
> > > > >> >> >> > >> >>>> > > > >>>> Please see the following master log.
> > > > >> >> >> > >> >>>> > > > >>>>
> > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:02,114 INFO
> > > > >> >> >> > >> >>>> > > > org.apache.hadoop.hbase.master.BaseScanner:
> > > > >> >> >> > >> >>>> > > > >>>> RegionManager.metaScanner scan of 4658
> > > row(s)
> > > > >> of
> > > > >> >> meta
> > > > >> >> >> > >> region
> > > > >> >> >> > >> >>>> > > {server:
> > > > >> >> >> > >> >>>> > > > >>>> 192.168.100.128:60021, regionname:
> > > > .META.,,1,
> > > > >> >> >> startKey:
> > > > >> >> >> > >> <>}
> > > > >> >> >> > >> >>>> > > complete
> > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:02,114 INFO
> > > > >> >> >> > >> >>>> > > > org.apache.hadoop.hbase.master.BaseScanner:
> > > > >> >> >> > >> >>>> > > > >>>>
> > > > >> >> >> > >> >>>> > > > >>> All
> > > > >> >> >> > >> >>>> > > > >>>
> > > > >> >> >> > >> >>>> > > > >>>> 1 .META. region(s) scanned
> > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:07,677 DEBUG
> > > > >> >> >> > >> org.apache.zookeeper.ClientCnxn:
> > > > >> >> >> > >> >>>> Got
> > > > >> >> >> > >> >>>> > > > ping
> > > > >> >> >> > >> >>>> > > > >>>> response for
> sessionid:0x424eebf1c10004c
> > > > after
> > > > >> 3ms
> > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:08,178 INFO
> > > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > > >> >> >> > >> >>>> > > > Exception
> > > > >> >> >> > >> >>>> > > > >>>>
> > > > >> >> >> > >> >>>> > > > >>> in
> > > > >> >> >> > >> >>>> > > > >>>
> > > > >> >> >> > >> >>>> > > > >>>> createBlockOutputStream
> > > java.io.IOException:
> > > > >> Bad
> > > > >> >> >> connect
> > > > >> >> >> > >> ack
> > > > >> >> >> > >> >>>> with
> > > > >> >> >> > >> >>>> > > > >>>> firstBadLink 192.168.100.123:50010
> > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:08,178 INFO
> > > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > > >> >> >> > >> >>>> > > > >>>> Abandoning
> > > > >> >> >> > >> >>>> > > > >>>> block blk_-2808245019291145247_5478039
> > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:09,682 INFO
> > > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > > >> >> >> > >> >>>> > > > Exception
> > > > >> >> >> > >> >>>> > > > >>>>
> > > > >> >> >> > >> >>>> > > > >>> in
> > > > >> >> >> > >> >>>> > > > >>>
> > > > >> >> >> > >> >>>> > > > >>>> createBlockOutputStream
> > > java.io.EOFException
> > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:09,682 INFO
> > > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > > >> >> >> > >> >>>> > > > >>>> Abandoning
> > > > >> >> >> > >> >>>> > > > >>>> block blk_1074853606841896259_5478048
> > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:10,334 DEBUG
> > > > >> >> >> > >> org.apache.zookeeper.ClientCnxn:
> > > > >> >> >> > >> >>>> Got
> > > > >> >> >> > >> >>>> > > > ping
> > > > >> >> >> > >> >>>> > > > >>>> response for sessionid:0x24eebf1043003c
> > > after
> > > > >> 1ms
> > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:21,018 DEBUG
> > > > >> >> >> > >> org.apache.zookeeper.ClientCnxn:
> > > > >> >> >> > >> >>>> Got
> > > > >> >> >> > >> >>>> > > > ping
> > > > >> >> >> > >> >>>> > > > >>>> response for
> sessionid:0x424eebf1c10004c
> > > > after
> > > > >> 0ms
> > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:23,674 DEBUG
> > > > >> >> >> > >> org.apache.zookeeper.ClientCnxn:
> > > > >> >> >> > >> >>>> Got
> > > > >> >> >> > >> >>>> > > > ping
> > > > >> >> >> > >> >>>> > > > >>>> response for sessionid:0x24eebf1043003c
> > > after
> > > > >> 0ms
> > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:24,828 INFO
> > > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > > >> >> >> > >> >>>> > > > Exception
> > > > >> >> >> > >> >>>> > > > >>>>
> > > > >> >> >> > >> >>>> > > > >>> in
> > > > >> >> >> > >> >>>> > > > >>>
> > > > >> >> >> > >> >>>> > > > >>>> createBlockOutputStream
> > > java.io.IOException:
> > > > >> Bad
> > > > >> >> >> connect
> > > > >> >> >> > >> ack
> > > > >> >> >> > >> >>>> with
> > > > >> >> >> > >> >>>> > > > >>>> firstBadLink 192.168.100.123:50010
> > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:24,828 INFO
> > > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > > >> >> >> > >> >>>> > > > >>>> Abandoning
> > > > >> >> >> > >> >>>> > > > >>>> block blk_-6642544517082142289_5478063
> > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:24,828 INFO
> > > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > > >> >> >> > >> >>>> > > > Exception
> > > > >> >> >> > >> >>>> > > > >>>>
> > > > >> >> >> > >> >>>> > > > >>> in
> > > > >> >> >> > >> >>>> > > > >>>
> > > > >> >> >> > >> >>>> > > > >>>> createBlockOutputStream
> > > > >> java.net.SocketException:
> > > > >> >> Too
> > > > >> >> >> > many
> > > > >> >> >> > >> >>>> open
> > > > >> >> >> > >> >>>> > > files
> > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:24,828 INFO
> > > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > > >> >> >> > >> >>>> > > > >>>> Abandoning
> > > > >> >> >> > >> >>>> > > > >>>> block blk_2057511041109796090_5478063
> > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:24,928 INFO
> > > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > > >> >> >> > >> >>>> > > > Exception
> > > > >> >> >> > >> >>>> > > > >>>>
> > > > >> >> >> > >> >>>> > > > >>> in
> > > > >> >> >> > >> >>>> > > > >>>
> > > > >> >> >> > >> >>>> > > > >>>> createBlockOutputStream
> > > > >> java.net.SocketException:
> > > > >> >> Too
> > > > >> >> >> > many
> > > > >> >> >> > >> >>>> open
> > > > >> >> >> > >> >>>> > > files
> > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:24,928 INFO
> > > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > > >> >> >> > >> >>>> > > > >>>> Abandoning
> > > > >> >> >> > >> >>>> > > > >>>> block blk_8219260302213892894_5478064
> > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:30,855 INFO
> > > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > > >> >> >> > >> >>>> > > > Exception
> > > > >> >> >> > >> >>>> > > > >>>>
> > > > >> >> >> > >> >>>> > > > >>> in
> > > > >> >> >> > >> >>>> > > > >>>
> > > > >> >> >> > >> >>>> > > > >>>> createBlockOutputStream
> > > > >> java.net.SocketException:
> > > > >> >> Too
> > > > >> >> >> > many
> > > > >> >> >> > >> >>>> open
> > > > >> >> >> > >> >>>> > > files
> > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:30,855 INFO
> > > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > > >> >> >> > >> >>>> > > > >>>> Abandoning
> > > > >> >> >> > >> >>>> > > > >>>> block blk_1669205542853067709_5478235
> > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:30,905 INFO
> > > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > > >> >> >> > >> >>>> > > > Exception
> > > > >> >> >> > >> >>>> > > > >>>>
> > > > >> >> >> > >> >>>> > > > >>> in
> > > > >> >> >> > >> >>>> > > > >>>
> > > > >> >> >> > >> >>>> > > > >>>> createBlockOutputStream
> > > > >> java.net.SocketException:
> > > > >> >> Too
> > > > >> >> >> > many
> > > > >> >> >> > >> >>>> open
> > > > >> >> >> > >> >>>> > > files
> > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:30,905 INFO
> > > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > > >> >> >> > >> >>>> > > > >>>> Abandoning
> > > > >> >> >> > >> >>>> > > > >>>> block blk_9128897691346270351_5478237
> > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:30,955 INFO
> > > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > > >> >> >> > >> >>>> > > > Exception
> > > > >> >> >> > >> >>>> > > > >>>>
> > > > >> >> >> > >> >>>> > > > >>> in
> > > > >> >> >> > >> >>>> > > > >>>
> > > > >> >> >> > >> >>>> > > > >>>> createBlockOutputStream
> > > > >> java.net.SocketException:
> > > > >> >> Too
> > > > >> >> >> > many
> > > > >> >> >> > >> >>>> open
> > > > >> >> >> > >> >>>> > > files
> > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:30,955 INFO
> > > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > > >> >> >> > >> >>>> > > > >>>> Abandoning
> > > > >> >> >> > >> >>>> > > > >>>> block blk_1116845144864123018_5478240
> > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:34,372 DEBUG
> > > > >> >> >> > >> org.apache.zookeeper.ClientCnxn:
> > > > >> >> >> > >> >>>> Got
> > > > >> >> >> > >> >>>> > > > ping
> > > > >> >> >> > >> >>>> > > > >>>> response for
> sessionid:0x424eebf1c10004c
> > > > after
> > > > >> 0ms
> > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:37,034 DEBUG
> > > > >> >> >> > >> org.apache.zookeeper.ClientCnxn:
> > > > >> >> >> > >> >>>> Got
> > > > >> >> >> > >> >>>> > > > ping
> > > > >> >> >> > >> >>>> > > > >>>> response for sessionid:0x24eebf1043003c
> > > after
> > > > >> 0ms
> > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:37,235 WARN
> > > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > > >> >> >> > >> >>>> > > > >>>>
> > > > >> >> >> > >> >>>> > > > >>> DataStreamer
> > > > >> >> >> > >> >>>> > > > >>>
> > > > >> >> >> > >> >>>> > > > >>>> Exception: java.io.IOException: Too
> many
> > > open
> > > > >> >> files
> > > > >> >> >> > >> >>>> > > > >>>> at
> sun.nio.ch.IOUtil.initPipe(Native
> > > > >> Method)
> > > > >> >> >> > >> >>>> > > > >>>> at
> > > > >> >> >> > >> >>>> > >
> > > > >> >> >>
> sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:49)
> > > > >> >> >> > >> >>>> > > > >>>> at
> > > > >> >> >> > >> >>>> > > > >>>>
> > > > >> >> >> > >> >>>> > > > >>>>
> > > > >> >> >> > >> >>>> > > >
> > > > >> >> >> > >> >>>> > >
> > > > >> >> >> > >> >>>> >
> > > > >> >> >> > >> >>>>
> > > > >> >> >> > >>
> > > > >> >> >> >
> > > > >> >> >>
> > > > >> >>
> > > > >>
> > > >
> > >
> >
> sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:18)
> > > > >> >> >> > >> >>>> > > > >>>
> > > > >> >> >> > >> >>>> > > > >>>> at
> > > > >> >> >> > >> >>>> > > > >>>>
> > > > >> >> >> > >> >>>> > > > >>>>
> > > > >> >> >> > >> >>>> > > >
> > > > >> >> >> > >> >>>> > >
> > > > >> >> >> > >> >>>> >
> > > > >> >> >> > >> >>>>
> > > > >> >> >> > >>
> > > > >> >> >> >
> > > > >> >> >>
> > > > >> >>
> > > > >>
> > > >
> > >
> >
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.get(SocketIOWithTimeout.java:407)
> > > > >> >> >> > >> >>>> > > > >>>
> > > > >> >> >> > >> >>>> > > > >>>> at
> > > > >> >> >> > >> >>>> > > > >>>>
> > > > >> >> >> > >> >>>> > > > >>>>
> > > > >> >> >> > >> >>>> > > >
> > > > >> >> >> > >> >>>> > >
> > > > >> >> >> > >> >>>> >
> > > > >> >> >> > >> >>>>
> > > > >> >> >> > >>
> > > > >> >> >> >
> > > > >> >> >>
> > > > >> >>
> > > > >>
> > > >
> > >
> >
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:322)
> > > > >> >> >> > >> >>>> > > > >>>
> > > > >> >> >> > >> >>>> > > > >>>> at
> > > > >> >> >> > >> >>>> > > > >>>>
> > > > >> >> >> > >> >>>> > > > >>>>
> > > > >> >> >> > >> >>>> > > >
> > > > >> >> >> > >> >>>> > >
> > > > >> >> >> > >> >>>> >
> > > > >> >> >> > >> >>>>
> > > > >> >> >> > >>
> > > > >> >> >> >
> > > > >> >> >>
> > > > >> >>
> > > > >>
> > > >
> > >
> >
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
> > > > >> >> >> > >> >>>> > > > >>>
> > > > >> >> >> > >> >>>> > > > >>>> at
> > > > >> >> >> > >> >>>> > > > >>>>
> > > > >> >> >> > >> >>>> > > > >>>>
> > > > >> >> >> > >> >>>> > > >
> > > > >> >> >> > >> >>>> > >
> > > > >> >> >> > >> >>>> >
> > > > >> >> >> > >> >>>>
> > > > >> >> >> > >>
> > > > >> >> >> >
> > > > >> >> >>
> > > > >> >>
> > > > >>
> > > >
> > >
> >
> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
> > > > >> >> >> > >> >>>> > > > >>>
> > > > >> >> >> > >> >>>> > > > >>>> at
> > > > >> >> >> > >> >>>> > > > >>>>
> > > > >> >> >> > >> >>>> > > > >>>>
> > > > >> >> >> > >> >>>> > > >
> > > > >> >> >> > >> >>>> > >
> > > > >> >> >> > >> >>>> >
> > > > >> >> >> > >> >>>>
> > > > >> >> >> > >>
> > > > >> >> >> >
> > > > >> >> >>
> > > > >> >>
> > > > >>
> > > >
> > >
> >
> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
> > > > >> >> >> > >> >>>> > > > >>>
> > > > >> >> >> > >> >>>> > > > >>>> at
> > > > >> >> >> > >> >>>> > > > >>>>
> > > > >> >> >> > >> >>>> > > > >>>
> > > > >> >> >> > >> >>>>
> > > > >> >> >>
> > > java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
> > > > >> >> >> > >> >>>> > > > >>>
> > > > >> >> >> > >> >>>> > > > >>>> at
> > > > >> >> >> > >> >>>>
> > > java.io.DataOutputStream.write(DataOutputStream.java:90)
> > > > >> >> >> > >> >>>> > > > >>>> at
> > > > >> >> >> > >> >>>> > > > >>>>
> > > > >> >> >> > >> >>>> > > > >>>>
> > > > >> >> >> > >> >>>> > > >
> > > > >> >> >> > >> >>>> > >
> > > > >> >> >> > >> >>>> >
> > > > >> >> >> > >> >>>>
> > > > >> >> >> > >>
> > > > >> >> >> >
> > > > >> >> >>
> > > > >> >>
> > > > >>
> > > >
> > >
> >
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2290)
> > > > >> >> >> > >> >>>> > > > >>>
> > > > >> >> >> > >> >>>> > > > >>>>
> > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:37,235 WARN
> > > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > > >> >> >> > >> >>>> > Error
> > > > >> >> >> > >> >>>> > > > >>>> Recovery for block
> > > > >> blk_8148813491785406356_5478475
> > > > >> >> >> bad
> > > > >> >> >> > >> >>>> datanode[0]
> > > > >> >> >> > >> >>>> > > > >>>> 192.168.100.123:50010
> > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:37,235 WARN
> > > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > > >> >> >> > >> >>>> > Error
> > > > >> >> >> > >> >>>> > > > >>>> Recovery for block
> > > > >> blk_8148813491785406356_5478475
> > > > >> >> in
> > > > >> >> >> > >> pipeline
> > > > >> >> >> > >> >>>> > > > >>>> 192.168.100.123:50010,
> > > 192.168.100.134:50010
> > > > ,
> > > > >> >> >> > >> >>>> > 192.168.100.122:50010
> > > > >> >> >> > >> >>>> > > :
> > > > >> >> >> > >> >>>> > > > >>>> bad
> > > > >> >> >> > >> >>>> > > > >>>> datanode 192.168.100.123:50010
> > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:37,436 INFO
> > > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > > >> >> >> > >> >>>> > > > Exception
> > > > >> >> >> > >> >>>> > > > >>>>
> > > > >> >> >> > >> >>>> > > > >>> in
> > > > >> >> >> > >> >>>> > > > >>>
> > > > >> >> >> > >> >>>> > > > >>>> createBlockOutputStream
> > > > >> java.net.SocketException:
> > > > >> >> Too
> > > > >> >> >> > many
> > > > >> >> >> > >> >>>> open
> > > > >> >> >> > >> >>>> > > files
> > > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:37,436 INFO
> > > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > > >> >> >> > >> >>>> > > > >>>> Abandoning
> > > > >> >> >> > >> >>>> > > > >>>> block blk_2119727700857186236_5478498
> > > > >> >> >> > >> >>>> > > > >>>>
> > > > >> >> >> > >> >>>> > > > >>>>
> > > > >> >> >> > >> >>>> > > > >>>>
> > > > >> >> >> > >> >>>> > > > >>>>
> > > > >> >> >> > >> >>>> > > > >>>>
> > > > >> >> >> > >> >>>> > > > >>>> On Thu, Nov 12, 2009 at 4:21 PM, Zhenyu
> > > Zhong
> > > > <
> > > > >> >> >> > >> >>>> > > > zhongresearch@gmail.com
> > > > >> >> >> > >> >>>> > > > >>>> wrote:
> > > > >> >> >> > >> >>>> > > > >>>>
> > > > >> >> >> > >> >>>> > > > >>>> Will do.
> > > > >> >> >> > >> >>>> > > > >>>>>
> > > > >> >> >> > >> >>>> > > > >>>>> thanks
> > > > >> >> >> > >> >>>> > > > >>>>> zhenyu
> > > > >> >> >> > >> >>>> > > > >>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>
> > > > >> >> >> > >> >>>> > > > >>>>> On Thu, Nov 12, 2009 at 3:33 PM, stack
> <
> > > > >> >> >> > stack@duboce.net
> > > > >> >> >> > >> >
> > > > >> >> >> > >> >>>> wrote:
> > > > >> >> >> > >> >>>> > > > >>>>>
> > > > >> >> >> > >> >>>> > > > >>>>> Enable GC logging too on this next
> run
> > > (see
> > > > >> >> >> > >> hbase-env.sh).
> > > > >> >> >> > >> >>>> Lets
> > > > >> >> >> > >> >>>> > > try
> > > > >> >> >> > >> >>>> > > > >>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>> and
> > > > >> >> >> > >> >>>> > > > >>>
> > > > >> >> >> > >> >>>> > > > >>>> get
> > > > >> >> >> > >> >>>> > > > >>>>>> to the bottom of whats going on.
> > > > >> >> >> > >> >>>> > > > >>>>>> Thanks,
> > > > >> >> >> > >> >>>> > > > >>>>>> St.Ack
> > > > >> >> >> > >> >>>> > > > >>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>> On Thu, Nov 12, 2009 at 12:29 PM,
> > Zhenyu
> > > > >> Zhong <
> > > > >> >> >> > >> >>>> > > > >>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>> zhongresearch@gmail.com
> > > > >> >> >> > >> >>>> > > > >>>
> > > > >> >> >> > >> >>>> > > > >>>> wrote:
> > > > >> >> >> > >> >>>> > > > >>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>> I can switch the boxes that run
> > > zookeeper
> > > > >> with
> > > > >> >> the
> > > > >> >> >> > ones
> > > > >> >> >> > >> >>>> that
> > > > >> >> >> > >> >>>> > run
> > > > >> >> >> > >> >>>> > > > >>>>>>> regionservers.
> > > > >> >> >> > >> >>>> > > > >>>>>>> I will see the results later.
> > > > >> >> >> > >> >>>> > > > >>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>> FYI. The node does have the 10
> minutes
> > > > >> >> >> > >> >>>> > zookeeper.session.timeout
> > > > >> >> >> > >> >>>> > > > >>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>> value
> > > > >> >> >> > >> >>>> > > > >>>
> > > > >> >> >> > >> >>>> > > > >>>> in
> > > > >> >> >> > >> >>>> > > > >>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>> place.
> > > > >> >> >> > >> >>>> > > > >>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>> thanks
> > > > >> >> >> > >> >>>> > > > >>>>>>> zhenyu
> > > > >> >> >> > >> >>>> > > > >>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>> On Thu, Nov 12, 2009 at 3:21 PM,
> stack
> > <
> > > > >> >> >> > >> stack@duboce.net>
> > > > >> >> >> > >> >>>> > wrote:
> > > > >> >> >> > >> >>>> > > > >>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>> On Thu, Nov 12, 2009 at 11:50 AM,
> > > Zhenyu
> > > > >> Zhong
> > > > >> >> <
> > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>> zhongresearch@gmail.com
> > > > >> >> >> > >> >>>> > > > >>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>> wrote:
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>> In my cluster, half of the cluster
> > > have
> > > > 2
> > > > >> >> disks
> > > > >> >> >> > 400GB
> > > > >> >> >> > >> >>>> each
> > > > >> >> >> > >> >>>> > per
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>> machine,
> > > > >> >> >> > >> >>>> > > > >>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>> and
> > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>> half of the cluster have 6 disks
> per
> > > > >> machine.
> > > > >> >> >> > Maybe
> > > > >> >> >> > >> we
> > > > >> >> >> > >> >>>> > should
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>> run
> > > > >> >> >> > >> >>>> > > > >>>
> > > > >> >> >> > >> >>>> > > > >>>> zookeeper
> > > > >> >> >> > >> >>>> > > > >>>>>>>>> on the machines with 2 disks and
> RS
> > on
> > > > >> >> machines
> > > > >> >> >> > with
> > > > >> >> >> > >> 6
> > > > >> >> >> > >> >>>> disks?
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>> That would make most sense only in
> > the
> > > > >> below,
> > > > >> >> it
> > > > >> >> >> > >> looks
> > > > >> >> >> > >> >>>> like
> > > > >> >> >> > >> >>>> > the
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>> RS
> > > > >> >> >> > >> >>>> > > > >>>
> > > > >> >> >> > >> >>>> > > > >>>> that
> > > > >> >> >> > >> >>>> > > > >>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>> had
> > > > >> >> >> > >> >>>> > > > >>>>>>>> issue had 4 disks?
> > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>> BTW, the 10 minutes
> > > > >> zookeeper.session.timeout
> > > > >> >> >> has
> > > > >> >> >> > >> been
> > > > >> >> >> > >> >>>> set
> > > > >> >> >> > >> >>>> > > during
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>> the
> > > > >> >> >> > >> >>>> > > > >>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>> experiment.
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>> And for sure this node had it in
> > place?
> > > > >> >> >> > >> >>>> > > > >>>>>>>> St.Ack
> > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>> thanks
> > > > >> >> >> > >> >>>> > > > >>>>>>>>> zhenyu
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>> On Thu, Nov 12, 2009 at 2:08 PM,
> > stack
> > > <
> > > > >> >> >> > >> stack@duboce.net
> > > > >> >> >> > >> >>>> >
> > > > >> >> >> > >> >>>> > > wrote:
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>> On Thu, Nov 12, 2009 at 8:40 AM,
> > > Zhenyu
> > > > >> >> Zhong <
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>> zhongresearch@gmail.com
> > > > >> >> >> > >> >>>> > > > >>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>> wrote:
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>> Though I experienced 2
> > regionserver
> > > > >> >> >> disconnection
> > > > >> >> >> > >> this
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>> morning,
> > > > >> >> >> > >> >>>> > > > >>>
> > > > >> >> >> > >> >>>> > > > >>>> it
> > > > >> >> >> > >> >>>> > > > >>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>> looks
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>> better from the regionserver log.
> > > > (please
> > > > >> >> see
> > > > >> >> >> the
> > > > >> >> >> > >> >>>> following
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>> log)
> > > > >> >> >> > >> >>>> > > > >>>
> > > > >> >> >> > >> >>>> > > > >>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> http://pastebin.com/m496dbfae
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> I checked diskIO for one of the
> > > > >> >> >> > >> >>>> > regionserver(192.168.100.116)
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>> during
> > > > >> >> >> > >> >>>> > > > >>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>> the
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>> time it disconnected.
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> Time: 03:04:58 AM
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> Device: tps
> > Blk_read/s
> > > > >> >> >> Blk_wrtn/s
> > > > >> >> >> > >> >>>> Blk_read
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>> Blk_wrtn
> > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>> sda 105.31
> 5458.83
> > > > >> >> 19503.64
> > > > >> >> >> > >> >>>> 9043873239
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>> 32312473676
> > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>> sda1 2.90
> 6.64
> > > > >> >> 99.25
> > > > >> >> >> > >> >>>> 10993934
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>> 164433464
> > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>> sda2 1.72
> 23.76
> > > > >> >> 30.25
> > > > >> >> >> > >> >>>> 39365817
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>> 50124008
> > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>> sda3 0.30
> 0.38
> > > > >> >> 3.58
> > > > >> >> >> > >> >>>> 624930
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>> 5923000
> > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>> sda4 100.39
> 5428.06
> > > > >> >> 19370.56
> > > > >> >> >> > >> >>>> 8992888270
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>> 32091993204
> > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> Is this high for you? 20k
> > > > >> blocks/second
> > > > >> >> >> would
> > > > >> >> >> > >> seem to
> > > > >> >> >> > >> >>>> be
> > > > >> >> >> > >> >>>> > > high
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>> but
> > > > >> >> >> > >> >>>> > > > >>>
> > > > >> >> >> > >> >>>> > > > >>>> its
> > > > >> >> >> > >> >>>> > > > >>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>> one
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>> disk only and its not being
> shared
> > by
> > > > zk
> > > > >> >> >> anymore
> > > > >> >> >> > so
> > > > >> >> >> > >> >>>> > shouldn't
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>> matter?
> > > > >> >> >> > >> >>>> > > > >>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>> I also checked the zookeeper
> > quorum
> > > > >> server
> > > > >> >> >> that
> > > > >> >> >> > the
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>> regionserver
> > > > >> >> >> > >> >>>> > > > >>>
> > > > >> >> >> > >> >>>> > > > >>>> tried
> > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>> to
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>> connect according to the log.
> > > However,
> > > > I
> > > > >> >> don't
> > > > >> >> >> see
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>> 192.168.100.116
> > > > >> >> >> > >> >>>> > > > >>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>> in
> > > > >> >> >> > >> >>>> > > > >>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>> the
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>> client list of the zookeeper
> quorum
> > > > >> server
> > > > >> >> that
> > > > >> >> >> > the
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>> regionserver
> > > > >> >> >> > >> >>>> > > > >>>
> > > > >> >> >> > >> >>>> > > > >>>> tried
> > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>> to
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>> connect.
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> Would that be a problem?
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> Is that because the ephemeral
> > node
> > > > for
> > > > >> the
> > > > >> >> >> > >> >>>> regionserver
> > > > >> >> >> > >> >>>> > had
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>> evaporated?
> > > > >> >> >> > >> >>>> > > > >>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>> Lost
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>> its lease w/ zk by the time you
> > went
> > > to
> > > > >> >> look?
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>> Thu Nov 12 15:04:52 UTC 2009
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> Zookeeper version: 3.2.1-808558,
> > > built
> > > > >> on
> > > > >> >> >> > >> 08/27/2009
> > > > >> >> >> > >> >>>> 18:48
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>> GMT
> > > > >> >> >> > >> >>>> > > > >>>
> > > > >> >> >> > >> >>>> > > > >>>> Clients:
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.127:43045
> > > > >> >> >> > >> [1](queued=0,recved=26,sent=0)
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.131:39091
> > > > >> >> >> > >> [1](queued=0,recved=964,sent=0)
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.124:35961
> > > > >> >> >> > >> [1](queued=0,recved=3266,sent=0)
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.123:47935
> > > > >> >> >> > >> [1](queued=0,recved=233,sent=0)
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.125:46931
> > > > >> >> >> > >> [1](queued=0,recved=2,sent=0)
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.118:54924
> > > > >> >> >> > >> [1](queued=0,recved=295,sent=0)
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.118:41390
> > > > >> >> >> > >> [1](queued=0,recved=2290,sent=0)
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.136:42243
> > > > >> >> >> > >> [1](queued=0,recved=0,sent=0)
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> Latency min/avg/max: 0/17/6333
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> Received: 47111
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> Sent: 0
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> Outstanding: 0
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> Zxid: 0x77000083f4
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> Mode: leader
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> Node count: 23
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>> That 6 second maximum latency is
> > > pretty
> > > > >> bad
> > > > >> >> but
> > > > >> >> >> > >> should
> > > > >> >> >> > >> >>>> be
> > > > >> >> >> > >> >>>> > well
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>> within
> > > > >> >> >> > >> >>>> > > > >>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>> the
> > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>> zk
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>> session timeout.
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>> So, problem is likely on the zk
> > > client
> > > > >> side
> > > > >> >> of
> > > > >> >> >> the
> > > > >> >> >> > >> >>>> session;
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>> i.e.
> > > > >> >> >> > >> >>>> > > > >>>
> > > > >> >> >> > >> >>>> > > > >>>> in
> > > > >> >> >> > >> >>>> > > > >>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>> the
> > > > >> >> >> > >> >>>> > > > >>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>> RS.
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>> You could enable GC logging as
> J-D
> > > > >> suggested
> > > > >> >> to
> > > > >> >> >> > see
> > > > >> >> >> > >> if
> > > > >> >> >> > >> >>>> you
> > > > >> >> >> > >> >>>> > > have
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>> any
> > > > >> >> >> > >> >>>> > > > >>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>> big
> > > > >> >> >> > >> >>>> > > > >>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>> pauses, pauses > zk session
> timeout.
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>> When the RS went down, it didn't
> > look
> > > > too
> > > > >> >> >> heavily
> > > > >> >> >> > >> >>>> loaded:
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>> 1. 2009-11-12 15:04:52,830 INFO
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > >> >> >> > org.apache.hadoop.hbase.regionserver.HRegionServer:
> > > > >> >> >> > >> >>>> Dump of
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>> metrics:
> > > > >> >> >> > >> >>>> > > > >>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>> request=1.5166667, regions=322,
> > > > >> stores=657,
> > > > >> >> >> > >> >>>> storefiles=631,
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>> storefileIndexSize=61,
> > > > >> memstoreSize=1472,
> > > > >> >> >> > >> >>>> usedHeap=2819,
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>> maxHeap=4079,
> > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>> blockCacheSize=658110960,
> > > > >> >> >> > blockCacheFree=197395984,
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>> blockCacheCount=9903,
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>> blockCacheHitRatio=99
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>> Its serving a few reads? The
> > number
> > > of
> > > > >> >> store
> > > > >> >> >> > files
> > > > >> >> >> > >> >>>> seems
> > > > >> >> >> > >> >>>> > > fine.
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>> Not
> > > > >> >> >> > >> >>>> > > > >>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>> too
> > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>> much memory used.
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>> Looking at the logs, I see the
> > Lease
> > > > >> Still
> > > > >> >> Held
> > > > >> >> >> > >> >>>> exception.
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>> This
> > > > >> >> >> > >> >>>> > > > >>>
> > > > >> >> >> > >> >>>> > > > >>>> happens
> > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>> when the RS does its regular
> report
> > to
> > > > the
> > > > >> >> >> master
> > > > >> >> >> > but
> > > > >> >> >> > >> the
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>> master
> > > > >> >> >> > >> >>>> > > > >>>
> > > > >> >> >> > >> >>>> > > > >>>> thinks
> > > > >> >> >> > >> >>>> > > > >>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>> the
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>> RS has since restarted. It'll
> > think
> > > > this
> > > > >> >> >> probably
> > > > >> >> >> > >> >>>> because
> > > > >> >> >> > >> >>>> > it
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>> noticed
> > > > >> >> >> > >> >>>> > > > >>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>> that
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>> the RS's znode in zk had gone
> away
> > > and
> > > > it
> > > > >> >> >> > considered
> > > > >> >> >> > >> the
> > > > >> >> >> > >> >>>> RS
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>> dead.
> > > > >> >> >> > >> >>>> > > > >>>
> > > > >> >> >> > >> >>>> > > > >>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>> Looking too at your logs I see
> > this
> > > > gap
> > > > >> in
> > > > >> >> the
> > > > >> >> >> zk
> > > > >> >> >> > >> >>>> pinging:
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>> 1. 2009-11-12 15:03:39,325 DEBUG
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>> org.apache.zookeeper.ClientCnxn:
> > > > >> >> >> > >> >>>> > > > >>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>> Got
> > > > >> >> >> > >> >>>> > > > >>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>> ping response for
> > > > >> sessionid:0x224e55436ad0004
> > > > >> >> >> after
> > > > >> >> >> > >> 0ms
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>> 2. 2009-11-12 15:03:43,113 DEBUG
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>> org.apache.zookeeper.ClientCnxn:
> > > > >> >> >> > >> >>>> > > > >>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>> Got
> > > > >> >> >> > >> >>>> > > > >>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>> ping response for
> > > > >> sessionid:0x24e55436a0007d
> > > > >> >> >> after
> > > > >> >> >> > >> 0ms
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>> Where in the lines above it, its
> > > > >> reporting
> > > > >> >> >> about
> > > > >> >> >> > >> every
> > > > >> >> >> > >> >>>> ten
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>> seconds,
> > > > >> >> >> > >> >>>> > > > >>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>> here
> > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>> there is a big gap.
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>> Do you have ganglia or something
> > that
> > > > >> will
> > > > >> >> let
> > > > >> >> >> you
> > > > >> >> >> > >> look
> > > > >> >> >> > >> >>>> more
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>> into
> > > > >> >> >> > >> >>>> > > > >>>
> > > > >> >> >> > >> >>>> > > > >>>> what
> > > > >> >> >> > >> >>>> > > > >>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>> was
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>> happening on this machine around
> > this
> > > > >> time?
> > > > >> >> Is
> > > > >> >> >> > the
> > > > >> >> >> > >> >>>> machine
> > > > >> >> >> > >> >>>> > > OK?
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>> It
> > > > >> >> >> > >> >>>> > > > >>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>> looks
> > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>> lightly loaded and you have your
> > > cluster
> > > > >> >> nicely
> > > > >> >> >> > laid
> > > > >> >> >> > >> out.
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>> Something
> > > > >> >> >> > >> >>>> > > > >>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>> odd
> > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>> is
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>> going on. What about things like
> > the
> > > > >> write
> > > > >> >> >> speed
> > > > >> >> >> > to
> > > > >> >> >> > >> >>>> disk?
> > > > >> >> >> > >> >>>> > In
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>> the
> > > > >> >> >> > >> >>>> > > > >>>
> > > > >> >> >> > >> >>>> > > > >>>> past
> > > > >> >> >> > >> >>>> > > > >>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>> strange issues have been explained
> by
> > > > >> >> incorrectly
> > > > >> >> >> > set
> > > > >> >> >> > >> BIOS
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>> which
> > > > >> >> >> > >> >>>> > > > >>>
> > > > >> >> >> > >> >>>> > > > >>>> made
> > > > >> >> >> > >> >>>> > > > >>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>> disks
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>> run at 1/100th of their proper
> > speed.
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>> St.Ack
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> Best,
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> zhenyu
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> On Wed, Nov 11, 2009 at 3:58 PM,
> > > > Zhenyu
> > > > >> >> Zhong
> > > > >> >> >> <
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>> zhongresearch@gmail.com
> > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>> wrote:
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> Stack
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>> I am very appreciated for your
> > > > >> comments.
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>> I will use the zookeeper
> > monitoring
> > > > >> script
> > > > >> >> on
> > > > >> >> >> my
> > > > >> >> >> > >> >>>> cluster
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> and
> > > > >> >> >> > >> >>>> > > > >>>
> > > > >> >> >> > >> >>>> > > > >>>> let
> > > > >> >> >> > >> >>>> > > > >>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>> it
> > > > >> >> >> > >> >>>> > > > >>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>> run
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>> overnight to see the result.
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>> I will follow up the thread
> when
> > I
> > > > get
> > > > >> >> >> anything.
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>> thanks
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>> zhenyu
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>> On Wed, Nov 11, 2009 at 3:52
> PM,
> > > > stack
> > > > >> <
> > > > >> >> >> > >> >>>> stack@duboce.net>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> wrote:
> > > > >> >> >> > >> >>>> > > > >>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>> I see these in your log too:
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>> 1. 2009-11-11 04:27:19,018
> > DEBUG
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>
> org.apache.zookeeper.ClientCnxn:
> > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>> Got
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> ping response for
> > > > >> >> sessionid:0x424dfd908c50009
> > > > >> >> >> > >> after
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>> 4544ms
> > > > >> >> >> > >> >>>> > > > >>>
> > > > >> >> >> > >> >>>> > > > >>>> 2. 2009-11-11 04:27:19,018 DEBUG
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>
> org.apache.zookeeper.ClientCnxn:
> > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>> Got
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> ping response for
> > > > >> >> sessionid:0x24dfd90c810002
> > > > >> >> >> > after
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>> 4548ms
> > > > >> >> >> > >> >>>> > > > >>>
> > > > >> >> >> > >> >>>> > > > >>>> 3. 2009-11-11 04:27:43,960 DEBUG
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>
> org.apache.zookeeper.ClientCnxn:
> > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>> Got
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>> ping response for
> > > > >> >> sessionid:0x424dfd908c50009
> > > > >> >> >> > >> after
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>> 9030ms
> > > > >> >> >> > >> >>>> > > > >>>
> > > > >> >> >> > >> >>>> > > > >>>> 4. 2009-11-11 04:27:43,960 DEBUG
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>>>>>> org.apache.zookeeper.
> > > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > > >> >> >> > >> >>>> > > > >>>>>>>
> > > > >> >> >> > >> >>>> > > >
> > > > >> >> >> > >> >>>> > >
> > > > >> >> >> > >> >>>> >
> > > > >> >> >> > >> >>>>
> > > > >> >> >> > >> >>>
> > > > >> >> >> > >> >>>
> > > > >> >> >> > >> >>
> > > > >> >> >> > >> >
> > > > >> >> >> > >>
> > > > >> >> >> > >
> > > > >> >> >> > >
> > > > >> >> >> >
> > > > >> >> >>
> > > > >> >> >
> > > > >> >> >
> > > > >> >>
> > > > >> >
> > > > >> >
> > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>
Re: regionserver disconnection
Posted by stack <st...@duboce.net>.
So, the time at which we see
2009-11-18 12:41:56,782 ERROR
org.apache.zookeeper.server.quorum.FollowerHandler: Unexpected exception
causing shutdown while sock still open
... corresponds to the time at which the regionserver loses is session?
St.Ack
On Thu, Nov 19, 2009 at 8:38 AM, Zhenyu Zhong <zh...@gmail.com>wrote:
> Here is the zookeeper leader log. It had an unexpected exception. I don't
> know why.
>
> http://pastebin.com/m6594ccc
>
> During that time, I don't think the system load is high. Memory usage
> should
> be normal.
>
> Best,
> zhenyu
>
>
>
> On Thu, Nov 19, 2009 at 11:09 AM, stack <st...@duboce.net> wrote:
>
> > What is in the other zk logs? Why did the leader go away? Memory issue
> or
> > something? Why'd it shutdown? Thanks for pastebin'ing this stuff.
> > St.Ack
> >
> > On Thu, Nov 19, 2009 at 7:53 AM, Zhenyu Zhong <zhongresearch@gmail.com
> > >wrote:
> >
> > > After digging more on the zookeeper log, I found that there seemed to
> be
> > a
> > > socketconnection timeout. Also some warnings indicate that zookeeper
> > server
> > > is not running. Probably I need to understand the mechanism of
> zookeeper
> > > first.
> > >
> > > Please find the zookeeper log around that time.
> > >
> > > http://pastebin.com/m37bb4ad1
> > >
> > > Many thanks
> > > zhenyu
> > >
> > >
> > > On Thu, Nov 19, 2009 at 10:31 AM, Zhenyu Zhong <
> zhongresearch@gmail.com
> > > >wrote:
> > >
> > > > Stack,
> > > >
> > > > I am very appreciated. Let me dig into the zookeeper log more.
> > > >
> > > > FYI, I saw the sleeper complained about the delay for the first time.
> > It
> > > > looks like there are different reasons for the RS disconnections.
> > > >
> > > > 2009-11-19 08:08:32,708 DEBUG org.apache.zookeeper.ClientCnxn: Got
> ping
> > > > response for sessionid:0x42508716bfc0001 after 0ms
> > > > 2009-11-19 08:08:37,595 DEBUG org.apache.zookeeper.ClientCnxn: Got
> ping
> > > > response for sessionid:0x42508716bfc0082 after 0ms
> > > > 2009-11-19 08:08:46,073 DEBUG org.apache.zookeeper.ClientCnxn: Got
> ping
> > > > response for sessionid:0x42508716bfc0001 after 0ms
> > > > 2009-11-19 08:09:39,880 WARN org.apache.zookeeper.ClientCnxn:
> Exception
> > > > closing session 0x42508716bfc0001 to
> > sun.nio.ch.SelectionKeyImpl@4f3a608f
> > > > java.io.IOException: TIMED OUT
> > > > at
> > > > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:906)
> > > > 2009-11-19 08:09:39,880 WARN org.apache.zookeeper.ClientCnxn:
> Exception
> > > > closing session 0x42508716bfc0082 to
> > sun.nio.ch.SelectionKeyImpl@14c8875
> > > > java.io.IOException: TIMED OUT
> > > > at
> > > > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:906)
> > > > 2009-11-19 08:09:39,880 WARN org.apache.hadoop.hbase.util.Sleeper: We
> > > slept
> > > > 53556ms, ten times longer than scheduled: 3000
> > > > 2009-11-19 08:09:39,980 WARN
> > > > org.apache.hadoop.hbase.regionserver.HRegionServer: Attempt=1
> > > > org.apache.hadoop.hbase.Leases$LeaseStillHeldException
> > > > at
> > sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> > > > Method)
> > > > at
> > > >
> > >
> >
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> > > > at
> > > >
> > >
> >
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> > > > at
> > > java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> > > > at
> > > >
> > >
> >
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
> > > > at
> > > >
> > >
> >
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
> > > > at
> > > >
> > >
> >
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
> > > > at
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:571)
> > > > at java.lang.Thread.run(Thread.java:619)
> > > > 2009-11-19 08:09:39,980 INFO
> > > > org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper
> > event,
> > > > state: Disconnected, type: None, path: null
> > > > 2009-11-19 08:09:40,530 INFO org.apache.zookeeper.ClientCnxn:
> > Attempting
> > > > connection to server superpyxis0005/192.168.100.119:2181
> > > > 2009-11-19 08:09:40,530 INFO org.apache.zookeeper.ClientCnxn: Priming
> > > > connection to java.nio.channels.SocketChannel[connected local=/
> > > > 192.168.100.122:40659 remote=superpyxis0005/192.168.100.119:2181]
> > > >
> > > >
> > > >
> > > > Best,
> > > > zhenyu
> > > >
> > > > On Wed, Nov 18, 2009 at 6:52 PM, stack <st...@duboce.net> wrote:
> > > >
> > > >> Patrick Hunt, one of the zk lads, answered with the following:
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> According to the following snippet (I cut to just session
> > > >> 0x425035c48210002)
> > > >> the client is getting pings for this session id regularly every 6
> > > seconds
> > > >> or
> > > >> so, then after 41,016 there's a 20 second gap (corresponds to a 30
> > > second
> > > >> timeout being set during session creation) at which point we
> > disconnect
> > > >> from
> > > >> the server.
> > > >>
> > > >>
> > > >> > 2009-11-18 12:41:34,788 DEBUG org.apache.zookeeper.ClientCnxn: Got
> > > ping
> > > >> > response for sessionid:0x425035c48210002 after 0ms
> > > >> > 2009-11-18 12:42:01,482 WARN org.apache.zookeeper.ClientCnxn:
> > > Exception
> > > >> > closing session 0x425035c48210002 to
> > > >> sun.nio.ch.SelectionKeyImpl@421690ab
> > > >> > java.io.IOException: TIMED OUT
> > > >>
> > > >> the client notifies hbase of the disco, and attempts to reconnect to
> a
> > > >> server
> > > >>
> > > >>
> > > >> > org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper
> > > event,
> > > >> > state: Disconnected, type: None, path: null
> > > >> > 2009-11-18 12:42:01,782 INFO org.apache.zookeeper.ClientCnxn:
> > > Attempting
> > > >> > connection to server superpyxis0003/192.168.100.117:2181
> > > >>
> > > >> but the server won't allow the session be be established for some
> > reason
> > > >> (is
> > > >> that the right server:port that's it's trying to connect to?)
> > > >>
> > > >>
> > > >> > 2009-11-18 12:42:02,182 INFO org.apache.zookeeper.ClientCnxn:
> Server
> > > >> > connection successful
> > > >> > 2009-11-18 12:42:02,182 WARN org.apache.zookeeper.ClientCnxn:
> > > Exception
> > > >> > closing session 0x425035c48210002 to
> > > >> sun.nio.ch.SelectionKeyImpl@5c07076b
> > > >> > java.io.IOException: Read error rc = -1
> > > java.nio.DirectByteBuffer[pos=0
> > > >> > lim=4 cap=4]
> > > >>
> > > >> The "read error rc = -1" means that the client read from the socket
> > and
> > > >> got
> > > >> EOS. But it's not a session expiration afaict (and should not be at
> > this
> > > >> point in time).
> > > >>
> > > >>
> > > >> You need to look at the ZK server logs for this period in time (for
> > this
> > > >> server superpyxis0003:2181, and superpyxis0001:2181, the client
> tries
> > to
> > > >> connect to both of these). There should be some indication there why
> > the
> > > >> server is closing down the connection and not allowing the session
> to
> > be
> > > >> re-established.
> > > >>
> > > >> One thought - by default we limit clients to a max of 10 connections
> > > from
> > > >> a
> > > >> particular host to the server. If you exceed this limit the zk
> server
> > > >> (it's
> > > >> per server though, not per cluster, so perhaps this is not it) will
> > not
> > > >> allow any additional connections from the client host (you can't
> > exceed
> > > >> 10
> > > >> on a server). Could this be it? i.e. the user is running a number of
> > ZK
> > > >> clients (say hbase processes) from the same host? Regardless, get
> the
> > > >> server
> > > >> log for this time period and it should be more clear.
> > > >>
> > > >> Patrick
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> On Wed, Nov 18, 2009 at 11:18 AM, stack <st...@duboce.net> wrote:
> > > >>
> > > >> > And running stat on each of your zookeeper nodes tells you what
> > about
> > > >> > averages for connect-times?
> > > >> >
> > > >> > Its odd that there is a ping just 20 seconds beforehand and you
> have
> > > set
> > > >> > the zk session timeout at 60 seconds (or ten minutes?). Let me
> ask
> > > the
> > > >> zk
> > > >> > lads about it.
> > > >> >
> > > >> > If you look in the GC log for around this time, do you see
> anything?
> > > >> >
> > > >> > St.Ack
> > > >> >
> > > >> >
> > > >> >
> > > >> > On Wed, Nov 18, 2009 at 7:32 AM, Zhenyu Zhong <
> > > zhongresearch@gmail.com
> > > >> >wrote:
> > > >> >
> > > >> >> After running more experiment, it seems that the disconnection is
> > not
> > > >> >> related to the heavy load job, because the load average is low,
> the
> > > >> diskio
> > > >> >> is normal, memory Heap was not reached, also the virtual memory
> > stats
> > > >> >> shows
> > > >> >> that no swappings. However, the disconnection still happens.
> > > >> >>
> > > >> >> It looks like this time it pauses for 20 seconds. No idea why
> > > >> regionserver
> > > >> >> disconnected.
> > > >> >>
> > > >> >> Any other suggestions please?
> > > >> >>
> > > >> >>
> > > >> >> 2009-11-18 12:41:00,956 DEBUG org.apache.zookeeper.ClientCnxn:
> Got
> > > ping
> > > >> >> response for sessionid:0x125035c495d02bb after 0ms
> > > >> >> 2009-11-18 12:41:08,074 DEBUG org.apache.zookeeper.ClientCnxn:
> Got
> > > ping
> > > >> >> response for sessionid:0x425035c48210002 after 0ms
> > > >> >> 2009-11-18 12:41:14,302 DEBUG org.apache.zookeeper.ClientCnxn:
> Got
> > > ping
> > > >> >> response for sessionid:0x125035c495d02bb after 0ms
> > > >> >> 2009-11-18 12:41:21,433 DEBUG org.apache.zookeeper.ClientCnxn:
> Got
> > > ping
> > > >> >> response for sessionid:0x425035c48210002 after 0ms
> > > >> >> 2009-11-18 12:41:27,656 DEBUG org.apache.zookeeper.ClientCnxn:
> Got
> > > ping
> > > >> >> response for sessionid:0x125035c495d02bb after 0ms
> > > >> >> 2009-11-18 12:41:34,788 DEBUG org.apache.zookeeper.ClientCnxn:
> Got
> > > ping
> > > >> >> response for sessionid:0x425035c48210002 after 0ms
> > > >> >> 2009-11-18 12:41:41,016 DEBUG org.apache.zookeeper.ClientCnxn:
> Got
> > > ping
> > > >> >> response for sessionid:0x125035c495d02bb after 1ms
> > > >> >> 2009-11-18 12:42:01,482 WARN org.apache.zookeeper.ClientCnxn:
> > > Exception
> > > >> >> closing session 0x425035c48210002 to
> > > >> sun.nio.ch.SelectionKeyImpl@421690ab
> > > >> >> java.io.IOException: TIMED OUT
> > > >> >> at
> > > >> >>
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:906)
> > > >> >> 2009-11-18 12:42:01,582 INFO
> > > >> >> org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper
> > > >> event,
> > > >> >> state: Disconnected, type: None, path: null
> > > >> >> 2009-11-18 12:42:01,782 INFO org.apache.zookeeper.ClientCnxn:
> > > >> Attempting
> > > >> >> connection to server superpyxis0003/192.168.100.117:2181
> > > >> >> 2009-11-18 12:42:01,782 INFO org.apache.zookeeper.ClientCnxn:
> > Priming
> > > >> >> connection to java.nio.channels.SocketChannel[connected local=/
> > > >> >> 192.168.100.132:46610 remote=superpyxis0003/192.168.100.117:2181
> ]
> > > >> >> 2009-11-18 12:42:01,782 INFO org.apache.zookeeper.ClientCnxn:
> > Server
> > > >> >> connection successful
> > > >> >> 2009-11-18 12:42:01,782 WARN org.apache.zookeeper.ClientCnxn:
> > > Exception
> > > >> >> closing session 0x425035c48210002 to
> > > >> sun.nio.ch.SelectionKeyImpl@56dc6fac
> > > >> >> java.io.IOException: Read error rc = -1
> > > java.nio.DirectByteBuffer[pos=0
> > > >> >> lim=4 cap=4]
> > > >> >> at
> > > >> >>
> > org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
> > > >> >> at
> > > >> >>
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
> > > >> >> 2009-11-18 12:42:01,782 WARN org.apache.zookeeper.ClientCnxn:
> > > Ignoring
> > > >> >> exception during shutdown input
> > > >> >> java.net.SocketException: Transport endpoint is not connected
> > > >> >> at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
> > > >> >> at
> > > >> >>
> > > sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
> > > >> >> at
> > > >> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> > > >> >> at
> > > >> >>
> > > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> > > >> >> at
> > > >> >>
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> > > >> >> 2009-11-18 12:42:01,782 WARN org.apache.zookeeper.ClientCnxn:
> > > Ignoring
> > > >> >> exception during shutdown output
> > > >> >> java.net.SocketException: Transport endpoint is not connected
> > > >> >> at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
> > > >> >> at
> > > >> >>
> > > sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
> > > >> >> at
> > > >> sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> > > >> >> at
> > > >> >>
> > > >>
> > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> > > >> >> at
> > > >> >>
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> > > >> >> 2009-11-18 12:42:02,182 INFO org.apache.zookeeper.ClientCnxn:
> > > >> Attempting
> > > >> >> connection to server superpyxis0001/192.168.100.115:2181
> > > >> >> 2009-11-18 12:42:02,182 INFO org.apache.zookeeper.ClientCnxn:
> > Priming
> > > >> >> connection to java.nio.channels.SocketChannel[connected local=/
> > > >> >> 192.168.100.132:36197 remote=superpyxis0001/192.168.100.115:2181
> ]
> > > >> >> 2009-11-18 12:42:02,182 INFO org.apache.zookeeper.ClientCnxn:
> > Server
> > > >> >> connection successful
> > > >> >> 2009-11-18 12:42:02,182 WARN org.apache.zookeeper.ClientCnxn:
> > > Exception
> > > >> >> closing session 0x425035c48210002 to
> > > >> sun.nio.ch.SelectionKeyImpl@5c07076b
> > > >> >> java.io.IOException: Read error rc = -1
> > > java.nio.DirectByteBuffer[pos=0
> > > >> >> lim=4 cap=4]
> > > >> >> at
> > > >> >>
> > org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
> > > >> >> at
> > > >> >>
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
> > > >> >> 2009-11-18 12:42:02,182 WARN org.apache.zookeeper.ClientCnxn:
> > > Ignoring
> > > >> >> exception during shutdown input
> > > >> >> java.net.SocketException: Transport endpoint is not connected
> > > >> >> at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
> > > >> >> at
> > > >> >>
> > > sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
> > > >> >> at
> > > >> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> > > >> >> at
> > > >> >>
> > > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> > > >> >> at
> > > >> >>
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> > > >> >> 2009-11-18 12:42:02,182 WARN org.apache.zookeeper.ClientCnxn:
> > > Ignoring
> > > >> >> exception during shutdown output
> > > >> >> java.net.SocketException: Transport endpoint is not connected
> > > >> >>
> > > >> >>
> > > >> >> On Mon, Nov 16, 2009 at 4:16 PM, Zhenyu Zhong <
> > > zhongresearch@gmail.com
> > > >> >> >wrote:
> > > >> >>
> > > >> >> > Here is the diskIO and CPU around the time we had RS
> > disconnection
> > > on
> > > >> >> one
> > > >> >> > machine that runs RegionServer. It doesn't seem to be high.
> > Similar
> > > >> disk
> > > >> >> and
> > > >> >> > cpu usage have been seen before and the HBase was running fine.
> > > >> >> >
> > > >> >> >
> > > >> >> > So far I haven't found why my 10 minutes session timeout
> doesn't
> > > >> apply.
> > > >> >> > Still digging.
> > > >> >> >
> > > >> >> > Many thanks!
> > > >> >> >
> > > >> >> > Time: 03:01:08 PM
> > > >> >> > avg-cpu: %user %nice %system %iowait %steal %idle
> > > >> >> > 3.82 0.00 0.89 3.97 0.00 91.32
> > > >> >> >
> > > >> >> >
> > > >> >> > Device: tps Blk_read/s Blk_wrtn/s Blk_read
> > > >> Blk_wrtn
> > > >> >> > sda 24.30 1605.39 6679.35 6238623388
> > > >> 25956251720
> > > >> >> > sda1 0.51 1.94 15.71 7547265
> > > >> 61067904
> > > >> >> > sda2 0.13 2.11 5.20 8202079
> > > >> 20189232
> > > >> >> > sda3 0.13 0.25 1.63 987846
> > > >> 6323336
> > > >> >> > sda4 0.00 0.00 0.00 4
> > > >> 0
> > > >> >> > sda5 1.78 0.06 24.14 220010
> > > >> 93817208
> > > >> >> > sda6 0.38 0.42 8.41 1630584
> > > >> 32688152
> > > >> >> > sda7 21.38 1600.61 6624.26 6220035272
> > > >> 25742165888
> > > >> >> > sdb 4.52 767.33 565.91 2981868690
> > > >> 2199132380
> > > >> >> > sdb1 4.52 767.33 565.91 2981866755
> > > >> 2199132372
> > > >> >> > sdc 4.42 742.95 563.00 2887151482
> > > >> 2187823092
> > > >> >> > sdc1 4.42 742.95 563.00 2887149547
> > > >> 2187823084
> > > >> >> > sdd 4.49 750.78 557.25 2917554074
> > > >> 2165513500
> > > >> >> > sdd1 4.49 750.78 557.25 2917552139
> > > >> 2165513492
> > > >> >> > sde 4.52 758.51 569.46 2947593394
> > > >> 2212964236
> > > >> >> > sde1 4.52 758.51 569.46 2947591459
> > > >> 2212964228
> > > >> >> > sdf 4.51 747.22 571.78 2903740266
> > > >> 2221972652
> > > >> >> > sdf1 4.51 747.22 571.78 2903738331
> > > >> 2221972644
> > > >> >> >
> > > >> >> > Best,
> > > >> >> > zhenyu
> > > >> >> >
> > > >> >> >
> > > >> >> >
> > > >> >> > On Mon, Nov 16, 2009 at 4:07 PM, stack <st...@duboce.net>
> wrote:
> > > >> >> >
> > > >> >> >> On Mon, Nov 16, 2009 at 12:05 PM, Zhenyu Zhong <
> > > >> >> zhongresearch@gmail.com
> > > >> >> >> >wrote:
> > > >> >> >>
> > > >> >> >> > I just realized that there was a MapReduce job running
> during
> > > the
> > > >> >> time
> > > >> >> >> the
> > > >> >> >> > regionserver disconnected from the zookeeper.
> > > >> >> >> > That MapReduce Job was processing 500GB data and took about
> 8
> > > >> minutes
> > > >> >> to
> > > >> >> >> > finish. It launched over 2000 map tasks.
> > > >> >> >>
> > > >> >> >>
> > > >> >> >> There was a tasktracker running on the RegionServer that
> > > >> disconnected?
> > > >> >> Is
> > > >> >> >> the map i/o or cpu heavy? Do you think it could have stole
> life
> > > >> from
> > > >> >> the
> > > >> >> >> datanode/regionserver?
> > > >> >> >>
> > > >> >> >>
> > > >> >> >>
> > > >> >> >> > I doubt that this introduced
> > > >> >> >> > resource contention between DataNode and RegionServer Node.
> > > >> >> >> > Also during the time that MapReduce job ran, I saw a few
> > errors
> > > >> >> >> indicating
> > > >> >> >> > that
> > > >> >> >> >
> > > >> >> >> > java.io.IOException: Could not obtain block:
> > > >> >> >> > blk_7661247556283580226_4450462
> > file=/data/xxx.txt.200910141159
> > > >> >> >> > at
> > > >> >> >> >
> > > >> >> >>
> > > >> >>
> > > >>
> > >
> >
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1787)
> > > >> >> >> > at
> > > >> >> >> >
> > > >> >> >>
> > > >> >>
> > > >>
> > >
> >
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1615)
> > > >> >> >> > at
> > > >> >> >> >
> > > >> >> >>
> > > >> >>
> > > >>
> > >
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1742)
> > > >> >> >> > at
> > java.io.DataInputStream.read(DataInputStream.java:83)
> > > >> >> >> > at
> > > >> >> >>
> org.apache.hadoop.util.LineReader.readLine(LineReader.java:134)
> > > >> >> >> > at
> > > >> >> >> >
> > > >> >> >>
> > > >> >>
> > > >>
> > >
> org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:136)
> > > >> >> >> > at
> > > >> >> >> >
> > > >> >>
> > > >>
> > org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:40)
> > > >> >> >> > at
> > > >> >> >> >
> > > >> >> >>
> > > >> >>
> > > >>
> > >
> >
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:192)
> > > >> >> >> > at
> > > >> >> >> >
> > > >> >> >>
> > > >> >>
> > > >>
> > >
> >
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:176)
> > > >> >> >> > at
> > > >> org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
> > > >> >> >> > at
> > > >> >> >>
> org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
> > > >> >> >> > at
> > org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
> > > >> >> >> > at
> org.apache.hadoop.mapred.Child.main(Child.java:170)
> > > >> >> >> >
> > > >> >> >>
> > > >> >> >> This is interesting. Your map task is running into hdfs-127.
> > > Yeah,
> > > >> >> you
> > > >> >> >> should patch your cluster if you want to get rid of these
> > > (hdfs-127
> > > >> has
> > > >> >> >> been
> > > >> >> >> applied to 0.21 hadoop and will be in the next hadoop release
> on
> > > >> 0.20.x
> > > >> >> >> branch, hadoop 0.20.2).
> > > >> >> >>
> > > >> >> >>
> > > >> >> >>
> > > >> >> >> >
> > > >> >> >> > Possibly it is related to HDFS-127, but I don't see any
> > > datanodes
> > > >> >> went
> > > >> >> >> > down. Does that affect Regionserver? Shall we apply the
> patch?
> > > >> >> >> >
> > > >> >> >> >
> > > >> >> >> Read up on hdfs-127 for explaination of whats going on.
> > > >> >> >>
> > > >> >> >>
> > > >> >> >>
> > > >> >> >> > Now I start to keep tracking of the virtual memory stats to
> > see
> > > if
> > > >> >> the
> > > >> >> >> same
> > > >> >> >> > issue happens tomorrow around the same time period.
> > > >> >> >> >
> > > >> >> >>
> > > >> >> >> Excellent. Thanks for doing this.
> > > >> >> >>
> > > >> >> >>
> > > >> >> >>
> > > >> >> >> > And I highly suspect that this particular MapReduce job
> hurts
> > > >> HBase
> > > >> >> >> > Regionserver.
> > > >> >> >> >
> > > >> >> >> > PS. I also use gcviewer to parse the GC-log, I only see
> around
> > > 30
> > > >> >> >> seconds
> > > >> >> >> > pauses maximum.
> > > >> >> >> >
> > > >> >> >>
> > > >> >> >>
> > > >> >> >> Did you figure why your ten minute session is not applying?
> > > >> >> >>
> > > >> >> >> St.Ack
> > > >> >> >>
> > > >> >> >>
> > > >> >> >>
> > > >> >> >> >
> > > >> >> >> > Thanks a lot.
> > > >> >> >> > zhenyu
> > > >> >> >> >
> > > >> >> >> >
> > > >> >> >> >
> > > >> >> >> > On Sun, Nov 15, 2009 at 12:52 AM, Zhenyu Zhong <
> > > >> >> zhongresearch@gmail.com
> > > >> >> >> > >wrote:
> > > >> >> >> >
> > > >> >> >> > > J-D,
> > > >> >> >> > >
> > > >> >> >> > > Thank you very much for your comments.
> > > >> >> >> > > My company block the IRC port, so I have trouble to
> connect
> > to
> > > >> IRC
> > > >> >> >> > channel.
> > > >> >> >> > > I have been trying to ask the IT to open the IRC port for
> > me,
> > > it
> > > >> >> might
> > > >> >> >> > take
> > > >> >> >> > > a while.
> > > >> >> >> > >
> > > >> >> >> > > Best,
> > > >> >> >> > > zhenyu
> > > >> >> >> > >
> > > >> >> >> > >
> > > >> >> >> > > On Sat, Nov 14, 2009 at 2:21 PM, Jean-Daniel Cryans <
> > > >> >> >> jdcryans@apache.org
> > > >> >> >> > >wrote:
> > > >> >> >> > >
> > > >> >> >> > >> The error you are getting is a disconnection from a
> > zookeeper
> > > >> >> server
> > > >> >> >> > >> and is very generic.
> > > >> >> >> > >>
> > > >> >> >> > >> ZK-86 is still opened and the last comment refers to
> ZK-111
> > > >> saying
> > > >> >> >> > >> that the bug (in unit tests) was probably fixed in
> release
> > > >> 3.0.0
> > > >> >> last
> > > >> >> >> > >> year.
> > > >> >> >> > >>
> > > >> >> >> > >> To figure the hang you have, you can try to jstack the
> > > process
> > > >> pid
> > > >> >> >> and
> > > >> >> >> > >> see exactly what's holding the RS from shutting down.
> > > >> >> >> > >>
> > > >> >> >> > >> Would it be possible for you to drop by the IRC channel?
> > This
> > > >> way
> > > >> >> we
> > > >> >> >> > >> can debug this a at much faster pace.
> > > >> >> >> > >>
> > > >> >> >> > >> Thx!
> > > >> >> >> > >>
> > > >> >> >> > >> J-D
> > > >> >> >> > >>
> > > >> >> >> > >> On Sat, Nov 14, 2009 at 9:57 AM, Zhenyu Zhong <
> > > >> >> >> zhongresearch@gmail.com>
> > > >> >> >> > >> wrote:
> > > >> >> >> > >> > I found this.
> > > >> >> >> > >> > http://issues.apache.org/jira/browse/ZOOKEEPER-86
> > > >> >> >> > >> >
> > > >> >> >> > >> > It looks like the same error I had. Is it a zookeeper
> > bug?
> > > >> When
> > > >> >> >> will
> > > >> >> >> > >> HBase
> > > >> >> >> > >> > take the zookeeper version 3.3.0?
> > > >> >> >> > >> >
> > > >> >> >> > >> > thanks
> > > >> >> >> > >> > zhenyu
> > > >> >> >> > >> >
> > > >> >> >> > >> > On Sat, Nov 14, 2009 at 11:31 AM, Zhenyu Zhong <
> > > >> >> >> > zhongresearch@gmail.com
> > > >> >> >> > >> >wrote:
> > > >> >> >> > >> >
> > > >> >> >> > >> >> Now I really doubt about the zookeeper, from the log I
> > saw
> > > >> >> errors
> > > >> >> >> > like
> > > >> >> >> > >> >> IOException Read Error, while the zookeeper client
> > > >> >> (regionserver )
> > > >> >> >> > >> tried to
> > > >> >> >> > >> >> read. But it got disconneted status from zookeeper.
> > > >> >> >> > >> >>
> > > >> >> >> > >> >> I don't see any load on any zookeeper quorum servers.
> > > DiskIO
> > > >> is
> > > >> >> >> fine.
> > > >> >> >> > >> >>
> > > >> >> >> > >> >> Also when the regionserver decides to exit due to the
> > > >> >> disconnect
> > > >> >> >> > status
> > > >> >> >> > >> >> from zookeeper, sometimes the regionserver hangs
> during
> > > the
> > > >> >> >> exiting.
> > > >> >> >> > We
> > > >> >> >> > >> can
> > > >> >> >> > >> >> still see the HRegionServer process even we don't see
> it
> > > in
> > > >> the
> > > >> >> >> > Master
> > > >> >> >> > >> web
> > > >> >> >> > >> >> interface.
> > > >> >> >> > >> >>
> > > >> >> >> > >> >> I also notice that there is an zookeeper.retries and
> > > >> >> >> zookeeper.pauses
> > > >> >> >> > >> >> settings in hbase-default.xml, would that matter ?
> > > >> >> >> > >> >>
> > > >> >> >> > >> >> thanks
> > > >> >> >> > >> >>
> > > >> >> >> > >> >> You may see the IOException error in the following log
> > > when
> > > >> the
> > > >> >> >> > >> >> Regionserver lost connection to the zookeeper.
> > > >> >> >> > >> >>
> > > >> >> >> > >> >> 2009-11-14 15:58:31,993 INFO
> > > >> >> >> > org.apache.hadoop.hbase.regionserver.HLog:
> > > >> >> >> > >> >> Roll
> > > >> >> >> > >> >>
> > > >> >> >> > >>
> > > >> >> >> >
> > > >> >> >>
> > > >> >>
> > > >>
> > >
> >
> /hbase/.logs/superpyxis0008.scur.colo,60021,1258179101617/hlog.dat.1258179106921,
> > > >> >> >> > >> >> entries=1062964, calcsize=255014936,
> filesize=139566093.
> > > New
> > > >> >> hlog
> > > >> >> >> > >> >>
> > > >> >> >> > >>
> > > >> >> >> >
> > > >> >> >>
> > > >> >>
> > > >>
> > >
> >
> /hbase/.logs/superpyxis0008.scur.colo,60021,1258179101617/hlog.dat.1258214311993
> > > >> >> >> > >> >> 2009-11-14 15:58:39,295 DEBUG
> > > >> org.apache.zookeeper.ClientCnxn:
> > > >> >> Got
> > > >> >> >> > ping
> > > >> >> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
> > > >> >> >> > >> >> 2009-11-14 15:58:52,648 DEBUG
> > > >> org.apache.zookeeper.ClientCnxn:
> > > >> >> Got
> > > >> >> >> > ping
> > > >> >> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
> > > >> >> >> > >> >> 2009-11-14 15:59:06,007 DEBUG
> > > >> org.apache.zookeeper.ClientCnxn:
> > > >> >> Got
> > > >> >> >> > ping
> > > >> >> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
> > > >> >> >> > >> >> 2009-11-14 15:59:19,365 DEBUG
> > > >> org.apache.zookeeper.ClientCnxn:
> > > >> >> Got
> > > >> >> >> > ping
> > > >> >> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
> > > >> >> >> > >> >> 2009-11-14 15:59:32,867 DEBUG
> > > >> org.apache.zookeeper.ClientCnxn:
> > > >> >> Got
> > > >> >> >> > ping
> > > >> >> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 186ms
> > > >> >> >> > >> >> 2009-11-14 15:59:46,070 DEBUG
> > > >> org.apache.zookeeper.ClientCnxn:
> > > >> >> Got
> > > >> >> >> > ping
> > > >> >> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
> > > >> >> >> > >> >> 2009-11-14 15:59:59,423 DEBUG
> > > >> org.apache.zookeeper.ClientCnxn:
> > > >> >> Got
> > > >> >> >> > ping
> > > >> >> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
> > > >> >> >> > >> >> 2009-11-14 16:00:12,775 DEBUG
> > > >> org.apache.zookeeper.ClientCnxn:
> > > >> >> Got
> > > >> >> >> > ping
> > > >> >> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
> > > >> >> >> > >> >> 2009-11-14 16:00:28,141 DEBUG
> > > >> org.apache.zookeeper.ClientCnxn:
> > > >> >> Got
> > > >> >> >> > ping
> > > >> >> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 2010ms
> > > >> >> >> > >> >> 2009-11-14 16:01:10,378 WARN
> > > >> org.apache.zookeeper.ClientCnxn:
> > > >> >> >> > Exception
> > > >> >> >> > >> >> closing session 0x424f0ed01ea00a6 to
> > > >> >> >> > >> sun.nio.ch.SelectionKeyImpl@39fbb2d6
> > > >> >> >> > >> >>
> > > >> >> >> > >> >> java.io.IOException: Read error rc = -1
> > > >> >> >> > java.nio.DirectByteBuffer[pos=0
> > > >> >> >> > >> >> lim=4 cap=4]
> > > >> >> >> > >> >> at
> > > >> >> >> > >> >>
> > > >> >> >>
> > > org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
> > > >> >> >> > >> >> at
> > > >> >> >> > >> >>
> > > >> >> >>
> > > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
> > > >> >> >> > >> >> 2009-11-14 16:01:10,478 INFO
> > > >> >> >> > >> >> org.apache.hadoop.hbase.regionserver.HRegionServer:
> Got
> > > >> >> ZooKeeper
> > > >> >> >> > >> event,
> > > >> >> >> > >> >> state: Disconnected, type: None, path: null
> > > >> >> >> > >> >> 2009-11-14 16:01:11,333 WARN
> > > >> >> >> > >> >> org.apache.hadoop.hbase.regionserver.HRegionServer:
> > > >> Attempt=1
> > > >> >> >> > >> >>
> > > >> >> >> > >> >> org.apache.hadoop.hbase.Leases$LeaseStillHeldException
> > > >> >> >> > >> >> at
> > > >> >> >> > >>
> > sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> > > >> >> >> > >> >> Method)
> > > >> >> >> > >> >> at
> > > >> >> >> > >> >>
> > > >> >> >> > >>
> > > >> >> >> >
> > > >> >> >>
> > > >> >>
> > > >>
> > >
> >
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> > > >> >> >> > >> >> at
> > > >> >> >> > >> >>
> > > >> >> >> > >>
> > > >> >> >> >
> > > >> >> >>
> > > >> >>
> > > >>
> > >
> >
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> > > >> >> >> > >> >> at
> > > >> >> >> > >>
> > > java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> > > >> >> >> > >> >> at
> > > >> >> >> > >> >>
> > > >> >> >> > >>
> > > >> >> >> >
> > > >> >> >>
> > > >> >>
> > > >>
> > >
> >
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
> > > >> >> >> > >> >> at
> > > >> >> >> > >> >>
> > > >> >> >> > >>
> > > >> >> >> >
> > > >> >> >>
> > > >> >>
> > > >>
> > >
> >
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
> > > >> >> >> > >> >> at
> > > >> >> >> > >> >>
> > > >> >> >> > >>
> > > >> >> >> >
> > > >> >> >>
> > > >> >>
> > > >>
> > >
> >
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
> > > >> >> >> > >> >> at
> > > >> >> >> > >> >>
> > > >> >> >> > >>
> > > >> >> >> >
> > > >> >> >>
> > > >> >>
> > > >>
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:571)
> > > >> >> >> > >> >> at java.lang.Thread.run(Thread.java:619)
> > > >> >> >> > >> >> 2009-11-14 16:01:11,433 INFO
> > > >> org.apache.zookeeper.ClientCnxn:
> > > >> >> >> > >> Attempting
> > > >> >> >> > >> >> connection to server superpyxis0001/
> > 192.168.100.115:2181
> > > >> >> >> > >> >> 2009-11-14 16:01:11,433 INFO
> > > >> org.apache.zookeeper.ClientCnxn:
> > > >> >> >> Priming
> > > >> >> >> > >> >> connection to
> java.nio.channels.SocketChannel[connected
> > > >> local=/
> > > >> >> >> > >> >> 192.168.100.122:59575 remote=superpyxis0001/
> > > >> >> 192.168.100.115:2181]
> > > >> >> >> > >> >> 2009-11-14 16:01:11,433 INFO
> > > >> org.apache.zookeeper.ClientCnxn:
> > > >> >> >> Server
> > > >> >> >> > >> >> connection successful
> > > >> >> >> > >> >> 2009-11-14 16:01:11,433 INFO
> > > >> >> >> > >> >> org.apache.hadoop.hbase.regionserver.HRegionServer:
> Got
> > > >> >> ZooKeeper
> > > >> >> >> > >> event,
> > > >> >> >> > >> >> state: Expired, type: None, path: null
> > > >> >> >> > >> >> 2009-11-14 16:01:11,433 WARN
> > > >> org.apache.zookeeper.ClientCnxn:
> > > >> >> >> > Exception
> > > >> >> >> > >> >> closing session 0x424f0ed01ea00a6 to
> > > >> >> >> > >> sun.nio.ch.SelectionKeyImpl@6b93d343
> > > >> >> >> > >> >> java.io.IOException: Session Expired
> > > >> >> >> > >> >> at
> > > >> >> >> > >> >>
> > > >> >> >> > >>
> > > >> >> >> >
> > > >> >> >>
> > > >> >>
> > > >>
> > >
> >
> org.apache.zookeeper.ClientCnxn$SendThread.readConnectResult(ClientCnxn.java:589)
> > > >> >> >> > >> >> at
> > > >> >> >> > >> >>
> > > >> >> >>
> > > org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:709)
> > > >> >> >> > >> >>
> > > >> >> >> > >> >> at
> > > >> >> >> > >> >>
> > > >> >> >>
> > > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
> > > >> >> >> > >> >> 2009-11-14 16:01:11,433 ERROR
> > > >> >> >> > >> >> org.apache.hadoop.hbase.regionserver.HRegionServer:
> > > >> ZooKeeper
> > > >> >> >> session
> > > >> >> >> > >> >> expired
> > > >> >> >> > >> >> 2009-11-14 16:01:13,513 INFO
> > > >> org.apache.hadoop.ipc.HBaseServer:
> > > >> >> >> IPC
> > > >> >> >> > >> Server
> > > >> >> >> > >> >> handler 3 on 60021, call put([B@73deb204,
> > > >> >> >> > >> >> [Lorg.apache.hadoop.hbase.client.Put;@2179600a) from
> > > >> >> >> > >> 192.168.100.132:40728:
> > > >> >> >> > >> >> error: java.io.IOException: Server not running,
> aborting
> > > >> >> >> > >> >> java.io.IOException: Server not running, aborting
> > > >> >> >> > >> >>
> > > >> >> >> > >> >>
> > > >> >> >> > >> >>
> > > >> >> >> > >> >>
> > > >> >> >> > >> >>
> > > >> >> >> > >> >> On Sat, Nov 14, 2009 at 1:01 AM, Zhenyu Zhong <
> > > >> >> >> > zhongresearch@gmail.com
> > > >> >> >> > >> >wrote:
> > > >> >> >> > >> >>
> > > >> >> >> > >> >>> Stack,
> > > >> >> >> > >> >>>
> > > >> >> >> > >> >>> Thanks a lot!
> > > >> >> >> > >> >>>
> > > >> >> >> > >> >>> I found out that the reason why HBase doesn't take
> the
> > > >> system
> > > >> >> >> file
> > > >> >> >> > >> >>> descriptor value I set before. I started the HBase
> > using
> > > >> root
> > > >> >> >> > instead
> > > >> >> >> > >> of the
> > > >> >> >> > >> >>> normal hadoop user, while my system configures hadoop
> > > with
> > > >> >> higher
> > > >> >> >> > file
> > > >> >> >> > >> >>> descriptor value but configures root with 1024
> default
> > > >> value.
> > > >> >> >> > >> >>>
> > > >> >> >> > >> >>>
> > > >> >> >> > >> >>> Now my system has a clean start. However, it seems
> that
> > > the
> > > >> >> >> > >> >>> zookeeper.session.timeout value doesn't take into
> > effect.
> > > I
> > > >> >> still
> > > >> >> >> > >> found
> > > >> >> >> > >> >>> around 60 seconds pauses from the disconnected
> > > >> regionserver. I
> > > >> >> >> > really
> > > >> >> >> > >> don't
> > > >> >> >> > >> >>> know why regionserver only times out after 60 seconds
> > > >> instead
> > > >> >> of
> > > >> >> >> 10
> > > >> >> >> > >> minutes
> > > >> >> >> > >> >>> which I set for zookeeper.session.timeout.
> > > >> >> >> > >> >>>
> > > >> >> >> > >> >>> Is there any other timeout value coming into play
> > before
> > > >> the
> > > >> >> >> actual
> > > >> >> >> > >> >>> session times out?
> > > >> >> >> > >> >>>
> > > >> >> >> > >> >>> zhenyu
> > > >> >> >> > >> >>>
> > > >> >> >> > >> >>>
> > > >> >> >> > >> >>>
> > > >> >> >> > >> >>> On Fri, Nov 13, 2009 at 7:08 PM, stack <
> > stack@duboce.net
> > > >
> > > >> >> wrote:
> > > >> >> >> > >> >>>
> > > >> >> >> > >> >>>> Ok. Lack of file descriptors manifests in all kinds
> > of
> > > >> weird
> > > >> >> >> ways.
> > > >> >> >> > >> >>>> Hopefully thats it. If not, lets keep digging.
> > > >> >> >> > >> >>>> St.Ack
> > > >> >> >> > >> >>>>
> > > >> >> >> > >> >>>> On Fri, Nov 13, 2009 at 3:44 PM, Zhenyu Zhong <
> > > >> >> >> > >> zhongresearch@gmail.com
> > > >> >> >> > >> >>>> >wrote:
> > > >> >> >> > >> >>>>
> > > >> >> >> > >> >>>> > Stack,
> > > >> >> >> > >> >>>> >
> > > >> >> >> > >> >>>> > You are right, the master started with ulimit -n
> > 1024.
> > > >> It
> > > >> >> >> doesn't
> > > >> >> >> > >> take
> > > >> >> >> > >> >>>> the
> > > >> >> >> > >> >>>> > system value.
> > > >> >> >> > >> >>>> >
> > > >> >> >> > >> >>>> > Regarding to the too many open files, it looks
> like
> > > the
> > > >> >> same
> > > >> >> >> as
> > > >> >> >> > the
> > > >> >> >> > >> one
> > > >> >> >> > >> >>>> J-D
> > > >> >> >> > >> >>>> > put up. But I will get the Master start with
> higher
> > > >> value
> > > >> >> >> first
> > > >> >> >> > and
> > > >> >> >> > >> see
> > > >> >> >> > >> >>>> if
> > > >> >> >> > >> >>>> > this kind of error goes away.
> > > >> >> >> > >> >>>> >
> > > >> >> >> > >> >>>> > thanks a lot!
> > > >> >> >> > >> >>>> > zhenyu
> > > >> >> >> > >> >>>> >
> > > >> >> >> > >> >>>> > On Fri, Nov 13, 2009 at 6:02 PM, stack <
> > > >> stack@duboce.net>
> > > >> >> >> wrote:
> > > >> >> >> > >> >>>> >
> > > >> >> >> > >> >>>> > > Does it say
> > > >> >> >> > >> >>>> > >
> > > >> >> >> > >> >>>> > > ulimit -n 32768
> > > >> >> >> > >> >>>> > >
> > > >> >> >> > >> >>>> > > ...as the second line in your log file on start
> of
> > > the
> > > >> >> >> master?
> > > >> >> >> > >> >>>> > >
> > > >> >> >> > >> >>>> > > Is the stack trace that complains about too many
> > > open
> > > >> >> files
> > > >> >> >> > same
> > > >> >> >> > >> as
> > > >> >> >> > >> >>>> the
> > > >> >> >> > >> >>>> > one
> > > >> >> >> > >> >>>> > > in the blog post J-D put up?
> > > >> >> >> > >> >>>> > >
> > > >> >> >> > >> >>>> > > St.Ack
> > > >> >> >> > >> >>>> > >
> > > >> >> >> > >> >>>> > >
> > > >> >> >> > >> >>>> > > On Fri, Nov 13, 2009 at 1:37 PM, Zhenyu Zhong <
> > > >> >> >> > >> >>>> zhongresearch@gmail.com
> > > >> >> >> > >> >>>> > > >wrote:
> > > >> >> >> > >> >>>> > >
> > > >> >> >> > >> >>>> > > > The ulimit file descriptors was set to
> > fs.file-max
> > > >> >> >> =1578334,
> > > >> >> >> > >> also
> > > >> >> >> > >> >>>> in
> > > >> >> >> > >> >>>> > the
> > > >> >> >> > >> >>>> > > > limits.conf the value was set to 32768.
> > > >> >> >> > >> >>>> > > > So these are way higher than the open
> > descriptors
> > > >> for
> > > >> >> the
> > > >> >> >> > >> running
> > > >> >> >> > >> >>>> > > > processes.
> > > >> >> >> > >> >>>> > > >
> > > >> >> >> > >> >>>> > > > thanks
> > > >> >> >> > >> >>>> > > > zhenyu
> > > >> >> >> > >> >>>> > > >
> > > >> >> >> > >> >>>> > > >
> > > >> >> >> > >> >>>> > > > On Fri, Nov 13, 2009 at 4:33 PM, Stack <
> > > >> >> >> saint.ack@gmail.com>
> > > >> >> >> > >> >>>> wrote:
> > > >> >> >> > >> >>>> > > >
> > > >> >> >> > >> >>>> > > > > You upped the ulimit file descriptors as per
> > the
> > > >> >> getting
> > > >> >> >> > >> started
> > > >> >> >> > >> >>>> doc?
> > > >> >> >> > >> >>>> > > > >
> > > >> >> >> > >> >>>> > > > >
> > > >> >> >> > >> >>>> > > > >
> > > >> >> >> > >> >>>> > > > > On Nov 13, 2009, at 1:26 PM, Zhenyu Zhong <
> > > >> >> >> > >> >>>> zhongresearch@gmail.com>
> > > >> >> >> > >> >>>> > > > wrote:
> > > >> >> >> > >> >>>> > > > >
> > > >> >> >> > >> >>>> > > > > Thanks a lot.
> > > >> >> >> > >> >>>> > > > >>
> > > >> >> >> > >> >>>> > > > >>
> > > >> >> >> > >> >>>> > > > >> Bad news is my kernel is still 2.6.26.
> > > >> >> >> > >> >>>> > > > >> But it was not a problem before.
> > > >> >> >> > >> >>>> > > > >>
> > > >> >> >> > >> >>>> > > > >> Very strange.
> > > >> >> >> > >> >>>> > > > >>
> > > >> >> >> > >> >>>> > > > >> zhenyu
> > > >> >> >> > >> >>>> > > > >>
> > > >> >> >> > >> >>>> > > > >> On Fri, Nov 13, 2009 at 4:16 PM,
> Jean-Daniel
> > > >> Cryans
> > > >> >> <
> > > >> >> >> > >> >>>> > > > jdcryans@apache.org
> > > >> >> >> > >> >>>> > > > >> >wrote:
> > > >> >> >> > >> >>>> > > > >>
> > > >> >> >> > >> >>>> > > > >> Looks like
> > > >> >> >> > >> >>>> > > > >>>
> > > >> >> >> > >> >>>> > > > >>>
> > > >> >> >> > >> >>>> > > >
> > > >> >> >> > >> >>>> > >
> > > >> >> >> > >> >>>> >
> > > >> >> >> > >> >>>>
> > > >> >> >> > >>
> > > >> >> >> >
> > > >> >> >>
> > > >> >>
> > > >>
> > >
> >
> http://pero.blogs.aprilmayjune.org/2009/01/22/hadoop-and-linux-kernel-2627-epoll-limits/
> > > >> >> >> > >> >>>> > > > >>>
> > > >> >> >> > >> >>>> > > > >>> J-D
> > > >> >> >> > >> >>>> > > > >>>
> > > >> >> >> > >> >>>> > > > >>> On Fri, Nov 13, 2009 at 1:12 PM, Zhenyu
> > Zhong
> > > <
> > > >> >> >> > >> >>>> > > zhongresearch@gmail.com
> > > >> >> >> > >> >>>> > > > >
> > > >> >> >> > >> >>>> > > > >>> wrote:
> > > >> >> >> > >> >>>> > > > >>>
> > > >> >> >> > >> >>>> > > > >>>> Hi,
> > > >> >> >> > >> >>>> > > > >>>>
> > > >> >> >> > >> >>>> > > > >>>>
> > > >> >> >> > >> >>>> > > > >>>> After I re-organize the cluster, the
> > > experiment
> > > >> >> ran
> > > >> >> >> into
> > > >> >> >> > >> >>>> problem
> > > >> >> >> > >> >>>> > > > faster
> > > >> >> >> > >> >>>> > > > >>>>
> > > >> >> >> > >> >>>> > > > >>> than
> > > >> >> >> > >> >>>> > > > >>>
> > > >> >> >> > >> >>>> > > > >>>> before.
> > > >> >> >> > >> >>>> > > > >>>>
> > > >> >> >> > >> >>>> > > > >>>> Basically the changes are to use machines
> > > with
> > > >> >> less
> > > >> >> >> > >> resources
> > > >> >> >> > >> >>>> as
> > > >> >> >> > >> >>>> > > > >>>>
> > > >> >> >> > >> >>>> > > > >>> zookeeper
> > > >> >> >> > >> >>>> > > > >>>
> > > >> >> >> > >> >>>> > > > >>>> quorums and machines with more resources
> as
> > > >> >> >> > regionserver.
> > > >> >> >> > >> >>>> > > > >>>>
> > > >> >> >> > >> >>>> > > > >>>> From the log, I still see the pause
> around
> > 1
> > > >> >> minute.
> > > >> >> >> > >> >>>> > > > >>>> I enabled the GC logging, please see
> > > >> >> >> > >> >>>> > > > >>>>
> > > >> >> >> > >> >>>> > > > >>>> http://pastebin.com/m1d4ce0f1
> > > >> >> >> > >> >>>> > > > >>>>
> > > >> >> >> > >> >>>> > > > >>>> for details.
> > > >> >> >> > >> >>>> > > > >>>> In general I don't see many pauses in the
> > GC.
> > > >> >> >> > >> >>>> > > > >>>>
> > > >> >> >> > >> >>>> > > > >>>> What is more interesting, 1 hour after
> the
> > > 1st
> > > >> >> >> > >> regionserver
> > > >> >> >> > >> >>>> > > > >>>> disconnected,
> > > >> >> >> > >> >>>> > > > >>>> the master log complained about too many
> > open
> > > >> >> files.
> > > >> >> >> > This
> > > >> >> >> > >> >>>> didn't
> > > >> >> >> > >> >>>> > > > happen
> > > >> >> >> > >> >>>> > > > >>>> before.
> > > >> >> >> > >> >>>> > > > >>>> I checked the system OS setup as well as
> > the
> > > >> >> >> > limits.conf.
> > > >> >> >> > >> I
> > > >> >> >> > >> >>>> also
> > > >> >> >> > >> >>>> > > > checked
> > > >> >> >> > >> >>>> > > > >>>>
> > > >> >> >> > >> >>>> > > > >>> the
> > > >> >> >> > >> >>>> > > > >>>
> > > >> >> >> > >> >>>> > > > >>>> running processes. The total open files
> > don't
> > > >> >> reach
> > > >> >> >> the
> > > >> >> >> > >> limit.
> > > >> >> >> > >> >>>> I
> > > >> >> >> > >> >>>> > am
> > > >> >> >> > >> >>>> > > > >>>>
> > > >> >> >> > >> >>>> > > > >>> confused
> > > >> >> >> > >> >>>> > > > >>>
> > > >> >> >> > >> >>>> > > > >>>> a bit.
> > > >> >> >> > >> >>>> > > > >>>>
> > > >> >> >> > >> >>>> > > > >>>>
> > > >> >> >> > >> >>>> > > > >>>> Please see the following master log.
> > > >> >> >> > >> >>>> > > > >>>>
> > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:02,114 INFO
> > > >> >> >> > >> >>>> > > > org.apache.hadoop.hbase.master.BaseScanner:
> > > >> >> >> > >> >>>> > > > >>>> RegionManager.metaScanner scan of 4658
> > row(s)
> > > >> of
> > > >> >> meta
> > > >> >> >> > >> region
> > > >> >> >> > >> >>>> > > {server:
> > > >> >> >> > >> >>>> > > > >>>> 192.168.100.128:60021, regionname:
> > > .META.,,1,
> > > >> >> >> startKey:
> > > >> >> >> > >> <>}
> > > >> >> >> > >> >>>> > > complete
> > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:02,114 INFO
> > > >> >> >> > >> >>>> > > > org.apache.hadoop.hbase.master.BaseScanner:
> > > >> >> >> > >> >>>> > > > >>>>
> > > >> >> >> > >> >>>> > > > >>> All
> > > >> >> >> > >> >>>> > > > >>>
> > > >> >> >> > >> >>>> > > > >>>> 1 .META. region(s) scanned
> > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:07,677 DEBUG
> > > >> >> >> > >> org.apache.zookeeper.ClientCnxn:
> > > >> >> >> > >> >>>> Got
> > > >> >> >> > >> >>>> > > > ping
> > > >> >> >> > >> >>>> > > > >>>> response for sessionid:0x424eebf1c10004c
> > > after
> > > >> 3ms
> > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:08,178 INFO
> > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > >> >> >> > >> >>>> > > > Exception
> > > >> >> >> > >> >>>> > > > >>>>
> > > >> >> >> > >> >>>> > > > >>> in
> > > >> >> >> > >> >>>> > > > >>>
> > > >> >> >> > >> >>>> > > > >>>> createBlockOutputStream
> > java.io.IOException:
> > > >> Bad
> > > >> >> >> connect
> > > >> >> >> > >> ack
> > > >> >> >> > >> >>>> with
> > > >> >> >> > >> >>>> > > > >>>> firstBadLink 192.168.100.123:50010
> > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:08,178 INFO
> > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > >> >> >> > >> >>>> > > > >>>> Abandoning
> > > >> >> >> > >> >>>> > > > >>>> block blk_-2808245019291145247_5478039
> > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:09,682 INFO
> > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > >> >> >> > >> >>>> > > > Exception
> > > >> >> >> > >> >>>> > > > >>>>
> > > >> >> >> > >> >>>> > > > >>> in
> > > >> >> >> > >> >>>> > > > >>>
> > > >> >> >> > >> >>>> > > > >>>> createBlockOutputStream
> > java.io.EOFException
> > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:09,682 INFO
> > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > >> >> >> > >> >>>> > > > >>>> Abandoning
> > > >> >> >> > >> >>>> > > > >>>> block blk_1074853606841896259_5478048
> > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:10,334 DEBUG
> > > >> >> >> > >> org.apache.zookeeper.ClientCnxn:
> > > >> >> >> > >> >>>> Got
> > > >> >> >> > >> >>>> > > > ping
> > > >> >> >> > >> >>>> > > > >>>> response for sessionid:0x24eebf1043003c
> > after
> > > >> 1ms
> > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:21,018 DEBUG
> > > >> >> >> > >> org.apache.zookeeper.ClientCnxn:
> > > >> >> >> > >> >>>> Got
> > > >> >> >> > >> >>>> > > > ping
> > > >> >> >> > >> >>>> > > > >>>> response for sessionid:0x424eebf1c10004c
> > > after
> > > >> 0ms
> > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:23,674 DEBUG
> > > >> >> >> > >> org.apache.zookeeper.ClientCnxn:
> > > >> >> >> > >> >>>> Got
> > > >> >> >> > >> >>>> > > > ping
> > > >> >> >> > >> >>>> > > > >>>> response for sessionid:0x24eebf1043003c
> > after
> > > >> 0ms
> > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:24,828 INFO
> > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > >> >> >> > >> >>>> > > > Exception
> > > >> >> >> > >> >>>> > > > >>>>
> > > >> >> >> > >> >>>> > > > >>> in
> > > >> >> >> > >> >>>> > > > >>>
> > > >> >> >> > >> >>>> > > > >>>> createBlockOutputStream
> > java.io.IOException:
> > > >> Bad
> > > >> >> >> connect
> > > >> >> >> > >> ack
> > > >> >> >> > >> >>>> with
> > > >> >> >> > >> >>>> > > > >>>> firstBadLink 192.168.100.123:50010
> > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:24,828 INFO
> > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > >> >> >> > >> >>>> > > > >>>> Abandoning
> > > >> >> >> > >> >>>> > > > >>>> block blk_-6642544517082142289_5478063
> > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:24,828 INFO
> > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > >> >> >> > >> >>>> > > > Exception
> > > >> >> >> > >> >>>> > > > >>>>
> > > >> >> >> > >> >>>> > > > >>> in
> > > >> >> >> > >> >>>> > > > >>>
> > > >> >> >> > >> >>>> > > > >>>> createBlockOutputStream
> > > >> java.net.SocketException:
> > > >> >> Too
> > > >> >> >> > many
> > > >> >> >> > >> >>>> open
> > > >> >> >> > >> >>>> > > files
> > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:24,828 INFO
> > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > >> >> >> > >> >>>> > > > >>>> Abandoning
> > > >> >> >> > >> >>>> > > > >>>> block blk_2057511041109796090_5478063
> > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:24,928 INFO
> > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > >> >> >> > >> >>>> > > > Exception
> > > >> >> >> > >> >>>> > > > >>>>
> > > >> >> >> > >> >>>> > > > >>> in
> > > >> >> >> > >> >>>> > > > >>>
> > > >> >> >> > >> >>>> > > > >>>> createBlockOutputStream
> > > >> java.net.SocketException:
> > > >> >> Too
> > > >> >> >> > many
> > > >> >> >> > >> >>>> open
> > > >> >> >> > >> >>>> > > files
> > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:24,928 INFO
> > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > >> >> >> > >> >>>> > > > >>>> Abandoning
> > > >> >> >> > >> >>>> > > > >>>> block blk_8219260302213892894_5478064
> > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:30,855 INFO
> > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > >> >> >> > >> >>>> > > > Exception
> > > >> >> >> > >> >>>> > > > >>>>
> > > >> >> >> > >> >>>> > > > >>> in
> > > >> >> >> > >> >>>> > > > >>>
> > > >> >> >> > >> >>>> > > > >>>> createBlockOutputStream
> > > >> java.net.SocketException:
> > > >> >> Too
> > > >> >> >> > many
> > > >> >> >> > >> >>>> open
> > > >> >> >> > >> >>>> > > files
> > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:30,855 INFO
> > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > >> >> >> > >> >>>> > > > >>>> Abandoning
> > > >> >> >> > >> >>>> > > > >>>> block blk_1669205542853067709_5478235
> > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:30,905 INFO
> > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > >> >> >> > >> >>>> > > > Exception
> > > >> >> >> > >> >>>> > > > >>>>
> > > >> >> >> > >> >>>> > > > >>> in
> > > >> >> >> > >> >>>> > > > >>>
> > > >> >> >> > >> >>>> > > > >>>> createBlockOutputStream
> > > >> java.net.SocketException:
> > > >> >> Too
> > > >> >> >> > many
> > > >> >> >> > >> >>>> open
> > > >> >> >> > >> >>>> > > files
> > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:30,905 INFO
> > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > >> >> >> > >> >>>> > > > >>>> Abandoning
> > > >> >> >> > >> >>>> > > > >>>> block blk_9128897691346270351_5478237
> > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:30,955 INFO
> > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > >> >> >> > >> >>>> > > > Exception
> > > >> >> >> > >> >>>> > > > >>>>
> > > >> >> >> > >> >>>> > > > >>> in
> > > >> >> >> > >> >>>> > > > >>>
> > > >> >> >> > >> >>>> > > > >>>> createBlockOutputStream
> > > >> java.net.SocketException:
> > > >> >> Too
> > > >> >> >> > many
> > > >> >> >> > >> >>>> open
> > > >> >> >> > >> >>>> > > files
> > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:30,955 INFO
> > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > >> >> >> > >> >>>> > > > >>>> Abandoning
> > > >> >> >> > >> >>>> > > > >>>> block blk_1116845144864123018_5478240
> > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:34,372 DEBUG
> > > >> >> >> > >> org.apache.zookeeper.ClientCnxn:
> > > >> >> >> > >> >>>> Got
> > > >> >> >> > >> >>>> > > > ping
> > > >> >> >> > >> >>>> > > > >>>> response for sessionid:0x424eebf1c10004c
> > > after
> > > >> 0ms
> > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:37,034 DEBUG
> > > >> >> >> > >> org.apache.zookeeper.ClientCnxn:
> > > >> >> >> > >> >>>> Got
> > > >> >> >> > >> >>>> > > > ping
> > > >> >> >> > >> >>>> > > > >>>> response for sessionid:0x24eebf1043003c
> > after
> > > >> 0ms
> > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:37,235 WARN
> > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > >> >> >> > >> >>>> > > > >>>>
> > > >> >> >> > >> >>>> > > > >>> DataStreamer
> > > >> >> >> > >> >>>> > > > >>>
> > > >> >> >> > >> >>>> > > > >>>> Exception: java.io.IOException: Too many
> > open
> > > >> >> files
> > > >> >> >> > >> >>>> > > > >>>> at sun.nio.ch.IOUtil.initPipe(Native
> > > >> Method)
> > > >> >> >> > >> >>>> > > > >>>> at
> > > >> >> >> > >> >>>> > >
> > > >> >> >> sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:49)
> > > >> >> >> > >> >>>> > > > >>>> at
> > > >> >> >> > >> >>>> > > > >>>>
> > > >> >> >> > >> >>>> > > > >>>>
> > > >> >> >> > >> >>>> > > >
> > > >> >> >> > >> >>>> > >
> > > >> >> >> > >> >>>> >
> > > >> >> >> > >> >>>>
> > > >> >> >> > >>
> > > >> >> >> >
> > > >> >> >>
> > > >> >>
> > > >>
> > >
> >
> sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:18)
> > > >> >> >> > >> >>>> > > > >>>
> > > >> >> >> > >> >>>> > > > >>>> at
> > > >> >> >> > >> >>>> > > > >>>>
> > > >> >> >> > >> >>>> > > > >>>>
> > > >> >> >> > >> >>>> > > >
> > > >> >> >> > >> >>>> > >
> > > >> >> >> > >> >>>> >
> > > >> >> >> > >> >>>>
> > > >> >> >> > >>
> > > >> >> >> >
> > > >> >> >>
> > > >> >>
> > > >>
> > >
> >
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.get(SocketIOWithTimeout.java:407)
> > > >> >> >> > >> >>>> > > > >>>
> > > >> >> >> > >> >>>> > > > >>>> at
> > > >> >> >> > >> >>>> > > > >>>>
> > > >> >> >> > >> >>>> > > > >>>>
> > > >> >> >> > >> >>>> > > >
> > > >> >> >> > >> >>>> > >
> > > >> >> >> > >> >>>> >
> > > >> >> >> > >> >>>>
> > > >> >> >> > >>
> > > >> >> >> >
> > > >> >> >>
> > > >> >>
> > > >>
> > >
> >
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:322)
> > > >> >> >> > >> >>>> > > > >>>
> > > >> >> >> > >> >>>> > > > >>>> at
> > > >> >> >> > >> >>>> > > > >>>>
> > > >> >> >> > >> >>>> > > > >>>>
> > > >> >> >> > >> >>>> > > >
> > > >> >> >> > >> >>>> > >
> > > >> >> >> > >> >>>> >
> > > >> >> >> > >> >>>>
> > > >> >> >> > >>
> > > >> >> >> >
> > > >> >> >>
> > > >> >>
> > > >>
> > >
> >
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
> > > >> >> >> > >> >>>> > > > >>>
> > > >> >> >> > >> >>>> > > > >>>> at
> > > >> >> >> > >> >>>> > > > >>>>
> > > >> >> >> > >> >>>> > > > >>>>
> > > >> >> >> > >> >>>> > > >
> > > >> >> >> > >> >>>> > >
> > > >> >> >> > >> >>>> >
> > > >> >> >> > >> >>>>
> > > >> >> >> > >>
> > > >> >> >> >
> > > >> >> >>
> > > >> >>
> > > >>
> > >
> >
> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
> > > >> >> >> > >> >>>> > > > >>>
> > > >> >> >> > >> >>>> > > > >>>> at
> > > >> >> >> > >> >>>> > > > >>>>
> > > >> >> >> > >> >>>> > > > >>>>
> > > >> >> >> > >> >>>> > > >
> > > >> >> >> > >> >>>> > >
> > > >> >> >> > >> >>>> >
> > > >> >> >> > >> >>>>
> > > >> >> >> > >>
> > > >> >> >> >
> > > >> >> >>
> > > >> >>
> > > >>
> > >
> >
> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
> > > >> >> >> > >> >>>> > > > >>>
> > > >> >> >> > >> >>>> > > > >>>> at
> > > >> >> >> > >> >>>> > > > >>>>
> > > >> >> >> > >> >>>> > > > >>>
> > > >> >> >> > >> >>>>
> > > >> >> >>
> > java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
> > > >> >> >> > >> >>>> > > > >>>
> > > >> >> >> > >> >>>> > > > >>>> at
> > > >> >> >> > >> >>>>
> > java.io.DataOutputStream.write(DataOutputStream.java:90)
> > > >> >> >> > >> >>>> > > > >>>> at
> > > >> >> >> > >> >>>> > > > >>>>
> > > >> >> >> > >> >>>> > > > >>>>
> > > >> >> >> > >> >>>> > > >
> > > >> >> >> > >> >>>> > >
> > > >> >> >> > >> >>>> >
> > > >> >> >> > >> >>>>
> > > >> >> >> > >>
> > > >> >> >> >
> > > >> >> >>
> > > >> >>
> > > >>
> > >
> >
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2290)
> > > >> >> >> > >> >>>> > > > >>>
> > > >> >> >> > >> >>>> > > > >>>>
> > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:37,235 WARN
> > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > >> >> >> > >> >>>> > Error
> > > >> >> >> > >> >>>> > > > >>>> Recovery for block
> > > >> blk_8148813491785406356_5478475
> > > >> >> >> bad
> > > >> >> >> > >> >>>> datanode[0]
> > > >> >> >> > >> >>>> > > > >>>> 192.168.100.123:50010
> > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:37,235 WARN
> > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > >> >> >> > >> >>>> > Error
> > > >> >> >> > >> >>>> > > > >>>> Recovery for block
> > > >> blk_8148813491785406356_5478475
> > > >> >> in
> > > >> >> >> > >> pipeline
> > > >> >> >> > >> >>>> > > > >>>> 192.168.100.123:50010,
> > 192.168.100.134:50010
> > > ,
> > > >> >> >> > >> >>>> > 192.168.100.122:50010
> > > >> >> >> > >> >>>> > > :
> > > >> >> >> > >> >>>> > > > >>>> bad
> > > >> >> >> > >> >>>> > > > >>>> datanode 192.168.100.123:50010
> > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:37,436 INFO
> > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > >> >> >> > >> >>>> > > > Exception
> > > >> >> >> > >> >>>> > > > >>>>
> > > >> >> >> > >> >>>> > > > >>> in
> > > >> >> >> > >> >>>> > > > >>>
> > > >> >> >> > >> >>>> > > > >>>> createBlockOutputStream
> > > >> java.net.SocketException:
> > > >> >> Too
> > > >> >> >> > many
> > > >> >> >> > >> >>>> open
> > > >> >> >> > >> >>>> > > files
> > > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:37,436 INFO
> > > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > > >> >> >> > >> >>>> > > > >>>> Abandoning
> > > >> >> >> > >> >>>> > > > >>>> block blk_2119727700857186236_5478498
> > > >> >> >> > >> >>>> > > > >>>>
> > > >> >> >> > >> >>>> > > > >>>>
> > > >> >> >> > >> >>>> > > > >>>>
> > > >> >> >> > >> >>>> > > > >>>>
> > > >> >> >> > >> >>>> > > > >>>>
> > > >> >> >> > >> >>>> > > > >>>> On Thu, Nov 12, 2009 at 4:21 PM, Zhenyu
> > Zhong
> > > <
> > > >> >> >> > >> >>>> > > > zhongresearch@gmail.com
> > > >> >> >> > >> >>>> > > > >>>> wrote:
> > > >> >> >> > >> >>>> > > > >>>>
> > > >> >> >> > >> >>>> > > > >>>> Will do.
> > > >> >> >> > >> >>>> > > > >>>>>
> > > >> >> >> > >> >>>> > > > >>>>> thanks
> > > >> >> >> > >> >>>> > > > >>>>> zhenyu
> > > >> >> >> > >> >>>> > > > >>>>>
> > > >> >> >> > >> >>>> > > > >>>>>
> > > >> >> >> > >> >>>> > > > >>>>> On Thu, Nov 12, 2009 at 3:33 PM, stack <
> > > >> >> >> > stack@duboce.net
> > > >> >> >> > >> >
> > > >> >> >> > >> >>>> wrote:
> > > >> >> >> > >> >>>> > > > >>>>>
> > > >> >> >> > >> >>>> > > > >>>>> Enable GC logging too on this next run
> > (see
> > > >> >> >> > >> hbase-env.sh).
> > > >> >> >> > >> >>>> Lets
> > > >> >> >> > >> >>>> > > try
> > > >> >> >> > >> >>>> > > > >>>>>>
> > > >> >> >> > >> >>>> > > > >>>>> and
> > > >> >> >> > >> >>>> > > > >>>
> > > >> >> >> > >> >>>> > > > >>>> get
> > > >> >> >> > >> >>>> > > > >>>>>> to the bottom of whats going on.
> > > >> >> >> > >> >>>> > > > >>>>>> Thanks,
> > > >> >> >> > >> >>>> > > > >>>>>> St.Ack
> > > >> >> >> > >> >>>> > > > >>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>> On Thu, Nov 12, 2009 at 12:29 PM,
> Zhenyu
> > > >> Zhong <
> > > >> >> >> > >> >>>> > > > >>>>>>
> > > >> >> >> > >> >>>> > > > >>>>> zhongresearch@gmail.com
> > > >> >> >> > >> >>>> > > > >>>
> > > >> >> >> > >> >>>> > > > >>>> wrote:
> > > >> >> >> > >> >>>> > > > >>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>> I can switch the boxes that run
> > zookeeper
> > > >> with
> > > >> >> the
> > > >> >> >> > ones
> > > >> >> >> > >> >>>> that
> > > >> >> >> > >> >>>> > run
> > > >> >> >> > >> >>>> > > > >>>>>>> regionservers.
> > > >> >> >> > >> >>>> > > > >>>>>>> I will see the results later.
> > > >> >> >> > >> >>>> > > > >>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>> FYI. The node does have the 10 minutes
> > > >> >> >> > >> >>>> > zookeeper.session.timeout
> > > >> >> >> > >> >>>> > > > >>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>> value
> > > >> >> >> > >> >>>> > > > >>>
> > > >> >> >> > >> >>>> > > > >>>> in
> > > >> >> >> > >> >>>> > > > >>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>> place.
> > > >> >> >> > >> >>>> > > > >>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>> thanks
> > > >> >> >> > >> >>>> > > > >>>>>>> zhenyu
> > > >> >> >> > >> >>>> > > > >>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>> On Thu, Nov 12, 2009 at 3:21 PM, stack
> <
> > > >> >> >> > >> stack@duboce.net>
> > > >> >> >> > >> >>>> > wrote:
> > > >> >> >> > >> >>>> > > > >>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>> On Thu, Nov 12, 2009 at 11:50 AM,
> > Zhenyu
> > > >> Zhong
> > > >> >> <
> > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>> zhongresearch@gmail.com
> > > >> >> >> > >> >>>> > > > >>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>> wrote:
> > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>> In my cluster, half of the cluster
> > have
> > > 2
> > > >> >> disks
> > > >> >> >> > 400GB
> > > >> >> >> > >> >>>> each
> > > >> >> >> > >> >>>> > per
> > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>> machine,
> > > >> >> >> > >> >>>> > > > >>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>> and
> > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>> half of the cluster have 6 disks per
> > > >> machine.
> > > >> >> >> > Maybe
> > > >> >> >> > >> we
> > > >> >> >> > >> >>>> > should
> > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>> run
> > > >> >> >> > >> >>>> > > > >>>
> > > >> >> >> > >> >>>> > > > >>>> zookeeper
> > > >> >> >> > >> >>>> > > > >>>>>>>>> on the machines with 2 disks and RS
> on
> > > >> >> machines
> > > >> >> >> > with
> > > >> >> >> > >> 6
> > > >> >> >> > >> >>>> disks?
> > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>> That would make most sense only in
> the
> > > >> below,
> > > >> >> it
> > > >> >> >> > >> looks
> > > >> >> >> > >> >>>> like
> > > >> >> >> > >> >>>> > the
> > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>> RS
> > > >> >> >> > >> >>>> > > > >>>
> > > >> >> >> > >> >>>> > > > >>>> that
> > > >> >> >> > >> >>>> > > > >>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>> had
> > > >> >> >> > >> >>>> > > > >>>>>>>> issue had 4 disks?
> > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>> BTW, the 10 minutes
> > > >> zookeeper.session.timeout
> > > >> >> >> has
> > > >> >> >> > >> been
> > > >> >> >> > >> >>>> set
> > > >> >> >> > >> >>>> > > during
> > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>> the
> > > >> >> >> > >> >>>> > > > >>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>> experiment.
> > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>> And for sure this node had it in
> place?
> > > >> >> >> > >> >>>> > > > >>>>>>>> St.Ack
> > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>> thanks
> > > >> >> >> > >> >>>> > > > >>>>>>>>> zhenyu
> > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>> On Thu, Nov 12, 2009 at 2:08 PM,
> stack
> > <
> > > >> >> >> > >> stack@duboce.net
> > > >> >> >> > >> >>>> >
> > > >> >> >> > >> >>>> > > wrote:
> > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>> On Thu, Nov 12, 2009 at 8:40 AM,
> > Zhenyu
> > > >> >> Zhong <
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>> zhongresearch@gmail.com
> > > >> >> >> > >> >>>> > > > >>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>> wrote:
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>> Though I experienced 2
> regionserver
> > > >> >> >> disconnection
> > > >> >> >> > >> this
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>> morning,
> > > >> >> >> > >> >>>> > > > >>>
> > > >> >> >> > >> >>>> > > > >>>> it
> > > >> >> >> > >> >>>> > > > >>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>> looks
> > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>> better from the regionserver log.
> > > (please
> > > >> >> see
> > > >> >> >> the
> > > >> >> >> > >> >>>> following
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>> log)
> > > >> >> >> > >> >>>> > > > >>>
> > > >> >> >> > >> >>>> > > > >>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>> http://pastebin.com/m496dbfae
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>> I checked diskIO for one of the
> > > >> >> >> > >> >>>> > regionserver(192.168.100.116)
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>> during
> > > >> >> >> > >> >>>> > > > >>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>> the
> > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>> time it disconnected.
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>> Time: 03:04:58 AM
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>> Device: tps
> Blk_read/s
> > > >> >> >> Blk_wrtn/s
> > > >> >> >> > >> >>>> Blk_read
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>> Blk_wrtn
> > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>> sda 105.31 5458.83
> > > >> >> 19503.64
> > > >> >> >> > >> >>>> 9043873239
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>> 32312473676
> > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>> sda1 2.90 6.64
> > > >> >> 99.25
> > > >> >> >> > >> >>>> 10993934
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>> 164433464
> > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>> sda2 1.72 23.76
> > > >> >> 30.25
> > > >> >> >> > >> >>>> 39365817
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>> 50124008
> > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>> sda3 0.30 0.38
> > > >> >> 3.58
> > > >> >> >> > >> >>>> 624930
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>> 5923000
> > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>> sda4 100.39 5428.06
> > > >> >> 19370.56
> > > >> >> >> > >> >>>> 8992888270
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>> 32091993204
> > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>> Is this high for you? 20k
> > > >> blocks/second
> > > >> >> >> would
> > > >> >> >> > >> seem to
> > > >> >> >> > >> >>>> be
> > > >> >> >> > >> >>>> > > high
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>> but
> > > >> >> >> > >> >>>> > > > >>>
> > > >> >> >> > >> >>>> > > > >>>> its
> > > >> >> >> > >> >>>> > > > >>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>> one
> > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>> disk only and its not being shared
> by
> > > zk
> > > >> >> >> anymore
> > > >> >> >> > so
> > > >> >> >> > >> >>>> > shouldn't
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>> matter?
> > > >> >> >> > >> >>>> > > > >>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>> I also checked the zookeeper
> quorum
> > > >> server
> > > >> >> >> that
> > > >> >> >> > the
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>> regionserver
> > > >> >> >> > >> >>>> > > > >>>
> > > >> >> >> > >> >>>> > > > >>>> tried
> > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>> to
> > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>> connect according to the log.
> > However,
> > > I
> > > >> >> don't
> > > >> >> >> see
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>> 192.168.100.116
> > > >> >> >> > >> >>>> > > > >>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>> in
> > > >> >> >> > >> >>>> > > > >>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>> the
> > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>> client list of the zookeeper quorum
> > > >> server
> > > >> >> that
> > > >> >> >> > the
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>> regionserver
> > > >> >> >> > >> >>>> > > > >>>
> > > >> >> >> > >> >>>> > > > >>>> tried
> > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>> to
> > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>> connect.
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>> Would that be a problem?
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>> Is that because the ephemeral
> node
> > > for
> > > >> the
> > > >> >> >> > >> >>>> regionserver
> > > >> >> >> > >> >>>> > had
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>> evaporated?
> > > >> >> >> > >> >>>> > > > >>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>> Lost
> > > >> >> >> > >> >>>> > > > >>>>>>>>>> its lease w/ zk by the time you
> went
> > to
> > > >> >> look?
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>> Thu Nov 12 15:04:52 UTC 2009
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>> Zookeeper version: 3.2.1-808558,
> > built
> > > >> on
> > > >> >> >> > >> 08/27/2009
> > > >> >> >> > >> >>>> 18:48
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>> GMT
> > > >> >> >> > >> >>>> > > > >>>
> > > >> >> >> > >> >>>> > > > >>>> Clients:
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.127:43045
> > > >> >> >> > >> [1](queued=0,recved=26,sent=0)
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.131:39091
> > > >> >> >> > >> [1](queued=0,recved=964,sent=0)
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.124:35961
> > > >> >> >> > >> [1](queued=0,recved=3266,sent=0)
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.123:47935
> > > >> >> >> > >> [1](queued=0,recved=233,sent=0)
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.125:46931
> > > >> >> >> > >> [1](queued=0,recved=2,sent=0)
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.118:54924
> > > >> >> >> > >> [1](queued=0,recved=295,sent=0)
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.118:41390
> > > >> >> >> > >> [1](queued=0,recved=2290,sent=0)
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.136:42243
> > > >> >> >> > >> [1](queued=0,recved=0,sent=0)
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>> Latency min/avg/max: 0/17/6333
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>> Received: 47111
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>> Sent: 0
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>> Outstanding: 0
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>> Zxid: 0x77000083f4
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>> Mode: leader
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>> Node count: 23
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>> That 6 second maximum latency is
> > pretty
> > > >> bad
> > > >> >> but
> > > >> >> >> > >> should
> > > >> >> >> > >> >>>> be
> > > >> >> >> > >> >>>> > well
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>> within
> > > >> >> >> > >> >>>> > > > >>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>> the
> > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>> zk
> > > >> >> >> > >> >>>> > > > >>>>>>>>>> session timeout.
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>> So, problem is likely on the zk
> > client
> > > >> side
> > > >> >> of
> > > >> >> >> the
> > > >> >> >> > >> >>>> session;
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>> i.e.
> > > >> >> >> > >> >>>> > > > >>>
> > > >> >> >> > >> >>>> > > > >>>> in
> > > >> >> >> > >> >>>> > > > >>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>> the
> > > >> >> >> > >> >>>> > > > >>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>> RS.
> > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>> You could enable GC logging as J-D
> > > >> suggested
> > > >> >> to
> > > >> >> >> > see
> > > >> >> >> > >> if
> > > >> >> >> > >> >>>> you
> > > >> >> >> > >> >>>> > > have
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>> any
> > > >> >> >> > >> >>>> > > > >>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>> big
> > > >> >> >> > >> >>>> > > > >>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>> pauses, pauses > zk session timeout.
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>> When the RS went down, it didn't
> look
> > > too
> > > >> >> >> heavily
> > > >> >> >> > >> >>>> loaded:
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>> 1. 2009-11-12 15:04:52,830 INFO
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > >> >> >> > org.apache.hadoop.hbase.regionserver.HRegionServer:
> > > >> >> >> > >> >>>> Dump of
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>> metrics:
> > > >> >> >> > >> >>>> > > > >>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>> request=1.5166667, regions=322,
> > > >> stores=657,
> > > >> >> >> > >> >>>> storefiles=631,
> > > >> >> >> > >> >>>> > > > >>>>>>>>>> storefileIndexSize=61,
> > > >> memstoreSize=1472,
> > > >> >> >> > >> >>>> usedHeap=2819,
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>> maxHeap=4079,
> > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>> blockCacheSize=658110960,
> > > >> >> >> > blockCacheFree=197395984,
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>> blockCacheCount=9903,
> > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>> blockCacheHitRatio=99
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>> Its serving a few reads? The
> number
> > of
> > > >> >> store
> > > >> >> >> > files
> > > >> >> >> > >> >>>> seems
> > > >> >> >> > >> >>>> > > fine.
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>> Not
> > > >> >> >> > >> >>>> > > > >>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>> too
> > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>> much memory used.
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>> Looking at the logs, I see the
> Lease
> > > >> Still
> > > >> >> Held
> > > >> >> >> > >> >>>> exception.
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>> This
> > > >> >> >> > >> >>>> > > > >>>
> > > >> >> >> > >> >>>> > > > >>>> happens
> > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>> when the RS does its regular report
> to
> > > the
> > > >> >> >> master
> > > >> >> >> > but
> > > >> >> >> > >> the
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>> master
> > > >> >> >> > >> >>>> > > > >>>
> > > >> >> >> > >> >>>> > > > >>>> thinks
> > > >> >> >> > >> >>>> > > > >>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>> the
> > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>> RS has since restarted. It'll
> think
> > > this
> > > >> >> >> probably
> > > >> >> >> > >> >>>> because
> > > >> >> >> > >> >>>> > it
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>> noticed
> > > >> >> >> > >> >>>> > > > >>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>> that
> > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>> the RS's znode in zk had gone away
> > and
> > > it
> > > >> >> >> > considered
> > > >> >> >> > >> the
> > > >> >> >> > >> >>>> RS
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>> dead.
> > > >> >> >> > >> >>>> > > > >>>
> > > >> >> >> > >> >>>> > > > >>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>> Looking too at your logs I see
> this
> > > gap
> > > >> in
> > > >> >> the
> > > >> >> >> zk
> > > >> >> >> > >> >>>> pinging:
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>> 1. 2009-11-12 15:03:39,325 DEBUG
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>> org.apache.zookeeper.ClientCnxn:
> > > >> >> >> > >> >>>> > > > >>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>> Got
> > > >> >> >> > >> >>>> > > > >>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>> ping response for
> > > >> sessionid:0x224e55436ad0004
> > > >> >> >> after
> > > >> >> >> > >> 0ms
> > > >> >> >> > >> >>>> > > > >>>>>>>>>> 2. 2009-11-12 15:03:43,113 DEBUG
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>> org.apache.zookeeper.ClientCnxn:
> > > >> >> >> > >> >>>> > > > >>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>> Got
> > > >> >> >> > >> >>>> > > > >>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>> ping response for
> > > >> sessionid:0x24e55436a0007d
> > > >> >> >> after
> > > >> >> >> > >> 0ms
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>> Where in the lines above it, its
> > > >> reporting
> > > >> >> >> about
> > > >> >> >> > >> every
> > > >> >> >> > >> >>>> ten
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>> seconds,
> > > >> >> >> > >> >>>> > > > >>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>> here
> > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>> there is a big gap.
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>> Do you have ganglia or something
> that
> > > >> will
> > > >> >> let
> > > >> >> >> you
> > > >> >> >> > >> look
> > > >> >> >> > >> >>>> more
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>> into
> > > >> >> >> > >> >>>> > > > >>>
> > > >> >> >> > >> >>>> > > > >>>> what
> > > >> >> >> > >> >>>> > > > >>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>> was
> > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>> happening on this machine around
> this
> > > >> time?
> > > >> >> Is
> > > >> >> >> > the
> > > >> >> >> > >> >>>> machine
> > > >> >> >> > >> >>>> > > OK?
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>> It
> > > >> >> >> > >> >>>> > > > >>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>> looks
> > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>> lightly loaded and you have your
> > cluster
> > > >> >> nicely
> > > >> >> >> > laid
> > > >> >> >> > >> out.
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>> Something
> > > >> >> >> > >> >>>> > > > >>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>> odd
> > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>> is
> > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>> going on. What about things like
> the
> > > >> write
> > > >> >> >> speed
> > > >> >> >> > to
> > > >> >> >> > >> >>>> disk?
> > > >> >> >> > >> >>>> > In
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>> the
> > > >> >> >> > >> >>>> > > > >>>
> > > >> >> >> > >> >>>> > > > >>>> past
> > > >> >> >> > >> >>>> > > > >>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>> strange issues have been explained by
> > > >> >> incorrectly
> > > >> >> >> > set
> > > >> >> >> > >> BIOS
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>> which
> > > >> >> >> > >> >>>> > > > >>>
> > > >> >> >> > >> >>>> > > > >>>> made
> > > >> >> >> > >> >>>> > > > >>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>> disks
> > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>> run at 1/100th of their proper
> speed.
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>> St.Ack
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>> Best,
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>> zhenyu
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>> On Wed, Nov 11, 2009 at 3:58 PM,
> > > Zhenyu
> > > >> >> Zhong
> > > >> >> >> <
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>> zhongresearch@gmail.com
> > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>> wrote:
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>> Stack
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>>> I am very appreciated for your
> > > >> comments.
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>>> I will use the zookeeper
> monitoring
> > > >> script
> > > >> >> on
> > > >> >> >> my
> > > >> >> >> > >> >>>> cluster
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>> and
> > > >> >> >> > >> >>>> > > > >>>
> > > >> >> >> > >> >>>> > > > >>>> let
> > > >> >> >> > >> >>>> > > > >>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>> it
> > > >> >> >> > >> >>>> > > > >>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>> run
> > > >> >> >> > >> >>>> > > > >>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>> overnight to see the result.
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>>> I will follow up the thread when
> I
> > > get
> > > >> >> >> anything.
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>>> thanks
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>>> zhenyu
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>>> On Wed, Nov 11, 2009 at 3:52 PM,
> > > stack
> > > >> <
> > > >> >> >> > >> >>>> stack@duboce.net>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>> wrote:
> > > >> >> >> > >> >>>> > > > >>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>>> I see these in your log too:
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>> 1. 2009-11-11 04:27:19,018
> DEBUG
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
> > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>> Got
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>> ping response for
> > > >> >> sessionid:0x424dfd908c50009
> > > >> >> >> > >> after
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>>> 4544ms
> > > >> >> >> > >> >>>> > > > >>>
> > > >> >> >> > >> >>>> > > > >>>> 2. 2009-11-11 04:27:19,018 DEBUG
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
> > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>> Got
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>> ping response for
> > > >> >> sessionid:0x24dfd90c810002
> > > >> >> >> > after
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>>> 4548ms
> > > >> >> >> > >> >>>> > > > >>>
> > > >> >> >> > >> >>>> > > > >>>> 3. 2009-11-11 04:27:43,960 DEBUG
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
> > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>> Got
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>> ping response for
> > > >> >> sessionid:0x424dfd908c50009
> > > >> >> >> > >> after
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>>> 9030ms
> > > >> >> >> > >> >>>> > > > >>>
> > > >> >> >> > >> >>>> > > > >>>> 4. 2009-11-11 04:27:43,960 DEBUG
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>>>>>> org.apache.zookeeper.
> > > >> >> >> > >> >>>> > > > >>>>>>>>
> > > >> >> >> > >> >>>> > > > >>>>>>>
> > > >> >> >> > >> >>>> > > >
> > > >> >> >> > >> >>>> > >
> > > >> >> >> > >> >>>> >
> > > >> >> >> > >> >>>>
> > > >> >> >> > >> >>>
> > > >> >> >> > >> >>>
> > > >> >> >> > >> >>
> > > >> >> >> > >> >
> > > >> >> >> > >>
> > > >> >> >> > >
> > > >> >> >> > >
> > > >> >> >> >
> > > >> >> >>
> > > >> >> >
> > > >> >> >
> > > >> >>
> > > >> >
> > > >> >
> > > >>
> > > >
> > > >
> > >
> >
>
Re: regionserver disconnection
Posted by Zhenyu Zhong <zh...@gmail.com>.
Here is the zookeeper leader log. It had an unexpected exception. I don't
know why.
http://pastebin.com/m6594ccc
During that time, I don't think the system load is high. Memory usage should
be normal.
Best,
zhenyu
On Thu, Nov 19, 2009 at 11:09 AM, stack <st...@duboce.net> wrote:
> What is in the other zk logs? Why did the leader go away? Memory issue or
> something? Why'd it shutdown? Thanks for pastebin'ing this stuff.
> St.Ack
>
> On Thu, Nov 19, 2009 at 7:53 AM, Zhenyu Zhong <zhongresearch@gmail.com
> >wrote:
>
> > After digging more on the zookeeper log, I found that there seemed to be
> a
> > socketconnection timeout. Also some warnings indicate that zookeeper
> server
> > is not running. Probably I need to understand the mechanism of zookeeper
> > first.
> >
> > Please find the zookeeper log around that time.
> >
> > http://pastebin.com/m37bb4ad1
> >
> > Many thanks
> > zhenyu
> >
> >
> > On Thu, Nov 19, 2009 at 10:31 AM, Zhenyu Zhong <zhongresearch@gmail.com
> > >wrote:
> >
> > > Stack,
> > >
> > > I am very appreciated. Let me dig into the zookeeper log more.
> > >
> > > FYI, I saw the sleeper complained about the delay for the first time.
> It
> > > looks like there are different reasons for the RS disconnections.
> > >
> > > 2009-11-19 08:08:32,708 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
> > > response for sessionid:0x42508716bfc0001 after 0ms
> > > 2009-11-19 08:08:37,595 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
> > > response for sessionid:0x42508716bfc0082 after 0ms
> > > 2009-11-19 08:08:46,073 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
> > > response for sessionid:0x42508716bfc0001 after 0ms
> > > 2009-11-19 08:09:39,880 WARN org.apache.zookeeper.ClientCnxn: Exception
> > > closing session 0x42508716bfc0001 to
> sun.nio.ch.SelectionKeyImpl@4f3a608f
> > > java.io.IOException: TIMED OUT
> > > at
> > > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:906)
> > > 2009-11-19 08:09:39,880 WARN org.apache.zookeeper.ClientCnxn: Exception
> > > closing session 0x42508716bfc0082 to
> sun.nio.ch.SelectionKeyImpl@14c8875
> > > java.io.IOException: TIMED OUT
> > > at
> > > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:906)
> > > 2009-11-19 08:09:39,880 WARN org.apache.hadoop.hbase.util.Sleeper: We
> > slept
> > > 53556ms, ten times longer than scheduled: 3000
> > > 2009-11-19 08:09:39,980 WARN
> > > org.apache.hadoop.hbase.regionserver.HRegionServer: Attempt=1
> > > org.apache.hadoop.hbase.Leases$LeaseStillHeldException
> > > at
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> > > Method)
> > > at
> > >
> >
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> > > at
> > >
> >
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> > > at
> > java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> > > at
> > >
> >
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
> > > at
> > >
> >
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
> > > at
> > >
> >
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
> > > at
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:571)
> > > at java.lang.Thread.run(Thread.java:619)
> > > 2009-11-19 08:09:39,980 INFO
> > > org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper
> event,
> > > state: Disconnected, type: None, path: null
> > > 2009-11-19 08:09:40,530 INFO org.apache.zookeeper.ClientCnxn:
> Attempting
> > > connection to server superpyxis0005/192.168.100.119:2181
> > > 2009-11-19 08:09:40,530 INFO org.apache.zookeeper.ClientCnxn: Priming
> > > connection to java.nio.channels.SocketChannel[connected local=/
> > > 192.168.100.122:40659 remote=superpyxis0005/192.168.100.119:2181]
> > >
> > >
> > >
> > > Best,
> > > zhenyu
> > >
> > > On Wed, Nov 18, 2009 at 6:52 PM, stack <st...@duboce.net> wrote:
> > >
> > >> Patrick Hunt, one of the zk lads, answered with the following:
> > >>
> > >>
> > >>
> > >>
> > >> According to the following snippet (I cut to just session
> > >> 0x425035c48210002)
> > >> the client is getting pings for this session id regularly every 6
> > seconds
> > >> or
> > >> so, then after 41,016 there's a 20 second gap (corresponds to a 30
> > second
> > >> timeout being set during session creation) at which point we
> disconnect
> > >> from
> > >> the server.
> > >>
> > >>
> > >> > 2009-11-18 12:41:34,788 DEBUG org.apache.zookeeper.ClientCnxn: Got
> > ping
> > >> > response for sessionid:0x425035c48210002 after 0ms
> > >> > 2009-11-18 12:42:01,482 WARN org.apache.zookeeper.ClientCnxn:
> > Exception
> > >> > closing session 0x425035c48210002 to
> > >> sun.nio.ch.SelectionKeyImpl@421690ab
> > >> > java.io.IOException: TIMED OUT
> > >>
> > >> the client notifies hbase of the disco, and attempts to reconnect to a
> > >> server
> > >>
> > >>
> > >> > org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper
> > event,
> > >> > state: Disconnected, type: None, path: null
> > >> > 2009-11-18 12:42:01,782 INFO org.apache.zookeeper.ClientCnxn:
> > Attempting
> > >> > connection to server superpyxis0003/192.168.100.117:2181
> > >>
> > >> but the server won't allow the session be be established for some
> reason
> > >> (is
> > >> that the right server:port that's it's trying to connect to?)
> > >>
> > >>
> > >> > 2009-11-18 12:42:02,182 INFO org.apache.zookeeper.ClientCnxn: Server
> > >> > connection successful
> > >> > 2009-11-18 12:42:02,182 WARN org.apache.zookeeper.ClientCnxn:
> > Exception
> > >> > closing session 0x425035c48210002 to
> > >> sun.nio.ch.SelectionKeyImpl@5c07076b
> > >> > java.io.IOException: Read error rc = -1
> > java.nio.DirectByteBuffer[pos=0
> > >> > lim=4 cap=4]
> > >>
> > >> The "read error rc = -1" means that the client read from the socket
> and
> > >> got
> > >> EOS. But it's not a session expiration afaict (and should not be at
> this
> > >> point in time).
> > >>
> > >>
> > >> You need to look at the ZK server logs for this period in time (for
> this
> > >> server superpyxis0003:2181, and superpyxis0001:2181, the client tries
> to
> > >> connect to both of these). There should be some indication there why
> the
> > >> server is closing down the connection and not allowing the session to
> be
> > >> re-established.
> > >>
> > >> One thought - by default we limit clients to a max of 10 connections
> > from
> > >> a
> > >> particular host to the server. If you exceed this limit the zk server
> > >> (it's
> > >> per server though, not per cluster, so perhaps this is not it) will
> not
> > >> allow any additional connections from the client host (you can't
> exceed
> > >> 10
> > >> on a server). Could this be it? i.e. the user is running a number of
> ZK
> > >> clients (say hbase processes) from the same host? Regardless, get the
> > >> server
> > >> log for this time period and it should be more clear.
> > >>
> > >> Patrick
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> On Wed, Nov 18, 2009 at 11:18 AM, stack <st...@duboce.net> wrote:
> > >>
> > >> > And running stat on each of your zookeeper nodes tells you what
> about
> > >> > averages for connect-times?
> > >> >
> > >> > Its odd that there is a ping just 20 seconds beforehand and you have
> > set
> > >> > the zk session timeout at 60 seconds (or ten minutes?). Let me ask
> > the
> > >> zk
> > >> > lads about it.
> > >> >
> > >> > If you look in the GC log for around this time, do you see anything?
> > >> >
> > >> > St.Ack
> > >> >
> > >> >
> > >> >
> > >> > On Wed, Nov 18, 2009 at 7:32 AM, Zhenyu Zhong <
> > zhongresearch@gmail.com
> > >> >wrote:
> > >> >
> > >> >> After running more experiment, it seems that the disconnection is
> not
> > >> >> related to the heavy load job, because the load average is low, the
> > >> diskio
> > >> >> is normal, memory Heap was not reached, also the virtual memory
> stats
> > >> >> shows
> > >> >> that no swappings. However, the disconnection still happens.
> > >> >>
> > >> >> It looks like this time it pauses for 20 seconds. No idea why
> > >> regionserver
> > >> >> disconnected.
> > >> >>
> > >> >> Any other suggestions please?
> > >> >>
> > >> >>
> > >> >> 2009-11-18 12:41:00,956 DEBUG org.apache.zookeeper.ClientCnxn: Got
> > ping
> > >> >> response for sessionid:0x125035c495d02bb after 0ms
> > >> >> 2009-11-18 12:41:08,074 DEBUG org.apache.zookeeper.ClientCnxn: Got
> > ping
> > >> >> response for sessionid:0x425035c48210002 after 0ms
> > >> >> 2009-11-18 12:41:14,302 DEBUG org.apache.zookeeper.ClientCnxn: Got
> > ping
> > >> >> response for sessionid:0x125035c495d02bb after 0ms
> > >> >> 2009-11-18 12:41:21,433 DEBUG org.apache.zookeeper.ClientCnxn: Got
> > ping
> > >> >> response for sessionid:0x425035c48210002 after 0ms
> > >> >> 2009-11-18 12:41:27,656 DEBUG org.apache.zookeeper.ClientCnxn: Got
> > ping
> > >> >> response for sessionid:0x125035c495d02bb after 0ms
> > >> >> 2009-11-18 12:41:34,788 DEBUG org.apache.zookeeper.ClientCnxn: Got
> > ping
> > >> >> response for sessionid:0x425035c48210002 after 0ms
> > >> >> 2009-11-18 12:41:41,016 DEBUG org.apache.zookeeper.ClientCnxn: Got
> > ping
> > >> >> response for sessionid:0x125035c495d02bb after 1ms
> > >> >> 2009-11-18 12:42:01,482 WARN org.apache.zookeeper.ClientCnxn:
> > Exception
> > >> >> closing session 0x425035c48210002 to
> > >> sun.nio.ch.SelectionKeyImpl@421690ab
> > >> >> java.io.IOException: TIMED OUT
> > >> >> at
> > >> >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:906)
> > >> >> 2009-11-18 12:42:01,582 INFO
> > >> >> org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper
> > >> event,
> > >> >> state: Disconnected, type: None, path: null
> > >> >> 2009-11-18 12:42:01,782 INFO org.apache.zookeeper.ClientCnxn:
> > >> Attempting
> > >> >> connection to server superpyxis0003/192.168.100.117:2181
> > >> >> 2009-11-18 12:42:01,782 INFO org.apache.zookeeper.ClientCnxn:
> Priming
> > >> >> connection to java.nio.channels.SocketChannel[connected local=/
> > >> >> 192.168.100.132:46610 remote=superpyxis0003/192.168.100.117:2181]
> > >> >> 2009-11-18 12:42:01,782 INFO org.apache.zookeeper.ClientCnxn:
> Server
> > >> >> connection successful
> > >> >> 2009-11-18 12:42:01,782 WARN org.apache.zookeeper.ClientCnxn:
> > Exception
> > >> >> closing session 0x425035c48210002 to
> > >> sun.nio.ch.SelectionKeyImpl@56dc6fac
> > >> >> java.io.IOException: Read error rc = -1
> > java.nio.DirectByteBuffer[pos=0
> > >> >> lim=4 cap=4]
> > >> >> at
> > >> >>
> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
> > >> >> at
> > >> >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
> > >> >> 2009-11-18 12:42:01,782 WARN org.apache.zookeeper.ClientCnxn:
> > Ignoring
> > >> >> exception during shutdown input
> > >> >> java.net.SocketException: Transport endpoint is not connected
> > >> >> at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
> > >> >> at
> > >> >>
> > sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
> > >> >> at
> > >> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> > >> >> at
> > >> >>
> > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> > >> >> at
> > >> >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> > >> >> 2009-11-18 12:42:01,782 WARN org.apache.zookeeper.ClientCnxn:
> > Ignoring
> > >> >> exception during shutdown output
> > >> >> java.net.SocketException: Transport endpoint is not connected
> > >> >> at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
> > >> >> at
> > >> >>
> > sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
> > >> >> at
> > >> sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> > >> >> at
> > >> >>
> > >>
> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> > >> >> at
> > >> >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> > >> >> 2009-11-18 12:42:02,182 INFO org.apache.zookeeper.ClientCnxn:
> > >> Attempting
> > >> >> connection to server superpyxis0001/192.168.100.115:2181
> > >> >> 2009-11-18 12:42:02,182 INFO org.apache.zookeeper.ClientCnxn:
> Priming
> > >> >> connection to java.nio.channels.SocketChannel[connected local=/
> > >> >> 192.168.100.132:36197 remote=superpyxis0001/192.168.100.115:2181]
> > >> >> 2009-11-18 12:42:02,182 INFO org.apache.zookeeper.ClientCnxn:
> Server
> > >> >> connection successful
> > >> >> 2009-11-18 12:42:02,182 WARN org.apache.zookeeper.ClientCnxn:
> > Exception
> > >> >> closing session 0x425035c48210002 to
> > >> sun.nio.ch.SelectionKeyImpl@5c07076b
> > >> >> java.io.IOException: Read error rc = -1
> > java.nio.DirectByteBuffer[pos=0
> > >> >> lim=4 cap=4]
> > >> >> at
> > >> >>
> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
> > >> >> at
> > >> >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
> > >> >> 2009-11-18 12:42:02,182 WARN org.apache.zookeeper.ClientCnxn:
> > Ignoring
> > >> >> exception during shutdown input
> > >> >> java.net.SocketException: Transport endpoint is not connected
> > >> >> at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
> > >> >> at
> > >> >>
> > sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
> > >> >> at
> > >> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> > >> >> at
> > >> >>
> > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> > >> >> at
> > >> >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> > >> >> 2009-11-18 12:42:02,182 WARN org.apache.zookeeper.ClientCnxn:
> > Ignoring
> > >> >> exception during shutdown output
> > >> >> java.net.SocketException: Transport endpoint is not connected
> > >> >>
> > >> >>
> > >> >> On Mon, Nov 16, 2009 at 4:16 PM, Zhenyu Zhong <
> > zhongresearch@gmail.com
> > >> >> >wrote:
> > >> >>
> > >> >> > Here is the diskIO and CPU around the time we had RS
> disconnection
> > on
> > >> >> one
> > >> >> > machine that runs RegionServer. It doesn't seem to be high.
> Similar
> > >> disk
> > >> >> and
> > >> >> > cpu usage have been seen before and the HBase was running fine.
> > >> >> >
> > >> >> >
> > >> >> > So far I haven't found why my 10 minutes session timeout doesn't
> > >> apply.
> > >> >> > Still digging.
> > >> >> >
> > >> >> > Many thanks!
> > >> >> >
> > >> >> > Time: 03:01:08 PM
> > >> >> > avg-cpu: %user %nice %system %iowait %steal %idle
> > >> >> > 3.82 0.00 0.89 3.97 0.00 91.32
> > >> >> >
> > >> >> >
> > >> >> > Device: tps Blk_read/s Blk_wrtn/s Blk_read
> > >> Blk_wrtn
> > >> >> > sda 24.30 1605.39 6679.35 6238623388
> > >> 25956251720
> > >> >> > sda1 0.51 1.94 15.71 7547265
> > >> 61067904
> > >> >> > sda2 0.13 2.11 5.20 8202079
> > >> 20189232
> > >> >> > sda3 0.13 0.25 1.63 987846
> > >> 6323336
> > >> >> > sda4 0.00 0.00 0.00 4
> > >> 0
> > >> >> > sda5 1.78 0.06 24.14 220010
> > >> 93817208
> > >> >> > sda6 0.38 0.42 8.41 1630584
> > >> 32688152
> > >> >> > sda7 21.38 1600.61 6624.26 6220035272
> > >> 25742165888
> > >> >> > sdb 4.52 767.33 565.91 2981868690
> > >> 2199132380
> > >> >> > sdb1 4.52 767.33 565.91 2981866755
> > >> 2199132372
> > >> >> > sdc 4.42 742.95 563.00 2887151482
> > >> 2187823092
> > >> >> > sdc1 4.42 742.95 563.00 2887149547
> > >> 2187823084
> > >> >> > sdd 4.49 750.78 557.25 2917554074
> > >> 2165513500
> > >> >> > sdd1 4.49 750.78 557.25 2917552139
> > >> 2165513492
> > >> >> > sde 4.52 758.51 569.46 2947593394
> > >> 2212964236
> > >> >> > sde1 4.52 758.51 569.46 2947591459
> > >> 2212964228
> > >> >> > sdf 4.51 747.22 571.78 2903740266
> > >> 2221972652
> > >> >> > sdf1 4.51 747.22 571.78 2903738331
> > >> 2221972644
> > >> >> >
> > >> >> > Best,
> > >> >> > zhenyu
> > >> >> >
> > >> >> >
> > >> >> >
> > >> >> > On Mon, Nov 16, 2009 at 4:07 PM, stack <st...@duboce.net> wrote:
> > >> >> >
> > >> >> >> On Mon, Nov 16, 2009 at 12:05 PM, Zhenyu Zhong <
> > >> >> zhongresearch@gmail.com
> > >> >> >> >wrote:
> > >> >> >>
> > >> >> >> > I just realized that there was a MapReduce job running during
> > the
> > >> >> time
> > >> >> >> the
> > >> >> >> > regionserver disconnected from the zookeeper.
> > >> >> >> > That MapReduce Job was processing 500GB data and took about 8
> > >> minutes
> > >> >> to
> > >> >> >> > finish. It launched over 2000 map tasks.
> > >> >> >>
> > >> >> >>
> > >> >> >> There was a tasktracker running on the RegionServer that
> > >> disconnected?
> > >> >> Is
> > >> >> >> the map i/o or cpu heavy? Do you think it could have stole life
> > >> from
> > >> >> the
> > >> >> >> datanode/regionserver?
> > >> >> >>
> > >> >> >>
> > >> >> >>
> > >> >> >> > I doubt that this introduced
> > >> >> >> > resource contention between DataNode and RegionServer Node.
> > >> >> >> > Also during the time that MapReduce job ran, I saw a few
> errors
> > >> >> >> indicating
> > >> >> >> > that
> > >> >> >> >
> > >> >> >> > java.io.IOException: Could not obtain block:
> > >> >> >> > blk_7661247556283580226_4450462
> file=/data/xxx.txt.200910141159
> > >> >> >> > at
> > >> >> >> >
> > >> >> >>
> > >> >>
> > >>
> >
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1787)
> > >> >> >> > at
> > >> >> >> >
> > >> >> >>
> > >> >>
> > >>
> >
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1615)
> > >> >> >> > at
> > >> >> >> >
> > >> >> >>
> > >> >>
> > >>
> > org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1742)
> > >> >> >> > at
> java.io.DataInputStream.read(DataInputStream.java:83)
> > >> >> >> > at
> > >> >> >> org.apache.hadoop.util.LineReader.readLine(LineReader.java:134)
> > >> >> >> > at
> > >> >> >> >
> > >> >> >>
> > >> >>
> > >>
> > org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:136)
> > >> >> >> > at
> > >> >> >> >
> > >> >>
> > >>
> org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:40)
> > >> >> >> > at
> > >> >> >> >
> > >> >> >>
> > >> >>
> > >>
> >
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:192)
> > >> >> >> > at
> > >> >> >> >
> > >> >> >>
> > >> >>
> > >>
> >
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:176)
> > >> >> >> > at
> > >> org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
> > >> >> >> > at
> > >> >> >> org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
> > >> >> >> > at
> org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
> > >> >> >> > at org.apache.hadoop.mapred.Child.main(Child.java:170)
> > >> >> >> >
> > >> >> >>
> > >> >> >> This is interesting. Your map task is running into hdfs-127.
> > Yeah,
> > >> >> you
> > >> >> >> should patch your cluster if you want to get rid of these
> > (hdfs-127
> > >> has
> > >> >> >> been
> > >> >> >> applied to 0.21 hadoop and will be in the next hadoop release on
> > >> 0.20.x
> > >> >> >> branch, hadoop 0.20.2).
> > >> >> >>
> > >> >> >>
> > >> >> >>
> > >> >> >> >
> > >> >> >> > Possibly it is related to HDFS-127, but I don't see any
> > datanodes
> > >> >> went
> > >> >> >> > down. Does that affect Regionserver? Shall we apply the patch?
> > >> >> >> >
> > >> >> >> >
> > >> >> >> Read up on hdfs-127 for explaination of whats going on.
> > >> >> >>
> > >> >> >>
> > >> >> >>
> > >> >> >> > Now I start to keep tracking of the virtual memory stats to
> see
> > if
> > >> >> the
> > >> >> >> same
> > >> >> >> > issue happens tomorrow around the same time period.
> > >> >> >> >
> > >> >> >>
> > >> >> >> Excellent. Thanks for doing this.
> > >> >> >>
> > >> >> >>
> > >> >> >>
> > >> >> >> > And I highly suspect that this particular MapReduce job hurts
> > >> HBase
> > >> >> >> > Regionserver.
> > >> >> >> >
> > >> >> >> > PS. I also use gcviewer to parse the GC-log, I only see around
> > 30
> > >> >> >> seconds
> > >> >> >> > pauses maximum.
> > >> >> >> >
> > >> >> >>
> > >> >> >>
> > >> >> >> Did you figure why your ten minute session is not applying?
> > >> >> >>
> > >> >> >> St.Ack
> > >> >> >>
> > >> >> >>
> > >> >> >>
> > >> >> >> >
> > >> >> >> > Thanks a lot.
> > >> >> >> > zhenyu
> > >> >> >> >
> > >> >> >> >
> > >> >> >> >
> > >> >> >> > On Sun, Nov 15, 2009 at 12:52 AM, Zhenyu Zhong <
> > >> >> zhongresearch@gmail.com
> > >> >> >> > >wrote:
> > >> >> >> >
> > >> >> >> > > J-D,
> > >> >> >> > >
> > >> >> >> > > Thank you very much for your comments.
> > >> >> >> > > My company block the IRC port, so I have trouble to connect
> to
> > >> IRC
> > >> >> >> > channel.
> > >> >> >> > > I have been trying to ask the IT to open the IRC port for
> me,
> > it
> > >> >> might
> > >> >> >> > take
> > >> >> >> > > a while.
> > >> >> >> > >
> > >> >> >> > > Best,
> > >> >> >> > > zhenyu
> > >> >> >> > >
> > >> >> >> > >
> > >> >> >> > > On Sat, Nov 14, 2009 at 2:21 PM, Jean-Daniel Cryans <
> > >> >> >> jdcryans@apache.org
> > >> >> >> > >wrote:
> > >> >> >> > >
> > >> >> >> > >> The error you are getting is a disconnection from a
> zookeeper
> > >> >> server
> > >> >> >> > >> and is very generic.
> > >> >> >> > >>
> > >> >> >> > >> ZK-86 is still opened and the last comment refers to ZK-111
> > >> saying
> > >> >> >> > >> that the bug (in unit tests) was probably fixed in release
> > >> 3.0.0
> > >> >> last
> > >> >> >> > >> year.
> > >> >> >> > >>
> > >> >> >> > >> To figure the hang you have, you can try to jstack the
> > process
> > >> pid
> > >> >> >> and
> > >> >> >> > >> see exactly what's holding the RS from shutting down.
> > >> >> >> > >>
> > >> >> >> > >> Would it be possible for you to drop by the IRC channel?
> This
> > >> way
> > >> >> we
> > >> >> >> > >> can debug this a at much faster pace.
> > >> >> >> > >>
> > >> >> >> > >> Thx!
> > >> >> >> > >>
> > >> >> >> > >> J-D
> > >> >> >> > >>
> > >> >> >> > >> On Sat, Nov 14, 2009 at 9:57 AM, Zhenyu Zhong <
> > >> >> >> zhongresearch@gmail.com>
> > >> >> >> > >> wrote:
> > >> >> >> > >> > I found this.
> > >> >> >> > >> > http://issues.apache.org/jira/browse/ZOOKEEPER-86
> > >> >> >> > >> >
> > >> >> >> > >> > It looks like the same error I had. Is it a zookeeper
> bug?
> > >> When
> > >> >> >> will
> > >> >> >> > >> HBase
> > >> >> >> > >> > take the zookeeper version 3.3.0?
> > >> >> >> > >> >
> > >> >> >> > >> > thanks
> > >> >> >> > >> > zhenyu
> > >> >> >> > >> >
> > >> >> >> > >> > On Sat, Nov 14, 2009 at 11:31 AM, Zhenyu Zhong <
> > >> >> >> > zhongresearch@gmail.com
> > >> >> >> > >> >wrote:
> > >> >> >> > >> >
> > >> >> >> > >> >> Now I really doubt about the zookeeper, from the log I
> saw
> > >> >> errors
> > >> >> >> > like
> > >> >> >> > >> >> IOException Read Error, while the zookeeper client
> > >> >> (regionserver )
> > >> >> >> > >> tried to
> > >> >> >> > >> >> read. But it got disconneted status from zookeeper.
> > >> >> >> > >> >>
> > >> >> >> > >> >> I don't see any load on any zookeeper quorum servers.
> > DiskIO
> > >> is
> > >> >> >> fine.
> > >> >> >> > >> >>
> > >> >> >> > >> >> Also when the regionserver decides to exit due to the
> > >> >> disconnect
> > >> >> >> > status
> > >> >> >> > >> >> from zookeeper, sometimes the regionserver hangs during
> > the
> > >> >> >> exiting.
> > >> >> >> > We
> > >> >> >> > >> can
> > >> >> >> > >> >> still see the HRegionServer process even we don't see it
> > in
> > >> the
> > >> >> >> > Master
> > >> >> >> > >> web
> > >> >> >> > >> >> interface.
> > >> >> >> > >> >>
> > >> >> >> > >> >> I also notice that there is an zookeeper.retries and
> > >> >> >> zookeeper.pauses
> > >> >> >> > >> >> settings in hbase-default.xml, would that matter ?
> > >> >> >> > >> >>
> > >> >> >> > >> >> thanks
> > >> >> >> > >> >>
> > >> >> >> > >> >> You may see the IOException error in the following log
> > when
> > >> the
> > >> >> >> > >> >> Regionserver lost connection to the zookeeper.
> > >> >> >> > >> >>
> > >> >> >> > >> >> 2009-11-14 15:58:31,993 INFO
> > >> >> >> > org.apache.hadoop.hbase.regionserver.HLog:
> > >> >> >> > >> >> Roll
> > >> >> >> > >> >>
> > >> >> >> > >>
> > >> >> >> >
> > >> >> >>
> > >> >>
> > >>
> >
> /hbase/.logs/superpyxis0008.scur.colo,60021,1258179101617/hlog.dat.1258179106921,
> > >> >> >> > >> >> entries=1062964, calcsize=255014936, filesize=139566093.
> > New
> > >> >> hlog
> > >> >> >> > >> >>
> > >> >> >> > >>
> > >> >> >> >
> > >> >> >>
> > >> >>
> > >>
> >
> /hbase/.logs/superpyxis0008.scur.colo,60021,1258179101617/hlog.dat.1258214311993
> > >> >> >> > >> >> 2009-11-14 15:58:39,295 DEBUG
> > >> org.apache.zookeeper.ClientCnxn:
> > >> >> Got
> > >> >> >> > ping
> > >> >> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
> > >> >> >> > >> >> 2009-11-14 15:58:52,648 DEBUG
> > >> org.apache.zookeeper.ClientCnxn:
> > >> >> Got
> > >> >> >> > ping
> > >> >> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
> > >> >> >> > >> >> 2009-11-14 15:59:06,007 DEBUG
> > >> org.apache.zookeeper.ClientCnxn:
> > >> >> Got
> > >> >> >> > ping
> > >> >> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
> > >> >> >> > >> >> 2009-11-14 15:59:19,365 DEBUG
> > >> org.apache.zookeeper.ClientCnxn:
> > >> >> Got
> > >> >> >> > ping
> > >> >> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
> > >> >> >> > >> >> 2009-11-14 15:59:32,867 DEBUG
> > >> org.apache.zookeeper.ClientCnxn:
> > >> >> Got
> > >> >> >> > ping
> > >> >> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 186ms
> > >> >> >> > >> >> 2009-11-14 15:59:46,070 DEBUG
> > >> org.apache.zookeeper.ClientCnxn:
> > >> >> Got
> > >> >> >> > ping
> > >> >> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
> > >> >> >> > >> >> 2009-11-14 15:59:59,423 DEBUG
> > >> org.apache.zookeeper.ClientCnxn:
> > >> >> Got
> > >> >> >> > ping
> > >> >> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
> > >> >> >> > >> >> 2009-11-14 16:00:12,775 DEBUG
> > >> org.apache.zookeeper.ClientCnxn:
> > >> >> Got
> > >> >> >> > ping
> > >> >> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
> > >> >> >> > >> >> 2009-11-14 16:00:28,141 DEBUG
> > >> org.apache.zookeeper.ClientCnxn:
> > >> >> Got
> > >> >> >> > ping
> > >> >> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 2010ms
> > >> >> >> > >> >> 2009-11-14 16:01:10,378 WARN
> > >> org.apache.zookeeper.ClientCnxn:
> > >> >> >> > Exception
> > >> >> >> > >> >> closing session 0x424f0ed01ea00a6 to
> > >> >> >> > >> sun.nio.ch.SelectionKeyImpl@39fbb2d6
> > >> >> >> > >> >>
> > >> >> >> > >> >> java.io.IOException: Read error rc = -1
> > >> >> >> > java.nio.DirectByteBuffer[pos=0
> > >> >> >> > >> >> lim=4 cap=4]
> > >> >> >> > >> >> at
> > >> >> >> > >> >>
> > >> >> >>
> > org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
> > >> >> >> > >> >> at
> > >> >> >> > >> >>
> > >> >> >>
> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
> > >> >> >> > >> >> 2009-11-14 16:01:10,478 INFO
> > >> >> >> > >> >> org.apache.hadoop.hbase.regionserver.HRegionServer: Got
> > >> >> ZooKeeper
> > >> >> >> > >> event,
> > >> >> >> > >> >> state: Disconnected, type: None, path: null
> > >> >> >> > >> >> 2009-11-14 16:01:11,333 WARN
> > >> >> >> > >> >> org.apache.hadoop.hbase.regionserver.HRegionServer:
> > >> Attempt=1
> > >> >> >> > >> >>
> > >> >> >> > >> >> org.apache.hadoop.hbase.Leases$LeaseStillHeldException
> > >> >> >> > >> >> at
> > >> >> >> > >>
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> > >> >> >> > >> >> Method)
> > >> >> >> > >> >> at
> > >> >> >> > >> >>
> > >> >> >> > >>
> > >> >> >> >
> > >> >> >>
> > >> >>
> > >>
> >
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> > >> >> >> > >> >> at
> > >> >> >> > >> >>
> > >> >> >> > >>
> > >> >> >> >
> > >> >> >>
> > >> >>
> > >>
> >
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> > >> >> >> > >> >> at
> > >> >> >> > >>
> > java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> > >> >> >> > >> >> at
> > >> >> >> > >> >>
> > >> >> >> > >>
> > >> >> >> >
> > >> >> >>
> > >> >>
> > >>
> >
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
> > >> >> >> > >> >> at
> > >> >> >> > >> >>
> > >> >> >> > >>
> > >> >> >> >
> > >> >> >>
> > >> >>
> > >>
> >
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
> > >> >> >> > >> >> at
> > >> >> >> > >> >>
> > >> >> >> > >>
> > >> >> >> >
> > >> >> >>
> > >> >>
> > >>
> >
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
> > >> >> >> > >> >> at
> > >> >> >> > >> >>
> > >> >> >> > >>
> > >> >> >> >
> > >> >> >>
> > >> >>
> > >>
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:571)
> > >> >> >> > >> >> at java.lang.Thread.run(Thread.java:619)
> > >> >> >> > >> >> 2009-11-14 16:01:11,433 INFO
> > >> org.apache.zookeeper.ClientCnxn:
> > >> >> >> > >> Attempting
> > >> >> >> > >> >> connection to server superpyxis0001/
> 192.168.100.115:2181
> > >> >> >> > >> >> 2009-11-14 16:01:11,433 INFO
> > >> org.apache.zookeeper.ClientCnxn:
> > >> >> >> Priming
> > >> >> >> > >> >> connection to java.nio.channels.SocketChannel[connected
> > >> local=/
> > >> >> >> > >> >> 192.168.100.122:59575 remote=superpyxis0001/
> > >> >> 192.168.100.115:2181]
> > >> >> >> > >> >> 2009-11-14 16:01:11,433 INFO
> > >> org.apache.zookeeper.ClientCnxn:
> > >> >> >> Server
> > >> >> >> > >> >> connection successful
> > >> >> >> > >> >> 2009-11-14 16:01:11,433 INFO
> > >> >> >> > >> >> org.apache.hadoop.hbase.regionserver.HRegionServer: Got
> > >> >> ZooKeeper
> > >> >> >> > >> event,
> > >> >> >> > >> >> state: Expired, type: None, path: null
> > >> >> >> > >> >> 2009-11-14 16:01:11,433 WARN
> > >> org.apache.zookeeper.ClientCnxn:
> > >> >> >> > Exception
> > >> >> >> > >> >> closing session 0x424f0ed01ea00a6 to
> > >> >> >> > >> sun.nio.ch.SelectionKeyImpl@6b93d343
> > >> >> >> > >> >> java.io.IOException: Session Expired
> > >> >> >> > >> >> at
> > >> >> >> > >> >>
> > >> >> >> > >>
> > >> >> >> >
> > >> >> >>
> > >> >>
> > >>
> >
> org.apache.zookeeper.ClientCnxn$SendThread.readConnectResult(ClientCnxn.java:589)
> > >> >> >> > >> >> at
> > >> >> >> > >> >>
> > >> >> >>
> > org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:709)
> > >> >> >> > >> >>
> > >> >> >> > >> >> at
> > >> >> >> > >> >>
> > >> >> >>
> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
> > >> >> >> > >> >> 2009-11-14 16:01:11,433 ERROR
> > >> >> >> > >> >> org.apache.hadoop.hbase.regionserver.HRegionServer:
> > >> ZooKeeper
> > >> >> >> session
> > >> >> >> > >> >> expired
> > >> >> >> > >> >> 2009-11-14 16:01:13,513 INFO
> > >> org.apache.hadoop.ipc.HBaseServer:
> > >> >> >> IPC
> > >> >> >> > >> Server
> > >> >> >> > >> >> handler 3 on 60021, call put([B@73deb204,
> > >> >> >> > >> >> [Lorg.apache.hadoop.hbase.client.Put;@2179600a) from
> > >> >> >> > >> 192.168.100.132:40728:
> > >> >> >> > >> >> error: java.io.IOException: Server not running, aborting
> > >> >> >> > >> >> java.io.IOException: Server not running, aborting
> > >> >> >> > >> >>
> > >> >> >> > >> >>
> > >> >> >> > >> >>
> > >> >> >> > >> >>
> > >> >> >> > >> >>
> > >> >> >> > >> >> On Sat, Nov 14, 2009 at 1:01 AM, Zhenyu Zhong <
> > >> >> >> > zhongresearch@gmail.com
> > >> >> >> > >> >wrote:
> > >> >> >> > >> >>
> > >> >> >> > >> >>> Stack,
> > >> >> >> > >> >>>
> > >> >> >> > >> >>> Thanks a lot!
> > >> >> >> > >> >>>
> > >> >> >> > >> >>> I found out that the reason why HBase doesn't take the
> > >> system
> > >> >> >> file
> > >> >> >> > >> >>> descriptor value I set before. I started the HBase
> using
> > >> root
> > >> >> >> > instead
> > >> >> >> > >> of the
> > >> >> >> > >> >>> normal hadoop user, while my system configures hadoop
> > with
> > >> >> higher
> > >> >> >> > file
> > >> >> >> > >> >>> descriptor value but configures root with 1024 default
> > >> value.
> > >> >> >> > >> >>>
> > >> >> >> > >> >>>
> > >> >> >> > >> >>> Now my system has a clean start. However, it seems that
> > the
> > >> >> >> > >> >>> zookeeper.session.timeout value doesn't take into
> effect.
> > I
> > >> >> still
> > >> >> >> > >> found
> > >> >> >> > >> >>> around 60 seconds pauses from the disconnected
> > >> regionserver. I
> > >> >> >> > really
> > >> >> >> > >> don't
> > >> >> >> > >> >>> know why regionserver only times out after 60 seconds
> > >> instead
> > >> >> of
> > >> >> >> 10
> > >> >> >> > >> minutes
> > >> >> >> > >> >>> which I set for zookeeper.session.timeout.
> > >> >> >> > >> >>>
> > >> >> >> > >> >>> Is there any other timeout value coming into play
> before
> > >> the
> > >> >> >> actual
> > >> >> >> > >> >>> session times out?
> > >> >> >> > >> >>>
> > >> >> >> > >> >>> zhenyu
> > >> >> >> > >> >>>
> > >> >> >> > >> >>>
> > >> >> >> > >> >>>
> > >> >> >> > >> >>> On Fri, Nov 13, 2009 at 7:08 PM, stack <
> stack@duboce.net
> > >
> > >> >> wrote:
> > >> >> >> > >> >>>
> > >> >> >> > >> >>>> Ok. Lack of file descriptors manifests in all kinds
> of
> > >> weird
> > >> >> >> ways.
> > >> >> >> > >> >>>> Hopefully thats it. If not, lets keep digging.
> > >> >> >> > >> >>>> St.Ack
> > >> >> >> > >> >>>>
> > >> >> >> > >> >>>> On Fri, Nov 13, 2009 at 3:44 PM, Zhenyu Zhong <
> > >> >> >> > >> zhongresearch@gmail.com
> > >> >> >> > >> >>>> >wrote:
> > >> >> >> > >> >>>>
> > >> >> >> > >> >>>> > Stack,
> > >> >> >> > >> >>>> >
> > >> >> >> > >> >>>> > You are right, the master started with ulimit -n
> 1024.
> > >> It
> > >> >> >> doesn't
> > >> >> >> > >> take
> > >> >> >> > >> >>>> the
> > >> >> >> > >> >>>> > system value.
> > >> >> >> > >> >>>> >
> > >> >> >> > >> >>>> > Regarding to the too many open files, it looks like
> > the
> > >> >> same
> > >> >> >> as
> > >> >> >> > the
> > >> >> >> > >> one
> > >> >> >> > >> >>>> J-D
> > >> >> >> > >> >>>> > put up. But I will get the Master start with higher
> > >> value
> > >> >> >> first
> > >> >> >> > and
> > >> >> >> > >> see
> > >> >> >> > >> >>>> if
> > >> >> >> > >> >>>> > this kind of error goes away.
> > >> >> >> > >> >>>> >
> > >> >> >> > >> >>>> > thanks a lot!
> > >> >> >> > >> >>>> > zhenyu
> > >> >> >> > >> >>>> >
> > >> >> >> > >> >>>> > On Fri, Nov 13, 2009 at 6:02 PM, stack <
> > >> stack@duboce.net>
> > >> >> >> wrote:
> > >> >> >> > >> >>>> >
> > >> >> >> > >> >>>> > > Does it say
> > >> >> >> > >> >>>> > >
> > >> >> >> > >> >>>> > > ulimit -n 32768
> > >> >> >> > >> >>>> > >
> > >> >> >> > >> >>>> > > ...as the second line in your log file on start of
> > the
> > >> >> >> master?
> > >> >> >> > >> >>>> > >
> > >> >> >> > >> >>>> > > Is the stack trace that complains about too many
> > open
> > >> >> files
> > >> >> >> > same
> > >> >> >> > >> as
> > >> >> >> > >> >>>> the
> > >> >> >> > >> >>>> > one
> > >> >> >> > >> >>>> > > in the blog post J-D put up?
> > >> >> >> > >> >>>> > >
> > >> >> >> > >> >>>> > > St.Ack
> > >> >> >> > >> >>>> > >
> > >> >> >> > >> >>>> > >
> > >> >> >> > >> >>>> > > On Fri, Nov 13, 2009 at 1:37 PM, Zhenyu Zhong <
> > >> >> >> > >> >>>> zhongresearch@gmail.com
> > >> >> >> > >> >>>> > > >wrote:
> > >> >> >> > >> >>>> > >
> > >> >> >> > >> >>>> > > > The ulimit file descriptors was set to
> fs.file-max
> > >> >> >> =1578334,
> > >> >> >> > >> also
> > >> >> >> > >> >>>> in
> > >> >> >> > >> >>>> > the
> > >> >> >> > >> >>>> > > > limits.conf the value was set to 32768.
> > >> >> >> > >> >>>> > > > So these are way higher than the open
> descriptors
> > >> for
> > >> >> the
> > >> >> >> > >> running
> > >> >> >> > >> >>>> > > > processes.
> > >> >> >> > >> >>>> > > >
> > >> >> >> > >> >>>> > > > thanks
> > >> >> >> > >> >>>> > > > zhenyu
> > >> >> >> > >> >>>> > > >
> > >> >> >> > >> >>>> > > >
> > >> >> >> > >> >>>> > > > On Fri, Nov 13, 2009 at 4:33 PM, Stack <
> > >> >> >> saint.ack@gmail.com>
> > >> >> >> > >> >>>> wrote:
> > >> >> >> > >> >>>> > > >
> > >> >> >> > >> >>>> > > > > You upped the ulimit file descriptors as per
> the
> > >> >> getting
> > >> >> >> > >> started
> > >> >> >> > >> >>>> doc?
> > >> >> >> > >> >>>> > > > >
> > >> >> >> > >> >>>> > > > >
> > >> >> >> > >> >>>> > > > >
> > >> >> >> > >> >>>> > > > > On Nov 13, 2009, at 1:26 PM, Zhenyu Zhong <
> > >> >> >> > >> >>>> zhongresearch@gmail.com>
> > >> >> >> > >> >>>> > > > wrote:
> > >> >> >> > >> >>>> > > > >
> > >> >> >> > >> >>>> > > > > Thanks a lot.
> > >> >> >> > >> >>>> > > > >>
> > >> >> >> > >> >>>> > > > >>
> > >> >> >> > >> >>>> > > > >> Bad news is my kernel is still 2.6.26.
> > >> >> >> > >> >>>> > > > >> But it was not a problem before.
> > >> >> >> > >> >>>> > > > >>
> > >> >> >> > >> >>>> > > > >> Very strange.
> > >> >> >> > >> >>>> > > > >>
> > >> >> >> > >> >>>> > > > >> zhenyu
> > >> >> >> > >> >>>> > > > >>
> > >> >> >> > >> >>>> > > > >> On Fri, Nov 13, 2009 at 4:16 PM, Jean-Daniel
> > >> Cryans
> > >> >> <
> > >> >> >> > >> >>>> > > > jdcryans@apache.org
> > >> >> >> > >> >>>> > > > >> >wrote:
> > >> >> >> > >> >>>> > > > >>
> > >> >> >> > >> >>>> > > > >> Looks like
> > >> >> >> > >> >>>> > > > >>>
> > >> >> >> > >> >>>> > > > >>>
> > >> >> >> > >> >>>> > > >
> > >> >> >> > >> >>>> > >
> > >> >> >> > >> >>>> >
> > >> >> >> > >> >>>>
> > >> >> >> > >>
> > >> >> >> >
> > >> >> >>
> > >> >>
> > >>
> >
> http://pero.blogs.aprilmayjune.org/2009/01/22/hadoop-and-linux-kernel-2627-epoll-limits/
> > >> >> >> > >> >>>> > > > >>>
> > >> >> >> > >> >>>> > > > >>> J-D
> > >> >> >> > >> >>>> > > > >>>
> > >> >> >> > >> >>>> > > > >>> On Fri, Nov 13, 2009 at 1:12 PM, Zhenyu
> Zhong
> > <
> > >> >> >> > >> >>>> > > zhongresearch@gmail.com
> > >> >> >> > >> >>>> > > > >
> > >> >> >> > >> >>>> > > > >>> wrote:
> > >> >> >> > >> >>>> > > > >>>
> > >> >> >> > >> >>>> > > > >>>> Hi,
> > >> >> >> > >> >>>> > > > >>>>
> > >> >> >> > >> >>>> > > > >>>>
> > >> >> >> > >> >>>> > > > >>>> After I re-organize the cluster, the
> > experiment
> > >> >> ran
> > >> >> >> into
> > >> >> >> > >> >>>> problem
> > >> >> >> > >> >>>> > > > faster
> > >> >> >> > >> >>>> > > > >>>>
> > >> >> >> > >> >>>> > > > >>> than
> > >> >> >> > >> >>>> > > > >>>
> > >> >> >> > >> >>>> > > > >>>> before.
> > >> >> >> > >> >>>> > > > >>>>
> > >> >> >> > >> >>>> > > > >>>> Basically the changes are to use machines
> > with
> > >> >> less
> > >> >> >> > >> resources
> > >> >> >> > >> >>>> as
> > >> >> >> > >> >>>> > > > >>>>
> > >> >> >> > >> >>>> > > > >>> zookeeper
> > >> >> >> > >> >>>> > > > >>>
> > >> >> >> > >> >>>> > > > >>>> quorums and machines with more resources as
> > >> >> >> > regionserver.
> > >> >> >> > >> >>>> > > > >>>>
> > >> >> >> > >> >>>> > > > >>>> From the log, I still see the pause around
> 1
> > >> >> minute.
> > >> >> >> > >> >>>> > > > >>>> I enabled the GC logging, please see
> > >> >> >> > >> >>>> > > > >>>>
> > >> >> >> > >> >>>> > > > >>>> http://pastebin.com/m1d4ce0f1
> > >> >> >> > >> >>>> > > > >>>>
> > >> >> >> > >> >>>> > > > >>>> for details.
> > >> >> >> > >> >>>> > > > >>>> In general I don't see many pauses in the
> GC.
> > >> >> >> > >> >>>> > > > >>>>
> > >> >> >> > >> >>>> > > > >>>> What is more interesting, 1 hour after the
> > 1st
> > >> >> >> > >> regionserver
> > >> >> >> > >> >>>> > > > >>>> disconnected,
> > >> >> >> > >> >>>> > > > >>>> the master log complained about too many
> open
> > >> >> files.
> > >> >> >> > This
> > >> >> >> > >> >>>> didn't
> > >> >> >> > >> >>>> > > > happen
> > >> >> >> > >> >>>> > > > >>>> before.
> > >> >> >> > >> >>>> > > > >>>> I checked the system OS setup as well as
> the
> > >> >> >> > limits.conf.
> > >> >> >> > >> I
> > >> >> >> > >> >>>> also
> > >> >> >> > >> >>>> > > > checked
> > >> >> >> > >> >>>> > > > >>>>
> > >> >> >> > >> >>>> > > > >>> the
> > >> >> >> > >> >>>> > > > >>>
> > >> >> >> > >> >>>> > > > >>>> running processes. The total open files
> don't
> > >> >> reach
> > >> >> >> the
> > >> >> >> > >> limit.
> > >> >> >> > >> >>>> I
> > >> >> >> > >> >>>> > am
> > >> >> >> > >> >>>> > > > >>>>
> > >> >> >> > >> >>>> > > > >>> confused
> > >> >> >> > >> >>>> > > > >>>
> > >> >> >> > >> >>>> > > > >>>> a bit.
> > >> >> >> > >> >>>> > > > >>>>
> > >> >> >> > >> >>>> > > > >>>>
> > >> >> >> > >> >>>> > > > >>>> Please see the following master log.
> > >> >> >> > >> >>>> > > > >>>>
> > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:02,114 INFO
> > >> >> >> > >> >>>> > > > org.apache.hadoop.hbase.master.BaseScanner:
> > >> >> >> > >> >>>> > > > >>>> RegionManager.metaScanner scan of 4658
> row(s)
> > >> of
> > >> >> meta
> > >> >> >> > >> region
> > >> >> >> > >> >>>> > > {server:
> > >> >> >> > >> >>>> > > > >>>> 192.168.100.128:60021, regionname:
> > .META.,,1,
> > >> >> >> startKey:
> > >> >> >> > >> <>}
> > >> >> >> > >> >>>> > > complete
> > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:02,114 INFO
> > >> >> >> > >> >>>> > > > org.apache.hadoop.hbase.master.BaseScanner:
> > >> >> >> > >> >>>> > > > >>>>
> > >> >> >> > >> >>>> > > > >>> All
> > >> >> >> > >> >>>> > > > >>>
> > >> >> >> > >> >>>> > > > >>>> 1 .META. region(s) scanned
> > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:07,677 DEBUG
> > >> >> >> > >> org.apache.zookeeper.ClientCnxn:
> > >> >> >> > >> >>>> Got
> > >> >> >> > >> >>>> > > > ping
> > >> >> >> > >> >>>> > > > >>>> response for sessionid:0x424eebf1c10004c
> > after
> > >> 3ms
> > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:08,178 INFO
> > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > >> >> >> > >> >>>> > > > Exception
> > >> >> >> > >> >>>> > > > >>>>
> > >> >> >> > >> >>>> > > > >>> in
> > >> >> >> > >> >>>> > > > >>>
> > >> >> >> > >> >>>> > > > >>>> createBlockOutputStream
> java.io.IOException:
> > >> Bad
> > >> >> >> connect
> > >> >> >> > >> ack
> > >> >> >> > >> >>>> with
> > >> >> >> > >> >>>> > > > >>>> firstBadLink 192.168.100.123:50010
> > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:08,178 INFO
> > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > >> >> >> > >> >>>> > > > >>>> Abandoning
> > >> >> >> > >> >>>> > > > >>>> block blk_-2808245019291145247_5478039
> > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:09,682 INFO
> > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > >> >> >> > >> >>>> > > > Exception
> > >> >> >> > >> >>>> > > > >>>>
> > >> >> >> > >> >>>> > > > >>> in
> > >> >> >> > >> >>>> > > > >>>
> > >> >> >> > >> >>>> > > > >>>> createBlockOutputStream
> java.io.EOFException
> > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:09,682 INFO
> > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > >> >> >> > >> >>>> > > > >>>> Abandoning
> > >> >> >> > >> >>>> > > > >>>> block blk_1074853606841896259_5478048
> > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:10,334 DEBUG
> > >> >> >> > >> org.apache.zookeeper.ClientCnxn:
> > >> >> >> > >> >>>> Got
> > >> >> >> > >> >>>> > > > ping
> > >> >> >> > >> >>>> > > > >>>> response for sessionid:0x24eebf1043003c
> after
> > >> 1ms
> > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:21,018 DEBUG
> > >> >> >> > >> org.apache.zookeeper.ClientCnxn:
> > >> >> >> > >> >>>> Got
> > >> >> >> > >> >>>> > > > ping
> > >> >> >> > >> >>>> > > > >>>> response for sessionid:0x424eebf1c10004c
> > after
> > >> 0ms
> > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:23,674 DEBUG
> > >> >> >> > >> org.apache.zookeeper.ClientCnxn:
> > >> >> >> > >> >>>> Got
> > >> >> >> > >> >>>> > > > ping
> > >> >> >> > >> >>>> > > > >>>> response for sessionid:0x24eebf1043003c
> after
> > >> 0ms
> > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:24,828 INFO
> > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > >> >> >> > >> >>>> > > > Exception
> > >> >> >> > >> >>>> > > > >>>>
> > >> >> >> > >> >>>> > > > >>> in
> > >> >> >> > >> >>>> > > > >>>
> > >> >> >> > >> >>>> > > > >>>> createBlockOutputStream
> java.io.IOException:
> > >> Bad
> > >> >> >> connect
> > >> >> >> > >> ack
> > >> >> >> > >> >>>> with
> > >> >> >> > >> >>>> > > > >>>> firstBadLink 192.168.100.123:50010
> > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:24,828 INFO
> > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > >> >> >> > >> >>>> > > > >>>> Abandoning
> > >> >> >> > >> >>>> > > > >>>> block blk_-6642544517082142289_5478063
> > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:24,828 INFO
> > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > >> >> >> > >> >>>> > > > Exception
> > >> >> >> > >> >>>> > > > >>>>
> > >> >> >> > >> >>>> > > > >>> in
> > >> >> >> > >> >>>> > > > >>>
> > >> >> >> > >> >>>> > > > >>>> createBlockOutputStream
> > >> java.net.SocketException:
> > >> >> Too
> > >> >> >> > many
> > >> >> >> > >> >>>> open
> > >> >> >> > >> >>>> > > files
> > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:24,828 INFO
> > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > >> >> >> > >> >>>> > > > >>>> Abandoning
> > >> >> >> > >> >>>> > > > >>>> block blk_2057511041109796090_5478063
> > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:24,928 INFO
> > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > >> >> >> > >> >>>> > > > Exception
> > >> >> >> > >> >>>> > > > >>>>
> > >> >> >> > >> >>>> > > > >>> in
> > >> >> >> > >> >>>> > > > >>>
> > >> >> >> > >> >>>> > > > >>>> createBlockOutputStream
> > >> java.net.SocketException:
> > >> >> Too
> > >> >> >> > many
> > >> >> >> > >> >>>> open
> > >> >> >> > >> >>>> > > files
> > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:24,928 INFO
> > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > >> >> >> > >> >>>> > > > >>>> Abandoning
> > >> >> >> > >> >>>> > > > >>>> block blk_8219260302213892894_5478064
> > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:30,855 INFO
> > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > >> >> >> > >> >>>> > > > Exception
> > >> >> >> > >> >>>> > > > >>>>
> > >> >> >> > >> >>>> > > > >>> in
> > >> >> >> > >> >>>> > > > >>>
> > >> >> >> > >> >>>> > > > >>>> createBlockOutputStream
> > >> java.net.SocketException:
> > >> >> Too
> > >> >> >> > many
> > >> >> >> > >> >>>> open
> > >> >> >> > >> >>>> > > files
> > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:30,855 INFO
> > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > >> >> >> > >> >>>> > > > >>>> Abandoning
> > >> >> >> > >> >>>> > > > >>>> block blk_1669205542853067709_5478235
> > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:30,905 INFO
> > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > >> >> >> > >> >>>> > > > Exception
> > >> >> >> > >> >>>> > > > >>>>
> > >> >> >> > >> >>>> > > > >>> in
> > >> >> >> > >> >>>> > > > >>>
> > >> >> >> > >> >>>> > > > >>>> createBlockOutputStream
> > >> java.net.SocketException:
> > >> >> Too
> > >> >> >> > many
> > >> >> >> > >> >>>> open
> > >> >> >> > >> >>>> > > files
> > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:30,905 INFO
> > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > >> >> >> > >> >>>> > > > >>>> Abandoning
> > >> >> >> > >> >>>> > > > >>>> block blk_9128897691346270351_5478237
> > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:30,955 INFO
> > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > >> >> >> > >> >>>> > > > Exception
> > >> >> >> > >> >>>> > > > >>>>
> > >> >> >> > >> >>>> > > > >>> in
> > >> >> >> > >> >>>> > > > >>>
> > >> >> >> > >> >>>> > > > >>>> createBlockOutputStream
> > >> java.net.SocketException:
> > >> >> Too
> > >> >> >> > many
> > >> >> >> > >> >>>> open
> > >> >> >> > >> >>>> > > files
> > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:30,955 INFO
> > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > >> >> >> > >> >>>> > > > >>>> Abandoning
> > >> >> >> > >> >>>> > > > >>>> block blk_1116845144864123018_5478240
> > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:34,372 DEBUG
> > >> >> >> > >> org.apache.zookeeper.ClientCnxn:
> > >> >> >> > >> >>>> Got
> > >> >> >> > >> >>>> > > > ping
> > >> >> >> > >> >>>> > > > >>>> response for sessionid:0x424eebf1c10004c
> > after
> > >> 0ms
> > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:37,034 DEBUG
> > >> >> >> > >> org.apache.zookeeper.ClientCnxn:
> > >> >> >> > >> >>>> Got
> > >> >> >> > >> >>>> > > > ping
> > >> >> >> > >> >>>> > > > >>>> response for sessionid:0x24eebf1043003c
> after
> > >> 0ms
> > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:37,235 WARN
> > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > >> >> >> > >> >>>> > > > >>>>
> > >> >> >> > >> >>>> > > > >>> DataStreamer
> > >> >> >> > >> >>>> > > > >>>
> > >> >> >> > >> >>>> > > > >>>> Exception: java.io.IOException: Too many
> open
> > >> >> files
> > >> >> >> > >> >>>> > > > >>>> at sun.nio.ch.IOUtil.initPipe(Native
> > >> Method)
> > >> >> >> > >> >>>> > > > >>>> at
> > >> >> >> > >> >>>> > >
> > >> >> >> sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:49)
> > >> >> >> > >> >>>> > > > >>>> at
> > >> >> >> > >> >>>> > > > >>>>
> > >> >> >> > >> >>>> > > > >>>>
> > >> >> >> > >> >>>> > > >
> > >> >> >> > >> >>>> > >
> > >> >> >> > >> >>>> >
> > >> >> >> > >> >>>>
> > >> >> >> > >>
> > >> >> >> >
> > >> >> >>
> > >> >>
> > >>
> >
> sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:18)
> > >> >> >> > >> >>>> > > > >>>
> > >> >> >> > >> >>>> > > > >>>> at
> > >> >> >> > >> >>>> > > > >>>>
> > >> >> >> > >> >>>> > > > >>>>
> > >> >> >> > >> >>>> > > >
> > >> >> >> > >> >>>> > >
> > >> >> >> > >> >>>> >
> > >> >> >> > >> >>>>
> > >> >> >> > >>
> > >> >> >> >
> > >> >> >>
> > >> >>
> > >>
> >
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.get(SocketIOWithTimeout.java:407)
> > >> >> >> > >> >>>> > > > >>>
> > >> >> >> > >> >>>> > > > >>>> at
> > >> >> >> > >> >>>> > > > >>>>
> > >> >> >> > >> >>>> > > > >>>>
> > >> >> >> > >> >>>> > > >
> > >> >> >> > >> >>>> > >
> > >> >> >> > >> >>>> >
> > >> >> >> > >> >>>>
> > >> >> >> > >>
> > >> >> >> >
> > >> >> >>
> > >> >>
> > >>
> >
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:322)
> > >> >> >> > >> >>>> > > > >>>
> > >> >> >> > >> >>>> > > > >>>> at
> > >> >> >> > >> >>>> > > > >>>>
> > >> >> >> > >> >>>> > > > >>>>
> > >> >> >> > >> >>>> > > >
> > >> >> >> > >> >>>> > >
> > >> >> >> > >> >>>> >
> > >> >> >> > >> >>>>
> > >> >> >> > >>
> > >> >> >> >
> > >> >> >>
> > >> >>
> > >>
> >
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
> > >> >> >> > >> >>>> > > > >>>
> > >> >> >> > >> >>>> > > > >>>> at
> > >> >> >> > >> >>>> > > > >>>>
> > >> >> >> > >> >>>> > > > >>>>
> > >> >> >> > >> >>>> > > >
> > >> >> >> > >> >>>> > >
> > >> >> >> > >> >>>> >
> > >> >> >> > >> >>>>
> > >> >> >> > >>
> > >> >> >> >
> > >> >> >>
> > >> >>
> > >>
> >
> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
> > >> >> >> > >> >>>> > > > >>>
> > >> >> >> > >> >>>> > > > >>>> at
> > >> >> >> > >> >>>> > > > >>>>
> > >> >> >> > >> >>>> > > > >>>>
> > >> >> >> > >> >>>> > > >
> > >> >> >> > >> >>>> > >
> > >> >> >> > >> >>>> >
> > >> >> >> > >> >>>>
> > >> >> >> > >>
> > >> >> >> >
> > >> >> >>
> > >> >>
> > >>
> >
> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
> > >> >> >> > >> >>>> > > > >>>
> > >> >> >> > >> >>>> > > > >>>> at
> > >> >> >> > >> >>>> > > > >>>>
> > >> >> >> > >> >>>> > > > >>>
> > >> >> >> > >> >>>>
> > >> >> >>
> java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
> > >> >> >> > >> >>>> > > > >>>
> > >> >> >> > >> >>>> > > > >>>> at
> > >> >> >> > >> >>>>
> java.io.DataOutputStream.write(DataOutputStream.java:90)
> > >> >> >> > >> >>>> > > > >>>> at
> > >> >> >> > >> >>>> > > > >>>>
> > >> >> >> > >> >>>> > > > >>>>
> > >> >> >> > >> >>>> > > >
> > >> >> >> > >> >>>> > >
> > >> >> >> > >> >>>> >
> > >> >> >> > >> >>>>
> > >> >> >> > >>
> > >> >> >> >
> > >> >> >>
> > >> >>
> > >>
> >
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2290)
> > >> >> >> > >> >>>> > > > >>>
> > >> >> >> > >> >>>> > > > >>>>
> > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:37,235 WARN
> > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > >> >> >> > >> >>>> > Error
> > >> >> >> > >> >>>> > > > >>>> Recovery for block
> > >> blk_8148813491785406356_5478475
> > >> >> >> bad
> > >> >> >> > >> >>>> datanode[0]
> > >> >> >> > >> >>>> > > > >>>> 192.168.100.123:50010
> > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:37,235 WARN
> > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > >> >> >> > >> >>>> > Error
> > >> >> >> > >> >>>> > > > >>>> Recovery for block
> > >> blk_8148813491785406356_5478475
> > >> >> in
> > >> >> >> > >> pipeline
> > >> >> >> > >> >>>> > > > >>>> 192.168.100.123:50010,
> 192.168.100.134:50010
> > ,
> > >> >> >> > >> >>>> > 192.168.100.122:50010
> > >> >> >> > >> >>>> > > :
> > >> >> >> > >> >>>> > > > >>>> bad
> > >> >> >> > >> >>>> > > > >>>> datanode 192.168.100.123:50010
> > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:37,436 INFO
> > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > >> >> >> > >> >>>> > > > Exception
> > >> >> >> > >> >>>> > > > >>>>
> > >> >> >> > >> >>>> > > > >>> in
> > >> >> >> > >> >>>> > > > >>>
> > >> >> >> > >> >>>> > > > >>>> createBlockOutputStream
> > >> java.net.SocketException:
> > >> >> Too
> > >> >> >> > many
> > >> >> >> > >> >>>> open
> > >> >> >> > >> >>>> > > files
> > >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:37,436 INFO
> > >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> > >> >> >> > >> >>>> > > > >>>> Abandoning
> > >> >> >> > >> >>>> > > > >>>> block blk_2119727700857186236_5478498
> > >> >> >> > >> >>>> > > > >>>>
> > >> >> >> > >> >>>> > > > >>>>
> > >> >> >> > >> >>>> > > > >>>>
> > >> >> >> > >> >>>> > > > >>>>
> > >> >> >> > >> >>>> > > > >>>>
> > >> >> >> > >> >>>> > > > >>>> On Thu, Nov 12, 2009 at 4:21 PM, Zhenyu
> Zhong
> > <
> > >> >> >> > >> >>>> > > > zhongresearch@gmail.com
> > >> >> >> > >> >>>> > > > >>>> wrote:
> > >> >> >> > >> >>>> > > > >>>>
> > >> >> >> > >> >>>> > > > >>>> Will do.
> > >> >> >> > >> >>>> > > > >>>>>
> > >> >> >> > >> >>>> > > > >>>>> thanks
> > >> >> >> > >> >>>> > > > >>>>> zhenyu
> > >> >> >> > >> >>>> > > > >>>>>
> > >> >> >> > >> >>>> > > > >>>>>
> > >> >> >> > >> >>>> > > > >>>>> On Thu, Nov 12, 2009 at 3:33 PM, stack <
> > >> >> >> > stack@duboce.net
> > >> >> >> > >> >
> > >> >> >> > >> >>>> wrote:
> > >> >> >> > >> >>>> > > > >>>>>
> > >> >> >> > >> >>>> > > > >>>>> Enable GC logging too on this next run
> (see
> > >> >> >> > >> hbase-env.sh).
> > >> >> >> > >> >>>> Lets
> > >> >> >> > >> >>>> > > try
> > >> >> >> > >> >>>> > > > >>>>>>
> > >> >> >> > >> >>>> > > > >>>>> and
> > >> >> >> > >> >>>> > > > >>>
> > >> >> >> > >> >>>> > > > >>>> get
> > >> >> >> > >> >>>> > > > >>>>>> to the bottom of whats going on.
> > >> >> >> > >> >>>> > > > >>>>>> Thanks,
> > >> >> >> > >> >>>> > > > >>>>>> St.Ack
> > >> >> >> > >> >>>> > > > >>>>>>
> > >> >> >> > >> >>>> > > > >>>>>> On Thu, Nov 12, 2009 at 12:29 PM, Zhenyu
> > >> Zhong <
> > >> >> >> > >> >>>> > > > >>>>>>
> > >> >> >> > >> >>>> > > > >>>>> zhongresearch@gmail.com
> > >> >> >> > >> >>>> > > > >>>
> > >> >> >> > >> >>>> > > > >>>> wrote:
> > >> >> >> > >> >>>> > > > >>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>
> > >> >> >> > >> >>>> > > > >>>>>> I can switch the boxes that run
> zookeeper
> > >> with
> > >> >> the
> > >> >> >> > ones
> > >> >> >> > >> >>>> that
> > >> >> >> > >> >>>> > run
> > >> >> >> > >> >>>> > > > >>>>>>> regionservers.
> > >> >> >> > >> >>>> > > > >>>>>>> I will see the results later.
> > >> >> >> > >> >>>> > > > >>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>> FYI. The node does have the 10 minutes
> > >> >> >> > >> >>>> > zookeeper.session.timeout
> > >> >> >> > >> >>>> > > > >>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>> value
> > >> >> >> > >> >>>> > > > >>>
> > >> >> >> > >> >>>> > > > >>>> in
> > >> >> >> > >> >>>> > > > >>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>> place.
> > >> >> >> > >> >>>> > > > >>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>> thanks
> > >> >> >> > >> >>>> > > > >>>>>>> zhenyu
> > >> >> >> > >> >>>> > > > >>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>> On Thu, Nov 12, 2009 at 3:21 PM, stack <
> > >> >> >> > >> stack@duboce.net>
> > >> >> >> > >> >>>> > wrote:
> > >> >> >> > >> >>>> > > > >>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>> On Thu, Nov 12, 2009 at 11:50 AM,
> Zhenyu
> > >> Zhong
> > >> >> <
> > >> >> >> > >> >>>> > > > >>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>> zhongresearch@gmail.com
> > >> >> >> > >> >>>> > > > >>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>> wrote:
> > >> >> >> > >> >>>> > > > >>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>> In my cluster, half of the cluster
> have
> > 2
> > >> >> disks
> > >> >> >> > 400GB
> > >> >> >> > >> >>>> each
> > >> >> >> > >> >>>> > per
> > >> >> >> > >> >>>> > > > >>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>> machine,
> > >> >> >> > >> >>>> > > > >>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>> and
> > >> >> >> > >> >>>> > > > >>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>> half of the cluster have 6 disks per
> > >> machine.
> > >> >> >> > Maybe
> > >> >> >> > >> we
> > >> >> >> > >> >>>> > should
> > >> >> >> > >> >>>> > > > >>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>> run
> > >> >> >> > >> >>>> > > > >>>
> > >> >> >> > >> >>>> > > > >>>> zookeeper
> > >> >> >> > >> >>>> > > > >>>>>>>>> on the machines with 2 disks and RS on
> > >> >> machines
> > >> >> >> > with
> > >> >> >> > >> 6
> > >> >> >> > >> >>>> disks?
> > >> >> >> > >> >>>> > > > >>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>> That would make most sense only in the
> > >> below,
> > >> >> it
> > >> >> >> > >> looks
> > >> >> >> > >> >>>> like
> > >> >> >> > >> >>>> > the
> > >> >> >> > >> >>>> > > > >>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>> RS
> > >> >> >> > >> >>>> > > > >>>
> > >> >> >> > >> >>>> > > > >>>> that
> > >> >> >> > >> >>>> > > > >>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>> had
> > >> >> >> > >> >>>> > > > >>>>>>>> issue had 4 disks?
> > >> >> >> > >> >>>> > > > >>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>> BTW, the 10 minutes
> > >> zookeeper.session.timeout
> > >> >> >> has
> > >> >> >> > >> been
> > >> >> >> > >> >>>> set
> > >> >> >> > >> >>>> > > during
> > >> >> >> > >> >>>> > > > >>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>> the
> > >> >> >> > >> >>>> > > > >>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>> experiment.
> > >> >> >> > >> >>>> > > > >>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>> And for sure this node had it in place?
> > >> >> >> > >> >>>> > > > >>>>>>>> St.Ack
> > >> >> >> > >> >>>> > > > >>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>> thanks
> > >> >> >> > >> >>>> > > > >>>>>>>>> zhenyu
> > >> >> >> > >> >>>> > > > >>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>> On Thu, Nov 12, 2009 at 2:08 PM, stack
> <
> > >> >> >> > >> stack@duboce.net
> > >> >> >> > >> >>>> >
> > >> >> >> > >> >>>> > > wrote:
> > >> >> >> > >> >>>> > > > >>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>> On Thu, Nov 12, 2009 at 8:40 AM,
> Zhenyu
> > >> >> Zhong <
> > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>> zhongresearch@gmail.com
> > >> >> >> > >> >>>> > > > >>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>> wrote:
> > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>> Though I experienced 2 regionserver
> > >> >> >> disconnection
> > >> >> >> > >> this
> > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>> morning,
> > >> >> >> > >> >>>> > > > >>>
> > >> >> >> > >> >>>> > > > >>>> it
> > >> >> >> > >> >>>> > > > >>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>> looks
> > >> >> >> > >> >>>> > > > >>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>> better from the regionserver log.
> > (please
> > >> >> see
> > >> >> >> the
> > >> >> >> > >> >>>> following
> > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>> log)
> > >> >> >> > >> >>>> > > > >>>
> > >> >> >> > >> >>>> > > > >>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>>> http://pastebin.com/m496dbfae
> > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>>> I checked diskIO for one of the
> > >> >> >> > >> >>>> > regionserver(192.168.100.116)
> > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>> during
> > >> >> >> > >> >>>> > > > >>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>> the
> > >> >> >> > >> >>>> > > > >>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>> time it disconnected.
> > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>>> Time: 03:04:58 AM
> > >> >> >> > >> >>>> > > > >>>>>>>>>>> Device: tps Blk_read/s
> > >> >> >> Blk_wrtn/s
> > >> >> >> > >> >>>> Blk_read
> > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>> Blk_wrtn
> > >> >> >> > >> >>>> > > > >>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>> sda 105.31 5458.83
> > >> >> 19503.64
> > >> >> >> > >> >>>> 9043873239
> > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>> 32312473676
> > >> >> >> > >> >>>> > > > >>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>> sda1 2.90 6.64
> > >> >> 99.25
> > >> >> >> > >> >>>> 10993934
> > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>> 164433464
> > >> >> >> > >> >>>> > > > >>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>> sda2 1.72 23.76
> > >> >> 30.25
> > >> >> >> > >> >>>> 39365817
> > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>> 50124008
> > >> >> >> > >> >>>> > > > >>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>> sda3 0.30 0.38
> > >> >> 3.58
> > >> >> >> > >> >>>> 624930
> > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>> 5923000
> > >> >> >> > >> >>>> > > > >>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>> sda4 100.39 5428.06
> > >> >> 19370.56
> > >> >> >> > >> >>>> 8992888270
> > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>> 32091993204
> > >> >> >> > >> >>>> > > > >>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>>> Is this high for you? 20k
> > >> blocks/second
> > >> >> >> would
> > >> >> >> > >> seem to
> > >> >> >> > >> >>>> be
> > >> >> >> > >> >>>> > > high
> > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>> but
> > >> >> >> > >> >>>> > > > >>>
> > >> >> >> > >> >>>> > > > >>>> its
> > >> >> >> > >> >>>> > > > >>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>> one
> > >> >> >> > >> >>>> > > > >>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>> disk only and its not being shared by
> > zk
> > >> >> >> anymore
> > >> >> >> > so
> > >> >> >> > >> >>>> > shouldn't
> > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>> matter?
> > >> >> >> > >> >>>> > > > >>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>> I also checked the zookeeper quorum
> > >> server
> > >> >> >> that
> > >> >> >> > the
> > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>> regionserver
> > >> >> >> > >> >>>> > > > >>>
> > >> >> >> > >> >>>> > > > >>>> tried
> > >> >> >> > >> >>>> > > > >>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>> to
> > >> >> >> > >> >>>> > > > >>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>> connect according to the log.
> However,
> > I
> > >> >> don't
> > >> >> >> see
> > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>> 192.168.100.116
> > >> >> >> > >> >>>> > > > >>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>> in
> > >> >> >> > >> >>>> > > > >>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>> the
> > >> >> >> > >> >>>> > > > >>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>> client list of the zookeeper quorum
> > >> server
> > >> >> that
> > >> >> >> > the
> > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>> regionserver
> > >> >> >> > >> >>>> > > > >>>
> > >> >> >> > >> >>>> > > > >>>> tried
> > >> >> >> > >> >>>> > > > >>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>> to
> > >> >> >> > >> >>>> > > > >>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>> connect.
> > >> >> >> > >> >>>> > > > >>>>>>>>>>> Would that be a problem?
> > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>>> Is that because the ephemeral node
> > for
> > >> the
> > >> >> >> > >> >>>> regionserver
> > >> >> >> > >> >>>> > had
> > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>> evaporated?
> > >> >> >> > >> >>>> > > > >>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>> Lost
> > >> >> >> > >> >>>> > > > >>>>>>>>>> its lease w/ zk by the time you went
> to
> > >> >> look?
> > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>> Thu Nov 12 15:04:52 UTC 2009
> > >> >> >> > >> >>>> > > > >>>>>>>>>>> Zookeeper version: 3.2.1-808558,
> built
> > >> on
> > >> >> >> > >> 08/27/2009
> > >> >> >> > >> >>>> 18:48
> > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>> GMT
> > >> >> >> > >> >>>> > > > >>>
> > >> >> >> > >> >>>> > > > >>>> Clients:
> > >> >> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.127:43045
> > >> >> >> > >> [1](queued=0,recved=26,sent=0)
> > >> >> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.131:39091
> > >> >> >> > >> [1](queued=0,recved=964,sent=0)
> > >> >> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.124:35961
> > >> >> >> > >> [1](queued=0,recved=3266,sent=0)
> > >> >> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.123:47935
> > >> >> >> > >> [1](queued=0,recved=233,sent=0)
> > >> >> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.125:46931
> > >> >> >> > >> [1](queued=0,recved=2,sent=0)
> > >> >> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.118:54924
> > >> >> >> > >> [1](queued=0,recved=295,sent=0)
> > >> >> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.118:41390
> > >> >> >> > >> [1](queued=0,recved=2290,sent=0)
> > >> >> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.136:42243
> > >> >> >> > >> [1](queued=0,recved=0,sent=0)
> > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>>> Latency min/avg/max: 0/17/6333
> > >> >> >> > >> >>>> > > > >>>>>>>>>>> Received: 47111
> > >> >> >> > >> >>>> > > > >>>>>>>>>>> Sent: 0
> > >> >> >> > >> >>>> > > > >>>>>>>>>>> Outstanding: 0
> > >> >> >> > >> >>>> > > > >>>>>>>>>>> Zxid: 0x77000083f4
> > >> >> >> > >> >>>> > > > >>>>>>>>>>> Mode: leader
> > >> >> >> > >> >>>> > > > >>>>>>>>>>> Node count: 23
> > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>> That 6 second maximum latency is
> pretty
> > >> bad
> > >> >> but
> > >> >> >> > >> should
> > >> >> >> > >> >>>> be
> > >> >> >> > >> >>>> > well
> > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>> within
> > >> >> >> > >> >>>> > > > >>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>> the
> > >> >> >> > >> >>>> > > > >>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>> zk
> > >> >> >> > >> >>>> > > > >>>>>>>>>> session timeout.
> > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>> So, problem is likely on the zk
> client
> > >> side
> > >> >> of
> > >> >> >> the
> > >> >> >> > >> >>>> session;
> > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>> i.e.
> > >> >> >> > >> >>>> > > > >>>
> > >> >> >> > >> >>>> > > > >>>> in
> > >> >> >> > >> >>>> > > > >>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>> the
> > >> >> >> > >> >>>> > > > >>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>> RS.
> > >> >> >> > >> >>>> > > > >>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>> You could enable GC logging as J-D
> > >> suggested
> > >> >> to
> > >> >> >> > see
> > >> >> >> > >> if
> > >> >> >> > >> >>>> you
> > >> >> >> > >> >>>> > > have
> > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>> any
> > >> >> >> > >> >>>> > > > >>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>> big
> > >> >> >> > >> >>>> > > > >>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>> pauses, pauses > zk session timeout.
> > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>> When the RS went down, it didn't look
> > too
> > >> >> >> heavily
> > >> >> >> > >> >>>> loaded:
> > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>> 1. 2009-11-12 15:04:52,830 INFO
> > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > >> >> >> > org.apache.hadoop.hbase.regionserver.HRegionServer:
> > >> >> >> > >> >>>> Dump of
> > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>> metrics:
> > >> >> >> > >> >>>> > > > >>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>> request=1.5166667, regions=322,
> > >> stores=657,
> > >> >> >> > >> >>>> storefiles=631,
> > >> >> >> > >> >>>> > > > >>>>>>>>>> storefileIndexSize=61,
> > >> memstoreSize=1472,
> > >> >> >> > >> >>>> usedHeap=2819,
> > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>> maxHeap=4079,
> > >> >> >> > >> >>>> > > > >>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>> blockCacheSize=658110960,
> > >> >> >> > blockCacheFree=197395984,
> > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>> blockCacheCount=9903,
> > >> >> >> > >> >>>> > > > >>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>> blockCacheHitRatio=99
> > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>> Its serving a few reads? The number
> of
> > >> >> store
> > >> >> >> > files
> > >> >> >> > >> >>>> seems
> > >> >> >> > >> >>>> > > fine.
> > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>> Not
> > >> >> >> > >> >>>> > > > >>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>> too
> > >> >> >> > >> >>>> > > > >>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>> much memory used.
> > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>> Looking at the logs, I see the Lease
> > >> Still
> > >> >> Held
> > >> >> >> > >> >>>> exception.
> > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>> This
> > >> >> >> > >> >>>> > > > >>>
> > >> >> >> > >> >>>> > > > >>>> happens
> > >> >> >> > >> >>>> > > > >>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>> when the RS does its regular report to
> > the
> > >> >> >> master
> > >> >> >> > but
> > >> >> >> > >> the
> > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>> master
> > >> >> >> > >> >>>> > > > >>>
> > >> >> >> > >> >>>> > > > >>>> thinks
> > >> >> >> > >> >>>> > > > >>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>> the
> > >> >> >> > >> >>>> > > > >>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>> RS has since restarted. It'll think
> > this
> > >> >> >> probably
> > >> >> >> > >> >>>> because
> > >> >> >> > >> >>>> > it
> > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>> noticed
> > >> >> >> > >> >>>> > > > >>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>> that
> > >> >> >> > >> >>>> > > > >>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>> the RS's znode in zk had gone away
> and
> > it
> > >> >> >> > considered
> > >> >> >> > >> the
> > >> >> >> > >> >>>> RS
> > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>> dead.
> > >> >> >> > >> >>>> > > > >>>
> > >> >> >> > >> >>>> > > > >>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>> Looking too at your logs I see this
> > gap
> > >> in
> > >> >> the
> > >> >> >> zk
> > >> >> >> > >> >>>> pinging:
> > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>> 1. 2009-11-12 15:03:39,325 DEBUG
> > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>> org.apache.zookeeper.ClientCnxn:
> > >> >> >> > >> >>>> > > > >>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>> Got
> > >> >> >> > >> >>>> > > > >>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>> ping response for
> > >> sessionid:0x224e55436ad0004
> > >> >> >> after
> > >> >> >> > >> 0ms
> > >> >> >> > >> >>>> > > > >>>>>>>>>> 2. 2009-11-12 15:03:43,113 DEBUG
> > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>> org.apache.zookeeper.ClientCnxn:
> > >> >> >> > >> >>>> > > > >>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>> Got
> > >> >> >> > >> >>>> > > > >>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>> ping response for
> > >> sessionid:0x24e55436a0007d
> > >> >> >> after
> > >> >> >> > >> 0ms
> > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>> Where in the lines above it, its
> > >> reporting
> > >> >> >> about
> > >> >> >> > >> every
> > >> >> >> > >> >>>> ten
> > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>> seconds,
> > >> >> >> > >> >>>> > > > >>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>> here
> > >> >> >> > >> >>>> > > > >>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>> there is a big gap.
> > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>> Do you have ganglia or something that
> > >> will
> > >> >> let
> > >> >> >> you
> > >> >> >> > >> look
> > >> >> >> > >> >>>> more
> > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>> into
> > >> >> >> > >> >>>> > > > >>>
> > >> >> >> > >> >>>> > > > >>>> what
> > >> >> >> > >> >>>> > > > >>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>> was
> > >> >> >> > >> >>>> > > > >>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>> happening on this machine around this
> > >> time?
> > >> >> Is
> > >> >> >> > the
> > >> >> >> > >> >>>> machine
> > >> >> >> > >> >>>> > > OK?
> > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>> It
> > >> >> >> > >> >>>> > > > >>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>> looks
> > >> >> >> > >> >>>> > > > >>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>> lightly loaded and you have your
> cluster
> > >> >> nicely
> > >> >> >> > laid
> > >> >> >> > >> out.
> > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>> Something
> > >> >> >> > >> >>>> > > > >>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>> odd
> > >> >> >> > >> >>>> > > > >>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>> is
> > >> >> >> > >> >>>> > > > >>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>> going on. What about things like the
> > >> write
> > >> >> >> speed
> > >> >> >> > to
> > >> >> >> > >> >>>> disk?
> > >> >> >> > >> >>>> > In
> > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>> the
> > >> >> >> > >> >>>> > > > >>>
> > >> >> >> > >> >>>> > > > >>>> past
> > >> >> >> > >> >>>> > > > >>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>> strange issues have been explained by
> > >> >> incorrectly
> > >> >> >> > set
> > >> >> >> > >> BIOS
> > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>> which
> > >> >> >> > >> >>>> > > > >>>
> > >> >> >> > >> >>>> > > > >>>> made
> > >> >> >> > >> >>>> > > > >>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>> disks
> > >> >> >> > >> >>>> > > > >>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>> run at 1/100th of their proper speed.
> > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>> St.Ack
> > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>>> Best,
> > >> >> >> > >> >>>> > > > >>>>>>>>>>> zhenyu
> > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>>> On Wed, Nov 11, 2009 at 3:58 PM,
> > Zhenyu
> > >> >> Zhong
> > >> >> >> <
> > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>> zhongresearch@gmail.com
> > >> >> >> > >> >>>> > > > >>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>> wrote:
> > >> >> >> > >> >>>> > > > >>>>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>>> Stack
> > >> >> >> > >> >>>> > > > >>>>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>>>> I am very appreciated for your
> > >> comments.
> > >> >> >> > >> >>>> > > > >>>>>>>>>>>> I will use the zookeeper monitoring
> > >> script
> > >> >> on
> > >> >> >> my
> > >> >> >> > >> >>>> cluster
> > >> >> >> > >> >>>> > > > >>>>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>>> and
> > >> >> >> > >> >>>> > > > >>>
> > >> >> >> > >> >>>> > > > >>>> let
> > >> >> >> > >> >>>> > > > >>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>> it
> > >> >> >> > >> >>>> > > > >>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>> run
> > >> >> >> > >> >>>> > > > >>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>> overnight to see the result.
> > >> >> >> > >> >>>> > > > >>>>>>>>>>>> I will follow up the thread when I
> > get
> > >> >> >> anything.
> > >> >> >> > >> >>>> > > > >>>>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>>>> thanks
> > >> >> >> > >> >>>> > > > >>>>>>>>>>>> zhenyu
> > >> >> >> > >> >>>> > > > >>>>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>>>> On Wed, Nov 11, 2009 at 3:52 PM,
> > stack
> > >> <
> > >> >> >> > >> >>>> stack@duboce.net>
> > >> >> >> > >> >>>> > > > >>>>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>>> wrote:
> > >> >> >> > >> >>>> > > > >>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>>>> I see these in your log too:
> > >> >> >> > >> >>>> > > > >>>>>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>>>>> 1. 2009-11-11 04:27:19,018 DEBUG
> > >> >> >> > >> >>>> > > > >>>>>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
> > >> >> >> > >> >>>> > > > >>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>> Got
> > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>>> ping response for
> > >> >> sessionid:0x424dfd908c50009
> > >> >> >> > >> after
> > >> >> >> > >> >>>> > > > >>>>>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>>>> 4544ms
> > >> >> >> > >> >>>> > > > >>>
> > >> >> >> > >> >>>> > > > >>>> 2. 2009-11-11 04:27:19,018 DEBUG
> > >> >> >> > >> >>>> > > > >>>>>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
> > >> >> >> > >> >>>> > > > >>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>> Got
> > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>>> ping response for
> > >> >> sessionid:0x24dfd90c810002
> > >> >> >> > after
> > >> >> >> > >> >>>> > > > >>>>>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>>>> 4548ms
> > >> >> >> > >> >>>> > > > >>>
> > >> >> >> > >> >>>> > > > >>>> 3. 2009-11-11 04:27:43,960 DEBUG
> > >> >> >> > >> >>>> > > > >>>>>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
> > >> >> >> > >> >>>> > > > >>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>> Got
> > >> >> >> > >> >>>> > > > >>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>>> ping response for
> > >> >> sessionid:0x424dfd908c50009
> > >> >> >> > >> after
> > >> >> >> > >> >>>> > > > >>>>>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>>>> 9030ms
> > >> >> >> > >> >>>> > > > >>>
> > >> >> >> > >> >>>> > > > >>>> 4. 2009-11-11 04:27:43,960 DEBUG
> > >> >> >> > >> >>>> > > > >>>>>>>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>>>>>> org.apache.zookeeper.
> > >> >> >> > >> >>>> > > > >>>>>>>>
> > >> >> >> > >> >>>> > > > >>>>>>>
> > >> >> >> > >> >>>> > > >
> > >> >> >> > >> >>>> > >
> > >> >> >> > >> >>>> >
> > >> >> >> > >> >>>>
> > >> >> >> > >> >>>
> > >> >> >> > >> >>>
> > >> >> >> > >> >>
> > >> >> >> > >> >
> > >> >> >> > >>
> > >> >> >> > >
> > >> >> >> > >
> > >> >> >> >
> > >> >> >>
> > >> >> >
> > >> >> >
> > >> >>
> > >> >
> > >> >
> > >>
> > >
> > >
> >
>
Re: regionserver disconnection
Posted by stack <st...@duboce.net>.
What is in the other zk logs? Why did the leader go away? Memory issue or
something? Why'd it shutdown? Thanks for pastebin'ing this stuff.
St.Ack
On Thu, Nov 19, 2009 at 7:53 AM, Zhenyu Zhong <zh...@gmail.com>wrote:
> After digging more on the zookeeper log, I found that there seemed to be a
> socketconnection timeout. Also some warnings indicate that zookeeper server
> is not running. Probably I need to understand the mechanism of zookeeper
> first.
>
> Please find the zookeeper log around that time.
>
> http://pastebin.com/m37bb4ad1
>
> Many thanks
> zhenyu
>
>
> On Thu, Nov 19, 2009 at 10:31 AM, Zhenyu Zhong <zhongresearch@gmail.com
> >wrote:
>
> > Stack,
> >
> > I am very appreciated. Let me dig into the zookeeper log more.
> >
> > FYI, I saw the sleeper complained about the delay for the first time. It
> > looks like there are different reasons for the RS disconnections.
> >
> > 2009-11-19 08:08:32,708 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
> > response for sessionid:0x42508716bfc0001 after 0ms
> > 2009-11-19 08:08:37,595 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
> > response for sessionid:0x42508716bfc0082 after 0ms
> > 2009-11-19 08:08:46,073 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
> > response for sessionid:0x42508716bfc0001 after 0ms
> > 2009-11-19 08:09:39,880 WARN org.apache.zookeeper.ClientCnxn: Exception
> > closing session 0x42508716bfc0001 to sun.nio.ch.SelectionKeyImpl@4f3a608f
> > java.io.IOException: TIMED OUT
> > at
> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:906)
> > 2009-11-19 08:09:39,880 WARN org.apache.zookeeper.ClientCnxn: Exception
> > closing session 0x42508716bfc0082 to sun.nio.ch.SelectionKeyImpl@14c8875
> > java.io.IOException: TIMED OUT
> > at
> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:906)
> > 2009-11-19 08:09:39,880 WARN org.apache.hadoop.hbase.util.Sleeper: We
> slept
> > 53556ms, ten times longer than scheduled: 3000
> > 2009-11-19 08:09:39,980 WARN
> > org.apache.hadoop.hbase.regionserver.HRegionServer: Attempt=1
> > org.apache.hadoop.hbase.Leases$LeaseStillHeldException
> > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> > Method)
> > at
> >
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> > at
> java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> > at
> >
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
> > at
> >
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
> > at
> >
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
> > at
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:571)
> > at java.lang.Thread.run(Thread.java:619)
> > 2009-11-19 08:09:39,980 INFO
> > org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
> > state: Disconnected, type: None, path: null
> > 2009-11-19 08:09:40,530 INFO org.apache.zookeeper.ClientCnxn: Attempting
> > connection to server superpyxis0005/192.168.100.119:2181
> > 2009-11-19 08:09:40,530 INFO org.apache.zookeeper.ClientCnxn: Priming
> > connection to java.nio.channels.SocketChannel[connected local=/
> > 192.168.100.122:40659 remote=superpyxis0005/192.168.100.119:2181]
> >
> >
> >
> > Best,
> > zhenyu
> >
> > On Wed, Nov 18, 2009 at 6:52 PM, stack <st...@duboce.net> wrote:
> >
> >> Patrick Hunt, one of the zk lads, answered with the following:
> >>
> >>
> >>
> >>
> >> According to the following snippet (I cut to just session
> >> 0x425035c48210002)
> >> the client is getting pings for this session id regularly every 6
> seconds
> >> or
> >> so, then after 41,016 there's a 20 second gap (corresponds to a 30
> second
> >> timeout being set during session creation) at which point we disconnect
> >> from
> >> the server.
> >>
> >>
> >> > 2009-11-18 12:41:34,788 DEBUG org.apache.zookeeper.ClientCnxn: Got
> ping
> >> > response for sessionid:0x425035c48210002 after 0ms
> >> > 2009-11-18 12:42:01,482 WARN org.apache.zookeeper.ClientCnxn:
> Exception
> >> > closing session 0x425035c48210002 to
> >> sun.nio.ch.SelectionKeyImpl@421690ab
> >> > java.io.IOException: TIMED OUT
> >>
> >> the client notifies hbase of the disco, and attempts to reconnect to a
> >> server
> >>
> >>
> >> > org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper
> event,
> >> > state: Disconnected, type: None, path: null
> >> > 2009-11-18 12:42:01,782 INFO org.apache.zookeeper.ClientCnxn:
> Attempting
> >> > connection to server superpyxis0003/192.168.100.117:2181
> >>
> >> but the server won't allow the session be be established for some reason
> >> (is
> >> that the right server:port that's it's trying to connect to?)
> >>
> >>
> >> > 2009-11-18 12:42:02,182 INFO org.apache.zookeeper.ClientCnxn: Server
> >> > connection successful
> >> > 2009-11-18 12:42:02,182 WARN org.apache.zookeeper.ClientCnxn:
> Exception
> >> > closing session 0x425035c48210002 to
> >> sun.nio.ch.SelectionKeyImpl@5c07076b
> >> > java.io.IOException: Read error rc = -1
> java.nio.DirectByteBuffer[pos=0
> >> > lim=4 cap=4]
> >>
> >> The "read error rc = -1" means that the client read from the socket and
> >> got
> >> EOS. But it's not a session expiration afaict (and should not be at this
> >> point in time).
> >>
> >>
> >> You need to look at the ZK server logs for this period in time (for this
> >> server superpyxis0003:2181, and superpyxis0001:2181, the client tries to
> >> connect to both of these). There should be some indication there why the
> >> server is closing down the connection and not allowing the session to be
> >> re-established.
> >>
> >> One thought - by default we limit clients to a max of 10 connections
> from
> >> a
> >> particular host to the server. If you exceed this limit the zk server
> >> (it's
> >> per server though, not per cluster, so perhaps this is not it) will not
> >> allow any additional connections from the client host (you can't exceed
> >> 10
> >> on a server). Could this be it? i.e. the user is running a number of ZK
> >> clients (say hbase processes) from the same host? Regardless, get the
> >> server
> >> log for this time period and it should be more clear.
> >>
> >> Patrick
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> On Wed, Nov 18, 2009 at 11:18 AM, stack <st...@duboce.net> wrote:
> >>
> >> > And running stat on each of your zookeeper nodes tells you what about
> >> > averages for connect-times?
> >> >
> >> > Its odd that there is a ping just 20 seconds beforehand and you have
> set
> >> > the zk session timeout at 60 seconds (or ten minutes?). Let me ask
> the
> >> zk
> >> > lads about it.
> >> >
> >> > If you look in the GC log for around this time, do you see anything?
> >> >
> >> > St.Ack
> >> >
> >> >
> >> >
> >> > On Wed, Nov 18, 2009 at 7:32 AM, Zhenyu Zhong <
> zhongresearch@gmail.com
> >> >wrote:
> >> >
> >> >> After running more experiment, it seems that the disconnection is not
> >> >> related to the heavy load job, because the load average is low, the
> >> diskio
> >> >> is normal, memory Heap was not reached, also the virtual memory stats
> >> >> shows
> >> >> that no swappings. However, the disconnection still happens.
> >> >>
> >> >> It looks like this time it pauses for 20 seconds. No idea why
> >> regionserver
> >> >> disconnected.
> >> >>
> >> >> Any other suggestions please?
> >> >>
> >> >>
> >> >> 2009-11-18 12:41:00,956 DEBUG org.apache.zookeeper.ClientCnxn: Got
> ping
> >> >> response for sessionid:0x125035c495d02bb after 0ms
> >> >> 2009-11-18 12:41:08,074 DEBUG org.apache.zookeeper.ClientCnxn: Got
> ping
> >> >> response for sessionid:0x425035c48210002 after 0ms
> >> >> 2009-11-18 12:41:14,302 DEBUG org.apache.zookeeper.ClientCnxn: Got
> ping
> >> >> response for sessionid:0x125035c495d02bb after 0ms
> >> >> 2009-11-18 12:41:21,433 DEBUG org.apache.zookeeper.ClientCnxn: Got
> ping
> >> >> response for sessionid:0x425035c48210002 after 0ms
> >> >> 2009-11-18 12:41:27,656 DEBUG org.apache.zookeeper.ClientCnxn: Got
> ping
> >> >> response for sessionid:0x125035c495d02bb after 0ms
> >> >> 2009-11-18 12:41:34,788 DEBUG org.apache.zookeeper.ClientCnxn: Got
> ping
> >> >> response for sessionid:0x425035c48210002 after 0ms
> >> >> 2009-11-18 12:41:41,016 DEBUG org.apache.zookeeper.ClientCnxn: Got
> ping
> >> >> response for sessionid:0x125035c495d02bb after 1ms
> >> >> 2009-11-18 12:42:01,482 WARN org.apache.zookeeper.ClientCnxn:
> Exception
> >> >> closing session 0x425035c48210002 to
> >> sun.nio.ch.SelectionKeyImpl@421690ab
> >> >> java.io.IOException: TIMED OUT
> >> >> at
> >> >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:906)
> >> >> 2009-11-18 12:42:01,582 INFO
> >> >> org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper
> >> event,
> >> >> state: Disconnected, type: None, path: null
> >> >> 2009-11-18 12:42:01,782 INFO org.apache.zookeeper.ClientCnxn:
> >> Attempting
> >> >> connection to server superpyxis0003/192.168.100.117:2181
> >> >> 2009-11-18 12:42:01,782 INFO org.apache.zookeeper.ClientCnxn: Priming
> >> >> connection to java.nio.channels.SocketChannel[connected local=/
> >> >> 192.168.100.132:46610 remote=superpyxis0003/192.168.100.117:2181]
> >> >> 2009-11-18 12:42:01,782 INFO org.apache.zookeeper.ClientCnxn: Server
> >> >> connection successful
> >> >> 2009-11-18 12:42:01,782 WARN org.apache.zookeeper.ClientCnxn:
> Exception
> >> >> closing session 0x425035c48210002 to
> >> sun.nio.ch.SelectionKeyImpl@56dc6fac
> >> >> java.io.IOException: Read error rc = -1
> java.nio.DirectByteBuffer[pos=0
> >> >> lim=4 cap=4]
> >> >> at
> >> >> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
> >> >> at
> >> >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
> >> >> 2009-11-18 12:42:01,782 WARN org.apache.zookeeper.ClientCnxn:
> Ignoring
> >> >> exception during shutdown input
> >> >> java.net.SocketException: Transport endpoint is not connected
> >> >> at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
> >> >> at
> >> >>
> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
> >> >> at
> >> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> >> >> at
> >> >>
> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> >> >> at
> >> >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> >> >> 2009-11-18 12:42:01,782 WARN org.apache.zookeeper.ClientCnxn:
> Ignoring
> >> >> exception during shutdown output
> >> >> java.net.SocketException: Transport endpoint is not connected
> >> >> at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
> >> >> at
> >> >>
> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
> >> >> at
> >> sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> >> >> at
> >> >>
> >> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> >> >> at
> >> >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> >> >> 2009-11-18 12:42:02,182 INFO org.apache.zookeeper.ClientCnxn:
> >> Attempting
> >> >> connection to server superpyxis0001/192.168.100.115:2181
> >> >> 2009-11-18 12:42:02,182 INFO org.apache.zookeeper.ClientCnxn: Priming
> >> >> connection to java.nio.channels.SocketChannel[connected local=/
> >> >> 192.168.100.132:36197 remote=superpyxis0001/192.168.100.115:2181]
> >> >> 2009-11-18 12:42:02,182 INFO org.apache.zookeeper.ClientCnxn: Server
> >> >> connection successful
> >> >> 2009-11-18 12:42:02,182 WARN org.apache.zookeeper.ClientCnxn:
> Exception
> >> >> closing session 0x425035c48210002 to
> >> sun.nio.ch.SelectionKeyImpl@5c07076b
> >> >> java.io.IOException: Read error rc = -1
> java.nio.DirectByteBuffer[pos=0
> >> >> lim=4 cap=4]
> >> >> at
> >> >> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
> >> >> at
> >> >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
> >> >> 2009-11-18 12:42:02,182 WARN org.apache.zookeeper.ClientCnxn:
> Ignoring
> >> >> exception during shutdown input
> >> >> java.net.SocketException: Transport endpoint is not connected
> >> >> at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
> >> >> at
> >> >>
> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
> >> >> at
> >> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> >> >> at
> >> >>
> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> >> >> at
> >> >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> >> >> 2009-11-18 12:42:02,182 WARN org.apache.zookeeper.ClientCnxn:
> Ignoring
> >> >> exception during shutdown output
> >> >> java.net.SocketException: Transport endpoint is not connected
> >> >>
> >> >>
> >> >> On Mon, Nov 16, 2009 at 4:16 PM, Zhenyu Zhong <
> zhongresearch@gmail.com
> >> >> >wrote:
> >> >>
> >> >> > Here is the diskIO and CPU around the time we had RS disconnection
> on
> >> >> one
> >> >> > machine that runs RegionServer. It doesn't seem to be high. Similar
> >> disk
> >> >> and
> >> >> > cpu usage have been seen before and the HBase was running fine.
> >> >> >
> >> >> >
> >> >> > So far I haven't found why my 10 minutes session timeout doesn't
> >> apply.
> >> >> > Still digging.
> >> >> >
> >> >> > Many thanks!
> >> >> >
> >> >> > Time: 03:01:08 PM
> >> >> > avg-cpu: %user %nice %system %iowait %steal %idle
> >> >> > 3.82 0.00 0.89 3.97 0.00 91.32
> >> >> >
> >> >> >
> >> >> > Device: tps Blk_read/s Blk_wrtn/s Blk_read
> >> Blk_wrtn
> >> >> > sda 24.30 1605.39 6679.35 6238623388
> >> 25956251720
> >> >> > sda1 0.51 1.94 15.71 7547265
> >> 61067904
> >> >> > sda2 0.13 2.11 5.20 8202079
> >> 20189232
> >> >> > sda3 0.13 0.25 1.63 987846
> >> 6323336
> >> >> > sda4 0.00 0.00 0.00 4
> >> 0
> >> >> > sda5 1.78 0.06 24.14 220010
> >> 93817208
> >> >> > sda6 0.38 0.42 8.41 1630584
> >> 32688152
> >> >> > sda7 21.38 1600.61 6624.26 6220035272
> >> 25742165888
> >> >> > sdb 4.52 767.33 565.91 2981868690
> >> 2199132380
> >> >> > sdb1 4.52 767.33 565.91 2981866755
> >> 2199132372
> >> >> > sdc 4.42 742.95 563.00 2887151482
> >> 2187823092
> >> >> > sdc1 4.42 742.95 563.00 2887149547
> >> 2187823084
> >> >> > sdd 4.49 750.78 557.25 2917554074
> >> 2165513500
> >> >> > sdd1 4.49 750.78 557.25 2917552139
> >> 2165513492
> >> >> > sde 4.52 758.51 569.46 2947593394
> >> 2212964236
> >> >> > sde1 4.52 758.51 569.46 2947591459
> >> 2212964228
> >> >> > sdf 4.51 747.22 571.78 2903740266
> >> 2221972652
> >> >> > sdf1 4.51 747.22 571.78 2903738331
> >> 2221972644
> >> >> >
> >> >> > Best,
> >> >> > zhenyu
> >> >> >
> >> >> >
> >> >> >
> >> >> > On Mon, Nov 16, 2009 at 4:07 PM, stack <st...@duboce.net> wrote:
> >> >> >
> >> >> >> On Mon, Nov 16, 2009 at 12:05 PM, Zhenyu Zhong <
> >> >> zhongresearch@gmail.com
> >> >> >> >wrote:
> >> >> >>
> >> >> >> > I just realized that there was a MapReduce job running during
> the
> >> >> time
> >> >> >> the
> >> >> >> > regionserver disconnected from the zookeeper.
> >> >> >> > That MapReduce Job was processing 500GB data and took about 8
> >> minutes
> >> >> to
> >> >> >> > finish. It launched over 2000 map tasks.
> >> >> >>
> >> >> >>
> >> >> >> There was a tasktracker running on the RegionServer that
> >> disconnected?
> >> >> Is
> >> >> >> the map i/o or cpu heavy? Do you think it could have stole life
> >> from
> >> >> the
> >> >> >> datanode/regionserver?
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> > I doubt that this introduced
> >> >> >> > resource contention between DataNode and RegionServer Node.
> >> >> >> > Also during the time that MapReduce job ran, I saw a few errors
> >> >> >> indicating
> >> >> >> > that
> >> >> >> >
> >> >> >> > java.io.IOException: Could not obtain block:
> >> >> >> > blk_7661247556283580226_4450462 file=/data/xxx.txt.200910141159
> >> >> >> > at
> >> >> >> >
> >> >> >>
> >> >>
> >>
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1787)
> >> >> >> > at
> >> >> >> >
> >> >> >>
> >> >>
> >>
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1615)
> >> >> >> > at
> >> >> >> >
> >> >> >>
> >> >>
> >>
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1742)
> >> >> >> > at java.io.DataInputStream.read(DataInputStream.java:83)
> >> >> >> > at
> >> >> >> org.apache.hadoop.util.LineReader.readLine(LineReader.java:134)
> >> >> >> > at
> >> >> >> >
> >> >> >>
> >> >>
> >>
> org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:136)
> >> >> >> > at
> >> >> >> >
> >> >>
> >> org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:40)
> >> >> >> > at
> >> >> >> >
> >> >> >>
> >> >>
> >>
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:192)
> >> >> >> > at
> >> >> >> >
> >> >> >>
> >> >>
> >>
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:176)
> >> >> >> > at
> >> org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
> >> >> >> > at
> >> >> >> org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
> >> >> >> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
> >> >> >> > at org.apache.hadoop.mapred.Child.main(Child.java:170)
> >> >> >> >
> >> >> >>
> >> >> >> This is interesting. Your map task is running into hdfs-127.
> Yeah,
> >> >> you
> >> >> >> should patch your cluster if you want to get rid of these
> (hdfs-127
> >> has
> >> >> >> been
> >> >> >> applied to 0.21 hadoop and will be in the next hadoop release on
> >> 0.20.x
> >> >> >> branch, hadoop 0.20.2).
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> >
> >> >> >> > Possibly it is related to HDFS-127, but I don't see any
> datanodes
> >> >> went
> >> >> >> > down. Does that affect Regionserver? Shall we apply the patch?
> >> >> >> >
> >> >> >> >
> >> >> >> Read up on hdfs-127 for explaination of whats going on.
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> > Now I start to keep tracking of the virtual memory stats to see
> if
> >> >> the
> >> >> >> same
> >> >> >> > issue happens tomorrow around the same time period.
> >> >> >> >
> >> >> >>
> >> >> >> Excellent. Thanks for doing this.
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> > And I highly suspect that this particular MapReduce job hurts
> >> HBase
> >> >> >> > Regionserver.
> >> >> >> >
> >> >> >> > PS. I also use gcviewer to parse the GC-log, I only see around
> 30
> >> >> >> seconds
> >> >> >> > pauses maximum.
> >> >> >> >
> >> >> >>
> >> >> >>
> >> >> >> Did you figure why your ten minute session is not applying?
> >> >> >>
> >> >> >> St.Ack
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> >
> >> >> >> > Thanks a lot.
> >> >> >> > zhenyu
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> > On Sun, Nov 15, 2009 at 12:52 AM, Zhenyu Zhong <
> >> >> zhongresearch@gmail.com
> >> >> >> > >wrote:
> >> >> >> >
> >> >> >> > > J-D,
> >> >> >> > >
> >> >> >> > > Thank you very much for your comments.
> >> >> >> > > My company block the IRC port, so I have trouble to connect to
> >> IRC
> >> >> >> > channel.
> >> >> >> > > I have been trying to ask the IT to open the IRC port for me,
> it
> >> >> might
> >> >> >> > take
> >> >> >> > > a while.
> >> >> >> > >
> >> >> >> > > Best,
> >> >> >> > > zhenyu
> >> >> >> > >
> >> >> >> > >
> >> >> >> > > On Sat, Nov 14, 2009 at 2:21 PM, Jean-Daniel Cryans <
> >> >> >> jdcryans@apache.org
> >> >> >> > >wrote:
> >> >> >> > >
> >> >> >> > >> The error you are getting is a disconnection from a zookeeper
> >> >> server
> >> >> >> > >> and is very generic.
> >> >> >> > >>
> >> >> >> > >> ZK-86 is still opened and the last comment refers to ZK-111
> >> saying
> >> >> >> > >> that the bug (in unit tests) was probably fixed in release
> >> 3.0.0
> >> >> last
> >> >> >> > >> year.
> >> >> >> > >>
> >> >> >> > >> To figure the hang you have, you can try to jstack the
> process
> >> pid
> >> >> >> and
> >> >> >> > >> see exactly what's holding the RS from shutting down.
> >> >> >> > >>
> >> >> >> > >> Would it be possible for you to drop by the IRC channel? This
> >> way
> >> >> we
> >> >> >> > >> can debug this a at much faster pace.
> >> >> >> > >>
> >> >> >> > >> Thx!
> >> >> >> > >>
> >> >> >> > >> J-D
> >> >> >> > >>
> >> >> >> > >> On Sat, Nov 14, 2009 at 9:57 AM, Zhenyu Zhong <
> >> >> >> zhongresearch@gmail.com>
> >> >> >> > >> wrote:
> >> >> >> > >> > I found this.
> >> >> >> > >> > http://issues.apache.org/jira/browse/ZOOKEEPER-86
> >> >> >> > >> >
> >> >> >> > >> > It looks like the same error I had. Is it a zookeeper bug?
> >> When
> >> >> >> will
> >> >> >> > >> HBase
> >> >> >> > >> > take the zookeeper version 3.3.0?
> >> >> >> > >> >
> >> >> >> > >> > thanks
> >> >> >> > >> > zhenyu
> >> >> >> > >> >
> >> >> >> > >> > On Sat, Nov 14, 2009 at 11:31 AM, Zhenyu Zhong <
> >> >> >> > zhongresearch@gmail.com
> >> >> >> > >> >wrote:
> >> >> >> > >> >
> >> >> >> > >> >> Now I really doubt about the zookeeper, from the log I saw
> >> >> errors
> >> >> >> > like
> >> >> >> > >> >> IOException Read Error, while the zookeeper client
> >> >> (regionserver )
> >> >> >> > >> tried to
> >> >> >> > >> >> read. But it got disconneted status from zookeeper.
> >> >> >> > >> >>
> >> >> >> > >> >> I don't see any load on any zookeeper quorum servers.
> DiskIO
> >> is
> >> >> >> fine.
> >> >> >> > >> >>
> >> >> >> > >> >> Also when the regionserver decides to exit due to the
> >> >> disconnect
> >> >> >> > status
> >> >> >> > >> >> from zookeeper, sometimes the regionserver hangs during
> the
> >> >> >> exiting.
> >> >> >> > We
> >> >> >> > >> can
> >> >> >> > >> >> still see the HRegionServer process even we don't see it
> in
> >> the
> >> >> >> > Master
> >> >> >> > >> web
> >> >> >> > >> >> interface.
> >> >> >> > >> >>
> >> >> >> > >> >> I also notice that there is an zookeeper.retries and
> >> >> >> zookeeper.pauses
> >> >> >> > >> >> settings in hbase-default.xml, would that matter ?
> >> >> >> > >> >>
> >> >> >> > >> >> thanks
> >> >> >> > >> >>
> >> >> >> > >> >> You may see the IOException error in the following log
> when
> >> the
> >> >> >> > >> >> Regionserver lost connection to the zookeeper.
> >> >> >> > >> >>
> >> >> >> > >> >> 2009-11-14 15:58:31,993 INFO
> >> >> >> > org.apache.hadoop.hbase.regionserver.HLog:
> >> >> >> > >> >> Roll
> >> >> >> > >> >>
> >> >> >> > >>
> >> >> >> >
> >> >> >>
> >> >>
> >>
> /hbase/.logs/superpyxis0008.scur.colo,60021,1258179101617/hlog.dat.1258179106921,
> >> >> >> > >> >> entries=1062964, calcsize=255014936, filesize=139566093.
> New
> >> >> hlog
> >> >> >> > >> >>
> >> >> >> > >>
> >> >> >> >
> >> >> >>
> >> >>
> >>
> /hbase/.logs/superpyxis0008.scur.colo,60021,1258179101617/hlog.dat.1258214311993
> >> >> >> > >> >> 2009-11-14 15:58:39,295 DEBUG
> >> org.apache.zookeeper.ClientCnxn:
> >> >> Got
> >> >> >> > ping
> >> >> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
> >> >> >> > >> >> 2009-11-14 15:58:52,648 DEBUG
> >> org.apache.zookeeper.ClientCnxn:
> >> >> Got
> >> >> >> > ping
> >> >> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
> >> >> >> > >> >> 2009-11-14 15:59:06,007 DEBUG
> >> org.apache.zookeeper.ClientCnxn:
> >> >> Got
> >> >> >> > ping
> >> >> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
> >> >> >> > >> >> 2009-11-14 15:59:19,365 DEBUG
> >> org.apache.zookeeper.ClientCnxn:
> >> >> Got
> >> >> >> > ping
> >> >> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
> >> >> >> > >> >> 2009-11-14 15:59:32,867 DEBUG
> >> org.apache.zookeeper.ClientCnxn:
> >> >> Got
> >> >> >> > ping
> >> >> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 186ms
> >> >> >> > >> >> 2009-11-14 15:59:46,070 DEBUG
> >> org.apache.zookeeper.ClientCnxn:
> >> >> Got
> >> >> >> > ping
> >> >> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
> >> >> >> > >> >> 2009-11-14 15:59:59,423 DEBUG
> >> org.apache.zookeeper.ClientCnxn:
> >> >> Got
> >> >> >> > ping
> >> >> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
> >> >> >> > >> >> 2009-11-14 16:00:12,775 DEBUG
> >> org.apache.zookeeper.ClientCnxn:
> >> >> Got
> >> >> >> > ping
> >> >> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
> >> >> >> > >> >> 2009-11-14 16:00:28,141 DEBUG
> >> org.apache.zookeeper.ClientCnxn:
> >> >> Got
> >> >> >> > ping
> >> >> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 2010ms
> >> >> >> > >> >> 2009-11-14 16:01:10,378 WARN
> >> org.apache.zookeeper.ClientCnxn:
> >> >> >> > Exception
> >> >> >> > >> >> closing session 0x424f0ed01ea00a6 to
> >> >> >> > >> sun.nio.ch.SelectionKeyImpl@39fbb2d6
> >> >> >> > >> >>
> >> >> >> > >> >> java.io.IOException: Read error rc = -1
> >> >> >> > java.nio.DirectByteBuffer[pos=0
> >> >> >> > >> >> lim=4 cap=4]
> >> >> >> > >> >> at
> >> >> >> > >> >>
> >> >> >>
> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
> >> >> >> > >> >> at
> >> >> >> > >> >>
> >> >> >>
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
> >> >> >> > >> >> 2009-11-14 16:01:10,478 INFO
> >> >> >> > >> >> org.apache.hadoop.hbase.regionserver.HRegionServer: Got
> >> >> ZooKeeper
> >> >> >> > >> event,
> >> >> >> > >> >> state: Disconnected, type: None, path: null
> >> >> >> > >> >> 2009-11-14 16:01:11,333 WARN
> >> >> >> > >> >> org.apache.hadoop.hbase.regionserver.HRegionServer:
> >> Attempt=1
> >> >> >> > >> >>
> >> >> >> > >> >> org.apache.hadoop.hbase.Leases$LeaseStillHeldException
> >> >> >> > >> >> at
> >> >> >> > >> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> >> >> >> > >> >> Method)
> >> >> >> > >> >> at
> >> >> >> > >> >>
> >> >> >> > >>
> >> >> >> >
> >> >> >>
> >> >>
> >>
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> >> >> >> > >> >> at
> >> >> >> > >> >>
> >> >> >> > >>
> >> >> >> >
> >> >> >>
> >> >>
> >>
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> >> >> >> > >> >> at
> >> >> >> > >>
> java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> >> >> >> > >> >> at
> >> >> >> > >> >>
> >> >> >> > >>
> >> >> >> >
> >> >> >>
> >> >>
> >>
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
> >> >> >> > >> >> at
> >> >> >> > >> >>
> >> >> >> > >>
> >> >> >> >
> >> >> >>
> >> >>
> >>
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
> >> >> >> > >> >> at
> >> >> >> > >> >>
> >> >> >> > >>
> >> >> >> >
> >> >> >>
> >> >>
> >>
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
> >> >> >> > >> >> at
> >> >> >> > >> >>
> >> >> >> > >>
> >> >> >> >
> >> >> >>
> >> >>
> >>
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:571)
> >> >> >> > >> >> at java.lang.Thread.run(Thread.java:619)
> >> >> >> > >> >> 2009-11-14 16:01:11,433 INFO
> >> org.apache.zookeeper.ClientCnxn:
> >> >> >> > >> Attempting
> >> >> >> > >> >> connection to server superpyxis0001/192.168.100.115:2181
> >> >> >> > >> >> 2009-11-14 16:01:11,433 INFO
> >> org.apache.zookeeper.ClientCnxn:
> >> >> >> Priming
> >> >> >> > >> >> connection to java.nio.channels.SocketChannel[connected
> >> local=/
> >> >> >> > >> >> 192.168.100.122:59575 remote=superpyxis0001/
> >> >> 192.168.100.115:2181]
> >> >> >> > >> >> 2009-11-14 16:01:11,433 INFO
> >> org.apache.zookeeper.ClientCnxn:
> >> >> >> Server
> >> >> >> > >> >> connection successful
> >> >> >> > >> >> 2009-11-14 16:01:11,433 INFO
> >> >> >> > >> >> org.apache.hadoop.hbase.regionserver.HRegionServer: Got
> >> >> ZooKeeper
> >> >> >> > >> event,
> >> >> >> > >> >> state: Expired, type: None, path: null
> >> >> >> > >> >> 2009-11-14 16:01:11,433 WARN
> >> org.apache.zookeeper.ClientCnxn:
> >> >> >> > Exception
> >> >> >> > >> >> closing session 0x424f0ed01ea00a6 to
> >> >> >> > >> sun.nio.ch.SelectionKeyImpl@6b93d343
> >> >> >> > >> >> java.io.IOException: Session Expired
> >> >> >> > >> >> at
> >> >> >> > >> >>
> >> >> >> > >>
> >> >> >> >
> >> >> >>
> >> >>
> >>
> org.apache.zookeeper.ClientCnxn$SendThread.readConnectResult(ClientCnxn.java:589)
> >> >> >> > >> >> at
> >> >> >> > >> >>
> >> >> >>
> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:709)
> >> >> >> > >> >>
> >> >> >> > >> >> at
> >> >> >> > >> >>
> >> >> >>
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
> >> >> >> > >> >> 2009-11-14 16:01:11,433 ERROR
> >> >> >> > >> >> org.apache.hadoop.hbase.regionserver.HRegionServer:
> >> ZooKeeper
> >> >> >> session
> >> >> >> > >> >> expired
> >> >> >> > >> >> 2009-11-14 16:01:13,513 INFO
> >> org.apache.hadoop.ipc.HBaseServer:
> >> >> >> IPC
> >> >> >> > >> Server
> >> >> >> > >> >> handler 3 on 60021, call put([B@73deb204,
> >> >> >> > >> >> [Lorg.apache.hadoop.hbase.client.Put;@2179600a) from
> >> >> >> > >> 192.168.100.132:40728:
> >> >> >> > >> >> error: java.io.IOException: Server not running, aborting
> >> >> >> > >> >> java.io.IOException: Server not running, aborting
> >> >> >> > >> >>
> >> >> >> > >> >>
> >> >> >> > >> >>
> >> >> >> > >> >>
> >> >> >> > >> >>
> >> >> >> > >> >> On Sat, Nov 14, 2009 at 1:01 AM, Zhenyu Zhong <
> >> >> >> > zhongresearch@gmail.com
> >> >> >> > >> >wrote:
> >> >> >> > >> >>
> >> >> >> > >> >>> Stack,
> >> >> >> > >> >>>
> >> >> >> > >> >>> Thanks a lot!
> >> >> >> > >> >>>
> >> >> >> > >> >>> I found out that the reason why HBase doesn't take the
> >> system
> >> >> >> file
> >> >> >> > >> >>> descriptor value I set before. I started the HBase using
> >> root
> >> >> >> > instead
> >> >> >> > >> of the
> >> >> >> > >> >>> normal hadoop user, while my system configures hadoop
> with
> >> >> higher
> >> >> >> > file
> >> >> >> > >> >>> descriptor value but configures root with 1024 default
> >> value.
> >> >> >> > >> >>>
> >> >> >> > >> >>>
> >> >> >> > >> >>> Now my system has a clean start. However, it seems that
> the
> >> >> >> > >> >>> zookeeper.session.timeout value doesn't take into effect.
> I
> >> >> still
> >> >> >> > >> found
> >> >> >> > >> >>> around 60 seconds pauses from the disconnected
> >> regionserver. I
> >> >> >> > really
> >> >> >> > >> don't
> >> >> >> > >> >>> know why regionserver only times out after 60 seconds
> >> instead
> >> >> of
> >> >> >> 10
> >> >> >> > >> minutes
> >> >> >> > >> >>> which I set for zookeeper.session.timeout.
> >> >> >> > >> >>>
> >> >> >> > >> >>> Is there any other timeout value coming into play before
> >> the
> >> >> >> actual
> >> >> >> > >> >>> session times out?
> >> >> >> > >> >>>
> >> >> >> > >> >>> zhenyu
> >> >> >> > >> >>>
> >> >> >> > >> >>>
> >> >> >> > >> >>>
> >> >> >> > >> >>> On Fri, Nov 13, 2009 at 7:08 PM, stack <stack@duboce.net
> >
> >> >> wrote:
> >> >> >> > >> >>>
> >> >> >> > >> >>>> Ok. Lack of file descriptors manifests in all kinds of
> >> weird
> >> >> >> ways.
> >> >> >> > >> >>>> Hopefully thats it. If not, lets keep digging.
> >> >> >> > >> >>>> St.Ack
> >> >> >> > >> >>>>
> >> >> >> > >> >>>> On Fri, Nov 13, 2009 at 3:44 PM, Zhenyu Zhong <
> >> >> >> > >> zhongresearch@gmail.com
> >> >> >> > >> >>>> >wrote:
> >> >> >> > >> >>>>
> >> >> >> > >> >>>> > Stack,
> >> >> >> > >> >>>> >
> >> >> >> > >> >>>> > You are right, the master started with ulimit -n 1024.
> >> It
> >> >> >> doesn't
> >> >> >> > >> take
> >> >> >> > >> >>>> the
> >> >> >> > >> >>>> > system value.
> >> >> >> > >> >>>> >
> >> >> >> > >> >>>> > Regarding to the too many open files, it looks like
> the
> >> >> same
> >> >> >> as
> >> >> >> > the
> >> >> >> > >> one
> >> >> >> > >> >>>> J-D
> >> >> >> > >> >>>> > put up. But I will get the Master start with higher
> >> value
> >> >> >> first
> >> >> >> > and
> >> >> >> > >> see
> >> >> >> > >> >>>> if
> >> >> >> > >> >>>> > this kind of error goes away.
> >> >> >> > >> >>>> >
> >> >> >> > >> >>>> > thanks a lot!
> >> >> >> > >> >>>> > zhenyu
> >> >> >> > >> >>>> >
> >> >> >> > >> >>>> > On Fri, Nov 13, 2009 at 6:02 PM, stack <
> >> stack@duboce.net>
> >> >> >> wrote:
> >> >> >> > >> >>>> >
> >> >> >> > >> >>>> > > Does it say
> >> >> >> > >> >>>> > >
> >> >> >> > >> >>>> > > ulimit -n 32768
> >> >> >> > >> >>>> > >
> >> >> >> > >> >>>> > > ...as the second line in your log file on start of
> the
> >> >> >> master?
> >> >> >> > >> >>>> > >
> >> >> >> > >> >>>> > > Is the stack trace that complains about too many
> open
> >> >> files
> >> >> >> > same
> >> >> >> > >> as
> >> >> >> > >> >>>> the
> >> >> >> > >> >>>> > one
> >> >> >> > >> >>>> > > in the blog post J-D put up?
> >> >> >> > >> >>>> > >
> >> >> >> > >> >>>> > > St.Ack
> >> >> >> > >> >>>> > >
> >> >> >> > >> >>>> > >
> >> >> >> > >> >>>> > > On Fri, Nov 13, 2009 at 1:37 PM, Zhenyu Zhong <
> >> >> >> > >> >>>> zhongresearch@gmail.com
> >> >> >> > >> >>>> > > >wrote:
> >> >> >> > >> >>>> > >
> >> >> >> > >> >>>> > > > The ulimit file descriptors was set to fs.file-max
> >> >> >> =1578334,
> >> >> >> > >> also
> >> >> >> > >> >>>> in
> >> >> >> > >> >>>> > the
> >> >> >> > >> >>>> > > > limits.conf the value was set to 32768.
> >> >> >> > >> >>>> > > > So these are way higher than the open descriptors
> >> for
> >> >> the
> >> >> >> > >> running
> >> >> >> > >> >>>> > > > processes.
> >> >> >> > >> >>>> > > >
> >> >> >> > >> >>>> > > > thanks
> >> >> >> > >> >>>> > > > zhenyu
> >> >> >> > >> >>>> > > >
> >> >> >> > >> >>>> > > >
> >> >> >> > >> >>>> > > > On Fri, Nov 13, 2009 at 4:33 PM, Stack <
> >> >> >> saint.ack@gmail.com>
> >> >> >> > >> >>>> wrote:
> >> >> >> > >> >>>> > > >
> >> >> >> > >> >>>> > > > > You upped the ulimit file descriptors as per the
> >> >> getting
> >> >> >> > >> started
> >> >> >> > >> >>>> doc?
> >> >> >> > >> >>>> > > > >
> >> >> >> > >> >>>> > > > >
> >> >> >> > >> >>>> > > > >
> >> >> >> > >> >>>> > > > > On Nov 13, 2009, at 1:26 PM, Zhenyu Zhong <
> >> >> >> > >> >>>> zhongresearch@gmail.com>
> >> >> >> > >> >>>> > > > wrote:
> >> >> >> > >> >>>> > > > >
> >> >> >> > >> >>>> > > > > Thanks a lot.
> >> >> >> > >> >>>> > > > >>
> >> >> >> > >> >>>> > > > >>
> >> >> >> > >> >>>> > > > >> Bad news is my kernel is still 2.6.26.
> >> >> >> > >> >>>> > > > >> But it was not a problem before.
> >> >> >> > >> >>>> > > > >>
> >> >> >> > >> >>>> > > > >> Very strange.
> >> >> >> > >> >>>> > > > >>
> >> >> >> > >> >>>> > > > >> zhenyu
> >> >> >> > >> >>>> > > > >>
> >> >> >> > >> >>>> > > > >> On Fri, Nov 13, 2009 at 4:16 PM, Jean-Daniel
> >> Cryans
> >> >> <
> >> >> >> > >> >>>> > > > jdcryans@apache.org
> >> >> >> > >> >>>> > > > >> >wrote:
> >> >> >> > >> >>>> > > > >>
> >> >> >> > >> >>>> > > > >> Looks like
> >> >> >> > >> >>>> > > > >>>
> >> >> >> > >> >>>> > > > >>>
> >> >> >> > >> >>>> > > >
> >> >> >> > >> >>>> > >
> >> >> >> > >> >>>> >
> >> >> >> > >> >>>>
> >> >> >> > >>
> >> >> >> >
> >> >> >>
> >> >>
> >>
> http://pero.blogs.aprilmayjune.org/2009/01/22/hadoop-and-linux-kernel-2627-epoll-limits/
> >> >> >> > >> >>>> > > > >>>
> >> >> >> > >> >>>> > > > >>> J-D
> >> >> >> > >> >>>> > > > >>>
> >> >> >> > >> >>>> > > > >>> On Fri, Nov 13, 2009 at 1:12 PM, Zhenyu Zhong
> <
> >> >> >> > >> >>>> > > zhongresearch@gmail.com
> >> >> >> > >> >>>> > > > >
> >> >> >> > >> >>>> > > > >>> wrote:
> >> >> >> > >> >>>> > > > >>>
> >> >> >> > >> >>>> > > > >>>> Hi,
> >> >> >> > >> >>>> > > > >>>>
> >> >> >> > >> >>>> > > > >>>>
> >> >> >> > >> >>>> > > > >>>> After I re-organize the cluster, the
> experiment
> >> >> ran
> >> >> >> into
> >> >> >> > >> >>>> problem
> >> >> >> > >> >>>> > > > faster
> >> >> >> > >> >>>> > > > >>>>
> >> >> >> > >> >>>> > > > >>> than
> >> >> >> > >> >>>> > > > >>>
> >> >> >> > >> >>>> > > > >>>> before.
> >> >> >> > >> >>>> > > > >>>>
> >> >> >> > >> >>>> > > > >>>> Basically the changes are to use machines
> with
> >> >> less
> >> >> >> > >> resources
> >> >> >> > >> >>>> as
> >> >> >> > >> >>>> > > > >>>>
> >> >> >> > >> >>>> > > > >>> zookeeper
> >> >> >> > >> >>>> > > > >>>
> >> >> >> > >> >>>> > > > >>>> quorums and machines with more resources as
> >> >> >> > regionserver.
> >> >> >> > >> >>>> > > > >>>>
> >> >> >> > >> >>>> > > > >>>> From the log, I still see the pause around 1
> >> >> minute.
> >> >> >> > >> >>>> > > > >>>> I enabled the GC logging, please see
> >> >> >> > >> >>>> > > > >>>>
> >> >> >> > >> >>>> > > > >>>> http://pastebin.com/m1d4ce0f1
> >> >> >> > >> >>>> > > > >>>>
> >> >> >> > >> >>>> > > > >>>> for details.
> >> >> >> > >> >>>> > > > >>>> In general I don't see many pauses in the GC.
> >> >> >> > >> >>>> > > > >>>>
> >> >> >> > >> >>>> > > > >>>> What is more interesting, 1 hour after the
> 1st
> >> >> >> > >> regionserver
> >> >> >> > >> >>>> > > > >>>> disconnected,
> >> >> >> > >> >>>> > > > >>>> the master log complained about too many open
> >> >> files.
> >> >> >> > This
> >> >> >> > >> >>>> didn't
> >> >> >> > >> >>>> > > > happen
> >> >> >> > >> >>>> > > > >>>> before.
> >> >> >> > >> >>>> > > > >>>> I checked the system OS setup as well as the
> >> >> >> > limits.conf.
> >> >> >> > >> I
> >> >> >> > >> >>>> also
> >> >> >> > >> >>>> > > > checked
> >> >> >> > >> >>>> > > > >>>>
> >> >> >> > >> >>>> > > > >>> the
> >> >> >> > >> >>>> > > > >>>
> >> >> >> > >> >>>> > > > >>>> running processes. The total open files don't
> >> >> reach
> >> >> >> the
> >> >> >> > >> limit.
> >> >> >> > >> >>>> I
> >> >> >> > >> >>>> > am
> >> >> >> > >> >>>> > > > >>>>
> >> >> >> > >> >>>> > > > >>> confused
> >> >> >> > >> >>>> > > > >>>
> >> >> >> > >> >>>> > > > >>>> a bit.
> >> >> >> > >> >>>> > > > >>>>
> >> >> >> > >> >>>> > > > >>>>
> >> >> >> > >> >>>> > > > >>>> Please see the following master log.
> >> >> >> > >> >>>> > > > >>>>
> >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:02,114 INFO
> >> >> >> > >> >>>> > > > org.apache.hadoop.hbase.master.BaseScanner:
> >> >> >> > >> >>>> > > > >>>> RegionManager.metaScanner scan of 4658 row(s)
> >> of
> >> >> meta
> >> >> >> > >> region
> >> >> >> > >> >>>> > > {server:
> >> >> >> > >> >>>> > > > >>>> 192.168.100.128:60021, regionname:
> .META.,,1,
> >> >> >> startKey:
> >> >> >> > >> <>}
> >> >> >> > >> >>>> > > complete
> >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:02,114 INFO
> >> >> >> > >> >>>> > > > org.apache.hadoop.hbase.master.BaseScanner:
> >> >> >> > >> >>>> > > > >>>>
> >> >> >> > >> >>>> > > > >>> All
> >> >> >> > >> >>>> > > > >>>
> >> >> >> > >> >>>> > > > >>>> 1 .META. region(s) scanned
> >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:07,677 DEBUG
> >> >> >> > >> org.apache.zookeeper.ClientCnxn:
> >> >> >> > >> >>>> Got
> >> >> >> > >> >>>> > > > ping
> >> >> >> > >> >>>> > > > >>>> response for sessionid:0x424eebf1c10004c
> after
> >> 3ms
> >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:08,178 INFO
> >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> >> >> >> > >> >>>> > > > Exception
> >> >> >> > >> >>>> > > > >>>>
> >> >> >> > >> >>>> > > > >>> in
> >> >> >> > >> >>>> > > > >>>
> >> >> >> > >> >>>> > > > >>>> createBlockOutputStream java.io.IOException:
> >> Bad
> >> >> >> connect
> >> >> >> > >> ack
> >> >> >> > >> >>>> with
> >> >> >> > >> >>>> > > > >>>> firstBadLink 192.168.100.123:50010
> >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:08,178 INFO
> >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> >> >> >> > >> >>>> > > > >>>> Abandoning
> >> >> >> > >> >>>> > > > >>>> block blk_-2808245019291145247_5478039
> >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:09,682 INFO
> >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> >> >> >> > >> >>>> > > > Exception
> >> >> >> > >> >>>> > > > >>>>
> >> >> >> > >> >>>> > > > >>> in
> >> >> >> > >> >>>> > > > >>>
> >> >> >> > >> >>>> > > > >>>> createBlockOutputStream java.io.EOFException
> >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:09,682 INFO
> >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> >> >> >> > >> >>>> > > > >>>> Abandoning
> >> >> >> > >> >>>> > > > >>>> block blk_1074853606841896259_5478048
> >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:10,334 DEBUG
> >> >> >> > >> org.apache.zookeeper.ClientCnxn:
> >> >> >> > >> >>>> Got
> >> >> >> > >> >>>> > > > ping
> >> >> >> > >> >>>> > > > >>>> response for sessionid:0x24eebf1043003c after
> >> 1ms
> >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:21,018 DEBUG
> >> >> >> > >> org.apache.zookeeper.ClientCnxn:
> >> >> >> > >> >>>> Got
> >> >> >> > >> >>>> > > > ping
> >> >> >> > >> >>>> > > > >>>> response for sessionid:0x424eebf1c10004c
> after
> >> 0ms
> >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:23,674 DEBUG
> >> >> >> > >> org.apache.zookeeper.ClientCnxn:
> >> >> >> > >> >>>> Got
> >> >> >> > >> >>>> > > > ping
> >> >> >> > >> >>>> > > > >>>> response for sessionid:0x24eebf1043003c after
> >> 0ms
> >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:24,828 INFO
> >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> >> >> >> > >> >>>> > > > Exception
> >> >> >> > >> >>>> > > > >>>>
> >> >> >> > >> >>>> > > > >>> in
> >> >> >> > >> >>>> > > > >>>
> >> >> >> > >> >>>> > > > >>>> createBlockOutputStream java.io.IOException:
> >> Bad
> >> >> >> connect
> >> >> >> > >> ack
> >> >> >> > >> >>>> with
> >> >> >> > >> >>>> > > > >>>> firstBadLink 192.168.100.123:50010
> >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:24,828 INFO
> >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> >> >> >> > >> >>>> > > > >>>> Abandoning
> >> >> >> > >> >>>> > > > >>>> block blk_-6642544517082142289_5478063
> >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:24,828 INFO
> >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> >> >> >> > >> >>>> > > > Exception
> >> >> >> > >> >>>> > > > >>>>
> >> >> >> > >> >>>> > > > >>> in
> >> >> >> > >> >>>> > > > >>>
> >> >> >> > >> >>>> > > > >>>> createBlockOutputStream
> >> java.net.SocketException:
> >> >> Too
> >> >> >> > many
> >> >> >> > >> >>>> open
> >> >> >> > >> >>>> > > files
> >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:24,828 INFO
> >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> >> >> >> > >> >>>> > > > >>>> Abandoning
> >> >> >> > >> >>>> > > > >>>> block blk_2057511041109796090_5478063
> >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:24,928 INFO
> >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> >> >> >> > >> >>>> > > > Exception
> >> >> >> > >> >>>> > > > >>>>
> >> >> >> > >> >>>> > > > >>> in
> >> >> >> > >> >>>> > > > >>>
> >> >> >> > >> >>>> > > > >>>> createBlockOutputStream
> >> java.net.SocketException:
> >> >> Too
> >> >> >> > many
> >> >> >> > >> >>>> open
> >> >> >> > >> >>>> > > files
> >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:24,928 INFO
> >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> >> >> >> > >> >>>> > > > >>>> Abandoning
> >> >> >> > >> >>>> > > > >>>> block blk_8219260302213892894_5478064
> >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:30,855 INFO
> >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> >> >> >> > >> >>>> > > > Exception
> >> >> >> > >> >>>> > > > >>>>
> >> >> >> > >> >>>> > > > >>> in
> >> >> >> > >> >>>> > > > >>>
> >> >> >> > >> >>>> > > > >>>> createBlockOutputStream
> >> java.net.SocketException:
> >> >> Too
> >> >> >> > many
> >> >> >> > >> >>>> open
> >> >> >> > >> >>>> > > files
> >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:30,855 INFO
> >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> >> >> >> > >> >>>> > > > >>>> Abandoning
> >> >> >> > >> >>>> > > > >>>> block blk_1669205542853067709_5478235
> >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:30,905 INFO
> >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> >> >> >> > >> >>>> > > > Exception
> >> >> >> > >> >>>> > > > >>>>
> >> >> >> > >> >>>> > > > >>> in
> >> >> >> > >> >>>> > > > >>>
> >> >> >> > >> >>>> > > > >>>> createBlockOutputStream
> >> java.net.SocketException:
> >> >> Too
> >> >> >> > many
> >> >> >> > >> >>>> open
> >> >> >> > >> >>>> > > files
> >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:30,905 INFO
> >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> >> >> >> > >> >>>> > > > >>>> Abandoning
> >> >> >> > >> >>>> > > > >>>> block blk_9128897691346270351_5478237
> >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:30,955 INFO
> >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> >> >> >> > >> >>>> > > > Exception
> >> >> >> > >> >>>> > > > >>>>
> >> >> >> > >> >>>> > > > >>> in
> >> >> >> > >> >>>> > > > >>>
> >> >> >> > >> >>>> > > > >>>> createBlockOutputStream
> >> java.net.SocketException:
> >> >> Too
> >> >> >> > many
> >> >> >> > >> >>>> open
> >> >> >> > >> >>>> > > files
> >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:30,955 INFO
> >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> >> >> >> > >> >>>> > > > >>>> Abandoning
> >> >> >> > >> >>>> > > > >>>> block blk_1116845144864123018_5478240
> >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:34,372 DEBUG
> >> >> >> > >> org.apache.zookeeper.ClientCnxn:
> >> >> >> > >> >>>> Got
> >> >> >> > >> >>>> > > > ping
> >> >> >> > >> >>>> > > > >>>> response for sessionid:0x424eebf1c10004c
> after
> >> 0ms
> >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:37,034 DEBUG
> >> >> >> > >> org.apache.zookeeper.ClientCnxn:
> >> >> >> > >> >>>> Got
> >> >> >> > >> >>>> > > > ping
> >> >> >> > >> >>>> > > > >>>> response for sessionid:0x24eebf1043003c after
> >> 0ms
> >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:37,235 WARN
> >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> >> >> >> > >> >>>> > > > >>>>
> >> >> >> > >> >>>> > > > >>> DataStreamer
> >> >> >> > >> >>>> > > > >>>
> >> >> >> > >> >>>> > > > >>>> Exception: java.io.IOException: Too many open
> >> >> files
> >> >> >> > >> >>>> > > > >>>> at sun.nio.ch.IOUtil.initPipe(Native
> >> Method)
> >> >> >> > >> >>>> > > > >>>> at
> >> >> >> > >> >>>> > >
> >> >> >> sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:49)
> >> >> >> > >> >>>> > > > >>>> at
> >> >> >> > >> >>>> > > > >>>>
> >> >> >> > >> >>>> > > > >>>>
> >> >> >> > >> >>>> > > >
> >> >> >> > >> >>>> > >
> >> >> >> > >> >>>> >
> >> >> >> > >> >>>>
> >> >> >> > >>
> >> >> >> >
> >> >> >>
> >> >>
> >>
> sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:18)
> >> >> >> > >> >>>> > > > >>>
> >> >> >> > >> >>>> > > > >>>> at
> >> >> >> > >> >>>> > > > >>>>
> >> >> >> > >> >>>> > > > >>>>
> >> >> >> > >> >>>> > > >
> >> >> >> > >> >>>> > >
> >> >> >> > >> >>>> >
> >> >> >> > >> >>>>
> >> >> >> > >>
> >> >> >> >
> >> >> >>
> >> >>
> >>
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.get(SocketIOWithTimeout.java:407)
> >> >> >> > >> >>>> > > > >>>
> >> >> >> > >> >>>> > > > >>>> at
> >> >> >> > >> >>>> > > > >>>>
> >> >> >> > >> >>>> > > > >>>>
> >> >> >> > >> >>>> > > >
> >> >> >> > >> >>>> > >
> >> >> >> > >> >>>> >
> >> >> >> > >> >>>>
> >> >> >> > >>
> >> >> >> >
> >> >> >>
> >> >>
> >>
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:322)
> >> >> >> > >> >>>> > > > >>>
> >> >> >> > >> >>>> > > > >>>> at
> >> >> >> > >> >>>> > > > >>>>
> >> >> >> > >> >>>> > > > >>>>
> >> >> >> > >> >>>> > > >
> >> >> >> > >> >>>> > >
> >> >> >> > >> >>>> >
> >> >> >> > >> >>>>
> >> >> >> > >>
> >> >> >> >
> >> >> >>
> >> >>
> >>
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
> >> >> >> > >> >>>> > > > >>>
> >> >> >> > >> >>>> > > > >>>> at
> >> >> >> > >> >>>> > > > >>>>
> >> >> >> > >> >>>> > > > >>>>
> >> >> >> > >> >>>> > > >
> >> >> >> > >> >>>> > >
> >> >> >> > >> >>>> >
> >> >> >> > >> >>>>
> >> >> >> > >>
> >> >> >> >
> >> >> >>
> >> >>
> >>
> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
> >> >> >> > >> >>>> > > > >>>
> >> >> >> > >> >>>> > > > >>>> at
> >> >> >> > >> >>>> > > > >>>>
> >> >> >> > >> >>>> > > > >>>>
> >> >> >> > >> >>>> > > >
> >> >> >> > >> >>>> > >
> >> >> >> > >> >>>> >
> >> >> >> > >> >>>>
> >> >> >> > >>
> >> >> >> >
> >> >> >>
> >> >>
> >>
> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
> >> >> >> > >> >>>> > > > >>>
> >> >> >> > >> >>>> > > > >>>> at
> >> >> >> > >> >>>> > > > >>>>
> >> >> >> > >> >>>> > > > >>>
> >> >> >> > >> >>>>
> >> >> >> java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
> >> >> >> > >> >>>> > > > >>>
> >> >> >> > >> >>>> > > > >>>> at
> >> >> >> > >> >>>> java.io.DataOutputStream.write(DataOutputStream.java:90)
> >> >> >> > >> >>>> > > > >>>> at
> >> >> >> > >> >>>> > > > >>>>
> >> >> >> > >> >>>> > > > >>>>
> >> >> >> > >> >>>> > > >
> >> >> >> > >> >>>> > >
> >> >> >> > >> >>>> >
> >> >> >> > >> >>>>
> >> >> >> > >>
> >> >> >> >
> >> >> >>
> >> >>
> >>
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2290)
> >> >> >> > >> >>>> > > > >>>
> >> >> >> > >> >>>> > > > >>>>
> >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:37,235 WARN
> >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> >> >> >> > >> >>>> > Error
> >> >> >> > >> >>>> > > > >>>> Recovery for block
> >> blk_8148813491785406356_5478475
> >> >> >> bad
> >> >> >> > >> >>>> datanode[0]
> >> >> >> > >> >>>> > > > >>>> 192.168.100.123:50010
> >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:37,235 WARN
> >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> >> >> >> > >> >>>> > Error
> >> >> >> > >> >>>> > > > >>>> Recovery for block
> >> blk_8148813491785406356_5478475
> >> >> in
> >> >> >> > >> pipeline
> >> >> >> > >> >>>> > > > >>>> 192.168.100.123:50010, 192.168.100.134:50010
> ,
> >> >> >> > >> >>>> > 192.168.100.122:50010
> >> >> >> > >> >>>> > > :
> >> >> >> > >> >>>> > > > >>>> bad
> >> >> >> > >> >>>> > > > >>>> datanode 192.168.100.123:50010
> >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:37,436 INFO
> >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> >> >> >> > >> >>>> > > > Exception
> >> >> >> > >> >>>> > > > >>>>
> >> >> >> > >> >>>> > > > >>> in
> >> >> >> > >> >>>> > > > >>>
> >> >> >> > >> >>>> > > > >>>> createBlockOutputStream
> >> java.net.SocketException:
> >> >> Too
> >> >> >> > many
> >> >> >> > >> >>>> open
> >> >> >> > >> >>>> > > files
> >> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:37,436 INFO
> >> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> >> >> >> > >> >>>> > > > >>>> Abandoning
> >> >> >> > >> >>>> > > > >>>> block blk_2119727700857186236_5478498
> >> >> >> > >> >>>> > > > >>>>
> >> >> >> > >> >>>> > > > >>>>
> >> >> >> > >> >>>> > > > >>>>
> >> >> >> > >> >>>> > > > >>>>
> >> >> >> > >> >>>> > > > >>>>
> >> >> >> > >> >>>> > > > >>>> On Thu, Nov 12, 2009 at 4:21 PM, Zhenyu Zhong
> <
> >> >> >> > >> >>>> > > > zhongresearch@gmail.com
> >> >> >> > >> >>>> > > > >>>> wrote:
> >> >> >> > >> >>>> > > > >>>>
> >> >> >> > >> >>>> > > > >>>> Will do.
> >> >> >> > >> >>>> > > > >>>>>
> >> >> >> > >> >>>> > > > >>>>> thanks
> >> >> >> > >> >>>> > > > >>>>> zhenyu
> >> >> >> > >> >>>> > > > >>>>>
> >> >> >> > >> >>>> > > > >>>>>
> >> >> >> > >> >>>> > > > >>>>> On Thu, Nov 12, 2009 at 3:33 PM, stack <
> >> >> >> > stack@duboce.net
> >> >> >> > >> >
> >> >> >> > >> >>>> wrote:
> >> >> >> > >> >>>> > > > >>>>>
> >> >> >> > >> >>>> > > > >>>>> Enable GC logging too on this next run (see
> >> >> >> > >> hbase-env.sh).
> >> >> >> > >> >>>> Lets
> >> >> >> > >> >>>> > > try
> >> >> >> > >> >>>> > > > >>>>>>
> >> >> >> > >> >>>> > > > >>>>> and
> >> >> >> > >> >>>> > > > >>>
> >> >> >> > >> >>>> > > > >>>> get
> >> >> >> > >> >>>> > > > >>>>>> to the bottom of whats going on.
> >> >> >> > >> >>>> > > > >>>>>> Thanks,
> >> >> >> > >> >>>> > > > >>>>>> St.Ack
> >> >> >> > >> >>>> > > > >>>>>>
> >> >> >> > >> >>>> > > > >>>>>> On Thu, Nov 12, 2009 at 12:29 PM, Zhenyu
> >> Zhong <
> >> >> >> > >> >>>> > > > >>>>>>
> >> >> >> > >> >>>> > > > >>>>> zhongresearch@gmail.com
> >> >> >> > >> >>>> > > > >>>
> >> >> >> > >> >>>> > > > >>>> wrote:
> >> >> >> > >> >>>> > > > >>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>
> >> >> >> > >> >>>> > > > >>>>>> I can switch the boxes that run zookeeper
> >> with
> >> >> the
> >> >> >> > ones
> >> >> >> > >> >>>> that
> >> >> >> > >> >>>> > run
> >> >> >> > >> >>>> > > > >>>>>>> regionservers.
> >> >> >> > >> >>>> > > > >>>>>>> I will see the results later.
> >> >> >> > >> >>>> > > > >>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>> FYI. The node does have the 10 minutes
> >> >> >> > >> >>>> > zookeeper.session.timeout
> >> >> >> > >> >>>> > > > >>>>>>>
> >> >> >> > >> >>>> > > > >>>>>> value
> >> >> >> > >> >>>> > > > >>>
> >> >> >> > >> >>>> > > > >>>> in
> >> >> >> > >> >>>> > > > >>>>>>
> >> >> >> > >> >>>> > > > >>>>>>> place.
> >> >> >> > >> >>>> > > > >>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>> thanks
> >> >> >> > >> >>>> > > > >>>>>>> zhenyu
> >> >> >> > >> >>>> > > > >>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>> On Thu, Nov 12, 2009 at 3:21 PM, stack <
> >> >> >> > >> stack@duboce.net>
> >> >> >> > >> >>>> > wrote:
> >> >> >> > >> >>>> > > > >>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>> On Thu, Nov 12, 2009 at 11:50 AM, Zhenyu
> >> Zhong
> >> >> <
> >> >> >> > >> >>>> > > > >>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>> zhongresearch@gmail.com
> >> >> >> > >> >>>> > > > >>>>>>
> >> >> >> > >> >>>> > > > >>>>>>> wrote:
> >> >> >> > >> >>>> > > > >>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>> In my cluster, half of the cluster have
> 2
> >> >> disks
> >> >> >> > 400GB
> >> >> >> > >> >>>> each
> >> >> >> > >> >>>> > per
> >> >> >> > >> >>>> > > > >>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>> machine,
> >> >> >> > >> >>>> > > > >>>>>>
> >> >> >> > >> >>>> > > > >>>>>>> and
> >> >> >> > >> >>>> > > > >>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>> half of the cluster have 6 disks per
> >> machine.
> >> >> >> > Maybe
> >> >> >> > >> we
> >> >> >> > >> >>>> > should
> >> >> >> > >> >>>> > > > >>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>> run
> >> >> >> > >> >>>> > > > >>>
> >> >> >> > >> >>>> > > > >>>> zookeeper
> >> >> >> > >> >>>> > > > >>>>>>>>> on the machines with 2 disks and RS on
> >> >> machines
> >> >> >> > with
> >> >> >> > >> 6
> >> >> >> > >> >>>> disks?
> >> >> >> > >> >>>> > > > >>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>> That would make most sense only in the
> >> below,
> >> >> it
> >> >> >> > >> looks
> >> >> >> > >> >>>> like
> >> >> >> > >> >>>> > the
> >> >> >> > >> >>>> > > > >>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>> RS
> >> >> >> > >> >>>> > > > >>>
> >> >> >> > >> >>>> > > > >>>> that
> >> >> >> > >> >>>> > > > >>>>>>
> >> >> >> > >> >>>> > > > >>>>>>> had
> >> >> >> > >> >>>> > > > >>>>>>>> issue had 4 disks?
> >> >> >> > >> >>>> > > > >>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>> BTW, the 10 minutes
> >> zookeeper.session.timeout
> >> >> >> has
> >> >> >> > >> been
> >> >> >> > >> >>>> set
> >> >> >> > >> >>>> > > during
> >> >> >> > >> >>>> > > > >>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>> the
> >> >> >> > >> >>>> > > > >>>>>>
> >> >> >> > >> >>>> > > > >>>>>>> experiment.
> >> >> >> > >> >>>> > > > >>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>> And for sure this node had it in place?
> >> >> >> > >> >>>> > > > >>>>>>>> St.Ack
> >> >> >> > >> >>>> > > > >>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>> thanks
> >> >> >> > >> >>>> > > > >>>>>>>>> zhenyu
> >> >> >> > >> >>>> > > > >>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>> On Thu, Nov 12, 2009 at 2:08 PM, stack <
> >> >> >> > >> stack@duboce.net
> >> >> >> > >> >>>> >
> >> >> >> > >> >>>> > > wrote:
> >> >> >> > >> >>>> > > > >>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>> On Thu, Nov 12, 2009 at 8:40 AM, Zhenyu
> >> >> Zhong <
> >> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>> zhongresearch@gmail.com
> >> >> >> > >> >>>> > > > >>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>> wrote:
> >> >> >> > >> >>>> > > > >>>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>> Though I experienced 2 regionserver
> >> >> >> disconnection
> >> >> >> > >> this
> >> >> >> > >> >>>> > > > >>>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>> morning,
> >> >> >> > >> >>>> > > > >>>
> >> >> >> > >> >>>> > > > >>>> it
> >> >> >> > >> >>>> > > > >>>>>>
> >> >> >> > >> >>>> > > > >>>>>>> looks
> >> >> >> > >> >>>> > > > >>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>> better from the regionserver log.
> (please
> >> >> see
> >> >> >> the
> >> >> >> > >> >>>> following
> >> >> >> > >> >>>> > > > >>>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>> log)
> >> >> >> > >> >>>> > > > >>>
> >> >> >> > >> >>>> > > > >>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>>> http://pastebin.com/m496dbfae
> >> >> >> > >> >>>> > > > >>>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>>> I checked diskIO for one of the
> >> >> >> > >> >>>> > regionserver(192.168.100.116)
> >> >> >> > >> >>>> > > > >>>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>> during
> >> >> >> > >> >>>> > > > >>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>> the
> >> >> >> > >> >>>> > > > >>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>> time it disconnected.
> >> >> >> > >> >>>> > > > >>>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>>> Time: 03:04:58 AM
> >> >> >> > >> >>>> > > > >>>>>>>>>>> Device: tps Blk_read/s
> >> >> >> Blk_wrtn/s
> >> >> >> > >> >>>> Blk_read
> >> >> >> > >> >>>> > > > >>>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>> Blk_wrtn
> >> >> >> > >> >>>> > > > >>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>> sda 105.31 5458.83
> >> >> 19503.64
> >> >> >> > >> >>>> 9043873239
> >> >> >> > >> >>>> > > > >>>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>> 32312473676
> >> >> >> > >> >>>> > > > >>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>> sda1 2.90 6.64
> >> >> 99.25
> >> >> >> > >> >>>> 10993934
> >> >> >> > >> >>>> > > > >>>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>> 164433464
> >> >> >> > >> >>>> > > > >>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>> sda2 1.72 23.76
> >> >> 30.25
> >> >> >> > >> >>>> 39365817
> >> >> >> > >> >>>> > > > >>>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>> 50124008
> >> >> >> > >> >>>> > > > >>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>> sda3 0.30 0.38
> >> >> 3.58
> >> >> >> > >> >>>> 624930
> >> >> >> > >> >>>> > > > >>>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>> 5923000
> >> >> >> > >> >>>> > > > >>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>> sda4 100.39 5428.06
> >> >> 19370.56
> >> >> >> > >> >>>> 8992888270
> >> >> >> > >> >>>> > > > >>>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>> 32091993204
> >> >> >> > >> >>>> > > > >>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>>> Is this high for you? 20k
> >> blocks/second
> >> >> >> would
> >> >> >> > >> seem to
> >> >> >> > >> >>>> be
> >> >> >> > >> >>>> > > high
> >> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>> but
> >> >> >> > >> >>>> > > > >>>
> >> >> >> > >> >>>> > > > >>>> its
> >> >> >> > >> >>>> > > > >>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>> one
> >> >> >> > >> >>>> > > > >>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>> disk only and its not being shared by
> zk
> >> >> >> anymore
> >> >> >> > so
> >> >> >> > >> >>>> > shouldn't
> >> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>> matter?
> >> >> >> > >> >>>> > > > >>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>> I also checked the zookeeper quorum
> >> server
> >> >> >> that
> >> >> >> > the
> >> >> >> > >> >>>> > > > >>>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>> regionserver
> >> >> >> > >> >>>> > > > >>>
> >> >> >> > >> >>>> > > > >>>> tried
> >> >> >> > >> >>>> > > > >>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>> to
> >> >> >> > >> >>>> > > > >>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>> connect according to the log. However,
> I
> >> >> don't
> >> >> >> see
> >> >> >> > >> >>>> > > > >>>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>> 192.168.100.116
> >> >> >> > >> >>>> > > > >>>>>>
> >> >> >> > >> >>>> > > > >>>>>>> in
> >> >> >> > >> >>>> > > > >>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>> the
> >> >> >> > >> >>>> > > > >>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>> client list of the zookeeper quorum
> >> server
> >> >> that
> >> >> >> > the
> >> >> >> > >> >>>> > > > >>>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>> regionserver
> >> >> >> > >> >>>> > > > >>>
> >> >> >> > >> >>>> > > > >>>> tried
> >> >> >> > >> >>>> > > > >>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>> to
> >> >> >> > >> >>>> > > > >>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>> connect.
> >> >> >> > >> >>>> > > > >>>>>>>>>>> Would that be a problem?
> >> >> >> > >> >>>> > > > >>>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>>> Is that because the ephemeral node
> for
> >> the
> >> >> >> > >> >>>> regionserver
> >> >> >> > >> >>>> > had
> >> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>> evaporated?
> >> >> >> > >> >>>> > > > >>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>> Lost
> >> >> >> > >> >>>> > > > >>>>>>>>>> its lease w/ zk by the time you went to
> >> >> look?
> >> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>> Thu Nov 12 15:04:52 UTC 2009
> >> >> >> > >> >>>> > > > >>>>>>>>>>> Zookeeper version: 3.2.1-808558, built
> >> on
> >> >> >> > >> 08/27/2009
> >> >> >> > >> >>>> 18:48
> >> >> >> > >> >>>> > > > >>>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>> GMT
> >> >> >> > >> >>>> > > > >>>
> >> >> >> > >> >>>> > > > >>>> Clients:
> >> >> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.127:43045
> >> >> >> > >> [1](queued=0,recved=26,sent=0)
> >> >> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.131:39091
> >> >> >> > >> [1](queued=0,recved=964,sent=0)
> >> >> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.124:35961
> >> >> >> > >> [1](queued=0,recved=3266,sent=0)
> >> >> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.123:47935
> >> >> >> > >> [1](queued=0,recved=233,sent=0)
> >> >> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.125:46931
> >> >> >> > >> [1](queued=0,recved=2,sent=0)
> >> >> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.118:54924
> >> >> >> > >> [1](queued=0,recved=295,sent=0)
> >> >> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.118:41390
> >> >> >> > >> [1](queued=0,recved=2290,sent=0)
> >> >> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.136:42243
> >> >> >> > >> [1](queued=0,recved=0,sent=0)
> >> >> >> > >> >>>> > > > >>>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>>> Latency min/avg/max: 0/17/6333
> >> >> >> > >> >>>> > > > >>>>>>>>>>> Received: 47111
> >> >> >> > >> >>>> > > > >>>>>>>>>>> Sent: 0
> >> >> >> > >> >>>> > > > >>>>>>>>>>> Outstanding: 0
> >> >> >> > >> >>>> > > > >>>>>>>>>>> Zxid: 0x77000083f4
> >> >> >> > >> >>>> > > > >>>>>>>>>>> Mode: leader
> >> >> >> > >> >>>> > > > >>>>>>>>>>> Node count: 23
> >> >> >> > >> >>>> > > > >>>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>> That 6 second maximum latency is pretty
> >> bad
> >> >> but
> >> >> >> > >> should
> >> >> >> > >> >>>> be
> >> >> >> > >> >>>> > well
> >> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>> within
> >> >> >> > >> >>>> > > > >>>>>>
> >> >> >> > >> >>>> > > > >>>>>>> the
> >> >> >> > >> >>>> > > > >>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>> zk
> >> >> >> > >> >>>> > > > >>>>>>>>>> session timeout.
> >> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>> So, problem is likely on the zk client
> >> side
> >> >> of
> >> >> >> the
> >> >> >> > >> >>>> session;
> >> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>> i.e.
> >> >> >> > >> >>>> > > > >>>
> >> >> >> > >> >>>> > > > >>>> in
> >> >> >> > >> >>>> > > > >>>>>>
> >> >> >> > >> >>>> > > > >>>>>>> the
> >> >> >> > >> >>>> > > > >>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>> RS.
> >> >> >> > >> >>>> > > > >>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>> You could enable GC logging as J-D
> >> suggested
> >> >> to
> >> >> >> > see
> >> >> >> > >> if
> >> >> >> > >> >>>> you
> >> >> >> > >> >>>> > > have
> >> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>> any
> >> >> >> > >> >>>> > > > >>>>>>
> >> >> >> > >> >>>> > > > >>>>>>> big
> >> >> >> > >> >>>> > > > >>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>> pauses, pauses > zk session timeout.
> >> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>> When the RS went down, it didn't look
> too
> >> >> >> heavily
> >> >> >> > >> >>>> loaded:
> >> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>> 1. 2009-11-12 15:04:52,830 INFO
> >> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> >> > org.apache.hadoop.hbase.regionserver.HRegionServer:
> >> >> >> > >> >>>> Dump of
> >> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>> metrics:
> >> >> >> > >> >>>> > > > >>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>> request=1.5166667, regions=322,
> >> stores=657,
> >> >> >> > >> >>>> storefiles=631,
> >> >> >> > >> >>>> > > > >>>>>>>>>> storefileIndexSize=61,
> >> memstoreSize=1472,
> >> >> >> > >> >>>> usedHeap=2819,
> >> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>> maxHeap=4079,
> >> >> >> > >> >>>> > > > >>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>> blockCacheSize=658110960,
> >> >> >> > blockCacheFree=197395984,
> >> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>> blockCacheCount=9903,
> >> >> >> > >> >>>> > > > >>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>> blockCacheHitRatio=99
> >> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>> Its serving a few reads? The number of
> >> >> store
> >> >> >> > files
> >> >> >> > >> >>>> seems
> >> >> >> > >> >>>> > > fine.
> >> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>> Not
> >> >> >> > >> >>>> > > > >>>>>>
> >> >> >> > >> >>>> > > > >>>>>>> too
> >> >> >> > >> >>>> > > > >>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>> much memory used.
> >> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>> Looking at the logs, I see the Lease
> >> Still
> >> >> Held
> >> >> >> > >> >>>> exception.
> >> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>> This
> >> >> >> > >> >>>> > > > >>>
> >> >> >> > >> >>>> > > > >>>> happens
> >> >> >> > >> >>>> > > > >>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>> when the RS does its regular report to
> the
> >> >> >> master
> >> >> >> > but
> >> >> >> > >> the
> >> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>> master
> >> >> >> > >> >>>> > > > >>>
> >> >> >> > >> >>>> > > > >>>> thinks
> >> >> >> > >> >>>> > > > >>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>> the
> >> >> >> > >> >>>> > > > >>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>> RS has since restarted. It'll think
> this
> >> >> >> probably
> >> >> >> > >> >>>> because
> >> >> >> > >> >>>> > it
> >> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>> noticed
> >> >> >> > >> >>>> > > > >>>>>>
> >> >> >> > >> >>>> > > > >>>>>>> that
> >> >> >> > >> >>>> > > > >>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>> the RS's znode in zk had gone away and
> it
> >> >> >> > considered
> >> >> >> > >> the
> >> >> >> > >> >>>> RS
> >> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>> dead.
> >> >> >> > >> >>>> > > > >>>
> >> >> >> > >> >>>> > > > >>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>> Looking too at your logs I see this
> gap
> >> in
> >> >> the
> >> >> >> zk
> >> >> >> > >> >>>> pinging:
> >> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>> 1. 2009-11-12 15:03:39,325 DEBUG
> >> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>> org.apache.zookeeper.ClientCnxn:
> >> >> >> > >> >>>> > > > >>>>>>
> >> >> >> > >> >>>> > > > >>>>>>> Got
> >> >> >> > >> >>>> > > > >>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>> ping response for
> >> sessionid:0x224e55436ad0004
> >> >> >> after
> >> >> >> > >> 0ms
> >> >> >> > >> >>>> > > > >>>>>>>>>> 2. 2009-11-12 15:03:43,113 DEBUG
> >> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>> org.apache.zookeeper.ClientCnxn:
> >> >> >> > >> >>>> > > > >>>>>>
> >> >> >> > >> >>>> > > > >>>>>>> Got
> >> >> >> > >> >>>> > > > >>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>> ping response for
> >> sessionid:0x24e55436a0007d
> >> >> >> after
> >> >> >> > >> 0ms
> >> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>> Where in the lines above it, its
> >> reporting
> >> >> >> about
> >> >> >> > >> every
> >> >> >> > >> >>>> ten
> >> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>> seconds,
> >> >> >> > >> >>>> > > > >>>>>>
> >> >> >> > >> >>>> > > > >>>>>>> here
> >> >> >> > >> >>>> > > > >>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>> there is a big gap.
> >> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>> Do you have ganglia or something that
> >> will
> >> >> let
> >> >> >> you
> >> >> >> > >> look
> >> >> >> > >> >>>> more
> >> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>> into
> >> >> >> > >> >>>> > > > >>>
> >> >> >> > >> >>>> > > > >>>> what
> >> >> >> > >> >>>> > > > >>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>> was
> >> >> >> > >> >>>> > > > >>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>> happening on this machine around this
> >> time?
> >> >> Is
> >> >> >> > the
> >> >> >> > >> >>>> machine
> >> >> >> > >> >>>> > > OK?
> >> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>> It
> >> >> >> > >> >>>> > > > >>>>>>
> >> >> >> > >> >>>> > > > >>>>>>> looks
> >> >> >> > >> >>>> > > > >>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>> lightly loaded and you have your cluster
> >> >> nicely
> >> >> >> > laid
> >> >> >> > >> out.
> >> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>> Something
> >> >> >> > >> >>>> > > > >>>>>>
> >> >> >> > >> >>>> > > > >>>>>>> odd
> >> >> >> > >> >>>> > > > >>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>> is
> >> >> >> > >> >>>> > > > >>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>> going on. What about things like the
> >> write
> >> >> >> speed
> >> >> >> > to
> >> >> >> > >> >>>> disk?
> >> >> >> > >> >>>> > In
> >> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>> the
> >> >> >> > >> >>>> > > > >>>
> >> >> >> > >> >>>> > > > >>>> past
> >> >> >> > >> >>>> > > > >>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>> strange issues have been explained by
> >> >> incorrectly
> >> >> >> > set
> >> >> >> > >> BIOS
> >> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>> which
> >> >> >> > >> >>>> > > > >>>
> >> >> >> > >> >>>> > > > >>>> made
> >> >> >> > >> >>>> > > > >>>>>>
> >> >> >> > >> >>>> > > > >>>>>>> disks
> >> >> >> > >> >>>> > > > >>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>> run at 1/100th of their proper speed.
> >> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>> St.Ack
> >> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>>> Best,
> >> >> >> > >> >>>> > > > >>>>>>>>>>> zhenyu
> >> >> >> > >> >>>> > > > >>>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>>> On Wed, Nov 11, 2009 at 3:58 PM,
> Zhenyu
> >> >> Zhong
> >> >> >> <
> >> >> >> > >> >>>> > > > >>>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>> zhongresearch@gmail.com
> >> >> >> > >> >>>> > > > >>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>> wrote:
> >> >> >> > >> >>>> > > > >>>>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>>> Stack
> >> >> >> > >> >>>> > > > >>>>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>>>> I am very appreciated for your
> >> comments.
> >> >> >> > >> >>>> > > > >>>>>>>>>>>> I will use the zookeeper monitoring
> >> script
> >> >> on
> >> >> >> my
> >> >> >> > >> >>>> cluster
> >> >> >> > >> >>>> > > > >>>>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>>> and
> >> >> >> > >> >>>> > > > >>>
> >> >> >> > >> >>>> > > > >>>> let
> >> >> >> > >> >>>> > > > >>>>>>
> >> >> >> > >> >>>> > > > >>>>>>> it
> >> >> >> > >> >>>> > > > >>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>> run
> >> >> >> > >> >>>> > > > >>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>> overnight to see the result.
> >> >> >> > >> >>>> > > > >>>>>>>>>>>> I will follow up the thread when I
> get
> >> >> >> anything.
> >> >> >> > >> >>>> > > > >>>>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>>>> thanks
> >> >> >> > >> >>>> > > > >>>>>>>>>>>> zhenyu
> >> >> >> > >> >>>> > > > >>>>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>>>> On Wed, Nov 11, 2009 at 3:52 PM,
> stack
> >> <
> >> >> >> > >> >>>> stack@duboce.net>
> >> >> >> > >> >>>> > > > >>>>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>>> wrote:
> >> >> >> > >> >>>> > > > >>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>>>> I see these in your log too:
> >> >> >> > >> >>>> > > > >>>>>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>>>>> 1. 2009-11-11 04:27:19,018 DEBUG
> >> >> >> > >> >>>> > > > >>>>>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
> >> >> >> > >> >>>> > > > >>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>> Got
> >> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>>> ping response for
> >> >> sessionid:0x424dfd908c50009
> >> >> >> > >> after
> >> >> >> > >> >>>> > > > >>>>>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>>>> 4544ms
> >> >> >> > >> >>>> > > > >>>
> >> >> >> > >> >>>> > > > >>>> 2. 2009-11-11 04:27:19,018 DEBUG
> >> >> >> > >> >>>> > > > >>>>>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
> >> >> >> > >> >>>> > > > >>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>> Got
> >> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>>> ping response for
> >> >> sessionid:0x24dfd90c810002
> >> >> >> > after
> >> >> >> > >> >>>> > > > >>>>>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>>>> 4548ms
> >> >> >> > >> >>>> > > > >>>
> >> >> >> > >> >>>> > > > >>>> 3. 2009-11-11 04:27:43,960 DEBUG
> >> >> >> > >> >>>> > > > >>>>>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
> >> >> >> > >> >>>> > > > >>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>> Got
> >> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>>> ping response for
> >> >> sessionid:0x424dfd908c50009
> >> >> >> > >> after
> >> >> >> > >> >>>> > > > >>>>>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>>>> 9030ms
> >> >> >> > >> >>>> > > > >>>
> >> >> >> > >> >>>> > > > >>>> 4. 2009-11-11 04:27:43,960 DEBUG
> >> >> >> > >> >>>> > > > >>>>>>>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>>>>>> org.apache.zookeeper.
> >> >> >> > >> >>>> > > > >>>>>>>>
> >> >> >> > >> >>>> > > > >>>>>>>
> >> >> >> > >> >>>> > > >
> >> >> >> > >> >>>> > >
> >> >> >> > >> >>>> >
> >> >> >> > >> >>>>
> >> >> >> > >> >>>
> >> >> >> > >> >>>
> >> >> >> > >> >>
> >> >> >> > >> >
> >> >> >> > >>
> >> >> >> > >
> >> >> >> > >
> >> >> >> >
> >> >> >>
> >> >> >
> >> >> >
> >> >>
> >> >
> >> >
> >>
> >
> >
>
Re: regionserver disconnection
Posted by Zhenyu Zhong <zh...@gmail.com>.
After digging more on the zookeeper log, I found that there seemed to be a
socketconnection timeout. Also some warnings indicate that zookeeper server
is not running. Probably I need to understand the mechanism of zookeeper
first.
Please find the zookeeper log around that time.
http://pastebin.com/m37bb4ad1
Many thanks
zhenyu
On Thu, Nov 19, 2009 at 10:31 AM, Zhenyu Zhong <zh...@gmail.com>wrote:
> Stack,
>
> I am very appreciated. Let me dig into the zookeeper log more.
>
> FYI, I saw the sleeper complained about the delay for the first time. It
> looks like there are different reasons for the RS disconnections.
>
> 2009-11-19 08:08:32,708 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
> response for sessionid:0x42508716bfc0001 after 0ms
> 2009-11-19 08:08:37,595 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
> response for sessionid:0x42508716bfc0082 after 0ms
> 2009-11-19 08:08:46,073 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
> response for sessionid:0x42508716bfc0001 after 0ms
> 2009-11-19 08:09:39,880 WARN org.apache.zookeeper.ClientCnxn: Exception
> closing session 0x42508716bfc0001 to sun.nio.ch.SelectionKeyImpl@4f3a608f
> java.io.IOException: TIMED OUT
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:906)
> 2009-11-19 08:09:39,880 WARN org.apache.zookeeper.ClientCnxn: Exception
> closing session 0x42508716bfc0082 to sun.nio.ch.SelectionKeyImpl@14c8875
> java.io.IOException: TIMED OUT
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:906)
> 2009-11-19 08:09:39,880 WARN org.apache.hadoop.hbase.util.Sleeper: We slept
> 53556ms, ten times longer than scheduled: 3000
> 2009-11-19 08:09:39,980 WARN
> org.apache.hadoop.hbase.regionserver.HRegionServer: Attempt=1
> org.apache.hadoop.hbase.Leases$LeaseStillHeldException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
> at
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
> at
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:571)
> at java.lang.Thread.run(Thread.java:619)
> 2009-11-19 08:09:39,980 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
> state: Disconnected, type: None, path: null
> 2009-11-19 08:09:40,530 INFO org.apache.zookeeper.ClientCnxn: Attempting
> connection to server superpyxis0005/192.168.100.119:2181
> 2009-11-19 08:09:40,530 INFO org.apache.zookeeper.ClientCnxn: Priming
> connection to java.nio.channels.SocketChannel[connected local=/
> 192.168.100.122:40659 remote=superpyxis0005/192.168.100.119:2181]
>
>
>
> Best,
> zhenyu
>
> On Wed, Nov 18, 2009 at 6:52 PM, stack <st...@duboce.net> wrote:
>
>> Patrick Hunt, one of the zk lads, answered with the following:
>>
>>
>>
>>
>> According to the following snippet (I cut to just session
>> 0x425035c48210002)
>> the client is getting pings for this session id regularly every 6 seconds
>> or
>> so, then after 41,016 there's a 20 second gap (corresponds to a 30 second
>> timeout being set during session creation) at which point we disconnect
>> from
>> the server.
>>
>>
>> > 2009-11-18 12:41:34,788 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
>> > response for sessionid:0x425035c48210002 after 0ms
>> > 2009-11-18 12:42:01,482 WARN org.apache.zookeeper.ClientCnxn: Exception
>> > closing session 0x425035c48210002 to
>> sun.nio.ch.SelectionKeyImpl@421690ab
>> > java.io.IOException: TIMED OUT
>>
>> the client notifies hbase of the disco, and attempts to reconnect to a
>> server
>>
>>
>> > org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
>> > state: Disconnected, type: None, path: null
>> > 2009-11-18 12:42:01,782 INFO org.apache.zookeeper.ClientCnxn: Attempting
>> > connection to server superpyxis0003/192.168.100.117:2181
>>
>> but the server won't allow the session be be established for some reason
>> (is
>> that the right server:port that's it's trying to connect to?)
>>
>>
>> > 2009-11-18 12:42:02,182 INFO org.apache.zookeeper.ClientCnxn: Server
>> > connection successful
>> > 2009-11-18 12:42:02,182 WARN org.apache.zookeeper.ClientCnxn: Exception
>> > closing session 0x425035c48210002 to
>> sun.nio.ch.SelectionKeyImpl@5c07076b
>> > java.io.IOException: Read error rc = -1 java.nio.DirectByteBuffer[pos=0
>> > lim=4 cap=4]
>>
>> The "read error rc = -1" means that the client read from the socket and
>> got
>> EOS. But it's not a session expiration afaict (and should not be at this
>> point in time).
>>
>>
>> You need to look at the ZK server logs for this period in time (for this
>> server superpyxis0003:2181, and superpyxis0001:2181, the client tries to
>> connect to both of these). There should be some indication there why the
>> server is closing down the connection and not allowing the session to be
>> re-established.
>>
>> One thought - by default we limit clients to a max of 10 connections from
>> a
>> particular host to the server. If you exceed this limit the zk server
>> (it's
>> per server though, not per cluster, so perhaps this is not it) will not
>> allow any additional connections from the client host (you can't exceed
>> 10
>> on a server). Could this be it? i.e. the user is running a number of ZK
>> clients (say hbase processes) from the same host? Regardless, get the
>> server
>> log for this time period and it should be more clear.
>>
>> Patrick
>>
>>
>>
>>
>>
>>
>>
>> On Wed, Nov 18, 2009 at 11:18 AM, stack <st...@duboce.net> wrote:
>>
>> > And running stat on each of your zookeeper nodes tells you what about
>> > averages for connect-times?
>> >
>> > Its odd that there is a ping just 20 seconds beforehand and you have set
>> > the zk session timeout at 60 seconds (or ten minutes?). Let me ask the
>> zk
>> > lads about it.
>> >
>> > If you look in the GC log for around this time, do you see anything?
>> >
>> > St.Ack
>> >
>> >
>> >
>> > On Wed, Nov 18, 2009 at 7:32 AM, Zhenyu Zhong <zhongresearch@gmail.com
>> >wrote:
>> >
>> >> After running more experiment, it seems that the disconnection is not
>> >> related to the heavy load job, because the load average is low, the
>> diskio
>> >> is normal, memory Heap was not reached, also the virtual memory stats
>> >> shows
>> >> that no swappings. However, the disconnection still happens.
>> >>
>> >> It looks like this time it pauses for 20 seconds. No idea why
>> regionserver
>> >> disconnected.
>> >>
>> >> Any other suggestions please?
>> >>
>> >>
>> >> 2009-11-18 12:41:00,956 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
>> >> response for sessionid:0x125035c495d02bb after 0ms
>> >> 2009-11-18 12:41:08,074 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
>> >> response for sessionid:0x425035c48210002 after 0ms
>> >> 2009-11-18 12:41:14,302 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
>> >> response for sessionid:0x125035c495d02bb after 0ms
>> >> 2009-11-18 12:41:21,433 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
>> >> response for sessionid:0x425035c48210002 after 0ms
>> >> 2009-11-18 12:41:27,656 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
>> >> response for sessionid:0x125035c495d02bb after 0ms
>> >> 2009-11-18 12:41:34,788 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
>> >> response for sessionid:0x425035c48210002 after 0ms
>> >> 2009-11-18 12:41:41,016 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
>> >> response for sessionid:0x125035c495d02bb after 1ms
>> >> 2009-11-18 12:42:01,482 WARN org.apache.zookeeper.ClientCnxn: Exception
>> >> closing session 0x425035c48210002 to
>> sun.nio.ch.SelectionKeyImpl@421690ab
>> >> java.io.IOException: TIMED OUT
>> >> at
>> >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:906)
>> >> 2009-11-18 12:42:01,582 INFO
>> >> org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper
>> event,
>> >> state: Disconnected, type: None, path: null
>> >> 2009-11-18 12:42:01,782 INFO org.apache.zookeeper.ClientCnxn:
>> Attempting
>> >> connection to server superpyxis0003/192.168.100.117:2181
>> >> 2009-11-18 12:42:01,782 INFO org.apache.zookeeper.ClientCnxn: Priming
>> >> connection to java.nio.channels.SocketChannel[connected local=/
>> >> 192.168.100.132:46610 remote=superpyxis0003/192.168.100.117:2181]
>> >> 2009-11-18 12:42:01,782 INFO org.apache.zookeeper.ClientCnxn: Server
>> >> connection successful
>> >> 2009-11-18 12:42:01,782 WARN org.apache.zookeeper.ClientCnxn: Exception
>> >> closing session 0x425035c48210002 to
>> sun.nio.ch.SelectionKeyImpl@56dc6fac
>> >> java.io.IOException: Read error rc = -1 java.nio.DirectByteBuffer[pos=0
>> >> lim=4 cap=4]
>> >> at
>> >> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
>> >> at
>> >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
>> >> 2009-11-18 12:42:01,782 WARN org.apache.zookeeper.ClientCnxn: Ignoring
>> >> exception during shutdown input
>> >> java.net.SocketException: Transport endpoint is not connected
>> >> at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
>> >> at
>> >> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
>> >> at
>> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>> >> at
>> >> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
>> >> at
>> >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
>> >> 2009-11-18 12:42:01,782 WARN org.apache.zookeeper.ClientCnxn: Ignoring
>> >> exception during shutdown output
>> >> java.net.SocketException: Transport endpoint is not connected
>> >> at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
>> >> at
>> >> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
>> >> at
>> sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
>> >> at
>> >>
>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
>> >> at
>> >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
>> >> 2009-11-18 12:42:02,182 INFO org.apache.zookeeper.ClientCnxn:
>> Attempting
>> >> connection to server superpyxis0001/192.168.100.115:2181
>> >> 2009-11-18 12:42:02,182 INFO org.apache.zookeeper.ClientCnxn: Priming
>> >> connection to java.nio.channels.SocketChannel[connected local=/
>> >> 192.168.100.132:36197 remote=superpyxis0001/192.168.100.115:2181]
>> >> 2009-11-18 12:42:02,182 INFO org.apache.zookeeper.ClientCnxn: Server
>> >> connection successful
>> >> 2009-11-18 12:42:02,182 WARN org.apache.zookeeper.ClientCnxn: Exception
>> >> closing session 0x425035c48210002 to
>> sun.nio.ch.SelectionKeyImpl@5c07076b
>> >> java.io.IOException: Read error rc = -1 java.nio.DirectByteBuffer[pos=0
>> >> lim=4 cap=4]
>> >> at
>> >> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
>> >> at
>> >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
>> >> 2009-11-18 12:42:02,182 WARN org.apache.zookeeper.ClientCnxn: Ignoring
>> >> exception during shutdown input
>> >> java.net.SocketException: Transport endpoint is not connected
>> >> at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
>> >> at
>> >> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
>> >> at
>> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>> >> at
>> >> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
>> >> at
>> >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
>> >> 2009-11-18 12:42:02,182 WARN org.apache.zookeeper.ClientCnxn: Ignoring
>> >> exception during shutdown output
>> >> java.net.SocketException: Transport endpoint is not connected
>> >>
>> >>
>> >> On Mon, Nov 16, 2009 at 4:16 PM, Zhenyu Zhong <zhongresearch@gmail.com
>> >> >wrote:
>> >>
>> >> > Here is the diskIO and CPU around the time we had RS disconnection on
>> >> one
>> >> > machine that runs RegionServer. It doesn't seem to be high. Similar
>> disk
>> >> and
>> >> > cpu usage have been seen before and the HBase was running fine.
>> >> >
>> >> >
>> >> > So far I haven't found why my 10 minutes session timeout doesn't
>> apply.
>> >> > Still digging.
>> >> >
>> >> > Many thanks!
>> >> >
>> >> > Time: 03:01:08 PM
>> >> > avg-cpu: %user %nice %system %iowait %steal %idle
>> >> > 3.82 0.00 0.89 3.97 0.00 91.32
>> >> >
>> >> >
>> >> > Device: tps Blk_read/s Blk_wrtn/s Blk_read
>> Blk_wrtn
>> >> > sda 24.30 1605.39 6679.35 6238623388
>> 25956251720
>> >> > sda1 0.51 1.94 15.71 7547265
>> 61067904
>> >> > sda2 0.13 2.11 5.20 8202079
>> 20189232
>> >> > sda3 0.13 0.25 1.63 987846
>> 6323336
>> >> > sda4 0.00 0.00 0.00 4
>> 0
>> >> > sda5 1.78 0.06 24.14 220010
>> 93817208
>> >> > sda6 0.38 0.42 8.41 1630584
>> 32688152
>> >> > sda7 21.38 1600.61 6624.26 6220035272
>> 25742165888
>> >> > sdb 4.52 767.33 565.91 2981868690
>> 2199132380
>> >> > sdb1 4.52 767.33 565.91 2981866755
>> 2199132372
>> >> > sdc 4.42 742.95 563.00 2887151482
>> 2187823092
>> >> > sdc1 4.42 742.95 563.00 2887149547
>> 2187823084
>> >> > sdd 4.49 750.78 557.25 2917554074
>> 2165513500
>> >> > sdd1 4.49 750.78 557.25 2917552139
>> 2165513492
>> >> > sde 4.52 758.51 569.46 2947593394
>> 2212964236
>> >> > sde1 4.52 758.51 569.46 2947591459
>> 2212964228
>> >> > sdf 4.51 747.22 571.78 2903740266
>> 2221972652
>> >> > sdf1 4.51 747.22 571.78 2903738331
>> 2221972644
>> >> >
>> >> > Best,
>> >> > zhenyu
>> >> >
>> >> >
>> >> >
>> >> > On Mon, Nov 16, 2009 at 4:07 PM, stack <st...@duboce.net> wrote:
>> >> >
>> >> >> On Mon, Nov 16, 2009 at 12:05 PM, Zhenyu Zhong <
>> >> zhongresearch@gmail.com
>> >> >> >wrote:
>> >> >>
>> >> >> > I just realized that there was a MapReduce job running during the
>> >> time
>> >> >> the
>> >> >> > regionserver disconnected from the zookeeper.
>> >> >> > That MapReduce Job was processing 500GB data and took about 8
>> minutes
>> >> to
>> >> >> > finish. It launched over 2000 map tasks.
>> >> >>
>> >> >>
>> >> >> There was a tasktracker running on the RegionServer that
>> disconnected?
>> >> Is
>> >> >> the map i/o or cpu heavy? Do you think it could have stole life
>> from
>> >> the
>> >> >> datanode/regionserver?
>> >> >>
>> >> >>
>> >> >>
>> >> >> > I doubt that this introduced
>> >> >> > resource contention between DataNode and RegionServer Node.
>> >> >> > Also during the time that MapReduce job ran, I saw a few errors
>> >> >> indicating
>> >> >> > that
>> >> >> >
>> >> >> > java.io.IOException: Could not obtain block:
>> >> >> > blk_7661247556283580226_4450462 file=/data/xxx.txt.200910141159
>> >> >> > at
>> >> >> >
>> >> >>
>> >>
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1787)
>> >> >> > at
>> >> >> >
>> >> >>
>> >>
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1615)
>> >> >> > at
>> >> >> >
>> >> >>
>> >>
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1742)
>> >> >> > at java.io.DataInputStream.read(DataInputStream.java:83)
>> >> >> > at
>> >> >> org.apache.hadoop.util.LineReader.readLine(LineReader.java:134)
>> >> >> > at
>> >> >> >
>> >> >>
>> >>
>> org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:136)
>> >> >> > at
>> >> >> >
>> >>
>> org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:40)
>> >> >> > at
>> >> >> >
>> >> >>
>> >>
>> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:192)
>> >> >> > at
>> >> >> >
>> >> >>
>> >>
>> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:176)
>> >> >> > at
>> org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
>> >> >> > at
>> >> >> org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
>> >> >> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
>> >> >> > at org.apache.hadoop.mapred.Child.main(Child.java:170)
>> >> >> >
>> >> >>
>> >> >> This is interesting. Your map task is running into hdfs-127. Yeah,
>> >> you
>> >> >> should patch your cluster if you want to get rid of these (hdfs-127
>> has
>> >> >> been
>> >> >> applied to 0.21 hadoop and will be in the next hadoop release on
>> 0.20.x
>> >> >> branch, hadoop 0.20.2).
>> >> >>
>> >> >>
>> >> >>
>> >> >> >
>> >> >> > Possibly it is related to HDFS-127, but I don't see any datanodes
>> >> went
>> >> >> > down. Does that affect Regionserver? Shall we apply the patch?
>> >> >> >
>> >> >> >
>> >> >> Read up on hdfs-127 for explaination of whats going on.
>> >> >>
>> >> >>
>> >> >>
>> >> >> > Now I start to keep tracking of the virtual memory stats to see if
>> >> the
>> >> >> same
>> >> >> > issue happens tomorrow around the same time period.
>> >> >> >
>> >> >>
>> >> >> Excellent. Thanks for doing this.
>> >> >>
>> >> >>
>> >> >>
>> >> >> > And I highly suspect that this particular MapReduce job hurts
>> HBase
>> >> >> > Regionserver.
>> >> >> >
>> >> >> > PS. I also use gcviewer to parse the GC-log, I only see around 30
>> >> >> seconds
>> >> >> > pauses maximum.
>> >> >> >
>> >> >>
>> >> >>
>> >> >> Did you figure why your ten minute session is not applying?
>> >> >>
>> >> >> St.Ack
>> >> >>
>> >> >>
>> >> >>
>> >> >> >
>> >> >> > Thanks a lot.
>> >> >> > zhenyu
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > On Sun, Nov 15, 2009 at 12:52 AM, Zhenyu Zhong <
>> >> zhongresearch@gmail.com
>> >> >> > >wrote:
>> >> >> >
>> >> >> > > J-D,
>> >> >> > >
>> >> >> > > Thank you very much for your comments.
>> >> >> > > My company block the IRC port, so I have trouble to connect to
>> IRC
>> >> >> > channel.
>> >> >> > > I have been trying to ask the IT to open the IRC port for me, it
>> >> might
>> >> >> > take
>> >> >> > > a while.
>> >> >> > >
>> >> >> > > Best,
>> >> >> > > zhenyu
>> >> >> > >
>> >> >> > >
>> >> >> > > On Sat, Nov 14, 2009 at 2:21 PM, Jean-Daniel Cryans <
>> >> >> jdcryans@apache.org
>> >> >> > >wrote:
>> >> >> > >
>> >> >> > >> The error you are getting is a disconnection from a zookeeper
>> >> server
>> >> >> > >> and is very generic.
>> >> >> > >>
>> >> >> > >> ZK-86 is still opened and the last comment refers to ZK-111
>> saying
>> >> >> > >> that the bug (in unit tests) was probably fixed in release
>> 3.0.0
>> >> last
>> >> >> > >> year.
>> >> >> > >>
>> >> >> > >> To figure the hang you have, you can try to jstack the process
>> pid
>> >> >> and
>> >> >> > >> see exactly what's holding the RS from shutting down.
>> >> >> > >>
>> >> >> > >> Would it be possible for you to drop by the IRC channel? This
>> way
>> >> we
>> >> >> > >> can debug this a at much faster pace.
>> >> >> > >>
>> >> >> > >> Thx!
>> >> >> > >>
>> >> >> > >> J-D
>> >> >> > >>
>> >> >> > >> On Sat, Nov 14, 2009 at 9:57 AM, Zhenyu Zhong <
>> >> >> zhongresearch@gmail.com>
>> >> >> > >> wrote:
>> >> >> > >> > I found this.
>> >> >> > >> > http://issues.apache.org/jira/browse/ZOOKEEPER-86
>> >> >> > >> >
>> >> >> > >> > It looks like the same error I had. Is it a zookeeper bug?
>> When
>> >> >> will
>> >> >> > >> HBase
>> >> >> > >> > take the zookeeper version 3.3.0?
>> >> >> > >> >
>> >> >> > >> > thanks
>> >> >> > >> > zhenyu
>> >> >> > >> >
>> >> >> > >> > On Sat, Nov 14, 2009 at 11:31 AM, Zhenyu Zhong <
>> >> >> > zhongresearch@gmail.com
>> >> >> > >> >wrote:
>> >> >> > >> >
>> >> >> > >> >> Now I really doubt about the zookeeper, from the log I saw
>> >> errors
>> >> >> > like
>> >> >> > >> >> IOException Read Error, while the zookeeper client
>> >> (regionserver )
>> >> >> > >> tried to
>> >> >> > >> >> read. But it got disconneted status from zookeeper.
>> >> >> > >> >>
>> >> >> > >> >> I don't see any load on any zookeeper quorum servers. DiskIO
>> is
>> >> >> fine.
>> >> >> > >> >>
>> >> >> > >> >> Also when the regionserver decides to exit due to the
>> >> disconnect
>> >> >> > status
>> >> >> > >> >> from zookeeper, sometimes the regionserver hangs during the
>> >> >> exiting.
>> >> >> > We
>> >> >> > >> can
>> >> >> > >> >> still see the HRegionServer process even we don't see it in
>> the
>> >> >> > Master
>> >> >> > >> web
>> >> >> > >> >> interface.
>> >> >> > >> >>
>> >> >> > >> >> I also notice that there is an zookeeper.retries and
>> >> >> zookeeper.pauses
>> >> >> > >> >> settings in hbase-default.xml, would that matter ?
>> >> >> > >> >>
>> >> >> > >> >> thanks
>> >> >> > >> >>
>> >> >> > >> >> You may see the IOException error in the following log when
>> the
>> >> >> > >> >> Regionserver lost connection to the zookeeper.
>> >> >> > >> >>
>> >> >> > >> >> 2009-11-14 15:58:31,993 INFO
>> >> >> > org.apache.hadoop.hbase.regionserver.HLog:
>> >> >> > >> >> Roll
>> >> >> > >> >>
>> >> >> > >>
>> >> >> >
>> >> >>
>> >>
>> /hbase/.logs/superpyxis0008.scur.colo,60021,1258179101617/hlog.dat.1258179106921,
>> >> >> > >> >> entries=1062964, calcsize=255014936, filesize=139566093. New
>> >> hlog
>> >> >> > >> >>
>> >> >> > >>
>> >> >> >
>> >> >>
>> >>
>> /hbase/.logs/superpyxis0008.scur.colo,60021,1258179101617/hlog.dat.1258214311993
>> >> >> > >> >> 2009-11-14 15:58:39,295 DEBUG
>> org.apache.zookeeper.ClientCnxn:
>> >> Got
>> >> >> > ping
>> >> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
>> >> >> > >> >> 2009-11-14 15:58:52,648 DEBUG
>> org.apache.zookeeper.ClientCnxn:
>> >> Got
>> >> >> > ping
>> >> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
>> >> >> > >> >> 2009-11-14 15:59:06,007 DEBUG
>> org.apache.zookeeper.ClientCnxn:
>> >> Got
>> >> >> > ping
>> >> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
>> >> >> > >> >> 2009-11-14 15:59:19,365 DEBUG
>> org.apache.zookeeper.ClientCnxn:
>> >> Got
>> >> >> > ping
>> >> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
>> >> >> > >> >> 2009-11-14 15:59:32,867 DEBUG
>> org.apache.zookeeper.ClientCnxn:
>> >> Got
>> >> >> > ping
>> >> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 186ms
>> >> >> > >> >> 2009-11-14 15:59:46,070 DEBUG
>> org.apache.zookeeper.ClientCnxn:
>> >> Got
>> >> >> > ping
>> >> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
>> >> >> > >> >> 2009-11-14 15:59:59,423 DEBUG
>> org.apache.zookeeper.ClientCnxn:
>> >> Got
>> >> >> > ping
>> >> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
>> >> >> > >> >> 2009-11-14 16:00:12,775 DEBUG
>> org.apache.zookeeper.ClientCnxn:
>> >> Got
>> >> >> > ping
>> >> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
>> >> >> > >> >> 2009-11-14 16:00:28,141 DEBUG
>> org.apache.zookeeper.ClientCnxn:
>> >> Got
>> >> >> > ping
>> >> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 2010ms
>> >> >> > >> >> 2009-11-14 16:01:10,378 WARN
>> org.apache.zookeeper.ClientCnxn:
>> >> >> > Exception
>> >> >> > >> >> closing session 0x424f0ed01ea00a6 to
>> >> >> > >> sun.nio.ch.SelectionKeyImpl@39fbb2d6
>> >> >> > >> >>
>> >> >> > >> >> java.io.IOException: Read error rc = -1
>> >> >> > java.nio.DirectByteBuffer[pos=0
>> >> >> > >> >> lim=4 cap=4]
>> >> >> > >> >> at
>> >> >> > >> >>
>> >> >> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
>> >> >> > >> >> at
>> >> >> > >> >>
>> >> >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
>> >> >> > >> >> 2009-11-14 16:01:10,478 INFO
>> >> >> > >> >> org.apache.hadoop.hbase.regionserver.HRegionServer: Got
>> >> ZooKeeper
>> >> >> > >> event,
>> >> >> > >> >> state: Disconnected, type: None, path: null
>> >> >> > >> >> 2009-11-14 16:01:11,333 WARN
>> >> >> > >> >> org.apache.hadoop.hbase.regionserver.HRegionServer:
>> Attempt=1
>> >> >> > >> >>
>> >> >> > >> >> org.apache.hadoop.hbase.Leases$LeaseStillHeldException
>> >> >> > >> >> at
>> >> >> > >> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>> >> >> > >> >> Method)
>> >> >> > >> >> at
>> >> >> > >> >>
>> >> >> > >>
>> >> >> >
>> >> >>
>> >>
>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>> >> >> > >> >> at
>> >> >> > >> >>
>> >> >> > >>
>> >> >> >
>> >> >>
>> >>
>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>> >> >> > >> >> at
>> >> >> > >> java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>> >> >> > >> >> at
>> >> >> > >> >>
>> >> >> > >>
>> >> >> >
>> >> >>
>> >>
>> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
>> >> >> > >> >> at
>> >> >> > >> >>
>> >> >> > >>
>> >> >> >
>> >> >>
>> >>
>> org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
>> >> >> > >> >> at
>> >> >> > >> >>
>> >> >> > >>
>> >> >> >
>> >> >>
>> >>
>> org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
>> >> >> > >> >> at
>> >> >> > >> >>
>> >> >> > >>
>> >> >> >
>> >> >>
>> >>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:571)
>> >> >> > >> >> at java.lang.Thread.run(Thread.java:619)
>> >> >> > >> >> 2009-11-14 16:01:11,433 INFO
>> org.apache.zookeeper.ClientCnxn:
>> >> >> > >> Attempting
>> >> >> > >> >> connection to server superpyxis0001/192.168.100.115:2181
>> >> >> > >> >> 2009-11-14 16:01:11,433 INFO
>> org.apache.zookeeper.ClientCnxn:
>> >> >> Priming
>> >> >> > >> >> connection to java.nio.channels.SocketChannel[connected
>> local=/
>> >> >> > >> >> 192.168.100.122:59575 remote=superpyxis0001/
>> >> 192.168.100.115:2181]
>> >> >> > >> >> 2009-11-14 16:01:11,433 INFO
>> org.apache.zookeeper.ClientCnxn:
>> >> >> Server
>> >> >> > >> >> connection successful
>> >> >> > >> >> 2009-11-14 16:01:11,433 INFO
>> >> >> > >> >> org.apache.hadoop.hbase.regionserver.HRegionServer: Got
>> >> ZooKeeper
>> >> >> > >> event,
>> >> >> > >> >> state: Expired, type: None, path: null
>> >> >> > >> >> 2009-11-14 16:01:11,433 WARN
>> org.apache.zookeeper.ClientCnxn:
>> >> >> > Exception
>> >> >> > >> >> closing session 0x424f0ed01ea00a6 to
>> >> >> > >> sun.nio.ch.SelectionKeyImpl@6b93d343
>> >> >> > >> >> java.io.IOException: Session Expired
>> >> >> > >> >> at
>> >> >> > >> >>
>> >> >> > >>
>> >> >> >
>> >> >>
>> >>
>> org.apache.zookeeper.ClientCnxn$SendThread.readConnectResult(ClientCnxn.java:589)
>> >> >> > >> >> at
>> >> >> > >> >>
>> >> >> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:709)
>> >> >> > >> >>
>> >> >> > >> >> at
>> >> >> > >> >>
>> >> >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
>> >> >> > >> >> 2009-11-14 16:01:11,433 ERROR
>> >> >> > >> >> org.apache.hadoop.hbase.regionserver.HRegionServer:
>> ZooKeeper
>> >> >> session
>> >> >> > >> >> expired
>> >> >> > >> >> 2009-11-14 16:01:13,513 INFO
>> org.apache.hadoop.ipc.HBaseServer:
>> >> >> IPC
>> >> >> > >> Server
>> >> >> > >> >> handler 3 on 60021, call put([B@73deb204,
>> >> >> > >> >> [Lorg.apache.hadoop.hbase.client.Put;@2179600a) from
>> >> >> > >> 192.168.100.132:40728:
>> >> >> > >> >> error: java.io.IOException: Server not running, aborting
>> >> >> > >> >> java.io.IOException: Server not running, aborting
>> >> >> > >> >>
>> >> >> > >> >>
>> >> >> > >> >>
>> >> >> > >> >>
>> >> >> > >> >>
>> >> >> > >> >> On Sat, Nov 14, 2009 at 1:01 AM, Zhenyu Zhong <
>> >> >> > zhongresearch@gmail.com
>> >> >> > >> >wrote:
>> >> >> > >> >>
>> >> >> > >> >>> Stack,
>> >> >> > >> >>>
>> >> >> > >> >>> Thanks a lot!
>> >> >> > >> >>>
>> >> >> > >> >>> I found out that the reason why HBase doesn't take the
>> system
>> >> >> file
>> >> >> > >> >>> descriptor value I set before. I started the HBase using
>> root
>> >> >> > instead
>> >> >> > >> of the
>> >> >> > >> >>> normal hadoop user, while my system configures hadoop with
>> >> higher
>> >> >> > file
>> >> >> > >> >>> descriptor value but configures root with 1024 default
>> value.
>> >> >> > >> >>>
>> >> >> > >> >>>
>> >> >> > >> >>> Now my system has a clean start. However, it seems that the
>> >> >> > >> >>> zookeeper.session.timeout value doesn't take into effect. I
>> >> still
>> >> >> > >> found
>> >> >> > >> >>> around 60 seconds pauses from the disconnected
>> regionserver. I
>> >> >> > really
>> >> >> > >> don't
>> >> >> > >> >>> know why regionserver only times out after 60 seconds
>> instead
>> >> of
>> >> >> 10
>> >> >> > >> minutes
>> >> >> > >> >>> which I set for zookeeper.session.timeout.
>> >> >> > >> >>>
>> >> >> > >> >>> Is there any other timeout value coming into play before
>> the
>> >> >> actual
>> >> >> > >> >>> session times out?
>> >> >> > >> >>>
>> >> >> > >> >>> zhenyu
>> >> >> > >> >>>
>> >> >> > >> >>>
>> >> >> > >> >>>
>> >> >> > >> >>> On Fri, Nov 13, 2009 at 7:08 PM, stack <st...@duboce.net>
>> >> wrote:
>> >> >> > >> >>>
>> >> >> > >> >>>> Ok. Lack of file descriptors manifests in all kinds of
>> weird
>> >> >> ways.
>> >> >> > >> >>>> Hopefully thats it. If not, lets keep digging.
>> >> >> > >> >>>> St.Ack
>> >> >> > >> >>>>
>> >> >> > >> >>>> On Fri, Nov 13, 2009 at 3:44 PM, Zhenyu Zhong <
>> >> >> > >> zhongresearch@gmail.com
>> >> >> > >> >>>> >wrote:
>> >> >> > >> >>>>
>> >> >> > >> >>>> > Stack,
>> >> >> > >> >>>> >
>> >> >> > >> >>>> > You are right, the master started with ulimit -n 1024.
>> It
>> >> >> doesn't
>> >> >> > >> take
>> >> >> > >> >>>> the
>> >> >> > >> >>>> > system value.
>> >> >> > >> >>>> >
>> >> >> > >> >>>> > Regarding to the too many open files, it looks like the
>> >> same
>> >> >> as
>> >> >> > the
>> >> >> > >> one
>> >> >> > >> >>>> J-D
>> >> >> > >> >>>> > put up. But I will get the Master start with higher
>> value
>> >> >> first
>> >> >> > and
>> >> >> > >> see
>> >> >> > >> >>>> if
>> >> >> > >> >>>> > this kind of error goes away.
>> >> >> > >> >>>> >
>> >> >> > >> >>>> > thanks a lot!
>> >> >> > >> >>>> > zhenyu
>> >> >> > >> >>>> >
>> >> >> > >> >>>> > On Fri, Nov 13, 2009 at 6:02 PM, stack <
>> stack@duboce.net>
>> >> >> wrote:
>> >> >> > >> >>>> >
>> >> >> > >> >>>> > > Does it say
>> >> >> > >> >>>> > >
>> >> >> > >> >>>> > > ulimit -n 32768
>> >> >> > >> >>>> > >
>> >> >> > >> >>>> > > ...as the second line in your log file on start of the
>> >> >> master?
>> >> >> > >> >>>> > >
>> >> >> > >> >>>> > > Is the stack trace that complains about too many open
>> >> files
>> >> >> > same
>> >> >> > >> as
>> >> >> > >> >>>> the
>> >> >> > >> >>>> > one
>> >> >> > >> >>>> > > in the blog post J-D put up?
>> >> >> > >> >>>> > >
>> >> >> > >> >>>> > > St.Ack
>> >> >> > >> >>>> > >
>> >> >> > >> >>>> > >
>> >> >> > >> >>>> > > On Fri, Nov 13, 2009 at 1:37 PM, Zhenyu Zhong <
>> >> >> > >> >>>> zhongresearch@gmail.com
>> >> >> > >> >>>> > > >wrote:
>> >> >> > >> >>>> > >
>> >> >> > >> >>>> > > > The ulimit file descriptors was set to fs.file-max
>> >> >> =1578334,
>> >> >> > >> also
>> >> >> > >> >>>> in
>> >> >> > >> >>>> > the
>> >> >> > >> >>>> > > > limits.conf the value was set to 32768.
>> >> >> > >> >>>> > > > So these are way higher than the open descriptors
>> for
>> >> the
>> >> >> > >> running
>> >> >> > >> >>>> > > > processes.
>> >> >> > >> >>>> > > >
>> >> >> > >> >>>> > > > thanks
>> >> >> > >> >>>> > > > zhenyu
>> >> >> > >> >>>> > > >
>> >> >> > >> >>>> > > >
>> >> >> > >> >>>> > > > On Fri, Nov 13, 2009 at 4:33 PM, Stack <
>> >> >> saint.ack@gmail.com>
>> >> >> > >> >>>> wrote:
>> >> >> > >> >>>> > > >
>> >> >> > >> >>>> > > > > You upped the ulimit file descriptors as per the
>> >> getting
>> >> >> > >> started
>> >> >> > >> >>>> doc?
>> >> >> > >> >>>> > > > >
>> >> >> > >> >>>> > > > >
>> >> >> > >> >>>> > > > >
>> >> >> > >> >>>> > > > > On Nov 13, 2009, at 1:26 PM, Zhenyu Zhong <
>> >> >> > >> >>>> zhongresearch@gmail.com>
>> >> >> > >> >>>> > > > wrote:
>> >> >> > >> >>>> > > > >
>> >> >> > >> >>>> > > > > Thanks a lot.
>> >> >> > >> >>>> > > > >>
>> >> >> > >> >>>> > > > >>
>> >> >> > >> >>>> > > > >> Bad news is my kernel is still 2.6.26.
>> >> >> > >> >>>> > > > >> But it was not a problem before.
>> >> >> > >> >>>> > > > >>
>> >> >> > >> >>>> > > > >> Very strange.
>> >> >> > >> >>>> > > > >>
>> >> >> > >> >>>> > > > >> zhenyu
>> >> >> > >> >>>> > > > >>
>> >> >> > >> >>>> > > > >> On Fri, Nov 13, 2009 at 4:16 PM, Jean-Daniel
>> Cryans
>> >> <
>> >> >> > >> >>>> > > > jdcryans@apache.org
>> >> >> > >> >>>> > > > >> >wrote:
>> >> >> > >> >>>> > > > >>
>> >> >> > >> >>>> > > > >> Looks like
>> >> >> > >> >>>> > > > >>>
>> >> >> > >> >>>> > > > >>>
>> >> >> > >> >>>> > > >
>> >> >> > >> >>>> > >
>> >> >> > >> >>>> >
>> >> >> > >> >>>>
>> >> >> > >>
>> >> >> >
>> >> >>
>> >>
>> http://pero.blogs.aprilmayjune.org/2009/01/22/hadoop-and-linux-kernel-2627-epoll-limits/
>> >> >> > >> >>>> > > > >>>
>> >> >> > >> >>>> > > > >>> J-D
>> >> >> > >> >>>> > > > >>>
>> >> >> > >> >>>> > > > >>> On Fri, Nov 13, 2009 at 1:12 PM, Zhenyu Zhong <
>> >> >> > >> >>>> > > zhongresearch@gmail.com
>> >> >> > >> >>>> > > > >
>> >> >> > >> >>>> > > > >>> wrote:
>> >> >> > >> >>>> > > > >>>
>> >> >> > >> >>>> > > > >>>> Hi,
>> >> >> > >> >>>> > > > >>>>
>> >> >> > >> >>>> > > > >>>>
>> >> >> > >> >>>> > > > >>>> After I re-organize the cluster, the experiment
>> >> ran
>> >> >> into
>> >> >> > >> >>>> problem
>> >> >> > >> >>>> > > > faster
>> >> >> > >> >>>> > > > >>>>
>> >> >> > >> >>>> > > > >>> than
>> >> >> > >> >>>> > > > >>>
>> >> >> > >> >>>> > > > >>>> before.
>> >> >> > >> >>>> > > > >>>>
>> >> >> > >> >>>> > > > >>>> Basically the changes are to use machines with
>> >> less
>> >> >> > >> resources
>> >> >> > >> >>>> as
>> >> >> > >> >>>> > > > >>>>
>> >> >> > >> >>>> > > > >>> zookeeper
>> >> >> > >> >>>> > > > >>>
>> >> >> > >> >>>> > > > >>>> quorums and machines with more resources as
>> >> >> > regionserver.
>> >> >> > >> >>>> > > > >>>>
>> >> >> > >> >>>> > > > >>>> From the log, I still see the pause around 1
>> >> minute.
>> >> >> > >> >>>> > > > >>>> I enabled the GC logging, please see
>> >> >> > >> >>>> > > > >>>>
>> >> >> > >> >>>> > > > >>>> http://pastebin.com/m1d4ce0f1
>> >> >> > >> >>>> > > > >>>>
>> >> >> > >> >>>> > > > >>>> for details.
>> >> >> > >> >>>> > > > >>>> In general I don't see many pauses in the GC.
>> >> >> > >> >>>> > > > >>>>
>> >> >> > >> >>>> > > > >>>> What is more interesting, 1 hour after the 1st
>> >> >> > >> regionserver
>> >> >> > >> >>>> > > > >>>> disconnected,
>> >> >> > >> >>>> > > > >>>> the master log complained about too many open
>> >> files.
>> >> >> > This
>> >> >> > >> >>>> didn't
>> >> >> > >> >>>> > > > happen
>> >> >> > >> >>>> > > > >>>> before.
>> >> >> > >> >>>> > > > >>>> I checked the system OS setup as well as the
>> >> >> > limits.conf.
>> >> >> > >> I
>> >> >> > >> >>>> also
>> >> >> > >> >>>> > > > checked
>> >> >> > >> >>>> > > > >>>>
>> >> >> > >> >>>> > > > >>> the
>> >> >> > >> >>>> > > > >>>
>> >> >> > >> >>>> > > > >>>> running processes. The total open files don't
>> >> reach
>> >> >> the
>> >> >> > >> limit.
>> >> >> > >> >>>> I
>> >> >> > >> >>>> > am
>> >> >> > >> >>>> > > > >>>>
>> >> >> > >> >>>> > > > >>> confused
>> >> >> > >> >>>> > > > >>>
>> >> >> > >> >>>> > > > >>>> a bit.
>> >> >> > >> >>>> > > > >>>>
>> >> >> > >> >>>> > > > >>>>
>> >> >> > >> >>>> > > > >>>> Please see the following master log.
>> >> >> > >> >>>> > > > >>>>
>> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:02,114 INFO
>> >> >> > >> >>>> > > > org.apache.hadoop.hbase.master.BaseScanner:
>> >> >> > >> >>>> > > > >>>> RegionManager.metaScanner scan of 4658 row(s)
>> of
>> >> meta
>> >> >> > >> region
>> >> >> > >> >>>> > > {server:
>> >> >> > >> >>>> > > > >>>> 192.168.100.128:60021, regionname: .META.,,1,
>> >> >> startKey:
>> >> >> > >> <>}
>> >> >> > >> >>>> > > complete
>> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:02,114 INFO
>> >> >> > >> >>>> > > > org.apache.hadoop.hbase.master.BaseScanner:
>> >> >> > >> >>>> > > > >>>>
>> >> >> > >> >>>> > > > >>> All
>> >> >> > >> >>>> > > > >>>
>> >> >> > >> >>>> > > > >>>> 1 .META. region(s) scanned
>> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:07,677 DEBUG
>> >> >> > >> org.apache.zookeeper.ClientCnxn:
>> >> >> > >> >>>> Got
>> >> >> > >> >>>> > > > ping
>> >> >> > >> >>>> > > > >>>> response for sessionid:0x424eebf1c10004c after
>> 3ms
>> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:08,178 INFO
>> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
>> >> >> > >> >>>> > > > Exception
>> >> >> > >> >>>> > > > >>>>
>> >> >> > >> >>>> > > > >>> in
>> >> >> > >> >>>> > > > >>>
>> >> >> > >> >>>> > > > >>>> createBlockOutputStream java.io.IOException:
>> Bad
>> >> >> connect
>> >> >> > >> ack
>> >> >> > >> >>>> with
>> >> >> > >> >>>> > > > >>>> firstBadLink 192.168.100.123:50010
>> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:08,178 INFO
>> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
>> >> >> > >> >>>> > > > >>>> Abandoning
>> >> >> > >> >>>> > > > >>>> block blk_-2808245019291145247_5478039
>> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:09,682 INFO
>> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
>> >> >> > >> >>>> > > > Exception
>> >> >> > >> >>>> > > > >>>>
>> >> >> > >> >>>> > > > >>> in
>> >> >> > >> >>>> > > > >>>
>> >> >> > >> >>>> > > > >>>> createBlockOutputStream java.io.EOFException
>> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:09,682 INFO
>> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
>> >> >> > >> >>>> > > > >>>> Abandoning
>> >> >> > >> >>>> > > > >>>> block blk_1074853606841896259_5478048
>> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:10,334 DEBUG
>> >> >> > >> org.apache.zookeeper.ClientCnxn:
>> >> >> > >> >>>> Got
>> >> >> > >> >>>> > > > ping
>> >> >> > >> >>>> > > > >>>> response for sessionid:0x24eebf1043003c after
>> 1ms
>> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:21,018 DEBUG
>> >> >> > >> org.apache.zookeeper.ClientCnxn:
>> >> >> > >> >>>> Got
>> >> >> > >> >>>> > > > ping
>> >> >> > >> >>>> > > > >>>> response for sessionid:0x424eebf1c10004c after
>> 0ms
>> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:23,674 DEBUG
>> >> >> > >> org.apache.zookeeper.ClientCnxn:
>> >> >> > >> >>>> Got
>> >> >> > >> >>>> > > > ping
>> >> >> > >> >>>> > > > >>>> response for sessionid:0x24eebf1043003c after
>> 0ms
>> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:24,828 INFO
>> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
>> >> >> > >> >>>> > > > Exception
>> >> >> > >> >>>> > > > >>>>
>> >> >> > >> >>>> > > > >>> in
>> >> >> > >> >>>> > > > >>>
>> >> >> > >> >>>> > > > >>>> createBlockOutputStream java.io.IOException:
>> Bad
>> >> >> connect
>> >> >> > >> ack
>> >> >> > >> >>>> with
>> >> >> > >> >>>> > > > >>>> firstBadLink 192.168.100.123:50010
>> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:24,828 INFO
>> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
>> >> >> > >> >>>> > > > >>>> Abandoning
>> >> >> > >> >>>> > > > >>>> block blk_-6642544517082142289_5478063
>> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:24,828 INFO
>> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
>> >> >> > >> >>>> > > > Exception
>> >> >> > >> >>>> > > > >>>>
>> >> >> > >> >>>> > > > >>> in
>> >> >> > >> >>>> > > > >>>
>> >> >> > >> >>>> > > > >>>> createBlockOutputStream
>> java.net.SocketException:
>> >> Too
>> >> >> > many
>> >> >> > >> >>>> open
>> >> >> > >> >>>> > > files
>> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:24,828 INFO
>> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
>> >> >> > >> >>>> > > > >>>> Abandoning
>> >> >> > >> >>>> > > > >>>> block blk_2057511041109796090_5478063
>> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:24,928 INFO
>> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
>> >> >> > >> >>>> > > > Exception
>> >> >> > >> >>>> > > > >>>>
>> >> >> > >> >>>> > > > >>> in
>> >> >> > >> >>>> > > > >>>
>> >> >> > >> >>>> > > > >>>> createBlockOutputStream
>> java.net.SocketException:
>> >> Too
>> >> >> > many
>> >> >> > >> >>>> open
>> >> >> > >> >>>> > > files
>> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:24,928 INFO
>> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
>> >> >> > >> >>>> > > > >>>> Abandoning
>> >> >> > >> >>>> > > > >>>> block blk_8219260302213892894_5478064
>> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:30,855 INFO
>> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
>> >> >> > >> >>>> > > > Exception
>> >> >> > >> >>>> > > > >>>>
>> >> >> > >> >>>> > > > >>> in
>> >> >> > >> >>>> > > > >>>
>> >> >> > >> >>>> > > > >>>> createBlockOutputStream
>> java.net.SocketException:
>> >> Too
>> >> >> > many
>> >> >> > >> >>>> open
>> >> >> > >> >>>> > > files
>> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:30,855 INFO
>> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
>> >> >> > >> >>>> > > > >>>> Abandoning
>> >> >> > >> >>>> > > > >>>> block blk_1669205542853067709_5478235
>> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:30,905 INFO
>> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
>> >> >> > >> >>>> > > > Exception
>> >> >> > >> >>>> > > > >>>>
>> >> >> > >> >>>> > > > >>> in
>> >> >> > >> >>>> > > > >>>
>> >> >> > >> >>>> > > > >>>> createBlockOutputStream
>> java.net.SocketException:
>> >> Too
>> >> >> > many
>> >> >> > >> >>>> open
>> >> >> > >> >>>> > > files
>> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:30,905 INFO
>> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
>> >> >> > >> >>>> > > > >>>> Abandoning
>> >> >> > >> >>>> > > > >>>> block blk_9128897691346270351_5478237
>> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:30,955 INFO
>> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
>> >> >> > >> >>>> > > > Exception
>> >> >> > >> >>>> > > > >>>>
>> >> >> > >> >>>> > > > >>> in
>> >> >> > >> >>>> > > > >>>
>> >> >> > >> >>>> > > > >>>> createBlockOutputStream
>> java.net.SocketException:
>> >> Too
>> >> >> > many
>> >> >> > >> >>>> open
>> >> >> > >> >>>> > > files
>> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:30,955 INFO
>> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
>> >> >> > >> >>>> > > > >>>> Abandoning
>> >> >> > >> >>>> > > > >>>> block blk_1116845144864123018_5478240
>> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:34,372 DEBUG
>> >> >> > >> org.apache.zookeeper.ClientCnxn:
>> >> >> > >> >>>> Got
>> >> >> > >> >>>> > > > ping
>> >> >> > >> >>>> > > > >>>> response for sessionid:0x424eebf1c10004c after
>> 0ms
>> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:37,034 DEBUG
>> >> >> > >> org.apache.zookeeper.ClientCnxn:
>> >> >> > >> >>>> Got
>> >> >> > >> >>>> > > > ping
>> >> >> > >> >>>> > > > >>>> response for sessionid:0x24eebf1043003c after
>> 0ms
>> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:37,235 WARN
>> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
>> >> >> > >> >>>> > > > >>>>
>> >> >> > >> >>>> > > > >>> DataStreamer
>> >> >> > >> >>>> > > > >>>
>> >> >> > >> >>>> > > > >>>> Exception: java.io.IOException: Too many open
>> >> files
>> >> >> > >> >>>> > > > >>>> at sun.nio.ch.IOUtil.initPipe(Native
>> Method)
>> >> >> > >> >>>> > > > >>>> at
>> >> >> > >> >>>> > >
>> >> >> sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:49)
>> >> >> > >> >>>> > > > >>>> at
>> >> >> > >> >>>> > > > >>>>
>> >> >> > >> >>>> > > > >>>>
>> >> >> > >> >>>> > > >
>> >> >> > >> >>>> > >
>> >> >> > >> >>>> >
>> >> >> > >> >>>>
>> >> >> > >>
>> >> >> >
>> >> >>
>> >>
>> sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:18)
>> >> >> > >> >>>> > > > >>>
>> >> >> > >> >>>> > > > >>>> at
>> >> >> > >> >>>> > > > >>>>
>> >> >> > >> >>>> > > > >>>>
>> >> >> > >> >>>> > > >
>> >> >> > >> >>>> > >
>> >> >> > >> >>>> >
>> >> >> > >> >>>>
>> >> >> > >>
>> >> >> >
>> >> >>
>> >>
>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.get(SocketIOWithTimeout.java:407)
>> >> >> > >> >>>> > > > >>>
>> >> >> > >> >>>> > > > >>>> at
>> >> >> > >> >>>> > > > >>>>
>> >> >> > >> >>>> > > > >>>>
>> >> >> > >> >>>> > > >
>> >> >> > >> >>>> > >
>> >> >> > >> >>>> >
>> >> >> > >> >>>>
>> >> >> > >>
>> >> >> >
>> >> >>
>> >>
>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:322)
>> >> >> > >> >>>> > > > >>>
>> >> >> > >> >>>> > > > >>>> at
>> >> >> > >> >>>> > > > >>>>
>> >> >> > >> >>>> > > > >>>>
>> >> >> > >> >>>> > > >
>> >> >> > >> >>>> > >
>> >> >> > >> >>>> >
>> >> >> > >> >>>>
>> >> >> > >>
>> >> >> >
>> >> >>
>> >>
>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
>> >> >> > >> >>>> > > > >>>
>> >> >> > >> >>>> > > > >>>> at
>> >> >> > >> >>>> > > > >>>>
>> >> >> > >> >>>> > > > >>>>
>> >> >> > >> >>>> > > >
>> >> >> > >> >>>> > >
>> >> >> > >> >>>> >
>> >> >> > >> >>>>
>> >> >> > >>
>> >> >> >
>> >> >>
>> >>
>> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
>> >> >> > >> >>>> > > > >>>
>> >> >> > >> >>>> > > > >>>> at
>> >> >> > >> >>>> > > > >>>>
>> >> >> > >> >>>> > > > >>>>
>> >> >> > >> >>>> > > >
>> >> >> > >> >>>> > >
>> >> >> > >> >>>> >
>> >> >> > >> >>>>
>> >> >> > >>
>> >> >> >
>> >> >>
>> >>
>> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
>> >> >> > >> >>>> > > > >>>
>> >> >> > >> >>>> > > > >>>> at
>> >> >> > >> >>>> > > > >>>>
>> >> >> > >> >>>> > > > >>>
>> >> >> > >> >>>>
>> >> >> java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
>> >> >> > >> >>>> > > > >>>
>> >> >> > >> >>>> > > > >>>> at
>> >> >> > >> >>>> java.io.DataOutputStream.write(DataOutputStream.java:90)
>> >> >> > >> >>>> > > > >>>> at
>> >> >> > >> >>>> > > > >>>>
>> >> >> > >> >>>> > > > >>>>
>> >> >> > >> >>>> > > >
>> >> >> > >> >>>> > >
>> >> >> > >> >>>> >
>> >> >> > >> >>>>
>> >> >> > >>
>> >> >> >
>> >> >>
>> >>
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2290)
>> >> >> > >> >>>> > > > >>>
>> >> >> > >> >>>> > > > >>>>
>> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:37,235 WARN
>> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
>> >> >> > >> >>>> > Error
>> >> >> > >> >>>> > > > >>>> Recovery for block
>> blk_8148813491785406356_5478475
>> >> >> bad
>> >> >> > >> >>>> datanode[0]
>> >> >> > >> >>>> > > > >>>> 192.168.100.123:50010
>> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:37,235 WARN
>> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
>> >> >> > >> >>>> > Error
>> >> >> > >> >>>> > > > >>>> Recovery for block
>> blk_8148813491785406356_5478475
>> >> in
>> >> >> > >> pipeline
>> >> >> > >> >>>> > > > >>>> 192.168.100.123:50010, 192.168.100.134:50010,
>> >> >> > >> >>>> > 192.168.100.122:50010
>> >> >> > >> >>>> > > :
>> >> >> > >> >>>> > > > >>>> bad
>> >> >> > >> >>>> > > > >>>> datanode 192.168.100.123:50010
>> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:37,436 INFO
>> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
>> >> >> > >> >>>> > > > Exception
>> >> >> > >> >>>> > > > >>>>
>> >> >> > >> >>>> > > > >>> in
>> >> >> > >> >>>> > > > >>>
>> >> >> > >> >>>> > > > >>>> createBlockOutputStream
>> java.net.SocketException:
>> >> Too
>> >> >> > many
>> >> >> > >> >>>> open
>> >> >> > >> >>>> > > files
>> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:37,436 INFO
>> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
>> >> >> > >> >>>> > > > >>>> Abandoning
>> >> >> > >> >>>> > > > >>>> block blk_2119727700857186236_5478498
>> >> >> > >> >>>> > > > >>>>
>> >> >> > >> >>>> > > > >>>>
>> >> >> > >> >>>> > > > >>>>
>> >> >> > >> >>>> > > > >>>>
>> >> >> > >> >>>> > > > >>>>
>> >> >> > >> >>>> > > > >>>> On Thu, Nov 12, 2009 at 4:21 PM, Zhenyu Zhong <
>> >> >> > >> >>>> > > > zhongresearch@gmail.com
>> >> >> > >> >>>> > > > >>>> wrote:
>> >> >> > >> >>>> > > > >>>>
>> >> >> > >> >>>> > > > >>>> Will do.
>> >> >> > >> >>>> > > > >>>>>
>> >> >> > >> >>>> > > > >>>>> thanks
>> >> >> > >> >>>> > > > >>>>> zhenyu
>> >> >> > >> >>>> > > > >>>>>
>> >> >> > >> >>>> > > > >>>>>
>> >> >> > >> >>>> > > > >>>>> On Thu, Nov 12, 2009 at 3:33 PM, stack <
>> >> >> > stack@duboce.net
>> >> >> > >> >
>> >> >> > >> >>>> wrote:
>> >> >> > >> >>>> > > > >>>>>
>> >> >> > >> >>>> > > > >>>>> Enable GC logging too on this next run (see
>> >> >> > >> hbase-env.sh).
>> >> >> > >> >>>> Lets
>> >> >> > >> >>>> > > try
>> >> >> > >> >>>> > > > >>>>>>
>> >> >> > >> >>>> > > > >>>>> and
>> >> >> > >> >>>> > > > >>>
>> >> >> > >> >>>> > > > >>>> get
>> >> >> > >> >>>> > > > >>>>>> to the bottom of whats going on.
>> >> >> > >> >>>> > > > >>>>>> Thanks,
>> >> >> > >> >>>> > > > >>>>>> St.Ack
>> >> >> > >> >>>> > > > >>>>>>
>> >> >> > >> >>>> > > > >>>>>> On Thu, Nov 12, 2009 at 12:29 PM, Zhenyu
>> Zhong <
>> >> >> > >> >>>> > > > >>>>>>
>> >> >> > >> >>>> > > > >>>>> zhongresearch@gmail.com
>> >> >> > >> >>>> > > > >>>
>> >> >> > >> >>>> > > > >>>> wrote:
>> >> >> > >> >>>> > > > >>>>>>>
>> >> >> > >> >>>> > > > >>>>>>
>> >> >> > >> >>>> > > > >>>>>> I can switch the boxes that run zookeeper
>> with
>> >> the
>> >> >> > ones
>> >> >> > >> >>>> that
>> >> >> > >> >>>> > run
>> >> >> > >> >>>> > > > >>>>>>> regionservers.
>> >> >> > >> >>>> > > > >>>>>>> I will see the results later.
>> >> >> > >> >>>> > > > >>>>>>>
>> >> >> > >> >>>> > > > >>>>>>> FYI. The node does have the 10 minutes
>> >> >> > >> >>>> > zookeeper.session.timeout
>> >> >> > >> >>>> > > > >>>>>>>
>> >> >> > >> >>>> > > > >>>>>> value
>> >> >> > >> >>>> > > > >>>
>> >> >> > >> >>>> > > > >>>> in
>> >> >> > >> >>>> > > > >>>>>>
>> >> >> > >> >>>> > > > >>>>>>> place.
>> >> >> > >> >>>> > > > >>>>>>>
>> >> >> > >> >>>> > > > >>>>>>> thanks
>> >> >> > >> >>>> > > > >>>>>>> zhenyu
>> >> >> > >> >>>> > > > >>>>>>>
>> >> >> > >> >>>> > > > >>>>>>> On Thu, Nov 12, 2009 at 3:21 PM, stack <
>> >> >> > >> stack@duboce.net>
>> >> >> > >> >>>> > wrote:
>> >> >> > >> >>>> > > > >>>>>>>
>> >> >> > >> >>>> > > > >>>>>>> On Thu, Nov 12, 2009 at 11:50 AM, Zhenyu
>> Zhong
>> >> <
>> >> >> > >> >>>> > > > >>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>> zhongresearch@gmail.com
>> >> >> > >> >>>> > > > >>>>>>
>> >> >> > >> >>>> > > > >>>>>>> wrote:
>> >> >> > >> >>>> > > > >>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>> In my cluster, half of the cluster have 2
>> >> disks
>> >> >> > 400GB
>> >> >> > >> >>>> each
>> >> >> > >> >>>> > per
>> >> >> > >> >>>> > > > >>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>> machine,
>> >> >> > >> >>>> > > > >>>>>>
>> >> >> > >> >>>> > > > >>>>>>> and
>> >> >> > >> >>>> > > > >>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>> half of the cluster have 6 disks per
>> machine.
>> >> >> > Maybe
>> >> >> > >> we
>> >> >> > >> >>>> > should
>> >> >> > >> >>>> > > > >>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>> run
>> >> >> > >> >>>> > > > >>>
>> >> >> > >> >>>> > > > >>>> zookeeper
>> >> >> > >> >>>> > > > >>>>>>>>> on the machines with 2 disks and RS on
>> >> machines
>> >> >> > with
>> >> >> > >> 6
>> >> >> > >> >>>> disks?
>> >> >> > >> >>>> > > > >>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>> That would make most sense only in the
>> below,
>> >> it
>> >> >> > >> looks
>> >> >> > >> >>>> like
>> >> >> > >> >>>> > the
>> >> >> > >> >>>> > > > >>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>> RS
>> >> >> > >> >>>> > > > >>>
>> >> >> > >> >>>> > > > >>>> that
>> >> >> > >> >>>> > > > >>>>>>
>> >> >> > >> >>>> > > > >>>>>>> had
>> >> >> > >> >>>> > > > >>>>>>>> issue had 4 disks?
>> >> >> > >> >>>> > > > >>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>> BTW, the 10 minutes
>> zookeeper.session.timeout
>> >> >> has
>> >> >> > >> been
>> >> >> > >> >>>> set
>> >> >> > >> >>>> > > during
>> >> >> > >> >>>> > > > >>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>> the
>> >> >> > >> >>>> > > > >>>>>>
>> >> >> > >> >>>> > > > >>>>>>> experiment.
>> >> >> > >> >>>> > > > >>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>> And for sure this node had it in place?
>> >> >> > >> >>>> > > > >>>>>>>> St.Ack
>> >> >> > >> >>>> > > > >>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>> thanks
>> >> >> > >> >>>> > > > >>>>>>>>> zhenyu
>> >> >> > >> >>>> > > > >>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>> On Thu, Nov 12, 2009 at 2:08 PM, stack <
>> >> >> > >> stack@duboce.net
>> >> >> > >> >>>> >
>> >> >> > >> >>>> > > wrote:
>> >> >> > >> >>>> > > > >>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>> On Thu, Nov 12, 2009 at 8:40 AM, Zhenyu
>> >> Zhong <
>> >> >> > >> >>>> > > > >>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>> zhongresearch@gmail.com
>> >> >> > >> >>>> > > > >>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>> wrote:
>> >> >> > >> >>>> > > > >>>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>> Though I experienced 2 regionserver
>> >> >> disconnection
>> >> >> > >> this
>> >> >> > >> >>>> > > > >>>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>> morning,
>> >> >> > >> >>>> > > > >>>
>> >> >> > >> >>>> > > > >>>> it
>> >> >> > >> >>>> > > > >>>>>>
>> >> >> > >> >>>> > > > >>>>>>> looks
>> >> >> > >> >>>> > > > >>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>> better from the regionserver log. (please
>> >> see
>> >> >> the
>> >> >> > >> >>>> following
>> >> >> > >> >>>> > > > >>>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>> log)
>> >> >> > >> >>>> > > > >>>
>> >> >> > >> >>>> > > > >>>>
>> >> >> > >> >>>> > > > >>>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>>> http://pastebin.com/m496dbfae
>> >> >> > >> >>>> > > > >>>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>>> I checked diskIO for one of the
>> >> >> > >> >>>> > regionserver(192.168.100.116)
>> >> >> > >> >>>> > > > >>>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>> during
>> >> >> > >> >>>> > > > >>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>> the
>> >> >> > >> >>>> > > > >>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>> time it disconnected.
>> >> >> > >> >>>> > > > >>>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>>> Time: 03:04:58 AM
>> >> >> > >> >>>> > > > >>>>>>>>>>> Device: tps Blk_read/s
>> >> >> Blk_wrtn/s
>> >> >> > >> >>>> Blk_read
>> >> >> > >> >>>> > > > >>>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>> Blk_wrtn
>> >> >> > >> >>>> > > > >>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>> sda 105.31 5458.83
>> >> 19503.64
>> >> >> > >> >>>> 9043873239
>> >> >> > >> >>>> > > > >>>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>> 32312473676
>> >> >> > >> >>>> > > > >>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>> sda1 2.90 6.64
>> >> 99.25
>> >> >> > >> >>>> 10993934
>> >> >> > >> >>>> > > > >>>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>> 164433464
>> >> >> > >> >>>> > > > >>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>> sda2 1.72 23.76
>> >> 30.25
>> >> >> > >> >>>> 39365817
>> >> >> > >> >>>> > > > >>>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>> 50124008
>> >> >> > >> >>>> > > > >>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>> sda3 0.30 0.38
>> >> 3.58
>> >> >> > >> >>>> 624930
>> >> >> > >> >>>> > > > >>>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>> 5923000
>> >> >> > >> >>>> > > > >>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>> sda4 100.39 5428.06
>> >> 19370.56
>> >> >> > >> >>>> 8992888270
>> >> >> > >> >>>> > > > >>>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>> 32091993204
>> >> >> > >> >>>> > > > >>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>>> Is this high for you? 20k
>> blocks/second
>> >> >> would
>> >> >> > >> seem to
>> >> >> > >> >>>> be
>> >> >> > >> >>>> > > high
>> >> >> > >> >>>> > > > >>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>> but
>> >> >> > >> >>>> > > > >>>
>> >> >> > >> >>>> > > > >>>> its
>> >> >> > >> >>>> > > > >>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>> one
>> >> >> > >> >>>> > > > >>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>> disk only and its not being shared by zk
>> >> >> anymore
>> >> >> > so
>> >> >> > >> >>>> > shouldn't
>> >> >> > >> >>>> > > > >>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>> matter?
>> >> >> > >> >>>> > > > >>>>>>
>> >> >> > >> >>>> > > > >>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>> I also checked the zookeeper quorum
>> server
>> >> >> that
>> >> >> > the
>> >> >> > >> >>>> > > > >>>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>> regionserver
>> >> >> > >> >>>> > > > >>>
>> >> >> > >> >>>> > > > >>>> tried
>> >> >> > >> >>>> > > > >>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>> to
>> >> >> > >> >>>> > > > >>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>> connect according to the log. However, I
>> >> don't
>> >> >> see
>> >> >> > >> >>>> > > > >>>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>> 192.168.100.116
>> >> >> > >> >>>> > > > >>>>>>
>> >> >> > >> >>>> > > > >>>>>>> in
>> >> >> > >> >>>> > > > >>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>> the
>> >> >> > >> >>>> > > > >>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>> client list of the zookeeper quorum
>> server
>> >> that
>> >> >> > the
>> >> >> > >> >>>> > > > >>>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>> regionserver
>> >> >> > >> >>>> > > > >>>
>> >> >> > >> >>>> > > > >>>> tried
>> >> >> > >> >>>> > > > >>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>> to
>> >> >> > >> >>>> > > > >>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>> connect.
>> >> >> > >> >>>> > > > >>>>>>>>>>> Would that be a problem?
>> >> >> > >> >>>> > > > >>>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>>> Is that because the ephemeral node for
>> the
>> >> >> > >> >>>> regionserver
>> >> >> > >> >>>> > had
>> >> >> > >> >>>> > > > >>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>> evaporated?
>> >> >> > >> >>>> > > > >>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>> Lost
>> >> >> > >> >>>> > > > >>>>>>>>>> its lease w/ zk by the time you went to
>> >> look?
>> >> >> > >> >>>> > > > >>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>> Thu Nov 12 15:04:52 UTC 2009
>> >> >> > >> >>>> > > > >>>>>>>>>>> Zookeeper version: 3.2.1-808558, built
>> on
>> >> >> > >> 08/27/2009
>> >> >> > >> >>>> 18:48
>> >> >> > >> >>>> > > > >>>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>> GMT
>> >> >> > >> >>>> > > > >>>
>> >> >> > >> >>>> > > > >>>> Clients:
>> >> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.127:43045
>> >> >> > >> [1](queued=0,recved=26,sent=0)
>> >> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.131:39091
>> >> >> > >> [1](queued=0,recved=964,sent=0)
>> >> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.124:35961
>> >> >> > >> [1](queued=0,recved=3266,sent=0)
>> >> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.123:47935
>> >> >> > >> [1](queued=0,recved=233,sent=0)
>> >> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.125:46931
>> >> >> > >> [1](queued=0,recved=2,sent=0)
>> >> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.118:54924
>> >> >> > >> [1](queued=0,recved=295,sent=0)
>> >> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.118:41390
>> >> >> > >> [1](queued=0,recved=2290,sent=0)
>> >> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.136:42243
>> >> >> > >> [1](queued=0,recved=0,sent=0)
>> >> >> > >> >>>> > > > >>>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>>> Latency min/avg/max: 0/17/6333
>> >> >> > >> >>>> > > > >>>>>>>>>>> Received: 47111
>> >> >> > >> >>>> > > > >>>>>>>>>>> Sent: 0
>> >> >> > >> >>>> > > > >>>>>>>>>>> Outstanding: 0
>> >> >> > >> >>>> > > > >>>>>>>>>>> Zxid: 0x77000083f4
>> >> >> > >> >>>> > > > >>>>>>>>>>> Mode: leader
>> >> >> > >> >>>> > > > >>>>>>>>>>> Node count: 23
>> >> >> > >> >>>> > > > >>>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>> That 6 second maximum latency is pretty
>> bad
>> >> but
>> >> >> > >> should
>> >> >> > >> >>>> be
>> >> >> > >> >>>> > well
>> >> >> > >> >>>> > > > >>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>> within
>> >> >> > >> >>>> > > > >>>>>>
>> >> >> > >> >>>> > > > >>>>>>> the
>> >> >> > >> >>>> > > > >>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>> zk
>> >> >> > >> >>>> > > > >>>>>>>>>> session timeout.
>> >> >> > >> >>>> > > > >>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>> So, problem is likely on the zk client
>> side
>> >> of
>> >> >> the
>> >> >> > >> >>>> session;
>> >> >> > >> >>>> > > > >>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>> i.e.
>> >> >> > >> >>>> > > > >>>
>> >> >> > >> >>>> > > > >>>> in
>> >> >> > >> >>>> > > > >>>>>>
>> >> >> > >> >>>> > > > >>>>>>> the
>> >> >> > >> >>>> > > > >>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>> RS.
>> >> >> > >> >>>> > > > >>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>> You could enable GC logging as J-D
>> suggested
>> >> to
>> >> >> > see
>> >> >> > >> if
>> >> >> > >> >>>> you
>> >> >> > >> >>>> > > have
>> >> >> > >> >>>> > > > >>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>> any
>> >> >> > >> >>>> > > > >>>>>>
>> >> >> > >> >>>> > > > >>>>>>> big
>> >> >> > >> >>>> > > > >>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>> pauses, pauses > zk session timeout.
>> >> >> > >> >>>> > > > >>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>> When the RS went down, it didn't look too
>> >> >> heavily
>> >> >> > >> >>>> loaded:
>> >> >> > >> >>>> > > > >>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>> 1. 2009-11-12 15:04:52,830 INFO
>> >> >> > >> >>>> > > > >>>>>>>>>>
>> >> >> > org.apache.hadoop.hbase.regionserver.HRegionServer:
>> >> >> > >> >>>> Dump of
>> >> >> > >> >>>> > > > >>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>> metrics:
>> >> >> > >> >>>> > > > >>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>> request=1.5166667, regions=322,
>> stores=657,
>> >> >> > >> >>>> storefiles=631,
>> >> >> > >> >>>> > > > >>>>>>>>>> storefileIndexSize=61,
>> memstoreSize=1472,
>> >> >> > >> >>>> usedHeap=2819,
>> >> >> > >> >>>> > > > >>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>> maxHeap=4079,
>> >> >> > >> >>>> > > > >>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>> blockCacheSize=658110960,
>> >> >> > blockCacheFree=197395984,
>> >> >> > >> >>>> > > > >>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>> blockCacheCount=9903,
>> >> >> > >> >>>> > > > >>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>> blockCacheHitRatio=99
>> >> >> > >> >>>> > > > >>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>> Its serving a few reads? The number of
>> >> store
>> >> >> > files
>> >> >> > >> >>>> seems
>> >> >> > >> >>>> > > fine.
>> >> >> > >> >>>> > > > >>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>> Not
>> >> >> > >> >>>> > > > >>>>>>
>> >> >> > >> >>>> > > > >>>>>>> too
>> >> >> > >> >>>> > > > >>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>> much memory used.
>> >> >> > >> >>>> > > > >>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>> Looking at the logs, I see the Lease
>> Still
>> >> Held
>> >> >> > >> >>>> exception.
>> >> >> > >> >>>> > > > >>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>> This
>> >> >> > >> >>>> > > > >>>
>> >> >> > >> >>>> > > > >>>> happens
>> >> >> > >> >>>> > > > >>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>> when the RS does its regular report to the
>> >> >> master
>> >> >> > but
>> >> >> > >> the
>> >> >> > >> >>>> > > > >>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>> master
>> >> >> > >> >>>> > > > >>>
>> >> >> > >> >>>> > > > >>>> thinks
>> >> >> > >> >>>> > > > >>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>> the
>> >> >> > >> >>>> > > > >>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>> RS has since restarted. It'll think this
>> >> >> probably
>> >> >> > >> >>>> because
>> >> >> > >> >>>> > it
>> >> >> > >> >>>> > > > >>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>> noticed
>> >> >> > >> >>>> > > > >>>>>>
>> >> >> > >> >>>> > > > >>>>>>> that
>> >> >> > >> >>>> > > > >>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>> the RS's znode in zk had gone away and it
>> >> >> > considered
>> >> >> > >> the
>> >> >> > >> >>>> RS
>> >> >> > >> >>>> > > > >>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>> dead.
>> >> >> > >> >>>> > > > >>>
>> >> >> > >> >>>> > > > >>>>
>> >> >> > >> >>>> > > > >>>>>>>>>> Looking too at your logs I see this gap
>> in
>> >> the
>> >> >> zk
>> >> >> > >> >>>> pinging:
>> >> >> > >> >>>> > > > >>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>> 1. 2009-11-12 15:03:39,325 DEBUG
>> >> >> > >> >>>> > > > >>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>> org.apache.zookeeper.ClientCnxn:
>> >> >> > >> >>>> > > > >>>>>>
>> >> >> > >> >>>> > > > >>>>>>> Got
>> >> >> > >> >>>> > > > >>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>> ping response for
>> sessionid:0x224e55436ad0004
>> >> >> after
>> >> >> > >> 0ms
>> >> >> > >> >>>> > > > >>>>>>>>>> 2. 2009-11-12 15:03:43,113 DEBUG
>> >> >> > >> >>>> > > > >>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>> org.apache.zookeeper.ClientCnxn:
>> >> >> > >> >>>> > > > >>>>>>
>> >> >> > >> >>>> > > > >>>>>>> Got
>> >> >> > >> >>>> > > > >>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>> ping response for
>> sessionid:0x24e55436a0007d
>> >> >> after
>> >> >> > >> 0ms
>> >> >> > >> >>>> > > > >>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>> Where in the lines above it, its
>> reporting
>> >> >> about
>> >> >> > >> every
>> >> >> > >> >>>> ten
>> >> >> > >> >>>> > > > >>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>> seconds,
>> >> >> > >> >>>> > > > >>>>>>
>> >> >> > >> >>>> > > > >>>>>>> here
>> >> >> > >> >>>> > > > >>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>> there is a big gap.
>> >> >> > >> >>>> > > > >>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>> Do you have ganglia or something that
>> will
>> >> let
>> >> >> you
>> >> >> > >> look
>> >> >> > >> >>>> more
>> >> >> > >> >>>> > > > >>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>> into
>> >> >> > >> >>>> > > > >>>
>> >> >> > >> >>>> > > > >>>> what
>> >> >> > >> >>>> > > > >>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>> was
>> >> >> > >> >>>> > > > >>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>> happening on this machine around this
>> time?
>> >> Is
>> >> >> > the
>> >> >> > >> >>>> machine
>> >> >> > >> >>>> > > OK?
>> >> >> > >> >>>> > > > >>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>> It
>> >> >> > >> >>>> > > > >>>>>>
>> >> >> > >> >>>> > > > >>>>>>> looks
>> >> >> > >> >>>> > > > >>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>> lightly loaded and you have your cluster
>> >> nicely
>> >> >> > laid
>> >> >> > >> out.
>> >> >> > >> >>>> > > > >>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>> Something
>> >> >> > >> >>>> > > > >>>>>>
>> >> >> > >> >>>> > > > >>>>>>> odd
>> >> >> > >> >>>> > > > >>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>> is
>> >> >> > >> >>>> > > > >>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>> going on. What about things like the
>> write
>> >> >> speed
>> >> >> > to
>> >> >> > >> >>>> disk?
>> >> >> > >> >>>> > In
>> >> >> > >> >>>> > > > >>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>> the
>> >> >> > >> >>>> > > > >>>
>> >> >> > >> >>>> > > > >>>> past
>> >> >> > >> >>>> > > > >>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>> strange issues have been explained by
>> >> incorrectly
>> >> >> > set
>> >> >> > >> BIOS
>> >> >> > >> >>>> > > > >>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>> which
>> >> >> > >> >>>> > > > >>>
>> >> >> > >> >>>> > > > >>>> made
>> >> >> > >> >>>> > > > >>>>>>
>> >> >> > >> >>>> > > > >>>>>>> disks
>> >> >> > >> >>>> > > > >>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>> run at 1/100th of their proper speed.
>> >> >> > >> >>>> > > > >>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>> St.Ack
>> >> >> > >> >>>> > > > >>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>>> Best,
>> >> >> > >> >>>> > > > >>>>>>>>>>> zhenyu
>> >> >> > >> >>>> > > > >>>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>>> On Wed, Nov 11, 2009 at 3:58 PM, Zhenyu
>> >> Zhong
>> >> >> <
>> >> >> > >> >>>> > > > >>>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>> zhongresearch@gmail.com
>> >> >> > >> >>>> > > > >>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>> wrote:
>> >> >> > >> >>>> > > > >>>>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>>> Stack
>> >> >> > >> >>>> > > > >>>>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>>>> I am very appreciated for your
>> comments.
>> >> >> > >> >>>> > > > >>>>>>>>>>>> I will use the zookeeper monitoring
>> script
>> >> on
>> >> >> my
>> >> >> > >> >>>> cluster
>> >> >> > >> >>>> > > > >>>>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>>> and
>> >> >> > >> >>>> > > > >>>
>> >> >> > >> >>>> > > > >>>> let
>> >> >> > >> >>>> > > > >>>>>>
>> >> >> > >> >>>> > > > >>>>>>> it
>> >> >> > >> >>>> > > > >>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>> run
>> >> >> > >> >>>> > > > >>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>> overnight to see the result.
>> >> >> > >> >>>> > > > >>>>>>>>>>>> I will follow up the thread when I get
>> >> >> anything.
>> >> >> > >> >>>> > > > >>>>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>>>> thanks
>> >> >> > >> >>>> > > > >>>>>>>>>>>> zhenyu
>> >> >> > >> >>>> > > > >>>>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>>>> On Wed, Nov 11, 2009 at 3:52 PM, stack
>> <
>> >> >> > >> >>>> stack@duboce.net>
>> >> >> > >> >>>> > > > >>>>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>>> wrote:
>> >> >> > >> >>>> > > > >>>>>>
>> >> >> > >> >>>> > > > >>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>>>> I see these in your log too:
>> >> >> > >> >>>> > > > >>>>>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>>>>> 1. 2009-11-11 04:27:19,018 DEBUG
>> >> >> > >> >>>> > > > >>>>>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
>> >> >> > >> >>>> > > > >>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>> Got
>> >> >> > >> >>>> > > > >>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>>> ping response for
>> >> sessionid:0x424dfd908c50009
>> >> >> > >> after
>> >> >> > >> >>>> > > > >>>>>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>>>> 4544ms
>> >> >> > >> >>>> > > > >>>
>> >> >> > >> >>>> > > > >>>> 2. 2009-11-11 04:27:19,018 DEBUG
>> >> >> > >> >>>> > > > >>>>>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
>> >> >> > >> >>>> > > > >>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>> Got
>> >> >> > >> >>>> > > > >>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>>> ping response for
>> >> sessionid:0x24dfd90c810002
>> >> >> > after
>> >> >> > >> >>>> > > > >>>>>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>>>> 4548ms
>> >> >> > >> >>>> > > > >>>
>> >> >> > >> >>>> > > > >>>> 3. 2009-11-11 04:27:43,960 DEBUG
>> >> >> > >> >>>> > > > >>>>>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
>> >> >> > >> >>>> > > > >>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>> Got
>> >> >> > >> >>>> > > > >>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>>> ping response for
>> >> sessionid:0x424dfd908c50009
>> >> >> > >> after
>> >> >> > >> >>>> > > > >>>>>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>>>> 9030ms
>> >> >> > >> >>>> > > > >>>
>> >> >> > >> >>>> > > > >>>> 4. 2009-11-11 04:27:43,960 DEBUG
>> >> >> > >> >>>> > > > >>>>>>>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>>>>>> org.apache.zookeeper.
>> >> >> > >> >>>> > > > >>>>>>>>
>> >> >> > >> >>>> > > > >>>>>>>
>> >> >> > >> >>>> > > >
>> >> >> > >> >>>> > >
>> >> >> > >> >>>> >
>> >> >> > >> >>>>
>> >> >> > >> >>>
>> >> >> > >> >>>
>> >> >> > >> >>
>> >> >> > >> >
>> >> >> > >>
>> >> >> > >
>> >> >> > >
>> >> >> >
>> >> >>
>> >> >
>> >> >
>> >>
>> >
>> >
>>
>
>
Re: regionserver disconnection
Posted by Zhenyu Zhong <zh...@gmail.com>.
Stack,
I am very appreciated. Let me dig into the zookeeper log more.
FYI, I saw the sleeper complained about the delay for the first time. It
looks like there are different reasons for the RS disconnections.
2009-11-19 08:08:32,708 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
response for sessionid:0x42508716bfc0001 after 0ms
2009-11-19 08:08:37,595 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
response for sessionid:0x42508716bfc0082 after 0ms
2009-11-19 08:08:46,073 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
response for sessionid:0x42508716bfc0001 after 0ms
2009-11-19 08:09:39,880 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x42508716bfc0001 to sun.nio.ch.SelectionKeyImpl@4f3a608f
java.io.IOException: TIMED OUT
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:906)
2009-11-19 08:09:39,880 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x42508716bfc0082 to sun.nio.ch.SelectionKeyImpl@14c8875
java.io.IOException: TIMED OUT
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:906)
2009-11-19 08:09:39,880 WARN org.apache.hadoop.hbase.util.Sleeper: We slept
53556ms, ten times longer than scheduled: 3000
2009-11-19 08:09:39,980 WARN
org.apache.hadoop.hbase.regionserver.HRegionServer: Attempt=1
org.apache.hadoop.hbase.Leases$LeaseStillHeldException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:571)
at java.lang.Thread.run(Thread.java:619)
2009-11-19 08:09:39,980 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
state: Disconnected, type: None, path: null
2009-11-19 08:09:40,530 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server superpyxis0005/192.168.100.119:2181
2009-11-19 08:09:40,530 INFO org.apache.zookeeper.ClientCnxn: Priming
connection to java.nio.channels.SocketChannel[connected local=/
192.168.100.122:40659 remote=superpyxis0005/192.168.100.119:2181]
Best,
zhenyu
On Wed, Nov 18, 2009 at 6:52 PM, stack <st...@duboce.net> wrote:
> Patrick Hunt, one of the zk lads, answered with the following:
>
>
>
>
> According to the following snippet (I cut to just session
> 0x425035c48210002)
> the client is getting pings for this session id regularly every 6 seconds
> or
> so, then after 41,016 there's a 20 second gap (corresponds to a 30 second
> timeout being set during session creation) at which point we disconnect
> from
> the server.
>
>
> > 2009-11-18 12:41:34,788 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
> > response for sessionid:0x425035c48210002 after 0ms
> > 2009-11-18 12:42:01,482 WARN org.apache.zookeeper.ClientCnxn: Exception
> > closing session 0x425035c48210002 to sun.nio.ch.SelectionKeyImpl@421690ab
> > java.io.IOException: TIMED OUT
>
> the client notifies hbase of the disco, and attempts to reconnect to a
> server
>
>
> > org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
> > state: Disconnected, type: None, path: null
> > 2009-11-18 12:42:01,782 INFO org.apache.zookeeper.ClientCnxn: Attempting
> > connection to server superpyxis0003/192.168.100.117:2181
>
> but the server won't allow the session be be established for some reason
> (is
> that the right server:port that's it's trying to connect to?)
>
>
> > 2009-11-18 12:42:02,182 INFO org.apache.zookeeper.ClientCnxn: Server
> > connection successful
> > 2009-11-18 12:42:02,182 WARN org.apache.zookeeper.ClientCnxn: Exception
> > closing session 0x425035c48210002 to sun.nio.ch.SelectionKeyImpl@5c07076b
> > java.io.IOException: Read error rc = -1 java.nio.DirectByteBuffer[pos=0
> > lim=4 cap=4]
>
> The "read error rc = -1" means that the client read from the socket and got
> EOS. But it's not a session expiration afaict (and should not be at this
> point in time).
>
>
> You need to look at the ZK server logs for this period in time (for this
> server superpyxis0003:2181, and superpyxis0001:2181, the client tries to
> connect to both of these). There should be some indication there why the
> server is closing down the connection and not allowing the session to be
> re-established.
>
> One thought - by default we limit clients to a max of 10 connections from a
> particular host to the server. If you exceed this limit the zk server (it's
> per server though, not per cluster, so perhaps this is not it) will not
> allow any additional connections from the client host (you can't exceed 10
> on a server). Could this be it? i.e. the user is running a number of ZK
> clients (say hbase processes) from the same host? Regardless, get the
> server
> log for this time period and it should be more clear.
>
> Patrick
>
>
>
>
>
>
>
> On Wed, Nov 18, 2009 at 11:18 AM, stack <st...@duboce.net> wrote:
>
> > And running stat on each of your zookeeper nodes tells you what about
> > averages for connect-times?
> >
> > Its odd that there is a ping just 20 seconds beforehand and you have set
> > the zk session timeout at 60 seconds (or ten minutes?). Let me ask the
> zk
> > lads about it.
> >
> > If you look in the GC log for around this time, do you see anything?
> >
> > St.Ack
> >
> >
> >
> > On Wed, Nov 18, 2009 at 7:32 AM, Zhenyu Zhong <zhongresearch@gmail.com
> >wrote:
> >
> >> After running more experiment, it seems that the disconnection is not
> >> related to the heavy load job, because the load average is low, the
> diskio
> >> is normal, memory Heap was not reached, also the virtual memory stats
> >> shows
> >> that no swappings. However, the disconnection still happens.
> >>
> >> It looks like this time it pauses for 20 seconds. No idea why
> regionserver
> >> disconnected.
> >>
> >> Any other suggestions please?
> >>
> >>
> >> 2009-11-18 12:41:00,956 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
> >> response for sessionid:0x125035c495d02bb after 0ms
> >> 2009-11-18 12:41:08,074 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
> >> response for sessionid:0x425035c48210002 after 0ms
> >> 2009-11-18 12:41:14,302 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
> >> response for sessionid:0x125035c495d02bb after 0ms
> >> 2009-11-18 12:41:21,433 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
> >> response for sessionid:0x425035c48210002 after 0ms
> >> 2009-11-18 12:41:27,656 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
> >> response for sessionid:0x125035c495d02bb after 0ms
> >> 2009-11-18 12:41:34,788 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
> >> response for sessionid:0x425035c48210002 after 0ms
> >> 2009-11-18 12:41:41,016 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
> >> response for sessionid:0x125035c495d02bb after 1ms
> >> 2009-11-18 12:42:01,482 WARN org.apache.zookeeper.ClientCnxn: Exception
> >> closing session 0x425035c48210002 to
> sun.nio.ch.SelectionKeyImpl@421690ab
> >> java.io.IOException: TIMED OUT
> >> at
> >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:906)
> >> 2009-11-18 12:42:01,582 INFO
> >> org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
> >> state: Disconnected, type: None, path: null
> >> 2009-11-18 12:42:01,782 INFO org.apache.zookeeper.ClientCnxn: Attempting
> >> connection to server superpyxis0003/192.168.100.117:2181
> >> 2009-11-18 12:42:01,782 INFO org.apache.zookeeper.ClientCnxn: Priming
> >> connection to java.nio.channels.SocketChannel[connected local=/
> >> 192.168.100.132:46610 remote=superpyxis0003/192.168.100.117:2181]
> >> 2009-11-18 12:42:01,782 INFO org.apache.zookeeper.ClientCnxn: Server
> >> connection successful
> >> 2009-11-18 12:42:01,782 WARN org.apache.zookeeper.ClientCnxn: Exception
> >> closing session 0x425035c48210002 to
> sun.nio.ch.SelectionKeyImpl@56dc6fac
> >> java.io.IOException: Read error rc = -1 java.nio.DirectByteBuffer[pos=0
> >> lim=4 cap=4]
> >> at
> >> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
> >> at
> >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
> >> 2009-11-18 12:42:01,782 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> >> exception during shutdown input
> >> java.net.SocketException: Transport endpoint is not connected
> >> at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
> >> at
> >> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
> >> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> >> at
> >> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> >> at
> >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> >> 2009-11-18 12:42:01,782 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> >> exception during shutdown output
> >> java.net.SocketException: Transport endpoint is not connected
> >> at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
> >> at
> >> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
> >> at
> sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> >> at
> >> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> >> at
> >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> >> 2009-11-18 12:42:02,182 INFO org.apache.zookeeper.ClientCnxn: Attempting
> >> connection to server superpyxis0001/192.168.100.115:2181
> >> 2009-11-18 12:42:02,182 INFO org.apache.zookeeper.ClientCnxn: Priming
> >> connection to java.nio.channels.SocketChannel[connected local=/
> >> 192.168.100.132:36197 remote=superpyxis0001/192.168.100.115:2181]
> >> 2009-11-18 12:42:02,182 INFO org.apache.zookeeper.ClientCnxn: Server
> >> connection successful
> >> 2009-11-18 12:42:02,182 WARN org.apache.zookeeper.ClientCnxn: Exception
> >> closing session 0x425035c48210002 to
> sun.nio.ch.SelectionKeyImpl@5c07076b
> >> java.io.IOException: Read error rc = -1 java.nio.DirectByteBuffer[pos=0
> >> lim=4 cap=4]
> >> at
> >> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
> >> at
> >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
> >> 2009-11-18 12:42:02,182 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> >> exception during shutdown input
> >> java.net.SocketException: Transport endpoint is not connected
> >> at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
> >> at
> >> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
> >> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> >> at
> >> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> >> at
> >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> >> 2009-11-18 12:42:02,182 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> >> exception during shutdown output
> >> java.net.SocketException: Transport endpoint is not connected
> >>
> >>
> >> On Mon, Nov 16, 2009 at 4:16 PM, Zhenyu Zhong <zhongresearch@gmail.com
> >> >wrote:
> >>
> >> > Here is the diskIO and CPU around the time we had RS disconnection on
> >> one
> >> > machine that runs RegionServer. It doesn't seem to be high. Similar
> disk
> >> and
> >> > cpu usage have been seen before and the HBase was running fine.
> >> >
> >> >
> >> > So far I haven't found why my 10 minutes session timeout doesn't
> apply.
> >> > Still digging.
> >> >
> >> > Many thanks!
> >> >
> >> > Time: 03:01:08 PM
> >> > avg-cpu: %user %nice %system %iowait %steal %idle
> >> > 3.82 0.00 0.89 3.97 0.00 91.32
> >> >
> >> >
> >> > Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
> >> > sda 24.30 1605.39 6679.35 6238623388
> 25956251720
> >> > sda1 0.51 1.94 15.71 7547265 61067904
> >> > sda2 0.13 2.11 5.20 8202079 20189232
> >> > sda3 0.13 0.25 1.63 987846 6323336
> >> > sda4 0.00 0.00 0.00 4 0
> >> > sda5 1.78 0.06 24.14 220010 93817208
> >> > sda6 0.38 0.42 8.41 1630584 32688152
> >> > sda7 21.38 1600.61 6624.26 6220035272
> 25742165888
> >> > sdb 4.52 767.33 565.91 2981868690 2199132380
> >> > sdb1 4.52 767.33 565.91 2981866755 2199132372
> >> > sdc 4.42 742.95 563.00 2887151482 2187823092
> >> > sdc1 4.42 742.95 563.00 2887149547 2187823084
> >> > sdd 4.49 750.78 557.25 2917554074 2165513500
> >> > sdd1 4.49 750.78 557.25 2917552139 2165513492
> >> > sde 4.52 758.51 569.46 2947593394 2212964236
> >> > sde1 4.52 758.51 569.46 2947591459 2212964228
> >> > sdf 4.51 747.22 571.78 2903740266 2221972652
> >> > sdf1 4.51 747.22 571.78 2903738331 2221972644
> >> >
> >> > Best,
> >> > zhenyu
> >> >
> >> >
> >> >
> >> > On Mon, Nov 16, 2009 at 4:07 PM, stack <st...@duboce.net> wrote:
> >> >
> >> >> On Mon, Nov 16, 2009 at 12:05 PM, Zhenyu Zhong <
> >> zhongresearch@gmail.com
> >> >> >wrote:
> >> >>
> >> >> > I just realized that there was a MapReduce job running during the
> >> time
> >> >> the
> >> >> > regionserver disconnected from the zookeeper.
> >> >> > That MapReduce Job was processing 500GB data and took about 8
> minutes
> >> to
> >> >> > finish. It launched over 2000 map tasks.
> >> >>
> >> >>
> >> >> There was a tasktracker running on the RegionServer that
> disconnected?
> >> Is
> >> >> the map i/o or cpu heavy? Do you think it could have stole life from
> >> the
> >> >> datanode/regionserver?
> >> >>
> >> >>
> >> >>
> >> >> > I doubt that this introduced
> >> >> > resource contention between DataNode and RegionServer Node.
> >> >> > Also during the time that MapReduce job ran, I saw a few errors
> >> >> indicating
> >> >> > that
> >> >> >
> >> >> > java.io.IOException: Could not obtain block:
> >> >> > blk_7661247556283580226_4450462 file=/data/xxx.txt.200910141159
> >> >> > at
> >> >> >
> >> >>
> >>
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1787)
> >> >> > at
> >> >> >
> >> >>
> >>
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1615)
> >> >> > at
> >> >> >
> >> >>
> >>
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1742)
> >> >> > at java.io.DataInputStream.read(DataInputStream.java:83)
> >> >> > at
> >> >> org.apache.hadoop.util.LineReader.readLine(LineReader.java:134)
> >> >> > at
> >> >> >
> >> >>
> >>
> org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:136)
> >> >> > at
> >> >> >
> >> org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:40)
> >> >> > at
> >> >> >
> >> >>
> >>
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:192)
> >> >> > at
> >> >> >
> >> >>
> >>
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:176)
> >> >> > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
> >> >> > at
> >> >> org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
> >> >> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
> >> >> > at org.apache.hadoop.mapred.Child.main(Child.java:170)
> >> >> >
> >> >>
> >> >> This is interesting. Your map task is running into hdfs-127. Yeah,
> >> you
> >> >> should patch your cluster if you want to get rid of these (hdfs-127
> has
> >> >> been
> >> >> applied to 0.21 hadoop and will be in the next hadoop release on
> 0.20.x
> >> >> branch, hadoop 0.20.2).
> >> >>
> >> >>
> >> >>
> >> >> >
> >> >> > Possibly it is related to HDFS-127, but I don't see any datanodes
> >> went
> >> >> > down. Does that affect Regionserver? Shall we apply the patch?
> >> >> >
> >> >> >
> >> >> Read up on hdfs-127 for explaination of whats going on.
> >> >>
> >> >>
> >> >>
> >> >> > Now I start to keep tracking of the virtual memory stats to see if
> >> the
> >> >> same
> >> >> > issue happens tomorrow around the same time period.
> >> >> >
> >> >>
> >> >> Excellent. Thanks for doing this.
> >> >>
> >> >>
> >> >>
> >> >> > And I highly suspect that this particular MapReduce job hurts HBase
> >> >> > Regionserver.
> >> >> >
> >> >> > PS. I also use gcviewer to parse the GC-log, I only see around 30
> >> >> seconds
> >> >> > pauses maximum.
> >> >> >
> >> >>
> >> >>
> >> >> Did you figure why your ten minute session is not applying?
> >> >>
> >> >> St.Ack
> >> >>
> >> >>
> >> >>
> >> >> >
> >> >> > Thanks a lot.
> >> >> > zhenyu
> >> >> >
> >> >> >
> >> >> >
> >> >> > On Sun, Nov 15, 2009 at 12:52 AM, Zhenyu Zhong <
> >> zhongresearch@gmail.com
> >> >> > >wrote:
> >> >> >
> >> >> > > J-D,
> >> >> > >
> >> >> > > Thank you very much for your comments.
> >> >> > > My company block the IRC port, so I have trouble to connect to
> IRC
> >> >> > channel.
> >> >> > > I have been trying to ask the IT to open the IRC port for me, it
> >> might
> >> >> > take
> >> >> > > a while.
> >> >> > >
> >> >> > > Best,
> >> >> > > zhenyu
> >> >> > >
> >> >> > >
> >> >> > > On Sat, Nov 14, 2009 at 2:21 PM, Jean-Daniel Cryans <
> >> >> jdcryans@apache.org
> >> >> > >wrote:
> >> >> > >
> >> >> > >> The error you are getting is a disconnection from a zookeeper
> >> server
> >> >> > >> and is very generic.
> >> >> > >>
> >> >> > >> ZK-86 is still opened and the last comment refers to ZK-111
> saying
> >> >> > >> that the bug (in unit tests) was probably fixed in release 3.0.0
> >> last
> >> >> > >> year.
> >> >> > >>
> >> >> > >> To figure the hang you have, you can try to jstack the process
> pid
> >> >> and
> >> >> > >> see exactly what's holding the RS from shutting down.
> >> >> > >>
> >> >> > >> Would it be possible for you to drop by the IRC channel? This
> way
> >> we
> >> >> > >> can debug this a at much faster pace.
> >> >> > >>
> >> >> > >> Thx!
> >> >> > >>
> >> >> > >> J-D
> >> >> > >>
> >> >> > >> On Sat, Nov 14, 2009 at 9:57 AM, Zhenyu Zhong <
> >> >> zhongresearch@gmail.com>
> >> >> > >> wrote:
> >> >> > >> > I found this.
> >> >> > >> > http://issues.apache.org/jira/browse/ZOOKEEPER-86
> >> >> > >> >
> >> >> > >> > It looks like the same error I had. Is it a zookeeper bug?
> When
> >> >> will
> >> >> > >> HBase
> >> >> > >> > take the zookeeper version 3.3.0?
> >> >> > >> >
> >> >> > >> > thanks
> >> >> > >> > zhenyu
> >> >> > >> >
> >> >> > >> > On Sat, Nov 14, 2009 at 11:31 AM, Zhenyu Zhong <
> >> >> > zhongresearch@gmail.com
> >> >> > >> >wrote:
> >> >> > >> >
> >> >> > >> >> Now I really doubt about the zookeeper, from the log I saw
> >> errors
> >> >> > like
> >> >> > >> >> IOException Read Error, while the zookeeper client
> >> (regionserver )
> >> >> > >> tried to
> >> >> > >> >> read. But it got disconneted status from zookeeper.
> >> >> > >> >>
> >> >> > >> >> I don't see any load on any zookeeper quorum servers. DiskIO
> is
> >> >> fine.
> >> >> > >> >>
> >> >> > >> >> Also when the regionserver decides to exit due to the
> >> disconnect
> >> >> > status
> >> >> > >> >> from zookeeper, sometimes the regionserver hangs during the
> >> >> exiting.
> >> >> > We
> >> >> > >> can
> >> >> > >> >> still see the HRegionServer process even we don't see it in
> the
> >> >> > Master
> >> >> > >> web
> >> >> > >> >> interface.
> >> >> > >> >>
> >> >> > >> >> I also notice that there is an zookeeper.retries and
> >> >> zookeeper.pauses
> >> >> > >> >> settings in hbase-default.xml, would that matter ?
> >> >> > >> >>
> >> >> > >> >> thanks
> >> >> > >> >>
> >> >> > >> >> You may see the IOException error in the following log when
> the
> >> >> > >> >> Regionserver lost connection to the zookeeper.
> >> >> > >> >>
> >> >> > >> >> 2009-11-14 15:58:31,993 INFO
> >> >> > org.apache.hadoop.hbase.regionserver.HLog:
> >> >> > >> >> Roll
> >> >> > >> >>
> >> >> > >>
> >> >> >
> >> >>
> >>
> /hbase/.logs/superpyxis0008.scur.colo,60021,1258179101617/hlog.dat.1258179106921,
> >> >> > >> >> entries=1062964, calcsize=255014936, filesize=139566093. New
> >> hlog
> >> >> > >> >>
> >> >> > >>
> >> >> >
> >> >>
> >>
> /hbase/.logs/superpyxis0008.scur.colo,60021,1258179101617/hlog.dat.1258214311993
> >> >> > >> >> 2009-11-14 15:58:39,295 DEBUG
> org.apache.zookeeper.ClientCnxn:
> >> Got
> >> >> > ping
> >> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
> >> >> > >> >> 2009-11-14 15:58:52,648 DEBUG
> org.apache.zookeeper.ClientCnxn:
> >> Got
> >> >> > ping
> >> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
> >> >> > >> >> 2009-11-14 15:59:06,007 DEBUG
> org.apache.zookeeper.ClientCnxn:
> >> Got
> >> >> > ping
> >> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
> >> >> > >> >> 2009-11-14 15:59:19,365 DEBUG
> org.apache.zookeeper.ClientCnxn:
> >> Got
> >> >> > ping
> >> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
> >> >> > >> >> 2009-11-14 15:59:32,867 DEBUG
> org.apache.zookeeper.ClientCnxn:
> >> Got
> >> >> > ping
> >> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 186ms
> >> >> > >> >> 2009-11-14 15:59:46,070 DEBUG
> org.apache.zookeeper.ClientCnxn:
> >> Got
> >> >> > ping
> >> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
> >> >> > >> >> 2009-11-14 15:59:59,423 DEBUG
> org.apache.zookeeper.ClientCnxn:
> >> Got
> >> >> > ping
> >> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
> >> >> > >> >> 2009-11-14 16:00:12,775 DEBUG
> org.apache.zookeeper.ClientCnxn:
> >> Got
> >> >> > ping
> >> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
> >> >> > >> >> 2009-11-14 16:00:28,141 DEBUG
> org.apache.zookeeper.ClientCnxn:
> >> Got
> >> >> > ping
> >> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 2010ms
> >> >> > >> >> 2009-11-14 16:01:10,378 WARN org.apache.zookeeper.ClientCnxn:
> >> >> > Exception
> >> >> > >> >> closing session 0x424f0ed01ea00a6 to
> >> >> > >> sun.nio.ch.SelectionKeyImpl@39fbb2d6
> >> >> > >> >>
> >> >> > >> >> java.io.IOException: Read error rc = -1
> >> >> > java.nio.DirectByteBuffer[pos=0
> >> >> > >> >> lim=4 cap=4]
> >> >> > >> >> at
> >> >> > >> >>
> >> >> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
> >> >> > >> >> at
> >> >> > >> >>
> >> >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
> >> >> > >> >> 2009-11-14 16:01:10,478 INFO
> >> >> > >> >> org.apache.hadoop.hbase.regionserver.HRegionServer: Got
> >> ZooKeeper
> >> >> > >> event,
> >> >> > >> >> state: Disconnected, type: None, path: null
> >> >> > >> >> 2009-11-14 16:01:11,333 WARN
> >> >> > >> >> org.apache.hadoop.hbase.regionserver.HRegionServer: Attempt=1
> >> >> > >> >>
> >> >> > >> >> org.apache.hadoop.hbase.Leases$LeaseStillHeldException
> >> >> > >> >> at
> >> >> > >> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> >> >> > >> >> Method)
> >> >> > >> >> at
> >> >> > >> >>
> >> >> > >>
> >> >> >
> >> >>
> >>
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> >> >> > >> >> at
> >> >> > >> >>
> >> >> > >>
> >> >> >
> >> >>
> >>
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> >> >> > >> >> at
> >> >> > >> java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> >> >> > >> >> at
> >> >> > >> >>
> >> >> > >>
> >> >> >
> >> >>
> >>
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
> >> >> > >> >> at
> >> >> > >> >>
> >> >> > >>
> >> >> >
> >> >>
> >>
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
> >> >> > >> >> at
> >> >> > >> >>
> >> >> > >>
> >> >> >
> >> >>
> >>
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
> >> >> > >> >> at
> >> >> > >> >>
> >> >> > >>
> >> >> >
> >> >>
> >>
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:571)
> >> >> > >> >> at java.lang.Thread.run(Thread.java:619)
> >> >> > >> >> 2009-11-14 16:01:11,433 INFO org.apache.zookeeper.ClientCnxn:
> >> >> > >> Attempting
> >> >> > >> >> connection to server superpyxis0001/192.168.100.115:2181
> >> >> > >> >> 2009-11-14 16:01:11,433 INFO org.apache.zookeeper.ClientCnxn:
> >> >> Priming
> >> >> > >> >> connection to java.nio.channels.SocketChannel[connected
> local=/
> >> >> > >> >> 192.168.100.122:59575 remote=superpyxis0001/
> >> 192.168.100.115:2181]
> >> >> > >> >> 2009-11-14 16:01:11,433 INFO org.apache.zookeeper.ClientCnxn:
> >> >> Server
> >> >> > >> >> connection successful
> >> >> > >> >> 2009-11-14 16:01:11,433 INFO
> >> >> > >> >> org.apache.hadoop.hbase.regionserver.HRegionServer: Got
> >> ZooKeeper
> >> >> > >> event,
> >> >> > >> >> state: Expired, type: None, path: null
> >> >> > >> >> 2009-11-14 16:01:11,433 WARN org.apache.zookeeper.ClientCnxn:
> >> >> > Exception
> >> >> > >> >> closing session 0x424f0ed01ea00a6 to
> >> >> > >> sun.nio.ch.SelectionKeyImpl@6b93d343
> >> >> > >> >> java.io.IOException: Session Expired
> >> >> > >> >> at
> >> >> > >> >>
> >> >> > >>
> >> >> >
> >> >>
> >>
> org.apache.zookeeper.ClientCnxn$SendThread.readConnectResult(ClientCnxn.java:589)
> >> >> > >> >> at
> >> >> > >> >>
> >> >> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:709)
> >> >> > >> >>
> >> >> > >> >> at
> >> >> > >> >>
> >> >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
> >> >> > >> >> 2009-11-14 16:01:11,433 ERROR
> >> >> > >> >> org.apache.hadoop.hbase.regionserver.HRegionServer: ZooKeeper
> >> >> session
> >> >> > >> >> expired
> >> >> > >> >> 2009-11-14 16:01:13,513 INFO
> org.apache.hadoop.ipc.HBaseServer:
> >> >> IPC
> >> >> > >> Server
> >> >> > >> >> handler 3 on 60021, call put([B@73deb204,
> >> >> > >> >> [Lorg.apache.hadoop.hbase.client.Put;@2179600a) from
> >> >> > >> 192.168.100.132:40728:
> >> >> > >> >> error: java.io.IOException: Server not running, aborting
> >> >> > >> >> java.io.IOException: Server not running, aborting
> >> >> > >> >>
> >> >> > >> >>
> >> >> > >> >>
> >> >> > >> >>
> >> >> > >> >>
> >> >> > >> >> On Sat, Nov 14, 2009 at 1:01 AM, Zhenyu Zhong <
> >> >> > zhongresearch@gmail.com
> >> >> > >> >wrote:
> >> >> > >> >>
> >> >> > >> >>> Stack,
> >> >> > >> >>>
> >> >> > >> >>> Thanks a lot!
> >> >> > >> >>>
> >> >> > >> >>> I found out that the reason why HBase doesn't take the
> system
> >> >> file
> >> >> > >> >>> descriptor value I set before. I started the HBase using
> root
> >> >> > instead
> >> >> > >> of the
> >> >> > >> >>> normal hadoop user, while my system configures hadoop with
> >> higher
> >> >> > file
> >> >> > >> >>> descriptor value but configures root with 1024 default
> value.
> >> >> > >> >>>
> >> >> > >> >>>
> >> >> > >> >>> Now my system has a clean start. However, it seems that the
> >> >> > >> >>> zookeeper.session.timeout value doesn't take into effect. I
> >> still
> >> >> > >> found
> >> >> > >> >>> around 60 seconds pauses from the disconnected regionserver.
> I
> >> >> > really
> >> >> > >> don't
> >> >> > >> >>> know why regionserver only times out after 60 seconds
> instead
> >> of
> >> >> 10
> >> >> > >> minutes
> >> >> > >> >>> which I set for zookeeper.session.timeout.
> >> >> > >> >>>
> >> >> > >> >>> Is there any other timeout value coming into play before the
> >> >> actual
> >> >> > >> >>> session times out?
> >> >> > >> >>>
> >> >> > >> >>> zhenyu
> >> >> > >> >>>
> >> >> > >> >>>
> >> >> > >> >>>
> >> >> > >> >>> On Fri, Nov 13, 2009 at 7:08 PM, stack <st...@duboce.net>
> >> wrote:
> >> >> > >> >>>
> >> >> > >> >>>> Ok. Lack of file descriptors manifests in all kinds of
> weird
> >> >> ways.
> >> >> > >> >>>> Hopefully thats it. If not, lets keep digging.
> >> >> > >> >>>> St.Ack
> >> >> > >> >>>>
> >> >> > >> >>>> On Fri, Nov 13, 2009 at 3:44 PM, Zhenyu Zhong <
> >> >> > >> zhongresearch@gmail.com
> >> >> > >> >>>> >wrote:
> >> >> > >> >>>>
> >> >> > >> >>>> > Stack,
> >> >> > >> >>>> >
> >> >> > >> >>>> > You are right, the master started with ulimit -n 1024. It
> >> >> doesn't
> >> >> > >> take
> >> >> > >> >>>> the
> >> >> > >> >>>> > system value.
> >> >> > >> >>>> >
> >> >> > >> >>>> > Regarding to the too many open files, it looks like the
> >> same
> >> >> as
> >> >> > the
> >> >> > >> one
> >> >> > >> >>>> J-D
> >> >> > >> >>>> > put up. But I will get the Master start with higher value
> >> >> first
> >> >> > and
> >> >> > >> see
> >> >> > >> >>>> if
> >> >> > >> >>>> > this kind of error goes away.
> >> >> > >> >>>> >
> >> >> > >> >>>> > thanks a lot!
> >> >> > >> >>>> > zhenyu
> >> >> > >> >>>> >
> >> >> > >> >>>> > On Fri, Nov 13, 2009 at 6:02 PM, stack <stack@duboce.net
> >
> >> >> wrote:
> >> >> > >> >>>> >
> >> >> > >> >>>> > > Does it say
> >> >> > >> >>>> > >
> >> >> > >> >>>> > > ulimit -n 32768
> >> >> > >> >>>> > >
> >> >> > >> >>>> > > ...as the second line in your log file on start of the
> >> >> master?
> >> >> > >> >>>> > >
> >> >> > >> >>>> > > Is the stack trace that complains about too many open
> >> files
> >> >> > same
> >> >> > >> as
> >> >> > >> >>>> the
> >> >> > >> >>>> > one
> >> >> > >> >>>> > > in the blog post J-D put up?
> >> >> > >> >>>> > >
> >> >> > >> >>>> > > St.Ack
> >> >> > >> >>>> > >
> >> >> > >> >>>> > >
> >> >> > >> >>>> > > On Fri, Nov 13, 2009 at 1:37 PM, Zhenyu Zhong <
> >> >> > >> >>>> zhongresearch@gmail.com
> >> >> > >> >>>> > > >wrote:
> >> >> > >> >>>> > >
> >> >> > >> >>>> > > > The ulimit file descriptors was set to fs.file-max
> >> >> =1578334,
> >> >> > >> also
> >> >> > >> >>>> in
> >> >> > >> >>>> > the
> >> >> > >> >>>> > > > limits.conf the value was set to 32768.
> >> >> > >> >>>> > > > So these are way higher than the open descriptors for
> >> the
> >> >> > >> running
> >> >> > >> >>>> > > > processes.
> >> >> > >> >>>> > > >
> >> >> > >> >>>> > > > thanks
> >> >> > >> >>>> > > > zhenyu
> >> >> > >> >>>> > > >
> >> >> > >> >>>> > > >
> >> >> > >> >>>> > > > On Fri, Nov 13, 2009 at 4:33 PM, Stack <
> >> >> saint.ack@gmail.com>
> >> >> > >> >>>> wrote:
> >> >> > >> >>>> > > >
> >> >> > >> >>>> > > > > You upped the ulimit file descriptors as per the
> >> getting
> >> >> > >> started
> >> >> > >> >>>> doc?
> >> >> > >> >>>> > > > >
> >> >> > >> >>>> > > > >
> >> >> > >> >>>> > > > >
> >> >> > >> >>>> > > > > On Nov 13, 2009, at 1:26 PM, Zhenyu Zhong <
> >> >> > >> >>>> zhongresearch@gmail.com>
> >> >> > >> >>>> > > > wrote:
> >> >> > >> >>>> > > > >
> >> >> > >> >>>> > > > > Thanks a lot.
> >> >> > >> >>>> > > > >>
> >> >> > >> >>>> > > > >>
> >> >> > >> >>>> > > > >> Bad news is my kernel is still 2.6.26.
> >> >> > >> >>>> > > > >> But it was not a problem before.
> >> >> > >> >>>> > > > >>
> >> >> > >> >>>> > > > >> Very strange.
> >> >> > >> >>>> > > > >>
> >> >> > >> >>>> > > > >> zhenyu
> >> >> > >> >>>> > > > >>
> >> >> > >> >>>> > > > >> On Fri, Nov 13, 2009 at 4:16 PM, Jean-Daniel
> Cryans
> >> <
> >> >> > >> >>>> > > > jdcryans@apache.org
> >> >> > >> >>>> > > > >> >wrote:
> >> >> > >> >>>> > > > >>
> >> >> > >> >>>> > > > >> Looks like
> >> >> > >> >>>> > > > >>>
> >> >> > >> >>>> > > > >>>
> >> >> > >> >>>> > > >
> >> >> > >> >>>> > >
> >> >> > >> >>>> >
> >> >> > >> >>>>
> >> >> > >>
> >> >> >
> >> >>
> >>
> http://pero.blogs.aprilmayjune.org/2009/01/22/hadoop-and-linux-kernel-2627-epoll-limits/
> >> >> > >> >>>> > > > >>>
> >> >> > >> >>>> > > > >>> J-D
> >> >> > >> >>>> > > > >>>
> >> >> > >> >>>> > > > >>> On Fri, Nov 13, 2009 at 1:12 PM, Zhenyu Zhong <
> >> >> > >> >>>> > > zhongresearch@gmail.com
> >> >> > >> >>>> > > > >
> >> >> > >> >>>> > > > >>> wrote:
> >> >> > >> >>>> > > > >>>
> >> >> > >> >>>> > > > >>>> Hi,
> >> >> > >> >>>> > > > >>>>
> >> >> > >> >>>> > > > >>>>
> >> >> > >> >>>> > > > >>>> After I re-organize the cluster, the experiment
> >> ran
> >> >> into
> >> >> > >> >>>> problem
> >> >> > >> >>>> > > > faster
> >> >> > >> >>>> > > > >>>>
> >> >> > >> >>>> > > > >>> than
> >> >> > >> >>>> > > > >>>
> >> >> > >> >>>> > > > >>>> before.
> >> >> > >> >>>> > > > >>>>
> >> >> > >> >>>> > > > >>>> Basically the changes are to use machines with
> >> less
> >> >> > >> resources
> >> >> > >> >>>> as
> >> >> > >> >>>> > > > >>>>
> >> >> > >> >>>> > > > >>> zookeeper
> >> >> > >> >>>> > > > >>>
> >> >> > >> >>>> > > > >>>> quorums and machines with more resources as
> >> >> > regionserver.
> >> >> > >> >>>> > > > >>>>
> >> >> > >> >>>> > > > >>>> From the log, I still see the pause around 1
> >> minute.
> >> >> > >> >>>> > > > >>>> I enabled the GC logging, please see
> >> >> > >> >>>> > > > >>>>
> >> >> > >> >>>> > > > >>>> http://pastebin.com/m1d4ce0f1
> >> >> > >> >>>> > > > >>>>
> >> >> > >> >>>> > > > >>>> for details.
> >> >> > >> >>>> > > > >>>> In general I don't see many pauses in the GC.
> >> >> > >> >>>> > > > >>>>
> >> >> > >> >>>> > > > >>>> What is more interesting, 1 hour after the 1st
> >> >> > >> regionserver
> >> >> > >> >>>> > > > >>>> disconnected,
> >> >> > >> >>>> > > > >>>> the master log complained about too many open
> >> files.
> >> >> > This
> >> >> > >> >>>> didn't
> >> >> > >> >>>> > > > happen
> >> >> > >> >>>> > > > >>>> before.
> >> >> > >> >>>> > > > >>>> I checked the system OS setup as well as the
> >> >> > limits.conf.
> >> >> > >> I
> >> >> > >> >>>> also
> >> >> > >> >>>> > > > checked
> >> >> > >> >>>> > > > >>>>
> >> >> > >> >>>> > > > >>> the
> >> >> > >> >>>> > > > >>>
> >> >> > >> >>>> > > > >>>> running processes. The total open files don't
> >> reach
> >> >> the
> >> >> > >> limit.
> >> >> > >> >>>> I
> >> >> > >> >>>> > am
> >> >> > >> >>>> > > > >>>>
> >> >> > >> >>>> > > > >>> confused
> >> >> > >> >>>> > > > >>>
> >> >> > >> >>>> > > > >>>> a bit.
> >> >> > >> >>>> > > > >>>>
> >> >> > >> >>>> > > > >>>>
> >> >> > >> >>>> > > > >>>> Please see the following master log.
> >> >> > >> >>>> > > > >>>>
> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:02,114 INFO
> >> >> > >> >>>> > > > org.apache.hadoop.hbase.master.BaseScanner:
> >> >> > >> >>>> > > > >>>> RegionManager.metaScanner scan of 4658 row(s) of
> >> meta
> >> >> > >> region
> >> >> > >> >>>> > > {server:
> >> >> > >> >>>> > > > >>>> 192.168.100.128:60021, regionname: .META.,,1,
> >> >> startKey:
> >> >> > >> <>}
> >> >> > >> >>>> > > complete
> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:02,114 INFO
> >> >> > >> >>>> > > > org.apache.hadoop.hbase.master.BaseScanner:
> >> >> > >> >>>> > > > >>>>
> >> >> > >> >>>> > > > >>> All
> >> >> > >> >>>> > > > >>>
> >> >> > >> >>>> > > > >>>> 1 .META. region(s) scanned
> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:07,677 DEBUG
> >> >> > >> org.apache.zookeeper.ClientCnxn:
> >> >> > >> >>>> Got
> >> >> > >> >>>> > > > ping
> >> >> > >> >>>> > > > >>>> response for sessionid:0x424eebf1c10004c after
> 3ms
> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:08,178 INFO
> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> >> >> > >> >>>> > > > Exception
> >> >> > >> >>>> > > > >>>>
> >> >> > >> >>>> > > > >>> in
> >> >> > >> >>>> > > > >>>
> >> >> > >> >>>> > > > >>>> createBlockOutputStream java.io.IOException: Bad
> >> >> connect
> >> >> > >> ack
> >> >> > >> >>>> with
> >> >> > >> >>>> > > > >>>> firstBadLink 192.168.100.123:50010
> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:08,178 INFO
> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> >> >> > >> >>>> > > > >>>> Abandoning
> >> >> > >> >>>> > > > >>>> block blk_-2808245019291145247_5478039
> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:09,682 INFO
> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> >> >> > >> >>>> > > > Exception
> >> >> > >> >>>> > > > >>>>
> >> >> > >> >>>> > > > >>> in
> >> >> > >> >>>> > > > >>>
> >> >> > >> >>>> > > > >>>> createBlockOutputStream java.io.EOFException
> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:09,682 INFO
> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> >> >> > >> >>>> > > > >>>> Abandoning
> >> >> > >> >>>> > > > >>>> block blk_1074853606841896259_5478048
> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:10,334 DEBUG
> >> >> > >> org.apache.zookeeper.ClientCnxn:
> >> >> > >> >>>> Got
> >> >> > >> >>>> > > > ping
> >> >> > >> >>>> > > > >>>> response for sessionid:0x24eebf1043003c after
> 1ms
> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:21,018 DEBUG
> >> >> > >> org.apache.zookeeper.ClientCnxn:
> >> >> > >> >>>> Got
> >> >> > >> >>>> > > > ping
> >> >> > >> >>>> > > > >>>> response for sessionid:0x424eebf1c10004c after
> 0ms
> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:23,674 DEBUG
> >> >> > >> org.apache.zookeeper.ClientCnxn:
> >> >> > >> >>>> Got
> >> >> > >> >>>> > > > ping
> >> >> > >> >>>> > > > >>>> response for sessionid:0x24eebf1043003c after
> 0ms
> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:24,828 INFO
> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> >> >> > >> >>>> > > > Exception
> >> >> > >> >>>> > > > >>>>
> >> >> > >> >>>> > > > >>> in
> >> >> > >> >>>> > > > >>>
> >> >> > >> >>>> > > > >>>> createBlockOutputStream java.io.IOException: Bad
> >> >> connect
> >> >> > >> ack
> >> >> > >> >>>> with
> >> >> > >> >>>> > > > >>>> firstBadLink 192.168.100.123:50010
> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:24,828 INFO
> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> >> >> > >> >>>> > > > >>>> Abandoning
> >> >> > >> >>>> > > > >>>> block blk_-6642544517082142289_5478063
> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:24,828 INFO
> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> >> >> > >> >>>> > > > Exception
> >> >> > >> >>>> > > > >>>>
> >> >> > >> >>>> > > > >>> in
> >> >> > >> >>>> > > > >>>
> >> >> > >> >>>> > > > >>>> createBlockOutputStream
> java.net.SocketException:
> >> Too
> >> >> > many
> >> >> > >> >>>> open
> >> >> > >> >>>> > > files
> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:24,828 INFO
> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> >> >> > >> >>>> > > > >>>> Abandoning
> >> >> > >> >>>> > > > >>>> block blk_2057511041109796090_5478063
> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:24,928 INFO
> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> >> >> > >> >>>> > > > Exception
> >> >> > >> >>>> > > > >>>>
> >> >> > >> >>>> > > > >>> in
> >> >> > >> >>>> > > > >>>
> >> >> > >> >>>> > > > >>>> createBlockOutputStream
> java.net.SocketException:
> >> Too
> >> >> > many
> >> >> > >> >>>> open
> >> >> > >> >>>> > > files
> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:24,928 INFO
> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> >> >> > >> >>>> > > > >>>> Abandoning
> >> >> > >> >>>> > > > >>>> block blk_8219260302213892894_5478064
> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:30,855 INFO
> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> >> >> > >> >>>> > > > Exception
> >> >> > >> >>>> > > > >>>>
> >> >> > >> >>>> > > > >>> in
> >> >> > >> >>>> > > > >>>
> >> >> > >> >>>> > > > >>>> createBlockOutputStream
> java.net.SocketException:
> >> Too
> >> >> > many
> >> >> > >> >>>> open
> >> >> > >> >>>> > > files
> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:30,855 INFO
> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> >> >> > >> >>>> > > > >>>> Abandoning
> >> >> > >> >>>> > > > >>>> block blk_1669205542853067709_5478235
> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:30,905 INFO
> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> >> >> > >> >>>> > > > Exception
> >> >> > >> >>>> > > > >>>>
> >> >> > >> >>>> > > > >>> in
> >> >> > >> >>>> > > > >>>
> >> >> > >> >>>> > > > >>>> createBlockOutputStream
> java.net.SocketException:
> >> Too
> >> >> > many
> >> >> > >> >>>> open
> >> >> > >> >>>> > > files
> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:30,905 INFO
> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> >> >> > >> >>>> > > > >>>> Abandoning
> >> >> > >> >>>> > > > >>>> block blk_9128897691346270351_5478237
> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:30,955 INFO
> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> >> >> > >> >>>> > > > Exception
> >> >> > >> >>>> > > > >>>>
> >> >> > >> >>>> > > > >>> in
> >> >> > >> >>>> > > > >>>
> >> >> > >> >>>> > > > >>>> createBlockOutputStream
> java.net.SocketException:
> >> Too
> >> >> > many
> >> >> > >> >>>> open
> >> >> > >> >>>> > > files
> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:30,955 INFO
> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> >> >> > >> >>>> > > > >>>> Abandoning
> >> >> > >> >>>> > > > >>>> block blk_1116845144864123018_5478240
> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:34,372 DEBUG
> >> >> > >> org.apache.zookeeper.ClientCnxn:
> >> >> > >> >>>> Got
> >> >> > >> >>>> > > > ping
> >> >> > >> >>>> > > > >>>> response for sessionid:0x424eebf1c10004c after
> 0ms
> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:37,034 DEBUG
> >> >> > >> org.apache.zookeeper.ClientCnxn:
> >> >> > >> >>>> Got
> >> >> > >> >>>> > > > ping
> >> >> > >> >>>> > > > >>>> response for sessionid:0x24eebf1043003c after
> 0ms
> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:37,235 WARN
> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> >> >> > >> >>>> > > > >>>>
> >> >> > >> >>>> > > > >>> DataStreamer
> >> >> > >> >>>> > > > >>>
> >> >> > >> >>>> > > > >>>> Exception: java.io.IOException: Too many open
> >> files
> >> >> > >> >>>> > > > >>>> at sun.nio.ch.IOUtil.initPipe(Native
> Method)
> >> >> > >> >>>> > > > >>>> at
> >> >> > >> >>>> > >
> >> >> sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:49)
> >> >> > >> >>>> > > > >>>> at
> >> >> > >> >>>> > > > >>>>
> >> >> > >> >>>> > > > >>>>
> >> >> > >> >>>> > > >
> >> >> > >> >>>> > >
> >> >> > >> >>>> >
> >> >> > >> >>>>
> >> >> > >>
> >> >> >
> >> >>
> >>
> sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:18)
> >> >> > >> >>>> > > > >>>
> >> >> > >> >>>> > > > >>>> at
> >> >> > >> >>>> > > > >>>>
> >> >> > >> >>>> > > > >>>>
> >> >> > >> >>>> > > >
> >> >> > >> >>>> > >
> >> >> > >> >>>> >
> >> >> > >> >>>>
> >> >> > >>
> >> >> >
> >> >>
> >>
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.get(SocketIOWithTimeout.java:407)
> >> >> > >> >>>> > > > >>>
> >> >> > >> >>>> > > > >>>> at
> >> >> > >> >>>> > > > >>>>
> >> >> > >> >>>> > > > >>>>
> >> >> > >> >>>> > > >
> >> >> > >> >>>> > >
> >> >> > >> >>>> >
> >> >> > >> >>>>
> >> >> > >>
> >> >> >
> >> >>
> >>
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:322)
> >> >> > >> >>>> > > > >>>
> >> >> > >> >>>> > > > >>>> at
> >> >> > >> >>>> > > > >>>>
> >> >> > >> >>>> > > > >>>>
> >> >> > >> >>>> > > >
> >> >> > >> >>>> > >
> >> >> > >> >>>> >
> >> >> > >> >>>>
> >> >> > >>
> >> >> >
> >> >>
> >>
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
> >> >> > >> >>>> > > > >>>
> >> >> > >> >>>> > > > >>>> at
> >> >> > >> >>>> > > > >>>>
> >> >> > >> >>>> > > > >>>>
> >> >> > >> >>>> > > >
> >> >> > >> >>>> > >
> >> >> > >> >>>> >
> >> >> > >> >>>>
> >> >> > >>
> >> >> >
> >> >>
> >>
> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
> >> >> > >> >>>> > > > >>>
> >> >> > >> >>>> > > > >>>> at
> >> >> > >> >>>> > > > >>>>
> >> >> > >> >>>> > > > >>>>
> >> >> > >> >>>> > > >
> >> >> > >> >>>> > >
> >> >> > >> >>>> >
> >> >> > >> >>>>
> >> >> > >>
> >> >> >
> >> >>
> >>
> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
> >> >> > >> >>>> > > > >>>
> >> >> > >> >>>> > > > >>>> at
> >> >> > >> >>>> > > > >>>>
> >> >> > >> >>>> > > > >>>
> >> >> > >> >>>>
> >> >> java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
> >> >> > >> >>>> > > > >>>
> >> >> > >> >>>> > > > >>>> at
> >> >> > >> >>>> java.io.DataOutputStream.write(DataOutputStream.java:90)
> >> >> > >> >>>> > > > >>>> at
> >> >> > >> >>>> > > > >>>>
> >> >> > >> >>>> > > > >>>>
> >> >> > >> >>>> > > >
> >> >> > >> >>>> > >
> >> >> > >> >>>> >
> >> >> > >> >>>>
> >> >> > >>
> >> >> >
> >> >>
> >>
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2290)
> >> >> > >> >>>> > > > >>>
> >> >> > >> >>>> > > > >>>>
> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:37,235 WARN
> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> >> >> > >> >>>> > Error
> >> >> > >> >>>> > > > >>>> Recovery for block
> blk_8148813491785406356_5478475
> >> >> bad
> >> >> > >> >>>> datanode[0]
> >> >> > >> >>>> > > > >>>> 192.168.100.123:50010
> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:37,235 WARN
> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> >> >> > >> >>>> > Error
> >> >> > >> >>>> > > > >>>> Recovery for block
> blk_8148813491785406356_5478475
> >> in
> >> >> > >> pipeline
> >> >> > >> >>>> > > > >>>> 192.168.100.123:50010, 192.168.100.134:50010,
> >> >> > >> >>>> > 192.168.100.122:50010
> >> >> > >> >>>> > > :
> >> >> > >> >>>> > > > >>>> bad
> >> >> > >> >>>> > > > >>>> datanode 192.168.100.123:50010
> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:37,436 INFO
> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> >> >> > >> >>>> > > > Exception
> >> >> > >> >>>> > > > >>>>
> >> >> > >> >>>> > > > >>> in
> >> >> > >> >>>> > > > >>>
> >> >> > >> >>>> > > > >>>> createBlockOutputStream
> java.net.SocketException:
> >> Too
> >> >> > many
> >> >> > >> >>>> open
> >> >> > >> >>>> > > files
> >> >> > >> >>>> > > > >>>> 2009-11-13 20:06:37,436 INFO
> >> >> > >> org.apache.hadoop.hdfs.DFSClient:
> >> >> > >> >>>> > > > >>>> Abandoning
> >> >> > >> >>>> > > > >>>> block blk_2119727700857186236_5478498
> >> >> > >> >>>> > > > >>>>
> >> >> > >> >>>> > > > >>>>
> >> >> > >> >>>> > > > >>>>
> >> >> > >> >>>> > > > >>>>
> >> >> > >> >>>> > > > >>>>
> >> >> > >> >>>> > > > >>>> On Thu, Nov 12, 2009 at 4:21 PM, Zhenyu Zhong <
> >> >> > >> >>>> > > > zhongresearch@gmail.com
> >> >> > >> >>>> > > > >>>> wrote:
> >> >> > >> >>>> > > > >>>>
> >> >> > >> >>>> > > > >>>> Will do.
> >> >> > >> >>>> > > > >>>>>
> >> >> > >> >>>> > > > >>>>> thanks
> >> >> > >> >>>> > > > >>>>> zhenyu
> >> >> > >> >>>> > > > >>>>>
> >> >> > >> >>>> > > > >>>>>
> >> >> > >> >>>> > > > >>>>> On Thu, Nov 12, 2009 at 3:33 PM, stack <
> >> >> > stack@duboce.net
> >> >> > >> >
> >> >> > >> >>>> wrote:
> >> >> > >> >>>> > > > >>>>>
> >> >> > >> >>>> > > > >>>>> Enable GC logging too on this next run (see
> >> >> > >> hbase-env.sh).
> >> >> > >> >>>> Lets
> >> >> > >> >>>> > > try
> >> >> > >> >>>> > > > >>>>>>
> >> >> > >> >>>> > > > >>>>> and
> >> >> > >> >>>> > > > >>>
> >> >> > >> >>>> > > > >>>> get
> >> >> > >> >>>> > > > >>>>>> to the bottom of whats going on.
> >> >> > >> >>>> > > > >>>>>> Thanks,
> >> >> > >> >>>> > > > >>>>>> St.Ack
> >> >> > >> >>>> > > > >>>>>>
> >> >> > >> >>>> > > > >>>>>> On Thu, Nov 12, 2009 at 12:29 PM, Zhenyu Zhong
> <
> >> >> > >> >>>> > > > >>>>>>
> >> >> > >> >>>> > > > >>>>> zhongresearch@gmail.com
> >> >> > >> >>>> > > > >>>
> >> >> > >> >>>> > > > >>>> wrote:
> >> >> > >> >>>> > > > >>>>>>>
> >> >> > >> >>>> > > > >>>>>>
> >> >> > >> >>>> > > > >>>>>> I can switch the boxes that run zookeeper
> with
> >> the
> >> >> > ones
> >> >> > >> >>>> that
> >> >> > >> >>>> > run
> >> >> > >> >>>> > > > >>>>>>> regionservers.
> >> >> > >> >>>> > > > >>>>>>> I will see the results later.
> >> >> > >> >>>> > > > >>>>>>>
> >> >> > >> >>>> > > > >>>>>>> FYI. The node does have the 10 minutes
> >> >> > >> >>>> > zookeeper.session.timeout
> >> >> > >> >>>> > > > >>>>>>>
> >> >> > >> >>>> > > > >>>>>> value
> >> >> > >> >>>> > > > >>>
> >> >> > >> >>>> > > > >>>> in
> >> >> > >> >>>> > > > >>>>>>
> >> >> > >> >>>> > > > >>>>>>> place.
> >> >> > >> >>>> > > > >>>>>>>
> >> >> > >> >>>> > > > >>>>>>> thanks
> >> >> > >> >>>> > > > >>>>>>> zhenyu
> >> >> > >> >>>> > > > >>>>>>>
> >> >> > >> >>>> > > > >>>>>>> On Thu, Nov 12, 2009 at 3:21 PM, stack <
> >> >> > >> stack@duboce.net>
> >> >> > >> >>>> > wrote:
> >> >> > >> >>>> > > > >>>>>>>
> >> >> > >> >>>> > > > >>>>>>> On Thu, Nov 12, 2009 at 11:50 AM, Zhenyu
> Zhong
> >> <
> >> >> > >> >>>> > > > >>>>>>>>
> >> >> > >> >>>> > > > >>>>>>> zhongresearch@gmail.com
> >> >> > >> >>>> > > > >>>>>>
> >> >> > >> >>>> > > > >>>>>>> wrote:
> >> >> > >> >>>> > > > >>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>> In my cluster, half of the cluster have 2
> >> disks
> >> >> > 400GB
> >> >> > >> >>>> each
> >> >> > >> >>>> > per
> >> >> > >> >>>> > > > >>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>> machine,
> >> >> > >> >>>> > > > >>>>>>
> >> >> > >> >>>> > > > >>>>>>> and
> >> >> > >> >>>> > > > >>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>> half of the cluster have 6 disks per
> machine.
> >> >> > Maybe
> >> >> > >> we
> >> >> > >> >>>> > should
> >> >> > >> >>>> > > > >>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>> run
> >> >> > >> >>>> > > > >>>
> >> >> > >> >>>> > > > >>>> zookeeper
> >> >> > >> >>>> > > > >>>>>>>>> on the machines with 2 disks and RS on
> >> machines
> >> >> > with
> >> >> > >> 6
> >> >> > >> >>>> disks?
> >> >> > >> >>>> > > > >>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>> That would make most sense only in the
> below,
> >> it
> >> >> > >> looks
> >> >> > >> >>>> like
> >> >> > >> >>>> > the
> >> >> > >> >>>> > > > >>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>> RS
> >> >> > >> >>>> > > > >>>
> >> >> > >> >>>> > > > >>>> that
> >> >> > >> >>>> > > > >>>>>>
> >> >> > >> >>>> > > > >>>>>>> had
> >> >> > >> >>>> > > > >>>>>>>> issue had 4 disks?
> >> >> > >> >>>> > > > >>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>> BTW, the 10 minutes
> zookeeper.session.timeout
> >> >> has
> >> >> > >> been
> >> >> > >> >>>> set
> >> >> > >> >>>> > > during
> >> >> > >> >>>> > > > >>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>> the
> >> >> > >> >>>> > > > >>>>>>
> >> >> > >> >>>> > > > >>>>>>> experiment.
> >> >> > >> >>>> > > > >>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>> And for sure this node had it in place?
> >> >> > >> >>>> > > > >>>>>>>> St.Ack
> >> >> > >> >>>> > > > >>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>> thanks
> >> >> > >> >>>> > > > >>>>>>>>> zhenyu
> >> >> > >> >>>> > > > >>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>> On Thu, Nov 12, 2009 at 2:08 PM, stack <
> >> >> > >> stack@duboce.net
> >> >> > >> >>>> >
> >> >> > >> >>>> > > wrote:
> >> >> > >> >>>> > > > >>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>> On Thu, Nov 12, 2009 at 8:40 AM, Zhenyu
> >> Zhong <
> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>> zhongresearch@gmail.com
> >> >> > >> >>>> > > > >>>>>>>
> >> >> > >> >>>> > > > >>>>>>>> wrote:
> >> >> > >> >>>> > > > >>>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>> Though I experienced 2 regionserver
> >> >> disconnection
> >> >> > >> this
> >> >> > >> >>>> > > > >>>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>> morning,
> >> >> > >> >>>> > > > >>>
> >> >> > >> >>>> > > > >>>> it
> >> >> > >> >>>> > > > >>>>>>
> >> >> > >> >>>> > > > >>>>>>> looks
> >> >> > >> >>>> > > > >>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>> better from the regionserver log. (please
> >> see
> >> >> the
> >> >> > >> >>>> following
> >> >> > >> >>>> > > > >>>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>> log)
> >> >> > >> >>>> > > > >>>
> >> >> > >> >>>> > > > >>>>
> >> >> > >> >>>> > > > >>>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>>> http://pastebin.com/m496dbfae
> >> >> > >> >>>> > > > >>>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>>> I checked diskIO for one of the
> >> >> > >> >>>> > regionserver(192.168.100.116)
> >> >> > >> >>>> > > > >>>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>> during
> >> >> > >> >>>> > > > >>>>>>>
> >> >> > >> >>>> > > > >>>>>>>> the
> >> >> > >> >>>> > > > >>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>> time it disconnected.
> >> >> > >> >>>> > > > >>>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>>> Time: 03:04:58 AM
> >> >> > >> >>>> > > > >>>>>>>>>>> Device: tps Blk_read/s
> >> >> Blk_wrtn/s
> >> >> > >> >>>> Blk_read
> >> >> > >> >>>> > > > >>>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>> Blk_wrtn
> >> >> > >> >>>> > > > >>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>> sda 105.31 5458.83
> >> 19503.64
> >> >> > >> >>>> 9043873239
> >> >> > >> >>>> > > > >>>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>> 32312473676
> >> >> > >> >>>> > > > >>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>> sda1 2.90 6.64
> >> 99.25
> >> >> > >> >>>> 10993934
> >> >> > >> >>>> > > > >>>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>> 164433464
> >> >> > >> >>>> > > > >>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>> sda2 1.72 23.76
> >> 30.25
> >> >> > >> >>>> 39365817
> >> >> > >> >>>> > > > >>>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>> 50124008
> >> >> > >> >>>> > > > >>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>> sda3 0.30 0.38
> >> 3.58
> >> >> > >> >>>> 624930
> >> >> > >> >>>> > > > >>>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>> 5923000
> >> >> > >> >>>> > > > >>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>> sda4 100.39 5428.06
> >> 19370.56
> >> >> > >> >>>> 8992888270
> >> >> > >> >>>> > > > >>>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>> 32091993204
> >> >> > >> >>>> > > > >>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>>> Is this high for you? 20k blocks/second
> >> >> would
> >> >> > >> seem to
> >> >> > >> >>>> be
> >> >> > >> >>>> > > high
> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>> but
> >> >> > >> >>>> > > > >>>
> >> >> > >> >>>> > > > >>>> its
> >> >> > >> >>>> > > > >>>>>>>
> >> >> > >> >>>> > > > >>>>>>>> one
> >> >> > >> >>>> > > > >>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>> disk only and its not being shared by zk
> >> >> anymore
> >> >> > so
> >> >> > >> >>>> > shouldn't
> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>> matter?
> >> >> > >> >>>> > > > >>>>>>
> >> >> > >> >>>> > > > >>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>> I also checked the zookeeper quorum
> server
> >> >> that
> >> >> > the
> >> >> > >> >>>> > > > >>>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>> regionserver
> >> >> > >> >>>> > > > >>>
> >> >> > >> >>>> > > > >>>> tried
> >> >> > >> >>>> > > > >>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>> to
> >> >> > >> >>>> > > > >>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>> connect according to the log. However, I
> >> don't
> >> >> see
> >> >> > >> >>>> > > > >>>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>> 192.168.100.116
> >> >> > >> >>>> > > > >>>>>>
> >> >> > >> >>>> > > > >>>>>>> in
> >> >> > >> >>>> > > > >>>>>>>
> >> >> > >> >>>> > > > >>>>>>>> the
> >> >> > >> >>>> > > > >>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>> client list of the zookeeper quorum server
> >> that
> >> >> > the
> >> >> > >> >>>> > > > >>>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>> regionserver
> >> >> > >> >>>> > > > >>>
> >> >> > >> >>>> > > > >>>> tried
> >> >> > >> >>>> > > > >>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>> to
> >> >> > >> >>>> > > > >>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>> connect.
> >> >> > >> >>>> > > > >>>>>>>>>>> Would that be a problem?
> >> >> > >> >>>> > > > >>>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>>> Is that because the ephemeral node for
> the
> >> >> > >> >>>> regionserver
> >> >> > >> >>>> > had
> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>> evaporated?
> >> >> > >> >>>> > > > >>>>>>>
> >> >> > >> >>>> > > > >>>>>>>> Lost
> >> >> > >> >>>> > > > >>>>>>>>>> its lease w/ zk by the time you went to
> >> look?
> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>> Thu Nov 12 15:04:52 UTC 2009
> >> >> > >> >>>> > > > >>>>>>>>>>> Zookeeper version: 3.2.1-808558, built on
> >> >> > >> 08/27/2009
> >> >> > >> >>>> 18:48
> >> >> > >> >>>> > > > >>>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>> GMT
> >> >> > >> >>>> > > > >>>
> >> >> > >> >>>> > > > >>>> Clients:
> >> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.127:43045
> >> >> > >> [1](queued=0,recved=26,sent=0)
> >> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.131:39091
> >> >> > >> [1](queued=0,recved=964,sent=0)
> >> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.124:35961
> >> >> > >> [1](queued=0,recved=3266,sent=0)
> >> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.123:47935
> >> >> > >> [1](queued=0,recved=233,sent=0)
> >> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.125:46931
> >> >> > >> [1](queued=0,recved=2,sent=0)
> >> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.118:54924
> >> >> > >> [1](queued=0,recved=295,sent=0)
> >> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.118:41390
> >> >> > >> [1](queued=0,recved=2290,sent=0)
> >> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.136:42243
> >> >> > >> [1](queued=0,recved=0,sent=0)
> >> >> > >> >>>> > > > >>>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>>> Latency min/avg/max: 0/17/6333
> >> >> > >> >>>> > > > >>>>>>>>>>> Received: 47111
> >> >> > >> >>>> > > > >>>>>>>>>>> Sent: 0
> >> >> > >> >>>> > > > >>>>>>>>>>> Outstanding: 0
> >> >> > >> >>>> > > > >>>>>>>>>>> Zxid: 0x77000083f4
> >> >> > >> >>>> > > > >>>>>>>>>>> Mode: leader
> >> >> > >> >>>> > > > >>>>>>>>>>> Node count: 23
> >> >> > >> >>>> > > > >>>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>> That 6 second maximum latency is pretty
> bad
> >> but
> >> >> > >> should
> >> >> > >> >>>> be
> >> >> > >> >>>> > well
> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>> within
> >> >> > >> >>>> > > > >>>>>>
> >> >> > >> >>>> > > > >>>>>>> the
> >> >> > >> >>>> > > > >>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>> zk
> >> >> > >> >>>> > > > >>>>>>>>>> session timeout.
> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>> So, problem is likely on the zk client
> side
> >> of
> >> >> the
> >> >> > >> >>>> session;
> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>> i.e.
> >> >> > >> >>>> > > > >>>
> >> >> > >> >>>> > > > >>>> in
> >> >> > >> >>>> > > > >>>>>>
> >> >> > >> >>>> > > > >>>>>>> the
> >> >> > >> >>>> > > > >>>>>>>
> >> >> > >> >>>> > > > >>>>>>>> RS.
> >> >> > >> >>>> > > > >>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>> You could enable GC logging as J-D
> suggested
> >> to
> >> >> > see
> >> >> > >> if
> >> >> > >> >>>> you
> >> >> > >> >>>> > > have
> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>> any
> >> >> > >> >>>> > > > >>>>>>
> >> >> > >> >>>> > > > >>>>>>> big
> >> >> > >> >>>> > > > >>>>>>>
> >> >> > >> >>>> > > > >>>>>>>> pauses, pauses > zk session timeout.
> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>> When the RS went down, it didn't look too
> >> >> heavily
> >> >> > >> >>>> loaded:
> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>> 1. 2009-11-12 15:04:52,830 INFO
> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> > org.apache.hadoop.hbase.regionserver.HRegionServer:
> >> >> > >> >>>> Dump of
> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>> metrics:
> >> >> > >> >>>> > > > >>>>>>>
> >> >> > >> >>>> > > > >>>>>>>> request=1.5166667, regions=322, stores=657,
> >> >> > >> >>>> storefiles=631,
> >> >> > >> >>>> > > > >>>>>>>>>> storefileIndexSize=61, memstoreSize=1472,
> >> >> > >> >>>> usedHeap=2819,
> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>> maxHeap=4079,
> >> >> > >> >>>> > > > >>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>> blockCacheSize=658110960,
> >> >> > blockCacheFree=197395984,
> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>> blockCacheCount=9903,
> >> >> > >> >>>> > > > >>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>> blockCacheHitRatio=99
> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>> Its serving a few reads? The number of
> >> store
> >> >> > files
> >> >> > >> >>>> seems
> >> >> > >> >>>> > > fine.
> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>> Not
> >> >> > >> >>>> > > > >>>>>>
> >> >> > >> >>>> > > > >>>>>>> too
> >> >> > >> >>>> > > > >>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>> much memory used.
> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>> Looking at the logs, I see the Lease Still
> >> Held
> >> >> > >> >>>> exception.
> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>> This
> >> >> > >> >>>> > > > >>>
> >> >> > >> >>>> > > > >>>> happens
> >> >> > >> >>>> > > > >>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>> when the RS does its regular report to the
> >> >> master
> >> >> > but
> >> >> > >> the
> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>> master
> >> >> > >> >>>> > > > >>>
> >> >> > >> >>>> > > > >>>> thinks
> >> >> > >> >>>> > > > >>>>>>>
> >> >> > >> >>>> > > > >>>>>>>> the
> >> >> > >> >>>> > > > >>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>> RS has since restarted. It'll think this
> >> >> probably
> >> >> > >> >>>> because
> >> >> > >> >>>> > it
> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>> noticed
> >> >> > >> >>>> > > > >>>>>>
> >> >> > >> >>>> > > > >>>>>>> that
> >> >> > >> >>>> > > > >>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>> the RS's znode in zk had gone away and it
> >> >> > considered
> >> >> > >> the
> >> >> > >> >>>> RS
> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>> dead.
> >> >> > >> >>>> > > > >>>
> >> >> > >> >>>> > > > >>>>
> >> >> > >> >>>> > > > >>>>>>>>>> Looking too at your logs I see this gap
> in
> >> the
> >> >> zk
> >> >> > >> >>>> pinging:
> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>> 1. 2009-11-12 15:03:39,325 DEBUG
> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>> org.apache.zookeeper.ClientCnxn:
> >> >> > >> >>>> > > > >>>>>>
> >> >> > >> >>>> > > > >>>>>>> Got
> >> >> > >> >>>> > > > >>>>>>>
> >> >> > >> >>>> > > > >>>>>>>> ping response for
> sessionid:0x224e55436ad0004
> >> >> after
> >> >> > >> 0ms
> >> >> > >> >>>> > > > >>>>>>>>>> 2. 2009-11-12 15:03:43,113 DEBUG
> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>> org.apache.zookeeper.ClientCnxn:
> >> >> > >> >>>> > > > >>>>>>
> >> >> > >> >>>> > > > >>>>>>> Got
> >> >> > >> >>>> > > > >>>>>>>
> >> >> > >> >>>> > > > >>>>>>>> ping response for
> sessionid:0x24e55436a0007d
> >> >> after
> >> >> > >> 0ms
> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>> Where in the lines above it, its reporting
> >> >> about
> >> >> > >> every
> >> >> > >> >>>> ten
> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>> seconds,
> >> >> > >> >>>> > > > >>>>>>
> >> >> > >> >>>> > > > >>>>>>> here
> >> >> > >> >>>> > > > >>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>> there is a big gap.
> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>> Do you have ganglia or something that will
> >> let
> >> >> you
> >> >> > >> look
> >> >> > >> >>>> more
> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>> into
> >> >> > >> >>>> > > > >>>
> >> >> > >> >>>> > > > >>>> what
> >> >> > >> >>>> > > > >>>>>>>
> >> >> > >> >>>> > > > >>>>>>>> was
> >> >> > >> >>>> > > > >>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>> happening on this machine around this
> time?
> >> Is
> >> >> > the
> >> >> > >> >>>> machine
> >> >> > >> >>>> > > OK?
> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>> It
> >> >> > >> >>>> > > > >>>>>>
> >> >> > >> >>>> > > > >>>>>>> looks
> >> >> > >> >>>> > > > >>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>> lightly loaded and you have your cluster
> >> nicely
> >> >> > laid
> >> >> > >> out.
> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>> Something
> >> >> > >> >>>> > > > >>>>>>
> >> >> > >> >>>> > > > >>>>>>> odd
> >> >> > >> >>>> > > > >>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>> is
> >> >> > >> >>>> > > > >>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>> going on. What about things like the
> write
> >> >> speed
> >> >> > to
> >> >> > >> >>>> disk?
> >> >> > >> >>>> > In
> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>> the
> >> >> > >> >>>> > > > >>>
> >> >> > >> >>>> > > > >>>> past
> >> >> > >> >>>> > > > >>>>>>>
> >> >> > >> >>>> > > > >>>>>>>> strange issues have been explained by
> >> incorrectly
> >> >> > set
> >> >> > >> BIOS
> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>> which
> >> >> > >> >>>> > > > >>>
> >> >> > >> >>>> > > > >>>> made
> >> >> > >> >>>> > > > >>>>>>
> >> >> > >> >>>> > > > >>>>>>> disks
> >> >> > >> >>>> > > > >>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>> run at 1/100th of their proper speed.
> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>> St.Ack
> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>>> Best,
> >> >> > >> >>>> > > > >>>>>>>>>>> zhenyu
> >> >> > >> >>>> > > > >>>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>>> On Wed, Nov 11, 2009 at 3:58 PM, Zhenyu
> >> Zhong
> >> >> <
> >> >> > >> >>>> > > > >>>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>> zhongresearch@gmail.com
> >> >> > >> >>>> > > > >>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>> wrote:
> >> >> > >> >>>> > > > >>>>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>>> Stack
> >> >> > >> >>>> > > > >>>>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>>>> I am very appreciated for your comments.
> >> >> > >> >>>> > > > >>>>>>>>>>>> I will use the zookeeper monitoring
> script
> >> on
> >> >> my
> >> >> > >> >>>> cluster
> >> >> > >> >>>> > > > >>>>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>>> and
> >> >> > >> >>>> > > > >>>
> >> >> > >> >>>> > > > >>>> let
> >> >> > >> >>>> > > > >>>>>>
> >> >> > >> >>>> > > > >>>>>>> it
> >> >> > >> >>>> > > > >>>>>>>
> >> >> > >> >>>> > > > >>>>>>>> run
> >> >> > >> >>>> > > > >>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>> overnight to see the result.
> >> >> > >> >>>> > > > >>>>>>>>>>>> I will follow up the thread when I get
> >> >> anything.
> >> >> > >> >>>> > > > >>>>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>>>> thanks
> >> >> > >> >>>> > > > >>>>>>>>>>>> zhenyu
> >> >> > >> >>>> > > > >>>>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>>>> On Wed, Nov 11, 2009 at 3:52 PM, stack <
> >> >> > >> >>>> stack@duboce.net>
> >> >> > >> >>>> > > > >>>>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>>> wrote:
> >> >> > >> >>>> > > > >>>>>>
> >> >> > >> >>>> > > > >>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>>>> I see these in your log too:
> >> >> > >> >>>> > > > >>>>>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>>>>> 1. 2009-11-11 04:27:19,018 DEBUG
> >> >> > >> >>>> > > > >>>>>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
> >> >> > >> >>>> > > > >>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>> Got
> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>>> ping response for
> >> sessionid:0x424dfd908c50009
> >> >> > >> after
> >> >> > >> >>>> > > > >>>>>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>>>> 4544ms
> >> >> > >> >>>> > > > >>>
> >> >> > >> >>>> > > > >>>> 2. 2009-11-11 04:27:19,018 DEBUG
> >> >> > >> >>>> > > > >>>>>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
> >> >> > >> >>>> > > > >>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>> Got
> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>>> ping response for
> >> sessionid:0x24dfd90c810002
> >> >> > after
> >> >> > >> >>>> > > > >>>>>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>>>> 4548ms
> >> >> > >> >>>> > > > >>>
> >> >> > >> >>>> > > > >>>> 3. 2009-11-11 04:27:43,960 DEBUG
> >> >> > >> >>>> > > > >>>>>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
> >> >> > >> >>>> > > > >>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>> Got
> >> >> > >> >>>> > > > >>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>>> ping response for
> >> sessionid:0x424dfd908c50009
> >> >> > >> after
> >> >> > >> >>>> > > > >>>>>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>>>> 9030ms
> >> >> > >> >>>> > > > >>>
> >> >> > >> >>>> > > > >>>> 4. 2009-11-11 04:27:43,960 DEBUG
> >> >> > >> >>>> > > > >>>>>>>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>>>>>> org.apache.zookeeper.
> >> >> > >> >>>> > > > >>>>>>>>
> >> >> > >> >>>> > > > >>>>>>>
> >> >> > >> >>>> > > >
> >> >> > >> >>>> > >
> >> >> > >> >>>> >
> >> >> > >> >>>>
> >> >> > >> >>>
> >> >> > >> >>>
> >> >> > >> >>
> >> >> > >> >
> >> >> > >>
> >> >> > >
> >> >> > >
> >> >> >
> >> >>
> >> >
> >> >
> >>
> >
> >
>
Re: regionserver disconnection
Posted by stack <st...@duboce.net>.
Patrick Hunt, one of the zk lads, answered with the following:
According to the following snippet (I cut to just session 0x425035c48210002)
the client is getting pings for this session id regularly every 6 seconds or
so, then after 41,016 there's a 20 second gap (corresponds to a 30 second
timeout being set during session creation) at which point we disconnect from
the server.
> 2009-11-18 12:41:34,788 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
> response for sessionid:0x425035c48210002 after 0ms
> 2009-11-18 12:42:01,482 WARN org.apache.zookeeper.ClientCnxn: Exception
> closing session 0x425035c48210002 to sun.nio.ch.SelectionKeyImpl@421690ab
> java.io.IOException: TIMED OUT
the client notifies hbase of the disco, and attempts to reconnect to a
server
> org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
> state: Disconnected, type: None, path: null
> 2009-11-18 12:42:01,782 INFO org.apache.zookeeper.ClientCnxn: Attempting
> connection to server superpyxis0003/192.168.100.117:2181
but the server won't allow the session be be established for some reason (is
that the right server:port that's it's trying to connect to?)
> 2009-11-18 12:42:02,182 INFO org.apache.zookeeper.ClientCnxn: Server
> connection successful
> 2009-11-18 12:42:02,182 WARN org.apache.zookeeper.ClientCnxn: Exception
> closing session 0x425035c48210002 to sun.nio.ch.SelectionKeyImpl@5c07076b
> java.io.IOException: Read error rc = -1 java.nio.DirectByteBuffer[pos=0
> lim=4 cap=4]
The "read error rc = -1" means that the client read from the socket and got
EOS. But it's not a session expiration afaict (and should not be at this
point in time).
You need to look at the ZK server logs for this period in time (for this
server superpyxis0003:2181, and superpyxis0001:2181, the client tries to
connect to both of these). There should be some indication there why the
server is closing down the connection and not allowing the session to be
re-established.
One thought - by default we limit clients to a max of 10 connections from a
particular host to the server. If you exceed this limit the zk server (it's
per server though, not per cluster, so perhaps this is not it) will not
allow any additional connections from the client host (you can't exceed 10
on a server). Could this be it? i.e. the user is running a number of ZK
clients (say hbase processes) from the same host? Regardless, get the server
log for this time period and it should be more clear.
Patrick
On Wed, Nov 18, 2009 at 11:18 AM, stack <st...@duboce.net> wrote:
> And running stat on each of your zookeeper nodes tells you what about
> averages for connect-times?
>
> Its odd that there is a ping just 20 seconds beforehand and you have set
> the zk session timeout at 60 seconds (or ten minutes?). Let me ask the zk
> lads about it.
>
> If you look in the GC log for around this time, do you see anything?
>
> St.Ack
>
>
>
> On Wed, Nov 18, 2009 at 7:32 AM, Zhenyu Zhong <zh...@gmail.com>wrote:
>
>> After running more experiment, it seems that the disconnection is not
>> related to the heavy load job, because the load average is low, the diskio
>> is normal, memory Heap was not reached, also the virtual memory stats
>> shows
>> that no swappings. However, the disconnection still happens.
>>
>> It looks like this time it pauses for 20 seconds. No idea why regionserver
>> disconnected.
>>
>> Any other suggestions please?
>>
>>
>> 2009-11-18 12:41:00,956 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
>> response for sessionid:0x125035c495d02bb after 0ms
>> 2009-11-18 12:41:08,074 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
>> response for sessionid:0x425035c48210002 after 0ms
>> 2009-11-18 12:41:14,302 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
>> response for sessionid:0x125035c495d02bb after 0ms
>> 2009-11-18 12:41:21,433 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
>> response for sessionid:0x425035c48210002 after 0ms
>> 2009-11-18 12:41:27,656 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
>> response for sessionid:0x125035c495d02bb after 0ms
>> 2009-11-18 12:41:34,788 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
>> response for sessionid:0x425035c48210002 after 0ms
>> 2009-11-18 12:41:41,016 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
>> response for sessionid:0x125035c495d02bb after 1ms
>> 2009-11-18 12:42:01,482 WARN org.apache.zookeeper.ClientCnxn: Exception
>> closing session 0x425035c48210002 to sun.nio.ch.SelectionKeyImpl@421690ab
>> java.io.IOException: TIMED OUT
>> at
>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:906)
>> 2009-11-18 12:42:01,582 INFO
>> org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
>> state: Disconnected, type: None, path: null
>> 2009-11-18 12:42:01,782 INFO org.apache.zookeeper.ClientCnxn: Attempting
>> connection to server superpyxis0003/192.168.100.117:2181
>> 2009-11-18 12:42:01,782 INFO org.apache.zookeeper.ClientCnxn: Priming
>> connection to java.nio.channels.SocketChannel[connected local=/
>> 192.168.100.132:46610 remote=superpyxis0003/192.168.100.117:2181]
>> 2009-11-18 12:42:01,782 INFO org.apache.zookeeper.ClientCnxn: Server
>> connection successful
>> 2009-11-18 12:42:01,782 WARN org.apache.zookeeper.ClientCnxn: Exception
>> closing session 0x425035c48210002 to sun.nio.ch.SelectionKeyImpl@56dc6fac
>> java.io.IOException: Read error rc = -1 java.nio.DirectByteBuffer[pos=0
>> lim=4 cap=4]
>> at
>> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
>> at
>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
>> 2009-11-18 12:42:01,782 WARN org.apache.zookeeper.ClientCnxn: Ignoring
>> exception during shutdown input
>> java.net.SocketException: Transport endpoint is not connected
>> at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
>> at
>> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
>> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>> at
>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
>> at
>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
>> 2009-11-18 12:42:01,782 WARN org.apache.zookeeper.ClientCnxn: Ignoring
>> exception during shutdown output
>> java.net.SocketException: Transport endpoint is not connected
>> at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
>> at
>> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
>> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
>> at
>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
>> at
>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
>> 2009-11-18 12:42:02,182 INFO org.apache.zookeeper.ClientCnxn: Attempting
>> connection to server superpyxis0001/192.168.100.115:2181
>> 2009-11-18 12:42:02,182 INFO org.apache.zookeeper.ClientCnxn: Priming
>> connection to java.nio.channels.SocketChannel[connected local=/
>> 192.168.100.132:36197 remote=superpyxis0001/192.168.100.115:2181]
>> 2009-11-18 12:42:02,182 INFO org.apache.zookeeper.ClientCnxn: Server
>> connection successful
>> 2009-11-18 12:42:02,182 WARN org.apache.zookeeper.ClientCnxn: Exception
>> closing session 0x425035c48210002 to sun.nio.ch.SelectionKeyImpl@5c07076b
>> java.io.IOException: Read error rc = -1 java.nio.DirectByteBuffer[pos=0
>> lim=4 cap=4]
>> at
>> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
>> at
>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
>> 2009-11-18 12:42:02,182 WARN org.apache.zookeeper.ClientCnxn: Ignoring
>> exception during shutdown input
>> java.net.SocketException: Transport endpoint is not connected
>> at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
>> at
>> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
>> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>> at
>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
>> at
>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
>> 2009-11-18 12:42:02,182 WARN org.apache.zookeeper.ClientCnxn: Ignoring
>> exception during shutdown output
>> java.net.SocketException: Transport endpoint is not connected
>>
>>
>> On Mon, Nov 16, 2009 at 4:16 PM, Zhenyu Zhong <zhongresearch@gmail.com
>> >wrote:
>>
>> > Here is the diskIO and CPU around the time we had RS disconnection on
>> one
>> > machine that runs RegionServer. It doesn't seem to be high. Similar disk
>> and
>> > cpu usage have been seen before and the HBase was running fine.
>> >
>> >
>> > So far I haven't found why my 10 minutes session timeout doesn't apply.
>> > Still digging.
>> >
>> > Many thanks!
>> >
>> > Time: 03:01:08 PM
>> > avg-cpu: %user %nice %system %iowait %steal %idle
>> > 3.82 0.00 0.89 3.97 0.00 91.32
>> >
>> >
>> > Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
>> > sda 24.30 1605.39 6679.35 6238623388 25956251720
>> > sda1 0.51 1.94 15.71 7547265 61067904
>> > sda2 0.13 2.11 5.20 8202079 20189232
>> > sda3 0.13 0.25 1.63 987846 6323336
>> > sda4 0.00 0.00 0.00 4 0
>> > sda5 1.78 0.06 24.14 220010 93817208
>> > sda6 0.38 0.42 8.41 1630584 32688152
>> > sda7 21.38 1600.61 6624.26 6220035272 25742165888
>> > sdb 4.52 767.33 565.91 2981868690 2199132380
>> > sdb1 4.52 767.33 565.91 2981866755 2199132372
>> > sdc 4.42 742.95 563.00 2887151482 2187823092
>> > sdc1 4.42 742.95 563.00 2887149547 2187823084
>> > sdd 4.49 750.78 557.25 2917554074 2165513500
>> > sdd1 4.49 750.78 557.25 2917552139 2165513492
>> > sde 4.52 758.51 569.46 2947593394 2212964236
>> > sde1 4.52 758.51 569.46 2947591459 2212964228
>> > sdf 4.51 747.22 571.78 2903740266 2221972652
>> > sdf1 4.51 747.22 571.78 2903738331 2221972644
>> >
>> > Best,
>> > zhenyu
>> >
>> >
>> >
>> > On Mon, Nov 16, 2009 at 4:07 PM, stack <st...@duboce.net> wrote:
>> >
>> >> On Mon, Nov 16, 2009 at 12:05 PM, Zhenyu Zhong <
>> zhongresearch@gmail.com
>> >> >wrote:
>> >>
>> >> > I just realized that there was a MapReduce job running during the
>> time
>> >> the
>> >> > regionserver disconnected from the zookeeper.
>> >> > That MapReduce Job was processing 500GB data and took about 8 minutes
>> to
>> >> > finish. It launched over 2000 map tasks.
>> >>
>> >>
>> >> There was a tasktracker running on the RegionServer that disconnected?
>> Is
>> >> the map i/o or cpu heavy? Do you think it could have stole life from
>> the
>> >> datanode/regionserver?
>> >>
>> >>
>> >>
>> >> > I doubt that this introduced
>> >> > resource contention between DataNode and RegionServer Node.
>> >> > Also during the time that MapReduce job ran, I saw a few errors
>> >> indicating
>> >> > that
>> >> >
>> >> > java.io.IOException: Could not obtain block:
>> >> > blk_7661247556283580226_4450462 file=/data/xxx.txt.200910141159
>> >> > at
>> >> >
>> >>
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1787)
>> >> > at
>> >> >
>> >>
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1615)
>> >> > at
>> >> >
>> >>
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1742)
>> >> > at java.io.DataInputStream.read(DataInputStream.java:83)
>> >> > at
>> >> org.apache.hadoop.util.LineReader.readLine(LineReader.java:134)
>> >> > at
>> >> >
>> >>
>> org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:136)
>> >> > at
>> >> >
>> org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:40)
>> >> > at
>> >> >
>> >>
>> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:192)
>> >> > at
>> >> >
>> >>
>> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:176)
>> >> > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
>> >> > at
>> >> org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
>> >> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
>> >> > at org.apache.hadoop.mapred.Child.main(Child.java:170)
>> >> >
>> >>
>> >> This is interesting. Your map task is running into hdfs-127. Yeah,
>> you
>> >> should patch your cluster if you want to get rid of these (hdfs-127 has
>> >> been
>> >> applied to 0.21 hadoop and will be in the next hadoop release on 0.20.x
>> >> branch, hadoop 0.20.2).
>> >>
>> >>
>> >>
>> >> >
>> >> > Possibly it is related to HDFS-127, but I don't see any datanodes
>> went
>> >> > down. Does that affect Regionserver? Shall we apply the patch?
>> >> >
>> >> >
>> >> Read up on hdfs-127 for explaination of whats going on.
>> >>
>> >>
>> >>
>> >> > Now I start to keep tracking of the virtual memory stats to see if
>> the
>> >> same
>> >> > issue happens tomorrow around the same time period.
>> >> >
>> >>
>> >> Excellent. Thanks for doing this.
>> >>
>> >>
>> >>
>> >> > And I highly suspect that this particular MapReduce job hurts HBase
>> >> > Regionserver.
>> >> >
>> >> > PS. I also use gcviewer to parse the GC-log, I only see around 30
>> >> seconds
>> >> > pauses maximum.
>> >> >
>> >>
>> >>
>> >> Did you figure why your ten minute session is not applying?
>> >>
>> >> St.Ack
>> >>
>> >>
>> >>
>> >> >
>> >> > Thanks a lot.
>> >> > zhenyu
>> >> >
>> >> >
>> >> >
>> >> > On Sun, Nov 15, 2009 at 12:52 AM, Zhenyu Zhong <
>> zhongresearch@gmail.com
>> >> > >wrote:
>> >> >
>> >> > > J-D,
>> >> > >
>> >> > > Thank you very much for your comments.
>> >> > > My company block the IRC port, so I have trouble to connect to IRC
>> >> > channel.
>> >> > > I have been trying to ask the IT to open the IRC port for me, it
>> might
>> >> > take
>> >> > > a while.
>> >> > >
>> >> > > Best,
>> >> > > zhenyu
>> >> > >
>> >> > >
>> >> > > On Sat, Nov 14, 2009 at 2:21 PM, Jean-Daniel Cryans <
>> >> jdcryans@apache.org
>> >> > >wrote:
>> >> > >
>> >> > >> The error you are getting is a disconnection from a zookeeper
>> server
>> >> > >> and is very generic.
>> >> > >>
>> >> > >> ZK-86 is still opened and the last comment refers to ZK-111 saying
>> >> > >> that the bug (in unit tests) was probably fixed in release 3.0.0
>> last
>> >> > >> year.
>> >> > >>
>> >> > >> To figure the hang you have, you can try to jstack the process pid
>> >> and
>> >> > >> see exactly what's holding the RS from shutting down.
>> >> > >>
>> >> > >> Would it be possible for you to drop by the IRC channel? This way
>> we
>> >> > >> can debug this a at much faster pace.
>> >> > >>
>> >> > >> Thx!
>> >> > >>
>> >> > >> J-D
>> >> > >>
>> >> > >> On Sat, Nov 14, 2009 at 9:57 AM, Zhenyu Zhong <
>> >> zhongresearch@gmail.com>
>> >> > >> wrote:
>> >> > >> > I found this.
>> >> > >> > http://issues.apache.org/jira/browse/ZOOKEEPER-86
>> >> > >> >
>> >> > >> > It looks like the same error I had. Is it a zookeeper bug? When
>> >> will
>> >> > >> HBase
>> >> > >> > take the zookeeper version 3.3.0?
>> >> > >> >
>> >> > >> > thanks
>> >> > >> > zhenyu
>> >> > >> >
>> >> > >> > On Sat, Nov 14, 2009 at 11:31 AM, Zhenyu Zhong <
>> >> > zhongresearch@gmail.com
>> >> > >> >wrote:
>> >> > >> >
>> >> > >> >> Now I really doubt about the zookeeper, from the log I saw
>> errors
>> >> > like
>> >> > >> >> IOException Read Error, while the zookeeper client
>> (regionserver )
>> >> > >> tried to
>> >> > >> >> read. But it got disconneted status from zookeeper.
>> >> > >> >>
>> >> > >> >> I don't see any load on any zookeeper quorum servers. DiskIO is
>> >> fine.
>> >> > >> >>
>> >> > >> >> Also when the regionserver decides to exit due to the
>> disconnect
>> >> > status
>> >> > >> >> from zookeeper, sometimes the regionserver hangs during the
>> >> exiting.
>> >> > We
>> >> > >> can
>> >> > >> >> still see the HRegionServer process even we don't see it in the
>> >> > Master
>> >> > >> web
>> >> > >> >> interface.
>> >> > >> >>
>> >> > >> >> I also notice that there is an zookeeper.retries and
>> >> zookeeper.pauses
>> >> > >> >> settings in hbase-default.xml, would that matter ?
>> >> > >> >>
>> >> > >> >> thanks
>> >> > >> >>
>> >> > >> >> You may see the IOException error in the following log when the
>> >> > >> >> Regionserver lost connection to the zookeeper.
>> >> > >> >>
>> >> > >> >> 2009-11-14 15:58:31,993 INFO
>> >> > org.apache.hadoop.hbase.regionserver.HLog:
>> >> > >> >> Roll
>> >> > >> >>
>> >> > >>
>> >> >
>> >>
>> /hbase/.logs/superpyxis0008.scur.colo,60021,1258179101617/hlog.dat.1258179106921,
>> >> > >> >> entries=1062964, calcsize=255014936, filesize=139566093. New
>> hlog
>> >> > >> >>
>> >> > >>
>> >> >
>> >>
>> /hbase/.logs/superpyxis0008.scur.colo,60021,1258179101617/hlog.dat.1258214311993
>> >> > >> >> 2009-11-14 15:58:39,295 DEBUG org.apache.zookeeper.ClientCnxn:
>> Got
>> >> > ping
>> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
>> >> > >> >> 2009-11-14 15:58:52,648 DEBUG org.apache.zookeeper.ClientCnxn:
>> Got
>> >> > ping
>> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
>> >> > >> >> 2009-11-14 15:59:06,007 DEBUG org.apache.zookeeper.ClientCnxn:
>> Got
>> >> > ping
>> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
>> >> > >> >> 2009-11-14 15:59:19,365 DEBUG org.apache.zookeeper.ClientCnxn:
>> Got
>> >> > ping
>> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
>> >> > >> >> 2009-11-14 15:59:32,867 DEBUG org.apache.zookeeper.ClientCnxn:
>> Got
>> >> > ping
>> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 186ms
>> >> > >> >> 2009-11-14 15:59:46,070 DEBUG org.apache.zookeeper.ClientCnxn:
>> Got
>> >> > ping
>> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
>> >> > >> >> 2009-11-14 15:59:59,423 DEBUG org.apache.zookeeper.ClientCnxn:
>> Got
>> >> > ping
>> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
>> >> > >> >> 2009-11-14 16:00:12,775 DEBUG org.apache.zookeeper.ClientCnxn:
>> Got
>> >> > ping
>> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
>> >> > >> >> 2009-11-14 16:00:28,141 DEBUG org.apache.zookeeper.ClientCnxn:
>> Got
>> >> > ping
>> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 2010ms
>> >> > >> >> 2009-11-14 16:01:10,378 WARN org.apache.zookeeper.ClientCnxn:
>> >> > Exception
>> >> > >> >> closing session 0x424f0ed01ea00a6 to
>> >> > >> sun.nio.ch.SelectionKeyImpl@39fbb2d6
>> >> > >> >>
>> >> > >> >> java.io.IOException: Read error rc = -1
>> >> > java.nio.DirectByteBuffer[pos=0
>> >> > >> >> lim=4 cap=4]
>> >> > >> >> at
>> >> > >> >>
>> >> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
>> >> > >> >> at
>> >> > >> >>
>> >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
>> >> > >> >> 2009-11-14 16:01:10,478 INFO
>> >> > >> >> org.apache.hadoop.hbase.regionserver.HRegionServer: Got
>> ZooKeeper
>> >> > >> event,
>> >> > >> >> state: Disconnected, type: None, path: null
>> >> > >> >> 2009-11-14 16:01:11,333 WARN
>> >> > >> >> org.apache.hadoop.hbase.regionserver.HRegionServer: Attempt=1
>> >> > >> >>
>> >> > >> >> org.apache.hadoop.hbase.Leases$LeaseStillHeldException
>> >> > >> >> at
>> >> > >> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>> >> > >> >> Method)
>> >> > >> >> at
>> >> > >> >>
>> >> > >>
>> >> >
>> >>
>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>> >> > >> >> at
>> >> > >> >>
>> >> > >>
>> >> >
>> >>
>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>> >> > >> >> at
>> >> > >> java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>> >> > >> >> at
>> >> > >> >>
>> >> > >>
>> >> >
>> >>
>> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
>> >> > >> >> at
>> >> > >> >>
>> >> > >>
>> >> >
>> >>
>> org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
>> >> > >> >> at
>> >> > >> >>
>> >> > >>
>> >> >
>> >>
>> org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
>> >> > >> >> at
>> >> > >> >>
>> >> > >>
>> >> >
>> >>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:571)
>> >> > >> >> at java.lang.Thread.run(Thread.java:619)
>> >> > >> >> 2009-11-14 16:01:11,433 INFO org.apache.zookeeper.ClientCnxn:
>> >> > >> Attempting
>> >> > >> >> connection to server superpyxis0001/192.168.100.115:2181
>> >> > >> >> 2009-11-14 16:01:11,433 INFO org.apache.zookeeper.ClientCnxn:
>> >> Priming
>> >> > >> >> connection to java.nio.channels.SocketChannel[connected local=/
>> >> > >> >> 192.168.100.122:59575 remote=superpyxis0001/
>> 192.168.100.115:2181]
>> >> > >> >> 2009-11-14 16:01:11,433 INFO org.apache.zookeeper.ClientCnxn:
>> >> Server
>> >> > >> >> connection successful
>> >> > >> >> 2009-11-14 16:01:11,433 INFO
>> >> > >> >> org.apache.hadoop.hbase.regionserver.HRegionServer: Got
>> ZooKeeper
>> >> > >> event,
>> >> > >> >> state: Expired, type: None, path: null
>> >> > >> >> 2009-11-14 16:01:11,433 WARN org.apache.zookeeper.ClientCnxn:
>> >> > Exception
>> >> > >> >> closing session 0x424f0ed01ea00a6 to
>> >> > >> sun.nio.ch.SelectionKeyImpl@6b93d343
>> >> > >> >> java.io.IOException: Session Expired
>> >> > >> >> at
>> >> > >> >>
>> >> > >>
>> >> >
>> >>
>> org.apache.zookeeper.ClientCnxn$SendThread.readConnectResult(ClientCnxn.java:589)
>> >> > >> >> at
>> >> > >> >>
>> >> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:709)
>> >> > >> >>
>> >> > >> >> at
>> >> > >> >>
>> >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
>> >> > >> >> 2009-11-14 16:01:11,433 ERROR
>> >> > >> >> org.apache.hadoop.hbase.regionserver.HRegionServer: ZooKeeper
>> >> session
>> >> > >> >> expired
>> >> > >> >> 2009-11-14 16:01:13,513 INFO org.apache.hadoop.ipc.HBaseServer:
>> >> IPC
>> >> > >> Server
>> >> > >> >> handler 3 on 60021, call put([B@73deb204,
>> >> > >> >> [Lorg.apache.hadoop.hbase.client.Put;@2179600a) from
>> >> > >> 192.168.100.132:40728:
>> >> > >> >> error: java.io.IOException: Server not running, aborting
>> >> > >> >> java.io.IOException: Server not running, aborting
>> >> > >> >>
>> >> > >> >>
>> >> > >> >>
>> >> > >> >>
>> >> > >> >>
>> >> > >> >> On Sat, Nov 14, 2009 at 1:01 AM, Zhenyu Zhong <
>> >> > zhongresearch@gmail.com
>> >> > >> >wrote:
>> >> > >> >>
>> >> > >> >>> Stack,
>> >> > >> >>>
>> >> > >> >>> Thanks a lot!
>> >> > >> >>>
>> >> > >> >>> I found out that the reason why HBase doesn't take the system
>> >> file
>> >> > >> >>> descriptor value I set before. I started the HBase using root
>> >> > instead
>> >> > >> of the
>> >> > >> >>> normal hadoop user, while my system configures hadoop with
>> higher
>> >> > file
>> >> > >> >>> descriptor value but configures root with 1024 default value.
>> >> > >> >>>
>> >> > >> >>>
>> >> > >> >>> Now my system has a clean start. However, it seems that the
>> >> > >> >>> zookeeper.session.timeout value doesn't take into effect. I
>> still
>> >> > >> found
>> >> > >> >>> around 60 seconds pauses from the disconnected regionserver. I
>> >> > really
>> >> > >> don't
>> >> > >> >>> know why regionserver only times out after 60 seconds instead
>> of
>> >> 10
>> >> > >> minutes
>> >> > >> >>> which I set for zookeeper.session.timeout.
>> >> > >> >>>
>> >> > >> >>> Is there any other timeout value coming into play before the
>> >> actual
>> >> > >> >>> session times out?
>> >> > >> >>>
>> >> > >> >>> zhenyu
>> >> > >> >>>
>> >> > >> >>>
>> >> > >> >>>
>> >> > >> >>> On Fri, Nov 13, 2009 at 7:08 PM, stack <st...@duboce.net>
>> wrote:
>> >> > >> >>>
>> >> > >> >>>> Ok. Lack of file descriptors manifests in all kinds of weird
>> >> ways.
>> >> > >> >>>> Hopefully thats it. If not, lets keep digging.
>> >> > >> >>>> St.Ack
>> >> > >> >>>>
>> >> > >> >>>> On Fri, Nov 13, 2009 at 3:44 PM, Zhenyu Zhong <
>> >> > >> zhongresearch@gmail.com
>> >> > >> >>>> >wrote:
>> >> > >> >>>>
>> >> > >> >>>> > Stack,
>> >> > >> >>>> >
>> >> > >> >>>> > You are right, the master started with ulimit -n 1024. It
>> >> doesn't
>> >> > >> take
>> >> > >> >>>> the
>> >> > >> >>>> > system value.
>> >> > >> >>>> >
>> >> > >> >>>> > Regarding to the too many open files, it looks like the
>> same
>> >> as
>> >> > the
>> >> > >> one
>> >> > >> >>>> J-D
>> >> > >> >>>> > put up. But I will get the Master start with higher value
>> >> first
>> >> > and
>> >> > >> see
>> >> > >> >>>> if
>> >> > >> >>>> > this kind of error goes away.
>> >> > >> >>>> >
>> >> > >> >>>> > thanks a lot!
>> >> > >> >>>> > zhenyu
>> >> > >> >>>> >
>> >> > >> >>>> > On Fri, Nov 13, 2009 at 6:02 PM, stack <st...@duboce.net>
>> >> wrote:
>> >> > >> >>>> >
>> >> > >> >>>> > > Does it say
>> >> > >> >>>> > >
>> >> > >> >>>> > > ulimit -n 32768
>> >> > >> >>>> > >
>> >> > >> >>>> > > ...as the second line in your log file on start of the
>> >> master?
>> >> > >> >>>> > >
>> >> > >> >>>> > > Is the stack trace that complains about too many open
>> files
>> >> > same
>> >> > >> as
>> >> > >> >>>> the
>> >> > >> >>>> > one
>> >> > >> >>>> > > in the blog post J-D put up?
>> >> > >> >>>> > >
>> >> > >> >>>> > > St.Ack
>> >> > >> >>>> > >
>> >> > >> >>>> > >
>> >> > >> >>>> > > On Fri, Nov 13, 2009 at 1:37 PM, Zhenyu Zhong <
>> >> > >> >>>> zhongresearch@gmail.com
>> >> > >> >>>> > > >wrote:
>> >> > >> >>>> > >
>> >> > >> >>>> > > > The ulimit file descriptors was set to fs.file-max
>> >> =1578334,
>> >> > >> also
>> >> > >> >>>> in
>> >> > >> >>>> > the
>> >> > >> >>>> > > > limits.conf the value was set to 32768.
>> >> > >> >>>> > > > So these are way higher than the open descriptors for
>> the
>> >> > >> running
>> >> > >> >>>> > > > processes.
>> >> > >> >>>> > > >
>> >> > >> >>>> > > > thanks
>> >> > >> >>>> > > > zhenyu
>> >> > >> >>>> > > >
>> >> > >> >>>> > > >
>> >> > >> >>>> > > > On Fri, Nov 13, 2009 at 4:33 PM, Stack <
>> >> saint.ack@gmail.com>
>> >> > >> >>>> wrote:
>> >> > >> >>>> > > >
>> >> > >> >>>> > > > > You upped the ulimit file descriptors as per the
>> getting
>> >> > >> started
>> >> > >> >>>> doc?
>> >> > >> >>>> > > > >
>> >> > >> >>>> > > > >
>> >> > >> >>>> > > > >
>> >> > >> >>>> > > > > On Nov 13, 2009, at 1:26 PM, Zhenyu Zhong <
>> >> > >> >>>> zhongresearch@gmail.com>
>> >> > >> >>>> > > > wrote:
>> >> > >> >>>> > > > >
>> >> > >> >>>> > > > > Thanks a lot.
>> >> > >> >>>> > > > >>
>> >> > >> >>>> > > > >>
>> >> > >> >>>> > > > >> Bad news is my kernel is still 2.6.26.
>> >> > >> >>>> > > > >> But it was not a problem before.
>> >> > >> >>>> > > > >>
>> >> > >> >>>> > > > >> Very strange.
>> >> > >> >>>> > > > >>
>> >> > >> >>>> > > > >> zhenyu
>> >> > >> >>>> > > > >>
>> >> > >> >>>> > > > >> On Fri, Nov 13, 2009 at 4:16 PM, Jean-Daniel Cryans
>> <
>> >> > >> >>>> > > > jdcryans@apache.org
>> >> > >> >>>> > > > >> >wrote:
>> >> > >> >>>> > > > >>
>> >> > >> >>>> > > > >> Looks like
>> >> > >> >>>> > > > >>>
>> >> > >> >>>> > > > >>>
>> >> > >> >>>> > > >
>> >> > >> >>>> > >
>> >> > >> >>>> >
>> >> > >> >>>>
>> >> > >>
>> >> >
>> >>
>> http://pero.blogs.aprilmayjune.org/2009/01/22/hadoop-and-linux-kernel-2627-epoll-limits/
>> >> > >> >>>> > > > >>>
>> >> > >> >>>> > > > >>> J-D
>> >> > >> >>>> > > > >>>
>> >> > >> >>>> > > > >>> On Fri, Nov 13, 2009 at 1:12 PM, Zhenyu Zhong <
>> >> > >> >>>> > > zhongresearch@gmail.com
>> >> > >> >>>> > > > >
>> >> > >> >>>> > > > >>> wrote:
>> >> > >> >>>> > > > >>>
>> >> > >> >>>> > > > >>>> Hi,
>> >> > >> >>>> > > > >>>>
>> >> > >> >>>> > > > >>>>
>> >> > >> >>>> > > > >>>> After I re-organize the cluster, the experiment
>> ran
>> >> into
>> >> > >> >>>> problem
>> >> > >> >>>> > > > faster
>> >> > >> >>>> > > > >>>>
>> >> > >> >>>> > > > >>> than
>> >> > >> >>>> > > > >>>
>> >> > >> >>>> > > > >>>> before.
>> >> > >> >>>> > > > >>>>
>> >> > >> >>>> > > > >>>> Basically the changes are to use machines with
>> less
>> >> > >> resources
>> >> > >> >>>> as
>> >> > >> >>>> > > > >>>>
>> >> > >> >>>> > > > >>> zookeeper
>> >> > >> >>>> > > > >>>
>> >> > >> >>>> > > > >>>> quorums and machines with more resources as
>> >> > regionserver.
>> >> > >> >>>> > > > >>>>
>> >> > >> >>>> > > > >>>> From the log, I still see the pause around 1
>> minute.
>> >> > >> >>>> > > > >>>> I enabled the GC logging, please see
>> >> > >> >>>> > > > >>>>
>> >> > >> >>>> > > > >>>> http://pastebin.com/m1d4ce0f1
>> >> > >> >>>> > > > >>>>
>> >> > >> >>>> > > > >>>> for details.
>> >> > >> >>>> > > > >>>> In general I don't see many pauses in the GC.
>> >> > >> >>>> > > > >>>>
>> >> > >> >>>> > > > >>>> What is more interesting, 1 hour after the 1st
>> >> > >> regionserver
>> >> > >> >>>> > > > >>>> disconnected,
>> >> > >> >>>> > > > >>>> the master log complained about too many open
>> files.
>> >> > This
>> >> > >> >>>> didn't
>> >> > >> >>>> > > > happen
>> >> > >> >>>> > > > >>>> before.
>> >> > >> >>>> > > > >>>> I checked the system OS setup as well as the
>> >> > limits.conf.
>> >> > >> I
>> >> > >> >>>> also
>> >> > >> >>>> > > > checked
>> >> > >> >>>> > > > >>>>
>> >> > >> >>>> > > > >>> the
>> >> > >> >>>> > > > >>>
>> >> > >> >>>> > > > >>>> running processes. The total open files don't
>> reach
>> >> the
>> >> > >> limit.
>> >> > >> >>>> I
>> >> > >> >>>> > am
>> >> > >> >>>> > > > >>>>
>> >> > >> >>>> > > > >>> confused
>> >> > >> >>>> > > > >>>
>> >> > >> >>>> > > > >>>> a bit.
>> >> > >> >>>> > > > >>>>
>> >> > >> >>>> > > > >>>>
>> >> > >> >>>> > > > >>>> Please see the following master log.
>> >> > >> >>>> > > > >>>>
>> >> > >> >>>> > > > >>>> 2009-11-13 20:06:02,114 INFO
>> >> > >> >>>> > > > org.apache.hadoop.hbase.master.BaseScanner:
>> >> > >> >>>> > > > >>>> RegionManager.metaScanner scan of 4658 row(s) of
>> meta
>> >> > >> region
>> >> > >> >>>> > > {server:
>> >> > >> >>>> > > > >>>> 192.168.100.128:60021, regionname: .META.,,1,
>> >> startKey:
>> >> > >> <>}
>> >> > >> >>>> > > complete
>> >> > >> >>>> > > > >>>> 2009-11-13 20:06:02,114 INFO
>> >> > >> >>>> > > > org.apache.hadoop.hbase.master.BaseScanner:
>> >> > >> >>>> > > > >>>>
>> >> > >> >>>> > > > >>> All
>> >> > >> >>>> > > > >>>
>> >> > >> >>>> > > > >>>> 1 .META. region(s) scanned
>> >> > >> >>>> > > > >>>> 2009-11-13 20:06:07,677 DEBUG
>> >> > >> org.apache.zookeeper.ClientCnxn:
>> >> > >> >>>> Got
>> >> > >> >>>> > > > ping
>> >> > >> >>>> > > > >>>> response for sessionid:0x424eebf1c10004c after 3ms
>> >> > >> >>>> > > > >>>> 2009-11-13 20:06:08,178 INFO
>> >> > >> org.apache.hadoop.hdfs.DFSClient:
>> >> > >> >>>> > > > Exception
>> >> > >> >>>> > > > >>>>
>> >> > >> >>>> > > > >>> in
>> >> > >> >>>> > > > >>>
>> >> > >> >>>> > > > >>>> createBlockOutputStream java.io.IOException: Bad
>> >> connect
>> >> > >> ack
>> >> > >> >>>> with
>> >> > >> >>>> > > > >>>> firstBadLink 192.168.100.123:50010
>> >> > >> >>>> > > > >>>> 2009-11-13 20:06:08,178 INFO
>> >> > >> org.apache.hadoop.hdfs.DFSClient:
>> >> > >> >>>> > > > >>>> Abandoning
>> >> > >> >>>> > > > >>>> block blk_-2808245019291145247_5478039
>> >> > >> >>>> > > > >>>> 2009-11-13 20:06:09,682 INFO
>> >> > >> org.apache.hadoop.hdfs.DFSClient:
>> >> > >> >>>> > > > Exception
>> >> > >> >>>> > > > >>>>
>> >> > >> >>>> > > > >>> in
>> >> > >> >>>> > > > >>>
>> >> > >> >>>> > > > >>>> createBlockOutputStream java.io.EOFException
>> >> > >> >>>> > > > >>>> 2009-11-13 20:06:09,682 INFO
>> >> > >> org.apache.hadoop.hdfs.DFSClient:
>> >> > >> >>>> > > > >>>> Abandoning
>> >> > >> >>>> > > > >>>> block blk_1074853606841896259_5478048
>> >> > >> >>>> > > > >>>> 2009-11-13 20:06:10,334 DEBUG
>> >> > >> org.apache.zookeeper.ClientCnxn:
>> >> > >> >>>> Got
>> >> > >> >>>> > > > ping
>> >> > >> >>>> > > > >>>> response for sessionid:0x24eebf1043003c after 1ms
>> >> > >> >>>> > > > >>>> 2009-11-13 20:06:21,018 DEBUG
>> >> > >> org.apache.zookeeper.ClientCnxn:
>> >> > >> >>>> Got
>> >> > >> >>>> > > > ping
>> >> > >> >>>> > > > >>>> response for sessionid:0x424eebf1c10004c after 0ms
>> >> > >> >>>> > > > >>>> 2009-11-13 20:06:23,674 DEBUG
>> >> > >> org.apache.zookeeper.ClientCnxn:
>> >> > >> >>>> Got
>> >> > >> >>>> > > > ping
>> >> > >> >>>> > > > >>>> response for sessionid:0x24eebf1043003c after 0ms
>> >> > >> >>>> > > > >>>> 2009-11-13 20:06:24,828 INFO
>> >> > >> org.apache.hadoop.hdfs.DFSClient:
>> >> > >> >>>> > > > Exception
>> >> > >> >>>> > > > >>>>
>> >> > >> >>>> > > > >>> in
>> >> > >> >>>> > > > >>>
>> >> > >> >>>> > > > >>>> createBlockOutputStream java.io.IOException: Bad
>> >> connect
>> >> > >> ack
>> >> > >> >>>> with
>> >> > >> >>>> > > > >>>> firstBadLink 192.168.100.123:50010
>> >> > >> >>>> > > > >>>> 2009-11-13 20:06:24,828 INFO
>> >> > >> org.apache.hadoop.hdfs.DFSClient:
>> >> > >> >>>> > > > >>>> Abandoning
>> >> > >> >>>> > > > >>>> block blk_-6642544517082142289_5478063
>> >> > >> >>>> > > > >>>> 2009-11-13 20:06:24,828 INFO
>> >> > >> org.apache.hadoop.hdfs.DFSClient:
>> >> > >> >>>> > > > Exception
>> >> > >> >>>> > > > >>>>
>> >> > >> >>>> > > > >>> in
>> >> > >> >>>> > > > >>>
>> >> > >> >>>> > > > >>>> createBlockOutputStream java.net.SocketException:
>> Too
>> >> > many
>> >> > >> >>>> open
>> >> > >> >>>> > > files
>> >> > >> >>>> > > > >>>> 2009-11-13 20:06:24,828 INFO
>> >> > >> org.apache.hadoop.hdfs.DFSClient:
>> >> > >> >>>> > > > >>>> Abandoning
>> >> > >> >>>> > > > >>>> block blk_2057511041109796090_5478063
>> >> > >> >>>> > > > >>>> 2009-11-13 20:06:24,928 INFO
>> >> > >> org.apache.hadoop.hdfs.DFSClient:
>> >> > >> >>>> > > > Exception
>> >> > >> >>>> > > > >>>>
>> >> > >> >>>> > > > >>> in
>> >> > >> >>>> > > > >>>
>> >> > >> >>>> > > > >>>> createBlockOutputStream java.net.SocketException:
>> Too
>> >> > many
>> >> > >> >>>> open
>> >> > >> >>>> > > files
>> >> > >> >>>> > > > >>>> 2009-11-13 20:06:24,928 INFO
>> >> > >> org.apache.hadoop.hdfs.DFSClient:
>> >> > >> >>>> > > > >>>> Abandoning
>> >> > >> >>>> > > > >>>> block blk_8219260302213892894_5478064
>> >> > >> >>>> > > > >>>> 2009-11-13 20:06:30,855 INFO
>> >> > >> org.apache.hadoop.hdfs.DFSClient:
>> >> > >> >>>> > > > Exception
>> >> > >> >>>> > > > >>>>
>> >> > >> >>>> > > > >>> in
>> >> > >> >>>> > > > >>>
>> >> > >> >>>> > > > >>>> createBlockOutputStream java.net.SocketException:
>> Too
>> >> > many
>> >> > >> >>>> open
>> >> > >> >>>> > > files
>> >> > >> >>>> > > > >>>> 2009-11-13 20:06:30,855 INFO
>> >> > >> org.apache.hadoop.hdfs.DFSClient:
>> >> > >> >>>> > > > >>>> Abandoning
>> >> > >> >>>> > > > >>>> block blk_1669205542853067709_5478235
>> >> > >> >>>> > > > >>>> 2009-11-13 20:06:30,905 INFO
>> >> > >> org.apache.hadoop.hdfs.DFSClient:
>> >> > >> >>>> > > > Exception
>> >> > >> >>>> > > > >>>>
>> >> > >> >>>> > > > >>> in
>> >> > >> >>>> > > > >>>
>> >> > >> >>>> > > > >>>> createBlockOutputStream java.net.SocketException:
>> Too
>> >> > many
>> >> > >> >>>> open
>> >> > >> >>>> > > files
>> >> > >> >>>> > > > >>>> 2009-11-13 20:06:30,905 INFO
>> >> > >> org.apache.hadoop.hdfs.DFSClient:
>> >> > >> >>>> > > > >>>> Abandoning
>> >> > >> >>>> > > > >>>> block blk_9128897691346270351_5478237
>> >> > >> >>>> > > > >>>> 2009-11-13 20:06:30,955 INFO
>> >> > >> org.apache.hadoop.hdfs.DFSClient:
>> >> > >> >>>> > > > Exception
>> >> > >> >>>> > > > >>>>
>> >> > >> >>>> > > > >>> in
>> >> > >> >>>> > > > >>>
>> >> > >> >>>> > > > >>>> createBlockOutputStream java.net.SocketException:
>> Too
>> >> > many
>> >> > >> >>>> open
>> >> > >> >>>> > > files
>> >> > >> >>>> > > > >>>> 2009-11-13 20:06:30,955 INFO
>> >> > >> org.apache.hadoop.hdfs.DFSClient:
>> >> > >> >>>> > > > >>>> Abandoning
>> >> > >> >>>> > > > >>>> block blk_1116845144864123018_5478240
>> >> > >> >>>> > > > >>>> 2009-11-13 20:06:34,372 DEBUG
>> >> > >> org.apache.zookeeper.ClientCnxn:
>> >> > >> >>>> Got
>> >> > >> >>>> > > > ping
>> >> > >> >>>> > > > >>>> response for sessionid:0x424eebf1c10004c after 0ms
>> >> > >> >>>> > > > >>>> 2009-11-13 20:06:37,034 DEBUG
>> >> > >> org.apache.zookeeper.ClientCnxn:
>> >> > >> >>>> Got
>> >> > >> >>>> > > > ping
>> >> > >> >>>> > > > >>>> response for sessionid:0x24eebf1043003c after 0ms
>> >> > >> >>>> > > > >>>> 2009-11-13 20:06:37,235 WARN
>> >> > >> org.apache.hadoop.hdfs.DFSClient:
>> >> > >> >>>> > > > >>>>
>> >> > >> >>>> > > > >>> DataStreamer
>> >> > >> >>>> > > > >>>
>> >> > >> >>>> > > > >>>> Exception: java.io.IOException: Too many open
>> files
>> >> > >> >>>> > > > >>>> at sun.nio.ch.IOUtil.initPipe(Native Method)
>> >> > >> >>>> > > > >>>> at
>> >> > >> >>>> > >
>> >> sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:49)
>> >> > >> >>>> > > > >>>> at
>> >> > >> >>>> > > > >>>>
>> >> > >> >>>> > > > >>>>
>> >> > >> >>>> > > >
>> >> > >> >>>> > >
>> >> > >> >>>> >
>> >> > >> >>>>
>> >> > >>
>> >> >
>> >>
>> sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:18)
>> >> > >> >>>> > > > >>>
>> >> > >> >>>> > > > >>>> at
>> >> > >> >>>> > > > >>>>
>> >> > >> >>>> > > > >>>>
>> >> > >> >>>> > > >
>> >> > >> >>>> > >
>> >> > >> >>>> >
>> >> > >> >>>>
>> >> > >>
>> >> >
>> >>
>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.get(SocketIOWithTimeout.java:407)
>> >> > >> >>>> > > > >>>
>> >> > >> >>>> > > > >>>> at
>> >> > >> >>>> > > > >>>>
>> >> > >> >>>> > > > >>>>
>> >> > >> >>>> > > >
>> >> > >> >>>> > >
>> >> > >> >>>> >
>> >> > >> >>>>
>> >> > >>
>> >> >
>> >>
>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:322)
>> >> > >> >>>> > > > >>>
>> >> > >> >>>> > > > >>>> at
>> >> > >> >>>> > > > >>>>
>> >> > >> >>>> > > > >>>>
>> >> > >> >>>> > > >
>> >> > >> >>>> > >
>> >> > >> >>>> >
>> >> > >> >>>>
>> >> > >>
>> >> >
>> >>
>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
>> >> > >> >>>> > > > >>>
>> >> > >> >>>> > > > >>>> at
>> >> > >> >>>> > > > >>>>
>> >> > >> >>>> > > > >>>>
>> >> > >> >>>> > > >
>> >> > >> >>>> > >
>> >> > >> >>>> >
>> >> > >> >>>>
>> >> > >>
>> >> >
>> >>
>> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
>> >> > >> >>>> > > > >>>
>> >> > >> >>>> > > > >>>> at
>> >> > >> >>>> > > > >>>>
>> >> > >> >>>> > > > >>>>
>> >> > >> >>>> > > >
>> >> > >> >>>> > >
>> >> > >> >>>> >
>> >> > >> >>>>
>> >> > >>
>> >> >
>> >>
>> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
>> >> > >> >>>> > > > >>>
>> >> > >> >>>> > > > >>>> at
>> >> > >> >>>> > > > >>>>
>> >> > >> >>>> > > > >>>
>> >> > >> >>>>
>> >> java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
>> >> > >> >>>> > > > >>>
>> >> > >> >>>> > > > >>>> at
>> >> > >> >>>> java.io.DataOutputStream.write(DataOutputStream.java:90)
>> >> > >> >>>> > > > >>>> at
>> >> > >> >>>> > > > >>>>
>> >> > >> >>>> > > > >>>>
>> >> > >> >>>> > > >
>> >> > >> >>>> > >
>> >> > >> >>>> >
>> >> > >> >>>>
>> >> > >>
>> >> >
>> >>
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2290)
>> >> > >> >>>> > > > >>>
>> >> > >> >>>> > > > >>>>
>> >> > >> >>>> > > > >>>> 2009-11-13 20:06:37,235 WARN
>> >> > >> org.apache.hadoop.hdfs.DFSClient:
>> >> > >> >>>> > Error
>> >> > >> >>>> > > > >>>> Recovery for block blk_8148813491785406356_5478475
>> >> bad
>> >> > >> >>>> datanode[0]
>> >> > >> >>>> > > > >>>> 192.168.100.123:50010
>> >> > >> >>>> > > > >>>> 2009-11-13 20:06:37,235 WARN
>> >> > >> org.apache.hadoop.hdfs.DFSClient:
>> >> > >> >>>> > Error
>> >> > >> >>>> > > > >>>> Recovery for block blk_8148813491785406356_5478475
>> in
>> >> > >> pipeline
>> >> > >> >>>> > > > >>>> 192.168.100.123:50010, 192.168.100.134:50010,
>> >> > >> >>>> > 192.168.100.122:50010
>> >> > >> >>>> > > :
>> >> > >> >>>> > > > >>>> bad
>> >> > >> >>>> > > > >>>> datanode 192.168.100.123:50010
>> >> > >> >>>> > > > >>>> 2009-11-13 20:06:37,436 INFO
>> >> > >> org.apache.hadoop.hdfs.DFSClient:
>> >> > >> >>>> > > > Exception
>> >> > >> >>>> > > > >>>>
>> >> > >> >>>> > > > >>> in
>> >> > >> >>>> > > > >>>
>> >> > >> >>>> > > > >>>> createBlockOutputStream java.net.SocketException:
>> Too
>> >> > many
>> >> > >> >>>> open
>> >> > >> >>>> > > files
>> >> > >> >>>> > > > >>>> 2009-11-13 20:06:37,436 INFO
>> >> > >> org.apache.hadoop.hdfs.DFSClient:
>> >> > >> >>>> > > > >>>> Abandoning
>> >> > >> >>>> > > > >>>> block blk_2119727700857186236_5478498
>> >> > >> >>>> > > > >>>>
>> >> > >> >>>> > > > >>>>
>> >> > >> >>>> > > > >>>>
>> >> > >> >>>> > > > >>>>
>> >> > >> >>>> > > > >>>>
>> >> > >> >>>> > > > >>>> On Thu, Nov 12, 2009 at 4:21 PM, Zhenyu Zhong <
>> >> > >> >>>> > > > zhongresearch@gmail.com
>> >> > >> >>>> > > > >>>> wrote:
>> >> > >> >>>> > > > >>>>
>> >> > >> >>>> > > > >>>> Will do.
>> >> > >> >>>> > > > >>>>>
>> >> > >> >>>> > > > >>>>> thanks
>> >> > >> >>>> > > > >>>>> zhenyu
>> >> > >> >>>> > > > >>>>>
>> >> > >> >>>> > > > >>>>>
>> >> > >> >>>> > > > >>>>> On Thu, Nov 12, 2009 at 3:33 PM, stack <
>> >> > stack@duboce.net
>> >> > >> >
>> >> > >> >>>> wrote:
>> >> > >> >>>> > > > >>>>>
>> >> > >> >>>> > > > >>>>> Enable GC logging too on this next run (see
>> >> > >> hbase-env.sh).
>> >> > >> >>>> Lets
>> >> > >> >>>> > > try
>> >> > >> >>>> > > > >>>>>>
>> >> > >> >>>> > > > >>>>> and
>> >> > >> >>>> > > > >>>
>> >> > >> >>>> > > > >>>> get
>> >> > >> >>>> > > > >>>>>> to the bottom of whats going on.
>> >> > >> >>>> > > > >>>>>> Thanks,
>> >> > >> >>>> > > > >>>>>> St.Ack
>> >> > >> >>>> > > > >>>>>>
>> >> > >> >>>> > > > >>>>>> On Thu, Nov 12, 2009 at 12:29 PM, Zhenyu Zhong <
>> >> > >> >>>> > > > >>>>>>
>> >> > >> >>>> > > > >>>>> zhongresearch@gmail.com
>> >> > >> >>>> > > > >>>
>> >> > >> >>>> > > > >>>> wrote:
>> >> > >> >>>> > > > >>>>>>>
>> >> > >> >>>> > > > >>>>>>
>> >> > >> >>>> > > > >>>>>> I can switch the boxes that run zookeeper with
>> the
>> >> > ones
>> >> > >> >>>> that
>> >> > >> >>>> > run
>> >> > >> >>>> > > > >>>>>>> regionservers.
>> >> > >> >>>> > > > >>>>>>> I will see the results later.
>> >> > >> >>>> > > > >>>>>>>
>> >> > >> >>>> > > > >>>>>>> FYI. The node does have the 10 minutes
>> >> > >> >>>> > zookeeper.session.timeout
>> >> > >> >>>> > > > >>>>>>>
>> >> > >> >>>> > > > >>>>>> value
>> >> > >> >>>> > > > >>>
>> >> > >> >>>> > > > >>>> in
>> >> > >> >>>> > > > >>>>>>
>> >> > >> >>>> > > > >>>>>>> place.
>> >> > >> >>>> > > > >>>>>>>
>> >> > >> >>>> > > > >>>>>>> thanks
>> >> > >> >>>> > > > >>>>>>> zhenyu
>> >> > >> >>>> > > > >>>>>>>
>> >> > >> >>>> > > > >>>>>>> On Thu, Nov 12, 2009 at 3:21 PM, stack <
>> >> > >> stack@duboce.net>
>> >> > >> >>>> > wrote:
>> >> > >> >>>> > > > >>>>>>>
>> >> > >> >>>> > > > >>>>>>> On Thu, Nov 12, 2009 at 11:50 AM, Zhenyu Zhong
>> <
>> >> > >> >>>> > > > >>>>>>>>
>> >> > >> >>>> > > > >>>>>>> zhongresearch@gmail.com
>> >> > >> >>>> > > > >>>>>>
>> >> > >> >>>> > > > >>>>>>> wrote:
>> >> > >> >>>> > > > >>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>> In my cluster, half of the cluster have 2
>> disks
>> >> > 400GB
>> >> > >> >>>> each
>> >> > >> >>>> > per
>> >> > >> >>>> > > > >>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>> machine,
>> >> > >> >>>> > > > >>>>>>
>> >> > >> >>>> > > > >>>>>>> and
>> >> > >> >>>> > > > >>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>> half of the cluster have 6 disks per machine.
>> >> > Maybe
>> >> > >> we
>> >> > >> >>>> > should
>> >> > >> >>>> > > > >>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>> run
>> >> > >> >>>> > > > >>>
>> >> > >> >>>> > > > >>>> zookeeper
>> >> > >> >>>> > > > >>>>>>>>> on the machines with 2 disks and RS on
>> machines
>> >> > with
>> >> > >> 6
>> >> > >> >>>> disks?
>> >> > >> >>>> > > > >>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>> That would make most sense only in the below,
>> it
>> >> > >> looks
>> >> > >> >>>> like
>> >> > >> >>>> > the
>> >> > >> >>>> > > > >>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>> RS
>> >> > >> >>>> > > > >>>
>> >> > >> >>>> > > > >>>> that
>> >> > >> >>>> > > > >>>>>>
>> >> > >> >>>> > > > >>>>>>> had
>> >> > >> >>>> > > > >>>>>>>> issue had 4 disks?
>> >> > >> >>>> > > > >>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>
>> >> > >> >>>> > > > >>>>>>>> BTW, the 10 minutes zookeeper.session.timeout
>> >> has
>> >> > >> been
>> >> > >> >>>> set
>> >> > >> >>>> > > during
>> >> > >> >>>> > > > >>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>> the
>> >> > >> >>>> > > > >>>>>>
>> >> > >> >>>> > > > >>>>>>> experiment.
>> >> > >> >>>> > > > >>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>> And for sure this node had it in place?
>> >> > >> >>>> > > > >>>>>>>> St.Ack
>> >> > >> >>>> > > > >>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>> thanks
>> >> > >> >>>> > > > >>>>>>>>> zhenyu
>> >> > >> >>>> > > > >>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>> On Thu, Nov 12, 2009 at 2:08 PM, stack <
>> >> > >> stack@duboce.net
>> >> > >> >>>> >
>> >> > >> >>>> > > wrote:
>> >> > >> >>>> > > > >>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>> On Thu, Nov 12, 2009 at 8:40 AM, Zhenyu
>> Zhong <
>> >> > >> >>>> > > > >>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>> zhongresearch@gmail.com
>> >> > >> >>>> > > > >>>>>>>
>> >> > >> >>>> > > > >>>>>>>> wrote:
>> >> > >> >>>> > > > >>>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>> Though I experienced 2 regionserver
>> >> disconnection
>> >> > >> this
>> >> > >> >>>> > > > >>>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>> morning,
>> >> > >> >>>> > > > >>>
>> >> > >> >>>> > > > >>>> it
>> >> > >> >>>> > > > >>>>>>
>> >> > >> >>>> > > > >>>>>>> looks
>> >> > >> >>>> > > > >>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>> better from the regionserver log. (please
>> see
>> >> the
>> >> > >> >>>> following
>> >> > >> >>>> > > > >>>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>> log)
>> >> > >> >>>> > > > >>>
>> >> > >> >>>> > > > >>>>
>> >> > >> >>>> > > > >>>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>>> http://pastebin.com/m496dbfae
>> >> > >> >>>> > > > >>>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>>> I checked diskIO for one of the
>> >> > >> >>>> > regionserver(192.168.100.116)
>> >> > >> >>>> > > > >>>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>> during
>> >> > >> >>>> > > > >>>>>>>
>> >> > >> >>>> > > > >>>>>>>> the
>> >> > >> >>>> > > > >>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>> time it disconnected.
>> >> > >> >>>> > > > >>>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>>> Time: 03:04:58 AM
>> >> > >> >>>> > > > >>>>>>>>>>> Device: tps Blk_read/s
>> >> Blk_wrtn/s
>> >> > >> >>>> Blk_read
>> >> > >> >>>> > > > >>>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>> Blk_wrtn
>> >> > >> >>>> > > > >>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>> sda 105.31 5458.83
>> 19503.64
>> >> > >> >>>> 9043873239
>> >> > >> >>>> > > > >>>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>> 32312473676
>> >> > >> >>>> > > > >>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>> sda1 2.90 6.64
>> 99.25
>> >> > >> >>>> 10993934
>> >> > >> >>>> > > > >>>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>> 164433464
>> >> > >> >>>> > > > >>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>> sda2 1.72 23.76
>> 30.25
>> >> > >> >>>> 39365817
>> >> > >> >>>> > > > >>>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>> 50124008
>> >> > >> >>>> > > > >>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>> sda3 0.30 0.38
>> 3.58
>> >> > >> >>>> 624930
>> >> > >> >>>> > > > >>>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>> 5923000
>> >> > >> >>>> > > > >>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>> sda4 100.39 5428.06
>> 19370.56
>> >> > >> >>>> 8992888270
>> >> > >> >>>> > > > >>>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>> 32091993204
>> >> > >> >>>> > > > >>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>>> Is this high for you? 20k blocks/second
>> >> would
>> >> > >> seem to
>> >> > >> >>>> be
>> >> > >> >>>> > > high
>> >> > >> >>>> > > > >>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>> but
>> >> > >> >>>> > > > >>>
>> >> > >> >>>> > > > >>>> its
>> >> > >> >>>> > > > >>>>>>>
>> >> > >> >>>> > > > >>>>>>>> one
>> >> > >> >>>> > > > >>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>> disk only and its not being shared by zk
>> >> anymore
>> >> > so
>> >> > >> >>>> > shouldn't
>> >> > >> >>>> > > > >>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>> matter?
>> >> > >> >>>> > > > >>>>>>
>> >> > >> >>>> > > > >>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>> I also checked the zookeeper quorum server
>> >> that
>> >> > the
>> >> > >> >>>> > > > >>>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>> regionserver
>> >> > >> >>>> > > > >>>
>> >> > >> >>>> > > > >>>> tried
>> >> > >> >>>> > > > >>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>> to
>> >> > >> >>>> > > > >>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>> connect according to the log. However, I
>> don't
>> >> see
>> >> > >> >>>> > > > >>>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>> 192.168.100.116
>> >> > >> >>>> > > > >>>>>>
>> >> > >> >>>> > > > >>>>>>> in
>> >> > >> >>>> > > > >>>>>>>
>> >> > >> >>>> > > > >>>>>>>> the
>> >> > >> >>>> > > > >>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>> client list of the zookeeper quorum server
>> that
>> >> > the
>> >> > >> >>>> > > > >>>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>> regionserver
>> >> > >> >>>> > > > >>>
>> >> > >> >>>> > > > >>>> tried
>> >> > >> >>>> > > > >>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>> to
>> >> > >> >>>> > > > >>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>> connect.
>> >> > >> >>>> > > > >>>>>>>>>>> Would that be a problem?
>> >> > >> >>>> > > > >>>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>>> Is that because the ephemeral node for the
>> >> > >> >>>> regionserver
>> >> > >> >>>> > had
>> >> > >> >>>> > > > >>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>> evaporated?
>> >> > >> >>>> > > > >>>>>>>
>> >> > >> >>>> > > > >>>>>>>> Lost
>> >> > >> >>>> > > > >>>>>>>>>> its lease w/ zk by the time you went to
>> look?
>> >> > >> >>>> > > > >>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>> Thu Nov 12 15:04:52 UTC 2009
>> >> > >> >>>> > > > >>>>>>>>>>> Zookeeper version: 3.2.1-808558, built on
>> >> > >> 08/27/2009
>> >> > >> >>>> 18:48
>> >> > >> >>>> > > > >>>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>> GMT
>> >> > >> >>>> > > > >>>
>> >> > >> >>>> > > > >>>> Clients:
>> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.127:43045
>> >> > >> [1](queued=0,recved=26,sent=0)
>> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.131:39091
>> >> > >> [1](queued=0,recved=964,sent=0)
>> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.124:35961
>> >> > >> [1](queued=0,recved=3266,sent=0)
>> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.123:47935
>> >> > >> [1](queued=0,recved=233,sent=0)
>> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.125:46931
>> >> > >> [1](queued=0,recved=2,sent=0)
>> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.118:54924
>> >> > >> [1](queued=0,recved=295,sent=0)
>> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.118:41390
>> >> > >> [1](queued=0,recved=2290,sent=0)
>> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.136:42243
>> >> > >> [1](queued=0,recved=0,sent=0)
>> >> > >> >>>> > > > >>>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>>> Latency min/avg/max: 0/17/6333
>> >> > >> >>>> > > > >>>>>>>>>>> Received: 47111
>> >> > >> >>>> > > > >>>>>>>>>>> Sent: 0
>> >> > >> >>>> > > > >>>>>>>>>>> Outstanding: 0
>> >> > >> >>>> > > > >>>>>>>>>>> Zxid: 0x77000083f4
>> >> > >> >>>> > > > >>>>>>>>>>> Mode: leader
>> >> > >> >>>> > > > >>>>>>>>>>> Node count: 23
>> >> > >> >>>> > > > >>>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>> That 6 second maximum latency is pretty bad
>> but
>> >> > >> should
>> >> > >> >>>> be
>> >> > >> >>>> > well
>> >> > >> >>>> > > > >>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>> within
>> >> > >> >>>> > > > >>>>>>
>> >> > >> >>>> > > > >>>>>>> the
>> >> > >> >>>> > > > >>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>> zk
>> >> > >> >>>> > > > >>>>>>>>>> session timeout.
>> >> > >> >>>> > > > >>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>> So, problem is likely on the zk client side
>> of
>> >> the
>> >> > >> >>>> session;
>> >> > >> >>>> > > > >>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>> i.e.
>> >> > >> >>>> > > > >>>
>> >> > >> >>>> > > > >>>> in
>> >> > >> >>>> > > > >>>>>>
>> >> > >> >>>> > > > >>>>>>> the
>> >> > >> >>>> > > > >>>>>>>
>> >> > >> >>>> > > > >>>>>>>> RS.
>> >> > >> >>>> > > > >>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>> You could enable GC logging as J-D suggested
>> to
>> >> > see
>> >> > >> if
>> >> > >> >>>> you
>> >> > >> >>>> > > have
>> >> > >> >>>> > > > >>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>> any
>> >> > >> >>>> > > > >>>>>>
>> >> > >> >>>> > > > >>>>>>> big
>> >> > >> >>>> > > > >>>>>>>
>> >> > >> >>>> > > > >>>>>>>> pauses, pauses > zk session timeout.
>> >> > >> >>>> > > > >>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>> When the RS went down, it didn't look too
>> >> heavily
>> >> > >> >>>> loaded:
>> >> > >> >>>> > > > >>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>> 1. 2009-11-12 15:04:52,830 INFO
>> >> > >> >>>> > > > >>>>>>>>>>
>> >> > org.apache.hadoop.hbase.regionserver.HRegionServer:
>> >> > >> >>>> Dump of
>> >> > >> >>>> > > > >>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>> metrics:
>> >> > >> >>>> > > > >>>>>>>
>> >> > >> >>>> > > > >>>>>>>> request=1.5166667, regions=322, stores=657,
>> >> > >> >>>> storefiles=631,
>> >> > >> >>>> > > > >>>>>>>>>> storefileIndexSize=61, memstoreSize=1472,
>> >> > >> >>>> usedHeap=2819,
>> >> > >> >>>> > > > >>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>> maxHeap=4079,
>> >> > >> >>>> > > > >>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>> blockCacheSize=658110960,
>> >> > blockCacheFree=197395984,
>> >> > >> >>>> > > > >>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>> blockCacheCount=9903,
>> >> > >> >>>> > > > >>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>> blockCacheHitRatio=99
>> >> > >> >>>> > > > >>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>> Its serving a few reads? The number of
>> store
>> >> > files
>> >> > >> >>>> seems
>> >> > >> >>>> > > fine.
>> >> > >> >>>> > > > >>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>> Not
>> >> > >> >>>> > > > >>>>>>
>> >> > >> >>>> > > > >>>>>>> too
>> >> > >> >>>> > > > >>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>> much memory used.
>> >> > >> >>>> > > > >>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>> Looking at the logs, I see the Lease Still
>> Held
>> >> > >> >>>> exception.
>> >> > >> >>>> > > > >>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>> This
>> >> > >> >>>> > > > >>>
>> >> > >> >>>> > > > >>>> happens
>> >> > >> >>>> > > > >>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>> when the RS does its regular report to the
>> >> master
>> >> > but
>> >> > >> the
>> >> > >> >>>> > > > >>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>> master
>> >> > >> >>>> > > > >>>
>> >> > >> >>>> > > > >>>> thinks
>> >> > >> >>>> > > > >>>>>>>
>> >> > >> >>>> > > > >>>>>>>> the
>> >> > >> >>>> > > > >>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>> RS has since restarted. It'll think this
>> >> probably
>> >> > >> >>>> because
>> >> > >> >>>> > it
>> >> > >> >>>> > > > >>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>> noticed
>> >> > >> >>>> > > > >>>>>>
>> >> > >> >>>> > > > >>>>>>> that
>> >> > >> >>>> > > > >>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>> the RS's znode in zk had gone away and it
>> >> > considered
>> >> > >> the
>> >> > >> >>>> RS
>> >> > >> >>>> > > > >>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>> dead.
>> >> > >> >>>> > > > >>>
>> >> > >> >>>> > > > >>>>
>> >> > >> >>>> > > > >>>>>>>>>> Looking too at your logs I see this gap in
>> the
>> >> zk
>> >> > >> >>>> pinging:
>> >> > >> >>>> > > > >>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>> 1. 2009-11-12 15:03:39,325 DEBUG
>> >> > >> >>>> > > > >>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>> org.apache.zookeeper.ClientCnxn:
>> >> > >> >>>> > > > >>>>>>
>> >> > >> >>>> > > > >>>>>>> Got
>> >> > >> >>>> > > > >>>>>>>
>> >> > >> >>>> > > > >>>>>>>> ping response for sessionid:0x224e55436ad0004
>> >> after
>> >> > >> 0ms
>> >> > >> >>>> > > > >>>>>>>>>> 2. 2009-11-12 15:03:43,113 DEBUG
>> >> > >> >>>> > > > >>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>> org.apache.zookeeper.ClientCnxn:
>> >> > >> >>>> > > > >>>>>>
>> >> > >> >>>> > > > >>>>>>> Got
>> >> > >> >>>> > > > >>>>>>>
>> >> > >> >>>> > > > >>>>>>>> ping response for sessionid:0x24e55436a0007d
>> >> after
>> >> > >> 0ms
>> >> > >> >>>> > > > >>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>> Where in the lines above it, its reporting
>> >> about
>> >> > >> every
>> >> > >> >>>> ten
>> >> > >> >>>> > > > >>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>> seconds,
>> >> > >> >>>> > > > >>>>>>
>> >> > >> >>>> > > > >>>>>>> here
>> >> > >> >>>> > > > >>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>> there is a big gap.
>> >> > >> >>>> > > > >>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>> Do you have ganglia or something that will
>> let
>> >> you
>> >> > >> look
>> >> > >> >>>> more
>> >> > >> >>>> > > > >>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>> into
>> >> > >> >>>> > > > >>>
>> >> > >> >>>> > > > >>>> what
>> >> > >> >>>> > > > >>>>>>>
>> >> > >> >>>> > > > >>>>>>>> was
>> >> > >> >>>> > > > >>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>> happening on this machine around this time?
>> Is
>> >> > the
>> >> > >> >>>> machine
>> >> > >> >>>> > > OK?
>> >> > >> >>>> > > > >>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>> It
>> >> > >> >>>> > > > >>>>>>
>> >> > >> >>>> > > > >>>>>>> looks
>> >> > >> >>>> > > > >>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>> lightly loaded and you have your cluster
>> nicely
>> >> > laid
>> >> > >> out.
>> >> > >> >>>> > > > >>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>> Something
>> >> > >> >>>> > > > >>>>>>
>> >> > >> >>>> > > > >>>>>>> odd
>> >> > >> >>>> > > > >>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>> is
>> >> > >> >>>> > > > >>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>> going on. What about things like the write
>> >> speed
>> >> > to
>> >> > >> >>>> disk?
>> >> > >> >>>> > In
>> >> > >> >>>> > > > >>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>> the
>> >> > >> >>>> > > > >>>
>> >> > >> >>>> > > > >>>> past
>> >> > >> >>>> > > > >>>>>>>
>> >> > >> >>>> > > > >>>>>>>> strange issues have been explained by
>> incorrectly
>> >> > set
>> >> > >> BIOS
>> >> > >> >>>> > > > >>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>> which
>> >> > >> >>>> > > > >>>
>> >> > >> >>>> > > > >>>> made
>> >> > >> >>>> > > > >>>>>>
>> >> > >> >>>> > > > >>>>>>> disks
>> >> > >> >>>> > > > >>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>> run at 1/100th of their proper speed.
>> >> > >> >>>> > > > >>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>> St.Ack
>> >> > >> >>>> > > > >>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>>> Best,
>> >> > >> >>>> > > > >>>>>>>>>>> zhenyu
>> >> > >> >>>> > > > >>>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>>> On Wed, Nov 11, 2009 at 3:58 PM, Zhenyu
>> Zhong
>> >> <
>> >> > >> >>>> > > > >>>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>> zhongresearch@gmail.com
>> >> > >> >>>> > > > >>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>> wrote:
>> >> > >> >>>> > > > >>>>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>>> Stack
>> >> > >> >>>> > > > >>>>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>>>> I am very appreciated for your comments.
>> >> > >> >>>> > > > >>>>>>>>>>>> I will use the zookeeper monitoring script
>> on
>> >> my
>> >> > >> >>>> cluster
>> >> > >> >>>> > > > >>>>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>>> and
>> >> > >> >>>> > > > >>>
>> >> > >> >>>> > > > >>>> let
>> >> > >> >>>> > > > >>>>>>
>> >> > >> >>>> > > > >>>>>>> it
>> >> > >> >>>> > > > >>>>>>>
>> >> > >> >>>> > > > >>>>>>>> run
>> >> > >> >>>> > > > >>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>> overnight to see the result.
>> >> > >> >>>> > > > >>>>>>>>>>>> I will follow up the thread when I get
>> >> anything.
>> >> > >> >>>> > > > >>>>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>>>> thanks
>> >> > >> >>>> > > > >>>>>>>>>>>> zhenyu
>> >> > >> >>>> > > > >>>>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>>>> On Wed, Nov 11, 2009 at 3:52 PM, stack <
>> >> > >> >>>> stack@duboce.net>
>> >> > >> >>>> > > > >>>>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>>> wrote:
>> >> > >> >>>> > > > >>>>>>
>> >> > >> >>>> > > > >>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>>>> I see these in your log too:
>> >> > >> >>>> > > > >>>>>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>>>>> 1. 2009-11-11 04:27:19,018 DEBUG
>> >> > >> >>>> > > > >>>>>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
>> >> > >> >>>> > > > >>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>> Got
>> >> > >> >>>> > > > >>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>>> ping response for
>> sessionid:0x424dfd908c50009
>> >> > >> after
>> >> > >> >>>> > > > >>>>>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>>>> 4544ms
>> >> > >> >>>> > > > >>>
>> >> > >> >>>> > > > >>>> 2. 2009-11-11 04:27:19,018 DEBUG
>> >> > >> >>>> > > > >>>>>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
>> >> > >> >>>> > > > >>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>> Got
>> >> > >> >>>> > > > >>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>>> ping response for
>> sessionid:0x24dfd90c810002
>> >> > after
>> >> > >> >>>> > > > >>>>>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>>>> 4548ms
>> >> > >> >>>> > > > >>>
>> >> > >> >>>> > > > >>>> 3. 2009-11-11 04:27:43,960 DEBUG
>> >> > >> >>>> > > > >>>>>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
>> >> > >> >>>> > > > >>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>> Got
>> >> > >> >>>> > > > >>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>>> ping response for
>> sessionid:0x424dfd908c50009
>> >> > >> after
>> >> > >> >>>> > > > >>>>>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>>>> 9030ms
>> >> > >> >>>> > > > >>>
>> >> > >> >>>> > > > >>>> 4. 2009-11-11 04:27:43,960 DEBUG
>> >> > >> >>>> > > > >>>>>>>>>>>>>
>> >> > >> >>>> > > > >>>>>>>>>>>> org.apache.zookeeper.
>> >> > >> >>>> > > > >>>>>>>>
>> >> > >> >>>> > > > >>>>>>>
>> >> > >> >>>> > > >
>> >> > >> >>>> > >
>> >> > >> >>>> >
>> >> > >> >>>>
>> >> > >> >>>
>> >> > >> >>>
>> >> > >> >>
>> >> > >> >
>> >> > >>
>> >> > >
>> >> > >
>> >> >
>> >>
>> >
>> >
>>
>
>
Re: regionserver disconnection
Posted by stack <st...@duboce.net>.
And running stat on each of your zookeeper nodes tells you what about
averages for connect-times?
Its odd that there is a ping just 20 seconds beforehand and you have set the
zk session timeout at 60 seconds (or ten minutes?). Let me ask the zk lads
about it.
If you look in the GC log for around this time, do you see anything?
St.Ack
On Wed, Nov 18, 2009 at 7:32 AM, Zhenyu Zhong <zh...@gmail.com>wrote:
> After running more experiment, it seems that the disconnection is not
> related to the heavy load job, because the load average is low, the diskio
> is normal, memory Heap was not reached, also the virtual memory stats shows
> that no swappings. However, the disconnection still happens.
>
> It looks like this time it pauses for 20 seconds. No idea why regionserver
> disconnected.
>
> Any other suggestions please?
>
>
> 2009-11-18 12:41:00,956 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
> response for sessionid:0x125035c495d02bb after 0ms
> 2009-11-18 12:41:08,074 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
> response for sessionid:0x425035c48210002 after 0ms
> 2009-11-18 12:41:14,302 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
> response for sessionid:0x125035c495d02bb after 0ms
> 2009-11-18 12:41:21,433 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
> response for sessionid:0x425035c48210002 after 0ms
> 2009-11-18 12:41:27,656 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
> response for sessionid:0x125035c495d02bb after 0ms
> 2009-11-18 12:41:34,788 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
> response for sessionid:0x425035c48210002 after 0ms
> 2009-11-18 12:41:41,016 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
> response for sessionid:0x125035c495d02bb after 1ms
> 2009-11-18 12:42:01,482 WARN org.apache.zookeeper.ClientCnxn: Exception
> closing session 0x425035c48210002 to sun.nio.ch.SelectionKeyImpl@421690ab
> java.io.IOException: TIMED OUT
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:906)
> 2009-11-18 12:42:01,582 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
> state: Disconnected, type: None, path: null
> 2009-11-18 12:42:01,782 INFO org.apache.zookeeper.ClientCnxn: Attempting
> connection to server superpyxis0003/192.168.100.117:2181
> 2009-11-18 12:42:01,782 INFO org.apache.zookeeper.ClientCnxn: Priming
> connection to java.nio.channels.SocketChannel[connected local=/
> 192.168.100.132:46610 remote=superpyxis0003/192.168.100.117:2181]
> 2009-11-18 12:42:01,782 INFO org.apache.zookeeper.ClientCnxn: Server
> connection successful
> 2009-11-18 12:42:01,782 WARN org.apache.zookeeper.ClientCnxn: Exception
> closing session 0x425035c48210002 to sun.nio.ch.SelectionKeyImpl@56dc6fac
> java.io.IOException: Read error rc = -1 java.nio.DirectByteBuffer[pos=0
> lim=4 cap=4]
> at
> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
> 2009-11-18 12:42:01,782 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> exception during shutdown input
> java.net.SocketException: Transport endpoint is not connected
> at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2009-11-18 12:42:01,782 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> exception during shutdown output
> java.net.SocketException: Transport endpoint is not connected
> at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2009-11-18 12:42:02,182 INFO org.apache.zookeeper.ClientCnxn: Attempting
> connection to server superpyxis0001/192.168.100.115:2181
> 2009-11-18 12:42:02,182 INFO org.apache.zookeeper.ClientCnxn: Priming
> connection to java.nio.channels.SocketChannel[connected local=/
> 192.168.100.132:36197 remote=superpyxis0001/192.168.100.115:2181]
> 2009-11-18 12:42:02,182 INFO org.apache.zookeeper.ClientCnxn: Server
> connection successful
> 2009-11-18 12:42:02,182 WARN org.apache.zookeeper.ClientCnxn: Exception
> closing session 0x425035c48210002 to sun.nio.ch.SelectionKeyImpl@5c07076b
> java.io.IOException: Read error rc = -1 java.nio.DirectByteBuffer[pos=0
> lim=4 cap=4]
> at
> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
> 2009-11-18 12:42:02,182 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> exception during shutdown input
> java.net.SocketException: Transport endpoint is not connected
> at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2009-11-18 12:42:02,182 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> exception during shutdown output
> java.net.SocketException: Transport endpoint is not connected
>
>
> On Mon, Nov 16, 2009 at 4:16 PM, Zhenyu Zhong <zhongresearch@gmail.com
> >wrote:
>
> > Here is the diskIO and CPU around the time we had RS disconnection on one
> > machine that runs RegionServer. It doesn't seem to be high. Similar disk
> and
> > cpu usage have been seen before and the HBase was running fine.
> >
> >
> > So far I haven't found why my 10 minutes session timeout doesn't apply.
> > Still digging.
> >
> > Many thanks!
> >
> > Time: 03:01:08 PM
> > avg-cpu: %user %nice %system %iowait %steal %idle
> > 3.82 0.00 0.89 3.97 0.00 91.32
> >
> >
> > Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
> > sda 24.30 1605.39 6679.35 6238623388 25956251720
> > sda1 0.51 1.94 15.71 7547265 61067904
> > sda2 0.13 2.11 5.20 8202079 20189232
> > sda3 0.13 0.25 1.63 987846 6323336
> > sda4 0.00 0.00 0.00 4 0
> > sda5 1.78 0.06 24.14 220010 93817208
> > sda6 0.38 0.42 8.41 1630584 32688152
> > sda7 21.38 1600.61 6624.26 6220035272 25742165888
> > sdb 4.52 767.33 565.91 2981868690 2199132380
> > sdb1 4.52 767.33 565.91 2981866755 2199132372
> > sdc 4.42 742.95 563.00 2887151482 2187823092
> > sdc1 4.42 742.95 563.00 2887149547 2187823084
> > sdd 4.49 750.78 557.25 2917554074 2165513500
> > sdd1 4.49 750.78 557.25 2917552139 2165513492
> > sde 4.52 758.51 569.46 2947593394 2212964236
> > sde1 4.52 758.51 569.46 2947591459 2212964228
> > sdf 4.51 747.22 571.78 2903740266 2221972652
> > sdf1 4.51 747.22 571.78 2903738331 2221972644
> >
> > Best,
> > zhenyu
> >
> >
> >
> > On Mon, Nov 16, 2009 at 4:07 PM, stack <st...@duboce.net> wrote:
> >
> >> On Mon, Nov 16, 2009 at 12:05 PM, Zhenyu Zhong <zhongresearch@gmail.com
> >> >wrote:
> >>
> >> > I just realized that there was a MapReduce job running during the time
> >> the
> >> > regionserver disconnected from the zookeeper.
> >> > That MapReduce Job was processing 500GB data and took about 8 minutes
> to
> >> > finish. It launched over 2000 map tasks.
> >>
> >>
> >> There was a tasktracker running on the RegionServer that disconnected?
> Is
> >> the map i/o or cpu heavy? Do you think it could have stole life from
> the
> >> datanode/regionserver?
> >>
> >>
> >>
> >> > I doubt that this introduced
> >> > resource contention between DataNode and RegionServer Node.
> >> > Also during the time that MapReduce job ran, I saw a few errors
> >> indicating
> >> > that
> >> >
> >> > java.io.IOException: Could not obtain block:
> >> > blk_7661247556283580226_4450462 file=/data/xxx.txt.200910141159
> >> > at
> >> >
> >>
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1787)
> >> > at
> >> >
> >>
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1615)
> >> > at
> >> >
> >>
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1742)
> >> > at java.io.DataInputStream.read(DataInputStream.java:83)
> >> > at
> >> org.apache.hadoop.util.LineReader.readLine(LineReader.java:134)
> >> > at
> >> >
> >>
> org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:136)
> >> > at
> >> >
> org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:40)
> >> > at
> >> >
> >>
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:192)
> >> > at
> >> >
> >>
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:176)
> >> > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
> >> > at
> >> org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
> >> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
> >> > at org.apache.hadoop.mapred.Child.main(Child.java:170)
> >> >
> >>
> >> This is interesting. Your map task is running into hdfs-127. Yeah, you
> >> should patch your cluster if you want to get rid of these (hdfs-127 has
> >> been
> >> applied to 0.21 hadoop and will be in the next hadoop release on 0.20.x
> >> branch, hadoop 0.20.2).
> >>
> >>
> >>
> >> >
> >> > Possibly it is related to HDFS-127, but I don't see any datanodes went
> >> > down. Does that affect Regionserver? Shall we apply the patch?
> >> >
> >> >
> >> Read up on hdfs-127 for explaination of whats going on.
> >>
> >>
> >>
> >> > Now I start to keep tracking of the virtual memory stats to see if the
> >> same
> >> > issue happens tomorrow around the same time period.
> >> >
> >>
> >> Excellent. Thanks for doing this.
> >>
> >>
> >>
> >> > And I highly suspect that this particular MapReduce job hurts HBase
> >> > Regionserver.
> >> >
> >> > PS. I also use gcviewer to parse the GC-log, I only see around 30
> >> seconds
> >> > pauses maximum.
> >> >
> >>
> >>
> >> Did you figure why your ten minute session is not applying?
> >>
> >> St.Ack
> >>
> >>
> >>
> >> >
> >> > Thanks a lot.
> >> > zhenyu
> >> >
> >> >
> >> >
> >> > On Sun, Nov 15, 2009 at 12:52 AM, Zhenyu Zhong <
> zhongresearch@gmail.com
> >> > >wrote:
> >> >
> >> > > J-D,
> >> > >
> >> > > Thank you very much for your comments.
> >> > > My company block the IRC port, so I have trouble to connect to IRC
> >> > channel.
> >> > > I have been trying to ask the IT to open the IRC port for me, it
> might
> >> > take
> >> > > a while.
> >> > >
> >> > > Best,
> >> > > zhenyu
> >> > >
> >> > >
> >> > > On Sat, Nov 14, 2009 at 2:21 PM, Jean-Daniel Cryans <
> >> jdcryans@apache.org
> >> > >wrote:
> >> > >
> >> > >> The error you are getting is a disconnection from a zookeeper
> server
> >> > >> and is very generic.
> >> > >>
> >> > >> ZK-86 is still opened and the last comment refers to ZK-111 saying
> >> > >> that the bug (in unit tests) was probably fixed in release 3.0.0
> last
> >> > >> year.
> >> > >>
> >> > >> To figure the hang you have, you can try to jstack the process pid
> >> and
> >> > >> see exactly what's holding the RS from shutting down.
> >> > >>
> >> > >> Would it be possible for you to drop by the IRC channel? This way
> we
> >> > >> can debug this a at much faster pace.
> >> > >>
> >> > >> Thx!
> >> > >>
> >> > >> J-D
> >> > >>
> >> > >> On Sat, Nov 14, 2009 at 9:57 AM, Zhenyu Zhong <
> >> zhongresearch@gmail.com>
> >> > >> wrote:
> >> > >> > I found this.
> >> > >> > http://issues.apache.org/jira/browse/ZOOKEEPER-86
> >> > >> >
> >> > >> > It looks like the same error I had. Is it a zookeeper bug? When
> >> will
> >> > >> HBase
> >> > >> > take the zookeeper version 3.3.0?
> >> > >> >
> >> > >> > thanks
> >> > >> > zhenyu
> >> > >> >
> >> > >> > On Sat, Nov 14, 2009 at 11:31 AM, Zhenyu Zhong <
> >> > zhongresearch@gmail.com
> >> > >> >wrote:
> >> > >> >
> >> > >> >> Now I really doubt about the zookeeper, from the log I saw
> errors
> >> > like
> >> > >> >> IOException Read Error, while the zookeeper client (regionserver
> )
> >> > >> tried to
> >> > >> >> read. But it got disconneted status from zookeeper.
> >> > >> >>
> >> > >> >> I don't see any load on any zookeeper quorum servers. DiskIO is
> >> fine.
> >> > >> >>
> >> > >> >> Also when the regionserver decides to exit due to the disconnect
> >> > status
> >> > >> >> from zookeeper, sometimes the regionserver hangs during the
> >> exiting.
> >> > We
> >> > >> can
> >> > >> >> still see the HRegionServer process even we don't see it in the
> >> > Master
> >> > >> web
> >> > >> >> interface.
> >> > >> >>
> >> > >> >> I also notice that there is an zookeeper.retries and
> >> zookeeper.pauses
> >> > >> >> settings in hbase-default.xml, would that matter ?
> >> > >> >>
> >> > >> >> thanks
> >> > >> >>
> >> > >> >> You may see the IOException error in the following log when the
> >> > >> >> Regionserver lost connection to the zookeeper.
> >> > >> >>
> >> > >> >> 2009-11-14 15:58:31,993 INFO
> >> > org.apache.hadoop.hbase.regionserver.HLog:
> >> > >> >> Roll
> >> > >> >>
> >> > >>
> >> >
> >>
> /hbase/.logs/superpyxis0008.scur.colo,60021,1258179101617/hlog.dat.1258179106921,
> >> > >> >> entries=1062964, calcsize=255014936, filesize=139566093. New
> hlog
> >> > >> >>
> >> > >>
> >> >
> >>
> /hbase/.logs/superpyxis0008.scur.colo,60021,1258179101617/hlog.dat.1258214311993
> >> > >> >> 2009-11-14 15:58:39,295 DEBUG org.apache.zookeeper.ClientCnxn:
> Got
> >> > ping
> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
> >> > >> >> 2009-11-14 15:58:52,648 DEBUG org.apache.zookeeper.ClientCnxn:
> Got
> >> > ping
> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
> >> > >> >> 2009-11-14 15:59:06,007 DEBUG org.apache.zookeeper.ClientCnxn:
> Got
> >> > ping
> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
> >> > >> >> 2009-11-14 15:59:19,365 DEBUG org.apache.zookeeper.ClientCnxn:
> Got
> >> > ping
> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
> >> > >> >> 2009-11-14 15:59:32,867 DEBUG org.apache.zookeeper.ClientCnxn:
> Got
> >> > ping
> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 186ms
> >> > >> >> 2009-11-14 15:59:46,070 DEBUG org.apache.zookeeper.ClientCnxn:
> Got
> >> > ping
> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
> >> > >> >> 2009-11-14 15:59:59,423 DEBUG org.apache.zookeeper.ClientCnxn:
> Got
> >> > ping
> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
> >> > >> >> 2009-11-14 16:00:12,775 DEBUG org.apache.zookeeper.ClientCnxn:
> Got
> >> > ping
> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
> >> > >> >> 2009-11-14 16:00:28,141 DEBUG org.apache.zookeeper.ClientCnxn:
> Got
> >> > ping
> >> > >> >> response for sessionid:0x424f0ed01ea00a6 after 2010ms
> >> > >> >> 2009-11-14 16:01:10,378 WARN org.apache.zookeeper.ClientCnxn:
> >> > Exception
> >> > >> >> closing session 0x424f0ed01ea00a6 to
> >> > >> sun.nio.ch.SelectionKeyImpl@39fbb2d6
> >> > >> >>
> >> > >> >> java.io.IOException: Read error rc = -1
> >> > java.nio.DirectByteBuffer[pos=0
> >> > >> >> lim=4 cap=4]
> >> > >> >> at
> >> > >> >>
> >> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
> >> > >> >> at
> >> > >> >>
> >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
> >> > >> >> 2009-11-14 16:01:10,478 INFO
> >> > >> >> org.apache.hadoop.hbase.regionserver.HRegionServer: Got
> ZooKeeper
> >> > >> event,
> >> > >> >> state: Disconnected, type: None, path: null
> >> > >> >> 2009-11-14 16:01:11,333 WARN
> >> > >> >> org.apache.hadoop.hbase.regionserver.HRegionServer: Attempt=1
> >> > >> >>
> >> > >> >> org.apache.hadoop.hbase.Leases$LeaseStillHeldException
> >> > >> >> at
> >> > >> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> >> > >> >> Method)
> >> > >> >> at
> >> > >> >>
> >> > >>
> >> >
> >>
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> >> > >> >> at
> >> > >> >>
> >> > >>
> >> >
> >>
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> >> > >> >> at
> >> > >> java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> >> > >> >> at
> >> > >> >>
> >> > >>
> >> >
> >>
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
> >> > >> >> at
> >> > >> >>
> >> > >>
> >> >
> >>
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
> >> > >> >> at
> >> > >> >>
> >> > >>
> >> >
> >>
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
> >> > >> >> at
> >> > >> >>
> >> > >>
> >> >
> >>
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:571)
> >> > >> >> at java.lang.Thread.run(Thread.java:619)
> >> > >> >> 2009-11-14 16:01:11,433 INFO org.apache.zookeeper.ClientCnxn:
> >> > >> Attempting
> >> > >> >> connection to server superpyxis0001/192.168.100.115:2181
> >> > >> >> 2009-11-14 16:01:11,433 INFO org.apache.zookeeper.ClientCnxn:
> >> Priming
> >> > >> >> connection to java.nio.channels.SocketChannel[connected local=/
> >> > >> >> 192.168.100.122:59575 remote=superpyxis0001/
> 192.168.100.115:2181]
> >> > >> >> 2009-11-14 16:01:11,433 INFO org.apache.zookeeper.ClientCnxn:
> >> Server
> >> > >> >> connection successful
> >> > >> >> 2009-11-14 16:01:11,433 INFO
> >> > >> >> org.apache.hadoop.hbase.regionserver.HRegionServer: Got
> ZooKeeper
> >> > >> event,
> >> > >> >> state: Expired, type: None, path: null
> >> > >> >> 2009-11-14 16:01:11,433 WARN org.apache.zookeeper.ClientCnxn:
> >> > Exception
> >> > >> >> closing session 0x424f0ed01ea00a6 to
> >> > >> sun.nio.ch.SelectionKeyImpl@6b93d343
> >> > >> >> java.io.IOException: Session Expired
> >> > >> >> at
> >> > >> >>
> >> > >>
> >> >
> >>
> org.apache.zookeeper.ClientCnxn$SendThread.readConnectResult(ClientCnxn.java:589)
> >> > >> >> at
> >> > >> >>
> >> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:709)
> >> > >> >>
> >> > >> >> at
> >> > >> >>
> >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
> >> > >> >> 2009-11-14 16:01:11,433 ERROR
> >> > >> >> org.apache.hadoop.hbase.regionserver.HRegionServer: ZooKeeper
> >> session
> >> > >> >> expired
> >> > >> >> 2009-11-14 16:01:13,513 INFO org.apache.hadoop.ipc.HBaseServer:
> >> IPC
> >> > >> Server
> >> > >> >> handler 3 on 60021, call put([B@73deb204,
> >> > >> >> [Lorg.apache.hadoop.hbase.client.Put;@2179600a) from
> >> > >> 192.168.100.132:40728:
> >> > >> >> error: java.io.IOException: Server not running, aborting
> >> > >> >> java.io.IOException: Server not running, aborting
> >> > >> >>
> >> > >> >>
> >> > >> >>
> >> > >> >>
> >> > >> >>
> >> > >> >> On Sat, Nov 14, 2009 at 1:01 AM, Zhenyu Zhong <
> >> > zhongresearch@gmail.com
> >> > >> >wrote:
> >> > >> >>
> >> > >> >>> Stack,
> >> > >> >>>
> >> > >> >>> Thanks a lot!
> >> > >> >>>
> >> > >> >>> I found out that the reason why HBase doesn't take the system
> >> file
> >> > >> >>> descriptor value I set before. I started the HBase using root
> >> > instead
> >> > >> of the
> >> > >> >>> normal hadoop user, while my system configures hadoop with
> higher
> >> > file
> >> > >> >>> descriptor value but configures root with 1024 default value.
> >> > >> >>>
> >> > >> >>>
> >> > >> >>> Now my system has a clean start. However, it seems that the
> >> > >> >>> zookeeper.session.timeout value doesn't take into effect. I
> still
> >> > >> found
> >> > >> >>> around 60 seconds pauses from the disconnected regionserver. I
> >> > really
> >> > >> don't
> >> > >> >>> know why regionserver only times out after 60 seconds instead
> of
> >> 10
> >> > >> minutes
> >> > >> >>> which I set for zookeeper.session.timeout.
> >> > >> >>>
> >> > >> >>> Is there any other timeout value coming into play before the
> >> actual
> >> > >> >>> session times out?
> >> > >> >>>
> >> > >> >>> zhenyu
> >> > >> >>>
> >> > >> >>>
> >> > >> >>>
> >> > >> >>> On Fri, Nov 13, 2009 at 7:08 PM, stack <st...@duboce.net>
> wrote:
> >> > >> >>>
> >> > >> >>>> Ok. Lack of file descriptors manifests in all kinds of weird
> >> ways.
> >> > >> >>>> Hopefully thats it. If not, lets keep digging.
> >> > >> >>>> St.Ack
> >> > >> >>>>
> >> > >> >>>> On Fri, Nov 13, 2009 at 3:44 PM, Zhenyu Zhong <
> >> > >> zhongresearch@gmail.com
> >> > >> >>>> >wrote:
> >> > >> >>>>
> >> > >> >>>> > Stack,
> >> > >> >>>> >
> >> > >> >>>> > You are right, the master started with ulimit -n 1024. It
> >> doesn't
> >> > >> take
> >> > >> >>>> the
> >> > >> >>>> > system value.
> >> > >> >>>> >
> >> > >> >>>> > Regarding to the too many open files, it looks like the same
> >> as
> >> > the
> >> > >> one
> >> > >> >>>> J-D
> >> > >> >>>> > put up. But I will get the Master start with higher value
> >> first
> >> > and
> >> > >> see
> >> > >> >>>> if
> >> > >> >>>> > this kind of error goes away.
> >> > >> >>>> >
> >> > >> >>>> > thanks a lot!
> >> > >> >>>> > zhenyu
> >> > >> >>>> >
> >> > >> >>>> > On Fri, Nov 13, 2009 at 6:02 PM, stack <st...@duboce.net>
> >> wrote:
> >> > >> >>>> >
> >> > >> >>>> > > Does it say
> >> > >> >>>> > >
> >> > >> >>>> > > ulimit -n 32768
> >> > >> >>>> > >
> >> > >> >>>> > > ...as the second line in your log file on start of the
> >> master?
> >> > >> >>>> > >
> >> > >> >>>> > > Is the stack trace that complains about too many open
> files
> >> > same
> >> > >> as
> >> > >> >>>> the
> >> > >> >>>> > one
> >> > >> >>>> > > in the blog post J-D put up?
> >> > >> >>>> > >
> >> > >> >>>> > > St.Ack
> >> > >> >>>> > >
> >> > >> >>>> > >
> >> > >> >>>> > > On Fri, Nov 13, 2009 at 1:37 PM, Zhenyu Zhong <
> >> > >> >>>> zhongresearch@gmail.com
> >> > >> >>>> > > >wrote:
> >> > >> >>>> > >
> >> > >> >>>> > > > The ulimit file descriptors was set to fs.file-max
> >> =1578334,
> >> > >> also
> >> > >> >>>> in
> >> > >> >>>> > the
> >> > >> >>>> > > > limits.conf the value was set to 32768.
> >> > >> >>>> > > > So these are way higher than the open descriptors for
> the
> >> > >> running
> >> > >> >>>> > > > processes.
> >> > >> >>>> > > >
> >> > >> >>>> > > > thanks
> >> > >> >>>> > > > zhenyu
> >> > >> >>>> > > >
> >> > >> >>>> > > >
> >> > >> >>>> > > > On Fri, Nov 13, 2009 at 4:33 PM, Stack <
> >> saint.ack@gmail.com>
> >> > >> >>>> wrote:
> >> > >> >>>> > > >
> >> > >> >>>> > > > > You upped the ulimit file descriptors as per the
> getting
> >> > >> started
> >> > >> >>>> doc?
> >> > >> >>>> > > > >
> >> > >> >>>> > > > >
> >> > >> >>>> > > > >
> >> > >> >>>> > > > > On Nov 13, 2009, at 1:26 PM, Zhenyu Zhong <
> >> > >> >>>> zhongresearch@gmail.com>
> >> > >> >>>> > > > wrote:
> >> > >> >>>> > > > >
> >> > >> >>>> > > > > Thanks a lot.
> >> > >> >>>> > > > >>
> >> > >> >>>> > > > >>
> >> > >> >>>> > > > >> Bad news is my kernel is still 2.6.26.
> >> > >> >>>> > > > >> But it was not a problem before.
> >> > >> >>>> > > > >>
> >> > >> >>>> > > > >> Very strange.
> >> > >> >>>> > > > >>
> >> > >> >>>> > > > >> zhenyu
> >> > >> >>>> > > > >>
> >> > >> >>>> > > > >> On Fri, Nov 13, 2009 at 4:16 PM, Jean-Daniel Cryans <
> >> > >> >>>> > > > jdcryans@apache.org
> >> > >> >>>> > > > >> >wrote:
> >> > >> >>>> > > > >>
> >> > >> >>>> > > > >> Looks like
> >> > >> >>>> > > > >>>
> >> > >> >>>> > > > >>>
> >> > >> >>>> > > >
> >> > >> >>>> > >
> >> > >> >>>> >
> >> > >> >>>>
> >> > >>
> >> >
> >>
> http://pero.blogs.aprilmayjune.org/2009/01/22/hadoop-and-linux-kernel-2627-epoll-limits/
> >> > >> >>>> > > > >>>
> >> > >> >>>> > > > >>> J-D
> >> > >> >>>> > > > >>>
> >> > >> >>>> > > > >>> On Fri, Nov 13, 2009 at 1:12 PM, Zhenyu Zhong <
> >> > >> >>>> > > zhongresearch@gmail.com
> >> > >> >>>> > > > >
> >> > >> >>>> > > > >>> wrote:
> >> > >> >>>> > > > >>>
> >> > >> >>>> > > > >>>> Hi,
> >> > >> >>>> > > > >>>>
> >> > >> >>>> > > > >>>>
> >> > >> >>>> > > > >>>> After I re-organize the cluster, the experiment ran
> >> into
> >> > >> >>>> problem
> >> > >> >>>> > > > faster
> >> > >> >>>> > > > >>>>
> >> > >> >>>> > > > >>> than
> >> > >> >>>> > > > >>>
> >> > >> >>>> > > > >>>> before.
> >> > >> >>>> > > > >>>>
> >> > >> >>>> > > > >>>> Basically the changes are to use machines with less
> >> > >> resources
> >> > >> >>>> as
> >> > >> >>>> > > > >>>>
> >> > >> >>>> > > > >>> zookeeper
> >> > >> >>>> > > > >>>
> >> > >> >>>> > > > >>>> quorums and machines with more resources as
> >> > regionserver.
> >> > >> >>>> > > > >>>>
> >> > >> >>>> > > > >>>> From the log, I still see the pause around 1
> minute.
> >> > >> >>>> > > > >>>> I enabled the GC logging, please see
> >> > >> >>>> > > > >>>>
> >> > >> >>>> > > > >>>> http://pastebin.com/m1d4ce0f1
> >> > >> >>>> > > > >>>>
> >> > >> >>>> > > > >>>> for details.
> >> > >> >>>> > > > >>>> In general I don't see many pauses in the GC.
> >> > >> >>>> > > > >>>>
> >> > >> >>>> > > > >>>> What is more interesting, 1 hour after the 1st
> >> > >> regionserver
> >> > >> >>>> > > > >>>> disconnected,
> >> > >> >>>> > > > >>>> the master log complained about too many open
> files.
> >> > This
> >> > >> >>>> didn't
> >> > >> >>>> > > > happen
> >> > >> >>>> > > > >>>> before.
> >> > >> >>>> > > > >>>> I checked the system OS setup as well as the
> >> > limits.conf.
> >> > >> I
> >> > >> >>>> also
> >> > >> >>>> > > > checked
> >> > >> >>>> > > > >>>>
> >> > >> >>>> > > > >>> the
> >> > >> >>>> > > > >>>
> >> > >> >>>> > > > >>>> running processes. The total open files don't reach
> >> the
> >> > >> limit.
> >> > >> >>>> I
> >> > >> >>>> > am
> >> > >> >>>> > > > >>>>
> >> > >> >>>> > > > >>> confused
> >> > >> >>>> > > > >>>
> >> > >> >>>> > > > >>>> a bit.
> >> > >> >>>> > > > >>>>
> >> > >> >>>> > > > >>>>
> >> > >> >>>> > > > >>>> Please see the following master log.
> >> > >> >>>> > > > >>>>
> >> > >> >>>> > > > >>>> 2009-11-13 20:06:02,114 INFO
> >> > >> >>>> > > > org.apache.hadoop.hbase.master.BaseScanner:
> >> > >> >>>> > > > >>>> RegionManager.metaScanner scan of 4658 row(s) of
> meta
> >> > >> region
> >> > >> >>>> > > {server:
> >> > >> >>>> > > > >>>> 192.168.100.128:60021, regionname: .META.,,1,
> >> startKey:
> >> > >> <>}
> >> > >> >>>> > > complete
> >> > >> >>>> > > > >>>> 2009-11-13 20:06:02,114 INFO
> >> > >> >>>> > > > org.apache.hadoop.hbase.master.BaseScanner:
> >> > >> >>>> > > > >>>>
> >> > >> >>>> > > > >>> All
> >> > >> >>>> > > > >>>
> >> > >> >>>> > > > >>>> 1 .META. region(s) scanned
> >> > >> >>>> > > > >>>> 2009-11-13 20:06:07,677 DEBUG
> >> > >> org.apache.zookeeper.ClientCnxn:
> >> > >> >>>> Got
> >> > >> >>>> > > > ping
> >> > >> >>>> > > > >>>> response for sessionid:0x424eebf1c10004c after 3ms
> >> > >> >>>> > > > >>>> 2009-11-13 20:06:08,178 INFO
> >> > >> org.apache.hadoop.hdfs.DFSClient:
> >> > >> >>>> > > > Exception
> >> > >> >>>> > > > >>>>
> >> > >> >>>> > > > >>> in
> >> > >> >>>> > > > >>>
> >> > >> >>>> > > > >>>> createBlockOutputStream java.io.IOException: Bad
> >> connect
> >> > >> ack
> >> > >> >>>> with
> >> > >> >>>> > > > >>>> firstBadLink 192.168.100.123:50010
> >> > >> >>>> > > > >>>> 2009-11-13 20:06:08,178 INFO
> >> > >> org.apache.hadoop.hdfs.DFSClient:
> >> > >> >>>> > > > >>>> Abandoning
> >> > >> >>>> > > > >>>> block blk_-2808245019291145247_5478039
> >> > >> >>>> > > > >>>> 2009-11-13 20:06:09,682 INFO
> >> > >> org.apache.hadoop.hdfs.DFSClient:
> >> > >> >>>> > > > Exception
> >> > >> >>>> > > > >>>>
> >> > >> >>>> > > > >>> in
> >> > >> >>>> > > > >>>
> >> > >> >>>> > > > >>>> createBlockOutputStream java.io.EOFException
> >> > >> >>>> > > > >>>> 2009-11-13 20:06:09,682 INFO
> >> > >> org.apache.hadoop.hdfs.DFSClient:
> >> > >> >>>> > > > >>>> Abandoning
> >> > >> >>>> > > > >>>> block blk_1074853606841896259_5478048
> >> > >> >>>> > > > >>>> 2009-11-13 20:06:10,334 DEBUG
> >> > >> org.apache.zookeeper.ClientCnxn:
> >> > >> >>>> Got
> >> > >> >>>> > > > ping
> >> > >> >>>> > > > >>>> response for sessionid:0x24eebf1043003c after 1ms
> >> > >> >>>> > > > >>>> 2009-11-13 20:06:21,018 DEBUG
> >> > >> org.apache.zookeeper.ClientCnxn:
> >> > >> >>>> Got
> >> > >> >>>> > > > ping
> >> > >> >>>> > > > >>>> response for sessionid:0x424eebf1c10004c after 0ms
> >> > >> >>>> > > > >>>> 2009-11-13 20:06:23,674 DEBUG
> >> > >> org.apache.zookeeper.ClientCnxn:
> >> > >> >>>> Got
> >> > >> >>>> > > > ping
> >> > >> >>>> > > > >>>> response for sessionid:0x24eebf1043003c after 0ms
> >> > >> >>>> > > > >>>> 2009-11-13 20:06:24,828 INFO
> >> > >> org.apache.hadoop.hdfs.DFSClient:
> >> > >> >>>> > > > Exception
> >> > >> >>>> > > > >>>>
> >> > >> >>>> > > > >>> in
> >> > >> >>>> > > > >>>
> >> > >> >>>> > > > >>>> createBlockOutputStream java.io.IOException: Bad
> >> connect
> >> > >> ack
> >> > >> >>>> with
> >> > >> >>>> > > > >>>> firstBadLink 192.168.100.123:50010
> >> > >> >>>> > > > >>>> 2009-11-13 20:06:24,828 INFO
> >> > >> org.apache.hadoop.hdfs.DFSClient:
> >> > >> >>>> > > > >>>> Abandoning
> >> > >> >>>> > > > >>>> block blk_-6642544517082142289_5478063
> >> > >> >>>> > > > >>>> 2009-11-13 20:06:24,828 INFO
> >> > >> org.apache.hadoop.hdfs.DFSClient:
> >> > >> >>>> > > > Exception
> >> > >> >>>> > > > >>>>
> >> > >> >>>> > > > >>> in
> >> > >> >>>> > > > >>>
> >> > >> >>>> > > > >>>> createBlockOutputStream java.net.SocketException:
> Too
> >> > many
> >> > >> >>>> open
> >> > >> >>>> > > files
> >> > >> >>>> > > > >>>> 2009-11-13 20:06:24,828 INFO
> >> > >> org.apache.hadoop.hdfs.DFSClient:
> >> > >> >>>> > > > >>>> Abandoning
> >> > >> >>>> > > > >>>> block blk_2057511041109796090_5478063
> >> > >> >>>> > > > >>>> 2009-11-13 20:06:24,928 INFO
> >> > >> org.apache.hadoop.hdfs.DFSClient:
> >> > >> >>>> > > > Exception
> >> > >> >>>> > > > >>>>
> >> > >> >>>> > > > >>> in
> >> > >> >>>> > > > >>>
> >> > >> >>>> > > > >>>> createBlockOutputStream java.net.SocketException:
> Too
> >> > many
> >> > >> >>>> open
> >> > >> >>>> > > files
> >> > >> >>>> > > > >>>> 2009-11-13 20:06:24,928 INFO
> >> > >> org.apache.hadoop.hdfs.DFSClient:
> >> > >> >>>> > > > >>>> Abandoning
> >> > >> >>>> > > > >>>> block blk_8219260302213892894_5478064
> >> > >> >>>> > > > >>>> 2009-11-13 20:06:30,855 INFO
> >> > >> org.apache.hadoop.hdfs.DFSClient:
> >> > >> >>>> > > > Exception
> >> > >> >>>> > > > >>>>
> >> > >> >>>> > > > >>> in
> >> > >> >>>> > > > >>>
> >> > >> >>>> > > > >>>> createBlockOutputStream java.net.SocketException:
> Too
> >> > many
> >> > >> >>>> open
> >> > >> >>>> > > files
> >> > >> >>>> > > > >>>> 2009-11-13 20:06:30,855 INFO
> >> > >> org.apache.hadoop.hdfs.DFSClient:
> >> > >> >>>> > > > >>>> Abandoning
> >> > >> >>>> > > > >>>> block blk_1669205542853067709_5478235
> >> > >> >>>> > > > >>>> 2009-11-13 20:06:30,905 INFO
> >> > >> org.apache.hadoop.hdfs.DFSClient:
> >> > >> >>>> > > > Exception
> >> > >> >>>> > > > >>>>
> >> > >> >>>> > > > >>> in
> >> > >> >>>> > > > >>>
> >> > >> >>>> > > > >>>> createBlockOutputStream java.net.SocketException:
> Too
> >> > many
> >> > >> >>>> open
> >> > >> >>>> > > files
> >> > >> >>>> > > > >>>> 2009-11-13 20:06:30,905 INFO
> >> > >> org.apache.hadoop.hdfs.DFSClient:
> >> > >> >>>> > > > >>>> Abandoning
> >> > >> >>>> > > > >>>> block blk_9128897691346270351_5478237
> >> > >> >>>> > > > >>>> 2009-11-13 20:06:30,955 INFO
> >> > >> org.apache.hadoop.hdfs.DFSClient:
> >> > >> >>>> > > > Exception
> >> > >> >>>> > > > >>>>
> >> > >> >>>> > > > >>> in
> >> > >> >>>> > > > >>>
> >> > >> >>>> > > > >>>> createBlockOutputStream java.net.SocketException:
> Too
> >> > many
> >> > >> >>>> open
> >> > >> >>>> > > files
> >> > >> >>>> > > > >>>> 2009-11-13 20:06:30,955 INFO
> >> > >> org.apache.hadoop.hdfs.DFSClient:
> >> > >> >>>> > > > >>>> Abandoning
> >> > >> >>>> > > > >>>> block blk_1116845144864123018_5478240
> >> > >> >>>> > > > >>>> 2009-11-13 20:06:34,372 DEBUG
> >> > >> org.apache.zookeeper.ClientCnxn:
> >> > >> >>>> Got
> >> > >> >>>> > > > ping
> >> > >> >>>> > > > >>>> response for sessionid:0x424eebf1c10004c after 0ms
> >> > >> >>>> > > > >>>> 2009-11-13 20:06:37,034 DEBUG
> >> > >> org.apache.zookeeper.ClientCnxn:
> >> > >> >>>> Got
> >> > >> >>>> > > > ping
> >> > >> >>>> > > > >>>> response for sessionid:0x24eebf1043003c after 0ms
> >> > >> >>>> > > > >>>> 2009-11-13 20:06:37,235 WARN
> >> > >> org.apache.hadoop.hdfs.DFSClient:
> >> > >> >>>> > > > >>>>
> >> > >> >>>> > > > >>> DataStreamer
> >> > >> >>>> > > > >>>
> >> > >> >>>> > > > >>>> Exception: java.io.IOException: Too many open files
> >> > >> >>>> > > > >>>> at sun.nio.ch.IOUtil.initPipe(Native Method)
> >> > >> >>>> > > > >>>> at
> >> > >> >>>> > >
> >> sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:49)
> >> > >> >>>> > > > >>>> at
> >> > >> >>>> > > > >>>>
> >> > >> >>>> > > > >>>>
> >> > >> >>>> > > >
> >> > >> >>>> > >
> >> > >> >>>> >
> >> > >> >>>>
> >> > >>
> >> >
> >>
> sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:18)
> >> > >> >>>> > > > >>>
> >> > >> >>>> > > > >>>> at
> >> > >> >>>> > > > >>>>
> >> > >> >>>> > > > >>>>
> >> > >> >>>> > > >
> >> > >> >>>> > >
> >> > >> >>>> >
> >> > >> >>>>
> >> > >>
> >> >
> >>
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.get(SocketIOWithTimeout.java:407)
> >> > >> >>>> > > > >>>
> >> > >> >>>> > > > >>>> at
> >> > >> >>>> > > > >>>>
> >> > >> >>>> > > > >>>>
> >> > >> >>>> > > >
> >> > >> >>>> > >
> >> > >> >>>> >
> >> > >> >>>>
> >> > >>
> >> >
> >>
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:322)
> >> > >> >>>> > > > >>>
> >> > >> >>>> > > > >>>> at
> >> > >> >>>> > > > >>>>
> >> > >> >>>> > > > >>>>
> >> > >> >>>> > > >
> >> > >> >>>> > >
> >> > >> >>>> >
> >> > >> >>>>
> >> > >>
> >> >
> >>
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
> >> > >> >>>> > > > >>>
> >> > >> >>>> > > > >>>> at
> >> > >> >>>> > > > >>>>
> >> > >> >>>> > > > >>>>
> >> > >> >>>> > > >
> >> > >> >>>> > >
> >> > >> >>>> >
> >> > >> >>>>
> >> > >>
> >> >
> >>
> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
> >> > >> >>>> > > > >>>
> >> > >> >>>> > > > >>>> at
> >> > >> >>>> > > > >>>>
> >> > >> >>>> > > > >>>>
> >> > >> >>>> > > >
> >> > >> >>>> > >
> >> > >> >>>> >
> >> > >> >>>>
> >> > >>
> >> >
> >>
> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
> >> > >> >>>> > > > >>>
> >> > >> >>>> > > > >>>> at
> >> > >> >>>> > > > >>>>
> >> > >> >>>> > > > >>>
> >> > >> >>>>
> >> java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
> >> > >> >>>> > > > >>>
> >> > >> >>>> > > > >>>> at
> >> > >> >>>> java.io.DataOutputStream.write(DataOutputStream.java:90)
> >> > >> >>>> > > > >>>> at
> >> > >> >>>> > > > >>>>
> >> > >> >>>> > > > >>>>
> >> > >> >>>> > > >
> >> > >> >>>> > >
> >> > >> >>>> >
> >> > >> >>>>
> >> > >>
> >> >
> >>
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2290)
> >> > >> >>>> > > > >>>
> >> > >> >>>> > > > >>>>
> >> > >> >>>> > > > >>>> 2009-11-13 20:06:37,235 WARN
> >> > >> org.apache.hadoop.hdfs.DFSClient:
> >> > >> >>>> > Error
> >> > >> >>>> > > > >>>> Recovery for block blk_8148813491785406356_5478475
> >> bad
> >> > >> >>>> datanode[0]
> >> > >> >>>> > > > >>>> 192.168.100.123:50010
> >> > >> >>>> > > > >>>> 2009-11-13 20:06:37,235 WARN
> >> > >> org.apache.hadoop.hdfs.DFSClient:
> >> > >> >>>> > Error
> >> > >> >>>> > > > >>>> Recovery for block blk_8148813491785406356_5478475
> in
> >> > >> pipeline
> >> > >> >>>> > > > >>>> 192.168.100.123:50010, 192.168.100.134:50010,
> >> > >> >>>> > 192.168.100.122:50010
> >> > >> >>>> > > :
> >> > >> >>>> > > > >>>> bad
> >> > >> >>>> > > > >>>> datanode 192.168.100.123:50010
> >> > >> >>>> > > > >>>> 2009-11-13 20:06:37,436 INFO
> >> > >> org.apache.hadoop.hdfs.DFSClient:
> >> > >> >>>> > > > Exception
> >> > >> >>>> > > > >>>>
> >> > >> >>>> > > > >>> in
> >> > >> >>>> > > > >>>
> >> > >> >>>> > > > >>>> createBlockOutputStream java.net.SocketException:
> Too
> >> > many
> >> > >> >>>> open
> >> > >> >>>> > > files
> >> > >> >>>> > > > >>>> 2009-11-13 20:06:37,436 INFO
> >> > >> org.apache.hadoop.hdfs.DFSClient:
> >> > >> >>>> > > > >>>> Abandoning
> >> > >> >>>> > > > >>>> block blk_2119727700857186236_5478498
> >> > >> >>>> > > > >>>>
> >> > >> >>>> > > > >>>>
> >> > >> >>>> > > > >>>>
> >> > >> >>>> > > > >>>>
> >> > >> >>>> > > > >>>>
> >> > >> >>>> > > > >>>> On Thu, Nov 12, 2009 at 4:21 PM, Zhenyu Zhong <
> >> > >> >>>> > > > zhongresearch@gmail.com
> >> > >> >>>> > > > >>>> wrote:
> >> > >> >>>> > > > >>>>
> >> > >> >>>> > > > >>>> Will do.
> >> > >> >>>> > > > >>>>>
> >> > >> >>>> > > > >>>>> thanks
> >> > >> >>>> > > > >>>>> zhenyu
> >> > >> >>>> > > > >>>>>
> >> > >> >>>> > > > >>>>>
> >> > >> >>>> > > > >>>>> On Thu, Nov 12, 2009 at 3:33 PM, stack <
> >> > stack@duboce.net
> >> > >> >
> >> > >> >>>> wrote:
> >> > >> >>>> > > > >>>>>
> >> > >> >>>> > > > >>>>> Enable GC logging too on this next run (see
> >> > >> hbase-env.sh).
> >> > >> >>>> Lets
> >> > >> >>>> > > try
> >> > >> >>>> > > > >>>>>>
> >> > >> >>>> > > > >>>>> and
> >> > >> >>>> > > > >>>
> >> > >> >>>> > > > >>>> get
> >> > >> >>>> > > > >>>>>> to the bottom of whats going on.
> >> > >> >>>> > > > >>>>>> Thanks,
> >> > >> >>>> > > > >>>>>> St.Ack
> >> > >> >>>> > > > >>>>>>
> >> > >> >>>> > > > >>>>>> On Thu, Nov 12, 2009 at 12:29 PM, Zhenyu Zhong <
> >> > >> >>>> > > > >>>>>>
> >> > >> >>>> > > > >>>>> zhongresearch@gmail.com
> >> > >> >>>> > > > >>>
> >> > >> >>>> > > > >>>> wrote:
> >> > >> >>>> > > > >>>>>>>
> >> > >> >>>> > > > >>>>>>
> >> > >> >>>> > > > >>>>>> I can switch the boxes that run zookeeper with
> the
> >> > ones
> >> > >> >>>> that
> >> > >> >>>> > run
> >> > >> >>>> > > > >>>>>>> regionservers.
> >> > >> >>>> > > > >>>>>>> I will see the results later.
> >> > >> >>>> > > > >>>>>>>
> >> > >> >>>> > > > >>>>>>> FYI. The node does have the 10 minutes
> >> > >> >>>> > zookeeper.session.timeout
> >> > >> >>>> > > > >>>>>>>
> >> > >> >>>> > > > >>>>>> value
> >> > >> >>>> > > > >>>
> >> > >> >>>> > > > >>>> in
> >> > >> >>>> > > > >>>>>>
> >> > >> >>>> > > > >>>>>>> place.
> >> > >> >>>> > > > >>>>>>>
> >> > >> >>>> > > > >>>>>>> thanks
> >> > >> >>>> > > > >>>>>>> zhenyu
> >> > >> >>>> > > > >>>>>>>
> >> > >> >>>> > > > >>>>>>> On Thu, Nov 12, 2009 at 3:21 PM, stack <
> >> > >> stack@duboce.net>
> >> > >> >>>> > wrote:
> >> > >> >>>> > > > >>>>>>>
> >> > >> >>>> > > > >>>>>>> On Thu, Nov 12, 2009 at 11:50 AM, Zhenyu Zhong
> <
> >> > >> >>>> > > > >>>>>>>>
> >> > >> >>>> > > > >>>>>>> zhongresearch@gmail.com
> >> > >> >>>> > > > >>>>>>
> >> > >> >>>> > > > >>>>>>> wrote:
> >> > >> >>>> > > > >>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>
> >> > >> >>>> > > > >>>>>>>>
> >> > >> >>>> > > > >>>>>>>>> In my cluster, half of the cluster have 2
> disks
> >> > 400GB
> >> > >> >>>> each
> >> > >> >>>> > per
> >> > >> >>>> > > > >>>>>>>>>
> >> > >> >>>> > > > >>>>>>>> machine,
> >> > >> >>>> > > > >>>>>>
> >> > >> >>>> > > > >>>>>>> and
> >> > >> >>>> > > > >>>>>>>>
> >> > >> >>>> > > > >>>>>>>>> half of the cluster have 6 disks per machine.
> >> > Maybe
> >> > >> we
> >> > >> >>>> > should
> >> > >> >>>> > > > >>>>>>>>>
> >> > >> >>>> > > > >>>>>>>> run
> >> > >> >>>> > > > >>>
> >> > >> >>>> > > > >>>> zookeeper
> >> > >> >>>> > > > >>>>>>>>> on the machines with 2 disks and RS on
> machines
> >> > with
> >> > >> 6
> >> > >> >>>> disks?
> >> > >> >>>> > > > >>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>> That would make most sense only in the below,
> it
> >> > >> looks
> >> > >> >>>> like
> >> > >> >>>> > the
> >> > >> >>>> > > > >>>>>>>>>
> >> > >> >>>> > > > >>>>>>>> RS
> >> > >> >>>> > > > >>>
> >> > >> >>>> > > > >>>> that
> >> > >> >>>> > > > >>>>>>
> >> > >> >>>> > > > >>>>>>> had
> >> > >> >>>> > > > >>>>>>>> issue had 4 disks?
> >> > >> >>>> > > > >>>>>>>>
> >> > >> >>>> > > > >>>>>>>>
> >> > >> >>>> > > > >>>>>>>>
> >> > >> >>>> > > > >>>>>>>> BTW, the 10 minutes zookeeper.session.timeout
> >> has
> >> > >> been
> >> > >> >>>> set
> >> > >> >>>> > > during
> >> > >> >>>> > > > >>>>>>>>>
> >> > >> >>>> > > > >>>>>>>> the
> >> > >> >>>> > > > >>>>>>
> >> > >> >>>> > > > >>>>>>> experiment.
> >> > >> >>>> > > > >>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>
> >> > >> >>>> > > > >>>>>>>> And for sure this node had it in place?
> >> > >> >>>> > > > >>>>>>>> St.Ack
> >> > >> >>>> > > > >>>>>>>>
> >> > >> >>>> > > > >>>>>>>>
> >> > >> >>>> > > > >>>>>>>>
> >> > >> >>>> > > > >>>>>>>>
> >> > >> >>>> > > > >>>>>>>>> thanks
> >> > >> >>>> > > > >>>>>>>>> zhenyu
> >> > >> >>>> > > > >>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>> On Thu, Nov 12, 2009 at 2:08 PM, stack <
> >> > >> stack@duboce.net
> >> > >> >>>> >
> >> > >> >>>> > > wrote:
> >> > >> >>>> > > > >>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>> On Thu, Nov 12, 2009 at 8:40 AM, Zhenyu Zhong
> <
> >> > >> >>>> > > > >>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>> zhongresearch@gmail.com
> >> > >> >>>> > > > >>>>>>>
> >> > >> >>>> > > > >>>>>>>> wrote:
> >> > >> >>>> > > > >>>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>> Though I experienced 2 regionserver
> >> disconnection
> >> > >> this
> >> > >> >>>> > > > >>>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>> morning,
> >> > >> >>>> > > > >>>
> >> > >> >>>> > > > >>>> it
> >> > >> >>>> > > > >>>>>>
> >> > >> >>>> > > > >>>>>>> looks
> >> > >> >>>> > > > >>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>> better from the regionserver log. (please see
> >> the
> >> > >> >>>> following
> >> > >> >>>> > > > >>>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>> log)
> >> > >> >>>> > > > >>>
> >> > >> >>>> > > > >>>>
> >> > >> >>>> > > > >>>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>>> http://pastebin.com/m496dbfae
> >> > >> >>>> > > > >>>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>>> I checked diskIO for one of the
> >> > >> >>>> > regionserver(192.168.100.116)
> >> > >> >>>> > > > >>>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>> during
> >> > >> >>>> > > > >>>>>>>
> >> > >> >>>> > > > >>>>>>>> the
> >> > >> >>>> > > > >>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>> time it disconnected.
> >> > >> >>>> > > > >>>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>>> Time: 03:04:58 AM
> >> > >> >>>> > > > >>>>>>>>>>> Device: tps Blk_read/s
> >> Blk_wrtn/s
> >> > >> >>>> Blk_read
> >> > >> >>>> > > > >>>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>> Blk_wrtn
> >> > >> >>>> > > > >>>>>>>>
> >> > >> >>>> > > > >>>>>>>>> sda 105.31 5458.83
> 19503.64
> >> > >> >>>> 9043873239
> >> > >> >>>> > > > >>>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>> 32312473676
> >> > >> >>>> > > > >>>>>>>>
> >> > >> >>>> > > > >>>>>>>>> sda1 2.90 6.64
> 99.25
> >> > >> >>>> 10993934
> >> > >> >>>> > > > >>>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>> 164433464
> >> > >> >>>> > > > >>>>>>>>
> >> > >> >>>> > > > >>>>>>>>> sda2 1.72 23.76
> 30.25
> >> > >> >>>> 39365817
> >> > >> >>>> > > > >>>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>> 50124008
> >> > >> >>>> > > > >>>>>>>>
> >> > >> >>>> > > > >>>>>>>>> sda3 0.30 0.38
> 3.58
> >> > >> >>>> 624930
> >> > >> >>>> > > > >>>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>> 5923000
> >> > >> >>>> > > > >>>>>>>>
> >> > >> >>>> > > > >>>>>>>>> sda4 100.39 5428.06
> 19370.56
> >> > >> >>>> 8992888270
> >> > >> >>>> > > > >>>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>> 32091993204
> >> > >> >>>> > > > >>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>>> Is this high for you? 20k blocks/second
> >> would
> >> > >> seem to
> >> > >> >>>> be
> >> > >> >>>> > > high
> >> > >> >>>> > > > >>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>> but
> >> > >> >>>> > > > >>>
> >> > >> >>>> > > > >>>> its
> >> > >> >>>> > > > >>>>>>>
> >> > >> >>>> > > > >>>>>>>> one
> >> > >> >>>> > > > >>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>> disk only and its not being shared by zk
> >> anymore
> >> > so
> >> > >> >>>> > shouldn't
> >> > >> >>>> > > > >>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>> matter?
> >> > >> >>>> > > > >>>>>>
> >> > >> >>>> > > > >>>>>>>
> >> > >> >>>> > > > >>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>> I also checked the zookeeper quorum server
> >> that
> >> > the
> >> > >> >>>> > > > >>>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>> regionserver
> >> > >> >>>> > > > >>>
> >> > >> >>>> > > > >>>> tried
> >> > >> >>>> > > > >>>>>>>>
> >> > >> >>>> > > > >>>>>>>>> to
> >> > >> >>>> > > > >>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>> connect according to the log. However, I
> don't
> >> see
> >> > >> >>>> > > > >>>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>> 192.168.100.116
> >> > >> >>>> > > > >>>>>>
> >> > >> >>>> > > > >>>>>>> in
> >> > >> >>>> > > > >>>>>>>
> >> > >> >>>> > > > >>>>>>>> the
> >> > >> >>>> > > > >>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>> client list of the zookeeper quorum server
> that
> >> > the
> >> > >> >>>> > > > >>>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>> regionserver
> >> > >> >>>> > > > >>>
> >> > >> >>>> > > > >>>> tried
> >> > >> >>>> > > > >>>>>>>>
> >> > >> >>>> > > > >>>>>>>>> to
> >> > >> >>>> > > > >>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>> connect.
> >> > >> >>>> > > > >>>>>>>>>>> Would that be a problem?
> >> > >> >>>> > > > >>>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>>> Is that because the ephemeral node for the
> >> > >> >>>> regionserver
> >> > >> >>>> > had
> >> > >> >>>> > > > >>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>> evaporated?
> >> > >> >>>> > > > >>>>>>>
> >> > >> >>>> > > > >>>>>>>> Lost
> >> > >> >>>> > > > >>>>>>>>>> its lease w/ zk by the time you went to look?
> >> > >> >>>> > > > >>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>> Thu Nov 12 15:04:52 UTC 2009
> >> > >> >>>> > > > >>>>>>>>>>> Zookeeper version: 3.2.1-808558, built on
> >> > >> 08/27/2009
> >> > >> >>>> 18:48
> >> > >> >>>> > > > >>>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>> GMT
> >> > >> >>>> > > > >>>
> >> > >> >>>> > > > >>>> Clients:
> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.127:43045
> >> > >> [1](queued=0,recved=26,sent=0)
> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.131:39091
> >> > >> [1](queued=0,recved=964,sent=0)
> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.124:35961
> >> > >> [1](queued=0,recved=3266,sent=0)
> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.123:47935
> >> > >> [1](queued=0,recved=233,sent=0)
> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.125:46931
> >> > >> [1](queued=0,recved=2,sent=0)
> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.118:54924
> >> > >> [1](queued=0,recved=295,sent=0)
> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.118:41390
> >> > >> [1](queued=0,recved=2290,sent=0)
> >> > >> >>>> > > > >>>>>>>>>>> /192.168.100.136:42243
> >> > >> [1](queued=0,recved=0,sent=0)
> >> > >> >>>> > > > >>>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>>> Latency min/avg/max: 0/17/6333
> >> > >> >>>> > > > >>>>>>>>>>> Received: 47111
> >> > >> >>>> > > > >>>>>>>>>>> Sent: 0
> >> > >> >>>> > > > >>>>>>>>>>> Outstanding: 0
> >> > >> >>>> > > > >>>>>>>>>>> Zxid: 0x77000083f4
> >> > >> >>>> > > > >>>>>>>>>>> Mode: leader
> >> > >> >>>> > > > >>>>>>>>>>> Node count: 23
> >> > >> >>>> > > > >>>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>> That 6 second maximum latency is pretty bad
> but
> >> > >> should
> >> > >> >>>> be
> >> > >> >>>> > well
> >> > >> >>>> > > > >>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>> within
> >> > >> >>>> > > > >>>>>>
> >> > >> >>>> > > > >>>>>>> the
> >> > >> >>>> > > > >>>>>>>>
> >> > >> >>>> > > > >>>>>>>>> zk
> >> > >> >>>> > > > >>>>>>>>>> session timeout.
> >> > >> >>>> > > > >>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>> So, problem is likely on the zk client side
> of
> >> the
> >> > >> >>>> session;
> >> > >> >>>> > > > >>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>> i.e.
> >> > >> >>>> > > > >>>
> >> > >> >>>> > > > >>>> in
> >> > >> >>>> > > > >>>>>>
> >> > >> >>>> > > > >>>>>>> the
> >> > >> >>>> > > > >>>>>>>
> >> > >> >>>> > > > >>>>>>>> RS.
> >> > >> >>>> > > > >>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>> You could enable GC logging as J-D suggested
> to
> >> > see
> >> > >> if
> >> > >> >>>> you
> >> > >> >>>> > > have
> >> > >> >>>> > > > >>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>> any
> >> > >> >>>> > > > >>>>>>
> >> > >> >>>> > > > >>>>>>> big
> >> > >> >>>> > > > >>>>>>>
> >> > >> >>>> > > > >>>>>>>> pauses, pauses > zk session timeout.
> >> > >> >>>> > > > >>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>> When the RS went down, it didn't look too
> >> heavily
> >> > >> >>>> loaded:
> >> > >> >>>> > > > >>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>> 1. 2009-11-12 15:04:52,830 INFO
> >> > >> >>>> > > > >>>>>>>>>>
> >> > org.apache.hadoop.hbase.regionserver.HRegionServer:
> >> > >> >>>> Dump of
> >> > >> >>>> > > > >>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>> metrics:
> >> > >> >>>> > > > >>>>>>>
> >> > >> >>>> > > > >>>>>>>> request=1.5166667, regions=322, stores=657,
> >> > >> >>>> storefiles=631,
> >> > >> >>>> > > > >>>>>>>>>> storefileIndexSize=61, memstoreSize=1472,
> >> > >> >>>> usedHeap=2819,
> >> > >> >>>> > > > >>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>> maxHeap=4079,
> >> > >> >>>> > > > >>>>>>>>
> >> > >> >>>> > > > >>>>>>>>> blockCacheSize=658110960,
> >> > blockCacheFree=197395984,
> >> > >> >>>> > > > >>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>> blockCacheCount=9903,
> >> > >> >>>> > > > >>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>> blockCacheHitRatio=99
> >> > >> >>>> > > > >>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>> Its serving a few reads? The number of store
> >> > files
> >> > >> >>>> seems
> >> > >> >>>> > > fine.
> >> > >> >>>> > > > >>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>> Not
> >> > >> >>>> > > > >>>>>>
> >> > >> >>>> > > > >>>>>>> too
> >> > >> >>>> > > > >>>>>>>>
> >> > >> >>>> > > > >>>>>>>>> much memory used.
> >> > >> >>>> > > > >>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>> Looking at the logs, I see the Lease Still
> Held
> >> > >> >>>> exception.
> >> > >> >>>> > > > >>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>> This
> >> > >> >>>> > > > >>>
> >> > >> >>>> > > > >>>> happens
> >> > >> >>>> > > > >>>>>>>>
> >> > >> >>>> > > > >>>>>>>>> when the RS does its regular report to the
> >> master
> >> > but
> >> > >> the
> >> > >> >>>> > > > >>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>> master
> >> > >> >>>> > > > >>>
> >> > >> >>>> > > > >>>> thinks
> >> > >> >>>> > > > >>>>>>>
> >> > >> >>>> > > > >>>>>>>> the
> >> > >> >>>> > > > >>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>> RS has since restarted. It'll think this
> >> probably
> >> > >> >>>> because
> >> > >> >>>> > it
> >> > >> >>>> > > > >>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>> noticed
> >> > >> >>>> > > > >>>>>>
> >> > >> >>>> > > > >>>>>>> that
> >> > >> >>>> > > > >>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>> the RS's znode in zk had gone away and it
> >> > considered
> >> > >> the
> >> > >> >>>> RS
> >> > >> >>>> > > > >>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>> dead.
> >> > >> >>>> > > > >>>
> >> > >> >>>> > > > >>>>
> >> > >> >>>> > > > >>>>>>>>>> Looking too at your logs I see this gap in
> the
> >> zk
> >> > >> >>>> pinging:
> >> > >> >>>> > > > >>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>> 1. 2009-11-12 15:03:39,325 DEBUG
> >> > >> >>>> > > > >>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>> org.apache.zookeeper.ClientCnxn:
> >> > >> >>>> > > > >>>>>>
> >> > >> >>>> > > > >>>>>>> Got
> >> > >> >>>> > > > >>>>>>>
> >> > >> >>>> > > > >>>>>>>> ping response for sessionid:0x224e55436ad0004
> >> after
> >> > >> 0ms
> >> > >> >>>> > > > >>>>>>>>>> 2. 2009-11-12 15:03:43,113 DEBUG
> >> > >> >>>> > > > >>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>> org.apache.zookeeper.ClientCnxn:
> >> > >> >>>> > > > >>>>>>
> >> > >> >>>> > > > >>>>>>> Got
> >> > >> >>>> > > > >>>>>>>
> >> > >> >>>> > > > >>>>>>>> ping response for sessionid:0x24e55436a0007d
> >> after
> >> > >> 0ms
> >> > >> >>>> > > > >>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>> Where in the lines above it, its reporting
> >> about
> >> > >> every
> >> > >> >>>> ten
> >> > >> >>>> > > > >>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>> seconds,
> >> > >> >>>> > > > >>>>>>
> >> > >> >>>> > > > >>>>>>> here
> >> > >> >>>> > > > >>>>>>>>
> >> > >> >>>> > > > >>>>>>>>> there is a big gap.
> >> > >> >>>> > > > >>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>> Do you have ganglia or something that will
> let
> >> you
> >> > >> look
> >> > >> >>>> more
> >> > >> >>>> > > > >>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>> into
> >> > >> >>>> > > > >>>
> >> > >> >>>> > > > >>>> what
> >> > >> >>>> > > > >>>>>>>
> >> > >> >>>> > > > >>>>>>>> was
> >> > >> >>>> > > > >>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>> happening on this machine around this time?
> Is
> >> > the
> >> > >> >>>> machine
> >> > >> >>>> > > OK?
> >> > >> >>>> > > > >>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>> It
> >> > >> >>>> > > > >>>>>>
> >> > >> >>>> > > > >>>>>>> looks
> >> > >> >>>> > > > >>>>>>>>
> >> > >> >>>> > > > >>>>>>>>> lightly loaded and you have your cluster
> nicely
> >> > laid
> >> > >> out.
> >> > >> >>>> > > > >>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>> Something
> >> > >> >>>> > > > >>>>>>
> >> > >> >>>> > > > >>>>>>> odd
> >> > >> >>>> > > > >>>>>>>>
> >> > >> >>>> > > > >>>>>>>>> is
> >> > >> >>>> > > > >>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>> going on. What about things like the write
> >> speed
> >> > to
> >> > >> >>>> disk?
> >> > >> >>>> > In
> >> > >> >>>> > > > >>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>> the
> >> > >> >>>> > > > >>>
> >> > >> >>>> > > > >>>> past
> >> > >> >>>> > > > >>>>>>>
> >> > >> >>>> > > > >>>>>>>> strange issues have been explained by
> incorrectly
> >> > set
> >> > >> BIOS
> >> > >> >>>> > > > >>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>> which
> >> > >> >>>> > > > >>>
> >> > >> >>>> > > > >>>> made
> >> > >> >>>> > > > >>>>>>
> >> > >> >>>> > > > >>>>>>> disks
> >> > >> >>>> > > > >>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>> run at 1/100th of their proper speed.
> >> > >> >>>> > > > >>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>> St.Ack
> >> > >> >>>> > > > >>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>>> Best,
> >> > >> >>>> > > > >>>>>>>>>>> zhenyu
> >> > >> >>>> > > > >>>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>>> On Wed, Nov 11, 2009 at 3:58 PM, Zhenyu
> Zhong
> >> <
> >> > >> >>>> > > > >>>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>> zhongresearch@gmail.com
> >> > >> >>>> > > > >>>>>>>>
> >> > >> >>>> > > > >>>>>>>>> wrote:
> >> > >> >>>> > > > >>>>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>>> Stack
> >> > >> >>>> > > > >>>>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>>>> I am very appreciated for your comments.
> >> > >> >>>> > > > >>>>>>>>>>>> I will use the zookeeper monitoring script
> on
> >> my
> >> > >> >>>> cluster
> >> > >> >>>> > > > >>>>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>>> and
> >> > >> >>>> > > > >>>
> >> > >> >>>> > > > >>>> let
> >> > >> >>>> > > > >>>>>>
> >> > >> >>>> > > > >>>>>>> it
> >> > >> >>>> > > > >>>>>>>
> >> > >> >>>> > > > >>>>>>>> run
> >> > >> >>>> > > > >>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>> overnight to see the result.
> >> > >> >>>> > > > >>>>>>>>>>>> I will follow up the thread when I get
> >> anything.
> >> > >> >>>> > > > >>>>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>>>> thanks
> >> > >> >>>> > > > >>>>>>>>>>>> zhenyu
> >> > >> >>>> > > > >>>>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>>>> On Wed, Nov 11, 2009 at 3:52 PM, stack <
> >> > >> >>>> stack@duboce.net>
> >> > >> >>>> > > > >>>>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>>> wrote:
> >> > >> >>>> > > > >>>>>>
> >> > >> >>>> > > > >>>>>>>
> >> > >> >>>> > > > >>>>>>>>>>>> I see these in your log too:
> >> > >> >>>> > > > >>>>>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>>>>> 1. 2009-11-11 04:27:19,018 DEBUG
> >> > >> >>>> > > > >>>>>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
> >> > >> >>>> > > > >>>>>>>>
> >> > >> >>>> > > > >>>>>>>>> Got
> >> > >> >>>> > > > >>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>>> ping response for
> sessionid:0x424dfd908c50009
> >> > >> after
> >> > >> >>>> > > > >>>>>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>>>> 4544ms
> >> > >> >>>> > > > >>>
> >> > >> >>>> > > > >>>> 2. 2009-11-11 04:27:19,018 DEBUG
> >> > >> >>>> > > > >>>>>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
> >> > >> >>>> > > > >>>>>>>>
> >> > >> >>>> > > > >>>>>>>>> Got
> >> > >> >>>> > > > >>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>>> ping response for
> sessionid:0x24dfd90c810002
> >> > after
> >> > >> >>>> > > > >>>>>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>>>> 4548ms
> >> > >> >>>> > > > >>>
> >> > >> >>>> > > > >>>> 3. 2009-11-11 04:27:43,960 DEBUG
> >> > >> >>>> > > > >>>>>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
> >> > >> >>>> > > > >>>>>>>>
> >> > >> >>>> > > > >>>>>>>>> Got
> >> > >> >>>> > > > >>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>>> ping response for
> sessionid:0x424dfd908c50009
> >> > >> after
> >> > >> >>>> > > > >>>>>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>>>> 9030ms
> >> > >> >>>> > > > >>>
> >> > >> >>>> > > > >>>> 4. 2009-11-11 04:27:43,960 DEBUG
> >> > >> >>>> > > > >>>>>>>>>>>>>
> >> > >> >>>> > > > >>>>>>>>>>>> org.apache.zookeeper.
> >> > >> >>>> > > > >>>>>>>>
> >> > >> >>>> > > > >>>>>>>
> >> > >> >>>> > > >
> >> > >> >>>> > >
> >> > >> >>>> >
> >> > >> >>>>
> >> > >> >>>
> >> > >> >>>
> >> > >> >>
> >> > >> >
> >> > >>
> >> > >
> >> > >
> >> >
> >>
> >
> >
>
Re: regionserver disconnection
Posted by Zhenyu Zhong <zh...@gmail.com>.
After running more experiment, it seems that the disconnection is not
related to the heavy load job, because the load average is low, the diskio
is normal, memory Heap was not reached, also the virtual memory stats shows
that no swappings. However, the disconnection still happens.
It looks like this time it pauses for 20 seconds. No idea why regionserver
disconnected.
Any other suggestions please?
2009-11-18 12:41:00,956 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
response for sessionid:0x125035c495d02bb after 0ms
2009-11-18 12:41:08,074 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
response for sessionid:0x425035c48210002 after 0ms
2009-11-18 12:41:14,302 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
response for sessionid:0x125035c495d02bb after 0ms
2009-11-18 12:41:21,433 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
response for sessionid:0x425035c48210002 after 0ms
2009-11-18 12:41:27,656 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
response for sessionid:0x125035c495d02bb after 0ms
2009-11-18 12:41:34,788 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
response for sessionid:0x425035c48210002 after 0ms
2009-11-18 12:41:41,016 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
response for sessionid:0x125035c495d02bb after 1ms
2009-11-18 12:42:01,482 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x425035c48210002 to sun.nio.ch.SelectionKeyImpl@421690ab
java.io.IOException: TIMED OUT
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:906)
2009-11-18 12:42:01,582 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
state: Disconnected, type: None, path: null
2009-11-18 12:42:01,782 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server superpyxis0003/192.168.100.117:2181
2009-11-18 12:42:01,782 INFO org.apache.zookeeper.ClientCnxn: Priming
connection to java.nio.channels.SocketChannel[connected local=/
192.168.100.132:46610 remote=superpyxis0003/192.168.100.117:2181]
2009-11-18 12:42:01,782 INFO org.apache.zookeeper.ClientCnxn: Server
connection successful
2009-11-18 12:42:01,782 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x425035c48210002 to sun.nio.ch.SelectionKeyImpl@56dc6fac
java.io.IOException: Read error rc = -1 java.nio.DirectByteBuffer[pos=0
lim=4 cap=4]
at
org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
2009-11-18 12:42:01,782 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown input
java.net.SocketException: Transport endpoint is not connected
at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
at
sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
at
org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
2009-11-18 12:42:01,782 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown output
java.net.SocketException: Transport endpoint is not connected
at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
at
sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
at
org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
2009-11-18 12:42:02,182 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server superpyxis0001/192.168.100.115:2181
2009-11-18 12:42:02,182 INFO org.apache.zookeeper.ClientCnxn: Priming
connection to java.nio.channels.SocketChannel[connected local=/
192.168.100.132:36197 remote=superpyxis0001/192.168.100.115:2181]
2009-11-18 12:42:02,182 INFO org.apache.zookeeper.ClientCnxn: Server
connection successful
2009-11-18 12:42:02,182 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x425035c48210002 to sun.nio.ch.SelectionKeyImpl@5c07076b
java.io.IOException: Read error rc = -1 java.nio.DirectByteBuffer[pos=0
lim=4 cap=4]
at
org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
2009-11-18 12:42:02,182 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown input
java.net.SocketException: Transport endpoint is not connected
at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
at
sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
at
org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
2009-11-18 12:42:02,182 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown output
java.net.SocketException: Transport endpoint is not connected
On Mon, Nov 16, 2009 at 4:16 PM, Zhenyu Zhong <zh...@gmail.com>wrote:
> Here is the diskIO and CPU around the time we had RS disconnection on one
> machine that runs RegionServer. It doesn't seem to be high. Similar disk and
> cpu usage have been seen before and the HBase was running fine.
>
>
> So far I haven't found why my 10 minutes session timeout doesn't apply.
> Still digging.
>
> Many thanks!
>
> Time: 03:01:08 PM
> avg-cpu: %user %nice %system %iowait %steal %idle
> 3.82 0.00 0.89 3.97 0.00 91.32
>
>
> Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
> sda 24.30 1605.39 6679.35 6238623388 25956251720
> sda1 0.51 1.94 15.71 7547265 61067904
> sda2 0.13 2.11 5.20 8202079 20189232
> sda3 0.13 0.25 1.63 987846 6323336
> sda4 0.00 0.00 0.00 4 0
> sda5 1.78 0.06 24.14 220010 93817208
> sda6 0.38 0.42 8.41 1630584 32688152
> sda7 21.38 1600.61 6624.26 6220035272 25742165888
> sdb 4.52 767.33 565.91 2981868690 2199132380
> sdb1 4.52 767.33 565.91 2981866755 2199132372
> sdc 4.42 742.95 563.00 2887151482 2187823092
> sdc1 4.42 742.95 563.00 2887149547 2187823084
> sdd 4.49 750.78 557.25 2917554074 2165513500
> sdd1 4.49 750.78 557.25 2917552139 2165513492
> sde 4.52 758.51 569.46 2947593394 2212964236
> sde1 4.52 758.51 569.46 2947591459 2212964228
> sdf 4.51 747.22 571.78 2903740266 2221972652
> sdf1 4.51 747.22 571.78 2903738331 2221972644
>
> Best,
> zhenyu
>
>
>
> On Mon, Nov 16, 2009 at 4:07 PM, stack <st...@duboce.net> wrote:
>
>> On Mon, Nov 16, 2009 at 12:05 PM, Zhenyu Zhong <zhongresearch@gmail.com
>> >wrote:
>>
>> > I just realized that there was a MapReduce job running during the time
>> the
>> > regionserver disconnected from the zookeeper.
>> > That MapReduce Job was processing 500GB data and took about 8 minutes to
>> > finish. It launched over 2000 map tasks.
>>
>>
>> There was a tasktracker running on the RegionServer that disconnected? Is
>> the map i/o or cpu heavy? Do you think it could have stole life from the
>> datanode/regionserver?
>>
>>
>>
>> > I doubt that this introduced
>> > resource contention between DataNode and RegionServer Node.
>> > Also during the time that MapReduce job ran, I saw a few errors
>> indicating
>> > that
>> >
>> > java.io.IOException: Could not obtain block:
>> > blk_7661247556283580226_4450462 file=/data/xxx.txt.200910141159
>> > at
>> >
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1787)
>> > at
>> >
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1615)
>> > at
>> >
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1742)
>> > at java.io.DataInputStream.read(DataInputStream.java:83)
>> > at
>> org.apache.hadoop.util.LineReader.readLine(LineReader.java:134)
>> > at
>> >
>> org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:136)
>> > at
>> > org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:40)
>> > at
>> >
>> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:192)
>> > at
>> >
>> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:176)
>> > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
>> > at
>> org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
>> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
>> > at org.apache.hadoop.mapred.Child.main(Child.java:170)
>> >
>>
>> This is interesting. Your map task is running into hdfs-127. Yeah, you
>> should patch your cluster if you want to get rid of these (hdfs-127 has
>> been
>> applied to 0.21 hadoop and will be in the next hadoop release on 0.20.x
>> branch, hadoop 0.20.2).
>>
>>
>>
>> >
>> > Possibly it is related to HDFS-127, but I don't see any datanodes went
>> > down. Does that affect Regionserver? Shall we apply the patch?
>> >
>> >
>> Read up on hdfs-127 for explaination of whats going on.
>>
>>
>>
>> > Now I start to keep tracking of the virtual memory stats to see if the
>> same
>> > issue happens tomorrow around the same time period.
>> >
>>
>> Excellent. Thanks for doing this.
>>
>>
>>
>> > And I highly suspect that this particular MapReduce job hurts HBase
>> > Regionserver.
>> >
>> > PS. I also use gcviewer to parse the GC-log, I only see around 30
>> seconds
>> > pauses maximum.
>> >
>>
>>
>> Did you figure why your ten minute session is not applying?
>>
>> St.Ack
>>
>>
>>
>> >
>> > Thanks a lot.
>> > zhenyu
>> >
>> >
>> >
>> > On Sun, Nov 15, 2009 at 12:52 AM, Zhenyu Zhong <zhongresearch@gmail.com
>> > >wrote:
>> >
>> > > J-D,
>> > >
>> > > Thank you very much for your comments.
>> > > My company block the IRC port, so I have trouble to connect to IRC
>> > channel.
>> > > I have been trying to ask the IT to open the IRC port for me, it might
>> > take
>> > > a while.
>> > >
>> > > Best,
>> > > zhenyu
>> > >
>> > >
>> > > On Sat, Nov 14, 2009 at 2:21 PM, Jean-Daniel Cryans <
>> jdcryans@apache.org
>> > >wrote:
>> > >
>> > >> The error you are getting is a disconnection from a zookeeper server
>> > >> and is very generic.
>> > >>
>> > >> ZK-86 is still opened and the last comment refers to ZK-111 saying
>> > >> that the bug (in unit tests) was probably fixed in release 3.0.0 last
>> > >> year.
>> > >>
>> > >> To figure the hang you have, you can try to jstack the process pid
>> and
>> > >> see exactly what's holding the RS from shutting down.
>> > >>
>> > >> Would it be possible for you to drop by the IRC channel? This way we
>> > >> can debug this a at much faster pace.
>> > >>
>> > >> Thx!
>> > >>
>> > >> J-D
>> > >>
>> > >> On Sat, Nov 14, 2009 at 9:57 AM, Zhenyu Zhong <
>> zhongresearch@gmail.com>
>> > >> wrote:
>> > >> > I found this.
>> > >> > http://issues.apache.org/jira/browse/ZOOKEEPER-86
>> > >> >
>> > >> > It looks like the same error I had. Is it a zookeeper bug? When
>> will
>> > >> HBase
>> > >> > take the zookeeper version 3.3.0?
>> > >> >
>> > >> > thanks
>> > >> > zhenyu
>> > >> >
>> > >> > On Sat, Nov 14, 2009 at 11:31 AM, Zhenyu Zhong <
>> > zhongresearch@gmail.com
>> > >> >wrote:
>> > >> >
>> > >> >> Now I really doubt about the zookeeper, from the log I saw errors
>> > like
>> > >> >> IOException Read Error, while the zookeeper client (regionserver )
>> > >> tried to
>> > >> >> read. But it got disconneted status from zookeeper.
>> > >> >>
>> > >> >> I don't see any load on any zookeeper quorum servers. DiskIO is
>> fine.
>> > >> >>
>> > >> >> Also when the regionserver decides to exit due to the disconnect
>> > status
>> > >> >> from zookeeper, sometimes the regionserver hangs during the
>> exiting.
>> > We
>> > >> can
>> > >> >> still see the HRegionServer process even we don't see it in the
>> > Master
>> > >> web
>> > >> >> interface.
>> > >> >>
>> > >> >> I also notice that there is an zookeeper.retries and
>> zookeeper.pauses
>> > >> >> settings in hbase-default.xml, would that matter ?
>> > >> >>
>> > >> >> thanks
>> > >> >>
>> > >> >> You may see the IOException error in the following log when the
>> > >> >> Regionserver lost connection to the zookeeper.
>> > >> >>
>> > >> >> 2009-11-14 15:58:31,993 INFO
>> > org.apache.hadoop.hbase.regionserver.HLog:
>> > >> >> Roll
>> > >> >>
>> > >>
>> >
>> /hbase/.logs/superpyxis0008.scur.colo,60021,1258179101617/hlog.dat.1258179106921,
>> > >> >> entries=1062964, calcsize=255014936, filesize=139566093. New hlog
>> > >> >>
>> > >>
>> >
>> /hbase/.logs/superpyxis0008.scur.colo,60021,1258179101617/hlog.dat.1258214311993
>> > >> >> 2009-11-14 15:58:39,295 DEBUG org.apache.zookeeper.ClientCnxn: Got
>> > ping
>> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
>> > >> >> 2009-11-14 15:58:52,648 DEBUG org.apache.zookeeper.ClientCnxn: Got
>> > ping
>> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
>> > >> >> 2009-11-14 15:59:06,007 DEBUG org.apache.zookeeper.ClientCnxn: Got
>> > ping
>> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
>> > >> >> 2009-11-14 15:59:19,365 DEBUG org.apache.zookeeper.ClientCnxn: Got
>> > ping
>> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
>> > >> >> 2009-11-14 15:59:32,867 DEBUG org.apache.zookeeper.ClientCnxn: Got
>> > ping
>> > >> >> response for sessionid:0x424f0ed01ea00a6 after 186ms
>> > >> >> 2009-11-14 15:59:46,070 DEBUG org.apache.zookeeper.ClientCnxn: Got
>> > ping
>> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
>> > >> >> 2009-11-14 15:59:59,423 DEBUG org.apache.zookeeper.ClientCnxn: Got
>> > ping
>> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
>> > >> >> 2009-11-14 16:00:12,775 DEBUG org.apache.zookeeper.ClientCnxn: Got
>> > ping
>> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
>> > >> >> 2009-11-14 16:00:28,141 DEBUG org.apache.zookeeper.ClientCnxn: Got
>> > ping
>> > >> >> response for sessionid:0x424f0ed01ea00a6 after 2010ms
>> > >> >> 2009-11-14 16:01:10,378 WARN org.apache.zookeeper.ClientCnxn:
>> > Exception
>> > >> >> closing session 0x424f0ed01ea00a6 to
>> > >> sun.nio.ch.SelectionKeyImpl@39fbb2d6
>> > >> >>
>> > >> >> java.io.IOException: Read error rc = -1
>> > java.nio.DirectByteBuffer[pos=0
>> > >> >> lim=4 cap=4]
>> > >> >> at
>> > >> >>
>> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
>> > >> >> at
>> > >> >>
>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
>> > >> >> 2009-11-14 16:01:10,478 INFO
>> > >> >> org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper
>> > >> event,
>> > >> >> state: Disconnected, type: None, path: null
>> > >> >> 2009-11-14 16:01:11,333 WARN
>> > >> >> org.apache.hadoop.hbase.regionserver.HRegionServer: Attempt=1
>> > >> >>
>> > >> >> org.apache.hadoop.hbase.Leases$LeaseStillHeldException
>> > >> >> at
>> > >> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>> > >> >> Method)
>> > >> >> at
>> > >> >>
>> > >>
>> >
>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>> > >> >> at
>> > >> >>
>> > >>
>> >
>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>> > >> >> at
>> > >> java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>> > >> >> at
>> > >> >>
>> > >>
>> >
>> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
>> > >> >> at
>> > >> >>
>> > >>
>> >
>> org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
>> > >> >> at
>> > >> >>
>> > >>
>> >
>> org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
>> > >> >> at
>> > >> >>
>> > >>
>> >
>> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:571)
>> > >> >> at java.lang.Thread.run(Thread.java:619)
>> > >> >> 2009-11-14 16:01:11,433 INFO org.apache.zookeeper.ClientCnxn:
>> > >> Attempting
>> > >> >> connection to server superpyxis0001/192.168.100.115:2181
>> > >> >> 2009-11-14 16:01:11,433 INFO org.apache.zookeeper.ClientCnxn:
>> Priming
>> > >> >> connection to java.nio.channels.SocketChannel[connected local=/
>> > >> >> 192.168.100.122:59575 remote=superpyxis0001/192.168.100.115:2181]
>> > >> >> 2009-11-14 16:01:11,433 INFO org.apache.zookeeper.ClientCnxn:
>> Server
>> > >> >> connection successful
>> > >> >> 2009-11-14 16:01:11,433 INFO
>> > >> >> org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper
>> > >> event,
>> > >> >> state: Expired, type: None, path: null
>> > >> >> 2009-11-14 16:01:11,433 WARN org.apache.zookeeper.ClientCnxn:
>> > Exception
>> > >> >> closing session 0x424f0ed01ea00a6 to
>> > >> sun.nio.ch.SelectionKeyImpl@6b93d343
>> > >> >> java.io.IOException: Session Expired
>> > >> >> at
>> > >> >>
>> > >>
>> >
>> org.apache.zookeeper.ClientCnxn$SendThread.readConnectResult(ClientCnxn.java:589)
>> > >> >> at
>> > >> >>
>> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:709)
>> > >> >>
>> > >> >> at
>> > >> >>
>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
>> > >> >> 2009-11-14 16:01:11,433 ERROR
>> > >> >> org.apache.hadoop.hbase.regionserver.HRegionServer: ZooKeeper
>> session
>> > >> >> expired
>> > >> >> 2009-11-14 16:01:13,513 INFO org.apache.hadoop.ipc.HBaseServer:
>> IPC
>> > >> Server
>> > >> >> handler 3 on 60021, call put([B@73deb204,
>> > >> >> [Lorg.apache.hadoop.hbase.client.Put;@2179600a) from
>> > >> 192.168.100.132:40728:
>> > >> >> error: java.io.IOException: Server not running, aborting
>> > >> >> java.io.IOException: Server not running, aborting
>> > >> >>
>> > >> >>
>> > >> >>
>> > >> >>
>> > >> >>
>> > >> >> On Sat, Nov 14, 2009 at 1:01 AM, Zhenyu Zhong <
>> > zhongresearch@gmail.com
>> > >> >wrote:
>> > >> >>
>> > >> >>> Stack,
>> > >> >>>
>> > >> >>> Thanks a lot!
>> > >> >>>
>> > >> >>> I found out that the reason why HBase doesn't take the system
>> file
>> > >> >>> descriptor value I set before. I started the HBase using root
>> > instead
>> > >> of the
>> > >> >>> normal hadoop user, while my system configures hadoop with higher
>> > file
>> > >> >>> descriptor value but configures root with 1024 default value.
>> > >> >>>
>> > >> >>>
>> > >> >>> Now my system has a clean start. However, it seems that the
>> > >> >>> zookeeper.session.timeout value doesn't take into effect. I still
>> > >> found
>> > >> >>> around 60 seconds pauses from the disconnected regionserver. I
>> > really
>> > >> don't
>> > >> >>> know why regionserver only times out after 60 seconds instead of
>> 10
>> > >> minutes
>> > >> >>> which I set for zookeeper.session.timeout.
>> > >> >>>
>> > >> >>> Is there any other timeout value coming into play before the
>> actual
>> > >> >>> session times out?
>> > >> >>>
>> > >> >>> zhenyu
>> > >> >>>
>> > >> >>>
>> > >> >>>
>> > >> >>> On Fri, Nov 13, 2009 at 7:08 PM, stack <st...@duboce.net> wrote:
>> > >> >>>
>> > >> >>>> Ok. Lack of file descriptors manifests in all kinds of weird
>> ways.
>> > >> >>>> Hopefully thats it. If not, lets keep digging.
>> > >> >>>> St.Ack
>> > >> >>>>
>> > >> >>>> On Fri, Nov 13, 2009 at 3:44 PM, Zhenyu Zhong <
>> > >> zhongresearch@gmail.com
>> > >> >>>> >wrote:
>> > >> >>>>
>> > >> >>>> > Stack,
>> > >> >>>> >
>> > >> >>>> > You are right, the master started with ulimit -n 1024. It
>> doesn't
>> > >> take
>> > >> >>>> the
>> > >> >>>> > system value.
>> > >> >>>> >
>> > >> >>>> > Regarding to the too many open files, it looks like the same
>> as
>> > the
>> > >> one
>> > >> >>>> J-D
>> > >> >>>> > put up. But I will get the Master start with higher value
>> first
>> > and
>> > >> see
>> > >> >>>> if
>> > >> >>>> > this kind of error goes away.
>> > >> >>>> >
>> > >> >>>> > thanks a lot!
>> > >> >>>> > zhenyu
>> > >> >>>> >
>> > >> >>>> > On Fri, Nov 13, 2009 at 6:02 PM, stack <st...@duboce.net>
>> wrote:
>> > >> >>>> >
>> > >> >>>> > > Does it say
>> > >> >>>> > >
>> > >> >>>> > > ulimit -n 32768
>> > >> >>>> > >
>> > >> >>>> > > ...as the second line in your log file on start of the
>> master?
>> > >> >>>> > >
>> > >> >>>> > > Is the stack trace that complains about too many open files
>> > same
>> > >> as
>> > >> >>>> the
>> > >> >>>> > one
>> > >> >>>> > > in the blog post J-D put up?
>> > >> >>>> > >
>> > >> >>>> > > St.Ack
>> > >> >>>> > >
>> > >> >>>> > >
>> > >> >>>> > > On Fri, Nov 13, 2009 at 1:37 PM, Zhenyu Zhong <
>> > >> >>>> zhongresearch@gmail.com
>> > >> >>>> > > >wrote:
>> > >> >>>> > >
>> > >> >>>> > > > The ulimit file descriptors was set to fs.file-max
>> =1578334,
>> > >> also
>> > >> >>>> in
>> > >> >>>> > the
>> > >> >>>> > > > limits.conf the value was set to 32768.
>> > >> >>>> > > > So these are way higher than the open descriptors for the
>> > >> running
>> > >> >>>> > > > processes.
>> > >> >>>> > > >
>> > >> >>>> > > > thanks
>> > >> >>>> > > > zhenyu
>> > >> >>>> > > >
>> > >> >>>> > > >
>> > >> >>>> > > > On Fri, Nov 13, 2009 at 4:33 PM, Stack <
>> saint.ack@gmail.com>
>> > >> >>>> wrote:
>> > >> >>>> > > >
>> > >> >>>> > > > > You upped the ulimit file descriptors as per the getting
>> > >> started
>> > >> >>>> doc?
>> > >> >>>> > > > >
>> > >> >>>> > > > >
>> > >> >>>> > > > >
>> > >> >>>> > > > > On Nov 13, 2009, at 1:26 PM, Zhenyu Zhong <
>> > >> >>>> zhongresearch@gmail.com>
>> > >> >>>> > > > wrote:
>> > >> >>>> > > > >
>> > >> >>>> > > > > Thanks a lot.
>> > >> >>>> > > > >>
>> > >> >>>> > > > >>
>> > >> >>>> > > > >> Bad news is my kernel is still 2.6.26.
>> > >> >>>> > > > >> But it was not a problem before.
>> > >> >>>> > > > >>
>> > >> >>>> > > > >> Very strange.
>> > >> >>>> > > > >>
>> > >> >>>> > > > >> zhenyu
>> > >> >>>> > > > >>
>> > >> >>>> > > > >> On Fri, Nov 13, 2009 at 4:16 PM, Jean-Daniel Cryans <
>> > >> >>>> > > > jdcryans@apache.org
>> > >> >>>> > > > >> >wrote:
>> > >> >>>> > > > >>
>> > >> >>>> > > > >> Looks like
>> > >> >>>> > > > >>>
>> > >> >>>> > > > >>>
>> > >> >>>> > > >
>> > >> >>>> > >
>> > >> >>>> >
>> > >> >>>>
>> > >>
>> >
>> http://pero.blogs.aprilmayjune.org/2009/01/22/hadoop-and-linux-kernel-2627-epoll-limits/
>> > >> >>>> > > > >>>
>> > >> >>>> > > > >>> J-D
>> > >> >>>> > > > >>>
>> > >> >>>> > > > >>> On Fri, Nov 13, 2009 at 1:12 PM, Zhenyu Zhong <
>> > >> >>>> > > zhongresearch@gmail.com
>> > >> >>>> > > > >
>> > >> >>>> > > > >>> wrote:
>> > >> >>>> > > > >>>
>> > >> >>>> > > > >>>> Hi,
>> > >> >>>> > > > >>>>
>> > >> >>>> > > > >>>>
>> > >> >>>> > > > >>>> After I re-organize the cluster, the experiment ran
>> into
>> > >> >>>> problem
>> > >> >>>> > > > faster
>> > >> >>>> > > > >>>>
>> > >> >>>> > > > >>> than
>> > >> >>>> > > > >>>
>> > >> >>>> > > > >>>> before.
>> > >> >>>> > > > >>>>
>> > >> >>>> > > > >>>> Basically the changes are to use machines with less
>> > >> resources
>> > >> >>>> as
>> > >> >>>> > > > >>>>
>> > >> >>>> > > > >>> zookeeper
>> > >> >>>> > > > >>>
>> > >> >>>> > > > >>>> quorums and machines with more resources as
>> > regionserver.
>> > >> >>>> > > > >>>>
>> > >> >>>> > > > >>>> From the log, I still see the pause around 1 minute.
>> > >> >>>> > > > >>>> I enabled the GC logging, please see
>> > >> >>>> > > > >>>>
>> > >> >>>> > > > >>>> http://pastebin.com/m1d4ce0f1
>> > >> >>>> > > > >>>>
>> > >> >>>> > > > >>>> for details.
>> > >> >>>> > > > >>>> In general I don't see many pauses in the GC.
>> > >> >>>> > > > >>>>
>> > >> >>>> > > > >>>> What is more interesting, 1 hour after the 1st
>> > >> regionserver
>> > >> >>>> > > > >>>> disconnected,
>> > >> >>>> > > > >>>> the master log complained about too many open files.
>> > This
>> > >> >>>> didn't
>> > >> >>>> > > > happen
>> > >> >>>> > > > >>>> before.
>> > >> >>>> > > > >>>> I checked the system OS setup as well as the
>> > limits.conf.
>> > >> I
>> > >> >>>> also
>> > >> >>>> > > > checked
>> > >> >>>> > > > >>>>
>> > >> >>>> > > > >>> the
>> > >> >>>> > > > >>>
>> > >> >>>> > > > >>>> running processes. The total open files don't reach
>> the
>> > >> limit.
>> > >> >>>> I
>> > >> >>>> > am
>> > >> >>>> > > > >>>>
>> > >> >>>> > > > >>> confused
>> > >> >>>> > > > >>>
>> > >> >>>> > > > >>>> a bit.
>> > >> >>>> > > > >>>>
>> > >> >>>> > > > >>>>
>> > >> >>>> > > > >>>> Please see the following master log.
>> > >> >>>> > > > >>>>
>> > >> >>>> > > > >>>> 2009-11-13 20:06:02,114 INFO
>> > >> >>>> > > > org.apache.hadoop.hbase.master.BaseScanner:
>> > >> >>>> > > > >>>> RegionManager.metaScanner scan of 4658 row(s) of meta
>> > >> region
>> > >> >>>> > > {server:
>> > >> >>>> > > > >>>> 192.168.100.128:60021, regionname: .META.,,1,
>> startKey:
>> > >> <>}
>> > >> >>>> > > complete
>> > >> >>>> > > > >>>> 2009-11-13 20:06:02,114 INFO
>> > >> >>>> > > > org.apache.hadoop.hbase.master.BaseScanner:
>> > >> >>>> > > > >>>>
>> > >> >>>> > > > >>> All
>> > >> >>>> > > > >>>
>> > >> >>>> > > > >>>> 1 .META. region(s) scanned
>> > >> >>>> > > > >>>> 2009-11-13 20:06:07,677 DEBUG
>> > >> org.apache.zookeeper.ClientCnxn:
>> > >> >>>> Got
>> > >> >>>> > > > ping
>> > >> >>>> > > > >>>> response for sessionid:0x424eebf1c10004c after 3ms
>> > >> >>>> > > > >>>> 2009-11-13 20:06:08,178 INFO
>> > >> org.apache.hadoop.hdfs.DFSClient:
>> > >> >>>> > > > Exception
>> > >> >>>> > > > >>>>
>> > >> >>>> > > > >>> in
>> > >> >>>> > > > >>>
>> > >> >>>> > > > >>>> createBlockOutputStream java.io.IOException: Bad
>> connect
>> > >> ack
>> > >> >>>> with
>> > >> >>>> > > > >>>> firstBadLink 192.168.100.123:50010
>> > >> >>>> > > > >>>> 2009-11-13 20:06:08,178 INFO
>> > >> org.apache.hadoop.hdfs.DFSClient:
>> > >> >>>> > > > >>>> Abandoning
>> > >> >>>> > > > >>>> block blk_-2808245019291145247_5478039
>> > >> >>>> > > > >>>> 2009-11-13 20:06:09,682 INFO
>> > >> org.apache.hadoop.hdfs.DFSClient:
>> > >> >>>> > > > Exception
>> > >> >>>> > > > >>>>
>> > >> >>>> > > > >>> in
>> > >> >>>> > > > >>>
>> > >> >>>> > > > >>>> createBlockOutputStream java.io.EOFException
>> > >> >>>> > > > >>>> 2009-11-13 20:06:09,682 INFO
>> > >> org.apache.hadoop.hdfs.DFSClient:
>> > >> >>>> > > > >>>> Abandoning
>> > >> >>>> > > > >>>> block blk_1074853606841896259_5478048
>> > >> >>>> > > > >>>> 2009-11-13 20:06:10,334 DEBUG
>> > >> org.apache.zookeeper.ClientCnxn:
>> > >> >>>> Got
>> > >> >>>> > > > ping
>> > >> >>>> > > > >>>> response for sessionid:0x24eebf1043003c after 1ms
>> > >> >>>> > > > >>>> 2009-11-13 20:06:21,018 DEBUG
>> > >> org.apache.zookeeper.ClientCnxn:
>> > >> >>>> Got
>> > >> >>>> > > > ping
>> > >> >>>> > > > >>>> response for sessionid:0x424eebf1c10004c after 0ms
>> > >> >>>> > > > >>>> 2009-11-13 20:06:23,674 DEBUG
>> > >> org.apache.zookeeper.ClientCnxn:
>> > >> >>>> Got
>> > >> >>>> > > > ping
>> > >> >>>> > > > >>>> response for sessionid:0x24eebf1043003c after 0ms
>> > >> >>>> > > > >>>> 2009-11-13 20:06:24,828 INFO
>> > >> org.apache.hadoop.hdfs.DFSClient:
>> > >> >>>> > > > Exception
>> > >> >>>> > > > >>>>
>> > >> >>>> > > > >>> in
>> > >> >>>> > > > >>>
>> > >> >>>> > > > >>>> createBlockOutputStream java.io.IOException: Bad
>> connect
>> > >> ack
>> > >> >>>> with
>> > >> >>>> > > > >>>> firstBadLink 192.168.100.123:50010
>> > >> >>>> > > > >>>> 2009-11-13 20:06:24,828 INFO
>> > >> org.apache.hadoop.hdfs.DFSClient:
>> > >> >>>> > > > >>>> Abandoning
>> > >> >>>> > > > >>>> block blk_-6642544517082142289_5478063
>> > >> >>>> > > > >>>> 2009-11-13 20:06:24,828 INFO
>> > >> org.apache.hadoop.hdfs.DFSClient:
>> > >> >>>> > > > Exception
>> > >> >>>> > > > >>>>
>> > >> >>>> > > > >>> in
>> > >> >>>> > > > >>>
>> > >> >>>> > > > >>>> createBlockOutputStream java.net.SocketException: Too
>> > many
>> > >> >>>> open
>> > >> >>>> > > files
>> > >> >>>> > > > >>>> 2009-11-13 20:06:24,828 INFO
>> > >> org.apache.hadoop.hdfs.DFSClient:
>> > >> >>>> > > > >>>> Abandoning
>> > >> >>>> > > > >>>> block blk_2057511041109796090_5478063
>> > >> >>>> > > > >>>> 2009-11-13 20:06:24,928 INFO
>> > >> org.apache.hadoop.hdfs.DFSClient:
>> > >> >>>> > > > Exception
>> > >> >>>> > > > >>>>
>> > >> >>>> > > > >>> in
>> > >> >>>> > > > >>>
>> > >> >>>> > > > >>>> createBlockOutputStream java.net.SocketException: Too
>> > many
>> > >> >>>> open
>> > >> >>>> > > files
>> > >> >>>> > > > >>>> 2009-11-13 20:06:24,928 INFO
>> > >> org.apache.hadoop.hdfs.DFSClient:
>> > >> >>>> > > > >>>> Abandoning
>> > >> >>>> > > > >>>> block blk_8219260302213892894_5478064
>> > >> >>>> > > > >>>> 2009-11-13 20:06:30,855 INFO
>> > >> org.apache.hadoop.hdfs.DFSClient:
>> > >> >>>> > > > Exception
>> > >> >>>> > > > >>>>
>> > >> >>>> > > > >>> in
>> > >> >>>> > > > >>>
>> > >> >>>> > > > >>>> createBlockOutputStream java.net.SocketException: Too
>> > many
>> > >> >>>> open
>> > >> >>>> > > files
>> > >> >>>> > > > >>>> 2009-11-13 20:06:30,855 INFO
>> > >> org.apache.hadoop.hdfs.DFSClient:
>> > >> >>>> > > > >>>> Abandoning
>> > >> >>>> > > > >>>> block blk_1669205542853067709_5478235
>> > >> >>>> > > > >>>> 2009-11-13 20:06:30,905 INFO
>> > >> org.apache.hadoop.hdfs.DFSClient:
>> > >> >>>> > > > Exception
>> > >> >>>> > > > >>>>
>> > >> >>>> > > > >>> in
>> > >> >>>> > > > >>>
>> > >> >>>> > > > >>>> createBlockOutputStream java.net.SocketException: Too
>> > many
>> > >> >>>> open
>> > >> >>>> > > files
>> > >> >>>> > > > >>>> 2009-11-13 20:06:30,905 INFO
>> > >> org.apache.hadoop.hdfs.DFSClient:
>> > >> >>>> > > > >>>> Abandoning
>> > >> >>>> > > > >>>> block blk_9128897691346270351_5478237
>> > >> >>>> > > > >>>> 2009-11-13 20:06:30,955 INFO
>> > >> org.apache.hadoop.hdfs.DFSClient:
>> > >> >>>> > > > Exception
>> > >> >>>> > > > >>>>
>> > >> >>>> > > > >>> in
>> > >> >>>> > > > >>>
>> > >> >>>> > > > >>>> createBlockOutputStream java.net.SocketException: Too
>> > many
>> > >> >>>> open
>> > >> >>>> > > files
>> > >> >>>> > > > >>>> 2009-11-13 20:06:30,955 INFO
>> > >> org.apache.hadoop.hdfs.DFSClient:
>> > >> >>>> > > > >>>> Abandoning
>> > >> >>>> > > > >>>> block blk_1116845144864123018_5478240
>> > >> >>>> > > > >>>> 2009-11-13 20:06:34,372 DEBUG
>> > >> org.apache.zookeeper.ClientCnxn:
>> > >> >>>> Got
>> > >> >>>> > > > ping
>> > >> >>>> > > > >>>> response for sessionid:0x424eebf1c10004c after 0ms
>> > >> >>>> > > > >>>> 2009-11-13 20:06:37,034 DEBUG
>> > >> org.apache.zookeeper.ClientCnxn:
>> > >> >>>> Got
>> > >> >>>> > > > ping
>> > >> >>>> > > > >>>> response for sessionid:0x24eebf1043003c after 0ms
>> > >> >>>> > > > >>>> 2009-11-13 20:06:37,235 WARN
>> > >> org.apache.hadoop.hdfs.DFSClient:
>> > >> >>>> > > > >>>>
>> > >> >>>> > > > >>> DataStreamer
>> > >> >>>> > > > >>>
>> > >> >>>> > > > >>>> Exception: java.io.IOException: Too many open files
>> > >> >>>> > > > >>>> at sun.nio.ch.IOUtil.initPipe(Native Method)
>> > >> >>>> > > > >>>> at
>> > >> >>>> > >
>> sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:49)
>> > >> >>>> > > > >>>> at
>> > >> >>>> > > > >>>>
>> > >> >>>> > > > >>>>
>> > >> >>>> > > >
>> > >> >>>> > >
>> > >> >>>> >
>> > >> >>>>
>> > >>
>> >
>> sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:18)
>> > >> >>>> > > > >>>
>> > >> >>>> > > > >>>> at
>> > >> >>>> > > > >>>>
>> > >> >>>> > > > >>>>
>> > >> >>>> > > >
>> > >> >>>> > >
>> > >> >>>> >
>> > >> >>>>
>> > >>
>> >
>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.get(SocketIOWithTimeout.java:407)
>> > >> >>>> > > > >>>
>> > >> >>>> > > > >>>> at
>> > >> >>>> > > > >>>>
>> > >> >>>> > > > >>>>
>> > >> >>>> > > >
>> > >> >>>> > >
>> > >> >>>> >
>> > >> >>>>
>> > >>
>> >
>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:322)
>> > >> >>>> > > > >>>
>> > >> >>>> > > > >>>> at
>> > >> >>>> > > > >>>>
>> > >> >>>> > > > >>>>
>> > >> >>>> > > >
>> > >> >>>> > >
>> > >> >>>> >
>> > >> >>>>
>> > >>
>> >
>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
>> > >> >>>> > > > >>>
>> > >> >>>> > > > >>>> at
>> > >> >>>> > > > >>>>
>> > >> >>>> > > > >>>>
>> > >> >>>> > > >
>> > >> >>>> > >
>> > >> >>>> >
>> > >> >>>>
>> > >>
>> >
>> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
>> > >> >>>> > > > >>>
>> > >> >>>> > > > >>>> at
>> > >> >>>> > > > >>>>
>> > >> >>>> > > > >>>>
>> > >> >>>> > > >
>> > >> >>>> > >
>> > >> >>>> >
>> > >> >>>>
>> > >>
>> >
>> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
>> > >> >>>> > > > >>>
>> > >> >>>> > > > >>>> at
>> > >> >>>> > > > >>>>
>> > >> >>>> > > > >>>
>> > >> >>>>
>> java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
>> > >> >>>> > > > >>>
>> > >> >>>> > > > >>>> at
>> > >> >>>> java.io.DataOutputStream.write(DataOutputStream.java:90)
>> > >> >>>> > > > >>>> at
>> > >> >>>> > > > >>>>
>> > >> >>>> > > > >>>>
>> > >> >>>> > > >
>> > >> >>>> > >
>> > >> >>>> >
>> > >> >>>>
>> > >>
>> >
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2290)
>> > >> >>>> > > > >>>
>> > >> >>>> > > > >>>>
>> > >> >>>> > > > >>>> 2009-11-13 20:06:37,235 WARN
>> > >> org.apache.hadoop.hdfs.DFSClient:
>> > >> >>>> > Error
>> > >> >>>> > > > >>>> Recovery for block blk_8148813491785406356_5478475
>> bad
>> > >> >>>> datanode[0]
>> > >> >>>> > > > >>>> 192.168.100.123:50010
>> > >> >>>> > > > >>>> 2009-11-13 20:06:37,235 WARN
>> > >> org.apache.hadoop.hdfs.DFSClient:
>> > >> >>>> > Error
>> > >> >>>> > > > >>>> Recovery for block blk_8148813491785406356_5478475 in
>> > >> pipeline
>> > >> >>>> > > > >>>> 192.168.100.123:50010, 192.168.100.134:50010,
>> > >> >>>> > 192.168.100.122:50010
>> > >> >>>> > > :
>> > >> >>>> > > > >>>> bad
>> > >> >>>> > > > >>>> datanode 192.168.100.123:50010
>> > >> >>>> > > > >>>> 2009-11-13 20:06:37,436 INFO
>> > >> org.apache.hadoop.hdfs.DFSClient:
>> > >> >>>> > > > Exception
>> > >> >>>> > > > >>>>
>> > >> >>>> > > > >>> in
>> > >> >>>> > > > >>>
>> > >> >>>> > > > >>>> createBlockOutputStream java.net.SocketException: Too
>> > many
>> > >> >>>> open
>> > >> >>>> > > files
>> > >> >>>> > > > >>>> 2009-11-13 20:06:37,436 INFO
>> > >> org.apache.hadoop.hdfs.DFSClient:
>> > >> >>>> > > > >>>> Abandoning
>> > >> >>>> > > > >>>> block blk_2119727700857186236_5478498
>> > >> >>>> > > > >>>>
>> > >> >>>> > > > >>>>
>> > >> >>>> > > > >>>>
>> > >> >>>> > > > >>>>
>> > >> >>>> > > > >>>>
>> > >> >>>> > > > >>>> On Thu, Nov 12, 2009 at 4:21 PM, Zhenyu Zhong <
>> > >> >>>> > > > zhongresearch@gmail.com
>> > >> >>>> > > > >>>> wrote:
>> > >> >>>> > > > >>>>
>> > >> >>>> > > > >>>> Will do.
>> > >> >>>> > > > >>>>>
>> > >> >>>> > > > >>>>> thanks
>> > >> >>>> > > > >>>>> zhenyu
>> > >> >>>> > > > >>>>>
>> > >> >>>> > > > >>>>>
>> > >> >>>> > > > >>>>> On Thu, Nov 12, 2009 at 3:33 PM, stack <
>> > stack@duboce.net
>> > >> >
>> > >> >>>> wrote:
>> > >> >>>> > > > >>>>>
>> > >> >>>> > > > >>>>> Enable GC logging too on this next run (see
>> > >> hbase-env.sh).
>> > >> >>>> Lets
>> > >> >>>> > > try
>> > >> >>>> > > > >>>>>>
>> > >> >>>> > > > >>>>> and
>> > >> >>>> > > > >>>
>> > >> >>>> > > > >>>> get
>> > >> >>>> > > > >>>>>> to the bottom of whats going on.
>> > >> >>>> > > > >>>>>> Thanks,
>> > >> >>>> > > > >>>>>> St.Ack
>> > >> >>>> > > > >>>>>>
>> > >> >>>> > > > >>>>>> On Thu, Nov 12, 2009 at 12:29 PM, Zhenyu Zhong <
>> > >> >>>> > > > >>>>>>
>> > >> >>>> > > > >>>>> zhongresearch@gmail.com
>> > >> >>>> > > > >>>
>> > >> >>>> > > > >>>> wrote:
>> > >> >>>> > > > >>>>>>>
>> > >> >>>> > > > >>>>>>
>> > >> >>>> > > > >>>>>> I can switch the boxes that run zookeeper with the
>> > ones
>> > >> >>>> that
>> > >> >>>> > run
>> > >> >>>> > > > >>>>>>> regionservers.
>> > >> >>>> > > > >>>>>>> I will see the results later.
>> > >> >>>> > > > >>>>>>>
>> > >> >>>> > > > >>>>>>> FYI. The node does have the 10 minutes
>> > >> >>>> > zookeeper.session.timeout
>> > >> >>>> > > > >>>>>>>
>> > >> >>>> > > > >>>>>> value
>> > >> >>>> > > > >>>
>> > >> >>>> > > > >>>> in
>> > >> >>>> > > > >>>>>>
>> > >> >>>> > > > >>>>>>> place.
>> > >> >>>> > > > >>>>>>>
>> > >> >>>> > > > >>>>>>> thanks
>> > >> >>>> > > > >>>>>>> zhenyu
>> > >> >>>> > > > >>>>>>>
>> > >> >>>> > > > >>>>>>> On Thu, Nov 12, 2009 at 3:21 PM, stack <
>> > >> stack@duboce.net>
>> > >> >>>> > wrote:
>> > >> >>>> > > > >>>>>>>
>> > >> >>>> > > > >>>>>>> On Thu, Nov 12, 2009 at 11:50 AM, Zhenyu Zhong <
>> > >> >>>> > > > >>>>>>>>
>> > >> >>>> > > > >>>>>>> zhongresearch@gmail.com
>> > >> >>>> > > > >>>>>>
>> > >> >>>> > > > >>>>>>> wrote:
>> > >> >>>> > > > >>>>>>>>>
>> > >> >>>> > > > >>>>>>>>
>> > >> >>>> > > > >>>>>>>>
>> > >> >>>> > > > >>>>>>>>> In my cluster, half of the cluster have 2 disks
>> > 400GB
>> > >> >>>> each
>> > >> >>>> > per
>> > >> >>>> > > > >>>>>>>>>
>> > >> >>>> > > > >>>>>>>> machine,
>> > >> >>>> > > > >>>>>>
>> > >> >>>> > > > >>>>>>> and
>> > >> >>>> > > > >>>>>>>>
>> > >> >>>> > > > >>>>>>>>> half of the cluster have 6 disks per machine.
>> > Maybe
>> > >> we
>> > >> >>>> > should
>> > >> >>>> > > > >>>>>>>>>
>> > >> >>>> > > > >>>>>>>> run
>> > >> >>>> > > > >>>
>> > >> >>>> > > > >>>> zookeeper
>> > >> >>>> > > > >>>>>>>>> on the machines with 2 disks and RS on machines
>> > with
>> > >> 6
>> > >> >>>> disks?
>> > >> >>>> > > > >>>>>>>>>
>> > >> >>>> > > > >>>>>>>>> That would make most sense only in the below, it
>> > >> looks
>> > >> >>>> like
>> > >> >>>> > the
>> > >> >>>> > > > >>>>>>>>>
>> > >> >>>> > > > >>>>>>>> RS
>> > >> >>>> > > > >>>
>> > >> >>>> > > > >>>> that
>> > >> >>>> > > > >>>>>>
>> > >> >>>> > > > >>>>>>> had
>> > >> >>>> > > > >>>>>>>> issue had 4 disks?
>> > >> >>>> > > > >>>>>>>>
>> > >> >>>> > > > >>>>>>>>
>> > >> >>>> > > > >>>>>>>>
>> > >> >>>> > > > >>>>>>>> BTW, the 10 minutes zookeeper.session.timeout
>> has
>> > >> been
>> > >> >>>> set
>> > >> >>>> > > during
>> > >> >>>> > > > >>>>>>>>>
>> > >> >>>> > > > >>>>>>>> the
>> > >> >>>> > > > >>>>>>
>> > >> >>>> > > > >>>>>>> experiment.
>> > >> >>>> > > > >>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>
>> > >> >>>> > > > >>>>>>>> And for sure this node had it in place?
>> > >> >>>> > > > >>>>>>>> St.Ack
>> > >> >>>> > > > >>>>>>>>
>> > >> >>>> > > > >>>>>>>>
>> > >> >>>> > > > >>>>>>>>
>> > >> >>>> > > > >>>>>>>>
>> > >> >>>> > > > >>>>>>>>> thanks
>> > >> >>>> > > > >>>>>>>>> zhenyu
>> > >> >>>> > > > >>>>>>>>>
>> > >> >>>> > > > >>>>>>>>> On Thu, Nov 12, 2009 at 2:08 PM, stack <
>> > >> stack@duboce.net
>> > >> >>>> >
>> > >> >>>> > > wrote:
>> > >> >>>> > > > >>>>>>>>>
>> > >> >>>> > > > >>>>>>>>> On Thu, Nov 12, 2009 at 8:40 AM, Zhenyu Zhong <
>> > >> >>>> > > > >>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>> zhongresearch@gmail.com
>> > >> >>>> > > > >>>>>>>
>> > >> >>>> > > > >>>>>>>> wrote:
>> > >> >>>> > > > >>>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>> Though I experienced 2 regionserver
>> disconnection
>> > >> this
>> > >> >>>> > > > >>>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>> morning,
>> > >> >>>> > > > >>>
>> > >> >>>> > > > >>>> it
>> > >> >>>> > > > >>>>>>
>> > >> >>>> > > > >>>>>>> looks
>> > >> >>>> > > > >>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>> better from the regionserver log. (please see
>> the
>> > >> >>>> following
>> > >> >>>> > > > >>>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>> log)
>> > >> >>>> > > > >>>
>> > >> >>>> > > > >>>>
>> > >> >>>> > > > >>>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>>> http://pastebin.com/m496dbfae
>> > >> >>>> > > > >>>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>>> I checked diskIO for one of the
>> > >> >>>> > regionserver(192.168.100.116)
>> > >> >>>> > > > >>>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>> during
>> > >> >>>> > > > >>>>>>>
>> > >> >>>> > > > >>>>>>>> the
>> > >> >>>> > > > >>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>> time it disconnected.
>> > >> >>>> > > > >>>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>>> Time: 03:04:58 AM
>> > >> >>>> > > > >>>>>>>>>>> Device: tps Blk_read/s
>> Blk_wrtn/s
>> > >> >>>> Blk_read
>> > >> >>>> > > > >>>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>> Blk_wrtn
>> > >> >>>> > > > >>>>>>>>
>> > >> >>>> > > > >>>>>>>>> sda 105.31 5458.83 19503.64
>> > >> >>>> 9043873239
>> > >> >>>> > > > >>>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>> 32312473676
>> > >> >>>> > > > >>>>>>>>
>> > >> >>>> > > > >>>>>>>>> sda1 2.90 6.64 99.25
>> > >> >>>> 10993934
>> > >> >>>> > > > >>>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>> 164433464
>> > >> >>>> > > > >>>>>>>>
>> > >> >>>> > > > >>>>>>>>> sda2 1.72 23.76 30.25
>> > >> >>>> 39365817
>> > >> >>>> > > > >>>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>> 50124008
>> > >> >>>> > > > >>>>>>>>
>> > >> >>>> > > > >>>>>>>>> sda3 0.30 0.38 3.58
>> > >> >>>> 624930
>> > >> >>>> > > > >>>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>> 5923000
>> > >> >>>> > > > >>>>>>>>
>> > >> >>>> > > > >>>>>>>>> sda4 100.39 5428.06 19370.56
>> > >> >>>> 8992888270
>> > >> >>>> > > > >>>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>> 32091993204
>> > >> >>>> > > > >>>>>>>>
>> > >> >>>> > > > >>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>>> Is this high for you? 20k blocks/second
>> would
>> > >> seem to
>> > >> >>>> be
>> > >> >>>> > > high
>> > >> >>>> > > > >>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>> but
>> > >> >>>> > > > >>>
>> > >> >>>> > > > >>>> its
>> > >> >>>> > > > >>>>>>>
>> > >> >>>> > > > >>>>>>>> one
>> > >> >>>> > > > >>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>> disk only and its not being shared by zk
>> anymore
>> > so
>> > >> >>>> > shouldn't
>> > >> >>>> > > > >>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>> matter?
>> > >> >>>> > > > >>>>>>
>> > >> >>>> > > > >>>>>>>
>> > >> >>>> > > > >>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>> I also checked the zookeeper quorum server
>> that
>> > the
>> > >> >>>> > > > >>>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>> regionserver
>> > >> >>>> > > > >>>
>> > >> >>>> > > > >>>> tried
>> > >> >>>> > > > >>>>>>>>
>> > >> >>>> > > > >>>>>>>>> to
>> > >> >>>> > > > >>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>> connect according to the log. However, I don't
>> see
>> > >> >>>> > > > >>>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>> 192.168.100.116
>> > >> >>>> > > > >>>>>>
>> > >> >>>> > > > >>>>>>> in
>> > >> >>>> > > > >>>>>>>
>> > >> >>>> > > > >>>>>>>> the
>> > >> >>>> > > > >>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>> client list of the zookeeper quorum server that
>> > the
>> > >> >>>> > > > >>>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>> regionserver
>> > >> >>>> > > > >>>
>> > >> >>>> > > > >>>> tried
>> > >> >>>> > > > >>>>>>>>
>> > >> >>>> > > > >>>>>>>>> to
>> > >> >>>> > > > >>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>> connect.
>> > >> >>>> > > > >>>>>>>>>>> Would that be a problem?
>> > >> >>>> > > > >>>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>>> Is that because the ephemeral node for the
>> > >> >>>> regionserver
>> > >> >>>> > had
>> > >> >>>> > > > >>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>> evaporated?
>> > >> >>>> > > > >>>>>>>
>> > >> >>>> > > > >>>>>>>> Lost
>> > >> >>>> > > > >>>>>>>>>> its lease w/ zk by the time you went to look?
>> > >> >>>> > > > >>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>> Thu Nov 12 15:04:52 UTC 2009
>> > >> >>>> > > > >>>>>>>>>>> Zookeeper version: 3.2.1-808558, built on
>> > >> 08/27/2009
>> > >> >>>> 18:48
>> > >> >>>> > > > >>>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>> GMT
>> > >> >>>> > > > >>>
>> > >> >>>> > > > >>>> Clients:
>> > >> >>>> > > > >>>>>>>>>>> /192.168.100.127:43045
>> > >> [1](queued=0,recved=26,sent=0)
>> > >> >>>> > > > >>>>>>>>>>> /192.168.100.131:39091
>> > >> [1](queued=0,recved=964,sent=0)
>> > >> >>>> > > > >>>>>>>>>>> /192.168.100.124:35961
>> > >> [1](queued=0,recved=3266,sent=0)
>> > >> >>>> > > > >>>>>>>>>>> /192.168.100.123:47935
>> > >> [1](queued=0,recved=233,sent=0)
>> > >> >>>> > > > >>>>>>>>>>> /192.168.100.125:46931
>> > >> [1](queued=0,recved=2,sent=0)
>> > >> >>>> > > > >>>>>>>>>>> /192.168.100.118:54924
>> > >> [1](queued=0,recved=295,sent=0)
>> > >> >>>> > > > >>>>>>>>>>> /192.168.100.118:41390
>> > >> [1](queued=0,recved=2290,sent=0)
>> > >> >>>> > > > >>>>>>>>>>> /192.168.100.136:42243
>> > >> [1](queued=0,recved=0,sent=0)
>> > >> >>>> > > > >>>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>>> Latency min/avg/max: 0/17/6333
>> > >> >>>> > > > >>>>>>>>>>> Received: 47111
>> > >> >>>> > > > >>>>>>>>>>> Sent: 0
>> > >> >>>> > > > >>>>>>>>>>> Outstanding: 0
>> > >> >>>> > > > >>>>>>>>>>> Zxid: 0x77000083f4
>> > >> >>>> > > > >>>>>>>>>>> Mode: leader
>> > >> >>>> > > > >>>>>>>>>>> Node count: 23
>> > >> >>>> > > > >>>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>> That 6 second maximum latency is pretty bad but
>> > >> should
>> > >> >>>> be
>> > >> >>>> > well
>> > >> >>>> > > > >>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>> within
>> > >> >>>> > > > >>>>>>
>> > >> >>>> > > > >>>>>>> the
>> > >> >>>> > > > >>>>>>>>
>> > >> >>>> > > > >>>>>>>>> zk
>> > >> >>>> > > > >>>>>>>>>> session timeout.
>> > >> >>>> > > > >>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>> So, problem is likely on the zk client side of
>> the
>> > >> >>>> session;
>> > >> >>>> > > > >>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>> i.e.
>> > >> >>>> > > > >>>
>> > >> >>>> > > > >>>> in
>> > >> >>>> > > > >>>>>>
>> > >> >>>> > > > >>>>>>> the
>> > >> >>>> > > > >>>>>>>
>> > >> >>>> > > > >>>>>>>> RS.
>> > >> >>>> > > > >>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>> You could enable GC logging as J-D suggested to
>> > see
>> > >> if
>> > >> >>>> you
>> > >> >>>> > > have
>> > >> >>>> > > > >>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>> any
>> > >> >>>> > > > >>>>>>
>> > >> >>>> > > > >>>>>>> big
>> > >> >>>> > > > >>>>>>>
>> > >> >>>> > > > >>>>>>>> pauses, pauses > zk session timeout.
>> > >> >>>> > > > >>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>> When the RS went down, it didn't look too
>> heavily
>> > >> >>>> loaded:
>> > >> >>>> > > > >>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>> 1. 2009-11-12 15:04:52,830 INFO
>> > >> >>>> > > > >>>>>>>>>>
>> > org.apache.hadoop.hbase.regionserver.HRegionServer:
>> > >> >>>> Dump of
>> > >> >>>> > > > >>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>> metrics:
>> > >> >>>> > > > >>>>>>>
>> > >> >>>> > > > >>>>>>>> request=1.5166667, regions=322, stores=657,
>> > >> >>>> storefiles=631,
>> > >> >>>> > > > >>>>>>>>>> storefileIndexSize=61, memstoreSize=1472,
>> > >> >>>> usedHeap=2819,
>> > >> >>>> > > > >>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>> maxHeap=4079,
>> > >> >>>> > > > >>>>>>>>
>> > >> >>>> > > > >>>>>>>>> blockCacheSize=658110960,
>> > blockCacheFree=197395984,
>> > >> >>>> > > > >>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>> blockCacheCount=9903,
>> > >> >>>> > > > >>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>> blockCacheHitRatio=99
>> > >> >>>> > > > >>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>> Its serving a few reads? The number of store
>> > files
>> > >> >>>> seems
>> > >> >>>> > > fine.
>> > >> >>>> > > > >>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>> Not
>> > >> >>>> > > > >>>>>>
>> > >> >>>> > > > >>>>>>> too
>> > >> >>>> > > > >>>>>>>>
>> > >> >>>> > > > >>>>>>>>> much memory used.
>> > >> >>>> > > > >>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>> Looking at the logs, I see the Lease Still Held
>> > >> >>>> exception.
>> > >> >>>> > > > >>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>> This
>> > >> >>>> > > > >>>
>> > >> >>>> > > > >>>> happens
>> > >> >>>> > > > >>>>>>>>
>> > >> >>>> > > > >>>>>>>>> when the RS does its regular report to the
>> master
>> > but
>> > >> the
>> > >> >>>> > > > >>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>> master
>> > >> >>>> > > > >>>
>> > >> >>>> > > > >>>> thinks
>> > >> >>>> > > > >>>>>>>
>> > >> >>>> > > > >>>>>>>> the
>> > >> >>>> > > > >>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>> RS has since restarted. It'll think this
>> probably
>> > >> >>>> because
>> > >> >>>> > it
>> > >> >>>> > > > >>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>> noticed
>> > >> >>>> > > > >>>>>>
>> > >> >>>> > > > >>>>>>> that
>> > >> >>>> > > > >>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>> the RS's znode in zk had gone away and it
>> > considered
>> > >> the
>> > >> >>>> RS
>> > >> >>>> > > > >>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>> dead.
>> > >> >>>> > > > >>>
>> > >> >>>> > > > >>>>
>> > >> >>>> > > > >>>>>>>>>> Looking too at your logs I see this gap in the
>> zk
>> > >> >>>> pinging:
>> > >> >>>> > > > >>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>> 1. 2009-11-12 15:03:39,325 DEBUG
>> > >> >>>> > > > >>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>> org.apache.zookeeper.ClientCnxn:
>> > >> >>>> > > > >>>>>>
>> > >> >>>> > > > >>>>>>> Got
>> > >> >>>> > > > >>>>>>>
>> > >> >>>> > > > >>>>>>>> ping response for sessionid:0x224e55436ad0004
>> after
>> > >> 0ms
>> > >> >>>> > > > >>>>>>>>>> 2. 2009-11-12 15:03:43,113 DEBUG
>> > >> >>>> > > > >>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>> org.apache.zookeeper.ClientCnxn:
>> > >> >>>> > > > >>>>>>
>> > >> >>>> > > > >>>>>>> Got
>> > >> >>>> > > > >>>>>>>
>> > >> >>>> > > > >>>>>>>> ping response for sessionid:0x24e55436a0007d
>> after
>> > >> 0ms
>> > >> >>>> > > > >>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>> Where in the lines above it, its reporting
>> about
>> > >> every
>> > >> >>>> ten
>> > >> >>>> > > > >>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>> seconds,
>> > >> >>>> > > > >>>>>>
>> > >> >>>> > > > >>>>>>> here
>> > >> >>>> > > > >>>>>>>>
>> > >> >>>> > > > >>>>>>>>> there is a big gap.
>> > >> >>>> > > > >>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>> Do you have ganglia or something that will let
>> you
>> > >> look
>> > >> >>>> more
>> > >> >>>> > > > >>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>> into
>> > >> >>>> > > > >>>
>> > >> >>>> > > > >>>> what
>> > >> >>>> > > > >>>>>>>
>> > >> >>>> > > > >>>>>>>> was
>> > >> >>>> > > > >>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>> happening on this machine around this time? Is
>> > the
>> > >> >>>> machine
>> > >> >>>> > > OK?
>> > >> >>>> > > > >>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>> It
>> > >> >>>> > > > >>>>>>
>> > >> >>>> > > > >>>>>>> looks
>> > >> >>>> > > > >>>>>>>>
>> > >> >>>> > > > >>>>>>>>> lightly loaded and you have your cluster nicely
>> > laid
>> > >> out.
>> > >> >>>> > > > >>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>> Something
>> > >> >>>> > > > >>>>>>
>> > >> >>>> > > > >>>>>>> odd
>> > >> >>>> > > > >>>>>>>>
>> > >> >>>> > > > >>>>>>>>> is
>> > >> >>>> > > > >>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>> going on. What about things like the write
>> speed
>> > to
>> > >> >>>> disk?
>> > >> >>>> > In
>> > >> >>>> > > > >>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>> the
>> > >> >>>> > > > >>>
>> > >> >>>> > > > >>>> past
>> > >> >>>> > > > >>>>>>>
>> > >> >>>> > > > >>>>>>>> strange issues have been explained by incorrectly
>> > set
>> > >> BIOS
>> > >> >>>> > > > >>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>> which
>> > >> >>>> > > > >>>
>> > >> >>>> > > > >>>> made
>> > >> >>>> > > > >>>>>>
>> > >> >>>> > > > >>>>>>> disks
>> > >> >>>> > > > >>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>> run at 1/100th of their proper speed.
>> > >> >>>> > > > >>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>> St.Ack
>> > >> >>>> > > > >>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>>> Best,
>> > >> >>>> > > > >>>>>>>>>>> zhenyu
>> > >> >>>> > > > >>>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>>> On Wed, Nov 11, 2009 at 3:58 PM, Zhenyu Zhong
>> <
>> > >> >>>> > > > >>>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>> zhongresearch@gmail.com
>> > >> >>>> > > > >>>>>>>>
>> > >> >>>> > > > >>>>>>>>> wrote:
>> > >> >>>> > > > >>>>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>>> Stack
>> > >> >>>> > > > >>>>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>>>> I am very appreciated for your comments.
>> > >> >>>> > > > >>>>>>>>>>>> I will use the zookeeper monitoring script on
>> my
>> > >> >>>> cluster
>> > >> >>>> > > > >>>>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>>> and
>> > >> >>>> > > > >>>
>> > >> >>>> > > > >>>> let
>> > >> >>>> > > > >>>>>>
>> > >> >>>> > > > >>>>>>> it
>> > >> >>>> > > > >>>>>>>
>> > >> >>>> > > > >>>>>>>> run
>> > >> >>>> > > > >>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>> overnight to see the result.
>> > >> >>>> > > > >>>>>>>>>>>> I will follow up the thread when I get
>> anything.
>> > >> >>>> > > > >>>>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>>>> thanks
>> > >> >>>> > > > >>>>>>>>>>>> zhenyu
>> > >> >>>> > > > >>>>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>>>> On Wed, Nov 11, 2009 at 3:52 PM, stack <
>> > >> >>>> stack@duboce.net>
>> > >> >>>> > > > >>>>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>>> wrote:
>> > >> >>>> > > > >>>>>>
>> > >> >>>> > > > >>>>>>>
>> > >> >>>> > > > >>>>>>>>>>>> I see these in your log too:
>> > >> >>>> > > > >>>>>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>>>>> 1. 2009-11-11 04:27:19,018 DEBUG
>> > >> >>>> > > > >>>>>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
>> > >> >>>> > > > >>>>>>>>
>> > >> >>>> > > > >>>>>>>>> Got
>> > >> >>>> > > > >>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>>> ping response for sessionid:0x424dfd908c50009
>> > >> after
>> > >> >>>> > > > >>>>>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>>>> 4544ms
>> > >> >>>> > > > >>>
>> > >> >>>> > > > >>>> 2. 2009-11-11 04:27:19,018 DEBUG
>> > >> >>>> > > > >>>>>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
>> > >> >>>> > > > >>>>>>>>
>> > >> >>>> > > > >>>>>>>>> Got
>> > >> >>>> > > > >>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>>> ping response for sessionid:0x24dfd90c810002
>> > after
>> > >> >>>> > > > >>>>>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>>>> 4548ms
>> > >> >>>> > > > >>>
>> > >> >>>> > > > >>>> 3. 2009-11-11 04:27:43,960 DEBUG
>> > >> >>>> > > > >>>>>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
>> > >> >>>> > > > >>>>>>>>
>> > >> >>>> > > > >>>>>>>>> Got
>> > >> >>>> > > > >>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>>> ping response for sessionid:0x424dfd908c50009
>> > >> after
>> > >> >>>> > > > >>>>>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>>>> 9030ms
>> > >> >>>> > > > >>>
>> > >> >>>> > > > >>>> 4. 2009-11-11 04:27:43,960 DEBUG
>> > >> >>>> > > > >>>>>>>>>>>>>
>> > >> >>>> > > > >>>>>>>>>>>> org.apache.zookeeper.
>> > >> >>>> > > > >>>>>>>>
>> > >> >>>> > > > >>>>>>>
>> > >> >>>> > > >
>> > >> >>>> > >
>> > >> >>>> >
>> > >> >>>>
>> > >> >>>
>> > >> >>>
>> > >> >>
>> > >> >
>> > >>
>> > >
>> > >
>> >
>>
>
>
Re: regionserver disconnection
Posted by Zhenyu Zhong <zh...@gmail.com>.
Here is the diskIO and CPU around the time we had RS disconnection on one
machine that runs RegionServer. It doesn't seem to be high. Similar disk and
cpu usage have been seen before and the HBase was running fine.
So far I haven't found why my 10 minutes session timeout doesn't apply.
Still digging.
Many thanks!
Time: 03:01:08 PM
avg-cpu: %user %nice %system %iowait %steal %idle
3.82 0.00 0.89 3.97 0.00 91.32
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 24.30 1605.39 6679.35 6238623388 25956251720
sda1 0.51 1.94 15.71 7547265 61067904
sda2 0.13 2.11 5.20 8202079 20189232
sda3 0.13 0.25 1.63 987846 6323336
sda4 0.00 0.00 0.00 4 0
sda5 1.78 0.06 24.14 220010 93817208
sda6 0.38 0.42 8.41 1630584 32688152
sda7 21.38 1600.61 6624.26 6220035272 25742165888
sdb 4.52 767.33 565.91 2981868690 2199132380
sdb1 4.52 767.33 565.91 2981866755 2199132372
sdc 4.42 742.95 563.00 2887151482 2187823092
sdc1 4.42 742.95 563.00 2887149547 2187823084
sdd 4.49 750.78 557.25 2917554074 2165513500
sdd1 4.49 750.78 557.25 2917552139 2165513492
sde 4.52 758.51 569.46 2947593394 2212964236
sde1 4.52 758.51 569.46 2947591459 2212964228
sdf 4.51 747.22 571.78 2903740266 2221972652
sdf1 4.51 747.22 571.78 2903738331 2221972644
Best,
zhenyu
On Mon, Nov 16, 2009 at 4:07 PM, stack <st...@duboce.net> wrote:
> On Mon, Nov 16, 2009 at 12:05 PM, Zhenyu Zhong <zhongresearch@gmail.com
> >wrote:
>
> > I just realized that there was a MapReduce job running during the time
> the
> > regionserver disconnected from the zookeeper.
> > That MapReduce Job was processing 500GB data and took about 8 minutes to
> > finish. It launched over 2000 map tasks.
>
>
> There was a tasktracker running on the RegionServer that disconnected? Is
> the map i/o or cpu heavy? Do you think it could have stole life from the
> datanode/regionserver?
>
>
>
> > I doubt that this introduced
> > resource contention between DataNode and RegionServer Node.
> > Also during the time that MapReduce job ran, I saw a few errors
> indicating
> > that
> >
> > java.io.IOException: Could not obtain block:
> > blk_7661247556283580226_4450462 file=/data/xxx.txt.200910141159
> > at
> >
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1787)
> > at
> >
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1615)
> > at
> > org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1742)
> > at java.io.DataInputStream.read(DataInputStream.java:83)
> > at org.apache.hadoop.util.LineReader.readLine(LineReader.java:134)
> > at
> > org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:136)
> > at
> > org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:40)
> > at
> >
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:192)
> > at
> >
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:176)
> > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
> > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
> > at org.apache.hadoop.mapred.Child.main(Child.java:170)
> >
>
> This is interesting. Your map task is running into hdfs-127. Yeah, you
> should patch your cluster if you want to get rid of these (hdfs-127 has
> been
> applied to 0.21 hadoop and will be in the next hadoop release on 0.20.x
> branch, hadoop 0.20.2).
>
>
>
> >
> > Possibly it is related to HDFS-127, but I don't see any datanodes went
> > down. Does that affect Regionserver? Shall we apply the patch?
> >
> >
> Read up on hdfs-127 for explaination of whats going on.
>
>
>
> > Now I start to keep tracking of the virtual memory stats to see if the
> same
> > issue happens tomorrow around the same time period.
> >
>
> Excellent. Thanks for doing this.
>
>
>
> > And I highly suspect that this particular MapReduce job hurts HBase
> > Regionserver.
> >
> > PS. I also use gcviewer to parse the GC-log, I only see around 30 seconds
> > pauses maximum.
> >
>
>
> Did you figure why your ten minute session is not applying?
>
> St.Ack
>
>
>
> >
> > Thanks a lot.
> > zhenyu
> >
> >
> >
> > On Sun, Nov 15, 2009 at 12:52 AM, Zhenyu Zhong <zhongresearch@gmail.com
> > >wrote:
> >
> > > J-D,
> > >
> > > Thank you very much for your comments.
> > > My company block the IRC port, so I have trouble to connect to IRC
> > channel.
> > > I have been trying to ask the IT to open the IRC port for me, it might
> > take
> > > a while.
> > >
> > > Best,
> > > zhenyu
> > >
> > >
> > > On Sat, Nov 14, 2009 at 2:21 PM, Jean-Daniel Cryans <
> jdcryans@apache.org
> > >wrote:
> > >
> > >> The error you are getting is a disconnection from a zookeeper server
> > >> and is very generic.
> > >>
> > >> ZK-86 is still opened and the last comment refers to ZK-111 saying
> > >> that the bug (in unit tests) was probably fixed in release 3.0.0 last
> > >> year.
> > >>
> > >> To figure the hang you have, you can try to jstack the process pid and
> > >> see exactly what's holding the RS from shutting down.
> > >>
> > >> Would it be possible for you to drop by the IRC channel? This way we
> > >> can debug this a at much faster pace.
> > >>
> > >> Thx!
> > >>
> > >> J-D
> > >>
> > >> On Sat, Nov 14, 2009 at 9:57 AM, Zhenyu Zhong <
> zhongresearch@gmail.com>
> > >> wrote:
> > >> > I found this.
> > >> > http://issues.apache.org/jira/browse/ZOOKEEPER-86
> > >> >
> > >> > It looks like the same error I had. Is it a zookeeper bug? When will
> > >> HBase
> > >> > take the zookeeper version 3.3.0?
> > >> >
> > >> > thanks
> > >> > zhenyu
> > >> >
> > >> > On Sat, Nov 14, 2009 at 11:31 AM, Zhenyu Zhong <
> > zhongresearch@gmail.com
> > >> >wrote:
> > >> >
> > >> >> Now I really doubt about the zookeeper, from the log I saw errors
> > like
> > >> >> IOException Read Error, while the zookeeper client (regionserver )
> > >> tried to
> > >> >> read. But it got disconneted status from zookeeper.
> > >> >>
> > >> >> I don't see any load on any zookeeper quorum servers. DiskIO is
> fine.
> > >> >>
> > >> >> Also when the regionserver decides to exit due to the disconnect
> > status
> > >> >> from zookeeper, sometimes the regionserver hangs during the
> exiting.
> > We
> > >> can
> > >> >> still see the HRegionServer process even we don't see it in the
> > Master
> > >> web
> > >> >> interface.
> > >> >>
> > >> >> I also notice that there is an zookeeper.retries and
> zookeeper.pauses
> > >> >> settings in hbase-default.xml, would that matter ?
> > >> >>
> > >> >> thanks
> > >> >>
> > >> >> You may see the IOException error in the following log when the
> > >> >> Regionserver lost connection to the zookeeper.
> > >> >>
> > >> >> 2009-11-14 15:58:31,993 INFO
> > org.apache.hadoop.hbase.regionserver.HLog:
> > >> >> Roll
> > >> >>
> > >>
> >
> /hbase/.logs/superpyxis0008.scur.colo,60021,1258179101617/hlog.dat.1258179106921,
> > >> >> entries=1062964, calcsize=255014936, filesize=139566093. New hlog
> > >> >>
> > >>
> >
> /hbase/.logs/superpyxis0008.scur.colo,60021,1258179101617/hlog.dat.1258214311993
> > >> >> 2009-11-14 15:58:39,295 DEBUG org.apache.zookeeper.ClientCnxn: Got
> > ping
> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
> > >> >> 2009-11-14 15:58:52,648 DEBUG org.apache.zookeeper.ClientCnxn: Got
> > ping
> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
> > >> >> 2009-11-14 15:59:06,007 DEBUG org.apache.zookeeper.ClientCnxn: Got
> > ping
> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
> > >> >> 2009-11-14 15:59:19,365 DEBUG org.apache.zookeeper.ClientCnxn: Got
> > ping
> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
> > >> >> 2009-11-14 15:59:32,867 DEBUG org.apache.zookeeper.ClientCnxn: Got
> > ping
> > >> >> response for sessionid:0x424f0ed01ea00a6 after 186ms
> > >> >> 2009-11-14 15:59:46,070 DEBUG org.apache.zookeeper.ClientCnxn: Got
> > ping
> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
> > >> >> 2009-11-14 15:59:59,423 DEBUG org.apache.zookeeper.ClientCnxn: Got
> > ping
> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
> > >> >> 2009-11-14 16:00:12,775 DEBUG org.apache.zookeeper.ClientCnxn: Got
> > ping
> > >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
> > >> >> 2009-11-14 16:00:28,141 DEBUG org.apache.zookeeper.ClientCnxn: Got
> > ping
> > >> >> response for sessionid:0x424f0ed01ea00a6 after 2010ms
> > >> >> 2009-11-14 16:01:10,378 WARN org.apache.zookeeper.ClientCnxn:
> > Exception
> > >> >> closing session 0x424f0ed01ea00a6 to
> > >> sun.nio.ch.SelectionKeyImpl@39fbb2d6
> > >> >>
> > >> >> java.io.IOException: Read error rc = -1
> > java.nio.DirectByteBuffer[pos=0
> > >> >> lim=4 cap=4]
> > >> >> at
> > >> >>
> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
> > >> >> at
> > >> >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
> > >> >> 2009-11-14 16:01:10,478 INFO
> > >> >> org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper
> > >> event,
> > >> >> state: Disconnected, type: None, path: null
> > >> >> 2009-11-14 16:01:11,333 WARN
> > >> >> org.apache.hadoop.hbase.regionserver.HRegionServer: Attempt=1
> > >> >>
> > >> >> org.apache.hadoop.hbase.Leases$LeaseStillHeldException
> > >> >> at
> > >> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> > >> >> Method)
> > >> >> at
> > >> >>
> > >>
> >
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> > >> >> at
> > >> >>
> > >>
> >
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> > >> >> at
> > >> java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> > >> >> at
> > >> >>
> > >>
> >
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
> > >> >> at
> > >> >>
> > >>
> >
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
> > >> >> at
> > >> >>
> > >>
> >
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
> > >> >> at
> > >> >>
> > >>
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:571)
> > >> >> at java.lang.Thread.run(Thread.java:619)
> > >> >> 2009-11-14 16:01:11,433 INFO org.apache.zookeeper.ClientCnxn:
> > >> Attempting
> > >> >> connection to server superpyxis0001/192.168.100.115:2181
> > >> >> 2009-11-14 16:01:11,433 INFO org.apache.zookeeper.ClientCnxn:
> Priming
> > >> >> connection to java.nio.channels.SocketChannel[connected local=/
> > >> >> 192.168.100.122:59575 remote=superpyxis0001/192.168.100.115:2181]
> > >> >> 2009-11-14 16:01:11,433 INFO org.apache.zookeeper.ClientCnxn:
> Server
> > >> >> connection successful
> > >> >> 2009-11-14 16:01:11,433 INFO
> > >> >> org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper
> > >> event,
> > >> >> state: Expired, type: None, path: null
> > >> >> 2009-11-14 16:01:11,433 WARN org.apache.zookeeper.ClientCnxn:
> > Exception
> > >> >> closing session 0x424f0ed01ea00a6 to
> > >> sun.nio.ch.SelectionKeyImpl@6b93d343
> > >> >> java.io.IOException: Session Expired
> > >> >> at
> > >> >>
> > >>
> >
> org.apache.zookeeper.ClientCnxn$SendThread.readConnectResult(ClientCnxn.java:589)
> > >> >> at
> > >> >>
> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:709)
> > >> >>
> > >> >> at
> > >> >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
> > >> >> 2009-11-14 16:01:11,433 ERROR
> > >> >> org.apache.hadoop.hbase.regionserver.HRegionServer: ZooKeeper
> session
> > >> >> expired
> > >> >> 2009-11-14 16:01:13,513 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> > >> Server
> > >> >> handler 3 on 60021, call put([B@73deb204,
> > >> >> [Lorg.apache.hadoop.hbase.client.Put;@2179600a) from
> > >> 192.168.100.132:40728:
> > >> >> error: java.io.IOException: Server not running, aborting
> > >> >> java.io.IOException: Server not running, aborting
> > >> >>
> > >> >>
> > >> >>
> > >> >>
> > >> >>
> > >> >> On Sat, Nov 14, 2009 at 1:01 AM, Zhenyu Zhong <
> > zhongresearch@gmail.com
> > >> >wrote:
> > >> >>
> > >> >>> Stack,
> > >> >>>
> > >> >>> Thanks a lot!
> > >> >>>
> > >> >>> I found out that the reason why HBase doesn't take the system file
> > >> >>> descriptor value I set before. I started the HBase using root
> > instead
> > >> of the
> > >> >>> normal hadoop user, while my system configures hadoop with higher
> > file
> > >> >>> descriptor value but configures root with 1024 default value.
> > >> >>>
> > >> >>>
> > >> >>> Now my system has a clean start. However, it seems that the
> > >> >>> zookeeper.session.timeout value doesn't take into effect. I still
> > >> found
> > >> >>> around 60 seconds pauses from the disconnected regionserver. I
> > really
> > >> don't
> > >> >>> know why regionserver only times out after 60 seconds instead of
> 10
> > >> minutes
> > >> >>> which I set for zookeeper.session.timeout.
> > >> >>>
> > >> >>> Is there any other timeout value coming into play before the
> actual
> > >> >>> session times out?
> > >> >>>
> > >> >>> zhenyu
> > >> >>>
> > >> >>>
> > >> >>>
> > >> >>> On Fri, Nov 13, 2009 at 7:08 PM, stack <st...@duboce.net> wrote:
> > >> >>>
> > >> >>>> Ok. Lack of file descriptors manifests in all kinds of weird
> ways.
> > >> >>>> Hopefully thats it. If not, lets keep digging.
> > >> >>>> St.Ack
> > >> >>>>
> > >> >>>> On Fri, Nov 13, 2009 at 3:44 PM, Zhenyu Zhong <
> > >> zhongresearch@gmail.com
> > >> >>>> >wrote:
> > >> >>>>
> > >> >>>> > Stack,
> > >> >>>> >
> > >> >>>> > You are right, the master started with ulimit -n 1024. It
> doesn't
> > >> take
> > >> >>>> the
> > >> >>>> > system value.
> > >> >>>> >
> > >> >>>> > Regarding to the too many open files, it looks like the same as
> > the
> > >> one
> > >> >>>> J-D
> > >> >>>> > put up. But I will get the Master start with higher value first
> > and
> > >> see
> > >> >>>> if
> > >> >>>> > this kind of error goes away.
> > >> >>>> >
> > >> >>>> > thanks a lot!
> > >> >>>> > zhenyu
> > >> >>>> >
> > >> >>>> > On Fri, Nov 13, 2009 at 6:02 PM, stack <st...@duboce.net>
> wrote:
> > >> >>>> >
> > >> >>>> > > Does it say
> > >> >>>> > >
> > >> >>>> > > ulimit -n 32768
> > >> >>>> > >
> > >> >>>> > > ...as the second line in your log file on start of the
> master?
> > >> >>>> > >
> > >> >>>> > > Is the stack trace that complains about too many open files
> > same
> > >> as
> > >> >>>> the
> > >> >>>> > one
> > >> >>>> > > in the blog post J-D put up?
> > >> >>>> > >
> > >> >>>> > > St.Ack
> > >> >>>> > >
> > >> >>>> > >
> > >> >>>> > > On Fri, Nov 13, 2009 at 1:37 PM, Zhenyu Zhong <
> > >> >>>> zhongresearch@gmail.com
> > >> >>>> > > >wrote:
> > >> >>>> > >
> > >> >>>> > > > The ulimit file descriptors was set to fs.file-max
> =1578334,
> > >> also
> > >> >>>> in
> > >> >>>> > the
> > >> >>>> > > > limits.conf the value was set to 32768.
> > >> >>>> > > > So these are way higher than the open descriptors for the
> > >> running
> > >> >>>> > > > processes.
> > >> >>>> > > >
> > >> >>>> > > > thanks
> > >> >>>> > > > zhenyu
> > >> >>>> > > >
> > >> >>>> > > >
> > >> >>>> > > > On Fri, Nov 13, 2009 at 4:33 PM, Stack <
> saint.ack@gmail.com>
> > >> >>>> wrote:
> > >> >>>> > > >
> > >> >>>> > > > > You upped the ulimit file descriptors as per the getting
> > >> started
> > >> >>>> doc?
> > >> >>>> > > > >
> > >> >>>> > > > >
> > >> >>>> > > > >
> > >> >>>> > > > > On Nov 13, 2009, at 1:26 PM, Zhenyu Zhong <
> > >> >>>> zhongresearch@gmail.com>
> > >> >>>> > > > wrote:
> > >> >>>> > > > >
> > >> >>>> > > > > Thanks a lot.
> > >> >>>> > > > >>
> > >> >>>> > > > >>
> > >> >>>> > > > >> Bad news is my kernel is still 2.6.26.
> > >> >>>> > > > >> But it was not a problem before.
> > >> >>>> > > > >>
> > >> >>>> > > > >> Very strange.
> > >> >>>> > > > >>
> > >> >>>> > > > >> zhenyu
> > >> >>>> > > > >>
> > >> >>>> > > > >> On Fri, Nov 13, 2009 at 4:16 PM, Jean-Daniel Cryans <
> > >> >>>> > > > jdcryans@apache.org
> > >> >>>> > > > >> >wrote:
> > >> >>>> > > > >>
> > >> >>>> > > > >> Looks like
> > >> >>>> > > > >>>
> > >> >>>> > > > >>>
> > >> >>>> > > >
> > >> >>>> > >
> > >> >>>> >
> > >> >>>>
> > >>
> >
> http://pero.blogs.aprilmayjune.org/2009/01/22/hadoop-and-linux-kernel-2627-epoll-limits/
> > >> >>>> > > > >>>
> > >> >>>> > > > >>> J-D
> > >> >>>> > > > >>>
> > >> >>>> > > > >>> On Fri, Nov 13, 2009 at 1:12 PM, Zhenyu Zhong <
> > >> >>>> > > zhongresearch@gmail.com
> > >> >>>> > > > >
> > >> >>>> > > > >>> wrote:
> > >> >>>> > > > >>>
> > >> >>>> > > > >>>> Hi,
> > >> >>>> > > > >>>>
> > >> >>>> > > > >>>>
> > >> >>>> > > > >>>> After I re-organize the cluster, the experiment ran
> into
> > >> >>>> problem
> > >> >>>> > > > faster
> > >> >>>> > > > >>>>
> > >> >>>> > > > >>> than
> > >> >>>> > > > >>>
> > >> >>>> > > > >>>> before.
> > >> >>>> > > > >>>>
> > >> >>>> > > > >>>> Basically the changes are to use machines with less
> > >> resources
> > >> >>>> as
> > >> >>>> > > > >>>>
> > >> >>>> > > > >>> zookeeper
> > >> >>>> > > > >>>
> > >> >>>> > > > >>>> quorums and machines with more resources as
> > regionserver.
> > >> >>>> > > > >>>>
> > >> >>>> > > > >>>> From the log, I still see the pause around 1 minute.
> > >> >>>> > > > >>>> I enabled the GC logging, please see
> > >> >>>> > > > >>>>
> > >> >>>> > > > >>>> http://pastebin.com/m1d4ce0f1
> > >> >>>> > > > >>>>
> > >> >>>> > > > >>>> for details.
> > >> >>>> > > > >>>> In general I don't see many pauses in the GC.
> > >> >>>> > > > >>>>
> > >> >>>> > > > >>>> What is more interesting, 1 hour after the 1st
> > >> regionserver
> > >> >>>> > > > >>>> disconnected,
> > >> >>>> > > > >>>> the master log complained about too many open files.
> > This
> > >> >>>> didn't
> > >> >>>> > > > happen
> > >> >>>> > > > >>>> before.
> > >> >>>> > > > >>>> I checked the system OS setup as well as the
> > limits.conf.
> > >> I
> > >> >>>> also
> > >> >>>> > > > checked
> > >> >>>> > > > >>>>
> > >> >>>> > > > >>> the
> > >> >>>> > > > >>>
> > >> >>>> > > > >>>> running processes. The total open files don't reach
> the
> > >> limit.
> > >> >>>> I
> > >> >>>> > am
> > >> >>>> > > > >>>>
> > >> >>>> > > > >>> confused
> > >> >>>> > > > >>>
> > >> >>>> > > > >>>> a bit.
> > >> >>>> > > > >>>>
> > >> >>>> > > > >>>>
> > >> >>>> > > > >>>> Please see the following master log.
> > >> >>>> > > > >>>>
> > >> >>>> > > > >>>> 2009-11-13 20:06:02,114 INFO
> > >> >>>> > > > org.apache.hadoop.hbase.master.BaseScanner:
> > >> >>>> > > > >>>> RegionManager.metaScanner scan of 4658 row(s) of meta
> > >> region
> > >> >>>> > > {server:
> > >> >>>> > > > >>>> 192.168.100.128:60021, regionname: .META.,,1,
> startKey:
> > >> <>}
> > >> >>>> > > complete
> > >> >>>> > > > >>>> 2009-11-13 20:06:02,114 INFO
> > >> >>>> > > > org.apache.hadoop.hbase.master.BaseScanner:
> > >> >>>> > > > >>>>
> > >> >>>> > > > >>> All
> > >> >>>> > > > >>>
> > >> >>>> > > > >>>> 1 .META. region(s) scanned
> > >> >>>> > > > >>>> 2009-11-13 20:06:07,677 DEBUG
> > >> org.apache.zookeeper.ClientCnxn:
> > >> >>>> Got
> > >> >>>> > > > ping
> > >> >>>> > > > >>>> response for sessionid:0x424eebf1c10004c after 3ms
> > >> >>>> > > > >>>> 2009-11-13 20:06:08,178 INFO
> > >> org.apache.hadoop.hdfs.DFSClient:
> > >> >>>> > > > Exception
> > >> >>>> > > > >>>>
> > >> >>>> > > > >>> in
> > >> >>>> > > > >>>
> > >> >>>> > > > >>>> createBlockOutputStream java.io.IOException: Bad
> connect
> > >> ack
> > >> >>>> with
> > >> >>>> > > > >>>> firstBadLink 192.168.100.123:50010
> > >> >>>> > > > >>>> 2009-11-13 20:06:08,178 INFO
> > >> org.apache.hadoop.hdfs.DFSClient:
> > >> >>>> > > > >>>> Abandoning
> > >> >>>> > > > >>>> block blk_-2808245019291145247_5478039
> > >> >>>> > > > >>>> 2009-11-13 20:06:09,682 INFO
> > >> org.apache.hadoop.hdfs.DFSClient:
> > >> >>>> > > > Exception
> > >> >>>> > > > >>>>
> > >> >>>> > > > >>> in
> > >> >>>> > > > >>>
> > >> >>>> > > > >>>> createBlockOutputStream java.io.EOFException
> > >> >>>> > > > >>>> 2009-11-13 20:06:09,682 INFO
> > >> org.apache.hadoop.hdfs.DFSClient:
> > >> >>>> > > > >>>> Abandoning
> > >> >>>> > > > >>>> block blk_1074853606841896259_5478048
> > >> >>>> > > > >>>> 2009-11-13 20:06:10,334 DEBUG
> > >> org.apache.zookeeper.ClientCnxn:
> > >> >>>> Got
> > >> >>>> > > > ping
> > >> >>>> > > > >>>> response for sessionid:0x24eebf1043003c after 1ms
> > >> >>>> > > > >>>> 2009-11-13 20:06:21,018 DEBUG
> > >> org.apache.zookeeper.ClientCnxn:
> > >> >>>> Got
> > >> >>>> > > > ping
> > >> >>>> > > > >>>> response for sessionid:0x424eebf1c10004c after 0ms
> > >> >>>> > > > >>>> 2009-11-13 20:06:23,674 DEBUG
> > >> org.apache.zookeeper.ClientCnxn:
> > >> >>>> Got
> > >> >>>> > > > ping
> > >> >>>> > > > >>>> response for sessionid:0x24eebf1043003c after 0ms
> > >> >>>> > > > >>>> 2009-11-13 20:06:24,828 INFO
> > >> org.apache.hadoop.hdfs.DFSClient:
> > >> >>>> > > > Exception
> > >> >>>> > > > >>>>
> > >> >>>> > > > >>> in
> > >> >>>> > > > >>>
> > >> >>>> > > > >>>> createBlockOutputStream java.io.IOException: Bad
> connect
> > >> ack
> > >> >>>> with
> > >> >>>> > > > >>>> firstBadLink 192.168.100.123:50010
> > >> >>>> > > > >>>> 2009-11-13 20:06:24,828 INFO
> > >> org.apache.hadoop.hdfs.DFSClient:
> > >> >>>> > > > >>>> Abandoning
> > >> >>>> > > > >>>> block blk_-6642544517082142289_5478063
> > >> >>>> > > > >>>> 2009-11-13 20:06:24,828 INFO
> > >> org.apache.hadoop.hdfs.DFSClient:
> > >> >>>> > > > Exception
> > >> >>>> > > > >>>>
> > >> >>>> > > > >>> in
> > >> >>>> > > > >>>
> > >> >>>> > > > >>>> createBlockOutputStream java.net.SocketException: Too
> > many
> > >> >>>> open
> > >> >>>> > > files
> > >> >>>> > > > >>>> 2009-11-13 20:06:24,828 INFO
> > >> org.apache.hadoop.hdfs.DFSClient:
> > >> >>>> > > > >>>> Abandoning
> > >> >>>> > > > >>>> block blk_2057511041109796090_5478063
> > >> >>>> > > > >>>> 2009-11-13 20:06:24,928 INFO
> > >> org.apache.hadoop.hdfs.DFSClient:
> > >> >>>> > > > Exception
> > >> >>>> > > > >>>>
> > >> >>>> > > > >>> in
> > >> >>>> > > > >>>
> > >> >>>> > > > >>>> createBlockOutputStream java.net.SocketException: Too
> > many
> > >> >>>> open
> > >> >>>> > > files
> > >> >>>> > > > >>>> 2009-11-13 20:06:24,928 INFO
> > >> org.apache.hadoop.hdfs.DFSClient:
> > >> >>>> > > > >>>> Abandoning
> > >> >>>> > > > >>>> block blk_8219260302213892894_5478064
> > >> >>>> > > > >>>> 2009-11-13 20:06:30,855 INFO
> > >> org.apache.hadoop.hdfs.DFSClient:
> > >> >>>> > > > Exception
> > >> >>>> > > > >>>>
> > >> >>>> > > > >>> in
> > >> >>>> > > > >>>
> > >> >>>> > > > >>>> createBlockOutputStream java.net.SocketException: Too
> > many
> > >> >>>> open
> > >> >>>> > > files
> > >> >>>> > > > >>>> 2009-11-13 20:06:30,855 INFO
> > >> org.apache.hadoop.hdfs.DFSClient:
> > >> >>>> > > > >>>> Abandoning
> > >> >>>> > > > >>>> block blk_1669205542853067709_5478235
> > >> >>>> > > > >>>> 2009-11-13 20:06:30,905 INFO
> > >> org.apache.hadoop.hdfs.DFSClient:
> > >> >>>> > > > Exception
> > >> >>>> > > > >>>>
> > >> >>>> > > > >>> in
> > >> >>>> > > > >>>
> > >> >>>> > > > >>>> createBlockOutputStream java.net.SocketException: Too
> > many
> > >> >>>> open
> > >> >>>> > > files
> > >> >>>> > > > >>>> 2009-11-13 20:06:30,905 INFO
> > >> org.apache.hadoop.hdfs.DFSClient:
> > >> >>>> > > > >>>> Abandoning
> > >> >>>> > > > >>>> block blk_9128897691346270351_5478237
> > >> >>>> > > > >>>> 2009-11-13 20:06:30,955 INFO
> > >> org.apache.hadoop.hdfs.DFSClient:
> > >> >>>> > > > Exception
> > >> >>>> > > > >>>>
> > >> >>>> > > > >>> in
> > >> >>>> > > > >>>
> > >> >>>> > > > >>>> createBlockOutputStream java.net.SocketException: Too
> > many
> > >> >>>> open
> > >> >>>> > > files
> > >> >>>> > > > >>>> 2009-11-13 20:06:30,955 INFO
> > >> org.apache.hadoop.hdfs.DFSClient:
> > >> >>>> > > > >>>> Abandoning
> > >> >>>> > > > >>>> block blk_1116845144864123018_5478240
> > >> >>>> > > > >>>> 2009-11-13 20:06:34,372 DEBUG
> > >> org.apache.zookeeper.ClientCnxn:
> > >> >>>> Got
> > >> >>>> > > > ping
> > >> >>>> > > > >>>> response for sessionid:0x424eebf1c10004c after 0ms
> > >> >>>> > > > >>>> 2009-11-13 20:06:37,034 DEBUG
> > >> org.apache.zookeeper.ClientCnxn:
> > >> >>>> Got
> > >> >>>> > > > ping
> > >> >>>> > > > >>>> response for sessionid:0x24eebf1043003c after 0ms
> > >> >>>> > > > >>>> 2009-11-13 20:06:37,235 WARN
> > >> org.apache.hadoop.hdfs.DFSClient:
> > >> >>>> > > > >>>>
> > >> >>>> > > > >>> DataStreamer
> > >> >>>> > > > >>>
> > >> >>>> > > > >>>> Exception: java.io.IOException: Too many open files
> > >> >>>> > > > >>>> at sun.nio.ch.IOUtil.initPipe(Native Method)
> > >> >>>> > > > >>>> at
> > >> >>>> > >
> sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:49)
> > >> >>>> > > > >>>> at
> > >> >>>> > > > >>>>
> > >> >>>> > > > >>>>
> > >> >>>> > > >
> > >> >>>> > >
> > >> >>>> >
> > >> >>>>
> > >>
> >
> sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:18)
> > >> >>>> > > > >>>
> > >> >>>> > > > >>>> at
> > >> >>>> > > > >>>>
> > >> >>>> > > > >>>>
> > >> >>>> > > >
> > >> >>>> > >
> > >> >>>> >
> > >> >>>>
> > >>
> >
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.get(SocketIOWithTimeout.java:407)
> > >> >>>> > > > >>>
> > >> >>>> > > > >>>> at
> > >> >>>> > > > >>>>
> > >> >>>> > > > >>>>
> > >> >>>> > > >
> > >> >>>> > >
> > >> >>>> >
> > >> >>>>
> > >>
> >
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:322)
> > >> >>>> > > > >>>
> > >> >>>> > > > >>>> at
> > >> >>>> > > > >>>>
> > >> >>>> > > > >>>>
> > >> >>>> > > >
> > >> >>>> > >
> > >> >>>> >
> > >> >>>>
> > >>
> >
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
> > >> >>>> > > > >>>
> > >> >>>> > > > >>>> at
> > >> >>>> > > > >>>>
> > >> >>>> > > > >>>>
> > >> >>>> > > >
> > >> >>>> > >
> > >> >>>> >
> > >> >>>>
> > >>
> >
> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
> > >> >>>> > > > >>>
> > >> >>>> > > > >>>> at
> > >> >>>> > > > >>>>
> > >> >>>> > > > >>>>
> > >> >>>> > > >
> > >> >>>> > >
> > >> >>>> >
> > >> >>>>
> > >>
> >
> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
> > >> >>>> > > > >>>
> > >> >>>> > > > >>>> at
> > >> >>>> > > > >>>>
> > >> >>>> > > > >>>
> > >> >>>> java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
> > >> >>>> > > > >>>
> > >> >>>> > > > >>>> at
> > >> >>>> java.io.DataOutputStream.write(DataOutputStream.java:90)
> > >> >>>> > > > >>>> at
> > >> >>>> > > > >>>>
> > >> >>>> > > > >>>>
> > >> >>>> > > >
> > >> >>>> > >
> > >> >>>> >
> > >> >>>>
> > >>
> >
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2290)
> > >> >>>> > > > >>>
> > >> >>>> > > > >>>>
> > >> >>>> > > > >>>> 2009-11-13 20:06:37,235 WARN
> > >> org.apache.hadoop.hdfs.DFSClient:
> > >> >>>> > Error
> > >> >>>> > > > >>>> Recovery for block blk_8148813491785406356_5478475 bad
> > >> >>>> datanode[0]
> > >> >>>> > > > >>>> 192.168.100.123:50010
> > >> >>>> > > > >>>> 2009-11-13 20:06:37,235 WARN
> > >> org.apache.hadoop.hdfs.DFSClient:
> > >> >>>> > Error
> > >> >>>> > > > >>>> Recovery for block blk_8148813491785406356_5478475 in
> > >> pipeline
> > >> >>>> > > > >>>> 192.168.100.123:50010, 192.168.100.134:50010,
> > >> >>>> > 192.168.100.122:50010
> > >> >>>> > > :
> > >> >>>> > > > >>>> bad
> > >> >>>> > > > >>>> datanode 192.168.100.123:50010
> > >> >>>> > > > >>>> 2009-11-13 20:06:37,436 INFO
> > >> org.apache.hadoop.hdfs.DFSClient:
> > >> >>>> > > > Exception
> > >> >>>> > > > >>>>
> > >> >>>> > > > >>> in
> > >> >>>> > > > >>>
> > >> >>>> > > > >>>> createBlockOutputStream java.net.SocketException: Too
> > many
> > >> >>>> open
> > >> >>>> > > files
> > >> >>>> > > > >>>> 2009-11-13 20:06:37,436 INFO
> > >> org.apache.hadoop.hdfs.DFSClient:
> > >> >>>> > > > >>>> Abandoning
> > >> >>>> > > > >>>> block blk_2119727700857186236_5478498
> > >> >>>> > > > >>>>
> > >> >>>> > > > >>>>
> > >> >>>> > > > >>>>
> > >> >>>> > > > >>>>
> > >> >>>> > > > >>>>
> > >> >>>> > > > >>>> On Thu, Nov 12, 2009 at 4:21 PM, Zhenyu Zhong <
> > >> >>>> > > > zhongresearch@gmail.com
> > >> >>>> > > > >>>> wrote:
> > >> >>>> > > > >>>>
> > >> >>>> > > > >>>> Will do.
> > >> >>>> > > > >>>>>
> > >> >>>> > > > >>>>> thanks
> > >> >>>> > > > >>>>> zhenyu
> > >> >>>> > > > >>>>>
> > >> >>>> > > > >>>>>
> > >> >>>> > > > >>>>> On Thu, Nov 12, 2009 at 3:33 PM, stack <
> > stack@duboce.net
> > >> >
> > >> >>>> wrote:
> > >> >>>> > > > >>>>>
> > >> >>>> > > > >>>>> Enable GC logging too on this next run (see
> > >> hbase-env.sh).
> > >> >>>> Lets
> > >> >>>> > > try
> > >> >>>> > > > >>>>>>
> > >> >>>> > > > >>>>> and
> > >> >>>> > > > >>>
> > >> >>>> > > > >>>> get
> > >> >>>> > > > >>>>>> to the bottom of whats going on.
> > >> >>>> > > > >>>>>> Thanks,
> > >> >>>> > > > >>>>>> St.Ack
> > >> >>>> > > > >>>>>>
> > >> >>>> > > > >>>>>> On Thu, Nov 12, 2009 at 12:29 PM, Zhenyu Zhong <
> > >> >>>> > > > >>>>>>
> > >> >>>> > > > >>>>> zhongresearch@gmail.com
> > >> >>>> > > > >>>
> > >> >>>> > > > >>>> wrote:
> > >> >>>> > > > >>>>>>>
> > >> >>>> > > > >>>>>>
> > >> >>>> > > > >>>>>> I can switch the boxes that run zookeeper with the
> > ones
> > >> >>>> that
> > >> >>>> > run
> > >> >>>> > > > >>>>>>> regionservers.
> > >> >>>> > > > >>>>>>> I will see the results later.
> > >> >>>> > > > >>>>>>>
> > >> >>>> > > > >>>>>>> FYI. The node does have the 10 minutes
> > >> >>>> > zookeeper.session.timeout
> > >> >>>> > > > >>>>>>>
> > >> >>>> > > > >>>>>> value
> > >> >>>> > > > >>>
> > >> >>>> > > > >>>> in
> > >> >>>> > > > >>>>>>
> > >> >>>> > > > >>>>>>> place.
> > >> >>>> > > > >>>>>>>
> > >> >>>> > > > >>>>>>> thanks
> > >> >>>> > > > >>>>>>> zhenyu
> > >> >>>> > > > >>>>>>>
> > >> >>>> > > > >>>>>>> On Thu, Nov 12, 2009 at 3:21 PM, stack <
> > >> stack@duboce.net>
> > >> >>>> > wrote:
> > >> >>>> > > > >>>>>>>
> > >> >>>> > > > >>>>>>> On Thu, Nov 12, 2009 at 11:50 AM, Zhenyu Zhong <
> > >> >>>> > > > >>>>>>>>
> > >> >>>> > > > >>>>>>> zhongresearch@gmail.com
> > >> >>>> > > > >>>>>>
> > >> >>>> > > > >>>>>>> wrote:
> > >> >>>> > > > >>>>>>>>>
> > >> >>>> > > > >>>>>>>>
> > >> >>>> > > > >>>>>>>>
> > >> >>>> > > > >>>>>>>>> In my cluster, half of the cluster have 2 disks
> > 400GB
> > >> >>>> each
> > >> >>>> > per
> > >> >>>> > > > >>>>>>>>>
> > >> >>>> > > > >>>>>>>> machine,
> > >> >>>> > > > >>>>>>
> > >> >>>> > > > >>>>>>> and
> > >> >>>> > > > >>>>>>>>
> > >> >>>> > > > >>>>>>>>> half of the cluster have 6 disks per machine.
> > Maybe
> > >> we
> > >> >>>> > should
> > >> >>>> > > > >>>>>>>>>
> > >> >>>> > > > >>>>>>>> run
> > >> >>>> > > > >>>
> > >> >>>> > > > >>>> zookeeper
> > >> >>>> > > > >>>>>>>>> on the machines with 2 disks and RS on machines
> > with
> > >> 6
> > >> >>>> disks?
> > >> >>>> > > > >>>>>>>>>
> > >> >>>> > > > >>>>>>>>> That would make most sense only in the below, it
> > >> looks
> > >> >>>> like
> > >> >>>> > the
> > >> >>>> > > > >>>>>>>>>
> > >> >>>> > > > >>>>>>>> RS
> > >> >>>> > > > >>>
> > >> >>>> > > > >>>> that
> > >> >>>> > > > >>>>>>
> > >> >>>> > > > >>>>>>> had
> > >> >>>> > > > >>>>>>>> issue had 4 disks?
> > >> >>>> > > > >>>>>>>>
> > >> >>>> > > > >>>>>>>>
> > >> >>>> > > > >>>>>>>>
> > >> >>>> > > > >>>>>>>> BTW, the 10 minutes zookeeper.session.timeout has
> > >> been
> > >> >>>> set
> > >> >>>> > > during
> > >> >>>> > > > >>>>>>>>>
> > >> >>>> > > > >>>>>>>> the
> > >> >>>> > > > >>>>>>
> > >> >>>> > > > >>>>>>> experiment.
> > >> >>>> > > > >>>>>>>>>
> > >> >>>> > > > >>>>>>>>>
> > >> >>>> > > > >>>>>>>> And for sure this node had it in place?
> > >> >>>> > > > >>>>>>>> St.Ack
> > >> >>>> > > > >>>>>>>>
> > >> >>>> > > > >>>>>>>>
> > >> >>>> > > > >>>>>>>>
> > >> >>>> > > > >>>>>>>>
> > >> >>>> > > > >>>>>>>>> thanks
> > >> >>>> > > > >>>>>>>>> zhenyu
> > >> >>>> > > > >>>>>>>>>
> > >> >>>> > > > >>>>>>>>> On Thu, Nov 12, 2009 at 2:08 PM, stack <
> > >> stack@duboce.net
> > >> >>>> >
> > >> >>>> > > wrote:
> > >> >>>> > > > >>>>>>>>>
> > >> >>>> > > > >>>>>>>>> On Thu, Nov 12, 2009 at 8:40 AM, Zhenyu Zhong <
> > >> >>>> > > > >>>>>>>>>>
> > >> >>>> > > > >>>>>>>>> zhongresearch@gmail.com
> > >> >>>> > > > >>>>>>>
> > >> >>>> > > > >>>>>>>> wrote:
> > >> >>>> > > > >>>>>>>>>>>
> > >> >>>> > > > >>>>>>>>>>
> > >> >>>> > > > >>>>>>>>>>
> > >> >>>> > > > >>>>>>>>>> Though I experienced 2 regionserver
> disconnection
> > >> this
> > >> >>>> > > > >>>>>>>>>>>
> > >> >>>> > > > >>>>>>>>>> morning,
> > >> >>>> > > > >>>
> > >> >>>> > > > >>>> it
> > >> >>>> > > > >>>>>>
> > >> >>>> > > > >>>>>>> looks
> > >> >>>> > > > >>>>>>>>>
> > >> >>>> > > > >>>>>>>>>> better from the regionserver log. (please see
> the
> > >> >>>> following
> > >> >>>> > > > >>>>>>>>>>>
> > >> >>>> > > > >>>>>>>>>> log)
> > >> >>>> > > > >>>
> > >> >>>> > > > >>>>
> > >> >>>> > > > >>>>>>>>>>>
> > >> >>>> > > > >>>>>>>>>>> http://pastebin.com/m496dbfae
> > >> >>>> > > > >>>>>>>>>>>
> > >> >>>> > > > >>>>>>>>>>>
> > >> >>>> > > > >>>>>>>>>>> I checked diskIO for one of the
> > >> >>>> > regionserver(192.168.100.116)
> > >> >>>> > > > >>>>>>>>>>>
> > >> >>>> > > > >>>>>>>>>> during
> > >> >>>> > > > >>>>>>>
> > >> >>>> > > > >>>>>>>> the
> > >> >>>> > > > >>>>>>>>>
> > >> >>>> > > > >>>>>>>>>> time it disconnected.
> > >> >>>> > > > >>>>>>>>>>>
> > >> >>>> > > > >>>>>>>>>>> Time: 03:04:58 AM
> > >> >>>> > > > >>>>>>>>>>> Device: tps Blk_read/s
> Blk_wrtn/s
> > >> >>>> Blk_read
> > >> >>>> > > > >>>>>>>>>>>
> > >> >>>> > > > >>>>>>>>>> Blk_wrtn
> > >> >>>> > > > >>>>>>>>
> > >> >>>> > > > >>>>>>>>> sda 105.31 5458.83 19503.64
> > >> >>>> 9043873239
> > >> >>>> > > > >>>>>>>>>>>
> > >> >>>> > > > >>>>>>>>>> 32312473676
> > >> >>>> > > > >>>>>>>>
> > >> >>>> > > > >>>>>>>>> sda1 2.90 6.64 99.25
> > >> >>>> 10993934
> > >> >>>> > > > >>>>>>>>>>>
> > >> >>>> > > > >>>>>>>>>> 164433464
> > >> >>>> > > > >>>>>>>>
> > >> >>>> > > > >>>>>>>>> sda2 1.72 23.76 30.25
> > >> >>>> 39365817
> > >> >>>> > > > >>>>>>>>>>>
> > >> >>>> > > > >>>>>>>>>> 50124008
> > >> >>>> > > > >>>>>>>>
> > >> >>>> > > > >>>>>>>>> sda3 0.30 0.38 3.58
> > >> >>>> 624930
> > >> >>>> > > > >>>>>>>>>>>
> > >> >>>> > > > >>>>>>>>>> 5923000
> > >> >>>> > > > >>>>>>>>
> > >> >>>> > > > >>>>>>>>> sda4 100.39 5428.06 19370.56
> > >> >>>> 8992888270
> > >> >>>> > > > >>>>>>>>>>>
> > >> >>>> > > > >>>>>>>>>> 32091993204
> > >> >>>> > > > >>>>>>>>
> > >> >>>> > > > >>>>>>>>>
> > >> >>>> > > > >>>>>>>>>>>
> > >> >>>> > > > >>>>>>>>>>> Is this high for you? 20k blocks/second would
> > >> seem to
> > >> >>>> be
> > >> >>>> > > high
> > >> >>>> > > > >>>>>>>>>>
> > >> >>>> > > > >>>>>>>>> but
> > >> >>>> > > > >>>
> > >> >>>> > > > >>>> its
> > >> >>>> > > > >>>>>>>
> > >> >>>> > > > >>>>>>>> one
> > >> >>>> > > > >>>>>>>>>
> > >> >>>> > > > >>>>>>>>>> disk only and its not being shared by zk anymore
> > so
> > >> >>>> > shouldn't
> > >> >>>> > > > >>>>>>>>>>
> > >> >>>> > > > >>>>>>>>> matter?
> > >> >>>> > > > >>>>>>
> > >> >>>> > > > >>>>>>>
> > >> >>>> > > > >>>>>>>>>>
> > >> >>>> > > > >>>>>>>>>> I also checked the zookeeper quorum server that
> > the
> > >> >>>> > > > >>>>>>>>>>>
> > >> >>>> > > > >>>>>>>>>> regionserver
> > >> >>>> > > > >>>
> > >> >>>> > > > >>>> tried
> > >> >>>> > > > >>>>>>>>
> > >> >>>> > > > >>>>>>>>> to
> > >> >>>> > > > >>>>>>>>>
> > >> >>>> > > > >>>>>>>>>> connect according to the log. However, I don't
> see
> > >> >>>> > > > >>>>>>>>>>>
> > >> >>>> > > > >>>>>>>>>> 192.168.100.116
> > >> >>>> > > > >>>>>>
> > >> >>>> > > > >>>>>>> in
> > >> >>>> > > > >>>>>>>
> > >> >>>> > > > >>>>>>>> the
> > >> >>>> > > > >>>>>>>>>
> > >> >>>> > > > >>>>>>>>>> client list of the zookeeper quorum server that
> > the
> > >> >>>> > > > >>>>>>>>>>>
> > >> >>>> > > > >>>>>>>>>> regionserver
> > >> >>>> > > > >>>
> > >> >>>> > > > >>>> tried
> > >> >>>> > > > >>>>>>>>
> > >> >>>> > > > >>>>>>>>> to
> > >> >>>> > > > >>>>>>>>>
> > >> >>>> > > > >>>>>>>>>> connect.
> > >> >>>> > > > >>>>>>>>>>> Would that be a problem?
> > >> >>>> > > > >>>>>>>>>>>
> > >> >>>> > > > >>>>>>>>>>>
> > >> >>>> > > > >>>>>>>>>>> Is that because the ephemeral node for the
> > >> >>>> regionserver
> > >> >>>> > had
> > >> >>>> > > > >>>>>>>>>>
> > >> >>>> > > > >>>>>>>>> evaporated?
> > >> >>>> > > > >>>>>>>
> > >> >>>> > > > >>>>>>>> Lost
> > >> >>>> > > > >>>>>>>>>> its lease w/ zk by the time you went to look?
> > >> >>>> > > > >>>>>>>>>>
> > >> >>>> > > > >>>>>>>>>>
> > >> >>>> > > > >>>>>>>>>>
> > >> >>>> > > > >>>>>>>>>>
> > >> >>>> > > > >>>>>>>>>> Thu Nov 12 15:04:52 UTC 2009
> > >> >>>> > > > >>>>>>>>>>> Zookeeper version: 3.2.1-808558, built on
> > >> 08/27/2009
> > >> >>>> 18:48
> > >> >>>> > > > >>>>>>>>>>>
> > >> >>>> > > > >>>>>>>>>> GMT
> > >> >>>> > > > >>>
> > >> >>>> > > > >>>> Clients:
> > >> >>>> > > > >>>>>>>>>>> /192.168.100.127:43045
> > >> [1](queued=0,recved=26,sent=0)
> > >> >>>> > > > >>>>>>>>>>> /192.168.100.131:39091
> > >> [1](queued=0,recved=964,sent=0)
> > >> >>>> > > > >>>>>>>>>>> /192.168.100.124:35961
> > >> [1](queued=0,recved=3266,sent=0)
> > >> >>>> > > > >>>>>>>>>>> /192.168.100.123:47935
> > >> [1](queued=0,recved=233,sent=0)
> > >> >>>> > > > >>>>>>>>>>> /192.168.100.125:46931
> > >> [1](queued=0,recved=2,sent=0)
> > >> >>>> > > > >>>>>>>>>>> /192.168.100.118:54924
> > >> [1](queued=0,recved=295,sent=0)
> > >> >>>> > > > >>>>>>>>>>> /192.168.100.118:41390
> > >> [1](queued=0,recved=2290,sent=0)
> > >> >>>> > > > >>>>>>>>>>> /192.168.100.136:42243
> > >> [1](queued=0,recved=0,sent=0)
> > >> >>>> > > > >>>>>>>>>>>
> > >> >>>> > > > >>>>>>>>>>> Latency min/avg/max: 0/17/6333
> > >> >>>> > > > >>>>>>>>>>> Received: 47111
> > >> >>>> > > > >>>>>>>>>>> Sent: 0
> > >> >>>> > > > >>>>>>>>>>> Outstanding: 0
> > >> >>>> > > > >>>>>>>>>>> Zxid: 0x77000083f4
> > >> >>>> > > > >>>>>>>>>>> Mode: leader
> > >> >>>> > > > >>>>>>>>>>> Node count: 23
> > >> >>>> > > > >>>>>>>>>>>
> > >> >>>> > > > >>>>>>>>>>>
> > >> >>>> > > > >>>>>>>>>>
> > >> >>>> > > > >>>>>>>>>> That 6 second maximum latency is pretty bad but
> > >> should
> > >> >>>> be
> > >> >>>> > well
> > >> >>>> > > > >>>>>>>>>>
> > >> >>>> > > > >>>>>>>>> within
> > >> >>>> > > > >>>>>>
> > >> >>>> > > > >>>>>>> the
> > >> >>>> > > > >>>>>>>>
> > >> >>>> > > > >>>>>>>>> zk
> > >> >>>> > > > >>>>>>>>>> session timeout.
> > >> >>>> > > > >>>>>>>>>>
> > >> >>>> > > > >>>>>>>>>> So, problem is likely on the zk client side of
> the
> > >> >>>> session;
> > >> >>>> > > > >>>>>>>>>>
> > >> >>>> > > > >>>>>>>>> i.e.
> > >> >>>> > > > >>>
> > >> >>>> > > > >>>> in
> > >> >>>> > > > >>>>>>
> > >> >>>> > > > >>>>>>> the
> > >> >>>> > > > >>>>>>>
> > >> >>>> > > > >>>>>>>> RS.
> > >> >>>> > > > >>>>>>>>>
> > >> >>>> > > > >>>>>>>>>> You could enable GC logging as J-D suggested to
> > see
> > >> if
> > >> >>>> you
> > >> >>>> > > have
> > >> >>>> > > > >>>>>>>>>>
> > >> >>>> > > > >>>>>>>>> any
> > >> >>>> > > > >>>>>>
> > >> >>>> > > > >>>>>>> big
> > >> >>>> > > > >>>>>>>
> > >> >>>> > > > >>>>>>>> pauses, pauses > zk session timeout.
> > >> >>>> > > > >>>>>>>>>>
> > >> >>>> > > > >>>>>>>>>> When the RS went down, it didn't look too
> heavily
> > >> >>>> loaded:
> > >> >>>> > > > >>>>>>>>>>
> > >> >>>> > > > >>>>>>>>>>
> > >> >>>> > > > >>>>>>>>>> 1. 2009-11-12 15:04:52,830 INFO
> > >> >>>> > > > >>>>>>>>>>
> > org.apache.hadoop.hbase.regionserver.HRegionServer:
> > >> >>>> Dump of
> > >> >>>> > > > >>>>>>>>>>
> > >> >>>> > > > >>>>>>>>> metrics:
> > >> >>>> > > > >>>>>>>
> > >> >>>> > > > >>>>>>>> request=1.5166667, regions=322, stores=657,
> > >> >>>> storefiles=631,
> > >> >>>> > > > >>>>>>>>>> storefileIndexSize=61, memstoreSize=1472,
> > >> >>>> usedHeap=2819,
> > >> >>>> > > > >>>>>>>>>>
> > >> >>>> > > > >>>>>>>>> maxHeap=4079,
> > >> >>>> > > > >>>>>>>>
> > >> >>>> > > > >>>>>>>>> blockCacheSize=658110960,
> > blockCacheFree=197395984,
> > >> >>>> > > > >>>>>>>>>>
> > >> >>>> > > > >>>>>>>>> blockCacheCount=9903,
> > >> >>>> > > > >>>>>>>>>
> > >> >>>> > > > >>>>>>>>>> blockCacheHitRatio=99
> > >> >>>> > > > >>>>>>>>>>
> > >> >>>> > > > >>>>>>>>>> Its serving a few reads? The number of store
> > files
> > >> >>>> seems
> > >> >>>> > > fine.
> > >> >>>> > > > >>>>>>>>>>
> > >> >>>> > > > >>>>>>>>> Not
> > >> >>>> > > > >>>>>>
> > >> >>>> > > > >>>>>>> too
> > >> >>>> > > > >>>>>>>>
> > >> >>>> > > > >>>>>>>>> much memory used.
> > >> >>>> > > > >>>>>>>>>>
> > >> >>>> > > > >>>>>>>>>> Looking at the logs, I see the Lease Still Held
> > >> >>>> exception.
> > >> >>>> > > > >>>>>>>>>>
> > >> >>>> > > > >>>>>>>>> This
> > >> >>>> > > > >>>
> > >> >>>> > > > >>>> happens
> > >> >>>> > > > >>>>>>>>
> > >> >>>> > > > >>>>>>>>> when the RS does its regular report to the master
> > but
> > >> the
> > >> >>>> > > > >>>>>>>>>>
> > >> >>>> > > > >>>>>>>>> master
> > >> >>>> > > > >>>
> > >> >>>> > > > >>>> thinks
> > >> >>>> > > > >>>>>>>
> > >> >>>> > > > >>>>>>>> the
> > >> >>>> > > > >>>>>>>>>
> > >> >>>> > > > >>>>>>>>>> RS has since restarted. It'll think this
> probably
> > >> >>>> because
> > >> >>>> > it
> > >> >>>> > > > >>>>>>>>>>
> > >> >>>> > > > >>>>>>>>> noticed
> > >> >>>> > > > >>>>>>
> > >> >>>> > > > >>>>>>> that
> > >> >>>> > > > >>>>>>>>>
> > >> >>>> > > > >>>>>>>>>> the RS's znode in zk had gone away and it
> > considered
> > >> the
> > >> >>>> RS
> > >> >>>> > > > >>>>>>>>>>
> > >> >>>> > > > >>>>>>>>> dead.
> > >> >>>> > > > >>>
> > >> >>>> > > > >>>>
> > >> >>>> > > > >>>>>>>>>> Looking too at your logs I see this gap in the
> zk
> > >> >>>> pinging:
> > >> >>>> > > > >>>>>>>>>>
> > >> >>>> > > > >>>>>>>>>>
> > >> >>>> > > > >>>>>>>>>> 1. 2009-11-12 15:03:39,325 DEBUG
> > >> >>>> > > > >>>>>>>>>>
> > >> >>>> > > > >>>>>>>>> org.apache.zookeeper.ClientCnxn:
> > >> >>>> > > > >>>>>>
> > >> >>>> > > > >>>>>>> Got
> > >> >>>> > > > >>>>>>>
> > >> >>>> > > > >>>>>>>> ping response for sessionid:0x224e55436ad0004
> after
> > >> 0ms
> > >> >>>> > > > >>>>>>>>>> 2. 2009-11-12 15:03:43,113 DEBUG
> > >> >>>> > > > >>>>>>>>>>
> > >> >>>> > > > >>>>>>>>> org.apache.zookeeper.ClientCnxn:
> > >> >>>> > > > >>>>>>
> > >> >>>> > > > >>>>>>> Got
> > >> >>>> > > > >>>>>>>
> > >> >>>> > > > >>>>>>>> ping response for sessionid:0x24e55436a0007d
> after
> > >> 0ms
> > >> >>>> > > > >>>>>>>>>>
> > >> >>>> > > > >>>>>>>>>> Where in the lines above it, its reporting about
> > >> every
> > >> >>>> ten
> > >> >>>> > > > >>>>>>>>>>
> > >> >>>> > > > >>>>>>>>> seconds,
> > >> >>>> > > > >>>>>>
> > >> >>>> > > > >>>>>>> here
> > >> >>>> > > > >>>>>>>>
> > >> >>>> > > > >>>>>>>>> there is a big gap.
> > >> >>>> > > > >>>>>>>>>>
> > >> >>>> > > > >>>>>>>>>> Do you have ganglia or something that will let
> you
> > >> look
> > >> >>>> more
> > >> >>>> > > > >>>>>>>>>>
> > >> >>>> > > > >>>>>>>>> into
> > >> >>>> > > > >>>
> > >> >>>> > > > >>>> what
> > >> >>>> > > > >>>>>>>
> > >> >>>> > > > >>>>>>>> was
> > >> >>>> > > > >>>>>>>>>
> > >> >>>> > > > >>>>>>>>>> happening on this machine around this time? Is
> > the
> > >> >>>> machine
> > >> >>>> > > OK?
> > >> >>>> > > > >>>>>>>>>>
> > >> >>>> > > > >>>>>>>>> It
> > >> >>>> > > > >>>>>>
> > >> >>>> > > > >>>>>>> looks
> > >> >>>> > > > >>>>>>>>
> > >> >>>> > > > >>>>>>>>> lightly loaded and you have your cluster nicely
> > laid
> > >> out.
> > >> >>>> > > > >>>>>>>>>>
> > >> >>>> > > > >>>>>>>>> Something
> > >> >>>> > > > >>>>>>
> > >> >>>> > > > >>>>>>> odd
> > >> >>>> > > > >>>>>>>>
> > >> >>>> > > > >>>>>>>>> is
> > >> >>>> > > > >>>>>>>>>
> > >> >>>> > > > >>>>>>>>>> going on. What about things like the write
> speed
> > to
> > >> >>>> disk?
> > >> >>>> > In
> > >> >>>> > > > >>>>>>>>>>
> > >> >>>> > > > >>>>>>>>> the
> > >> >>>> > > > >>>
> > >> >>>> > > > >>>> past
> > >> >>>> > > > >>>>>>>
> > >> >>>> > > > >>>>>>>> strange issues have been explained by incorrectly
> > set
> > >> BIOS
> > >> >>>> > > > >>>>>>>>>>
> > >> >>>> > > > >>>>>>>>> which
> > >> >>>> > > > >>>
> > >> >>>> > > > >>>> made
> > >> >>>> > > > >>>>>>
> > >> >>>> > > > >>>>>>> disks
> > >> >>>> > > > >>>>>>>>>
> > >> >>>> > > > >>>>>>>>>> run at 1/100th of their proper speed.
> > >> >>>> > > > >>>>>>>>>>
> > >> >>>> > > > >>>>>>>>>> St.Ack
> > >> >>>> > > > >>>>>>>>>>
> > >> >>>> > > > >>>>>>>>>>
> > >> >>>> > > > >>>>>>>>>>
> > >> >>>> > > > >>>>>>>>>>
> > >> >>>> > > > >>>>>>>>>>
> > >> >>>> > > > >>>>>>>>>>
> > >> >>>> > > > >>>>>>>>>>
> > >> >>>> > > > >>>>>>>>>>
> > >> >>>> > > > >>>>>>>>>>> Best,
> > >> >>>> > > > >>>>>>>>>>> zhenyu
> > >> >>>> > > > >>>>>>>>>>>
> > >> >>>> > > > >>>>>>>>>>>
> > >> >>>> > > > >>>>>>>>>>> On Wed, Nov 11, 2009 at 3:58 PM, Zhenyu Zhong <
> > >> >>>> > > > >>>>>>>>>>>
> > >> >>>> > > > >>>>>>>>>> zhongresearch@gmail.com
> > >> >>>> > > > >>>>>>>>
> > >> >>>> > > > >>>>>>>>> wrote:
> > >> >>>> > > > >>>>>>>>>>>>
> > >> >>>> > > > >>>>>>>>>>>
> > >> >>>> > > > >>>>>>>>>>> Stack
> > >> >>>> > > > >>>>>>>>>>>>
> > >> >>>> > > > >>>>>>>>>>>> I am very appreciated for your comments.
> > >> >>>> > > > >>>>>>>>>>>> I will use the zookeeper monitoring script on
> my
> > >> >>>> cluster
> > >> >>>> > > > >>>>>>>>>>>>
> > >> >>>> > > > >>>>>>>>>>> and
> > >> >>>> > > > >>>
> > >> >>>> > > > >>>> let
> > >> >>>> > > > >>>>>>
> > >> >>>> > > > >>>>>>> it
> > >> >>>> > > > >>>>>>>
> > >> >>>> > > > >>>>>>>> run
> > >> >>>> > > > >>>>>>>>>
> > >> >>>> > > > >>>>>>>>>> overnight to see the result.
> > >> >>>> > > > >>>>>>>>>>>> I will follow up the thread when I get
> anything.
> > >> >>>> > > > >>>>>>>>>>>>
> > >> >>>> > > > >>>>>>>>>>>> thanks
> > >> >>>> > > > >>>>>>>>>>>> zhenyu
> > >> >>>> > > > >>>>>>>>>>>>
> > >> >>>> > > > >>>>>>>>>>>>
> > >> >>>> > > > >>>>>>>>>>>> On Wed, Nov 11, 2009 at 3:52 PM, stack <
> > >> >>>> stack@duboce.net>
> > >> >>>> > > > >>>>>>>>>>>>
> > >> >>>> > > > >>>>>>>>>>> wrote:
> > >> >>>> > > > >>>>>>
> > >> >>>> > > > >>>>>>>
> > >> >>>> > > > >>>>>>>>>>>> I see these in your log too:
> > >> >>>> > > > >>>>>>>>>>>>>
> > >> >>>> > > > >>>>>>>>>>>>>
> > >> >>>> > > > >>>>>>>>>>>>> 1. 2009-11-11 04:27:19,018 DEBUG
> > >> >>>> > > > >>>>>>>>>>>>>
> > >> >>>> > > > >>>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
> > >> >>>> > > > >>>>>>>>
> > >> >>>> > > > >>>>>>>>> Got
> > >> >>>> > > > >>>>>>>>>>
> > >> >>>> > > > >>>>>>>>>>> ping response for sessionid:0x424dfd908c50009
> > >> after
> > >> >>>> > > > >>>>>>>>>>>>>
> > >> >>>> > > > >>>>>>>>>>>> 4544ms
> > >> >>>> > > > >>>
> > >> >>>> > > > >>>> 2. 2009-11-11 04:27:19,018 DEBUG
> > >> >>>> > > > >>>>>>>>>>>>>
> > >> >>>> > > > >>>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
> > >> >>>> > > > >>>>>>>>
> > >> >>>> > > > >>>>>>>>> Got
> > >> >>>> > > > >>>>>>>>>>
> > >> >>>> > > > >>>>>>>>>>> ping response for sessionid:0x24dfd90c810002
> > after
> > >> >>>> > > > >>>>>>>>>>>>>
> > >> >>>> > > > >>>>>>>>>>>> 4548ms
> > >> >>>> > > > >>>
> > >> >>>> > > > >>>> 3. 2009-11-11 04:27:43,960 DEBUG
> > >> >>>> > > > >>>>>>>>>>>>>
> > >> >>>> > > > >>>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
> > >> >>>> > > > >>>>>>>>
> > >> >>>> > > > >>>>>>>>> Got
> > >> >>>> > > > >>>>>>>>>>
> > >> >>>> > > > >>>>>>>>>>> ping response for sessionid:0x424dfd908c50009
> > >> after
> > >> >>>> > > > >>>>>>>>>>>>>
> > >> >>>> > > > >>>>>>>>>>>> 9030ms
> > >> >>>> > > > >>>
> > >> >>>> > > > >>>> 4. 2009-11-11 04:27:43,960 DEBUG
> > >> >>>> > > > >>>>>>>>>>>>>
> > >> >>>> > > > >>>>>>>>>>>> org.apache.zookeeper.
> > >> >>>> > > > >>>>>>>>
> > >> >>>> > > > >>>>>>>
> > >> >>>> > > >
> > >> >>>> > >
> > >> >>>> >
> > >> >>>>
> > >> >>>
> > >> >>>
> > >> >>
> > >> >
> > >>
> > >
> > >
> >
>
Re: regionserver disconnection
Posted by stack <st...@duboce.net>.
On Mon, Nov 16, 2009 at 12:05 PM, Zhenyu Zhong <zh...@gmail.com>wrote:
> I just realized that there was a MapReduce job running during the time the
> regionserver disconnected from the zookeeper.
> That MapReduce Job was processing 500GB data and took about 8 minutes to
> finish. It launched over 2000 map tasks.
There was a tasktracker running on the RegionServer that disconnected? Is
the map i/o or cpu heavy? Do you think it could have stole life from the
datanode/regionserver?
> I doubt that this introduced
> resource contention between DataNode and RegionServer Node.
> Also during the time that MapReduce job ran, I saw a few errors indicating
> that
>
> java.io.IOException: Could not obtain block:
> blk_7661247556283580226_4450462 file=/data/xxx.txt.200910141159
> at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1787)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1615)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1742)
> at java.io.DataInputStream.read(DataInputStream.java:83)
> at org.apache.hadoop.util.LineReader.readLine(LineReader.java:134)
> at
> org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:136)
> at
> org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:40)
> at
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:192)
> at
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:176)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
> at org.apache.hadoop.mapred.Child.main(Child.java:170)
>
This is interesting. Your map task is running into hdfs-127. Yeah, you
should patch your cluster if you want to get rid of these (hdfs-127 has been
applied to 0.21 hadoop and will be in the next hadoop release on 0.20.x
branch, hadoop 0.20.2).
>
> Possibly it is related to HDFS-127, but I don't see any datanodes went
> down. Does that affect Regionserver? Shall we apply the patch?
>
>
Read up on hdfs-127 for explaination of whats going on.
> Now I start to keep tracking of the virtual memory stats to see if the same
> issue happens tomorrow around the same time period.
>
Excellent. Thanks for doing this.
> And I highly suspect that this particular MapReduce job hurts HBase
> Regionserver.
>
> PS. I also use gcviewer to parse the GC-log, I only see around 30 seconds
> pauses maximum.
>
Did you figure why your ten minute session is not applying?
St.Ack
>
> Thanks a lot.
> zhenyu
>
>
>
> On Sun, Nov 15, 2009 at 12:52 AM, Zhenyu Zhong <zhongresearch@gmail.com
> >wrote:
>
> > J-D,
> >
> > Thank you very much for your comments.
> > My company block the IRC port, so I have trouble to connect to IRC
> channel.
> > I have been trying to ask the IT to open the IRC port for me, it might
> take
> > a while.
> >
> > Best,
> > zhenyu
> >
> >
> > On Sat, Nov 14, 2009 at 2:21 PM, Jean-Daniel Cryans <jdcryans@apache.org
> >wrote:
> >
> >> The error you are getting is a disconnection from a zookeeper server
> >> and is very generic.
> >>
> >> ZK-86 is still opened and the last comment refers to ZK-111 saying
> >> that the bug (in unit tests) was probably fixed in release 3.0.0 last
> >> year.
> >>
> >> To figure the hang you have, you can try to jstack the process pid and
> >> see exactly what's holding the RS from shutting down.
> >>
> >> Would it be possible for you to drop by the IRC channel? This way we
> >> can debug this a at much faster pace.
> >>
> >> Thx!
> >>
> >> J-D
> >>
> >> On Sat, Nov 14, 2009 at 9:57 AM, Zhenyu Zhong <zh...@gmail.com>
> >> wrote:
> >> > I found this.
> >> > http://issues.apache.org/jira/browse/ZOOKEEPER-86
> >> >
> >> > It looks like the same error I had. Is it a zookeeper bug? When will
> >> HBase
> >> > take the zookeeper version 3.3.0?
> >> >
> >> > thanks
> >> > zhenyu
> >> >
> >> > On Sat, Nov 14, 2009 at 11:31 AM, Zhenyu Zhong <
> zhongresearch@gmail.com
> >> >wrote:
> >> >
> >> >> Now I really doubt about the zookeeper, from the log I saw errors
> like
> >> >> IOException Read Error, while the zookeeper client (regionserver )
> >> tried to
> >> >> read. But it got disconneted status from zookeeper.
> >> >>
> >> >> I don't see any load on any zookeeper quorum servers. DiskIO is fine.
> >> >>
> >> >> Also when the regionserver decides to exit due to the disconnect
> status
> >> >> from zookeeper, sometimes the regionserver hangs during the exiting.
> We
> >> can
> >> >> still see the HRegionServer process even we don't see it in the
> Master
> >> web
> >> >> interface.
> >> >>
> >> >> I also notice that there is an zookeeper.retries and zookeeper.pauses
> >> >> settings in hbase-default.xml, would that matter ?
> >> >>
> >> >> thanks
> >> >>
> >> >> You may see the IOException error in the following log when the
> >> >> Regionserver lost connection to the zookeeper.
> >> >>
> >> >> 2009-11-14 15:58:31,993 INFO
> org.apache.hadoop.hbase.regionserver.HLog:
> >> >> Roll
> >> >>
> >>
> /hbase/.logs/superpyxis0008.scur.colo,60021,1258179101617/hlog.dat.1258179106921,
> >> >> entries=1062964, calcsize=255014936, filesize=139566093. New hlog
> >> >>
> >>
> /hbase/.logs/superpyxis0008.scur.colo,60021,1258179101617/hlog.dat.1258214311993
> >> >> 2009-11-14 15:58:39,295 DEBUG org.apache.zookeeper.ClientCnxn: Got
> ping
> >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
> >> >> 2009-11-14 15:58:52,648 DEBUG org.apache.zookeeper.ClientCnxn: Got
> ping
> >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
> >> >> 2009-11-14 15:59:06,007 DEBUG org.apache.zookeeper.ClientCnxn: Got
> ping
> >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
> >> >> 2009-11-14 15:59:19,365 DEBUG org.apache.zookeeper.ClientCnxn: Got
> ping
> >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
> >> >> 2009-11-14 15:59:32,867 DEBUG org.apache.zookeeper.ClientCnxn: Got
> ping
> >> >> response for sessionid:0x424f0ed01ea00a6 after 186ms
> >> >> 2009-11-14 15:59:46,070 DEBUG org.apache.zookeeper.ClientCnxn: Got
> ping
> >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
> >> >> 2009-11-14 15:59:59,423 DEBUG org.apache.zookeeper.ClientCnxn: Got
> ping
> >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
> >> >> 2009-11-14 16:00:12,775 DEBUG org.apache.zookeeper.ClientCnxn: Got
> ping
> >> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
> >> >> 2009-11-14 16:00:28,141 DEBUG org.apache.zookeeper.ClientCnxn: Got
> ping
> >> >> response for sessionid:0x424f0ed01ea00a6 after 2010ms
> >> >> 2009-11-14 16:01:10,378 WARN org.apache.zookeeper.ClientCnxn:
> Exception
> >> >> closing session 0x424f0ed01ea00a6 to
> >> sun.nio.ch.SelectionKeyImpl@39fbb2d6
> >> >>
> >> >> java.io.IOException: Read error rc = -1
> java.nio.DirectByteBuffer[pos=0
> >> >> lim=4 cap=4]
> >> >> at
> >> >> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
> >> >> at
> >> >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
> >> >> 2009-11-14 16:01:10,478 INFO
> >> >> org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper
> >> event,
> >> >> state: Disconnected, type: None, path: null
> >> >> 2009-11-14 16:01:11,333 WARN
> >> >> org.apache.hadoop.hbase.regionserver.HRegionServer: Attempt=1
> >> >>
> >> >> org.apache.hadoop.hbase.Leases$LeaseStillHeldException
> >> >> at
> >> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> >> >> Method)
> >> >> at
> >> >>
> >>
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> >> >> at
> >> >>
> >>
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> >> >> at
> >> java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> >> >> at
> >> >>
> >>
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
> >> >> at
> >> >>
> >>
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
> >> >> at
> >> >>
> >>
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
> >> >> at
> >> >>
> >>
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:571)
> >> >> at java.lang.Thread.run(Thread.java:619)
> >> >> 2009-11-14 16:01:11,433 INFO org.apache.zookeeper.ClientCnxn:
> >> Attempting
> >> >> connection to server superpyxis0001/192.168.100.115:2181
> >> >> 2009-11-14 16:01:11,433 INFO org.apache.zookeeper.ClientCnxn: Priming
> >> >> connection to java.nio.channels.SocketChannel[connected local=/
> >> >> 192.168.100.122:59575 remote=superpyxis0001/192.168.100.115:2181]
> >> >> 2009-11-14 16:01:11,433 INFO org.apache.zookeeper.ClientCnxn: Server
> >> >> connection successful
> >> >> 2009-11-14 16:01:11,433 INFO
> >> >> org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper
> >> event,
> >> >> state: Expired, type: None, path: null
> >> >> 2009-11-14 16:01:11,433 WARN org.apache.zookeeper.ClientCnxn:
> Exception
> >> >> closing session 0x424f0ed01ea00a6 to
> >> sun.nio.ch.SelectionKeyImpl@6b93d343
> >> >> java.io.IOException: Session Expired
> >> >> at
> >> >>
> >>
> org.apache.zookeeper.ClientCnxn$SendThread.readConnectResult(ClientCnxn.java:589)
> >> >> at
> >> >> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:709)
> >> >>
> >> >> at
> >> >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
> >> >> 2009-11-14 16:01:11,433 ERROR
> >> >> org.apache.hadoop.hbase.regionserver.HRegionServer: ZooKeeper session
> >> >> expired
> >> >> 2009-11-14 16:01:13,513 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> >> Server
> >> >> handler 3 on 60021, call put([B@73deb204,
> >> >> [Lorg.apache.hadoop.hbase.client.Put;@2179600a) from
> >> 192.168.100.132:40728:
> >> >> error: java.io.IOException: Server not running, aborting
> >> >> java.io.IOException: Server not running, aborting
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> On Sat, Nov 14, 2009 at 1:01 AM, Zhenyu Zhong <
> zhongresearch@gmail.com
> >> >wrote:
> >> >>
> >> >>> Stack,
> >> >>>
> >> >>> Thanks a lot!
> >> >>>
> >> >>> I found out that the reason why HBase doesn't take the system file
> >> >>> descriptor value I set before. I started the HBase using root
> instead
> >> of the
> >> >>> normal hadoop user, while my system configures hadoop with higher
> file
> >> >>> descriptor value but configures root with 1024 default value.
> >> >>>
> >> >>>
> >> >>> Now my system has a clean start. However, it seems that the
> >> >>> zookeeper.session.timeout value doesn't take into effect. I still
> >> found
> >> >>> around 60 seconds pauses from the disconnected regionserver. I
> really
> >> don't
> >> >>> know why regionserver only times out after 60 seconds instead of 10
> >> minutes
> >> >>> which I set for zookeeper.session.timeout.
> >> >>>
> >> >>> Is there any other timeout value coming into play before the actual
> >> >>> session times out?
> >> >>>
> >> >>> zhenyu
> >> >>>
> >> >>>
> >> >>>
> >> >>> On Fri, Nov 13, 2009 at 7:08 PM, stack <st...@duboce.net> wrote:
> >> >>>
> >> >>>> Ok. Lack of file descriptors manifests in all kinds of weird ways.
> >> >>>> Hopefully thats it. If not, lets keep digging.
> >> >>>> St.Ack
> >> >>>>
> >> >>>> On Fri, Nov 13, 2009 at 3:44 PM, Zhenyu Zhong <
> >> zhongresearch@gmail.com
> >> >>>> >wrote:
> >> >>>>
> >> >>>> > Stack,
> >> >>>> >
> >> >>>> > You are right, the master started with ulimit -n 1024. It doesn't
> >> take
> >> >>>> the
> >> >>>> > system value.
> >> >>>> >
> >> >>>> > Regarding to the too many open files, it looks like the same as
> the
> >> one
> >> >>>> J-D
> >> >>>> > put up. But I will get the Master start with higher value first
> and
> >> see
> >> >>>> if
> >> >>>> > this kind of error goes away.
> >> >>>> >
> >> >>>> > thanks a lot!
> >> >>>> > zhenyu
> >> >>>> >
> >> >>>> > On Fri, Nov 13, 2009 at 6:02 PM, stack <st...@duboce.net> wrote:
> >> >>>> >
> >> >>>> > > Does it say
> >> >>>> > >
> >> >>>> > > ulimit -n 32768
> >> >>>> > >
> >> >>>> > > ...as the second line in your log file on start of the master?
> >> >>>> > >
> >> >>>> > > Is the stack trace that complains about too many open files
> same
> >> as
> >> >>>> the
> >> >>>> > one
> >> >>>> > > in the blog post J-D put up?
> >> >>>> > >
> >> >>>> > > St.Ack
> >> >>>> > >
> >> >>>> > >
> >> >>>> > > On Fri, Nov 13, 2009 at 1:37 PM, Zhenyu Zhong <
> >> >>>> zhongresearch@gmail.com
> >> >>>> > > >wrote:
> >> >>>> > >
> >> >>>> > > > The ulimit file descriptors was set to fs.file-max =1578334,
> >> also
> >> >>>> in
> >> >>>> > the
> >> >>>> > > > limits.conf the value was set to 32768.
> >> >>>> > > > So these are way higher than the open descriptors for the
> >> running
> >> >>>> > > > processes.
> >> >>>> > > >
> >> >>>> > > > thanks
> >> >>>> > > > zhenyu
> >> >>>> > > >
> >> >>>> > > >
> >> >>>> > > > On Fri, Nov 13, 2009 at 4:33 PM, Stack <sa...@gmail.com>
> >> >>>> wrote:
> >> >>>> > > >
> >> >>>> > > > > You upped the ulimit file descriptors as per the getting
> >> started
> >> >>>> doc?
> >> >>>> > > > >
> >> >>>> > > > >
> >> >>>> > > > >
> >> >>>> > > > > On Nov 13, 2009, at 1:26 PM, Zhenyu Zhong <
> >> >>>> zhongresearch@gmail.com>
> >> >>>> > > > wrote:
> >> >>>> > > > >
> >> >>>> > > > > Thanks a lot.
> >> >>>> > > > >>
> >> >>>> > > > >>
> >> >>>> > > > >> Bad news is my kernel is still 2.6.26.
> >> >>>> > > > >> But it was not a problem before.
> >> >>>> > > > >>
> >> >>>> > > > >> Very strange.
> >> >>>> > > > >>
> >> >>>> > > > >> zhenyu
> >> >>>> > > > >>
> >> >>>> > > > >> On Fri, Nov 13, 2009 at 4:16 PM, Jean-Daniel Cryans <
> >> >>>> > > > jdcryans@apache.org
> >> >>>> > > > >> >wrote:
> >> >>>> > > > >>
> >> >>>> > > > >> Looks like
> >> >>>> > > > >>>
> >> >>>> > > > >>>
> >> >>>> > > >
> >> >>>> > >
> >> >>>> >
> >> >>>>
> >>
> http://pero.blogs.aprilmayjune.org/2009/01/22/hadoop-and-linux-kernel-2627-epoll-limits/
> >> >>>> > > > >>>
> >> >>>> > > > >>> J-D
> >> >>>> > > > >>>
> >> >>>> > > > >>> On Fri, Nov 13, 2009 at 1:12 PM, Zhenyu Zhong <
> >> >>>> > > zhongresearch@gmail.com
> >> >>>> > > > >
> >> >>>> > > > >>> wrote:
> >> >>>> > > > >>>
> >> >>>> > > > >>>> Hi,
> >> >>>> > > > >>>>
> >> >>>> > > > >>>>
> >> >>>> > > > >>>> After I re-organize the cluster, the experiment ran into
> >> >>>> problem
> >> >>>> > > > faster
> >> >>>> > > > >>>>
> >> >>>> > > > >>> than
> >> >>>> > > > >>>
> >> >>>> > > > >>>> before.
> >> >>>> > > > >>>>
> >> >>>> > > > >>>> Basically the changes are to use machines with less
> >> resources
> >> >>>> as
> >> >>>> > > > >>>>
> >> >>>> > > > >>> zookeeper
> >> >>>> > > > >>>
> >> >>>> > > > >>>> quorums and machines with more resources as
> regionserver.
> >> >>>> > > > >>>>
> >> >>>> > > > >>>> From the log, I still see the pause around 1 minute.
> >> >>>> > > > >>>> I enabled the GC logging, please see
> >> >>>> > > > >>>>
> >> >>>> > > > >>>> http://pastebin.com/m1d4ce0f1
> >> >>>> > > > >>>>
> >> >>>> > > > >>>> for details.
> >> >>>> > > > >>>> In general I don't see many pauses in the GC.
> >> >>>> > > > >>>>
> >> >>>> > > > >>>> What is more interesting, 1 hour after the 1st
> >> regionserver
> >> >>>> > > > >>>> disconnected,
> >> >>>> > > > >>>> the master log complained about too many open files.
> This
> >> >>>> didn't
> >> >>>> > > > happen
> >> >>>> > > > >>>> before.
> >> >>>> > > > >>>> I checked the system OS setup as well as the
> limits.conf.
> >> I
> >> >>>> also
> >> >>>> > > > checked
> >> >>>> > > > >>>>
> >> >>>> > > > >>> the
> >> >>>> > > > >>>
> >> >>>> > > > >>>> running processes. The total open files don't reach the
> >> limit.
> >> >>>> I
> >> >>>> > am
> >> >>>> > > > >>>>
> >> >>>> > > > >>> confused
> >> >>>> > > > >>>
> >> >>>> > > > >>>> a bit.
> >> >>>> > > > >>>>
> >> >>>> > > > >>>>
> >> >>>> > > > >>>> Please see the following master log.
> >> >>>> > > > >>>>
> >> >>>> > > > >>>> 2009-11-13 20:06:02,114 INFO
> >> >>>> > > > org.apache.hadoop.hbase.master.BaseScanner:
> >> >>>> > > > >>>> RegionManager.metaScanner scan of 4658 row(s) of meta
> >> region
> >> >>>> > > {server:
> >> >>>> > > > >>>> 192.168.100.128:60021, regionname: .META.,,1, startKey:
> >> <>}
> >> >>>> > > complete
> >> >>>> > > > >>>> 2009-11-13 20:06:02,114 INFO
> >> >>>> > > > org.apache.hadoop.hbase.master.BaseScanner:
> >> >>>> > > > >>>>
> >> >>>> > > > >>> All
> >> >>>> > > > >>>
> >> >>>> > > > >>>> 1 .META. region(s) scanned
> >> >>>> > > > >>>> 2009-11-13 20:06:07,677 DEBUG
> >> org.apache.zookeeper.ClientCnxn:
> >> >>>> Got
> >> >>>> > > > ping
> >> >>>> > > > >>>> response for sessionid:0x424eebf1c10004c after 3ms
> >> >>>> > > > >>>> 2009-11-13 20:06:08,178 INFO
> >> org.apache.hadoop.hdfs.DFSClient:
> >> >>>> > > > Exception
> >> >>>> > > > >>>>
> >> >>>> > > > >>> in
> >> >>>> > > > >>>
> >> >>>> > > > >>>> createBlockOutputStream java.io.IOException: Bad connect
> >> ack
> >> >>>> with
> >> >>>> > > > >>>> firstBadLink 192.168.100.123:50010
> >> >>>> > > > >>>> 2009-11-13 20:06:08,178 INFO
> >> org.apache.hadoop.hdfs.DFSClient:
> >> >>>> > > > >>>> Abandoning
> >> >>>> > > > >>>> block blk_-2808245019291145247_5478039
> >> >>>> > > > >>>> 2009-11-13 20:06:09,682 INFO
> >> org.apache.hadoop.hdfs.DFSClient:
> >> >>>> > > > Exception
> >> >>>> > > > >>>>
> >> >>>> > > > >>> in
> >> >>>> > > > >>>
> >> >>>> > > > >>>> createBlockOutputStream java.io.EOFException
> >> >>>> > > > >>>> 2009-11-13 20:06:09,682 INFO
> >> org.apache.hadoop.hdfs.DFSClient:
> >> >>>> > > > >>>> Abandoning
> >> >>>> > > > >>>> block blk_1074853606841896259_5478048
> >> >>>> > > > >>>> 2009-11-13 20:06:10,334 DEBUG
> >> org.apache.zookeeper.ClientCnxn:
> >> >>>> Got
> >> >>>> > > > ping
> >> >>>> > > > >>>> response for sessionid:0x24eebf1043003c after 1ms
> >> >>>> > > > >>>> 2009-11-13 20:06:21,018 DEBUG
> >> org.apache.zookeeper.ClientCnxn:
> >> >>>> Got
> >> >>>> > > > ping
> >> >>>> > > > >>>> response for sessionid:0x424eebf1c10004c after 0ms
> >> >>>> > > > >>>> 2009-11-13 20:06:23,674 DEBUG
> >> org.apache.zookeeper.ClientCnxn:
> >> >>>> Got
> >> >>>> > > > ping
> >> >>>> > > > >>>> response for sessionid:0x24eebf1043003c after 0ms
> >> >>>> > > > >>>> 2009-11-13 20:06:24,828 INFO
> >> org.apache.hadoop.hdfs.DFSClient:
> >> >>>> > > > Exception
> >> >>>> > > > >>>>
> >> >>>> > > > >>> in
> >> >>>> > > > >>>
> >> >>>> > > > >>>> createBlockOutputStream java.io.IOException: Bad connect
> >> ack
> >> >>>> with
> >> >>>> > > > >>>> firstBadLink 192.168.100.123:50010
> >> >>>> > > > >>>> 2009-11-13 20:06:24,828 INFO
> >> org.apache.hadoop.hdfs.DFSClient:
> >> >>>> > > > >>>> Abandoning
> >> >>>> > > > >>>> block blk_-6642544517082142289_5478063
> >> >>>> > > > >>>> 2009-11-13 20:06:24,828 INFO
> >> org.apache.hadoop.hdfs.DFSClient:
> >> >>>> > > > Exception
> >> >>>> > > > >>>>
> >> >>>> > > > >>> in
> >> >>>> > > > >>>
> >> >>>> > > > >>>> createBlockOutputStream java.net.SocketException: Too
> many
> >> >>>> open
> >> >>>> > > files
> >> >>>> > > > >>>> 2009-11-13 20:06:24,828 INFO
> >> org.apache.hadoop.hdfs.DFSClient:
> >> >>>> > > > >>>> Abandoning
> >> >>>> > > > >>>> block blk_2057511041109796090_5478063
> >> >>>> > > > >>>> 2009-11-13 20:06:24,928 INFO
> >> org.apache.hadoop.hdfs.DFSClient:
> >> >>>> > > > Exception
> >> >>>> > > > >>>>
> >> >>>> > > > >>> in
> >> >>>> > > > >>>
> >> >>>> > > > >>>> createBlockOutputStream java.net.SocketException: Too
> many
> >> >>>> open
> >> >>>> > > files
> >> >>>> > > > >>>> 2009-11-13 20:06:24,928 INFO
> >> org.apache.hadoop.hdfs.DFSClient:
> >> >>>> > > > >>>> Abandoning
> >> >>>> > > > >>>> block blk_8219260302213892894_5478064
> >> >>>> > > > >>>> 2009-11-13 20:06:30,855 INFO
> >> org.apache.hadoop.hdfs.DFSClient:
> >> >>>> > > > Exception
> >> >>>> > > > >>>>
> >> >>>> > > > >>> in
> >> >>>> > > > >>>
> >> >>>> > > > >>>> createBlockOutputStream java.net.SocketException: Too
> many
> >> >>>> open
> >> >>>> > > files
> >> >>>> > > > >>>> 2009-11-13 20:06:30,855 INFO
> >> org.apache.hadoop.hdfs.DFSClient:
> >> >>>> > > > >>>> Abandoning
> >> >>>> > > > >>>> block blk_1669205542853067709_5478235
> >> >>>> > > > >>>> 2009-11-13 20:06:30,905 INFO
> >> org.apache.hadoop.hdfs.DFSClient:
> >> >>>> > > > Exception
> >> >>>> > > > >>>>
> >> >>>> > > > >>> in
> >> >>>> > > > >>>
> >> >>>> > > > >>>> createBlockOutputStream java.net.SocketException: Too
> many
> >> >>>> open
> >> >>>> > > files
> >> >>>> > > > >>>> 2009-11-13 20:06:30,905 INFO
> >> org.apache.hadoop.hdfs.DFSClient:
> >> >>>> > > > >>>> Abandoning
> >> >>>> > > > >>>> block blk_9128897691346270351_5478237
> >> >>>> > > > >>>> 2009-11-13 20:06:30,955 INFO
> >> org.apache.hadoop.hdfs.DFSClient:
> >> >>>> > > > Exception
> >> >>>> > > > >>>>
> >> >>>> > > > >>> in
> >> >>>> > > > >>>
> >> >>>> > > > >>>> createBlockOutputStream java.net.SocketException: Too
> many
> >> >>>> open
> >> >>>> > > files
> >> >>>> > > > >>>> 2009-11-13 20:06:30,955 INFO
> >> org.apache.hadoop.hdfs.DFSClient:
> >> >>>> > > > >>>> Abandoning
> >> >>>> > > > >>>> block blk_1116845144864123018_5478240
> >> >>>> > > > >>>> 2009-11-13 20:06:34,372 DEBUG
> >> org.apache.zookeeper.ClientCnxn:
> >> >>>> Got
> >> >>>> > > > ping
> >> >>>> > > > >>>> response for sessionid:0x424eebf1c10004c after 0ms
> >> >>>> > > > >>>> 2009-11-13 20:06:37,034 DEBUG
> >> org.apache.zookeeper.ClientCnxn:
> >> >>>> Got
> >> >>>> > > > ping
> >> >>>> > > > >>>> response for sessionid:0x24eebf1043003c after 0ms
> >> >>>> > > > >>>> 2009-11-13 20:06:37,235 WARN
> >> org.apache.hadoop.hdfs.DFSClient:
> >> >>>> > > > >>>>
> >> >>>> > > > >>> DataStreamer
> >> >>>> > > > >>>
> >> >>>> > > > >>>> Exception: java.io.IOException: Too many open files
> >> >>>> > > > >>>> at sun.nio.ch.IOUtil.initPipe(Native Method)
> >> >>>> > > > >>>> at
> >> >>>> > > sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:49)
> >> >>>> > > > >>>> at
> >> >>>> > > > >>>>
> >> >>>> > > > >>>>
> >> >>>> > > >
> >> >>>> > >
> >> >>>> >
> >> >>>>
> >>
> sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:18)
> >> >>>> > > > >>>
> >> >>>> > > > >>>> at
> >> >>>> > > > >>>>
> >> >>>> > > > >>>>
> >> >>>> > > >
> >> >>>> > >
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.get(SocketIOWithTimeout.java:407)
> >> >>>> > > > >>>
> >> >>>> > > > >>>> at
> >> >>>> > > > >>>>
> >> >>>> > > > >>>>
> >> >>>> > > >
> >> >>>> > >
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:322)
> >> >>>> > > > >>>
> >> >>>> > > > >>>> at
> >> >>>> > > > >>>>
> >> >>>> > > > >>>>
> >> >>>> > > >
> >> >>>> > >
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
> >> >>>> > > > >>>
> >> >>>> > > > >>>> at
> >> >>>> > > > >>>>
> >> >>>> > > > >>>>
> >> >>>> > > >
> >> >>>> > >
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
> >> >>>> > > > >>>
> >> >>>> > > > >>>> at
> >> >>>> > > > >>>>
> >> >>>> > > > >>>>
> >> >>>> > > >
> >> >>>> > >
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
> >> >>>> > > > >>>
> >> >>>> > > > >>>> at
> >> >>>> > > > >>>>
> >> >>>> > > > >>>
> >> >>>> java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
> >> >>>> > > > >>>
> >> >>>> > > > >>>> at
> >> >>>> java.io.DataOutputStream.write(DataOutputStream.java:90)
> >> >>>> > > > >>>> at
> >> >>>> > > > >>>>
> >> >>>> > > > >>>>
> >> >>>> > > >
> >> >>>> > >
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2290)
> >> >>>> > > > >>>
> >> >>>> > > > >>>>
> >> >>>> > > > >>>> 2009-11-13 20:06:37,235 WARN
> >> org.apache.hadoop.hdfs.DFSClient:
> >> >>>> > Error
> >> >>>> > > > >>>> Recovery for block blk_8148813491785406356_5478475 bad
> >> >>>> datanode[0]
> >> >>>> > > > >>>> 192.168.100.123:50010
> >> >>>> > > > >>>> 2009-11-13 20:06:37,235 WARN
> >> org.apache.hadoop.hdfs.DFSClient:
> >> >>>> > Error
> >> >>>> > > > >>>> Recovery for block blk_8148813491785406356_5478475 in
> >> pipeline
> >> >>>> > > > >>>> 192.168.100.123:50010, 192.168.100.134:50010,
> >> >>>> > 192.168.100.122:50010
> >> >>>> > > :
> >> >>>> > > > >>>> bad
> >> >>>> > > > >>>> datanode 192.168.100.123:50010
> >> >>>> > > > >>>> 2009-11-13 20:06:37,436 INFO
> >> org.apache.hadoop.hdfs.DFSClient:
> >> >>>> > > > Exception
> >> >>>> > > > >>>>
> >> >>>> > > > >>> in
> >> >>>> > > > >>>
> >> >>>> > > > >>>> createBlockOutputStream java.net.SocketException: Too
> many
> >> >>>> open
> >> >>>> > > files
> >> >>>> > > > >>>> 2009-11-13 20:06:37,436 INFO
> >> org.apache.hadoop.hdfs.DFSClient:
> >> >>>> > > > >>>> Abandoning
> >> >>>> > > > >>>> block blk_2119727700857186236_5478498
> >> >>>> > > > >>>>
> >> >>>> > > > >>>>
> >> >>>> > > > >>>>
> >> >>>> > > > >>>>
> >> >>>> > > > >>>>
> >> >>>> > > > >>>> On Thu, Nov 12, 2009 at 4:21 PM, Zhenyu Zhong <
> >> >>>> > > > zhongresearch@gmail.com
> >> >>>> > > > >>>> wrote:
> >> >>>> > > > >>>>
> >> >>>> > > > >>>> Will do.
> >> >>>> > > > >>>>>
> >> >>>> > > > >>>>> thanks
> >> >>>> > > > >>>>> zhenyu
> >> >>>> > > > >>>>>
> >> >>>> > > > >>>>>
> >> >>>> > > > >>>>> On Thu, Nov 12, 2009 at 3:33 PM, stack <
> stack@duboce.net
> >> >
> >> >>>> wrote:
> >> >>>> > > > >>>>>
> >> >>>> > > > >>>>> Enable GC logging too on this next run (see
> >> hbase-env.sh).
> >> >>>> Lets
> >> >>>> > > try
> >> >>>> > > > >>>>>>
> >> >>>> > > > >>>>> and
> >> >>>> > > > >>>
> >> >>>> > > > >>>> get
> >> >>>> > > > >>>>>> to the bottom of whats going on.
> >> >>>> > > > >>>>>> Thanks,
> >> >>>> > > > >>>>>> St.Ack
> >> >>>> > > > >>>>>>
> >> >>>> > > > >>>>>> On Thu, Nov 12, 2009 at 12:29 PM, Zhenyu Zhong <
> >> >>>> > > > >>>>>>
> >> >>>> > > > >>>>> zhongresearch@gmail.com
> >> >>>> > > > >>>
> >> >>>> > > > >>>> wrote:
> >> >>>> > > > >>>>>>>
> >> >>>> > > > >>>>>>
> >> >>>> > > > >>>>>> I can switch the boxes that run zookeeper with the
> ones
> >> >>>> that
> >> >>>> > run
> >> >>>> > > > >>>>>>> regionservers.
> >> >>>> > > > >>>>>>> I will see the results later.
> >> >>>> > > > >>>>>>>
> >> >>>> > > > >>>>>>> FYI. The node does have the 10 minutes
> >> >>>> > zookeeper.session.timeout
> >> >>>> > > > >>>>>>>
> >> >>>> > > > >>>>>> value
> >> >>>> > > > >>>
> >> >>>> > > > >>>> in
> >> >>>> > > > >>>>>>
> >> >>>> > > > >>>>>>> place.
> >> >>>> > > > >>>>>>>
> >> >>>> > > > >>>>>>> thanks
> >> >>>> > > > >>>>>>> zhenyu
> >> >>>> > > > >>>>>>>
> >> >>>> > > > >>>>>>> On Thu, Nov 12, 2009 at 3:21 PM, stack <
> >> stack@duboce.net>
> >> >>>> > wrote:
> >> >>>> > > > >>>>>>>
> >> >>>> > > > >>>>>>> On Thu, Nov 12, 2009 at 11:50 AM, Zhenyu Zhong <
> >> >>>> > > > >>>>>>>>
> >> >>>> > > > >>>>>>> zhongresearch@gmail.com
> >> >>>> > > > >>>>>>
> >> >>>> > > > >>>>>>> wrote:
> >> >>>> > > > >>>>>>>>>
> >> >>>> > > > >>>>>>>>
> >> >>>> > > > >>>>>>>>
> >> >>>> > > > >>>>>>>>> In my cluster, half of the cluster have 2 disks
> 400GB
> >> >>>> each
> >> >>>> > per
> >> >>>> > > > >>>>>>>>>
> >> >>>> > > > >>>>>>>> machine,
> >> >>>> > > > >>>>>>
> >> >>>> > > > >>>>>>> and
> >> >>>> > > > >>>>>>>>
> >> >>>> > > > >>>>>>>>> half of the cluster have 6 disks per machine.
> Maybe
> >> we
> >> >>>> > should
> >> >>>> > > > >>>>>>>>>
> >> >>>> > > > >>>>>>>> run
> >> >>>> > > > >>>
> >> >>>> > > > >>>> zookeeper
> >> >>>> > > > >>>>>>>>> on the machines with 2 disks and RS on machines
> with
> >> 6
> >> >>>> disks?
> >> >>>> > > > >>>>>>>>>
> >> >>>> > > > >>>>>>>>> That would make most sense only in the below, it
> >> looks
> >> >>>> like
> >> >>>> > the
> >> >>>> > > > >>>>>>>>>
> >> >>>> > > > >>>>>>>> RS
> >> >>>> > > > >>>
> >> >>>> > > > >>>> that
> >> >>>> > > > >>>>>>
> >> >>>> > > > >>>>>>> had
> >> >>>> > > > >>>>>>>> issue had 4 disks?
> >> >>>> > > > >>>>>>>>
> >> >>>> > > > >>>>>>>>
> >> >>>> > > > >>>>>>>>
> >> >>>> > > > >>>>>>>> BTW, the 10 minutes zookeeper.session.timeout has
> >> been
> >> >>>> set
> >> >>>> > > during
> >> >>>> > > > >>>>>>>>>
> >> >>>> > > > >>>>>>>> the
> >> >>>> > > > >>>>>>
> >> >>>> > > > >>>>>>> experiment.
> >> >>>> > > > >>>>>>>>>
> >> >>>> > > > >>>>>>>>>
> >> >>>> > > > >>>>>>>> And for sure this node had it in place?
> >> >>>> > > > >>>>>>>> St.Ack
> >> >>>> > > > >>>>>>>>
> >> >>>> > > > >>>>>>>>
> >> >>>> > > > >>>>>>>>
> >> >>>> > > > >>>>>>>>
> >> >>>> > > > >>>>>>>>> thanks
> >> >>>> > > > >>>>>>>>> zhenyu
> >> >>>> > > > >>>>>>>>>
> >> >>>> > > > >>>>>>>>> On Thu, Nov 12, 2009 at 2:08 PM, stack <
> >> stack@duboce.net
> >> >>>> >
> >> >>>> > > wrote:
> >> >>>> > > > >>>>>>>>>
> >> >>>> > > > >>>>>>>>> On Thu, Nov 12, 2009 at 8:40 AM, Zhenyu Zhong <
> >> >>>> > > > >>>>>>>>>>
> >> >>>> > > > >>>>>>>>> zhongresearch@gmail.com
> >> >>>> > > > >>>>>>>
> >> >>>> > > > >>>>>>>> wrote:
> >> >>>> > > > >>>>>>>>>>>
> >> >>>> > > > >>>>>>>>>>
> >> >>>> > > > >>>>>>>>>>
> >> >>>> > > > >>>>>>>>>> Though I experienced 2 regionserver disconnection
> >> this
> >> >>>> > > > >>>>>>>>>>>
> >> >>>> > > > >>>>>>>>>> morning,
> >> >>>> > > > >>>
> >> >>>> > > > >>>> it
> >> >>>> > > > >>>>>>
> >> >>>> > > > >>>>>>> looks
> >> >>>> > > > >>>>>>>>>
> >> >>>> > > > >>>>>>>>>> better from the regionserver log. (please see the
> >> >>>> following
> >> >>>> > > > >>>>>>>>>>>
> >> >>>> > > > >>>>>>>>>> log)
> >> >>>> > > > >>>
> >> >>>> > > > >>>>
> >> >>>> > > > >>>>>>>>>>>
> >> >>>> > > > >>>>>>>>>>> http://pastebin.com/m496dbfae
> >> >>>> > > > >>>>>>>>>>>
> >> >>>> > > > >>>>>>>>>>>
> >> >>>> > > > >>>>>>>>>>> I checked diskIO for one of the
> >> >>>> > regionserver(192.168.100.116)
> >> >>>> > > > >>>>>>>>>>>
> >> >>>> > > > >>>>>>>>>> during
> >> >>>> > > > >>>>>>>
> >> >>>> > > > >>>>>>>> the
> >> >>>> > > > >>>>>>>>>
> >> >>>> > > > >>>>>>>>>> time it disconnected.
> >> >>>> > > > >>>>>>>>>>>
> >> >>>> > > > >>>>>>>>>>> Time: 03:04:58 AM
> >> >>>> > > > >>>>>>>>>>> Device: tps Blk_read/s Blk_wrtn/s
> >> >>>> Blk_read
> >> >>>> > > > >>>>>>>>>>>
> >> >>>> > > > >>>>>>>>>> Blk_wrtn
> >> >>>> > > > >>>>>>>>
> >> >>>> > > > >>>>>>>>> sda 105.31 5458.83 19503.64
> >> >>>> 9043873239
> >> >>>> > > > >>>>>>>>>>>
> >> >>>> > > > >>>>>>>>>> 32312473676
> >> >>>> > > > >>>>>>>>
> >> >>>> > > > >>>>>>>>> sda1 2.90 6.64 99.25
> >> >>>> 10993934
> >> >>>> > > > >>>>>>>>>>>
> >> >>>> > > > >>>>>>>>>> 164433464
> >> >>>> > > > >>>>>>>>
> >> >>>> > > > >>>>>>>>> sda2 1.72 23.76 30.25
> >> >>>> 39365817
> >> >>>> > > > >>>>>>>>>>>
> >> >>>> > > > >>>>>>>>>> 50124008
> >> >>>> > > > >>>>>>>>
> >> >>>> > > > >>>>>>>>> sda3 0.30 0.38 3.58
> >> >>>> 624930
> >> >>>> > > > >>>>>>>>>>>
> >> >>>> > > > >>>>>>>>>> 5923000
> >> >>>> > > > >>>>>>>>
> >> >>>> > > > >>>>>>>>> sda4 100.39 5428.06 19370.56
> >> >>>> 8992888270
> >> >>>> > > > >>>>>>>>>>>
> >> >>>> > > > >>>>>>>>>> 32091993204
> >> >>>> > > > >>>>>>>>
> >> >>>> > > > >>>>>>>>>
> >> >>>> > > > >>>>>>>>>>>
> >> >>>> > > > >>>>>>>>>>> Is this high for you? 20k blocks/second would
> >> seem to
> >> >>>> be
> >> >>>> > > high
> >> >>>> > > > >>>>>>>>>>
> >> >>>> > > > >>>>>>>>> but
> >> >>>> > > > >>>
> >> >>>> > > > >>>> its
> >> >>>> > > > >>>>>>>
> >> >>>> > > > >>>>>>>> one
> >> >>>> > > > >>>>>>>>>
> >> >>>> > > > >>>>>>>>>> disk only and its not being shared by zk anymore
> so
> >> >>>> > shouldn't
> >> >>>> > > > >>>>>>>>>>
> >> >>>> > > > >>>>>>>>> matter?
> >> >>>> > > > >>>>>>
> >> >>>> > > > >>>>>>>
> >> >>>> > > > >>>>>>>>>>
> >> >>>> > > > >>>>>>>>>> I also checked the zookeeper quorum server that
> the
> >> >>>> > > > >>>>>>>>>>>
> >> >>>> > > > >>>>>>>>>> regionserver
> >> >>>> > > > >>>
> >> >>>> > > > >>>> tried
> >> >>>> > > > >>>>>>>>
> >> >>>> > > > >>>>>>>>> to
> >> >>>> > > > >>>>>>>>>
> >> >>>> > > > >>>>>>>>>> connect according to the log. However, I don't see
> >> >>>> > > > >>>>>>>>>>>
> >> >>>> > > > >>>>>>>>>> 192.168.100.116
> >> >>>> > > > >>>>>>
> >> >>>> > > > >>>>>>> in
> >> >>>> > > > >>>>>>>
> >> >>>> > > > >>>>>>>> the
> >> >>>> > > > >>>>>>>>>
> >> >>>> > > > >>>>>>>>>> client list of the zookeeper quorum server that
> the
> >> >>>> > > > >>>>>>>>>>>
> >> >>>> > > > >>>>>>>>>> regionserver
> >> >>>> > > > >>>
> >> >>>> > > > >>>> tried
> >> >>>> > > > >>>>>>>>
> >> >>>> > > > >>>>>>>>> to
> >> >>>> > > > >>>>>>>>>
> >> >>>> > > > >>>>>>>>>> connect.
> >> >>>> > > > >>>>>>>>>>> Would that be a problem?
> >> >>>> > > > >>>>>>>>>>>
> >> >>>> > > > >>>>>>>>>>>
> >> >>>> > > > >>>>>>>>>>> Is that because the ephemeral node for the
> >> >>>> regionserver
> >> >>>> > had
> >> >>>> > > > >>>>>>>>>>
> >> >>>> > > > >>>>>>>>> evaporated?
> >> >>>> > > > >>>>>>>
> >> >>>> > > > >>>>>>>> Lost
> >> >>>> > > > >>>>>>>>>> its lease w/ zk by the time you went to look?
> >> >>>> > > > >>>>>>>>>>
> >> >>>> > > > >>>>>>>>>>
> >> >>>> > > > >>>>>>>>>>
> >> >>>> > > > >>>>>>>>>>
> >> >>>> > > > >>>>>>>>>> Thu Nov 12 15:04:52 UTC 2009
> >> >>>> > > > >>>>>>>>>>> Zookeeper version: 3.2.1-808558, built on
> >> 08/27/2009
> >> >>>> 18:48
> >> >>>> > > > >>>>>>>>>>>
> >> >>>> > > > >>>>>>>>>> GMT
> >> >>>> > > > >>>
> >> >>>> > > > >>>> Clients:
> >> >>>> > > > >>>>>>>>>>> /192.168.100.127:43045
> >> [1](queued=0,recved=26,sent=0)
> >> >>>> > > > >>>>>>>>>>> /192.168.100.131:39091
> >> [1](queued=0,recved=964,sent=0)
> >> >>>> > > > >>>>>>>>>>> /192.168.100.124:35961
> >> [1](queued=0,recved=3266,sent=0)
> >> >>>> > > > >>>>>>>>>>> /192.168.100.123:47935
> >> [1](queued=0,recved=233,sent=0)
> >> >>>> > > > >>>>>>>>>>> /192.168.100.125:46931
> >> [1](queued=0,recved=2,sent=0)
> >> >>>> > > > >>>>>>>>>>> /192.168.100.118:54924
> >> [1](queued=0,recved=295,sent=0)
> >> >>>> > > > >>>>>>>>>>> /192.168.100.118:41390
> >> [1](queued=0,recved=2290,sent=0)
> >> >>>> > > > >>>>>>>>>>> /192.168.100.136:42243
> >> [1](queued=0,recved=0,sent=0)
> >> >>>> > > > >>>>>>>>>>>
> >> >>>> > > > >>>>>>>>>>> Latency min/avg/max: 0/17/6333
> >> >>>> > > > >>>>>>>>>>> Received: 47111
> >> >>>> > > > >>>>>>>>>>> Sent: 0
> >> >>>> > > > >>>>>>>>>>> Outstanding: 0
> >> >>>> > > > >>>>>>>>>>> Zxid: 0x77000083f4
> >> >>>> > > > >>>>>>>>>>> Mode: leader
> >> >>>> > > > >>>>>>>>>>> Node count: 23
> >> >>>> > > > >>>>>>>>>>>
> >> >>>> > > > >>>>>>>>>>>
> >> >>>> > > > >>>>>>>>>>
> >> >>>> > > > >>>>>>>>>> That 6 second maximum latency is pretty bad but
> >> should
> >> >>>> be
> >> >>>> > well
> >> >>>> > > > >>>>>>>>>>
> >> >>>> > > > >>>>>>>>> within
> >> >>>> > > > >>>>>>
> >> >>>> > > > >>>>>>> the
> >> >>>> > > > >>>>>>>>
> >> >>>> > > > >>>>>>>>> zk
> >> >>>> > > > >>>>>>>>>> session timeout.
> >> >>>> > > > >>>>>>>>>>
> >> >>>> > > > >>>>>>>>>> So, problem is likely on the zk client side of the
> >> >>>> session;
> >> >>>> > > > >>>>>>>>>>
> >> >>>> > > > >>>>>>>>> i.e.
> >> >>>> > > > >>>
> >> >>>> > > > >>>> in
> >> >>>> > > > >>>>>>
> >> >>>> > > > >>>>>>> the
> >> >>>> > > > >>>>>>>
> >> >>>> > > > >>>>>>>> RS.
> >> >>>> > > > >>>>>>>>>
> >> >>>> > > > >>>>>>>>>> You could enable GC logging as J-D suggested to
> see
> >> if
> >> >>>> you
> >> >>>> > > have
> >> >>>> > > > >>>>>>>>>>
> >> >>>> > > > >>>>>>>>> any
> >> >>>> > > > >>>>>>
> >> >>>> > > > >>>>>>> big
> >> >>>> > > > >>>>>>>
> >> >>>> > > > >>>>>>>> pauses, pauses > zk session timeout.
> >> >>>> > > > >>>>>>>>>>
> >> >>>> > > > >>>>>>>>>> When the RS went down, it didn't look too heavily
> >> >>>> loaded:
> >> >>>> > > > >>>>>>>>>>
> >> >>>> > > > >>>>>>>>>>
> >> >>>> > > > >>>>>>>>>> 1. 2009-11-12 15:04:52,830 INFO
> >> >>>> > > > >>>>>>>>>>
> org.apache.hadoop.hbase.regionserver.HRegionServer:
> >> >>>> Dump of
> >> >>>> > > > >>>>>>>>>>
> >> >>>> > > > >>>>>>>>> metrics:
> >> >>>> > > > >>>>>>>
> >> >>>> > > > >>>>>>>> request=1.5166667, regions=322, stores=657,
> >> >>>> storefiles=631,
> >> >>>> > > > >>>>>>>>>> storefileIndexSize=61, memstoreSize=1472,
> >> >>>> usedHeap=2819,
> >> >>>> > > > >>>>>>>>>>
> >> >>>> > > > >>>>>>>>> maxHeap=4079,
> >> >>>> > > > >>>>>>>>
> >> >>>> > > > >>>>>>>>> blockCacheSize=658110960,
> blockCacheFree=197395984,
> >> >>>> > > > >>>>>>>>>>
> >> >>>> > > > >>>>>>>>> blockCacheCount=9903,
> >> >>>> > > > >>>>>>>>>
> >> >>>> > > > >>>>>>>>>> blockCacheHitRatio=99
> >> >>>> > > > >>>>>>>>>>
> >> >>>> > > > >>>>>>>>>> Its serving a few reads? The number of store
> files
> >> >>>> seems
> >> >>>> > > fine.
> >> >>>> > > > >>>>>>>>>>
> >> >>>> > > > >>>>>>>>> Not
> >> >>>> > > > >>>>>>
> >> >>>> > > > >>>>>>> too
> >> >>>> > > > >>>>>>>>
> >> >>>> > > > >>>>>>>>> much memory used.
> >> >>>> > > > >>>>>>>>>>
> >> >>>> > > > >>>>>>>>>> Looking at the logs, I see the Lease Still Held
> >> >>>> exception.
> >> >>>> > > > >>>>>>>>>>
> >> >>>> > > > >>>>>>>>> This
> >> >>>> > > > >>>
> >> >>>> > > > >>>> happens
> >> >>>> > > > >>>>>>>>
> >> >>>> > > > >>>>>>>>> when the RS does its regular report to the master
> but
> >> the
> >> >>>> > > > >>>>>>>>>>
> >> >>>> > > > >>>>>>>>> master
> >> >>>> > > > >>>
> >> >>>> > > > >>>> thinks
> >> >>>> > > > >>>>>>>
> >> >>>> > > > >>>>>>>> the
> >> >>>> > > > >>>>>>>>>
> >> >>>> > > > >>>>>>>>>> RS has since restarted. It'll think this probably
> >> >>>> because
> >> >>>> > it
> >> >>>> > > > >>>>>>>>>>
> >> >>>> > > > >>>>>>>>> noticed
> >> >>>> > > > >>>>>>
> >> >>>> > > > >>>>>>> that
> >> >>>> > > > >>>>>>>>>
> >> >>>> > > > >>>>>>>>>> the RS's znode in zk had gone away and it
> considered
> >> the
> >> >>>> RS
> >> >>>> > > > >>>>>>>>>>
> >> >>>> > > > >>>>>>>>> dead.
> >> >>>> > > > >>>
> >> >>>> > > > >>>>
> >> >>>> > > > >>>>>>>>>> Looking too at your logs I see this gap in the zk
> >> >>>> pinging:
> >> >>>> > > > >>>>>>>>>>
> >> >>>> > > > >>>>>>>>>>
> >> >>>> > > > >>>>>>>>>> 1. 2009-11-12 15:03:39,325 DEBUG
> >> >>>> > > > >>>>>>>>>>
> >> >>>> > > > >>>>>>>>> org.apache.zookeeper.ClientCnxn:
> >> >>>> > > > >>>>>>
> >> >>>> > > > >>>>>>> Got
> >> >>>> > > > >>>>>>>
> >> >>>> > > > >>>>>>>> ping response for sessionid:0x224e55436ad0004 after
> >> 0ms
> >> >>>> > > > >>>>>>>>>> 2. 2009-11-12 15:03:43,113 DEBUG
> >> >>>> > > > >>>>>>>>>>
> >> >>>> > > > >>>>>>>>> org.apache.zookeeper.ClientCnxn:
> >> >>>> > > > >>>>>>
> >> >>>> > > > >>>>>>> Got
> >> >>>> > > > >>>>>>>
> >> >>>> > > > >>>>>>>> ping response for sessionid:0x24e55436a0007d after
> >> 0ms
> >> >>>> > > > >>>>>>>>>>
> >> >>>> > > > >>>>>>>>>> Where in the lines above it, its reporting about
> >> every
> >> >>>> ten
> >> >>>> > > > >>>>>>>>>>
> >> >>>> > > > >>>>>>>>> seconds,
> >> >>>> > > > >>>>>>
> >> >>>> > > > >>>>>>> here
> >> >>>> > > > >>>>>>>>
> >> >>>> > > > >>>>>>>>> there is a big gap.
> >> >>>> > > > >>>>>>>>>>
> >> >>>> > > > >>>>>>>>>> Do you have ganglia or something that will let you
> >> look
> >> >>>> more
> >> >>>> > > > >>>>>>>>>>
> >> >>>> > > > >>>>>>>>> into
> >> >>>> > > > >>>
> >> >>>> > > > >>>> what
> >> >>>> > > > >>>>>>>
> >> >>>> > > > >>>>>>>> was
> >> >>>> > > > >>>>>>>>>
> >> >>>> > > > >>>>>>>>>> happening on this machine around this time? Is
> the
> >> >>>> machine
> >> >>>> > > OK?
> >> >>>> > > > >>>>>>>>>>
> >> >>>> > > > >>>>>>>>> It
> >> >>>> > > > >>>>>>
> >> >>>> > > > >>>>>>> looks
> >> >>>> > > > >>>>>>>>
> >> >>>> > > > >>>>>>>>> lightly loaded and you have your cluster nicely
> laid
> >> out.
> >> >>>> > > > >>>>>>>>>>
> >> >>>> > > > >>>>>>>>> Something
> >> >>>> > > > >>>>>>
> >> >>>> > > > >>>>>>> odd
> >> >>>> > > > >>>>>>>>
> >> >>>> > > > >>>>>>>>> is
> >> >>>> > > > >>>>>>>>>
> >> >>>> > > > >>>>>>>>>> going on. What about things like the write speed
> to
> >> >>>> disk?
> >> >>>> > In
> >> >>>> > > > >>>>>>>>>>
> >> >>>> > > > >>>>>>>>> the
> >> >>>> > > > >>>
> >> >>>> > > > >>>> past
> >> >>>> > > > >>>>>>>
> >> >>>> > > > >>>>>>>> strange issues have been explained by incorrectly
> set
> >> BIOS
> >> >>>> > > > >>>>>>>>>>
> >> >>>> > > > >>>>>>>>> which
> >> >>>> > > > >>>
> >> >>>> > > > >>>> made
> >> >>>> > > > >>>>>>
> >> >>>> > > > >>>>>>> disks
> >> >>>> > > > >>>>>>>>>
> >> >>>> > > > >>>>>>>>>> run at 1/100th of their proper speed.
> >> >>>> > > > >>>>>>>>>>
> >> >>>> > > > >>>>>>>>>> St.Ack
> >> >>>> > > > >>>>>>>>>>
> >> >>>> > > > >>>>>>>>>>
> >> >>>> > > > >>>>>>>>>>
> >> >>>> > > > >>>>>>>>>>
> >> >>>> > > > >>>>>>>>>>
> >> >>>> > > > >>>>>>>>>>
> >> >>>> > > > >>>>>>>>>>
> >> >>>> > > > >>>>>>>>>>
> >> >>>> > > > >>>>>>>>>>> Best,
> >> >>>> > > > >>>>>>>>>>> zhenyu
> >> >>>> > > > >>>>>>>>>>>
> >> >>>> > > > >>>>>>>>>>>
> >> >>>> > > > >>>>>>>>>>> On Wed, Nov 11, 2009 at 3:58 PM, Zhenyu Zhong <
> >> >>>> > > > >>>>>>>>>>>
> >> >>>> > > > >>>>>>>>>> zhongresearch@gmail.com
> >> >>>> > > > >>>>>>>>
> >> >>>> > > > >>>>>>>>> wrote:
> >> >>>> > > > >>>>>>>>>>>>
> >> >>>> > > > >>>>>>>>>>>
> >> >>>> > > > >>>>>>>>>>> Stack
> >> >>>> > > > >>>>>>>>>>>>
> >> >>>> > > > >>>>>>>>>>>> I am very appreciated for your comments.
> >> >>>> > > > >>>>>>>>>>>> I will use the zookeeper monitoring script on my
> >> >>>> cluster
> >> >>>> > > > >>>>>>>>>>>>
> >> >>>> > > > >>>>>>>>>>> and
> >> >>>> > > > >>>
> >> >>>> > > > >>>> let
> >> >>>> > > > >>>>>>
> >> >>>> > > > >>>>>>> it
> >> >>>> > > > >>>>>>>
> >> >>>> > > > >>>>>>>> run
> >> >>>> > > > >>>>>>>>>
> >> >>>> > > > >>>>>>>>>> overnight to see the result.
> >> >>>> > > > >>>>>>>>>>>> I will follow up the thread when I get anything.
> >> >>>> > > > >>>>>>>>>>>>
> >> >>>> > > > >>>>>>>>>>>> thanks
> >> >>>> > > > >>>>>>>>>>>> zhenyu
> >> >>>> > > > >>>>>>>>>>>>
> >> >>>> > > > >>>>>>>>>>>>
> >> >>>> > > > >>>>>>>>>>>> On Wed, Nov 11, 2009 at 3:52 PM, stack <
> >> >>>> stack@duboce.net>
> >> >>>> > > > >>>>>>>>>>>>
> >> >>>> > > > >>>>>>>>>>> wrote:
> >> >>>> > > > >>>>>>
> >> >>>> > > > >>>>>>>
> >> >>>> > > > >>>>>>>>>>>> I see these in your log too:
> >> >>>> > > > >>>>>>>>>>>>>
> >> >>>> > > > >>>>>>>>>>>>>
> >> >>>> > > > >>>>>>>>>>>>> 1. 2009-11-11 04:27:19,018 DEBUG
> >> >>>> > > > >>>>>>>>>>>>>
> >> >>>> > > > >>>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
> >> >>>> > > > >>>>>>>>
> >> >>>> > > > >>>>>>>>> Got
> >> >>>> > > > >>>>>>>>>>
> >> >>>> > > > >>>>>>>>>>> ping response for sessionid:0x424dfd908c50009
> >> after
> >> >>>> > > > >>>>>>>>>>>>>
> >> >>>> > > > >>>>>>>>>>>> 4544ms
> >> >>>> > > > >>>
> >> >>>> > > > >>>> 2. 2009-11-11 04:27:19,018 DEBUG
> >> >>>> > > > >>>>>>>>>>>>>
> >> >>>> > > > >>>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
> >> >>>> > > > >>>>>>>>
> >> >>>> > > > >>>>>>>>> Got
> >> >>>> > > > >>>>>>>>>>
> >> >>>> > > > >>>>>>>>>>> ping response for sessionid:0x24dfd90c810002
> after
> >> >>>> > > > >>>>>>>>>>>>>
> >> >>>> > > > >>>>>>>>>>>> 4548ms
> >> >>>> > > > >>>
> >> >>>> > > > >>>> 3. 2009-11-11 04:27:43,960 DEBUG
> >> >>>> > > > >>>>>>>>>>>>>
> >> >>>> > > > >>>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
> >> >>>> > > > >>>>>>>>
> >> >>>> > > > >>>>>>>>> Got
> >> >>>> > > > >>>>>>>>>>
> >> >>>> > > > >>>>>>>>>>> ping response for sessionid:0x424dfd908c50009
> >> after
> >> >>>> > > > >>>>>>>>>>>>>
> >> >>>> > > > >>>>>>>>>>>> 9030ms
> >> >>>> > > > >>>
> >> >>>> > > > >>>> 4. 2009-11-11 04:27:43,960 DEBUG
> >> >>>> > > > >>>>>>>>>>>>>
> >> >>>> > > > >>>>>>>>>>>> org.apache.zookeeper.
> >> >>>> > > > >>>>>>>>
> >> >>>> > > > >>>>>>>
> >> >>>> > > >
> >> >>>> > >
> >> >>>> >
> >> >>>>
> >> >>>
> >> >>>
> >> >>
> >> >
> >>
> >
> >
>
Re: regionserver disconnection
Posted by Zhenyu Zhong <zh...@gmail.com>.
I just realized that there was a MapReduce job running during the time the
regionserver disconnected from the zookeeper.
That MapReduce Job was processing 500GB data and took about 8 minutes to
finish. It launched over 2000 map tasks. I doubt that this introduced
resource contention between DataNode and RegionServer Node.
Also during the time that MapReduce job ran, I saw a few errors indicating
that
java.io.IOException: Could not obtain block:
blk_7661247556283580226_4450462 file=/data/xxx.txt.200910141159
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1787)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1615)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1742)
at java.io.DataInputStream.read(DataInputStream.java:83)
at org.apache.hadoop.util.LineReader.readLine(LineReader.java:134)
at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:136)
at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:40)
at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:192)
at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:176)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
Possibly it is related to HDFS-127, but I don't see any datanodes went
down. Does that affect Regionserver? Shall we apply the patch?
Now I start to keep tracking of the virtual memory stats to see if the same
issue happens tomorrow around the same time period.
And I highly suspect that this particular MapReduce job hurts HBase
Regionserver.
PS. I also use gcviewer to parse the GC-log, I only see around 30 seconds
pauses maximum.
Thanks a lot.
zhenyu
On Sun, Nov 15, 2009 at 12:52 AM, Zhenyu Zhong <zh...@gmail.com>wrote:
> J-D,
>
> Thank you very much for your comments.
> My company block the IRC port, so I have trouble to connect to IRC channel.
> I have been trying to ask the IT to open the IRC port for me, it might take
> a while.
>
> Best,
> zhenyu
>
>
> On Sat, Nov 14, 2009 at 2:21 PM, Jean-Daniel Cryans <jd...@apache.org>wrote:
>
>> The error you are getting is a disconnection from a zookeeper server
>> and is very generic.
>>
>> ZK-86 is still opened and the last comment refers to ZK-111 saying
>> that the bug (in unit tests) was probably fixed in release 3.0.0 last
>> year.
>>
>> To figure the hang you have, you can try to jstack the process pid and
>> see exactly what's holding the RS from shutting down.
>>
>> Would it be possible for you to drop by the IRC channel? This way we
>> can debug this a at much faster pace.
>>
>> Thx!
>>
>> J-D
>>
>> On Sat, Nov 14, 2009 at 9:57 AM, Zhenyu Zhong <zh...@gmail.com>
>> wrote:
>> > I found this.
>> > http://issues.apache.org/jira/browse/ZOOKEEPER-86
>> >
>> > It looks like the same error I had. Is it a zookeeper bug? When will
>> HBase
>> > take the zookeeper version 3.3.0?
>> >
>> > thanks
>> > zhenyu
>> >
>> > On Sat, Nov 14, 2009 at 11:31 AM, Zhenyu Zhong <zhongresearch@gmail.com
>> >wrote:
>> >
>> >> Now I really doubt about the zookeeper, from the log I saw errors like
>> >> IOException Read Error, while the zookeeper client (regionserver )
>> tried to
>> >> read. But it got disconneted status from zookeeper.
>> >>
>> >> I don't see any load on any zookeeper quorum servers. DiskIO is fine.
>> >>
>> >> Also when the regionserver decides to exit due to the disconnect status
>> >> from zookeeper, sometimes the regionserver hangs during the exiting. We
>> can
>> >> still see the HRegionServer process even we don't see it in the Master
>> web
>> >> interface.
>> >>
>> >> I also notice that there is an zookeeper.retries and zookeeper.pauses
>> >> settings in hbase-default.xml, would that matter ?
>> >>
>> >> thanks
>> >>
>> >> You may see the IOException error in the following log when the
>> >> Regionserver lost connection to the zookeeper.
>> >>
>> >> 2009-11-14 15:58:31,993 INFO org.apache.hadoop.hbase.regionserver.HLog:
>> >> Roll
>> >>
>> /hbase/.logs/superpyxis0008.scur.colo,60021,1258179101617/hlog.dat.1258179106921,
>> >> entries=1062964, calcsize=255014936, filesize=139566093. New hlog
>> >>
>> /hbase/.logs/superpyxis0008.scur.colo,60021,1258179101617/hlog.dat.1258214311993
>> >> 2009-11-14 15:58:39,295 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
>> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
>> >> 2009-11-14 15:58:52,648 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
>> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
>> >> 2009-11-14 15:59:06,007 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
>> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
>> >> 2009-11-14 15:59:19,365 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
>> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
>> >> 2009-11-14 15:59:32,867 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
>> >> response for sessionid:0x424f0ed01ea00a6 after 186ms
>> >> 2009-11-14 15:59:46,070 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
>> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
>> >> 2009-11-14 15:59:59,423 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
>> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
>> >> 2009-11-14 16:00:12,775 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
>> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
>> >> 2009-11-14 16:00:28,141 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
>> >> response for sessionid:0x424f0ed01ea00a6 after 2010ms
>> >> 2009-11-14 16:01:10,378 WARN org.apache.zookeeper.ClientCnxn: Exception
>> >> closing session 0x424f0ed01ea00a6 to
>> sun.nio.ch.SelectionKeyImpl@39fbb2d6
>> >>
>> >> java.io.IOException: Read error rc = -1 java.nio.DirectByteBuffer[pos=0
>> >> lim=4 cap=4]
>> >> at
>> >> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
>> >> at
>> >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
>> >> 2009-11-14 16:01:10,478 INFO
>> >> org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper
>> event,
>> >> state: Disconnected, type: None, path: null
>> >> 2009-11-14 16:01:11,333 WARN
>> >> org.apache.hadoop.hbase.regionserver.HRegionServer: Attempt=1
>> >>
>> >> org.apache.hadoop.hbase.Leases$LeaseStillHeldException
>> >> at
>> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>> >> Method)
>> >> at
>> >>
>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>> >> at
>> >>
>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>> >> at
>> java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>> >> at
>> >>
>> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
>> >> at
>> >>
>> org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
>> >> at
>> >>
>> org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
>> >> at
>> >>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:571)
>> >> at java.lang.Thread.run(Thread.java:619)
>> >> 2009-11-14 16:01:11,433 INFO org.apache.zookeeper.ClientCnxn:
>> Attempting
>> >> connection to server superpyxis0001/192.168.100.115:2181
>> >> 2009-11-14 16:01:11,433 INFO org.apache.zookeeper.ClientCnxn: Priming
>> >> connection to java.nio.channels.SocketChannel[connected local=/
>> >> 192.168.100.122:59575 remote=superpyxis0001/192.168.100.115:2181]
>> >> 2009-11-14 16:01:11,433 INFO org.apache.zookeeper.ClientCnxn: Server
>> >> connection successful
>> >> 2009-11-14 16:01:11,433 INFO
>> >> org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper
>> event,
>> >> state: Expired, type: None, path: null
>> >> 2009-11-14 16:01:11,433 WARN org.apache.zookeeper.ClientCnxn: Exception
>> >> closing session 0x424f0ed01ea00a6 to
>> sun.nio.ch.SelectionKeyImpl@6b93d343
>> >> java.io.IOException: Session Expired
>> >> at
>> >>
>> org.apache.zookeeper.ClientCnxn$SendThread.readConnectResult(ClientCnxn.java:589)
>> >> at
>> >> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:709)
>> >>
>> >> at
>> >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
>> >> 2009-11-14 16:01:11,433 ERROR
>> >> org.apache.hadoop.hbase.regionserver.HRegionServer: ZooKeeper session
>> >> expired
>> >> 2009-11-14 16:01:13,513 INFO org.apache.hadoop.ipc.HBaseServer: IPC
>> Server
>> >> handler 3 on 60021, call put([B@73deb204,
>> >> [Lorg.apache.hadoop.hbase.client.Put;@2179600a) from
>> 192.168.100.132:40728:
>> >> error: java.io.IOException: Server not running, aborting
>> >> java.io.IOException: Server not running, aborting
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> On Sat, Nov 14, 2009 at 1:01 AM, Zhenyu Zhong <zhongresearch@gmail.com
>> >wrote:
>> >>
>> >>> Stack,
>> >>>
>> >>> Thanks a lot!
>> >>>
>> >>> I found out that the reason why HBase doesn't take the system file
>> >>> descriptor value I set before. I started the HBase using root instead
>> of the
>> >>> normal hadoop user, while my system configures hadoop with higher file
>> >>> descriptor value but configures root with 1024 default value.
>> >>>
>> >>>
>> >>> Now my system has a clean start. However, it seems that the
>> >>> zookeeper.session.timeout value doesn't take into effect. I still
>> found
>> >>> around 60 seconds pauses from the disconnected regionserver. I really
>> don't
>> >>> know why regionserver only times out after 60 seconds instead of 10
>> minutes
>> >>> which I set for zookeeper.session.timeout.
>> >>>
>> >>> Is there any other timeout value coming into play before the actual
>> >>> session times out?
>> >>>
>> >>> zhenyu
>> >>>
>> >>>
>> >>>
>> >>> On Fri, Nov 13, 2009 at 7:08 PM, stack <st...@duboce.net> wrote:
>> >>>
>> >>>> Ok. Lack of file descriptors manifests in all kinds of weird ways.
>> >>>> Hopefully thats it. If not, lets keep digging.
>> >>>> St.Ack
>> >>>>
>> >>>> On Fri, Nov 13, 2009 at 3:44 PM, Zhenyu Zhong <
>> zhongresearch@gmail.com
>> >>>> >wrote:
>> >>>>
>> >>>> > Stack,
>> >>>> >
>> >>>> > You are right, the master started with ulimit -n 1024. It doesn't
>> take
>> >>>> the
>> >>>> > system value.
>> >>>> >
>> >>>> > Regarding to the too many open files, it looks like the same as the
>> one
>> >>>> J-D
>> >>>> > put up. But I will get the Master start with higher value first and
>> see
>> >>>> if
>> >>>> > this kind of error goes away.
>> >>>> >
>> >>>> > thanks a lot!
>> >>>> > zhenyu
>> >>>> >
>> >>>> > On Fri, Nov 13, 2009 at 6:02 PM, stack <st...@duboce.net> wrote:
>> >>>> >
>> >>>> > > Does it say
>> >>>> > >
>> >>>> > > ulimit -n 32768
>> >>>> > >
>> >>>> > > ...as the second line in your log file on start of the master?
>> >>>> > >
>> >>>> > > Is the stack trace that complains about too many open files same
>> as
>> >>>> the
>> >>>> > one
>> >>>> > > in the blog post J-D put up?
>> >>>> > >
>> >>>> > > St.Ack
>> >>>> > >
>> >>>> > >
>> >>>> > > On Fri, Nov 13, 2009 at 1:37 PM, Zhenyu Zhong <
>> >>>> zhongresearch@gmail.com
>> >>>> > > >wrote:
>> >>>> > >
>> >>>> > > > The ulimit file descriptors was set to fs.file-max =1578334,
>> also
>> >>>> in
>> >>>> > the
>> >>>> > > > limits.conf the value was set to 32768.
>> >>>> > > > So these are way higher than the open descriptors for the
>> running
>> >>>> > > > processes.
>> >>>> > > >
>> >>>> > > > thanks
>> >>>> > > > zhenyu
>> >>>> > > >
>> >>>> > > >
>> >>>> > > > On Fri, Nov 13, 2009 at 4:33 PM, Stack <sa...@gmail.com>
>> >>>> wrote:
>> >>>> > > >
>> >>>> > > > > You upped the ulimit file descriptors as per the getting
>> started
>> >>>> doc?
>> >>>> > > > >
>> >>>> > > > >
>> >>>> > > > >
>> >>>> > > > > On Nov 13, 2009, at 1:26 PM, Zhenyu Zhong <
>> >>>> zhongresearch@gmail.com>
>> >>>> > > > wrote:
>> >>>> > > > >
>> >>>> > > > > Thanks a lot.
>> >>>> > > > >>
>> >>>> > > > >>
>> >>>> > > > >> Bad news is my kernel is still 2.6.26.
>> >>>> > > > >> But it was not a problem before.
>> >>>> > > > >>
>> >>>> > > > >> Very strange.
>> >>>> > > > >>
>> >>>> > > > >> zhenyu
>> >>>> > > > >>
>> >>>> > > > >> On Fri, Nov 13, 2009 at 4:16 PM, Jean-Daniel Cryans <
>> >>>> > > > jdcryans@apache.org
>> >>>> > > > >> >wrote:
>> >>>> > > > >>
>> >>>> > > > >> Looks like
>> >>>> > > > >>>
>> >>>> > > > >>>
>> >>>> > > >
>> >>>> > >
>> >>>> >
>> >>>>
>> http://pero.blogs.aprilmayjune.org/2009/01/22/hadoop-and-linux-kernel-2627-epoll-limits/
>> >>>> > > > >>>
>> >>>> > > > >>> J-D
>> >>>> > > > >>>
>> >>>> > > > >>> On Fri, Nov 13, 2009 at 1:12 PM, Zhenyu Zhong <
>> >>>> > > zhongresearch@gmail.com
>> >>>> > > > >
>> >>>> > > > >>> wrote:
>> >>>> > > > >>>
>> >>>> > > > >>>> Hi,
>> >>>> > > > >>>>
>> >>>> > > > >>>>
>> >>>> > > > >>>> After I re-organize the cluster, the experiment ran into
>> >>>> problem
>> >>>> > > > faster
>> >>>> > > > >>>>
>> >>>> > > > >>> than
>> >>>> > > > >>>
>> >>>> > > > >>>> before.
>> >>>> > > > >>>>
>> >>>> > > > >>>> Basically the changes are to use machines with less
>> resources
>> >>>> as
>> >>>> > > > >>>>
>> >>>> > > > >>> zookeeper
>> >>>> > > > >>>
>> >>>> > > > >>>> quorums and machines with more resources as regionserver.
>> >>>> > > > >>>>
>> >>>> > > > >>>> From the log, I still see the pause around 1 minute.
>> >>>> > > > >>>> I enabled the GC logging, please see
>> >>>> > > > >>>>
>> >>>> > > > >>>> http://pastebin.com/m1d4ce0f1
>> >>>> > > > >>>>
>> >>>> > > > >>>> for details.
>> >>>> > > > >>>> In general I don't see many pauses in the GC.
>> >>>> > > > >>>>
>> >>>> > > > >>>> What is more interesting, 1 hour after the 1st
>> regionserver
>> >>>> > > > >>>> disconnected,
>> >>>> > > > >>>> the master log complained about too many open files. This
>> >>>> didn't
>> >>>> > > > happen
>> >>>> > > > >>>> before.
>> >>>> > > > >>>> I checked the system OS setup as well as the limits.conf.
>> I
>> >>>> also
>> >>>> > > > checked
>> >>>> > > > >>>>
>> >>>> > > > >>> the
>> >>>> > > > >>>
>> >>>> > > > >>>> running processes. The total open files don't reach the
>> limit.
>> >>>> I
>> >>>> > am
>> >>>> > > > >>>>
>> >>>> > > > >>> confused
>> >>>> > > > >>>
>> >>>> > > > >>>> a bit.
>> >>>> > > > >>>>
>> >>>> > > > >>>>
>> >>>> > > > >>>> Please see the following master log.
>> >>>> > > > >>>>
>> >>>> > > > >>>> 2009-11-13 20:06:02,114 INFO
>> >>>> > > > org.apache.hadoop.hbase.master.BaseScanner:
>> >>>> > > > >>>> RegionManager.metaScanner scan of 4658 row(s) of meta
>> region
>> >>>> > > {server:
>> >>>> > > > >>>> 192.168.100.128:60021, regionname: .META.,,1, startKey:
>> <>}
>> >>>> > > complete
>> >>>> > > > >>>> 2009-11-13 20:06:02,114 INFO
>> >>>> > > > org.apache.hadoop.hbase.master.BaseScanner:
>> >>>> > > > >>>>
>> >>>> > > > >>> All
>> >>>> > > > >>>
>> >>>> > > > >>>> 1 .META. region(s) scanned
>> >>>> > > > >>>> 2009-11-13 20:06:07,677 DEBUG
>> org.apache.zookeeper.ClientCnxn:
>> >>>> Got
>> >>>> > > > ping
>> >>>> > > > >>>> response for sessionid:0x424eebf1c10004c after 3ms
>> >>>> > > > >>>> 2009-11-13 20:06:08,178 INFO
>> org.apache.hadoop.hdfs.DFSClient:
>> >>>> > > > Exception
>> >>>> > > > >>>>
>> >>>> > > > >>> in
>> >>>> > > > >>>
>> >>>> > > > >>>> createBlockOutputStream java.io.IOException: Bad connect
>> ack
>> >>>> with
>> >>>> > > > >>>> firstBadLink 192.168.100.123:50010
>> >>>> > > > >>>> 2009-11-13 20:06:08,178 INFO
>> org.apache.hadoop.hdfs.DFSClient:
>> >>>> > > > >>>> Abandoning
>> >>>> > > > >>>> block blk_-2808245019291145247_5478039
>> >>>> > > > >>>> 2009-11-13 20:06:09,682 INFO
>> org.apache.hadoop.hdfs.DFSClient:
>> >>>> > > > Exception
>> >>>> > > > >>>>
>> >>>> > > > >>> in
>> >>>> > > > >>>
>> >>>> > > > >>>> createBlockOutputStream java.io.EOFException
>> >>>> > > > >>>> 2009-11-13 20:06:09,682 INFO
>> org.apache.hadoop.hdfs.DFSClient:
>> >>>> > > > >>>> Abandoning
>> >>>> > > > >>>> block blk_1074853606841896259_5478048
>> >>>> > > > >>>> 2009-11-13 20:06:10,334 DEBUG
>> org.apache.zookeeper.ClientCnxn:
>> >>>> Got
>> >>>> > > > ping
>> >>>> > > > >>>> response for sessionid:0x24eebf1043003c after 1ms
>> >>>> > > > >>>> 2009-11-13 20:06:21,018 DEBUG
>> org.apache.zookeeper.ClientCnxn:
>> >>>> Got
>> >>>> > > > ping
>> >>>> > > > >>>> response for sessionid:0x424eebf1c10004c after 0ms
>> >>>> > > > >>>> 2009-11-13 20:06:23,674 DEBUG
>> org.apache.zookeeper.ClientCnxn:
>> >>>> Got
>> >>>> > > > ping
>> >>>> > > > >>>> response for sessionid:0x24eebf1043003c after 0ms
>> >>>> > > > >>>> 2009-11-13 20:06:24,828 INFO
>> org.apache.hadoop.hdfs.DFSClient:
>> >>>> > > > Exception
>> >>>> > > > >>>>
>> >>>> > > > >>> in
>> >>>> > > > >>>
>> >>>> > > > >>>> createBlockOutputStream java.io.IOException: Bad connect
>> ack
>> >>>> with
>> >>>> > > > >>>> firstBadLink 192.168.100.123:50010
>> >>>> > > > >>>> 2009-11-13 20:06:24,828 INFO
>> org.apache.hadoop.hdfs.DFSClient:
>> >>>> > > > >>>> Abandoning
>> >>>> > > > >>>> block blk_-6642544517082142289_5478063
>> >>>> > > > >>>> 2009-11-13 20:06:24,828 INFO
>> org.apache.hadoop.hdfs.DFSClient:
>> >>>> > > > Exception
>> >>>> > > > >>>>
>> >>>> > > > >>> in
>> >>>> > > > >>>
>> >>>> > > > >>>> createBlockOutputStream java.net.SocketException: Too many
>> >>>> open
>> >>>> > > files
>> >>>> > > > >>>> 2009-11-13 20:06:24,828 INFO
>> org.apache.hadoop.hdfs.DFSClient:
>> >>>> > > > >>>> Abandoning
>> >>>> > > > >>>> block blk_2057511041109796090_5478063
>> >>>> > > > >>>> 2009-11-13 20:06:24,928 INFO
>> org.apache.hadoop.hdfs.DFSClient:
>> >>>> > > > Exception
>> >>>> > > > >>>>
>> >>>> > > > >>> in
>> >>>> > > > >>>
>> >>>> > > > >>>> createBlockOutputStream java.net.SocketException: Too many
>> >>>> open
>> >>>> > > files
>> >>>> > > > >>>> 2009-11-13 20:06:24,928 INFO
>> org.apache.hadoop.hdfs.DFSClient:
>> >>>> > > > >>>> Abandoning
>> >>>> > > > >>>> block blk_8219260302213892894_5478064
>> >>>> > > > >>>> 2009-11-13 20:06:30,855 INFO
>> org.apache.hadoop.hdfs.DFSClient:
>> >>>> > > > Exception
>> >>>> > > > >>>>
>> >>>> > > > >>> in
>> >>>> > > > >>>
>> >>>> > > > >>>> createBlockOutputStream java.net.SocketException: Too many
>> >>>> open
>> >>>> > > files
>> >>>> > > > >>>> 2009-11-13 20:06:30,855 INFO
>> org.apache.hadoop.hdfs.DFSClient:
>> >>>> > > > >>>> Abandoning
>> >>>> > > > >>>> block blk_1669205542853067709_5478235
>> >>>> > > > >>>> 2009-11-13 20:06:30,905 INFO
>> org.apache.hadoop.hdfs.DFSClient:
>> >>>> > > > Exception
>> >>>> > > > >>>>
>> >>>> > > > >>> in
>> >>>> > > > >>>
>> >>>> > > > >>>> createBlockOutputStream java.net.SocketException: Too many
>> >>>> open
>> >>>> > > files
>> >>>> > > > >>>> 2009-11-13 20:06:30,905 INFO
>> org.apache.hadoop.hdfs.DFSClient:
>> >>>> > > > >>>> Abandoning
>> >>>> > > > >>>> block blk_9128897691346270351_5478237
>> >>>> > > > >>>> 2009-11-13 20:06:30,955 INFO
>> org.apache.hadoop.hdfs.DFSClient:
>> >>>> > > > Exception
>> >>>> > > > >>>>
>> >>>> > > > >>> in
>> >>>> > > > >>>
>> >>>> > > > >>>> createBlockOutputStream java.net.SocketException: Too many
>> >>>> open
>> >>>> > > files
>> >>>> > > > >>>> 2009-11-13 20:06:30,955 INFO
>> org.apache.hadoop.hdfs.DFSClient:
>> >>>> > > > >>>> Abandoning
>> >>>> > > > >>>> block blk_1116845144864123018_5478240
>> >>>> > > > >>>> 2009-11-13 20:06:34,372 DEBUG
>> org.apache.zookeeper.ClientCnxn:
>> >>>> Got
>> >>>> > > > ping
>> >>>> > > > >>>> response for sessionid:0x424eebf1c10004c after 0ms
>> >>>> > > > >>>> 2009-11-13 20:06:37,034 DEBUG
>> org.apache.zookeeper.ClientCnxn:
>> >>>> Got
>> >>>> > > > ping
>> >>>> > > > >>>> response for sessionid:0x24eebf1043003c after 0ms
>> >>>> > > > >>>> 2009-11-13 20:06:37,235 WARN
>> org.apache.hadoop.hdfs.DFSClient:
>> >>>> > > > >>>>
>> >>>> > > > >>> DataStreamer
>> >>>> > > > >>>
>> >>>> > > > >>>> Exception: java.io.IOException: Too many open files
>> >>>> > > > >>>> at sun.nio.ch.IOUtil.initPipe(Native Method)
>> >>>> > > > >>>> at
>> >>>> > > sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:49)
>> >>>> > > > >>>> at
>> >>>> > > > >>>>
>> >>>> > > > >>>>
>> >>>> > > >
>> >>>> > >
>> >>>> >
>> >>>>
>> sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:18)
>> >>>> > > > >>>
>> >>>> > > > >>>> at
>> >>>> > > > >>>>
>> >>>> > > > >>>>
>> >>>> > > >
>> >>>> > >
>> >>>> >
>> >>>>
>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.get(SocketIOWithTimeout.java:407)
>> >>>> > > > >>>
>> >>>> > > > >>>> at
>> >>>> > > > >>>>
>> >>>> > > > >>>>
>> >>>> > > >
>> >>>> > >
>> >>>> >
>> >>>>
>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:322)
>> >>>> > > > >>>
>> >>>> > > > >>>> at
>> >>>> > > > >>>>
>> >>>> > > > >>>>
>> >>>> > > >
>> >>>> > >
>> >>>> >
>> >>>>
>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
>> >>>> > > > >>>
>> >>>> > > > >>>> at
>> >>>> > > > >>>>
>> >>>> > > > >>>>
>> >>>> > > >
>> >>>> > >
>> >>>> >
>> >>>>
>> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
>> >>>> > > > >>>
>> >>>> > > > >>>> at
>> >>>> > > > >>>>
>> >>>> > > > >>>>
>> >>>> > > >
>> >>>> > >
>> >>>> >
>> >>>>
>> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
>> >>>> > > > >>>
>> >>>> > > > >>>> at
>> >>>> > > > >>>>
>> >>>> > > > >>>
>> >>>> java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
>> >>>> > > > >>>
>> >>>> > > > >>>> at
>> >>>> java.io.DataOutputStream.write(DataOutputStream.java:90)
>> >>>> > > > >>>> at
>> >>>> > > > >>>>
>> >>>> > > > >>>>
>> >>>> > > >
>> >>>> > >
>> >>>> >
>> >>>>
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2290)
>> >>>> > > > >>>
>> >>>> > > > >>>>
>> >>>> > > > >>>> 2009-11-13 20:06:37,235 WARN
>> org.apache.hadoop.hdfs.DFSClient:
>> >>>> > Error
>> >>>> > > > >>>> Recovery for block blk_8148813491785406356_5478475 bad
>> >>>> datanode[0]
>> >>>> > > > >>>> 192.168.100.123:50010
>> >>>> > > > >>>> 2009-11-13 20:06:37,235 WARN
>> org.apache.hadoop.hdfs.DFSClient:
>> >>>> > Error
>> >>>> > > > >>>> Recovery for block blk_8148813491785406356_5478475 in
>> pipeline
>> >>>> > > > >>>> 192.168.100.123:50010, 192.168.100.134:50010,
>> >>>> > 192.168.100.122:50010
>> >>>> > > :
>> >>>> > > > >>>> bad
>> >>>> > > > >>>> datanode 192.168.100.123:50010
>> >>>> > > > >>>> 2009-11-13 20:06:37,436 INFO
>> org.apache.hadoop.hdfs.DFSClient:
>> >>>> > > > Exception
>> >>>> > > > >>>>
>> >>>> > > > >>> in
>> >>>> > > > >>>
>> >>>> > > > >>>> createBlockOutputStream java.net.SocketException: Too many
>> >>>> open
>> >>>> > > files
>> >>>> > > > >>>> 2009-11-13 20:06:37,436 INFO
>> org.apache.hadoop.hdfs.DFSClient:
>> >>>> > > > >>>> Abandoning
>> >>>> > > > >>>> block blk_2119727700857186236_5478498
>> >>>> > > > >>>>
>> >>>> > > > >>>>
>> >>>> > > > >>>>
>> >>>> > > > >>>>
>> >>>> > > > >>>>
>> >>>> > > > >>>> On Thu, Nov 12, 2009 at 4:21 PM, Zhenyu Zhong <
>> >>>> > > > zhongresearch@gmail.com
>> >>>> > > > >>>> wrote:
>> >>>> > > > >>>>
>> >>>> > > > >>>> Will do.
>> >>>> > > > >>>>>
>> >>>> > > > >>>>> thanks
>> >>>> > > > >>>>> zhenyu
>> >>>> > > > >>>>>
>> >>>> > > > >>>>>
>> >>>> > > > >>>>> On Thu, Nov 12, 2009 at 3:33 PM, stack <stack@duboce.net
>> >
>> >>>> wrote:
>> >>>> > > > >>>>>
>> >>>> > > > >>>>> Enable GC logging too on this next run (see
>> hbase-env.sh).
>> >>>> Lets
>> >>>> > > try
>> >>>> > > > >>>>>>
>> >>>> > > > >>>>> and
>> >>>> > > > >>>
>> >>>> > > > >>>> get
>> >>>> > > > >>>>>> to the bottom of whats going on.
>> >>>> > > > >>>>>> Thanks,
>> >>>> > > > >>>>>> St.Ack
>> >>>> > > > >>>>>>
>> >>>> > > > >>>>>> On Thu, Nov 12, 2009 at 12:29 PM, Zhenyu Zhong <
>> >>>> > > > >>>>>>
>> >>>> > > > >>>>> zhongresearch@gmail.com
>> >>>> > > > >>>
>> >>>> > > > >>>> wrote:
>> >>>> > > > >>>>>>>
>> >>>> > > > >>>>>>
>> >>>> > > > >>>>>> I can switch the boxes that run zookeeper with the ones
>> >>>> that
>> >>>> > run
>> >>>> > > > >>>>>>> regionservers.
>> >>>> > > > >>>>>>> I will see the results later.
>> >>>> > > > >>>>>>>
>> >>>> > > > >>>>>>> FYI. The node does have the 10 minutes
>> >>>> > zookeeper.session.timeout
>> >>>> > > > >>>>>>>
>> >>>> > > > >>>>>> value
>> >>>> > > > >>>
>> >>>> > > > >>>> in
>> >>>> > > > >>>>>>
>> >>>> > > > >>>>>>> place.
>> >>>> > > > >>>>>>>
>> >>>> > > > >>>>>>> thanks
>> >>>> > > > >>>>>>> zhenyu
>> >>>> > > > >>>>>>>
>> >>>> > > > >>>>>>> On Thu, Nov 12, 2009 at 3:21 PM, stack <
>> stack@duboce.net>
>> >>>> > wrote:
>> >>>> > > > >>>>>>>
>> >>>> > > > >>>>>>> On Thu, Nov 12, 2009 at 11:50 AM, Zhenyu Zhong <
>> >>>> > > > >>>>>>>>
>> >>>> > > > >>>>>>> zhongresearch@gmail.com
>> >>>> > > > >>>>>>
>> >>>> > > > >>>>>>> wrote:
>> >>>> > > > >>>>>>>>>
>> >>>> > > > >>>>>>>>
>> >>>> > > > >>>>>>>>
>> >>>> > > > >>>>>>>>> In my cluster, half of the cluster have 2 disks 400GB
>> >>>> each
>> >>>> > per
>> >>>> > > > >>>>>>>>>
>> >>>> > > > >>>>>>>> machine,
>> >>>> > > > >>>>>>
>> >>>> > > > >>>>>>> and
>> >>>> > > > >>>>>>>>
>> >>>> > > > >>>>>>>>> half of the cluster have 6 disks per machine. Maybe
>> we
>> >>>> > should
>> >>>> > > > >>>>>>>>>
>> >>>> > > > >>>>>>>> run
>> >>>> > > > >>>
>> >>>> > > > >>>> zookeeper
>> >>>> > > > >>>>>>>>> on the machines with 2 disks and RS on machines with
>> 6
>> >>>> disks?
>> >>>> > > > >>>>>>>>>
>> >>>> > > > >>>>>>>>> That would make most sense only in the below, it
>> looks
>> >>>> like
>> >>>> > the
>> >>>> > > > >>>>>>>>>
>> >>>> > > > >>>>>>>> RS
>> >>>> > > > >>>
>> >>>> > > > >>>> that
>> >>>> > > > >>>>>>
>> >>>> > > > >>>>>>> had
>> >>>> > > > >>>>>>>> issue had 4 disks?
>> >>>> > > > >>>>>>>>
>> >>>> > > > >>>>>>>>
>> >>>> > > > >>>>>>>>
>> >>>> > > > >>>>>>>> BTW, the 10 minutes zookeeper.session.timeout has
>> been
>> >>>> set
>> >>>> > > during
>> >>>> > > > >>>>>>>>>
>> >>>> > > > >>>>>>>> the
>> >>>> > > > >>>>>>
>> >>>> > > > >>>>>>> experiment.
>> >>>> > > > >>>>>>>>>
>> >>>> > > > >>>>>>>>>
>> >>>> > > > >>>>>>>> And for sure this node had it in place?
>> >>>> > > > >>>>>>>> St.Ack
>> >>>> > > > >>>>>>>>
>> >>>> > > > >>>>>>>>
>> >>>> > > > >>>>>>>>
>> >>>> > > > >>>>>>>>
>> >>>> > > > >>>>>>>>> thanks
>> >>>> > > > >>>>>>>>> zhenyu
>> >>>> > > > >>>>>>>>>
>> >>>> > > > >>>>>>>>> On Thu, Nov 12, 2009 at 2:08 PM, stack <
>> stack@duboce.net
>> >>>> >
>> >>>> > > wrote:
>> >>>> > > > >>>>>>>>>
>> >>>> > > > >>>>>>>>> On Thu, Nov 12, 2009 at 8:40 AM, Zhenyu Zhong <
>> >>>> > > > >>>>>>>>>>
>> >>>> > > > >>>>>>>>> zhongresearch@gmail.com
>> >>>> > > > >>>>>>>
>> >>>> > > > >>>>>>>> wrote:
>> >>>> > > > >>>>>>>>>>>
>> >>>> > > > >>>>>>>>>>
>> >>>> > > > >>>>>>>>>>
>> >>>> > > > >>>>>>>>>> Though I experienced 2 regionserver disconnection
>> this
>> >>>> > > > >>>>>>>>>>>
>> >>>> > > > >>>>>>>>>> morning,
>> >>>> > > > >>>
>> >>>> > > > >>>> it
>> >>>> > > > >>>>>>
>> >>>> > > > >>>>>>> looks
>> >>>> > > > >>>>>>>>>
>> >>>> > > > >>>>>>>>>> better from the regionserver log. (please see the
>> >>>> following
>> >>>> > > > >>>>>>>>>>>
>> >>>> > > > >>>>>>>>>> log)
>> >>>> > > > >>>
>> >>>> > > > >>>>
>> >>>> > > > >>>>>>>>>>>
>> >>>> > > > >>>>>>>>>>> http://pastebin.com/m496dbfae
>> >>>> > > > >>>>>>>>>>>
>> >>>> > > > >>>>>>>>>>>
>> >>>> > > > >>>>>>>>>>> I checked diskIO for one of the
>> >>>> > regionserver(192.168.100.116)
>> >>>> > > > >>>>>>>>>>>
>> >>>> > > > >>>>>>>>>> during
>> >>>> > > > >>>>>>>
>> >>>> > > > >>>>>>>> the
>> >>>> > > > >>>>>>>>>
>> >>>> > > > >>>>>>>>>> time it disconnected.
>> >>>> > > > >>>>>>>>>>>
>> >>>> > > > >>>>>>>>>>> Time: 03:04:58 AM
>> >>>> > > > >>>>>>>>>>> Device: tps Blk_read/s Blk_wrtn/s
>> >>>> Blk_read
>> >>>> > > > >>>>>>>>>>>
>> >>>> > > > >>>>>>>>>> Blk_wrtn
>> >>>> > > > >>>>>>>>
>> >>>> > > > >>>>>>>>> sda 105.31 5458.83 19503.64
>> >>>> 9043873239
>> >>>> > > > >>>>>>>>>>>
>> >>>> > > > >>>>>>>>>> 32312473676
>> >>>> > > > >>>>>>>>
>> >>>> > > > >>>>>>>>> sda1 2.90 6.64 99.25
>> >>>> 10993934
>> >>>> > > > >>>>>>>>>>>
>> >>>> > > > >>>>>>>>>> 164433464
>> >>>> > > > >>>>>>>>
>> >>>> > > > >>>>>>>>> sda2 1.72 23.76 30.25
>> >>>> 39365817
>> >>>> > > > >>>>>>>>>>>
>> >>>> > > > >>>>>>>>>> 50124008
>> >>>> > > > >>>>>>>>
>> >>>> > > > >>>>>>>>> sda3 0.30 0.38 3.58
>> >>>> 624930
>> >>>> > > > >>>>>>>>>>>
>> >>>> > > > >>>>>>>>>> 5923000
>> >>>> > > > >>>>>>>>
>> >>>> > > > >>>>>>>>> sda4 100.39 5428.06 19370.56
>> >>>> 8992888270
>> >>>> > > > >>>>>>>>>>>
>> >>>> > > > >>>>>>>>>> 32091993204
>> >>>> > > > >>>>>>>>
>> >>>> > > > >>>>>>>>>
>> >>>> > > > >>>>>>>>>>>
>> >>>> > > > >>>>>>>>>>> Is this high for you? 20k blocks/second would
>> seem to
>> >>>> be
>> >>>> > > high
>> >>>> > > > >>>>>>>>>>
>> >>>> > > > >>>>>>>>> but
>> >>>> > > > >>>
>> >>>> > > > >>>> its
>> >>>> > > > >>>>>>>
>> >>>> > > > >>>>>>>> one
>> >>>> > > > >>>>>>>>>
>> >>>> > > > >>>>>>>>>> disk only and its not being shared by zk anymore so
>> >>>> > shouldn't
>> >>>> > > > >>>>>>>>>>
>> >>>> > > > >>>>>>>>> matter?
>> >>>> > > > >>>>>>
>> >>>> > > > >>>>>>>
>> >>>> > > > >>>>>>>>>>
>> >>>> > > > >>>>>>>>>> I also checked the zookeeper quorum server that the
>> >>>> > > > >>>>>>>>>>>
>> >>>> > > > >>>>>>>>>> regionserver
>> >>>> > > > >>>
>> >>>> > > > >>>> tried
>> >>>> > > > >>>>>>>>
>> >>>> > > > >>>>>>>>> to
>> >>>> > > > >>>>>>>>>
>> >>>> > > > >>>>>>>>>> connect according to the log. However, I don't see
>> >>>> > > > >>>>>>>>>>>
>> >>>> > > > >>>>>>>>>> 192.168.100.116
>> >>>> > > > >>>>>>
>> >>>> > > > >>>>>>> in
>> >>>> > > > >>>>>>>
>> >>>> > > > >>>>>>>> the
>> >>>> > > > >>>>>>>>>
>> >>>> > > > >>>>>>>>>> client list of the zookeeper quorum server that the
>> >>>> > > > >>>>>>>>>>>
>> >>>> > > > >>>>>>>>>> regionserver
>> >>>> > > > >>>
>> >>>> > > > >>>> tried
>> >>>> > > > >>>>>>>>
>> >>>> > > > >>>>>>>>> to
>> >>>> > > > >>>>>>>>>
>> >>>> > > > >>>>>>>>>> connect.
>> >>>> > > > >>>>>>>>>>> Would that be a problem?
>> >>>> > > > >>>>>>>>>>>
>> >>>> > > > >>>>>>>>>>>
>> >>>> > > > >>>>>>>>>>> Is that because the ephemeral node for the
>> >>>> regionserver
>> >>>> > had
>> >>>> > > > >>>>>>>>>>
>> >>>> > > > >>>>>>>>> evaporated?
>> >>>> > > > >>>>>>>
>> >>>> > > > >>>>>>>> Lost
>> >>>> > > > >>>>>>>>>> its lease w/ zk by the time you went to look?
>> >>>> > > > >>>>>>>>>>
>> >>>> > > > >>>>>>>>>>
>> >>>> > > > >>>>>>>>>>
>> >>>> > > > >>>>>>>>>>
>> >>>> > > > >>>>>>>>>> Thu Nov 12 15:04:52 UTC 2009
>> >>>> > > > >>>>>>>>>>> Zookeeper version: 3.2.1-808558, built on
>> 08/27/2009
>> >>>> 18:48
>> >>>> > > > >>>>>>>>>>>
>> >>>> > > > >>>>>>>>>> GMT
>> >>>> > > > >>>
>> >>>> > > > >>>> Clients:
>> >>>> > > > >>>>>>>>>>> /192.168.100.127:43045
>> [1](queued=0,recved=26,sent=0)
>> >>>> > > > >>>>>>>>>>> /192.168.100.131:39091
>> [1](queued=0,recved=964,sent=0)
>> >>>> > > > >>>>>>>>>>> /192.168.100.124:35961
>> [1](queued=0,recved=3266,sent=0)
>> >>>> > > > >>>>>>>>>>> /192.168.100.123:47935
>> [1](queued=0,recved=233,sent=0)
>> >>>> > > > >>>>>>>>>>> /192.168.100.125:46931
>> [1](queued=0,recved=2,sent=0)
>> >>>> > > > >>>>>>>>>>> /192.168.100.118:54924
>> [1](queued=0,recved=295,sent=0)
>> >>>> > > > >>>>>>>>>>> /192.168.100.118:41390
>> [1](queued=0,recved=2290,sent=0)
>> >>>> > > > >>>>>>>>>>> /192.168.100.136:42243
>> [1](queued=0,recved=0,sent=0)
>> >>>> > > > >>>>>>>>>>>
>> >>>> > > > >>>>>>>>>>> Latency min/avg/max: 0/17/6333
>> >>>> > > > >>>>>>>>>>> Received: 47111
>> >>>> > > > >>>>>>>>>>> Sent: 0
>> >>>> > > > >>>>>>>>>>> Outstanding: 0
>> >>>> > > > >>>>>>>>>>> Zxid: 0x77000083f4
>> >>>> > > > >>>>>>>>>>> Mode: leader
>> >>>> > > > >>>>>>>>>>> Node count: 23
>> >>>> > > > >>>>>>>>>>>
>> >>>> > > > >>>>>>>>>>>
>> >>>> > > > >>>>>>>>>>
>> >>>> > > > >>>>>>>>>> That 6 second maximum latency is pretty bad but
>> should
>> >>>> be
>> >>>> > well
>> >>>> > > > >>>>>>>>>>
>> >>>> > > > >>>>>>>>> within
>> >>>> > > > >>>>>>
>> >>>> > > > >>>>>>> the
>> >>>> > > > >>>>>>>>
>> >>>> > > > >>>>>>>>> zk
>> >>>> > > > >>>>>>>>>> session timeout.
>> >>>> > > > >>>>>>>>>>
>> >>>> > > > >>>>>>>>>> So, problem is likely on the zk client side of the
>> >>>> session;
>> >>>> > > > >>>>>>>>>>
>> >>>> > > > >>>>>>>>> i.e.
>> >>>> > > > >>>
>> >>>> > > > >>>> in
>> >>>> > > > >>>>>>
>> >>>> > > > >>>>>>> the
>> >>>> > > > >>>>>>>
>> >>>> > > > >>>>>>>> RS.
>> >>>> > > > >>>>>>>>>
>> >>>> > > > >>>>>>>>>> You could enable GC logging as J-D suggested to see
>> if
>> >>>> you
>> >>>> > > have
>> >>>> > > > >>>>>>>>>>
>> >>>> > > > >>>>>>>>> any
>> >>>> > > > >>>>>>
>> >>>> > > > >>>>>>> big
>> >>>> > > > >>>>>>>
>> >>>> > > > >>>>>>>> pauses, pauses > zk session timeout.
>> >>>> > > > >>>>>>>>>>
>> >>>> > > > >>>>>>>>>> When the RS went down, it didn't look too heavily
>> >>>> loaded:
>> >>>> > > > >>>>>>>>>>
>> >>>> > > > >>>>>>>>>>
>> >>>> > > > >>>>>>>>>> 1. 2009-11-12 15:04:52,830 INFO
>> >>>> > > > >>>>>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>> >>>> Dump of
>> >>>> > > > >>>>>>>>>>
>> >>>> > > > >>>>>>>>> metrics:
>> >>>> > > > >>>>>>>
>> >>>> > > > >>>>>>>> request=1.5166667, regions=322, stores=657,
>> >>>> storefiles=631,
>> >>>> > > > >>>>>>>>>> storefileIndexSize=61, memstoreSize=1472,
>> >>>> usedHeap=2819,
>> >>>> > > > >>>>>>>>>>
>> >>>> > > > >>>>>>>>> maxHeap=4079,
>> >>>> > > > >>>>>>>>
>> >>>> > > > >>>>>>>>> blockCacheSize=658110960, blockCacheFree=197395984,
>> >>>> > > > >>>>>>>>>>
>> >>>> > > > >>>>>>>>> blockCacheCount=9903,
>> >>>> > > > >>>>>>>>>
>> >>>> > > > >>>>>>>>>> blockCacheHitRatio=99
>> >>>> > > > >>>>>>>>>>
>> >>>> > > > >>>>>>>>>> Its serving a few reads? The number of store files
>> >>>> seems
>> >>>> > > fine.
>> >>>> > > > >>>>>>>>>>
>> >>>> > > > >>>>>>>>> Not
>> >>>> > > > >>>>>>
>> >>>> > > > >>>>>>> too
>> >>>> > > > >>>>>>>>
>> >>>> > > > >>>>>>>>> much memory used.
>> >>>> > > > >>>>>>>>>>
>> >>>> > > > >>>>>>>>>> Looking at the logs, I see the Lease Still Held
>> >>>> exception.
>> >>>> > > > >>>>>>>>>>
>> >>>> > > > >>>>>>>>> This
>> >>>> > > > >>>
>> >>>> > > > >>>> happens
>> >>>> > > > >>>>>>>>
>> >>>> > > > >>>>>>>>> when the RS does its regular report to the master but
>> the
>> >>>> > > > >>>>>>>>>>
>> >>>> > > > >>>>>>>>> master
>> >>>> > > > >>>
>> >>>> > > > >>>> thinks
>> >>>> > > > >>>>>>>
>> >>>> > > > >>>>>>>> the
>> >>>> > > > >>>>>>>>>
>> >>>> > > > >>>>>>>>>> RS has since restarted. It'll think this probably
>> >>>> because
>> >>>> > it
>> >>>> > > > >>>>>>>>>>
>> >>>> > > > >>>>>>>>> noticed
>> >>>> > > > >>>>>>
>> >>>> > > > >>>>>>> that
>> >>>> > > > >>>>>>>>>
>> >>>> > > > >>>>>>>>>> the RS's znode in zk had gone away and it considered
>> the
>> >>>> RS
>> >>>> > > > >>>>>>>>>>
>> >>>> > > > >>>>>>>>> dead.
>> >>>> > > > >>>
>> >>>> > > > >>>>
>> >>>> > > > >>>>>>>>>> Looking too at your logs I see this gap in the zk
>> >>>> pinging:
>> >>>> > > > >>>>>>>>>>
>> >>>> > > > >>>>>>>>>>
>> >>>> > > > >>>>>>>>>> 1. 2009-11-12 15:03:39,325 DEBUG
>> >>>> > > > >>>>>>>>>>
>> >>>> > > > >>>>>>>>> org.apache.zookeeper.ClientCnxn:
>> >>>> > > > >>>>>>
>> >>>> > > > >>>>>>> Got
>> >>>> > > > >>>>>>>
>> >>>> > > > >>>>>>>> ping response for sessionid:0x224e55436ad0004 after
>> 0ms
>> >>>> > > > >>>>>>>>>> 2. 2009-11-12 15:03:43,113 DEBUG
>> >>>> > > > >>>>>>>>>>
>> >>>> > > > >>>>>>>>> org.apache.zookeeper.ClientCnxn:
>> >>>> > > > >>>>>>
>> >>>> > > > >>>>>>> Got
>> >>>> > > > >>>>>>>
>> >>>> > > > >>>>>>>> ping response for sessionid:0x24e55436a0007d after
>> 0ms
>> >>>> > > > >>>>>>>>>>
>> >>>> > > > >>>>>>>>>> Where in the lines above it, its reporting about
>> every
>> >>>> ten
>> >>>> > > > >>>>>>>>>>
>> >>>> > > > >>>>>>>>> seconds,
>> >>>> > > > >>>>>>
>> >>>> > > > >>>>>>> here
>> >>>> > > > >>>>>>>>
>> >>>> > > > >>>>>>>>> there is a big gap.
>> >>>> > > > >>>>>>>>>>
>> >>>> > > > >>>>>>>>>> Do you have ganglia or something that will let you
>> look
>> >>>> more
>> >>>> > > > >>>>>>>>>>
>> >>>> > > > >>>>>>>>> into
>> >>>> > > > >>>
>> >>>> > > > >>>> what
>> >>>> > > > >>>>>>>
>> >>>> > > > >>>>>>>> was
>> >>>> > > > >>>>>>>>>
>> >>>> > > > >>>>>>>>>> happening on this machine around this time? Is the
>> >>>> machine
>> >>>> > > OK?
>> >>>> > > > >>>>>>>>>>
>> >>>> > > > >>>>>>>>> It
>> >>>> > > > >>>>>>
>> >>>> > > > >>>>>>> looks
>> >>>> > > > >>>>>>>>
>> >>>> > > > >>>>>>>>> lightly loaded and you have your cluster nicely laid
>> out.
>> >>>> > > > >>>>>>>>>>
>> >>>> > > > >>>>>>>>> Something
>> >>>> > > > >>>>>>
>> >>>> > > > >>>>>>> odd
>> >>>> > > > >>>>>>>>
>> >>>> > > > >>>>>>>>> is
>> >>>> > > > >>>>>>>>>
>> >>>> > > > >>>>>>>>>> going on. What about things like the write speed to
>> >>>> disk?
>> >>>> > In
>> >>>> > > > >>>>>>>>>>
>> >>>> > > > >>>>>>>>> the
>> >>>> > > > >>>
>> >>>> > > > >>>> past
>> >>>> > > > >>>>>>>
>> >>>> > > > >>>>>>>> strange issues have been explained by incorrectly set
>> BIOS
>> >>>> > > > >>>>>>>>>>
>> >>>> > > > >>>>>>>>> which
>> >>>> > > > >>>
>> >>>> > > > >>>> made
>> >>>> > > > >>>>>>
>> >>>> > > > >>>>>>> disks
>> >>>> > > > >>>>>>>>>
>> >>>> > > > >>>>>>>>>> run at 1/100th of their proper speed.
>> >>>> > > > >>>>>>>>>>
>> >>>> > > > >>>>>>>>>> St.Ack
>> >>>> > > > >>>>>>>>>>
>> >>>> > > > >>>>>>>>>>
>> >>>> > > > >>>>>>>>>>
>> >>>> > > > >>>>>>>>>>
>> >>>> > > > >>>>>>>>>>
>> >>>> > > > >>>>>>>>>>
>> >>>> > > > >>>>>>>>>>
>> >>>> > > > >>>>>>>>>>
>> >>>> > > > >>>>>>>>>>> Best,
>> >>>> > > > >>>>>>>>>>> zhenyu
>> >>>> > > > >>>>>>>>>>>
>> >>>> > > > >>>>>>>>>>>
>> >>>> > > > >>>>>>>>>>> On Wed, Nov 11, 2009 at 3:58 PM, Zhenyu Zhong <
>> >>>> > > > >>>>>>>>>>>
>> >>>> > > > >>>>>>>>>> zhongresearch@gmail.com
>> >>>> > > > >>>>>>>>
>> >>>> > > > >>>>>>>>> wrote:
>> >>>> > > > >>>>>>>>>>>>
>> >>>> > > > >>>>>>>>>>>
>> >>>> > > > >>>>>>>>>>> Stack
>> >>>> > > > >>>>>>>>>>>>
>> >>>> > > > >>>>>>>>>>>> I am very appreciated for your comments.
>> >>>> > > > >>>>>>>>>>>> I will use the zookeeper monitoring script on my
>> >>>> cluster
>> >>>> > > > >>>>>>>>>>>>
>> >>>> > > > >>>>>>>>>>> and
>> >>>> > > > >>>
>> >>>> > > > >>>> let
>> >>>> > > > >>>>>>
>> >>>> > > > >>>>>>> it
>> >>>> > > > >>>>>>>
>> >>>> > > > >>>>>>>> run
>> >>>> > > > >>>>>>>>>
>> >>>> > > > >>>>>>>>>> overnight to see the result.
>> >>>> > > > >>>>>>>>>>>> I will follow up the thread when I get anything.
>> >>>> > > > >>>>>>>>>>>>
>> >>>> > > > >>>>>>>>>>>> thanks
>> >>>> > > > >>>>>>>>>>>> zhenyu
>> >>>> > > > >>>>>>>>>>>>
>> >>>> > > > >>>>>>>>>>>>
>> >>>> > > > >>>>>>>>>>>> On Wed, Nov 11, 2009 at 3:52 PM, stack <
>> >>>> stack@duboce.net>
>> >>>> > > > >>>>>>>>>>>>
>> >>>> > > > >>>>>>>>>>> wrote:
>> >>>> > > > >>>>>>
>> >>>> > > > >>>>>>>
>> >>>> > > > >>>>>>>>>>>> I see these in your log too:
>> >>>> > > > >>>>>>>>>>>>>
>> >>>> > > > >>>>>>>>>>>>>
>> >>>> > > > >>>>>>>>>>>>> 1. 2009-11-11 04:27:19,018 DEBUG
>> >>>> > > > >>>>>>>>>>>>>
>> >>>> > > > >>>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
>> >>>> > > > >>>>>>>>
>> >>>> > > > >>>>>>>>> Got
>> >>>> > > > >>>>>>>>>>
>> >>>> > > > >>>>>>>>>>> ping response for sessionid:0x424dfd908c50009
>> after
>> >>>> > > > >>>>>>>>>>>>>
>> >>>> > > > >>>>>>>>>>>> 4544ms
>> >>>> > > > >>>
>> >>>> > > > >>>> 2. 2009-11-11 04:27:19,018 DEBUG
>> >>>> > > > >>>>>>>>>>>>>
>> >>>> > > > >>>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
>> >>>> > > > >>>>>>>>
>> >>>> > > > >>>>>>>>> Got
>> >>>> > > > >>>>>>>>>>
>> >>>> > > > >>>>>>>>>>> ping response for sessionid:0x24dfd90c810002 after
>> >>>> > > > >>>>>>>>>>>>>
>> >>>> > > > >>>>>>>>>>>> 4548ms
>> >>>> > > > >>>
>> >>>> > > > >>>> 3. 2009-11-11 04:27:43,960 DEBUG
>> >>>> > > > >>>>>>>>>>>>>
>> >>>> > > > >>>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
>> >>>> > > > >>>>>>>>
>> >>>> > > > >>>>>>>>> Got
>> >>>> > > > >>>>>>>>>>
>> >>>> > > > >>>>>>>>>>> ping response for sessionid:0x424dfd908c50009
>> after
>> >>>> > > > >>>>>>>>>>>>>
>> >>>> > > > >>>>>>>>>>>> 9030ms
>> >>>> > > > >>>
>> >>>> > > > >>>> 4. 2009-11-11 04:27:43,960 DEBUG
>> >>>> > > > >>>>>>>>>>>>>
>> >>>> > > > >>>>>>>>>>>> org.apache.zookeeper.
>> >>>> > > > >>>>>>>>
>> >>>> > > > >>>>>>>
>> >>>> > > >
>> >>>> > >
>> >>>> >
>> >>>>
>> >>>
>> >>>
>> >>
>> >
>>
>
>
Re: regionserver disconnection
Posted by Zhenyu Zhong <zh...@gmail.com>.
J-D,
Thank you very much for your comments.
My company block the IRC port, so I have trouble to connect to IRC channel.
I have been trying to ask the IT to open the IRC port for me, it might take
a while.
Best,
zhenyu
On Sat, Nov 14, 2009 at 2:21 PM, Jean-Daniel Cryans <jd...@apache.org>wrote:
> The error you are getting is a disconnection from a zookeeper server
> and is very generic.
>
> ZK-86 is still opened and the last comment refers to ZK-111 saying
> that the bug (in unit tests) was probably fixed in release 3.0.0 last
> year.
>
> To figure the hang you have, you can try to jstack the process pid and
> see exactly what's holding the RS from shutting down.
>
> Would it be possible for you to drop by the IRC channel? This way we
> can debug this a at much faster pace.
>
> Thx!
>
> J-D
>
> On Sat, Nov 14, 2009 at 9:57 AM, Zhenyu Zhong <zh...@gmail.com>
> wrote:
> > I found this.
> > http://issues.apache.org/jira/browse/ZOOKEEPER-86
> >
> > It looks like the same error I had. Is it a zookeeper bug? When will
> HBase
> > take the zookeeper version 3.3.0?
> >
> > thanks
> > zhenyu
> >
> > On Sat, Nov 14, 2009 at 11:31 AM, Zhenyu Zhong <zhongresearch@gmail.com
> >wrote:
> >
> >> Now I really doubt about the zookeeper, from the log I saw errors like
> >> IOException Read Error, while the zookeeper client (regionserver ) tried
> to
> >> read. But it got disconneted status from zookeeper.
> >>
> >> I don't see any load on any zookeeper quorum servers. DiskIO is fine.
> >>
> >> Also when the regionserver decides to exit due to the disconnect status
> >> from zookeeper, sometimes the regionserver hangs during the exiting. We
> can
> >> still see the HRegionServer process even we don't see it in the Master
> web
> >> interface.
> >>
> >> I also notice that there is an zookeeper.retries and zookeeper.pauses
> >> settings in hbase-default.xml, would that matter ?
> >>
> >> thanks
> >>
> >> You may see the IOException error in the following log when the
> >> Regionserver lost connection to the zookeeper.
> >>
> >> 2009-11-14 15:58:31,993 INFO org.apache.hadoop.hbase.regionserver.HLog:
> >> Roll
> >>
> /hbase/.logs/superpyxis0008.scur.colo,60021,1258179101617/hlog.dat.1258179106921,
> >> entries=1062964, calcsize=255014936, filesize=139566093. New hlog
> >>
> /hbase/.logs/superpyxis0008.scur.colo,60021,1258179101617/hlog.dat.1258214311993
> >> 2009-11-14 15:58:39,295 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
> >> 2009-11-14 15:58:52,648 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
> >> 2009-11-14 15:59:06,007 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
> >> 2009-11-14 15:59:19,365 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
> >> 2009-11-14 15:59:32,867 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
> >> response for sessionid:0x424f0ed01ea00a6 after 186ms
> >> 2009-11-14 15:59:46,070 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
> >> 2009-11-14 15:59:59,423 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
> >> 2009-11-14 16:00:12,775 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
> >> response for sessionid:0x424f0ed01ea00a6 after 0ms
> >> 2009-11-14 16:00:28,141 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
> >> response for sessionid:0x424f0ed01ea00a6 after 2010ms
> >> 2009-11-14 16:01:10,378 WARN org.apache.zookeeper.ClientCnxn: Exception
> >> closing session 0x424f0ed01ea00a6 to
> sun.nio.ch.SelectionKeyImpl@39fbb2d6
> >>
> >> java.io.IOException: Read error rc = -1 java.nio.DirectByteBuffer[pos=0
> >> lim=4 cap=4]
> >> at
> >> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
> >> at
> >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
> >> 2009-11-14 16:01:10,478 INFO
> >> org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
> >> state: Disconnected, type: None, path: null
> >> 2009-11-14 16:01:11,333 WARN
> >> org.apache.hadoop.hbase.regionserver.HRegionServer: Attempt=1
> >>
> >> org.apache.hadoop.hbase.Leases$LeaseStillHeldException
> >> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> >> Method)
> >> at
> >>
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> >> at
> >>
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> >> at
> java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> >> at
> >>
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
> >> at
> >>
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
> >> at
> >>
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
> >> at
> >>
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:571)
> >> at java.lang.Thread.run(Thread.java:619)
> >> 2009-11-14 16:01:11,433 INFO org.apache.zookeeper.ClientCnxn: Attempting
> >> connection to server superpyxis0001/192.168.100.115:2181
> >> 2009-11-14 16:01:11,433 INFO org.apache.zookeeper.ClientCnxn: Priming
> >> connection to java.nio.channels.SocketChannel[connected local=/
> >> 192.168.100.122:59575 remote=superpyxis0001/192.168.100.115:2181]
> >> 2009-11-14 16:01:11,433 INFO org.apache.zookeeper.ClientCnxn: Server
> >> connection successful
> >> 2009-11-14 16:01:11,433 INFO
> >> org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
> >> state: Expired, type: None, path: null
> >> 2009-11-14 16:01:11,433 WARN org.apache.zookeeper.ClientCnxn: Exception
> >> closing session 0x424f0ed01ea00a6 to
> sun.nio.ch.SelectionKeyImpl@6b93d343
> >> java.io.IOException: Session Expired
> >> at
> >>
> org.apache.zookeeper.ClientCnxn$SendThread.readConnectResult(ClientCnxn.java:589)
> >> at
> >> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:709)
> >>
> >> at
> >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
> >> 2009-11-14 16:01:11,433 ERROR
> >> org.apache.hadoop.hbase.regionserver.HRegionServer: ZooKeeper session
> >> expired
> >> 2009-11-14 16:01:13,513 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> Server
> >> handler 3 on 60021, call put([B@73deb204,
> >> [Lorg.apache.hadoop.hbase.client.Put;@2179600a) from
> 192.168.100.132:40728:
> >> error: java.io.IOException: Server not running, aborting
> >> java.io.IOException: Server not running, aborting
> >>
> >>
> >>
> >>
> >>
> >> On Sat, Nov 14, 2009 at 1:01 AM, Zhenyu Zhong <zhongresearch@gmail.com
> >wrote:
> >>
> >>> Stack,
> >>>
> >>> Thanks a lot!
> >>>
> >>> I found out that the reason why HBase doesn't take the system file
> >>> descriptor value I set before. I started the HBase using root instead
> of the
> >>> normal hadoop user, while my system configures hadoop with higher file
> >>> descriptor value but configures root with 1024 default value.
> >>>
> >>>
> >>> Now my system has a clean start. However, it seems that the
> >>> zookeeper.session.timeout value doesn't take into effect. I still found
> >>> around 60 seconds pauses from the disconnected regionserver. I really
> don't
> >>> know why regionserver only times out after 60 seconds instead of 10
> minutes
> >>> which I set for zookeeper.session.timeout.
> >>>
> >>> Is there any other timeout value coming into play before the actual
> >>> session times out?
> >>>
> >>> zhenyu
> >>>
> >>>
> >>>
> >>> On Fri, Nov 13, 2009 at 7:08 PM, stack <st...@duboce.net> wrote:
> >>>
> >>>> Ok. Lack of file descriptors manifests in all kinds of weird ways.
> >>>> Hopefully thats it. If not, lets keep digging.
> >>>> St.Ack
> >>>>
> >>>> On Fri, Nov 13, 2009 at 3:44 PM, Zhenyu Zhong <
> zhongresearch@gmail.com
> >>>> >wrote:
> >>>>
> >>>> > Stack,
> >>>> >
> >>>> > You are right, the master started with ulimit -n 1024. It doesn't
> take
> >>>> the
> >>>> > system value.
> >>>> >
> >>>> > Regarding to the too many open files, it looks like the same as the
> one
> >>>> J-D
> >>>> > put up. But I will get the Master start with higher value first and
> see
> >>>> if
> >>>> > this kind of error goes away.
> >>>> >
> >>>> > thanks a lot!
> >>>> > zhenyu
> >>>> >
> >>>> > On Fri, Nov 13, 2009 at 6:02 PM, stack <st...@duboce.net> wrote:
> >>>> >
> >>>> > > Does it say
> >>>> > >
> >>>> > > ulimit -n 32768
> >>>> > >
> >>>> > > ...as the second line in your log file on start of the master?
> >>>> > >
> >>>> > > Is the stack trace that complains about too many open files same
> as
> >>>> the
> >>>> > one
> >>>> > > in the blog post J-D put up?
> >>>> > >
> >>>> > > St.Ack
> >>>> > >
> >>>> > >
> >>>> > > On Fri, Nov 13, 2009 at 1:37 PM, Zhenyu Zhong <
> >>>> zhongresearch@gmail.com
> >>>> > > >wrote:
> >>>> > >
> >>>> > > > The ulimit file descriptors was set to fs.file-max =1578334,
> also
> >>>> in
> >>>> > the
> >>>> > > > limits.conf the value was set to 32768.
> >>>> > > > So these are way higher than the open descriptors for the
> running
> >>>> > > > processes.
> >>>> > > >
> >>>> > > > thanks
> >>>> > > > zhenyu
> >>>> > > >
> >>>> > > >
> >>>> > > > On Fri, Nov 13, 2009 at 4:33 PM, Stack <sa...@gmail.com>
> >>>> wrote:
> >>>> > > >
> >>>> > > > > You upped the ulimit file descriptors as per the getting
> started
> >>>> doc?
> >>>> > > > >
> >>>> > > > >
> >>>> > > > >
> >>>> > > > > On Nov 13, 2009, at 1:26 PM, Zhenyu Zhong <
> >>>> zhongresearch@gmail.com>
> >>>> > > > wrote:
> >>>> > > > >
> >>>> > > > > Thanks a lot.
> >>>> > > > >>
> >>>> > > > >>
> >>>> > > > >> Bad news is my kernel is still 2.6.26.
> >>>> > > > >> But it was not a problem before.
> >>>> > > > >>
> >>>> > > > >> Very strange.
> >>>> > > > >>
> >>>> > > > >> zhenyu
> >>>> > > > >>
> >>>> > > > >> On Fri, Nov 13, 2009 at 4:16 PM, Jean-Daniel Cryans <
> >>>> > > > jdcryans@apache.org
> >>>> > > > >> >wrote:
> >>>> > > > >>
> >>>> > > > >> Looks like
> >>>> > > > >>>
> >>>> > > > >>>
> >>>> > > >
> >>>> > >
> >>>> >
> >>>>
> http://pero.blogs.aprilmayjune.org/2009/01/22/hadoop-and-linux-kernel-2627-epoll-limits/
> >>>> > > > >>>
> >>>> > > > >>> J-D
> >>>> > > > >>>
> >>>> > > > >>> On Fri, Nov 13, 2009 at 1:12 PM, Zhenyu Zhong <
> >>>> > > zhongresearch@gmail.com
> >>>> > > > >
> >>>> > > > >>> wrote:
> >>>> > > > >>>
> >>>> > > > >>>> Hi,
> >>>> > > > >>>>
> >>>> > > > >>>>
> >>>> > > > >>>> After I re-organize the cluster, the experiment ran into
> >>>> problem
> >>>> > > > faster
> >>>> > > > >>>>
> >>>> > > > >>> than
> >>>> > > > >>>
> >>>> > > > >>>> before.
> >>>> > > > >>>>
> >>>> > > > >>>> Basically the changes are to use machines with less
> resources
> >>>> as
> >>>> > > > >>>>
> >>>> > > > >>> zookeeper
> >>>> > > > >>>
> >>>> > > > >>>> quorums and machines with more resources as regionserver.
> >>>> > > > >>>>
> >>>> > > > >>>> From the log, I still see the pause around 1 minute.
> >>>> > > > >>>> I enabled the GC logging, please see
> >>>> > > > >>>>
> >>>> > > > >>>> http://pastebin.com/m1d4ce0f1
> >>>> > > > >>>>
> >>>> > > > >>>> for details.
> >>>> > > > >>>> In general I don't see many pauses in the GC.
> >>>> > > > >>>>
> >>>> > > > >>>> What is more interesting, 1 hour after the 1st regionserver
> >>>> > > > >>>> disconnected,
> >>>> > > > >>>> the master log complained about too many open files. This
> >>>> didn't
> >>>> > > > happen
> >>>> > > > >>>> before.
> >>>> > > > >>>> I checked the system OS setup as well as the limits.conf. I
> >>>> also
> >>>> > > > checked
> >>>> > > > >>>>
> >>>> > > > >>> the
> >>>> > > > >>>
> >>>> > > > >>>> running processes. The total open files don't reach the
> limit.
> >>>> I
> >>>> > am
> >>>> > > > >>>>
> >>>> > > > >>> confused
> >>>> > > > >>>
> >>>> > > > >>>> a bit.
> >>>> > > > >>>>
> >>>> > > > >>>>
> >>>> > > > >>>> Please see the following master log.
> >>>> > > > >>>>
> >>>> > > > >>>> 2009-11-13 20:06:02,114 INFO
> >>>> > > > org.apache.hadoop.hbase.master.BaseScanner:
> >>>> > > > >>>> RegionManager.metaScanner scan of 4658 row(s) of meta
> region
> >>>> > > {server:
> >>>> > > > >>>> 192.168.100.128:60021, regionname: .META.,,1, startKey:
> <>}
> >>>> > > complete
> >>>> > > > >>>> 2009-11-13 20:06:02,114 INFO
> >>>> > > > org.apache.hadoop.hbase.master.BaseScanner:
> >>>> > > > >>>>
> >>>> > > > >>> All
> >>>> > > > >>>
> >>>> > > > >>>> 1 .META. region(s) scanned
> >>>> > > > >>>> 2009-11-13 20:06:07,677 DEBUG
> org.apache.zookeeper.ClientCnxn:
> >>>> Got
> >>>> > > > ping
> >>>> > > > >>>> response for sessionid:0x424eebf1c10004c after 3ms
> >>>> > > > >>>> 2009-11-13 20:06:08,178 INFO
> org.apache.hadoop.hdfs.DFSClient:
> >>>> > > > Exception
> >>>> > > > >>>>
> >>>> > > > >>> in
> >>>> > > > >>>
> >>>> > > > >>>> createBlockOutputStream java.io.IOException: Bad connect
> ack
> >>>> with
> >>>> > > > >>>> firstBadLink 192.168.100.123:50010
> >>>> > > > >>>> 2009-11-13 20:06:08,178 INFO
> org.apache.hadoop.hdfs.DFSClient:
> >>>> > > > >>>> Abandoning
> >>>> > > > >>>> block blk_-2808245019291145247_5478039
> >>>> > > > >>>> 2009-11-13 20:06:09,682 INFO
> org.apache.hadoop.hdfs.DFSClient:
> >>>> > > > Exception
> >>>> > > > >>>>
> >>>> > > > >>> in
> >>>> > > > >>>
> >>>> > > > >>>> createBlockOutputStream java.io.EOFException
> >>>> > > > >>>> 2009-11-13 20:06:09,682 INFO
> org.apache.hadoop.hdfs.DFSClient:
> >>>> > > > >>>> Abandoning
> >>>> > > > >>>> block blk_1074853606841896259_5478048
> >>>> > > > >>>> 2009-11-13 20:06:10,334 DEBUG
> org.apache.zookeeper.ClientCnxn:
> >>>> Got
> >>>> > > > ping
> >>>> > > > >>>> response for sessionid:0x24eebf1043003c after 1ms
> >>>> > > > >>>> 2009-11-13 20:06:21,018 DEBUG
> org.apache.zookeeper.ClientCnxn:
> >>>> Got
> >>>> > > > ping
> >>>> > > > >>>> response for sessionid:0x424eebf1c10004c after 0ms
> >>>> > > > >>>> 2009-11-13 20:06:23,674 DEBUG
> org.apache.zookeeper.ClientCnxn:
> >>>> Got
> >>>> > > > ping
> >>>> > > > >>>> response for sessionid:0x24eebf1043003c after 0ms
> >>>> > > > >>>> 2009-11-13 20:06:24,828 INFO
> org.apache.hadoop.hdfs.DFSClient:
> >>>> > > > Exception
> >>>> > > > >>>>
> >>>> > > > >>> in
> >>>> > > > >>>
> >>>> > > > >>>> createBlockOutputStream java.io.IOException: Bad connect
> ack
> >>>> with
> >>>> > > > >>>> firstBadLink 192.168.100.123:50010
> >>>> > > > >>>> 2009-11-13 20:06:24,828 INFO
> org.apache.hadoop.hdfs.DFSClient:
> >>>> > > > >>>> Abandoning
> >>>> > > > >>>> block blk_-6642544517082142289_5478063
> >>>> > > > >>>> 2009-11-13 20:06:24,828 INFO
> org.apache.hadoop.hdfs.DFSClient:
> >>>> > > > Exception
> >>>> > > > >>>>
> >>>> > > > >>> in
> >>>> > > > >>>
> >>>> > > > >>>> createBlockOutputStream java.net.SocketException: Too many
> >>>> open
> >>>> > > files
> >>>> > > > >>>> 2009-11-13 20:06:24,828 INFO
> org.apache.hadoop.hdfs.DFSClient:
> >>>> > > > >>>> Abandoning
> >>>> > > > >>>> block blk_2057511041109796090_5478063
> >>>> > > > >>>> 2009-11-13 20:06:24,928 INFO
> org.apache.hadoop.hdfs.DFSClient:
> >>>> > > > Exception
> >>>> > > > >>>>
> >>>> > > > >>> in
> >>>> > > > >>>
> >>>> > > > >>>> createBlockOutputStream java.net.SocketException: Too many
> >>>> open
> >>>> > > files
> >>>> > > > >>>> 2009-11-13 20:06:24,928 INFO
> org.apache.hadoop.hdfs.DFSClient:
> >>>> > > > >>>> Abandoning
> >>>> > > > >>>> block blk_8219260302213892894_5478064
> >>>> > > > >>>> 2009-11-13 20:06:30,855 INFO
> org.apache.hadoop.hdfs.DFSClient:
> >>>> > > > Exception
> >>>> > > > >>>>
> >>>> > > > >>> in
> >>>> > > > >>>
> >>>> > > > >>>> createBlockOutputStream java.net.SocketException: Too many
> >>>> open
> >>>> > > files
> >>>> > > > >>>> 2009-11-13 20:06:30,855 INFO
> org.apache.hadoop.hdfs.DFSClient:
> >>>> > > > >>>> Abandoning
> >>>> > > > >>>> block blk_1669205542853067709_5478235
> >>>> > > > >>>> 2009-11-13 20:06:30,905 INFO
> org.apache.hadoop.hdfs.DFSClient:
> >>>> > > > Exception
> >>>> > > > >>>>
> >>>> > > > >>> in
> >>>> > > > >>>
> >>>> > > > >>>> createBlockOutputStream java.net.SocketException: Too many
> >>>> open
> >>>> > > files
> >>>> > > > >>>> 2009-11-13 20:06:30,905 INFO
> org.apache.hadoop.hdfs.DFSClient:
> >>>> > > > >>>> Abandoning
> >>>> > > > >>>> block blk_9128897691346270351_5478237
> >>>> > > > >>>> 2009-11-13 20:06:30,955 INFO
> org.apache.hadoop.hdfs.DFSClient:
> >>>> > > > Exception
> >>>> > > > >>>>
> >>>> > > > >>> in
> >>>> > > > >>>
> >>>> > > > >>>> createBlockOutputStream java.net.SocketException: Too many
> >>>> open
> >>>> > > files
> >>>> > > > >>>> 2009-11-13 20:06:30,955 INFO
> org.apache.hadoop.hdfs.DFSClient:
> >>>> > > > >>>> Abandoning
> >>>> > > > >>>> block blk_1116845144864123018_5478240
> >>>> > > > >>>> 2009-11-13 20:06:34,372 DEBUG
> org.apache.zookeeper.ClientCnxn:
> >>>> Got
> >>>> > > > ping
> >>>> > > > >>>> response for sessionid:0x424eebf1c10004c after 0ms
> >>>> > > > >>>> 2009-11-13 20:06:37,034 DEBUG
> org.apache.zookeeper.ClientCnxn:
> >>>> Got
> >>>> > > > ping
> >>>> > > > >>>> response for sessionid:0x24eebf1043003c after 0ms
> >>>> > > > >>>> 2009-11-13 20:06:37,235 WARN
> org.apache.hadoop.hdfs.DFSClient:
> >>>> > > > >>>>
> >>>> > > > >>> DataStreamer
> >>>> > > > >>>
> >>>> > > > >>>> Exception: java.io.IOException: Too many open files
> >>>> > > > >>>> at sun.nio.ch.IOUtil.initPipe(Native Method)
> >>>> > > > >>>> at
> >>>> > > sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:49)
> >>>> > > > >>>> at
> >>>> > > > >>>>
> >>>> > > > >>>>
> >>>> > > >
> >>>> > >
> >>>> >
> >>>>
> sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:18)
> >>>> > > > >>>
> >>>> > > > >>>> at
> >>>> > > > >>>>
> >>>> > > > >>>>
> >>>> > > >
> >>>> > >
> >>>> >
> >>>>
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.get(SocketIOWithTimeout.java:407)
> >>>> > > > >>>
> >>>> > > > >>>> at
> >>>> > > > >>>>
> >>>> > > > >>>>
> >>>> > > >
> >>>> > >
> >>>> >
> >>>>
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:322)
> >>>> > > > >>>
> >>>> > > > >>>> at
> >>>> > > > >>>>
> >>>> > > > >>>>
> >>>> > > >
> >>>> > >
> >>>> >
> >>>>
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
> >>>> > > > >>>
> >>>> > > > >>>> at
> >>>> > > > >>>>
> >>>> > > > >>>>
> >>>> > > >
> >>>> > >
> >>>> >
> >>>>
> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
> >>>> > > > >>>
> >>>> > > > >>>> at
> >>>> > > > >>>>
> >>>> > > > >>>>
> >>>> > > >
> >>>> > >
> >>>> >
> >>>>
> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
> >>>> > > > >>>
> >>>> > > > >>>> at
> >>>> > > > >>>>
> >>>> > > > >>>
> >>>> java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
> >>>> > > > >>>
> >>>> > > > >>>> at
> >>>> java.io.DataOutputStream.write(DataOutputStream.java:90)
> >>>> > > > >>>> at
> >>>> > > > >>>>
> >>>> > > > >>>>
> >>>> > > >
> >>>> > >
> >>>> >
> >>>>
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2290)
> >>>> > > > >>>
> >>>> > > > >>>>
> >>>> > > > >>>> 2009-11-13 20:06:37,235 WARN
> org.apache.hadoop.hdfs.DFSClient:
> >>>> > Error
> >>>> > > > >>>> Recovery for block blk_8148813491785406356_5478475 bad
> >>>> datanode[0]
> >>>> > > > >>>> 192.168.100.123:50010
> >>>> > > > >>>> 2009-11-13 20:06:37,235 WARN
> org.apache.hadoop.hdfs.DFSClient:
> >>>> > Error
> >>>> > > > >>>> Recovery for block blk_8148813491785406356_5478475 in
> pipeline
> >>>> > > > >>>> 192.168.100.123:50010, 192.168.100.134:50010,
> >>>> > 192.168.100.122:50010
> >>>> > > :
> >>>> > > > >>>> bad
> >>>> > > > >>>> datanode 192.168.100.123:50010
> >>>> > > > >>>> 2009-11-13 20:06:37,436 INFO
> org.apache.hadoop.hdfs.DFSClient:
> >>>> > > > Exception
> >>>> > > > >>>>
> >>>> > > > >>> in
> >>>> > > > >>>
> >>>> > > > >>>> createBlockOutputStream java.net.SocketException: Too many
> >>>> open
> >>>> > > files
> >>>> > > > >>>> 2009-11-13 20:06:37,436 INFO
> org.apache.hadoop.hdfs.DFSClient:
> >>>> > > > >>>> Abandoning
> >>>> > > > >>>> block blk_2119727700857186236_5478498
> >>>> > > > >>>>
> >>>> > > > >>>>
> >>>> > > > >>>>
> >>>> > > > >>>>
> >>>> > > > >>>>
> >>>> > > > >>>> On Thu, Nov 12, 2009 at 4:21 PM, Zhenyu Zhong <
> >>>> > > > zhongresearch@gmail.com
> >>>> > > > >>>> wrote:
> >>>> > > > >>>>
> >>>> > > > >>>> Will do.
> >>>> > > > >>>>>
> >>>> > > > >>>>> thanks
> >>>> > > > >>>>> zhenyu
> >>>> > > > >>>>>
> >>>> > > > >>>>>
> >>>> > > > >>>>> On Thu, Nov 12, 2009 at 3:33 PM, stack <st...@duboce.net>
> >>>> wrote:
> >>>> > > > >>>>>
> >>>> > > > >>>>> Enable GC logging too on this next run (see
> hbase-env.sh).
> >>>> Lets
> >>>> > > try
> >>>> > > > >>>>>>
> >>>> > > > >>>>> and
> >>>> > > > >>>
> >>>> > > > >>>> get
> >>>> > > > >>>>>> to the bottom of whats going on.
> >>>> > > > >>>>>> Thanks,
> >>>> > > > >>>>>> St.Ack
> >>>> > > > >>>>>>
> >>>> > > > >>>>>> On Thu, Nov 12, 2009 at 12:29 PM, Zhenyu Zhong <
> >>>> > > > >>>>>>
> >>>> > > > >>>>> zhongresearch@gmail.com
> >>>> > > > >>>
> >>>> > > > >>>> wrote:
> >>>> > > > >>>>>>>
> >>>> > > > >>>>>>
> >>>> > > > >>>>>> I can switch the boxes that run zookeeper with the ones
> >>>> that
> >>>> > run
> >>>> > > > >>>>>>> regionservers.
> >>>> > > > >>>>>>> I will see the results later.
> >>>> > > > >>>>>>>
> >>>> > > > >>>>>>> FYI. The node does have the 10 minutes
> >>>> > zookeeper.session.timeout
> >>>> > > > >>>>>>>
> >>>> > > > >>>>>> value
> >>>> > > > >>>
> >>>> > > > >>>> in
> >>>> > > > >>>>>>
> >>>> > > > >>>>>>> place.
> >>>> > > > >>>>>>>
> >>>> > > > >>>>>>> thanks
> >>>> > > > >>>>>>> zhenyu
> >>>> > > > >>>>>>>
> >>>> > > > >>>>>>> On Thu, Nov 12, 2009 at 3:21 PM, stack <
> stack@duboce.net>
> >>>> > wrote:
> >>>> > > > >>>>>>>
> >>>> > > > >>>>>>> On Thu, Nov 12, 2009 at 11:50 AM, Zhenyu Zhong <
> >>>> > > > >>>>>>>>
> >>>> > > > >>>>>>> zhongresearch@gmail.com
> >>>> > > > >>>>>>
> >>>> > > > >>>>>>> wrote:
> >>>> > > > >>>>>>>>>
> >>>> > > > >>>>>>>>
> >>>> > > > >>>>>>>>
> >>>> > > > >>>>>>>>> In my cluster, half of the cluster have 2 disks 400GB
> >>>> each
> >>>> > per
> >>>> > > > >>>>>>>>>
> >>>> > > > >>>>>>>> machine,
> >>>> > > > >>>>>>
> >>>> > > > >>>>>>> and
> >>>> > > > >>>>>>>>
> >>>> > > > >>>>>>>>> half of the cluster have 6 disks per machine. Maybe
> we
> >>>> > should
> >>>> > > > >>>>>>>>>
> >>>> > > > >>>>>>>> run
> >>>> > > > >>>
> >>>> > > > >>>> zookeeper
> >>>> > > > >>>>>>>>> on the machines with 2 disks and RS on machines with 6
> >>>> disks?
> >>>> > > > >>>>>>>>>
> >>>> > > > >>>>>>>>> That would make most sense only in the below, it looks
> >>>> like
> >>>> > the
> >>>> > > > >>>>>>>>>
> >>>> > > > >>>>>>>> RS
> >>>> > > > >>>
> >>>> > > > >>>> that
> >>>> > > > >>>>>>
> >>>> > > > >>>>>>> had
> >>>> > > > >>>>>>>> issue had 4 disks?
> >>>> > > > >>>>>>>>
> >>>> > > > >>>>>>>>
> >>>> > > > >>>>>>>>
> >>>> > > > >>>>>>>> BTW, the 10 minutes zookeeper.session.timeout has been
> >>>> set
> >>>> > > during
> >>>> > > > >>>>>>>>>
> >>>> > > > >>>>>>>> the
> >>>> > > > >>>>>>
> >>>> > > > >>>>>>> experiment.
> >>>> > > > >>>>>>>>>
> >>>> > > > >>>>>>>>>
> >>>> > > > >>>>>>>> And for sure this node had it in place?
> >>>> > > > >>>>>>>> St.Ack
> >>>> > > > >>>>>>>>
> >>>> > > > >>>>>>>>
> >>>> > > > >>>>>>>>
> >>>> > > > >>>>>>>>
> >>>> > > > >>>>>>>>> thanks
> >>>> > > > >>>>>>>>> zhenyu
> >>>> > > > >>>>>>>>>
> >>>> > > > >>>>>>>>> On Thu, Nov 12, 2009 at 2:08 PM, stack <
> stack@duboce.net
> >>>> >
> >>>> > > wrote:
> >>>> > > > >>>>>>>>>
> >>>> > > > >>>>>>>>> On Thu, Nov 12, 2009 at 8:40 AM, Zhenyu Zhong <
> >>>> > > > >>>>>>>>>>
> >>>> > > > >>>>>>>>> zhongresearch@gmail.com
> >>>> > > > >>>>>>>
> >>>> > > > >>>>>>>> wrote:
> >>>> > > > >>>>>>>>>>>
> >>>> > > > >>>>>>>>>>
> >>>> > > > >>>>>>>>>>
> >>>> > > > >>>>>>>>>> Though I experienced 2 regionserver disconnection
> this
> >>>> > > > >>>>>>>>>>>
> >>>> > > > >>>>>>>>>> morning,
> >>>> > > > >>>
> >>>> > > > >>>> it
> >>>> > > > >>>>>>
> >>>> > > > >>>>>>> looks
> >>>> > > > >>>>>>>>>
> >>>> > > > >>>>>>>>>> better from the regionserver log. (please see the
> >>>> following
> >>>> > > > >>>>>>>>>>>
> >>>> > > > >>>>>>>>>> log)
> >>>> > > > >>>
> >>>> > > > >>>>
> >>>> > > > >>>>>>>>>>>
> >>>> > > > >>>>>>>>>>> http://pastebin.com/m496dbfae
> >>>> > > > >>>>>>>>>>>
> >>>> > > > >>>>>>>>>>>
> >>>> > > > >>>>>>>>>>> I checked diskIO for one of the
> >>>> > regionserver(192.168.100.116)
> >>>> > > > >>>>>>>>>>>
> >>>> > > > >>>>>>>>>> during
> >>>> > > > >>>>>>>
> >>>> > > > >>>>>>>> the
> >>>> > > > >>>>>>>>>
> >>>> > > > >>>>>>>>>> time it disconnected.
> >>>> > > > >>>>>>>>>>>
> >>>> > > > >>>>>>>>>>> Time: 03:04:58 AM
> >>>> > > > >>>>>>>>>>> Device: tps Blk_read/s Blk_wrtn/s
> >>>> Blk_read
> >>>> > > > >>>>>>>>>>>
> >>>> > > > >>>>>>>>>> Blk_wrtn
> >>>> > > > >>>>>>>>
> >>>> > > > >>>>>>>>> sda 105.31 5458.83 19503.64
> >>>> 9043873239
> >>>> > > > >>>>>>>>>>>
> >>>> > > > >>>>>>>>>> 32312473676
> >>>> > > > >>>>>>>>
> >>>> > > > >>>>>>>>> sda1 2.90 6.64 99.25
> >>>> 10993934
> >>>> > > > >>>>>>>>>>>
> >>>> > > > >>>>>>>>>> 164433464
> >>>> > > > >>>>>>>>
> >>>> > > > >>>>>>>>> sda2 1.72 23.76 30.25
> >>>> 39365817
> >>>> > > > >>>>>>>>>>>
> >>>> > > > >>>>>>>>>> 50124008
> >>>> > > > >>>>>>>>
> >>>> > > > >>>>>>>>> sda3 0.30 0.38 3.58
> >>>> 624930
> >>>> > > > >>>>>>>>>>>
> >>>> > > > >>>>>>>>>> 5923000
> >>>> > > > >>>>>>>>
> >>>> > > > >>>>>>>>> sda4 100.39 5428.06 19370.56
> >>>> 8992888270
> >>>> > > > >>>>>>>>>>>
> >>>> > > > >>>>>>>>>> 32091993204
> >>>> > > > >>>>>>>>
> >>>> > > > >>>>>>>>>
> >>>> > > > >>>>>>>>>>>
> >>>> > > > >>>>>>>>>>> Is this high for you? 20k blocks/second would seem
> to
> >>>> be
> >>>> > > high
> >>>> > > > >>>>>>>>>>
> >>>> > > > >>>>>>>>> but
> >>>> > > > >>>
> >>>> > > > >>>> its
> >>>> > > > >>>>>>>
> >>>> > > > >>>>>>>> one
> >>>> > > > >>>>>>>>>
> >>>> > > > >>>>>>>>>> disk only and its not being shared by zk anymore so
> >>>> > shouldn't
> >>>> > > > >>>>>>>>>>
> >>>> > > > >>>>>>>>> matter?
> >>>> > > > >>>>>>
> >>>> > > > >>>>>>>
> >>>> > > > >>>>>>>>>>
> >>>> > > > >>>>>>>>>> I also checked the zookeeper quorum server that the
> >>>> > > > >>>>>>>>>>>
> >>>> > > > >>>>>>>>>> regionserver
> >>>> > > > >>>
> >>>> > > > >>>> tried
> >>>> > > > >>>>>>>>
> >>>> > > > >>>>>>>>> to
> >>>> > > > >>>>>>>>>
> >>>> > > > >>>>>>>>>> connect according to the log. However, I don't see
> >>>> > > > >>>>>>>>>>>
> >>>> > > > >>>>>>>>>> 192.168.100.116
> >>>> > > > >>>>>>
> >>>> > > > >>>>>>> in
> >>>> > > > >>>>>>>
> >>>> > > > >>>>>>>> the
> >>>> > > > >>>>>>>>>
> >>>> > > > >>>>>>>>>> client list of the zookeeper quorum server that the
> >>>> > > > >>>>>>>>>>>
> >>>> > > > >>>>>>>>>> regionserver
> >>>> > > > >>>
> >>>> > > > >>>> tried
> >>>> > > > >>>>>>>>
> >>>> > > > >>>>>>>>> to
> >>>> > > > >>>>>>>>>
> >>>> > > > >>>>>>>>>> connect.
> >>>> > > > >>>>>>>>>>> Would that be a problem?
> >>>> > > > >>>>>>>>>>>
> >>>> > > > >>>>>>>>>>>
> >>>> > > > >>>>>>>>>>> Is that because the ephemeral node for the
> >>>> regionserver
> >>>> > had
> >>>> > > > >>>>>>>>>>
> >>>> > > > >>>>>>>>> evaporated?
> >>>> > > > >>>>>>>
> >>>> > > > >>>>>>>> Lost
> >>>> > > > >>>>>>>>>> its lease w/ zk by the time you went to look?
> >>>> > > > >>>>>>>>>>
> >>>> > > > >>>>>>>>>>
> >>>> > > > >>>>>>>>>>
> >>>> > > > >>>>>>>>>>
> >>>> > > > >>>>>>>>>> Thu Nov 12 15:04:52 UTC 2009
> >>>> > > > >>>>>>>>>>> Zookeeper version: 3.2.1-808558, built on 08/27/2009
> >>>> 18:48
> >>>> > > > >>>>>>>>>>>
> >>>> > > > >>>>>>>>>> GMT
> >>>> > > > >>>
> >>>> > > > >>>> Clients:
> >>>> > > > >>>>>>>>>>> /192.168.100.127:43045
> [1](queued=0,recved=26,sent=0)
> >>>> > > > >>>>>>>>>>> /192.168.100.131:39091
> [1](queued=0,recved=964,sent=0)
> >>>> > > > >>>>>>>>>>> /192.168.100.124:35961
> [1](queued=0,recved=3266,sent=0)
> >>>> > > > >>>>>>>>>>> /192.168.100.123:47935
> [1](queued=0,recved=233,sent=0)
> >>>> > > > >>>>>>>>>>> /192.168.100.125:46931[1](queued=0,recved=2,sent=0)
> >>>> > > > >>>>>>>>>>> /192.168.100.118:54924
> [1](queued=0,recved=295,sent=0)
> >>>> > > > >>>>>>>>>>> /192.168.100.118:41390
> [1](queued=0,recved=2290,sent=0)
> >>>> > > > >>>>>>>>>>> /192.168.100.136:42243[1](queued=0,recved=0,sent=0)
> >>>> > > > >>>>>>>>>>>
> >>>> > > > >>>>>>>>>>> Latency min/avg/max: 0/17/6333
> >>>> > > > >>>>>>>>>>> Received: 47111
> >>>> > > > >>>>>>>>>>> Sent: 0
> >>>> > > > >>>>>>>>>>> Outstanding: 0
> >>>> > > > >>>>>>>>>>> Zxid: 0x77000083f4
> >>>> > > > >>>>>>>>>>> Mode: leader
> >>>> > > > >>>>>>>>>>> Node count: 23
> >>>> > > > >>>>>>>>>>>
> >>>> > > > >>>>>>>>>>>
> >>>> > > > >>>>>>>>>>
> >>>> > > > >>>>>>>>>> That 6 second maximum latency is pretty bad but
> should
> >>>> be
> >>>> > well
> >>>> > > > >>>>>>>>>>
> >>>> > > > >>>>>>>>> within
> >>>> > > > >>>>>>
> >>>> > > > >>>>>>> the
> >>>> > > > >>>>>>>>
> >>>> > > > >>>>>>>>> zk
> >>>> > > > >>>>>>>>>> session timeout.
> >>>> > > > >>>>>>>>>>
> >>>> > > > >>>>>>>>>> So, problem is likely on the zk client side of the
> >>>> session;
> >>>> > > > >>>>>>>>>>
> >>>> > > > >>>>>>>>> i.e.
> >>>> > > > >>>
> >>>> > > > >>>> in
> >>>> > > > >>>>>>
> >>>> > > > >>>>>>> the
> >>>> > > > >>>>>>>
> >>>> > > > >>>>>>>> RS.
> >>>> > > > >>>>>>>>>
> >>>> > > > >>>>>>>>>> You could enable GC logging as J-D suggested to see
> if
> >>>> you
> >>>> > > have
> >>>> > > > >>>>>>>>>>
> >>>> > > > >>>>>>>>> any
> >>>> > > > >>>>>>
> >>>> > > > >>>>>>> big
> >>>> > > > >>>>>>>
> >>>> > > > >>>>>>>> pauses, pauses > zk session timeout.
> >>>> > > > >>>>>>>>>>
> >>>> > > > >>>>>>>>>> When the RS went down, it didn't look too heavily
> >>>> loaded:
> >>>> > > > >>>>>>>>>>
> >>>> > > > >>>>>>>>>>
> >>>> > > > >>>>>>>>>> 1. 2009-11-12 15:04:52,830 INFO
> >>>> > > > >>>>>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer:
> >>>> Dump of
> >>>> > > > >>>>>>>>>>
> >>>> > > > >>>>>>>>> metrics:
> >>>> > > > >>>>>>>
> >>>> > > > >>>>>>>> request=1.5166667, regions=322, stores=657,
> >>>> storefiles=631,
> >>>> > > > >>>>>>>>>> storefileIndexSize=61, memstoreSize=1472,
> >>>> usedHeap=2819,
> >>>> > > > >>>>>>>>>>
> >>>> > > > >>>>>>>>> maxHeap=4079,
> >>>> > > > >>>>>>>>
> >>>> > > > >>>>>>>>> blockCacheSize=658110960, blockCacheFree=197395984,
> >>>> > > > >>>>>>>>>>
> >>>> > > > >>>>>>>>> blockCacheCount=9903,
> >>>> > > > >>>>>>>>>
> >>>> > > > >>>>>>>>>> blockCacheHitRatio=99
> >>>> > > > >>>>>>>>>>
> >>>> > > > >>>>>>>>>> Its serving a few reads? The number of store files
> >>>> seems
> >>>> > > fine.
> >>>> > > > >>>>>>>>>>
> >>>> > > > >>>>>>>>> Not
> >>>> > > > >>>>>>
> >>>> > > > >>>>>>> too
> >>>> > > > >>>>>>>>
> >>>> > > > >>>>>>>>> much memory used.
> >>>> > > > >>>>>>>>>>
> >>>> > > > >>>>>>>>>> Looking at the logs, I see the Lease Still Held
> >>>> exception.
> >>>> > > > >>>>>>>>>>
> >>>> > > > >>>>>>>>> This
> >>>> > > > >>>
> >>>> > > > >>>> happens
> >>>> > > > >>>>>>>>
> >>>> > > > >>>>>>>>> when the RS does its regular report to the master but
> the
> >>>> > > > >>>>>>>>>>
> >>>> > > > >>>>>>>>> master
> >>>> > > > >>>
> >>>> > > > >>>> thinks
> >>>> > > > >>>>>>>
> >>>> > > > >>>>>>>> the
> >>>> > > > >>>>>>>>>
> >>>> > > > >>>>>>>>>> RS has since restarted. It'll think this probably
> >>>> because
> >>>> > it
> >>>> > > > >>>>>>>>>>
> >>>> > > > >>>>>>>>> noticed
> >>>> > > > >>>>>>
> >>>> > > > >>>>>>> that
> >>>> > > > >>>>>>>>>
> >>>> > > > >>>>>>>>>> the RS's znode in zk had gone away and it considered
> the
> >>>> RS
> >>>> > > > >>>>>>>>>>
> >>>> > > > >>>>>>>>> dead.
> >>>> > > > >>>
> >>>> > > > >>>>
> >>>> > > > >>>>>>>>>> Looking too at your logs I see this gap in the zk
> >>>> pinging:
> >>>> > > > >>>>>>>>>>
> >>>> > > > >>>>>>>>>>
> >>>> > > > >>>>>>>>>> 1. 2009-11-12 15:03:39,325 DEBUG
> >>>> > > > >>>>>>>>>>
> >>>> > > > >>>>>>>>> org.apache.zookeeper.ClientCnxn:
> >>>> > > > >>>>>>
> >>>> > > > >>>>>>> Got
> >>>> > > > >>>>>>>
> >>>> > > > >>>>>>>> ping response for sessionid:0x224e55436ad0004 after
> 0ms
> >>>> > > > >>>>>>>>>> 2. 2009-11-12 15:03:43,113 DEBUG
> >>>> > > > >>>>>>>>>>
> >>>> > > > >>>>>>>>> org.apache.zookeeper.ClientCnxn:
> >>>> > > > >>>>>>
> >>>> > > > >>>>>>> Got
> >>>> > > > >>>>>>>
> >>>> > > > >>>>>>>> ping response for sessionid:0x24e55436a0007d after 0ms
> >>>> > > > >>>>>>>>>>
> >>>> > > > >>>>>>>>>> Where in the lines above it, its reporting about
> every
> >>>> ten
> >>>> > > > >>>>>>>>>>
> >>>> > > > >>>>>>>>> seconds,
> >>>> > > > >>>>>>
> >>>> > > > >>>>>>> here
> >>>> > > > >>>>>>>>
> >>>> > > > >>>>>>>>> there is a big gap.
> >>>> > > > >>>>>>>>>>
> >>>> > > > >>>>>>>>>> Do you have ganglia or something that will let you
> look
> >>>> more
> >>>> > > > >>>>>>>>>>
> >>>> > > > >>>>>>>>> into
> >>>> > > > >>>
> >>>> > > > >>>> what
> >>>> > > > >>>>>>>
> >>>> > > > >>>>>>>> was
> >>>> > > > >>>>>>>>>
> >>>> > > > >>>>>>>>>> happening on this machine around this time? Is the
> >>>> machine
> >>>> > > OK?
> >>>> > > > >>>>>>>>>>
> >>>> > > > >>>>>>>>> It
> >>>> > > > >>>>>>
> >>>> > > > >>>>>>> looks
> >>>> > > > >>>>>>>>
> >>>> > > > >>>>>>>>> lightly loaded and you have your cluster nicely laid
> out.
> >>>> > > > >>>>>>>>>>
> >>>> > > > >>>>>>>>> Something
> >>>> > > > >>>>>>
> >>>> > > > >>>>>>> odd
> >>>> > > > >>>>>>>>
> >>>> > > > >>>>>>>>> is
> >>>> > > > >>>>>>>>>
> >>>> > > > >>>>>>>>>> going on. What about things like the write speed to
> >>>> disk?
> >>>> > In
> >>>> > > > >>>>>>>>>>
> >>>> > > > >>>>>>>>> the
> >>>> > > > >>>
> >>>> > > > >>>> past
> >>>> > > > >>>>>>>
> >>>> > > > >>>>>>>> strange issues have been explained by incorrectly set
> BIOS
> >>>> > > > >>>>>>>>>>
> >>>> > > > >>>>>>>>> which
> >>>> > > > >>>
> >>>> > > > >>>> made
> >>>> > > > >>>>>>
> >>>> > > > >>>>>>> disks
> >>>> > > > >>>>>>>>>
> >>>> > > > >>>>>>>>>> run at 1/100th of their proper speed.
> >>>> > > > >>>>>>>>>>
> >>>> > > > >>>>>>>>>> St.Ack
> >>>> > > > >>>>>>>>>>
> >>>> > > > >>>>>>>>>>
> >>>> > > > >>>>>>>>>>
> >>>> > > > >>>>>>>>>>
> >>>> > > > >>>>>>>>>>
> >>>> > > > >>>>>>>>>>
> >>>> > > > >>>>>>>>>>
> >>>> > > > >>>>>>>>>>
> >>>> > > > >>>>>>>>>>> Best,
> >>>> > > > >>>>>>>>>>> zhenyu
> >>>> > > > >>>>>>>>>>>
> >>>> > > > >>>>>>>>>>>
> >>>> > > > >>>>>>>>>>> On Wed, Nov 11, 2009 at 3:58 PM, Zhenyu Zhong <
> >>>> > > > >>>>>>>>>>>
> >>>> > > > >>>>>>>>>> zhongresearch@gmail.com
> >>>> > > > >>>>>>>>
> >>>> > > > >>>>>>>>> wrote:
> >>>> > > > >>>>>>>>>>>>
> >>>> > > > >>>>>>>>>>>
> >>>> > > > >>>>>>>>>>> Stack
> >>>> > > > >>>>>>>>>>>>
> >>>> > > > >>>>>>>>>>>> I am very appreciated for your comments.
> >>>> > > > >>>>>>>>>>>> I will use the zookeeper monitoring script on my
> >>>> cluster
> >>>> > > > >>>>>>>>>>>>
> >>>> > > > >>>>>>>>>>> and
> >>>> > > > >>>
> >>>> > > > >>>> let
> >>>> > > > >>>>>>
> >>>> > > > >>>>>>> it
> >>>> > > > >>>>>>>
> >>>> > > > >>>>>>>> run
> >>>> > > > >>>>>>>>>
> >>>> > > > >>>>>>>>>> overnight to see the result.
> >>>> > > > >>>>>>>>>>>> I will follow up the thread when I get anything.
> >>>> > > > >>>>>>>>>>>>
> >>>> > > > >>>>>>>>>>>> thanks
> >>>> > > > >>>>>>>>>>>> zhenyu
> >>>> > > > >>>>>>>>>>>>
> >>>> > > > >>>>>>>>>>>>
> >>>> > > > >>>>>>>>>>>> On Wed, Nov 11, 2009 at 3:52 PM, stack <
> >>>> stack@duboce.net>
> >>>> > > > >>>>>>>>>>>>
> >>>> > > > >>>>>>>>>>> wrote:
> >>>> > > > >>>>>>
> >>>> > > > >>>>>>>
> >>>> > > > >>>>>>>>>>>> I see these in your log too:
> >>>> > > > >>>>>>>>>>>>>
> >>>> > > > >>>>>>>>>>>>>
> >>>> > > > >>>>>>>>>>>>> 1. 2009-11-11 04:27:19,018 DEBUG
> >>>> > > > >>>>>>>>>>>>>
> >>>> > > > >>>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
> >>>> > > > >>>>>>>>
> >>>> > > > >>>>>>>>> Got
> >>>> > > > >>>>>>>>>>
> >>>> > > > >>>>>>>>>>> ping response for sessionid:0x424dfd908c50009 after
> >>>> > > > >>>>>>>>>>>>>
> >>>> > > > >>>>>>>>>>>> 4544ms
> >>>> > > > >>>
> >>>> > > > >>>> 2. 2009-11-11 04:27:19,018 DEBUG
> >>>> > > > >>>>>>>>>>>>>
> >>>> > > > >>>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
> >>>> > > > >>>>>>>>
> >>>> > > > >>>>>>>>> Got
> >>>> > > > >>>>>>>>>>
> >>>> > > > >>>>>>>>>>> ping response for sessionid:0x24dfd90c810002 after
> >>>> > > > >>>>>>>>>>>>>
> >>>> > > > >>>>>>>>>>>> 4548ms
> >>>> > > > >>>
> >>>> > > > >>>> 3. 2009-11-11 04:27:43,960 DEBUG
> >>>> > > > >>>>>>>>>>>>>
> >>>> > > > >>>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
> >>>> > > > >>>>>>>>
> >>>> > > > >>>>>>>>> Got
> >>>> > > > >>>>>>>>>>
> >>>> > > > >>>>>>>>>>> ping response for sessionid:0x424dfd908c50009 after
> >>>> > > > >>>>>>>>>>>>>
> >>>> > > > >>>>>>>>>>>> 9030ms
> >>>> > > > >>>
> >>>> > > > >>>> 4. 2009-11-11 04:27:43,960 DEBUG
> >>>> > > > >>>>>>>>>>>>>
> >>>> > > > >>>>>>>>>>>> org.apache.zookeeper.
> >>>> > > > >>>>>>>>
> >>>> > > > >>>>>>>
> >>>> > > >
> >>>> > >
> >>>> >
> >>>>
> >>>
> >>>
> >>
> >
>
Re: regionserver disconnection
Posted by Jean-Daniel Cryans <jd...@apache.org>.
The error you are getting is a disconnection from a zookeeper server
and is very generic.
ZK-86 is still opened and the last comment refers to ZK-111 saying
that the bug (in unit tests) was probably fixed in release 3.0.0 last
year.
To figure the hang you have, you can try to jstack the process pid and
see exactly what's holding the RS from shutting down.
Would it be possible for you to drop by the IRC channel? This way we
can debug this a at much faster pace.
Thx!
J-D
On Sat, Nov 14, 2009 at 9:57 AM, Zhenyu Zhong <zh...@gmail.com> wrote:
> I found this.
> http://issues.apache.org/jira/browse/ZOOKEEPER-86
>
> It looks like the same error I had. Is it a zookeeper bug? When will HBase
> take the zookeeper version 3.3.0?
>
> thanks
> zhenyu
>
> On Sat, Nov 14, 2009 at 11:31 AM, Zhenyu Zhong <zh...@gmail.com>wrote:
>
>> Now I really doubt about the zookeeper, from the log I saw errors like
>> IOException Read Error, while the zookeeper client (regionserver ) tried to
>> read. But it got disconneted status from zookeeper.
>>
>> I don't see any load on any zookeeper quorum servers. DiskIO is fine.
>>
>> Also when the regionserver decides to exit due to the disconnect status
>> from zookeeper, sometimes the regionserver hangs during the exiting. We can
>> still see the HRegionServer process even we don't see it in the Master web
>> interface.
>>
>> I also notice that there is an zookeeper.retries and zookeeper.pauses
>> settings in hbase-default.xml, would that matter ?
>>
>> thanks
>>
>> You may see the IOException error in the following log when the
>> Regionserver lost connection to the zookeeper.
>>
>> 2009-11-14 15:58:31,993 INFO org.apache.hadoop.hbase.regionserver.HLog:
>> Roll
>> /hbase/.logs/superpyxis0008.scur.colo,60021,1258179101617/hlog.dat.1258179106921,
>> entries=1062964, calcsize=255014936, filesize=139566093. New hlog
>> /hbase/.logs/superpyxis0008.scur.colo,60021,1258179101617/hlog.dat.1258214311993
>> 2009-11-14 15:58:39,295 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
>> response for sessionid:0x424f0ed01ea00a6 after 0ms
>> 2009-11-14 15:58:52,648 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
>> response for sessionid:0x424f0ed01ea00a6 after 0ms
>> 2009-11-14 15:59:06,007 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
>> response for sessionid:0x424f0ed01ea00a6 after 0ms
>> 2009-11-14 15:59:19,365 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
>> response for sessionid:0x424f0ed01ea00a6 after 0ms
>> 2009-11-14 15:59:32,867 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
>> response for sessionid:0x424f0ed01ea00a6 after 186ms
>> 2009-11-14 15:59:46,070 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
>> response for sessionid:0x424f0ed01ea00a6 after 0ms
>> 2009-11-14 15:59:59,423 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
>> response for sessionid:0x424f0ed01ea00a6 after 0ms
>> 2009-11-14 16:00:12,775 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
>> response for sessionid:0x424f0ed01ea00a6 after 0ms
>> 2009-11-14 16:00:28,141 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
>> response for sessionid:0x424f0ed01ea00a6 after 2010ms
>> 2009-11-14 16:01:10,378 WARN org.apache.zookeeper.ClientCnxn: Exception
>> closing session 0x424f0ed01ea00a6 to sun.nio.ch.SelectionKeyImpl@39fbb2d6
>>
>> java.io.IOException: Read error rc = -1 java.nio.DirectByteBuffer[pos=0
>> lim=4 cap=4]
>> at
>> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
>> at
>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
>> 2009-11-14 16:01:10,478 INFO
>> org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
>> state: Disconnected, type: None, path: null
>> 2009-11-14 16:01:11,333 WARN
>> org.apache.hadoop.hbase.regionserver.HRegionServer: Attempt=1
>>
>> org.apache.hadoop.hbase.Leases$LeaseStillHeldException
>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>> Method)
>> at
>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>> at
>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>> at
>> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
>> at
>> org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
>> at
>> org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:571)
>> at java.lang.Thread.run(Thread.java:619)
>> 2009-11-14 16:01:11,433 INFO org.apache.zookeeper.ClientCnxn: Attempting
>> connection to server superpyxis0001/192.168.100.115:2181
>> 2009-11-14 16:01:11,433 INFO org.apache.zookeeper.ClientCnxn: Priming
>> connection to java.nio.channels.SocketChannel[connected local=/
>> 192.168.100.122:59575 remote=superpyxis0001/192.168.100.115:2181]
>> 2009-11-14 16:01:11,433 INFO org.apache.zookeeper.ClientCnxn: Server
>> connection successful
>> 2009-11-14 16:01:11,433 INFO
>> org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
>> state: Expired, type: None, path: null
>> 2009-11-14 16:01:11,433 WARN org.apache.zookeeper.ClientCnxn: Exception
>> closing session 0x424f0ed01ea00a6 to sun.nio.ch.SelectionKeyImpl@6b93d343
>> java.io.IOException: Session Expired
>> at
>> org.apache.zookeeper.ClientCnxn$SendThread.readConnectResult(ClientCnxn.java:589)
>> at
>> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:709)
>>
>> at
>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
>> 2009-11-14 16:01:11,433 ERROR
>> org.apache.hadoop.hbase.regionserver.HRegionServer: ZooKeeper session
>> expired
>> 2009-11-14 16:01:13,513 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
>> handler 3 on 60021, call put([B@73deb204,
>> [Lorg.apache.hadoop.hbase.client.Put;@2179600a) from 192.168.100.132:40728:
>> error: java.io.IOException: Server not running, aborting
>> java.io.IOException: Server not running, aborting
>>
>>
>>
>>
>>
>> On Sat, Nov 14, 2009 at 1:01 AM, Zhenyu Zhong <zh...@gmail.com>wrote:
>>
>>> Stack,
>>>
>>> Thanks a lot!
>>>
>>> I found out that the reason why HBase doesn't take the system file
>>> descriptor value I set before. I started the HBase using root instead of the
>>> normal hadoop user, while my system configures hadoop with higher file
>>> descriptor value but configures root with 1024 default value.
>>>
>>>
>>> Now my system has a clean start. However, it seems that the
>>> zookeeper.session.timeout value doesn't take into effect. I still found
>>> around 60 seconds pauses from the disconnected regionserver. I really don't
>>> know why regionserver only times out after 60 seconds instead of 10 minutes
>>> which I set for zookeeper.session.timeout.
>>>
>>> Is there any other timeout value coming into play before the actual
>>> session times out?
>>>
>>> zhenyu
>>>
>>>
>>>
>>> On Fri, Nov 13, 2009 at 7:08 PM, stack <st...@duboce.net> wrote:
>>>
>>>> Ok. Lack of file descriptors manifests in all kinds of weird ways.
>>>> Hopefully thats it. If not, lets keep digging.
>>>> St.Ack
>>>>
>>>> On Fri, Nov 13, 2009 at 3:44 PM, Zhenyu Zhong <zhongresearch@gmail.com
>>>> >wrote:
>>>>
>>>> > Stack,
>>>> >
>>>> > You are right, the master started with ulimit -n 1024. It doesn't take
>>>> the
>>>> > system value.
>>>> >
>>>> > Regarding to the too many open files, it looks like the same as the one
>>>> J-D
>>>> > put up. But I will get the Master start with higher value first and see
>>>> if
>>>> > this kind of error goes away.
>>>> >
>>>> > thanks a lot!
>>>> > zhenyu
>>>> >
>>>> > On Fri, Nov 13, 2009 at 6:02 PM, stack <st...@duboce.net> wrote:
>>>> >
>>>> > > Does it say
>>>> > >
>>>> > > ulimit -n 32768
>>>> > >
>>>> > > ...as the second line in your log file on start of the master?
>>>> > >
>>>> > > Is the stack trace that complains about too many open files same as
>>>> the
>>>> > one
>>>> > > in the blog post J-D put up?
>>>> > >
>>>> > > St.Ack
>>>> > >
>>>> > >
>>>> > > On Fri, Nov 13, 2009 at 1:37 PM, Zhenyu Zhong <
>>>> zhongresearch@gmail.com
>>>> > > >wrote:
>>>> > >
>>>> > > > The ulimit file descriptors was set to fs.file-max =1578334, also
>>>> in
>>>> > the
>>>> > > > limits.conf the value was set to 32768.
>>>> > > > So these are way higher than the open descriptors for the running
>>>> > > > processes.
>>>> > > >
>>>> > > > thanks
>>>> > > > zhenyu
>>>> > > >
>>>> > > >
>>>> > > > On Fri, Nov 13, 2009 at 4:33 PM, Stack <sa...@gmail.com>
>>>> wrote:
>>>> > > >
>>>> > > > > You upped the ulimit file descriptors as per the getting started
>>>> doc?
>>>> > > > >
>>>> > > > >
>>>> > > > >
>>>> > > > > On Nov 13, 2009, at 1:26 PM, Zhenyu Zhong <
>>>> zhongresearch@gmail.com>
>>>> > > > wrote:
>>>> > > > >
>>>> > > > > Thanks a lot.
>>>> > > > >>
>>>> > > > >>
>>>> > > > >> Bad news is my kernel is still 2.6.26.
>>>> > > > >> But it was not a problem before.
>>>> > > > >>
>>>> > > > >> Very strange.
>>>> > > > >>
>>>> > > > >> zhenyu
>>>> > > > >>
>>>> > > > >> On Fri, Nov 13, 2009 at 4:16 PM, Jean-Daniel Cryans <
>>>> > > > jdcryans@apache.org
>>>> > > > >> >wrote:
>>>> > > > >>
>>>> > > > >> Looks like
>>>> > > > >>>
>>>> > > > >>>
>>>> > > >
>>>> > >
>>>> >
>>>> http://pero.blogs.aprilmayjune.org/2009/01/22/hadoop-and-linux-kernel-2627-epoll-limits/
>>>> > > > >>>
>>>> > > > >>> J-D
>>>> > > > >>>
>>>> > > > >>> On Fri, Nov 13, 2009 at 1:12 PM, Zhenyu Zhong <
>>>> > > zhongresearch@gmail.com
>>>> > > > >
>>>> > > > >>> wrote:
>>>> > > > >>>
>>>> > > > >>>> Hi,
>>>> > > > >>>>
>>>> > > > >>>>
>>>> > > > >>>> After I re-organize the cluster, the experiment ran into
>>>> problem
>>>> > > > faster
>>>> > > > >>>>
>>>> > > > >>> than
>>>> > > > >>>
>>>> > > > >>>> before.
>>>> > > > >>>>
>>>> > > > >>>> Basically the changes are to use machines with less resources
>>>> as
>>>> > > > >>>>
>>>> > > > >>> zookeeper
>>>> > > > >>>
>>>> > > > >>>> quorums and machines with more resources as regionserver.
>>>> > > > >>>>
>>>> > > > >>>> From the log, I still see the pause around 1 minute.
>>>> > > > >>>> I enabled the GC logging, please see
>>>> > > > >>>>
>>>> > > > >>>> http://pastebin.com/m1d4ce0f1
>>>> > > > >>>>
>>>> > > > >>>> for details.
>>>> > > > >>>> In general I don't see many pauses in the GC.
>>>> > > > >>>>
>>>> > > > >>>> What is more interesting, 1 hour after the 1st regionserver
>>>> > > > >>>> disconnected,
>>>> > > > >>>> the master log complained about too many open files. This
>>>> didn't
>>>> > > > happen
>>>> > > > >>>> before.
>>>> > > > >>>> I checked the system OS setup as well as the limits.conf. I
>>>> also
>>>> > > > checked
>>>> > > > >>>>
>>>> > > > >>> the
>>>> > > > >>>
>>>> > > > >>>> running processes. The total open files don't reach the limit.
>>>> I
>>>> > am
>>>> > > > >>>>
>>>> > > > >>> confused
>>>> > > > >>>
>>>> > > > >>>> a bit.
>>>> > > > >>>>
>>>> > > > >>>>
>>>> > > > >>>> Please see the following master log.
>>>> > > > >>>>
>>>> > > > >>>> 2009-11-13 20:06:02,114 INFO
>>>> > > > org.apache.hadoop.hbase.master.BaseScanner:
>>>> > > > >>>> RegionManager.metaScanner scan of 4658 row(s) of meta region
>>>> > > {server:
>>>> > > > >>>> 192.168.100.128:60021, regionname: .META.,,1, startKey: <>}
>>>> > > complete
>>>> > > > >>>> 2009-11-13 20:06:02,114 INFO
>>>> > > > org.apache.hadoop.hbase.master.BaseScanner:
>>>> > > > >>>>
>>>> > > > >>> All
>>>> > > > >>>
>>>> > > > >>>> 1 .META. region(s) scanned
>>>> > > > >>>> 2009-11-13 20:06:07,677 DEBUG org.apache.zookeeper.ClientCnxn:
>>>> Got
>>>> > > > ping
>>>> > > > >>>> response for sessionid:0x424eebf1c10004c after 3ms
>>>> > > > >>>> 2009-11-13 20:06:08,178 INFO org.apache.hadoop.hdfs.DFSClient:
>>>> > > > Exception
>>>> > > > >>>>
>>>> > > > >>> in
>>>> > > > >>>
>>>> > > > >>>> createBlockOutputStream java.io.IOException: Bad connect ack
>>>> with
>>>> > > > >>>> firstBadLink 192.168.100.123:50010
>>>> > > > >>>> 2009-11-13 20:06:08,178 INFO org.apache.hadoop.hdfs.DFSClient:
>>>> > > > >>>> Abandoning
>>>> > > > >>>> block blk_-2808245019291145247_5478039
>>>> > > > >>>> 2009-11-13 20:06:09,682 INFO org.apache.hadoop.hdfs.DFSClient:
>>>> > > > Exception
>>>> > > > >>>>
>>>> > > > >>> in
>>>> > > > >>>
>>>> > > > >>>> createBlockOutputStream java.io.EOFException
>>>> > > > >>>> 2009-11-13 20:06:09,682 INFO org.apache.hadoop.hdfs.DFSClient:
>>>> > > > >>>> Abandoning
>>>> > > > >>>> block blk_1074853606841896259_5478048
>>>> > > > >>>> 2009-11-13 20:06:10,334 DEBUG org.apache.zookeeper.ClientCnxn:
>>>> Got
>>>> > > > ping
>>>> > > > >>>> response for sessionid:0x24eebf1043003c after 1ms
>>>> > > > >>>> 2009-11-13 20:06:21,018 DEBUG org.apache.zookeeper.ClientCnxn:
>>>> Got
>>>> > > > ping
>>>> > > > >>>> response for sessionid:0x424eebf1c10004c after 0ms
>>>> > > > >>>> 2009-11-13 20:06:23,674 DEBUG org.apache.zookeeper.ClientCnxn:
>>>> Got
>>>> > > > ping
>>>> > > > >>>> response for sessionid:0x24eebf1043003c after 0ms
>>>> > > > >>>> 2009-11-13 20:06:24,828 INFO org.apache.hadoop.hdfs.DFSClient:
>>>> > > > Exception
>>>> > > > >>>>
>>>> > > > >>> in
>>>> > > > >>>
>>>> > > > >>>> createBlockOutputStream java.io.IOException: Bad connect ack
>>>> with
>>>> > > > >>>> firstBadLink 192.168.100.123:50010
>>>> > > > >>>> 2009-11-13 20:06:24,828 INFO org.apache.hadoop.hdfs.DFSClient:
>>>> > > > >>>> Abandoning
>>>> > > > >>>> block blk_-6642544517082142289_5478063
>>>> > > > >>>> 2009-11-13 20:06:24,828 INFO org.apache.hadoop.hdfs.DFSClient:
>>>> > > > Exception
>>>> > > > >>>>
>>>> > > > >>> in
>>>> > > > >>>
>>>> > > > >>>> createBlockOutputStream java.net.SocketException: Too many
>>>> open
>>>> > > files
>>>> > > > >>>> 2009-11-13 20:06:24,828 INFO org.apache.hadoop.hdfs.DFSClient:
>>>> > > > >>>> Abandoning
>>>> > > > >>>> block blk_2057511041109796090_5478063
>>>> > > > >>>> 2009-11-13 20:06:24,928 INFO org.apache.hadoop.hdfs.DFSClient:
>>>> > > > Exception
>>>> > > > >>>>
>>>> > > > >>> in
>>>> > > > >>>
>>>> > > > >>>> createBlockOutputStream java.net.SocketException: Too many
>>>> open
>>>> > > files
>>>> > > > >>>> 2009-11-13 20:06:24,928 INFO org.apache.hadoop.hdfs.DFSClient:
>>>> > > > >>>> Abandoning
>>>> > > > >>>> block blk_8219260302213892894_5478064
>>>> > > > >>>> 2009-11-13 20:06:30,855 INFO org.apache.hadoop.hdfs.DFSClient:
>>>> > > > Exception
>>>> > > > >>>>
>>>> > > > >>> in
>>>> > > > >>>
>>>> > > > >>>> createBlockOutputStream java.net.SocketException: Too many
>>>> open
>>>> > > files
>>>> > > > >>>> 2009-11-13 20:06:30,855 INFO org.apache.hadoop.hdfs.DFSClient:
>>>> > > > >>>> Abandoning
>>>> > > > >>>> block blk_1669205542853067709_5478235
>>>> > > > >>>> 2009-11-13 20:06:30,905 INFO org.apache.hadoop.hdfs.DFSClient:
>>>> > > > Exception
>>>> > > > >>>>
>>>> > > > >>> in
>>>> > > > >>>
>>>> > > > >>>> createBlockOutputStream java.net.SocketException: Too many
>>>> open
>>>> > > files
>>>> > > > >>>> 2009-11-13 20:06:30,905 INFO org.apache.hadoop.hdfs.DFSClient:
>>>> > > > >>>> Abandoning
>>>> > > > >>>> block blk_9128897691346270351_5478237
>>>> > > > >>>> 2009-11-13 20:06:30,955 INFO org.apache.hadoop.hdfs.DFSClient:
>>>> > > > Exception
>>>> > > > >>>>
>>>> > > > >>> in
>>>> > > > >>>
>>>> > > > >>>> createBlockOutputStream java.net.SocketException: Too many
>>>> open
>>>> > > files
>>>> > > > >>>> 2009-11-13 20:06:30,955 INFO org.apache.hadoop.hdfs.DFSClient:
>>>> > > > >>>> Abandoning
>>>> > > > >>>> block blk_1116845144864123018_5478240
>>>> > > > >>>> 2009-11-13 20:06:34,372 DEBUG org.apache.zookeeper.ClientCnxn:
>>>> Got
>>>> > > > ping
>>>> > > > >>>> response for sessionid:0x424eebf1c10004c after 0ms
>>>> > > > >>>> 2009-11-13 20:06:37,034 DEBUG org.apache.zookeeper.ClientCnxn:
>>>> Got
>>>> > > > ping
>>>> > > > >>>> response for sessionid:0x24eebf1043003c after 0ms
>>>> > > > >>>> 2009-11-13 20:06:37,235 WARN org.apache.hadoop.hdfs.DFSClient:
>>>> > > > >>>>
>>>> > > > >>> DataStreamer
>>>> > > > >>>
>>>> > > > >>>> Exception: java.io.IOException: Too many open files
>>>> > > > >>>> at sun.nio.ch.IOUtil.initPipe(Native Method)
>>>> > > > >>>> at
>>>> > > sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:49)
>>>> > > > >>>> at
>>>> > > > >>>>
>>>> > > > >>>>
>>>> > > >
>>>> > >
>>>> >
>>>> sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:18)
>>>> > > > >>>
>>>> > > > >>>> at
>>>> > > > >>>>
>>>> > > > >>>>
>>>> > > >
>>>> > >
>>>> >
>>>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.get(SocketIOWithTimeout.java:407)
>>>> > > > >>>
>>>> > > > >>>> at
>>>> > > > >>>>
>>>> > > > >>>>
>>>> > > >
>>>> > >
>>>> >
>>>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:322)
>>>> > > > >>>
>>>> > > > >>>> at
>>>> > > > >>>>
>>>> > > > >>>>
>>>> > > >
>>>> > >
>>>> >
>>>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
>>>> > > > >>>
>>>> > > > >>>> at
>>>> > > > >>>>
>>>> > > > >>>>
>>>> > > >
>>>> > >
>>>> >
>>>> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
>>>> > > > >>>
>>>> > > > >>>> at
>>>> > > > >>>>
>>>> > > > >>>>
>>>> > > >
>>>> > >
>>>> >
>>>> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
>>>> > > > >>>
>>>> > > > >>>> at
>>>> > > > >>>>
>>>> > > > >>>
>>>> java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
>>>> > > > >>>
>>>> > > > >>>> at
>>>> java.io.DataOutputStream.write(DataOutputStream.java:90)
>>>> > > > >>>> at
>>>> > > > >>>>
>>>> > > > >>>>
>>>> > > >
>>>> > >
>>>> >
>>>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2290)
>>>> > > > >>>
>>>> > > > >>>>
>>>> > > > >>>> 2009-11-13 20:06:37,235 WARN org.apache.hadoop.hdfs.DFSClient:
>>>> > Error
>>>> > > > >>>> Recovery for block blk_8148813491785406356_5478475 bad
>>>> datanode[0]
>>>> > > > >>>> 192.168.100.123:50010
>>>> > > > >>>> 2009-11-13 20:06:37,235 WARN org.apache.hadoop.hdfs.DFSClient:
>>>> > Error
>>>> > > > >>>> Recovery for block blk_8148813491785406356_5478475 in pipeline
>>>> > > > >>>> 192.168.100.123:50010, 192.168.100.134:50010,
>>>> > 192.168.100.122:50010
>>>> > > :
>>>> > > > >>>> bad
>>>> > > > >>>> datanode 192.168.100.123:50010
>>>> > > > >>>> 2009-11-13 20:06:37,436 INFO org.apache.hadoop.hdfs.DFSClient:
>>>> > > > Exception
>>>> > > > >>>>
>>>> > > > >>> in
>>>> > > > >>>
>>>> > > > >>>> createBlockOutputStream java.net.SocketException: Too many
>>>> open
>>>> > > files
>>>> > > > >>>> 2009-11-13 20:06:37,436 INFO org.apache.hadoop.hdfs.DFSClient:
>>>> > > > >>>> Abandoning
>>>> > > > >>>> block blk_2119727700857186236_5478498
>>>> > > > >>>>
>>>> > > > >>>>
>>>> > > > >>>>
>>>> > > > >>>>
>>>> > > > >>>>
>>>> > > > >>>> On Thu, Nov 12, 2009 at 4:21 PM, Zhenyu Zhong <
>>>> > > > zhongresearch@gmail.com
>>>> > > > >>>> wrote:
>>>> > > > >>>>
>>>> > > > >>>> Will do.
>>>> > > > >>>>>
>>>> > > > >>>>> thanks
>>>> > > > >>>>> zhenyu
>>>> > > > >>>>>
>>>> > > > >>>>>
>>>> > > > >>>>> On Thu, Nov 12, 2009 at 3:33 PM, stack <st...@duboce.net>
>>>> wrote:
>>>> > > > >>>>>
>>>> > > > >>>>> Enable GC logging too on this next run (see hbase-env.sh).
>>>> Lets
>>>> > > try
>>>> > > > >>>>>>
>>>> > > > >>>>> and
>>>> > > > >>>
>>>> > > > >>>> get
>>>> > > > >>>>>> to the bottom of whats going on.
>>>> > > > >>>>>> Thanks,
>>>> > > > >>>>>> St.Ack
>>>> > > > >>>>>>
>>>> > > > >>>>>> On Thu, Nov 12, 2009 at 12:29 PM, Zhenyu Zhong <
>>>> > > > >>>>>>
>>>> > > > >>>>> zhongresearch@gmail.com
>>>> > > > >>>
>>>> > > > >>>> wrote:
>>>> > > > >>>>>>>
>>>> > > > >>>>>>
>>>> > > > >>>>>> I can switch the boxes that run zookeeper with the ones
>>>> that
>>>> > run
>>>> > > > >>>>>>> regionservers.
>>>> > > > >>>>>>> I will see the results later.
>>>> > > > >>>>>>>
>>>> > > > >>>>>>> FYI. The node does have the 10 minutes
>>>> > zookeeper.session.timeout
>>>> > > > >>>>>>>
>>>> > > > >>>>>> value
>>>> > > > >>>
>>>> > > > >>>> in
>>>> > > > >>>>>>
>>>> > > > >>>>>>> place.
>>>> > > > >>>>>>>
>>>> > > > >>>>>>> thanks
>>>> > > > >>>>>>> zhenyu
>>>> > > > >>>>>>>
>>>> > > > >>>>>>> On Thu, Nov 12, 2009 at 3:21 PM, stack <st...@duboce.net>
>>>> > wrote:
>>>> > > > >>>>>>>
>>>> > > > >>>>>>> On Thu, Nov 12, 2009 at 11:50 AM, Zhenyu Zhong <
>>>> > > > >>>>>>>>
>>>> > > > >>>>>>> zhongresearch@gmail.com
>>>> > > > >>>>>>
>>>> > > > >>>>>>> wrote:
>>>> > > > >>>>>>>>>
>>>> > > > >>>>>>>>
>>>> > > > >>>>>>>>
>>>> > > > >>>>>>>>> In my cluster, half of the cluster have 2 disks 400GB
>>>> each
>>>> > per
>>>> > > > >>>>>>>>>
>>>> > > > >>>>>>>> machine,
>>>> > > > >>>>>>
>>>> > > > >>>>>>> and
>>>> > > > >>>>>>>>
>>>> > > > >>>>>>>>> half of the cluster have 6 disks per machine. Maybe we
>>>> > should
>>>> > > > >>>>>>>>>
>>>> > > > >>>>>>>> run
>>>> > > > >>>
>>>> > > > >>>> zookeeper
>>>> > > > >>>>>>>>> on the machines with 2 disks and RS on machines with 6
>>>> disks?
>>>> > > > >>>>>>>>>
>>>> > > > >>>>>>>>> That would make most sense only in the below, it looks
>>>> like
>>>> > the
>>>> > > > >>>>>>>>>
>>>> > > > >>>>>>>> RS
>>>> > > > >>>
>>>> > > > >>>> that
>>>> > > > >>>>>>
>>>> > > > >>>>>>> had
>>>> > > > >>>>>>>> issue had 4 disks?
>>>> > > > >>>>>>>>
>>>> > > > >>>>>>>>
>>>> > > > >>>>>>>>
>>>> > > > >>>>>>>> BTW, the 10 minutes zookeeper.session.timeout has been
>>>> set
>>>> > > during
>>>> > > > >>>>>>>>>
>>>> > > > >>>>>>>> the
>>>> > > > >>>>>>
>>>> > > > >>>>>>> experiment.
>>>> > > > >>>>>>>>>
>>>> > > > >>>>>>>>>
>>>> > > > >>>>>>>> And for sure this node had it in place?
>>>> > > > >>>>>>>> St.Ack
>>>> > > > >>>>>>>>
>>>> > > > >>>>>>>>
>>>> > > > >>>>>>>>
>>>> > > > >>>>>>>>
>>>> > > > >>>>>>>>> thanks
>>>> > > > >>>>>>>>> zhenyu
>>>> > > > >>>>>>>>>
>>>> > > > >>>>>>>>> On Thu, Nov 12, 2009 at 2:08 PM, stack <stack@duboce.net
>>>> >
>>>> > > wrote:
>>>> > > > >>>>>>>>>
>>>> > > > >>>>>>>>> On Thu, Nov 12, 2009 at 8:40 AM, Zhenyu Zhong <
>>>> > > > >>>>>>>>>>
>>>> > > > >>>>>>>>> zhongresearch@gmail.com
>>>> > > > >>>>>>>
>>>> > > > >>>>>>>> wrote:
>>>> > > > >>>>>>>>>>>
>>>> > > > >>>>>>>>>>
>>>> > > > >>>>>>>>>>
>>>> > > > >>>>>>>>>> Though I experienced 2 regionserver disconnection this
>>>> > > > >>>>>>>>>>>
>>>> > > > >>>>>>>>>> morning,
>>>> > > > >>>
>>>> > > > >>>> it
>>>> > > > >>>>>>
>>>> > > > >>>>>>> looks
>>>> > > > >>>>>>>>>
>>>> > > > >>>>>>>>>> better from the regionserver log. (please see the
>>>> following
>>>> > > > >>>>>>>>>>>
>>>> > > > >>>>>>>>>> log)
>>>> > > > >>>
>>>> > > > >>>>
>>>> > > > >>>>>>>>>>>
>>>> > > > >>>>>>>>>>> http://pastebin.com/m496dbfae
>>>> > > > >>>>>>>>>>>
>>>> > > > >>>>>>>>>>>
>>>> > > > >>>>>>>>>>> I checked diskIO for one of the
>>>> > regionserver(192.168.100.116)
>>>> > > > >>>>>>>>>>>
>>>> > > > >>>>>>>>>> during
>>>> > > > >>>>>>>
>>>> > > > >>>>>>>> the
>>>> > > > >>>>>>>>>
>>>> > > > >>>>>>>>>> time it disconnected.
>>>> > > > >>>>>>>>>>>
>>>> > > > >>>>>>>>>>> Time: 03:04:58 AM
>>>> > > > >>>>>>>>>>> Device: tps Blk_read/s Blk_wrtn/s
>>>> Blk_read
>>>> > > > >>>>>>>>>>>
>>>> > > > >>>>>>>>>> Blk_wrtn
>>>> > > > >>>>>>>>
>>>> > > > >>>>>>>>> sda 105.31 5458.83 19503.64
>>>> 9043873239
>>>> > > > >>>>>>>>>>>
>>>> > > > >>>>>>>>>> 32312473676
>>>> > > > >>>>>>>>
>>>> > > > >>>>>>>>> sda1 2.90 6.64 99.25
>>>> 10993934
>>>> > > > >>>>>>>>>>>
>>>> > > > >>>>>>>>>> 164433464
>>>> > > > >>>>>>>>
>>>> > > > >>>>>>>>> sda2 1.72 23.76 30.25
>>>> 39365817
>>>> > > > >>>>>>>>>>>
>>>> > > > >>>>>>>>>> 50124008
>>>> > > > >>>>>>>>
>>>> > > > >>>>>>>>> sda3 0.30 0.38 3.58
>>>> 624930
>>>> > > > >>>>>>>>>>>
>>>> > > > >>>>>>>>>> 5923000
>>>> > > > >>>>>>>>
>>>> > > > >>>>>>>>> sda4 100.39 5428.06 19370.56
>>>> 8992888270
>>>> > > > >>>>>>>>>>>
>>>> > > > >>>>>>>>>> 32091993204
>>>> > > > >>>>>>>>
>>>> > > > >>>>>>>>>
>>>> > > > >>>>>>>>>>>
>>>> > > > >>>>>>>>>>> Is this high for you? 20k blocks/second would seem to
>>>> be
>>>> > > high
>>>> > > > >>>>>>>>>>
>>>> > > > >>>>>>>>> but
>>>> > > > >>>
>>>> > > > >>>> its
>>>> > > > >>>>>>>
>>>> > > > >>>>>>>> one
>>>> > > > >>>>>>>>>
>>>> > > > >>>>>>>>>> disk only and its not being shared by zk anymore so
>>>> > shouldn't
>>>> > > > >>>>>>>>>>
>>>> > > > >>>>>>>>> matter?
>>>> > > > >>>>>>
>>>> > > > >>>>>>>
>>>> > > > >>>>>>>>>>
>>>> > > > >>>>>>>>>> I also checked the zookeeper quorum server that the
>>>> > > > >>>>>>>>>>>
>>>> > > > >>>>>>>>>> regionserver
>>>> > > > >>>
>>>> > > > >>>> tried
>>>> > > > >>>>>>>>
>>>> > > > >>>>>>>>> to
>>>> > > > >>>>>>>>>
>>>> > > > >>>>>>>>>> connect according to the log. However, I don't see
>>>> > > > >>>>>>>>>>>
>>>> > > > >>>>>>>>>> 192.168.100.116
>>>> > > > >>>>>>
>>>> > > > >>>>>>> in
>>>> > > > >>>>>>>
>>>> > > > >>>>>>>> the
>>>> > > > >>>>>>>>>
>>>> > > > >>>>>>>>>> client list of the zookeeper quorum server that the
>>>> > > > >>>>>>>>>>>
>>>> > > > >>>>>>>>>> regionserver
>>>> > > > >>>
>>>> > > > >>>> tried
>>>> > > > >>>>>>>>
>>>> > > > >>>>>>>>> to
>>>> > > > >>>>>>>>>
>>>> > > > >>>>>>>>>> connect.
>>>> > > > >>>>>>>>>>> Would that be a problem?
>>>> > > > >>>>>>>>>>>
>>>> > > > >>>>>>>>>>>
>>>> > > > >>>>>>>>>>> Is that because the ephemeral node for the
>>>> regionserver
>>>> > had
>>>> > > > >>>>>>>>>>
>>>> > > > >>>>>>>>> evaporated?
>>>> > > > >>>>>>>
>>>> > > > >>>>>>>> Lost
>>>> > > > >>>>>>>>>> its lease w/ zk by the time you went to look?
>>>> > > > >>>>>>>>>>
>>>> > > > >>>>>>>>>>
>>>> > > > >>>>>>>>>>
>>>> > > > >>>>>>>>>>
>>>> > > > >>>>>>>>>> Thu Nov 12 15:04:52 UTC 2009
>>>> > > > >>>>>>>>>>> Zookeeper version: 3.2.1-808558, built on 08/27/2009
>>>> 18:48
>>>> > > > >>>>>>>>>>>
>>>> > > > >>>>>>>>>> GMT
>>>> > > > >>>
>>>> > > > >>>> Clients:
>>>> > > > >>>>>>>>>>> /192.168.100.127:43045[1](queued=0,recved=26,sent=0)
>>>> > > > >>>>>>>>>>> /192.168.100.131:39091[1](queued=0,recved=964,sent=0)
>>>> > > > >>>>>>>>>>> /192.168.100.124:35961[1](queued=0,recved=3266,sent=0)
>>>> > > > >>>>>>>>>>> /192.168.100.123:47935[1](queued=0,recved=233,sent=0)
>>>> > > > >>>>>>>>>>> /192.168.100.125:46931[1](queued=0,recved=2,sent=0)
>>>> > > > >>>>>>>>>>> /192.168.100.118:54924[1](queued=0,recved=295,sent=0)
>>>> > > > >>>>>>>>>>> /192.168.100.118:41390[1](queued=0,recved=2290,sent=0)
>>>> > > > >>>>>>>>>>> /192.168.100.136:42243[1](queued=0,recved=0,sent=0)
>>>> > > > >>>>>>>>>>>
>>>> > > > >>>>>>>>>>> Latency min/avg/max: 0/17/6333
>>>> > > > >>>>>>>>>>> Received: 47111
>>>> > > > >>>>>>>>>>> Sent: 0
>>>> > > > >>>>>>>>>>> Outstanding: 0
>>>> > > > >>>>>>>>>>> Zxid: 0x77000083f4
>>>> > > > >>>>>>>>>>> Mode: leader
>>>> > > > >>>>>>>>>>> Node count: 23
>>>> > > > >>>>>>>>>>>
>>>> > > > >>>>>>>>>>>
>>>> > > > >>>>>>>>>>
>>>> > > > >>>>>>>>>> That 6 second maximum latency is pretty bad but should
>>>> be
>>>> > well
>>>> > > > >>>>>>>>>>
>>>> > > > >>>>>>>>> within
>>>> > > > >>>>>>
>>>> > > > >>>>>>> the
>>>> > > > >>>>>>>>
>>>> > > > >>>>>>>>> zk
>>>> > > > >>>>>>>>>> session timeout.
>>>> > > > >>>>>>>>>>
>>>> > > > >>>>>>>>>> So, problem is likely on the zk client side of the
>>>> session;
>>>> > > > >>>>>>>>>>
>>>> > > > >>>>>>>>> i.e.
>>>> > > > >>>
>>>> > > > >>>> in
>>>> > > > >>>>>>
>>>> > > > >>>>>>> the
>>>> > > > >>>>>>>
>>>> > > > >>>>>>>> RS.
>>>> > > > >>>>>>>>>
>>>> > > > >>>>>>>>>> You could enable GC logging as J-D suggested to see if
>>>> you
>>>> > > have
>>>> > > > >>>>>>>>>>
>>>> > > > >>>>>>>>> any
>>>> > > > >>>>>>
>>>> > > > >>>>>>> big
>>>> > > > >>>>>>>
>>>> > > > >>>>>>>> pauses, pauses > zk session timeout.
>>>> > > > >>>>>>>>>>
>>>> > > > >>>>>>>>>> When the RS went down, it didn't look too heavily
>>>> loaded:
>>>> > > > >>>>>>>>>>
>>>> > > > >>>>>>>>>>
>>>> > > > >>>>>>>>>> 1. 2009-11-12 15:04:52,830 INFO
>>>> > > > >>>>>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>>>> Dump of
>>>> > > > >>>>>>>>>>
>>>> > > > >>>>>>>>> metrics:
>>>> > > > >>>>>>>
>>>> > > > >>>>>>>> request=1.5166667, regions=322, stores=657,
>>>> storefiles=631,
>>>> > > > >>>>>>>>>> storefileIndexSize=61, memstoreSize=1472,
>>>> usedHeap=2819,
>>>> > > > >>>>>>>>>>
>>>> > > > >>>>>>>>> maxHeap=4079,
>>>> > > > >>>>>>>>
>>>> > > > >>>>>>>>> blockCacheSize=658110960, blockCacheFree=197395984,
>>>> > > > >>>>>>>>>>
>>>> > > > >>>>>>>>> blockCacheCount=9903,
>>>> > > > >>>>>>>>>
>>>> > > > >>>>>>>>>> blockCacheHitRatio=99
>>>> > > > >>>>>>>>>>
>>>> > > > >>>>>>>>>> Its serving a few reads? The number of store files
>>>> seems
>>>> > > fine.
>>>> > > > >>>>>>>>>>
>>>> > > > >>>>>>>>> Not
>>>> > > > >>>>>>
>>>> > > > >>>>>>> too
>>>> > > > >>>>>>>>
>>>> > > > >>>>>>>>> much memory used.
>>>> > > > >>>>>>>>>>
>>>> > > > >>>>>>>>>> Looking at the logs, I see the Lease Still Held
>>>> exception.
>>>> > > > >>>>>>>>>>
>>>> > > > >>>>>>>>> This
>>>> > > > >>>
>>>> > > > >>>> happens
>>>> > > > >>>>>>>>
>>>> > > > >>>>>>>>> when the RS does its regular report to the master but the
>>>> > > > >>>>>>>>>>
>>>> > > > >>>>>>>>> master
>>>> > > > >>>
>>>> > > > >>>> thinks
>>>> > > > >>>>>>>
>>>> > > > >>>>>>>> the
>>>> > > > >>>>>>>>>
>>>> > > > >>>>>>>>>> RS has since restarted. It'll think this probably
>>>> because
>>>> > it
>>>> > > > >>>>>>>>>>
>>>> > > > >>>>>>>>> noticed
>>>> > > > >>>>>>
>>>> > > > >>>>>>> that
>>>> > > > >>>>>>>>>
>>>> > > > >>>>>>>>>> the RS's znode in zk had gone away and it considered the
>>>> RS
>>>> > > > >>>>>>>>>>
>>>> > > > >>>>>>>>> dead.
>>>> > > > >>>
>>>> > > > >>>>
>>>> > > > >>>>>>>>>> Looking too at your logs I see this gap in the zk
>>>> pinging:
>>>> > > > >>>>>>>>>>
>>>> > > > >>>>>>>>>>
>>>> > > > >>>>>>>>>> 1. 2009-11-12 15:03:39,325 DEBUG
>>>> > > > >>>>>>>>>>
>>>> > > > >>>>>>>>> org.apache.zookeeper.ClientCnxn:
>>>> > > > >>>>>>
>>>> > > > >>>>>>> Got
>>>> > > > >>>>>>>
>>>> > > > >>>>>>>> ping response for sessionid:0x224e55436ad0004 after 0ms
>>>> > > > >>>>>>>>>> 2. 2009-11-12 15:03:43,113 DEBUG
>>>> > > > >>>>>>>>>>
>>>> > > > >>>>>>>>> org.apache.zookeeper.ClientCnxn:
>>>> > > > >>>>>>
>>>> > > > >>>>>>> Got
>>>> > > > >>>>>>>
>>>> > > > >>>>>>>> ping response for sessionid:0x24e55436a0007d after 0ms
>>>> > > > >>>>>>>>>>
>>>> > > > >>>>>>>>>> Where in the lines above it, its reporting about every
>>>> ten
>>>> > > > >>>>>>>>>>
>>>> > > > >>>>>>>>> seconds,
>>>> > > > >>>>>>
>>>> > > > >>>>>>> here
>>>> > > > >>>>>>>>
>>>> > > > >>>>>>>>> there is a big gap.
>>>> > > > >>>>>>>>>>
>>>> > > > >>>>>>>>>> Do you have ganglia or something that will let you look
>>>> more
>>>> > > > >>>>>>>>>>
>>>> > > > >>>>>>>>> into
>>>> > > > >>>
>>>> > > > >>>> what
>>>> > > > >>>>>>>
>>>> > > > >>>>>>>> was
>>>> > > > >>>>>>>>>
>>>> > > > >>>>>>>>>> happening on this machine around this time? Is the
>>>> machine
>>>> > > OK?
>>>> > > > >>>>>>>>>>
>>>> > > > >>>>>>>>> It
>>>> > > > >>>>>>
>>>> > > > >>>>>>> looks
>>>> > > > >>>>>>>>
>>>> > > > >>>>>>>>> lightly loaded and you have your cluster nicely laid out.
>>>> > > > >>>>>>>>>>
>>>> > > > >>>>>>>>> Something
>>>> > > > >>>>>>
>>>> > > > >>>>>>> odd
>>>> > > > >>>>>>>>
>>>> > > > >>>>>>>>> is
>>>> > > > >>>>>>>>>
>>>> > > > >>>>>>>>>> going on. What about things like the write speed to
>>>> disk?
>>>> > In
>>>> > > > >>>>>>>>>>
>>>> > > > >>>>>>>>> the
>>>> > > > >>>
>>>> > > > >>>> past
>>>> > > > >>>>>>>
>>>> > > > >>>>>>>> strange issues have been explained by incorrectly set BIOS
>>>> > > > >>>>>>>>>>
>>>> > > > >>>>>>>>> which
>>>> > > > >>>
>>>> > > > >>>> made
>>>> > > > >>>>>>
>>>> > > > >>>>>>> disks
>>>> > > > >>>>>>>>>
>>>> > > > >>>>>>>>>> run at 1/100th of their proper speed.
>>>> > > > >>>>>>>>>>
>>>> > > > >>>>>>>>>> St.Ack
>>>> > > > >>>>>>>>>>
>>>> > > > >>>>>>>>>>
>>>> > > > >>>>>>>>>>
>>>> > > > >>>>>>>>>>
>>>> > > > >>>>>>>>>>
>>>> > > > >>>>>>>>>>
>>>> > > > >>>>>>>>>>
>>>> > > > >>>>>>>>>>
>>>> > > > >>>>>>>>>>> Best,
>>>> > > > >>>>>>>>>>> zhenyu
>>>> > > > >>>>>>>>>>>
>>>> > > > >>>>>>>>>>>
>>>> > > > >>>>>>>>>>> On Wed, Nov 11, 2009 at 3:58 PM, Zhenyu Zhong <
>>>> > > > >>>>>>>>>>>
>>>> > > > >>>>>>>>>> zhongresearch@gmail.com
>>>> > > > >>>>>>>>
>>>> > > > >>>>>>>>> wrote:
>>>> > > > >>>>>>>>>>>>
>>>> > > > >>>>>>>>>>>
>>>> > > > >>>>>>>>>>> Stack
>>>> > > > >>>>>>>>>>>>
>>>> > > > >>>>>>>>>>>> I am very appreciated for your comments.
>>>> > > > >>>>>>>>>>>> I will use the zookeeper monitoring script on my
>>>> cluster
>>>> > > > >>>>>>>>>>>>
>>>> > > > >>>>>>>>>>> and
>>>> > > > >>>
>>>> > > > >>>> let
>>>> > > > >>>>>>
>>>> > > > >>>>>>> it
>>>> > > > >>>>>>>
>>>> > > > >>>>>>>> run
>>>> > > > >>>>>>>>>
>>>> > > > >>>>>>>>>> overnight to see the result.
>>>> > > > >>>>>>>>>>>> I will follow up the thread when I get anything.
>>>> > > > >>>>>>>>>>>>
>>>> > > > >>>>>>>>>>>> thanks
>>>> > > > >>>>>>>>>>>> zhenyu
>>>> > > > >>>>>>>>>>>>
>>>> > > > >>>>>>>>>>>>
>>>> > > > >>>>>>>>>>>> On Wed, Nov 11, 2009 at 3:52 PM, stack <
>>>> stack@duboce.net>
>>>> > > > >>>>>>>>>>>>
>>>> > > > >>>>>>>>>>> wrote:
>>>> > > > >>>>>>
>>>> > > > >>>>>>>
>>>> > > > >>>>>>>>>>>> I see these in your log too:
>>>> > > > >>>>>>>>>>>>>
>>>> > > > >>>>>>>>>>>>>
>>>> > > > >>>>>>>>>>>>> 1. 2009-11-11 04:27:19,018 DEBUG
>>>> > > > >>>>>>>>>>>>>
>>>> > > > >>>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
>>>> > > > >>>>>>>>
>>>> > > > >>>>>>>>> Got
>>>> > > > >>>>>>>>>>
>>>> > > > >>>>>>>>>>> ping response for sessionid:0x424dfd908c50009 after
>>>> > > > >>>>>>>>>>>>>
>>>> > > > >>>>>>>>>>>> 4544ms
>>>> > > > >>>
>>>> > > > >>>> 2. 2009-11-11 04:27:19,018 DEBUG
>>>> > > > >>>>>>>>>>>>>
>>>> > > > >>>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
>>>> > > > >>>>>>>>
>>>> > > > >>>>>>>>> Got
>>>> > > > >>>>>>>>>>
>>>> > > > >>>>>>>>>>> ping response for sessionid:0x24dfd90c810002 after
>>>> > > > >>>>>>>>>>>>>
>>>> > > > >>>>>>>>>>>> 4548ms
>>>> > > > >>>
>>>> > > > >>>> 3. 2009-11-11 04:27:43,960 DEBUG
>>>> > > > >>>>>>>>>>>>>
>>>> > > > >>>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
>>>> > > > >>>>>>>>
>>>> > > > >>>>>>>>> Got
>>>> > > > >>>>>>>>>>
>>>> > > > >>>>>>>>>>> ping response for sessionid:0x424dfd908c50009 after
>>>> > > > >>>>>>>>>>>>>
>>>> > > > >>>>>>>>>>>> 9030ms
>>>> > > > >>>
>>>> > > > >>>> 4. 2009-11-11 04:27:43,960 DEBUG
>>>> > > > >>>>>>>>>>>>>
>>>> > > > >>>>>>>>>>>> org.apache.zookeeper.
>>>> > > > >>>>>>>>
>>>> > > > >>>>>>>
>>>> > > >
>>>> > >
>>>> >
>>>>
>>>
>>>
>>
>
Re: regionserver disconnection
Posted by Zhenyu Zhong <zh...@gmail.com>.
I found this.
http://issues.apache.org/jira/browse/ZOOKEEPER-86
It looks like the same error I had. Is it a zookeeper bug? When will HBase
take the zookeeper version 3.3.0?
thanks
zhenyu
On Sat, Nov 14, 2009 at 11:31 AM, Zhenyu Zhong <zh...@gmail.com>wrote:
> Now I really doubt about the zookeeper, from the log I saw errors like
> IOException Read Error, while the zookeeper client (regionserver ) tried to
> read. But it got disconneted status from zookeeper.
>
> I don't see any load on any zookeeper quorum servers. DiskIO is fine.
>
> Also when the regionserver decides to exit due to the disconnect status
> from zookeeper, sometimes the regionserver hangs during the exiting. We can
> still see the HRegionServer process even we don't see it in the Master web
> interface.
>
> I also notice that there is an zookeeper.retries and zookeeper.pauses
> settings in hbase-default.xml, would that matter ?
>
> thanks
>
> You may see the IOException error in the following log when the
> Regionserver lost connection to the zookeeper.
>
> 2009-11-14 15:58:31,993 INFO org.apache.hadoop.hbase.regionserver.HLog:
> Roll
> /hbase/.logs/superpyxis0008.scur.colo,60021,1258179101617/hlog.dat.1258179106921,
> entries=1062964, calcsize=255014936, filesize=139566093. New hlog
> /hbase/.logs/superpyxis0008.scur.colo,60021,1258179101617/hlog.dat.1258214311993
> 2009-11-14 15:58:39,295 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
> response for sessionid:0x424f0ed01ea00a6 after 0ms
> 2009-11-14 15:58:52,648 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
> response for sessionid:0x424f0ed01ea00a6 after 0ms
> 2009-11-14 15:59:06,007 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
> response for sessionid:0x424f0ed01ea00a6 after 0ms
> 2009-11-14 15:59:19,365 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
> response for sessionid:0x424f0ed01ea00a6 after 0ms
> 2009-11-14 15:59:32,867 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
> response for sessionid:0x424f0ed01ea00a6 after 186ms
> 2009-11-14 15:59:46,070 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
> response for sessionid:0x424f0ed01ea00a6 after 0ms
> 2009-11-14 15:59:59,423 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
> response for sessionid:0x424f0ed01ea00a6 after 0ms
> 2009-11-14 16:00:12,775 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
> response for sessionid:0x424f0ed01ea00a6 after 0ms
> 2009-11-14 16:00:28,141 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
> response for sessionid:0x424f0ed01ea00a6 after 2010ms
> 2009-11-14 16:01:10,378 WARN org.apache.zookeeper.ClientCnxn: Exception
> closing session 0x424f0ed01ea00a6 to sun.nio.ch.SelectionKeyImpl@39fbb2d6
>
> java.io.IOException: Read error rc = -1 java.nio.DirectByteBuffer[pos=0
> lim=4 cap=4]
> at
> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
> 2009-11-14 16:01:10,478 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
> state: Disconnected, type: None, path: null
> 2009-11-14 16:01:11,333 WARN
> org.apache.hadoop.hbase.regionserver.HRegionServer: Attempt=1
>
> org.apache.hadoop.hbase.Leases$LeaseStillHeldException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
> at
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
> at
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:571)
> at java.lang.Thread.run(Thread.java:619)
> 2009-11-14 16:01:11,433 INFO org.apache.zookeeper.ClientCnxn: Attempting
> connection to server superpyxis0001/192.168.100.115:2181
> 2009-11-14 16:01:11,433 INFO org.apache.zookeeper.ClientCnxn: Priming
> connection to java.nio.channels.SocketChannel[connected local=/
> 192.168.100.122:59575 remote=superpyxis0001/192.168.100.115:2181]
> 2009-11-14 16:01:11,433 INFO org.apache.zookeeper.ClientCnxn: Server
> connection successful
> 2009-11-14 16:01:11,433 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
> state: Expired, type: None, path: null
> 2009-11-14 16:01:11,433 WARN org.apache.zookeeper.ClientCnxn: Exception
> closing session 0x424f0ed01ea00a6 to sun.nio.ch.SelectionKeyImpl@6b93d343
> java.io.IOException: Session Expired
> at
> org.apache.zookeeper.ClientCnxn$SendThread.readConnectResult(ClientCnxn.java:589)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:709)
>
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
> 2009-11-14 16:01:11,433 ERROR
> org.apache.hadoop.hbase.regionserver.HRegionServer: ZooKeeper session
> expired
> 2009-11-14 16:01:13,513 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 3 on 60021, call put([B@73deb204,
> [Lorg.apache.hadoop.hbase.client.Put;@2179600a) from 192.168.100.132:40728:
> error: java.io.IOException: Server not running, aborting
> java.io.IOException: Server not running, aborting
>
>
>
>
>
> On Sat, Nov 14, 2009 at 1:01 AM, Zhenyu Zhong <zh...@gmail.com>wrote:
>
>> Stack,
>>
>> Thanks a lot!
>>
>> I found out that the reason why HBase doesn't take the system file
>> descriptor value I set before. I started the HBase using root instead of the
>> normal hadoop user, while my system configures hadoop with higher file
>> descriptor value but configures root with 1024 default value.
>>
>>
>> Now my system has a clean start. However, it seems that the
>> zookeeper.session.timeout value doesn't take into effect. I still found
>> around 60 seconds pauses from the disconnected regionserver. I really don't
>> know why regionserver only times out after 60 seconds instead of 10 minutes
>> which I set for zookeeper.session.timeout.
>>
>> Is there any other timeout value coming into play before the actual
>> session times out?
>>
>> zhenyu
>>
>>
>>
>> On Fri, Nov 13, 2009 at 7:08 PM, stack <st...@duboce.net> wrote:
>>
>>> Ok. Lack of file descriptors manifests in all kinds of weird ways.
>>> Hopefully thats it. If not, lets keep digging.
>>> St.Ack
>>>
>>> On Fri, Nov 13, 2009 at 3:44 PM, Zhenyu Zhong <zhongresearch@gmail.com
>>> >wrote:
>>>
>>> > Stack,
>>> >
>>> > You are right, the master started with ulimit -n 1024. It doesn't take
>>> the
>>> > system value.
>>> >
>>> > Regarding to the too many open files, it looks like the same as the one
>>> J-D
>>> > put up. But I will get the Master start with higher value first and see
>>> if
>>> > this kind of error goes away.
>>> >
>>> > thanks a lot!
>>> > zhenyu
>>> >
>>> > On Fri, Nov 13, 2009 at 6:02 PM, stack <st...@duboce.net> wrote:
>>> >
>>> > > Does it say
>>> > >
>>> > > ulimit -n 32768
>>> > >
>>> > > ...as the second line in your log file on start of the master?
>>> > >
>>> > > Is the stack trace that complains about too many open files same as
>>> the
>>> > one
>>> > > in the blog post J-D put up?
>>> > >
>>> > > St.Ack
>>> > >
>>> > >
>>> > > On Fri, Nov 13, 2009 at 1:37 PM, Zhenyu Zhong <
>>> zhongresearch@gmail.com
>>> > > >wrote:
>>> > >
>>> > > > The ulimit file descriptors was set to fs.file-max =1578334, also
>>> in
>>> > the
>>> > > > limits.conf the value was set to 32768.
>>> > > > So these are way higher than the open descriptors for the running
>>> > > > processes.
>>> > > >
>>> > > > thanks
>>> > > > zhenyu
>>> > > >
>>> > > >
>>> > > > On Fri, Nov 13, 2009 at 4:33 PM, Stack <sa...@gmail.com>
>>> wrote:
>>> > > >
>>> > > > > You upped the ulimit file descriptors as per the getting started
>>> doc?
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > > On Nov 13, 2009, at 1:26 PM, Zhenyu Zhong <
>>> zhongresearch@gmail.com>
>>> > > > wrote:
>>> > > > >
>>> > > > > Thanks a lot.
>>> > > > >>
>>> > > > >>
>>> > > > >> Bad news is my kernel is still 2.6.26.
>>> > > > >> But it was not a problem before.
>>> > > > >>
>>> > > > >> Very strange.
>>> > > > >>
>>> > > > >> zhenyu
>>> > > > >>
>>> > > > >> On Fri, Nov 13, 2009 at 4:16 PM, Jean-Daniel Cryans <
>>> > > > jdcryans@apache.org
>>> > > > >> >wrote:
>>> > > > >>
>>> > > > >> Looks like
>>> > > > >>>
>>> > > > >>>
>>> > > >
>>> > >
>>> >
>>> http://pero.blogs.aprilmayjune.org/2009/01/22/hadoop-and-linux-kernel-2627-epoll-limits/
>>> > > > >>>
>>> > > > >>> J-D
>>> > > > >>>
>>> > > > >>> On Fri, Nov 13, 2009 at 1:12 PM, Zhenyu Zhong <
>>> > > zhongresearch@gmail.com
>>> > > > >
>>> > > > >>> wrote:
>>> > > > >>>
>>> > > > >>>> Hi,
>>> > > > >>>>
>>> > > > >>>>
>>> > > > >>>> After I re-organize the cluster, the experiment ran into
>>> problem
>>> > > > faster
>>> > > > >>>>
>>> > > > >>> than
>>> > > > >>>
>>> > > > >>>> before.
>>> > > > >>>>
>>> > > > >>>> Basically the changes are to use machines with less resources
>>> as
>>> > > > >>>>
>>> > > > >>> zookeeper
>>> > > > >>>
>>> > > > >>>> quorums and machines with more resources as regionserver.
>>> > > > >>>>
>>> > > > >>>> From the log, I still see the pause around 1 minute.
>>> > > > >>>> I enabled the GC logging, please see
>>> > > > >>>>
>>> > > > >>>> http://pastebin.com/m1d4ce0f1
>>> > > > >>>>
>>> > > > >>>> for details.
>>> > > > >>>> In general I don't see many pauses in the GC.
>>> > > > >>>>
>>> > > > >>>> What is more interesting, 1 hour after the 1st regionserver
>>> > > > >>>> disconnected,
>>> > > > >>>> the master log complained about too many open files. This
>>> didn't
>>> > > > happen
>>> > > > >>>> before.
>>> > > > >>>> I checked the system OS setup as well as the limits.conf. I
>>> also
>>> > > > checked
>>> > > > >>>>
>>> > > > >>> the
>>> > > > >>>
>>> > > > >>>> running processes. The total open files don't reach the limit.
>>> I
>>> > am
>>> > > > >>>>
>>> > > > >>> confused
>>> > > > >>>
>>> > > > >>>> a bit.
>>> > > > >>>>
>>> > > > >>>>
>>> > > > >>>> Please see the following master log.
>>> > > > >>>>
>>> > > > >>>> 2009-11-13 20:06:02,114 INFO
>>> > > > org.apache.hadoop.hbase.master.BaseScanner:
>>> > > > >>>> RegionManager.metaScanner scan of 4658 row(s) of meta region
>>> > > {server:
>>> > > > >>>> 192.168.100.128:60021, regionname: .META.,,1, startKey: <>}
>>> > > complete
>>> > > > >>>> 2009-11-13 20:06:02,114 INFO
>>> > > > org.apache.hadoop.hbase.master.BaseScanner:
>>> > > > >>>>
>>> > > > >>> All
>>> > > > >>>
>>> > > > >>>> 1 .META. region(s) scanned
>>> > > > >>>> 2009-11-13 20:06:07,677 DEBUG org.apache.zookeeper.ClientCnxn:
>>> Got
>>> > > > ping
>>> > > > >>>> response for sessionid:0x424eebf1c10004c after 3ms
>>> > > > >>>> 2009-11-13 20:06:08,178 INFO org.apache.hadoop.hdfs.DFSClient:
>>> > > > Exception
>>> > > > >>>>
>>> > > > >>> in
>>> > > > >>>
>>> > > > >>>> createBlockOutputStream java.io.IOException: Bad connect ack
>>> with
>>> > > > >>>> firstBadLink 192.168.100.123:50010
>>> > > > >>>> 2009-11-13 20:06:08,178 INFO org.apache.hadoop.hdfs.DFSClient:
>>> > > > >>>> Abandoning
>>> > > > >>>> block blk_-2808245019291145247_5478039
>>> > > > >>>> 2009-11-13 20:06:09,682 INFO org.apache.hadoop.hdfs.DFSClient:
>>> > > > Exception
>>> > > > >>>>
>>> > > > >>> in
>>> > > > >>>
>>> > > > >>>> createBlockOutputStream java.io.EOFException
>>> > > > >>>> 2009-11-13 20:06:09,682 INFO org.apache.hadoop.hdfs.DFSClient:
>>> > > > >>>> Abandoning
>>> > > > >>>> block blk_1074853606841896259_5478048
>>> > > > >>>> 2009-11-13 20:06:10,334 DEBUG org.apache.zookeeper.ClientCnxn:
>>> Got
>>> > > > ping
>>> > > > >>>> response for sessionid:0x24eebf1043003c after 1ms
>>> > > > >>>> 2009-11-13 20:06:21,018 DEBUG org.apache.zookeeper.ClientCnxn:
>>> Got
>>> > > > ping
>>> > > > >>>> response for sessionid:0x424eebf1c10004c after 0ms
>>> > > > >>>> 2009-11-13 20:06:23,674 DEBUG org.apache.zookeeper.ClientCnxn:
>>> Got
>>> > > > ping
>>> > > > >>>> response for sessionid:0x24eebf1043003c after 0ms
>>> > > > >>>> 2009-11-13 20:06:24,828 INFO org.apache.hadoop.hdfs.DFSClient:
>>> > > > Exception
>>> > > > >>>>
>>> > > > >>> in
>>> > > > >>>
>>> > > > >>>> createBlockOutputStream java.io.IOException: Bad connect ack
>>> with
>>> > > > >>>> firstBadLink 192.168.100.123:50010
>>> > > > >>>> 2009-11-13 20:06:24,828 INFO org.apache.hadoop.hdfs.DFSClient:
>>> > > > >>>> Abandoning
>>> > > > >>>> block blk_-6642544517082142289_5478063
>>> > > > >>>> 2009-11-13 20:06:24,828 INFO org.apache.hadoop.hdfs.DFSClient:
>>> > > > Exception
>>> > > > >>>>
>>> > > > >>> in
>>> > > > >>>
>>> > > > >>>> createBlockOutputStream java.net.SocketException: Too many
>>> open
>>> > > files
>>> > > > >>>> 2009-11-13 20:06:24,828 INFO org.apache.hadoop.hdfs.DFSClient:
>>> > > > >>>> Abandoning
>>> > > > >>>> block blk_2057511041109796090_5478063
>>> > > > >>>> 2009-11-13 20:06:24,928 INFO org.apache.hadoop.hdfs.DFSClient:
>>> > > > Exception
>>> > > > >>>>
>>> > > > >>> in
>>> > > > >>>
>>> > > > >>>> createBlockOutputStream java.net.SocketException: Too many
>>> open
>>> > > files
>>> > > > >>>> 2009-11-13 20:06:24,928 INFO org.apache.hadoop.hdfs.DFSClient:
>>> > > > >>>> Abandoning
>>> > > > >>>> block blk_8219260302213892894_5478064
>>> > > > >>>> 2009-11-13 20:06:30,855 INFO org.apache.hadoop.hdfs.DFSClient:
>>> > > > Exception
>>> > > > >>>>
>>> > > > >>> in
>>> > > > >>>
>>> > > > >>>> createBlockOutputStream java.net.SocketException: Too many
>>> open
>>> > > files
>>> > > > >>>> 2009-11-13 20:06:30,855 INFO org.apache.hadoop.hdfs.DFSClient:
>>> > > > >>>> Abandoning
>>> > > > >>>> block blk_1669205542853067709_5478235
>>> > > > >>>> 2009-11-13 20:06:30,905 INFO org.apache.hadoop.hdfs.DFSClient:
>>> > > > Exception
>>> > > > >>>>
>>> > > > >>> in
>>> > > > >>>
>>> > > > >>>> createBlockOutputStream java.net.SocketException: Too many
>>> open
>>> > > files
>>> > > > >>>> 2009-11-13 20:06:30,905 INFO org.apache.hadoop.hdfs.DFSClient:
>>> > > > >>>> Abandoning
>>> > > > >>>> block blk_9128897691346270351_5478237
>>> > > > >>>> 2009-11-13 20:06:30,955 INFO org.apache.hadoop.hdfs.DFSClient:
>>> > > > Exception
>>> > > > >>>>
>>> > > > >>> in
>>> > > > >>>
>>> > > > >>>> createBlockOutputStream java.net.SocketException: Too many
>>> open
>>> > > files
>>> > > > >>>> 2009-11-13 20:06:30,955 INFO org.apache.hadoop.hdfs.DFSClient:
>>> > > > >>>> Abandoning
>>> > > > >>>> block blk_1116845144864123018_5478240
>>> > > > >>>> 2009-11-13 20:06:34,372 DEBUG org.apache.zookeeper.ClientCnxn:
>>> Got
>>> > > > ping
>>> > > > >>>> response for sessionid:0x424eebf1c10004c after 0ms
>>> > > > >>>> 2009-11-13 20:06:37,034 DEBUG org.apache.zookeeper.ClientCnxn:
>>> Got
>>> > > > ping
>>> > > > >>>> response for sessionid:0x24eebf1043003c after 0ms
>>> > > > >>>> 2009-11-13 20:06:37,235 WARN org.apache.hadoop.hdfs.DFSClient:
>>> > > > >>>>
>>> > > > >>> DataStreamer
>>> > > > >>>
>>> > > > >>>> Exception: java.io.IOException: Too many open files
>>> > > > >>>> at sun.nio.ch.IOUtil.initPipe(Native Method)
>>> > > > >>>> at
>>> > > sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:49)
>>> > > > >>>> at
>>> > > > >>>>
>>> > > > >>>>
>>> > > >
>>> > >
>>> >
>>> sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:18)
>>> > > > >>>
>>> > > > >>>> at
>>> > > > >>>>
>>> > > > >>>>
>>> > > >
>>> > >
>>> >
>>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.get(SocketIOWithTimeout.java:407)
>>> > > > >>>
>>> > > > >>>> at
>>> > > > >>>>
>>> > > > >>>>
>>> > > >
>>> > >
>>> >
>>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:322)
>>> > > > >>>
>>> > > > >>>> at
>>> > > > >>>>
>>> > > > >>>>
>>> > > >
>>> > >
>>> >
>>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
>>> > > > >>>
>>> > > > >>>> at
>>> > > > >>>>
>>> > > > >>>>
>>> > > >
>>> > >
>>> >
>>> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
>>> > > > >>>
>>> > > > >>>> at
>>> > > > >>>>
>>> > > > >>>>
>>> > > >
>>> > >
>>> >
>>> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
>>> > > > >>>
>>> > > > >>>> at
>>> > > > >>>>
>>> > > > >>>
>>> java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
>>> > > > >>>
>>> > > > >>>> at
>>> java.io.DataOutputStream.write(DataOutputStream.java:90)
>>> > > > >>>> at
>>> > > > >>>>
>>> > > > >>>>
>>> > > >
>>> > >
>>> >
>>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2290)
>>> > > > >>>
>>> > > > >>>>
>>> > > > >>>> 2009-11-13 20:06:37,235 WARN org.apache.hadoop.hdfs.DFSClient:
>>> > Error
>>> > > > >>>> Recovery for block blk_8148813491785406356_5478475 bad
>>> datanode[0]
>>> > > > >>>> 192.168.100.123:50010
>>> > > > >>>> 2009-11-13 20:06:37,235 WARN org.apache.hadoop.hdfs.DFSClient:
>>> > Error
>>> > > > >>>> Recovery for block blk_8148813491785406356_5478475 in pipeline
>>> > > > >>>> 192.168.100.123:50010, 192.168.100.134:50010,
>>> > 192.168.100.122:50010
>>> > > :
>>> > > > >>>> bad
>>> > > > >>>> datanode 192.168.100.123:50010
>>> > > > >>>> 2009-11-13 20:06:37,436 INFO org.apache.hadoop.hdfs.DFSClient:
>>> > > > Exception
>>> > > > >>>>
>>> > > > >>> in
>>> > > > >>>
>>> > > > >>>> createBlockOutputStream java.net.SocketException: Too many
>>> open
>>> > > files
>>> > > > >>>> 2009-11-13 20:06:37,436 INFO org.apache.hadoop.hdfs.DFSClient:
>>> > > > >>>> Abandoning
>>> > > > >>>> block blk_2119727700857186236_5478498
>>> > > > >>>>
>>> > > > >>>>
>>> > > > >>>>
>>> > > > >>>>
>>> > > > >>>>
>>> > > > >>>> On Thu, Nov 12, 2009 at 4:21 PM, Zhenyu Zhong <
>>> > > > zhongresearch@gmail.com
>>> > > > >>>> wrote:
>>> > > > >>>>
>>> > > > >>>> Will do.
>>> > > > >>>>>
>>> > > > >>>>> thanks
>>> > > > >>>>> zhenyu
>>> > > > >>>>>
>>> > > > >>>>>
>>> > > > >>>>> On Thu, Nov 12, 2009 at 3:33 PM, stack <st...@duboce.net>
>>> wrote:
>>> > > > >>>>>
>>> > > > >>>>> Enable GC logging too on this next run (see hbase-env.sh).
>>> Lets
>>> > > try
>>> > > > >>>>>>
>>> > > > >>>>> and
>>> > > > >>>
>>> > > > >>>> get
>>> > > > >>>>>> to the bottom of whats going on.
>>> > > > >>>>>> Thanks,
>>> > > > >>>>>> St.Ack
>>> > > > >>>>>>
>>> > > > >>>>>> On Thu, Nov 12, 2009 at 12:29 PM, Zhenyu Zhong <
>>> > > > >>>>>>
>>> > > > >>>>> zhongresearch@gmail.com
>>> > > > >>>
>>> > > > >>>> wrote:
>>> > > > >>>>>>>
>>> > > > >>>>>>
>>> > > > >>>>>> I can switch the boxes that run zookeeper with the ones
>>> that
>>> > run
>>> > > > >>>>>>> regionservers.
>>> > > > >>>>>>> I will see the results later.
>>> > > > >>>>>>>
>>> > > > >>>>>>> FYI. The node does have the 10 minutes
>>> > zookeeper.session.timeout
>>> > > > >>>>>>>
>>> > > > >>>>>> value
>>> > > > >>>
>>> > > > >>>> in
>>> > > > >>>>>>
>>> > > > >>>>>>> place.
>>> > > > >>>>>>>
>>> > > > >>>>>>> thanks
>>> > > > >>>>>>> zhenyu
>>> > > > >>>>>>>
>>> > > > >>>>>>> On Thu, Nov 12, 2009 at 3:21 PM, stack <st...@duboce.net>
>>> > wrote:
>>> > > > >>>>>>>
>>> > > > >>>>>>> On Thu, Nov 12, 2009 at 11:50 AM, Zhenyu Zhong <
>>> > > > >>>>>>>>
>>> > > > >>>>>>> zhongresearch@gmail.com
>>> > > > >>>>>>
>>> > > > >>>>>>> wrote:
>>> > > > >>>>>>>>>
>>> > > > >>>>>>>>
>>> > > > >>>>>>>>
>>> > > > >>>>>>>>> In my cluster, half of the cluster have 2 disks 400GB
>>> each
>>> > per
>>> > > > >>>>>>>>>
>>> > > > >>>>>>>> machine,
>>> > > > >>>>>>
>>> > > > >>>>>>> and
>>> > > > >>>>>>>>
>>> > > > >>>>>>>>> half of the cluster have 6 disks per machine. Maybe we
>>> > should
>>> > > > >>>>>>>>>
>>> > > > >>>>>>>> run
>>> > > > >>>
>>> > > > >>>> zookeeper
>>> > > > >>>>>>>>> on the machines with 2 disks and RS on machines with 6
>>> disks?
>>> > > > >>>>>>>>>
>>> > > > >>>>>>>>> That would make most sense only in the below, it looks
>>> like
>>> > the
>>> > > > >>>>>>>>>
>>> > > > >>>>>>>> RS
>>> > > > >>>
>>> > > > >>>> that
>>> > > > >>>>>>
>>> > > > >>>>>>> had
>>> > > > >>>>>>>> issue had 4 disks?
>>> > > > >>>>>>>>
>>> > > > >>>>>>>>
>>> > > > >>>>>>>>
>>> > > > >>>>>>>> BTW, the 10 minutes zookeeper.session.timeout has been
>>> set
>>> > > during
>>> > > > >>>>>>>>>
>>> > > > >>>>>>>> the
>>> > > > >>>>>>
>>> > > > >>>>>>> experiment.
>>> > > > >>>>>>>>>
>>> > > > >>>>>>>>>
>>> > > > >>>>>>>> And for sure this node had it in place?
>>> > > > >>>>>>>> St.Ack
>>> > > > >>>>>>>>
>>> > > > >>>>>>>>
>>> > > > >>>>>>>>
>>> > > > >>>>>>>>
>>> > > > >>>>>>>>> thanks
>>> > > > >>>>>>>>> zhenyu
>>> > > > >>>>>>>>>
>>> > > > >>>>>>>>> On Thu, Nov 12, 2009 at 2:08 PM, stack <stack@duboce.net
>>> >
>>> > > wrote:
>>> > > > >>>>>>>>>
>>> > > > >>>>>>>>> On Thu, Nov 12, 2009 at 8:40 AM, Zhenyu Zhong <
>>> > > > >>>>>>>>>>
>>> > > > >>>>>>>>> zhongresearch@gmail.com
>>> > > > >>>>>>>
>>> > > > >>>>>>>> wrote:
>>> > > > >>>>>>>>>>>
>>> > > > >>>>>>>>>>
>>> > > > >>>>>>>>>>
>>> > > > >>>>>>>>>> Though I experienced 2 regionserver disconnection this
>>> > > > >>>>>>>>>>>
>>> > > > >>>>>>>>>> morning,
>>> > > > >>>
>>> > > > >>>> it
>>> > > > >>>>>>
>>> > > > >>>>>>> looks
>>> > > > >>>>>>>>>
>>> > > > >>>>>>>>>> better from the regionserver log. (please see the
>>> following
>>> > > > >>>>>>>>>>>
>>> > > > >>>>>>>>>> log)
>>> > > > >>>
>>> > > > >>>>
>>> > > > >>>>>>>>>>>
>>> > > > >>>>>>>>>>> http://pastebin.com/m496dbfae
>>> > > > >>>>>>>>>>>
>>> > > > >>>>>>>>>>>
>>> > > > >>>>>>>>>>> I checked diskIO for one of the
>>> > regionserver(192.168.100.116)
>>> > > > >>>>>>>>>>>
>>> > > > >>>>>>>>>> during
>>> > > > >>>>>>>
>>> > > > >>>>>>>> the
>>> > > > >>>>>>>>>
>>> > > > >>>>>>>>>> time it disconnected.
>>> > > > >>>>>>>>>>>
>>> > > > >>>>>>>>>>> Time: 03:04:58 AM
>>> > > > >>>>>>>>>>> Device: tps Blk_read/s Blk_wrtn/s
>>> Blk_read
>>> > > > >>>>>>>>>>>
>>> > > > >>>>>>>>>> Blk_wrtn
>>> > > > >>>>>>>>
>>> > > > >>>>>>>>> sda 105.31 5458.83 19503.64
>>> 9043873239
>>> > > > >>>>>>>>>>>
>>> > > > >>>>>>>>>> 32312473676
>>> > > > >>>>>>>>
>>> > > > >>>>>>>>> sda1 2.90 6.64 99.25
>>> 10993934
>>> > > > >>>>>>>>>>>
>>> > > > >>>>>>>>>> 164433464
>>> > > > >>>>>>>>
>>> > > > >>>>>>>>> sda2 1.72 23.76 30.25
>>> 39365817
>>> > > > >>>>>>>>>>>
>>> > > > >>>>>>>>>> 50124008
>>> > > > >>>>>>>>
>>> > > > >>>>>>>>> sda3 0.30 0.38 3.58
>>> 624930
>>> > > > >>>>>>>>>>>
>>> > > > >>>>>>>>>> 5923000
>>> > > > >>>>>>>>
>>> > > > >>>>>>>>> sda4 100.39 5428.06 19370.56
>>> 8992888270
>>> > > > >>>>>>>>>>>
>>> > > > >>>>>>>>>> 32091993204
>>> > > > >>>>>>>>
>>> > > > >>>>>>>>>
>>> > > > >>>>>>>>>>>
>>> > > > >>>>>>>>>>> Is this high for you? 20k blocks/second would seem to
>>> be
>>> > > high
>>> > > > >>>>>>>>>>
>>> > > > >>>>>>>>> but
>>> > > > >>>
>>> > > > >>>> its
>>> > > > >>>>>>>
>>> > > > >>>>>>>> one
>>> > > > >>>>>>>>>
>>> > > > >>>>>>>>>> disk only and its not being shared by zk anymore so
>>> > shouldn't
>>> > > > >>>>>>>>>>
>>> > > > >>>>>>>>> matter?
>>> > > > >>>>>>
>>> > > > >>>>>>>
>>> > > > >>>>>>>>>>
>>> > > > >>>>>>>>>> I also checked the zookeeper quorum server that the
>>> > > > >>>>>>>>>>>
>>> > > > >>>>>>>>>> regionserver
>>> > > > >>>
>>> > > > >>>> tried
>>> > > > >>>>>>>>
>>> > > > >>>>>>>>> to
>>> > > > >>>>>>>>>
>>> > > > >>>>>>>>>> connect according to the log. However, I don't see
>>> > > > >>>>>>>>>>>
>>> > > > >>>>>>>>>> 192.168.100.116
>>> > > > >>>>>>
>>> > > > >>>>>>> in
>>> > > > >>>>>>>
>>> > > > >>>>>>>> the
>>> > > > >>>>>>>>>
>>> > > > >>>>>>>>>> client list of the zookeeper quorum server that the
>>> > > > >>>>>>>>>>>
>>> > > > >>>>>>>>>> regionserver
>>> > > > >>>
>>> > > > >>>> tried
>>> > > > >>>>>>>>
>>> > > > >>>>>>>>> to
>>> > > > >>>>>>>>>
>>> > > > >>>>>>>>>> connect.
>>> > > > >>>>>>>>>>> Would that be a problem?
>>> > > > >>>>>>>>>>>
>>> > > > >>>>>>>>>>>
>>> > > > >>>>>>>>>>> Is that because the ephemeral node for the
>>> regionserver
>>> > had
>>> > > > >>>>>>>>>>
>>> > > > >>>>>>>>> evaporated?
>>> > > > >>>>>>>
>>> > > > >>>>>>>> Lost
>>> > > > >>>>>>>>>> its lease w/ zk by the time you went to look?
>>> > > > >>>>>>>>>>
>>> > > > >>>>>>>>>>
>>> > > > >>>>>>>>>>
>>> > > > >>>>>>>>>>
>>> > > > >>>>>>>>>> Thu Nov 12 15:04:52 UTC 2009
>>> > > > >>>>>>>>>>> Zookeeper version: 3.2.1-808558, built on 08/27/2009
>>> 18:48
>>> > > > >>>>>>>>>>>
>>> > > > >>>>>>>>>> GMT
>>> > > > >>>
>>> > > > >>>> Clients:
>>> > > > >>>>>>>>>>> /192.168.100.127:43045[1](queued=0,recved=26,sent=0)
>>> > > > >>>>>>>>>>> /192.168.100.131:39091[1](queued=0,recved=964,sent=0)
>>> > > > >>>>>>>>>>> /192.168.100.124:35961[1](queued=0,recved=3266,sent=0)
>>> > > > >>>>>>>>>>> /192.168.100.123:47935[1](queued=0,recved=233,sent=0)
>>> > > > >>>>>>>>>>> /192.168.100.125:46931[1](queued=0,recved=2,sent=0)
>>> > > > >>>>>>>>>>> /192.168.100.118:54924[1](queued=0,recved=295,sent=0)
>>> > > > >>>>>>>>>>> /192.168.100.118:41390[1](queued=0,recved=2290,sent=0)
>>> > > > >>>>>>>>>>> /192.168.100.136:42243[1](queued=0,recved=0,sent=0)
>>> > > > >>>>>>>>>>>
>>> > > > >>>>>>>>>>> Latency min/avg/max: 0/17/6333
>>> > > > >>>>>>>>>>> Received: 47111
>>> > > > >>>>>>>>>>> Sent: 0
>>> > > > >>>>>>>>>>> Outstanding: 0
>>> > > > >>>>>>>>>>> Zxid: 0x77000083f4
>>> > > > >>>>>>>>>>> Mode: leader
>>> > > > >>>>>>>>>>> Node count: 23
>>> > > > >>>>>>>>>>>
>>> > > > >>>>>>>>>>>
>>> > > > >>>>>>>>>>
>>> > > > >>>>>>>>>> That 6 second maximum latency is pretty bad but should
>>> be
>>> > well
>>> > > > >>>>>>>>>>
>>> > > > >>>>>>>>> within
>>> > > > >>>>>>
>>> > > > >>>>>>> the
>>> > > > >>>>>>>>
>>> > > > >>>>>>>>> zk
>>> > > > >>>>>>>>>> session timeout.
>>> > > > >>>>>>>>>>
>>> > > > >>>>>>>>>> So, problem is likely on the zk client side of the
>>> session;
>>> > > > >>>>>>>>>>
>>> > > > >>>>>>>>> i.e.
>>> > > > >>>
>>> > > > >>>> in
>>> > > > >>>>>>
>>> > > > >>>>>>> the
>>> > > > >>>>>>>
>>> > > > >>>>>>>> RS.
>>> > > > >>>>>>>>>
>>> > > > >>>>>>>>>> You could enable GC logging as J-D suggested to see if
>>> you
>>> > > have
>>> > > > >>>>>>>>>>
>>> > > > >>>>>>>>> any
>>> > > > >>>>>>
>>> > > > >>>>>>> big
>>> > > > >>>>>>>
>>> > > > >>>>>>>> pauses, pauses > zk session timeout.
>>> > > > >>>>>>>>>>
>>> > > > >>>>>>>>>> When the RS went down, it didn't look too heavily
>>> loaded:
>>> > > > >>>>>>>>>>
>>> > > > >>>>>>>>>>
>>> > > > >>>>>>>>>> 1. 2009-11-12 15:04:52,830 INFO
>>> > > > >>>>>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>>> Dump of
>>> > > > >>>>>>>>>>
>>> > > > >>>>>>>>> metrics:
>>> > > > >>>>>>>
>>> > > > >>>>>>>> request=1.5166667, regions=322, stores=657,
>>> storefiles=631,
>>> > > > >>>>>>>>>> storefileIndexSize=61, memstoreSize=1472,
>>> usedHeap=2819,
>>> > > > >>>>>>>>>>
>>> > > > >>>>>>>>> maxHeap=4079,
>>> > > > >>>>>>>>
>>> > > > >>>>>>>>> blockCacheSize=658110960, blockCacheFree=197395984,
>>> > > > >>>>>>>>>>
>>> > > > >>>>>>>>> blockCacheCount=9903,
>>> > > > >>>>>>>>>
>>> > > > >>>>>>>>>> blockCacheHitRatio=99
>>> > > > >>>>>>>>>>
>>> > > > >>>>>>>>>> Its serving a few reads? The number of store files
>>> seems
>>> > > fine.
>>> > > > >>>>>>>>>>
>>> > > > >>>>>>>>> Not
>>> > > > >>>>>>
>>> > > > >>>>>>> too
>>> > > > >>>>>>>>
>>> > > > >>>>>>>>> much memory used.
>>> > > > >>>>>>>>>>
>>> > > > >>>>>>>>>> Looking at the logs, I see the Lease Still Held
>>> exception.
>>> > > > >>>>>>>>>>
>>> > > > >>>>>>>>> This
>>> > > > >>>
>>> > > > >>>> happens
>>> > > > >>>>>>>>
>>> > > > >>>>>>>>> when the RS does its regular report to the master but the
>>> > > > >>>>>>>>>>
>>> > > > >>>>>>>>> master
>>> > > > >>>
>>> > > > >>>> thinks
>>> > > > >>>>>>>
>>> > > > >>>>>>>> the
>>> > > > >>>>>>>>>
>>> > > > >>>>>>>>>> RS has since restarted. It'll think this probably
>>> because
>>> > it
>>> > > > >>>>>>>>>>
>>> > > > >>>>>>>>> noticed
>>> > > > >>>>>>
>>> > > > >>>>>>> that
>>> > > > >>>>>>>>>
>>> > > > >>>>>>>>>> the RS's znode in zk had gone away and it considered the
>>> RS
>>> > > > >>>>>>>>>>
>>> > > > >>>>>>>>> dead.
>>> > > > >>>
>>> > > > >>>>
>>> > > > >>>>>>>>>> Looking too at your logs I see this gap in the zk
>>> pinging:
>>> > > > >>>>>>>>>>
>>> > > > >>>>>>>>>>
>>> > > > >>>>>>>>>> 1. 2009-11-12 15:03:39,325 DEBUG
>>> > > > >>>>>>>>>>
>>> > > > >>>>>>>>> org.apache.zookeeper.ClientCnxn:
>>> > > > >>>>>>
>>> > > > >>>>>>> Got
>>> > > > >>>>>>>
>>> > > > >>>>>>>> ping response for sessionid:0x224e55436ad0004 after 0ms
>>> > > > >>>>>>>>>> 2. 2009-11-12 15:03:43,113 DEBUG
>>> > > > >>>>>>>>>>
>>> > > > >>>>>>>>> org.apache.zookeeper.ClientCnxn:
>>> > > > >>>>>>
>>> > > > >>>>>>> Got
>>> > > > >>>>>>>
>>> > > > >>>>>>>> ping response for sessionid:0x24e55436a0007d after 0ms
>>> > > > >>>>>>>>>>
>>> > > > >>>>>>>>>> Where in the lines above it, its reporting about every
>>> ten
>>> > > > >>>>>>>>>>
>>> > > > >>>>>>>>> seconds,
>>> > > > >>>>>>
>>> > > > >>>>>>> here
>>> > > > >>>>>>>>
>>> > > > >>>>>>>>> there is a big gap.
>>> > > > >>>>>>>>>>
>>> > > > >>>>>>>>>> Do you have ganglia or something that will let you look
>>> more
>>> > > > >>>>>>>>>>
>>> > > > >>>>>>>>> into
>>> > > > >>>
>>> > > > >>>> what
>>> > > > >>>>>>>
>>> > > > >>>>>>>> was
>>> > > > >>>>>>>>>
>>> > > > >>>>>>>>>> happening on this machine around this time? Is the
>>> machine
>>> > > OK?
>>> > > > >>>>>>>>>>
>>> > > > >>>>>>>>> It
>>> > > > >>>>>>
>>> > > > >>>>>>> looks
>>> > > > >>>>>>>>
>>> > > > >>>>>>>>> lightly loaded and you have your cluster nicely laid out.
>>> > > > >>>>>>>>>>
>>> > > > >>>>>>>>> Something
>>> > > > >>>>>>
>>> > > > >>>>>>> odd
>>> > > > >>>>>>>>
>>> > > > >>>>>>>>> is
>>> > > > >>>>>>>>>
>>> > > > >>>>>>>>>> going on. What about things like the write speed to
>>> disk?
>>> > In
>>> > > > >>>>>>>>>>
>>> > > > >>>>>>>>> the
>>> > > > >>>
>>> > > > >>>> past
>>> > > > >>>>>>>
>>> > > > >>>>>>>> strange issues have been explained by incorrectly set BIOS
>>> > > > >>>>>>>>>>
>>> > > > >>>>>>>>> which
>>> > > > >>>
>>> > > > >>>> made
>>> > > > >>>>>>
>>> > > > >>>>>>> disks
>>> > > > >>>>>>>>>
>>> > > > >>>>>>>>>> run at 1/100th of their proper speed.
>>> > > > >>>>>>>>>>
>>> > > > >>>>>>>>>> St.Ack
>>> > > > >>>>>>>>>>
>>> > > > >>>>>>>>>>
>>> > > > >>>>>>>>>>
>>> > > > >>>>>>>>>>
>>> > > > >>>>>>>>>>
>>> > > > >>>>>>>>>>
>>> > > > >>>>>>>>>>
>>> > > > >>>>>>>>>>
>>> > > > >>>>>>>>>>> Best,
>>> > > > >>>>>>>>>>> zhenyu
>>> > > > >>>>>>>>>>>
>>> > > > >>>>>>>>>>>
>>> > > > >>>>>>>>>>> On Wed, Nov 11, 2009 at 3:58 PM, Zhenyu Zhong <
>>> > > > >>>>>>>>>>>
>>> > > > >>>>>>>>>> zhongresearch@gmail.com
>>> > > > >>>>>>>>
>>> > > > >>>>>>>>> wrote:
>>> > > > >>>>>>>>>>>>
>>> > > > >>>>>>>>>>>
>>> > > > >>>>>>>>>>> Stack
>>> > > > >>>>>>>>>>>>
>>> > > > >>>>>>>>>>>> I am very appreciated for your comments.
>>> > > > >>>>>>>>>>>> I will use the zookeeper monitoring script on my
>>> cluster
>>> > > > >>>>>>>>>>>>
>>> > > > >>>>>>>>>>> and
>>> > > > >>>
>>> > > > >>>> let
>>> > > > >>>>>>
>>> > > > >>>>>>> it
>>> > > > >>>>>>>
>>> > > > >>>>>>>> run
>>> > > > >>>>>>>>>
>>> > > > >>>>>>>>>> overnight to see the result.
>>> > > > >>>>>>>>>>>> I will follow up the thread when I get anything.
>>> > > > >>>>>>>>>>>>
>>> > > > >>>>>>>>>>>> thanks
>>> > > > >>>>>>>>>>>> zhenyu
>>> > > > >>>>>>>>>>>>
>>> > > > >>>>>>>>>>>>
>>> > > > >>>>>>>>>>>> On Wed, Nov 11, 2009 at 3:52 PM, stack <
>>> stack@duboce.net>
>>> > > > >>>>>>>>>>>>
>>> > > > >>>>>>>>>>> wrote:
>>> > > > >>>>>>
>>> > > > >>>>>>>
>>> > > > >>>>>>>>>>>> I see these in your log too:
>>> > > > >>>>>>>>>>>>>
>>> > > > >>>>>>>>>>>>>
>>> > > > >>>>>>>>>>>>> 1. 2009-11-11 04:27:19,018 DEBUG
>>> > > > >>>>>>>>>>>>>
>>> > > > >>>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
>>> > > > >>>>>>>>
>>> > > > >>>>>>>>> Got
>>> > > > >>>>>>>>>>
>>> > > > >>>>>>>>>>> ping response for sessionid:0x424dfd908c50009 after
>>> > > > >>>>>>>>>>>>>
>>> > > > >>>>>>>>>>>> 4544ms
>>> > > > >>>
>>> > > > >>>> 2. 2009-11-11 04:27:19,018 DEBUG
>>> > > > >>>>>>>>>>>>>
>>> > > > >>>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
>>> > > > >>>>>>>>
>>> > > > >>>>>>>>> Got
>>> > > > >>>>>>>>>>
>>> > > > >>>>>>>>>>> ping response for sessionid:0x24dfd90c810002 after
>>> > > > >>>>>>>>>>>>>
>>> > > > >>>>>>>>>>>> 4548ms
>>> > > > >>>
>>> > > > >>>> 3. 2009-11-11 04:27:43,960 DEBUG
>>> > > > >>>>>>>>>>>>>
>>> > > > >>>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
>>> > > > >>>>>>>>
>>> > > > >>>>>>>>> Got
>>> > > > >>>>>>>>>>
>>> > > > >>>>>>>>>>> ping response for sessionid:0x424dfd908c50009 after
>>> > > > >>>>>>>>>>>>>
>>> > > > >>>>>>>>>>>> 9030ms
>>> > > > >>>
>>> > > > >>>> 4. 2009-11-11 04:27:43,960 DEBUG
>>> > > > >>>>>>>>>>>>>
>>> > > > >>>>>>>>>>>> org.apache.zookeeper.
>>> > > > >>>>>>>>
>>> > > > >>>>>>>
>>> > > >
>>> > >
>>> >
>>>
>>
>>
>
Re: regionserver disconnection
Posted by Zhenyu Zhong <zh...@gmail.com>.
Sorry for the delay.
Here is the link to the GC log.
http://pastebin.com/m13b52962
At line 1867, you will see a big number 2170
My GC configuration is
export HBASE_OPTS="-XX:+HeapDumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC
-XX:+CMSIncrementalMode -XX:+AggressiveOpts -XX:+UseFastAccessorMethods
-XX:ParallelGCThreads=8"
My GC logging configuration is:
-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
-Xloggc:/data/hbase_logs/gc-hbase.log
Have a wonderful weekend!
Best,
zhenyu
On Sat, Nov 14, 2009 at 5:56 PM, stack <st...@duboce.net> wrote:
> So, tell us more about your GC configuration and please post your GC log
> somewhere. A few of the lads here on this list are good at looking at GC
> logs.
> Thanks,
> St.Ack
>
> On Sat, Nov 14, 2009 at 1:20 PM, Zhenyu Zhong <zhongresearch@gmail.com
> >wrote:
>
> > Stack,
> >
> > I am very appreciated for your help.
> > I went back to check the GC log.
> > I found one entry that is
> > 8677.032: [CMS-concurrent-mark: 1.547/2170.359 secs] [Times: user=50.29
> > sys=3.65, real=2170.49 secs]
> > I couldn't tell what time this entry was logged.
> >
> > This is the only entry I found that has such huge number of GC time.
> > I also saw a few long GC time around 200 secs. (should be within the
> > session
> > timeout)
> >
> > Next, I need to find a way to correlate the timestamp with a friendly
> > dateformat on each GC log entry.
> >
> > Hopefully this is the reason.
> >
> > Best,
> > zhenyu
> >
> >
> >
> >
> > On Sat, Nov 14, 2009 at 3:03 PM, stack <st...@duboce.net> wrote:
> >
> > > On Sat, Nov 14, 2009 at 8:31 AM, Zhenyu Zhong <zhongresearch@gmail.com
> > > >wrote:
> > >
> > > ....
> > >
> > > > 2009-11-14 16:00:12,775 DEBUG org.apache.zookeeper.ClientCnxn: Got
> ping
> > > > response for sessionid:0x424f0ed01ea00a6 after 0ms
> > > > 2009-11-14 16:00:28,141 DEBUG org.apache.zookeeper.ClientCnxn: Got
> ping
> > > > response for sessionid:0x424f0ed01ea00a6 after 2010ms
> > > > 2009-11-14 16:01:10,378 WARN org.apache.zookeeper.ClientCnxn:
> Exception
> > > > closing session 0x424f0ed01ea00a6 to
> > sun.nio.ch.SelectionKeyImpl@39fbb2d6
> > > > java.io.IOException: Read error rc = -1
> java.nio.DirectByteBuffer[pos=0
> > > > lim=4 cap=4]
> > > > at
> > > > org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
> > > >
> > >
> > > Anything happening on the machine at this time? The ping time is
> > starting
> > > to climb... was 0ms for a long time and the last log is > 2seconds.
> > Whats
> > > in the GC logs? Anything else running on this machine?
> > > St.Ack
> > >
> >
>
Re: regionserver disconnection
Posted by stack <st...@duboce.net>.
So, tell us more about your GC configuration and please post your GC log
somewhere. A few of the lads here on this list are good at looking at GC
logs.
Thanks,
St.Ack
On Sat, Nov 14, 2009 at 1:20 PM, Zhenyu Zhong <zh...@gmail.com>wrote:
> Stack,
>
> I am very appreciated for your help.
> I went back to check the GC log.
> I found one entry that is
> 8677.032: [CMS-concurrent-mark: 1.547/2170.359 secs] [Times: user=50.29
> sys=3.65, real=2170.49 secs]
> I couldn't tell what time this entry was logged.
>
> This is the only entry I found that has such huge number of GC time.
> I also saw a few long GC time around 200 secs. (should be within the
> session
> timeout)
>
> Next, I need to find a way to correlate the timestamp with a friendly
> dateformat on each GC log entry.
>
> Hopefully this is the reason.
>
> Best,
> zhenyu
>
>
>
>
> On Sat, Nov 14, 2009 at 3:03 PM, stack <st...@duboce.net> wrote:
>
> > On Sat, Nov 14, 2009 at 8:31 AM, Zhenyu Zhong <zhongresearch@gmail.com
> > >wrote:
> >
> > ....
> >
> > > 2009-11-14 16:00:12,775 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
> > > response for sessionid:0x424f0ed01ea00a6 after 0ms
> > > 2009-11-14 16:00:28,141 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
> > > response for sessionid:0x424f0ed01ea00a6 after 2010ms
> > > 2009-11-14 16:01:10,378 WARN org.apache.zookeeper.ClientCnxn: Exception
> > > closing session 0x424f0ed01ea00a6 to
> sun.nio.ch.SelectionKeyImpl@39fbb2d6
> > > java.io.IOException: Read error rc = -1 java.nio.DirectByteBuffer[pos=0
> > > lim=4 cap=4]
> > > at
> > > org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
> > >
> >
> > Anything happening on the machine at this time? The ping time is
> starting
> > to climb... was 0ms for a long time and the last log is > 2seconds.
> Whats
> > in the GC logs? Anything else running on this machine?
> > St.Ack
> >
>
Re: regionserver disconnection
Posted by Zhenyu Zhong <zh...@gmail.com>.
Stack,
I am very appreciated for your help.
I went back to check the GC log.
I found one entry that is
8677.032: [CMS-concurrent-mark: 1.547/2170.359 secs] [Times: user=50.29
sys=3.65, real=2170.49 secs]
I couldn't tell what time this entry was logged.
This is the only entry I found that has such huge number of GC time.
I also saw a few long GC time around 200 secs. (should be within the session
timeout)
Next, I need to find a way to correlate the timestamp with a friendly
dateformat on each GC log entry.
Hopefully this is the reason.
Best,
zhenyu
On Sat, Nov 14, 2009 at 3:03 PM, stack <st...@duboce.net> wrote:
> On Sat, Nov 14, 2009 at 8:31 AM, Zhenyu Zhong <zhongresearch@gmail.com
> >wrote:
>
> ....
>
> > 2009-11-14 16:00:12,775 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
> > response for sessionid:0x424f0ed01ea00a6 after 0ms
> > 2009-11-14 16:00:28,141 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
> > response for sessionid:0x424f0ed01ea00a6 after 2010ms
> > 2009-11-14 16:01:10,378 WARN org.apache.zookeeper.ClientCnxn: Exception
> > closing session 0x424f0ed01ea00a6 to sun.nio.ch.SelectionKeyImpl@39fbb2d6
> > java.io.IOException: Read error rc = -1 java.nio.DirectByteBuffer[pos=0
> > lim=4 cap=4]
> > at
> > org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
> >
>
> Anything happening on the machine at this time? The ping time is starting
> to climb... was 0ms for a long time and the last log is > 2seconds. Whats
> in the GC logs? Anything else running on this machine?
> St.Ack
>
Re: regionserver disconnection
Posted by stack <st...@duboce.net>.
On Sat, Nov 14, 2009 at 8:31 AM, Zhenyu Zhong <zh...@gmail.com>wrote:
....
> 2009-11-14 16:00:12,775 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
> response for sessionid:0x424f0ed01ea00a6 after 0ms
> 2009-11-14 16:00:28,141 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
> response for sessionid:0x424f0ed01ea00a6 after 2010ms
> 2009-11-14 16:01:10,378 WARN org.apache.zookeeper.ClientCnxn: Exception
> closing session 0x424f0ed01ea00a6 to sun.nio.ch.SelectionKeyImpl@39fbb2d6
> java.io.IOException: Read error rc = -1 java.nio.DirectByteBuffer[pos=0
> lim=4 cap=4]
> at
> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
>
Anything happening on the machine at this time? The ping time is starting
to climb... was 0ms for a long time and the last log is > 2seconds. Whats
in the GC logs? Anything else running on this machine?
St.Ack
Re: regionserver disconnection
Posted by Zhenyu Zhong <zh...@gmail.com>.
Now I really doubt about the zookeeper, from the log I saw errors like
IOException Read Error, while the zookeeper client (regionserver ) tried to
read. But it got disconneted status from zookeeper.
I don't see any load on any zookeeper quorum servers. DiskIO is fine.
Also when the regionserver decides to exit due to the disconnect status from
zookeeper, sometimes the regionserver hangs during the exiting. We can still
see the HRegionServer process even we don't see it in the Master web
interface.
I also notice that there is an zookeeper.retries and zookeeper.pauses
settings in hbase-default.xml, would that matter ?
thanks
You may see the IOException error in the following log when the Regionserver
lost connection to the zookeeper.
2009-11-14 15:58:31,993 INFO org.apache.hadoop.hbase.regionserver.HLog: Roll
/hbase/.logs/superpyxis0008.scur.colo,60021,1258179101617/hlog.dat.1258179106921,
entries=1062964, calcsize=255014936, filesize=139566093. New hlog
/hbase/.logs/superpyxis0008.scur.colo,60021,1258179101617/hlog.dat.1258214311993
2009-11-14 15:58:39,295 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
response for sessionid:0x424f0ed01ea00a6 after 0ms
2009-11-14 15:58:52,648 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
response for sessionid:0x424f0ed01ea00a6 after 0ms
2009-11-14 15:59:06,007 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
response for sessionid:0x424f0ed01ea00a6 after 0ms
2009-11-14 15:59:19,365 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
response for sessionid:0x424f0ed01ea00a6 after 0ms
2009-11-14 15:59:32,867 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
response for sessionid:0x424f0ed01ea00a6 after 186ms
2009-11-14 15:59:46,070 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
response for sessionid:0x424f0ed01ea00a6 after 0ms
2009-11-14 15:59:59,423 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
response for sessionid:0x424f0ed01ea00a6 after 0ms
2009-11-14 16:00:12,775 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
response for sessionid:0x424f0ed01ea00a6 after 0ms
2009-11-14 16:00:28,141 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
response for sessionid:0x424f0ed01ea00a6 after 2010ms
2009-11-14 16:01:10,378 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x424f0ed01ea00a6 to sun.nio.ch.SelectionKeyImpl@39fbb2d6
java.io.IOException: Read error rc = -1 java.nio.DirectByteBuffer[pos=0
lim=4 cap=4]
at
org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
2009-11-14 16:01:10,478 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
state: Disconnected, type: None, path: null
2009-11-14 16:01:11,333 WARN
org.apache.hadoop.hbase.regionserver.HRegionServer: Attempt=1
org.apache.hadoop.hbase.Leases$LeaseStillHeldException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:571)
at java.lang.Thread.run(Thread.java:619)
2009-11-14 16:01:11,433 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server superpyxis0001/192.168.100.115:2181
2009-11-14 16:01:11,433 INFO org.apache.zookeeper.ClientCnxn: Priming
connection to java.nio.channels.SocketChannel[connected local=/
192.168.100.122:59575 remote=superpyxis0001/192.168.100.115:2181]
2009-11-14 16:01:11,433 INFO org.apache.zookeeper.ClientCnxn: Server
connection successful
2009-11-14 16:01:11,433 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
state: Expired, type: None, path: null
2009-11-14 16:01:11,433 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x424f0ed01ea00a6 to sun.nio.ch.SelectionKeyImpl@6b93d343
java.io.IOException: Session Expired
at
org.apache.zookeeper.ClientCnxn$SendThread.readConnectResult(ClientCnxn.java:589)
at
org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:709)
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
2009-11-14 16:01:11,433 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer: ZooKeeper session
expired
2009-11-14 16:01:13,513 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 3 on 60021, call put([B@73deb204,
[Lorg.apache.hadoop.hbase.client.Put;@2179600a) from 192.168.100.132:40728:
error: java.io.IOException: Server not running, aborting
java.io.IOException: Server not running, aborting
On Sat, Nov 14, 2009 at 1:01 AM, Zhenyu Zhong <zh...@gmail.com>wrote:
> Stack,
>
> Thanks a lot!
>
> I found out that the reason why HBase doesn't take the system file
> descriptor value I set before. I started the HBase using root instead of the
> normal hadoop user, while my system configures hadoop with higher file
> descriptor value but configures root with 1024 default value.
>
>
> Now my system has a clean start. However, it seems that the
> zookeeper.session.timeout value doesn't take into effect. I still found
> around 60 seconds pauses from the disconnected regionserver. I really don't
> know why regionserver only times out after 60 seconds instead of 10 minutes
> which I set for zookeeper.session.timeout.
>
> Is there any other timeout value coming into play before the actual session
> times out?
>
> zhenyu
>
>
>
> On Fri, Nov 13, 2009 at 7:08 PM, stack <st...@duboce.net> wrote:
>
>> Ok. Lack of file descriptors manifests in all kinds of weird ways.
>> Hopefully thats it. If not, lets keep digging.
>> St.Ack
>>
>> On Fri, Nov 13, 2009 at 3:44 PM, Zhenyu Zhong <zhongresearch@gmail.com
>> >wrote:
>>
>> > Stack,
>> >
>> > You are right, the master started with ulimit -n 1024. It doesn't take
>> the
>> > system value.
>> >
>> > Regarding to the too many open files, it looks like the same as the one
>> J-D
>> > put up. But I will get the Master start with higher value first and see
>> if
>> > this kind of error goes away.
>> >
>> > thanks a lot!
>> > zhenyu
>> >
>> > On Fri, Nov 13, 2009 at 6:02 PM, stack <st...@duboce.net> wrote:
>> >
>> > > Does it say
>> > >
>> > > ulimit -n 32768
>> > >
>> > > ...as the second line in your log file on start of the master?
>> > >
>> > > Is the stack trace that complains about too many open files same as
>> the
>> > one
>> > > in the blog post J-D put up?
>> > >
>> > > St.Ack
>> > >
>> > >
>> > > On Fri, Nov 13, 2009 at 1:37 PM, Zhenyu Zhong <
>> zhongresearch@gmail.com
>> > > >wrote:
>> > >
>> > > > The ulimit file descriptors was set to fs.file-max =1578334, also in
>> > the
>> > > > limits.conf the value was set to 32768.
>> > > > So these are way higher than the open descriptors for the running
>> > > > processes.
>> > > >
>> > > > thanks
>> > > > zhenyu
>> > > >
>> > > >
>> > > > On Fri, Nov 13, 2009 at 4:33 PM, Stack <sa...@gmail.com> wrote:
>> > > >
>> > > > > You upped the ulimit file descriptors as per the getting started
>> doc?
>> > > > >
>> > > > >
>> > > > >
>> > > > > On Nov 13, 2009, at 1:26 PM, Zhenyu Zhong <
>> zhongresearch@gmail.com>
>> > > > wrote:
>> > > > >
>> > > > > Thanks a lot.
>> > > > >>
>> > > > >>
>> > > > >> Bad news is my kernel is still 2.6.26.
>> > > > >> But it was not a problem before.
>> > > > >>
>> > > > >> Very strange.
>> > > > >>
>> > > > >> zhenyu
>> > > > >>
>> > > > >> On Fri, Nov 13, 2009 at 4:16 PM, Jean-Daniel Cryans <
>> > > > jdcryans@apache.org
>> > > > >> >wrote:
>> > > > >>
>> > > > >> Looks like
>> > > > >>>
>> > > > >>>
>> > > >
>> > >
>> >
>> http://pero.blogs.aprilmayjune.org/2009/01/22/hadoop-and-linux-kernel-2627-epoll-limits/
>> > > > >>>
>> > > > >>> J-D
>> > > > >>>
>> > > > >>> On Fri, Nov 13, 2009 at 1:12 PM, Zhenyu Zhong <
>> > > zhongresearch@gmail.com
>> > > > >
>> > > > >>> wrote:
>> > > > >>>
>> > > > >>>> Hi,
>> > > > >>>>
>> > > > >>>>
>> > > > >>>> After I re-organize the cluster, the experiment ran into
>> problem
>> > > > faster
>> > > > >>>>
>> > > > >>> than
>> > > > >>>
>> > > > >>>> before.
>> > > > >>>>
>> > > > >>>> Basically the changes are to use machines with less resources
>> as
>> > > > >>>>
>> > > > >>> zookeeper
>> > > > >>>
>> > > > >>>> quorums and machines with more resources as regionserver.
>> > > > >>>>
>> > > > >>>> From the log, I still see the pause around 1 minute.
>> > > > >>>> I enabled the GC logging, please see
>> > > > >>>>
>> > > > >>>> http://pastebin.com/m1d4ce0f1
>> > > > >>>>
>> > > > >>>> for details.
>> > > > >>>> In general I don't see many pauses in the GC.
>> > > > >>>>
>> > > > >>>> What is more interesting, 1 hour after the 1st regionserver
>> > > > >>>> disconnected,
>> > > > >>>> the master log complained about too many open files. This
>> didn't
>> > > > happen
>> > > > >>>> before.
>> > > > >>>> I checked the system OS setup as well as the limits.conf. I
>> also
>> > > > checked
>> > > > >>>>
>> > > > >>> the
>> > > > >>>
>> > > > >>>> running processes. The total open files don't reach the limit.
>> I
>> > am
>> > > > >>>>
>> > > > >>> confused
>> > > > >>>
>> > > > >>>> a bit.
>> > > > >>>>
>> > > > >>>>
>> > > > >>>> Please see the following master log.
>> > > > >>>>
>> > > > >>>> 2009-11-13 20:06:02,114 INFO
>> > > > org.apache.hadoop.hbase.master.BaseScanner:
>> > > > >>>> RegionManager.metaScanner scan of 4658 row(s) of meta region
>> > > {server:
>> > > > >>>> 192.168.100.128:60021, regionname: .META.,,1, startKey: <>}
>> > > complete
>> > > > >>>> 2009-11-13 20:06:02,114 INFO
>> > > > org.apache.hadoop.hbase.master.BaseScanner:
>> > > > >>>>
>> > > > >>> All
>> > > > >>>
>> > > > >>>> 1 .META. region(s) scanned
>> > > > >>>> 2009-11-13 20:06:07,677 DEBUG org.apache.zookeeper.ClientCnxn:
>> Got
>> > > > ping
>> > > > >>>> response for sessionid:0x424eebf1c10004c after 3ms
>> > > > >>>> 2009-11-13 20:06:08,178 INFO org.apache.hadoop.hdfs.DFSClient:
>> > > > Exception
>> > > > >>>>
>> > > > >>> in
>> > > > >>>
>> > > > >>>> createBlockOutputStream java.io.IOException: Bad connect ack
>> with
>> > > > >>>> firstBadLink 192.168.100.123:50010
>> > > > >>>> 2009-11-13 20:06:08,178 INFO org.apache.hadoop.hdfs.DFSClient:
>> > > > >>>> Abandoning
>> > > > >>>> block blk_-2808245019291145247_5478039
>> > > > >>>> 2009-11-13 20:06:09,682 INFO org.apache.hadoop.hdfs.DFSClient:
>> > > > Exception
>> > > > >>>>
>> > > > >>> in
>> > > > >>>
>> > > > >>>> createBlockOutputStream java.io.EOFException
>> > > > >>>> 2009-11-13 20:06:09,682 INFO org.apache.hadoop.hdfs.DFSClient:
>> > > > >>>> Abandoning
>> > > > >>>> block blk_1074853606841896259_5478048
>> > > > >>>> 2009-11-13 20:06:10,334 DEBUG org.apache.zookeeper.ClientCnxn:
>> Got
>> > > > ping
>> > > > >>>> response for sessionid:0x24eebf1043003c after 1ms
>> > > > >>>> 2009-11-13 20:06:21,018 DEBUG org.apache.zookeeper.ClientCnxn:
>> Got
>> > > > ping
>> > > > >>>> response for sessionid:0x424eebf1c10004c after 0ms
>> > > > >>>> 2009-11-13 20:06:23,674 DEBUG org.apache.zookeeper.ClientCnxn:
>> Got
>> > > > ping
>> > > > >>>> response for sessionid:0x24eebf1043003c after 0ms
>> > > > >>>> 2009-11-13 20:06:24,828 INFO org.apache.hadoop.hdfs.DFSClient:
>> > > > Exception
>> > > > >>>>
>> > > > >>> in
>> > > > >>>
>> > > > >>>> createBlockOutputStream java.io.IOException: Bad connect ack
>> with
>> > > > >>>> firstBadLink 192.168.100.123:50010
>> > > > >>>> 2009-11-13 20:06:24,828 INFO org.apache.hadoop.hdfs.DFSClient:
>> > > > >>>> Abandoning
>> > > > >>>> block blk_-6642544517082142289_5478063
>> > > > >>>> 2009-11-13 20:06:24,828 INFO org.apache.hadoop.hdfs.DFSClient:
>> > > > Exception
>> > > > >>>>
>> > > > >>> in
>> > > > >>>
>> > > > >>>> createBlockOutputStream java.net.SocketException: Too many open
>> > > files
>> > > > >>>> 2009-11-13 20:06:24,828 INFO org.apache.hadoop.hdfs.DFSClient:
>> > > > >>>> Abandoning
>> > > > >>>> block blk_2057511041109796090_5478063
>> > > > >>>> 2009-11-13 20:06:24,928 INFO org.apache.hadoop.hdfs.DFSClient:
>> > > > Exception
>> > > > >>>>
>> > > > >>> in
>> > > > >>>
>> > > > >>>> createBlockOutputStream java.net.SocketException: Too many open
>> > > files
>> > > > >>>> 2009-11-13 20:06:24,928 INFO org.apache.hadoop.hdfs.DFSClient:
>> > > > >>>> Abandoning
>> > > > >>>> block blk_8219260302213892894_5478064
>> > > > >>>> 2009-11-13 20:06:30,855 INFO org.apache.hadoop.hdfs.DFSClient:
>> > > > Exception
>> > > > >>>>
>> > > > >>> in
>> > > > >>>
>> > > > >>>> createBlockOutputStream java.net.SocketException: Too many open
>> > > files
>> > > > >>>> 2009-11-13 20:06:30,855 INFO org.apache.hadoop.hdfs.DFSClient:
>> > > > >>>> Abandoning
>> > > > >>>> block blk_1669205542853067709_5478235
>> > > > >>>> 2009-11-13 20:06:30,905 INFO org.apache.hadoop.hdfs.DFSClient:
>> > > > Exception
>> > > > >>>>
>> > > > >>> in
>> > > > >>>
>> > > > >>>> createBlockOutputStream java.net.SocketException: Too many open
>> > > files
>> > > > >>>> 2009-11-13 20:06:30,905 INFO org.apache.hadoop.hdfs.DFSClient:
>> > > > >>>> Abandoning
>> > > > >>>> block blk_9128897691346270351_5478237
>> > > > >>>> 2009-11-13 20:06:30,955 INFO org.apache.hadoop.hdfs.DFSClient:
>> > > > Exception
>> > > > >>>>
>> > > > >>> in
>> > > > >>>
>> > > > >>>> createBlockOutputStream java.net.SocketException: Too many open
>> > > files
>> > > > >>>> 2009-11-13 20:06:30,955 INFO org.apache.hadoop.hdfs.DFSClient:
>> > > > >>>> Abandoning
>> > > > >>>> block blk_1116845144864123018_5478240
>> > > > >>>> 2009-11-13 20:06:34,372 DEBUG org.apache.zookeeper.ClientCnxn:
>> Got
>> > > > ping
>> > > > >>>> response for sessionid:0x424eebf1c10004c after 0ms
>> > > > >>>> 2009-11-13 20:06:37,034 DEBUG org.apache.zookeeper.ClientCnxn:
>> Got
>> > > > ping
>> > > > >>>> response for sessionid:0x24eebf1043003c after 0ms
>> > > > >>>> 2009-11-13 20:06:37,235 WARN org.apache.hadoop.hdfs.DFSClient:
>> > > > >>>>
>> > > > >>> DataStreamer
>> > > > >>>
>> > > > >>>> Exception: java.io.IOException: Too many open files
>> > > > >>>> at sun.nio.ch.IOUtil.initPipe(Native Method)
>> > > > >>>> at
>> > > sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:49)
>> > > > >>>> at
>> > > > >>>>
>> > > > >>>>
>> > > >
>> > >
>> >
>> sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:18)
>> > > > >>>
>> > > > >>>> at
>> > > > >>>>
>> > > > >>>>
>> > > >
>> > >
>> >
>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.get(SocketIOWithTimeout.java:407)
>> > > > >>>
>> > > > >>>> at
>> > > > >>>>
>> > > > >>>>
>> > > >
>> > >
>> >
>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:322)
>> > > > >>>
>> > > > >>>> at
>> > > > >>>>
>> > > > >>>>
>> > > >
>> > >
>> >
>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
>> > > > >>>
>> > > > >>>> at
>> > > > >>>>
>> > > > >>>>
>> > > >
>> > >
>> >
>> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
>> > > > >>>
>> > > > >>>> at
>> > > > >>>>
>> > > > >>>>
>> > > >
>> > >
>> >
>> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
>> > > > >>>
>> > > > >>>> at
>> > > > >>>>
>> > > > >>>
>> java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
>> > > > >>>
>> > > > >>>> at
>> java.io.DataOutputStream.write(DataOutputStream.java:90)
>> > > > >>>> at
>> > > > >>>>
>> > > > >>>>
>> > > >
>> > >
>> >
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2290)
>> > > > >>>
>> > > > >>>>
>> > > > >>>> 2009-11-13 20:06:37,235 WARN org.apache.hadoop.hdfs.DFSClient:
>> > Error
>> > > > >>>> Recovery for block blk_8148813491785406356_5478475 bad
>> datanode[0]
>> > > > >>>> 192.168.100.123:50010
>> > > > >>>> 2009-11-13 20:06:37,235 WARN org.apache.hadoop.hdfs.DFSClient:
>> > Error
>> > > > >>>> Recovery for block blk_8148813491785406356_5478475 in pipeline
>> > > > >>>> 192.168.100.123:50010, 192.168.100.134:50010,
>> > 192.168.100.122:50010
>> > > :
>> > > > >>>> bad
>> > > > >>>> datanode 192.168.100.123:50010
>> > > > >>>> 2009-11-13 20:06:37,436 INFO org.apache.hadoop.hdfs.DFSClient:
>> > > > Exception
>> > > > >>>>
>> > > > >>> in
>> > > > >>>
>> > > > >>>> createBlockOutputStream java.net.SocketException: Too many open
>> > > files
>> > > > >>>> 2009-11-13 20:06:37,436 INFO org.apache.hadoop.hdfs.DFSClient:
>> > > > >>>> Abandoning
>> > > > >>>> block blk_2119727700857186236_5478498
>> > > > >>>>
>> > > > >>>>
>> > > > >>>>
>> > > > >>>>
>> > > > >>>>
>> > > > >>>> On Thu, Nov 12, 2009 at 4:21 PM, Zhenyu Zhong <
>> > > > zhongresearch@gmail.com
>> > > > >>>> wrote:
>> > > > >>>>
>> > > > >>>> Will do.
>> > > > >>>>>
>> > > > >>>>> thanks
>> > > > >>>>> zhenyu
>> > > > >>>>>
>> > > > >>>>>
>> > > > >>>>> On Thu, Nov 12, 2009 at 3:33 PM, stack <st...@duboce.net>
>> wrote:
>> > > > >>>>>
>> > > > >>>>> Enable GC logging too on this next run (see hbase-env.sh).
>> Lets
>> > > try
>> > > > >>>>>>
>> > > > >>>>> and
>> > > > >>>
>> > > > >>>> get
>> > > > >>>>>> to the bottom of whats going on.
>> > > > >>>>>> Thanks,
>> > > > >>>>>> St.Ack
>> > > > >>>>>>
>> > > > >>>>>> On Thu, Nov 12, 2009 at 12:29 PM, Zhenyu Zhong <
>> > > > >>>>>>
>> > > > >>>>> zhongresearch@gmail.com
>> > > > >>>
>> > > > >>>> wrote:
>> > > > >>>>>>>
>> > > > >>>>>>
>> > > > >>>>>> I can switch the boxes that run zookeeper with the ones that
>> > run
>> > > > >>>>>>> regionservers.
>> > > > >>>>>>> I will see the results later.
>> > > > >>>>>>>
>> > > > >>>>>>> FYI. The node does have the 10 minutes
>> > zookeeper.session.timeout
>> > > > >>>>>>>
>> > > > >>>>>> value
>> > > > >>>
>> > > > >>>> in
>> > > > >>>>>>
>> > > > >>>>>>> place.
>> > > > >>>>>>>
>> > > > >>>>>>> thanks
>> > > > >>>>>>> zhenyu
>> > > > >>>>>>>
>> > > > >>>>>>> On Thu, Nov 12, 2009 at 3:21 PM, stack <st...@duboce.net>
>> > wrote:
>> > > > >>>>>>>
>> > > > >>>>>>> On Thu, Nov 12, 2009 at 11:50 AM, Zhenyu Zhong <
>> > > > >>>>>>>>
>> > > > >>>>>>> zhongresearch@gmail.com
>> > > > >>>>>>
>> > > > >>>>>>> wrote:
>> > > > >>>>>>>>>
>> > > > >>>>>>>>
>> > > > >>>>>>>>
>> > > > >>>>>>>>> In my cluster, half of the cluster have 2 disks 400GB each
>> > per
>> > > > >>>>>>>>>
>> > > > >>>>>>>> machine,
>> > > > >>>>>>
>> > > > >>>>>>> and
>> > > > >>>>>>>>
>> > > > >>>>>>>>> half of the cluster have 6 disks per machine. Maybe we
>> > should
>> > > > >>>>>>>>>
>> > > > >>>>>>>> run
>> > > > >>>
>> > > > >>>> zookeeper
>> > > > >>>>>>>>> on the machines with 2 disks and RS on machines with 6
>> disks?
>> > > > >>>>>>>>>
>> > > > >>>>>>>>> That would make most sense only in the below, it looks
>> like
>> > the
>> > > > >>>>>>>>>
>> > > > >>>>>>>> RS
>> > > > >>>
>> > > > >>>> that
>> > > > >>>>>>
>> > > > >>>>>>> had
>> > > > >>>>>>>> issue had 4 disks?
>> > > > >>>>>>>>
>> > > > >>>>>>>>
>> > > > >>>>>>>>
>> > > > >>>>>>>> BTW, the 10 minutes zookeeper.session.timeout has been set
>> > > during
>> > > > >>>>>>>>>
>> > > > >>>>>>>> the
>> > > > >>>>>>
>> > > > >>>>>>> experiment.
>> > > > >>>>>>>>>
>> > > > >>>>>>>>>
>> > > > >>>>>>>> And for sure this node had it in place?
>> > > > >>>>>>>> St.Ack
>> > > > >>>>>>>>
>> > > > >>>>>>>>
>> > > > >>>>>>>>
>> > > > >>>>>>>>
>> > > > >>>>>>>>> thanks
>> > > > >>>>>>>>> zhenyu
>> > > > >>>>>>>>>
>> > > > >>>>>>>>> On Thu, Nov 12, 2009 at 2:08 PM, stack <st...@duboce.net>
>> > > wrote:
>> > > > >>>>>>>>>
>> > > > >>>>>>>>> On Thu, Nov 12, 2009 at 8:40 AM, Zhenyu Zhong <
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>> zhongresearch@gmail.com
>> > > > >>>>>>>
>> > > > >>>>>>>> wrote:
>> > > > >>>>>>>>>>>
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>>> Though I experienced 2 regionserver disconnection this
>> > > > >>>>>>>>>>>
>> > > > >>>>>>>>>> morning,
>> > > > >>>
>> > > > >>>> it
>> > > > >>>>>>
>> > > > >>>>>>> looks
>> > > > >>>>>>>>>
>> > > > >>>>>>>>>> better from the regionserver log. (please see the
>> following
>> > > > >>>>>>>>>>>
>> > > > >>>>>>>>>> log)
>> > > > >>>
>> > > > >>>>
>> > > > >>>>>>>>>>>
>> > > > >>>>>>>>>>> http://pastebin.com/m496dbfae
>> > > > >>>>>>>>>>>
>> > > > >>>>>>>>>>>
>> > > > >>>>>>>>>>> I checked diskIO for one of the
>> > regionserver(192.168.100.116)
>> > > > >>>>>>>>>>>
>> > > > >>>>>>>>>> during
>> > > > >>>>>>>
>> > > > >>>>>>>> the
>> > > > >>>>>>>>>
>> > > > >>>>>>>>>> time it disconnected.
>> > > > >>>>>>>>>>>
>> > > > >>>>>>>>>>> Time: 03:04:58 AM
>> > > > >>>>>>>>>>> Device: tps Blk_read/s Blk_wrtn/s
>> Blk_read
>> > > > >>>>>>>>>>>
>> > > > >>>>>>>>>> Blk_wrtn
>> > > > >>>>>>>>
>> > > > >>>>>>>>> sda 105.31 5458.83 19503.64
>> 9043873239
>> > > > >>>>>>>>>>>
>> > > > >>>>>>>>>> 32312473676
>> > > > >>>>>>>>
>> > > > >>>>>>>>> sda1 2.90 6.64 99.25
>> 10993934
>> > > > >>>>>>>>>>>
>> > > > >>>>>>>>>> 164433464
>> > > > >>>>>>>>
>> > > > >>>>>>>>> sda2 1.72 23.76 30.25
>> 39365817
>> > > > >>>>>>>>>>>
>> > > > >>>>>>>>>> 50124008
>> > > > >>>>>>>>
>> > > > >>>>>>>>> sda3 0.30 0.38 3.58
>> 624930
>> > > > >>>>>>>>>>>
>> > > > >>>>>>>>>> 5923000
>> > > > >>>>>>>>
>> > > > >>>>>>>>> sda4 100.39 5428.06 19370.56
>> 8992888270
>> > > > >>>>>>>>>>>
>> > > > >>>>>>>>>> 32091993204
>> > > > >>>>>>>>
>> > > > >>>>>>>>>
>> > > > >>>>>>>>>>>
>> > > > >>>>>>>>>>> Is this high for you? 20k blocks/second would seem to
>> be
>> > > high
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>> but
>> > > > >>>
>> > > > >>>> its
>> > > > >>>>>>>
>> > > > >>>>>>>> one
>> > > > >>>>>>>>>
>> > > > >>>>>>>>>> disk only and its not being shared by zk anymore so
>> > shouldn't
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>> matter?
>> > > > >>>>>>
>> > > > >>>>>>>
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>>> I also checked the zookeeper quorum server that the
>> > > > >>>>>>>>>>>
>> > > > >>>>>>>>>> regionserver
>> > > > >>>
>> > > > >>>> tried
>> > > > >>>>>>>>
>> > > > >>>>>>>>> to
>> > > > >>>>>>>>>
>> > > > >>>>>>>>>> connect according to the log. However, I don't see
>> > > > >>>>>>>>>>>
>> > > > >>>>>>>>>> 192.168.100.116
>> > > > >>>>>>
>> > > > >>>>>>> in
>> > > > >>>>>>>
>> > > > >>>>>>>> the
>> > > > >>>>>>>>>
>> > > > >>>>>>>>>> client list of the zookeeper quorum server that the
>> > > > >>>>>>>>>>>
>> > > > >>>>>>>>>> regionserver
>> > > > >>>
>> > > > >>>> tried
>> > > > >>>>>>>>
>> > > > >>>>>>>>> to
>> > > > >>>>>>>>>
>> > > > >>>>>>>>>> connect.
>> > > > >>>>>>>>>>> Would that be a problem?
>> > > > >>>>>>>>>>>
>> > > > >>>>>>>>>>>
>> > > > >>>>>>>>>>> Is that because the ephemeral node for the regionserver
>> > had
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>> evaporated?
>> > > > >>>>>>>
>> > > > >>>>>>>> Lost
>> > > > >>>>>>>>>> its lease w/ zk by the time you went to look?
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>>> Thu Nov 12 15:04:52 UTC 2009
>> > > > >>>>>>>>>>> Zookeeper version: 3.2.1-808558, built on 08/27/2009
>> 18:48
>> > > > >>>>>>>>>>>
>> > > > >>>>>>>>>> GMT
>> > > > >>>
>> > > > >>>> Clients:
>> > > > >>>>>>>>>>> /192.168.100.127:43045[1](queued=0,recved=26,sent=0)
>> > > > >>>>>>>>>>> /192.168.100.131:39091[1](queued=0,recved=964,sent=0)
>> > > > >>>>>>>>>>> /192.168.100.124:35961[1](queued=0,recved=3266,sent=0)
>> > > > >>>>>>>>>>> /192.168.100.123:47935[1](queued=0,recved=233,sent=0)
>> > > > >>>>>>>>>>> /192.168.100.125:46931[1](queued=0,recved=2,sent=0)
>> > > > >>>>>>>>>>> /192.168.100.118:54924[1](queued=0,recved=295,sent=0)
>> > > > >>>>>>>>>>> /192.168.100.118:41390[1](queued=0,recved=2290,sent=0)
>> > > > >>>>>>>>>>> /192.168.100.136:42243[1](queued=0,recved=0,sent=0)
>> > > > >>>>>>>>>>>
>> > > > >>>>>>>>>>> Latency min/avg/max: 0/17/6333
>> > > > >>>>>>>>>>> Received: 47111
>> > > > >>>>>>>>>>> Sent: 0
>> > > > >>>>>>>>>>> Outstanding: 0
>> > > > >>>>>>>>>>> Zxid: 0x77000083f4
>> > > > >>>>>>>>>>> Mode: leader
>> > > > >>>>>>>>>>> Node count: 23
>> > > > >>>>>>>>>>>
>> > > > >>>>>>>>>>>
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>>> That 6 second maximum latency is pretty bad but should be
>> > well
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>> within
>> > > > >>>>>>
>> > > > >>>>>>> the
>> > > > >>>>>>>>
>> > > > >>>>>>>>> zk
>> > > > >>>>>>>>>> session timeout.
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>>> So, problem is likely on the zk client side of the
>> session;
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>> i.e.
>> > > > >>>
>> > > > >>>> in
>> > > > >>>>>>
>> > > > >>>>>>> the
>> > > > >>>>>>>
>> > > > >>>>>>>> RS.
>> > > > >>>>>>>>>
>> > > > >>>>>>>>>> You could enable GC logging as J-D suggested to see if
>> you
>> > > have
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>> any
>> > > > >>>>>>
>> > > > >>>>>>> big
>> > > > >>>>>>>
>> > > > >>>>>>>> pauses, pauses > zk session timeout.
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>>> When the RS went down, it didn't look too heavily loaded:
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>>> 1. 2009-11-12 15:04:52,830 INFO
>> > > > >>>>>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Dump
>> of
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>> metrics:
>> > > > >>>>>>>
>> > > > >>>>>>>> request=1.5166667, regions=322, stores=657,
>> storefiles=631,
>> > > > >>>>>>>>>> storefileIndexSize=61, memstoreSize=1472, usedHeap=2819,
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>> maxHeap=4079,
>> > > > >>>>>>>>
>> > > > >>>>>>>>> blockCacheSize=658110960, blockCacheFree=197395984,
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>> blockCacheCount=9903,
>> > > > >>>>>>>>>
>> > > > >>>>>>>>>> blockCacheHitRatio=99
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>>> Its serving a few reads? The number of store files seems
>> > > fine.
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>> Not
>> > > > >>>>>>
>> > > > >>>>>>> too
>> > > > >>>>>>>>
>> > > > >>>>>>>>> much memory used.
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>>> Looking at the logs, I see the Lease Still Held
>> exception.
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>> This
>> > > > >>>
>> > > > >>>> happens
>> > > > >>>>>>>>
>> > > > >>>>>>>>> when the RS does its regular report to the master but the
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>> master
>> > > > >>>
>> > > > >>>> thinks
>> > > > >>>>>>>
>> > > > >>>>>>>> the
>> > > > >>>>>>>>>
>> > > > >>>>>>>>>> RS has since restarted. It'll think this probably
>> because
>> > it
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>> noticed
>> > > > >>>>>>
>> > > > >>>>>>> that
>> > > > >>>>>>>>>
>> > > > >>>>>>>>>> the RS's znode in zk had gone away and it considered the
>> RS
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>> dead.
>> > > > >>>
>> > > > >>>>
>> > > > >>>>>>>>>> Looking too at your logs I see this gap in the zk
>> pinging:
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>>> 1. 2009-11-12 15:03:39,325 DEBUG
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>> org.apache.zookeeper.ClientCnxn:
>> > > > >>>>>>
>> > > > >>>>>>> Got
>> > > > >>>>>>>
>> > > > >>>>>>>> ping response for sessionid:0x224e55436ad0004 after 0ms
>> > > > >>>>>>>>>> 2. 2009-11-12 15:03:43,113 DEBUG
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>> org.apache.zookeeper.ClientCnxn:
>> > > > >>>>>>
>> > > > >>>>>>> Got
>> > > > >>>>>>>
>> > > > >>>>>>>> ping response for sessionid:0x24e55436a0007d after 0ms
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>>> Where in the lines above it, its reporting about every
>> ten
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>> seconds,
>> > > > >>>>>>
>> > > > >>>>>>> here
>> > > > >>>>>>>>
>> > > > >>>>>>>>> there is a big gap.
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>>> Do you have ganglia or something that will let you look
>> more
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>> into
>> > > > >>>
>> > > > >>>> what
>> > > > >>>>>>>
>> > > > >>>>>>>> was
>> > > > >>>>>>>>>
>> > > > >>>>>>>>>> happening on this machine around this time? Is the
>> machine
>> > > OK?
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>> It
>> > > > >>>>>>
>> > > > >>>>>>> looks
>> > > > >>>>>>>>
>> > > > >>>>>>>>> lightly loaded and you have your cluster nicely laid out.
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>> Something
>> > > > >>>>>>
>> > > > >>>>>>> odd
>> > > > >>>>>>>>
>> > > > >>>>>>>>> is
>> > > > >>>>>>>>>
>> > > > >>>>>>>>>> going on. What about things like the write speed to
>> disk?
>> > In
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>> the
>> > > > >>>
>> > > > >>>> past
>> > > > >>>>>>>
>> > > > >>>>>>>> strange issues have been explained by incorrectly set BIOS
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>> which
>> > > > >>>
>> > > > >>>> made
>> > > > >>>>>>
>> > > > >>>>>>> disks
>> > > > >>>>>>>>>
>> > > > >>>>>>>>>> run at 1/100th of their proper speed.
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>>> St.Ack
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>>>> Best,
>> > > > >>>>>>>>>>> zhenyu
>> > > > >>>>>>>>>>>
>> > > > >>>>>>>>>>>
>> > > > >>>>>>>>>>> On Wed, Nov 11, 2009 at 3:58 PM, Zhenyu Zhong <
>> > > > >>>>>>>>>>>
>> > > > >>>>>>>>>> zhongresearch@gmail.com
>> > > > >>>>>>>>
>> > > > >>>>>>>>> wrote:
>> > > > >>>>>>>>>>>>
>> > > > >>>>>>>>>>>
>> > > > >>>>>>>>>>> Stack
>> > > > >>>>>>>>>>>>
>> > > > >>>>>>>>>>>> I am very appreciated for your comments.
>> > > > >>>>>>>>>>>> I will use the zookeeper monitoring script on my
>> cluster
>> > > > >>>>>>>>>>>>
>> > > > >>>>>>>>>>> and
>> > > > >>>
>> > > > >>>> let
>> > > > >>>>>>
>> > > > >>>>>>> it
>> > > > >>>>>>>
>> > > > >>>>>>>> run
>> > > > >>>>>>>>>
>> > > > >>>>>>>>>> overnight to see the result.
>> > > > >>>>>>>>>>>> I will follow up the thread when I get anything.
>> > > > >>>>>>>>>>>>
>> > > > >>>>>>>>>>>> thanks
>> > > > >>>>>>>>>>>> zhenyu
>> > > > >>>>>>>>>>>>
>> > > > >>>>>>>>>>>>
>> > > > >>>>>>>>>>>> On Wed, Nov 11, 2009 at 3:52 PM, stack <
>> stack@duboce.net>
>> > > > >>>>>>>>>>>>
>> > > > >>>>>>>>>>> wrote:
>> > > > >>>>>>
>> > > > >>>>>>>
>> > > > >>>>>>>>>>>> I see these in your log too:
>> > > > >>>>>>>>>>>>>
>> > > > >>>>>>>>>>>>>
>> > > > >>>>>>>>>>>>> 1. 2009-11-11 04:27:19,018 DEBUG
>> > > > >>>>>>>>>>>>>
>> > > > >>>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
>> > > > >>>>>>>>
>> > > > >>>>>>>>> Got
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>>>> ping response for sessionid:0x424dfd908c50009 after
>> > > > >>>>>>>>>>>>>
>> > > > >>>>>>>>>>>> 4544ms
>> > > > >>>
>> > > > >>>> 2. 2009-11-11 04:27:19,018 DEBUG
>> > > > >>>>>>>>>>>>>
>> > > > >>>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
>> > > > >>>>>>>>
>> > > > >>>>>>>>> Got
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>>>> ping response for sessionid:0x24dfd90c810002 after
>> > > > >>>>>>>>>>>>>
>> > > > >>>>>>>>>>>> 4548ms
>> > > > >>>
>> > > > >>>> 3. 2009-11-11 04:27:43,960 DEBUG
>> > > > >>>>>>>>>>>>>
>> > > > >>>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
>> > > > >>>>>>>>
>> > > > >>>>>>>>> Got
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>>>> ping response for sessionid:0x424dfd908c50009 after
>> > > > >>>>>>>>>>>>>
>> > > > >>>>>>>>>>>> 9030ms
>> > > > >>>
>> > > > >>>> 4. 2009-11-11 04:27:43,960 DEBUG
>> > > > >>>>>>>>>>>>>
>> > > > >>>>>>>>>>>> org.apache.zookeeper.
>> > > > >>>>>>>>
>> > > > >>>>>>>
>> > > >
>> > >
>> >
>>
>
>
Re: regionserver disconnection
Posted by Zhenyu Zhong <zh...@gmail.com>.
Stack,
Thanks a lot!
I found out that the reason why HBase doesn't take the system file
descriptor value I set before. I started the HBase using root instead of the
normal hadoop user, while my system configures hadoop with higher file
descriptor value but configures root with 1024 default value.
Now my system has a clean start. However, it seems that the
zookeeper.session.timeout value doesn't take into effect. I still found
around 60 seconds pauses from the disconnected regionserver. I really don't
know why regionserver only times out after 60 seconds instead of 10 minutes
which I set for zookeeper.session.timeout.
Is there any other timeout value coming into play before the actual session
times out?
zhenyu
On Fri, Nov 13, 2009 at 7:08 PM, stack <st...@duboce.net> wrote:
> Ok. Lack of file descriptors manifests in all kinds of weird ways.
> Hopefully thats it. If not, lets keep digging.
> St.Ack
>
> On Fri, Nov 13, 2009 at 3:44 PM, Zhenyu Zhong <zhongresearch@gmail.com
> >wrote:
>
> > Stack,
> >
> > You are right, the master started with ulimit -n 1024. It doesn't take
> the
> > system value.
> >
> > Regarding to the too many open files, it looks like the same as the one
> J-D
> > put up. But I will get the Master start with higher value first and see
> if
> > this kind of error goes away.
> >
> > thanks a lot!
> > zhenyu
> >
> > On Fri, Nov 13, 2009 at 6:02 PM, stack <st...@duboce.net> wrote:
> >
> > > Does it say
> > >
> > > ulimit -n 32768
> > >
> > > ...as the second line in your log file on start of the master?
> > >
> > > Is the stack trace that complains about too many open files same as the
> > one
> > > in the blog post J-D put up?
> > >
> > > St.Ack
> > >
> > >
> > > On Fri, Nov 13, 2009 at 1:37 PM, Zhenyu Zhong <zhongresearch@gmail.com
> > > >wrote:
> > >
> > > > The ulimit file descriptors was set to fs.file-max =1578334, also in
> > the
> > > > limits.conf the value was set to 32768.
> > > > So these are way higher than the open descriptors for the running
> > > > processes.
> > > >
> > > > thanks
> > > > zhenyu
> > > >
> > > >
> > > > On Fri, Nov 13, 2009 at 4:33 PM, Stack <sa...@gmail.com> wrote:
> > > >
> > > > > You upped the ulimit file descriptors as per the getting started
> doc?
> > > > >
> > > > >
> > > > >
> > > > > On Nov 13, 2009, at 1:26 PM, Zhenyu Zhong <zhongresearch@gmail.com
> >
> > > > wrote:
> > > > >
> > > > > Thanks a lot.
> > > > >>
> > > > >>
> > > > >> Bad news is my kernel is still 2.6.26.
> > > > >> But it was not a problem before.
> > > > >>
> > > > >> Very strange.
> > > > >>
> > > > >> zhenyu
> > > > >>
> > > > >> On Fri, Nov 13, 2009 at 4:16 PM, Jean-Daniel Cryans <
> > > > jdcryans@apache.org
> > > > >> >wrote:
> > > > >>
> > > > >> Looks like
> > > > >>>
> > > > >>>
> > > >
> > >
> >
> http://pero.blogs.aprilmayjune.org/2009/01/22/hadoop-and-linux-kernel-2627-epoll-limits/
> > > > >>>
> > > > >>> J-D
> > > > >>>
> > > > >>> On Fri, Nov 13, 2009 at 1:12 PM, Zhenyu Zhong <
> > > zhongresearch@gmail.com
> > > > >
> > > > >>> wrote:
> > > > >>>
> > > > >>>> Hi,
> > > > >>>>
> > > > >>>>
> > > > >>>> After I re-organize the cluster, the experiment ran into problem
> > > > faster
> > > > >>>>
> > > > >>> than
> > > > >>>
> > > > >>>> before.
> > > > >>>>
> > > > >>>> Basically the changes are to use machines with less resources as
> > > > >>>>
> > > > >>> zookeeper
> > > > >>>
> > > > >>>> quorums and machines with more resources as regionserver.
> > > > >>>>
> > > > >>>> From the log, I still see the pause around 1 minute.
> > > > >>>> I enabled the GC logging, please see
> > > > >>>>
> > > > >>>> http://pastebin.com/m1d4ce0f1
> > > > >>>>
> > > > >>>> for details.
> > > > >>>> In general I don't see many pauses in the GC.
> > > > >>>>
> > > > >>>> What is more interesting, 1 hour after the 1st regionserver
> > > > >>>> disconnected,
> > > > >>>> the master log complained about too many open files. This didn't
> > > > happen
> > > > >>>> before.
> > > > >>>> I checked the system OS setup as well as the limits.conf. I also
> > > > checked
> > > > >>>>
> > > > >>> the
> > > > >>>
> > > > >>>> running processes. The total open files don't reach the limit. I
> > am
> > > > >>>>
> > > > >>> confused
> > > > >>>
> > > > >>>> a bit.
> > > > >>>>
> > > > >>>>
> > > > >>>> Please see the following master log.
> > > > >>>>
> > > > >>>> 2009-11-13 20:06:02,114 INFO
> > > > org.apache.hadoop.hbase.master.BaseScanner:
> > > > >>>> RegionManager.metaScanner scan of 4658 row(s) of meta region
> > > {server:
> > > > >>>> 192.168.100.128:60021, regionname: .META.,,1, startKey: <>}
> > > complete
> > > > >>>> 2009-11-13 20:06:02,114 INFO
> > > > org.apache.hadoop.hbase.master.BaseScanner:
> > > > >>>>
> > > > >>> All
> > > > >>>
> > > > >>>> 1 .META. region(s) scanned
> > > > >>>> 2009-11-13 20:06:07,677 DEBUG org.apache.zookeeper.ClientCnxn:
> Got
> > > > ping
> > > > >>>> response for sessionid:0x424eebf1c10004c after 3ms
> > > > >>>> 2009-11-13 20:06:08,178 INFO org.apache.hadoop.hdfs.DFSClient:
> > > > Exception
> > > > >>>>
> > > > >>> in
> > > > >>>
> > > > >>>> createBlockOutputStream java.io.IOException: Bad connect ack
> with
> > > > >>>> firstBadLink 192.168.100.123:50010
> > > > >>>> 2009-11-13 20:06:08,178 INFO org.apache.hadoop.hdfs.DFSClient:
> > > > >>>> Abandoning
> > > > >>>> block blk_-2808245019291145247_5478039
> > > > >>>> 2009-11-13 20:06:09,682 INFO org.apache.hadoop.hdfs.DFSClient:
> > > > Exception
> > > > >>>>
> > > > >>> in
> > > > >>>
> > > > >>>> createBlockOutputStream java.io.EOFException
> > > > >>>> 2009-11-13 20:06:09,682 INFO org.apache.hadoop.hdfs.DFSClient:
> > > > >>>> Abandoning
> > > > >>>> block blk_1074853606841896259_5478048
> > > > >>>> 2009-11-13 20:06:10,334 DEBUG org.apache.zookeeper.ClientCnxn:
> Got
> > > > ping
> > > > >>>> response for sessionid:0x24eebf1043003c after 1ms
> > > > >>>> 2009-11-13 20:06:21,018 DEBUG org.apache.zookeeper.ClientCnxn:
> Got
> > > > ping
> > > > >>>> response for sessionid:0x424eebf1c10004c after 0ms
> > > > >>>> 2009-11-13 20:06:23,674 DEBUG org.apache.zookeeper.ClientCnxn:
> Got
> > > > ping
> > > > >>>> response for sessionid:0x24eebf1043003c after 0ms
> > > > >>>> 2009-11-13 20:06:24,828 INFO org.apache.hadoop.hdfs.DFSClient:
> > > > Exception
> > > > >>>>
> > > > >>> in
> > > > >>>
> > > > >>>> createBlockOutputStream java.io.IOException: Bad connect ack
> with
> > > > >>>> firstBadLink 192.168.100.123:50010
> > > > >>>> 2009-11-13 20:06:24,828 INFO org.apache.hadoop.hdfs.DFSClient:
> > > > >>>> Abandoning
> > > > >>>> block blk_-6642544517082142289_5478063
> > > > >>>> 2009-11-13 20:06:24,828 INFO org.apache.hadoop.hdfs.DFSClient:
> > > > Exception
> > > > >>>>
> > > > >>> in
> > > > >>>
> > > > >>>> createBlockOutputStream java.net.SocketException: Too many open
> > > files
> > > > >>>> 2009-11-13 20:06:24,828 INFO org.apache.hadoop.hdfs.DFSClient:
> > > > >>>> Abandoning
> > > > >>>> block blk_2057511041109796090_5478063
> > > > >>>> 2009-11-13 20:06:24,928 INFO org.apache.hadoop.hdfs.DFSClient:
> > > > Exception
> > > > >>>>
> > > > >>> in
> > > > >>>
> > > > >>>> createBlockOutputStream java.net.SocketException: Too many open
> > > files
> > > > >>>> 2009-11-13 20:06:24,928 INFO org.apache.hadoop.hdfs.DFSClient:
> > > > >>>> Abandoning
> > > > >>>> block blk_8219260302213892894_5478064
> > > > >>>> 2009-11-13 20:06:30,855 INFO org.apache.hadoop.hdfs.DFSClient:
> > > > Exception
> > > > >>>>
> > > > >>> in
> > > > >>>
> > > > >>>> createBlockOutputStream java.net.SocketException: Too many open
> > > files
> > > > >>>> 2009-11-13 20:06:30,855 INFO org.apache.hadoop.hdfs.DFSClient:
> > > > >>>> Abandoning
> > > > >>>> block blk_1669205542853067709_5478235
> > > > >>>> 2009-11-13 20:06:30,905 INFO org.apache.hadoop.hdfs.DFSClient:
> > > > Exception
> > > > >>>>
> > > > >>> in
> > > > >>>
> > > > >>>> createBlockOutputStream java.net.SocketException: Too many open
> > > files
> > > > >>>> 2009-11-13 20:06:30,905 INFO org.apache.hadoop.hdfs.DFSClient:
> > > > >>>> Abandoning
> > > > >>>> block blk_9128897691346270351_5478237
> > > > >>>> 2009-11-13 20:06:30,955 INFO org.apache.hadoop.hdfs.DFSClient:
> > > > Exception
> > > > >>>>
> > > > >>> in
> > > > >>>
> > > > >>>> createBlockOutputStream java.net.SocketException: Too many open
> > > files
> > > > >>>> 2009-11-13 20:06:30,955 INFO org.apache.hadoop.hdfs.DFSClient:
> > > > >>>> Abandoning
> > > > >>>> block blk_1116845144864123018_5478240
> > > > >>>> 2009-11-13 20:06:34,372 DEBUG org.apache.zookeeper.ClientCnxn:
> Got
> > > > ping
> > > > >>>> response for sessionid:0x424eebf1c10004c after 0ms
> > > > >>>> 2009-11-13 20:06:37,034 DEBUG org.apache.zookeeper.ClientCnxn:
> Got
> > > > ping
> > > > >>>> response for sessionid:0x24eebf1043003c after 0ms
> > > > >>>> 2009-11-13 20:06:37,235 WARN org.apache.hadoop.hdfs.DFSClient:
> > > > >>>>
> > > > >>> DataStreamer
> > > > >>>
> > > > >>>> Exception: java.io.IOException: Too many open files
> > > > >>>> at sun.nio.ch.IOUtil.initPipe(Native Method)
> > > > >>>> at
> > > sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:49)
> > > > >>>> at
> > > > >>>>
> > > > >>>>
> > > >
> > >
> >
> sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:18)
> > > > >>>
> > > > >>>> at
> > > > >>>>
> > > > >>>>
> > > >
> > >
> >
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.get(SocketIOWithTimeout.java:407)
> > > > >>>
> > > > >>>> at
> > > > >>>>
> > > > >>>>
> > > >
> > >
> >
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:322)
> > > > >>>
> > > > >>>> at
> > > > >>>>
> > > > >>>>
> > > >
> > >
> >
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
> > > > >>>
> > > > >>>> at
> > > > >>>>
> > > > >>>>
> > > >
> > >
> >
> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
> > > > >>>
> > > > >>>> at
> > > > >>>>
> > > > >>>>
> > > >
> > >
> >
> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
> > > > >>>
> > > > >>>> at
> > > > >>>>
> > > > >>> java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
> > > > >>>
> > > > >>>> at java.io.DataOutputStream.write(DataOutputStream.java:90)
> > > > >>>> at
> > > > >>>>
> > > > >>>>
> > > >
> > >
> >
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2290)
> > > > >>>
> > > > >>>>
> > > > >>>> 2009-11-13 20:06:37,235 WARN org.apache.hadoop.hdfs.DFSClient:
> > Error
> > > > >>>> Recovery for block blk_8148813491785406356_5478475 bad
> datanode[0]
> > > > >>>> 192.168.100.123:50010
> > > > >>>> 2009-11-13 20:06:37,235 WARN org.apache.hadoop.hdfs.DFSClient:
> > Error
> > > > >>>> Recovery for block blk_8148813491785406356_5478475 in pipeline
> > > > >>>> 192.168.100.123:50010, 192.168.100.134:50010,
> > 192.168.100.122:50010
> > > :
> > > > >>>> bad
> > > > >>>> datanode 192.168.100.123:50010
> > > > >>>> 2009-11-13 20:06:37,436 INFO org.apache.hadoop.hdfs.DFSClient:
> > > > Exception
> > > > >>>>
> > > > >>> in
> > > > >>>
> > > > >>>> createBlockOutputStream java.net.SocketException: Too many open
> > > files
> > > > >>>> 2009-11-13 20:06:37,436 INFO org.apache.hadoop.hdfs.DFSClient:
> > > > >>>> Abandoning
> > > > >>>> block blk_2119727700857186236_5478498
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > > >>>> On Thu, Nov 12, 2009 at 4:21 PM, Zhenyu Zhong <
> > > > zhongresearch@gmail.com
> > > > >>>> wrote:
> > > > >>>>
> > > > >>>> Will do.
> > > > >>>>>
> > > > >>>>> thanks
> > > > >>>>> zhenyu
> > > > >>>>>
> > > > >>>>>
> > > > >>>>> On Thu, Nov 12, 2009 at 3:33 PM, stack <st...@duboce.net>
> wrote:
> > > > >>>>>
> > > > >>>>> Enable GC logging too on this next run (see hbase-env.sh).
> Lets
> > > try
> > > > >>>>>>
> > > > >>>>> and
> > > > >>>
> > > > >>>> get
> > > > >>>>>> to the bottom of whats going on.
> > > > >>>>>> Thanks,
> > > > >>>>>> St.Ack
> > > > >>>>>>
> > > > >>>>>> On Thu, Nov 12, 2009 at 12:29 PM, Zhenyu Zhong <
> > > > >>>>>>
> > > > >>>>> zhongresearch@gmail.com
> > > > >>>
> > > > >>>> wrote:
> > > > >>>>>>>
> > > > >>>>>>
> > > > >>>>>> I can switch the boxes that run zookeeper with the ones that
> > run
> > > > >>>>>>> regionservers.
> > > > >>>>>>> I will see the results later.
> > > > >>>>>>>
> > > > >>>>>>> FYI. The node does have the 10 minutes
> > zookeeper.session.timeout
> > > > >>>>>>>
> > > > >>>>>> value
> > > > >>>
> > > > >>>> in
> > > > >>>>>>
> > > > >>>>>>> place.
> > > > >>>>>>>
> > > > >>>>>>> thanks
> > > > >>>>>>> zhenyu
> > > > >>>>>>>
> > > > >>>>>>> On Thu, Nov 12, 2009 at 3:21 PM, stack <st...@duboce.net>
> > wrote:
> > > > >>>>>>>
> > > > >>>>>>> On Thu, Nov 12, 2009 at 11:50 AM, Zhenyu Zhong <
> > > > >>>>>>>>
> > > > >>>>>>> zhongresearch@gmail.com
> > > > >>>>>>
> > > > >>>>>>> wrote:
> > > > >>>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>>>> In my cluster, half of the cluster have 2 disks 400GB each
> > per
> > > > >>>>>>>>>
> > > > >>>>>>>> machine,
> > > > >>>>>>
> > > > >>>>>>> and
> > > > >>>>>>>>
> > > > >>>>>>>>> half of the cluster have 6 disks per machine. Maybe we
> > should
> > > > >>>>>>>>>
> > > > >>>>>>>> run
> > > > >>>
> > > > >>>> zookeeper
> > > > >>>>>>>>> on the machines with 2 disks and RS on machines with 6
> disks?
> > > > >>>>>>>>>
> > > > >>>>>>>>> That would make most sense only in the below, it looks like
> > the
> > > > >>>>>>>>>
> > > > >>>>>>>> RS
> > > > >>>
> > > > >>>> that
> > > > >>>>>>
> > > > >>>>>>> had
> > > > >>>>>>>> issue had 4 disks?
> > > > >>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>>> BTW, the 10 minutes zookeeper.session.timeout has been set
> > > during
> > > > >>>>>>>>>
> > > > >>>>>>>> the
> > > > >>>>>>
> > > > >>>>>>> experiment.
> > > > >>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>> And for sure this node had it in place?
> > > > >>>>>>>> St.Ack
> > > > >>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>>>> thanks
> > > > >>>>>>>>> zhenyu
> > > > >>>>>>>>>
> > > > >>>>>>>>> On Thu, Nov 12, 2009 at 2:08 PM, stack <st...@duboce.net>
> > > wrote:
> > > > >>>>>>>>>
> > > > >>>>>>>>> On Thu, Nov 12, 2009 at 8:40 AM, Zhenyu Zhong <
> > > > >>>>>>>>>>
> > > > >>>>>>>>> zhongresearch@gmail.com
> > > > >>>>>>>
> > > > >>>>>>>> wrote:
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Though I experienced 2 regionserver disconnection this
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>> morning,
> > > > >>>
> > > > >>>> it
> > > > >>>>>>
> > > > >>>>>>> looks
> > > > >>>>>>>>>
> > > > >>>>>>>>>> better from the regionserver log. (please see the
> following
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>> log)
> > > > >>>
> > > > >>>>
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> http://pastebin.com/m496dbfae
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> I checked diskIO for one of the
> > regionserver(192.168.100.116)
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>> during
> > > > >>>>>>>
> > > > >>>>>>>> the
> > > > >>>>>>>>>
> > > > >>>>>>>>>> time it disconnected.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Time: 03:04:58 AM
> > > > >>>>>>>>>>> Device: tps Blk_read/s Blk_wrtn/s
> Blk_read
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>> Blk_wrtn
> > > > >>>>>>>>
> > > > >>>>>>>>> sda 105.31 5458.83 19503.64 9043873239
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>> 32312473676
> > > > >>>>>>>>
> > > > >>>>>>>>> sda1 2.90 6.64 99.25 10993934
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>> 164433464
> > > > >>>>>>>>
> > > > >>>>>>>>> sda2 1.72 23.76 30.25 39365817
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>> 50124008
> > > > >>>>>>>>
> > > > >>>>>>>>> sda3 0.30 0.38 3.58 624930
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>> 5923000
> > > > >>>>>>>>
> > > > >>>>>>>>> sda4 100.39 5428.06 19370.56 8992888270
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>> 32091993204
> > > > >>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Is this high for you? 20k blocks/second would seem to
> be
> > > high
> > > > >>>>>>>>>>
> > > > >>>>>>>>> but
> > > > >>>
> > > > >>>> its
> > > > >>>>>>>
> > > > >>>>>>>> one
> > > > >>>>>>>>>
> > > > >>>>>>>>>> disk only and its not being shared by zk anymore so
> > shouldn't
> > > > >>>>>>>>>>
> > > > >>>>>>>>> matter?
> > > > >>>>>>
> > > > >>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> I also checked the zookeeper quorum server that the
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>> regionserver
> > > > >>>
> > > > >>>> tried
> > > > >>>>>>>>
> > > > >>>>>>>>> to
> > > > >>>>>>>>>
> > > > >>>>>>>>>> connect according to the log. However, I don't see
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>> 192.168.100.116
> > > > >>>>>>
> > > > >>>>>>> in
> > > > >>>>>>>
> > > > >>>>>>>> the
> > > > >>>>>>>>>
> > > > >>>>>>>>>> client list of the zookeeper quorum server that the
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>> regionserver
> > > > >>>
> > > > >>>> tried
> > > > >>>>>>>>
> > > > >>>>>>>>> to
> > > > >>>>>>>>>
> > > > >>>>>>>>>> connect.
> > > > >>>>>>>>>>> Would that be a problem?
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Is that because the ephemeral node for the regionserver
> > had
> > > > >>>>>>>>>>
> > > > >>>>>>>>> evaporated?
> > > > >>>>>>>
> > > > >>>>>>>> Lost
> > > > >>>>>>>>>> its lease w/ zk by the time you went to look?
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Thu Nov 12 15:04:52 UTC 2009
> > > > >>>>>>>>>>> Zookeeper version: 3.2.1-808558, built on 08/27/2009
> 18:48
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>> GMT
> > > > >>>
> > > > >>>> Clients:
> > > > >>>>>>>>>>> /192.168.100.127:43045[1](queued=0,recved=26,sent=0)
> > > > >>>>>>>>>>> /192.168.100.131:39091[1](queued=0,recved=964,sent=0)
> > > > >>>>>>>>>>> /192.168.100.124:35961[1](queued=0,recved=3266,sent=0)
> > > > >>>>>>>>>>> /192.168.100.123:47935[1](queued=0,recved=233,sent=0)
> > > > >>>>>>>>>>> /192.168.100.125:46931[1](queued=0,recved=2,sent=0)
> > > > >>>>>>>>>>> /192.168.100.118:54924[1](queued=0,recved=295,sent=0)
> > > > >>>>>>>>>>> /192.168.100.118:41390[1](queued=0,recved=2290,sent=0)
> > > > >>>>>>>>>>> /192.168.100.136:42243[1](queued=0,recved=0,sent=0)
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Latency min/avg/max: 0/17/6333
> > > > >>>>>>>>>>> Received: 47111
> > > > >>>>>>>>>>> Sent: 0
> > > > >>>>>>>>>>> Outstanding: 0
> > > > >>>>>>>>>>> Zxid: 0x77000083f4
> > > > >>>>>>>>>>> Mode: leader
> > > > >>>>>>>>>>> Node count: 23
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> That 6 second maximum latency is pretty bad but should be
> > well
> > > > >>>>>>>>>>
> > > > >>>>>>>>> within
> > > > >>>>>>
> > > > >>>>>>> the
> > > > >>>>>>>>
> > > > >>>>>>>>> zk
> > > > >>>>>>>>>> session timeout.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> So, problem is likely on the zk client side of the
> session;
> > > > >>>>>>>>>>
> > > > >>>>>>>>> i.e.
> > > > >>>
> > > > >>>> in
> > > > >>>>>>
> > > > >>>>>>> the
> > > > >>>>>>>
> > > > >>>>>>>> RS.
> > > > >>>>>>>>>
> > > > >>>>>>>>>> You could enable GC logging as J-D suggested to see if you
> > > have
> > > > >>>>>>>>>>
> > > > >>>>>>>>> any
> > > > >>>>>>
> > > > >>>>>>> big
> > > > >>>>>>>
> > > > >>>>>>>> pauses, pauses > zk session timeout.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> When the RS went down, it didn't look too heavily loaded:
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> 1. 2009-11-12 15:04:52,830 INFO
> > > > >>>>>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Dump
> of
> > > > >>>>>>>>>>
> > > > >>>>>>>>> metrics:
> > > > >>>>>>>
> > > > >>>>>>>> request=1.5166667, regions=322, stores=657, storefiles=631,
> > > > >>>>>>>>>> storefileIndexSize=61, memstoreSize=1472, usedHeap=2819,
> > > > >>>>>>>>>>
> > > > >>>>>>>>> maxHeap=4079,
> > > > >>>>>>>>
> > > > >>>>>>>>> blockCacheSize=658110960, blockCacheFree=197395984,
> > > > >>>>>>>>>>
> > > > >>>>>>>>> blockCacheCount=9903,
> > > > >>>>>>>>>
> > > > >>>>>>>>>> blockCacheHitRatio=99
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Its serving a few reads? The number of store files seems
> > > fine.
> > > > >>>>>>>>>>
> > > > >>>>>>>>> Not
> > > > >>>>>>
> > > > >>>>>>> too
> > > > >>>>>>>>
> > > > >>>>>>>>> much memory used.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Looking at the logs, I see the Lease Still Held exception.
> > > > >>>>>>>>>>
> > > > >>>>>>>>> This
> > > > >>>
> > > > >>>> happens
> > > > >>>>>>>>
> > > > >>>>>>>>> when the RS does its regular report to the master but the
> > > > >>>>>>>>>>
> > > > >>>>>>>>> master
> > > > >>>
> > > > >>>> thinks
> > > > >>>>>>>
> > > > >>>>>>>> the
> > > > >>>>>>>>>
> > > > >>>>>>>>>> RS has since restarted. It'll think this probably because
> > it
> > > > >>>>>>>>>>
> > > > >>>>>>>>> noticed
> > > > >>>>>>
> > > > >>>>>>> that
> > > > >>>>>>>>>
> > > > >>>>>>>>>> the RS's znode in zk had gone away and it considered the
> RS
> > > > >>>>>>>>>>
> > > > >>>>>>>>> dead.
> > > > >>>
> > > > >>>>
> > > > >>>>>>>>>> Looking too at your logs I see this gap in the zk
> pinging:
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> 1. 2009-11-12 15:03:39,325 DEBUG
> > > > >>>>>>>>>>
> > > > >>>>>>>>> org.apache.zookeeper.ClientCnxn:
> > > > >>>>>>
> > > > >>>>>>> Got
> > > > >>>>>>>
> > > > >>>>>>>> ping response for sessionid:0x224e55436ad0004 after 0ms
> > > > >>>>>>>>>> 2. 2009-11-12 15:03:43,113 DEBUG
> > > > >>>>>>>>>>
> > > > >>>>>>>>> org.apache.zookeeper.ClientCnxn:
> > > > >>>>>>
> > > > >>>>>>> Got
> > > > >>>>>>>
> > > > >>>>>>>> ping response for sessionid:0x24e55436a0007d after 0ms
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Where in the lines above it, its reporting about every ten
> > > > >>>>>>>>>>
> > > > >>>>>>>>> seconds,
> > > > >>>>>>
> > > > >>>>>>> here
> > > > >>>>>>>>
> > > > >>>>>>>>> there is a big gap.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Do you have ganglia or something that will let you look
> more
> > > > >>>>>>>>>>
> > > > >>>>>>>>> into
> > > > >>>
> > > > >>>> what
> > > > >>>>>>>
> > > > >>>>>>>> was
> > > > >>>>>>>>>
> > > > >>>>>>>>>> happening on this machine around this time? Is the
> machine
> > > OK?
> > > > >>>>>>>>>>
> > > > >>>>>>>>> It
> > > > >>>>>>
> > > > >>>>>>> looks
> > > > >>>>>>>>
> > > > >>>>>>>>> lightly loaded and you have your cluster nicely laid out.
> > > > >>>>>>>>>>
> > > > >>>>>>>>> Something
> > > > >>>>>>
> > > > >>>>>>> odd
> > > > >>>>>>>>
> > > > >>>>>>>>> is
> > > > >>>>>>>>>
> > > > >>>>>>>>>> going on. What about things like the write speed to disk?
> > In
> > > > >>>>>>>>>>
> > > > >>>>>>>>> the
> > > > >>>
> > > > >>>> past
> > > > >>>>>>>
> > > > >>>>>>>> strange issues have been explained by incorrectly set BIOS
> > > > >>>>>>>>>>
> > > > >>>>>>>>> which
> > > > >>>
> > > > >>>> made
> > > > >>>>>>
> > > > >>>>>>> disks
> > > > >>>>>>>>>
> > > > >>>>>>>>>> run at 1/100th of their proper speed.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> St.Ack
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>> Best,
> > > > >>>>>>>>>>> zhenyu
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> On Wed, Nov 11, 2009 at 3:58 PM, Zhenyu Zhong <
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>> zhongresearch@gmail.com
> > > > >>>>>>>>
> > > > >>>>>>>>> wrote:
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Stack
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> I am very appreciated for your comments.
> > > > >>>>>>>>>>>> I will use the zookeeper monitoring script on my cluster
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>> and
> > > > >>>
> > > > >>>> let
> > > > >>>>>>
> > > > >>>>>>> it
> > > > >>>>>>>
> > > > >>>>>>>> run
> > > > >>>>>>>>>
> > > > >>>>>>>>>> overnight to see the result.
> > > > >>>>>>>>>>>> I will follow up the thread when I get anything.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> thanks
> > > > >>>>>>>>>>>> zhenyu
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> On Wed, Nov 11, 2009 at 3:52 PM, stack <
> stack@duboce.net>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>> wrote:
> > > > >>>>>>
> > > > >>>>>>>
> > > > >>>>>>>>>>>> I see these in your log too:
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> 1. 2009-11-11 04:27:19,018 DEBUG
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
> > > > >>>>>>>>
> > > > >>>>>>>>> Got
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>> ping response for sessionid:0x424dfd908c50009 after
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>> 4544ms
> > > > >>>
> > > > >>>> 2. 2009-11-11 04:27:19,018 DEBUG
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
> > > > >>>>>>>>
> > > > >>>>>>>>> Got
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>> ping response for sessionid:0x24dfd90c810002 after
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>> 4548ms
> > > > >>>
> > > > >>>> 3. 2009-11-11 04:27:43,960 DEBUG
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
> > > > >>>>>>>>
> > > > >>>>>>>>> Got
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>> ping response for sessionid:0x424dfd908c50009 after
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>> 9030ms
> > > > >>>
> > > > >>>> 4. 2009-11-11 04:27:43,960 DEBUG
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>> org.apache.zookeeper.
> > > > >>>>>>>>
> > > > >>>>>>>
> > > >
> > >
> >
>
Re: regionserver disconnection
Posted by stack <st...@duboce.net>.
Ok. Lack of file descriptors manifests in all kinds of weird ways.
Hopefully thats it. If not, lets keep digging.
St.Ack
On Fri, Nov 13, 2009 at 3:44 PM, Zhenyu Zhong <zh...@gmail.com>wrote:
> Stack,
>
> You are right, the master started with ulimit -n 1024. It doesn't take the
> system value.
>
> Regarding to the too many open files, it looks like the same as the one J-D
> put up. But I will get the Master start with higher value first and see if
> this kind of error goes away.
>
> thanks a lot!
> zhenyu
>
> On Fri, Nov 13, 2009 at 6:02 PM, stack <st...@duboce.net> wrote:
>
> > Does it say
> >
> > ulimit -n 32768
> >
> > ...as the second line in your log file on start of the master?
> >
> > Is the stack trace that complains about too many open files same as the
> one
> > in the blog post J-D put up?
> >
> > St.Ack
> >
> >
> > On Fri, Nov 13, 2009 at 1:37 PM, Zhenyu Zhong <zhongresearch@gmail.com
> > >wrote:
> >
> > > The ulimit file descriptors was set to fs.file-max =1578334, also in
> the
> > > limits.conf the value was set to 32768.
> > > So these are way higher than the open descriptors for the running
> > > processes.
> > >
> > > thanks
> > > zhenyu
> > >
> > >
> > > On Fri, Nov 13, 2009 at 4:33 PM, Stack <sa...@gmail.com> wrote:
> > >
> > > > You upped the ulimit file descriptors as per the getting started doc?
> > > >
> > > >
> > > >
> > > > On Nov 13, 2009, at 1:26 PM, Zhenyu Zhong <zh...@gmail.com>
> > > wrote:
> > > >
> > > > Thanks a lot.
> > > >>
> > > >>
> > > >> Bad news is my kernel is still 2.6.26.
> > > >> But it was not a problem before.
> > > >>
> > > >> Very strange.
> > > >>
> > > >> zhenyu
> > > >>
> > > >> On Fri, Nov 13, 2009 at 4:16 PM, Jean-Daniel Cryans <
> > > jdcryans@apache.org
> > > >> >wrote:
> > > >>
> > > >> Looks like
> > > >>>
> > > >>>
> > >
> >
> http://pero.blogs.aprilmayjune.org/2009/01/22/hadoop-and-linux-kernel-2627-epoll-limits/
> > > >>>
> > > >>> J-D
> > > >>>
> > > >>> On Fri, Nov 13, 2009 at 1:12 PM, Zhenyu Zhong <
> > zhongresearch@gmail.com
> > > >
> > > >>> wrote:
> > > >>>
> > > >>>> Hi,
> > > >>>>
> > > >>>>
> > > >>>> After I re-organize the cluster, the experiment ran into problem
> > > faster
> > > >>>>
> > > >>> than
> > > >>>
> > > >>>> before.
> > > >>>>
> > > >>>> Basically the changes are to use machines with less resources as
> > > >>>>
> > > >>> zookeeper
> > > >>>
> > > >>>> quorums and machines with more resources as regionserver.
> > > >>>>
> > > >>>> From the log, I still see the pause around 1 minute.
> > > >>>> I enabled the GC logging, please see
> > > >>>>
> > > >>>> http://pastebin.com/m1d4ce0f1
> > > >>>>
> > > >>>> for details.
> > > >>>> In general I don't see many pauses in the GC.
> > > >>>>
> > > >>>> What is more interesting, 1 hour after the 1st regionserver
> > > >>>> disconnected,
> > > >>>> the master log complained about too many open files. This didn't
> > > happen
> > > >>>> before.
> > > >>>> I checked the system OS setup as well as the limits.conf. I also
> > > checked
> > > >>>>
> > > >>> the
> > > >>>
> > > >>>> running processes. The total open files don't reach the limit. I
> am
> > > >>>>
> > > >>> confused
> > > >>>
> > > >>>> a bit.
> > > >>>>
> > > >>>>
> > > >>>> Please see the following master log.
> > > >>>>
> > > >>>> 2009-11-13 20:06:02,114 INFO
> > > org.apache.hadoop.hbase.master.BaseScanner:
> > > >>>> RegionManager.metaScanner scan of 4658 row(s) of meta region
> > {server:
> > > >>>> 192.168.100.128:60021, regionname: .META.,,1, startKey: <>}
> > complete
> > > >>>> 2009-11-13 20:06:02,114 INFO
> > > org.apache.hadoop.hbase.master.BaseScanner:
> > > >>>>
> > > >>> All
> > > >>>
> > > >>>> 1 .META. region(s) scanned
> > > >>>> 2009-11-13 20:06:07,677 DEBUG org.apache.zookeeper.ClientCnxn: Got
> > > ping
> > > >>>> response for sessionid:0x424eebf1c10004c after 3ms
> > > >>>> 2009-11-13 20:06:08,178 INFO org.apache.hadoop.hdfs.DFSClient:
> > > Exception
> > > >>>>
> > > >>> in
> > > >>>
> > > >>>> createBlockOutputStream java.io.IOException: Bad connect ack with
> > > >>>> firstBadLink 192.168.100.123:50010
> > > >>>> 2009-11-13 20:06:08,178 INFO org.apache.hadoop.hdfs.DFSClient:
> > > >>>> Abandoning
> > > >>>> block blk_-2808245019291145247_5478039
> > > >>>> 2009-11-13 20:06:09,682 INFO org.apache.hadoop.hdfs.DFSClient:
> > > Exception
> > > >>>>
> > > >>> in
> > > >>>
> > > >>>> createBlockOutputStream java.io.EOFException
> > > >>>> 2009-11-13 20:06:09,682 INFO org.apache.hadoop.hdfs.DFSClient:
> > > >>>> Abandoning
> > > >>>> block blk_1074853606841896259_5478048
> > > >>>> 2009-11-13 20:06:10,334 DEBUG org.apache.zookeeper.ClientCnxn: Got
> > > ping
> > > >>>> response for sessionid:0x24eebf1043003c after 1ms
> > > >>>> 2009-11-13 20:06:21,018 DEBUG org.apache.zookeeper.ClientCnxn: Got
> > > ping
> > > >>>> response for sessionid:0x424eebf1c10004c after 0ms
> > > >>>> 2009-11-13 20:06:23,674 DEBUG org.apache.zookeeper.ClientCnxn: Got
> > > ping
> > > >>>> response for sessionid:0x24eebf1043003c after 0ms
> > > >>>> 2009-11-13 20:06:24,828 INFO org.apache.hadoop.hdfs.DFSClient:
> > > Exception
> > > >>>>
> > > >>> in
> > > >>>
> > > >>>> createBlockOutputStream java.io.IOException: Bad connect ack with
> > > >>>> firstBadLink 192.168.100.123:50010
> > > >>>> 2009-11-13 20:06:24,828 INFO org.apache.hadoop.hdfs.DFSClient:
> > > >>>> Abandoning
> > > >>>> block blk_-6642544517082142289_5478063
> > > >>>> 2009-11-13 20:06:24,828 INFO org.apache.hadoop.hdfs.DFSClient:
> > > Exception
> > > >>>>
> > > >>> in
> > > >>>
> > > >>>> createBlockOutputStream java.net.SocketException: Too many open
> > files
> > > >>>> 2009-11-13 20:06:24,828 INFO org.apache.hadoop.hdfs.DFSClient:
> > > >>>> Abandoning
> > > >>>> block blk_2057511041109796090_5478063
> > > >>>> 2009-11-13 20:06:24,928 INFO org.apache.hadoop.hdfs.DFSClient:
> > > Exception
> > > >>>>
> > > >>> in
> > > >>>
> > > >>>> createBlockOutputStream java.net.SocketException: Too many open
> > files
> > > >>>> 2009-11-13 20:06:24,928 INFO org.apache.hadoop.hdfs.DFSClient:
> > > >>>> Abandoning
> > > >>>> block blk_8219260302213892894_5478064
> > > >>>> 2009-11-13 20:06:30,855 INFO org.apache.hadoop.hdfs.DFSClient:
> > > Exception
> > > >>>>
> > > >>> in
> > > >>>
> > > >>>> createBlockOutputStream java.net.SocketException: Too many open
> > files
> > > >>>> 2009-11-13 20:06:30,855 INFO org.apache.hadoop.hdfs.DFSClient:
> > > >>>> Abandoning
> > > >>>> block blk_1669205542853067709_5478235
> > > >>>> 2009-11-13 20:06:30,905 INFO org.apache.hadoop.hdfs.DFSClient:
> > > Exception
> > > >>>>
> > > >>> in
> > > >>>
> > > >>>> createBlockOutputStream java.net.SocketException: Too many open
> > files
> > > >>>> 2009-11-13 20:06:30,905 INFO org.apache.hadoop.hdfs.DFSClient:
> > > >>>> Abandoning
> > > >>>> block blk_9128897691346270351_5478237
> > > >>>> 2009-11-13 20:06:30,955 INFO org.apache.hadoop.hdfs.DFSClient:
> > > Exception
> > > >>>>
> > > >>> in
> > > >>>
> > > >>>> createBlockOutputStream java.net.SocketException: Too many open
> > files
> > > >>>> 2009-11-13 20:06:30,955 INFO org.apache.hadoop.hdfs.DFSClient:
> > > >>>> Abandoning
> > > >>>> block blk_1116845144864123018_5478240
> > > >>>> 2009-11-13 20:06:34,372 DEBUG org.apache.zookeeper.ClientCnxn: Got
> > > ping
> > > >>>> response for sessionid:0x424eebf1c10004c after 0ms
> > > >>>> 2009-11-13 20:06:37,034 DEBUG org.apache.zookeeper.ClientCnxn: Got
> > > ping
> > > >>>> response for sessionid:0x24eebf1043003c after 0ms
> > > >>>> 2009-11-13 20:06:37,235 WARN org.apache.hadoop.hdfs.DFSClient:
> > > >>>>
> > > >>> DataStreamer
> > > >>>
> > > >>>> Exception: java.io.IOException: Too many open files
> > > >>>> at sun.nio.ch.IOUtil.initPipe(Native Method)
> > > >>>> at
> > sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:49)
> > > >>>> at
> > > >>>>
> > > >>>>
> > >
> >
> sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:18)
> > > >>>
> > > >>>> at
> > > >>>>
> > > >>>>
> > >
> >
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.get(SocketIOWithTimeout.java:407)
> > > >>>
> > > >>>> at
> > > >>>>
> > > >>>>
> > >
> >
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:322)
> > > >>>
> > > >>>> at
> > > >>>>
> > > >>>>
> > >
> >
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
> > > >>>
> > > >>>> at
> > > >>>>
> > > >>>>
> > >
> >
> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
> > > >>>
> > > >>>> at
> > > >>>>
> > > >>>>
> > >
> >
> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
> > > >>>
> > > >>>> at
> > > >>>>
> > > >>> java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
> > > >>>
> > > >>>> at java.io.DataOutputStream.write(DataOutputStream.java:90)
> > > >>>> at
> > > >>>>
> > > >>>>
> > >
> >
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2290)
> > > >>>
> > > >>>>
> > > >>>> 2009-11-13 20:06:37,235 WARN org.apache.hadoop.hdfs.DFSClient:
> Error
> > > >>>> Recovery for block blk_8148813491785406356_5478475 bad datanode[0]
> > > >>>> 192.168.100.123:50010
> > > >>>> 2009-11-13 20:06:37,235 WARN org.apache.hadoop.hdfs.DFSClient:
> Error
> > > >>>> Recovery for block blk_8148813491785406356_5478475 in pipeline
> > > >>>> 192.168.100.123:50010, 192.168.100.134:50010,
> 192.168.100.122:50010
> > :
> > > >>>> bad
> > > >>>> datanode 192.168.100.123:50010
> > > >>>> 2009-11-13 20:06:37,436 INFO org.apache.hadoop.hdfs.DFSClient:
> > > Exception
> > > >>>>
> > > >>> in
> > > >>>
> > > >>>> createBlockOutputStream java.net.SocketException: Too many open
> > files
> > > >>>> 2009-11-13 20:06:37,436 INFO org.apache.hadoop.hdfs.DFSClient:
> > > >>>> Abandoning
> > > >>>> block blk_2119727700857186236_5478498
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> On Thu, Nov 12, 2009 at 4:21 PM, Zhenyu Zhong <
> > > zhongresearch@gmail.com
> > > >>>> wrote:
> > > >>>>
> > > >>>> Will do.
> > > >>>>>
> > > >>>>> thanks
> > > >>>>> zhenyu
> > > >>>>>
> > > >>>>>
> > > >>>>> On Thu, Nov 12, 2009 at 3:33 PM, stack <st...@duboce.net> wrote:
> > > >>>>>
> > > >>>>> Enable GC logging too on this next run (see hbase-env.sh). Lets
> > try
> > > >>>>>>
> > > >>>>> and
> > > >>>
> > > >>>> get
> > > >>>>>> to the bottom of whats going on.
> > > >>>>>> Thanks,
> > > >>>>>> St.Ack
> > > >>>>>>
> > > >>>>>> On Thu, Nov 12, 2009 at 12:29 PM, Zhenyu Zhong <
> > > >>>>>>
> > > >>>>> zhongresearch@gmail.com
> > > >>>
> > > >>>> wrote:
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>> I can switch the boxes that run zookeeper with the ones that
> run
> > > >>>>>>> regionservers.
> > > >>>>>>> I will see the results later.
> > > >>>>>>>
> > > >>>>>>> FYI. The node does have the 10 minutes
> zookeeper.session.timeout
> > > >>>>>>>
> > > >>>>>> value
> > > >>>
> > > >>>> in
> > > >>>>>>
> > > >>>>>>> place.
> > > >>>>>>>
> > > >>>>>>> thanks
> > > >>>>>>> zhenyu
> > > >>>>>>>
> > > >>>>>>> On Thu, Nov 12, 2009 at 3:21 PM, stack <st...@duboce.net>
> wrote:
> > > >>>>>>>
> > > >>>>>>> On Thu, Nov 12, 2009 at 11:50 AM, Zhenyu Zhong <
> > > >>>>>>>>
> > > >>>>>>> zhongresearch@gmail.com
> > > >>>>>>
> > > >>>>>>> wrote:
> > > >>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>>> In my cluster, half of the cluster have 2 disks 400GB each
> per
> > > >>>>>>>>>
> > > >>>>>>>> machine,
> > > >>>>>>
> > > >>>>>>> and
> > > >>>>>>>>
> > > >>>>>>>>> half of the cluster have 6 disks per machine. Maybe we
> should
> > > >>>>>>>>>
> > > >>>>>>>> run
> > > >>>
> > > >>>> zookeeper
> > > >>>>>>>>> on the machines with 2 disks and RS on machines with 6 disks?
> > > >>>>>>>>>
> > > >>>>>>>>> That would make most sense only in the below, it looks like
> the
> > > >>>>>>>>>
> > > >>>>>>>> RS
> > > >>>
> > > >>>> that
> > > >>>>>>
> > > >>>>>>> had
> > > >>>>>>>> issue had 4 disks?
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>> BTW, the 10 minutes zookeeper.session.timeout has been set
> > during
> > > >>>>>>>>>
> > > >>>>>>>> the
> > > >>>>>>
> > > >>>>>>> experiment.
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>> And for sure this node had it in place?
> > > >>>>>>>> St.Ack
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>>> thanks
> > > >>>>>>>>> zhenyu
> > > >>>>>>>>>
> > > >>>>>>>>> On Thu, Nov 12, 2009 at 2:08 PM, stack <st...@duboce.net>
> > wrote:
> > > >>>>>>>>>
> > > >>>>>>>>> On Thu, Nov 12, 2009 at 8:40 AM, Zhenyu Zhong <
> > > >>>>>>>>>>
> > > >>>>>>>>> zhongresearch@gmail.com
> > > >>>>>>>
> > > >>>>>>>> wrote:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>> Though I experienced 2 regionserver disconnection this
> > > >>>>>>>>>>>
> > > >>>>>>>>>> morning,
> > > >>>
> > > >>>> it
> > > >>>>>>
> > > >>>>>>> looks
> > > >>>>>>>>>
> > > >>>>>>>>>> better from the regionserver log. (please see the following
> > > >>>>>>>>>>>
> > > >>>>>>>>>> log)
> > > >>>
> > > >>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> http://pastebin.com/m496dbfae
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> I checked diskIO for one of the
> regionserver(192.168.100.116)
> > > >>>>>>>>>>>
> > > >>>>>>>>>> during
> > > >>>>>>>
> > > >>>>>>>> the
> > > >>>>>>>>>
> > > >>>>>>>>>> time it disconnected.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Time: 03:04:58 AM
> > > >>>>>>>>>>> Device: tps Blk_read/s Blk_wrtn/s Blk_read
> > > >>>>>>>>>>>
> > > >>>>>>>>>> Blk_wrtn
> > > >>>>>>>>
> > > >>>>>>>>> sda 105.31 5458.83 19503.64 9043873239
> > > >>>>>>>>>>>
> > > >>>>>>>>>> 32312473676
> > > >>>>>>>>
> > > >>>>>>>>> sda1 2.90 6.64 99.25 10993934
> > > >>>>>>>>>>>
> > > >>>>>>>>>> 164433464
> > > >>>>>>>>
> > > >>>>>>>>> sda2 1.72 23.76 30.25 39365817
> > > >>>>>>>>>>>
> > > >>>>>>>>>> 50124008
> > > >>>>>>>>
> > > >>>>>>>>> sda3 0.30 0.38 3.58 624930
> > > >>>>>>>>>>>
> > > >>>>>>>>>> 5923000
> > > >>>>>>>>
> > > >>>>>>>>> sda4 100.39 5428.06 19370.56 8992888270
> > > >>>>>>>>>>>
> > > >>>>>>>>>> 32091993204
> > > >>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Is this high for you? 20k blocks/second would seem to be
> > high
> > > >>>>>>>>>>
> > > >>>>>>>>> but
> > > >>>
> > > >>>> its
> > > >>>>>>>
> > > >>>>>>>> one
> > > >>>>>>>>>
> > > >>>>>>>>>> disk only and its not being shared by zk anymore so
> shouldn't
> > > >>>>>>>>>>
> > > >>>>>>>>> matter?
> > > >>>>>>
> > > >>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>> I also checked the zookeeper quorum server that the
> > > >>>>>>>>>>>
> > > >>>>>>>>>> regionserver
> > > >>>
> > > >>>> tried
> > > >>>>>>>>
> > > >>>>>>>>> to
> > > >>>>>>>>>
> > > >>>>>>>>>> connect according to the log. However, I don't see
> > > >>>>>>>>>>>
> > > >>>>>>>>>> 192.168.100.116
> > > >>>>>>
> > > >>>>>>> in
> > > >>>>>>>
> > > >>>>>>>> the
> > > >>>>>>>>>
> > > >>>>>>>>>> client list of the zookeeper quorum server that the
> > > >>>>>>>>>>>
> > > >>>>>>>>>> regionserver
> > > >>>
> > > >>>> tried
> > > >>>>>>>>
> > > >>>>>>>>> to
> > > >>>>>>>>>
> > > >>>>>>>>>> connect.
> > > >>>>>>>>>>> Would that be a problem?
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Is that because the ephemeral node for the regionserver
> had
> > > >>>>>>>>>>
> > > >>>>>>>>> evaporated?
> > > >>>>>>>
> > > >>>>>>>> Lost
> > > >>>>>>>>>> its lease w/ zk by the time you went to look?
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>> Thu Nov 12 15:04:52 UTC 2009
> > > >>>>>>>>>>> Zookeeper version: 3.2.1-808558, built on 08/27/2009 18:48
> > > >>>>>>>>>>>
> > > >>>>>>>>>> GMT
> > > >>>
> > > >>>> Clients:
> > > >>>>>>>>>>> /192.168.100.127:43045[1](queued=0,recved=26,sent=0)
> > > >>>>>>>>>>> /192.168.100.131:39091[1](queued=0,recved=964,sent=0)
> > > >>>>>>>>>>> /192.168.100.124:35961[1](queued=0,recved=3266,sent=0)
> > > >>>>>>>>>>> /192.168.100.123:47935[1](queued=0,recved=233,sent=0)
> > > >>>>>>>>>>> /192.168.100.125:46931[1](queued=0,recved=2,sent=0)
> > > >>>>>>>>>>> /192.168.100.118:54924[1](queued=0,recved=295,sent=0)
> > > >>>>>>>>>>> /192.168.100.118:41390[1](queued=0,recved=2290,sent=0)
> > > >>>>>>>>>>> /192.168.100.136:42243[1](queued=0,recved=0,sent=0)
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Latency min/avg/max: 0/17/6333
> > > >>>>>>>>>>> Received: 47111
> > > >>>>>>>>>>> Sent: 0
> > > >>>>>>>>>>> Outstanding: 0
> > > >>>>>>>>>>> Zxid: 0x77000083f4
> > > >>>>>>>>>>> Mode: leader
> > > >>>>>>>>>>> Node count: 23
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>> That 6 second maximum latency is pretty bad but should be
> well
> > > >>>>>>>>>>
> > > >>>>>>>>> within
> > > >>>>>>
> > > >>>>>>> the
> > > >>>>>>>>
> > > >>>>>>>>> zk
> > > >>>>>>>>>> session timeout.
> > > >>>>>>>>>>
> > > >>>>>>>>>> So, problem is likely on the zk client side of the session;
> > > >>>>>>>>>>
> > > >>>>>>>>> i.e.
> > > >>>
> > > >>>> in
> > > >>>>>>
> > > >>>>>>> the
> > > >>>>>>>
> > > >>>>>>>> RS.
> > > >>>>>>>>>
> > > >>>>>>>>>> You could enable GC logging as J-D suggested to see if you
> > have
> > > >>>>>>>>>>
> > > >>>>>>>>> any
> > > >>>>>>
> > > >>>>>>> big
> > > >>>>>>>
> > > >>>>>>>> pauses, pauses > zk session timeout.
> > > >>>>>>>>>>
> > > >>>>>>>>>> When the RS went down, it didn't look too heavily loaded:
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>> 1. 2009-11-12 15:04:52,830 INFO
> > > >>>>>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of
> > > >>>>>>>>>>
> > > >>>>>>>>> metrics:
> > > >>>>>>>
> > > >>>>>>>> request=1.5166667, regions=322, stores=657, storefiles=631,
> > > >>>>>>>>>> storefileIndexSize=61, memstoreSize=1472, usedHeap=2819,
> > > >>>>>>>>>>
> > > >>>>>>>>> maxHeap=4079,
> > > >>>>>>>>
> > > >>>>>>>>> blockCacheSize=658110960, blockCacheFree=197395984,
> > > >>>>>>>>>>
> > > >>>>>>>>> blockCacheCount=9903,
> > > >>>>>>>>>
> > > >>>>>>>>>> blockCacheHitRatio=99
> > > >>>>>>>>>>
> > > >>>>>>>>>> Its serving a few reads? The number of store files seems
> > fine.
> > > >>>>>>>>>>
> > > >>>>>>>>> Not
> > > >>>>>>
> > > >>>>>>> too
> > > >>>>>>>>
> > > >>>>>>>>> much memory used.
> > > >>>>>>>>>>
> > > >>>>>>>>>> Looking at the logs, I see the Lease Still Held exception.
> > > >>>>>>>>>>
> > > >>>>>>>>> This
> > > >>>
> > > >>>> happens
> > > >>>>>>>>
> > > >>>>>>>>> when the RS does its regular report to the master but the
> > > >>>>>>>>>>
> > > >>>>>>>>> master
> > > >>>
> > > >>>> thinks
> > > >>>>>>>
> > > >>>>>>>> the
> > > >>>>>>>>>
> > > >>>>>>>>>> RS has since restarted. It'll think this probably because
> it
> > > >>>>>>>>>>
> > > >>>>>>>>> noticed
> > > >>>>>>
> > > >>>>>>> that
> > > >>>>>>>>>
> > > >>>>>>>>>> the RS's znode in zk had gone away and it considered the RS
> > > >>>>>>>>>>
> > > >>>>>>>>> dead.
> > > >>>
> > > >>>>
> > > >>>>>>>>>> Looking too at your logs I see this gap in the zk pinging:
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>> 1. 2009-11-12 15:03:39,325 DEBUG
> > > >>>>>>>>>>
> > > >>>>>>>>> org.apache.zookeeper.ClientCnxn:
> > > >>>>>>
> > > >>>>>>> Got
> > > >>>>>>>
> > > >>>>>>>> ping response for sessionid:0x224e55436ad0004 after 0ms
> > > >>>>>>>>>> 2. 2009-11-12 15:03:43,113 DEBUG
> > > >>>>>>>>>>
> > > >>>>>>>>> org.apache.zookeeper.ClientCnxn:
> > > >>>>>>
> > > >>>>>>> Got
> > > >>>>>>>
> > > >>>>>>>> ping response for sessionid:0x24e55436a0007d after 0ms
> > > >>>>>>>>>>
> > > >>>>>>>>>> Where in the lines above it, its reporting about every ten
> > > >>>>>>>>>>
> > > >>>>>>>>> seconds,
> > > >>>>>>
> > > >>>>>>> here
> > > >>>>>>>>
> > > >>>>>>>>> there is a big gap.
> > > >>>>>>>>>>
> > > >>>>>>>>>> Do you have ganglia or something that will let you look more
> > > >>>>>>>>>>
> > > >>>>>>>>> into
> > > >>>
> > > >>>> what
> > > >>>>>>>
> > > >>>>>>>> was
> > > >>>>>>>>>
> > > >>>>>>>>>> happening on this machine around this time? Is the machine
> > OK?
> > > >>>>>>>>>>
> > > >>>>>>>>> It
> > > >>>>>>
> > > >>>>>>> looks
> > > >>>>>>>>
> > > >>>>>>>>> lightly loaded and you have your cluster nicely laid out.
> > > >>>>>>>>>>
> > > >>>>>>>>> Something
> > > >>>>>>
> > > >>>>>>> odd
> > > >>>>>>>>
> > > >>>>>>>>> is
> > > >>>>>>>>>
> > > >>>>>>>>>> going on. What about things like the write speed to disk?
> In
> > > >>>>>>>>>>
> > > >>>>>>>>> the
> > > >>>
> > > >>>> past
> > > >>>>>>>
> > > >>>>>>>> strange issues have been explained by incorrectly set BIOS
> > > >>>>>>>>>>
> > > >>>>>>>>> which
> > > >>>
> > > >>>> made
> > > >>>>>>
> > > >>>>>>> disks
> > > >>>>>>>>>
> > > >>>>>>>>>> run at 1/100th of their proper speed.
> > > >>>>>>>>>>
> > > >>>>>>>>>> St.Ack
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>> Best,
> > > >>>>>>>>>>> zhenyu
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> On Wed, Nov 11, 2009 at 3:58 PM, Zhenyu Zhong <
> > > >>>>>>>>>>>
> > > >>>>>>>>>> zhongresearch@gmail.com
> > > >>>>>>>>
> > > >>>>>>>>> wrote:
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Stack
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> I am very appreciated for your comments.
> > > >>>>>>>>>>>> I will use the zookeeper monitoring script on my cluster
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>> and
> > > >>>
> > > >>>> let
> > > >>>>>>
> > > >>>>>>> it
> > > >>>>>>>
> > > >>>>>>>> run
> > > >>>>>>>>>
> > > >>>>>>>>>> overnight to see the result.
> > > >>>>>>>>>>>> I will follow up the thread when I get anything.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> thanks
> > > >>>>>>>>>>>> zhenyu
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> On Wed, Nov 11, 2009 at 3:52 PM, stack <st...@duboce.net>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>> wrote:
> > > >>>>>>
> > > >>>>>>>
> > > >>>>>>>>>>>> I see these in your log too:
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> 1. 2009-11-11 04:27:19,018 DEBUG
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
> > > >>>>>>>>
> > > >>>>>>>>> Got
> > > >>>>>>>>>>
> > > >>>>>>>>>>> ping response for sessionid:0x424dfd908c50009 after
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>> 4544ms
> > > >>>
> > > >>>> 2. 2009-11-11 04:27:19,018 DEBUG
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
> > > >>>>>>>>
> > > >>>>>>>>> Got
> > > >>>>>>>>>>
> > > >>>>>>>>>>> ping response for sessionid:0x24dfd90c810002 after
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>> 4548ms
> > > >>>
> > > >>>> 3. 2009-11-11 04:27:43,960 DEBUG
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
> > > >>>>>>>>
> > > >>>>>>>>> Got
> > > >>>>>>>>>>
> > > >>>>>>>>>>> ping response for sessionid:0x424dfd908c50009 after
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>> 9030ms
> > > >>>
> > > >>>> 4. 2009-11-11 04:27:43,960 DEBUG
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>> org.apache.zookeeper.
> > > >>>>>>>>
> > > >>>>>>>
> > >
> >
>
Re: regionserver disconnection
Posted by Zhenyu Zhong <zh...@gmail.com>.
Stack,
You are right, the master started with ulimit -n 1024. It doesn't take the
system value.
Regarding to the too many open files, it looks like the same as the one J-D
put up. But I will get the Master start with higher value first and see if
this kind of error goes away.
thanks a lot!
zhenyu
On Fri, Nov 13, 2009 at 6:02 PM, stack <st...@duboce.net> wrote:
> Does it say
>
> ulimit -n 32768
>
> ...as the second line in your log file on start of the master?
>
> Is the stack trace that complains about too many open files same as the one
> in the blog post J-D put up?
>
> St.Ack
>
>
> On Fri, Nov 13, 2009 at 1:37 PM, Zhenyu Zhong <zhongresearch@gmail.com
> >wrote:
>
> > The ulimit file descriptors was set to fs.file-max =1578334, also in the
> > limits.conf the value was set to 32768.
> > So these are way higher than the open descriptors for the running
> > processes.
> >
> > thanks
> > zhenyu
> >
> >
> > On Fri, Nov 13, 2009 at 4:33 PM, Stack <sa...@gmail.com> wrote:
> >
> > > You upped the ulimit file descriptors as per the getting started doc?
> > >
> > >
> > >
> > > On Nov 13, 2009, at 1:26 PM, Zhenyu Zhong <zh...@gmail.com>
> > wrote:
> > >
> > > Thanks a lot.
> > >>
> > >>
> > >> Bad news is my kernel is still 2.6.26.
> > >> But it was not a problem before.
> > >>
> > >> Very strange.
> > >>
> > >> zhenyu
> > >>
> > >> On Fri, Nov 13, 2009 at 4:16 PM, Jean-Daniel Cryans <
> > jdcryans@apache.org
> > >> >wrote:
> > >>
> > >> Looks like
> > >>>
> > >>>
> >
> http://pero.blogs.aprilmayjune.org/2009/01/22/hadoop-and-linux-kernel-2627-epoll-limits/
> > >>>
> > >>> J-D
> > >>>
> > >>> On Fri, Nov 13, 2009 at 1:12 PM, Zhenyu Zhong <
> zhongresearch@gmail.com
> > >
> > >>> wrote:
> > >>>
> > >>>> Hi,
> > >>>>
> > >>>>
> > >>>> After I re-organize the cluster, the experiment ran into problem
> > faster
> > >>>>
> > >>> than
> > >>>
> > >>>> before.
> > >>>>
> > >>>> Basically the changes are to use machines with less resources as
> > >>>>
> > >>> zookeeper
> > >>>
> > >>>> quorums and machines with more resources as regionserver.
> > >>>>
> > >>>> From the log, I still see the pause around 1 minute.
> > >>>> I enabled the GC logging, please see
> > >>>>
> > >>>> http://pastebin.com/m1d4ce0f1
> > >>>>
> > >>>> for details.
> > >>>> In general I don't see many pauses in the GC.
> > >>>>
> > >>>> What is more interesting, 1 hour after the 1st regionserver
> > >>>> disconnected,
> > >>>> the master log complained about too many open files. This didn't
> > happen
> > >>>> before.
> > >>>> I checked the system OS setup as well as the limits.conf. I also
> > checked
> > >>>>
> > >>> the
> > >>>
> > >>>> running processes. The total open files don't reach the limit. I am
> > >>>>
> > >>> confused
> > >>>
> > >>>> a bit.
> > >>>>
> > >>>>
> > >>>> Please see the following master log.
> > >>>>
> > >>>> 2009-11-13 20:06:02,114 INFO
> > org.apache.hadoop.hbase.master.BaseScanner:
> > >>>> RegionManager.metaScanner scan of 4658 row(s) of meta region
> {server:
> > >>>> 192.168.100.128:60021, regionname: .META.,,1, startKey: <>}
> complete
> > >>>> 2009-11-13 20:06:02,114 INFO
> > org.apache.hadoop.hbase.master.BaseScanner:
> > >>>>
> > >>> All
> > >>>
> > >>>> 1 .META. region(s) scanned
> > >>>> 2009-11-13 20:06:07,677 DEBUG org.apache.zookeeper.ClientCnxn: Got
> > ping
> > >>>> response for sessionid:0x424eebf1c10004c after 3ms
> > >>>> 2009-11-13 20:06:08,178 INFO org.apache.hadoop.hdfs.DFSClient:
> > Exception
> > >>>>
> > >>> in
> > >>>
> > >>>> createBlockOutputStream java.io.IOException: Bad connect ack with
> > >>>> firstBadLink 192.168.100.123:50010
> > >>>> 2009-11-13 20:06:08,178 INFO org.apache.hadoop.hdfs.DFSClient:
> > >>>> Abandoning
> > >>>> block blk_-2808245019291145247_5478039
> > >>>> 2009-11-13 20:06:09,682 INFO org.apache.hadoop.hdfs.DFSClient:
> > Exception
> > >>>>
> > >>> in
> > >>>
> > >>>> createBlockOutputStream java.io.EOFException
> > >>>> 2009-11-13 20:06:09,682 INFO org.apache.hadoop.hdfs.DFSClient:
> > >>>> Abandoning
> > >>>> block blk_1074853606841896259_5478048
> > >>>> 2009-11-13 20:06:10,334 DEBUG org.apache.zookeeper.ClientCnxn: Got
> > ping
> > >>>> response for sessionid:0x24eebf1043003c after 1ms
> > >>>> 2009-11-13 20:06:21,018 DEBUG org.apache.zookeeper.ClientCnxn: Got
> > ping
> > >>>> response for sessionid:0x424eebf1c10004c after 0ms
> > >>>> 2009-11-13 20:06:23,674 DEBUG org.apache.zookeeper.ClientCnxn: Got
> > ping
> > >>>> response for sessionid:0x24eebf1043003c after 0ms
> > >>>> 2009-11-13 20:06:24,828 INFO org.apache.hadoop.hdfs.DFSClient:
> > Exception
> > >>>>
> > >>> in
> > >>>
> > >>>> createBlockOutputStream java.io.IOException: Bad connect ack with
> > >>>> firstBadLink 192.168.100.123:50010
> > >>>> 2009-11-13 20:06:24,828 INFO org.apache.hadoop.hdfs.DFSClient:
> > >>>> Abandoning
> > >>>> block blk_-6642544517082142289_5478063
> > >>>> 2009-11-13 20:06:24,828 INFO org.apache.hadoop.hdfs.DFSClient:
> > Exception
> > >>>>
> > >>> in
> > >>>
> > >>>> createBlockOutputStream java.net.SocketException: Too many open
> files
> > >>>> 2009-11-13 20:06:24,828 INFO org.apache.hadoop.hdfs.DFSClient:
> > >>>> Abandoning
> > >>>> block blk_2057511041109796090_5478063
> > >>>> 2009-11-13 20:06:24,928 INFO org.apache.hadoop.hdfs.DFSClient:
> > Exception
> > >>>>
> > >>> in
> > >>>
> > >>>> createBlockOutputStream java.net.SocketException: Too many open
> files
> > >>>> 2009-11-13 20:06:24,928 INFO org.apache.hadoop.hdfs.DFSClient:
> > >>>> Abandoning
> > >>>> block blk_8219260302213892894_5478064
> > >>>> 2009-11-13 20:06:30,855 INFO org.apache.hadoop.hdfs.DFSClient:
> > Exception
> > >>>>
> > >>> in
> > >>>
> > >>>> createBlockOutputStream java.net.SocketException: Too many open
> files
> > >>>> 2009-11-13 20:06:30,855 INFO org.apache.hadoop.hdfs.DFSClient:
> > >>>> Abandoning
> > >>>> block blk_1669205542853067709_5478235
> > >>>> 2009-11-13 20:06:30,905 INFO org.apache.hadoop.hdfs.DFSClient:
> > Exception
> > >>>>
> > >>> in
> > >>>
> > >>>> createBlockOutputStream java.net.SocketException: Too many open
> files
> > >>>> 2009-11-13 20:06:30,905 INFO org.apache.hadoop.hdfs.DFSClient:
> > >>>> Abandoning
> > >>>> block blk_9128897691346270351_5478237
> > >>>> 2009-11-13 20:06:30,955 INFO org.apache.hadoop.hdfs.DFSClient:
> > Exception
> > >>>>
> > >>> in
> > >>>
> > >>>> createBlockOutputStream java.net.SocketException: Too many open
> files
> > >>>> 2009-11-13 20:06:30,955 INFO org.apache.hadoop.hdfs.DFSClient:
> > >>>> Abandoning
> > >>>> block blk_1116845144864123018_5478240
> > >>>> 2009-11-13 20:06:34,372 DEBUG org.apache.zookeeper.ClientCnxn: Got
> > ping
> > >>>> response for sessionid:0x424eebf1c10004c after 0ms
> > >>>> 2009-11-13 20:06:37,034 DEBUG org.apache.zookeeper.ClientCnxn: Got
> > ping
> > >>>> response for sessionid:0x24eebf1043003c after 0ms
> > >>>> 2009-11-13 20:06:37,235 WARN org.apache.hadoop.hdfs.DFSClient:
> > >>>>
> > >>> DataStreamer
> > >>>
> > >>>> Exception: java.io.IOException: Too many open files
> > >>>> at sun.nio.ch.IOUtil.initPipe(Native Method)
> > >>>> at
> sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:49)
> > >>>> at
> > >>>>
> > >>>>
> >
> sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:18)
> > >>>
> > >>>> at
> > >>>>
> > >>>>
> >
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.get(SocketIOWithTimeout.java:407)
> > >>>
> > >>>> at
> > >>>>
> > >>>>
> >
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:322)
> > >>>
> > >>>> at
> > >>>>
> > >>>>
> >
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
> > >>>
> > >>>> at
> > >>>>
> > >>>>
> >
> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
> > >>>
> > >>>> at
> > >>>>
> > >>>>
> >
> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
> > >>>
> > >>>> at
> > >>>>
> > >>> java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
> > >>>
> > >>>> at java.io.DataOutputStream.write(DataOutputStream.java:90)
> > >>>> at
> > >>>>
> > >>>>
> >
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2290)
> > >>>
> > >>>>
> > >>>> 2009-11-13 20:06:37,235 WARN org.apache.hadoop.hdfs.DFSClient: Error
> > >>>> Recovery for block blk_8148813491785406356_5478475 bad datanode[0]
> > >>>> 192.168.100.123:50010
> > >>>> 2009-11-13 20:06:37,235 WARN org.apache.hadoop.hdfs.DFSClient: Error
> > >>>> Recovery for block blk_8148813491785406356_5478475 in pipeline
> > >>>> 192.168.100.123:50010, 192.168.100.134:50010, 192.168.100.122:50010
> :
> > >>>> bad
> > >>>> datanode 192.168.100.123:50010
> > >>>> 2009-11-13 20:06:37,436 INFO org.apache.hadoop.hdfs.DFSClient:
> > Exception
> > >>>>
> > >>> in
> > >>>
> > >>>> createBlockOutputStream java.net.SocketException: Too many open
> files
> > >>>> 2009-11-13 20:06:37,436 INFO org.apache.hadoop.hdfs.DFSClient:
> > >>>> Abandoning
> > >>>> block blk_2119727700857186236_5478498
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>> On Thu, Nov 12, 2009 at 4:21 PM, Zhenyu Zhong <
> > zhongresearch@gmail.com
> > >>>> wrote:
> > >>>>
> > >>>> Will do.
> > >>>>>
> > >>>>> thanks
> > >>>>> zhenyu
> > >>>>>
> > >>>>>
> > >>>>> On Thu, Nov 12, 2009 at 3:33 PM, stack <st...@duboce.net> wrote:
> > >>>>>
> > >>>>> Enable GC logging too on this next run (see hbase-env.sh). Lets
> try
> > >>>>>>
> > >>>>> and
> > >>>
> > >>>> get
> > >>>>>> to the bottom of whats going on.
> > >>>>>> Thanks,
> > >>>>>> St.Ack
> > >>>>>>
> > >>>>>> On Thu, Nov 12, 2009 at 12:29 PM, Zhenyu Zhong <
> > >>>>>>
> > >>>>> zhongresearch@gmail.com
> > >>>
> > >>>> wrote:
> > >>>>>>>
> > >>>>>>
> > >>>>>> I can switch the boxes that run zookeeper with the ones that run
> > >>>>>>> regionservers.
> > >>>>>>> I will see the results later.
> > >>>>>>>
> > >>>>>>> FYI. The node does have the 10 minutes zookeeper.session.timeout
> > >>>>>>>
> > >>>>>> value
> > >>>
> > >>>> in
> > >>>>>>
> > >>>>>>> place.
> > >>>>>>>
> > >>>>>>> thanks
> > >>>>>>> zhenyu
> > >>>>>>>
> > >>>>>>> On Thu, Nov 12, 2009 at 3:21 PM, stack <st...@duboce.net> wrote:
> > >>>>>>>
> > >>>>>>> On Thu, Nov 12, 2009 at 11:50 AM, Zhenyu Zhong <
> > >>>>>>>>
> > >>>>>>> zhongresearch@gmail.com
> > >>>>>>
> > >>>>>>> wrote:
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>> In my cluster, half of the cluster have 2 disks 400GB each per
> > >>>>>>>>>
> > >>>>>>>> machine,
> > >>>>>>
> > >>>>>>> and
> > >>>>>>>>
> > >>>>>>>>> half of the cluster have 6 disks per machine. Maybe we should
> > >>>>>>>>>
> > >>>>>>>> run
> > >>>
> > >>>> zookeeper
> > >>>>>>>>> on the machines with 2 disks and RS on machines with 6 disks?
> > >>>>>>>>>
> > >>>>>>>>> That would make most sense only in the below, it looks like the
> > >>>>>>>>>
> > >>>>>>>> RS
> > >>>
> > >>>> that
> > >>>>>>
> > >>>>>>> had
> > >>>>>>>> issue had 4 disks?
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> BTW, the 10 minutes zookeeper.session.timeout has been set
> during
> > >>>>>>>>>
> > >>>>>>>> the
> > >>>>>>
> > >>>>>>> experiment.
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>> And for sure this node had it in place?
> > >>>>>>>> St.Ack
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>> thanks
> > >>>>>>>>> zhenyu
> > >>>>>>>>>
> > >>>>>>>>> On Thu, Nov 12, 2009 at 2:08 PM, stack <st...@duboce.net>
> wrote:
> > >>>>>>>>>
> > >>>>>>>>> On Thu, Nov 12, 2009 at 8:40 AM, Zhenyu Zhong <
> > >>>>>>>>>>
> > >>>>>>>>> zhongresearch@gmail.com
> > >>>>>>>
> > >>>>>>>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> Though I experienced 2 regionserver disconnection this
> > >>>>>>>>>>>
> > >>>>>>>>>> morning,
> > >>>
> > >>>> it
> > >>>>>>
> > >>>>>>> looks
> > >>>>>>>>>
> > >>>>>>>>>> better from the regionserver log. (please see the following
> > >>>>>>>>>>>
> > >>>>>>>>>> log)
> > >>>
> > >>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> http://pastebin.com/m496dbfae
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> I checked diskIO for one of the regionserver(192.168.100.116)
> > >>>>>>>>>>>
> > >>>>>>>>>> during
> > >>>>>>>
> > >>>>>>>> the
> > >>>>>>>>>
> > >>>>>>>>>> time it disconnected.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Time: 03:04:58 AM
> > >>>>>>>>>>> Device: tps Blk_read/s Blk_wrtn/s Blk_read
> > >>>>>>>>>>>
> > >>>>>>>>>> Blk_wrtn
> > >>>>>>>>
> > >>>>>>>>> sda 105.31 5458.83 19503.64 9043873239
> > >>>>>>>>>>>
> > >>>>>>>>>> 32312473676
> > >>>>>>>>
> > >>>>>>>>> sda1 2.90 6.64 99.25 10993934
> > >>>>>>>>>>>
> > >>>>>>>>>> 164433464
> > >>>>>>>>
> > >>>>>>>>> sda2 1.72 23.76 30.25 39365817
> > >>>>>>>>>>>
> > >>>>>>>>>> 50124008
> > >>>>>>>>
> > >>>>>>>>> sda3 0.30 0.38 3.58 624930
> > >>>>>>>>>>>
> > >>>>>>>>>> 5923000
> > >>>>>>>>
> > >>>>>>>>> sda4 100.39 5428.06 19370.56 8992888270
> > >>>>>>>>>>>
> > >>>>>>>>>> 32091993204
> > >>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> Is this high for you? 20k blocks/second would seem to be
> high
> > >>>>>>>>>>
> > >>>>>>>>> but
> > >>>
> > >>>> its
> > >>>>>>>
> > >>>>>>>> one
> > >>>>>>>>>
> > >>>>>>>>>> disk only and its not being shared by zk anymore so shouldn't
> > >>>>>>>>>>
> > >>>>>>>>> matter?
> > >>>>>>
> > >>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> I also checked the zookeeper quorum server that the
> > >>>>>>>>>>>
> > >>>>>>>>>> regionserver
> > >>>
> > >>>> tried
> > >>>>>>>>
> > >>>>>>>>> to
> > >>>>>>>>>
> > >>>>>>>>>> connect according to the log. However, I don't see
> > >>>>>>>>>>>
> > >>>>>>>>>> 192.168.100.116
> > >>>>>>
> > >>>>>>> in
> > >>>>>>>
> > >>>>>>>> the
> > >>>>>>>>>
> > >>>>>>>>>> client list of the zookeeper quorum server that the
> > >>>>>>>>>>>
> > >>>>>>>>>> regionserver
> > >>>
> > >>>> tried
> > >>>>>>>>
> > >>>>>>>>> to
> > >>>>>>>>>
> > >>>>>>>>>> connect.
> > >>>>>>>>>>> Would that be a problem?
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> Is that because the ephemeral node for the regionserver had
> > >>>>>>>>>>
> > >>>>>>>>> evaporated?
> > >>>>>>>
> > >>>>>>>> Lost
> > >>>>>>>>>> its lease w/ zk by the time you went to look?
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> Thu Nov 12 15:04:52 UTC 2009
> > >>>>>>>>>>> Zookeeper version: 3.2.1-808558, built on 08/27/2009 18:48
> > >>>>>>>>>>>
> > >>>>>>>>>> GMT
> > >>>
> > >>>> Clients:
> > >>>>>>>>>>> /192.168.100.127:43045[1](queued=0,recved=26,sent=0)
> > >>>>>>>>>>> /192.168.100.131:39091[1](queued=0,recved=964,sent=0)
> > >>>>>>>>>>> /192.168.100.124:35961[1](queued=0,recved=3266,sent=0)
> > >>>>>>>>>>> /192.168.100.123:47935[1](queued=0,recved=233,sent=0)
> > >>>>>>>>>>> /192.168.100.125:46931[1](queued=0,recved=2,sent=0)
> > >>>>>>>>>>> /192.168.100.118:54924[1](queued=0,recved=295,sent=0)
> > >>>>>>>>>>> /192.168.100.118:41390[1](queued=0,recved=2290,sent=0)
> > >>>>>>>>>>> /192.168.100.136:42243[1](queued=0,recved=0,sent=0)
> > >>>>>>>>>>>
> > >>>>>>>>>>> Latency min/avg/max: 0/17/6333
> > >>>>>>>>>>> Received: 47111
> > >>>>>>>>>>> Sent: 0
> > >>>>>>>>>>> Outstanding: 0
> > >>>>>>>>>>> Zxid: 0x77000083f4
> > >>>>>>>>>>> Mode: leader
> > >>>>>>>>>>> Node count: 23
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> That 6 second maximum latency is pretty bad but should be well
> > >>>>>>>>>>
> > >>>>>>>>> within
> > >>>>>>
> > >>>>>>> the
> > >>>>>>>>
> > >>>>>>>>> zk
> > >>>>>>>>>> session timeout.
> > >>>>>>>>>>
> > >>>>>>>>>> So, problem is likely on the zk client side of the session;
> > >>>>>>>>>>
> > >>>>>>>>> i.e.
> > >>>
> > >>>> in
> > >>>>>>
> > >>>>>>> the
> > >>>>>>>
> > >>>>>>>> RS.
> > >>>>>>>>>
> > >>>>>>>>>> You could enable GC logging as J-D suggested to see if you
> have
> > >>>>>>>>>>
> > >>>>>>>>> any
> > >>>>>>
> > >>>>>>> big
> > >>>>>>>
> > >>>>>>>> pauses, pauses > zk session timeout.
> > >>>>>>>>>>
> > >>>>>>>>>> When the RS went down, it didn't look too heavily loaded:
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> 1. 2009-11-12 15:04:52,830 INFO
> > >>>>>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of
> > >>>>>>>>>>
> > >>>>>>>>> metrics:
> > >>>>>>>
> > >>>>>>>> request=1.5166667, regions=322, stores=657, storefiles=631,
> > >>>>>>>>>> storefileIndexSize=61, memstoreSize=1472, usedHeap=2819,
> > >>>>>>>>>>
> > >>>>>>>>> maxHeap=4079,
> > >>>>>>>>
> > >>>>>>>>> blockCacheSize=658110960, blockCacheFree=197395984,
> > >>>>>>>>>>
> > >>>>>>>>> blockCacheCount=9903,
> > >>>>>>>>>
> > >>>>>>>>>> blockCacheHitRatio=99
> > >>>>>>>>>>
> > >>>>>>>>>> Its serving a few reads? The number of store files seems
> fine.
> > >>>>>>>>>>
> > >>>>>>>>> Not
> > >>>>>>
> > >>>>>>> too
> > >>>>>>>>
> > >>>>>>>>> much memory used.
> > >>>>>>>>>>
> > >>>>>>>>>> Looking at the logs, I see the Lease Still Held exception.
> > >>>>>>>>>>
> > >>>>>>>>> This
> > >>>
> > >>>> happens
> > >>>>>>>>
> > >>>>>>>>> when the RS does its regular report to the master but the
> > >>>>>>>>>>
> > >>>>>>>>> master
> > >>>
> > >>>> thinks
> > >>>>>>>
> > >>>>>>>> the
> > >>>>>>>>>
> > >>>>>>>>>> RS has since restarted. It'll think this probably because it
> > >>>>>>>>>>
> > >>>>>>>>> noticed
> > >>>>>>
> > >>>>>>> that
> > >>>>>>>>>
> > >>>>>>>>>> the RS's znode in zk had gone away and it considered the RS
> > >>>>>>>>>>
> > >>>>>>>>> dead.
> > >>>
> > >>>>
> > >>>>>>>>>> Looking too at your logs I see this gap in the zk pinging:
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> 1. 2009-11-12 15:03:39,325 DEBUG
> > >>>>>>>>>>
> > >>>>>>>>> org.apache.zookeeper.ClientCnxn:
> > >>>>>>
> > >>>>>>> Got
> > >>>>>>>
> > >>>>>>>> ping response for sessionid:0x224e55436ad0004 after 0ms
> > >>>>>>>>>> 2. 2009-11-12 15:03:43,113 DEBUG
> > >>>>>>>>>>
> > >>>>>>>>> org.apache.zookeeper.ClientCnxn:
> > >>>>>>
> > >>>>>>> Got
> > >>>>>>>
> > >>>>>>>> ping response for sessionid:0x24e55436a0007d after 0ms
> > >>>>>>>>>>
> > >>>>>>>>>> Where in the lines above it, its reporting about every ten
> > >>>>>>>>>>
> > >>>>>>>>> seconds,
> > >>>>>>
> > >>>>>>> here
> > >>>>>>>>
> > >>>>>>>>> there is a big gap.
> > >>>>>>>>>>
> > >>>>>>>>>> Do you have ganglia or something that will let you look more
> > >>>>>>>>>>
> > >>>>>>>>> into
> > >>>
> > >>>> what
> > >>>>>>>
> > >>>>>>>> was
> > >>>>>>>>>
> > >>>>>>>>>> happening on this machine around this time? Is the machine
> OK?
> > >>>>>>>>>>
> > >>>>>>>>> It
> > >>>>>>
> > >>>>>>> looks
> > >>>>>>>>
> > >>>>>>>>> lightly loaded and you have your cluster nicely laid out.
> > >>>>>>>>>>
> > >>>>>>>>> Something
> > >>>>>>
> > >>>>>>> odd
> > >>>>>>>>
> > >>>>>>>>> is
> > >>>>>>>>>
> > >>>>>>>>>> going on. What about things like the write speed to disk? In
> > >>>>>>>>>>
> > >>>>>>>>> the
> > >>>
> > >>>> past
> > >>>>>>>
> > >>>>>>>> strange issues have been explained by incorrectly set BIOS
> > >>>>>>>>>>
> > >>>>>>>>> which
> > >>>
> > >>>> made
> > >>>>>>
> > >>>>>>> disks
> > >>>>>>>>>
> > >>>>>>>>>> run at 1/100th of their proper speed.
> > >>>>>>>>>>
> > >>>>>>>>>> St.Ack
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>> Best,
> > >>>>>>>>>>> zhenyu
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> On Wed, Nov 11, 2009 at 3:58 PM, Zhenyu Zhong <
> > >>>>>>>>>>>
> > >>>>>>>>>> zhongresearch@gmail.com
> > >>>>>>>>
> > >>>>>>>>> wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> Stack
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> I am very appreciated for your comments.
> > >>>>>>>>>>>> I will use the zookeeper monitoring script on my cluster
> > >>>>>>>>>>>>
> > >>>>>>>>>>> and
> > >>>
> > >>>> let
> > >>>>>>
> > >>>>>>> it
> > >>>>>>>
> > >>>>>>>> run
> > >>>>>>>>>
> > >>>>>>>>>> overnight to see the result.
> > >>>>>>>>>>>> I will follow up the thread when I get anything.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> thanks
> > >>>>>>>>>>>> zhenyu
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> On Wed, Nov 11, 2009 at 3:52 PM, stack <st...@duboce.net>
> > >>>>>>>>>>>>
> > >>>>>>>>>>> wrote:
> > >>>>>>
> > >>>>>>>
> > >>>>>>>>>>>> I see these in your log too:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> 1. 2009-11-11 04:27:19,018 DEBUG
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
> > >>>>>>>>
> > >>>>>>>>> Got
> > >>>>>>>>>>
> > >>>>>>>>>>> ping response for sessionid:0x424dfd908c50009 after
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>> 4544ms
> > >>>
> > >>>> 2. 2009-11-11 04:27:19,018 DEBUG
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
> > >>>>>>>>
> > >>>>>>>>> Got
> > >>>>>>>>>>
> > >>>>>>>>>>> ping response for sessionid:0x24dfd90c810002 after
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>> 4548ms
> > >>>
> > >>>> 3. 2009-11-11 04:27:43,960 DEBUG
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
> > >>>>>>>>
> > >>>>>>>>> Got
> > >>>>>>>>>>
> > >>>>>>>>>>> ping response for sessionid:0x424dfd908c50009 after
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>> 9030ms
> > >>>
> > >>>> 4. 2009-11-11 04:27:43,960 DEBUG
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>> org.apache.zookeeper.
> > >>>>>>>>
> > >>>>>>>
> >
>
Re: regionserver disconnection
Posted by stack <st...@duboce.net>.
Does it say
ulimit -n 32768
...as the second line in your log file on start of the master?
Is the stack trace that complains about too many open files same as the one
in the blog post J-D put up?
St.Ack
On Fri, Nov 13, 2009 at 1:37 PM, Zhenyu Zhong <zh...@gmail.com>wrote:
> The ulimit file descriptors was set to fs.file-max =1578334, also in the
> limits.conf the value was set to 32768.
> So these are way higher than the open descriptors for the running
> processes.
>
> thanks
> zhenyu
>
>
> On Fri, Nov 13, 2009 at 4:33 PM, Stack <sa...@gmail.com> wrote:
>
> > You upped the ulimit file descriptors as per the getting started doc?
> >
> >
> >
> > On Nov 13, 2009, at 1:26 PM, Zhenyu Zhong <zh...@gmail.com>
> wrote:
> >
> > Thanks a lot.
> >>
> >>
> >> Bad news is my kernel is still 2.6.26.
> >> But it was not a problem before.
> >>
> >> Very strange.
> >>
> >> zhenyu
> >>
> >> On Fri, Nov 13, 2009 at 4:16 PM, Jean-Daniel Cryans <
> jdcryans@apache.org
> >> >wrote:
> >>
> >> Looks like
> >>>
> >>>
> http://pero.blogs.aprilmayjune.org/2009/01/22/hadoop-and-linux-kernel-2627-epoll-limits/
> >>>
> >>> J-D
> >>>
> >>> On Fri, Nov 13, 2009 at 1:12 PM, Zhenyu Zhong <zhongresearch@gmail.com
> >
> >>> wrote:
> >>>
> >>>> Hi,
> >>>>
> >>>>
> >>>> After I re-organize the cluster, the experiment ran into problem
> faster
> >>>>
> >>> than
> >>>
> >>>> before.
> >>>>
> >>>> Basically the changes are to use machines with less resources as
> >>>>
> >>> zookeeper
> >>>
> >>>> quorums and machines with more resources as regionserver.
> >>>>
> >>>> From the log, I still see the pause around 1 minute.
> >>>> I enabled the GC logging, please see
> >>>>
> >>>> http://pastebin.com/m1d4ce0f1
> >>>>
> >>>> for details.
> >>>> In general I don't see many pauses in the GC.
> >>>>
> >>>> What is more interesting, 1 hour after the 1st regionserver
> >>>> disconnected,
> >>>> the master log complained about too many open files. This didn't
> happen
> >>>> before.
> >>>> I checked the system OS setup as well as the limits.conf. I also
> checked
> >>>>
> >>> the
> >>>
> >>>> running processes. The total open files don't reach the limit. I am
> >>>>
> >>> confused
> >>>
> >>>> a bit.
> >>>>
> >>>>
> >>>> Please see the following master log.
> >>>>
> >>>> 2009-11-13 20:06:02,114 INFO
> org.apache.hadoop.hbase.master.BaseScanner:
> >>>> RegionManager.metaScanner scan of 4658 row(s) of meta region {server:
> >>>> 192.168.100.128:60021, regionname: .META.,,1, startKey: <>} complete
> >>>> 2009-11-13 20:06:02,114 INFO
> org.apache.hadoop.hbase.master.BaseScanner:
> >>>>
> >>> All
> >>>
> >>>> 1 .META. region(s) scanned
> >>>> 2009-11-13 20:06:07,677 DEBUG org.apache.zookeeper.ClientCnxn: Got
> ping
> >>>> response for sessionid:0x424eebf1c10004c after 3ms
> >>>> 2009-11-13 20:06:08,178 INFO org.apache.hadoop.hdfs.DFSClient:
> Exception
> >>>>
> >>> in
> >>>
> >>>> createBlockOutputStream java.io.IOException: Bad connect ack with
> >>>> firstBadLink 192.168.100.123:50010
> >>>> 2009-11-13 20:06:08,178 INFO org.apache.hadoop.hdfs.DFSClient:
> >>>> Abandoning
> >>>> block blk_-2808245019291145247_5478039
> >>>> 2009-11-13 20:06:09,682 INFO org.apache.hadoop.hdfs.DFSClient:
> Exception
> >>>>
> >>> in
> >>>
> >>>> createBlockOutputStream java.io.EOFException
> >>>> 2009-11-13 20:06:09,682 INFO org.apache.hadoop.hdfs.DFSClient:
> >>>> Abandoning
> >>>> block blk_1074853606841896259_5478048
> >>>> 2009-11-13 20:06:10,334 DEBUG org.apache.zookeeper.ClientCnxn: Got
> ping
> >>>> response for sessionid:0x24eebf1043003c after 1ms
> >>>> 2009-11-13 20:06:21,018 DEBUG org.apache.zookeeper.ClientCnxn: Got
> ping
> >>>> response for sessionid:0x424eebf1c10004c after 0ms
> >>>> 2009-11-13 20:06:23,674 DEBUG org.apache.zookeeper.ClientCnxn: Got
> ping
> >>>> response for sessionid:0x24eebf1043003c after 0ms
> >>>> 2009-11-13 20:06:24,828 INFO org.apache.hadoop.hdfs.DFSClient:
> Exception
> >>>>
> >>> in
> >>>
> >>>> createBlockOutputStream java.io.IOException: Bad connect ack with
> >>>> firstBadLink 192.168.100.123:50010
> >>>> 2009-11-13 20:06:24,828 INFO org.apache.hadoop.hdfs.DFSClient:
> >>>> Abandoning
> >>>> block blk_-6642544517082142289_5478063
> >>>> 2009-11-13 20:06:24,828 INFO org.apache.hadoop.hdfs.DFSClient:
> Exception
> >>>>
> >>> in
> >>>
> >>>> createBlockOutputStream java.net.SocketException: Too many open files
> >>>> 2009-11-13 20:06:24,828 INFO org.apache.hadoop.hdfs.DFSClient:
> >>>> Abandoning
> >>>> block blk_2057511041109796090_5478063
> >>>> 2009-11-13 20:06:24,928 INFO org.apache.hadoop.hdfs.DFSClient:
> Exception
> >>>>
> >>> in
> >>>
> >>>> createBlockOutputStream java.net.SocketException: Too many open files
> >>>> 2009-11-13 20:06:24,928 INFO org.apache.hadoop.hdfs.DFSClient:
> >>>> Abandoning
> >>>> block blk_8219260302213892894_5478064
> >>>> 2009-11-13 20:06:30,855 INFO org.apache.hadoop.hdfs.DFSClient:
> Exception
> >>>>
> >>> in
> >>>
> >>>> createBlockOutputStream java.net.SocketException: Too many open files
> >>>> 2009-11-13 20:06:30,855 INFO org.apache.hadoop.hdfs.DFSClient:
> >>>> Abandoning
> >>>> block blk_1669205542853067709_5478235
> >>>> 2009-11-13 20:06:30,905 INFO org.apache.hadoop.hdfs.DFSClient:
> Exception
> >>>>
> >>> in
> >>>
> >>>> createBlockOutputStream java.net.SocketException: Too many open files
> >>>> 2009-11-13 20:06:30,905 INFO org.apache.hadoop.hdfs.DFSClient:
> >>>> Abandoning
> >>>> block blk_9128897691346270351_5478237
> >>>> 2009-11-13 20:06:30,955 INFO org.apache.hadoop.hdfs.DFSClient:
> Exception
> >>>>
> >>> in
> >>>
> >>>> createBlockOutputStream java.net.SocketException: Too many open files
> >>>> 2009-11-13 20:06:30,955 INFO org.apache.hadoop.hdfs.DFSClient:
> >>>> Abandoning
> >>>> block blk_1116845144864123018_5478240
> >>>> 2009-11-13 20:06:34,372 DEBUG org.apache.zookeeper.ClientCnxn: Got
> ping
> >>>> response for sessionid:0x424eebf1c10004c after 0ms
> >>>> 2009-11-13 20:06:37,034 DEBUG org.apache.zookeeper.ClientCnxn: Got
> ping
> >>>> response for sessionid:0x24eebf1043003c after 0ms
> >>>> 2009-11-13 20:06:37,235 WARN org.apache.hadoop.hdfs.DFSClient:
> >>>>
> >>> DataStreamer
> >>>
> >>>> Exception: java.io.IOException: Too many open files
> >>>> at sun.nio.ch.IOUtil.initPipe(Native Method)
> >>>> at sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:49)
> >>>> at
> >>>>
> >>>>
> sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:18)
> >>>
> >>>> at
> >>>>
> >>>>
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.get(SocketIOWithTimeout.java:407)
> >>>
> >>>> at
> >>>>
> >>>>
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:322)
> >>>
> >>>> at
> >>>>
> >>>>
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
> >>>
> >>>> at
> >>>>
> >>>>
> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
> >>>
> >>>> at
> >>>>
> >>>>
> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
> >>>
> >>>> at
> >>>>
> >>> java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
> >>>
> >>>> at java.io.DataOutputStream.write(DataOutputStream.java:90)
> >>>> at
> >>>>
> >>>>
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2290)
> >>>
> >>>>
> >>>> 2009-11-13 20:06:37,235 WARN org.apache.hadoop.hdfs.DFSClient: Error
> >>>> Recovery for block blk_8148813491785406356_5478475 bad datanode[0]
> >>>> 192.168.100.123:50010
> >>>> 2009-11-13 20:06:37,235 WARN org.apache.hadoop.hdfs.DFSClient: Error
> >>>> Recovery for block blk_8148813491785406356_5478475 in pipeline
> >>>> 192.168.100.123:50010, 192.168.100.134:50010, 192.168.100.122:50010:
> >>>> bad
> >>>> datanode 192.168.100.123:50010
> >>>> 2009-11-13 20:06:37,436 INFO org.apache.hadoop.hdfs.DFSClient:
> Exception
> >>>>
> >>> in
> >>>
> >>>> createBlockOutputStream java.net.SocketException: Too many open files
> >>>> 2009-11-13 20:06:37,436 INFO org.apache.hadoop.hdfs.DFSClient:
> >>>> Abandoning
> >>>> block blk_2119727700857186236_5478498
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> On Thu, Nov 12, 2009 at 4:21 PM, Zhenyu Zhong <
> zhongresearch@gmail.com
> >>>> wrote:
> >>>>
> >>>> Will do.
> >>>>>
> >>>>> thanks
> >>>>> zhenyu
> >>>>>
> >>>>>
> >>>>> On Thu, Nov 12, 2009 at 3:33 PM, stack <st...@duboce.net> wrote:
> >>>>>
> >>>>> Enable GC logging too on this next run (see hbase-env.sh). Lets try
> >>>>>>
> >>>>> and
> >>>
> >>>> get
> >>>>>> to the bottom of whats going on.
> >>>>>> Thanks,
> >>>>>> St.Ack
> >>>>>>
> >>>>>> On Thu, Nov 12, 2009 at 12:29 PM, Zhenyu Zhong <
> >>>>>>
> >>>>> zhongresearch@gmail.com
> >>>
> >>>> wrote:
> >>>>>>>
> >>>>>>
> >>>>>> I can switch the boxes that run zookeeper with the ones that run
> >>>>>>> regionservers.
> >>>>>>> I will see the results later.
> >>>>>>>
> >>>>>>> FYI. The node does have the 10 minutes zookeeper.session.timeout
> >>>>>>>
> >>>>>> value
> >>>
> >>>> in
> >>>>>>
> >>>>>>> place.
> >>>>>>>
> >>>>>>> thanks
> >>>>>>> zhenyu
> >>>>>>>
> >>>>>>> On Thu, Nov 12, 2009 at 3:21 PM, stack <st...@duboce.net> wrote:
> >>>>>>>
> >>>>>>> On Thu, Nov 12, 2009 at 11:50 AM, Zhenyu Zhong <
> >>>>>>>>
> >>>>>>> zhongresearch@gmail.com
> >>>>>>
> >>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> In my cluster, half of the cluster have 2 disks 400GB each per
> >>>>>>>>>
> >>>>>>>> machine,
> >>>>>>
> >>>>>>> and
> >>>>>>>>
> >>>>>>>>> half of the cluster have 6 disks per machine. Maybe we should
> >>>>>>>>>
> >>>>>>>> run
> >>>
> >>>> zookeeper
> >>>>>>>>> on the machines with 2 disks and RS on machines with 6 disks?
> >>>>>>>>>
> >>>>>>>>> That would make most sense only in the below, it looks like the
> >>>>>>>>>
> >>>>>>>> RS
> >>>
> >>>> that
> >>>>>>
> >>>>>>> had
> >>>>>>>> issue had 4 disks?
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> BTW, the 10 minutes zookeeper.session.timeout has been set during
> >>>>>>>>>
> >>>>>>>> the
> >>>>>>
> >>>>>>> experiment.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>> And for sure this node had it in place?
> >>>>>>>> St.Ack
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> thanks
> >>>>>>>>> zhenyu
> >>>>>>>>>
> >>>>>>>>> On Thu, Nov 12, 2009 at 2:08 PM, stack <st...@duboce.net> wrote:
> >>>>>>>>>
> >>>>>>>>> On Thu, Nov 12, 2009 at 8:40 AM, Zhenyu Zhong <
> >>>>>>>>>>
> >>>>>>>>> zhongresearch@gmail.com
> >>>>>>>
> >>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Though I experienced 2 regionserver disconnection this
> >>>>>>>>>>>
> >>>>>>>>>> morning,
> >>>
> >>>> it
> >>>>>>
> >>>>>>> looks
> >>>>>>>>>
> >>>>>>>>>> better from the regionserver log. (please see the following
> >>>>>>>>>>>
> >>>>>>>>>> log)
> >>>
> >>>>
> >>>>>>>>>>>
> >>>>>>>>>>> http://pastebin.com/m496dbfae
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> I checked diskIO for one of the regionserver(192.168.100.116)
> >>>>>>>>>>>
> >>>>>>>>>> during
> >>>>>>>
> >>>>>>>> the
> >>>>>>>>>
> >>>>>>>>>> time it disconnected.
> >>>>>>>>>>>
> >>>>>>>>>>> Time: 03:04:58 AM
> >>>>>>>>>>> Device: tps Blk_read/s Blk_wrtn/s Blk_read
> >>>>>>>>>>>
> >>>>>>>>>> Blk_wrtn
> >>>>>>>>
> >>>>>>>>> sda 105.31 5458.83 19503.64 9043873239
> >>>>>>>>>>>
> >>>>>>>>>> 32312473676
> >>>>>>>>
> >>>>>>>>> sda1 2.90 6.64 99.25 10993934
> >>>>>>>>>>>
> >>>>>>>>>> 164433464
> >>>>>>>>
> >>>>>>>>> sda2 1.72 23.76 30.25 39365817
> >>>>>>>>>>>
> >>>>>>>>>> 50124008
> >>>>>>>>
> >>>>>>>>> sda3 0.30 0.38 3.58 624930
> >>>>>>>>>>>
> >>>>>>>>>> 5923000
> >>>>>>>>
> >>>>>>>>> sda4 100.39 5428.06 19370.56 8992888270
> >>>>>>>>>>>
> >>>>>>>>>> 32091993204
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Is this high for you? 20k blocks/second would seem to be high
> >>>>>>>>>>
> >>>>>>>>> but
> >>>
> >>>> its
> >>>>>>>
> >>>>>>>> one
> >>>>>>>>>
> >>>>>>>>>> disk only and its not being shared by zk anymore so shouldn't
> >>>>>>>>>>
> >>>>>>>>> matter?
> >>>>>>
> >>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> I also checked the zookeeper quorum server that the
> >>>>>>>>>>>
> >>>>>>>>>> regionserver
> >>>
> >>>> tried
> >>>>>>>>
> >>>>>>>>> to
> >>>>>>>>>
> >>>>>>>>>> connect according to the log. However, I don't see
> >>>>>>>>>>>
> >>>>>>>>>> 192.168.100.116
> >>>>>>
> >>>>>>> in
> >>>>>>>
> >>>>>>>> the
> >>>>>>>>>
> >>>>>>>>>> client list of the zookeeper quorum server that the
> >>>>>>>>>>>
> >>>>>>>>>> regionserver
> >>>
> >>>> tried
> >>>>>>>>
> >>>>>>>>> to
> >>>>>>>>>
> >>>>>>>>>> connect.
> >>>>>>>>>>> Would that be a problem?
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Is that because the ephemeral node for the regionserver had
> >>>>>>>>>>
> >>>>>>>>> evaporated?
> >>>>>>>
> >>>>>>>> Lost
> >>>>>>>>>> its lease w/ zk by the time you went to look?
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Thu Nov 12 15:04:52 UTC 2009
> >>>>>>>>>>> Zookeeper version: 3.2.1-808558, built on 08/27/2009 18:48
> >>>>>>>>>>>
> >>>>>>>>>> GMT
> >>>
> >>>> Clients:
> >>>>>>>>>>> /192.168.100.127:43045[1](queued=0,recved=26,sent=0)
> >>>>>>>>>>> /192.168.100.131:39091[1](queued=0,recved=964,sent=0)
> >>>>>>>>>>> /192.168.100.124:35961[1](queued=0,recved=3266,sent=0)
> >>>>>>>>>>> /192.168.100.123:47935[1](queued=0,recved=233,sent=0)
> >>>>>>>>>>> /192.168.100.125:46931[1](queued=0,recved=2,sent=0)
> >>>>>>>>>>> /192.168.100.118:54924[1](queued=0,recved=295,sent=0)
> >>>>>>>>>>> /192.168.100.118:41390[1](queued=0,recved=2290,sent=0)
> >>>>>>>>>>> /192.168.100.136:42243[1](queued=0,recved=0,sent=0)
> >>>>>>>>>>>
> >>>>>>>>>>> Latency min/avg/max: 0/17/6333
> >>>>>>>>>>> Received: 47111
> >>>>>>>>>>> Sent: 0
> >>>>>>>>>>> Outstanding: 0
> >>>>>>>>>>> Zxid: 0x77000083f4
> >>>>>>>>>>> Mode: leader
> >>>>>>>>>>> Node count: 23
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> That 6 second maximum latency is pretty bad but should be well
> >>>>>>>>>>
> >>>>>>>>> within
> >>>>>>
> >>>>>>> the
> >>>>>>>>
> >>>>>>>>> zk
> >>>>>>>>>> session timeout.
> >>>>>>>>>>
> >>>>>>>>>> So, problem is likely on the zk client side of the session;
> >>>>>>>>>>
> >>>>>>>>> i.e.
> >>>
> >>>> in
> >>>>>>
> >>>>>>> the
> >>>>>>>
> >>>>>>>> RS.
> >>>>>>>>>
> >>>>>>>>>> You could enable GC logging as J-D suggested to see if you have
> >>>>>>>>>>
> >>>>>>>>> any
> >>>>>>
> >>>>>>> big
> >>>>>>>
> >>>>>>>> pauses, pauses > zk session timeout.
> >>>>>>>>>>
> >>>>>>>>>> When the RS went down, it didn't look too heavily loaded:
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> 1. 2009-11-12 15:04:52,830 INFO
> >>>>>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of
> >>>>>>>>>>
> >>>>>>>>> metrics:
> >>>>>>>
> >>>>>>>> request=1.5166667, regions=322, stores=657, storefiles=631,
> >>>>>>>>>> storefileIndexSize=61, memstoreSize=1472, usedHeap=2819,
> >>>>>>>>>>
> >>>>>>>>> maxHeap=4079,
> >>>>>>>>
> >>>>>>>>> blockCacheSize=658110960, blockCacheFree=197395984,
> >>>>>>>>>>
> >>>>>>>>> blockCacheCount=9903,
> >>>>>>>>>
> >>>>>>>>>> blockCacheHitRatio=99
> >>>>>>>>>>
> >>>>>>>>>> Its serving a few reads? The number of store files seems fine.
> >>>>>>>>>>
> >>>>>>>>> Not
> >>>>>>
> >>>>>>> too
> >>>>>>>>
> >>>>>>>>> much memory used.
> >>>>>>>>>>
> >>>>>>>>>> Looking at the logs, I see the Lease Still Held exception.
> >>>>>>>>>>
> >>>>>>>>> This
> >>>
> >>>> happens
> >>>>>>>>
> >>>>>>>>> when the RS does its regular report to the master but the
> >>>>>>>>>>
> >>>>>>>>> master
> >>>
> >>>> thinks
> >>>>>>>
> >>>>>>>> the
> >>>>>>>>>
> >>>>>>>>>> RS has since restarted. It'll think this probably because it
> >>>>>>>>>>
> >>>>>>>>> noticed
> >>>>>>
> >>>>>>> that
> >>>>>>>>>
> >>>>>>>>>> the RS's znode in zk had gone away and it considered the RS
> >>>>>>>>>>
> >>>>>>>>> dead.
> >>>
> >>>>
> >>>>>>>>>> Looking too at your logs I see this gap in the zk pinging:
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> 1. 2009-11-12 15:03:39,325 DEBUG
> >>>>>>>>>>
> >>>>>>>>> org.apache.zookeeper.ClientCnxn:
> >>>>>>
> >>>>>>> Got
> >>>>>>>
> >>>>>>>> ping response for sessionid:0x224e55436ad0004 after 0ms
> >>>>>>>>>> 2. 2009-11-12 15:03:43,113 DEBUG
> >>>>>>>>>>
> >>>>>>>>> org.apache.zookeeper.ClientCnxn:
> >>>>>>
> >>>>>>> Got
> >>>>>>>
> >>>>>>>> ping response for sessionid:0x24e55436a0007d after 0ms
> >>>>>>>>>>
> >>>>>>>>>> Where in the lines above it, its reporting about every ten
> >>>>>>>>>>
> >>>>>>>>> seconds,
> >>>>>>
> >>>>>>> here
> >>>>>>>>
> >>>>>>>>> there is a big gap.
> >>>>>>>>>>
> >>>>>>>>>> Do you have ganglia or something that will let you look more
> >>>>>>>>>>
> >>>>>>>>> into
> >>>
> >>>> what
> >>>>>>>
> >>>>>>>> was
> >>>>>>>>>
> >>>>>>>>>> happening on this machine around this time? Is the machine OK?
> >>>>>>>>>>
> >>>>>>>>> It
> >>>>>>
> >>>>>>> looks
> >>>>>>>>
> >>>>>>>>> lightly loaded and you have your cluster nicely laid out.
> >>>>>>>>>>
> >>>>>>>>> Something
> >>>>>>
> >>>>>>> odd
> >>>>>>>>
> >>>>>>>>> is
> >>>>>>>>>
> >>>>>>>>>> going on. What about things like the write speed to disk? In
> >>>>>>>>>>
> >>>>>>>>> the
> >>>
> >>>> past
> >>>>>>>
> >>>>>>>> strange issues have been explained by incorrectly set BIOS
> >>>>>>>>>>
> >>>>>>>>> which
> >>>
> >>>> made
> >>>>>>
> >>>>>>> disks
> >>>>>>>>>
> >>>>>>>>>> run at 1/100th of their proper speed.
> >>>>>>>>>>
> >>>>>>>>>> St.Ack
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>> Best,
> >>>>>>>>>>> zhenyu
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> On Wed, Nov 11, 2009 at 3:58 PM, Zhenyu Zhong <
> >>>>>>>>>>>
> >>>>>>>>>> zhongresearch@gmail.com
> >>>>>>>>
> >>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Stack
> >>>>>>>>>>>>
> >>>>>>>>>>>> I am very appreciated for your comments.
> >>>>>>>>>>>> I will use the zookeeper monitoring script on my cluster
> >>>>>>>>>>>>
> >>>>>>>>>>> and
> >>>
> >>>> let
> >>>>>>
> >>>>>>> it
> >>>>>>>
> >>>>>>>> run
> >>>>>>>>>
> >>>>>>>>>> overnight to see the result.
> >>>>>>>>>>>> I will follow up the thread when I get anything.
> >>>>>>>>>>>>
> >>>>>>>>>>>> thanks
> >>>>>>>>>>>> zhenyu
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Wed, Nov 11, 2009 at 3:52 PM, stack <st...@duboce.net>
> >>>>>>>>>>>>
> >>>>>>>>>>> wrote:
> >>>>>>
> >>>>>>>
> >>>>>>>>>>>> I see these in your log too:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> 1. 2009-11-11 04:27:19,018 DEBUG
> >>>>>>>>>>>>>
> >>>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
> >>>>>>>>
> >>>>>>>>> Got
> >>>>>>>>>>
> >>>>>>>>>>> ping response for sessionid:0x424dfd908c50009 after
> >>>>>>>>>>>>>
> >>>>>>>>>>>> 4544ms
> >>>
> >>>> 2. 2009-11-11 04:27:19,018 DEBUG
> >>>>>>>>>>>>>
> >>>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
> >>>>>>>>
> >>>>>>>>> Got
> >>>>>>>>>>
> >>>>>>>>>>> ping response for sessionid:0x24dfd90c810002 after
> >>>>>>>>>>>>>
> >>>>>>>>>>>> 4548ms
> >>>
> >>>> 3. 2009-11-11 04:27:43,960 DEBUG
> >>>>>>>>>>>>>
> >>>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
> >>>>>>>>
> >>>>>>>>> Got
> >>>>>>>>>>
> >>>>>>>>>>> ping response for sessionid:0x424dfd908c50009 after
> >>>>>>>>>>>>>
> >>>>>>>>>>>> 9030ms
> >>>
> >>>> 4. 2009-11-11 04:27:43,960 DEBUG
> >>>>>>>>>>>>>
> >>>>>>>>>>>> org.apache.zookeeper.
> >>>>>>>>
> >>>>>>>
>
Re: regionserver disconnection
Posted by Jean-Daniel Cryans <jd...@apache.org>.
AFAIK there's no requirement on the kernel version, 2.6.27 is the only
one that had that restriction and it was gone in the next version.
We do have a requirement for java6 per
http://hadoop.apache.org/hbase/docs/r0.20.1/api/overview-summary.html#overview_description
J-D
On Fri, Nov 13, 2009 at 1:58 PM, Zhenyu Zhong <zh...@gmail.com> wrote:
> Does HBase/Hadoop have to run on kernel 2.6.27 or jdk 1.6? It seems only
> that kernel provides epoll resource configuration.
>
> This is the first time I saw this error when I use machines with less
> resource for zookeeper.
>
> Probably, I should change it back.
>
>
> zhenyu
>
>
> On Fri, Nov 13, 2009 at 4:37 PM, Zhenyu Zhong <zh...@gmail.com>wrote:
>
>> The ulimit file descriptors was set to fs.file-max =1578334, also in the
>> limits.conf the value was set to 32768.
>> So these are way higher than the open descriptors for the running
>> processes.
>>
>> thanks
>> zhenyu
>>
>>
>>
>> On Fri, Nov 13, 2009 at 4:33 PM, Stack <sa...@gmail.com> wrote:
>>
>>> You upped the ulimit file descriptors as per the getting started doc?
>>>
>>>
>>>
>>> On Nov 13, 2009, at 1:26 PM, Zhenyu Zhong <zh...@gmail.com>
>>> wrote:
>>>
>>> Thanks a lot.
>>>>
>>>>
>>>> Bad news is my kernel is still 2.6.26.
>>>> But it was not a problem before.
>>>>
>>>> Very strange.
>>>>
>>>> zhenyu
>>>>
>>>> On Fri, Nov 13, 2009 at 4:16 PM, Jean-Daniel Cryans <jdcryans@apache.org
>>>> >wrote:
>>>>
>>>> Looks like
>>>>>
>>>>> http://pero.blogs.aprilmayjune.org/2009/01/22/hadoop-and-linux-kernel-2627-epoll-limits/
>>>>>
>>>>> J-D
>>>>>
>>>>> On Fri, Nov 13, 2009 at 1:12 PM, Zhenyu Zhong <zh...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>>
>>>>>> After I re-organize the cluster, the experiment ran into problem faster
>>>>>>
>>>>> than
>>>>>
>>>>>> before.
>>>>>>
>>>>>> Basically the changes are to use machines with less resources as
>>>>>>
>>>>> zookeeper
>>>>>
>>>>>> quorums and machines with more resources as regionserver.
>>>>>>
>>>>>> From the log, I still see the pause around 1 minute.
>>>>>> I enabled the GC logging, please see
>>>>>>
>>>>>> http://pastebin.com/m1d4ce0f1
>>>>>>
>>>>>> for details.
>>>>>> In general I don't see many pauses in the GC.
>>>>>>
>>>>>> What is more interesting, 1 hour after the 1st regionserver
>>>>>> disconnected,
>>>>>> the master log complained about too many open files. This didn't happen
>>>>>> before.
>>>>>> I checked the system OS setup as well as the limits.conf. I also
>>>>>> checked
>>>>>>
>>>>> the
>>>>>
>>>>>> running processes. The total open files don't reach the limit. I am
>>>>>>
>>>>> confused
>>>>>
>>>>>> a bit.
>>>>>>
>>>>>>
>>>>>> Please see the following master log.
>>>>>>
>>>>>> 2009-11-13 20:06:02,114 INFO
>>>>>> org.apache.hadoop.hbase.master.BaseScanner:
>>>>>> RegionManager.metaScanner scan of 4658 row(s) of meta region {server:
>>>>>> 192.168.100.128:60021, regionname: .META.,,1, startKey: <>} complete
>>>>>> 2009-11-13 20:06:02,114 INFO
>>>>>> org.apache.hadoop.hbase.master.BaseScanner:
>>>>>>
>>>>> All
>>>>>
>>>>>> 1 .META. region(s) scanned
>>>>>> 2009-11-13 20:06:07,677 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
>>>>>> response for sessionid:0x424eebf1c10004c after 3ms
>>>>>> 2009-11-13 20:06:08,178 INFO org.apache.hadoop.hdfs.DFSClient:
>>>>>> Exception
>>>>>>
>>>>> in
>>>>>
>>>>>> createBlockOutputStream java.io.IOException: Bad connect ack with
>>>>>> firstBadLink 192.168.100.123:50010
>>>>>> 2009-11-13 20:06:08,178 INFO org.apache.hadoop.hdfs.DFSClient:
>>>>>> Abandoning
>>>>>> block blk_-2808245019291145247_5478039
>>>>>> 2009-11-13 20:06:09,682 INFO org.apache.hadoop.hdfs.DFSClient:
>>>>>> Exception
>>>>>>
>>>>> in
>>>>>
>>>>>> createBlockOutputStream java.io.EOFException
>>>>>> 2009-11-13 20:06:09,682 INFO org.apache.hadoop.hdfs.DFSClient:
>>>>>> Abandoning
>>>>>> block blk_1074853606841896259_5478048
>>>>>> 2009-11-13 20:06:10,334 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
>>>>>> response for sessionid:0x24eebf1043003c after 1ms
>>>>>> 2009-11-13 20:06:21,018 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
>>>>>> response for sessionid:0x424eebf1c10004c after 0ms
>>>>>> 2009-11-13 20:06:23,674 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
>>>>>> response for sessionid:0x24eebf1043003c after 0ms
>>>>>> 2009-11-13 20:06:24,828 INFO org.apache.hadoop.hdfs.DFSClient:
>>>>>> Exception
>>>>>>
>>>>> in
>>>>>
>>>>>> createBlockOutputStream java.io.IOException: Bad connect ack with
>>>>>> firstBadLink 192.168.100.123:50010
>>>>>> 2009-11-13 20:06:24,828 INFO org.apache.hadoop.hdfs.DFSClient:
>>>>>> Abandoning
>>>>>> block blk_-6642544517082142289_5478063
>>>>>> 2009-11-13 20:06:24,828 INFO org.apache.hadoop.hdfs.DFSClient:
>>>>>> Exception
>>>>>>
>>>>> in
>>>>>
>>>>>> createBlockOutputStream java.net.SocketException: Too many open files
>>>>>> 2009-11-13 20:06:24,828 INFO org.apache.hadoop.hdfs.DFSClient:
>>>>>> Abandoning
>>>>>> block blk_2057511041109796090_5478063
>>>>>> 2009-11-13 20:06:24,928 INFO org.apache.hadoop.hdfs.DFSClient:
>>>>>> Exception
>>>>>>
>>>>> in
>>>>>
>>>>>> createBlockOutputStream java.net.SocketException: Too many open files
>>>>>> 2009-11-13 20:06:24,928 INFO org.apache.hadoop.hdfs.DFSClient:
>>>>>> Abandoning
>>>>>> block blk_8219260302213892894_5478064
>>>>>> 2009-11-13 20:06:30,855 INFO org.apache.hadoop.hdfs.DFSClient:
>>>>>> Exception
>>>>>>
>>>>> in
>>>>>
>>>>>> createBlockOutputStream java.net.SocketException: Too many open files
>>>>>> 2009-11-13 20:06:30,855 INFO org.apache.hadoop.hdfs.DFSClient:
>>>>>> Abandoning
>>>>>> block blk_1669205542853067709_5478235
>>>>>> 2009-11-13 20:06:30,905 INFO org.apache.hadoop.hdfs.DFSClient:
>>>>>> Exception
>>>>>>
>>>>> in
>>>>>
>>>>>> createBlockOutputStream java.net.SocketException: Too many open files
>>>>>> 2009-11-13 20:06:30,905 INFO org.apache.hadoop.hdfs.DFSClient:
>>>>>> Abandoning
>>>>>> block blk_9128897691346270351_5478237
>>>>>> 2009-11-13 20:06:30,955 INFO org.apache.hadoop.hdfs.DFSClient:
>>>>>> Exception
>>>>>>
>>>>> in
>>>>>
>>>>>> createBlockOutputStream java.net.SocketException: Too many open files
>>>>>> 2009-11-13 20:06:30,955 INFO org.apache.hadoop.hdfs.DFSClient:
>>>>>> Abandoning
>>>>>> block blk_1116845144864123018_5478240
>>>>>> 2009-11-13 20:06:34,372 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
>>>>>> response for sessionid:0x424eebf1c10004c after 0ms
>>>>>> 2009-11-13 20:06:37,034 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
>>>>>> response for sessionid:0x24eebf1043003c after 0ms
>>>>>> 2009-11-13 20:06:37,235 WARN org.apache.hadoop.hdfs.DFSClient:
>>>>>>
>>>>> DataStreamer
>>>>>
>>>>>> Exception: java.io.IOException: Too many open files
>>>>>> at sun.nio.ch.IOUtil.initPipe(Native Method)
>>>>>> at sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:49)
>>>>>> at
>>>>>>
>>>>>> sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:18)
>>>>>
>>>>>> at
>>>>>>
>>>>>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.get(SocketIOWithTimeout.java:407)
>>>>>
>>>>>> at
>>>>>>
>>>>>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:322)
>>>>>
>>>>>> at
>>>>>>
>>>>>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
>>>>>
>>>>>> at
>>>>>>
>>>>>> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
>>>>>
>>>>>> at
>>>>>>
>>>>>> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
>>>>>
>>>>>> at
>>>>>>
>>>>> java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
>>>>>
>>>>>> at java.io.DataOutputStream.write(DataOutputStream.java:90)
>>>>>> at
>>>>>>
>>>>>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2290)
>>>>>
>>>>>>
>>>>>> 2009-11-13 20:06:37,235 WARN org.apache.hadoop.hdfs.DFSClient: Error
>>>>>> Recovery for block blk_8148813491785406356_5478475 bad datanode[0]
>>>>>> 192.168.100.123:50010
>>>>>> 2009-11-13 20:06:37,235 WARN org.apache.hadoop.hdfs.DFSClient: Error
>>>>>> Recovery for block blk_8148813491785406356_5478475 in pipeline
>>>>>> 192.168.100.123:50010, 192.168.100.134:50010, 192.168.100.122:50010:
>>>>>> bad
>>>>>> datanode 192.168.100.123:50010
>>>>>> 2009-11-13 20:06:37,436 INFO org.apache.hadoop.hdfs.DFSClient:
>>>>>> Exception
>>>>>>
>>>>> in
>>>>>
>>>>>> createBlockOutputStream java.net.SocketException: Too many open files
>>>>>> 2009-11-13 20:06:37,436 INFO org.apache.hadoop.hdfs.DFSClient:
>>>>>> Abandoning
>>>>>> block blk_2119727700857186236_5478498
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Nov 12, 2009 at 4:21 PM, Zhenyu Zhong <zhongresearch@gmail.com
>>>>>> wrote:
>>>>>>
>>>>>> Will do.
>>>>>>>
>>>>>>> thanks
>>>>>>> zhenyu
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Nov 12, 2009 at 3:33 PM, stack <st...@duboce.net> wrote:
>>>>>>>
>>>>>>> Enable GC logging too on this next run (see hbase-env.sh). Lets try
>>>>>>>>
>>>>>>> and
>>>>>
>>>>>> get
>>>>>>>> to the bottom of whats going on.
>>>>>>>> Thanks,
>>>>>>>> St.Ack
>>>>>>>>
>>>>>>>> On Thu, Nov 12, 2009 at 12:29 PM, Zhenyu Zhong <
>>>>>>>>
>>>>>>> zhongresearch@gmail.com
>>>>>
>>>>>> wrote:
>>>>>>>>>
>>>>>>>>
>>>>>>>> I can switch the boxes that run zookeeper with the ones that run
>>>>>>>>> regionservers.
>>>>>>>>> I will see the results later.
>>>>>>>>>
>>>>>>>>> FYI. The node does have the 10 minutes zookeeper.session.timeout
>>>>>>>>>
>>>>>>>> value
>>>>>
>>>>>> in
>>>>>>>>
>>>>>>>>> place.
>>>>>>>>>
>>>>>>>>> thanks
>>>>>>>>> zhenyu
>>>>>>>>>
>>>>>>>>> On Thu, Nov 12, 2009 at 3:21 PM, stack <st...@duboce.net> wrote:
>>>>>>>>>
>>>>>>>>> On Thu, Nov 12, 2009 at 11:50 AM, Zhenyu Zhong <
>>>>>>>>>>
>>>>>>>>> zhongresearch@gmail.com
>>>>>>>>
>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> In my cluster, half of the cluster have 2 disks 400GB each per
>>>>>>>>>>>
>>>>>>>>>> machine,
>>>>>>>>
>>>>>>>>> and
>>>>>>>>>>
>>>>>>>>>>> half of the cluster have 6 disks per machine. Maybe we should
>>>>>>>>>>>
>>>>>>>>>> run
>>>>>
>>>>>> zookeeper
>>>>>>>>>>> on the machines with 2 disks and RS on machines with 6 disks?
>>>>>>>>>>>
>>>>>>>>>>> That would make most sense only in the below, it looks like the
>>>>>>>>>>>
>>>>>>>>>> RS
>>>>>
>>>>>> that
>>>>>>>>
>>>>>>>>> had
>>>>>>>>>> issue had 4 disks?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> BTW, the 10 minutes zookeeper.session.timeout has been set during
>>>>>>>>>>>
>>>>>>>>>> the
>>>>>>>>
>>>>>>>>> experiment.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> And for sure this node had it in place?
>>>>>>>>>> St.Ack
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> thanks
>>>>>>>>>>> zhenyu
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Nov 12, 2009 at 2:08 PM, stack <st...@duboce.net> wrote:
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Nov 12, 2009 at 8:40 AM, Zhenyu Zhong <
>>>>>>>>>>>>
>>>>>>>>>>> zhongresearch@gmail.com
>>>>>>>>>
>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Though I experienced 2 regionserver disconnection this
>>>>>>>>>>>>>
>>>>>>>>>>>> morning,
>>>>>
>>>>>> it
>>>>>>>>
>>>>>>>>> looks
>>>>>>>>>>>
>>>>>>>>>>>> better from the regionserver log. (please see the following
>>>>>>>>>>>>>
>>>>>>>>>>>> log)
>>>>>
>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> http://pastebin.com/m496dbfae
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> I checked diskIO for one of the regionserver(192.168.100.116)
>>>>>>>>>>>>>
>>>>>>>>>>>> during
>>>>>>>>>
>>>>>>>>>> the
>>>>>>>>>>>
>>>>>>>>>>>> time it disconnected.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Time: 03:04:58 AM
>>>>>>>>>>>>> Device: tps Blk_read/s Blk_wrtn/s Blk_read
>>>>>>>>>>>>>
>>>>>>>>>>>> Blk_wrtn
>>>>>>>>>>
>>>>>>>>>>> sda 105.31 5458.83 19503.64 9043873239
>>>>>>>>>>>>>
>>>>>>>>>>>> 32312473676
>>>>>>>>>>
>>>>>>>>>>> sda1 2.90 6.64 99.25 10993934
>>>>>>>>>>>>>
>>>>>>>>>>>> 164433464
>>>>>>>>>>
>>>>>>>>>>> sda2 1.72 23.76 30.25 39365817
>>>>>>>>>>>>>
>>>>>>>>>>>> 50124008
>>>>>>>>>>
>>>>>>>>>>> sda3 0.30 0.38 3.58 624930
>>>>>>>>>>>>>
>>>>>>>>>>>> 5923000
>>>>>>>>>>
>>>>>>>>>>> sda4 100.39 5428.06 19370.56 8992888270
>>>>>>>>>>>>>
>>>>>>>>>>>> 32091993204
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Is this high for you? 20k blocks/second would seem to be high
>>>>>>>>>>>>
>>>>>>>>>>> but
>>>>>
>>>>>> its
>>>>>>>>>
>>>>>>>>>> one
>>>>>>>>>>>
>>>>>>>>>>>> disk only and its not being shared by zk anymore so shouldn't
>>>>>>>>>>>>
>>>>>>>>>>> matter?
>>>>>>>>
>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I also checked the zookeeper quorum server that the
>>>>>>>>>>>>>
>>>>>>>>>>>> regionserver
>>>>>
>>>>>> tried
>>>>>>>>>>
>>>>>>>>>>> to
>>>>>>>>>>>
>>>>>>>>>>>> connect according to the log. However, I don't see
>>>>>>>>>>>>>
>>>>>>>>>>>> 192.168.100.116
>>>>>>>>
>>>>>>>>> in
>>>>>>>>>
>>>>>>>>>> the
>>>>>>>>>>>
>>>>>>>>>>>> client list of the zookeeper quorum server that the
>>>>>>>>>>>>>
>>>>>>>>>>>> regionserver
>>>>>
>>>>>> tried
>>>>>>>>>>
>>>>>>>>>>> to
>>>>>>>>>>>
>>>>>>>>>>>> connect.
>>>>>>>>>>>>> Would that be a problem?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Is that because the ephemeral node for the regionserver had
>>>>>>>>>>>>
>>>>>>>>>>> evaporated?
>>>>>>>>>
>>>>>>>>>> Lost
>>>>>>>>>>>> its lease w/ zk by the time you went to look?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Thu Nov 12 15:04:52 UTC 2009
>>>>>>>>>>>>> Zookeeper version: 3.2.1-808558, built on 08/27/2009 18:48
>>>>>>>>>>>>>
>>>>>>>>>>>> GMT
>>>>>
>>>>>> Clients:
>>>>>>>>>>>>> /192.168.100.127:43045[1](queued=0,recved=26,sent=0)
>>>>>>>>>>>>> /192.168.100.131:39091[1](queued=0,recved=964,sent=0)
>>>>>>>>>>>>> /192.168.100.124:35961[1](queued=0,recved=3266,sent=0)
>>>>>>>>>>>>> /192.168.100.123:47935[1](queued=0,recved=233,sent=0)
>>>>>>>>>>>>> /192.168.100.125:46931[1](queued=0,recved=2,sent=0)
>>>>>>>>>>>>> /192.168.100.118:54924[1](queued=0,recved=295,sent=0)
>>>>>>>>>>>>> /192.168.100.118:41390[1](queued=0,recved=2290,sent=0)
>>>>>>>>>>>>> /192.168.100.136:42243[1](queued=0,recved=0,sent=0)
>>>>>>>>>>>>>
>>>>>>>>>>>>> Latency min/avg/max: 0/17/6333
>>>>>>>>>>>>> Received: 47111
>>>>>>>>>>>>> Sent: 0
>>>>>>>>>>>>> Outstanding: 0
>>>>>>>>>>>>> Zxid: 0x77000083f4
>>>>>>>>>>>>> Mode: leader
>>>>>>>>>>>>> Node count: 23
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> That 6 second maximum latency is pretty bad but should be well
>>>>>>>>>>>>
>>>>>>>>>>> within
>>>>>>>>
>>>>>>>>> the
>>>>>>>>>>
>>>>>>>>>>> zk
>>>>>>>>>>>> session timeout.
>>>>>>>>>>>>
>>>>>>>>>>>> So, problem is likely on the zk client side of the session;
>>>>>>>>>>>>
>>>>>>>>>>> i.e.
>>>>>
>>>>>> in
>>>>>>>>
>>>>>>>>> the
>>>>>>>>>
>>>>>>>>>> RS.
>>>>>>>>>>>
>>>>>>>>>>>> You could enable GC logging as J-D suggested to see if you have
>>>>>>>>>>>>
>>>>>>>>>>> any
>>>>>>>>
>>>>>>>>> big
>>>>>>>>>
>>>>>>>>>> pauses, pauses > zk session timeout.
>>>>>>>>>>>>
>>>>>>>>>>>> When the RS went down, it didn't look too heavily loaded:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> 1. 2009-11-12 15:04:52,830 INFO
>>>>>>>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of
>>>>>>>>>>>>
>>>>>>>>>>> metrics:
>>>>>>>>>
>>>>>>>>>> request=1.5166667, regions=322, stores=657, storefiles=631,
>>>>>>>>>>>> storefileIndexSize=61, memstoreSize=1472, usedHeap=2819,
>>>>>>>>>>>>
>>>>>>>>>>> maxHeap=4079,
>>>>>>>>>>
>>>>>>>>>>> blockCacheSize=658110960, blockCacheFree=197395984,
>>>>>>>>>>>>
>>>>>>>>>>> blockCacheCount=9903,
>>>>>>>>>>>
>>>>>>>>>>>> blockCacheHitRatio=99
>>>>>>>>>>>>
>>>>>>>>>>>> Its serving a few reads? The number of store files seems fine.
>>>>>>>>>>>>
>>>>>>>>>>> Not
>>>>>>>>
>>>>>>>>> too
>>>>>>>>>>
>>>>>>>>>>> much memory used.
>>>>>>>>>>>>
>>>>>>>>>>>> Looking at the logs, I see the Lease Still Held exception.
>>>>>>>>>>>>
>>>>>>>>>>> This
>>>>>
>>>>>> happens
>>>>>>>>>>
>>>>>>>>>>> when the RS does its regular report to the master but the
>>>>>>>>>>>>
>>>>>>>>>>> master
>>>>>
>>>>>> thinks
>>>>>>>>>
>>>>>>>>>> the
>>>>>>>>>>>
>>>>>>>>>>>> RS has since restarted. It'll think this probably because it
>>>>>>>>>>>>
>>>>>>>>>>> noticed
>>>>>>>>
>>>>>>>>> that
>>>>>>>>>>>
>>>>>>>>>>>> the RS's znode in zk had gone away and it considered the RS
>>>>>>>>>>>>
>>>>>>>>>>> dead.
>>>>>
>>>>>>
>>>>>>>>>>>> Looking too at your logs I see this gap in the zk pinging:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> 1. 2009-11-12 15:03:39,325 DEBUG
>>>>>>>>>>>>
>>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>>>
>>>>>>>>> Got
>>>>>>>>>
>>>>>>>>>> ping response for sessionid:0x224e55436ad0004 after 0ms
>>>>>>>>>>>> 2. 2009-11-12 15:03:43,113 DEBUG
>>>>>>>>>>>>
>>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>>>
>>>>>>>>> Got
>>>>>>>>>
>>>>>>>>>> ping response for sessionid:0x24e55436a0007d after 0ms
>>>>>>>>>>>>
>>>>>>>>>>>> Where in the lines above it, its reporting about every ten
>>>>>>>>>>>>
>>>>>>>>>>> seconds,
>>>>>>>>
>>>>>>>>> here
>>>>>>>>>>
>>>>>>>>>>> there is a big gap.
>>>>>>>>>>>>
>>>>>>>>>>>> Do you have ganglia or something that will let you look more
>>>>>>>>>>>>
>>>>>>>>>>> into
>>>>>
>>>>>> what
>>>>>>>>>
>>>>>>>>>> was
>>>>>>>>>>>
>>>>>>>>>>>> happening on this machine around this time? Is the machine OK?
>>>>>>>>>>>>
>>>>>>>>>>> It
>>>>>>>>
>>>>>>>>> looks
>>>>>>>>>>
>>>>>>>>>>> lightly loaded and you have your cluster nicely laid out.
>>>>>>>>>>>>
>>>>>>>>>>> Something
>>>>>>>>
>>>>>>>>> odd
>>>>>>>>>>
>>>>>>>>>>> is
>>>>>>>>>>>
>>>>>>>>>>>> going on. What about things like the write speed to disk? In
>>>>>>>>>>>>
>>>>>>>>>>> the
>>>>>
>>>>>> past
>>>>>>>>>
>>>>>>>>>> strange issues have been explained by incorrectly set BIOS
>>>>>>>>>>>>
>>>>>>>>>>> which
>>>>>
>>>>>> made
>>>>>>>>
>>>>>>>>> disks
>>>>>>>>>>>
>>>>>>>>>>>> run at 1/100th of their proper speed.
>>>>>>>>>>>>
>>>>>>>>>>>> St.Ack
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> Best,
>>>>>>>>>>>>> zhenyu
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Nov 11, 2009 at 3:58 PM, Zhenyu Zhong <
>>>>>>>>>>>>>
>>>>>>>>>>>> zhongresearch@gmail.com
>>>>>>>>>>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Stack
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I am very appreciated for your comments.
>>>>>>>>>>>>>> I will use the zookeeper monitoring script on my cluster
>>>>>>>>>>>>>>
>>>>>>>>>>>>> and
>>>>>
>>>>>> let
>>>>>>>>
>>>>>>>>> it
>>>>>>>>>
>>>>>>>>>> run
>>>>>>>>>>>
>>>>>>>>>>>> overnight to see the result.
>>>>>>>>>>>>>> I will follow up the thread when I get anything.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> thanks
>>>>>>>>>>>>>> zhenyu
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wed, Nov 11, 2009 at 3:52 PM, stack <st...@duboce.net>
>>>>>>>>>>>>>>
>>>>>>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>>>>>>> I see these in your log too:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 1. 2009-11-11 04:27:19,018 DEBUG
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>>>>>
>>>>>>>>>>> Got
>>>>>>>>>>>>
>>>>>>>>>>>>> ping response for sessionid:0x424dfd908c50009 after
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 4544ms
>>>>>
>>>>>> 2. 2009-11-11 04:27:19,018 DEBUG
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>>>>>
>>>>>>>>>>> Got
>>>>>>>>>>>>
>>>>>>>>>>>>> ping response for sessionid:0x24dfd90c810002 after
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 4548ms
>>>>>
>>>>>> 3. 2009-11-11 04:27:43,960 DEBUG
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>>>>>
>>>>>>>>>>> Got
>>>>>>>>>>>>
>>>>>>>>>>>>> ping response for sessionid:0x424dfd908c50009 after
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 9030ms
>>>>>
>>>>>> 4. 2009-11-11 04:27:43,960 DEBUG
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> org.apache.zookeeper.
>>>>>>>>>>
>>>>>>>>>
>>
>
Re: regionserver disconnection
Posted by Zhenyu Zhong <zh...@gmail.com>.
Does HBase/Hadoop have to run on kernel 2.6.27 or jdk 1.6? It seems only
that kernel provides epoll resource configuration.
This is the first time I saw this error when I use machines with less
resource for zookeeper.
Probably, I should change it back.
zhenyu
On Fri, Nov 13, 2009 at 4:37 PM, Zhenyu Zhong <zh...@gmail.com>wrote:
> The ulimit file descriptors was set to fs.file-max =1578334, also in the
> limits.conf the value was set to 32768.
> So these are way higher than the open descriptors for the running
> processes.
>
> thanks
> zhenyu
>
>
>
> On Fri, Nov 13, 2009 at 4:33 PM, Stack <sa...@gmail.com> wrote:
>
>> You upped the ulimit file descriptors as per the getting started doc?
>>
>>
>>
>> On Nov 13, 2009, at 1:26 PM, Zhenyu Zhong <zh...@gmail.com>
>> wrote:
>>
>> Thanks a lot.
>>>
>>>
>>> Bad news is my kernel is still 2.6.26.
>>> But it was not a problem before.
>>>
>>> Very strange.
>>>
>>> zhenyu
>>>
>>> On Fri, Nov 13, 2009 at 4:16 PM, Jean-Daniel Cryans <jdcryans@apache.org
>>> >wrote:
>>>
>>> Looks like
>>>>
>>>> http://pero.blogs.aprilmayjune.org/2009/01/22/hadoop-and-linux-kernel-2627-epoll-limits/
>>>>
>>>> J-D
>>>>
>>>> On Fri, Nov 13, 2009 at 1:12 PM, Zhenyu Zhong <zh...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>>
>>>>> After I re-organize the cluster, the experiment ran into problem faster
>>>>>
>>>> than
>>>>
>>>>> before.
>>>>>
>>>>> Basically the changes are to use machines with less resources as
>>>>>
>>>> zookeeper
>>>>
>>>>> quorums and machines with more resources as regionserver.
>>>>>
>>>>> From the log, I still see the pause around 1 minute.
>>>>> I enabled the GC logging, please see
>>>>>
>>>>> http://pastebin.com/m1d4ce0f1
>>>>>
>>>>> for details.
>>>>> In general I don't see many pauses in the GC.
>>>>>
>>>>> What is more interesting, 1 hour after the 1st regionserver
>>>>> disconnected,
>>>>> the master log complained about too many open files. This didn't happen
>>>>> before.
>>>>> I checked the system OS setup as well as the limits.conf. I also
>>>>> checked
>>>>>
>>>> the
>>>>
>>>>> running processes. The total open files don't reach the limit. I am
>>>>>
>>>> confused
>>>>
>>>>> a bit.
>>>>>
>>>>>
>>>>> Please see the following master log.
>>>>>
>>>>> 2009-11-13 20:06:02,114 INFO
>>>>> org.apache.hadoop.hbase.master.BaseScanner:
>>>>> RegionManager.metaScanner scan of 4658 row(s) of meta region {server:
>>>>> 192.168.100.128:60021, regionname: .META.,,1, startKey: <>} complete
>>>>> 2009-11-13 20:06:02,114 INFO
>>>>> org.apache.hadoop.hbase.master.BaseScanner:
>>>>>
>>>> All
>>>>
>>>>> 1 .META. region(s) scanned
>>>>> 2009-11-13 20:06:07,677 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
>>>>> response for sessionid:0x424eebf1c10004c after 3ms
>>>>> 2009-11-13 20:06:08,178 INFO org.apache.hadoop.hdfs.DFSClient:
>>>>> Exception
>>>>>
>>>> in
>>>>
>>>>> createBlockOutputStream java.io.IOException: Bad connect ack with
>>>>> firstBadLink 192.168.100.123:50010
>>>>> 2009-11-13 20:06:08,178 INFO org.apache.hadoop.hdfs.DFSClient:
>>>>> Abandoning
>>>>> block blk_-2808245019291145247_5478039
>>>>> 2009-11-13 20:06:09,682 INFO org.apache.hadoop.hdfs.DFSClient:
>>>>> Exception
>>>>>
>>>> in
>>>>
>>>>> createBlockOutputStream java.io.EOFException
>>>>> 2009-11-13 20:06:09,682 INFO org.apache.hadoop.hdfs.DFSClient:
>>>>> Abandoning
>>>>> block blk_1074853606841896259_5478048
>>>>> 2009-11-13 20:06:10,334 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
>>>>> response for sessionid:0x24eebf1043003c after 1ms
>>>>> 2009-11-13 20:06:21,018 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
>>>>> response for sessionid:0x424eebf1c10004c after 0ms
>>>>> 2009-11-13 20:06:23,674 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
>>>>> response for sessionid:0x24eebf1043003c after 0ms
>>>>> 2009-11-13 20:06:24,828 INFO org.apache.hadoop.hdfs.DFSClient:
>>>>> Exception
>>>>>
>>>> in
>>>>
>>>>> createBlockOutputStream java.io.IOException: Bad connect ack with
>>>>> firstBadLink 192.168.100.123:50010
>>>>> 2009-11-13 20:06:24,828 INFO org.apache.hadoop.hdfs.DFSClient:
>>>>> Abandoning
>>>>> block blk_-6642544517082142289_5478063
>>>>> 2009-11-13 20:06:24,828 INFO org.apache.hadoop.hdfs.DFSClient:
>>>>> Exception
>>>>>
>>>> in
>>>>
>>>>> createBlockOutputStream java.net.SocketException: Too many open files
>>>>> 2009-11-13 20:06:24,828 INFO org.apache.hadoop.hdfs.DFSClient:
>>>>> Abandoning
>>>>> block blk_2057511041109796090_5478063
>>>>> 2009-11-13 20:06:24,928 INFO org.apache.hadoop.hdfs.DFSClient:
>>>>> Exception
>>>>>
>>>> in
>>>>
>>>>> createBlockOutputStream java.net.SocketException: Too many open files
>>>>> 2009-11-13 20:06:24,928 INFO org.apache.hadoop.hdfs.DFSClient:
>>>>> Abandoning
>>>>> block blk_8219260302213892894_5478064
>>>>> 2009-11-13 20:06:30,855 INFO org.apache.hadoop.hdfs.DFSClient:
>>>>> Exception
>>>>>
>>>> in
>>>>
>>>>> createBlockOutputStream java.net.SocketException: Too many open files
>>>>> 2009-11-13 20:06:30,855 INFO org.apache.hadoop.hdfs.DFSClient:
>>>>> Abandoning
>>>>> block blk_1669205542853067709_5478235
>>>>> 2009-11-13 20:06:30,905 INFO org.apache.hadoop.hdfs.DFSClient:
>>>>> Exception
>>>>>
>>>> in
>>>>
>>>>> createBlockOutputStream java.net.SocketException: Too many open files
>>>>> 2009-11-13 20:06:30,905 INFO org.apache.hadoop.hdfs.DFSClient:
>>>>> Abandoning
>>>>> block blk_9128897691346270351_5478237
>>>>> 2009-11-13 20:06:30,955 INFO org.apache.hadoop.hdfs.DFSClient:
>>>>> Exception
>>>>>
>>>> in
>>>>
>>>>> createBlockOutputStream java.net.SocketException: Too many open files
>>>>> 2009-11-13 20:06:30,955 INFO org.apache.hadoop.hdfs.DFSClient:
>>>>> Abandoning
>>>>> block blk_1116845144864123018_5478240
>>>>> 2009-11-13 20:06:34,372 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
>>>>> response for sessionid:0x424eebf1c10004c after 0ms
>>>>> 2009-11-13 20:06:37,034 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
>>>>> response for sessionid:0x24eebf1043003c after 0ms
>>>>> 2009-11-13 20:06:37,235 WARN org.apache.hadoop.hdfs.DFSClient:
>>>>>
>>>> DataStreamer
>>>>
>>>>> Exception: java.io.IOException: Too many open files
>>>>> at sun.nio.ch.IOUtil.initPipe(Native Method)
>>>>> at sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:49)
>>>>> at
>>>>>
>>>>> sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:18)
>>>>
>>>>> at
>>>>>
>>>>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.get(SocketIOWithTimeout.java:407)
>>>>
>>>>> at
>>>>>
>>>>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:322)
>>>>
>>>>> at
>>>>>
>>>>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
>>>>
>>>>> at
>>>>>
>>>>> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
>>>>
>>>>> at
>>>>>
>>>>> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
>>>>
>>>>> at
>>>>>
>>>> java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
>>>>
>>>>> at java.io.DataOutputStream.write(DataOutputStream.java:90)
>>>>> at
>>>>>
>>>>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2290)
>>>>
>>>>>
>>>>> 2009-11-13 20:06:37,235 WARN org.apache.hadoop.hdfs.DFSClient: Error
>>>>> Recovery for block blk_8148813491785406356_5478475 bad datanode[0]
>>>>> 192.168.100.123:50010
>>>>> 2009-11-13 20:06:37,235 WARN org.apache.hadoop.hdfs.DFSClient: Error
>>>>> Recovery for block blk_8148813491785406356_5478475 in pipeline
>>>>> 192.168.100.123:50010, 192.168.100.134:50010, 192.168.100.122:50010:
>>>>> bad
>>>>> datanode 192.168.100.123:50010
>>>>> 2009-11-13 20:06:37,436 INFO org.apache.hadoop.hdfs.DFSClient:
>>>>> Exception
>>>>>
>>>> in
>>>>
>>>>> createBlockOutputStream java.net.SocketException: Too many open files
>>>>> 2009-11-13 20:06:37,436 INFO org.apache.hadoop.hdfs.DFSClient:
>>>>> Abandoning
>>>>> block blk_2119727700857186236_5478498
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Nov 12, 2009 at 4:21 PM, Zhenyu Zhong <zhongresearch@gmail.com
>>>>> wrote:
>>>>>
>>>>> Will do.
>>>>>>
>>>>>> thanks
>>>>>> zhenyu
>>>>>>
>>>>>>
>>>>>> On Thu, Nov 12, 2009 at 3:33 PM, stack <st...@duboce.net> wrote:
>>>>>>
>>>>>> Enable GC logging too on this next run (see hbase-env.sh). Lets try
>>>>>>>
>>>>>> and
>>>>
>>>>> get
>>>>>>> to the bottom of whats going on.
>>>>>>> Thanks,
>>>>>>> St.Ack
>>>>>>>
>>>>>>> On Thu, Nov 12, 2009 at 12:29 PM, Zhenyu Zhong <
>>>>>>>
>>>>>> zhongresearch@gmail.com
>>>>
>>>>> wrote:
>>>>>>>>
>>>>>>>
>>>>>>> I can switch the boxes that run zookeeper with the ones that run
>>>>>>>> regionservers.
>>>>>>>> I will see the results later.
>>>>>>>>
>>>>>>>> FYI. The node does have the 10 minutes zookeeper.session.timeout
>>>>>>>>
>>>>>>> value
>>>>
>>>>> in
>>>>>>>
>>>>>>>> place.
>>>>>>>>
>>>>>>>> thanks
>>>>>>>> zhenyu
>>>>>>>>
>>>>>>>> On Thu, Nov 12, 2009 at 3:21 PM, stack <st...@duboce.net> wrote:
>>>>>>>>
>>>>>>>> On Thu, Nov 12, 2009 at 11:50 AM, Zhenyu Zhong <
>>>>>>>>>
>>>>>>>> zhongresearch@gmail.com
>>>>>>>
>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> In my cluster, half of the cluster have 2 disks 400GB each per
>>>>>>>>>>
>>>>>>>>> machine,
>>>>>>>
>>>>>>>> and
>>>>>>>>>
>>>>>>>>>> half of the cluster have 6 disks per machine. Maybe we should
>>>>>>>>>>
>>>>>>>>> run
>>>>
>>>>> zookeeper
>>>>>>>>>> on the machines with 2 disks and RS on machines with 6 disks?
>>>>>>>>>>
>>>>>>>>>> That would make most sense only in the below, it looks like the
>>>>>>>>>>
>>>>>>>>> RS
>>>>
>>>>> that
>>>>>>>
>>>>>>>> had
>>>>>>>>> issue had 4 disks?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> BTW, the 10 minutes zookeeper.session.timeout has been set during
>>>>>>>>>>
>>>>>>>>> the
>>>>>>>
>>>>>>>> experiment.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> And for sure this node had it in place?
>>>>>>>>> St.Ack
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> thanks
>>>>>>>>>> zhenyu
>>>>>>>>>>
>>>>>>>>>> On Thu, Nov 12, 2009 at 2:08 PM, stack <st...@duboce.net> wrote:
>>>>>>>>>>
>>>>>>>>>> On Thu, Nov 12, 2009 at 8:40 AM, Zhenyu Zhong <
>>>>>>>>>>>
>>>>>>>>>> zhongresearch@gmail.com
>>>>>>>>
>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Though I experienced 2 regionserver disconnection this
>>>>>>>>>>>>
>>>>>>>>>>> morning,
>>>>
>>>>> it
>>>>>>>
>>>>>>>> looks
>>>>>>>>>>
>>>>>>>>>>> better from the regionserver log. (please see the following
>>>>>>>>>>>>
>>>>>>>>>>> log)
>>>>
>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> http://pastebin.com/m496dbfae
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I checked diskIO for one of the regionserver(192.168.100.116)
>>>>>>>>>>>>
>>>>>>>>>>> during
>>>>>>>>
>>>>>>>>> the
>>>>>>>>>>
>>>>>>>>>>> time it disconnected.
>>>>>>>>>>>>
>>>>>>>>>>>> Time: 03:04:58 AM
>>>>>>>>>>>> Device: tps Blk_read/s Blk_wrtn/s Blk_read
>>>>>>>>>>>>
>>>>>>>>>>> Blk_wrtn
>>>>>>>>>
>>>>>>>>>> sda 105.31 5458.83 19503.64 9043873239
>>>>>>>>>>>>
>>>>>>>>>>> 32312473676
>>>>>>>>>
>>>>>>>>>> sda1 2.90 6.64 99.25 10993934
>>>>>>>>>>>>
>>>>>>>>>>> 164433464
>>>>>>>>>
>>>>>>>>>> sda2 1.72 23.76 30.25 39365817
>>>>>>>>>>>>
>>>>>>>>>>> 50124008
>>>>>>>>>
>>>>>>>>>> sda3 0.30 0.38 3.58 624930
>>>>>>>>>>>>
>>>>>>>>>>> 5923000
>>>>>>>>>
>>>>>>>>>> sda4 100.39 5428.06 19370.56 8992888270
>>>>>>>>>>>>
>>>>>>>>>>> 32091993204
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Is this high for you? 20k blocks/second would seem to be high
>>>>>>>>>>>
>>>>>>>>>> but
>>>>
>>>>> its
>>>>>>>>
>>>>>>>>> one
>>>>>>>>>>
>>>>>>>>>>> disk only and its not being shared by zk anymore so shouldn't
>>>>>>>>>>>
>>>>>>>>>> matter?
>>>>>>>
>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I also checked the zookeeper quorum server that the
>>>>>>>>>>>>
>>>>>>>>>>> regionserver
>>>>
>>>>> tried
>>>>>>>>>
>>>>>>>>>> to
>>>>>>>>>>
>>>>>>>>>>> connect according to the log. However, I don't see
>>>>>>>>>>>>
>>>>>>>>>>> 192.168.100.116
>>>>>>>
>>>>>>>> in
>>>>>>>>
>>>>>>>>> the
>>>>>>>>>>
>>>>>>>>>>> client list of the zookeeper quorum server that the
>>>>>>>>>>>>
>>>>>>>>>>> regionserver
>>>>
>>>>> tried
>>>>>>>>>
>>>>>>>>>> to
>>>>>>>>>>
>>>>>>>>>>> connect.
>>>>>>>>>>>> Would that be a problem?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Is that because the ephemeral node for the regionserver had
>>>>>>>>>>>
>>>>>>>>>> evaporated?
>>>>>>>>
>>>>>>>>> Lost
>>>>>>>>>>> its lease w/ zk by the time you went to look?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Thu Nov 12 15:04:52 UTC 2009
>>>>>>>>>>>> Zookeeper version: 3.2.1-808558, built on 08/27/2009 18:48
>>>>>>>>>>>>
>>>>>>>>>>> GMT
>>>>
>>>>> Clients:
>>>>>>>>>>>> /192.168.100.127:43045[1](queued=0,recved=26,sent=0)
>>>>>>>>>>>> /192.168.100.131:39091[1](queued=0,recved=964,sent=0)
>>>>>>>>>>>> /192.168.100.124:35961[1](queued=0,recved=3266,sent=0)
>>>>>>>>>>>> /192.168.100.123:47935[1](queued=0,recved=233,sent=0)
>>>>>>>>>>>> /192.168.100.125:46931[1](queued=0,recved=2,sent=0)
>>>>>>>>>>>> /192.168.100.118:54924[1](queued=0,recved=295,sent=0)
>>>>>>>>>>>> /192.168.100.118:41390[1](queued=0,recved=2290,sent=0)
>>>>>>>>>>>> /192.168.100.136:42243[1](queued=0,recved=0,sent=0)
>>>>>>>>>>>>
>>>>>>>>>>>> Latency min/avg/max: 0/17/6333
>>>>>>>>>>>> Received: 47111
>>>>>>>>>>>> Sent: 0
>>>>>>>>>>>> Outstanding: 0
>>>>>>>>>>>> Zxid: 0x77000083f4
>>>>>>>>>>>> Mode: leader
>>>>>>>>>>>> Node count: 23
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> That 6 second maximum latency is pretty bad but should be well
>>>>>>>>>>>
>>>>>>>>>> within
>>>>>>>
>>>>>>>> the
>>>>>>>>>
>>>>>>>>>> zk
>>>>>>>>>>> session timeout.
>>>>>>>>>>>
>>>>>>>>>>> So, problem is likely on the zk client side of the session;
>>>>>>>>>>>
>>>>>>>>>> i.e.
>>>>
>>>>> in
>>>>>>>
>>>>>>>> the
>>>>>>>>
>>>>>>>>> RS.
>>>>>>>>>>
>>>>>>>>>>> You could enable GC logging as J-D suggested to see if you have
>>>>>>>>>>>
>>>>>>>>>> any
>>>>>>>
>>>>>>>> big
>>>>>>>>
>>>>>>>>> pauses, pauses > zk session timeout.
>>>>>>>>>>>
>>>>>>>>>>> When the RS went down, it didn't look too heavily loaded:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 1. 2009-11-12 15:04:52,830 INFO
>>>>>>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of
>>>>>>>>>>>
>>>>>>>>>> metrics:
>>>>>>>>
>>>>>>>>> request=1.5166667, regions=322, stores=657, storefiles=631,
>>>>>>>>>>> storefileIndexSize=61, memstoreSize=1472, usedHeap=2819,
>>>>>>>>>>>
>>>>>>>>>> maxHeap=4079,
>>>>>>>>>
>>>>>>>>>> blockCacheSize=658110960, blockCacheFree=197395984,
>>>>>>>>>>>
>>>>>>>>>> blockCacheCount=9903,
>>>>>>>>>>
>>>>>>>>>>> blockCacheHitRatio=99
>>>>>>>>>>>
>>>>>>>>>>> Its serving a few reads? The number of store files seems fine.
>>>>>>>>>>>
>>>>>>>>>> Not
>>>>>>>
>>>>>>>> too
>>>>>>>>>
>>>>>>>>>> much memory used.
>>>>>>>>>>>
>>>>>>>>>>> Looking at the logs, I see the Lease Still Held exception.
>>>>>>>>>>>
>>>>>>>>>> This
>>>>
>>>>> happens
>>>>>>>>>
>>>>>>>>>> when the RS does its regular report to the master but the
>>>>>>>>>>>
>>>>>>>>>> master
>>>>
>>>>> thinks
>>>>>>>>
>>>>>>>>> the
>>>>>>>>>>
>>>>>>>>>>> RS has since restarted. It'll think this probably because it
>>>>>>>>>>>
>>>>>>>>>> noticed
>>>>>>>
>>>>>>>> that
>>>>>>>>>>
>>>>>>>>>>> the RS's znode in zk had gone away and it considered the RS
>>>>>>>>>>>
>>>>>>>>>> dead.
>>>>
>>>>>
>>>>>>>>>>> Looking too at your logs I see this gap in the zk pinging:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 1. 2009-11-12 15:03:39,325 DEBUG
>>>>>>>>>>>
>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>>
>>>>>>>> Got
>>>>>>>>
>>>>>>>>> ping response for sessionid:0x224e55436ad0004 after 0ms
>>>>>>>>>>> 2. 2009-11-12 15:03:43,113 DEBUG
>>>>>>>>>>>
>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>>
>>>>>>>> Got
>>>>>>>>
>>>>>>>>> ping response for sessionid:0x24e55436a0007d after 0ms
>>>>>>>>>>>
>>>>>>>>>>> Where in the lines above it, its reporting about every ten
>>>>>>>>>>>
>>>>>>>>>> seconds,
>>>>>>>
>>>>>>>> here
>>>>>>>>>
>>>>>>>>>> there is a big gap.
>>>>>>>>>>>
>>>>>>>>>>> Do you have ganglia or something that will let you look more
>>>>>>>>>>>
>>>>>>>>>> into
>>>>
>>>>> what
>>>>>>>>
>>>>>>>>> was
>>>>>>>>>>
>>>>>>>>>>> happening on this machine around this time? Is the machine OK?
>>>>>>>>>>>
>>>>>>>>>> It
>>>>>>>
>>>>>>>> looks
>>>>>>>>>
>>>>>>>>>> lightly loaded and you have your cluster nicely laid out.
>>>>>>>>>>>
>>>>>>>>>> Something
>>>>>>>
>>>>>>>> odd
>>>>>>>>>
>>>>>>>>>> is
>>>>>>>>>>
>>>>>>>>>>> going on. What about things like the write speed to disk? In
>>>>>>>>>>>
>>>>>>>>>> the
>>>>
>>>>> past
>>>>>>>>
>>>>>>>>> strange issues have been explained by incorrectly set BIOS
>>>>>>>>>>>
>>>>>>>>>> which
>>>>
>>>>> made
>>>>>>>
>>>>>>>> disks
>>>>>>>>>>
>>>>>>>>>>> run at 1/100th of their proper speed.
>>>>>>>>>>>
>>>>>>>>>>> St.Ack
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> Best,
>>>>>>>>>>>> zhenyu
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Nov 11, 2009 at 3:58 PM, Zhenyu Zhong <
>>>>>>>>>>>>
>>>>>>>>>>> zhongresearch@gmail.com
>>>>>>>>>
>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Stack
>>>>>>>>>>>>>
>>>>>>>>>>>>> I am very appreciated for your comments.
>>>>>>>>>>>>> I will use the zookeeper monitoring script on my cluster
>>>>>>>>>>>>>
>>>>>>>>>>>> and
>>>>
>>>>> let
>>>>>>>
>>>>>>>> it
>>>>>>>>
>>>>>>>>> run
>>>>>>>>>>
>>>>>>>>>>> overnight to see the result.
>>>>>>>>>>>>> I will follow up the thread when I get anything.
>>>>>>>>>>>>>
>>>>>>>>>>>>> thanks
>>>>>>>>>>>>> zhenyu
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Nov 11, 2009 at 3:52 PM, stack <st...@duboce.net>
>>>>>>>>>>>>>
>>>>>>>>>>>> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>>>>>>> I see these in your log too:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 1. 2009-11-11 04:27:19,018 DEBUG
>>>>>>>>>>>>>>
>>>>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>>>>
>>>>>>>>>> Got
>>>>>>>>>>>
>>>>>>>>>>>> ping response for sessionid:0x424dfd908c50009 after
>>>>>>>>>>>>>>
>>>>>>>>>>>>> 4544ms
>>>>
>>>>> 2. 2009-11-11 04:27:19,018 DEBUG
>>>>>>>>>>>>>>
>>>>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>>>>
>>>>>>>>>> Got
>>>>>>>>>>>
>>>>>>>>>>>> ping response for sessionid:0x24dfd90c810002 after
>>>>>>>>>>>>>>
>>>>>>>>>>>>> 4548ms
>>>>
>>>>> 3. 2009-11-11 04:27:43,960 DEBUG
>>>>>>>>>>>>>>
>>>>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>>>>
>>>>>>>>>> Got
>>>>>>>>>>>
>>>>>>>>>>>> ping response for sessionid:0x424dfd908c50009 after
>>>>>>>>>>>>>>
>>>>>>>>>>>>> 9030ms
>>>>
>>>>> 4. 2009-11-11 04:27:43,960 DEBUG
>>>>>>>>>>>>>>
>>>>>>>>>>>>> org.apache.zookeeper.
>>>>>>>>>
>>>>>>>>
>
Re: regionserver disconnection
Posted by Zhenyu Zhong <zh...@gmail.com>.
The ulimit file descriptors was set to fs.file-max =1578334, also in the
limits.conf the value was set to 32768.
So these are way higher than the open descriptors for the running processes.
thanks
zhenyu
On Fri, Nov 13, 2009 at 4:33 PM, Stack <sa...@gmail.com> wrote:
> You upped the ulimit file descriptors as per the getting started doc?
>
>
>
> On Nov 13, 2009, at 1:26 PM, Zhenyu Zhong <zh...@gmail.com> wrote:
>
> Thanks a lot.
>>
>>
>> Bad news is my kernel is still 2.6.26.
>> But it was not a problem before.
>>
>> Very strange.
>>
>> zhenyu
>>
>> On Fri, Nov 13, 2009 at 4:16 PM, Jean-Daniel Cryans <jdcryans@apache.org
>> >wrote:
>>
>> Looks like
>>>
>>> http://pero.blogs.aprilmayjune.org/2009/01/22/hadoop-and-linux-kernel-2627-epoll-limits/
>>>
>>> J-D
>>>
>>> On Fri, Nov 13, 2009 at 1:12 PM, Zhenyu Zhong <zh...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>>
>>>> After I re-organize the cluster, the experiment ran into problem faster
>>>>
>>> than
>>>
>>>> before.
>>>>
>>>> Basically the changes are to use machines with less resources as
>>>>
>>> zookeeper
>>>
>>>> quorums and machines with more resources as regionserver.
>>>>
>>>> From the log, I still see the pause around 1 minute.
>>>> I enabled the GC logging, please see
>>>>
>>>> http://pastebin.com/m1d4ce0f1
>>>>
>>>> for details.
>>>> In general I don't see many pauses in the GC.
>>>>
>>>> What is more interesting, 1 hour after the 1st regionserver
>>>> disconnected,
>>>> the master log complained about too many open files. This didn't happen
>>>> before.
>>>> I checked the system OS setup as well as the limits.conf. I also checked
>>>>
>>> the
>>>
>>>> running processes. The total open files don't reach the limit. I am
>>>>
>>> confused
>>>
>>>> a bit.
>>>>
>>>>
>>>> Please see the following master log.
>>>>
>>>> 2009-11-13 20:06:02,114 INFO org.apache.hadoop.hbase.master.BaseScanner:
>>>> RegionManager.metaScanner scan of 4658 row(s) of meta region {server:
>>>> 192.168.100.128:60021, regionname: .META.,,1, startKey: <>} complete
>>>> 2009-11-13 20:06:02,114 INFO org.apache.hadoop.hbase.master.BaseScanner:
>>>>
>>> All
>>>
>>>> 1 .META. region(s) scanned
>>>> 2009-11-13 20:06:07,677 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
>>>> response for sessionid:0x424eebf1c10004c after 3ms
>>>> 2009-11-13 20:06:08,178 INFO org.apache.hadoop.hdfs.DFSClient: Exception
>>>>
>>> in
>>>
>>>> createBlockOutputStream java.io.IOException: Bad connect ack with
>>>> firstBadLink 192.168.100.123:50010
>>>> 2009-11-13 20:06:08,178 INFO org.apache.hadoop.hdfs.DFSClient:
>>>> Abandoning
>>>> block blk_-2808245019291145247_5478039
>>>> 2009-11-13 20:06:09,682 INFO org.apache.hadoop.hdfs.DFSClient: Exception
>>>>
>>> in
>>>
>>>> createBlockOutputStream java.io.EOFException
>>>> 2009-11-13 20:06:09,682 INFO org.apache.hadoop.hdfs.DFSClient:
>>>> Abandoning
>>>> block blk_1074853606841896259_5478048
>>>> 2009-11-13 20:06:10,334 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
>>>> response for sessionid:0x24eebf1043003c after 1ms
>>>> 2009-11-13 20:06:21,018 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
>>>> response for sessionid:0x424eebf1c10004c after 0ms
>>>> 2009-11-13 20:06:23,674 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
>>>> response for sessionid:0x24eebf1043003c after 0ms
>>>> 2009-11-13 20:06:24,828 INFO org.apache.hadoop.hdfs.DFSClient: Exception
>>>>
>>> in
>>>
>>>> createBlockOutputStream java.io.IOException: Bad connect ack with
>>>> firstBadLink 192.168.100.123:50010
>>>> 2009-11-13 20:06:24,828 INFO org.apache.hadoop.hdfs.DFSClient:
>>>> Abandoning
>>>> block blk_-6642544517082142289_5478063
>>>> 2009-11-13 20:06:24,828 INFO org.apache.hadoop.hdfs.DFSClient: Exception
>>>>
>>> in
>>>
>>>> createBlockOutputStream java.net.SocketException: Too many open files
>>>> 2009-11-13 20:06:24,828 INFO org.apache.hadoop.hdfs.DFSClient:
>>>> Abandoning
>>>> block blk_2057511041109796090_5478063
>>>> 2009-11-13 20:06:24,928 INFO org.apache.hadoop.hdfs.DFSClient: Exception
>>>>
>>> in
>>>
>>>> createBlockOutputStream java.net.SocketException: Too many open files
>>>> 2009-11-13 20:06:24,928 INFO org.apache.hadoop.hdfs.DFSClient:
>>>> Abandoning
>>>> block blk_8219260302213892894_5478064
>>>> 2009-11-13 20:06:30,855 INFO org.apache.hadoop.hdfs.DFSClient: Exception
>>>>
>>> in
>>>
>>>> createBlockOutputStream java.net.SocketException: Too many open files
>>>> 2009-11-13 20:06:30,855 INFO org.apache.hadoop.hdfs.DFSClient:
>>>> Abandoning
>>>> block blk_1669205542853067709_5478235
>>>> 2009-11-13 20:06:30,905 INFO org.apache.hadoop.hdfs.DFSClient: Exception
>>>>
>>> in
>>>
>>>> createBlockOutputStream java.net.SocketException: Too many open files
>>>> 2009-11-13 20:06:30,905 INFO org.apache.hadoop.hdfs.DFSClient:
>>>> Abandoning
>>>> block blk_9128897691346270351_5478237
>>>> 2009-11-13 20:06:30,955 INFO org.apache.hadoop.hdfs.DFSClient: Exception
>>>>
>>> in
>>>
>>>> createBlockOutputStream java.net.SocketException: Too many open files
>>>> 2009-11-13 20:06:30,955 INFO org.apache.hadoop.hdfs.DFSClient:
>>>> Abandoning
>>>> block blk_1116845144864123018_5478240
>>>> 2009-11-13 20:06:34,372 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
>>>> response for sessionid:0x424eebf1c10004c after 0ms
>>>> 2009-11-13 20:06:37,034 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
>>>> response for sessionid:0x24eebf1043003c after 0ms
>>>> 2009-11-13 20:06:37,235 WARN org.apache.hadoop.hdfs.DFSClient:
>>>>
>>> DataStreamer
>>>
>>>> Exception: java.io.IOException: Too many open files
>>>> at sun.nio.ch.IOUtil.initPipe(Native Method)
>>>> at sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:49)
>>>> at
>>>>
>>>> sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:18)
>>>
>>>> at
>>>>
>>>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.get(SocketIOWithTimeout.java:407)
>>>
>>>> at
>>>>
>>>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:322)
>>>
>>>> at
>>>>
>>>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
>>>
>>>> at
>>>>
>>>> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
>>>
>>>> at
>>>>
>>>> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
>>>
>>>> at
>>>>
>>> java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
>>>
>>>> at java.io.DataOutputStream.write(DataOutputStream.java:90)
>>>> at
>>>>
>>>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2290)
>>>
>>>>
>>>> 2009-11-13 20:06:37,235 WARN org.apache.hadoop.hdfs.DFSClient: Error
>>>> Recovery for block blk_8148813491785406356_5478475 bad datanode[0]
>>>> 192.168.100.123:50010
>>>> 2009-11-13 20:06:37,235 WARN org.apache.hadoop.hdfs.DFSClient: Error
>>>> Recovery for block blk_8148813491785406356_5478475 in pipeline
>>>> 192.168.100.123:50010, 192.168.100.134:50010, 192.168.100.122:50010:
>>>> bad
>>>> datanode 192.168.100.123:50010
>>>> 2009-11-13 20:06:37,436 INFO org.apache.hadoop.hdfs.DFSClient: Exception
>>>>
>>> in
>>>
>>>> createBlockOutputStream java.net.SocketException: Too many open files
>>>> 2009-11-13 20:06:37,436 INFO org.apache.hadoop.hdfs.DFSClient:
>>>> Abandoning
>>>> block blk_2119727700857186236_5478498
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Thu, Nov 12, 2009 at 4:21 PM, Zhenyu Zhong <zhongresearch@gmail.com
>>>> wrote:
>>>>
>>>> Will do.
>>>>>
>>>>> thanks
>>>>> zhenyu
>>>>>
>>>>>
>>>>> On Thu, Nov 12, 2009 at 3:33 PM, stack <st...@duboce.net> wrote:
>>>>>
>>>>> Enable GC logging too on this next run (see hbase-env.sh). Lets try
>>>>>>
>>>>> and
>>>
>>>> get
>>>>>> to the bottom of whats going on.
>>>>>> Thanks,
>>>>>> St.Ack
>>>>>>
>>>>>> On Thu, Nov 12, 2009 at 12:29 PM, Zhenyu Zhong <
>>>>>>
>>>>> zhongresearch@gmail.com
>>>
>>>> wrote:
>>>>>>>
>>>>>>
>>>>>> I can switch the boxes that run zookeeper with the ones that run
>>>>>>> regionservers.
>>>>>>> I will see the results later.
>>>>>>>
>>>>>>> FYI. The node does have the 10 minutes zookeeper.session.timeout
>>>>>>>
>>>>>> value
>>>
>>>> in
>>>>>>
>>>>>>> place.
>>>>>>>
>>>>>>> thanks
>>>>>>> zhenyu
>>>>>>>
>>>>>>> On Thu, Nov 12, 2009 at 3:21 PM, stack <st...@duboce.net> wrote:
>>>>>>>
>>>>>>> On Thu, Nov 12, 2009 at 11:50 AM, Zhenyu Zhong <
>>>>>>>>
>>>>>>> zhongresearch@gmail.com
>>>>>>
>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> In my cluster, half of the cluster have 2 disks 400GB each per
>>>>>>>>>
>>>>>>>> machine,
>>>>>>
>>>>>>> and
>>>>>>>>
>>>>>>>>> half of the cluster have 6 disks per machine. Maybe we should
>>>>>>>>>
>>>>>>>> run
>>>
>>>> zookeeper
>>>>>>>>> on the machines with 2 disks and RS on machines with 6 disks?
>>>>>>>>>
>>>>>>>>> That would make most sense only in the below, it looks like the
>>>>>>>>>
>>>>>>>> RS
>>>
>>>> that
>>>>>>
>>>>>>> had
>>>>>>>> issue had 4 disks?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> BTW, the 10 minutes zookeeper.session.timeout has been set during
>>>>>>>>>
>>>>>>>> the
>>>>>>
>>>>>>> experiment.
>>>>>>>>>
>>>>>>>>>
>>>>>>>> And for sure this node had it in place?
>>>>>>>> St.Ack
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> thanks
>>>>>>>>> zhenyu
>>>>>>>>>
>>>>>>>>> On Thu, Nov 12, 2009 at 2:08 PM, stack <st...@duboce.net> wrote:
>>>>>>>>>
>>>>>>>>> On Thu, Nov 12, 2009 at 8:40 AM, Zhenyu Zhong <
>>>>>>>>>>
>>>>>>>>> zhongresearch@gmail.com
>>>>>>>
>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Though I experienced 2 regionserver disconnection this
>>>>>>>>>>>
>>>>>>>>>> morning,
>>>
>>>> it
>>>>>>
>>>>>>> looks
>>>>>>>>>
>>>>>>>>>> better from the regionserver log. (please see the following
>>>>>>>>>>>
>>>>>>>>>> log)
>>>
>>>>
>>>>>>>>>>>
>>>>>>>>>>> http://pastebin.com/m496dbfae
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I checked diskIO for one of the regionserver(192.168.100.116)
>>>>>>>>>>>
>>>>>>>>>> during
>>>>>>>
>>>>>>>> the
>>>>>>>>>
>>>>>>>>>> time it disconnected.
>>>>>>>>>>>
>>>>>>>>>>> Time: 03:04:58 AM
>>>>>>>>>>> Device: tps Blk_read/s Blk_wrtn/s Blk_read
>>>>>>>>>>>
>>>>>>>>>> Blk_wrtn
>>>>>>>>
>>>>>>>>> sda 105.31 5458.83 19503.64 9043873239
>>>>>>>>>>>
>>>>>>>>>> 32312473676
>>>>>>>>
>>>>>>>>> sda1 2.90 6.64 99.25 10993934
>>>>>>>>>>>
>>>>>>>>>> 164433464
>>>>>>>>
>>>>>>>>> sda2 1.72 23.76 30.25 39365817
>>>>>>>>>>>
>>>>>>>>>> 50124008
>>>>>>>>
>>>>>>>>> sda3 0.30 0.38 3.58 624930
>>>>>>>>>>>
>>>>>>>>>> 5923000
>>>>>>>>
>>>>>>>>> sda4 100.39 5428.06 19370.56 8992888270
>>>>>>>>>>>
>>>>>>>>>> 32091993204
>>>>>>>>
>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Is this high for you? 20k blocks/second would seem to be high
>>>>>>>>>>
>>>>>>>>> but
>>>
>>>> its
>>>>>>>
>>>>>>>> one
>>>>>>>>>
>>>>>>>>>> disk only and its not being shared by zk anymore so shouldn't
>>>>>>>>>>
>>>>>>>>> matter?
>>>>>>
>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I also checked the zookeeper quorum server that the
>>>>>>>>>>>
>>>>>>>>>> regionserver
>>>
>>>> tried
>>>>>>>>
>>>>>>>>> to
>>>>>>>>>
>>>>>>>>>> connect according to the log. However, I don't see
>>>>>>>>>>>
>>>>>>>>>> 192.168.100.116
>>>>>>
>>>>>>> in
>>>>>>>
>>>>>>>> the
>>>>>>>>>
>>>>>>>>>> client list of the zookeeper quorum server that the
>>>>>>>>>>>
>>>>>>>>>> regionserver
>>>
>>>> tried
>>>>>>>>
>>>>>>>>> to
>>>>>>>>>
>>>>>>>>>> connect.
>>>>>>>>>>> Would that be a problem?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Is that because the ephemeral node for the regionserver had
>>>>>>>>>>
>>>>>>>>> evaporated?
>>>>>>>
>>>>>>>> Lost
>>>>>>>>>> its lease w/ zk by the time you went to look?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thu Nov 12 15:04:52 UTC 2009
>>>>>>>>>>> Zookeeper version: 3.2.1-808558, built on 08/27/2009 18:48
>>>>>>>>>>>
>>>>>>>>>> GMT
>>>
>>>> Clients:
>>>>>>>>>>> /192.168.100.127:43045[1](queued=0,recved=26,sent=0)
>>>>>>>>>>> /192.168.100.131:39091[1](queued=0,recved=964,sent=0)
>>>>>>>>>>> /192.168.100.124:35961[1](queued=0,recved=3266,sent=0)
>>>>>>>>>>> /192.168.100.123:47935[1](queued=0,recved=233,sent=0)
>>>>>>>>>>> /192.168.100.125:46931[1](queued=0,recved=2,sent=0)
>>>>>>>>>>> /192.168.100.118:54924[1](queued=0,recved=295,sent=0)
>>>>>>>>>>> /192.168.100.118:41390[1](queued=0,recved=2290,sent=0)
>>>>>>>>>>> /192.168.100.136:42243[1](queued=0,recved=0,sent=0)
>>>>>>>>>>>
>>>>>>>>>>> Latency min/avg/max: 0/17/6333
>>>>>>>>>>> Received: 47111
>>>>>>>>>>> Sent: 0
>>>>>>>>>>> Outstanding: 0
>>>>>>>>>>> Zxid: 0x77000083f4
>>>>>>>>>>> Mode: leader
>>>>>>>>>>> Node count: 23
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> That 6 second maximum latency is pretty bad but should be well
>>>>>>>>>>
>>>>>>>>> within
>>>>>>
>>>>>>> the
>>>>>>>>
>>>>>>>>> zk
>>>>>>>>>> session timeout.
>>>>>>>>>>
>>>>>>>>>> So, problem is likely on the zk client side of the session;
>>>>>>>>>>
>>>>>>>>> i.e.
>>>
>>>> in
>>>>>>
>>>>>>> the
>>>>>>>
>>>>>>>> RS.
>>>>>>>>>
>>>>>>>>>> You could enable GC logging as J-D suggested to see if you have
>>>>>>>>>>
>>>>>>>>> any
>>>>>>
>>>>>>> big
>>>>>>>
>>>>>>>> pauses, pauses > zk session timeout.
>>>>>>>>>>
>>>>>>>>>> When the RS went down, it didn't look too heavily loaded:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 1. 2009-11-12 15:04:52,830 INFO
>>>>>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of
>>>>>>>>>>
>>>>>>>>> metrics:
>>>>>>>
>>>>>>>> request=1.5166667, regions=322, stores=657, storefiles=631,
>>>>>>>>>> storefileIndexSize=61, memstoreSize=1472, usedHeap=2819,
>>>>>>>>>>
>>>>>>>>> maxHeap=4079,
>>>>>>>>
>>>>>>>>> blockCacheSize=658110960, blockCacheFree=197395984,
>>>>>>>>>>
>>>>>>>>> blockCacheCount=9903,
>>>>>>>>>
>>>>>>>>>> blockCacheHitRatio=99
>>>>>>>>>>
>>>>>>>>>> Its serving a few reads? The number of store files seems fine.
>>>>>>>>>>
>>>>>>>>> Not
>>>>>>
>>>>>>> too
>>>>>>>>
>>>>>>>>> much memory used.
>>>>>>>>>>
>>>>>>>>>> Looking at the logs, I see the Lease Still Held exception.
>>>>>>>>>>
>>>>>>>>> This
>>>
>>>> happens
>>>>>>>>
>>>>>>>>> when the RS does its regular report to the master but the
>>>>>>>>>>
>>>>>>>>> master
>>>
>>>> thinks
>>>>>>>
>>>>>>>> the
>>>>>>>>>
>>>>>>>>>> RS has since restarted. It'll think this probably because it
>>>>>>>>>>
>>>>>>>>> noticed
>>>>>>
>>>>>>> that
>>>>>>>>>
>>>>>>>>>> the RS's znode in zk had gone away and it considered the RS
>>>>>>>>>>
>>>>>>>>> dead.
>>>
>>>>
>>>>>>>>>> Looking too at your logs I see this gap in the zk pinging:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 1. 2009-11-12 15:03:39,325 DEBUG
>>>>>>>>>>
>>>>>>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>
>>>>>>> Got
>>>>>>>
>>>>>>>> ping response for sessionid:0x224e55436ad0004 after 0ms
>>>>>>>>>> 2. 2009-11-12 15:03:43,113 DEBUG
>>>>>>>>>>
>>>>>>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>
>>>>>>> Got
>>>>>>>
>>>>>>>> ping response for sessionid:0x24e55436a0007d after 0ms
>>>>>>>>>>
>>>>>>>>>> Where in the lines above it, its reporting about every ten
>>>>>>>>>>
>>>>>>>>> seconds,
>>>>>>
>>>>>>> here
>>>>>>>>
>>>>>>>>> there is a big gap.
>>>>>>>>>>
>>>>>>>>>> Do you have ganglia or something that will let you look more
>>>>>>>>>>
>>>>>>>>> into
>>>
>>>> what
>>>>>>>
>>>>>>>> was
>>>>>>>>>
>>>>>>>>>> happening on this machine around this time? Is the machine OK?
>>>>>>>>>>
>>>>>>>>> It
>>>>>>
>>>>>>> looks
>>>>>>>>
>>>>>>>>> lightly loaded and you have your cluster nicely laid out.
>>>>>>>>>>
>>>>>>>>> Something
>>>>>>
>>>>>>> odd
>>>>>>>>
>>>>>>>>> is
>>>>>>>>>
>>>>>>>>>> going on. What about things like the write speed to disk? In
>>>>>>>>>>
>>>>>>>>> the
>>>
>>>> past
>>>>>>>
>>>>>>>> strange issues have been explained by incorrectly set BIOS
>>>>>>>>>>
>>>>>>>>> which
>>>
>>>> made
>>>>>>
>>>>>>> disks
>>>>>>>>>
>>>>>>>>>> run at 1/100th of their proper speed.
>>>>>>>>>>
>>>>>>>>>> St.Ack
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Best,
>>>>>>>>>>> zhenyu
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Nov 11, 2009 at 3:58 PM, Zhenyu Zhong <
>>>>>>>>>>>
>>>>>>>>>> zhongresearch@gmail.com
>>>>>>>>
>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Stack
>>>>>>>>>>>>
>>>>>>>>>>>> I am very appreciated for your comments.
>>>>>>>>>>>> I will use the zookeeper monitoring script on my cluster
>>>>>>>>>>>>
>>>>>>>>>>> and
>>>
>>>> let
>>>>>>
>>>>>>> it
>>>>>>>
>>>>>>>> run
>>>>>>>>>
>>>>>>>>>> overnight to see the result.
>>>>>>>>>>>> I will follow up the thread when I get anything.
>>>>>>>>>>>>
>>>>>>>>>>>> thanks
>>>>>>>>>>>> zhenyu
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Nov 11, 2009 at 3:52 PM, stack <st...@duboce.net>
>>>>>>>>>>>>
>>>>>>>>>>> wrote:
>>>>>>
>>>>>>>
>>>>>>>>>>>> I see these in your log too:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> 1. 2009-11-11 04:27:19,018 DEBUG
>>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>>>
>>>>>>>>> Got
>>>>>>>>>>
>>>>>>>>>>> ping response for sessionid:0x424dfd908c50009 after
>>>>>>>>>>>>>
>>>>>>>>>>>> 4544ms
>>>
>>>> 2. 2009-11-11 04:27:19,018 DEBUG
>>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>>>
>>>>>>>>> Got
>>>>>>>>>>
>>>>>>>>>>> ping response for sessionid:0x24dfd90c810002 after
>>>>>>>>>>>>>
>>>>>>>>>>>> 4548ms
>>>
>>>> 3. 2009-11-11 04:27:43,960 DEBUG
>>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>>>
>>>>>>>>> Got
>>>>>>>>>>
>>>>>>>>>>> ping response for sessionid:0x424dfd908c50009 after
>>>>>>>>>>>>>
>>>>>>>>>>>> 9030ms
>>>
>>>> 4. 2009-11-11 04:27:43,960 DEBUG
>>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.zookeeper.
>>>>>>>>
>>>>>>>
Re: regionserver disconnection
Posted by Stack <sa...@gmail.com>.
You upped the ulimit file descriptors as per the getting started doc?
On Nov 13, 2009, at 1:26 PM, Zhenyu Zhong <zh...@gmail.com>
wrote:
> Thanks a lot.
>
> Bad news is my kernel is still 2.6.26.
> But it was not a problem before.
>
> Very strange.
>
> zhenyu
>
> On Fri, Nov 13, 2009 at 4:16 PM, Jean-Daniel Cryans <jdcryans@apache.org
> >wrote:
>
>> Looks like
>> http://pero.blogs.aprilmayjune.org/2009/01/22/hadoop-and-linux-kernel-2627-epoll-limits/
>>
>> J-D
>>
>> On Fri, Nov 13, 2009 at 1:12 PM, Zhenyu Zhong <zhongresearch@gmail.com
>> >
>> wrote:
>>> Hi,
>>>
>>>
>>> After I re-organize the cluster, the experiment ran into problem
>>> faster
>> than
>>> before.
>>>
>>> Basically the changes are to use machines with less resources as
>> zookeeper
>>> quorums and machines with more resources as regionserver.
>>>
>>> From the log, I still see the pause around 1 minute.
>>> I enabled the GC logging, please see
>>>
>>> http://pastebin.com/m1d4ce0f1
>>>
>>> for details.
>>> In general I don't see many pauses in the GC.
>>>
>>> What is more interesting, 1 hour after the 1st regionserver
>>> disconnected,
>>> the master log complained about too many open files. This didn't
>>> happen
>>> before.
>>> I checked the system OS setup as well as the limits.conf. I also
>>> checked
>> the
>>> running processes. The total open files don't reach the limit. I am
>> confused
>>> a bit.
>>>
>>>
>>> Please see the following master log.
>>>
>>> 2009-11-13 20:06:02,114 INFO
>>> org.apache.hadoop.hbase.master.BaseScanner:
>>> RegionManager.metaScanner scan of 4658 row(s) of meta region
>>> {server:
>>> 192.168.100.128:60021, regionname: .META.,,1, startKey: <>} complete
>>> 2009-11-13 20:06:02,114 INFO
>>> org.apache.hadoop.hbase.master.BaseScanner:
>> All
>>> 1 .META. region(s) scanned
>>> 2009-11-13 20:06:07,677 DEBUG org.apache.zookeeper.ClientCnxn: Got
>>> ping
>>> response for sessionid:0x424eebf1c10004c after 3ms
>>> 2009-11-13 20:06:08,178 INFO org.apache.hadoop.hdfs.DFSClient:
>>> Exception
>> in
>>> createBlockOutputStream java.io.IOException: Bad connect ack with
>>> firstBadLink 192.168.100.123:50010
>>> 2009-11-13 20:06:08,178 INFO org.apache.hadoop.hdfs.DFSClient:
>>> Abandoning
>>> block blk_-2808245019291145247_5478039
>>> 2009-11-13 20:06:09,682 INFO org.apache.hadoop.hdfs.DFSClient:
>>> Exception
>> in
>>> createBlockOutputStream java.io.EOFException
>>> 2009-11-13 20:06:09,682 INFO org.apache.hadoop.hdfs.DFSClient:
>>> Abandoning
>>> block blk_1074853606841896259_5478048
>>> 2009-11-13 20:06:10,334 DEBUG org.apache.zookeeper.ClientCnxn: Got
>>> ping
>>> response for sessionid:0x24eebf1043003c after 1ms
>>> 2009-11-13 20:06:21,018 DEBUG org.apache.zookeeper.ClientCnxn: Got
>>> ping
>>> response for sessionid:0x424eebf1c10004c after 0ms
>>> 2009-11-13 20:06:23,674 DEBUG org.apache.zookeeper.ClientCnxn: Got
>>> ping
>>> response for sessionid:0x24eebf1043003c after 0ms
>>> 2009-11-13 20:06:24,828 INFO org.apache.hadoop.hdfs.DFSClient:
>>> Exception
>> in
>>> createBlockOutputStream java.io.IOException: Bad connect ack with
>>> firstBadLink 192.168.100.123:50010
>>> 2009-11-13 20:06:24,828 INFO org.apache.hadoop.hdfs.DFSClient:
>>> Abandoning
>>> block blk_-6642544517082142289_5478063
>>> 2009-11-13 20:06:24,828 INFO org.apache.hadoop.hdfs.DFSClient:
>>> Exception
>> in
>>> createBlockOutputStream java.net.SocketException: Too many open
>>> files
>>> 2009-11-13 20:06:24,828 INFO org.apache.hadoop.hdfs.DFSClient:
>>> Abandoning
>>> block blk_2057511041109796090_5478063
>>> 2009-11-13 20:06:24,928 INFO org.apache.hadoop.hdfs.DFSClient:
>>> Exception
>> in
>>> createBlockOutputStream java.net.SocketException: Too many open
>>> files
>>> 2009-11-13 20:06:24,928 INFO org.apache.hadoop.hdfs.DFSClient:
>>> Abandoning
>>> block blk_8219260302213892894_5478064
>>> 2009-11-13 20:06:30,855 INFO org.apache.hadoop.hdfs.DFSClient:
>>> Exception
>> in
>>> createBlockOutputStream java.net.SocketException: Too many open
>>> files
>>> 2009-11-13 20:06:30,855 INFO org.apache.hadoop.hdfs.DFSClient:
>>> Abandoning
>>> block blk_1669205542853067709_5478235
>>> 2009-11-13 20:06:30,905 INFO org.apache.hadoop.hdfs.DFSClient:
>>> Exception
>> in
>>> createBlockOutputStream java.net.SocketException: Too many open
>>> files
>>> 2009-11-13 20:06:30,905 INFO org.apache.hadoop.hdfs.DFSClient:
>>> Abandoning
>>> block blk_9128897691346270351_5478237
>>> 2009-11-13 20:06:30,955 INFO org.apache.hadoop.hdfs.DFSClient:
>>> Exception
>> in
>>> createBlockOutputStream java.net.SocketException: Too many open
>>> files
>>> 2009-11-13 20:06:30,955 INFO org.apache.hadoop.hdfs.DFSClient:
>>> Abandoning
>>> block blk_1116845144864123018_5478240
>>> 2009-11-13 20:06:34,372 DEBUG org.apache.zookeeper.ClientCnxn: Got
>>> ping
>>> response for sessionid:0x424eebf1c10004c after 0ms
>>> 2009-11-13 20:06:37,034 DEBUG org.apache.zookeeper.ClientCnxn: Got
>>> ping
>>> response for sessionid:0x24eebf1043003c after 0ms
>>> 2009-11-13 20:06:37,235 WARN org.apache.hadoop.hdfs.DFSClient:
>> DataStreamer
>>> Exception: java.io.IOException: Too many open files
>>> at sun.nio.ch.IOUtil.initPipe(Native Method)
>>> at sun.nio.ch.EPollSelectorImpl.<init>
>>> (EPollSelectorImpl.java:49)
>>> at
>>>
>> sun.nio.ch.EPollSelectorProvider.openSelector
>> (EPollSelectorProvider.java:18)
>>> at
>>>
>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.get
>> (SocketIOWithTimeout.java:407)
>>> at
>>>
>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select
>> (SocketIOWithTimeout.java:322)
>>> at
>>>
>> org.apache.hadoop.net.SocketIOWithTimeout.doIO
>> (SocketIOWithTimeout.java:157)
>>> at
>>>
>> org.apache.hadoop.net.SocketOutputStream.write
>> (SocketOutputStream.java:146)
>>> at
>>>
>> org.apache.hadoop.net.SocketOutputStream.write
>> (SocketOutputStream.java:107)
>>> at
>> java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
>>> at java.io.DataOutputStream.write(DataOutputStream.java:90)
>>> at
>>>
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run
>> (DFSClient.java:2290)
>>>
>>> 2009-11-13 20:06:37,235 WARN org.apache.hadoop.hdfs.DFSClient: Error
>>> Recovery for block blk_8148813491785406356_5478475 bad datanode[0]
>>> 192.168.100.123:50010
>>> 2009-11-13 20:06:37,235 WARN org.apache.hadoop.hdfs.DFSClient: Error
>>> Recovery for block blk_8148813491785406356_5478475 in pipeline
>>> 192.168.100.123:50010, 192.168.100.134:50010,
>>> 192.168.100.122:50010: bad
>>> datanode 192.168.100.123:50010
>>> 2009-11-13 20:06:37,436 INFO org.apache.hadoop.hdfs.DFSClient:
>>> Exception
>> in
>>> createBlockOutputStream java.net.SocketException: Too many open
>>> files
>>> 2009-11-13 20:06:37,436 INFO org.apache.hadoop.hdfs.DFSClient:
>>> Abandoning
>>> block blk_2119727700857186236_5478498
>>>
>>>
>>>
>>>
>>>
>>> On Thu, Nov 12, 2009 at 4:21 PM, Zhenyu Zhong <zhongresearch@gmail.com
>>> wrote:
>>>
>>>> Will do.
>>>>
>>>> thanks
>>>> zhenyu
>>>>
>>>>
>>>> On Thu, Nov 12, 2009 at 3:33 PM, stack <st...@duboce.net> wrote:
>>>>
>>>>> Enable GC logging too on this next run (see hbase-env.sh). Lets
>>>>> try
>> and
>>>>> get
>>>>> to the bottom of whats going on.
>>>>> Thanks,
>>>>> St.Ack
>>>>>
>>>>> On Thu, Nov 12, 2009 at 12:29 PM, Zhenyu Zhong <
>> zhongresearch@gmail.com
>>>>>> wrote:
>>>>>
>>>>>> I can switch the boxes that run zookeeper with the ones that run
>>>>>> regionservers.
>>>>>> I will see the results later.
>>>>>>
>>>>>> FYI. The node does have the 10 minutes zookeeper.session.timeout
>> value
>>>>> in
>>>>>> place.
>>>>>>
>>>>>> thanks
>>>>>> zhenyu
>>>>>>
>>>>>> On Thu, Nov 12, 2009 at 3:21 PM, stack <st...@duboce.net> wrote:
>>>>>>
>>>>>>> On Thu, Nov 12, 2009 at 11:50 AM, Zhenyu Zhong <
>>>>> zhongresearch@gmail.com
>>>>>>>> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>> In my cluster, half of the cluster have 2 disks 400GB each per
>>>>> machine,
>>>>>>> and
>>>>>>>> half of the cluster have 6 disks per machine. Maybe we should
>> run
>>>>>>>> zookeeper
>>>>>>>> on the machines with 2 disks and RS on machines with 6 disks?
>>>>>>>>
>>>>>>>> That would make most sense only in the below, it looks like the
>> RS
>>>>> that
>>>>>>> had
>>>>>>> issue had 4 disks?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> BTW, the 10 minutes zookeeper.session.timeout has been set
>>>>>>>> during
>>>>> the
>>>>>>>> experiment.
>>>>>>>>
>>>>>>>
>>>>>>> And for sure this node had it in place?
>>>>>>> St.Ack
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> thanks
>>>>>>>> zhenyu
>>>>>>>>
>>>>>>>> On Thu, Nov 12, 2009 at 2:08 PM, stack <st...@duboce.net>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> On Thu, Nov 12, 2009 at 8:40 AM, Zhenyu Zhong <
>>>>>> zhongresearch@gmail.com
>>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Though I experienced 2 regionserver disconnection this
>> morning,
>>>>> it
>>>>>>>> looks
>>>>>>>>>> better from the regionserver log. (please see the following
>> log)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> http://pastebin.com/m496dbfae
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I checked diskIO for one of the regionserver(192.168.100.116)
>>>>>> during
>>>>>>>> the
>>>>>>>>>> time it disconnected.
>>>>>>>>>>
>>>>>>>>>> Time: 03:04:58 AM
>>>>>>>>>> Device: tps Blk_read/s Blk_wrtn/s Blk_read
>>>>>>> Blk_wrtn
>>>>>>>>>> sda 105.31 5458.83 19503.64 9043873239
>>>>>>> 32312473676
>>>>>>>>>> sda1 2.90 6.64 99.25 10993934
>>>>>>> 164433464
>>>>>>>>>> sda2 1.72 23.76 30.25 39365817
>>>>>>> 50124008
>>>>>>>>>> sda3 0.30 0.38 3.58 624930
>>>>>>> 5923000
>>>>>>>>>> sda4 100.39 5428.06 19370.56 8992888270
>>>>>>> 32091993204
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> Is this high for you? 20k blocks/second would seem to be high
>> but
>>>>>> its
>>>>>>>> one
>>>>>>>>> disk only and its not being shared by zk anymore so shouldn't
>>>>> matter?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> I also checked the zookeeper quorum server that the
>> regionserver
>>>>>>> tried
>>>>>>>> to
>>>>>>>>>> connect according to the log. However, I don't see
>>>>> 192.168.100.116
>>>>>> in
>>>>>>>> the
>>>>>>>>>> client list of the zookeeper quorum server that the
>> regionserver
>>>>>>> tried
>>>>>>>> to
>>>>>>>>>> connect.
>>>>>>>>>> Would that be a problem?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> Is that because the ephemeral node for the regionserver had
>>>>>> evaporated?
>>>>>>>>> Lost
>>>>>>>>> its lease w/ zk by the time you went to look?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Thu Nov 12 15:04:52 UTC 2009
>>>>>>>>>> Zookeeper version: 3.2.1-808558, built on 08/27/2009 18:48
>> GMT
>>>>>>>>>> Clients:
>>>>>>>>>> /192.168.100.127:43045[1](queued=0,recved=26,sent=0)
>>>>>>>>>> /192.168.100.131:39091[1](queued=0,recved=964,sent=0)
>>>>>>>>>> /192.168.100.124:35961[1](queued=0,recved=3266,sent=0)
>>>>>>>>>> /192.168.100.123:47935[1](queued=0,recved=233,sent=0)
>>>>>>>>>> /192.168.100.125:46931[1](queued=0,recved=2,sent=0)
>>>>>>>>>> /192.168.100.118:54924[1](queued=0,recved=295,sent=0)
>>>>>>>>>> /192.168.100.118:41390[1](queued=0,recved=2290,sent=0)
>>>>>>>>>> /192.168.100.136:42243[1](queued=0,recved=0,sent=0)
>>>>>>>>>>
>>>>>>>>>> Latency min/avg/max: 0/17/6333
>>>>>>>>>> Received: 47111
>>>>>>>>>> Sent: 0
>>>>>>>>>> Outstanding: 0
>>>>>>>>>> Zxid: 0x77000083f4
>>>>>>>>>> Mode: leader
>>>>>>>>>> Node count: 23
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> That 6 second maximum latency is pretty bad but should be well
>>>>> within
>>>>>>> the
>>>>>>>>> zk
>>>>>>>>> session timeout.
>>>>>>>>>
>>>>>>>>> So, problem is likely on the zk client side of the session;
>> i.e.
>>>>> in
>>>>>> the
>>>>>>>> RS.
>>>>>>>>> You could enable GC logging as J-D suggested to see if you
>>>>>>>>> have
>>>>> any
>>>>>> big
>>>>>>>>> pauses, pauses > zk session timeout.
>>>>>>>>>
>>>>>>>>> When the RS went down, it didn't look too heavily loaded:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 1. 2009-11-12 15:04:52,830 INFO
>>>>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of
>>>>>> metrics:
>>>>>>>>> request=1.5166667, regions=322, stores=657, storefiles=631,
>>>>>>>>> storefileIndexSize=61, memstoreSize=1472, usedHeap=2819,
>>>>>>> maxHeap=4079,
>>>>>>>>> blockCacheSize=658110960, blockCacheFree=197395984,
>>>>>>>> blockCacheCount=9903,
>>>>>>>>> blockCacheHitRatio=99
>>>>>>>>>
>>>>>>>>> Its serving a few reads? The number of store files seems
>>>>>>>>> fine.
>>>>> Not
>>>>>>> too
>>>>>>>>> much memory used.
>>>>>>>>>
>>>>>>>>> Looking at the logs, I see the Lease Still Held exception.
>> This
>>>>>>> happens
>>>>>>>>> when the RS does its regular report to the master but the
>> master
>>>>>> thinks
>>>>>>>> the
>>>>>>>>> RS has since restarted. It'll think this probably because it
>>>>> noticed
>>>>>>>> that
>>>>>>>>> the RS's znode in zk had gone away and it considered the RS
>> dead.
>>>>>>>>>
>>>>>>>>> Looking too at your logs I see this gap in the zk pinging:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 1. 2009-11-12 15:03:39,325 DEBUG
>>>>> org.apache.zookeeper.ClientCnxn:
>>>>>> Got
>>>>>>>>> ping response for sessionid:0x224e55436ad0004 after 0ms
>>>>>>>>> 2. 2009-11-12 15:03:43,113 DEBUG
>>>>> org.apache.zookeeper.ClientCnxn:
>>>>>> Got
>>>>>>>>> ping response for sessionid:0x24e55436a0007d after 0ms
>>>>>>>>>
>>>>>>>>> Where in the lines above it, its reporting about every ten
>>>>> seconds,
>>>>>>> here
>>>>>>>>> there is a big gap.
>>>>>>>>>
>>>>>>>>> Do you have ganglia or something that will let you look more
>> into
>>>>>> what
>>>>>>>> was
>>>>>>>>> happening on this machine around this time? Is the machine
>>>>>>>>> OK?
>>>>> It
>>>>>>> looks
>>>>>>>>> lightly loaded and you have your cluster nicely laid out.
>>>>> Something
>>>>>>> odd
>>>>>>>> is
>>>>>>>>> going on. What about things like the write speed to disk? In
>> the
>>>>>> past
>>>>>>>>> strange issues have been explained by incorrectly set BIOS
>> which
>>>>> made
>>>>>>>> disks
>>>>>>>>> run at 1/100th of their proper speed.
>>>>>>>>>
>>>>>>>>> St.Ack
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Best,
>>>>>>>>>> zhenyu
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, Nov 11, 2009 at 3:58 PM, Zhenyu Zhong <
>>>>>>> zhongresearch@gmail.com
>>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Stack
>>>>>>>>>>>
>>>>>>>>>>> I am very appreciated for your comments.
>>>>>>>>>>> I will use the zookeeper monitoring script on my cluster
>> and
>>>>> let
>>>>>> it
>>>>>>>> run
>>>>>>>>>>> overnight to see the result.
>>>>>>>>>>> I will follow up the thread when I get anything.
>>>>>>>>>>>
>>>>>>>>>>> thanks
>>>>>>>>>>> zhenyu
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Nov 11, 2009 at 3:52 PM, stack <st...@duboce.net>
>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> I see these in your log too:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> 1. 2009-11-11 04:27:19,018 DEBUG
>>>>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>>>> Got
>>>>>>>>>>>> ping response for sessionid:0x424dfd908c50009 after
>> 4544ms
>>>>>>>>>>>> 2. 2009-11-11 04:27:19,018 DEBUG
>>>>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>>>> Got
>>>>>>>>>>>> ping response for sessionid:0x24dfd90c810002 after
>> 4548ms
>>>>>>>>>>>> 3. 2009-11-11 04:27:43,960 DEBUG
>>>>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>>>> Got
>>>>>>>>>>>> ping response for sessionid:0x424dfd908c50009 after
>> 9030ms
>>>>>>>>>>>> 4. 2009-11-11 04:27:43,960 DEBUG
>>>>>>> org.apache.zookeeper.
Re: regionserver disconnection
Posted by Zhenyu Zhong <zh...@gmail.com>.
Thanks a lot.
Bad news is my kernel is still 2.6.26.
But it was not a problem before.
Very strange.
zhenyu
On Fri, Nov 13, 2009 at 4:16 PM, Jean-Daniel Cryans <jd...@apache.org>wrote:
> Looks like
> http://pero.blogs.aprilmayjune.org/2009/01/22/hadoop-and-linux-kernel-2627-epoll-limits/
>
> J-D
>
> On Fri, Nov 13, 2009 at 1:12 PM, Zhenyu Zhong <zh...@gmail.com>
> wrote:
> > Hi,
> >
> >
> > After I re-organize the cluster, the experiment ran into problem faster
> than
> > before.
> >
> > Basically the changes are to use machines with less resources as
> zookeeper
> > quorums and machines with more resources as regionserver.
> >
> > From the log, I still see the pause around 1 minute.
> > I enabled the GC logging, please see
> >
> > http://pastebin.com/m1d4ce0f1
> >
> > for details.
> > In general I don't see many pauses in the GC.
> >
> > What is more interesting, 1 hour after the 1st regionserver disconnected,
> > the master log complained about too many open files. This didn't happen
> > before.
> > I checked the system OS setup as well as the limits.conf. I also checked
> the
> > running processes. The total open files don't reach the limit. I am
> confused
> > a bit.
> >
> >
> > Please see the following master log.
> >
> > 2009-11-13 20:06:02,114 INFO org.apache.hadoop.hbase.master.BaseScanner:
> > RegionManager.metaScanner scan of 4658 row(s) of meta region {server:
> > 192.168.100.128:60021, regionname: .META.,,1, startKey: <>} complete
> > 2009-11-13 20:06:02,114 INFO org.apache.hadoop.hbase.master.BaseScanner:
> All
> > 1 .META. region(s) scanned
> > 2009-11-13 20:06:07,677 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
> > response for sessionid:0x424eebf1c10004c after 3ms
> > 2009-11-13 20:06:08,178 INFO org.apache.hadoop.hdfs.DFSClient: Exception
> in
> > createBlockOutputStream java.io.IOException: Bad connect ack with
> > firstBadLink 192.168.100.123:50010
> > 2009-11-13 20:06:08,178 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning
> > block blk_-2808245019291145247_5478039
> > 2009-11-13 20:06:09,682 INFO org.apache.hadoop.hdfs.DFSClient: Exception
> in
> > createBlockOutputStream java.io.EOFException
> > 2009-11-13 20:06:09,682 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning
> > block blk_1074853606841896259_5478048
> > 2009-11-13 20:06:10,334 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
> > response for sessionid:0x24eebf1043003c after 1ms
> > 2009-11-13 20:06:21,018 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
> > response for sessionid:0x424eebf1c10004c after 0ms
> > 2009-11-13 20:06:23,674 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
> > response for sessionid:0x24eebf1043003c after 0ms
> > 2009-11-13 20:06:24,828 INFO org.apache.hadoop.hdfs.DFSClient: Exception
> in
> > createBlockOutputStream java.io.IOException: Bad connect ack with
> > firstBadLink 192.168.100.123:50010
> > 2009-11-13 20:06:24,828 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning
> > block blk_-6642544517082142289_5478063
> > 2009-11-13 20:06:24,828 INFO org.apache.hadoop.hdfs.DFSClient: Exception
> in
> > createBlockOutputStream java.net.SocketException: Too many open files
> > 2009-11-13 20:06:24,828 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning
> > block blk_2057511041109796090_5478063
> > 2009-11-13 20:06:24,928 INFO org.apache.hadoop.hdfs.DFSClient: Exception
> in
> > createBlockOutputStream java.net.SocketException: Too many open files
> > 2009-11-13 20:06:24,928 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning
> > block blk_8219260302213892894_5478064
> > 2009-11-13 20:06:30,855 INFO org.apache.hadoop.hdfs.DFSClient: Exception
> in
> > createBlockOutputStream java.net.SocketException: Too many open files
> > 2009-11-13 20:06:30,855 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning
> > block blk_1669205542853067709_5478235
> > 2009-11-13 20:06:30,905 INFO org.apache.hadoop.hdfs.DFSClient: Exception
> in
> > createBlockOutputStream java.net.SocketException: Too many open files
> > 2009-11-13 20:06:30,905 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning
> > block blk_9128897691346270351_5478237
> > 2009-11-13 20:06:30,955 INFO org.apache.hadoop.hdfs.DFSClient: Exception
> in
> > createBlockOutputStream java.net.SocketException: Too many open files
> > 2009-11-13 20:06:30,955 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning
> > block blk_1116845144864123018_5478240
> > 2009-11-13 20:06:34,372 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
> > response for sessionid:0x424eebf1c10004c after 0ms
> > 2009-11-13 20:06:37,034 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
> > response for sessionid:0x24eebf1043003c after 0ms
> > 2009-11-13 20:06:37,235 WARN org.apache.hadoop.hdfs.DFSClient:
> DataStreamer
> > Exception: java.io.IOException: Too many open files
> > at sun.nio.ch.IOUtil.initPipe(Native Method)
> > at sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:49)
> > at
> >
> sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:18)
> > at
> >
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.get(SocketIOWithTimeout.java:407)
> > at
> >
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:322)
> > at
> >
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
> > at
> >
> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
> > at
> >
> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
> > at
> java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
> > at java.io.DataOutputStream.write(DataOutputStream.java:90)
> > at
> >
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2290)
> >
> > 2009-11-13 20:06:37,235 WARN org.apache.hadoop.hdfs.DFSClient: Error
> > Recovery for block blk_8148813491785406356_5478475 bad datanode[0]
> > 192.168.100.123:50010
> > 2009-11-13 20:06:37,235 WARN org.apache.hadoop.hdfs.DFSClient: Error
> > Recovery for block blk_8148813491785406356_5478475 in pipeline
> > 192.168.100.123:50010, 192.168.100.134:50010, 192.168.100.122:50010: bad
> > datanode 192.168.100.123:50010
> > 2009-11-13 20:06:37,436 INFO org.apache.hadoop.hdfs.DFSClient: Exception
> in
> > createBlockOutputStream java.net.SocketException: Too many open files
> > 2009-11-13 20:06:37,436 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning
> > block blk_2119727700857186236_5478498
> >
> >
> >
> >
> >
> > On Thu, Nov 12, 2009 at 4:21 PM, Zhenyu Zhong <zhongresearch@gmail.com
> >wrote:
> >
> >> Will do.
> >>
> >> thanks
> >> zhenyu
> >>
> >>
> >> On Thu, Nov 12, 2009 at 3:33 PM, stack <st...@duboce.net> wrote:
> >>
> >>> Enable GC logging too on this next run (see hbase-env.sh). Lets try
> and
> >>> get
> >>> to the bottom of whats going on.
> >>> Thanks,
> >>> St.Ack
> >>>
> >>> On Thu, Nov 12, 2009 at 12:29 PM, Zhenyu Zhong <
> zhongresearch@gmail.com
> >>> >wrote:
> >>>
> >>> > I can switch the boxes that run zookeeper with the ones that run
> >>> > regionservers.
> >>> > I will see the results later.
> >>> >
> >>> > FYI. The node does have the 10 minutes zookeeper.session.timeout
> value
> >>> in
> >>> > place.
> >>> >
> >>> > thanks
> >>> > zhenyu
> >>> >
> >>> > On Thu, Nov 12, 2009 at 3:21 PM, stack <st...@duboce.net> wrote:
> >>> >
> >>> > > On Thu, Nov 12, 2009 at 11:50 AM, Zhenyu Zhong <
> >>> zhongresearch@gmail.com
> >>> > > >wrote:
> >>> > >
> >>> > > >
> >>> > > > In my cluster, half of the cluster have 2 disks 400GB each per
> >>> machine,
> >>> > > and
> >>> > > > half of the cluster have 6 disks per machine. Maybe we should
> run
> >>> > > > zookeeper
> >>> > > > on the machines with 2 disks and RS on machines with 6 disks?
> >>> > > >
> >>> > > > That would make most sense only in the below, it looks like the
> RS
> >>> that
> >>> > > had
> >>> > > issue had 4 disks?
> >>> > >
> >>> > >
> >>> > >
> >>> > > > BTW, the 10 minutes zookeeper.session.timeout has been set during
> >>> the
> >>> > > > experiment.
> >>> > > >
> >>> > >
> >>> > > And for sure this node had it in place?
> >>> > > St.Ack
> >>> > >
> >>> > >
> >>> > >
> >>> > > >
> >>> > > > thanks
> >>> > > > zhenyu
> >>> > > >
> >>> > > > On Thu, Nov 12, 2009 at 2:08 PM, stack <st...@duboce.net> wrote:
> >>> > > >
> >>> > > > > On Thu, Nov 12, 2009 at 8:40 AM, Zhenyu Zhong <
> >>> > zhongresearch@gmail.com
> >>> > > > > >wrote:
> >>> > > > >
> >>> > > > >
> >>> > > > > > Though I experienced 2 regionserver disconnection this
> morning,
> >>> it
> >>> > > > looks
> >>> > > > > > better from the regionserver log. (please see the following
> log)
> >>> > > > > >
> >>> > > > > >
> >>> > > > > > http://pastebin.com/m496dbfae
> >>> > > > > >
> >>> > > > > >
> >>> > > > > > I checked diskIO for one of the regionserver(192.168.100.116)
> >>> > during
> >>> > > > the
> >>> > > > > > time it disconnected.
> >>> > > > > >
> >>> > > > > > Time: 03:04:58 AM
> >>> > > > > > Device: tps Blk_read/s Blk_wrtn/s Blk_read
> >>> > > Blk_wrtn
> >>> > > > > > sda 105.31 5458.83 19503.64 9043873239
> >>> > > 32312473676
> >>> > > > > > sda1 2.90 6.64 99.25 10993934
> >>> > > 164433464
> >>> > > > > > sda2 1.72 23.76 30.25 39365817
> >>> > > 50124008
> >>> > > > > > sda3 0.30 0.38 3.58 624930
> >>> > > 5923000
> >>> > > > > > sda4 100.39 5428.06 19370.56 8992888270
> >>> > > 32091993204
> >>> > > > > >
> >>> > > > > >
> >>> > > > > Is this high for you? 20k blocks/second would seem to be high
> but
> >>> > its
> >>> > > > one
> >>> > > > > disk only and its not being shared by zk anymore so shouldn't
> >>> matter?
> >>> > > > >
> >>> > > > >
> >>> > > > > > I also checked the zookeeper quorum server that the
> regionserver
> >>> > > tried
> >>> > > > to
> >>> > > > > > connect according to the log. However, I don't see
> >>> 192.168.100.116
> >>> > in
> >>> > > > the
> >>> > > > > > client list of the zookeeper quorum server that the
> regionserver
> >>> > > tried
> >>> > > > to
> >>> > > > > > connect.
> >>> > > > > > Would that be a problem?
> >>> > > > > >
> >>> > > > > >
> >>> > > > > Is that because the ephemeral node for the regionserver had
> >>> > evaporated?
> >>> > > > > Lost
> >>> > > > > its lease w/ zk by the time you went to look?
> >>> > > > >
> >>> > > > >
> >>> > > > >
> >>> > > > >
> >>> > > > > > Thu Nov 12 15:04:52 UTC 2009
> >>> > > > > > Zookeeper version: 3.2.1-808558, built on 08/27/2009 18:48
> GMT
> >>> > > > > > Clients:
> >>> > > > > > /192.168.100.127:43045[1](queued=0,recved=26,sent=0)
> >>> > > > > > /192.168.100.131:39091[1](queued=0,recved=964,sent=0)
> >>> > > > > > /192.168.100.124:35961[1](queued=0,recved=3266,sent=0)
> >>> > > > > > /192.168.100.123:47935[1](queued=0,recved=233,sent=0)
> >>> > > > > > /192.168.100.125:46931[1](queued=0,recved=2,sent=0)
> >>> > > > > > /192.168.100.118:54924[1](queued=0,recved=295,sent=0)
> >>> > > > > > /192.168.100.118:41390[1](queued=0,recved=2290,sent=0)
> >>> > > > > > /192.168.100.136:42243[1](queued=0,recved=0,sent=0)
> >>> > > > > >
> >>> > > > > > Latency min/avg/max: 0/17/6333
> >>> > > > > > Received: 47111
> >>> > > > > > Sent: 0
> >>> > > > > > Outstanding: 0
> >>> > > > > > Zxid: 0x77000083f4
> >>> > > > > > Mode: leader
> >>> > > > > > Node count: 23
> >>> > > > > >
> >>> > > > >
> >>> > > > >
> >>> > > > > That 6 second maximum latency is pretty bad but should be well
> >>> within
> >>> > > the
> >>> > > > > zk
> >>> > > > > session timeout.
> >>> > > > >
> >>> > > > > So, problem is likely on the zk client side of the session;
> i.e.
> >>> in
> >>> > the
> >>> > > > RS.
> >>> > > > > You could enable GC logging as J-D suggested to see if you have
> >>> any
> >>> > big
> >>> > > > > pauses, pauses > zk session timeout.
> >>> > > > >
> >>> > > > > When the RS went down, it didn't look too heavily loaded:
> >>> > > > >
> >>> > > > >
> >>> > > > > 1. 2009-11-12 15:04:52,830 INFO
> >>> > > > > org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of
> >>> > metrics:
> >>> > > > > request=1.5166667, regions=322, stores=657, storefiles=631,
> >>> > > > > storefileIndexSize=61, memstoreSize=1472, usedHeap=2819,
> >>> > > maxHeap=4079,
> >>> > > > > blockCacheSize=658110960, blockCacheFree=197395984,
> >>> > > > blockCacheCount=9903,
> >>> > > > > blockCacheHitRatio=99
> >>> > > > >
> >>> > > > > Its serving a few reads? The number of store files seems fine.
> >>> Not
> >>> > > too
> >>> > > > > much memory used.
> >>> > > > >
> >>> > > > > Looking at the logs, I see the Lease Still Held exception.
> This
> >>> > > happens
> >>> > > > > when the RS does its regular report to the master but the
> master
> >>> > thinks
> >>> > > > the
> >>> > > > > RS has since restarted. It'll think this probably because it
> >>> noticed
> >>> > > > that
> >>> > > > > the RS's znode in zk had gone away and it considered the RS
> dead.
> >>> > > > >
> >>> > > > > Looking too at your logs I see this gap in the zk pinging:
> >>> > > > >
> >>> > > > >
> >>> > > > > 1. 2009-11-12 15:03:39,325 DEBUG
> >>> org.apache.zookeeper.ClientCnxn:
> >>> > Got
> >>> > > > > ping response for sessionid:0x224e55436ad0004 after 0ms
> >>> > > > > 2. 2009-11-12 15:03:43,113 DEBUG
> >>> org.apache.zookeeper.ClientCnxn:
> >>> > Got
> >>> > > > > ping response for sessionid:0x24e55436a0007d after 0ms
> >>> > > > >
> >>> > > > > Where in the lines above it, its reporting about every ten
> >>> seconds,
> >>> > > here
> >>> > > > > there is a big gap.
> >>> > > > >
> >>> > > > > Do you have ganglia or something that will let you look more
> into
> >>> > what
> >>> > > > was
> >>> > > > > happening on this machine around this time? Is the machine OK?
> >>> It
> >>> > > looks
> >>> > > > > lightly loaded and you have your cluster nicely laid out.
> >>> Something
> >>> > > odd
> >>> > > > is
> >>> > > > > going on. What about things like the write speed to disk? In
> the
> >>> > past
> >>> > > > > strange issues have been explained by incorrectly set BIOS
> which
> >>> made
> >>> > > > disks
> >>> > > > > run at 1/100th of their proper speed.
> >>> > > > >
> >>> > > > > St.Ack
> >>> > > > >
> >>> > > > >
> >>> > > > >
> >>> > > > >
> >>> > > > >
> >>> > > > >
> >>> > > > >
> >>> > > > > >
> >>> > > > > > Best,
> >>> > > > > > zhenyu
> >>> > > > > >
> >>> > > > > >
> >>> > > > > > On Wed, Nov 11, 2009 at 3:58 PM, Zhenyu Zhong <
> >>> > > zhongresearch@gmail.com
> >>> > > > > > >wrote:
> >>> > > > > >
> >>> > > > > > > Stack
> >>> > > > > > >
> >>> > > > > > > I am very appreciated for your comments.
> >>> > > > > > > I will use the zookeeper monitoring script on my cluster
> and
> >>> let
> >>> > it
> >>> > > > run
> >>> > > > > > > overnight to see the result.
> >>> > > > > > > I will follow up the thread when I get anything.
> >>> > > > > > >
> >>> > > > > > > thanks
> >>> > > > > > > zhenyu
> >>> > > > > > >
> >>> > > > > > >
> >>> > > > > > > On Wed, Nov 11, 2009 at 3:52 PM, stack <st...@duboce.net>
> >>> wrote:
> >>> > > > > > >
> >>> > > > > > >> I see these in your log too:
> >>> > > > > > >>
> >>> > > > > > >>
> >>> > > > > > >> 1. 2009-11-11 04:27:19,018 DEBUG
> >>> > > org.apache.zookeeper.ClientCnxn:
> >>> > > > > Got
> >>> > > > > > >> ping response for sessionid:0x424dfd908c50009 after
> 4544ms
> >>> > > > > > >> 2. 2009-11-11 04:27:19,018 DEBUG
> >>> > > org.apache.zookeeper.ClientCnxn:
> >>> > > > > Got
> >>> > > > > > >> ping response for sessionid:0x24dfd90c810002 after
> 4548ms
> >>> > > > > > >> 3. 2009-11-11 04:27:43,960 DEBUG
> >>> > > org.apache.zookeeper.ClientCnxn:
> >>> > > > > Got
> >>> > > > > > >> ping response for sessionid:0x424dfd908c50009 after
> 9030ms
> >>> > > > > > >> 4. 2009-11-11 04:27:43,960 DEBUG
> >>> > > org.apache.zookeeper.ClientCnxn:
> >>> > > > > Got
> >>> > > > > > >> ping response for sessionid:0x24dfd90c810002 after
> 9030ms
> >>> > > > > > >> 5. 2009-11-11 04:27:48,830 DEBUG
> >>> > > org.apache.zookeeper.ClientCnxn:
> >>> > > > > Got
> >>> > > > > > >> ping response for sessionid:0x424dfd908c50009 after 0ms
> >>> > > > > > >> 6. 2009-11-11 04:27:48,830 DEBUG
> >>> > > org.apache.zookeeper.ClientCnxn:
> >>> > > > > Got
> >>> > > > > > >> ping response for sessionid:0x24dfd90c810002 after 0ms
> >>> > > > > > >>
> >>> > > > > > >> Which would indicate lag talking to zk too.
> >>> > > > > > >>
> >>> > > > > > >> Appending HLog also seems to get steadily worse.
> >>> > > > > > >>
> >>> > > > > > >> I also see these:
> >>> > > > > > >>
> >>> > > > > > >>
> >>> > > > > > >> 1. 2009-11-11 04:27:01,233 WARN
> >>> > org.apache.zookeeper.ClientCnxn:
> >>> > > > > > >> Exception closing session 0x24dfd90c810002 to
> >>> > > > > > >> sun.nio.ch.SelectionKeyImpl@5e16a7e3
> >>> > > > > > >> 2. java.io.IOException: Read error rc = -1
> >>> > > > > > >> java.nio.DirectByteBuffer[pos=0 lim=4 cap=4]
> >>> > > > > > >> 3. at
> >>> > > > > > >>
> >>> > > > >
> >>> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
> >>> > > > > > >> 4. at
> >>> > > > > > >>
> >>> > > >
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
> >>> > > > > > >>
> >>> > > > > > >>
> >>> > > > > > >> Were all 5 quorum members participating? Check by doing
> >>> "echo
> >>> > > > stat|nc
> >>> > > > > > >> QUORUM_HOST port" for each member of quorum.... the below
> >>> might
> >>> > > > help:
> >>> > > > > > >>
> >>> > > > > > >> members="1,2,3,4,5"
> >>> > > > > > >> while true; do date; for i in $members; do echo $i; echo
> stat
> >>> |
> >>> > nc
> >>> > > > $i
> >>> > > > > > >> 2181;
> >>> > > > > > >> done; sleep 10; done
> >>> > > > > > >>
> >>> > > > > > >> We need to improve our zk_dump. It just reports quorum
> >>> > members...
> >>> > > > not
> >>> > > > > > if
> >>> > > > > > >> they are actually participating or not. The above will
> tell
> >>> you
> >>> > > if
> >>> > > > > dead
> >>> > > > > > >> or
> >>> > > > > > >> alive.
> >>> > > > > > >>
> >>> > > > > > >> Please report back after checking above and following the
> >>> > > > Jean-Daniel
> >>> > > > > > >> prescription.
> >>> > > > > > >>
> >>> > > > > > >> Thanks,
> >>> > > > > > >> St.Ack
> >>> > > > > > >>
> >>> > > > > > >>
> >>> > > > > > >>
> >>> > > > > > >> On Wed, Nov 11, 2009 at 12:29 PM, Zhenyu Zhong <
> >>> > > > > zhongresearch@gmail.com
> >>> > > > > > >> >wrote:
> >>> > > > > > >>
> >>> > > > > > >> > Thanks a lot for the help.
> >>> > > > > > >> >
> >>> > > > > > >> > I understand that the change of disk configurations
> should
> >>> be
> >>> > > done
> >>> > > > > > >> > progressively such that HDFS can balance and recover
> from a
> >>> > disk
> >>> > > > > going
> >>> > > > > > >> > away.
> >>> > > > > > >> >
> >>> > > > > > >> > The data we are trying to import into HBase is about
> >>> > 200million
> >>> > > > > rows,
> >>> > > > > > >> each
> >>> > > > > > >> > row is about hundreds of bytes. Basically, we read in a
> >>> text
> >>> > > file
> >>> > > > > from
> >>> > > > > > >> > HDFS,
> >>> > > > > > >> > then for each row in the text file, we fetch data from
> >>> HBase
> >>> > > > table,
> >>> > > > > > we
> >>> > > > > > >> > compare the column value and update the latest value
> back
> >>> into
> >>> > > > > HBase.
> >>> > > > > > I
> >>> > > > > > >> was
> >>> > > > > > >> > suggested to use CheckAndPut. Anyway this is the common
> >>> > usecase
> >>> > > > on
> >>> > > > > my
> >>> > > > > > >> > side.
> >>> > > > > > >> >
> >>> > > > > > >> >
> >>> > > > > > >> > Many thanks
> >>> > > > > > >> > zhenyu
> >>> > > > > > >> >
> >>> > > > > > >> >
> >>> > > > > > >> >
> >>> > > > > > >> > On Wed, Nov 11, 2009 at 3:03 PM, Jean-Daniel Cryans <
> >>> > > > > > >> jdcryans@apache.org
> >>> > > > > > >> > >wrote:
> >>> > > > > > >> >
> >>> > > > > > >> > > 6 disks, 19 machines and you see those kind of issues?
> >>> I'm
> >>> > > quite
> >>> > > > > > >> > > surprised and I want to learn more about what you
> trying
> >>> to
> >>> > > to.
> >>> > > > > > >> > >
> >>> > > > > > >> > > But first, if hbase is managing ZK for you then you
> can
> >>> > change
> >>> > > > the
> >>> > > > > > >> > > value of hbase.zookeeper.property.dataDir so that it
> >>> points
> >>> > to
> >>> > > > its
> >>> > > > > > own
> >>> > > > > > >> > > mount. Make sure you also change dfs.data.dir in
> >>> > > > > conf/hdfs-site.xml
> >>> > > > > > >> > > (hadoop stuff) so that it stops pointing there (you
> >>> should
> >>> > > have
> >>> > > > 6
> >>> > > > > > >> > > values there at the moment if you configured it
> >>> correctly).
> >>> > > > Doing
> >>> > > > > > this
> >>> > > > > > >> > > could result in data loss if you do it all at once so
> do
> >>> the
> >>> > > > > change
> >>> > > > > > >> > > progressively. This also means that you may have to
> keep
> >>> a
> >>> > > > > seperate
> >>> > > > > > >> > > HDFS configuration for those particular nodes. Please
> ask
> >>> > more
> >>> > > > > > >> > > questions if it's not very clear.
> >>> > > > > > >> > >
> >>> > > > > > >> > > WRT to dedicating nodes to Zookeeeper, it makes sense
> >>> only
> >>> > if
> >>> > > > you
> >>> > > > > > have
> >>> > > > > > >> > > a lot of nodes but in your case, unless there's also
> CPU
> >>> > > > > contention,
> >>> > > > > > >> > > you're good with only a dedicated disk.
> >>> > > > > > >> > >
> >>> > > > > > >> > > Now for what you are trying to do, is this a massive
> >>> import
> >>> > > that
> >>> > > > > > goes
> >>> > > > > > >> > > over many hours? How big is each row? I'd love to get
> a
> >>> lot
> >>> > of
> >>> > > > > > details
> >>> > > > > > >> > > to help you optimize your cluster for your particular
> >>> > problem.
> >>> > > > > > >> > >
> >>> > > > > > >> > > Thx,
> >>> > > > > > >> > >
> >>> > > > > > >> > > J-D
> >>> > > > > > >> > >
> >>> > > > > > >> > > On Wed, Nov 11, 2009 at 11:51 AM, Zhenyu Zhong <
> >>> > > > > > >> zhongresearch@gmail.com>
> >>> > > > > > >> > > wrote:
> >>> > > > > > >> > > > J-D,
> >>> > > > > > >> > > >
> >>> > > > > > >> > > > Thanks a lot for the insightful comments.
> >>> > > > > > >> > > > We are running 5 zookeeper quorums now. On these 5
> >>> quorum
> >>> > > > > servers,
> >>> > > > > > >> we
> >>> > > > > > >> > > also
> >>> > > > > > >> > > > run datanode, tasktracker and regionserver.
> >>> > > > > > >> > > > Now we have 6 disks per machine.
> >>> > > > > > >> > > > If it is the disk IO saturation, is there anyway to
> >>> > > configure
> >>> > > > > > >> zookeeper
> >>> > > > > > >> > > to
> >>> > > > > > >> > > > write to a seperate disk which is not the same one
> used
> >>> by
> >>> > > > HDFS?
> >>> > > > > > >> > > > Or shall we run zookeeper quorums on different nodes
> >>> other
> >>> > > > than
> >>> > > > > > >> HDFS?
> >>> > > > > > >> > > >
> >>> > > > > > >> > > > Are HLogs generated by Regionservers? Would it
> >>> alleviate
> >>> > the
> >>> > > > > issue
> >>> > > > > > >> if
> >>> > > > > > >> > we
> >>> > > > > > >> > > use
> >>> > > > > > >> > > > the first 14 for regionserver, and the rest of 5 for
> >>> > > > > zookeeper?(In
> >>> > > > > > >> that
> >>> > > > > > >> > > > sense, the node running zookeeper quorum won't run
> >>> > > > regionserver)
> >>> > > > > > >> > > >
> >>> > > > > > >> > > > I am very appreciated for your comments.
> >>> > > > > > >> > > >
> >>> > > > > > >> > > > Best,
> >>> > > > > > >> > > > zhenyu
> >>> > > > > > >> > > >
> >>> > > > > > >> > > >
> >>> > > > > > >> > > >
> >>> > > > > > >> > > > On Wed, Nov 11, 2009 at 2:30 PM, Jean-Daniel Cryans
> <
> >>> > > > > > >> > jdcryans@apache.org
> >>> > > > > > >> > > >wrote:
> >>> > > > > > >> > > >
> >>> > > > > > >> > > >> The problem seems to be that the IO is saturated.
> When
> >>> > you
> >>> > > > see
> >>> > > > > > >> this:
> >>> > > > > > >> > > >>
> >>> > > > > > >> > > >> #
> >>> > > > > > >> > > >> 2009-11-11 04:28:10,308 WARN
> >>> > > > > > >> > > >> org.apache.hadoop.hbase.regionserver.HLog: IPC
> Server
> >>> > > handler
> >>> > > > 9
> >>> > > > > > on
> >>> > > > > > >> > > >> 60021 took 6819ms appending an edit to hlog;
> >>> > editcount=370
> >>> > > > > > >> > > >> #
> >>> > > > > > >> > > >> 2009-11-11 04:28:10,308 WARN
> >>> > > > > > >> > > >> org.apache.hadoop.hbase.regionserver.HLog: IPC
> Server
> >>> > > handler
> >>> > > > 2
> >>> > > > > > on
> >>> > > > > > >> > > >> 60021 took 4412ms appending an edit to hlog;
> >>> > editcount=371
> >>> > > > > > >> > > >>
> >>> > > > > > >> > > >> It's a sign that HDFS is slow. I guess that the
> >>> zookeeper
> >>> > > > > quorum
> >>> > > > > > >> > > >> members are on those same nodes using the same
> disks?
> >>> If
> >>> > > so,
> >>> > > > > > >> there's
> >>> > > > > > >> > > >> the problem because Zookeeper needs to be able to
> >>> commit
> >>> > > > > > >> transactions
> >>> > > > > > >> > > >> very fast and if your disks are already hogged then
> >>> they
> >>> > > have
> >>> > > > > to
> >>> > > > > > >> wait
> >>> > > > > > >> > > >> and you can get disconnections. Do you still have
> 19
> >>> > quorum
> >>> > > > > > >> members?
> >>> > > > > > >> > > >> How many disks do you have per machine?
> >>> > > > > > >> > > >>
> >>> > > > > > >> > > >> Thx
> >>> > > > > > >> > > >>
> >>> > > > > > >> > > >> J-D
> >>> > > > > > >> > > >>
> >>> > > > > > >> > > >> On Wed, Nov 11, 2009 at 11:22 AM, Zhenyu Zhong <
> >>> > > > > > >> > zhongresearch@gmail.com
> >>> > > > > > >> > > >
> >>> > > > > > >> > > >> wrote:
> >>> > > > > > >> > > >> > Please take a look at the following regionserver
> >>> log.
> >>> > > > > > >> > > >> > The error started to come out from 4:24am UTC to
> >>> 4:29am
> >>> > > > UTC.
> >>> > > > > > >> > > >> >
> >>> > > > > > >> > > >> > http://pastebin.com/m3717d064
> >>> > > > > > >> > > >> >
> >>> > > > > > >> > > >> > Thanks in advance.
> >>> > > > > > >> > > >> >
> >>> > > > > > >> > > >> > Best,
> >>> > > > > > >> > > >> > zhenyu
> >>> > > > > > >> > > >> >
> >>> > > > > > >> > > >> >
> >>> > > > > > >> > > >> > On Wed, Nov 11, 2009 at 2:01 PM, Jean-Daniel
> Cryans
> >>> <
> >>> > > > > > >> > > jdcryans@apache.org
> >>> > > > > > >> > > >> >wrote:
> >>> > > > > > >> > > >> >
> >>> > > > > > >> > > >> >> Could you paste a bigger log trace (interested
> to
> >>> see
> >>> > at
> >>> > > > > least
> >>> > > > > > >> all
> >>> > > > > > >> > > 300
> >>> > > > > > >> > > >> >> lines before that) in a pastebin?
> >>> > > > > > >> > > >> >>
> >>> > > > > > >> > > >> >> Thx,
> >>> > > > > > >> > > >> >>
> >>> > > > > > >> > > >> >> J-D
> >>> > > > > > >> > > >> >>
> >>> > > > > > >> > > >> >> On Wed, Nov 11, 2009 at 10:56 AM, Zhenyu Zhong <
> >>> > > > > > >> > > zhongresearch@gmail.com
> >>> > > > > > >> > > >> >
> >>> > > > > > >> > > >> >> wrote:
> >>> > > > > > >> > > >> >> > J-D,
> >>> > > > > > >> > > >> >> >
> >>> > > > > > >> > > >> >> > Thanks for the reply.
> >>> > > > > > >> > > >> >> >
> >>> > > > > > >> > > >> >> > The problem I have is one or more of the
> >>> > regionservers
> >>> > > > > > >> disconnect
> >>> > > > > > >> > > from
> >>> > > > > > >> > > >> >> the
> >>> > > > > > >> > > >> >> > cluster.
> >>> > > > > > >> > > >> >> >
> >>> > > > > > >> > > >> >> > I checked the resource usage including memory
> >>> Heap
> >>> > > usage
> >>> > > > > for
> >>> > > > > > >> > > >> >> regionserver,
> >>> > > > > > >> > > >> >> > and found nothing suspicious. The regionserver
> >>> heap
> >>> > > > usage
> >>> > > > > is
> >>> > > > > > >> way
> >>> > > > > > >> > > below
> >>> > > > > > >> > > >> >> the
> >>> > > > > > >> > > >> >> > maxHeap defined in jvm.
> >>> > > > > > >> > > >> >> >
> >>> > > > > > >> > > >> >> > So I checked zookeeper log, then I found the
> >>> > following
> >>> > > > > > >> > > >> >> > BufferUnderflowException, eventually it leads
> to
> >>> > > > > > >> SocketException:
> >>> > > > > > >> > > >> >> Transport
> >>> > > > > > >> > > >> >> > endpoint is not connected.
> >>> > > > > > >> > > >> >> >
> >>> > > > > > >> > > >> >> >
> >>> > > > > > >> > > >> >> > I also checked the regionserver log. It looks
> >>> like
> >>> > the
> >>> > > > > > >> > regionserver
> >>> > > > > > >> > > >> tried
> >>> > > > > > >> > > >> >> to
> >>> > > > > > >> > > >> >> > connect to other zookeeper quorum server. It
> >>> > connects
> >>> > > > > > >> > successfully,
> >>> > > > > > >> > > >> but
> >>> > > > > > >> > > >> >> then
> >>> > > > > > >> > > >> >> > it complained about SocketException: transport
> >>> > > endpoint
> >>> > > > is
> >>> > > > > > not
> >>> > > > > > >> > > >> connected.
> >>> > > > > > >> > > >> >> >
> >>> > > > > > >> > > >> >> > So both zookeeper log and regionserver log
> told
> >>> me
> >>> > > that
> >>> > > > > the
> >>> > > > > > >> > > transport
> >>> > > > > > >> > > >> >> > endpoint is not connected. I don't know why.
> >>> > > > > > >> > > >> >> >
> >>> > > > > > >> > > >> >> > I would be very appreciated if you could point
> me
> >>> to
> >>> > > the
> >>> > > > > > right
> >>> > > > > > >> > > >> direction.
> >>> > > > > > >> > > >> >> >
> >>> > > > > > >> > > >> >> > Best,
> >>> > > > > > >> > > >> >> >
> >>> > > > > > >> > > >> >> >
> >>> > > > > > >> > > >> >> > Regionserver log:
> >>> > > > > > >> > > >> >> > 2009-11-11 04:24:53,718 INFO
> >>> > > > > > org.apache.zookeeper.ClientCnxn:
> >>> > > > > > >> > > >> Attempting
> >>> > > > > > >> > > >> >> > connection to server superpyxis0018/
> >>> > > > 192.168.100.132:2181
> >>> > > > > > >> > > >> >> > 2009-11-11 04:24:53,718 INFO
> >>> > > > > > org.apache.zookeeper.ClientCnxn:
> >>> > > > > > >> > > Priming
> >>> > > > > > >> > > >> >> > connection to
> >>> > > java.nio.channels.SocketChannel[connected
> >>> > > > > > >> local=/
> >>> > > > > > >> > > >> >> > 192.168.100.131:43065 remote=superpyxis0018/
> >>> > > > > > >> 192.168.100.132:2181
> >>> > > > > > >> > ]
> >>> > > > > > >> > > >> >> > 2009-11-11 04:24:53,718 INFO
> >>> > > > > > org.apache.zookeeper.ClientCnxn:
> >>> > > > > > >> > > Server
> >>> > > > > > >> > > >> >> > connection successful
> >>> > > > > > >> > > >> >> > 2009-11-11 04:24:53,768 INFO
> >>> > > > > > >> > > >> >> >
> >>> org.apache.hadoop.hbase.regionserver.HRegionServer:
> >>> > > Got
> >>> > > > > > >> ZooKeeper
> >>> > > > > > >> > > >> event,
> >>> > > > > > >> > > >> >> > state: Disconnected, type: None, path: null
> >>> > > > > > >> > > >> >> > 2009-11-11 04:24:54,118 INFO
> >>> > > > > > org.apache.zookeeper.ClientCnxn:
> >>> > > > > > >> > > >> Attempting
> >>> > > > > > >> > > >> >> > connection to server superpyxis0017/
> >>> > > > 192.168.100.131:2181
> >>> > > > > > >> > > >> >> > 2009-11-11 04:24:54,118 INFO
> >>> > > > > > org.apache.zookeeper.ClientCnxn:
> >>> > > > > > >> > > Priming
> >>> > > > > > >> > > >> >> > connection to
> >>> > > java.nio.channels.SocketChannel[connected
> >>> > > > > > >> local=/
> >>> > > > > > >> > > >> >> > 192.168.100.131:46903 remote=superpyxis0017/
> >>> > > > > > >> 192.168.100.131:2181
> >>> > > > > > >> > ]
> >>> > > > > > >> > > >> >> > 2009-11-11 04:24:54,118 INFO
> >>> > > > > > org.apache.zookeeper.ClientCnxn:
> >>> > > > > > >> > > Server
> >>> > > > > > >> > > >> >> > connection successful
> >>> > > > > > >> > > >> >> > 2009-11-11 04:24:54,118 WARN
> >>> > > > > > org.apache.zookeeper.ClientCnxn:
> >>> > > > > > >> > > >> Exception
> >>> > > > > > >> > > >> >> > closing session 0x24dfd90c810002 to
> >>> > > > > > >> > > >> sun.nio.ch.SelectionKeyImpl@2dd78ccf
> >>> > > > > > >> > > >> >> > java.io.IOException: Read error rc = -1
> >>> > > > > > >> > > >> java.nio.DirectByteBuffer[pos=0
> >>> > > > > > >> > > >> >> > lim=4 cap=4]
> >>> > > > > > >> > > >> >> > at
> >>> > > > > > >> > > >> >> >
> >>> > > > > > >> > >
> >>> > > > >
> >>> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
> >>> > > > > > >> > > >> >> > at
> >>> > > > > > >> > > >> >> >
> >>> > > > > > >> >
> >>> > > >
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
> >>> > > > > > >> > > >> >> > 2009-11-11 04:24:54,118 WARN
> >>> > > > > > org.apache.zookeeper.ClientCnxn:
> >>> > > > > > >> > > Ignoring
> >>> > > > > > >> > > >> >> > exception during shutdown input
> >>> > > > > > >> > > >> >> > java.net.SocketException: Transport endpoint
> is
> >>> not
> >>> > > > > > connected
> >>> > > > > > >> > > >> >> > at
> >>> > sun.nio.ch.SocketChannelImpl.shutdown(Native
> >>> > > > > > Method)
> >>> > > > > > >> > > >> >> > at
> >>> > > > > > >> > > >> >> >
> >>> > > > > > >> > >
> >>> > > > > >
> >>> > >
> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
> >>> > > > > > >> > > >> >> > at
> >>> > > > > > >> > > >>
> >>> > > > sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> >>> > > > > > >> > > >> >> > at
> >>> > > > > > >> > > >> >> >
> >>> > > > > > >> > > >>
> >>> > > > > > >> >
> >>> > > > > >
> >>> > >
> >>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> >>> > > > > > >> > > >> >> > at
> >>> > > > > > >> > > >> >> >
> >>> > > > > > >> >
> >>> > > >
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> >>> > > > > > >> > > >> >> >
> >>> > > > > > >> > > >> >> >
> >>> > > > > > >> > > >> >> >
> >>> > > > > > >> > > >> >> >
> >>> > > > > > >> > > >> >> >
> >>> > > > > > >> > > >> >> > On Wed, Nov 11, 2009 at 1:40 PM, Jean-Daniel
> >>> Cryans
> >>> > <
> >>> > > > > > >> > > >> jdcryans@apache.org
> >>> > > > > > >> > > >> >> >wrote:
> >>> > > > > > >> > > >> >> >
> >>> > > > > > >> > > >> >> >> I'm really not sure exactly what the
> situation
> >>> is
> >>> > > with
> >>> > > > > the
> >>> > > > > > >> > > >> >> >> java.nio.BufferUnderflowException, but what
> do
> >>> you
> >>> > > mean
> >>> > > > > by
> >>> > > > > > >> > > >> >> >> "regionserver disconnection"? Could you give
> >>> more
> >>> > > > > details?
> >>> > > > > > >> > > >> >> >>
> >>> > > > > > >> > > >> >> >> Thx,
> >>> > > > > > >> > > >> >> >>
> >>> > > > > > >> > > >> >> >> J-D
> >>> > > > > > >> > > >> >> >>
> >>> > > > > > >> > > >> >> >> On Wed, Nov 11, 2009 at 10:34 AM, Zhenyu
> Zhong <
> >>> > > > > > >> > > >> zhongresearch@gmail.com
> >>> > > > > > >> > > >> >> >
> >>> > > > > > >> > > >> >> >> wrote:
> >>> > > > > > >> > > >> >> >> > I searched around and found HBASE-1645
> "HBase
> >>> > could
> >>> > > > be
> >>> > > > > a
> >>> > > > > > >> > little
> >>> > > > > > >> > > >> more
> >>> > > > > > >> > > >> >> >> dainty
> >>> > > > > > >> > > >> >> >> > when zk is pulled out from under it"
> >>> > > > > > >> > > >> >> >> >
> >>> > > > > > >> > > >> >> >> > I am wondering whether my problem is
> related
> >>> to
> >>> > > this
> >>> > > > > > >> > HBase-1645.
> >>> > > > > > >> > > Or
> >>> > > > > > >> > > >> >> >> probably
> >>> > > > > > >> > > >> >> >> > has nothing to do with it?
> >>> > > > > > >> > > >> >> >> >
> >>> > > > > > >> > > >> >> >> > thanks
> >>> > > > > > >> > > >> >> >> >
> >>> > > > > > >> > > >> >> >> >
> >>> > > > > > >> > > >> >> >> > On Wed, Nov 11, 2009 at 12:02 PM, Zhenyu
> Zhong
> >>> <
> >>> > > > > > >> > > >> >> zhongresearch@gmail.com
> >>> > > > > > >> > > >> >> >> >wrote:
> >>> > > > > > >> > > >> >> >> >
> >>> > > > > > >> > > >> >> >> >> Hi,
> >>> > > > > > >> > > >> >> >> >>
> >>> > > > > > >> > > >> >> >> >> I have been monitoring the system
> resources
> >>> such
> >>> > > as
> >>> > > > > > memory
> >>> > > > > > >> > Heap
> >>> > > > > > >> > > >> usage
> >>> > > > > > >> > > >> >> >> for
> >>> > > > > > >> > > >> >> >> >> Regionserver, load average, network
> >>> bandwidth,
> >>> > > etc.
> >>> > > > > > >> > > >> >> >> >>
> >>> > > > > > >> > > >> >> >> >> I experienced a disconnection last night.
> >>> When I
> >>> > > > > checked
> >>> > > > > > >> the
> >>> > > > > > >> > > >> memory
> >>> > > > > > >> > > >> >> Heap
> >>> > > > > > >> > > >> >> >> >> for Regionserver, it was 2336 which is way
> >>> below
> >>> > > the
> >>> > > > > max
> >>> > > > > > >> Heap
> >>> > > > > > >> > > >> 4096. I
> >>> > > > > > >> > > >> >> >> don't
> >>> > > > > > >> > > >> >> >> >> think it is Regionserver problem. The
> >>> requests
> >>> > > that
> >>> > > > > the
> >>> > > > > > >> > > >> regionserver
> >>> > > > > > >> > > >> >> got
> >>> > > > > > >> > > >> >> >> was
> >>> > > > > > >> > > >> >> >> >> about 126 which doesn't seem to be high.
> The
> >>> > load
> >>> > > > > > average
> >>> > > > > > >> is
> >>> > > > > > >> > > kind
> >>> > > > > > >> > > >> of
> >>> > > > > > >> > > >> >> >> high
> >>> > > > > > >> > > >> >> >> >> >100. (but I saw disconnections while the
> >>> load
> >>> > > > average
> >>> > > > > > is
> >>> > > > > > >> > > around
> >>> > > > > > >> > > >> 30).
> >>> > > > > > >> > > >> >> >> The
> >>> > > > > > >> > > >> >> >> >> bandwidth is not a problem.
> >>> > > > > > >> > > >> >> >> >>
> >>> > > > > > >> > > >> >> >> >> However, I found error messages from the
> >>> > > zookeeper.
> >>> > > > > > Please
> >>> > > > > > >> > see
> >>> > > > > > >> > > the
> >>> > > > > > >> > > >> >> >> >> following error messages. There are some
> >>> > > > > > >> > > >> >> >> >> BufferUnderflowException error,
> IOException
> >>> > > > error,etc.
> >>> > > > > > >> > > >> >> >> >> May I ask what might be the causes that
> >>> generate
> >>> > > > these
> >>> > > > > > >> error
> >>> > > > > > >> > > >> >> messages?
> >>> > > > > > >> > > >> >> >> >>
> >>> > > > > > >> > > >> >> >> >> Many thanks
> >>> > > > > > >> > > >> >> >> >> zhenyu
> >>> > > > > > >> > > >> >> >> >>
> >>> > > > > > >> > > >> >> >> >>
> >>> > > > > > >> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
> >>> > > > > > >> > > org.apache.zookeeper.server.Request:
> >>> > > > > > >> > > >> >> >> Ignoring
> >>> > > > > > >> > > >> >> >> >> exception during toString
> >>> > > > > > >> > > >> >> >> >> java.nio.BufferUnderflowException
> >>> > > > > > >> > > >> >> >> >> at
> >>> > > > > > >> > java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
> >>> > > > > > >> > > >> >> >> >> at
> >>> > > > > java.nio.ByteBuffer.get(ByteBuffer.java:675)
> >>> > > > > > >> > > >> >> >> >> at
> >>> > > > > > >> > > >> >> >>
> >>> > > > > > >>
> >>> org.apache.zookeeper.server.Request.toString(Request.java:199)
> >>> > > > > > >> > > >> >> >> >> at
> >>> > > > java.lang.String.valueOf(String.java:2827)
> >>> > > > > > >> > > >> >> >> >> at
> >>> > > > > > >> > > java.lang.StringBuilder.append(StringBuilder.java:115)
> >>> > > > > > >> > > >> >> >> >> at
> >>> > > > > > >> > > >> >> >> >>
> >>> > > > > > >> > > >> >> >>
> >>> > > > > > >> > > >> >>
> >>> > > > > > >> > > >>
> >>> > > > > > >> > >
> >>> > > > > > >> >
> >>> > > > > > >>
> >>> > > > > >
> >>> > > > >
> >>> > > >
> >>> > >
> >>> >
> >>>
> org.apache.zookeeper.server.quorum.CommitProcessor.processRequest(CommitProcessor.java:167)
> >>> > > > > > >> > > >> >> >> >> at
> >>> > > > > > >> > > >> >> >> >>
> >>> > > > > > >> > > >> >> >>
> >>> > > > > > >> > > >> >>
> >>> > > > > > >> > > >>
> >>> > > > > > >> > >
> >>> > > > > > >> >
> >>> > > > > > >>
> >>> > > > > >
> >>> > > > >
> >>> > > >
> >>> > >
> >>> >
> >>>
> org.apache.zookeeper.server.quorum.FollowerRequestProcessor.run(FollowerRequestProcessor.java:68)
> >>> > > > > > >> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
> >>> > > > > > >> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
> >>> > > > > > >> > > >> >> >> >> Exception causing close of session
> >>> > > 0x424dfd908c50009
> >>> > > > > > >> > > >> >> >> >> due to java.io.IOException: Read error
> >>> > > > > > >> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
> >>> > > > > > >> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
> >>> > > > > > >> > > >> >> >> >> IOException stack trace
> >>> > > > > > >> > > >> >> >> >> java.io.IOException: Read error
> >>> > > > > > >> > > >> >> >> >> at
> >>> > > > > > >> > > >> >> >> >>
> >>> > > > > > >> > > >> >>
> >>> > > > > > >> > >
> >>> > > > > >
> >>> > >
> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:372)
> >>> > > > > > >> > > >> >> >> >> at
> >>> > > > > > >> > > >> >> >> >>
> >>> > > > > > >> > > >> >> >>
> >>> > > > > > >> > > >> >>
> >>> > > > > > >> > > >>
> >>> > > > > > >> > >
> >>> > > > > > >> >
> >>> > > > > > >>
> >>> > > > > >
> >>> > > > >
> >>> > > >
> >>> > >
> >>> >
> >>>
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:221)
> >>> > > > > > >> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
> >>> > > > > > >> > > >> >> >> >>
> >>> > > org.apache.zookeeper.server.quorum.CommitProcessor:
> >>> > > > > > >> > Processing
> >>> > > > > > >> > > >> >> request::
> >>> > > > > > >> > > >> >> >> >> sessionid:0x424dfd908c50009
> type:setWatches
> >>> > > > > > >> > > >> cxid:0xfffffffffffffff8
> >>> > > > > > >> > > >> >> >> >> zxid:0xfffffffffffffffe txntype:unknown
> n/a
> >>> > > > > > >> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
> >>> > > > > > >> > > org.apache.zookeeper.server.Request:
> >>> > > > > > >> > > >> >> >> Ignoring
> >>> > > > > > >> > > >> >> >> >> exception during toString
> >>> > > > > > >> > > >> >> >> >> java.nio.BufferUnderflowException
> >>> > > > > > >> > > >> >> >> >> at
> >>> > > > > > >> > java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
> >>> > > > > > >> > > >> >> >> >> at
> >>> > > > > java.nio.ByteBuffer.get(ByteBuffer.java:675)
> >>> > > > > > >> > > >> >> >> >> at
> >>> > > > > > >> > > >> >> >>
> >>> > > > > > >>
> >>> org.apache.zookeeper.server.Request.toString(Request.java:199)
> >>> > > > > > >> > > >> >> >> >> at
> >>> > > > java.lang.String.valueOf(String.java:2827)
> >>> > > > > > >> > > >> >> >> >> at
> >>> > > > > > >> > > java.lang.StringBuilder.append(StringBuilder.java:115)
> >>> > > > > > >> > > >> >> >> >> at
> >>> > > > > > >> > > >> >> >> >>
> >>> > > > > > >> > > >> >> >>
> >>> > > > > > >> > > >> >>
> >>> > > > > > >> > > >>
> >>> > > > > > >> > >
> >>> > > > > > >> >
> >>> > > > > > >>
> >>> > > > > >
> >>> > > > >
> >>> > > >
> >>> > >
> >>> >
> >>>
> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:74)
> >>> > > > > > >> > > >> >> >> >> at
> >>> > > > > > >> > > >> >> >> >>
> >>> > > > > > >> > > >> >> >>
> >>> > > > > > >> > > >> >>
> >>> > > > > > >> > > >>
> >>> > > > > > >> > >
> >>> > > > > > >> >
> >>> > > > > > >>
> >>> > > > > >
> >>> > > > >
> >>> > > >
> >>> > >
> >>> >
> >>>
> org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)
> >>> > > > > > >> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
> >>> > > > > > >> > > >> >> >> >>
> >>> > org.apache.zookeeper.server.FinalRequestProcessor:
> >>> > > > > > >> Processing
> >>> > > > > > >> > > >> >> request::
> >>> > > > > > >> > > >> >> >> >> sessionid:0x424dfd908c50009
> type:setWatches
> >>> > > > > > >> > > >> cxid:0xfffffffffffffff8
> >>> > > > > > >> > > >> >> >> >> zxid:0xfffffffffffffffe txntype:unknown
> n/a
> >>> > > > > > >> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
> >>> > > > > > >> > > org.apache.zookeeper.server.Request:
> >>> > > > > > >> > > >> >> >> Ignoring
> >>> > > > > > >> > > >> >> >> >> exception during toString
> >>> > > > > > >> > > >> >> >> >> java.nio.BufferUnderflowException
> >>> > > > > > >> > > >> >> >> >> at
> >>> > > > > > >> > java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
> >>> > > > > > >> > > >> >> >> >> at
> >>> > > > > java.nio.ByteBuffer.get(ByteBuffer.java:675)
> >>> > > > > > >> > > >> >> >> >> at
> >>> > > > > > >> > > >> >> >>
> >>> > > > > > >>
> >>> org.apache.zookeeper.server.Request.toString(Request.java:199)
> >>> > > > > > >> > > >> >> >> >> at
> >>> > > > > > >> > > >> >> >> >>
> >>> > > > > > >> > > >>
> >>> > > > > > >>
> >>> > > >
> >>> org.apache.log4j.or.DefaultRenderer.doRender(DefaultRenderer.java:36)
> >>> > > > > > >> > > >> >> >> >> at
> >>> > > > > > >> > > >> >> >> >>
> >>> > > > > > >> > >
> >>> > > >
> org.apache.log4j.or.RendererMap.findAndRender(RendererMap.java:80)
> >>> > > > > > >> > > >> >> >> >> at
> >>> > > > > > >> > > >> >> >> >>
> >>> > > > > > >> > > >> >> >>
> >>> > > > > > >> > > >> >>
> >>> > > > > > >> > > >>
> >>> > > > > > >> > >
> >>> > > > > > >> >
> >>> > > > > > >>
> >>> > > > > >
> >>> > > > >
> >>> > > >
> >>> > >
> >>> >
> >>>
> org.apache.log4j.spi.LoggingEvent.getRenderedMessage(LoggingEvent.java:362)
> >>> > > > > > >> > > >> >> >> >> at
> >>> > > > > > >> > > >> >> >> >>
> >>> > > > > > >> > > >> >> >>
> >>> > > > > > >> > > >> >>
> >>> > > > > > >> > > >>
> >>> > > > > > >> > >
> >>> > > > > > >> >
> >>> > > > > > >>
> >>> > > > > >
> >>> > > > >
> >>> > > >
> >>> > >
> >>> >
> >>>
> org.apache.log4j.helpers.PatternParser$BasicPatternConverter.convert(PatternParser.java:403)
> >>> > > > > > >> > > >> >> >> >> at
> >>> > > > > > >> > > >> >> >> >>
> >>> > > > > > >> > > >> >> >>
> >>> > > > > > >> > > >> >>
> >>> > > > > > >> > > >>
> >>> > > > > > >> > >
> >>> > > > > > >> >
> >>> > > > > > >>
> >>> > > > > >
> >>> > > > >
> >>> > > >
> >>> > >
> >>> >
> >>>
> org.apache.log4j.helpers.PatternConverter.format(PatternConverter.java:65)
> >>> > > > > > >> > > >> >> >> >> at
> >>> > > > > > >> > > >> >>
> >>> > > > > org.apache.log4j.PatternLayout.format(PatternLayout.java:502)
> >>> > > > > > >> > > >> >> >> >> at
> >>> > > > > > >> > > >> >> >> >>
> >>> > > > > > >> > >
> >>> > > >
> org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:302)
> >>> > > > > > >> > > >> >> >> >> at
> >>> > > > > > >> > > >> >> >> >>
> >>> > > > > > >> > > >> >> >>
> >>> > > > > > >> > > >> >>
> >>> > > > > > >> > > >>
> >>> > > > > > >> > >
> >>> > > > > > >> >
> >>> > > > > > >>
> >>> > > > > >
> >>> > > > >
> >>> > > >
> >>> > >
> >>> >
> >>>
> org.apache.log4j.DailyRollingFileAppender.subAppend(DailyRollingFileAppender.java:359)
> >>> > > > > > >> > > >> >> >> >> at
> >>> > > > > > >> > > >> >> >>
> >>> > > > > > >>
> >>> org.apache.log4j.WriterAppender.append(WriterAppender.java:160)
> >>> > > > > > >> > > >> >> >> >> at
> >>> > > > > > >> > > >> >> >> >>
> >>> > > > > > >> > > >>
> >>> > > > > > >>
> >>> > > >
> >>> org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
> >>> > > > > > >> > > >> >> >> >> at
> >>> > > > > > >> > > >> >> >> >>
> >>> > > > > > >> > > >> >> >>
> >>> > > > > > >> > > >> >>
> >>> > > > > > >> > > >>
> >>> > > > > > >> > >
> >>> > > > > > >> >
> >>> > > > > > >>
> >>> > > > > >
> >>> > > > >
> >>> > > >
> >>> > >
> >>> >
> >>>
> org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
> >>> > > > > > >> > > >> >> >> >> at
> >>> > > > > > >> > > >>
> >>> > org.apache.log4j.Category.callAppenders(Category.java:206)
> >>> > > > > > >> > > >> >> >> >> at
> >>> > > > > > >> > > org.apache.log4j.Category.forcedLog(Category.java:391)
> >>> > > > > > >> > > >> >> >> >> at
> >>> > > > > > >> org.apache.log4j.Category.debug(Category.java:260)
> >>> > > > > > >> > > >> >> >> >> at
> >>> > > > > > >> > > >> >> >> >>
> >>> > > > > > >> > > >> >> >>
> >>> > > > > > >> > > >> >>
> >>> > > > > > >> > > >>
> >>> > > > > > >> > >
> >>> > > > > > >> >
> >>> > > > > > >>
> >>> > > > > >
> >>> > > > >
> >>> > > >
> >>> > >
> >>> >
> >>>
> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:147)
> >>> > > > > > >> > > >> >> >> >> at
> >>> > > > > > >> > > >> >> >> >>
> >>> > > > > > >> > > >> >> >>
> >>> > > > > > >> > > >> >>
> >>> > > > > > >> > > >>
> >>> > > > > > >> > >
> >>> > > > > > >> >
> >>> > > > > > >>
> >>> > > > > >
> >>> > > > >
> >>> > > >
> >>> > >
> >>> >
> >>>
> org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)
> >>> > > > > > >> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
> >>> > > > > > >> > > org.apache.zookeeper.server.Request:
> >>> > > > > > >> > > >> >> >> Ignoring
> >>> > > > > > >> > > >> >> >> >> exception during toString
> >>> > > > > > >> > > >> >> >> >> sessionid:0x424dfd908c50009
> type:setWatches
> >>> > > > > > >> > > >> cxid:0xfffffffffffffff8
> >>> > > > > > >> > > >> >> >> >> zxid:0xfffffffffffffffe txntype:unknown
> n/a
> >>> > > > > > >> > > >> >> >> >> 2009-11-11 04:29:00,810 INFO
> >>> > > > > > >> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
> >>> > > > > > >> > > >> >> >> >> closing session:0x424dfd908c50009
> >>> NIOServerCnxn:
> >>> > > > > > >> > > >> >> >> >> java.nio.channels.SocketChannel[connected
> >>> > local=/
> >>> > > > > > >> > > >> >> 192.168.100.131:2181
> >>> > > > > > >> > > >> >> >> remote=/
> >>> > > > > > >> > > >> >> >> >> 192.168.100.131:35500]
> >>> > > > > > >> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
> >>> > > > > > >> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
> >>> > > > > > >> > > >> >> >> >> ignoring exception during input shutdown
> >>> > > > > > >> > > >> >> >> >> java.net.SocketException: Transport
> endpoint
> >>> is
> >>> > > not
> >>> > > > > > >> connected
> >>> > > > > > >> > > >> >> >> >> at
> >>> > > > > sun.nio.ch.SocketChannelImpl.shutdown(Native
> >>> > > > > > >> > Method)
> >>> > > > > > >> > > >> >> >> >> at
> >>> > > > > > >> > > >> >> >> >>
> >>> > > > > > >> > > >> >>
> >>> > > > > > >> > >
> >>> > > > > >
> >>> > >
> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
> >>> > > > > > >> > > >> >> >> >> at
> >>> > > > > > >> > > >> >> >>
> >>> > > > > > >>
> >>> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> >>> > > > > > >> > > >> >> >> >> at
> >>> > > > > > >> > > >> >> >> >>
> >>> > > > > > >> > > >> >>
> >>> > > > > > >> > >
> >>> > > > > > >>
> >>> > > > >
> >>> >
> org.apache.zookeeper.server.NIOServerCnxn.close(NIOServerCnxn.java:848)
> >>> > > > > > >> > > >> >> >> >> at
> >>> > > > > > >> > > >> >> >> >>
> >>> > > > > > >> > > >> >>
> >>> > > > > > >> > >
> >>> > > > > >
> >>> > >
> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:498)
> >>> > > > > > >> > > >> >> >> >> at
> >>> > > > > > >> > > >> >> >> >>
> >>> > > > > > >> > > >> >> >>
> >>> > > > > > >> > > >> >>
> >>> > > > > > >> > > >>
> >>> > > > > > >> > >
> >>> > > > > > >> >
> >>> > > > > > >>
> >>> > > > > >
> >>> > > > >
> >>> > > >
> >>> > >
> >>> >
> >>>
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:221)
> >>> > > > > > >> > > >> >> >> >>
> >>> > > > > > >> > > >> >> >> >
> >>> > > > > > >> > > >> >> >>
> >>> > > > > > >> > > >> >> >
> >>> > > > > > >> > > >> >>
> >>> > > > > > >> > > >> >
> >>> > > > > > >> > > >>
> >>> > > > > > >> > > >
> >>> > > > > > >> > >
> >>> > > > > > >> >
> >>> > > > > > >>
> >>> > > > > > >
> >>> > > > > > >
> >>> > > > > >
> >>> > > > >
> >>> > > >
> >>> > >
> >>> >
> >>>
> >>
> >>
> >
>
Re: regionserver disconnection
Posted by Jean-Daniel Cryans <jd...@apache.org>.
Looks like http://pero.blogs.aprilmayjune.org/2009/01/22/hadoop-and-linux-kernel-2627-epoll-limits/
J-D
On Fri, Nov 13, 2009 at 1:12 PM, Zhenyu Zhong <zh...@gmail.com> wrote:
> Hi,
>
>
> After I re-organize the cluster, the experiment ran into problem faster than
> before.
>
> Basically the changes are to use machines with less resources as zookeeper
> quorums and machines with more resources as regionserver.
>
> From the log, I still see the pause around 1 minute.
> I enabled the GC logging, please see
>
> http://pastebin.com/m1d4ce0f1
>
> for details.
> In general I don't see many pauses in the GC.
>
> What is more interesting, 1 hour after the 1st regionserver disconnected,
> the master log complained about too many open files. This didn't happen
> before.
> I checked the system OS setup as well as the limits.conf. I also checked the
> running processes. The total open files don't reach the limit. I am confused
> a bit.
>
>
> Please see the following master log.
>
> 2009-11-13 20:06:02,114 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.metaScanner scan of 4658 row(s) of meta region {server:
> 192.168.100.128:60021, regionname: .META.,,1, startKey: <>} complete
> 2009-11-13 20:06:02,114 INFO org.apache.hadoop.hbase.master.BaseScanner: All
> 1 .META. region(s) scanned
> 2009-11-13 20:06:07,677 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
> response for sessionid:0x424eebf1c10004c after 3ms
> 2009-11-13 20:06:08,178 INFO org.apache.hadoop.hdfs.DFSClient: Exception in
> createBlockOutputStream java.io.IOException: Bad connect ack with
> firstBadLink 192.168.100.123:50010
> 2009-11-13 20:06:08,178 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning
> block blk_-2808245019291145247_5478039
> 2009-11-13 20:06:09,682 INFO org.apache.hadoop.hdfs.DFSClient: Exception in
> createBlockOutputStream java.io.EOFException
> 2009-11-13 20:06:09,682 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning
> block blk_1074853606841896259_5478048
> 2009-11-13 20:06:10,334 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
> response for sessionid:0x24eebf1043003c after 1ms
> 2009-11-13 20:06:21,018 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
> response for sessionid:0x424eebf1c10004c after 0ms
> 2009-11-13 20:06:23,674 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
> response for sessionid:0x24eebf1043003c after 0ms
> 2009-11-13 20:06:24,828 INFO org.apache.hadoop.hdfs.DFSClient: Exception in
> createBlockOutputStream java.io.IOException: Bad connect ack with
> firstBadLink 192.168.100.123:50010
> 2009-11-13 20:06:24,828 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning
> block blk_-6642544517082142289_5478063
> 2009-11-13 20:06:24,828 INFO org.apache.hadoop.hdfs.DFSClient: Exception in
> createBlockOutputStream java.net.SocketException: Too many open files
> 2009-11-13 20:06:24,828 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning
> block blk_2057511041109796090_5478063
> 2009-11-13 20:06:24,928 INFO org.apache.hadoop.hdfs.DFSClient: Exception in
> createBlockOutputStream java.net.SocketException: Too many open files
> 2009-11-13 20:06:24,928 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning
> block blk_8219260302213892894_5478064
> 2009-11-13 20:06:30,855 INFO org.apache.hadoop.hdfs.DFSClient: Exception in
> createBlockOutputStream java.net.SocketException: Too many open files
> 2009-11-13 20:06:30,855 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning
> block blk_1669205542853067709_5478235
> 2009-11-13 20:06:30,905 INFO org.apache.hadoop.hdfs.DFSClient: Exception in
> createBlockOutputStream java.net.SocketException: Too many open files
> 2009-11-13 20:06:30,905 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning
> block blk_9128897691346270351_5478237
> 2009-11-13 20:06:30,955 INFO org.apache.hadoop.hdfs.DFSClient: Exception in
> createBlockOutputStream java.net.SocketException: Too many open files
> 2009-11-13 20:06:30,955 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning
> block blk_1116845144864123018_5478240
> 2009-11-13 20:06:34,372 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
> response for sessionid:0x424eebf1c10004c after 0ms
> 2009-11-13 20:06:37,034 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
> response for sessionid:0x24eebf1043003c after 0ms
> 2009-11-13 20:06:37,235 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer
> Exception: java.io.IOException: Too many open files
> at sun.nio.ch.IOUtil.initPipe(Native Method)
> at sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:49)
> at
> sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:18)
> at
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.get(SocketIOWithTimeout.java:407)
> at
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:322)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
> at
> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
> at
> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
> at java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
> at java.io.DataOutputStream.write(DataOutputStream.java:90)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2290)
>
> 2009-11-13 20:06:37,235 WARN org.apache.hadoop.hdfs.DFSClient: Error
> Recovery for block blk_8148813491785406356_5478475 bad datanode[0]
> 192.168.100.123:50010
> 2009-11-13 20:06:37,235 WARN org.apache.hadoop.hdfs.DFSClient: Error
> Recovery for block blk_8148813491785406356_5478475 in pipeline
> 192.168.100.123:50010, 192.168.100.134:50010, 192.168.100.122:50010: bad
> datanode 192.168.100.123:50010
> 2009-11-13 20:06:37,436 INFO org.apache.hadoop.hdfs.DFSClient: Exception in
> createBlockOutputStream java.net.SocketException: Too many open files
> 2009-11-13 20:06:37,436 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning
> block blk_2119727700857186236_5478498
>
>
>
>
>
> On Thu, Nov 12, 2009 at 4:21 PM, Zhenyu Zhong <zh...@gmail.com>wrote:
>
>> Will do.
>>
>> thanks
>> zhenyu
>>
>>
>> On Thu, Nov 12, 2009 at 3:33 PM, stack <st...@duboce.net> wrote:
>>
>>> Enable GC logging too on this next run (see hbase-env.sh). Lets try and
>>> get
>>> to the bottom of whats going on.
>>> Thanks,
>>> St.Ack
>>>
>>> On Thu, Nov 12, 2009 at 12:29 PM, Zhenyu Zhong <zhongresearch@gmail.com
>>> >wrote:
>>>
>>> > I can switch the boxes that run zookeeper with the ones that run
>>> > regionservers.
>>> > I will see the results later.
>>> >
>>> > FYI. The node does have the 10 minutes zookeeper.session.timeout value
>>> in
>>> > place.
>>> >
>>> > thanks
>>> > zhenyu
>>> >
>>> > On Thu, Nov 12, 2009 at 3:21 PM, stack <st...@duboce.net> wrote:
>>> >
>>> > > On Thu, Nov 12, 2009 at 11:50 AM, Zhenyu Zhong <
>>> zhongresearch@gmail.com
>>> > > >wrote:
>>> > >
>>> > > >
>>> > > > In my cluster, half of the cluster have 2 disks 400GB each per
>>> machine,
>>> > > and
>>> > > > half of the cluster have 6 disks per machine. Maybe we should run
>>> > > > zookeeper
>>> > > > on the machines with 2 disks and RS on machines with 6 disks?
>>> > > >
>>> > > > That would make most sense only in the below, it looks like the RS
>>> that
>>> > > had
>>> > > issue had 4 disks?
>>> > >
>>> > >
>>> > >
>>> > > > BTW, the 10 minutes zookeeper.session.timeout has been set during
>>> the
>>> > > > experiment.
>>> > > >
>>> > >
>>> > > And for sure this node had it in place?
>>> > > St.Ack
>>> > >
>>> > >
>>> > >
>>> > > >
>>> > > > thanks
>>> > > > zhenyu
>>> > > >
>>> > > > On Thu, Nov 12, 2009 at 2:08 PM, stack <st...@duboce.net> wrote:
>>> > > >
>>> > > > > On Thu, Nov 12, 2009 at 8:40 AM, Zhenyu Zhong <
>>> > zhongresearch@gmail.com
>>> > > > > >wrote:
>>> > > > >
>>> > > > >
>>> > > > > > Though I experienced 2 regionserver disconnection this morning,
>>> it
>>> > > > looks
>>> > > > > > better from the regionserver log. (please see the following log)
>>> > > > > >
>>> > > > > >
>>> > > > > > http://pastebin.com/m496dbfae
>>> > > > > >
>>> > > > > >
>>> > > > > > I checked diskIO for one of the regionserver(192.168.100.116)
>>> > during
>>> > > > the
>>> > > > > > time it disconnected.
>>> > > > > >
>>> > > > > > Time: 03:04:58 AM
>>> > > > > > Device: tps Blk_read/s Blk_wrtn/s Blk_read
>>> > > Blk_wrtn
>>> > > > > > sda 105.31 5458.83 19503.64 9043873239
>>> > > 32312473676
>>> > > > > > sda1 2.90 6.64 99.25 10993934
>>> > > 164433464
>>> > > > > > sda2 1.72 23.76 30.25 39365817
>>> > > 50124008
>>> > > > > > sda3 0.30 0.38 3.58 624930
>>> > > 5923000
>>> > > > > > sda4 100.39 5428.06 19370.56 8992888270
>>> > > 32091993204
>>> > > > > >
>>> > > > > >
>>> > > > > Is this high for you? 20k blocks/second would seem to be high but
>>> > its
>>> > > > one
>>> > > > > disk only and its not being shared by zk anymore so shouldn't
>>> matter?
>>> > > > >
>>> > > > >
>>> > > > > > I also checked the zookeeper quorum server that the regionserver
>>> > > tried
>>> > > > to
>>> > > > > > connect according to the log. However, I don't see
>>> 192.168.100.116
>>> > in
>>> > > > the
>>> > > > > > client list of the zookeeper quorum server that the regionserver
>>> > > tried
>>> > > > to
>>> > > > > > connect.
>>> > > > > > Would that be a problem?
>>> > > > > >
>>> > > > > >
>>> > > > > Is that because the ephemeral node for the regionserver had
>>> > evaporated?
>>> > > > > Lost
>>> > > > > its lease w/ zk by the time you went to look?
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > > > Thu Nov 12 15:04:52 UTC 2009
>>> > > > > > Zookeeper version: 3.2.1-808558, built on 08/27/2009 18:48 GMT
>>> > > > > > Clients:
>>> > > > > > /192.168.100.127:43045[1](queued=0,recved=26,sent=0)
>>> > > > > > /192.168.100.131:39091[1](queued=0,recved=964,sent=0)
>>> > > > > > /192.168.100.124:35961[1](queued=0,recved=3266,sent=0)
>>> > > > > > /192.168.100.123:47935[1](queued=0,recved=233,sent=0)
>>> > > > > > /192.168.100.125:46931[1](queued=0,recved=2,sent=0)
>>> > > > > > /192.168.100.118:54924[1](queued=0,recved=295,sent=0)
>>> > > > > > /192.168.100.118:41390[1](queued=0,recved=2290,sent=0)
>>> > > > > > /192.168.100.136:42243[1](queued=0,recved=0,sent=0)
>>> > > > > >
>>> > > > > > Latency min/avg/max: 0/17/6333
>>> > > > > > Received: 47111
>>> > > > > > Sent: 0
>>> > > > > > Outstanding: 0
>>> > > > > > Zxid: 0x77000083f4
>>> > > > > > Mode: leader
>>> > > > > > Node count: 23
>>> > > > > >
>>> > > > >
>>> > > > >
>>> > > > > That 6 second maximum latency is pretty bad but should be well
>>> within
>>> > > the
>>> > > > > zk
>>> > > > > session timeout.
>>> > > > >
>>> > > > > So, problem is likely on the zk client side of the session; i.e.
>>> in
>>> > the
>>> > > > RS.
>>> > > > > You could enable GC logging as J-D suggested to see if you have
>>> any
>>> > big
>>> > > > > pauses, pauses > zk session timeout.
>>> > > > >
>>> > > > > When the RS went down, it didn't look too heavily loaded:
>>> > > > >
>>> > > > >
>>> > > > > 1. 2009-11-12 15:04:52,830 INFO
>>> > > > > org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of
>>> > metrics:
>>> > > > > request=1.5166667, regions=322, stores=657, storefiles=631,
>>> > > > > storefileIndexSize=61, memstoreSize=1472, usedHeap=2819,
>>> > > maxHeap=4079,
>>> > > > > blockCacheSize=658110960, blockCacheFree=197395984,
>>> > > > blockCacheCount=9903,
>>> > > > > blockCacheHitRatio=99
>>> > > > >
>>> > > > > Its serving a few reads? The number of store files seems fine.
>>> Not
>>> > > too
>>> > > > > much memory used.
>>> > > > >
>>> > > > > Looking at the logs, I see the Lease Still Held exception. This
>>> > > happens
>>> > > > > when the RS does its regular report to the master but the master
>>> > thinks
>>> > > > the
>>> > > > > RS has since restarted. It'll think this probably because it
>>> noticed
>>> > > > that
>>> > > > > the RS's znode in zk had gone away and it considered the RS dead.
>>> > > > >
>>> > > > > Looking too at your logs I see this gap in the zk pinging:
>>> > > > >
>>> > > > >
>>> > > > > 1. 2009-11-12 15:03:39,325 DEBUG
>>> org.apache.zookeeper.ClientCnxn:
>>> > Got
>>> > > > > ping response for sessionid:0x224e55436ad0004 after 0ms
>>> > > > > 2. 2009-11-12 15:03:43,113 DEBUG
>>> org.apache.zookeeper.ClientCnxn:
>>> > Got
>>> > > > > ping response for sessionid:0x24e55436a0007d after 0ms
>>> > > > >
>>> > > > > Where in the lines above it, its reporting about every ten
>>> seconds,
>>> > > here
>>> > > > > there is a big gap.
>>> > > > >
>>> > > > > Do you have ganglia or something that will let you look more into
>>> > what
>>> > > > was
>>> > > > > happening on this machine around this time? Is the machine OK?
>>> It
>>> > > looks
>>> > > > > lightly loaded and you have your cluster nicely laid out.
>>> Something
>>> > > odd
>>> > > > is
>>> > > > > going on. What about things like the write speed to disk? In the
>>> > past
>>> > > > > strange issues have been explained by incorrectly set BIOS which
>>> made
>>> > > > disks
>>> > > > > run at 1/100th of their proper speed.
>>> > > > >
>>> > > > > St.Ack
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > > >
>>> > > > > > Best,
>>> > > > > > zhenyu
>>> > > > > >
>>> > > > > >
>>> > > > > > On Wed, Nov 11, 2009 at 3:58 PM, Zhenyu Zhong <
>>> > > zhongresearch@gmail.com
>>> > > > > > >wrote:
>>> > > > > >
>>> > > > > > > Stack
>>> > > > > > >
>>> > > > > > > I am very appreciated for your comments.
>>> > > > > > > I will use the zookeeper monitoring script on my cluster and
>>> let
>>> > it
>>> > > > run
>>> > > > > > > overnight to see the result.
>>> > > > > > > I will follow up the thread when I get anything.
>>> > > > > > >
>>> > > > > > > thanks
>>> > > > > > > zhenyu
>>> > > > > > >
>>> > > > > > >
>>> > > > > > > On Wed, Nov 11, 2009 at 3:52 PM, stack <st...@duboce.net>
>>> wrote:
>>> > > > > > >
>>> > > > > > >> I see these in your log too:
>>> > > > > > >>
>>> > > > > > >>
>>> > > > > > >> 1. 2009-11-11 04:27:19,018 DEBUG
>>> > > org.apache.zookeeper.ClientCnxn:
>>> > > > > Got
>>> > > > > > >> ping response for sessionid:0x424dfd908c50009 after 4544ms
>>> > > > > > >> 2. 2009-11-11 04:27:19,018 DEBUG
>>> > > org.apache.zookeeper.ClientCnxn:
>>> > > > > Got
>>> > > > > > >> ping response for sessionid:0x24dfd90c810002 after 4548ms
>>> > > > > > >> 3. 2009-11-11 04:27:43,960 DEBUG
>>> > > org.apache.zookeeper.ClientCnxn:
>>> > > > > Got
>>> > > > > > >> ping response for sessionid:0x424dfd908c50009 after 9030ms
>>> > > > > > >> 4. 2009-11-11 04:27:43,960 DEBUG
>>> > > org.apache.zookeeper.ClientCnxn:
>>> > > > > Got
>>> > > > > > >> ping response for sessionid:0x24dfd90c810002 after 9030ms
>>> > > > > > >> 5. 2009-11-11 04:27:48,830 DEBUG
>>> > > org.apache.zookeeper.ClientCnxn:
>>> > > > > Got
>>> > > > > > >> ping response for sessionid:0x424dfd908c50009 after 0ms
>>> > > > > > >> 6. 2009-11-11 04:27:48,830 DEBUG
>>> > > org.apache.zookeeper.ClientCnxn:
>>> > > > > Got
>>> > > > > > >> ping response for sessionid:0x24dfd90c810002 after 0ms
>>> > > > > > >>
>>> > > > > > >> Which would indicate lag talking to zk too.
>>> > > > > > >>
>>> > > > > > >> Appending HLog also seems to get steadily worse.
>>> > > > > > >>
>>> > > > > > >> I also see these:
>>> > > > > > >>
>>> > > > > > >>
>>> > > > > > >> 1. 2009-11-11 04:27:01,233 WARN
>>> > org.apache.zookeeper.ClientCnxn:
>>> > > > > > >> Exception closing session 0x24dfd90c810002 to
>>> > > > > > >> sun.nio.ch.SelectionKeyImpl@5e16a7e3
>>> > > > > > >> 2. java.io.IOException: Read error rc = -1
>>> > > > > > >> java.nio.DirectByteBuffer[pos=0 lim=4 cap=4]
>>> > > > > > >> 3. at
>>> > > > > > >>
>>> > > > >
>>> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
>>> > > > > > >> 4. at
>>> > > > > > >>
>>> > > > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
>>> > > > > > >>
>>> > > > > > >>
>>> > > > > > >> Were all 5 quorum members participating? Check by doing
>>> "echo
>>> > > > stat|nc
>>> > > > > > >> QUORUM_HOST port" for each member of quorum.... the below
>>> might
>>> > > > help:
>>> > > > > > >>
>>> > > > > > >> members="1,2,3,4,5"
>>> > > > > > >> while true; do date; for i in $members; do echo $i; echo stat
>>> |
>>> > nc
>>> > > > $i
>>> > > > > > >> 2181;
>>> > > > > > >> done; sleep 10; done
>>> > > > > > >>
>>> > > > > > >> We need to improve our zk_dump. It just reports quorum
>>> > members...
>>> > > > not
>>> > > > > > if
>>> > > > > > >> they are actually participating or not. The above will tell
>>> you
>>> > > if
>>> > > > > dead
>>> > > > > > >> or
>>> > > > > > >> alive.
>>> > > > > > >>
>>> > > > > > >> Please report back after checking above and following the
>>> > > > Jean-Daniel
>>> > > > > > >> prescription.
>>> > > > > > >>
>>> > > > > > >> Thanks,
>>> > > > > > >> St.Ack
>>> > > > > > >>
>>> > > > > > >>
>>> > > > > > >>
>>> > > > > > >> On Wed, Nov 11, 2009 at 12:29 PM, Zhenyu Zhong <
>>> > > > > zhongresearch@gmail.com
>>> > > > > > >> >wrote:
>>> > > > > > >>
>>> > > > > > >> > Thanks a lot for the help.
>>> > > > > > >> >
>>> > > > > > >> > I understand that the change of disk configurations should
>>> be
>>> > > done
>>> > > > > > >> > progressively such that HDFS can balance and recover from a
>>> > disk
>>> > > > > going
>>> > > > > > >> > away.
>>> > > > > > >> >
>>> > > > > > >> > The data we are trying to import into HBase is about
>>> > 200million
>>> > > > > rows,
>>> > > > > > >> each
>>> > > > > > >> > row is about hundreds of bytes. Basically, we read in a
>>> text
>>> > > file
>>> > > > > from
>>> > > > > > >> > HDFS,
>>> > > > > > >> > then for each row in the text file, we fetch data from
>>> HBase
>>> > > > table,
>>> > > > > > we
>>> > > > > > >> > compare the column value and update the latest value back
>>> into
>>> > > > > HBase.
>>> > > > > > I
>>> > > > > > >> was
>>> > > > > > >> > suggested to use CheckAndPut. Anyway this is the common
>>> > usecase
>>> > > > on
>>> > > > > my
>>> > > > > > >> > side.
>>> > > > > > >> >
>>> > > > > > >> >
>>> > > > > > >> > Many thanks
>>> > > > > > >> > zhenyu
>>> > > > > > >> >
>>> > > > > > >> >
>>> > > > > > >> >
>>> > > > > > >> > On Wed, Nov 11, 2009 at 3:03 PM, Jean-Daniel Cryans <
>>> > > > > > >> jdcryans@apache.org
>>> > > > > > >> > >wrote:
>>> > > > > > >> >
>>> > > > > > >> > > 6 disks, 19 machines and you see those kind of issues?
>>> I'm
>>> > > quite
>>> > > > > > >> > > surprised and I want to learn more about what you trying
>>> to
>>> > > to.
>>> > > > > > >> > >
>>> > > > > > >> > > But first, if hbase is managing ZK for you then you can
>>> > change
>>> > > > the
>>> > > > > > >> > > value of hbase.zookeeper.property.dataDir so that it
>>> points
>>> > to
>>> > > > its
>>> > > > > > own
>>> > > > > > >> > > mount. Make sure you also change dfs.data.dir in
>>> > > > > conf/hdfs-site.xml
>>> > > > > > >> > > (hadoop stuff) so that it stops pointing there (you
>>> should
>>> > > have
>>> > > > 6
>>> > > > > > >> > > values there at the moment if you configured it
>>> correctly).
>>> > > > Doing
>>> > > > > > this
>>> > > > > > >> > > could result in data loss if you do it all at once so do
>>> the
>>> > > > > change
>>> > > > > > >> > > progressively. This also means that you may have to keep
>>> a
>>> > > > > seperate
>>> > > > > > >> > > HDFS configuration for those particular nodes. Please ask
>>> > more
>>> > > > > > >> > > questions if it's not very clear.
>>> > > > > > >> > >
>>> > > > > > >> > > WRT to dedicating nodes to Zookeeeper, it makes sense
>>> only
>>> > if
>>> > > > you
>>> > > > > > have
>>> > > > > > >> > > a lot of nodes but in your case, unless there's also CPU
>>> > > > > contention,
>>> > > > > > >> > > you're good with only a dedicated disk.
>>> > > > > > >> > >
>>> > > > > > >> > > Now for what you are trying to do, is this a massive
>>> import
>>> > > that
>>> > > > > > goes
>>> > > > > > >> > > over many hours? How big is each row? I'd love to get a
>>> lot
>>> > of
>>> > > > > > details
>>> > > > > > >> > > to help you optimize your cluster for your particular
>>> > problem.
>>> > > > > > >> > >
>>> > > > > > >> > > Thx,
>>> > > > > > >> > >
>>> > > > > > >> > > J-D
>>> > > > > > >> > >
>>> > > > > > >> > > On Wed, Nov 11, 2009 at 11:51 AM, Zhenyu Zhong <
>>> > > > > > >> zhongresearch@gmail.com>
>>> > > > > > >> > > wrote:
>>> > > > > > >> > > > J-D,
>>> > > > > > >> > > >
>>> > > > > > >> > > > Thanks a lot for the insightful comments.
>>> > > > > > >> > > > We are running 5 zookeeper quorums now. On these 5
>>> quorum
>>> > > > > servers,
>>> > > > > > >> we
>>> > > > > > >> > > also
>>> > > > > > >> > > > run datanode, tasktracker and regionserver.
>>> > > > > > >> > > > Now we have 6 disks per machine.
>>> > > > > > >> > > > If it is the disk IO saturation, is there anyway to
>>> > > configure
>>> > > > > > >> zookeeper
>>> > > > > > >> > > to
>>> > > > > > >> > > > write to a seperate disk which is not the same one used
>>> by
>>> > > > HDFS?
>>> > > > > > >> > > > Or shall we run zookeeper quorums on different nodes
>>> other
>>> > > > than
>>> > > > > > >> HDFS?
>>> > > > > > >> > > >
>>> > > > > > >> > > > Are HLogs generated by Regionservers? Would it
>>> alleviate
>>> > the
>>> > > > > issue
>>> > > > > > >> if
>>> > > > > > >> > we
>>> > > > > > >> > > use
>>> > > > > > >> > > > the first 14 for regionserver, and the rest of 5 for
>>> > > > > zookeeper?(In
>>> > > > > > >> that
>>> > > > > > >> > > > sense, the node running zookeeper quorum won't run
>>> > > > regionserver)
>>> > > > > > >> > > >
>>> > > > > > >> > > > I am very appreciated for your comments.
>>> > > > > > >> > > >
>>> > > > > > >> > > > Best,
>>> > > > > > >> > > > zhenyu
>>> > > > > > >> > > >
>>> > > > > > >> > > >
>>> > > > > > >> > > >
>>> > > > > > >> > > > On Wed, Nov 11, 2009 at 2:30 PM, Jean-Daniel Cryans <
>>> > > > > > >> > jdcryans@apache.org
>>> > > > > > >> > > >wrote:
>>> > > > > > >> > > >
>>> > > > > > >> > > >> The problem seems to be that the IO is saturated. When
>>> > you
>>> > > > see
>>> > > > > > >> this:
>>> > > > > > >> > > >>
>>> > > > > > >> > > >> #
>>> > > > > > >> > > >> 2009-11-11 04:28:10,308 WARN
>>> > > > > > >> > > >> org.apache.hadoop.hbase.regionserver.HLog: IPC Server
>>> > > handler
>>> > > > 9
>>> > > > > > on
>>> > > > > > >> > > >> 60021 took 6819ms appending an edit to hlog;
>>> > editcount=370
>>> > > > > > >> > > >> #
>>> > > > > > >> > > >> 2009-11-11 04:28:10,308 WARN
>>> > > > > > >> > > >> org.apache.hadoop.hbase.regionserver.HLog: IPC Server
>>> > > handler
>>> > > > 2
>>> > > > > > on
>>> > > > > > >> > > >> 60021 took 4412ms appending an edit to hlog;
>>> > editcount=371
>>> > > > > > >> > > >>
>>> > > > > > >> > > >> It's a sign that HDFS is slow. I guess that the
>>> zookeeper
>>> > > > > quorum
>>> > > > > > >> > > >> members are on those same nodes using the same disks?
>>> If
>>> > > so,
>>> > > > > > >> there's
>>> > > > > > >> > > >> the problem because Zookeeper needs to be able to
>>> commit
>>> > > > > > >> transactions
>>> > > > > > >> > > >> very fast and if your disks are already hogged then
>>> they
>>> > > have
>>> > > > > to
>>> > > > > > >> wait
>>> > > > > > >> > > >> and you can get disconnections. Do you still have 19
>>> > quorum
>>> > > > > > >> members?
>>> > > > > > >> > > >> How many disks do you have per machine?
>>> > > > > > >> > > >>
>>> > > > > > >> > > >> Thx
>>> > > > > > >> > > >>
>>> > > > > > >> > > >> J-D
>>> > > > > > >> > > >>
>>> > > > > > >> > > >> On Wed, Nov 11, 2009 at 11:22 AM, Zhenyu Zhong <
>>> > > > > > >> > zhongresearch@gmail.com
>>> > > > > > >> > > >
>>> > > > > > >> > > >> wrote:
>>> > > > > > >> > > >> > Please take a look at the following regionserver
>>> log.
>>> > > > > > >> > > >> > The error started to come out from 4:24am UTC to
>>> 4:29am
>>> > > > UTC.
>>> > > > > > >> > > >> >
>>> > > > > > >> > > >> > http://pastebin.com/m3717d064
>>> > > > > > >> > > >> >
>>> > > > > > >> > > >> > Thanks in advance.
>>> > > > > > >> > > >> >
>>> > > > > > >> > > >> > Best,
>>> > > > > > >> > > >> > zhenyu
>>> > > > > > >> > > >> >
>>> > > > > > >> > > >> >
>>> > > > > > >> > > >> > On Wed, Nov 11, 2009 at 2:01 PM, Jean-Daniel Cryans
>>> <
>>> > > > > > >> > > jdcryans@apache.org
>>> > > > > > >> > > >> >wrote:
>>> > > > > > >> > > >> >
>>> > > > > > >> > > >> >> Could you paste a bigger log trace (interested to
>>> see
>>> > at
>>> > > > > least
>>> > > > > > >> all
>>> > > > > > >> > > 300
>>> > > > > > >> > > >> >> lines before that) in a pastebin?
>>> > > > > > >> > > >> >>
>>> > > > > > >> > > >> >> Thx,
>>> > > > > > >> > > >> >>
>>> > > > > > >> > > >> >> J-D
>>> > > > > > >> > > >> >>
>>> > > > > > >> > > >> >> On Wed, Nov 11, 2009 at 10:56 AM, Zhenyu Zhong <
>>> > > > > > >> > > zhongresearch@gmail.com
>>> > > > > > >> > > >> >
>>> > > > > > >> > > >> >> wrote:
>>> > > > > > >> > > >> >> > J-D,
>>> > > > > > >> > > >> >> >
>>> > > > > > >> > > >> >> > Thanks for the reply.
>>> > > > > > >> > > >> >> >
>>> > > > > > >> > > >> >> > The problem I have is one or more of the
>>> > regionservers
>>> > > > > > >> disconnect
>>> > > > > > >> > > from
>>> > > > > > >> > > >> >> the
>>> > > > > > >> > > >> >> > cluster.
>>> > > > > > >> > > >> >> >
>>> > > > > > >> > > >> >> > I checked the resource usage including memory
>>> Heap
>>> > > usage
>>> > > > > for
>>> > > > > > >> > > >> >> regionserver,
>>> > > > > > >> > > >> >> > and found nothing suspicious. The regionserver
>>> heap
>>> > > > usage
>>> > > > > is
>>> > > > > > >> way
>>> > > > > > >> > > below
>>> > > > > > >> > > >> >> the
>>> > > > > > >> > > >> >> > maxHeap defined in jvm.
>>> > > > > > >> > > >> >> >
>>> > > > > > >> > > >> >> > So I checked zookeeper log, then I found the
>>> > following
>>> > > > > > >> > > >> >> > BufferUnderflowException, eventually it leads to
>>> > > > > > >> SocketException:
>>> > > > > > >> > > >> >> Transport
>>> > > > > > >> > > >> >> > endpoint is not connected.
>>> > > > > > >> > > >> >> >
>>> > > > > > >> > > >> >> >
>>> > > > > > >> > > >> >> > I also checked the regionserver log. It looks
>>> like
>>> > the
>>> > > > > > >> > regionserver
>>> > > > > > >> > > >> tried
>>> > > > > > >> > > >> >> to
>>> > > > > > >> > > >> >> > connect to other zookeeper quorum server. It
>>> > connects
>>> > > > > > >> > successfully,
>>> > > > > > >> > > >> but
>>> > > > > > >> > > >> >> then
>>> > > > > > >> > > >> >> > it complained about SocketException: transport
>>> > > endpoint
>>> > > > is
>>> > > > > > not
>>> > > > > > >> > > >> connected.
>>> > > > > > >> > > >> >> >
>>> > > > > > >> > > >> >> > So both zookeeper log and regionserver log told
>>> me
>>> > > that
>>> > > > > the
>>> > > > > > >> > > transport
>>> > > > > > >> > > >> >> > endpoint is not connected. I don't know why.
>>> > > > > > >> > > >> >> >
>>> > > > > > >> > > >> >> > I would be very appreciated if you could point me
>>> to
>>> > > the
>>> > > > > > right
>>> > > > > > >> > > >> direction.
>>> > > > > > >> > > >> >> >
>>> > > > > > >> > > >> >> > Best,
>>> > > > > > >> > > >> >> >
>>> > > > > > >> > > >> >> >
>>> > > > > > >> > > >> >> > Regionserver log:
>>> > > > > > >> > > >> >> > 2009-11-11 04:24:53,718 INFO
>>> > > > > > org.apache.zookeeper.ClientCnxn:
>>> > > > > > >> > > >> Attempting
>>> > > > > > >> > > >> >> > connection to server superpyxis0018/
>>> > > > 192.168.100.132:2181
>>> > > > > > >> > > >> >> > 2009-11-11 04:24:53,718 INFO
>>> > > > > > org.apache.zookeeper.ClientCnxn:
>>> > > > > > >> > > Priming
>>> > > > > > >> > > >> >> > connection to
>>> > > java.nio.channels.SocketChannel[connected
>>> > > > > > >> local=/
>>> > > > > > >> > > >> >> > 192.168.100.131:43065 remote=superpyxis0018/
>>> > > > > > >> 192.168.100.132:2181
>>> > > > > > >> > ]
>>> > > > > > >> > > >> >> > 2009-11-11 04:24:53,718 INFO
>>> > > > > > org.apache.zookeeper.ClientCnxn:
>>> > > > > > >> > > Server
>>> > > > > > >> > > >> >> > connection successful
>>> > > > > > >> > > >> >> > 2009-11-11 04:24:53,768 INFO
>>> > > > > > >> > > >> >> >
>>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>>> > > Got
>>> > > > > > >> ZooKeeper
>>> > > > > > >> > > >> event,
>>> > > > > > >> > > >> >> > state: Disconnected, type: None, path: null
>>> > > > > > >> > > >> >> > 2009-11-11 04:24:54,118 INFO
>>> > > > > > org.apache.zookeeper.ClientCnxn:
>>> > > > > > >> > > >> Attempting
>>> > > > > > >> > > >> >> > connection to server superpyxis0017/
>>> > > > 192.168.100.131:2181
>>> > > > > > >> > > >> >> > 2009-11-11 04:24:54,118 INFO
>>> > > > > > org.apache.zookeeper.ClientCnxn:
>>> > > > > > >> > > Priming
>>> > > > > > >> > > >> >> > connection to
>>> > > java.nio.channels.SocketChannel[connected
>>> > > > > > >> local=/
>>> > > > > > >> > > >> >> > 192.168.100.131:46903 remote=superpyxis0017/
>>> > > > > > >> 192.168.100.131:2181
>>> > > > > > >> > ]
>>> > > > > > >> > > >> >> > 2009-11-11 04:24:54,118 INFO
>>> > > > > > org.apache.zookeeper.ClientCnxn:
>>> > > > > > >> > > Server
>>> > > > > > >> > > >> >> > connection successful
>>> > > > > > >> > > >> >> > 2009-11-11 04:24:54,118 WARN
>>> > > > > > org.apache.zookeeper.ClientCnxn:
>>> > > > > > >> > > >> Exception
>>> > > > > > >> > > >> >> > closing session 0x24dfd90c810002 to
>>> > > > > > >> > > >> sun.nio.ch.SelectionKeyImpl@2dd78ccf
>>> > > > > > >> > > >> >> > java.io.IOException: Read error rc = -1
>>> > > > > > >> > > >> java.nio.DirectByteBuffer[pos=0
>>> > > > > > >> > > >> >> > lim=4 cap=4]
>>> > > > > > >> > > >> >> > at
>>> > > > > > >> > > >> >> >
>>> > > > > > >> > >
>>> > > > >
>>> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
>>> > > > > > >> > > >> >> > at
>>> > > > > > >> > > >> >> >
>>> > > > > > >> >
>>> > > > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
>>> > > > > > >> > > >> >> > 2009-11-11 04:24:54,118 WARN
>>> > > > > > org.apache.zookeeper.ClientCnxn:
>>> > > > > > >> > > Ignoring
>>> > > > > > >> > > >> >> > exception during shutdown input
>>> > > > > > >> > > >> >> > java.net.SocketException: Transport endpoint is
>>> not
>>> > > > > > connected
>>> > > > > > >> > > >> >> > at
>>> > sun.nio.ch.SocketChannelImpl.shutdown(Native
>>> > > > > > Method)
>>> > > > > > >> > > >> >> > at
>>> > > > > > >> > > >> >> >
>>> > > > > > >> > >
>>> > > > > >
>>> > > sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
>>> > > > > > >> > > >> >> > at
>>> > > > > > >> > > >>
>>> > > > sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>>> > > > > > >> > > >> >> > at
>>> > > > > > >> > > >> >> >
>>> > > > > > >> > > >>
>>> > > > > > >> >
>>> > > > > >
>>> > >
>>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
>>> > > > > > >> > > >> >> > at
>>> > > > > > >> > > >> >> >
>>> > > > > > >> >
>>> > > > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
>>> > > > > > >> > > >> >> >
>>> > > > > > >> > > >> >> >
>>> > > > > > >> > > >> >> >
>>> > > > > > >> > > >> >> >
>>> > > > > > >> > > >> >> >
>>> > > > > > >> > > >> >> > On Wed, Nov 11, 2009 at 1:40 PM, Jean-Daniel
>>> Cryans
>>> > <
>>> > > > > > >> > > >> jdcryans@apache.org
>>> > > > > > >> > > >> >> >wrote:
>>> > > > > > >> > > >> >> >
>>> > > > > > >> > > >> >> >> I'm really not sure exactly what the situation
>>> is
>>> > > with
>>> > > > > the
>>> > > > > > >> > > >> >> >> java.nio.BufferUnderflowException, but what do
>>> you
>>> > > mean
>>> > > > > by
>>> > > > > > >> > > >> >> >> "regionserver disconnection"? Could you give
>>> more
>>> > > > > details?
>>> > > > > > >> > > >> >> >>
>>> > > > > > >> > > >> >> >> Thx,
>>> > > > > > >> > > >> >> >>
>>> > > > > > >> > > >> >> >> J-D
>>> > > > > > >> > > >> >> >>
>>> > > > > > >> > > >> >> >> On Wed, Nov 11, 2009 at 10:34 AM, Zhenyu Zhong <
>>> > > > > > >> > > >> zhongresearch@gmail.com
>>> > > > > > >> > > >> >> >
>>> > > > > > >> > > >> >> >> wrote:
>>> > > > > > >> > > >> >> >> > I searched around and found HBASE-1645 "HBase
>>> > could
>>> > > > be
>>> > > > > a
>>> > > > > > >> > little
>>> > > > > > >> > > >> more
>>> > > > > > >> > > >> >> >> dainty
>>> > > > > > >> > > >> >> >> > when zk is pulled out from under it"
>>> > > > > > >> > > >> >> >> >
>>> > > > > > >> > > >> >> >> > I am wondering whether my problem is related
>>> to
>>> > > this
>>> > > > > > >> > HBase-1645.
>>> > > > > > >> > > Or
>>> > > > > > >> > > >> >> >> probably
>>> > > > > > >> > > >> >> >> > has nothing to do with it?
>>> > > > > > >> > > >> >> >> >
>>> > > > > > >> > > >> >> >> > thanks
>>> > > > > > >> > > >> >> >> >
>>> > > > > > >> > > >> >> >> >
>>> > > > > > >> > > >> >> >> > On Wed, Nov 11, 2009 at 12:02 PM, Zhenyu Zhong
>>> <
>>> > > > > > >> > > >> >> zhongresearch@gmail.com
>>> > > > > > >> > > >> >> >> >wrote:
>>> > > > > > >> > > >> >> >> >
>>> > > > > > >> > > >> >> >> >> Hi,
>>> > > > > > >> > > >> >> >> >>
>>> > > > > > >> > > >> >> >> >> I have been monitoring the system resources
>>> such
>>> > > as
>>> > > > > > memory
>>> > > > > > >> > Heap
>>> > > > > > >> > > >> usage
>>> > > > > > >> > > >> >> >> for
>>> > > > > > >> > > >> >> >> >> Regionserver, load average, network
>>> bandwidth,
>>> > > etc.
>>> > > > > > >> > > >> >> >> >>
>>> > > > > > >> > > >> >> >> >> I experienced a disconnection last night.
>>> When I
>>> > > > > checked
>>> > > > > > >> the
>>> > > > > > >> > > >> memory
>>> > > > > > >> > > >> >> Heap
>>> > > > > > >> > > >> >> >> >> for Regionserver, it was 2336 which is way
>>> below
>>> > > the
>>> > > > > max
>>> > > > > > >> Heap
>>> > > > > > >> > > >> 4096. I
>>> > > > > > >> > > >> >> >> don't
>>> > > > > > >> > > >> >> >> >> think it is Regionserver problem. The
>>> requests
>>> > > that
>>> > > > > the
>>> > > > > > >> > > >> regionserver
>>> > > > > > >> > > >> >> got
>>> > > > > > >> > > >> >> >> was
>>> > > > > > >> > > >> >> >> >> about 126 which doesn't seem to be high. The
>>> > load
>>> > > > > > average
>>> > > > > > >> is
>>> > > > > > >> > > kind
>>> > > > > > >> > > >> of
>>> > > > > > >> > > >> >> >> high
>>> > > > > > >> > > >> >> >> >> >100. (but I saw disconnections while the
>>> load
>>> > > > average
>>> > > > > > is
>>> > > > > > >> > > around
>>> > > > > > >> > > >> 30).
>>> > > > > > >> > > >> >> >> The
>>> > > > > > >> > > >> >> >> >> bandwidth is not a problem.
>>> > > > > > >> > > >> >> >> >>
>>> > > > > > >> > > >> >> >> >> However, I found error messages from the
>>> > > zookeeper.
>>> > > > > > Please
>>> > > > > > >> > see
>>> > > > > > >> > > the
>>> > > > > > >> > > >> >> >> >> following error messages. There are some
>>> > > > > > >> > > >> >> >> >> BufferUnderflowException error, IOException
>>> > > > error,etc.
>>> > > > > > >> > > >> >> >> >> May I ask what might be the causes that
>>> generate
>>> > > > these
>>> > > > > > >> error
>>> > > > > > >> > > >> >> messages?
>>> > > > > > >> > > >> >> >> >>
>>> > > > > > >> > > >> >> >> >> Many thanks
>>> > > > > > >> > > >> >> >> >> zhenyu
>>> > > > > > >> > > >> >> >> >>
>>> > > > > > >> > > >> >> >> >>
>>> > > > > > >> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
>>> > > > > > >> > > org.apache.zookeeper.server.Request:
>>> > > > > > >> > > >> >> >> Ignoring
>>> > > > > > >> > > >> >> >> >> exception during toString
>>> > > > > > >> > > >> >> >> >> java.nio.BufferUnderflowException
>>> > > > > > >> > > >> >> >> >> at
>>> > > > > > >> > java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
>>> > > > > > >> > > >> >> >> >> at
>>> > > > > java.nio.ByteBuffer.get(ByteBuffer.java:675)
>>> > > > > > >> > > >> >> >> >> at
>>> > > > > > >> > > >> >> >>
>>> > > > > > >>
>>> org.apache.zookeeper.server.Request.toString(Request.java:199)
>>> > > > > > >> > > >> >> >> >> at
>>> > > > java.lang.String.valueOf(String.java:2827)
>>> > > > > > >> > > >> >> >> >> at
>>> > > > > > >> > > java.lang.StringBuilder.append(StringBuilder.java:115)
>>> > > > > > >> > > >> >> >> >> at
>>> > > > > > >> > > >> >> >> >>
>>> > > > > > >> > > >> >> >>
>>> > > > > > >> > > >> >>
>>> > > > > > >> > > >>
>>> > > > > > >> > >
>>> > > > > > >> >
>>> > > > > > >>
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> org.apache.zookeeper.server.quorum.CommitProcessor.processRequest(CommitProcessor.java:167)
>>> > > > > > >> > > >> >> >> >> at
>>> > > > > > >> > > >> >> >> >>
>>> > > > > > >> > > >> >> >>
>>> > > > > > >> > > >> >>
>>> > > > > > >> > > >>
>>> > > > > > >> > >
>>> > > > > > >> >
>>> > > > > > >>
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> org.apache.zookeeper.server.quorum.FollowerRequestProcessor.run(FollowerRequestProcessor.java:68)
>>> > > > > > >> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
>>> > > > > > >> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
>>> > > > > > >> > > >> >> >> >> Exception causing close of session
>>> > > 0x424dfd908c50009
>>> > > > > > >> > > >> >> >> >> due to java.io.IOException: Read error
>>> > > > > > >> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
>>> > > > > > >> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
>>> > > > > > >> > > >> >> >> >> IOException stack trace
>>> > > > > > >> > > >> >> >> >> java.io.IOException: Read error
>>> > > > > > >> > > >> >> >> >> at
>>> > > > > > >> > > >> >> >> >>
>>> > > > > > >> > > >> >>
>>> > > > > > >> > >
>>> > > > > >
>>> > > org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:372)
>>> > > > > > >> > > >> >> >> >> at
>>> > > > > > >> > > >> >> >> >>
>>> > > > > > >> > > >> >> >>
>>> > > > > > >> > > >> >>
>>> > > > > > >> > > >>
>>> > > > > > >> > >
>>> > > > > > >> >
>>> > > > > > >>
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:221)
>>> > > > > > >> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
>>> > > > > > >> > > >> >> >> >>
>>> > > org.apache.zookeeper.server.quorum.CommitProcessor:
>>> > > > > > >> > Processing
>>> > > > > > >> > > >> >> request::
>>> > > > > > >> > > >> >> >> >> sessionid:0x424dfd908c50009 type:setWatches
>>> > > > > > >> > > >> cxid:0xfffffffffffffff8
>>> > > > > > >> > > >> >> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
>>> > > > > > >> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
>>> > > > > > >> > > org.apache.zookeeper.server.Request:
>>> > > > > > >> > > >> >> >> Ignoring
>>> > > > > > >> > > >> >> >> >> exception during toString
>>> > > > > > >> > > >> >> >> >> java.nio.BufferUnderflowException
>>> > > > > > >> > > >> >> >> >> at
>>> > > > > > >> > java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
>>> > > > > > >> > > >> >> >> >> at
>>> > > > > java.nio.ByteBuffer.get(ByteBuffer.java:675)
>>> > > > > > >> > > >> >> >> >> at
>>> > > > > > >> > > >> >> >>
>>> > > > > > >>
>>> org.apache.zookeeper.server.Request.toString(Request.java:199)
>>> > > > > > >> > > >> >> >> >> at
>>> > > > java.lang.String.valueOf(String.java:2827)
>>> > > > > > >> > > >> >> >> >> at
>>> > > > > > >> > > java.lang.StringBuilder.append(StringBuilder.java:115)
>>> > > > > > >> > > >> >> >> >> at
>>> > > > > > >> > > >> >> >> >>
>>> > > > > > >> > > >> >> >>
>>> > > > > > >> > > >> >>
>>> > > > > > >> > > >>
>>> > > > > > >> > >
>>> > > > > > >> >
>>> > > > > > >>
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:74)
>>> > > > > > >> > > >> >> >> >> at
>>> > > > > > >> > > >> >> >> >>
>>> > > > > > >> > > >> >> >>
>>> > > > > > >> > > >> >>
>>> > > > > > >> > > >>
>>> > > > > > >> > >
>>> > > > > > >> >
>>> > > > > > >>
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)
>>> > > > > > >> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
>>> > > > > > >> > > >> >> >> >>
>>> > org.apache.zookeeper.server.FinalRequestProcessor:
>>> > > > > > >> Processing
>>> > > > > > >> > > >> >> request::
>>> > > > > > >> > > >> >> >> >> sessionid:0x424dfd908c50009 type:setWatches
>>> > > > > > >> > > >> cxid:0xfffffffffffffff8
>>> > > > > > >> > > >> >> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
>>> > > > > > >> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
>>> > > > > > >> > > org.apache.zookeeper.server.Request:
>>> > > > > > >> > > >> >> >> Ignoring
>>> > > > > > >> > > >> >> >> >> exception during toString
>>> > > > > > >> > > >> >> >> >> java.nio.BufferUnderflowException
>>> > > > > > >> > > >> >> >> >> at
>>> > > > > > >> > java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
>>> > > > > > >> > > >> >> >> >> at
>>> > > > > java.nio.ByteBuffer.get(ByteBuffer.java:675)
>>> > > > > > >> > > >> >> >> >> at
>>> > > > > > >> > > >> >> >>
>>> > > > > > >>
>>> org.apache.zookeeper.server.Request.toString(Request.java:199)
>>> > > > > > >> > > >> >> >> >> at
>>> > > > > > >> > > >> >> >> >>
>>> > > > > > >> > > >>
>>> > > > > > >>
>>> > > >
>>> org.apache.log4j.or.DefaultRenderer.doRender(DefaultRenderer.java:36)
>>> > > > > > >> > > >> >> >> >> at
>>> > > > > > >> > > >> >> >> >>
>>> > > > > > >> > >
>>> > > > org.apache.log4j.or.RendererMap.findAndRender(RendererMap.java:80)
>>> > > > > > >> > > >> >> >> >> at
>>> > > > > > >> > > >> >> >> >>
>>> > > > > > >> > > >> >> >>
>>> > > > > > >> > > >> >>
>>> > > > > > >> > > >>
>>> > > > > > >> > >
>>> > > > > > >> >
>>> > > > > > >>
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> org.apache.log4j.spi.LoggingEvent.getRenderedMessage(LoggingEvent.java:362)
>>> > > > > > >> > > >> >> >> >> at
>>> > > > > > >> > > >> >> >> >>
>>> > > > > > >> > > >> >> >>
>>> > > > > > >> > > >> >>
>>> > > > > > >> > > >>
>>> > > > > > >> > >
>>> > > > > > >> >
>>> > > > > > >>
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> org.apache.log4j.helpers.PatternParser$BasicPatternConverter.convert(PatternParser.java:403)
>>> > > > > > >> > > >> >> >> >> at
>>> > > > > > >> > > >> >> >> >>
>>> > > > > > >> > > >> >> >>
>>> > > > > > >> > > >> >>
>>> > > > > > >> > > >>
>>> > > > > > >> > >
>>> > > > > > >> >
>>> > > > > > >>
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> org.apache.log4j.helpers.PatternConverter.format(PatternConverter.java:65)
>>> > > > > > >> > > >> >> >> >> at
>>> > > > > > >> > > >> >>
>>> > > > > org.apache.log4j.PatternLayout.format(PatternLayout.java:502)
>>> > > > > > >> > > >> >> >> >> at
>>> > > > > > >> > > >> >> >> >>
>>> > > > > > >> > >
>>> > > > org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:302)
>>> > > > > > >> > > >> >> >> >> at
>>> > > > > > >> > > >> >> >> >>
>>> > > > > > >> > > >> >> >>
>>> > > > > > >> > > >> >>
>>> > > > > > >> > > >>
>>> > > > > > >> > >
>>> > > > > > >> >
>>> > > > > > >>
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> org.apache.log4j.DailyRollingFileAppender.subAppend(DailyRollingFileAppender.java:359)
>>> > > > > > >> > > >> >> >> >> at
>>> > > > > > >> > > >> >> >>
>>> > > > > > >>
>>> org.apache.log4j.WriterAppender.append(WriterAppender.java:160)
>>> > > > > > >> > > >> >> >> >> at
>>> > > > > > >> > > >> >> >> >>
>>> > > > > > >> > > >>
>>> > > > > > >>
>>> > > >
>>> org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
>>> > > > > > >> > > >> >> >> >> at
>>> > > > > > >> > > >> >> >> >>
>>> > > > > > >> > > >> >> >>
>>> > > > > > >> > > >> >>
>>> > > > > > >> > > >>
>>> > > > > > >> > >
>>> > > > > > >> >
>>> > > > > > >>
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
>>> > > > > > >> > > >> >> >> >> at
>>> > > > > > >> > > >>
>>> > org.apache.log4j.Category.callAppenders(Category.java:206)
>>> > > > > > >> > > >> >> >> >> at
>>> > > > > > >> > > org.apache.log4j.Category.forcedLog(Category.java:391)
>>> > > > > > >> > > >> >> >> >> at
>>> > > > > > >> org.apache.log4j.Category.debug(Category.java:260)
>>> > > > > > >> > > >> >> >> >> at
>>> > > > > > >> > > >> >> >> >>
>>> > > > > > >> > > >> >> >>
>>> > > > > > >> > > >> >>
>>> > > > > > >> > > >>
>>> > > > > > >> > >
>>> > > > > > >> >
>>> > > > > > >>
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:147)
>>> > > > > > >> > > >> >> >> >> at
>>> > > > > > >> > > >> >> >> >>
>>> > > > > > >> > > >> >> >>
>>> > > > > > >> > > >> >>
>>> > > > > > >> > > >>
>>> > > > > > >> > >
>>> > > > > > >> >
>>> > > > > > >>
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)
>>> > > > > > >> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
>>> > > > > > >> > > org.apache.zookeeper.server.Request:
>>> > > > > > >> > > >> >> >> Ignoring
>>> > > > > > >> > > >> >> >> >> exception during toString
>>> > > > > > >> > > >> >> >> >> sessionid:0x424dfd908c50009 type:setWatches
>>> > > > > > >> > > >> cxid:0xfffffffffffffff8
>>> > > > > > >> > > >> >> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
>>> > > > > > >> > > >> >> >> >> 2009-11-11 04:29:00,810 INFO
>>> > > > > > >> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
>>> > > > > > >> > > >> >> >> >> closing session:0x424dfd908c50009
>>> NIOServerCnxn:
>>> > > > > > >> > > >> >> >> >> java.nio.channels.SocketChannel[connected
>>> > local=/
>>> > > > > > >> > > >> >> 192.168.100.131:2181
>>> > > > > > >> > > >> >> >> remote=/
>>> > > > > > >> > > >> >> >> >> 192.168.100.131:35500]
>>> > > > > > >> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
>>> > > > > > >> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
>>> > > > > > >> > > >> >> >> >> ignoring exception during input shutdown
>>> > > > > > >> > > >> >> >> >> java.net.SocketException: Transport endpoint
>>> is
>>> > > not
>>> > > > > > >> connected
>>> > > > > > >> > > >> >> >> >> at
>>> > > > > sun.nio.ch.SocketChannelImpl.shutdown(Native
>>> > > > > > >> > Method)
>>> > > > > > >> > > >> >> >> >> at
>>> > > > > > >> > > >> >> >> >>
>>> > > > > > >> > > >> >>
>>> > > > > > >> > >
>>> > > > > >
>>> > > sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
>>> > > > > > >> > > >> >> >> >> at
>>> > > > > > >> > > >> >> >>
>>> > > > > > >>
>>> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>>> > > > > > >> > > >> >> >> >> at
>>> > > > > > >> > > >> >> >> >>
>>> > > > > > >> > > >> >>
>>> > > > > > >> > >
>>> > > > > > >>
>>> > > > >
>>> > org.apache.zookeeper.server.NIOServerCnxn.close(NIOServerCnxn.java:848)
>>> > > > > > >> > > >> >> >> >> at
>>> > > > > > >> > > >> >> >> >>
>>> > > > > > >> > > >> >>
>>> > > > > > >> > >
>>> > > > > >
>>> > > org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:498)
>>> > > > > > >> > > >> >> >> >> at
>>> > > > > > >> > > >> >> >> >>
>>> > > > > > >> > > >> >> >>
>>> > > > > > >> > > >> >>
>>> > > > > > >> > > >>
>>> > > > > > >> > >
>>> > > > > > >> >
>>> > > > > > >>
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:221)
>>> > > > > > >> > > >> >> >> >>
>>> > > > > > >> > > >> >> >> >
>>> > > > > > >> > > >> >> >>
>>> > > > > > >> > > >> >> >
>>> > > > > > >> > > >> >>
>>> > > > > > >> > > >> >
>>> > > > > > >> > > >>
>>> > > > > > >> > > >
>>> > > > > > >> > >
>>> > > > > > >> >
>>> > > > > > >>
>>> > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>>
>>
>>
>
Re: regionserver disconnection
Posted by Zhenyu Zhong <zh...@gmail.com>.
Hi,
After I re-organize the cluster, the experiment ran into problem faster than
before.
Basically the changes are to use machines with less resources as zookeeper
quorums and machines with more resources as regionserver.
>From the log, I still see the pause around 1 minute.
I enabled the GC logging, please see
http://pastebin.com/m1d4ce0f1
for details.
In general I don't see many pauses in the GC.
What is more interesting, 1 hour after the 1st regionserver disconnected,
the master log complained about too many open files. This didn't happen
before.
I checked the system OS setup as well as the limits.conf. I also checked the
running processes. The total open files don't reach the limit. I am confused
a bit.
Please see the following master log.
2009-11-13 20:06:02,114 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner scan of 4658 row(s) of meta region {server:
192.168.100.128:60021, regionname: .META.,,1, startKey: <>} complete
2009-11-13 20:06:02,114 INFO org.apache.hadoop.hbase.master.BaseScanner: All
1 .META. region(s) scanned
2009-11-13 20:06:07,677 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
response for sessionid:0x424eebf1c10004c after 3ms
2009-11-13 20:06:08,178 INFO org.apache.hadoop.hdfs.DFSClient: Exception in
createBlockOutputStream java.io.IOException: Bad connect ack with
firstBadLink 192.168.100.123:50010
2009-11-13 20:06:08,178 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning
block blk_-2808245019291145247_5478039
2009-11-13 20:06:09,682 INFO org.apache.hadoop.hdfs.DFSClient: Exception in
createBlockOutputStream java.io.EOFException
2009-11-13 20:06:09,682 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning
block blk_1074853606841896259_5478048
2009-11-13 20:06:10,334 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
response for sessionid:0x24eebf1043003c after 1ms
2009-11-13 20:06:21,018 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
response for sessionid:0x424eebf1c10004c after 0ms
2009-11-13 20:06:23,674 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
response for sessionid:0x24eebf1043003c after 0ms
2009-11-13 20:06:24,828 INFO org.apache.hadoop.hdfs.DFSClient: Exception in
createBlockOutputStream java.io.IOException: Bad connect ack with
firstBadLink 192.168.100.123:50010
2009-11-13 20:06:24,828 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning
block blk_-6642544517082142289_5478063
2009-11-13 20:06:24,828 INFO org.apache.hadoop.hdfs.DFSClient: Exception in
createBlockOutputStream java.net.SocketException: Too many open files
2009-11-13 20:06:24,828 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning
block blk_2057511041109796090_5478063
2009-11-13 20:06:24,928 INFO org.apache.hadoop.hdfs.DFSClient: Exception in
createBlockOutputStream java.net.SocketException: Too many open files
2009-11-13 20:06:24,928 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning
block blk_8219260302213892894_5478064
2009-11-13 20:06:30,855 INFO org.apache.hadoop.hdfs.DFSClient: Exception in
createBlockOutputStream java.net.SocketException: Too many open files
2009-11-13 20:06:30,855 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning
block blk_1669205542853067709_5478235
2009-11-13 20:06:30,905 INFO org.apache.hadoop.hdfs.DFSClient: Exception in
createBlockOutputStream java.net.SocketException: Too many open files
2009-11-13 20:06:30,905 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning
block blk_9128897691346270351_5478237
2009-11-13 20:06:30,955 INFO org.apache.hadoop.hdfs.DFSClient: Exception in
createBlockOutputStream java.net.SocketException: Too many open files
2009-11-13 20:06:30,955 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning
block blk_1116845144864123018_5478240
2009-11-13 20:06:34,372 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
response for sessionid:0x424eebf1c10004c after 0ms
2009-11-13 20:06:37,034 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
response for sessionid:0x24eebf1043003c after 0ms
2009-11-13 20:06:37,235 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer
Exception: java.io.IOException: Too many open files
at sun.nio.ch.IOUtil.initPipe(Native Method)
at sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:49)
at
sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:18)
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.get(SocketIOWithTimeout.java:407)
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:322)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
at
org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
at
org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
at java.io.DataOutputStream.write(DataOutputStream.java:90)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2290)
2009-11-13 20:06:37,235 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_8148813491785406356_5478475 bad datanode[0]
192.168.100.123:50010
2009-11-13 20:06:37,235 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_8148813491785406356_5478475 in pipeline
192.168.100.123:50010, 192.168.100.134:50010, 192.168.100.122:50010: bad
datanode 192.168.100.123:50010
2009-11-13 20:06:37,436 INFO org.apache.hadoop.hdfs.DFSClient: Exception in
createBlockOutputStream java.net.SocketException: Too many open files
2009-11-13 20:06:37,436 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning
block blk_2119727700857186236_5478498
On Thu, Nov 12, 2009 at 4:21 PM, Zhenyu Zhong <zh...@gmail.com>wrote:
> Will do.
>
> thanks
> zhenyu
>
>
> On Thu, Nov 12, 2009 at 3:33 PM, stack <st...@duboce.net> wrote:
>
>> Enable GC logging too on this next run (see hbase-env.sh). Lets try and
>> get
>> to the bottom of whats going on.
>> Thanks,
>> St.Ack
>>
>> On Thu, Nov 12, 2009 at 12:29 PM, Zhenyu Zhong <zhongresearch@gmail.com
>> >wrote:
>>
>> > I can switch the boxes that run zookeeper with the ones that run
>> > regionservers.
>> > I will see the results later.
>> >
>> > FYI. The node does have the 10 minutes zookeeper.session.timeout value
>> in
>> > place.
>> >
>> > thanks
>> > zhenyu
>> >
>> > On Thu, Nov 12, 2009 at 3:21 PM, stack <st...@duboce.net> wrote:
>> >
>> > > On Thu, Nov 12, 2009 at 11:50 AM, Zhenyu Zhong <
>> zhongresearch@gmail.com
>> > > >wrote:
>> > >
>> > > >
>> > > > In my cluster, half of the cluster have 2 disks 400GB each per
>> machine,
>> > > and
>> > > > half of the cluster have 6 disks per machine. Maybe we should run
>> > > > zookeeper
>> > > > on the machines with 2 disks and RS on machines with 6 disks?
>> > > >
>> > > > That would make most sense only in the below, it looks like the RS
>> that
>> > > had
>> > > issue had 4 disks?
>> > >
>> > >
>> > >
>> > > > BTW, the 10 minutes zookeeper.session.timeout has been set during
>> the
>> > > > experiment.
>> > > >
>> > >
>> > > And for sure this node had it in place?
>> > > St.Ack
>> > >
>> > >
>> > >
>> > > >
>> > > > thanks
>> > > > zhenyu
>> > > >
>> > > > On Thu, Nov 12, 2009 at 2:08 PM, stack <st...@duboce.net> wrote:
>> > > >
>> > > > > On Thu, Nov 12, 2009 at 8:40 AM, Zhenyu Zhong <
>> > zhongresearch@gmail.com
>> > > > > >wrote:
>> > > > >
>> > > > >
>> > > > > > Though I experienced 2 regionserver disconnection this morning,
>> it
>> > > > looks
>> > > > > > better from the regionserver log. (please see the following log)
>> > > > > >
>> > > > > >
>> > > > > > http://pastebin.com/m496dbfae
>> > > > > >
>> > > > > >
>> > > > > > I checked diskIO for one of the regionserver(192.168.100.116)
>> > during
>> > > > the
>> > > > > > time it disconnected.
>> > > > > >
>> > > > > > Time: 03:04:58 AM
>> > > > > > Device: tps Blk_read/s Blk_wrtn/s Blk_read
>> > > Blk_wrtn
>> > > > > > sda 105.31 5458.83 19503.64 9043873239
>> > > 32312473676
>> > > > > > sda1 2.90 6.64 99.25 10993934
>> > > 164433464
>> > > > > > sda2 1.72 23.76 30.25 39365817
>> > > 50124008
>> > > > > > sda3 0.30 0.38 3.58 624930
>> > > 5923000
>> > > > > > sda4 100.39 5428.06 19370.56 8992888270
>> > > 32091993204
>> > > > > >
>> > > > > >
>> > > > > Is this high for you? 20k blocks/second would seem to be high but
>> > its
>> > > > one
>> > > > > disk only and its not being shared by zk anymore so shouldn't
>> matter?
>> > > > >
>> > > > >
>> > > > > > I also checked the zookeeper quorum server that the regionserver
>> > > tried
>> > > > to
>> > > > > > connect according to the log. However, I don't see
>> 192.168.100.116
>> > in
>> > > > the
>> > > > > > client list of the zookeeper quorum server that the regionserver
>> > > tried
>> > > > to
>> > > > > > connect.
>> > > > > > Would that be a problem?
>> > > > > >
>> > > > > >
>> > > > > Is that because the ephemeral node for the regionserver had
>> > evaporated?
>> > > > > Lost
>> > > > > its lease w/ zk by the time you went to look?
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > > Thu Nov 12 15:04:52 UTC 2009
>> > > > > > Zookeeper version: 3.2.1-808558, built on 08/27/2009 18:48 GMT
>> > > > > > Clients:
>> > > > > > /192.168.100.127:43045[1](queued=0,recved=26,sent=0)
>> > > > > > /192.168.100.131:39091[1](queued=0,recved=964,sent=0)
>> > > > > > /192.168.100.124:35961[1](queued=0,recved=3266,sent=0)
>> > > > > > /192.168.100.123:47935[1](queued=0,recved=233,sent=0)
>> > > > > > /192.168.100.125:46931[1](queued=0,recved=2,sent=0)
>> > > > > > /192.168.100.118:54924[1](queued=0,recved=295,sent=0)
>> > > > > > /192.168.100.118:41390[1](queued=0,recved=2290,sent=0)
>> > > > > > /192.168.100.136:42243[1](queued=0,recved=0,sent=0)
>> > > > > >
>> > > > > > Latency min/avg/max: 0/17/6333
>> > > > > > Received: 47111
>> > > > > > Sent: 0
>> > > > > > Outstanding: 0
>> > > > > > Zxid: 0x77000083f4
>> > > > > > Mode: leader
>> > > > > > Node count: 23
>> > > > > >
>> > > > >
>> > > > >
>> > > > > That 6 second maximum latency is pretty bad but should be well
>> within
>> > > the
>> > > > > zk
>> > > > > session timeout.
>> > > > >
>> > > > > So, problem is likely on the zk client side of the session; i.e.
>> in
>> > the
>> > > > RS.
>> > > > > You could enable GC logging as J-D suggested to see if you have
>> any
>> > big
>> > > > > pauses, pauses > zk session timeout.
>> > > > >
>> > > > > When the RS went down, it didn't look too heavily loaded:
>> > > > >
>> > > > >
>> > > > > 1. 2009-11-12 15:04:52,830 INFO
>> > > > > org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of
>> > metrics:
>> > > > > request=1.5166667, regions=322, stores=657, storefiles=631,
>> > > > > storefileIndexSize=61, memstoreSize=1472, usedHeap=2819,
>> > > maxHeap=4079,
>> > > > > blockCacheSize=658110960, blockCacheFree=197395984,
>> > > > blockCacheCount=9903,
>> > > > > blockCacheHitRatio=99
>> > > > >
>> > > > > Its serving a few reads? The number of store files seems fine.
>> Not
>> > > too
>> > > > > much memory used.
>> > > > >
>> > > > > Looking at the logs, I see the Lease Still Held exception. This
>> > > happens
>> > > > > when the RS does its regular report to the master but the master
>> > thinks
>> > > > the
>> > > > > RS has since restarted. It'll think this probably because it
>> noticed
>> > > > that
>> > > > > the RS's znode in zk had gone away and it considered the RS dead.
>> > > > >
>> > > > > Looking too at your logs I see this gap in the zk pinging:
>> > > > >
>> > > > >
>> > > > > 1. 2009-11-12 15:03:39,325 DEBUG
>> org.apache.zookeeper.ClientCnxn:
>> > Got
>> > > > > ping response for sessionid:0x224e55436ad0004 after 0ms
>> > > > > 2. 2009-11-12 15:03:43,113 DEBUG
>> org.apache.zookeeper.ClientCnxn:
>> > Got
>> > > > > ping response for sessionid:0x24e55436a0007d after 0ms
>> > > > >
>> > > > > Where in the lines above it, its reporting about every ten
>> seconds,
>> > > here
>> > > > > there is a big gap.
>> > > > >
>> > > > > Do you have ganglia or something that will let you look more into
>> > what
>> > > > was
>> > > > > happening on this machine around this time? Is the machine OK?
>> It
>> > > looks
>> > > > > lightly loaded and you have your cluster nicely laid out.
>> Something
>> > > odd
>> > > > is
>> > > > > going on. What about things like the write speed to disk? In the
>> > past
>> > > > > strange issues have been explained by incorrectly set BIOS which
>> made
>> > > > disks
>> > > > > run at 1/100th of their proper speed.
>> > > > >
>> > > > > St.Ack
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > >
>> > > > > > Best,
>> > > > > > zhenyu
>> > > > > >
>> > > > > >
>> > > > > > On Wed, Nov 11, 2009 at 3:58 PM, Zhenyu Zhong <
>> > > zhongresearch@gmail.com
>> > > > > > >wrote:
>> > > > > >
>> > > > > > > Stack
>> > > > > > >
>> > > > > > > I am very appreciated for your comments.
>> > > > > > > I will use the zookeeper monitoring script on my cluster and
>> let
>> > it
>> > > > run
>> > > > > > > overnight to see the result.
>> > > > > > > I will follow up the thread when I get anything.
>> > > > > > >
>> > > > > > > thanks
>> > > > > > > zhenyu
>> > > > > > >
>> > > > > > >
>> > > > > > > On Wed, Nov 11, 2009 at 3:52 PM, stack <st...@duboce.net>
>> wrote:
>> > > > > > >
>> > > > > > >> I see these in your log too:
>> > > > > > >>
>> > > > > > >>
>> > > > > > >> 1. 2009-11-11 04:27:19,018 DEBUG
>> > > org.apache.zookeeper.ClientCnxn:
>> > > > > Got
>> > > > > > >> ping response for sessionid:0x424dfd908c50009 after 4544ms
>> > > > > > >> 2. 2009-11-11 04:27:19,018 DEBUG
>> > > org.apache.zookeeper.ClientCnxn:
>> > > > > Got
>> > > > > > >> ping response for sessionid:0x24dfd90c810002 after 4548ms
>> > > > > > >> 3. 2009-11-11 04:27:43,960 DEBUG
>> > > org.apache.zookeeper.ClientCnxn:
>> > > > > Got
>> > > > > > >> ping response for sessionid:0x424dfd908c50009 after 9030ms
>> > > > > > >> 4. 2009-11-11 04:27:43,960 DEBUG
>> > > org.apache.zookeeper.ClientCnxn:
>> > > > > Got
>> > > > > > >> ping response for sessionid:0x24dfd90c810002 after 9030ms
>> > > > > > >> 5. 2009-11-11 04:27:48,830 DEBUG
>> > > org.apache.zookeeper.ClientCnxn:
>> > > > > Got
>> > > > > > >> ping response for sessionid:0x424dfd908c50009 after 0ms
>> > > > > > >> 6. 2009-11-11 04:27:48,830 DEBUG
>> > > org.apache.zookeeper.ClientCnxn:
>> > > > > Got
>> > > > > > >> ping response for sessionid:0x24dfd90c810002 after 0ms
>> > > > > > >>
>> > > > > > >> Which would indicate lag talking to zk too.
>> > > > > > >>
>> > > > > > >> Appending HLog also seems to get steadily worse.
>> > > > > > >>
>> > > > > > >> I also see these:
>> > > > > > >>
>> > > > > > >>
>> > > > > > >> 1. 2009-11-11 04:27:01,233 WARN
>> > org.apache.zookeeper.ClientCnxn:
>> > > > > > >> Exception closing session 0x24dfd90c810002 to
>> > > > > > >> sun.nio.ch.SelectionKeyImpl@5e16a7e3
>> > > > > > >> 2. java.io.IOException: Read error rc = -1
>> > > > > > >> java.nio.DirectByteBuffer[pos=0 lim=4 cap=4]
>> > > > > > >> 3. at
>> > > > > > >>
>> > > > >
>> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
>> > > > > > >> 4. at
>> > > > > > >>
>> > > > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
>> > > > > > >>
>> > > > > > >>
>> > > > > > >> Were all 5 quorum members participating? Check by doing
>> "echo
>> > > > stat|nc
>> > > > > > >> QUORUM_HOST port" for each member of quorum.... the below
>> might
>> > > > help:
>> > > > > > >>
>> > > > > > >> members="1,2,3,4,5"
>> > > > > > >> while true; do date; for i in $members; do echo $i; echo stat
>> |
>> > nc
>> > > > $i
>> > > > > > >> 2181;
>> > > > > > >> done; sleep 10; done
>> > > > > > >>
>> > > > > > >> We need to improve our zk_dump. It just reports quorum
>> > members...
>> > > > not
>> > > > > > if
>> > > > > > >> they are actually participating or not. The above will tell
>> you
>> > > if
>> > > > > dead
>> > > > > > >> or
>> > > > > > >> alive.
>> > > > > > >>
>> > > > > > >> Please report back after checking above and following the
>> > > > Jean-Daniel
>> > > > > > >> prescription.
>> > > > > > >>
>> > > > > > >> Thanks,
>> > > > > > >> St.Ack
>> > > > > > >>
>> > > > > > >>
>> > > > > > >>
>> > > > > > >> On Wed, Nov 11, 2009 at 12:29 PM, Zhenyu Zhong <
>> > > > > zhongresearch@gmail.com
>> > > > > > >> >wrote:
>> > > > > > >>
>> > > > > > >> > Thanks a lot for the help.
>> > > > > > >> >
>> > > > > > >> > I understand that the change of disk configurations should
>> be
>> > > done
>> > > > > > >> > progressively such that HDFS can balance and recover from a
>> > disk
>> > > > > going
>> > > > > > >> > away.
>> > > > > > >> >
>> > > > > > >> > The data we are trying to import into HBase is about
>> > 200million
>> > > > > rows,
>> > > > > > >> each
>> > > > > > >> > row is about hundreds of bytes. Basically, we read in a
>> text
>> > > file
>> > > > > from
>> > > > > > >> > HDFS,
>> > > > > > >> > then for each row in the text file, we fetch data from
>> HBase
>> > > > table,
>> > > > > > we
>> > > > > > >> > compare the column value and update the latest value back
>> into
>> > > > > HBase.
>> > > > > > I
>> > > > > > >> was
>> > > > > > >> > suggested to use CheckAndPut. Anyway this is the common
>> > usecase
>> > > > on
>> > > > > my
>> > > > > > >> > side.
>> > > > > > >> >
>> > > > > > >> >
>> > > > > > >> > Many thanks
>> > > > > > >> > zhenyu
>> > > > > > >> >
>> > > > > > >> >
>> > > > > > >> >
>> > > > > > >> > On Wed, Nov 11, 2009 at 3:03 PM, Jean-Daniel Cryans <
>> > > > > > >> jdcryans@apache.org
>> > > > > > >> > >wrote:
>> > > > > > >> >
>> > > > > > >> > > 6 disks, 19 machines and you see those kind of issues?
>> I'm
>> > > quite
>> > > > > > >> > > surprised and I want to learn more about what you trying
>> to
>> > > to.
>> > > > > > >> > >
>> > > > > > >> > > But first, if hbase is managing ZK for you then you can
>> > change
>> > > > the
>> > > > > > >> > > value of hbase.zookeeper.property.dataDir so that it
>> points
>> > to
>> > > > its
>> > > > > > own
>> > > > > > >> > > mount. Make sure you also change dfs.data.dir in
>> > > > > conf/hdfs-site.xml
>> > > > > > >> > > (hadoop stuff) so that it stops pointing there (you
>> should
>> > > have
>> > > > 6
>> > > > > > >> > > values there at the moment if you configured it
>> correctly).
>> > > > Doing
>> > > > > > this
>> > > > > > >> > > could result in data loss if you do it all at once so do
>> the
>> > > > > change
>> > > > > > >> > > progressively. This also means that you may have to keep
>> a
>> > > > > seperate
>> > > > > > >> > > HDFS configuration for those particular nodes. Please ask
>> > more
>> > > > > > >> > > questions if it's not very clear.
>> > > > > > >> > >
>> > > > > > >> > > WRT to dedicating nodes to Zookeeeper, it makes sense
>> only
>> > if
>> > > > you
>> > > > > > have
>> > > > > > >> > > a lot of nodes but in your case, unless there's also CPU
>> > > > > contention,
>> > > > > > >> > > you're good with only a dedicated disk.
>> > > > > > >> > >
>> > > > > > >> > > Now for what you are trying to do, is this a massive
>> import
>> > > that
>> > > > > > goes
>> > > > > > >> > > over many hours? How big is each row? I'd love to get a
>> lot
>> > of
>> > > > > > details
>> > > > > > >> > > to help you optimize your cluster for your particular
>> > problem.
>> > > > > > >> > >
>> > > > > > >> > > Thx,
>> > > > > > >> > >
>> > > > > > >> > > J-D
>> > > > > > >> > >
>> > > > > > >> > > On Wed, Nov 11, 2009 at 11:51 AM, Zhenyu Zhong <
>> > > > > > >> zhongresearch@gmail.com>
>> > > > > > >> > > wrote:
>> > > > > > >> > > > J-D,
>> > > > > > >> > > >
>> > > > > > >> > > > Thanks a lot for the insightful comments.
>> > > > > > >> > > > We are running 5 zookeeper quorums now. On these 5
>> quorum
>> > > > > servers,
>> > > > > > >> we
>> > > > > > >> > > also
>> > > > > > >> > > > run datanode, tasktracker and regionserver.
>> > > > > > >> > > > Now we have 6 disks per machine.
>> > > > > > >> > > > If it is the disk IO saturation, is there anyway to
>> > > configure
>> > > > > > >> zookeeper
>> > > > > > >> > > to
>> > > > > > >> > > > write to a seperate disk which is not the same one used
>> by
>> > > > HDFS?
>> > > > > > >> > > > Or shall we run zookeeper quorums on different nodes
>> other
>> > > > than
>> > > > > > >> HDFS?
>> > > > > > >> > > >
>> > > > > > >> > > > Are HLogs generated by Regionservers? Would it
>> alleviate
>> > the
>> > > > > issue
>> > > > > > >> if
>> > > > > > >> > we
>> > > > > > >> > > use
>> > > > > > >> > > > the first 14 for regionserver, and the rest of 5 for
>> > > > > zookeeper?(In
>> > > > > > >> that
>> > > > > > >> > > > sense, the node running zookeeper quorum won't run
>> > > > regionserver)
>> > > > > > >> > > >
>> > > > > > >> > > > I am very appreciated for your comments.
>> > > > > > >> > > >
>> > > > > > >> > > > Best,
>> > > > > > >> > > > zhenyu
>> > > > > > >> > > >
>> > > > > > >> > > >
>> > > > > > >> > > >
>> > > > > > >> > > > On Wed, Nov 11, 2009 at 2:30 PM, Jean-Daniel Cryans <
>> > > > > > >> > jdcryans@apache.org
>> > > > > > >> > > >wrote:
>> > > > > > >> > > >
>> > > > > > >> > > >> The problem seems to be that the IO is saturated. When
>> > you
>> > > > see
>> > > > > > >> this:
>> > > > > > >> > > >>
>> > > > > > >> > > >> #
>> > > > > > >> > > >> 2009-11-11 04:28:10,308 WARN
>> > > > > > >> > > >> org.apache.hadoop.hbase.regionserver.HLog: IPC Server
>> > > handler
>> > > > 9
>> > > > > > on
>> > > > > > >> > > >> 60021 took 6819ms appending an edit to hlog;
>> > editcount=370
>> > > > > > >> > > >> #
>> > > > > > >> > > >> 2009-11-11 04:28:10,308 WARN
>> > > > > > >> > > >> org.apache.hadoop.hbase.regionserver.HLog: IPC Server
>> > > handler
>> > > > 2
>> > > > > > on
>> > > > > > >> > > >> 60021 took 4412ms appending an edit to hlog;
>> > editcount=371
>> > > > > > >> > > >>
>> > > > > > >> > > >> It's a sign that HDFS is slow. I guess that the
>> zookeeper
>> > > > > quorum
>> > > > > > >> > > >> members are on those same nodes using the same disks?
>> If
>> > > so,
>> > > > > > >> there's
>> > > > > > >> > > >> the problem because Zookeeper needs to be able to
>> commit
>> > > > > > >> transactions
>> > > > > > >> > > >> very fast and if your disks are already hogged then
>> they
>> > > have
>> > > > > to
>> > > > > > >> wait
>> > > > > > >> > > >> and you can get disconnections. Do you still have 19
>> > quorum
>> > > > > > >> members?
>> > > > > > >> > > >> How many disks do you have per machine?
>> > > > > > >> > > >>
>> > > > > > >> > > >> Thx
>> > > > > > >> > > >>
>> > > > > > >> > > >> J-D
>> > > > > > >> > > >>
>> > > > > > >> > > >> On Wed, Nov 11, 2009 at 11:22 AM, Zhenyu Zhong <
>> > > > > > >> > zhongresearch@gmail.com
>> > > > > > >> > > >
>> > > > > > >> > > >> wrote:
>> > > > > > >> > > >> > Please take a look at the following regionserver
>> log.
>> > > > > > >> > > >> > The error started to come out from 4:24am UTC to
>> 4:29am
>> > > > UTC.
>> > > > > > >> > > >> >
>> > > > > > >> > > >> > http://pastebin.com/m3717d064
>> > > > > > >> > > >> >
>> > > > > > >> > > >> > Thanks in advance.
>> > > > > > >> > > >> >
>> > > > > > >> > > >> > Best,
>> > > > > > >> > > >> > zhenyu
>> > > > > > >> > > >> >
>> > > > > > >> > > >> >
>> > > > > > >> > > >> > On Wed, Nov 11, 2009 at 2:01 PM, Jean-Daniel Cryans
>> <
>> > > > > > >> > > jdcryans@apache.org
>> > > > > > >> > > >> >wrote:
>> > > > > > >> > > >> >
>> > > > > > >> > > >> >> Could you paste a bigger log trace (interested to
>> see
>> > at
>> > > > > least
>> > > > > > >> all
>> > > > > > >> > > 300
>> > > > > > >> > > >> >> lines before that) in a pastebin?
>> > > > > > >> > > >> >>
>> > > > > > >> > > >> >> Thx,
>> > > > > > >> > > >> >>
>> > > > > > >> > > >> >> J-D
>> > > > > > >> > > >> >>
>> > > > > > >> > > >> >> On Wed, Nov 11, 2009 at 10:56 AM, Zhenyu Zhong <
>> > > > > > >> > > zhongresearch@gmail.com
>> > > > > > >> > > >> >
>> > > > > > >> > > >> >> wrote:
>> > > > > > >> > > >> >> > J-D,
>> > > > > > >> > > >> >> >
>> > > > > > >> > > >> >> > Thanks for the reply.
>> > > > > > >> > > >> >> >
>> > > > > > >> > > >> >> > The problem I have is one or more of the
>> > regionservers
>> > > > > > >> disconnect
>> > > > > > >> > > from
>> > > > > > >> > > >> >> the
>> > > > > > >> > > >> >> > cluster.
>> > > > > > >> > > >> >> >
>> > > > > > >> > > >> >> > I checked the resource usage including memory
>> Heap
>> > > usage
>> > > > > for
>> > > > > > >> > > >> >> regionserver,
>> > > > > > >> > > >> >> > and found nothing suspicious. The regionserver
>> heap
>> > > > usage
>> > > > > is
>> > > > > > >> way
>> > > > > > >> > > below
>> > > > > > >> > > >> >> the
>> > > > > > >> > > >> >> > maxHeap defined in jvm.
>> > > > > > >> > > >> >> >
>> > > > > > >> > > >> >> > So I checked zookeeper log, then I found the
>> > following
>> > > > > > >> > > >> >> > BufferUnderflowException, eventually it leads to
>> > > > > > >> SocketException:
>> > > > > > >> > > >> >> Transport
>> > > > > > >> > > >> >> > endpoint is not connected.
>> > > > > > >> > > >> >> >
>> > > > > > >> > > >> >> >
>> > > > > > >> > > >> >> > I also checked the regionserver log. It looks
>> like
>> > the
>> > > > > > >> > regionserver
>> > > > > > >> > > >> tried
>> > > > > > >> > > >> >> to
>> > > > > > >> > > >> >> > connect to other zookeeper quorum server. It
>> > connects
>> > > > > > >> > successfully,
>> > > > > > >> > > >> but
>> > > > > > >> > > >> >> then
>> > > > > > >> > > >> >> > it complained about SocketException: transport
>> > > endpoint
>> > > > is
>> > > > > > not
>> > > > > > >> > > >> connected.
>> > > > > > >> > > >> >> >
>> > > > > > >> > > >> >> > So both zookeeper log and regionserver log told
>> me
>> > > that
>> > > > > the
>> > > > > > >> > > transport
>> > > > > > >> > > >> >> > endpoint is not connected. I don't know why.
>> > > > > > >> > > >> >> >
>> > > > > > >> > > >> >> > I would be very appreciated if you could point me
>> to
>> > > the
>> > > > > > right
>> > > > > > >> > > >> direction.
>> > > > > > >> > > >> >> >
>> > > > > > >> > > >> >> > Best,
>> > > > > > >> > > >> >> >
>> > > > > > >> > > >> >> >
>> > > > > > >> > > >> >> > Regionserver log:
>> > > > > > >> > > >> >> > 2009-11-11 04:24:53,718 INFO
>> > > > > > org.apache.zookeeper.ClientCnxn:
>> > > > > > >> > > >> Attempting
>> > > > > > >> > > >> >> > connection to server superpyxis0018/
>> > > > 192.168.100.132:2181
>> > > > > > >> > > >> >> > 2009-11-11 04:24:53,718 INFO
>> > > > > > org.apache.zookeeper.ClientCnxn:
>> > > > > > >> > > Priming
>> > > > > > >> > > >> >> > connection to
>> > > java.nio.channels.SocketChannel[connected
>> > > > > > >> local=/
>> > > > > > >> > > >> >> > 192.168.100.131:43065 remote=superpyxis0018/
>> > > > > > >> 192.168.100.132:2181
>> > > > > > >> > ]
>> > > > > > >> > > >> >> > 2009-11-11 04:24:53,718 INFO
>> > > > > > org.apache.zookeeper.ClientCnxn:
>> > > > > > >> > > Server
>> > > > > > >> > > >> >> > connection successful
>> > > > > > >> > > >> >> > 2009-11-11 04:24:53,768 INFO
>> > > > > > >> > > >> >> >
>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>> > > Got
>> > > > > > >> ZooKeeper
>> > > > > > >> > > >> event,
>> > > > > > >> > > >> >> > state: Disconnected, type: None, path: null
>> > > > > > >> > > >> >> > 2009-11-11 04:24:54,118 INFO
>> > > > > > org.apache.zookeeper.ClientCnxn:
>> > > > > > >> > > >> Attempting
>> > > > > > >> > > >> >> > connection to server superpyxis0017/
>> > > > 192.168.100.131:2181
>> > > > > > >> > > >> >> > 2009-11-11 04:24:54,118 INFO
>> > > > > > org.apache.zookeeper.ClientCnxn:
>> > > > > > >> > > Priming
>> > > > > > >> > > >> >> > connection to
>> > > java.nio.channels.SocketChannel[connected
>> > > > > > >> local=/
>> > > > > > >> > > >> >> > 192.168.100.131:46903 remote=superpyxis0017/
>> > > > > > >> 192.168.100.131:2181
>> > > > > > >> > ]
>> > > > > > >> > > >> >> > 2009-11-11 04:24:54,118 INFO
>> > > > > > org.apache.zookeeper.ClientCnxn:
>> > > > > > >> > > Server
>> > > > > > >> > > >> >> > connection successful
>> > > > > > >> > > >> >> > 2009-11-11 04:24:54,118 WARN
>> > > > > > org.apache.zookeeper.ClientCnxn:
>> > > > > > >> > > >> Exception
>> > > > > > >> > > >> >> > closing session 0x24dfd90c810002 to
>> > > > > > >> > > >> sun.nio.ch.SelectionKeyImpl@2dd78ccf
>> > > > > > >> > > >> >> > java.io.IOException: Read error rc = -1
>> > > > > > >> > > >> java.nio.DirectByteBuffer[pos=0
>> > > > > > >> > > >> >> > lim=4 cap=4]
>> > > > > > >> > > >> >> > at
>> > > > > > >> > > >> >> >
>> > > > > > >> > >
>> > > > >
>> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
>> > > > > > >> > > >> >> > at
>> > > > > > >> > > >> >> >
>> > > > > > >> >
>> > > > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
>> > > > > > >> > > >> >> > 2009-11-11 04:24:54,118 WARN
>> > > > > > org.apache.zookeeper.ClientCnxn:
>> > > > > > >> > > Ignoring
>> > > > > > >> > > >> >> > exception during shutdown input
>> > > > > > >> > > >> >> > java.net.SocketException: Transport endpoint is
>> not
>> > > > > > connected
>> > > > > > >> > > >> >> > at
>> > sun.nio.ch.SocketChannelImpl.shutdown(Native
>> > > > > > Method)
>> > > > > > >> > > >> >> > at
>> > > > > > >> > > >> >> >
>> > > > > > >> > >
>> > > > > >
>> > > sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
>> > > > > > >> > > >> >> > at
>> > > > > > >> > > >>
>> > > > sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>> > > > > > >> > > >> >> > at
>> > > > > > >> > > >> >> >
>> > > > > > >> > > >>
>> > > > > > >> >
>> > > > > >
>> > >
>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
>> > > > > > >> > > >> >> > at
>> > > > > > >> > > >> >> >
>> > > > > > >> >
>> > > > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
>> > > > > > >> > > >> >> >
>> > > > > > >> > > >> >> >
>> > > > > > >> > > >> >> >
>> > > > > > >> > > >> >> >
>> > > > > > >> > > >> >> >
>> > > > > > >> > > >> >> > On Wed, Nov 11, 2009 at 1:40 PM, Jean-Daniel
>> Cryans
>> > <
>> > > > > > >> > > >> jdcryans@apache.org
>> > > > > > >> > > >> >> >wrote:
>> > > > > > >> > > >> >> >
>> > > > > > >> > > >> >> >> I'm really not sure exactly what the situation
>> is
>> > > with
>> > > > > the
>> > > > > > >> > > >> >> >> java.nio.BufferUnderflowException, but what do
>> you
>> > > mean
>> > > > > by
>> > > > > > >> > > >> >> >> "regionserver disconnection"? Could you give
>> more
>> > > > > details?
>> > > > > > >> > > >> >> >>
>> > > > > > >> > > >> >> >> Thx,
>> > > > > > >> > > >> >> >>
>> > > > > > >> > > >> >> >> J-D
>> > > > > > >> > > >> >> >>
>> > > > > > >> > > >> >> >> On Wed, Nov 11, 2009 at 10:34 AM, Zhenyu Zhong <
>> > > > > > >> > > >> zhongresearch@gmail.com
>> > > > > > >> > > >> >> >
>> > > > > > >> > > >> >> >> wrote:
>> > > > > > >> > > >> >> >> > I searched around and found HBASE-1645 "HBase
>> > could
>> > > > be
>> > > > > a
>> > > > > > >> > little
>> > > > > > >> > > >> more
>> > > > > > >> > > >> >> >> dainty
>> > > > > > >> > > >> >> >> > when zk is pulled out from under it"
>> > > > > > >> > > >> >> >> >
>> > > > > > >> > > >> >> >> > I am wondering whether my problem is related
>> to
>> > > this
>> > > > > > >> > HBase-1645.
>> > > > > > >> > > Or
>> > > > > > >> > > >> >> >> probably
>> > > > > > >> > > >> >> >> > has nothing to do with it?
>> > > > > > >> > > >> >> >> >
>> > > > > > >> > > >> >> >> > thanks
>> > > > > > >> > > >> >> >> >
>> > > > > > >> > > >> >> >> >
>> > > > > > >> > > >> >> >> > On Wed, Nov 11, 2009 at 12:02 PM, Zhenyu Zhong
>> <
>> > > > > > >> > > >> >> zhongresearch@gmail.com
>> > > > > > >> > > >> >> >> >wrote:
>> > > > > > >> > > >> >> >> >
>> > > > > > >> > > >> >> >> >> Hi,
>> > > > > > >> > > >> >> >> >>
>> > > > > > >> > > >> >> >> >> I have been monitoring the system resources
>> such
>> > > as
>> > > > > > memory
>> > > > > > >> > Heap
>> > > > > > >> > > >> usage
>> > > > > > >> > > >> >> >> for
>> > > > > > >> > > >> >> >> >> Regionserver, load average, network
>> bandwidth,
>> > > etc.
>> > > > > > >> > > >> >> >> >>
>> > > > > > >> > > >> >> >> >> I experienced a disconnection last night.
>> When I
>> > > > > checked
>> > > > > > >> the
>> > > > > > >> > > >> memory
>> > > > > > >> > > >> >> Heap
>> > > > > > >> > > >> >> >> >> for Regionserver, it was 2336 which is way
>> below
>> > > the
>> > > > > max
>> > > > > > >> Heap
>> > > > > > >> > > >> 4096. I
>> > > > > > >> > > >> >> >> don't
>> > > > > > >> > > >> >> >> >> think it is Regionserver problem. The
>> requests
>> > > that
>> > > > > the
>> > > > > > >> > > >> regionserver
>> > > > > > >> > > >> >> got
>> > > > > > >> > > >> >> >> was
>> > > > > > >> > > >> >> >> >> about 126 which doesn't seem to be high. The
>> > load
>> > > > > > average
>> > > > > > >> is
>> > > > > > >> > > kind
>> > > > > > >> > > >> of
>> > > > > > >> > > >> >> >> high
>> > > > > > >> > > >> >> >> >> >100. (but I saw disconnections while the
>> load
>> > > > average
>> > > > > > is
>> > > > > > >> > > around
>> > > > > > >> > > >> 30).
>> > > > > > >> > > >> >> >> The
>> > > > > > >> > > >> >> >> >> bandwidth is not a problem.
>> > > > > > >> > > >> >> >> >>
>> > > > > > >> > > >> >> >> >> However, I found error messages from the
>> > > zookeeper.
>> > > > > > Please
>> > > > > > >> > see
>> > > > > > >> > > the
>> > > > > > >> > > >> >> >> >> following error messages. There are some
>> > > > > > >> > > >> >> >> >> BufferUnderflowException error, IOException
>> > > > error,etc.
>> > > > > > >> > > >> >> >> >> May I ask what might be the causes that
>> generate
>> > > > these
>> > > > > > >> error
>> > > > > > >> > > >> >> messages?
>> > > > > > >> > > >> >> >> >>
>> > > > > > >> > > >> >> >> >> Many thanks
>> > > > > > >> > > >> >> >> >> zhenyu
>> > > > > > >> > > >> >> >> >>
>> > > > > > >> > > >> >> >> >>
>> > > > > > >> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
>> > > > > > >> > > org.apache.zookeeper.server.Request:
>> > > > > > >> > > >> >> >> Ignoring
>> > > > > > >> > > >> >> >> >> exception during toString
>> > > > > > >> > > >> >> >> >> java.nio.BufferUnderflowException
>> > > > > > >> > > >> >> >> >> at
>> > > > > > >> > java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
>> > > > > > >> > > >> >> >> >> at
>> > > > > java.nio.ByteBuffer.get(ByteBuffer.java:675)
>> > > > > > >> > > >> >> >> >> at
>> > > > > > >> > > >> >> >>
>> > > > > > >>
>> org.apache.zookeeper.server.Request.toString(Request.java:199)
>> > > > > > >> > > >> >> >> >> at
>> > > > java.lang.String.valueOf(String.java:2827)
>> > > > > > >> > > >> >> >> >> at
>> > > > > > >> > > java.lang.StringBuilder.append(StringBuilder.java:115)
>> > > > > > >> > > >> >> >> >> at
>> > > > > > >> > > >> >> >> >>
>> > > > > > >> > > >> >> >>
>> > > > > > >> > > >> >>
>> > > > > > >> > > >>
>> > > > > > >> > >
>> > > > > > >> >
>> > > > > > >>
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.apache.zookeeper.server.quorum.CommitProcessor.processRequest(CommitProcessor.java:167)
>> > > > > > >> > > >> >> >> >> at
>> > > > > > >> > > >> >> >> >>
>> > > > > > >> > > >> >> >>
>> > > > > > >> > > >> >>
>> > > > > > >> > > >>
>> > > > > > >> > >
>> > > > > > >> >
>> > > > > > >>
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.apache.zookeeper.server.quorum.FollowerRequestProcessor.run(FollowerRequestProcessor.java:68)
>> > > > > > >> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
>> > > > > > >> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
>> > > > > > >> > > >> >> >> >> Exception causing close of session
>> > > 0x424dfd908c50009
>> > > > > > >> > > >> >> >> >> due to java.io.IOException: Read error
>> > > > > > >> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
>> > > > > > >> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
>> > > > > > >> > > >> >> >> >> IOException stack trace
>> > > > > > >> > > >> >> >> >> java.io.IOException: Read error
>> > > > > > >> > > >> >> >> >> at
>> > > > > > >> > > >> >> >> >>
>> > > > > > >> > > >> >>
>> > > > > > >> > >
>> > > > > >
>> > > org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:372)
>> > > > > > >> > > >> >> >> >> at
>> > > > > > >> > > >> >> >> >>
>> > > > > > >> > > >> >> >>
>> > > > > > >> > > >> >>
>> > > > > > >> > > >>
>> > > > > > >> > >
>> > > > > > >> >
>> > > > > > >>
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:221)
>> > > > > > >> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
>> > > > > > >> > > >> >> >> >>
>> > > org.apache.zookeeper.server.quorum.CommitProcessor:
>> > > > > > >> > Processing
>> > > > > > >> > > >> >> request::
>> > > > > > >> > > >> >> >> >> sessionid:0x424dfd908c50009 type:setWatches
>> > > > > > >> > > >> cxid:0xfffffffffffffff8
>> > > > > > >> > > >> >> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
>> > > > > > >> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
>> > > > > > >> > > org.apache.zookeeper.server.Request:
>> > > > > > >> > > >> >> >> Ignoring
>> > > > > > >> > > >> >> >> >> exception during toString
>> > > > > > >> > > >> >> >> >> java.nio.BufferUnderflowException
>> > > > > > >> > > >> >> >> >> at
>> > > > > > >> > java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
>> > > > > > >> > > >> >> >> >> at
>> > > > > java.nio.ByteBuffer.get(ByteBuffer.java:675)
>> > > > > > >> > > >> >> >> >> at
>> > > > > > >> > > >> >> >>
>> > > > > > >>
>> org.apache.zookeeper.server.Request.toString(Request.java:199)
>> > > > > > >> > > >> >> >> >> at
>> > > > java.lang.String.valueOf(String.java:2827)
>> > > > > > >> > > >> >> >> >> at
>> > > > > > >> > > java.lang.StringBuilder.append(StringBuilder.java:115)
>> > > > > > >> > > >> >> >> >> at
>> > > > > > >> > > >> >> >> >>
>> > > > > > >> > > >> >> >>
>> > > > > > >> > > >> >>
>> > > > > > >> > > >>
>> > > > > > >> > >
>> > > > > > >> >
>> > > > > > >>
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:74)
>> > > > > > >> > > >> >> >> >> at
>> > > > > > >> > > >> >> >> >>
>> > > > > > >> > > >> >> >>
>> > > > > > >> > > >> >>
>> > > > > > >> > > >>
>> > > > > > >> > >
>> > > > > > >> >
>> > > > > > >>
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)
>> > > > > > >> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
>> > > > > > >> > > >> >> >> >>
>> > org.apache.zookeeper.server.FinalRequestProcessor:
>> > > > > > >> Processing
>> > > > > > >> > > >> >> request::
>> > > > > > >> > > >> >> >> >> sessionid:0x424dfd908c50009 type:setWatches
>> > > > > > >> > > >> cxid:0xfffffffffffffff8
>> > > > > > >> > > >> >> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
>> > > > > > >> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
>> > > > > > >> > > org.apache.zookeeper.server.Request:
>> > > > > > >> > > >> >> >> Ignoring
>> > > > > > >> > > >> >> >> >> exception during toString
>> > > > > > >> > > >> >> >> >> java.nio.BufferUnderflowException
>> > > > > > >> > > >> >> >> >> at
>> > > > > > >> > java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
>> > > > > > >> > > >> >> >> >> at
>> > > > > java.nio.ByteBuffer.get(ByteBuffer.java:675)
>> > > > > > >> > > >> >> >> >> at
>> > > > > > >> > > >> >> >>
>> > > > > > >>
>> org.apache.zookeeper.server.Request.toString(Request.java:199)
>> > > > > > >> > > >> >> >> >> at
>> > > > > > >> > > >> >> >> >>
>> > > > > > >> > > >>
>> > > > > > >>
>> > > >
>> org.apache.log4j.or.DefaultRenderer.doRender(DefaultRenderer.java:36)
>> > > > > > >> > > >> >> >> >> at
>> > > > > > >> > > >> >> >> >>
>> > > > > > >> > >
>> > > > org.apache.log4j.or.RendererMap.findAndRender(RendererMap.java:80)
>> > > > > > >> > > >> >> >> >> at
>> > > > > > >> > > >> >> >> >>
>> > > > > > >> > > >> >> >>
>> > > > > > >> > > >> >>
>> > > > > > >> > > >>
>> > > > > > >> > >
>> > > > > > >> >
>> > > > > > >>
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.apache.log4j.spi.LoggingEvent.getRenderedMessage(LoggingEvent.java:362)
>> > > > > > >> > > >> >> >> >> at
>> > > > > > >> > > >> >> >> >>
>> > > > > > >> > > >> >> >>
>> > > > > > >> > > >> >>
>> > > > > > >> > > >>
>> > > > > > >> > >
>> > > > > > >> >
>> > > > > > >>
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.apache.log4j.helpers.PatternParser$BasicPatternConverter.convert(PatternParser.java:403)
>> > > > > > >> > > >> >> >> >> at
>> > > > > > >> > > >> >> >> >>
>> > > > > > >> > > >> >> >>
>> > > > > > >> > > >> >>
>> > > > > > >> > > >>
>> > > > > > >> > >
>> > > > > > >> >
>> > > > > > >>
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.apache.log4j.helpers.PatternConverter.format(PatternConverter.java:65)
>> > > > > > >> > > >> >> >> >> at
>> > > > > > >> > > >> >>
>> > > > > org.apache.log4j.PatternLayout.format(PatternLayout.java:502)
>> > > > > > >> > > >> >> >> >> at
>> > > > > > >> > > >> >> >> >>
>> > > > > > >> > >
>> > > > org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:302)
>> > > > > > >> > > >> >> >> >> at
>> > > > > > >> > > >> >> >> >>
>> > > > > > >> > > >> >> >>
>> > > > > > >> > > >> >>
>> > > > > > >> > > >>
>> > > > > > >> > >
>> > > > > > >> >
>> > > > > > >>
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.apache.log4j.DailyRollingFileAppender.subAppend(DailyRollingFileAppender.java:359)
>> > > > > > >> > > >> >> >> >> at
>> > > > > > >> > > >> >> >>
>> > > > > > >>
>> org.apache.log4j.WriterAppender.append(WriterAppender.java:160)
>> > > > > > >> > > >> >> >> >> at
>> > > > > > >> > > >> >> >> >>
>> > > > > > >> > > >>
>> > > > > > >>
>> > > >
>> org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
>> > > > > > >> > > >> >> >> >> at
>> > > > > > >> > > >> >> >> >>
>> > > > > > >> > > >> >> >>
>> > > > > > >> > > >> >>
>> > > > > > >> > > >>
>> > > > > > >> > >
>> > > > > > >> >
>> > > > > > >>
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
>> > > > > > >> > > >> >> >> >> at
>> > > > > > >> > > >>
>> > org.apache.log4j.Category.callAppenders(Category.java:206)
>> > > > > > >> > > >> >> >> >> at
>> > > > > > >> > > org.apache.log4j.Category.forcedLog(Category.java:391)
>> > > > > > >> > > >> >> >> >> at
>> > > > > > >> org.apache.log4j.Category.debug(Category.java:260)
>> > > > > > >> > > >> >> >> >> at
>> > > > > > >> > > >> >> >> >>
>> > > > > > >> > > >> >> >>
>> > > > > > >> > > >> >>
>> > > > > > >> > > >>
>> > > > > > >> > >
>> > > > > > >> >
>> > > > > > >>
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:147)
>> > > > > > >> > > >> >> >> >> at
>> > > > > > >> > > >> >> >> >>
>> > > > > > >> > > >> >> >>
>> > > > > > >> > > >> >>
>> > > > > > >> > > >>
>> > > > > > >> > >
>> > > > > > >> >
>> > > > > > >>
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)
>> > > > > > >> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
>> > > > > > >> > > org.apache.zookeeper.server.Request:
>> > > > > > >> > > >> >> >> Ignoring
>> > > > > > >> > > >> >> >> >> exception during toString
>> > > > > > >> > > >> >> >> >> sessionid:0x424dfd908c50009 type:setWatches
>> > > > > > >> > > >> cxid:0xfffffffffffffff8
>> > > > > > >> > > >> >> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
>> > > > > > >> > > >> >> >> >> 2009-11-11 04:29:00,810 INFO
>> > > > > > >> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
>> > > > > > >> > > >> >> >> >> closing session:0x424dfd908c50009
>> NIOServerCnxn:
>> > > > > > >> > > >> >> >> >> java.nio.channels.SocketChannel[connected
>> > local=/
>> > > > > > >> > > >> >> 192.168.100.131:2181
>> > > > > > >> > > >> >> >> remote=/
>> > > > > > >> > > >> >> >> >> 192.168.100.131:35500]
>> > > > > > >> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
>> > > > > > >> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
>> > > > > > >> > > >> >> >> >> ignoring exception during input shutdown
>> > > > > > >> > > >> >> >> >> java.net.SocketException: Transport endpoint
>> is
>> > > not
>> > > > > > >> connected
>> > > > > > >> > > >> >> >> >> at
>> > > > > sun.nio.ch.SocketChannelImpl.shutdown(Native
>> > > > > > >> > Method)
>> > > > > > >> > > >> >> >> >> at
>> > > > > > >> > > >> >> >> >>
>> > > > > > >> > > >> >>
>> > > > > > >> > >
>> > > > > >
>> > > sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
>> > > > > > >> > > >> >> >> >> at
>> > > > > > >> > > >> >> >>
>> > > > > > >>
>> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>> > > > > > >> > > >> >> >> >> at
>> > > > > > >> > > >> >> >> >>
>> > > > > > >> > > >> >>
>> > > > > > >> > >
>> > > > > > >>
>> > > > >
>> > org.apache.zookeeper.server.NIOServerCnxn.close(NIOServerCnxn.java:848)
>> > > > > > >> > > >> >> >> >> at
>> > > > > > >> > > >> >> >> >>
>> > > > > > >> > > >> >>
>> > > > > > >> > >
>> > > > > >
>> > > org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:498)
>> > > > > > >> > > >> >> >> >> at
>> > > > > > >> > > >> >> >> >>
>> > > > > > >> > > >> >> >>
>> > > > > > >> > > >> >>
>> > > > > > >> > > >>
>> > > > > > >> > >
>> > > > > > >> >
>> > > > > > >>
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:221)
>> > > > > > >> > > >> >> >> >>
>> > > > > > >> > > >> >> >> >
>> > > > > > >> > > >> >> >>
>> > > > > > >> > > >> >> >
>> > > > > > >> > > >> >>
>> > > > > > >> > > >> >
>> > > > > > >> > > >>
>> > > > > > >> > > >
>> > > > > > >> > >
>> > > > > > >> >
>> > > > > > >>
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
>
Re: regionserver disconnection
Posted by Zhenyu Zhong <zh...@gmail.com>.
Will do.
thanks
zhenyu
On Thu, Nov 12, 2009 at 3:33 PM, stack <st...@duboce.net> wrote:
> Enable GC logging too on this next run (see hbase-env.sh). Lets try and
> get
> to the bottom of whats going on.
> Thanks,
> St.Ack
>
> On Thu, Nov 12, 2009 at 12:29 PM, Zhenyu Zhong <zhongresearch@gmail.com
> >wrote:
>
> > I can switch the boxes that run zookeeper with the ones that run
> > regionservers.
> > I will see the results later.
> >
> > FYI. The node does have the 10 minutes zookeeper.session.timeout value in
> > place.
> >
> > thanks
> > zhenyu
> >
> > On Thu, Nov 12, 2009 at 3:21 PM, stack <st...@duboce.net> wrote:
> >
> > > On Thu, Nov 12, 2009 at 11:50 AM, Zhenyu Zhong <
> zhongresearch@gmail.com
> > > >wrote:
> > >
> > > >
> > > > In my cluster, half of the cluster have 2 disks 400GB each per
> machine,
> > > and
> > > > half of the cluster have 6 disks per machine. Maybe we should run
> > > > zookeeper
> > > > on the machines with 2 disks and RS on machines with 6 disks?
> > > >
> > > > That would make most sense only in the below, it looks like the RS
> that
> > > had
> > > issue had 4 disks?
> > >
> > >
> > >
> > > > BTW, the 10 minutes zookeeper.session.timeout has been set during the
> > > > experiment.
> > > >
> > >
> > > And for sure this node had it in place?
> > > St.Ack
> > >
> > >
> > >
> > > >
> > > > thanks
> > > > zhenyu
> > > >
> > > > On Thu, Nov 12, 2009 at 2:08 PM, stack <st...@duboce.net> wrote:
> > > >
> > > > > On Thu, Nov 12, 2009 at 8:40 AM, Zhenyu Zhong <
> > zhongresearch@gmail.com
> > > > > >wrote:
> > > > >
> > > > >
> > > > > > Though I experienced 2 regionserver disconnection this morning,
> it
> > > > looks
> > > > > > better from the regionserver log. (please see the following log)
> > > > > >
> > > > > >
> > > > > > http://pastebin.com/m496dbfae
> > > > > >
> > > > > >
> > > > > > I checked diskIO for one of the regionserver(192.168.100.116)
> > during
> > > > the
> > > > > > time it disconnected.
> > > > > >
> > > > > > Time: 03:04:58 AM
> > > > > > Device: tps Blk_read/s Blk_wrtn/s Blk_read
> > > Blk_wrtn
> > > > > > sda 105.31 5458.83 19503.64 9043873239
> > > 32312473676
> > > > > > sda1 2.90 6.64 99.25 10993934
> > > 164433464
> > > > > > sda2 1.72 23.76 30.25 39365817
> > > 50124008
> > > > > > sda3 0.30 0.38 3.58 624930
> > > 5923000
> > > > > > sda4 100.39 5428.06 19370.56 8992888270
> > > 32091993204
> > > > > >
> > > > > >
> > > > > Is this high for you? 20k blocks/second would seem to be high but
> > its
> > > > one
> > > > > disk only and its not being shared by zk anymore so shouldn't
> matter?
> > > > >
> > > > >
> > > > > > I also checked the zookeeper quorum server that the regionserver
> > > tried
> > > > to
> > > > > > connect according to the log. However, I don't see
> 192.168.100.116
> > in
> > > > the
> > > > > > client list of the zookeeper quorum server that the regionserver
> > > tried
> > > > to
> > > > > > connect.
> > > > > > Would that be a problem?
> > > > > >
> > > > > >
> > > > > Is that because the ephemeral node for the regionserver had
> > evaporated?
> > > > > Lost
> > > > > its lease w/ zk by the time you went to look?
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > > Thu Nov 12 15:04:52 UTC 2009
> > > > > > Zookeeper version: 3.2.1-808558, built on 08/27/2009 18:48 GMT
> > > > > > Clients:
> > > > > > /192.168.100.127:43045[1](queued=0,recved=26,sent=0)
> > > > > > /192.168.100.131:39091[1](queued=0,recved=964,sent=0)
> > > > > > /192.168.100.124:35961[1](queued=0,recved=3266,sent=0)
> > > > > > /192.168.100.123:47935[1](queued=0,recved=233,sent=0)
> > > > > > /192.168.100.125:46931[1](queued=0,recved=2,sent=0)
> > > > > > /192.168.100.118:54924[1](queued=0,recved=295,sent=0)
> > > > > > /192.168.100.118:41390[1](queued=0,recved=2290,sent=0)
> > > > > > /192.168.100.136:42243[1](queued=0,recved=0,sent=0)
> > > > > >
> > > > > > Latency min/avg/max: 0/17/6333
> > > > > > Received: 47111
> > > > > > Sent: 0
> > > > > > Outstanding: 0
> > > > > > Zxid: 0x77000083f4
> > > > > > Mode: leader
> > > > > > Node count: 23
> > > > > >
> > > > >
> > > > >
> > > > > That 6 second maximum latency is pretty bad but should be well
> within
> > > the
> > > > > zk
> > > > > session timeout.
> > > > >
> > > > > So, problem is likely on the zk client side of the session; i.e. in
> > the
> > > > RS.
> > > > > You could enable GC logging as J-D suggested to see if you have any
> > big
> > > > > pauses, pauses > zk session timeout.
> > > > >
> > > > > When the RS went down, it didn't look too heavily loaded:
> > > > >
> > > > >
> > > > > 1. 2009-11-12 15:04:52,830 INFO
> > > > > org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of
> > metrics:
> > > > > request=1.5166667, regions=322, stores=657, storefiles=631,
> > > > > storefileIndexSize=61, memstoreSize=1472, usedHeap=2819,
> > > maxHeap=4079,
> > > > > blockCacheSize=658110960, blockCacheFree=197395984,
> > > > blockCacheCount=9903,
> > > > > blockCacheHitRatio=99
> > > > >
> > > > > Its serving a few reads? The number of store files seems fine.
> Not
> > > too
> > > > > much memory used.
> > > > >
> > > > > Looking at the logs, I see the Lease Still Held exception. This
> > > happens
> > > > > when the RS does its regular report to the master but the master
> > thinks
> > > > the
> > > > > RS has since restarted. It'll think this probably because it
> noticed
> > > > that
> > > > > the RS's znode in zk had gone away and it considered the RS dead.
> > > > >
> > > > > Looking too at your logs I see this gap in the zk pinging:
> > > > >
> > > > >
> > > > > 1. 2009-11-12 15:03:39,325 DEBUG org.apache.zookeeper.ClientCnxn:
> > Got
> > > > > ping response for sessionid:0x224e55436ad0004 after 0ms
> > > > > 2. 2009-11-12 15:03:43,113 DEBUG org.apache.zookeeper.ClientCnxn:
> > Got
> > > > > ping response for sessionid:0x24e55436a0007d after 0ms
> > > > >
> > > > > Where in the lines above it, its reporting about every ten seconds,
> > > here
> > > > > there is a big gap.
> > > > >
> > > > > Do you have ganglia or something that will let you look more into
> > what
> > > > was
> > > > > happening on this machine around this time? Is the machine OK? It
> > > looks
> > > > > lightly loaded and you have your cluster nicely laid out.
> Something
> > > odd
> > > > is
> > > > > going on. What about things like the write speed to disk? In the
> > past
> > > > > strange issues have been explained by incorrectly set BIOS which
> made
> > > > disks
> > > > > run at 1/100th of their proper speed.
> > > > >
> > > > > St.Ack
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > >
> > > > > > Best,
> > > > > > zhenyu
> > > > > >
> > > > > >
> > > > > > On Wed, Nov 11, 2009 at 3:58 PM, Zhenyu Zhong <
> > > zhongresearch@gmail.com
> > > > > > >wrote:
> > > > > >
> > > > > > > Stack
> > > > > > >
> > > > > > > I am very appreciated for your comments.
> > > > > > > I will use the zookeeper monitoring script on my cluster and
> let
> > it
> > > > run
> > > > > > > overnight to see the result.
> > > > > > > I will follow up the thread when I get anything.
> > > > > > >
> > > > > > > thanks
> > > > > > > zhenyu
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Nov 11, 2009 at 3:52 PM, stack <st...@duboce.net>
> wrote:
> > > > > > >
> > > > > > >> I see these in your log too:
> > > > > > >>
> > > > > > >>
> > > > > > >> 1. 2009-11-11 04:27:19,018 DEBUG
> > > org.apache.zookeeper.ClientCnxn:
> > > > > Got
> > > > > > >> ping response for sessionid:0x424dfd908c50009 after 4544ms
> > > > > > >> 2. 2009-11-11 04:27:19,018 DEBUG
> > > org.apache.zookeeper.ClientCnxn:
> > > > > Got
> > > > > > >> ping response for sessionid:0x24dfd90c810002 after 4548ms
> > > > > > >> 3. 2009-11-11 04:27:43,960 DEBUG
> > > org.apache.zookeeper.ClientCnxn:
> > > > > Got
> > > > > > >> ping response for sessionid:0x424dfd908c50009 after 9030ms
> > > > > > >> 4. 2009-11-11 04:27:43,960 DEBUG
> > > org.apache.zookeeper.ClientCnxn:
> > > > > Got
> > > > > > >> ping response for sessionid:0x24dfd90c810002 after 9030ms
> > > > > > >> 5. 2009-11-11 04:27:48,830 DEBUG
> > > org.apache.zookeeper.ClientCnxn:
> > > > > Got
> > > > > > >> ping response for sessionid:0x424dfd908c50009 after 0ms
> > > > > > >> 6. 2009-11-11 04:27:48,830 DEBUG
> > > org.apache.zookeeper.ClientCnxn:
> > > > > Got
> > > > > > >> ping response for sessionid:0x24dfd90c810002 after 0ms
> > > > > > >>
> > > > > > >> Which would indicate lag talking to zk too.
> > > > > > >>
> > > > > > >> Appending HLog also seems to get steadily worse.
> > > > > > >>
> > > > > > >> I also see these:
> > > > > > >>
> > > > > > >>
> > > > > > >> 1. 2009-11-11 04:27:01,233 WARN
> > org.apache.zookeeper.ClientCnxn:
> > > > > > >> Exception closing session 0x24dfd90c810002 to
> > > > > > >> sun.nio.ch.SelectionKeyImpl@5e16a7e3
> > > > > > >> 2. java.io.IOException: Read error rc = -1
> > > > > > >> java.nio.DirectByteBuffer[pos=0 lim=4 cap=4]
> > > > > > >> 3. at
> > > > > > >>
> > > > >
> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
> > > > > > >> 4. at
> > > > > > >>
> > > > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
> > > > > > >>
> > > > > > >>
> > > > > > >> Were all 5 quorum members participating? Check by doing "echo
> > > > stat|nc
> > > > > > >> QUORUM_HOST port" for each member of quorum.... the below
> might
> > > > help:
> > > > > > >>
> > > > > > >> members="1,2,3,4,5"
> > > > > > >> while true; do date; for i in $members; do echo $i; echo stat
> |
> > nc
> > > > $i
> > > > > > >> 2181;
> > > > > > >> done; sleep 10; done
> > > > > > >>
> > > > > > >> We need to improve our zk_dump. It just reports quorum
> > members...
> > > > not
> > > > > > if
> > > > > > >> they are actually participating or not. The above will tell
> you
> > > if
> > > > > dead
> > > > > > >> or
> > > > > > >> alive.
> > > > > > >>
> > > > > > >> Please report back after checking above and following the
> > > > Jean-Daniel
> > > > > > >> prescription.
> > > > > > >>
> > > > > > >> Thanks,
> > > > > > >> St.Ack
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >> On Wed, Nov 11, 2009 at 12:29 PM, Zhenyu Zhong <
> > > > > zhongresearch@gmail.com
> > > > > > >> >wrote:
> > > > > > >>
> > > > > > >> > Thanks a lot for the help.
> > > > > > >> >
> > > > > > >> > I understand that the change of disk configurations should
> be
> > > done
> > > > > > >> > progressively such that HDFS can balance and recover from a
> > disk
> > > > > going
> > > > > > >> > away.
> > > > > > >> >
> > > > > > >> > The data we are trying to import into HBase is about
> > 200million
> > > > > rows,
> > > > > > >> each
> > > > > > >> > row is about hundreds of bytes. Basically, we read in a text
> > > file
> > > > > from
> > > > > > >> > HDFS,
> > > > > > >> > then for each row in the text file, we fetch data from
> HBase
> > > > table,
> > > > > > we
> > > > > > >> > compare the column value and update the latest value back
> into
> > > > > HBase.
> > > > > > I
> > > > > > >> was
> > > > > > >> > suggested to use CheckAndPut. Anyway this is the common
> > usecase
> > > > on
> > > > > my
> > > > > > >> > side.
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > Many thanks
> > > > > > >> > zhenyu
> > > > > > >> >
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > On Wed, Nov 11, 2009 at 3:03 PM, Jean-Daniel Cryans <
> > > > > > >> jdcryans@apache.org
> > > > > > >> > >wrote:
> > > > > > >> >
> > > > > > >> > > 6 disks, 19 machines and you see those kind of issues? I'm
> > > quite
> > > > > > >> > > surprised and I want to learn more about what you trying
> to
> > > to.
> > > > > > >> > >
> > > > > > >> > > But first, if hbase is managing ZK for you then you can
> > change
> > > > the
> > > > > > >> > > value of hbase.zookeeper.property.dataDir so that it
> points
> > to
> > > > its
> > > > > > own
> > > > > > >> > > mount. Make sure you also change dfs.data.dir in
> > > > > conf/hdfs-site.xml
> > > > > > >> > > (hadoop stuff) so that it stops pointing there (you should
> > > have
> > > > 6
> > > > > > >> > > values there at the moment if you configured it
> correctly).
> > > > Doing
> > > > > > this
> > > > > > >> > > could result in data loss if you do it all at once so do
> the
> > > > > change
> > > > > > >> > > progressively. This also means that you may have to keep a
> > > > > seperate
> > > > > > >> > > HDFS configuration for those particular nodes. Please ask
> > more
> > > > > > >> > > questions if it's not very clear.
> > > > > > >> > >
> > > > > > >> > > WRT to dedicating nodes to Zookeeeper, it makes sense only
> > if
> > > > you
> > > > > > have
> > > > > > >> > > a lot of nodes but in your case, unless there's also CPU
> > > > > contention,
> > > > > > >> > > you're good with only a dedicated disk.
> > > > > > >> > >
> > > > > > >> > > Now for what you are trying to do, is this a massive
> import
> > > that
> > > > > > goes
> > > > > > >> > > over many hours? How big is each row? I'd love to get a
> lot
> > of
> > > > > > details
> > > > > > >> > > to help you optimize your cluster for your particular
> > problem.
> > > > > > >> > >
> > > > > > >> > > Thx,
> > > > > > >> > >
> > > > > > >> > > J-D
> > > > > > >> > >
> > > > > > >> > > On Wed, Nov 11, 2009 at 11:51 AM, Zhenyu Zhong <
> > > > > > >> zhongresearch@gmail.com>
> > > > > > >> > > wrote:
> > > > > > >> > > > J-D,
> > > > > > >> > > >
> > > > > > >> > > > Thanks a lot for the insightful comments.
> > > > > > >> > > > We are running 5 zookeeper quorums now. On these 5
> quorum
> > > > > servers,
> > > > > > >> we
> > > > > > >> > > also
> > > > > > >> > > > run datanode, tasktracker and regionserver.
> > > > > > >> > > > Now we have 6 disks per machine.
> > > > > > >> > > > If it is the disk IO saturation, is there anyway to
> > > configure
> > > > > > >> zookeeper
> > > > > > >> > > to
> > > > > > >> > > > write to a seperate disk which is not the same one used
> by
> > > > HDFS?
> > > > > > >> > > > Or shall we run zookeeper quorums on different nodes
> other
> > > > than
> > > > > > >> HDFS?
> > > > > > >> > > >
> > > > > > >> > > > Are HLogs generated by Regionservers? Would it alleviate
> > the
> > > > > issue
> > > > > > >> if
> > > > > > >> > we
> > > > > > >> > > use
> > > > > > >> > > > the first 14 for regionserver, and the rest of 5 for
> > > > > zookeeper?(In
> > > > > > >> that
> > > > > > >> > > > sense, the node running zookeeper quorum won't run
> > > > regionserver)
> > > > > > >> > > >
> > > > > > >> > > > I am very appreciated for your comments.
> > > > > > >> > > >
> > > > > > >> > > > Best,
> > > > > > >> > > > zhenyu
> > > > > > >> > > >
> > > > > > >> > > >
> > > > > > >> > > >
> > > > > > >> > > > On Wed, Nov 11, 2009 at 2:30 PM, Jean-Daniel Cryans <
> > > > > > >> > jdcryans@apache.org
> > > > > > >> > > >wrote:
> > > > > > >> > > >
> > > > > > >> > > >> The problem seems to be that the IO is saturated. When
> > you
> > > > see
> > > > > > >> this:
> > > > > > >> > > >>
> > > > > > >> > > >> #
> > > > > > >> > > >> 2009-11-11 04:28:10,308 WARN
> > > > > > >> > > >> org.apache.hadoop.hbase.regionserver.HLog: IPC Server
> > > handler
> > > > 9
> > > > > > on
> > > > > > >> > > >> 60021 took 6819ms appending an edit to hlog;
> > editcount=370
> > > > > > >> > > >> #
> > > > > > >> > > >> 2009-11-11 04:28:10,308 WARN
> > > > > > >> > > >> org.apache.hadoop.hbase.regionserver.HLog: IPC Server
> > > handler
> > > > 2
> > > > > > on
> > > > > > >> > > >> 60021 took 4412ms appending an edit to hlog;
> > editcount=371
> > > > > > >> > > >>
> > > > > > >> > > >> It's a sign that HDFS is slow. I guess that the
> zookeeper
> > > > > quorum
> > > > > > >> > > >> members are on those same nodes using the same disks?
> If
> > > so,
> > > > > > >> there's
> > > > > > >> > > >> the problem because Zookeeper needs to be able to
> commit
> > > > > > >> transactions
> > > > > > >> > > >> very fast and if your disks are already hogged then
> they
> > > have
> > > > > to
> > > > > > >> wait
> > > > > > >> > > >> and you can get disconnections. Do you still have 19
> > quorum
> > > > > > >> members?
> > > > > > >> > > >> How many disks do you have per machine?
> > > > > > >> > > >>
> > > > > > >> > > >> Thx
> > > > > > >> > > >>
> > > > > > >> > > >> J-D
> > > > > > >> > > >>
> > > > > > >> > > >> On Wed, Nov 11, 2009 at 11:22 AM, Zhenyu Zhong <
> > > > > > >> > zhongresearch@gmail.com
> > > > > > >> > > >
> > > > > > >> > > >> wrote:
> > > > > > >> > > >> > Please take a look at the following regionserver log.
> > > > > > >> > > >> > The error started to come out from 4:24am UTC to
> 4:29am
> > > > UTC.
> > > > > > >> > > >> >
> > > > > > >> > > >> > http://pastebin.com/m3717d064
> > > > > > >> > > >> >
> > > > > > >> > > >> > Thanks in advance.
> > > > > > >> > > >> >
> > > > > > >> > > >> > Best,
> > > > > > >> > > >> > zhenyu
> > > > > > >> > > >> >
> > > > > > >> > > >> >
> > > > > > >> > > >> > On Wed, Nov 11, 2009 at 2:01 PM, Jean-Daniel Cryans <
> > > > > > >> > > jdcryans@apache.org
> > > > > > >> > > >> >wrote:
> > > > > > >> > > >> >
> > > > > > >> > > >> >> Could you paste a bigger log trace (interested to
> see
> > at
> > > > > least
> > > > > > >> all
> > > > > > >> > > 300
> > > > > > >> > > >> >> lines before that) in a pastebin?
> > > > > > >> > > >> >>
> > > > > > >> > > >> >> Thx,
> > > > > > >> > > >> >>
> > > > > > >> > > >> >> J-D
> > > > > > >> > > >> >>
> > > > > > >> > > >> >> On Wed, Nov 11, 2009 at 10:56 AM, Zhenyu Zhong <
> > > > > > >> > > zhongresearch@gmail.com
> > > > > > >> > > >> >
> > > > > > >> > > >> >> wrote:
> > > > > > >> > > >> >> > J-D,
> > > > > > >> > > >> >> >
> > > > > > >> > > >> >> > Thanks for the reply.
> > > > > > >> > > >> >> >
> > > > > > >> > > >> >> > The problem I have is one or more of the
> > regionservers
> > > > > > >> disconnect
> > > > > > >> > > from
> > > > > > >> > > >> >> the
> > > > > > >> > > >> >> > cluster.
> > > > > > >> > > >> >> >
> > > > > > >> > > >> >> > I checked the resource usage including memory Heap
> > > usage
> > > > > for
> > > > > > >> > > >> >> regionserver,
> > > > > > >> > > >> >> > and found nothing suspicious. The regionserver
> heap
> > > > usage
> > > > > is
> > > > > > >> way
> > > > > > >> > > below
> > > > > > >> > > >> >> the
> > > > > > >> > > >> >> > maxHeap defined in jvm.
> > > > > > >> > > >> >> >
> > > > > > >> > > >> >> > So I checked zookeeper log, then I found the
> > following
> > > > > > >> > > >> >> > BufferUnderflowException, eventually it leads to
> > > > > > >> SocketException:
> > > > > > >> > > >> >> Transport
> > > > > > >> > > >> >> > endpoint is not connected.
> > > > > > >> > > >> >> >
> > > > > > >> > > >> >> >
> > > > > > >> > > >> >> > I also checked the regionserver log. It looks like
> > the
> > > > > > >> > regionserver
> > > > > > >> > > >> tried
> > > > > > >> > > >> >> to
> > > > > > >> > > >> >> > connect to other zookeeper quorum server. It
> > connects
> > > > > > >> > successfully,
> > > > > > >> > > >> but
> > > > > > >> > > >> >> then
> > > > > > >> > > >> >> > it complained about SocketException: transport
> > > endpoint
> > > > is
> > > > > > not
> > > > > > >> > > >> connected.
> > > > > > >> > > >> >> >
> > > > > > >> > > >> >> > So both zookeeper log and regionserver log told me
> > > that
> > > > > the
> > > > > > >> > > transport
> > > > > > >> > > >> >> > endpoint is not connected. I don't know why.
> > > > > > >> > > >> >> >
> > > > > > >> > > >> >> > I would be very appreciated if you could point me
> to
> > > the
> > > > > > right
> > > > > > >> > > >> direction.
> > > > > > >> > > >> >> >
> > > > > > >> > > >> >> > Best,
> > > > > > >> > > >> >> >
> > > > > > >> > > >> >> >
> > > > > > >> > > >> >> > Regionserver log:
> > > > > > >> > > >> >> > 2009-11-11 04:24:53,718 INFO
> > > > > > org.apache.zookeeper.ClientCnxn:
> > > > > > >> > > >> Attempting
> > > > > > >> > > >> >> > connection to server superpyxis0018/
> > > > 192.168.100.132:2181
> > > > > > >> > > >> >> > 2009-11-11 04:24:53,718 INFO
> > > > > > org.apache.zookeeper.ClientCnxn:
> > > > > > >> > > Priming
> > > > > > >> > > >> >> > connection to
> > > java.nio.channels.SocketChannel[connected
> > > > > > >> local=/
> > > > > > >> > > >> >> > 192.168.100.131:43065 remote=superpyxis0018/
> > > > > > >> 192.168.100.132:2181
> > > > > > >> > ]
> > > > > > >> > > >> >> > 2009-11-11 04:24:53,718 INFO
> > > > > > org.apache.zookeeper.ClientCnxn:
> > > > > > >> > > Server
> > > > > > >> > > >> >> > connection successful
> > > > > > >> > > >> >> > 2009-11-11 04:24:53,768 INFO
> > > > > > >> > > >> >> >
> org.apache.hadoop.hbase.regionserver.HRegionServer:
> > > Got
> > > > > > >> ZooKeeper
> > > > > > >> > > >> event,
> > > > > > >> > > >> >> > state: Disconnected, type: None, path: null
> > > > > > >> > > >> >> > 2009-11-11 04:24:54,118 INFO
> > > > > > org.apache.zookeeper.ClientCnxn:
> > > > > > >> > > >> Attempting
> > > > > > >> > > >> >> > connection to server superpyxis0017/
> > > > 192.168.100.131:2181
> > > > > > >> > > >> >> > 2009-11-11 04:24:54,118 INFO
> > > > > > org.apache.zookeeper.ClientCnxn:
> > > > > > >> > > Priming
> > > > > > >> > > >> >> > connection to
> > > java.nio.channels.SocketChannel[connected
> > > > > > >> local=/
> > > > > > >> > > >> >> > 192.168.100.131:46903 remote=superpyxis0017/
> > > > > > >> 192.168.100.131:2181
> > > > > > >> > ]
> > > > > > >> > > >> >> > 2009-11-11 04:24:54,118 INFO
> > > > > > org.apache.zookeeper.ClientCnxn:
> > > > > > >> > > Server
> > > > > > >> > > >> >> > connection successful
> > > > > > >> > > >> >> > 2009-11-11 04:24:54,118 WARN
> > > > > > org.apache.zookeeper.ClientCnxn:
> > > > > > >> > > >> Exception
> > > > > > >> > > >> >> > closing session 0x24dfd90c810002 to
> > > > > > >> > > >> sun.nio.ch.SelectionKeyImpl@2dd78ccf
> > > > > > >> > > >> >> > java.io.IOException: Read error rc = -1
> > > > > > >> > > >> java.nio.DirectByteBuffer[pos=0
> > > > > > >> > > >> >> > lim=4 cap=4]
> > > > > > >> > > >> >> > at
> > > > > > >> > > >> >> >
> > > > > > >> > >
> > > > >
> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
> > > > > > >> > > >> >> > at
> > > > > > >> > > >> >> >
> > > > > > >> >
> > > > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
> > > > > > >> > > >> >> > 2009-11-11 04:24:54,118 WARN
> > > > > > org.apache.zookeeper.ClientCnxn:
> > > > > > >> > > Ignoring
> > > > > > >> > > >> >> > exception during shutdown input
> > > > > > >> > > >> >> > java.net.SocketException: Transport endpoint is
> not
> > > > > > connected
> > > > > > >> > > >> >> > at
> > sun.nio.ch.SocketChannelImpl.shutdown(Native
> > > > > > Method)
> > > > > > >> > > >> >> > at
> > > > > > >> > > >> >> >
> > > > > > >> > >
> > > > > >
> > > sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
> > > > > > >> > > >> >> > at
> > > > > > >> > > >>
> > > > sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> > > > > > >> > > >> >> > at
> > > > > > >> > > >> >> >
> > > > > > >> > > >>
> > > > > > >> >
> > > > > >
> > > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> > > > > > >> > > >> >> > at
> > > > > > >> > > >> >> >
> > > > > > >> >
> > > > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> > > > > > >> > > >> >> >
> > > > > > >> > > >> >> >
> > > > > > >> > > >> >> >
> > > > > > >> > > >> >> >
> > > > > > >> > > >> >> >
> > > > > > >> > > >> >> > On Wed, Nov 11, 2009 at 1:40 PM, Jean-Daniel
> Cryans
> > <
> > > > > > >> > > >> jdcryans@apache.org
> > > > > > >> > > >> >> >wrote:
> > > > > > >> > > >> >> >
> > > > > > >> > > >> >> >> I'm really not sure exactly what the situation is
> > > with
> > > > > the
> > > > > > >> > > >> >> >> java.nio.BufferUnderflowException, but what do
> you
> > > mean
> > > > > by
> > > > > > >> > > >> >> >> "regionserver disconnection"? Could you give more
> > > > > details?
> > > > > > >> > > >> >> >>
> > > > > > >> > > >> >> >> Thx,
> > > > > > >> > > >> >> >>
> > > > > > >> > > >> >> >> J-D
> > > > > > >> > > >> >> >>
> > > > > > >> > > >> >> >> On Wed, Nov 11, 2009 at 10:34 AM, Zhenyu Zhong <
> > > > > > >> > > >> zhongresearch@gmail.com
> > > > > > >> > > >> >> >
> > > > > > >> > > >> >> >> wrote:
> > > > > > >> > > >> >> >> > I searched around and found HBASE-1645 "HBase
> > could
> > > > be
> > > > > a
> > > > > > >> > little
> > > > > > >> > > >> more
> > > > > > >> > > >> >> >> dainty
> > > > > > >> > > >> >> >> > when zk is pulled out from under it"
> > > > > > >> > > >> >> >> >
> > > > > > >> > > >> >> >> > I am wondering whether my problem is related to
> > > this
> > > > > > >> > HBase-1645.
> > > > > > >> > > Or
> > > > > > >> > > >> >> >> probably
> > > > > > >> > > >> >> >> > has nothing to do with it?
> > > > > > >> > > >> >> >> >
> > > > > > >> > > >> >> >> > thanks
> > > > > > >> > > >> >> >> >
> > > > > > >> > > >> >> >> >
> > > > > > >> > > >> >> >> > On Wed, Nov 11, 2009 at 12:02 PM, Zhenyu Zhong
> <
> > > > > > >> > > >> >> zhongresearch@gmail.com
> > > > > > >> > > >> >> >> >wrote:
> > > > > > >> > > >> >> >> >
> > > > > > >> > > >> >> >> >> Hi,
> > > > > > >> > > >> >> >> >>
> > > > > > >> > > >> >> >> >> I have been monitoring the system resources
> such
> > > as
> > > > > > memory
> > > > > > >> > Heap
> > > > > > >> > > >> usage
> > > > > > >> > > >> >> >> for
> > > > > > >> > > >> >> >> >> Regionserver, load average, network bandwidth,
> > > etc.
> > > > > > >> > > >> >> >> >>
> > > > > > >> > > >> >> >> >> I experienced a disconnection last night. When
> I
> > > > > checked
> > > > > > >> the
> > > > > > >> > > >> memory
> > > > > > >> > > >> >> Heap
> > > > > > >> > > >> >> >> >> for Regionserver, it was 2336 which is way
> below
> > > the
> > > > > max
> > > > > > >> Heap
> > > > > > >> > > >> 4096. I
> > > > > > >> > > >> >> >> don't
> > > > > > >> > > >> >> >> >> think it is Regionserver problem. The requests
> > > that
> > > > > the
> > > > > > >> > > >> regionserver
> > > > > > >> > > >> >> got
> > > > > > >> > > >> >> >> was
> > > > > > >> > > >> >> >> >> about 126 which doesn't seem to be high. The
> > load
> > > > > > average
> > > > > > >> is
> > > > > > >> > > kind
> > > > > > >> > > >> of
> > > > > > >> > > >> >> >> high
> > > > > > >> > > >> >> >> >> >100. (but I saw disconnections while the load
> > > > average
> > > > > > is
> > > > > > >> > > around
> > > > > > >> > > >> 30).
> > > > > > >> > > >> >> >> The
> > > > > > >> > > >> >> >> >> bandwidth is not a problem.
> > > > > > >> > > >> >> >> >>
> > > > > > >> > > >> >> >> >> However, I found error messages from the
> > > zookeeper.
> > > > > > Please
> > > > > > >> > see
> > > > > > >> > > the
> > > > > > >> > > >> >> >> >> following error messages. There are some
> > > > > > >> > > >> >> >> >> BufferUnderflowException error, IOException
> > > > error,etc.
> > > > > > >> > > >> >> >> >> May I ask what might be the causes that
> generate
> > > > these
> > > > > > >> error
> > > > > > >> > > >> >> messages?
> > > > > > >> > > >> >> >> >>
> > > > > > >> > > >> >> >> >> Many thanks
> > > > > > >> > > >> >> >> >> zhenyu
> > > > > > >> > > >> >> >> >>
> > > > > > >> > > >> >> >> >>
> > > > > > >> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
> > > > > > >> > > org.apache.zookeeper.server.Request:
> > > > > > >> > > >> >> >> Ignoring
> > > > > > >> > > >> >> >> >> exception during toString
> > > > > > >> > > >> >> >> >> java.nio.BufferUnderflowException
> > > > > > >> > > >> >> >> >> at
> > > > > > >> > java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
> > > > > > >> > > >> >> >> >> at
> > > > > java.nio.ByteBuffer.get(ByteBuffer.java:675)
> > > > > > >> > > >> >> >> >> at
> > > > > > >> > > >> >> >>
> > > > > > >> org.apache.zookeeper.server.Request.toString(Request.java:199)
> > > > > > >> > > >> >> >> >> at
> > > > java.lang.String.valueOf(String.java:2827)
> > > > > > >> > > >> >> >> >> at
> > > > > > >> > > java.lang.StringBuilder.append(StringBuilder.java:115)
> > > > > > >> > > >> >> >> >> at
> > > > > > >> > > >> >> >> >>
> > > > > > >> > > >> >> >>
> > > > > > >> > > >> >>
> > > > > > >> > > >>
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.zookeeper.server.quorum.CommitProcessor.processRequest(CommitProcessor.java:167)
> > > > > > >> > > >> >> >> >> at
> > > > > > >> > > >> >> >> >>
> > > > > > >> > > >> >> >>
> > > > > > >> > > >> >>
> > > > > > >> > > >>
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.zookeeper.server.quorum.FollowerRequestProcessor.run(FollowerRequestProcessor.java:68)
> > > > > > >> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
> > > > > > >> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
> > > > > > >> > > >> >> >> >> Exception causing close of session
> > > 0x424dfd908c50009
> > > > > > >> > > >> >> >> >> due to java.io.IOException: Read error
> > > > > > >> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
> > > > > > >> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
> > > > > > >> > > >> >> >> >> IOException stack trace
> > > > > > >> > > >> >> >> >> java.io.IOException: Read error
> > > > > > >> > > >> >> >> >> at
> > > > > > >> > > >> >> >> >>
> > > > > > >> > > >> >>
> > > > > > >> > >
> > > > > >
> > > org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:372)
> > > > > > >> > > >> >> >> >> at
> > > > > > >> > > >> >> >> >>
> > > > > > >> > > >> >> >>
> > > > > > >> > > >> >>
> > > > > > >> > > >>
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:221)
> > > > > > >> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
> > > > > > >> > > >> >> >> >>
> > > org.apache.zookeeper.server.quorum.CommitProcessor:
> > > > > > >> > Processing
> > > > > > >> > > >> >> request::
> > > > > > >> > > >> >> >> >> sessionid:0x424dfd908c50009 type:setWatches
> > > > > > >> > > >> cxid:0xfffffffffffffff8
> > > > > > >> > > >> >> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
> > > > > > >> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
> > > > > > >> > > org.apache.zookeeper.server.Request:
> > > > > > >> > > >> >> >> Ignoring
> > > > > > >> > > >> >> >> >> exception during toString
> > > > > > >> > > >> >> >> >> java.nio.BufferUnderflowException
> > > > > > >> > > >> >> >> >> at
> > > > > > >> > java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
> > > > > > >> > > >> >> >> >> at
> > > > > java.nio.ByteBuffer.get(ByteBuffer.java:675)
> > > > > > >> > > >> >> >> >> at
> > > > > > >> > > >> >> >>
> > > > > > >> org.apache.zookeeper.server.Request.toString(Request.java:199)
> > > > > > >> > > >> >> >> >> at
> > > > java.lang.String.valueOf(String.java:2827)
> > > > > > >> > > >> >> >> >> at
> > > > > > >> > > java.lang.StringBuilder.append(StringBuilder.java:115)
> > > > > > >> > > >> >> >> >> at
> > > > > > >> > > >> >> >> >>
> > > > > > >> > > >> >> >>
> > > > > > >> > > >> >>
> > > > > > >> > > >>
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:74)
> > > > > > >> > > >> >> >> >> at
> > > > > > >> > > >> >> >> >>
> > > > > > >> > > >> >> >>
> > > > > > >> > > >> >>
> > > > > > >> > > >>
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)
> > > > > > >> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
> > > > > > >> > > >> >> >> >>
> > org.apache.zookeeper.server.FinalRequestProcessor:
> > > > > > >> Processing
> > > > > > >> > > >> >> request::
> > > > > > >> > > >> >> >> >> sessionid:0x424dfd908c50009 type:setWatches
> > > > > > >> > > >> cxid:0xfffffffffffffff8
> > > > > > >> > > >> >> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
> > > > > > >> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
> > > > > > >> > > org.apache.zookeeper.server.Request:
> > > > > > >> > > >> >> >> Ignoring
> > > > > > >> > > >> >> >> >> exception during toString
> > > > > > >> > > >> >> >> >> java.nio.BufferUnderflowException
> > > > > > >> > > >> >> >> >> at
> > > > > > >> > java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
> > > > > > >> > > >> >> >> >> at
> > > > > java.nio.ByteBuffer.get(ByteBuffer.java:675)
> > > > > > >> > > >> >> >> >> at
> > > > > > >> > > >> >> >>
> > > > > > >> org.apache.zookeeper.server.Request.toString(Request.java:199)
> > > > > > >> > > >> >> >> >> at
> > > > > > >> > > >> >> >> >>
> > > > > > >> > > >>
> > > > > > >>
> > > > org.apache.log4j.or.DefaultRenderer.doRender(DefaultRenderer.java:36)
> > > > > > >> > > >> >> >> >> at
> > > > > > >> > > >> >> >> >>
> > > > > > >> > >
> > > > org.apache.log4j.or.RendererMap.findAndRender(RendererMap.java:80)
> > > > > > >> > > >> >> >> >> at
> > > > > > >> > > >> >> >> >>
> > > > > > >> > > >> >> >>
> > > > > > >> > > >> >>
> > > > > > >> > > >>
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.log4j.spi.LoggingEvent.getRenderedMessage(LoggingEvent.java:362)
> > > > > > >> > > >> >> >> >> at
> > > > > > >> > > >> >> >> >>
> > > > > > >> > > >> >> >>
> > > > > > >> > > >> >>
> > > > > > >> > > >>
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.log4j.helpers.PatternParser$BasicPatternConverter.convert(PatternParser.java:403)
> > > > > > >> > > >> >> >> >> at
> > > > > > >> > > >> >> >> >>
> > > > > > >> > > >> >> >>
> > > > > > >> > > >> >>
> > > > > > >> > > >>
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.log4j.helpers.PatternConverter.format(PatternConverter.java:65)
> > > > > > >> > > >> >> >> >> at
> > > > > > >> > > >> >>
> > > > > org.apache.log4j.PatternLayout.format(PatternLayout.java:502)
> > > > > > >> > > >> >> >> >> at
> > > > > > >> > > >> >> >> >>
> > > > > > >> > >
> > > > org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:302)
> > > > > > >> > > >> >> >> >> at
> > > > > > >> > > >> >> >> >>
> > > > > > >> > > >> >> >>
> > > > > > >> > > >> >>
> > > > > > >> > > >>
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.log4j.DailyRollingFileAppender.subAppend(DailyRollingFileAppender.java:359)
> > > > > > >> > > >> >> >> >> at
> > > > > > >> > > >> >> >>
> > > > > > >>
> org.apache.log4j.WriterAppender.append(WriterAppender.java:160)
> > > > > > >> > > >> >> >> >> at
> > > > > > >> > > >> >> >> >>
> > > > > > >> > > >>
> > > > > > >>
> > > > org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
> > > > > > >> > > >> >> >> >> at
> > > > > > >> > > >> >> >> >>
> > > > > > >> > > >> >> >>
> > > > > > >> > > >> >>
> > > > > > >> > > >>
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
> > > > > > >> > > >> >> >> >> at
> > > > > > >> > > >>
> > org.apache.log4j.Category.callAppenders(Category.java:206)
> > > > > > >> > > >> >> >> >> at
> > > > > > >> > > org.apache.log4j.Category.forcedLog(Category.java:391)
> > > > > > >> > > >> >> >> >> at
> > > > > > >> org.apache.log4j.Category.debug(Category.java:260)
> > > > > > >> > > >> >> >> >> at
> > > > > > >> > > >> >> >> >>
> > > > > > >> > > >> >> >>
> > > > > > >> > > >> >>
> > > > > > >> > > >>
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:147)
> > > > > > >> > > >> >> >> >> at
> > > > > > >> > > >> >> >> >>
> > > > > > >> > > >> >> >>
> > > > > > >> > > >> >>
> > > > > > >> > > >>
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)
> > > > > > >> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
> > > > > > >> > > org.apache.zookeeper.server.Request:
> > > > > > >> > > >> >> >> Ignoring
> > > > > > >> > > >> >> >> >> exception during toString
> > > > > > >> > > >> >> >> >> sessionid:0x424dfd908c50009 type:setWatches
> > > > > > >> > > >> cxid:0xfffffffffffffff8
> > > > > > >> > > >> >> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
> > > > > > >> > > >> >> >> >> 2009-11-11 04:29:00,810 INFO
> > > > > > >> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
> > > > > > >> > > >> >> >> >> closing session:0x424dfd908c50009
> NIOServerCnxn:
> > > > > > >> > > >> >> >> >> java.nio.channels.SocketChannel[connected
> > local=/
> > > > > > >> > > >> >> 192.168.100.131:2181
> > > > > > >> > > >> >> >> remote=/
> > > > > > >> > > >> >> >> >> 192.168.100.131:35500]
> > > > > > >> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
> > > > > > >> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
> > > > > > >> > > >> >> >> >> ignoring exception during input shutdown
> > > > > > >> > > >> >> >> >> java.net.SocketException: Transport endpoint
> is
> > > not
> > > > > > >> connected
> > > > > > >> > > >> >> >> >> at
> > > > > sun.nio.ch.SocketChannelImpl.shutdown(Native
> > > > > > >> > Method)
> > > > > > >> > > >> >> >> >> at
> > > > > > >> > > >> >> >> >>
> > > > > > >> > > >> >>
> > > > > > >> > >
> > > > > >
> > > sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
> > > > > > >> > > >> >> >> >> at
> > > > > > >> > > >> >> >>
> > > > > > >> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> > > > > > >> > > >> >> >> >> at
> > > > > > >> > > >> >> >> >>
> > > > > > >> > > >> >>
> > > > > > >> > >
> > > > > > >>
> > > > >
> > org.apache.zookeeper.server.NIOServerCnxn.close(NIOServerCnxn.java:848)
> > > > > > >> > > >> >> >> >> at
> > > > > > >> > > >> >> >> >>
> > > > > > >> > > >> >>
> > > > > > >> > >
> > > > > >
> > > org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:498)
> > > > > > >> > > >> >> >> >> at
> > > > > > >> > > >> >> >> >>
> > > > > > >> > > >> >> >>
> > > > > > >> > > >> >>
> > > > > > >> > > >>
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:221)
> > > > > > >> > > >> >> >> >>
> > > > > > >> > > >> >> >> >
> > > > > > >> > > >> >> >>
> > > > > > >> > > >> >> >
> > > > > > >> > > >> >>
> > > > > > >> > > >> >
> > > > > > >> > > >>
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: regionserver disconnection
Posted by stack <st...@duboce.net>.
Enable GC logging too on this next run (see hbase-env.sh). Lets try and get
to the bottom of whats going on.
Thanks,
St.Ack
On Thu, Nov 12, 2009 at 12:29 PM, Zhenyu Zhong <zh...@gmail.com>wrote:
> I can switch the boxes that run zookeeper with the ones that run
> regionservers.
> I will see the results later.
>
> FYI. The node does have the 10 minutes zookeeper.session.timeout value in
> place.
>
> thanks
> zhenyu
>
> On Thu, Nov 12, 2009 at 3:21 PM, stack <st...@duboce.net> wrote:
>
> > On Thu, Nov 12, 2009 at 11:50 AM, Zhenyu Zhong <zhongresearch@gmail.com
> > >wrote:
> >
> > >
> > > In my cluster, half of the cluster have 2 disks 400GB each per machine,
> > and
> > > half of the cluster have 6 disks per machine. Maybe we should run
> > > zookeeper
> > > on the machines with 2 disks and RS on machines with 6 disks?
> > >
> > > That would make most sense only in the below, it looks like the RS that
> > had
> > issue had 4 disks?
> >
> >
> >
> > > BTW, the 10 minutes zookeeper.session.timeout has been set during the
> > > experiment.
> > >
> >
> > And for sure this node had it in place?
> > St.Ack
> >
> >
> >
> > >
> > > thanks
> > > zhenyu
> > >
> > > On Thu, Nov 12, 2009 at 2:08 PM, stack <st...@duboce.net> wrote:
> > >
> > > > On Thu, Nov 12, 2009 at 8:40 AM, Zhenyu Zhong <
> zhongresearch@gmail.com
> > > > >wrote:
> > > >
> > > >
> > > > > Though I experienced 2 regionserver disconnection this morning, it
> > > looks
> > > > > better from the regionserver log. (please see the following log)
> > > > >
> > > > >
> > > > > http://pastebin.com/m496dbfae
> > > > >
> > > > >
> > > > > I checked diskIO for one of the regionserver(192.168.100.116)
> during
> > > the
> > > > > time it disconnected.
> > > > >
> > > > > Time: 03:04:58 AM
> > > > > Device: tps Blk_read/s Blk_wrtn/s Blk_read
> > Blk_wrtn
> > > > > sda 105.31 5458.83 19503.64 9043873239
> > 32312473676
> > > > > sda1 2.90 6.64 99.25 10993934
> > 164433464
> > > > > sda2 1.72 23.76 30.25 39365817
> > 50124008
> > > > > sda3 0.30 0.38 3.58 624930
> > 5923000
> > > > > sda4 100.39 5428.06 19370.56 8992888270
> > 32091993204
> > > > >
> > > > >
> > > > Is this high for you? 20k blocks/second would seem to be high but
> its
> > > one
> > > > disk only and its not being shared by zk anymore so shouldn't matter?
> > > >
> > > >
> > > > > I also checked the zookeeper quorum server that the regionserver
> > tried
> > > to
> > > > > connect according to the log. However, I don't see 192.168.100.116
> in
> > > the
> > > > > client list of the zookeeper quorum server that the regionserver
> > tried
> > > to
> > > > > connect.
> > > > > Would that be a problem?
> > > > >
> > > > >
> > > > Is that because the ephemeral node for the regionserver had
> evaporated?
> > > > Lost
> > > > its lease w/ zk by the time you went to look?
> > > >
> > > >
> > > >
> > > >
> > > > > Thu Nov 12 15:04:52 UTC 2009
> > > > > Zookeeper version: 3.2.1-808558, built on 08/27/2009 18:48 GMT
> > > > > Clients:
> > > > > /192.168.100.127:43045[1](queued=0,recved=26,sent=0)
> > > > > /192.168.100.131:39091[1](queued=0,recved=964,sent=0)
> > > > > /192.168.100.124:35961[1](queued=0,recved=3266,sent=0)
> > > > > /192.168.100.123:47935[1](queued=0,recved=233,sent=0)
> > > > > /192.168.100.125:46931[1](queued=0,recved=2,sent=0)
> > > > > /192.168.100.118:54924[1](queued=0,recved=295,sent=0)
> > > > > /192.168.100.118:41390[1](queued=0,recved=2290,sent=0)
> > > > > /192.168.100.136:42243[1](queued=0,recved=0,sent=0)
> > > > >
> > > > > Latency min/avg/max: 0/17/6333
> > > > > Received: 47111
> > > > > Sent: 0
> > > > > Outstanding: 0
> > > > > Zxid: 0x77000083f4
> > > > > Mode: leader
> > > > > Node count: 23
> > > > >
> > > >
> > > >
> > > > That 6 second maximum latency is pretty bad but should be well within
> > the
> > > > zk
> > > > session timeout.
> > > >
> > > > So, problem is likely on the zk client side of the session; i.e. in
> the
> > > RS.
> > > > You could enable GC logging as J-D suggested to see if you have any
> big
> > > > pauses, pauses > zk session timeout.
> > > >
> > > > When the RS went down, it didn't look too heavily loaded:
> > > >
> > > >
> > > > 1. 2009-11-12 15:04:52,830 INFO
> > > > org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of
> metrics:
> > > > request=1.5166667, regions=322, stores=657, storefiles=631,
> > > > storefileIndexSize=61, memstoreSize=1472, usedHeap=2819,
> > maxHeap=4079,
> > > > blockCacheSize=658110960, blockCacheFree=197395984,
> > > blockCacheCount=9903,
> > > > blockCacheHitRatio=99
> > > >
> > > > Its serving a few reads? The number of store files seems fine. Not
> > too
> > > > much memory used.
> > > >
> > > > Looking at the logs, I see the Lease Still Held exception. This
> > happens
> > > > when the RS does its regular report to the master but the master
> thinks
> > > the
> > > > RS has since restarted. It'll think this probably because it noticed
> > > that
> > > > the RS's znode in zk had gone away and it considered the RS dead.
> > > >
> > > > Looking too at your logs I see this gap in the zk pinging:
> > > >
> > > >
> > > > 1. 2009-11-12 15:03:39,325 DEBUG org.apache.zookeeper.ClientCnxn:
> Got
> > > > ping response for sessionid:0x224e55436ad0004 after 0ms
> > > > 2. 2009-11-12 15:03:43,113 DEBUG org.apache.zookeeper.ClientCnxn:
> Got
> > > > ping response for sessionid:0x24e55436a0007d after 0ms
> > > >
> > > > Where in the lines above it, its reporting about every ten seconds,
> > here
> > > > there is a big gap.
> > > >
> > > > Do you have ganglia or something that will let you look more into
> what
> > > was
> > > > happening on this machine around this time? Is the machine OK? It
> > looks
> > > > lightly loaded and you have your cluster nicely laid out. Something
> > odd
> > > is
> > > > going on. What about things like the write speed to disk? In the
> past
> > > > strange issues have been explained by incorrectly set BIOS which made
> > > disks
> > > > run at 1/100th of their proper speed.
> > > >
> > > > St.Ack
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > >
> > > > > Best,
> > > > > zhenyu
> > > > >
> > > > >
> > > > > On Wed, Nov 11, 2009 at 3:58 PM, Zhenyu Zhong <
> > zhongresearch@gmail.com
> > > > > >wrote:
> > > > >
> > > > > > Stack
> > > > > >
> > > > > > I am very appreciated for your comments.
> > > > > > I will use the zookeeper monitoring script on my cluster and let
> it
> > > run
> > > > > > overnight to see the result.
> > > > > > I will follow up the thread when I get anything.
> > > > > >
> > > > > > thanks
> > > > > > zhenyu
> > > > > >
> > > > > >
> > > > > > On Wed, Nov 11, 2009 at 3:52 PM, stack <st...@duboce.net> wrote:
> > > > > >
> > > > > >> I see these in your log too:
> > > > > >>
> > > > > >>
> > > > > >> 1. 2009-11-11 04:27:19,018 DEBUG
> > org.apache.zookeeper.ClientCnxn:
> > > > Got
> > > > > >> ping response for sessionid:0x424dfd908c50009 after 4544ms
> > > > > >> 2. 2009-11-11 04:27:19,018 DEBUG
> > org.apache.zookeeper.ClientCnxn:
> > > > Got
> > > > > >> ping response for sessionid:0x24dfd90c810002 after 4548ms
> > > > > >> 3. 2009-11-11 04:27:43,960 DEBUG
> > org.apache.zookeeper.ClientCnxn:
> > > > Got
> > > > > >> ping response for sessionid:0x424dfd908c50009 after 9030ms
> > > > > >> 4. 2009-11-11 04:27:43,960 DEBUG
> > org.apache.zookeeper.ClientCnxn:
> > > > Got
> > > > > >> ping response for sessionid:0x24dfd90c810002 after 9030ms
> > > > > >> 5. 2009-11-11 04:27:48,830 DEBUG
> > org.apache.zookeeper.ClientCnxn:
> > > > Got
> > > > > >> ping response for sessionid:0x424dfd908c50009 after 0ms
> > > > > >> 6. 2009-11-11 04:27:48,830 DEBUG
> > org.apache.zookeeper.ClientCnxn:
> > > > Got
> > > > > >> ping response for sessionid:0x24dfd90c810002 after 0ms
> > > > > >>
> > > > > >> Which would indicate lag talking to zk too.
> > > > > >>
> > > > > >> Appending HLog also seems to get steadily worse.
> > > > > >>
> > > > > >> I also see these:
> > > > > >>
> > > > > >>
> > > > > >> 1. 2009-11-11 04:27:01,233 WARN
> org.apache.zookeeper.ClientCnxn:
> > > > > >> Exception closing session 0x24dfd90c810002 to
> > > > > >> sun.nio.ch.SelectionKeyImpl@5e16a7e3
> > > > > >> 2. java.io.IOException: Read error rc = -1
> > > > > >> java.nio.DirectByteBuffer[pos=0 lim=4 cap=4]
> > > > > >> 3. at
> > > > > >>
> > > > org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
> > > > > >> 4. at
> > > > > >>
> > > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
> > > > > >>
> > > > > >>
> > > > > >> Were all 5 quorum members participating? Check by doing "echo
> > > stat|nc
> > > > > >> QUORUM_HOST port" for each member of quorum.... the below might
> > > help:
> > > > > >>
> > > > > >> members="1,2,3,4,5"
> > > > > >> while true; do date; for i in $members; do echo $i; echo stat |
> nc
> > > $i
> > > > > >> 2181;
> > > > > >> done; sleep 10; done
> > > > > >>
> > > > > >> We need to improve our zk_dump. It just reports quorum
> members...
> > > not
> > > > > if
> > > > > >> they are actually participating or not. The above will tell you
> > if
> > > > dead
> > > > > >> or
> > > > > >> alive.
> > > > > >>
> > > > > >> Please report back after checking above and following the
> > > Jean-Daniel
> > > > > >> prescription.
> > > > > >>
> > > > > >> Thanks,
> > > > > >> St.Ack
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> On Wed, Nov 11, 2009 at 12:29 PM, Zhenyu Zhong <
> > > > zhongresearch@gmail.com
> > > > > >> >wrote:
> > > > > >>
> > > > > >> > Thanks a lot for the help.
> > > > > >> >
> > > > > >> > I understand that the change of disk configurations should be
> > done
> > > > > >> > progressively such that HDFS can balance and recover from a
> disk
> > > > going
> > > > > >> > away.
> > > > > >> >
> > > > > >> > The data we are trying to import into HBase is about
> 200million
> > > > rows,
> > > > > >> each
> > > > > >> > row is about hundreds of bytes. Basically, we read in a text
> > file
> > > > from
> > > > > >> > HDFS,
> > > > > >> > then for each row in the text file, we fetch data from HBase
> > > table,
> > > > > we
> > > > > >> > compare the column value and update the latest value back into
> > > > HBase.
> > > > > I
> > > > > >> was
> > > > > >> > suggested to use CheckAndPut. Anyway this is the common
> usecase
> > > on
> > > > my
> > > > > >> > side.
> > > > > >> >
> > > > > >> >
> > > > > >> > Many thanks
> > > > > >> > zhenyu
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> > On Wed, Nov 11, 2009 at 3:03 PM, Jean-Daniel Cryans <
> > > > > >> jdcryans@apache.org
> > > > > >> > >wrote:
> > > > > >> >
> > > > > >> > > 6 disks, 19 machines and you see those kind of issues? I'm
> > quite
> > > > > >> > > surprised and I want to learn more about what you trying to
> > to.
> > > > > >> > >
> > > > > >> > > But first, if hbase is managing ZK for you then you can
> change
> > > the
> > > > > >> > > value of hbase.zookeeper.property.dataDir so that it points
> to
> > > its
> > > > > own
> > > > > >> > > mount. Make sure you also change dfs.data.dir in
> > > > conf/hdfs-site.xml
> > > > > >> > > (hadoop stuff) so that it stops pointing there (you should
> > have
> > > 6
> > > > > >> > > values there at the moment if you configured it correctly).
> > > Doing
> > > > > this
> > > > > >> > > could result in data loss if you do it all at once so do the
> > > > change
> > > > > >> > > progressively. This also means that you may have to keep a
> > > > seperate
> > > > > >> > > HDFS configuration for those particular nodes. Please ask
> more
> > > > > >> > > questions if it's not very clear.
> > > > > >> > >
> > > > > >> > > WRT to dedicating nodes to Zookeeeper, it makes sense only
> if
> > > you
> > > > > have
> > > > > >> > > a lot of nodes but in your case, unless there's also CPU
> > > > contention,
> > > > > >> > > you're good with only a dedicated disk.
> > > > > >> > >
> > > > > >> > > Now for what you are trying to do, is this a massive import
> > that
> > > > > goes
> > > > > >> > > over many hours? How big is each row? I'd love to get a lot
> of
> > > > > details
> > > > > >> > > to help you optimize your cluster for your particular
> problem.
> > > > > >> > >
> > > > > >> > > Thx,
> > > > > >> > >
> > > > > >> > > J-D
> > > > > >> > >
> > > > > >> > > On Wed, Nov 11, 2009 at 11:51 AM, Zhenyu Zhong <
> > > > > >> zhongresearch@gmail.com>
> > > > > >> > > wrote:
> > > > > >> > > > J-D,
> > > > > >> > > >
> > > > > >> > > > Thanks a lot for the insightful comments.
> > > > > >> > > > We are running 5 zookeeper quorums now. On these 5 quorum
> > > > servers,
> > > > > >> we
> > > > > >> > > also
> > > > > >> > > > run datanode, tasktracker and regionserver.
> > > > > >> > > > Now we have 6 disks per machine.
> > > > > >> > > > If it is the disk IO saturation, is there anyway to
> > configure
> > > > > >> zookeeper
> > > > > >> > > to
> > > > > >> > > > write to a seperate disk which is not the same one used by
> > > HDFS?
> > > > > >> > > > Or shall we run zookeeper quorums on different nodes other
> > > than
> > > > > >> HDFS?
> > > > > >> > > >
> > > > > >> > > > Are HLogs generated by Regionservers? Would it alleviate
> the
> > > > issue
> > > > > >> if
> > > > > >> > we
> > > > > >> > > use
> > > > > >> > > > the first 14 for regionserver, and the rest of 5 for
> > > > zookeeper?(In
> > > > > >> that
> > > > > >> > > > sense, the node running zookeeper quorum won't run
> > > regionserver)
> > > > > >> > > >
> > > > > >> > > > I am very appreciated for your comments.
> > > > > >> > > >
> > > > > >> > > > Best,
> > > > > >> > > > zhenyu
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > > > On Wed, Nov 11, 2009 at 2:30 PM, Jean-Daniel Cryans <
> > > > > >> > jdcryans@apache.org
> > > > > >> > > >wrote:
> > > > > >> > > >
> > > > > >> > > >> The problem seems to be that the IO is saturated. When
> you
> > > see
> > > > > >> this:
> > > > > >> > > >>
> > > > > >> > > >> #
> > > > > >> > > >> 2009-11-11 04:28:10,308 WARN
> > > > > >> > > >> org.apache.hadoop.hbase.regionserver.HLog: IPC Server
> > handler
> > > 9
> > > > > on
> > > > > >> > > >> 60021 took 6819ms appending an edit to hlog;
> editcount=370
> > > > > >> > > >> #
> > > > > >> > > >> 2009-11-11 04:28:10,308 WARN
> > > > > >> > > >> org.apache.hadoop.hbase.regionserver.HLog: IPC Server
> > handler
> > > 2
> > > > > on
> > > > > >> > > >> 60021 took 4412ms appending an edit to hlog;
> editcount=371
> > > > > >> > > >>
> > > > > >> > > >> It's a sign that HDFS is slow. I guess that the zookeeper
> > > > quorum
> > > > > >> > > >> members are on those same nodes using the same disks? If
> > so,
> > > > > >> there's
> > > > > >> > > >> the problem because Zookeeper needs to be able to commit
> > > > > >> transactions
> > > > > >> > > >> very fast and if your disks are already hogged then they
> > have
> > > > to
> > > > > >> wait
> > > > > >> > > >> and you can get disconnections. Do you still have 19
> quorum
> > > > > >> members?
> > > > > >> > > >> How many disks do you have per machine?
> > > > > >> > > >>
> > > > > >> > > >> Thx
> > > > > >> > > >>
> > > > > >> > > >> J-D
> > > > > >> > > >>
> > > > > >> > > >> On Wed, Nov 11, 2009 at 11:22 AM, Zhenyu Zhong <
> > > > > >> > zhongresearch@gmail.com
> > > > > >> > > >
> > > > > >> > > >> wrote:
> > > > > >> > > >> > Please take a look at the following regionserver log.
> > > > > >> > > >> > The error started to come out from 4:24am UTC to 4:29am
> > > UTC.
> > > > > >> > > >> >
> > > > > >> > > >> > http://pastebin.com/m3717d064
> > > > > >> > > >> >
> > > > > >> > > >> > Thanks in advance.
> > > > > >> > > >> >
> > > > > >> > > >> > Best,
> > > > > >> > > >> > zhenyu
> > > > > >> > > >> >
> > > > > >> > > >> >
> > > > > >> > > >> > On Wed, Nov 11, 2009 at 2:01 PM, Jean-Daniel Cryans <
> > > > > >> > > jdcryans@apache.org
> > > > > >> > > >> >wrote:
> > > > > >> > > >> >
> > > > > >> > > >> >> Could you paste a bigger log trace (interested to see
> at
> > > > least
> > > > > >> all
> > > > > >> > > 300
> > > > > >> > > >> >> lines before that) in a pastebin?
> > > > > >> > > >> >>
> > > > > >> > > >> >> Thx,
> > > > > >> > > >> >>
> > > > > >> > > >> >> J-D
> > > > > >> > > >> >>
> > > > > >> > > >> >> On Wed, Nov 11, 2009 at 10:56 AM, Zhenyu Zhong <
> > > > > >> > > zhongresearch@gmail.com
> > > > > >> > > >> >
> > > > > >> > > >> >> wrote:
> > > > > >> > > >> >> > J-D,
> > > > > >> > > >> >> >
> > > > > >> > > >> >> > Thanks for the reply.
> > > > > >> > > >> >> >
> > > > > >> > > >> >> > The problem I have is one or more of the
> regionservers
> > > > > >> disconnect
> > > > > >> > > from
> > > > > >> > > >> >> the
> > > > > >> > > >> >> > cluster.
> > > > > >> > > >> >> >
> > > > > >> > > >> >> > I checked the resource usage including memory Heap
> > usage
> > > > for
> > > > > >> > > >> >> regionserver,
> > > > > >> > > >> >> > and found nothing suspicious. The regionserver heap
> > > usage
> > > > is
> > > > > >> way
> > > > > >> > > below
> > > > > >> > > >> >> the
> > > > > >> > > >> >> > maxHeap defined in jvm.
> > > > > >> > > >> >> >
> > > > > >> > > >> >> > So I checked zookeeper log, then I found the
> following
> > > > > >> > > >> >> > BufferUnderflowException, eventually it leads to
> > > > > >> SocketException:
> > > > > >> > > >> >> Transport
> > > > > >> > > >> >> > endpoint is not connected.
> > > > > >> > > >> >> >
> > > > > >> > > >> >> >
> > > > > >> > > >> >> > I also checked the regionserver log. It looks like
> the
> > > > > >> > regionserver
> > > > > >> > > >> tried
> > > > > >> > > >> >> to
> > > > > >> > > >> >> > connect to other zookeeper quorum server. It
> connects
> > > > > >> > successfully,
> > > > > >> > > >> but
> > > > > >> > > >> >> then
> > > > > >> > > >> >> > it complained about SocketException: transport
> > endpoint
> > > is
> > > > > not
> > > > > >> > > >> connected.
> > > > > >> > > >> >> >
> > > > > >> > > >> >> > So both zookeeper log and regionserver log told me
> > that
> > > > the
> > > > > >> > > transport
> > > > > >> > > >> >> > endpoint is not connected. I don't know why.
> > > > > >> > > >> >> >
> > > > > >> > > >> >> > I would be very appreciated if you could point me to
> > the
> > > > > right
> > > > > >> > > >> direction.
> > > > > >> > > >> >> >
> > > > > >> > > >> >> > Best,
> > > > > >> > > >> >> >
> > > > > >> > > >> >> >
> > > > > >> > > >> >> > Regionserver log:
> > > > > >> > > >> >> > 2009-11-11 04:24:53,718 INFO
> > > > > org.apache.zookeeper.ClientCnxn:
> > > > > >> > > >> Attempting
> > > > > >> > > >> >> > connection to server superpyxis0018/
> > > 192.168.100.132:2181
> > > > > >> > > >> >> > 2009-11-11 04:24:53,718 INFO
> > > > > org.apache.zookeeper.ClientCnxn:
> > > > > >> > > Priming
> > > > > >> > > >> >> > connection to
> > java.nio.channels.SocketChannel[connected
> > > > > >> local=/
> > > > > >> > > >> >> > 192.168.100.131:43065 remote=superpyxis0018/
> > > > > >> 192.168.100.132:2181
> > > > > >> > ]
> > > > > >> > > >> >> > 2009-11-11 04:24:53,718 INFO
> > > > > org.apache.zookeeper.ClientCnxn:
> > > > > >> > > Server
> > > > > >> > > >> >> > connection successful
> > > > > >> > > >> >> > 2009-11-11 04:24:53,768 INFO
> > > > > >> > > >> >> > org.apache.hadoop.hbase.regionserver.HRegionServer:
> > Got
> > > > > >> ZooKeeper
> > > > > >> > > >> event,
> > > > > >> > > >> >> > state: Disconnected, type: None, path: null
> > > > > >> > > >> >> > 2009-11-11 04:24:54,118 INFO
> > > > > org.apache.zookeeper.ClientCnxn:
> > > > > >> > > >> Attempting
> > > > > >> > > >> >> > connection to server superpyxis0017/
> > > 192.168.100.131:2181
> > > > > >> > > >> >> > 2009-11-11 04:24:54,118 INFO
> > > > > org.apache.zookeeper.ClientCnxn:
> > > > > >> > > Priming
> > > > > >> > > >> >> > connection to
> > java.nio.channels.SocketChannel[connected
> > > > > >> local=/
> > > > > >> > > >> >> > 192.168.100.131:46903 remote=superpyxis0017/
> > > > > >> 192.168.100.131:2181
> > > > > >> > ]
> > > > > >> > > >> >> > 2009-11-11 04:24:54,118 INFO
> > > > > org.apache.zookeeper.ClientCnxn:
> > > > > >> > > Server
> > > > > >> > > >> >> > connection successful
> > > > > >> > > >> >> > 2009-11-11 04:24:54,118 WARN
> > > > > org.apache.zookeeper.ClientCnxn:
> > > > > >> > > >> Exception
> > > > > >> > > >> >> > closing session 0x24dfd90c810002 to
> > > > > >> > > >> sun.nio.ch.SelectionKeyImpl@2dd78ccf
> > > > > >> > > >> >> > java.io.IOException: Read error rc = -1
> > > > > >> > > >> java.nio.DirectByteBuffer[pos=0
> > > > > >> > > >> >> > lim=4 cap=4]
> > > > > >> > > >> >> > at
> > > > > >> > > >> >> >
> > > > > >> > >
> > > > org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
> > > > > >> > > >> >> > at
> > > > > >> > > >> >> >
> > > > > >> >
> > > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
> > > > > >> > > >> >> > 2009-11-11 04:24:54,118 WARN
> > > > > org.apache.zookeeper.ClientCnxn:
> > > > > >> > > Ignoring
> > > > > >> > > >> >> > exception during shutdown input
> > > > > >> > > >> >> > java.net.SocketException: Transport endpoint is not
> > > > > connected
> > > > > >> > > >> >> > at
> sun.nio.ch.SocketChannelImpl.shutdown(Native
> > > > > Method)
> > > > > >> > > >> >> > at
> > > > > >> > > >> >> >
> > > > > >> > >
> > > > >
> > sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
> > > > > >> > > >> >> > at
> > > > > >> > > >>
> > > sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> > > > > >> > > >> >> > at
> > > > > >> > > >> >> >
> > > > > >> > > >>
> > > > > >> >
> > > > >
> > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> > > > > >> > > >> >> > at
> > > > > >> > > >> >> >
> > > > > >> >
> > > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> > > > > >> > > >> >> >
> > > > > >> > > >> >> >
> > > > > >> > > >> >> >
> > > > > >> > > >> >> >
> > > > > >> > > >> >> >
> > > > > >> > > >> >> > On Wed, Nov 11, 2009 at 1:40 PM, Jean-Daniel Cryans
> <
> > > > > >> > > >> jdcryans@apache.org
> > > > > >> > > >> >> >wrote:
> > > > > >> > > >> >> >
> > > > > >> > > >> >> >> I'm really not sure exactly what the situation is
> > with
> > > > the
> > > > > >> > > >> >> >> java.nio.BufferUnderflowException, but what do you
> > mean
> > > > by
> > > > > >> > > >> >> >> "regionserver disconnection"? Could you give more
> > > > details?
> > > > > >> > > >> >> >>
> > > > > >> > > >> >> >> Thx,
> > > > > >> > > >> >> >>
> > > > > >> > > >> >> >> J-D
> > > > > >> > > >> >> >>
> > > > > >> > > >> >> >> On Wed, Nov 11, 2009 at 10:34 AM, Zhenyu Zhong <
> > > > > >> > > >> zhongresearch@gmail.com
> > > > > >> > > >> >> >
> > > > > >> > > >> >> >> wrote:
> > > > > >> > > >> >> >> > I searched around and found HBASE-1645 "HBase
> could
> > > be
> > > > a
> > > > > >> > little
> > > > > >> > > >> more
> > > > > >> > > >> >> >> dainty
> > > > > >> > > >> >> >> > when zk is pulled out from under it"
> > > > > >> > > >> >> >> >
> > > > > >> > > >> >> >> > I am wondering whether my problem is related to
> > this
> > > > > >> > HBase-1645.
> > > > > >> > > Or
> > > > > >> > > >> >> >> probably
> > > > > >> > > >> >> >> > has nothing to do with it?
> > > > > >> > > >> >> >> >
> > > > > >> > > >> >> >> > thanks
> > > > > >> > > >> >> >> >
> > > > > >> > > >> >> >> >
> > > > > >> > > >> >> >> > On Wed, Nov 11, 2009 at 12:02 PM, Zhenyu Zhong <
> > > > > >> > > >> >> zhongresearch@gmail.com
> > > > > >> > > >> >> >> >wrote:
> > > > > >> > > >> >> >> >
> > > > > >> > > >> >> >> >> Hi,
> > > > > >> > > >> >> >> >>
> > > > > >> > > >> >> >> >> I have been monitoring the system resources such
> > as
> > > > > memory
> > > > > >> > Heap
> > > > > >> > > >> usage
> > > > > >> > > >> >> >> for
> > > > > >> > > >> >> >> >> Regionserver, load average, network bandwidth,
> > etc.
> > > > > >> > > >> >> >> >>
> > > > > >> > > >> >> >> >> I experienced a disconnection last night. When I
> > > > checked
> > > > > >> the
> > > > > >> > > >> memory
> > > > > >> > > >> >> Heap
> > > > > >> > > >> >> >> >> for Regionserver, it was 2336 which is way below
> > the
> > > > max
> > > > > >> Heap
> > > > > >> > > >> 4096. I
> > > > > >> > > >> >> >> don't
> > > > > >> > > >> >> >> >> think it is Regionserver problem. The requests
> > that
> > > > the
> > > > > >> > > >> regionserver
> > > > > >> > > >> >> got
> > > > > >> > > >> >> >> was
> > > > > >> > > >> >> >> >> about 126 which doesn't seem to be high. The
> load
> > > > > average
> > > > > >> is
> > > > > >> > > kind
> > > > > >> > > >> of
> > > > > >> > > >> >> >> high
> > > > > >> > > >> >> >> >> >100. (but I saw disconnections while the load
> > > average
> > > > > is
> > > > > >> > > around
> > > > > >> > > >> 30).
> > > > > >> > > >> >> >> The
> > > > > >> > > >> >> >> >> bandwidth is not a problem.
> > > > > >> > > >> >> >> >>
> > > > > >> > > >> >> >> >> However, I found error messages from the
> > zookeeper.
> > > > > Please
> > > > > >> > see
> > > > > >> > > the
> > > > > >> > > >> >> >> >> following error messages. There are some
> > > > > >> > > >> >> >> >> BufferUnderflowException error, IOException
> > > error,etc.
> > > > > >> > > >> >> >> >> May I ask what might be the causes that generate
> > > these
> > > > > >> error
> > > > > >> > > >> >> messages?
> > > > > >> > > >> >> >> >>
> > > > > >> > > >> >> >> >> Many thanks
> > > > > >> > > >> >> >> >> zhenyu
> > > > > >> > > >> >> >> >>
> > > > > >> > > >> >> >> >>
> > > > > >> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
> > > > > >> > > org.apache.zookeeper.server.Request:
> > > > > >> > > >> >> >> Ignoring
> > > > > >> > > >> >> >> >> exception during toString
> > > > > >> > > >> >> >> >> java.nio.BufferUnderflowException
> > > > > >> > > >> >> >> >> at
> > > > > >> > java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
> > > > > >> > > >> >> >> >> at
> > > > java.nio.ByteBuffer.get(ByteBuffer.java:675)
> > > > > >> > > >> >> >> >> at
> > > > > >> > > >> >> >>
> > > > > >> org.apache.zookeeper.server.Request.toString(Request.java:199)
> > > > > >> > > >> >> >> >> at
> > > java.lang.String.valueOf(String.java:2827)
> > > > > >> > > >> >> >> >> at
> > > > > >> > > java.lang.StringBuilder.append(StringBuilder.java:115)
> > > > > >> > > >> >> >> >> at
> > > > > >> > > >> >> >> >>
> > > > > >> > > >> >> >>
> > > > > >> > > >> >>
> > > > > >> > > >>
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.zookeeper.server.quorum.CommitProcessor.processRequest(CommitProcessor.java:167)
> > > > > >> > > >> >> >> >> at
> > > > > >> > > >> >> >> >>
> > > > > >> > > >> >> >>
> > > > > >> > > >> >>
> > > > > >> > > >>
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.zookeeper.server.quorum.FollowerRequestProcessor.run(FollowerRequestProcessor.java:68)
> > > > > >> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
> > > > > >> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
> > > > > >> > > >> >> >> >> Exception causing close of session
> > 0x424dfd908c50009
> > > > > >> > > >> >> >> >> due to java.io.IOException: Read error
> > > > > >> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
> > > > > >> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
> > > > > >> > > >> >> >> >> IOException stack trace
> > > > > >> > > >> >> >> >> java.io.IOException: Read error
> > > > > >> > > >> >> >> >> at
> > > > > >> > > >> >> >> >>
> > > > > >> > > >> >>
> > > > > >> > >
> > > > >
> > org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:372)
> > > > > >> > > >> >> >> >> at
> > > > > >> > > >> >> >> >>
> > > > > >> > > >> >> >>
> > > > > >> > > >> >>
> > > > > >> > > >>
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:221)
> > > > > >> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
> > > > > >> > > >> >> >> >>
> > org.apache.zookeeper.server.quorum.CommitProcessor:
> > > > > >> > Processing
> > > > > >> > > >> >> request::
> > > > > >> > > >> >> >> >> sessionid:0x424dfd908c50009 type:setWatches
> > > > > >> > > >> cxid:0xfffffffffffffff8
> > > > > >> > > >> >> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
> > > > > >> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
> > > > > >> > > org.apache.zookeeper.server.Request:
> > > > > >> > > >> >> >> Ignoring
> > > > > >> > > >> >> >> >> exception during toString
> > > > > >> > > >> >> >> >> java.nio.BufferUnderflowException
> > > > > >> > > >> >> >> >> at
> > > > > >> > java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
> > > > > >> > > >> >> >> >> at
> > > > java.nio.ByteBuffer.get(ByteBuffer.java:675)
> > > > > >> > > >> >> >> >> at
> > > > > >> > > >> >> >>
> > > > > >> org.apache.zookeeper.server.Request.toString(Request.java:199)
> > > > > >> > > >> >> >> >> at
> > > java.lang.String.valueOf(String.java:2827)
> > > > > >> > > >> >> >> >> at
> > > > > >> > > java.lang.StringBuilder.append(StringBuilder.java:115)
> > > > > >> > > >> >> >> >> at
> > > > > >> > > >> >> >> >>
> > > > > >> > > >> >> >>
> > > > > >> > > >> >>
> > > > > >> > > >>
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:74)
> > > > > >> > > >> >> >> >> at
> > > > > >> > > >> >> >> >>
> > > > > >> > > >> >> >>
> > > > > >> > > >> >>
> > > > > >> > > >>
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)
> > > > > >> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
> > > > > >> > > >> >> >> >>
> org.apache.zookeeper.server.FinalRequestProcessor:
> > > > > >> Processing
> > > > > >> > > >> >> request::
> > > > > >> > > >> >> >> >> sessionid:0x424dfd908c50009 type:setWatches
> > > > > >> > > >> cxid:0xfffffffffffffff8
> > > > > >> > > >> >> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
> > > > > >> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
> > > > > >> > > org.apache.zookeeper.server.Request:
> > > > > >> > > >> >> >> Ignoring
> > > > > >> > > >> >> >> >> exception during toString
> > > > > >> > > >> >> >> >> java.nio.BufferUnderflowException
> > > > > >> > > >> >> >> >> at
> > > > > >> > java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
> > > > > >> > > >> >> >> >> at
> > > > java.nio.ByteBuffer.get(ByteBuffer.java:675)
> > > > > >> > > >> >> >> >> at
> > > > > >> > > >> >> >>
> > > > > >> org.apache.zookeeper.server.Request.toString(Request.java:199)
> > > > > >> > > >> >> >> >> at
> > > > > >> > > >> >> >> >>
> > > > > >> > > >>
> > > > > >>
> > > org.apache.log4j.or.DefaultRenderer.doRender(DefaultRenderer.java:36)
> > > > > >> > > >> >> >> >> at
> > > > > >> > > >> >> >> >>
> > > > > >> > >
> > > org.apache.log4j.or.RendererMap.findAndRender(RendererMap.java:80)
> > > > > >> > > >> >> >> >> at
> > > > > >> > > >> >> >> >>
> > > > > >> > > >> >> >>
> > > > > >> > > >> >>
> > > > > >> > > >>
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.log4j.spi.LoggingEvent.getRenderedMessage(LoggingEvent.java:362)
> > > > > >> > > >> >> >> >> at
> > > > > >> > > >> >> >> >>
> > > > > >> > > >> >> >>
> > > > > >> > > >> >>
> > > > > >> > > >>
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.log4j.helpers.PatternParser$BasicPatternConverter.convert(PatternParser.java:403)
> > > > > >> > > >> >> >> >> at
> > > > > >> > > >> >> >> >>
> > > > > >> > > >> >> >>
> > > > > >> > > >> >>
> > > > > >> > > >>
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.log4j.helpers.PatternConverter.format(PatternConverter.java:65)
> > > > > >> > > >> >> >> >> at
> > > > > >> > > >> >>
> > > > org.apache.log4j.PatternLayout.format(PatternLayout.java:502)
> > > > > >> > > >> >> >> >> at
> > > > > >> > > >> >> >> >>
> > > > > >> > >
> > > org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:302)
> > > > > >> > > >> >> >> >> at
> > > > > >> > > >> >> >> >>
> > > > > >> > > >> >> >>
> > > > > >> > > >> >>
> > > > > >> > > >>
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.log4j.DailyRollingFileAppender.subAppend(DailyRollingFileAppender.java:359)
> > > > > >> > > >> >> >> >> at
> > > > > >> > > >> >> >>
> > > > > >> org.apache.log4j.WriterAppender.append(WriterAppender.java:160)
> > > > > >> > > >> >> >> >> at
> > > > > >> > > >> >> >> >>
> > > > > >> > > >>
> > > > > >>
> > > org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
> > > > > >> > > >> >> >> >> at
> > > > > >> > > >> >> >> >>
> > > > > >> > > >> >> >>
> > > > > >> > > >> >>
> > > > > >> > > >>
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
> > > > > >> > > >> >> >> >> at
> > > > > >> > > >>
> org.apache.log4j.Category.callAppenders(Category.java:206)
> > > > > >> > > >> >> >> >> at
> > > > > >> > > org.apache.log4j.Category.forcedLog(Category.java:391)
> > > > > >> > > >> >> >> >> at
> > > > > >> org.apache.log4j.Category.debug(Category.java:260)
> > > > > >> > > >> >> >> >> at
> > > > > >> > > >> >> >> >>
> > > > > >> > > >> >> >>
> > > > > >> > > >> >>
> > > > > >> > > >>
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:147)
> > > > > >> > > >> >> >> >> at
> > > > > >> > > >> >> >> >>
> > > > > >> > > >> >> >>
> > > > > >> > > >> >>
> > > > > >> > > >>
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)
> > > > > >> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
> > > > > >> > > org.apache.zookeeper.server.Request:
> > > > > >> > > >> >> >> Ignoring
> > > > > >> > > >> >> >> >> exception during toString
> > > > > >> > > >> >> >> >> sessionid:0x424dfd908c50009 type:setWatches
> > > > > >> > > >> cxid:0xfffffffffffffff8
> > > > > >> > > >> >> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
> > > > > >> > > >> >> >> >> 2009-11-11 04:29:00,810 INFO
> > > > > >> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
> > > > > >> > > >> >> >> >> closing session:0x424dfd908c50009 NIOServerCnxn:
> > > > > >> > > >> >> >> >> java.nio.channels.SocketChannel[connected
> local=/
> > > > > >> > > >> >> 192.168.100.131:2181
> > > > > >> > > >> >> >> remote=/
> > > > > >> > > >> >> >> >> 192.168.100.131:35500]
> > > > > >> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
> > > > > >> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
> > > > > >> > > >> >> >> >> ignoring exception during input shutdown
> > > > > >> > > >> >> >> >> java.net.SocketException: Transport endpoint is
> > not
> > > > > >> connected
> > > > > >> > > >> >> >> >> at
> > > > sun.nio.ch.SocketChannelImpl.shutdown(Native
> > > > > >> > Method)
> > > > > >> > > >> >> >> >> at
> > > > > >> > > >> >> >> >>
> > > > > >> > > >> >>
> > > > > >> > >
> > > > >
> > sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
> > > > > >> > > >> >> >> >> at
> > > > > >> > > >> >> >>
> > > > > >> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> > > > > >> > > >> >> >> >> at
> > > > > >> > > >> >> >> >>
> > > > > >> > > >> >>
> > > > > >> > >
> > > > > >>
> > > >
> org.apache.zookeeper.server.NIOServerCnxn.close(NIOServerCnxn.java:848)
> > > > > >> > > >> >> >> >> at
> > > > > >> > > >> >> >> >>
> > > > > >> > > >> >>
> > > > > >> > >
> > > > >
> > org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:498)
> > > > > >> > > >> >> >> >> at
> > > > > >> > > >> >> >> >>
> > > > > >> > > >> >> >>
> > > > > >> > > >> >>
> > > > > >> > > >>
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:221)
> > > > > >> > > >> >> >> >>
> > > > > >> > > >> >> >> >
> > > > > >> > > >> >> >>
> > > > > >> > > >> >> >
> > > > > >> > > >> >>
> > > > > >> > > >> >
> > > > > >> > > >>
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: regionserver disconnection
Posted by Zhenyu Zhong <zh...@gmail.com>.
I can switch the boxes that run zookeeper with the ones that run
regionservers.
I will see the results later.
FYI. The node does have the 10 minutes zookeeper.session.timeout value in
place.
thanks
zhenyu
On Thu, Nov 12, 2009 at 3:21 PM, stack <st...@duboce.net> wrote:
> On Thu, Nov 12, 2009 at 11:50 AM, Zhenyu Zhong <zhongresearch@gmail.com
> >wrote:
>
> >
> > In my cluster, half of the cluster have 2 disks 400GB each per machine,
> and
> > half of the cluster have 6 disks per machine. Maybe we should run
> > zookeeper
> > on the machines with 2 disks and RS on machines with 6 disks?
> >
> > That would make most sense only in the below, it looks like the RS that
> had
> issue had 4 disks?
>
>
>
> > BTW, the 10 minutes zookeeper.session.timeout has been set during the
> > experiment.
> >
>
> And for sure this node had it in place?
> St.Ack
>
>
>
> >
> > thanks
> > zhenyu
> >
> > On Thu, Nov 12, 2009 at 2:08 PM, stack <st...@duboce.net> wrote:
> >
> > > On Thu, Nov 12, 2009 at 8:40 AM, Zhenyu Zhong <zhongresearch@gmail.com
> > > >wrote:
> > >
> > >
> > > > Though I experienced 2 regionserver disconnection this morning, it
> > looks
> > > > better from the regionserver log. (please see the following log)
> > > >
> > > >
> > > > http://pastebin.com/m496dbfae
> > > >
> > > >
> > > > I checked diskIO for one of the regionserver(192.168.100.116) during
> > the
> > > > time it disconnected.
> > > >
> > > > Time: 03:04:58 AM
> > > > Device: tps Blk_read/s Blk_wrtn/s Blk_read
> Blk_wrtn
> > > > sda 105.31 5458.83 19503.64 9043873239
> 32312473676
> > > > sda1 2.90 6.64 99.25 10993934
> 164433464
> > > > sda2 1.72 23.76 30.25 39365817
> 50124008
> > > > sda3 0.30 0.38 3.58 624930
> 5923000
> > > > sda4 100.39 5428.06 19370.56 8992888270
> 32091993204
> > > >
> > > >
> > > Is this high for you? 20k blocks/second would seem to be high but its
> > one
> > > disk only and its not being shared by zk anymore so shouldn't matter?
> > >
> > >
> > > > I also checked the zookeeper quorum server that the regionserver
> tried
> > to
> > > > connect according to the log. However, I don't see 192.168.100.116 in
> > the
> > > > client list of the zookeeper quorum server that the regionserver
> tried
> > to
> > > > connect.
> > > > Would that be a problem?
> > > >
> > > >
> > > Is that because the ephemeral node for the regionserver had evaporated?
> > > Lost
> > > its lease w/ zk by the time you went to look?
> > >
> > >
> > >
> > >
> > > > Thu Nov 12 15:04:52 UTC 2009
> > > > Zookeeper version: 3.2.1-808558, built on 08/27/2009 18:48 GMT
> > > > Clients:
> > > > /192.168.100.127:43045[1](queued=0,recved=26,sent=0)
> > > > /192.168.100.131:39091[1](queued=0,recved=964,sent=0)
> > > > /192.168.100.124:35961[1](queued=0,recved=3266,sent=0)
> > > > /192.168.100.123:47935[1](queued=0,recved=233,sent=0)
> > > > /192.168.100.125:46931[1](queued=0,recved=2,sent=0)
> > > > /192.168.100.118:54924[1](queued=0,recved=295,sent=0)
> > > > /192.168.100.118:41390[1](queued=0,recved=2290,sent=0)
> > > > /192.168.100.136:42243[1](queued=0,recved=0,sent=0)
> > > >
> > > > Latency min/avg/max: 0/17/6333
> > > > Received: 47111
> > > > Sent: 0
> > > > Outstanding: 0
> > > > Zxid: 0x77000083f4
> > > > Mode: leader
> > > > Node count: 23
> > > >
> > >
> > >
> > > That 6 second maximum latency is pretty bad but should be well within
> the
> > > zk
> > > session timeout.
> > >
> > > So, problem is likely on the zk client side of the session; i.e. in the
> > RS.
> > > You could enable GC logging as J-D suggested to see if you have any big
> > > pauses, pauses > zk session timeout.
> > >
> > > When the RS went down, it didn't look too heavily loaded:
> > >
> > >
> > > 1. 2009-11-12 15:04:52,830 INFO
> > > org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics:
> > > request=1.5166667, regions=322, stores=657, storefiles=631,
> > > storefileIndexSize=61, memstoreSize=1472, usedHeap=2819,
> maxHeap=4079,
> > > blockCacheSize=658110960, blockCacheFree=197395984,
> > blockCacheCount=9903,
> > > blockCacheHitRatio=99
> > >
> > > Its serving a few reads? The number of store files seems fine. Not
> too
> > > much memory used.
> > >
> > > Looking at the logs, I see the Lease Still Held exception. This
> happens
> > > when the RS does its regular report to the master but the master thinks
> > the
> > > RS has since restarted. It'll think this probably because it noticed
> > that
> > > the RS's znode in zk had gone away and it considered the RS dead.
> > >
> > > Looking too at your logs I see this gap in the zk pinging:
> > >
> > >
> > > 1. 2009-11-12 15:03:39,325 DEBUG org.apache.zookeeper.ClientCnxn: Got
> > > ping response for sessionid:0x224e55436ad0004 after 0ms
> > > 2. 2009-11-12 15:03:43,113 DEBUG org.apache.zookeeper.ClientCnxn: Got
> > > ping response for sessionid:0x24e55436a0007d after 0ms
> > >
> > > Where in the lines above it, its reporting about every ten seconds,
> here
> > > there is a big gap.
> > >
> > > Do you have ganglia or something that will let you look more into what
> > was
> > > happening on this machine around this time? Is the machine OK? It
> looks
> > > lightly loaded and you have your cluster nicely laid out. Something
> odd
> > is
> > > going on. What about things like the write speed to disk? In the past
> > > strange issues have been explained by incorrectly set BIOS which made
> > disks
> > > run at 1/100th of their proper speed.
> > >
> > > St.Ack
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > >
> > > > Best,
> > > > zhenyu
> > > >
> > > >
> > > > On Wed, Nov 11, 2009 at 3:58 PM, Zhenyu Zhong <
> zhongresearch@gmail.com
> > > > >wrote:
> > > >
> > > > > Stack
> > > > >
> > > > > I am very appreciated for your comments.
> > > > > I will use the zookeeper monitoring script on my cluster and let it
> > run
> > > > > overnight to see the result.
> > > > > I will follow up the thread when I get anything.
> > > > >
> > > > > thanks
> > > > > zhenyu
> > > > >
> > > > >
> > > > > On Wed, Nov 11, 2009 at 3:52 PM, stack <st...@duboce.net> wrote:
> > > > >
> > > > >> I see these in your log too:
> > > > >>
> > > > >>
> > > > >> 1. 2009-11-11 04:27:19,018 DEBUG
> org.apache.zookeeper.ClientCnxn:
> > > Got
> > > > >> ping response for sessionid:0x424dfd908c50009 after 4544ms
> > > > >> 2. 2009-11-11 04:27:19,018 DEBUG
> org.apache.zookeeper.ClientCnxn:
> > > Got
> > > > >> ping response for sessionid:0x24dfd90c810002 after 4548ms
> > > > >> 3. 2009-11-11 04:27:43,960 DEBUG
> org.apache.zookeeper.ClientCnxn:
> > > Got
> > > > >> ping response for sessionid:0x424dfd908c50009 after 9030ms
> > > > >> 4. 2009-11-11 04:27:43,960 DEBUG
> org.apache.zookeeper.ClientCnxn:
> > > Got
> > > > >> ping response for sessionid:0x24dfd90c810002 after 9030ms
> > > > >> 5. 2009-11-11 04:27:48,830 DEBUG
> org.apache.zookeeper.ClientCnxn:
> > > Got
> > > > >> ping response for sessionid:0x424dfd908c50009 after 0ms
> > > > >> 6. 2009-11-11 04:27:48,830 DEBUG
> org.apache.zookeeper.ClientCnxn:
> > > Got
> > > > >> ping response for sessionid:0x24dfd90c810002 after 0ms
> > > > >>
> > > > >> Which would indicate lag talking to zk too.
> > > > >>
> > > > >> Appending HLog also seems to get steadily worse.
> > > > >>
> > > > >> I also see these:
> > > > >>
> > > > >>
> > > > >> 1. 2009-11-11 04:27:01,233 WARN org.apache.zookeeper.ClientCnxn:
> > > > >> Exception closing session 0x24dfd90c810002 to
> > > > >> sun.nio.ch.SelectionKeyImpl@5e16a7e3
> > > > >> 2. java.io.IOException: Read error rc = -1
> > > > >> java.nio.DirectByteBuffer[pos=0 lim=4 cap=4]
> > > > >> 3. at
> > > > >>
> > > org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
> > > > >> 4. at
> > > > >>
> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
> > > > >>
> > > > >>
> > > > >> Were all 5 quorum members participating? Check by doing "echo
> > stat|nc
> > > > >> QUORUM_HOST port" for each member of quorum.... the below might
> > help:
> > > > >>
> > > > >> members="1,2,3,4,5"
> > > > >> while true; do date; for i in $members; do echo $i; echo stat | nc
> > $i
> > > > >> 2181;
> > > > >> done; sleep 10; done
> > > > >>
> > > > >> We need to improve our zk_dump. It just reports quorum members...
> > not
> > > > if
> > > > >> they are actually participating or not. The above will tell you
> if
> > > dead
> > > > >> or
> > > > >> alive.
> > > > >>
> > > > >> Please report back after checking above and following the
> > Jean-Daniel
> > > > >> prescription.
> > > > >>
> > > > >> Thanks,
> > > > >> St.Ack
> > > > >>
> > > > >>
> > > > >>
> > > > >> On Wed, Nov 11, 2009 at 12:29 PM, Zhenyu Zhong <
> > > zhongresearch@gmail.com
> > > > >> >wrote:
> > > > >>
> > > > >> > Thanks a lot for the help.
> > > > >> >
> > > > >> > I understand that the change of disk configurations should be
> done
> > > > >> > progressively such that HDFS can balance and recover from a disk
> > > going
> > > > >> > away.
> > > > >> >
> > > > >> > The data we are trying to import into HBase is about 200million
> > > rows,
> > > > >> each
> > > > >> > row is about hundreds of bytes. Basically, we read in a text
> file
> > > from
> > > > >> > HDFS,
> > > > >> > then for each row in the text file, we fetch data from HBase
> > table,
> > > > we
> > > > >> > compare the column value and update the latest value back into
> > > HBase.
> > > > I
> > > > >> was
> > > > >> > suggested to use CheckAndPut. Anyway this is the common usecase
> > on
> > > my
> > > > >> > side.
> > > > >> >
> > > > >> >
> > > > >> > Many thanks
> > > > >> > zhenyu
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > On Wed, Nov 11, 2009 at 3:03 PM, Jean-Daniel Cryans <
> > > > >> jdcryans@apache.org
> > > > >> > >wrote:
> > > > >> >
> > > > >> > > 6 disks, 19 machines and you see those kind of issues? I'm
> quite
> > > > >> > > surprised and I want to learn more about what you trying to
> to.
> > > > >> > >
> > > > >> > > But first, if hbase is managing ZK for you then you can change
> > the
> > > > >> > > value of hbase.zookeeper.property.dataDir so that it points to
> > its
> > > > own
> > > > >> > > mount. Make sure you also change dfs.data.dir in
> > > conf/hdfs-site.xml
> > > > >> > > (hadoop stuff) so that it stops pointing there (you should
> have
> > 6
> > > > >> > > values there at the moment if you configured it correctly).
> > Doing
> > > > this
> > > > >> > > could result in data loss if you do it all at once so do the
> > > change
> > > > >> > > progressively. This also means that you may have to keep a
> > > seperate
> > > > >> > > HDFS configuration for those particular nodes. Please ask more
> > > > >> > > questions if it's not very clear.
> > > > >> > >
> > > > >> > > WRT to dedicating nodes to Zookeeeper, it makes sense only if
> > you
> > > > have
> > > > >> > > a lot of nodes but in your case, unless there's also CPU
> > > contention,
> > > > >> > > you're good with only a dedicated disk.
> > > > >> > >
> > > > >> > > Now for what you are trying to do, is this a massive import
> that
> > > > goes
> > > > >> > > over many hours? How big is each row? I'd love to get a lot of
> > > > details
> > > > >> > > to help you optimize your cluster for your particular problem.
> > > > >> > >
> > > > >> > > Thx,
> > > > >> > >
> > > > >> > > J-D
> > > > >> > >
> > > > >> > > On Wed, Nov 11, 2009 at 11:51 AM, Zhenyu Zhong <
> > > > >> zhongresearch@gmail.com>
> > > > >> > > wrote:
> > > > >> > > > J-D,
> > > > >> > > >
> > > > >> > > > Thanks a lot for the insightful comments.
> > > > >> > > > We are running 5 zookeeper quorums now. On these 5 quorum
> > > servers,
> > > > >> we
> > > > >> > > also
> > > > >> > > > run datanode, tasktracker and regionserver.
> > > > >> > > > Now we have 6 disks per machine.
> > > > >> > > > If it is the disk IO saturation, is there anyway to
> configure
> > > > >> zookeeper
> > > > >> > > to
> > > > >> > > > write to a seperate disk which is not the same one used by
> > HDFS?
> > > > >> > > > Or shall we run zookeeper quorums on different nodes other
> > than
> > > > >> HDFS?
> > > > >> > > >
> > > > >> > > > Are HLogs generated by Regionservers? Would it alleviate the
> > > issue
> > > > >> if
> > > > >> > we
> > > > >> > > use
> > > > >> > > > the first 14 for regionserver, and the rest of 5 for
> > > zookeeper?(In
> > > > >> that
> > > > >> > > > sense, the node running zookeeper quorum won't run
> > regionserver)
> > > > >> > > >
> > > > >> > > > I am very appreciated for your comments.
> > > > >> > > >
> > > > >> > > > Best,
> > > > >> > > > zhenyu
> > > > >> > > >
> > > > >> > > >
> > > > >> > > >
> > > > >> > > > On Wed, Nov 11, 2009 at 2:30 PM, Jean-Daniel Cryans <
> > > > >> > jdcryans@apache.org
> > > > >> > > >wrote:
> > > > >> > > >
> > > > >> > > >> The problem seems to be that the IO is saturated. When you
> > see
> > > > >> this:
> > > > >> > > >>
> > > > >> > > >> #
> > > > >> > > >> 2009-11-11 04:28:10,308 WARN
> > > > >> > > >> org.apache.hadoop.hbase.regionserver.HLog: IPC Server
> handler
> > 9
> > > > on
> > > > >> > > >> 60021 took 6819ms appending an edit to hlog; editcount=370
> > > > >> > > >> #
> > > > >> > > >> 2009-11-11 04:28:10,308 WARN
> > > > >> > > >> org.apache.hadoop.hbase.regionserver.HLog: IPC Server
> handler
> > 2
> > > > on
> > > > >> > > >> 60021 took 4412ms appending an edit to hlog; editcount=371
> > > > >> > > >>
> > > > >> > > >> It's a sign that HDFS is slow. I guess that the zookeeper
> > > quorum
> > > > >> > > >> members are on those same nodes using the same disks? If
> so,
> > > > >> there's
> > > > >> > > >> the problem because Zookeeper needs to be able to commit
> > > > >> transactions
> > > > >> > > >> very fast and if your disks are already hogged then they
> have
> > > to
> > > > >> wait
> > > > >> > > >> and you can get disconnections. Do you still have 19 quorum
> > > > >> members?
> > > > >> > > >> How many disks do you have per machine?
> > > > >> > > >>
> > > > >> > > >> Thx
> > > > >> > > >>
> > > > >> > > >> J-D
> > > > >> > > >>
> > > > >> > > >> On Wed, Nov 11, 2009 at 11:22 AM, Zhenyu Zhong <
> > > > >> > zhongresearch@gmail.com
> > > > >> > > >
> > > > >> > > >> wrote:
> > > > >> > > >> > Please take a look at the following regionserver log.
> > > > >> > > >> > The error started to come out from 4:24am UTC to 4:29am
> > UTC.
> > > > >> > > >> >
> > > > >> > > >> > http://pastebin.com/m3717d064
> > > > >> > > >> >
> > > > >> > > >> > Thanks in advance.
> > > > >> > > >> >
> > > > >> > > >> > Best,
> > > > >> > > >> > zhenyu
> > > > >> > > >> >
> > > > >> > > >> >
> > > > >> > > >> > On Wed, Nov 11, 2009 at 2:01 PM, Jean-Daniel Cryans <
> > > > >> > > jdcryans@apache.org
> > > > >> > > >> >wrote:
> > > > >> > > >> >
> > > > >> > > >> >> Could you paste a bigger log trace (interested to see at
> > > least
> > > > >> all
> > > > >> > > 300
> > > > >> > > >> >> lines before that) in a pastebin?
> > > > >> > > >> >>
> > > > >> > > >> >> Thx,
> > > > >> > > >> >>
> > > > >> > > >> >> J-D
> > > > >> > > >> >>
> > > > >> > > >> >> On Wed, Nov 11, 2009 at 10:56 AM, Zhenyu Zhong <
> > > > >> > > zhongresearch@gmail.com
> > > > >> > > >> >
> > > > >> > > >> >> wrote:
> > > > >> > > >> >> > J-D,
> > > > >> > > >> >> >
> > > > >> > > >> >> > Thanks for the reply.
> > > > >> > > >> >> >
> > > > >> > > >> >> > The problem I have is one or more of the regionservers
> > > > >> disconnect
> > > > >> > > from
> > > > >> > > >> >> the
> > > > >> > > >> >> > cluster.
> > > > >> > > >> >> >
> > > > >> > > >> >> > I checked the resource usage including memory Heap
> usage
> > > for
> > > > >> > > >> >> regionserver,
> > > > >> > > >> >> > and found nothing suspicious. The regionserver heap
> > usage
> > > is
> > > > >> way
> > > > >> > > below
> > > > >> > > >> >> the
> > > > >> > > >> >> > maxHeap defined in jvm.
> > > > >> > > >> >> >
> > > > >> > > >> >> > So I checked zookeeper log, then I found the following
> > > > >> > > >> >> > BufferUnderflowException, eventually it leads to
> > > > >> SocketException:
> > > > >> > > >> >> Transport
> > > > >> > > >> >> > endpoint is not connected.
> > > > >> > > >> >> >
> > > > >> > > >> >> >
> > > > >> > > >> >> > I also checked the regionserver log. It looks like the
> > > > >> > regionserver
> > > > >> > > >> tried
> > > > >> > > >> >> to
> > > > >> > > >> >> > connect to other zookeeper quorum server. It connects
> > > > >> > successfully,
> > > > >> > > >> but
> > > > >> > > >> >> then
> > > > >> > > >> >> > it complained about SocketException: transport
> endpoint
> > is
> > > > not
> > > > >> > > >> connected.
> > > > >> > > >> >> >
> > > > >> > > >> >> > So both zookeeper log and regionserver log told me
> that
> > > the
> > > > >> > > transport
> > > > >> > > >> >> > endpoint is not connected. I don't know why.
> > > > >> > > >> >> >
> > > > >> > > >> >> > I would be very appreciated if you could point me to
> the
> > > > right
> > > > >> > > >> direction.
> > > > >> > > >> >> >
> > > > >> > > >> >> > Best,
> > > > >> > > >> >> >
> > > > >> > > >> >> >
> > > > >> > > >> >> > Regionserver log:
> > > > >> > > >> >> > 2009-11-11 04:24:53,718 INFO
> > > > org.apache.zookeeper.ClientCnxn:
> > > > >> > > >> Attempting
> > > > >> > > >> >> > connection to server superpyxis0018/
> > 192.168.100.132:2181
> > > > >> > > >> >> > 2009-11-11 04:24:53,718 INFO
> > > > org.apache.zookeeper.ClientCnxn:
> > > > >> > > Priming
> > > > >> > > >> >> > connection to
> java.nio.channels.SocketChannel[connected
> > > > >> local=/
> > > > >> > > >> >> > 192.168.100.131:43065 remote=superpyxis0018/
> > > > >> 192.168.100.132:2181
> > > > >> > ]
> > > > >> > > >> >> > 2009-11-11 04:24:53,718 INFO
> > > > org.apache.zookeeper.ClientCnxn:
> > > > >> > > Server
> > > > >> > > >> >> > connection successful
> > > > >> > > >> >> > 2009-11-11 04:24:53,768 INFO
> > > > >> > > >> >> > org.apache.hadoop.hbase.regionserver.HRegionServer:
> Got
> > > > >> ZooKeeper
> > > > >> > > >> event,
> > > > >> > > >> >> > state: Disconnected, type: None, path: null
> > > > >> > > >> >> > 2009-11-11 04:24:54,118 INFO
> > > > org.apache.zookeeper.ClientCnxn:
> > > > >> > > >> Attempting
> > > > >> > > >> >> > connection to server superpyxis0017/
> > 192.168.100.131:2181
> > > > >> > > >> >> > 2009-11-11 04:24:54,118 INFO
> > > > org.apache.zookeeper.ClientCnxn:
> > > > >> > > Priming
> > > > >> > > >> >> > connection to
> java.nio.channels.SocketChannel[connected
> > > > >> local=/
> > > > >> > > >> >> > 192.168.100.131:46903 remote=superpyxis0017/
> > > > >> 192.168.100.131:2181
> > > > >> > ]
> > > > >> > > >> >> > 2009-11-11 04:24:54,118 INFO
> > > > org.apache.zookeeper.ClientCnxn:
> > > > >> > > Server
> > > > >> > > >> >> > connection successful
> > > > >> > > >> >> > 2009-11-11 04:24:54,118 WARN
> > > > org.apache.zookeeper.ClientCnxn:
> > > > >> > > >> Exception
> > > > >> > > >> >> > closing session 0x24dfd90c810002 to
> > > > >> > > >> sun.nio.ch.SelectionKeyImpl@2dd78ccf
> > > > >> > > >> >> > java.io.IOException: Read error rc = -1
> > > > >> > > >> java.nio.DirectByteBuffer[pos=0
> > > > >> > > >> >> > lim=4 cap=4]
> > > > >> > > >> >> > at
> > > > >> > > >> >> >
> > > > >> > >
> > > org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
> > > > >> > > >> >> > at
> > > > >> > > >> >> >
> > > > >> >
> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
> > > > >> > > >> >> > 2009-11-11 04:24:54,118 WARN
> > > > org.apache.zookeeper.ClientCnxn:
> > > > >> > > Ignoring
> > > > >> > > >> >> > exception during shutdown input
> > > > >> > > >> >> > java.net.SocketException: Transport endpoint is not
> > > > connected
> > > > >> > > >> >> > at sun.nio.ch.SocketChannelImpl.shutdown(Native
> > > > Method)
> > > > >> > > >> >> > at
> > > > >> > > >> >> >
> > > > >> > >
> > > >
> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
> > > > >> > > >> >> > at
> > > > >> > > >>
> > sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> > > > >> > > >> >> > at
> > > > >> > > >> >> >
> > > > >> > > >>
> > > > >> >
> > > >
> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> > > > >> > > >> >> > at
> > > > >> > > >> >> >
> > > > >> >
> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> > > > >> > > >> >> >
> > > > >> > > >> >> >
> > > > >> > > >> >> >
> > > > >> > > >> >> >
> > > > >> > > >> >> >
> > > > >> > > >> >> > On Wed, Nov 11, 2009 at 1:40 PM, Jean-Daniel Cryans <
> > > > >> > > >> jdcryans@apache.org
> > > > >> > > >> >> >wrote:
> > > > >> > > >> >> >
> > > > >> > > >> >> >> I'm really not sure exactly what the situation is
> with
> > > the
> > > > >> > > >> >> >> java.nio.BufferUnderflowException, but what do you
> mean
> > > by
> > > > >> > > >> >> >> "regionserver disconnection"? Could you give more
> > > details?
> > > > >> > > >> >> >>
> > > > >> > > >> >> >> Thx,
> > > > >> > > >> >> >>
> > > > >> > > >> >> >> J-D
> > > > >> > > >> >> >>
> > > > >> > > >> >> >> On Wed, Nov 11, 2009 at 10:34 AM, Zhenyu Zhong <
> > > > >> > > >> zhongresearch@gmail.com
> > > > >> > > >> >> >
> > > > >> > > >> >> >> wrote:
> > > > >> > > >> >> >> > I searched around and found HBASE-1645 "HBase could
> > be
> > > a
> > > > >> > little
> > > > >> > > >> more
> > > > >> > > >> >> >> dainty
> > > > >> > > >> >> >> > when zk is pulled out from under it"
> > > > >> > > >> >> >> >
> > > > >> > > >> >> >> > I am wondering whether my problem is related to
> this
> > > > >> > HBase-1645.
> > > > >> > > Or
> > > > >> > > >> >> >> probably
> > > > >> > > >> >> >> > has nothing to do with it?
> > > > >> > > >> >> >> >
> > > > >> > > >> >> >> > thanks
> > > > >> > > >> >> >> >
> > > > >> > > >> >> >> >
> > > > >> > > >> >> >> > On Wed, Nov 11, 2009 at 12:02 PM, Zhenyu Zhong <
> > > > >> > > >> >> zhongresearch@gmail.com
> > > > >> > > >> >> >> >wrote:
> > > > >> > > >> >> >> >
> > > > >> > > >> >> >> >> Hi,
> > > > >> > > >> >> >> >>
> > > > >> > > >> >> >> >> I have been monitoring the system resources such
> as
> > > > memory
> > > > >> > Heap
> > > > >> > > >> usage
> > > > >> > > >> >> >> for
> > > > >> > > >> >> >> >> Regionserver, load average, network bandwidth,
> etc.
> > > > >> > > >> >> >> >>
> > > > >> > > >> >> >> >> I experienced a disconnection last night. When I
> > > checked
> > > > >> the
> > > > >> > > >> memory
> > > > >> > > >> >> Heap
> > > > >> > > >> >> >> >> for Regionserver, it was 2336 which is way below
> the
> > > max
> > > > >> Heap
> > > > >> > > >> 4096. I
> > > > >> > > >> >> >> don't
> > > > >> > > >> >> >> >> think it is Regionserver problem. The requests
> that
> > > the
> > > > >> > > >> regionserver
> > > > >> > > >> >> got
> > > > >> > > >> >> >> was
> > > > >> > > >> >> >> >> about 126 which doesn't seem to be high. The load
> > > > average
> > > > >> is
> > > > >> > > kind
> > > > >> > > >> of
> > > > >> > > >> >> >> high
> > > > >> > > >> >> >> >> >100. (but I saw disconnections while the load
> > average
> > > > is
> > > > >> > > around
> > > > >> > > >> 30).
> > > > >> > > >> >> >> The
> > > > >> > > >> >> >> >> bandwidth is not a problem.
> > > > >> > > >> >> >> >>
> > > > >> > > >> >> >> >> However, I found error messages from the
> zookeeper.
> > > > Please
> > > > >> > see
> > > > >> > > the
> > > > >> > > >> >> >> >> following error messages. There are some
> > > > >> > > >> >> >> >> BufferUnderflowException error, IOException
> > error,etc.
> > > > >> > > >> >> >> >> May I ask what might be the causes that generate
> > these
> > > > >> error
> > > > >> > > >> >> messages?
> > > > >> > > >> >> >> >>
> > > > >> > > >> >> >> >> Many thanks
> > > > >> > > >> >> >> >> zhenyu
> > > > >> > > >> >> >> >>
> > > > >> > > >> >> >> >>
> > > > >> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
> > > > >> > > org.apache.zookeeper.server.Request:
> > > > >> > > >> >> >> Ignoring
> > > > >> > > >> >> >> >> exception during toString
> > > > >> > > >> >> >> >> java.nio.BufferUnderflowException
> > > > >> > > >> >> >> >> at
> > > > >> > java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
> > > > >> > > >> >> >> >> at
> > > java.nio.ByteBuffer.get(ByteBuffer.java:675)
> > > > >> > > >> >> >> >> at
> > > > >> > > >> >> >>
> > > > >> org.apache.zookeeper.server.Request.toString(Request.java:199)
> > > > >> > > >> >> >> >> at
> > java.lang.String.valueOf(String.java:2827)
> > > > >> > > >> >> >> >> at
> > > > >> > > java.lang.StringBuilder.append(StringBuilder.java:115)
> > > > >> > > >> >> >> >> at
> > > > >> > > >> >> >> >>
> > > > >> > > >> >> >>
> > > > >> > > >> >>
> > > > >> > > >>
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.zookeeper.server.quorum.CommitProcessor.processRequest(CommitProcessor.java:167)
> > > > >> > > >> >> >> >> at
> > > > >> > > >> >> >> >>
> > > > >> > > >> >> >>
> > > > >> > > >> >>
> > > > >> > > >>
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.zookeeper.server.quorum.FollowerRequestProcessor.run(FollowerRequestProcessor.java:68)
> > > > >> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
> > > > >> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
> > > > >> > > >> >> >> >> Exception causing close of session
> 0x424dfd908c50009
> > > > >> > > >> >> >> >> due to java.io.IOException: Read error
> > > > >> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
> > > > >> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
> > > > >> > > >> >> >> >> IOException stack trace
> > > > >> > > >> >> >> >> java.io.IOException: Read error
> > > > >> > > >> >> >> >> at
> > > > >> > > >> >> >> >>
> > > > >> > > >> >>
> > > > >> > >
> > > >
> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:372)
> > > > >> > > >> >> >> >> at
> > > > >> > > >> >> >> >>
> > > > >> > > >> >> >>
> > > > >> > > >> >>
> > > > >> > > >>
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:221)
> > > > >> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
> > > > >> > > >> >> >> >>
> org.apache.zookeeper.server.quorum.CommitProcessor:
> > > > >> > Processing
> > > > >> > > >> >> request::
> > > > >> > > >> >> >> >> sessionid:0x424dfd908c50009 type:setWatches
> > > > >> > > >> cxid:0xfffffffffffffff8
> > > > >> > > >> >> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
> > > > >> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
> > > > >> > > org.apache.zookeeper.server.Request:
> > > > >> > > >> >> >> Ignoring
> > > > >> > > >> >> >> >> exception during toString
> > > > >> > > >> >> >> >> java.nio.BufferUnderflowException
> > > > >> > > >> >> >> >> at
> > > > >> > java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
> > > > >> > > >> >> >> >> at
> > > java.nio.ByteBuffer.get(ByteBuffer.java:675)
> > > > >> > > >> >> >> >> at
> > > > >> > > >> >> >>
> > > > >> org.apache.zookeeper.server.Request.toString(Request.java:199)
> > > > >> > > >> >> >> >> at
> > java.lang.String.valueOf(String.java:2827)
> > > > >> > > >> >> >> >> at
> > > > >> > > java.lang.StringBuilder.append(StringBuilder.java:115)
> > > > >> > > >> >> >> >> at
> > > > >> > > >> >> >> >>
> > > > >> > > >> >> >>
> > > > >> > > >> >>
> > > > >> > > >>
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:74)
> > > > >> > > >> >> >> >> at
> > > > >> > > >> >> >> >>
> > > > >> > > >> >> >>
> > > > >> > > >> >>
> > > > >> > > >>
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)
> > > > >> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
> > > > >> > > >> >> >> >> org.apache.zookeeper.server.FinalRequestProcessor:
> > > > >> Processing
> > > > >> > > >> >> request::
> > > > >> > > >> >> >> >> sessionid:0x424dfd908c50009 type:setWatches
> > > > >> > > >> cxid:0xfffffffffffffff8
> > > > >> > > >> >> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
> > > > >> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
> > > > >> > > org.apache.zookeeper.server.Request:
> > > > >> > > >> >> >> Ignoring
> > > > >> > > >> >> >> >> exception during toString
> > > > >> > > >> >> >> >> java.nio.BufferUnderflowException
> > > > >> > > >> >> >> >> at
> > > > >> > java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
> > > > >> > > >> >> >> >> at
> > > java.nio.ByteBuffer.get(ByteBuffer.java:675)
> > > > >> > > >> >> >> >> at
> > > > >> > > >> >> >>
> > > > >> org.apache.zookeeper.server.Request.toString(Request.java:199)
> > > > >> > > >> >> >> >> at
> > > > >> > > >> >> >> >>
> > > > >> > > >>
> > > > >>
> > org.apache.log4j.or.DefaultRenderer.doRender(DefaultRenderer.java:36)
> > > > >> > > >> >> >> >> at
> > > > >> > > >> >> >> >>
> > > > >> > >
> > org.apache.log4j.or.RendererMap.findAndRender(RendererMap.java:80)
> > > > >> > > >> >> >> >> at
> > > > >> > > >> >> >> >>
> > > > >> > > >> >> >>
> > > > >> > > >> >>
> > > > >> > > >>
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.log4j.spi.LoggingEvent.getRenderedMessage(LoggingEvent.java:362)
> > > > >> > > >> >> >> >> at
> > > > >> > > >> >> >> >>
> > > > >> > > >> >> >>
> > > > >> > > >> >>
> > > > >> > > >>
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.log4j.helpers.PatternParser$BasicPatternConverter.convert(PatternParser.java:403)
> > > > >> > > >> >> >> >> at
> > > > >> > > >> >> >> >>
> > > > >> > > >> >> >>
> > > > >> > > >> >>
> > > > >> > > >>
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.log4j.helpers.PatternConverter.format(PatternConverter.java:65)
> > > > >> > > >> >> >> >> at
> > > > >> > > >> >>
> > > org.apache.log4j.PatternLayout.format(PatternLayout.java:502)
> > > > >> > > >> >> >> >> at
> > > > >> > > >> >> >> >>
> > > > >> > >
> > org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:302)
> > > > >> > > >> >> >> >> at
> > > > >> > > >> >> >> >>
> > > > >> > > >> >> >>
> > > > >> > > >> >>
> > > > >> > > >>
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.log4j.DailyRollingFileAppender.subAppend(DailyRollingFileAppender.java:359)
> > > > >> > > >> >> >> >> at
> > > > >> > > >> >> >>
> > > > >> org.apache.log4j.WriterAppender.append(WriterAppender.java:160)
> > > > >> > > >> >> >> >> at
> > > > >> > > >> >> >> >>
> > > > >> > > >>
> > > > >>
> > org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
> > > > >> > > >> >> >> >> at
> > > > >> > > >> >> >> >>
> > > > >> > > >> >> >>
> > > > >> > > >> >>
> > > > >> > > >>
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
> > > > >> > > >> >> >> >> at
> > > > >> > > >> org.apache.log4j.Category.callAppenders(Category.java:206)
> > > > >> > > >> >> >> >> at
> > > > >> > > org.apache.log4j.Category.forcedLog(Category.java:391)
> > > > >> > > >> >> >> >> at
> > > > >> org.apache.log4j.Category.debug(Category.java:260)
> > > > >> > > >> >> >> >> at
> > > > >> > > >> >> >> >>
> > > > >> > > >> >> >>
> > > > >> > > >> >>
> > > > >> > > >>
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:147)
> > > > >> > > >> >> >> >> at
> > > > >> > > >> >> >> >>
> > > > >> > > >> >> >>
> > > > >> > > >> >>
> > > > >> > > >>
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)
> > > > >> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
> > > > >> > > org.apache.zookeeper.server.Request:
> > > > >> > > >> >> >> Ignoring
> > > > >> > > >> >> >> >> exception during toString
> > > > >> > > >> >> >> >> sessionid:0x424dfd908c50009 type:setWatches
> > > > >> > > >> cxid:0xfffffffffffffff8
> > > > >> > > >> >> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
> > > > >> > > >> >> >> >> 2009-11-11 04:29:00,810 INFO
> > > > >> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
> > > > >> > > >> >> >> >> closing session:0x424dfd908c50009 NIOServerCnxn:
> > > > >> > > >> >> >> >> java.nio.channels.SocketChannel[connected local=/
> > > > >> > > >> >> 192.168.100.131:2181
> > > > >> > > >> >> >> remote=/
> > > > >> > > >> >> >> >> 192.168.100.131:35500]
> > > > >> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
> > > > >> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
> > > > >> > > >> >> >> >> ignoring exception during input shutdown
> > > > >> > > >> >> >> >> java.net.SocketException: Transport endpoint is
> not
> > > > >> connected
> > > > >> > > >> >> >> >> at
> > > sun.nio.ch.SocketChannelImpl.shutdown(Native
> > > > >> > Method)
> > > > >> > > >> >> >> >> at
> > > > >> > > >> >> >> >>
> > > > >> > > >> >>
> > > > >> > >
> > > >
> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
> > > > >> > > >> >> >> >> at
> > > > >> > > >> >> >>
> > > > >> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> > > > >> > > >> >> >> >> at
> > > > >> > > >> >> >> >>
> > > > >> > > >> >>
> > > > >> > >
> > > > >>
> > > org.apache.zookeeper.server.NIOServerCnxn.close(NIOServerCnxn.java:848)
> > > > >> > > >> >> >> >> at
> > > > >> > > >> >> >> >>
> > > > >> > > >> >>
> > > > >> > >
> > > >
> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:498)
> > > > >> > > >> >> >> >> at
> > > > >> > > >> >> >> >>
> > > > >> > > >> >> >>
> > > > >> > > >> >>
> > > > >> > > >>
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:221)
> > > > >> > > >> >> >> >>
> > > > >> > > >> >> >> >
> > > > >> > > >> >> >>
> > > > >> > > >> >> >
> > > > >> > > >> >>
> > > > >> > > >> >
> > > > >> > > >>
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>
Re: regionserver disconnection
Posted by stack <st...@duboce.net>.
On Thu, Nov 12, 2009 at 11:50 AM, Zhenyu Zhong <zh...@gmail.com>wrote:
>
> In my cluster, half of the cluster have 2 disks 400GB each per machine, and
> half of the cluster have 6 disks per machine. Maybe we should run
> zookeeper
> on the machines with 2 disks and RS on machines with 6 disks?
>
> That would make most sense only in the below, it looks like the RS that had
issue had 4 disks?
> BTW, the 10 minutes zookeeper.session.timeout has been set during the
> experiment.
>
And for sure this node had it in place?
St.Ack
>
> thanks
> zhenyu
>
> On Thu, Nov 12, 2009 at 2:08 PM, stack <st...@duboce.net> wrote:
>
> > On Thu, Nov 12, 2009 at 8:40 AM, Zhenyu Zhong <zhongresearch@gmail.com
> > >wrote:
> >
> >
> > > Though I experienced 2 regionserver disconnection this morning, it
> looks
> > > better from the regionserver log. (please see the following log)
> > >
> > >
> > > http://pastebin.com/m496dbfae
> > >
> > >
> > > I checked diskIO for one of the regionserver(192.168.100.116) during
> the
> > > time it disconnected.
> > >
> > > Time: 03:04:58 AM
> > > Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
> > > sda 105.31 5458.83 19503.64 9043873239 32312473676
> > > sda1 2.90 6.64 99.25 10993934 164433464
> > > sda2 1.72 23.76 30.25 39365817 50124008
> > > sda3 0.30 0.38 3.58 624930 5923000
> > > sda4 100.39 5428.06 19370.56 8992888270 32091993204
> > >
> > >
> > Is this high for you? 20k blocks/second would seem to be high but its
> one
> > disk only and its not being shared by zk anymore so shouldn't matter?
> >
> >
> > > I also checked the zookeeper quorum server that the regionserver tried
> to
> > > connect according to the log. However, I don't see 192.168.100.116 in
> the
> > > client list of the zookeeper quorum server that the regionserver tried
> to
> > > connect.
> > > Would that be a problem?
> > >
> > >
> > Is that because the ephemeral node for the regionserver had evaporated?
> > Lost
> > its lease w/ zk by the time you went to look?
> >
> >
> >
> >
> > > Thu Nov 12 15:04:52 UTC 2009
> > > Zookeeper version: 3.2.1-808558, built on 08/27/2009 18:48 GMT
> > > Clients:
> > > /192.168.100.127:43045[1](queued=0,recved=26,sent=0)
> > > /192.168.100.131:39091[1](queued=0,recved=964,sent=0)
> > > /192.168.100.124:35961[1](queued=0,recved=3266,sent=0)
> > > /192.168.100.123:47935[1](queued=0,recved=233,sent=0)
> > > /192.168.100.125:46931[1](queued=0,recved=2,sent=0)
> > > /192.168.100.118:54924[1](queued=0,recved=295,sent=0)
> > > /192.168.100.118:41390[1](queued=0,recved=2290,sent=0)
> > > /192.168.100.136:42243[1](queued=0,recved=0,sent=0)
> > >
> > > Latency min/avg/max: 0/17/6333
> > > Received: 47111
> > > Sent: 0
> > > Outstanding: 0
> > > Zxid: 0x77000083f4
> > > Mode: leader
> > > Node count: 23
> > >
> >
> >
> > That 6 second maximum latency is pretty bad but should be well within the
> > zk
> > session timeout.
> >
> > So, problem is likely on the zk client side of the session; i.e. in the
> RS.
> > You could enable GC logging as J-D suggested to see if you have any big
> > pauses, pauses > zk session timeout.
> >
> > When the RS went down, it didn't look too heavily loaded:
> >
> >
> > 1. 2009-11-12 15:04:52,830 INFO
> > org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics:
> > request=1.5166667, regions=322, stores=657, storefiles=631,
> > storefileIndexSize=61, memstoreSize=1472, usedHeap=2819, maxHeap=4079,
> > blockCacheSize=658110960, blockCacheFree=197395984,
> blockCacheCount=9903,
> > blockCacheHitRatio=99
> >
> > Its serving a few reads? The number of store files seems fine. Not too
> > much memory used.
> >
> > Looking at the logs, I see the Lease Still Held exception. This happens
> > when the RS does its regular report to the master but the master thinks
> the
> > RS has since restarted. It'll think this probably because it noticed
> that
> > the RS's znode in zk had gone away and it considered the RS dead.
> >
> > Looking too at your logs I see this gap in the zk pinging:
> >
> >
> > 1. 2009-11-12 15:03:39,325 DEBUG org.apache.zookeeper.ClientCnxn: Got
> > ping response for sessionid:0x224e55436ad0004 after 0ms
> > 2. 2009-11-12 15:03:43,113 DEBUG org.apache.zookeeper.ClientCnxn: Got
> > ping response for sessionid:0x24e55436a0007d after 0ms
> >
> > Where in the lines above it, its reporting about every ten seconds, here
> > there is a big gap.
> >
> > Do you have ganglia or something that will let you look more into what
> was
> > happening on this machine around this time? Is the machine OK? It looks
> > lightly loaded and you have your cluster nicely laid out. Something odd
> is
> > going on. What about things like the write speed to disk? In the past
> > strange issues have been explained by incorrectly set BIOS which made
> disks
> > run at 1/100th of their proper speed.
> >
> > St.Ack
> >
> >
> >
> >
> >
> >
> >
> > >
> > > Best,
> > > zhenyu
> > >
> > >
> > > On Wed, Nov 11, 2009 at 3:58 PM, Zhenyu Zhong <zhongresearch@gmail.com
> > > >wrote:
> > >
> > > > Stack
> > > >
> > > > I am very appreciated for your comments.
> > > > I will use the zookeeper monitoring script on my cluster and let it
> run
> > > > overnight to see the result.
> > > > I will follow up the thread when I get anything.
> > > >
> > > > thanks
> > > > zhenyu
> > > >
> > > >
> > > > On Wed, Nov 11, 2009 at 3:52 PM, stack <st...@duboce.net> wrote:
> > > >
> > > >> I see these in your log too:
> > > >>
> > > >>
> > > >> 1. 2009-11-11 04:27:19,018 DEBUG org.apache.zookeeper.ClientCnxn:
> > Got
> > > >> ping response for sessionid:0x424dfd908c50009 after 4544ms
> > > >> 2. 2009-11-11 04:27:19,018 DEBUG org.apache.zookeeper.ClientCnxn:
> > Got
> > > >> ping response for sessionid:0x24dfd90c810002 after 4548ms
> > > >> 3. 2009-11-11 04:27:43,960 DEBUG org.apache.zookeeper.ClientCnxn:
> > Got
> > > >> ping response for sessionid:0x424dfd908c50009 after 9030ms
> > > >> 4. 2009-11-11 04:27:43,960 DEBUG org.apache.zookeeper.ClientCnxn:
> > Got
> > > >> ping response for sessionid:0x24dfd90c810002 after 9030ms
> > > >> 5. 2009-11-11 04:27:48,830 DEBUG org.apache.zookeeper.ClientCnxn:
> > Got
> > > >> ping response for sessionid:0x424dfd908c50009 after 0ms
> > > >> 6. 2009-11-11 04:27:48,830 DEBUG org.apache.zookeeper.ClientCnxn:
> > Got
> > > >> ping response for sessionid:0x24dfd90c810002 after 0ms
> > > >>
> > > >> Which would indicate lag talking to zk too.
> > > >>
> > > >> Appending HLog also seems to get steadily worse.
> > > >>
> > > >> I also see these:
> > > >>
> > > >>
> > > >> 1. 2009-11-11 04:27:01,233 WARN org.apache.zookeeper.ClientCnxn:
> > > >> Exception closing session 0x24dfd90c810002 to
> > > >> sun.nio.ch.SelectionKeyImpl@5e16a7e3
> > > >> 2. java.io.IOException: Read error rc = -1
> > > >> java.nio.DirectByteBuffer[pos=0 lim=4 cap=4]
> > > >> 3. at
> > > >>
> > org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
> > > >> 4. at
> > > >>
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
> > > >>
> > > >>
> > > >> Were all 5 quorum members participating? Check by doing "echo
> stat|nc
> > > >> QUORUM_HOST port" for each member of quorum.... the below might
> help:
> > > >>
> > > >> members="1,2,3,4,5"
> > > >> while true; do date; for i in $members; do echo $i; echo stat | nc
> $i
> > > >> 2181;
> > > >> done; sleep 10; done
> > > >>
> > > >> We need to improve our zk_dump. It just reports quorum members...
> not
> > > if
> > > >> they are actually participating or not. The above will tell you if
> > dead
> > > >> or
> > > >> alive.
> > > >>
> > > >> Please report back after checking above and following the
> Jean-Daniel
> > > >> prescription.
> > > >>
> > > >> Thanks,
> > > >> St.Ack
> > > >>
> > > >>
> > > >>
> > > >> On Wed, Nov 11, 2009 at 12:29 PM, Zhenyu Zhong <
> > zhongresearch@gmail.com
> > > >> >wrote:
> > > >>
> > > >> > Thanks a lot for the help.
> > > >> >
> > > >> > I understand that the change of disk configurations should be done
> > > >> > progressively such that HDFS can balance and recover from a disk
> > going
> > > >> > away.
> > > >> >
> > > >> > The data we are trying to import into HBase is about 200million
> > rows,
> > > >> each
> > > >> > row is about hundreds of bytes. Basically, we read in a text file
> > from
> > > >> > HDFS,
> > > >> > then for each row in the text file, we fetch data from HBase
> table,
> > > we
> > > >> > compare the column value and update the latest value back into
> > HBase.
> > > I
> > > >> was
> > > >> > suggested to use CheckAndPut. Anyway this is the common usecase
> on
> > my
> > > >> > side.
> > > >> >
> > > >> >
> > > >> > Many thanks
> > > >> > zhenyu
> > > >> >
> > > >> >
> > > >> >
> > > >> > On Wed, Nov 11, 2009 at 3:03 PM, Jean-Daniel Cryans <
> > > >> jdcryans@apache.org
> > > >> > >wrote:
> > > >> >
> > > >> > > 6 disks, 19 machines and you see those kind of issues? I'm quite
> > > >> > > surprised and I want to learn more about what you trying to to.
> > > >> > >
> > > >> > > But first, if hbase is managing ZK for you then you can change
> the
> > > >> > > value of hbase.zookeeper.property.dataDir so that it points to
> its
> > > own
> > > >> > > mount. Make sure you also change dfs.data.dir in
> > conf/hdfs-site.xml
> > > >> > > (hadoop stuff) so that it stops pointing there (you should have
> 6
> > > >> > > values there at the moment if you configured it correctly).
> Doing
> > > this
> > > >> > > could result in data loss if you do it all at once so do the
> > change
> > > >> > > progressively. This also means that you may have to keep a
> > seperate
> > > >> > > HDFS configuration for those particular nodes. Please ask more
> > > >> > > questions if it's not very clear.
> > > >> > >
> > > >> > > WRT to dedicating nodes to Zookeeeper, it makes sense only if
> you
> > > have
> > > >> > > a lot of nodes but in your case, unless there's also CPU
> > contention,
> > > >> > > you're good with only a dedicated disk.
> > > >> > >
> > > >> > > Now for what you are trying to do, is this a massive import that
> > > goes
> > > >> > > over many hours? How big is each row? I'd love to get a lot of
> > > details
> > > >> > > to help you optimize your cluster for your particular problem.
> > > >> > >
> > > >> > > Thx,
> > > >> > >
> > > >> > > J-D
> > > >> > >
> > > >> > > On Wed, Nov 11, 2009 at 11:51 AM, Zhenyu Zhong <
> > > >> zhongresearch@gmail.com>
> > > >> > > wrote:
> > > >> > > > J-D,
> > > >> > > >
> > > >> > > > Thanks a lot for the insightful comments.
> > > >> > > > We are running 5 zookeeper quorums now. On these 5 quorum
> > servers,
> > > >> we
> > > >> > > also
> > > >> > > > run datanode, tasktracker and regionserver.
> > > >> > > > Now we have 6 disks per machine.
> > > >> > > > If it is the disk IO saturation, is there anyway to configure
> > > >> zookeeper
> > > >> > > to
> > > >> > > > write to a seperate disk which is not the same one used by
> HDFS?
> > > >> > > > Or shall we run zookeeper quorums on different nodes other
> than
> > > >> HDFS?
> > > >> > > >
> > > >> > > > Are HLogs generated by Regionservers? Would it alleviate the
> > issue
> > > >> if
> > > >> > we
> > > >> > > use
> > > >> > > > the first 14 for regionserver, and the rest of 5 for
> > zookeeper?(In
> > > >> that
> > > >> > > > sense, the node running zookeeper quorum won't run
> regionserver)
> > > >> > > >
> > > >> > > > I am very appreciated for your comments.
> > > >> > > >
> > > >> > > > Best,
> > > >> > > > zhenyu
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > > On Wed, Nov 11, 2009 at 2:30 PM, Jean-Daniel Cryans <
> > > >> > jdcryans@apache.org
> > > >> > > >wrote:
> > > >> > > >
> > > >> > > >> The problem seems to be that the IO is saturated. When you
> see
> > > >> this:
> > > >> > > >>
> > > >> > > >> #
> > > >> > > >> 2009-11-11 04:28:10,308 WARN
> > > >> > > >> org.apache.hadoop.hbase.regionserver.HLog: IPC Server handler
> 9
> > > on
> > > >> > > >> 60021 took 6819ms appending an edit to hlog; editcount=370
> > > >> > > >> #
> > > >> > > >> 2009-11-11 04:28:10,308 WARN
> > > >> > > >> org.apache.hadoop.hbase.regionserver.HLog: IPC Server handler
> 2
> > > on
> > > >> > > >> 60021 took 4412ms appending an edit to hlog; editcount=371
> > > >> > > >>
> > > >> > > >> It's a sign that HDFS is slow. I guess that the zookeeper
> > quorum
> > > >> > > >> members are on those same nodes using the same disks? If so,
> > > >> there's
> > > >> > > >> the problem because Zookeeper needs to be able to commit
> > > >> transactions
> > > >> > > >> very fast and if your disks are already hogged then they have
> > to
> > > >> wait
> > > >> > > >> and you can get disconnections. Do you still have 19 quorum
> > > >> members?
> > > >> > > >> How many disks do you have per machine?
> > > >> > > >>
> > > >> > > >> Thx
> > > >> > > >>
> > > >> > > >> J-D
> > > >> > > >>
> > > >> > > >> On Wed, Nov 11, 2009 at 11:22 AM, Zhenyu Zhong <
> > > >> > zhongresearch@gmail.com
> > > >> > > >
> > > >> > > >> wrote:
> > > >> > > >> > Please take a look at the following regionserver log.
> > > >> > > >> > The error started to come out from 4:24am UTC to 4:29am
> UTC.
> > > >> > > >> >
> > > >> > > >> > http://pastebin.com/m3717d064
> > > >> > > >> >
> > > >> > > >> > Thanks in advance.
> > > >> > > >> >
> > > >> > > >> > Best,
> > > >> > > >> > zhenyu
> > > >> > > >> >
> > > >> > > >> >
> > > >> > > >> > On Wed, Nov 11, 2009 at 2:01 PM, Jean-Daniel Cryans <
> > > >> > > jdcryans@apache.org
> > > >> > > >> >wrote:
> > > >> > > >> >
> > > >> > > >> >> Could you paste a bigger log trace (interested to see at
> > least
> > > >> all
> > > >> > > 300
> > > >> > > >> >> lines before that) in a pastebin?
> > > >> > > >> >>
> > > >> > > >> >> Thx,
> > > >> > > >> >>
> > > >> > > >> >> J-D
> > > >> > > >> >>
> > > >> > > >> >> On Wed, Nov 11, 2009 at 10:56 AM, Zhenyu Zhong <
> > > >> > > zhongresearch@gmail.com
> > > >> > > >> >
> > > >> > > >> >> wrote:
> > > >> > > >> >> > J-D,
> > > >> > > >> >> >
> > > >> > > >> >> > Thanks for the reply.
> > > >> > > >> >> >
> > > >> > > >> >> > The problem I have is one or more of the regionservers
> > > >> disconnect
> > > >> > > from
> > > >> > > >> >> the
> > > >> > > >> >> > cluster.
> > > >> > > >> >> >
> > > >> > > >> >> > I checked the resource usage including memory Heap usage
> > for
> > > >> > > >> >> regionserver,
> > > >> > > >> >> > and found nothing suspicious. The regionserver heap
> usage
> > is
> > > >> way
> > > >> > > below
> > > >> > > >> >> the
> > > >> > > >> >> > maxHeap defined in jvm.
> > > >> > > >> >> >
> > > >> > > >> >> > So I checked zookeeper log, then I found the following
> > > >> > > >> >> > BufferUnderflowException, eventually it leads to
> > > >> SocketException:
> > > >> > > >> >> Transport
> > > >> > > >> >> > endpoint is not connected.
> > > >> > > >> >> >
> > > >> > > >> >> >
> > > >> > > >> >> > I also checked the regionserver log. It looks like the
> > > >> > regionserver
> > > >> > > >> tried
> > > >> > > >> >> to
> > > >> > > >> >> > connect to other zookeeper quorum server. It connects
> > > >> > successfully,
> > > >> > > >> but
> > > >> > > >> >> then
> > > >> > > >> >> > it complained about SocketException: transport endpoint
> is
> > > not
> > > >> > > >> connected.
> > > >> > > >> >> >
> > > >> > > >> >> > So both zookeeper log and regionserver log told me that
> > the
> > > >> > > transport
> > > >> > > >> >> > endpoint is not connected. I don't know why.
> > > >> > > >> >> >
> > > >> > > >> >> > I would be very appreciated if you could point me to the
> > > right
> > > >> > > >> direction.
> > > >> > > >> >> >
> > > >> > > >> >> > Best,
> > > >> > > >> >> >
> > > >> > > >> >> >
> > > >> > > >> >> > Regionserver log:
> > > >> > > >> >> > 2009-11-11 04:24:53,718 INFO
> > > org.apache.zookeeper.ClientCnxn:
> > > >> > > >> Attempting
> > > >> > > >> >> > connection to server superpyxis0018/
> 192.168.100.132:2181
> > > >> > > >> >> > 2009-11-11 04:24:53,718 INFO
> > > org.apache.zookeeper.ClientCnxn:
> > > >> > > Priming
> > > >> > > >> >> > connection to java.nio.channels.SocketChannel[connected
> > > >> local=/
> > > >> > > >> >> > 192.168.100.131:43065 remote=superpyxis0018/
> > > >> 192.168.100.132:2181
> > > >> > ]
> > > >> > > >> >> > 2009-11-11 04:24:53,718 INFO
> > > org.apache.zookeeper.ClientCnxn:
> > > >> > > Server
> > > >> > > >> >> > connection successful
> > > >> > > >> >> > 2009-11-11 04:24:53,768 INFO
> > > >> > > >> >> > org.apache.hadoop.hbase.regionserver.HRegionServer: Got
> > > >> ZooKeeper
> > > >> > > >> event,
> > > >> > > >> >> > state: Disconnected, type: None, path: null
> > > >> > > >> >> > 2009-11-11 04:24:54,118 INFO
> > > org.apache.zookeeper.ClientCnxn:
> > > >> > > >> Attempting
> > > >> > > >> >> > connection to server superpyxis0017/
> 192.168.100.131:2181
> > > >> > > >> >> > 2009-11-11 04:24:54,118 INFO
> > > org.apache.zookeeper.ClientCnxn:
> > > >> > > Priming
> > > >> > > >> >> > connection to java.nio.channels.SocketChannel[connected
> > > >> local=/
> > > >> > > >> >> > 192.168.100.131:46903 remote=superpyxis0017/
> > > >> 192.168.100.131:2181
> > > >> > ]
> > > >> > > >> >> > 2009-11-11 04:24:54,118 INFO
> > > org.apache.zookeeper.ClientCnxn:
> > > >> > > Server
> > > >> > > >> >> > connection successful
> > > >> > > >> >> > 2009-11-11 04:24:54,118 WARN
> > > org.apache.zookeeper.ClientCnxn:
> > > >> > > >> Exception
> > > >> > > >> >> > closing session 0x24dfd90c810002 to
> > > >> > > >> sun.nio.ch.SelectionKeyImpl@2dd78ccf
> > > >> > > >> >> > java.io.IOException: Read error rc = -1
> > > >> > > >> java.nio.DirectByteBuffer[pos=0
> > > >> > > >> >> > lim=4 cap=4]
> > > >> > > >> >> > at
> > > >> > > >> >> >
> > > >> > >
> > org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
> > > >> > > >> >> > at
> > > >> > > >> >> >
> > > >> >
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
> > > >> > > >> >> > 2009-11-11 04:24:54,118 WARN
> > > org.apache.zookeeper.ClientCnxn:
> > > >> > > Ignoring
> > > >> > > >> >> > exception during shutdown input
> > > >> > > >> >> > java.net.SocketException: Transport endpoint is not
> > > connected
> > > >> > > >> >> > at sun.nio.ch.SocketChannelImpl.shutdown(Native
> > > Method)
> > > >> > > >> >> > at
> > > >> > > >> >> >
> > > >> > >
> > > sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
> > > >> > > >> >> > at
> > > >> > > >>
> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> > > >> > > >> >> > at
> > > >> > > >> >> >
> > > >> > > >>
> > > >> >
> > > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> > > >> > > >> >> > at
> > > >> > > >> >> >
> > > >> >
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> > > >> > > >> >> >
> > > >> > > >> >> >
> > > >> > > >> >> >
> > > >> > > >> >> >
> > > >> > > >> >> >
> > > >> > > >> >> > On Wed, Nov 11, 2009 at 1:40 PM, Jean-Daniel Cryans <
> > > >> > > >> jdcryans@apache.org
> > > >> > > >> >> >wrote:
> > > >> > > >> >> >
> > > >> > > >> >> >> I'm really not sure exactly what the situation is with
> > the
> > > >> > > >> >> >> java.nio.BufferUnderflowException, but what do you mean
> > by
> > > >> > > >> >> >> "regionserver disconnection"? Could you give more
> > details?
> > > >> > > >> >> >>
> > > >> > > >> >> >> Thx,
> > > >> > > >> >> >>
> > > >> > > >> >> >> J-D
> > > >> > > >> >> >>
> > > >> > > >> >> >> On Wed, Nov 11, 2009 at 10:34 AM, Zhenyu Zhong <
> > > >> > > >> zhongresearch@gmail.com
> > > >> > > >> >> >
> > > >> > > >> >> >> wrote:
> > > >> > > >> >> >> > I searched around and found HBASE-1645 "HBase could
> be
> > a
> > > >> > little
> > > >> > > >> more
> > > >> > > >> >> >> dainty
> > > >> > > >> >> >> > when zk is pulled out from under it"
> > > >> > > >> >> >> >
> > > >> > > >> >> >> > I am wondering whether my problem is related to this
> > > >> > HBase-1645.
> > > >> > > Or
> > > >> > > >> >> >> probably
> > > >> > > >> >> >> > has nothing to do with it?
> > > >> > > >> >> >> >
> > > >> > > >> >> >> > thanks
> > > >> > > >> >> >> >
> > > >> > > >> >> >> >
> > > >> > > >> >> >> > On Wed, Nov 11, 2009 at 12:02 PM, Zhenyu Zhong <
> > > >> > > >> >> zhongresearch@gmail.com
> > > >> > > >> >> >> >wrote:
> > > >> > > >> >> >> >
> > > >> > > >> >> >> >> Hi,
> > > >> > > >> >> >> >>
> > > >> > > >> >> >> >> I have been monitoring the system resources such as
> > > memory
> > > >> > Heap
> > > >> > > >> usage
> > > >> > > >> >> >> for
> > > >> > > >> >> >> >> Regionserver, load average, network bandwidth, etc.
> > > >> > > >> >> >> >>
> > > >> > > >> >> >> >> I experienced a disconnection last night. When I
> > checked
> > > >> the
> > > >> > > >> memory
> > > >> > > >> >> Heap
> > > >> > > >> >> >> >> for Regionserver, it was 2336 which is way below the
> > max
> > > >> Heap
> > > >> > > >> 4096. I
> > > >> > > >> >> >> don't
> > > >> > > >> >> >> >> think it is Regionserver problem. The requests that
> > the
> > > >> > > >> regionserver
> > > >> > > >> >> got
> > > >> > > >> >> >> was
> > > >> > > >> >> >> >> about 126 which doesn't seem to be high. The load
> > > average
> > > >> is
> > > >> > > kind
> > > >> > > >> of
> > > >> > > >> >> >> high
> > > >> > > >> >> >> >> >100. (but I saw disconnections while the load
> average
> > > is
> > > >> > > around
> > > >> > > >> 30).
> > > >> > > >> >> >> The
> > > >> > > >> >> >> >> bandwidth is not a problem.
> > > >> > > >> >> >> >>
> > > >> > > >> >> >> >> However, I found error messages from the zookeeper.
> > > Please
> > > >> > see
> > > >> > > the
> > > >> > > >> >> >> >> following error messages. There are some
> > > >> > > >> >> >> >> BufferUnderflowException error, IOException
> error,etc.
> > > >> > > >> >> >> >> May I ask what might be the causes that generate
> these
> > > >> error
> > > >> > > >> >> messages?
> > > >> > > >> >> >> >>
> > > >> > > >> >> >> >> Many thanks
> > > >> > > >> >> >> >> zhenyu
> > > >> > > >> >> >> >>
> > > >> > > >> >> >> >>
> > > >> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
> > > >> > > org.apache.zookeeper.server.Request:
> > > >> > > >> >> >> Ignoring
> > > >> > > >> >> >> >> exception during toString
> > > >> > > >> >> >> >> java.nio.BufferUnderflowException
> > > >> > > >> >> >> >> at
> > > >> > java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
> > > >> > > >> >> >> >> at
> > java.nio.ByteBuffer.get(ByteBuffer.java:675)
> > > >> > > >> >> >> >> at
> > > >> > > >> >> >>
> > > >> org.apache.zookeeper.server.Request.toString(Request.java:199)
> > > >> > > >> >> >> >> at
> java.lang.String.valueOf(String.java:2827)
> > > >> > > >> >> >> >> at
> > > >> > > java.lang.StringBuilder.append(StringBuilder.java:115)
> > > >> > > >> >> >> >> at
> > > >> > > >> >> >> >>
> > > >> > > >> >> >>
> > > >> > > >> >>
> > > >> > > >>
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> org.apache.zookeeper.server.quorum.CommitProcessor.processRequest(CommitProcessor.java:167)
> > > >> > > >> >> >> >> at
> > > >> > > >> >> >> >>
> > > >> > > >> >> >>
> > > >> > > >> >>
> > > >> > > >>
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> org.apache.zookeeper.server.quorum.FollowerRequestProcessor.run(FollowerRequestProcessor.java:68)
> > > >> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
> > > >> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
> > > >> > > >> >> >> >> Exception causing close of session 0x424dfd908c50009
> > > >> > > >> >> >> >> due to java.io.IOException: Read error
> > > >> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
> > > >> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
> > > >> > > >> >> >> >> IOException stack trace
> > > >> > > >> >> >> >> java.io.IOException: Read error
> > > >> > > >> >> >> >> at
> > > >> > > >> >> >> >>
> > > >> > > >> >>
> > > >> > >
> > > org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:372)
> > > >> > > >> >> >> >> at
> > > >> > > >> >> >> >>
> > > >> > > >> >> >>
> > > >> > > >> >>
> > > >> > > >>
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:221)
> > > >> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
> > > >> > > >> >> >> >> org.apache.zookeeper.server.quorum.CommitProcessor:
> > > >> > Processing
> > > >> > > >> >> request::
> > > >> > > >> >> >> >> sessionid:0x424dfd908c50009 type:setWatches
> > > >> > > >> cxid:0xfffffffffffffff8
> > > >> > > >> >> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
> > > >> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
> > > >> > > org.apache.zookeeper.server.Request:
> > > >> > > >> >> >> Ignoring
> > > >> > > >> >> >> >> exception during toString
> > > >> > > >> >> >> >> java.nio.BufferUnderflowException
> > > >> > > >> >> >> >> at
> > > >> > java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
> > > >> > > >> >> >> >> at
> > java.nio.ByteBuffer.get(ByteBuffer.java:675)
> > > >> > > >> >> >> >> at
> > > >> > > >> >> >>
> > > >> org.apache.zookeeper.server.Request.toString(Request.java:199)
> > > >> > > >> >> >> >> at
> java.lang.String.valueOf(String.java:2827)
> > > >> > > >> >> >> >> at
> > > >> > > java.lang.StringBuilder.append(StringBuilder.java:115)
> > > >> > > >> >> >> >> at
> > > >> > > >> >> >> >>
> > > >> > > >> >> >>
> > > >> > > >> >>
> > > >> > > >>
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:74)
> > > >> > > >> >> >> >> at
> > > >> > > >> >> >> >>
> > > >> > > >> >> >>
> > > >> > > >> >>
> > > >> > > >>
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)
> > > >> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
> > > >> > > >> >> >> >> org.apache.zookeeper.server.FinalRequestProcessor:
> > > >> Processing
> > > >> > > >> >> request::
> > > >> > > >> >> >> >> sessionid:0x424dfd908c50009 type:setWatches
> > > >> > > >> cxid:0xfffffffffffffff8
> > > >> > > >> >> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
> > > >> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
> > > >> > > org.apache.zookeeper.server.Request:
> > > >> > > >> >> >> Ignoring
> > > >> > > >> >> >> >> exception during toString
> > > >> > > >> >> >> >> java.nio.BufferUnderflowException
> > > >> > > >> >> >> >> at
> > > >> > java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
> > > >> > > >> >> >> >> at
> > java.nio.ByteBuffer.get(ByteBuffer.java:675)
> > > >> > > >> >> >> >> at
> > > >> > > >> >> >>
> > > >> org.apache.zookeeper.server.Request.toString(Request.java:199)
> > > >> > > >> >> >> >> at
> > > >> > > >> >> >> >>
> > > >> > > >>
> > > >>
> org.apache.log4j.or.DefaultRenderer.doRender(DefaultRenderer.java:36)
> > > >> > > >> >> >> >> at
> > > >> > > >> >> >> >>
> > > >> > >
> org.apache.log4j.or.RendererMap.findAndRender(RendererMap.java:80)
> > > >> > > >> >> >> >> at
> > > >> > > >> >> >> >>
> > > >> > > >> >> >>
> > > >> > > >> >>
> > > >> > > >>
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> org.apache.log4j.spi.LoggingEvent.getRenderedMessage(LoggingEvent.java:362)
> > > >> > > >> >> >> >> at
> > > >> > > >> >> >> >>
> > > >> > > >> >> >>
> > > >> > > >> >>
> > > >> > > >>
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> org.apache.log4j.helpers.PatternParser$BasicPatternConverter.convert(PatternParser.java:403)
> > > >> > > >> >> >> >> at
> > > >> > > >> >> >> >>
> > > >> > > >> >> >>
> > > >> > > >> >>
> > > >> > > >>
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> org.apache.log4j.helpers.PatternConverter.format(PatternConverter.java:65)
> > > >> > > >> >> >> >> at
> > > >> > > >> >>
> > org.apache.log4j.PatternLayout.format(PatternLayout.java:502)
> > > >> > > >> >> >> >> at
> > > >> > > >> >> >> >>
> > > >> > >
> org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:302)
> > > >> > > >> >> >> >> at
> > > >> > > >> >> >> >>
> > > >> > > >> >> >>
> > > >> > > >> >>
> > > >> > > >>
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> org.apache.log4j.DailyRollingFileAppender.subAppend(DailyRollingFileAppender.java:359)
> > > >> > > >> >> >> >> at
> > > >> > > >> >> >>
> > > >> org.apache.log4j.WriterAppender.append(WriterAppender.java:160)
> > > >> > > >> >> >> >> at
> > > >> > > >> >> >> >>
> > > >> > > >>
> > > >>
> org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
> > > >> > > >> >> >> >> at
> > > >> > > >> >> >> >>
> > > >> > > >> >> >>
> > > >> > > >> >>
> > > >> > > >>
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
> > > >> > > >> >> >> >> at
> > > >> > > >> org.apache.log4j.Category.callAppenders(Category.java:206)
> > > >> > > >> >> >> >> at
> > > >> > > org.apache.log4j.Category.forcedLog(Category.java:391)
> > > >> > > >> >> >> >> at
> > > >> org.apache.log4j.Category.debug(Category.java:260)
> > > >> > > >> >> >> >> at
> > > >> > > >> >> >> >>
> > > >> > > >> >> >>
> > > >> > > >> >>
> > > >> > > >>
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:147)
> > > >> > > >> >> >> >> at
> > > >> > > >> >> >> >>
> > > >> > > >> >> >>
> > > >> > > >> >>
> > > >> > > >>
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)
> > > >> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
> > > >> > > org.apache.zookeeper.server.Request:
> > > >> > > >> >> >> Ignoring
> > > >> > > >> >> >> >> exception during toString
> > > >> > > >> >> >> >> sessionid:0x424dfd908c50009 type:setWatches
> > > >> > > >> cxid:0xfffffffffffffff8
> > > >> > > >> >> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
> > > >> > > >> >> >> >> 2009-11-11 04:29:00,810 INFO
> > > >> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
> > > >> > > >> >> >> >> closing session:0x424dfd908c50009 NIOServerCnxn:
> > > >> > > >> >> >> >> java.nio.channels.SocketChannel[connected local=/
> > > >> > > >> >> 192.168.100.131:2181
> > > >> > > >> >> >> remote=/
> > > >> > > >> >> >> >> 192.168.100.131:35500]
> > > >> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
> > > >> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
> > > >> > > >> >> >> >> ignoring exception during input shutdown
> > > >> > > >> >> >> >> java.net.SocketException: Transport endpoint is not
> > > >> connected
> > > >> > > >> >> >> >> at
> > sun.nio.ch.SocketChannelImpl.shutdown(Native
> > > >> > Method)
> > > >> > > >> >> >> >> at
> > > >> > > >> >> >> >>
> > > >> > > >> >>
> > > >> > >
> > > sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
> > > >> > > >> >> >> >> at
> > > >> > > >> >> >>
> > > >> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> > > >> > > >> >> >> >> at
> > > >> > > >> >> >> >>
> > > >> > > >> >>
> > > >> > >
> > > >>
> > org.apache.zookeeper.server.NIOServerCnxn.close(NIOServerCnxn.java:848)
> > > >> > > >> >> >> >> at
> > > >> > > >> >> >> >>
> > > >> > > >> >>
> > > >> > >
> > > org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:498)
> > > >> > > >> >> >> >> at
> > > >> > > >> >> >> >>
> > > >> > > >> >> >>
> > > >> > > >> >>
> > > >> > > >>
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:221)
> > > >> > > >> >> >> >>
> > > >> > > >> >> >> >
> > > >> > > >> >> >>
> > > >> > > >> >> >
> > > >> > > >> >>
> > > >> > > >> >
> > > >> > > >>
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > > >
> > > >
> > >
> >
>
Re: regionserver disconnection
Posted by Zhenyu Zhong <zh...@gmail.com>.
Stack,
I am very appreciated for your great help and insightful comments.
In my cluster, half of the cluster have 2 disks 400GB each per machine, and
half of the cluster have 6 disks per machine. Maybe we should run zookeeper
on the machines with 2 disks and RS on machines with 6 disks?
BTW, the 10 minutes zookeeper.session.timeout has been set during the
experiment.
thanks
zhenyu
On Thu, Nov 12, 2009 at 2:08 PM, stack <st...@duboce.net> wrote:
> On Thu, Nov 12, 2009 at 8:40 AM, Zhenyu Zhong <zhongresearch@gmail.com
> >wrote:
>
>
> > Though I experienced 2 regionserver disconnection this morning, it looks
> > better from the regionserver log. (please see the following log)
> >
> >
> > http://pastebin.com/m496dbfae
> >
> >
> > I checked diskIO for one of the regionserver(192.168.100.116) during the
> > time it disconnected.
> >
> > Time: 03:04:58 AM
> > Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
> > sda 105.31 5458.83 19503.64 9043873239 32312473676
> > sda1 2.90 6.64 99.25 10993934 164433464
> > sda2 1.72 23.76 30.25 39365817 50124008
> > sda3 0.30 0.38 3.58 624930 5923000
> > sda4 100.39 5428.06 19370.56 8992888270 32091993204
> >
> >
> Is this high for you? 20k blocks/second would seem to be high but its one
> disk only and its not being shared by zk anymore so shouldn't matter?
>
>
> > I also checked the zookeeper quorum server that the regionserver tried to
> > connect according to the log. However, I don't see 192.168.100.116 in the
> > client list of the zookeeper quorum server that the regionserver tried to
> > connect.
> > Would that be a problem?
> >
> >
> Is that because the ephemeral node for the regionserver had evaporated?
> Lost
> its lease w/ zk by the time you went to look?
>
>
>
>
> > Thu Nov 12 15:04:52 UTC 2009
> > Zookeeper version: 3.2.1-808558, built on 08/27/2009 18:48 GMT
> > Clients:
> > /192.168.100.127:43045[1](queued=0,recved=26,sent=0)
> > /192.168.100.131:39091[1](queued=0,recved=964,sent=0)
> > /192.168.100.124:35961[1](queued=0,recved=3266,sent=0)
> > /192.168.100.123:47935[1](queued=0,recved=233,sent=0)
> > /192.168.100.125:46931[1](queued=0,recved=2,sent=0)
> > /192.168.100.118:54924[1](queued=0,recved=295,sent=0)
> > /192.168.100.118:41390[1](queued=0,recved=2290,sent=0)
> > /192.168.100.136:42243[1](queued=0,recved=0,sent=0)
> >
> > Latency min/avg/max: 0/17/6333
> > Received: 47111
> > Sent: 0
> > Outstanding: 0
> > Zxid: 0x77000083f4
> > Mode: leader
> > Node count: 23
> >
>
>
> That 6 second maximum latency is pretty bad but should be well within the
> zk
> session timeout.
>
> So, problem is likely on the zk client side of the session; i.e. in the RS.
> You could enable GC logging as J-D suggested to see if you have any big
> pauses, pauses > zk session timeout.
>
> When the RS went down, it didn't look too heavily loaded:
>
>
> 1. 2009-11-12 15:04:52,830 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics:
> request=1.5166667, regions=322, stores=657, storefiles=631,
> storefileIndexSize=61, memstoreSize=1472, usedHeap=2819, maxHeap=4079,
> blockCacheSize=658110960, blockCacheFree=197395984, blockCacheCount=9903,
> blockCacheHitRatio=99
>
> Its serving a few reads? The number of store files seems fine. Not too
> much memory used.
>
> Looking at the logs, I see the Lease Still Held exception. This happens
> when the RS does its regular report to the master but the master thinks the
> RS has since restarted. It'll think this probably because it noticed that
> the RS's znode in zk had gone away and it considered the RS dead.
>
> Looking too at your logs I see this gap in the zk pinging:
>
>
> 1. 2009-11-12 15:03:39,325 DEBUG org.apache.zookeeper.ClientCnxn: Got
> ping response for sessionid:0x224e55436ad0004 after 0ms
> 2. 2009-11-12 15:03:43,113 DEBUG org.apache.zookeeper.ClientCnxn: Got
> ping response for sessionid:0x24e55436a0007d after 0ms
>
> Where in the lines above it, its reporting about every ten seconds, here
> there is a big gap.
>
> Do you have ganglia or something that will let you look more into what was
> happening on this machine around this time? Is the machine OK? It looks
> lightly loaded and you have your cluster nicely laid out. Something odd is
> going on. What about things like the write speed to disk? In the past
> strange issues have been explained by incorrectly set BIOS which made disks
> run at 1/100th of their proper speed.
>
> St.Ack
>
>
>
>
>
>
>
> >
> > Best,
> > zhenyu
> >
> >
> > On Wed, Nov 11, 2009 at 3:58 PM, Zhenyu Zhong <zhongresearch@gmail.com
> > >wrote:
> >
> > > Stack
> > >
> > > I am very appreciated for your comments.
> > > I will use the zookeeper monitoring script on my cluster and let it run
> > > overnight to see the result.
> > > I will follow up the thread when I get anything.
> > >
> > > thanks
> > > zhenyu
> > >
> > >
> > > On Wed, Nov 11, 2009 at 3:52 PM, stack <st...@duboce.net> wrote:
> > >
> > >> I see these in your log too:
> > >>
> > >>
> > >> 1. 2009-11-11 04:27:19,018 DEBUG org.apache.zookeeper.ClientCnxn:
> Got
> > >> ping response for sessionid:0x424dfd908c50009 after 4544ms
> > >> 2. 2009-11-11 04:27:19,018 DEBUG org.apache.zookeeper.ClientCnxn:
> Got
> > >> ping response for sessionid:0x24dfd90c810002 after 4548ms
> > >> 3. 2009-11-11 04:27:43,960 DEBUG org.apache.zookeeper.ClientCnxn:
> Got
> > >> ping response for sessionid:0x424dfd908c50009 after 9030ms
> > >> 4. 2009-11-11 04:27:43,960 DEBUG org.apache.zookeeper.ClientCnxn:
> Got
> > >> ping response for sessionid:0x24dfd90c810002 after 9030ms
> > >> 5. 2009-11-11 04:27:48,830 DEBUG org.apache.zookeeper.ClientCnxn:
> Got
> > >> ping response for sessionid:0x424dfd908c50009 after 0ms
> > >> 6. 2009-11-11 04:27:48,830 DEBUG org.apache.zookeeper.ClientCnxn:
> Got
> > >> ping response for sessionid:0x24dfd90c810002 after 0ms
> > >>
> > >> Which would indicate lag talking to zk too.
> > >>
> > >> Appending HLog also seems to get steadily worse.
> > >>
> > >> I also see these:
> > >>
> > >>
> > >> 1. 2009-11-11 04:27:01,233 WARN org.apache.zookeeper.ClientCnxn:
> > >> Exception closing session 0x24dfd90c810002 to
> > >> sun.nio.ch.SelectionKeyImpl@5e16a7e3
> > >> 2. java.io.IOException: Read error rc = -1
> > >> java.nio.DirectByteBuffer[pos=0 lim=4 cap=4]
> > >> 3. at
> > >>
> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
> > >> 4. at
> > >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
> > >>
> > >>
> > >> Were all 5 quorum members participating? Check by doing "echo stat|nc
> > >> QUORUM_HOST port" for each member of quorum.... the below might help:
> > >>
> > >> members="1,2,3,4,5"
> > >> while true; do date; for i in $members; do echo $i; echo stat | nc $i
> > >> 2181;
> > >> done; sleep 10; done
> > >>
> > >> We need to improve our zk_dump. It just reports quorum members... not
> > if
> > >> they are actually participating or not. The above will tell you if
> dead
> > >> or
> > >> alive.
> > >>
> > >> Please report back after checking above and following the Jean-Daniel
> > >> prescription.
> > >>
> > >> Thanks,
> > >> St.Ack
> > >>
> > >>
> > >>
> > >> On Wed, Nov 11, 2009 at 12:29 PM, Zhenyu Zhong <
> zhongresearch@gmail.com
> > >> >wrote:
> > >>
> > >> > Thanks a lot for the help.
> > >> >
> > >> > I understand that the change of disk configurations should be done
> > >> > progressively such that HDFS can balance and recover from a disk
> going
> > >> > away.
> > >> >
> > >> > The data we are trying to import into HBase is about 200million
> rows,
> > >> each
> > >> > row is about hundreds of bytes. Basically, we read in a text file
> from
> > >> > HDFS,
> > >> > then for each row in the text file, we fetch data from HBase table,
> > we
> > >> > compare the column value and update the latest value back into
> HBase.
> > I
> > >> was
> > >> > suggested to use CheckAndPut. Anyway this is the common usecase on
> my
> > >> > side.
> > >> >
> > >> >
> > >> > Many thanks
> > >> > zhenyu
> > >> >
> > >> >
> > >> >
> > >> > On Wed, Nov 11, 2009 at 3:03 PM, Jean-Daniel Cryans <
> > >> jdcryans@apache.org
> > >> > >wrote:
> > >> >
> > >> > > 6 disks, 19 machines and you see those kind of issues? I'm quite
> > >> > > surprised and I want to learn more about what you trying to to.
> > >> > >
> > >> > > But first, if hbase is managing ZK for you then you can change the
> > >> > > value of hbase.zookeeper.property.dataDir so that it points to its
> > own
> > >> > > mount. Make sure you also change dfs.data.dir in
> conf/hdfs-site.xml
> > >> > > (hadoop stuff) so that it stops pointing there (you should have 6
> > >> > > values there at the moment if you configured it correctly). Doing
> > this
> > >> > > could result in data loss if you do it all at once so do the
> change
> > >> > > progressively. This also means that you may have to keep a
> seperate
> > >> > > HDFS configuration for those particular nodes. Please ask more
> > >> > > questions if it's not very clear.
> > >> > >
> > >> > > WRT to dedicating nodes to Zookeeeper, it makes sense only if you
> > have
> > >> > > a lot of nodes but in your case, unless there's also CPU
> contention,
> > >> > > you're good with only a dedicated disk.
> > >> > >
> > >> > > Now for what you are trying to do, is this a massive import that
> > goes
> > >> > > over many hours? How big is each row? I'd love to get a lot of
> > details
> > >> > > to help you optimize your cluster for your particular problem.
> > >> > >
> > >> > > Thx,
> > >> > >
> > >> > > J-D
> > >> > >
> > >> > > On Wed, Nov 11, 2009 at 11:51 AM, Zhenyu Zhong <
> > >> zhongresearch@gmail.com>
> > >> > > wrote:
> > >> > > > J-D,
> > >> > > >
> > >> > > > Thanks a lot for the insightful comments.
> > >> > > > We are running 5 zookeeper quorums now. On these 5 quorum
> servers,
> > >> we
> > >> > > also
> > >> > > > run datanode, tasktracker and regionserver.
> > >> > > > Now we have 6 disks per machine.
> > >> > > > If it is the disk IO saturation, is there anyway to configure
> > >> zookeeper
> > >> > > to
> > >> > > > write to a seperate disk which is not the same one used by HDFS?
> > >> > > > Or shall we run zookeeper quorums on different nodes other than
> > >> HDFS?
> > >> > > >
> > >> > > > Are HLogs generated by Regionservers? Would it alleviate the
> issue
> > >> if
> > >> > we
> > >> > > use
> > >> > > > the first 14 for regionserver, and the rest of 5 for
> zookeeper?(In
> > >> that
> > >> > > > sense, the node running zookeeper quorum won't run regionserver)
> > >> > > >
> > >> > > > I am very appreciated for your comments.
> > >> > > >
> > >> > > > Best,
> > >> > > > zhenyu
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > > On Wed, Nov 11, 2009 at 2:30 PM, Jean-Daniel Cryans <
> > >> > jdcryans@apache.org
> > >> > > >wrote:
> > >> > > >
> > >> > > >> The problem seems to be that the IO is saturated. When you see
> > >> this:
> > >> > > >>
> > >> > > >> #
> > >> > > >> 2009-11-11 04:28:10,308 WARN
> > >> > > >> org.apache.hadoop.hbase.regionserver.HLog: IPC Server handler 9
> > on
> > >> > > >> 60021 took 6819ms appending an edit to hlog; editcount=370
> > >> > > >> #
> > >> > > >> 2009-11-11 04:28:10,308 WARN
> > >> > > >> org.apache.hadoop.hbase.regionserver.HLog: IPC Server handler 2
> > on
> > >> > > >> 60021 took 4412ms appending an edit to hlog; editcount=371
> > >> > > >>
> > >> > > >> It's a sign that HDFS is slow. I guess that the zookeeper
> quorum
> > >> > > >> members are on those same nodes using the same disks? If so,
> > >> there's
> > >> > > >> the problem because Zookeeper needs to be able to commit
> > >> transactions
> > >> > > >> very fast and if your disks are already hogged then they have
> to
> > >> wait
> > >> > > >> and you can get disconnections. Do you still have 19 quorum
> > >> members?
> > >> > > >> How many disks do you have per machine?
> > >> > > >>
> > >> > > >> Thx
> > >> > > >>
> > >> > > >> J-D
> > >> > > >>
> > >> > > >> On Wed, Nov 11, 2009 at 11:22 AM, Zhenyu Zhong <
> > >> > zhongresearch@gmail.com
> > >> > > >
> > >> > > >> wrote:
> > >> > > >> > Please take a look at the following regionserver log.
> > >> > > >> > The error started to come out from 4:24am UTC to 4:29am UTC.
> > >> > > >> >
> > >> > > >> > http://pastebin.com/m3717d064
> > >> > > >> >
> > >> > > >> > Thanks in advance.
> > >> > > >> >
> > >> > > >> > Best,
> > >> > > >> > zhenyu
> > >> > > >> >
> > >> > > >> >
> > >> > > >> > On Wed, Nov 11, 2009 at 2:01 PM, Jean-Daniel Cryans <
> > >> > > jdcryans@apache.org
> > >> > > >> >wrote:
> > >> > > >> >
> > >> > > >> >> Could you paste a bigger log trace (interested to see at
> least
> > >> all
> > >> > > 300
> > >> > > >> >> lines before that) in a pastebin?
> > >> > > >> >>
> > >> > > >> >> Thx,
> > >> > > >> >>
> > >> > > >> >> J-D
> > >> > > >> >>
> > >> > > >> >> On Wed, Nov 11, 2009 at 10:56 AM, Zhenyu Zhong <
> > >> > > zhongresearch@gmail.com
> > >> > > >> >
> > >> > > >> >> wrote:
> > >> > > >> >> > J-D,
> > >> > > >> >> >
> > >> > > >> >> > Thanks for the reply.
> > >> > > >> >> >
> > >> > > >> >> > The problem I have is one or more of the regionservers
> > >> disconnect
> > >> > > from
> > >> > > >> >> the
> > >> > > >> >> > cluster.
> > >> > > >> >> >
> > >> > > >> >> > I checked the resource usage including memory Heap usage
> for
> > >> > > >> >> regionserver,
> > >> > > >> >> > and found nothing suspicious. The regionserver heap usage
> is
> > >> way
> > >> > > below
> > >> > > >> >> the
> > >> > > >> >> > maxHeap defined in jvm.
> > >> > > >> >> >
> > >> > > >> >> > So I checked zookeeper log, then I found the following
> > >> > > >> >> > BufferUnderflowException, eventually it leads to
> > >> SocketException:
> > >> > > >> >> Transport
> > >> > > >> >> > endpoint is not connected.
> > >> > > >> >> >
> > >> > > >> >> >
> > >> > > >> >> > I also checked the regionserver log. It looks like the
> > >> > regionserver
> > >> > > >> tried
> > >> > > >> >> to
> > >> > > >> >> > connect to other zookeeper quorum server. It connects
> > >> > successfully,
> > >> > > >> but
> > >> > > >> >> then
> > >> > > >> >> > it complained about SocketException: transport endpoint is
> > not
> > >> > > >> connected.
> > >> > > >> >> >
> > >> > > >> >> > So both zookeeper log and regionserver log told me that
> the
> > >> > > transport
> > >> > > >> >> > endpoint is not connected. I don't know why.
> > >> > > >> >> >
> > >> > > >> >> > I would be very appreciated if you could point me to the
> > right
> > >> > > >> direction.
> > >> > > >> >> >
> > >> > > >> >> > Best,
> > >> > > >> >> >
> > >> > > >> >> >
> > >> > > >> >> > Regionserver log:
> > >> > > >> >> > 2009-11-11 04:24:53,718 INFO
> > org.apache.zookeeper.ClientCnxn:
> > >> > > >> Attempting
> > >> > > >> >> > connection to server superpyxis0018/192.168.100.132:2181
> > >> > > >> >> > 2009-11-11 04:24:53,718 INFO
> > org.apache.zookeeper.ClientCnxn:
> > >> > > Priming
> > >> > > >> >> > connection to java.nio.channels.SocketChannel[connected
> > >> local=/
> > >> > > >> >> > 192.168.100.131:43065 remote=superpyxis0018/
> > >> 192.168.100.132:2181
> > >> > ]
> > >> > > >> >> > 2009-11-11 04:24:53,718 INFO
> > org.apache.zookeeper.ClientCnxn:
> > >> > > Server
> > >> > > >> >> > connection successful
> > >> > > >> >> > 2009-11-11 04:24:53,768 INFO
> > >> > > >> >> > org.apache.hadoop.hbase.regionserver.HRegionServer: Got
> > >> ZooKeeper
> > >> > > >> event,
> > >> > > >> >> > state: Disconnected, type: None, path: null
> > >> > > >> >> > 2009-11-11 04:24:54,118 INFO
> > org.apache.zookeeper.ClientCnxn:
> > >> > > >> Attempting
> > >> > > >> >> > connection to server superpyxis0017/192.168.100.131:2181
> > >> > > >> >> > 2009-11-11 04:24:54,118 INFO
> > org.apache.zookeeper.ClientCnxn:
> > >> > > Priming
> > >> > > >> >> > connection to java.nio.channels.SocketChannel[connected
> > >> local=/
> > >> > > >> >> > 192.168.100.131:46903 remote=superpyxis0017/
> > >> 192.168.100.131:2181
> > >> > ]
> > >> > > >> >> > 2009-11-11 04:24:54,118 INFO
> > org.apache.zookeeper.ClientCnxn:
> > >> > > Server
> > >> > > >> >> > connection successful
> > >> > > >> >> > 2009-11-11 04:24:54,118 WARN
> > org.apache.zookeeper.ClientCnxn:
> > >> > > >> Exception
> > >> > > >> >> > closing session 0x24dfd90c810002 to
> > >> > > >> sun.nio.ch.SelectionKeyImpl@2dd78ccf
> > >> > > >> >> > java.io.IOException: Read error rc = -1
> > >> > > >> java.nio.DirectByteBuffer[pos=0
> > >> > > >> >> > lim=4 cap=4]
> > >> > > >> >> > at
> > >> > > >> >> >
> > >> > >
> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
> > >> > > >> >> > at
> > >> > > >> >> >
> > >> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
> > >> > > >> >> > 2009-11-11 04:24:54,118 WARN
> > org.apache.zookeeper.ClientCnxn:
> > >> > > Ignoring
> > >> > > >> >> > exception during shutdown input
> > >> > > >> >> > java.net.SocketException: Transport endpoint is not
> > connected
> > >> > > >> >> > at sun.nio.ch.SocketChannelImpl.shutdown(Native
> > Method)
> > >> > > >> >> > at
> > >> > > >> >> >
> > >> > >
> > sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
> > >> > > >> >> > at
> > >> > > >> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> > >> > > >> >> > at
> > >> > > >> >> >
> > >> > > >>
> > >> >
> > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> > >> > > >> >> > at
> > >> > > >> >> >
> > >> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> > >> > > >> >> >
> > >> > > >> >> >
> > >> > > >> >> >
> > >> > > >> >> >
> > >> > > >> >> >
> > >> > > >> >> > On Wed, Nov 11, 2009 at 1:40 PM, Jean-Daniel Cryans <
> > >> > > >> jdcryans@apache.org
> > >> > > >> >> >wrote:
> > >> > > >> >> >
> > >> > > >> >> >> I'm really not sure exactly what the situation is with
> the
> > >> > > >> >> >> java.nio.BufferUnderflowException, but what do you mean
> by
> > >> > > >> >> >> "regionserver disconnection"? Could you give more
> details?
> > >> > > >> >> >>
> > >> > > >> >> >> Thx,
> > >> > > >> >> >>
> > >> > > >> >> >> J-D
> > >> > > >> >> >>
> > >> > > >> >> >> On Wed, Nov 11, 2009 at 10:34 AM, Zhenyu Zhong <
> > >> > > >> zhongresearch@gmail.com
> > >> > > >> >> >
> > >> > > >> >> >> wrote:
> > >> > > >> >> >> > I searched around and found HBASE-1645 "HBase could be
> a
> > >> > little
> > >> > > >> more
> > >> > > >> >> >> dainty
> > >> > > >> >> >> > when zk is pulled out from under it"
> > >> > > >> >> >> >
> > >> > > >> >> >> > I am wondering whether my problem is related to this
> > >> > HBase-1645.
> > >> > > Or
> > >> > > >> >> >> probably
> > >> > > >> >> >> > has nothing to do with it?
> > >> > > >> >> >> >
> > >> > > >> >> >> > thanks
> > >> > > >> >> >> >
> > >> > > >> >> >> >
> > >> > > >> >> >> > On Wed, Nov 11, 2009 at 12:02 PM, Zhenyu Zhong <
> > >> > > >> >> zhongresearch@gmail.com
> > >> > > >> >> >> >wrote:
> > >> > > >> >> >> >
> > >> > > >> >> >> >> Hi,
> > >> > > >> >> >> >>
> > >> > > >> >> >> >> I have been monitoring the system resources such as
> > memory
> > >> > Heap
> > >> > > >> usage
> > >> > > >> >> >> for
> > >> > > >> >> >> >> Regionserver, load average, network bandwidth, etc.
> > >> > > >> >> >> >>
> > >> > > >> >> >> >> I experienced a disconnection last night. When I
> checked
> > >> the
> > >> > > >> memory
> > >> > > >> >> Heap
> > >> > > >> >> >> >> for Regionserver, it was 2336 which is way below the
> max
> > >> Heap
> > >> > > >> 4096. I
> > >> > > >> >> >> don't
> > >> > > >> >> >> >> think it is Regionserver problem. The requests that
> the
> > >> > > >> regionserver
> > >> > > >> >> got
> > >> > > >> >> >> was
> > >> > > >> >> >> >> about 126 which doesn't seem to be high. The load
> > average
> > >> is
> > >> > > kind
> > >> > > >> of
> > >> > > >> >> >> high
> > >> > > >> >> >> >> >100. (but I saw disconnections while the load average
> > is
> > >> > > around
> > >> > > >> 30).
> > >> > > >> >> >> The
> > >> > > >> >> >> >> bandwidth is not a problem.
> > >> > > >> >> >> >>
> > >> > > >> >> >> >> However, I found error messages from the zookeeper.
> > Please
> > >> > see
> > >> > > the
> > >> > > >> >> >> >> following error messages. There are some
> > >> > > >> >> >> >> BufferUnderflowException error, IOException error,etc.
> > >> > > >> >> >> >> May I ask what might be the causes that generate these
> > >> error
> > >> > > >> >> messages?
> > >> > > >> >> >> >>
> > >> > > >> >> >> >> Many thanks
> > >> > > >> >> >> >> zhenyu
> > >> > > >> >> >> >>
> > >> > > >> >> >> >>
> > >> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
> > >> > > org.apache.zookeeper.server.Request:
> > >> > > >> >> >> Ignoring
> > >> > > >> >> >> >> exception during toString
> > >> > > >> >> >> >> java.nio.BufferUnderflowException
> > >> > > >> >> >> >> at
> > >> > java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
> > >> > > >> >> >> >> at
> java.nio.ByteBuffer.get(ByteBuffer.java:675)
> > >> > > >> >> >> >> at
> > >> > > >> >> >>
> > >> org.apache.zookeeper.server.Request.toString(Request.java:199)
> > >> > > >> >> >> >> at java.lang.String.valueOf(String.java:2827)
> > >> > > >> >> >> >> at
> > >> > > java.lang.StringBuilder.append(StringBuilder.java:115)
> > >> > > >> >> >> >> at
> > >> > > >> >> >> >>
> > >> > > >> >> >>
> > >> > > >> >>
> > >> > > >>
> > >> > >
> > >> >
> > >>
> >
> org.apache.zookeeper.server.quorum.CommitProcessor.processRequest(CommitProcessor.java:167)
> > >> > > >> >> >> >> at
> > >> > > >> >> >> >>
> > >> > > >> >> >>
> > >> > > >> >>
> > >> > > >>
> > >> > >
> > >> >
> > >>
> >
> org.apache.zookeeper.server.quorum.FollowerRequestProcessor.run(FollowerRequestProcessor.java:68)
> > >> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
> > >> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
> > >> > > >> >> >> >> Exception causing close of session 0x424dfd908c50009
> > >> > > >> >> >> >> due to java.io.IOException: Read error
> > >> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
> > >> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
> > >> > > >> >> >> >> IOException stack trace
> > >> > > >> >> >> >> java.io.IOException: Read error
> > >> > > >> >> >> >> at
> > >> > > >> >> >> >>
> > >> > > >> >>
> > >> > >
> > org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:372)
> > >> > > >> >> >> >> at
> > >> > > >> >> >> >>
> > >> > > >> >> >>
> > >> > > >> >>
> > >> > > >>
> > >> > >
> > >> >
> > >>
> >
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:221)
> > >> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
> > >> > > >> >> >> >> org.apache.zookeeper.server.quorum.CommitProcessor:
> > >> > Processing
> > >> > > >> >> request::
> > >> > > >> >> >> >> sessionid:0x424dfd908c50009 type:setWatches
> > >> > > >> cxid:0xfffffffffffffff8
> > >> > > >> >> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
> > >> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
> > >> > > org.apache.zookeeper.server.Request:
> > >> > > >> >> >> Ignoring
> > >> > > >> >> >> >> exception during toString
> > >> > > >> >> >> >> java.nio.BufferUnderflowException
> > >> > > >> >> >> >> at
> > >> > java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
> > >> > > >> >> >> >> at
> java.nio.ByteBuffer.get(ByteBuffer.java:675)
> > >> > > >> >> >> >> at
> > >> > > >> >> >>
> > >> org.apache.zookeeper.server.Request.toString(Request.java:199)
> > >> > > >> >> >> >> at java.lang.String.valueOf(String.java:2827)
> > >> > > >> >> >> >> at
> > >> > > java.lang.StringBuilder.append(StringBuilder.java:115)
> > >> > > >> >> >> >> at
> > >> > > >> >> >> >>
> > >> > > >> >> >>
> > >> > > >> >>
> > >> > > >>
> > >> > >
> > >> >
> > >>
> >
> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:74)
> > >> > > >> >> >> >> at
> > >> > > >> >> >> >>
> > >> > > >> >> >>
> > >> > > >> >>
> > >> > > >>
> > >> > >
> > >> >
> > >>
> >
> org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)
> > >> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
> > >> > > >> >> >> >> org.apache.zookeeper.server.FinalRequestProcessor:
> > >> Processing
> > >> > > >> >> request::
> > >> > > >> >> >> >> sessionid:0x424dfd908c50009 type:setWatches
> > >> > > >> cxid:0xfffffffffffffff8
> > >> > > >> >> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
> > >> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
> > >> > > org.apache.zookeeper.server.Request:
> > >> > > >> >> >> Ignoring
> > >> > > >> >> >> >> exception during toString
> > >> > > >> >> >> >> java.nio.BufferUnderflowException
> > >> > > >> >> >> >> at
> > >> > java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
> > >> > > >> >> >> >> at
> java.nio.ByteBuffer.get(ByteBuffer.java:675)
> > >> > > >> >> >> >> at
> > >> > > >> >> >>
> > >> org.apache.zookeeper.server.Request.toString(Request.java:199)
> > >> > > >> >> >> >> at
> > >> > > >> >> >> >>
> > >> > > >>
> > >> org.apache.log4j.or.DefaultRenderer.doRender(DefaultRenderer.java:36)
> > >> > > >> >> >> >> at
> > >> > > >> >> >> >>
> > >> > > org.apache.log4j.or.RendererMap.findAndRender(RendererMap.java:80)
> > >> > > >> >> >> >> at
> > >> > > >> >> >> >>
> > >> > > >> >> >>
> > >> > > >> >>
> > >> > > >>
> > >> > >
> > >> >
> > >>
> >
> org.apache.log4j.spi.LoggingEvent.getRenderedMessage(LoggingEvent.java:362)
> > >> > > >> >> >> >> at
> > >> > > >> >> >> >>
> > >> > > >> >> >>
> > >> > > >> >>
> > >> > > >>
> > >> > >
> > >> >
> > >>
> >
> org.apache.log4j.helpers.PatternParser$BasicPatternConverter.convert(PatternParser.java:403)
> > >> > > >> >> >> >> at
> > >> > > >> >> >> >>
> > >> > > >> >> >>
> > >> > > >> >>
> > >> > > >>
> > >> > >
> > >> >
> > >>
> >
> org.apache.log4j.helpers.PatternConverter.format(PatternConverter.java:65)
> > >> > > >> >> >> >> at
> > >> > > >> >>
> org.apache.log4j.PatternLayout.format(PatternLayout.java:502)
> > >> > > >> >> >> >> at
> > >> > > >> >> >> >>
> > >> > > org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:302)
> > >> > > >> >> >> >> at
> > >> > > >> >> >> >>
> > >> > > >> >> >>
> > >> > > >> >>
> > >> > > >>
> > >> > >
> > >> >
> > >>
> >
> org.apache.log4j.DailyRollingFileAppender.subAppend(DailyRollingFileAppender.java:359)
> > >> > > >> >> >> >> at
> > >> > > >> >> >>
> > >> org.apache.log4j.WriterAppender.append(WriterAppender.java:160)
> > >> > > >> >> >> >> at
> > >> > > >> >> >> >>
> > >> > > >>
> > >> org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
> > >> > > >> >> >> >> at
> > >> > > >> >> >> >>
> > >> > > >> >> >>
> > >> > > >> >>
> > >> > > >>
> > >> > >
> > >> >
> > >>
> >
> org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
> > >> > > >> >> >> >> at
> > >> > > >> org.apache.log4j.Category.callAppenders(Category.java:206)
> > >> > > >> >> >> >> at
> > >> > > org.apache.log4j.Category.forcedLog(Category.java:391)
> > >> > > >> >> >> >> at
> > >> org.apache.log4j.Category.debug(Category.java:260)
> > >> > > >> >> >> >> at
> > >> > > >> >> >> >>
> > >> > > >> >> >>
> > >> > > >> >>
> > >> > > >>
> > >> > >
> > >> >
> > >>
> >
> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:147)
> > >> > > >> >> >> >> at
> > >> > > >> >> >> >>
> > >> > > >> >> >>
> > >> > > >> >>
> > >> > > >>
> > >> > >
> > >> >
> > >>
> >
> org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)
> > >> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
> > >> > > org.apache.zookeeper.server.Request:
> > >> > > >> >> >> Ignoring
> > >> > > >> >> >> >> exception during toString
> > >> > > >> >> >> >> sessionid:0x424dfd908c50009 type:setWatches
> > >> > > >> cxid:0xfffffffffffffff8
> > >> > > >> >> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
> > >> > > >> >> >> >> 2009-11-11 04:29:00,810 INFO
> > >> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
> > >> > > >> >> >> >> closing session:0x424dfd908c50009 NIOServerCnxn:
> > >> > > >> >> >> >> java.nio.channels.SocketChannel[connected local=/
> > >> > > >> >> 192.168.100.131:2181
> > >> > > >> >> >> remote=/
> > >> > > >> >> >> >> 192.168.100.131:35500]
> > >> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
> > >> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
> > >> > > >> >> >> >> ignoring exception during input shutdown
> > >> > > >> >> >> >> java.net.SocketException: Transport endpoint is not
> > >> connected
> > >> > > >> >> >> >> at
> sun.nio.ch.SocketChannelImpl.shutdown(Native
> > >> > Method)
> > >> > > >> >> >> >> at
> > >> > > >> >> >> >>
> > >> > > >> >>
> > >> > >
> > sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
> > >> > > >> >> >> >> at
> > >> > > >> >> >>
> > >> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> > >> > > >> >> >> >> at
> > >> > > >> >> >> >>
> > >> > > >> >>
> > >> > >
> > >>
> org.apache.zookeeper.server.NIOServerCnxn.close(NIOServerCnxn.java:848)
> > >> > > >> >> >> >> at
> > >> > > >> >> >> >>
> > >> > > >> >>
> > >> > >
> > org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:498)
> > >> > > >> >> >> >> at
> > >> > > >> >> >> >>
> > >> > > >> >> >>
> > >> > > >> >>
> > >> > > >>
> > >> > >
> > >> >
> > >>
> >
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:221)
> > >> > > >> >> >> >>
> > >> > > >> >> >> >
> > >> > > >> >> >>
> > >> > > >> >> >
> > >> > > >> >>
> > >> > > >> >
> > >> > > >>
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> > >
> >
>
Re: regionserver disconnection
Posted by stack <st...@duboce.net>.
On Thu, Nov 12, 2009 at 8:40 AM, Zhenyu Zhong <zh...@gmail.com>wrote:
> Though I experienced 2 regionserver disconnection this morning, it looks
> better from the regionserver log. (please see the following log)
>
>
> http://pastebin.com/m496dbfae
>
>
> I checked diskIO for one of the regionserver(192.168.100.116) during the
> time it disconnected.
>
> Time: 03:04:58 AM
> Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
> sda 105.31 5458.83 19503.64 9043873239 32312473676
> sda1 2.90 6.64 99.25 10993934 164433464
> sda2 1.72 23.76 30.25 39365817 50124008
> sda3 0.30 0.38 3.58 624930 5923000
> sda4 100.39 5428.06 19370.56 8992888270 32091993204
>
>
Is this high for you? 20k blocks/second would seem to be high but its one
disk only and its not being shared by zk anymore so shouldn't matter?
> I also checked the zookeeper quorum server that the regionserver tried to
> connect according to the log. However, I don't see 192.168.100.116 in the
> client list of the zookeeper quorum server that the regionserver tried to
> connect.
> Would that be a problem?
>
>
Is that because the ephemeral node for the regionserver had evaporated? Lost
its lease w/ zk by the time you went to look?
> Thu Nov 12 15:04:52 UTC 2009
> Zookeeper version: 3.2.1-808558, built on 08/27/2009 18:48 GMT
> Clients:
> /192.168.100.127:43045[1](queued=0,recved=26,sent=0)
> /192.168.100.131:39091[1](queued=0,recved=964,sent=0)
> /192.168.100.124:35961[1](queued=0,recved=3266,sent=0)
> /192.168.100.123:47935[1](queued=0,recved=233,sent=0)
> /192.168.100.125:46931[1](queued=0,recved=2,sent=0)
> /192.168.100.118:54924[1](queued=0,recved=295,sent=0)
> /192.168.100.118:41390[1](queued=0,recved=2290,sent=0)
> /192.168.100.136:42243[1](queued=0,recved=0,sent=0)
>
> Latency min/avg/max: 0/17/6333
> Received: 47111
> Sent: 0
> Outstanding: 0
> Zxid: 0x77000083f4
> Mode: leader
> Node count: 23
>
That 6 second maximum latency is pretty bad but should be well within the zk
session timeout.
So, problem is likely on the zk client side of the session; i.e. in the RS.
You could enable GC logging as J-D suggested to see if you have any big
pauses, pauses > zk session timeout.
When the RS went down, it didn't look too heavily loaded:
1. 2009-11-12 15:04:52,830 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics:
request=1.5166667, regions=322, stores=657, storefiles=631,
storefileIndexSize=61, memstoreSize=1472, usedHeap=2819, maxHeap=4079,
blockCacheSize=658110960, blockCacheFree=197395984, blockCacheCount=9903,
blockCacheHitRatio=99
Its serving a few reads? The number of store files seems fine. Not too
much memory used.
Looking at the logs, I see the Lease Still Held exception. This happens
when the RS does its regular report to the master but the master thinks the
RS has since restarted. It'll think this probably because it noticed that
the RS's znode in zk had gone away and it considered the RS dead.
Looking too at your logs I see this gap in the zk pinging:
1. 2009-11-12 15:03:39,325 DEBUG org.apache.zookeeper.ClientCnxn: Got
ping response for sessionid:0x224e55436ad0004 after 0ms
2. 2009-11-12 15:03:43,113 DEBUG org.apache.zookeeper.ClientCnxn: Got
ping response for sessionid:0x24e55436a0007d after 0ms
Where in the lines above it, its reporting about every ten seconds, here
there is a big gap.
Do you have ganglia or something that will let you look more into what was
happening on this machine around this time? Is the machine OK? It looks
lightly loaded and you have your cluster nicely laid out. Something odd is
going on. What about things like the write speed to disk? In the past
strange issues have been explained by incorrectly set BIOS which made disks
run at 1/100th of their proper speed.
St.Ack
>
> Best,
> zhenyu
>
>
> On Wed, Nov 11, 2009 at 3:58 PM, Zhenyu Zhong <zhongresearch@gmail.com
> >wrote:
>
> > Stack
> >
> > I am very appreciated for your comments.
> > I will use the zookeeper monitoring script on my cluster and let it run
> > overnight to see the result.
> > I will follow up the thread when I get anything.
> >
> > thanks
> > zhenyu
> >
> >
> > On Wed, Nov 11, 2009 at 3:52 PM, stack <st...@duboce.net> wrote:
> >
> >> I see these in your log too:
> >>
> >>
> >> 1. 2009-11-11 04:27:19,018 DEBUG org.apache.zookeeper.ClientCnxn: Got
> >> ping response for sessionid:0x424dfd908c50009 after 4544ms
> >> 2. 2009-11-11 04:27:19,018 DEBUG org.apache.zookeeper.ClientCnxn: Got
> >> ping response for sessionid:0x24dfd90c810002 after 4548ms
> >> 3. 2009-11-11 04:27:43,960 DEBUG org.apache.zookeeper.ClientCnxn: Got
> >> ping response for sessionid:0x424dfd908c50009 after 9030ms
> >> 4. 2009-11-11 04:27:43,960 DEBUG org.apache.zookeeper.ClientCnxn: Got
> >> ping response for sessionid:0x24dfd90c810002 after 9030ms
> >> 5. 2009-11-11 04:27:48,830 DEBUG org.apache.zookeeper.ClientCnxn: Got
> >> ping response for sessionid:0x424dfd908c50009 after 0ms
> >> 6. 2009-11-11 04:27:48,830 DEBUG org.apache.zookeeper.ClientCnxn: Got
> >> ping response for sessionid:0x24dfd90c810002 after 0ms
> >>
> >> Which would indicate lag talking to zk too.
> >>
> >> Appending HLog also seems to get steadily worse.
> >>
> >> I also see these:
> >>
> >>
> >> 1. 2009-11-11 04:27:01,233 WARN org.apache.zookeeper.ClientCnxn:
> >> Exception closing session 0x24dfd90c810002 to
> >> sun.nio.ch.SelectionKeyImpl@5e16a7e3
> >> 2. java.io.IOException: Read error rc = -1
> >> java.nio.DirectByteBuffer[pos=0 lim=4 cap=4]
> >> 3. at
> >> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
> >> 4. at
> >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
> >>
> >>
> >> Were all 5 quorum members participating? Check by doing "echo stat|nc
> >> QUORUM_HOST port" for each member of quorum.... the below might help:
> >>
> >> members="1,2,3,4,5"
> >> while true; do date; for i in $members; do echo $i; echo stat | nc $i
> >> 2181;
> >> done; sleep 10; done
> >>
> >> We need to improve our zk_dump. It just reports quorum members... not
> if
> >> they are actually participating or not. The above will tell you if dead
> >> or
> >> alive.
> >>
> >> Please report back after checking above and following the Jean-Daniel
> >> prescription.
> >>
> >> Thanks,
> >> St.Ack
> >>
> >>
> >>
> >> On Wed, Nov 11, 2009 at 12:29 PM, Zhenyu Zhong <zhongresearch@gmail.com
> >> >wrote:
> >>
> >> > Thanks a lot for the help.
> >> >
> >> > I understand that the change of disk configurations should be done
> >> > progressively such that HDFS can balance and recover from a disk going
> >> > away.
> >> >
> >> > The data we are trying to import into HBase is about 200million rows,
> >> each
> >> > row is about hundreds of bytes. Basically, we read in a text file from
> >> > HDFS,
> >> > then for each row in the text file, we fetch data from HBase table,
> we
> >> > compare the column value and update the latest value back into HBase.
> I
> >> was
> >> > suggested to use CheckAndPut. Anyway this is the common usecase on my
> >> > side.
> >> >
> >> >
> >> > Many thanks
> >> > zhenyu
> >> >
> >> >
> >> >
> >> > On Wed, Nov 11, 2009 at 3:03 PM, Jean-Daniel Cryans <
> >> jdcryans@apache.org
> >> > >wrote:
> >> >
> >> > > 6 disks, 19 machines and you see those kind of issues? I'm quite
> >> > > surprised and I want to learn more about what you trying to to.
> >> > >
> >> > > But first, if hbase is managing ZK for you then you can change the
> >> > > value of hbase.zookeeper.property.dataDir so that it points to its
> own
> >> > > mount. Make sure you also change dfs.data.dir in conf/hdfs-site.xml
> >> > > (hadoop stuff) so that it stops pointing there (you should have 6
> >> > > values there at the moment if you configured it correctly). Doing
> this
> >> > > could result in data loss if you do it all at once so do the change
> >> > > progressively. This also means that you may have to keep a seperate
> >> > > HDFS configuration for those particular nodes. Please ask more
> >> > > questions if it's not very clear.
> >> > >
> >> > > WRT to dedicating nodes to Zookeeeper, it makes sense only if you
> have
> >> > > a lot of nodes but in your case, unless there's also CPU contention,
> >> > > you're good with only a dedicated disk.
> >> > >
> >> > > Now for what you are trying to do, is this a massive import that
> goes
> >> > > over many hours? How big is each row? I'd love to get a lot of
> details
> >> > > to help you optimize your cluster for your particular problem.
> >> > >
> >> > > Thx,
> >> > >
> >> > > J-D
> >> > >
> >> > > On Wed, Nov 11, 2009 at 11:51 AM, Zhenyu Zhong <
> >> zhongresearch@gmail.com>
> >> > > wrote:
> >> > > > J-D,
> >> > > >
> >> > > > Thanks a lot for the insightful comments.
> >> > > > We are running 5 zookeeper quorums now. On these 5 quorum servers,
> >> we
> >> > > also
> >> > > > run datanode, tasktracker and regionserver.
> >> > > > Now we have 6 disks per machine.
> >> > > > If it is the disk IO saturation, is there anyway to configure
> >> zookeeper
> >> > > to
> >> > > > write to a seperate disk which is not the same one used by HDFS?
> >> > > > Or shall we run zookeeper quorums on different nodes other than
> >> HDFS?
> >> > > >
> >> > > > Are HLogs generated by Regionservers? Would it alleviate the issue
> >> if
> >> > we
> >> > > use
> >> > > > the first 14 for regionserver, and the rest of 5 for zookeeper?(In
> >> that
> >> > > > sense, the node running zookeeper quorum won't run regionserver)
> >> > > >
> >> > > > I am very appreciated for your comments.
> >> > > >
> >> > > > Best,
> >> > > > zhenyu
> >> > > >
> >> > > >
> >> > > >
> >> > > > On Wed, Nov 11, 2009 at 2:30 PM, Jean-Daniel Cryans <
> >> > jdcryans@apache.org
> >> > > >wrote:
> >> > > >
> >> > > >> The problem seems to be that the IO is saturated. When you see
> >> this:
> >> > > >>
> >> > > >> #
> >> > > >> 2009-11-11 04:28:10,308 WARN
> >> > > >> org.apache.hadoop.hbase.regionserver.HLog: IPC Server handler 9
> on
> >> > > >> 60021 took 6819ms appending an edit to hlog; editcount=370
> >> > > >> #
> >> > > >> 2009-11-11 04:28:10,308 WARN
> >> > > >> org.apache.hadoop.hbase.regionserver.HLog: IPC Server handler 2
> on
> >> > > >> 60021 took 4412ms appending an edit to hlog; editcount=371
> >> > > >>
> >> > > >> It's a sign that HDFS is slow. I guess that the zookeeper quorum
> >> > > >> members are on those same nodes using the same disks? If so,
> >> there's
> >> > > >> the problem because Zookeeper needs to be able to commit
> >> transactions
> >> > > >> very fast and if your disks are already hogged then they have to
> >> wait
> >> > > >> and you can get disconnections. Do you still have 19 quorum
> >> members?
> >> > > >> How many disks do you have per machine?
> >> > > >>
> >> > > >> Thx
> >> > > >>
> >> > > >> J-D
> >> > > >>
> >> > > >> On Wed, Nov 11, 2009 at 11:22 AM, Zhenyu Zhong <
> >> > zhongresearch@gmail.com
> >> > > >
> >> > > >> wrote:
> >> > > >> > Please take a look at the following regionserver log.
> >> > > >> > The error started to come out from 4:24am UTC to 4:29am UTC.
> >> > > >> >
> >> > > >> > http://pastebin.com/m3717d064
> >> > > >> >
> >> > > >> > Thanks in advance.
> >> > > >> >
> >> > > >> > Best,
> >> > > >> > zhenyu
> >> > > >> >
> >> > > >> >
> >> > > >> > On Wed, Nov 11, 2009 at 2:01 PM, Jean-Daniel Cryans <
> >> > > jdcryans@apache.org
> >> > > >> >wrote:
> >> > > >> >
> >> > > >> >> Could you paste a bigger log trace (interested to see at least
> >> all
> >> > > 300
> >> > > >> >> lines before that) in a pastebin?
> >> > > >> >>
> >> > > >> >> Thx,
> >> > > >> >>
> >> > > >> >> J-D
> >> > > >> >>
> >> > > >> >> On Wed, Nov 11, 2009 at 10:56 AM, Zhenyu Zhong <
> >> > > zhongresearch@gmail.com
> >> > > >> >
> >> > > >> >> wrote:
> >> > > >> >> > J-D,
> >> > > >> >> >
> >> > > >> >> > Thanks for the reply.
> >> > > >> >> >
> >> > > >> >> > The problem I have is one or more of the regionservers
> >> disconnect
> >> > > from
> >> > > >> >> the
> >> > > >> >> > cluster.
> >> > > >> >> >
> >> > > >> >> > I checked the resource usage including memory Heap usage for
> >> > > >> >> regionserver,
> >> > > >> >> > and found nothing suspicious. The regionserver heap usage is
> >> way
> >> > > below
> >> > > >> >> the
> >> > > >> >> > maxHeap defined in jvm.
> >> > > >> >> >
> >> > > >> >> > So I checked zookeeper log, then I found the following
> >> > > >> >> > BufferUnderflowException, eventually it leads to
> >> SocketException:
> >> > > >> >> Transport
> >> > > >> >> > endpoint is not connected.
> >> > > >> >> >
> >> > > >> >> >
> >> > > >> >> > I also checked the regionserver log. It looks like the
> >> > regionserver
> >> > > >> tried
> >> > > >> >> to
> >> > > >> >> > connect to other zookeeper quorum server. It connects
> >> > successfully,
> >> > > >> but
> >> > > >> >> then
> >> > > >> >> > it complained about SocketException: transport endpoint is
> not
> >> > > >> connected.
> >> > > >> >> >
> >> > > >> >> > So both zookeeper log and regionserver log told me that the
> >> > > transport
> >> > > >> >> > endpoint is not connected. I don't know why.
> >> > > >> >> >
> >> > > >> >> > I would be very appreciated if you could point me to the
> right
> >> > > >> direction.
> >> > > >> >> >
> >> > > >> >> > Best,
> >> > > >> >> >
> >> > > >> >> >
> >> > > >> >> > Regionserver log:
> >> > > >> >> > 2009-11-11 04:24:53,718 INFO
> org.apache.zookeeper.ClientCnxn:
> >> > > >> Attempting
> >> > > >> >> > connection to server superpyxis0018/192.168.100.132:2181
> >> > > >> >> > 2009-11-11 04:24:53,718 INFO
> org.apache.zookeeper.ClientCnxn:
> >> > > Priming
> >> > > >> >> > connection to java.nio.channels.SocketChannel[connected
> >> local=/
> >> > > >> >> > 192.168.100.131:43065 remote=superpyxis0018/
> >> 192.168.100.132:2181
> >> > ]
> >> > > >> >> > 2009-11-11 04:24:53,718 INFO
> org.apache.zookeeper.ClientCnxn:
> >> > > Server
> >> > > >> >> > connection successful
> >> > > >> >> > 2009-11-11 04:24:53,768 INFO
> >> > > >> >> > org.apache.hadoop.hbase.regionserver.HRegionServer: Got
> >> ZooKeeper
> >> > > >> event,
> >> > > >> >> > state: Disconnected, type: None, path: null
> >> > > >> >> > 2009-11-11 04:24:54,118 INFO
> org.apache.zookeeper.ClientCnxn:
> >> > > >> Attempting
> >> > > >> >> > connection to server superpyxis0017/192.168.100.131:2181
> >> > > >> >> > 2009-11-11 04:24:54,118 INFO
> org.apache.zookeeper.ClientCnxn:
> >> > > Priming
> >> > > >> >> > connection to java.nio.channels.SocketChannel[connected
> >> local=/
> >> > > >> >> > 192.168.100.131:46903 remote=superpyxis0017/
> >> 192.168.100.131:2181
> >> > ]
> >> > > >> >> > 2009-11-11 04:24:54,118 INFO
> org.apache.zookeeper.ClientCnxn:
> >> > > Server
> >> > > >> >> > connection successful
> >> > > >> >> > 2009-11-11 04:24:54,118 WARN
> org.apache.zookeeper.ClientCnxn:
> >> > > >> Exception
> >> > > >> >> > closing session 0x24dfd90c810002 to
> >> > > >> sun.nio.ch.SelectionKeyImpl@2dd78ccf
> >> > > >> >> > java.io.IOException: Read error rc = -1
> >> > > >> java.nio.DirectByteBuffer[pos=0
> >> > > >> >> > lim=4 cap=4]
> >> > > >> >> > at
> >> > > >> >> >
> >> > > org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
> >> > > >> >> > at
> >> > > >> >> >
> >> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
> >> > > >> >> > 2009-11-11 04:24:54,118 WARN
> org.apache.zookeeper.ClientCnxn:
> >> > > Ignoring
> >> > > >> >> > exception during shutdown input
> >> > > >> >> > java.net.SocketException: Transport endpoint is not
> connected
> >> > > >> >> > at sun.nio.ch.SocketChannelImpl.shutdown(Native
> Method)
> >> > > >> >> > at
> >> > > >> >> >
> >> > >
> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
> >> > > >> >> > at
> >> > > >> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> >> > > >> >> > at
> >> > > >> >> >
> >> > > >>
> >> >
> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> >> > > >> >> > at
> >> > > >> >> >
> >> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> >> > > >> >> >
> >> > > >> >> >
> >> > > >> >> >
> >> > > >> >> >
> >> > > >> >> >
> >> > > >> >> > On Wed, Nov 11, 2009 at 1:40 PM, Jean-Daniel Cryans <
> >> > > >> jdcryans@apache.org
> >> > > >> >> >wrote:
> >> > > >> >> >
> >> > > >> >> >> I'm really not sure exactly what the situation is with the
> >> > > >> >> >> java.nio.BufferUnderflowException, but what do you mean by
> >> > > >> >> >> "regionserver disconnection"? Could you give more details?
> >> > > >> >> >>
> >> > > >> >> >> Thx,
> >> > > >> >> >>
> >> > > >> >> >> J-D
> >> > > >> >> >>
> >> > > >> >> >> On Wed, Nov 11, 2009 at 10:34 AM, Zhenyu Zhong <
> >> > > >> zhongresearch@gmail.com
> >> > > >> >> >
> >> > > >> >> >> wrote:
> >> > > >> >> >> > I searched around and found HBASE-1645 "HBase could be a
> >> > little
> >> > > >> more
> >> > > >> >> >> dainty
> >> > > >> >> >> > when zk is pulled out from under it"
> >> > > >> >> >> >
> >> > > >> >> >> > I am wondering whether my problem is related to this
> >> > HBase-1645.
> >> > > Or
> >> > > >> >> >> probably
> >> > > >> >> >> > has nothing to do with it?
> >> > > >> >> >> >
> >> > > >> >> >> > thanks
> >> > > >> >> >> >
> >> > > >> >> >> >
> >> > > >> >> >> > On Wed, Nov 11, 2009 at 12:02 PM, Zhenyu Zhong <
> >> > > >> >> zhongresearch@gmail.com
> >> > > >> >> >> >wrote:
> >> > > >> >> >> >
> >> > > >> >> >> >> Hi,
> >> > > >> >> >> >>
> >> > > >> >> >> >> I have been monitoring the system resources such as
> memory
> >> > Heap
> >> > > >> usage
> >> > > >> >> >> for
> >> > > >> >> >> >> Regionserver, load average, network bandwidth, etc.
> >> > > >> >> >> >>
> >> > > >> >> >> >> I experienced a disconnection last night. When I checked
> >> the
> >> > > >> memory
> >> > > >> >> Heap
> >> > > >> >> >> >> for Regionserver, it was 2336 which is way below the max
> >> Heap
> >> > > >> 4096. I
> >> > > >> >> >> don't
> >> > > >> >> >> >> think it is Regionserver problem. The requests that the
> >> > > >> regionserver
> >> > > >> >> got
> >> > > >> >> >> was
> >> > > >> >> >> >> about 126 which doesn't seem to be high. The load
> average
> >> is
> >> > > kind
> >> > > >> of
> >> > > >> >> >> high
> >> > > >> >> >> >> >100. (but I saw disconnections while the load average
> is
> >> > > around
> >> > > >> 30).
> >> > > >> >> >> The
> >> > > >> >> >> >> bandwidth is not a problem.
> >> > > >> >> >> >>
> >> > > >> >> >> >> However, I found error messages from the zookeeper.
> Please
> >> > see
> >> > > the
> >> > > >> >> >> >> following error messages. There are some
> >> > > >> >> >> >> BufferUnderflowException error, IOException error,etc.
> >> > > >> >> >> >> May I ask what might be the causes that generate these
> >> error
> >> > > >> >> messages?
> >> > > >> >> >> >>
> >> > > >> >> >> >> Many thanks
> >> > > >> >> >> >> zhenyu
> >> > > >> >> >> >>
> >> > > >> >> >> >>
> >> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
> >> > > org.apache.zookeeper.server.Request:
> >> > > >> >> >> Ignoring
> >> > > >> >> >> >> exception during toString
> >> > > >> >> >> >> java.nio.BufferUnderflowException
> >> > > >> >> >> >> at
> >> > java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
> >> > > >> >> >> >> at java.nio.ByteBuffer.get(ByteBuffer.java:675)
> >> > > >> >> >> >> at
> >> > > >> >> >>
> >> org.apache.zookeeper.server.Request.toString(Request.java:199)
> >> > > >> >> >> >> at java.lang.String.valueOf(String.java:2827)
> >> > > >> >> >> >> at
> >> > > java.lang.StringBuilder.append(StringBuilder.java:115)
> >> > > >> >> >> >> at
> >> > > >> >> >> >>
> >> > > >> >> >>
> >> > > >> >>
> >> > > >>
> >> > >
> >> >
> >>
> org.apache.zookeeper.server.quorum.CommitProcessor.processRequest(CommitProcessor.java:167)
> >> > > >> >> >> >> at
> >> > > >> >> >> >>
> >> > > >> >> >>
> >> > > >> >>
> >> > > >>
> >> > >
> >> >
> >>
> org.apache.zookeeper.server.quorum.FollowerRequestProcessor.run(FollowerRequestProcessor.java:68)
> >> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
> >> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
> >> > > >> >> >> >> Exception causing close of session 0x424dfd908c50009
> >> > > >> >> >> >> due to java.io.IOException: Read error
> >> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
> >> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
> >> > > >> >> >> >> IOException stack trace
> >> > > >> >> >> >> java.io.IOException: Read error
> >> > > >> >> >> >> at
> >> > > >> >> >> >>
> >> > > >> >>
> >> > >
> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:372)
> >> > > >> >> >> >> at
> >> > > >> >> >> >>
> >> > > >> >> >>
> >> > > >> >>
> >> > > >>
> >> > >
> >> >
> >>
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:221)
> >> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
> >> > > >> >> >> >> org.apache.zookeeper.server.quorum.CommitProcessor:
> >> > Processing
> >> > > >> >> request::
> >> > > >> >> >> >> sessionid:0x424dfd908c50009 type:setWatches
> >> > > >> cxid:0xfffffffffffffff8
> >> > > >> >> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
> >> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
> >> > > org.apache.zookeeper.server.Request:
> >> > > >> >> >> Ignoring
> >> > > >> >> >> >> exception during toString
> >> > > >> >> >> >> java.nio.BufferUnderflowException
> >> > > >> >> >> >> at
> >> > java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
> >> > > >> >> >> >> at java.nio.ByteBuffer.get(ByteBuffer.java:675)
> >> > > >> >> >> >> at
> >> > > >> >> >>
> >> org.apache.zookeeper.server.Request.toString(Request.java:199)
> >> > > >> >> >> >> at java.lang.String.valueOf(String.java:2827)
> >> > > >> >> >> >> at
> >> > > java.lang.StringBuilder.append(StringBuilder.java:115)
> >> > > >> >> >> >> at
> >> > > >> >> >> >>
> >> > > >> >> >>
> >> > > >> >>
> >> > > >>
> >> > >
> >> >
> >>
> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:74)
> >> > > >> >> >> >> at
> >> > > >> >> >> >>
> >> > > >> >> >>
> >> > > >> >>
> >> > > >>
> >> > >
> >> >
> >>
> org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)
> >> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
> >> > > >> >> >> >> org.apache.zookeeper.server.FinalRequestProcessor:
> >> Processing
> >> > > >> >> request::
> >> > > >> >> >> >> sessionid:0x424dfd908c50009 type:setWatches
> >> > > >> cxid:0xfffffffffffffff8
> >> > > >> >> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
> >> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
> >> > > org.apache.zookeeper.server.Request:
> >> > > >> >> >> Ignoring
> >> > > >> >> >> >> exception during toString
> >> > > >> >> >> >> java.nio.BufferUnderflowException
> >> > > >> >> >> >> at
> >> > java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
> >> > > >> >> >> >> at java.nio.ByteBuffer.get(ByteBuffer.java:675)
> >> > > >> >> >> >> at
> >> > > >> >> >>
> >> org.apache.zookeeper.server.Request.toString(Request.java:199)
> >> > > >> >> >> >> at
> >> > > >> >> >> >>
> >> > > >>
> >> org.apache.log4j.or.DefaultRenderer.doRender(DefaultRenderer.java:36)
> >> > > >> >> >> >> at
> >> > > >> >> >> >>
> >> > > org.apache.log4j.or.RendererMap.findAndRender(RendererMap.java:80)
> >> > > >> >> >> >> at
> >> > > >> >> >> >>
> >> > > >> >> >>
> >> > > >> >>
> >> > > >>
> >> > >
> >> >
> >>
> org.apache.log4j.spi.LoggingEvent.getRenderedMessage(LoggingEvent.java:362)
> >> > > >> >> >> >> at
> >> > > >> >> >> >>
> >> > > >> >> >>
> >> > > >> >>
> >> > > >>
> >> > >
> >> >
> >>
> org.apache.log4j.helpers.PatternParser$BasicPatternConverter.convert(PatternParser.java:403)
> >> > > >> >> >> >> at
> >> > > >> >> >> >>
> >> > > >> >> >>
> >> > > >> >>
> >> > > >>
> >> > >
> >> >
> >>
> org.apache.log4j.helpers.PatternConverter.format(PatternConverter.java:65)
> >> > > >> >> >> >> at
> >> > > >> >> org.apache.log4j.PatternLayout.format(PatternLayout.java:502)
> >> > > >> >> >> >> at
> >> > > >> >> >> >>
> >> > > org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:302)
> >> > > >> >> >> >> at
> >> > > >> >> >> >>
> >> > > >> >> >>
> >> > > >> >>
> >> > > >>
> >> > >
> >> >
> >>
> org.apache.log4j.DailyRollingFileAppender.subAppend(DailyRollingFileAppender.java:359)
> >> > > >> >> >> >> at
> >> > > >> >> >>
> >> org.apache.log4j.WriterAppender.append(WriterAppender.java:160)
> >> > > >> >> >> >> at
> >> > > >> >> >> >>
> >> > > >>
> >> org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
> >> > > >> >> >> >> at
> >> > > >> >> >> >>
> >> > > >> >> >>
> >> > > >> >>
> >> > > >>
> >> > >
> >> >
> >>
> org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
> >> > > >> >> >> >> at
> >> > > >> org.apache.log4j.Category.callAppenders(Category.java:206)
> >> > > >> >> >> >> at
> >> > > org.apache.log4j.Category.forcedLog(Category.java:391)
> >> > > >> >> >> >> at
> >> org.apache.log4j.Category.debug(Category.java:260)
> >> > > >> >> >> >> at
> >> > > >> >> >> >>
> >> > > >> >> >>
> >> > > >> >>
> >> > > >>
> >> > >
> >> >
> >>
> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:147)
> >> > > >> >> >> >> at
> >> > > >> >> >> >>
> >> > > >> >> >>
> >> > > >> >>
> >> > > >>
> >> > >
> >> >
> >>
> org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)
> >> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
> >> > > org.apache.zookeeper.server.Request:
> >> > > >> >> >> Ignoring
> >> > > >> >> >> >> exception during toString
> >> > > >> >> >> >> sessionid:0x424dfd908c50009 type:setWatches
> >> > > >> cxid:0xfffffffffffffff8
> >> > > >> >> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
> >> > > >> >> >> >> 2009-11-11 04:29:00,810 INFO
> >> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
> >> > > >> >> >> >> closing session:0x424dfd908c50009 NIOServerCnxn:
> >> > > >> >> >> >> java.nio.channels.SocketChannel[connected local=/
> >> > > >> >> 192.168.100.131:2181
> >> > > >> >> >> remote=/
> >> > > >> >> >> >> 192.168.100.131:35500]
> >> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
> >> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
> >> > > >> >> >> >> ignoring exception during input shutdown
> >> > > >> >> >> >> java.net.SocketException: Transport endpoint is not
> >> connected
> >> > > >> >> >> >> at sun.nio.ch.SocketChannelImpl.shutdown(Native
> >> > Method)
> >> > > >> >> >> >> at
> >> > > >> >> >> >>
> >> > > >> >>
> >> > >
> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
> >> > > >> >> >> >> at
> >> > > >> >> >>
> >> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> >> > > >> >> >> >> at
> >> > > >> >> >> >>
> >> > > >> >>
> >> > >
> >> org.apache.zookeeper.server.NIOServerCnxn.close(NIOServerCnxn.java:848)
> >> > > >> >> >> >> at
> >> > > >> >> >> >>
> >> > > >> >>
> >> > >
> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:498)
> >> > > >> >> >> >> at
> >> > > >> >> >> >>
> >> > > >> >> >>
> >> > > >> >>
> >> > > >>
> >> > >
> >> >
> >>
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:221)
> >> > > >> >> >> >>
> >> > > >> >> >> >
> >> > > >> >> >>
> >> > > >> >> >
> >> > > >> >>
> >> > > >> >
> >> > > >>
> >> > > >
> >> > >
> >> >
> >>
> >
> >
>
Re: regionserver disconnection
Posted by Zhenyu Zhong <zh...@gmail.com>.
Yes, superpyxis0002 is the RS got disconnected.
Many thanks
zhenyu
On Thu, Nov 12, 2009 at 2:42 PM, stack <st...@duboce.net> wrote:
> superpyxis0002 is the name of the RS whose log you posted earlier?
> Thanks,
> St.Ack
>
> On Thu, Nov 12, 2009 at 11:36 AM, Zhenyu Zhong <zhongresearch@gmail.com
> >wrote:
>
> > J-D,
> >
> > Please take a look at the master log during that time.
> >
> > http://pastebin.com/m502a3581
> >
> >
> > We can see that during that time, one regionserver disconnected.
> >
> >
> > Many thanks!
> > zhenyu
> >
> >
> > On Thu, Nov 12, 2009 at 2:14 PM, Jean-Daniel Cryans <jdcryans@apache.org
> > >wrote:
> >
> > > Could we see the master log from around the same time the region
> > > server got a session expired?
> > >
> > > Thx,
> > >
> > > J-D
> > >
> > > On Thu, Nov 12, 2009 at 10:54 AM, Zhenyu Zhong <
> zhongresearch@gmail.com>
> > > wrote:
> > > > Sorry, there was a typo in my email. But it is 10 minutes that I set
> in
> > > the
> > > > hbase-site.xml.
> > > >
> > > > Please see the link.
> > > > http://pastebin.com/m564c51bc
> > > >
> > > > Very much appreciated.
> > > > zhenyu
> > > >
> > > > On Thu, Nov 12, 2009 at 1:42 PM, Jean-Daniel Cryans <
> > jdcryans@apache.org
> > > >wrote:
> > > >
> > > >> Usually if it was 10 minutes we would see sleepers complaining they
> > > >> slept for way too long. In the logs I can see a 1 minute pause but
> not
> > > >> 10 minutes because of the ZK debug. Are you sure you don't have a
> > > >> typo? Like what you just wrote is not valid, you wrote
> > > >> zookeeper.session.timout instead of zookeeper.session.timeout. Are
> you
> > > >> sure it's distributed over the cluster? Could you pastebin
> > > >> hbase-site.xml?
> > > >>
> > > >> Thx,
> > > >>
> > > >> J-D
> > > >>
> > > >> On Thu, Nov 12, 2009 at 10:28 AM, Zhenyu Zhong <
> > zhongresearch@gmail.com
> > > >
> > > >> wrote:
> > > >> > J-D,
> > > >> >
> > > >> > Yes, that is weird. But it is what I have in the log. Probably the
> > log
> > > is
> > > >> > flushed by different threads, and one thread is blocked for a
> while?
> > > >> >
> > > >> > I set the zookeeper.session.timout=600000, which is 10 minutes.
> > > However,
> > > >> we
> > > >> > still see this problem.
> > > >> >
> > > >> > Does GC pause for such a long time period?
> > > >> > I will try different GC options to see the differences.
> > > >> >
> > > >> > thanks
> > > >> > zhenyu
> > > >> >
> > > >> >
> > > >> > On Thu, Nov 12, 2009 at 1:10 PM, Jean-Daniel Cryans <
> > > jdcryans@apache.org
> > > >> >wrote:
> > > >> >
> > > >> >> I see something weird
> > > >> >>
> > > >> >> 2009-11-12 15:04:36,370 WARN
> > > >> >> org.apache.hadoop.hbase.regionserver.HRegionServer: Attempt=1
> > > >> >> org.apache.hadoop.hbase.Leases$LeaseStillHeldException
> > > >> >> at
> > > sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> > > >> >> Method)
> > > >> >> at
> > > >> >>
> > > >>
> > >
> >
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> > > >> >> at
> > > >> >>
> > > >>
> > >
> >
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> > > >> >> ...
> > > >> >> 2009-11-12 15:03:52,682 DEBUG org.apache.zookeeper.ClientCnxn:
> Got
> > > >> >> ping response for sessionid:0x224e55436ad0004 after 0ms
> > > >> >> 2009-11-12 15:04:52,480 WARN org.apache.zookeeper.ClientCnxn:
> > > >> >> Exception closing session 0x24e55436a0007d to
> > > >> >> sun.nio.ch.SelectionKeyImpl@136f6ed1
> > > >> >>
> > > >> >> Goes back in time? I does look better tho. Did you say you set
> the
> > > >> >> zookeeper session timeout higher in a previous thread? I'd be
> > > >> >> interested to see if a higher (2-3 minutes) value would solve the
> > > >> >> problem until java 7.
> > > >> >>
> > > >> >> The only thing that happens after that LeaseStillHeldException is
> > > that
> > > >> >> for some reason (probably GC) the region server paused. This
> caused
> > > >> >> the session to timeout and should be why the RS isn't connected
> to
> > > the
> > > >> >> ZK quorum member.
> > > >> >>
> > > >> >> BTW we worked with the Zookeeper dev team to have more meaningful
> > > >> >> exceptions and it's already committed in their trunk.
> > > >> >>
> > > >> >> J-D
> > > >> >>
> > > >> >> On Thu, Nov 12, 2009 at 8:40 AM, Zhenyu Zhong <
> > > zhongresearch@gmail.com>
> > > >> >> wrote:
> > > >> >> > Follow up on the regionserver disconnection:
> > > >> >> >
> > > >> >> > I made a few changes to the cluster setup.
> > > >> >> > 1. split the regionserver and zookeeper quorum servers, such
> that
> > > >> HRegion
> > > >> >> > doesn't run on the same box as HQuorumPeer. I assume that there
> > > should
> > > >> be
> > > >> >> no
> > > >> >> > disk io contention between the regionserver and zookeeper.
> > > >> >> > Now we have 14 regionservers, and 5 zookeeper quorum
> servers.
> > > >> >> >
> > > >> >> > Though I experienced 2 regionserver disconnection this morning,
> > it
> > > >> looks
> > > >> >> > better from the regionserver log. (please see the following
> log)
> > > >> >> >
> > > >> >> >
> > > >> >> > http://pastebin.com/m496dbfae
> > > >> >> >
> > > >> >> >
> > > >> >> > I checked diskIO for one of the regionserver(192.168.100.116)
> > > during
> > > >> the
> > > >> >> > time it disconnected.
> > > >> >> >
> > > >> >> > Time: 03:04:58 AM
> > > >> >> > Device: tps Blk_read/s Blk_wrtn/s Blk_read
> > > Blk_wrtn
> > > >> >> > sda 105.31 5458.83 19503.64 9043873239
> > > >> 32312473676
> > > >> >> > sda1 2.90 6.64 99.25 10993934
> > > 164433464
> > > >> >> > sda2 1.72 23.76 30.25 39365817
> > > 50124008
> > > >> >> > sda3 0.30 0.38 3.58 624930
> > > 5923000
> > > >> >> > sda4 100.39 5428.06 19370.56 8992888270
> > > >> 32091993204
> > > >> >> >
> > > >> >> >
> > > >> >> > I also checked the zookeeper quorum server that the
> regionserver
> > > tried
> > > >> to
> > > >> >> > connect according to the log. However, I don't see
> > 192.168.100.116
> > > in
> > > >> the
> > > >> >> > client list of the zookeeper quorum server that the
> regionserver
> > > tried
> > > >> to
> > > >> >> > connect.
> > > >> >> > Would that be a problem?
> > > >> >> >
> > > >> >> > Thu Nov 12 15:04:52 UTC 2009
> > > >> >> > Zookeeper version: 3.2.1-808558, built on 08/27/2009 18:48 GMT
> > > >> >> > Clients:
> > > >> >> > /192.168.100.127:43045[1](queued=0,recved=26,sent=0)
> > > >> >> > /192.168.100.131:39091[1](queued=0,recved=964,sent=0)
> > > >> >> > /192.168.100.124:35961[1](queued=0,recved=3266,sent=0)
> > > >> >> > /192.168.100.123:47935[1](queued=0,recved=233,sent=0)
> > > >> >> > /192.168.100.125:46931[1](queued=0,recved=2,sent=0)
> > > >> >> > /192.168.100.118:54924[1](queued=0,recved=295,sent=0)
> > > >> >> > /192.168.100.118:41390[1](queued=0,recved=2290,sent=0)
> > > >> >> > /192.168.100.136:42243[1](queued=0,recved=0,sent=0)
> > > >> >> >
> > > >> >> > Latency min/avg/max: 0/17/6333
> > > >> >> > Received: 47111
> > > >> >> > Sent: 0
> > > >> >> > Outstanding: 0
> > > >> >> > Zxid: 0x77000083f4
> > > >> >> > Mode: leader
> > > >> >> > Node count: 23
> > > >> >> >
> > > >> >> > Best,
> > > >> >> > zhenyu
> > > >> >> >
> > > >> >> >
> > > >> >> > On Wed, Nov 11, 2009 at 3:58 PM, Zhenyu Zhong <
> > > >> zhongresearch@gmail.com
> > > >> >> >wrote:
> > > >> >> >
> > > >> >> >> Stack
> > > >> >> >>
> > > >> >> >> I am very appreciated for your comments.
> > > >> >> >> I will use the zookeeper monitoring script on my cluster and
> let
> > > it
> > > >> run
> > > >> >> >> overnight to see the result.
> > > >> >> >> I will follow up the thread when I get anything.
> > > >> >> >>
> > > >> >> >> thanks
> > > >> >> >> zhenyu
> > > >> >> >>
> > > >> >> >>
> > > >> >> >> On Wed, Nov 11, 2009 at 3:52 PM, stack <st...@duboce.net>
> > wrote:
> > > >> >> >>
> > > >> >> >>> I see these in your log too:
> > > >> >> >>>
> > > >> >> >>>
> > > >> >> >>> 1. 2009-11-11 04:27:19,018 DEBUG
> > > org.apache.zookeeper.ClientCnxn:
> > > >> Got
> > > >> >> >>> ping response for sessionid:0x424dfd908c50009 after 4544ms
> > > >> >> >>> 2. 2009-11-11 04:27:19,018 DEBUG
> > > org.apache.zookeeper.ClientCnxn:
> > > >> Got
> > > >> >> >>> ping response for sessionid:0x24dfd90c810002 after 4548ms
> > > >> >> >>> 3. 2009-11-11 04:27:43,960 DEBUG
> > > org.apache.zookeeper.ClientCnxn:
> > > >> Got
> > > >> >> >>> ping response for sessionid:0x424dfd908c50009 after 9030ms
> > > >> >> >>> 4. 2009-11-11 04:27:43,960 DEBUG
> > > org.apache.zookeeper.ClientCnxn:
> > > >> Got
> > > >> >> >>> ping response for sessionid:0x24dfd90c810002 after 9030ms
> > > >> >> >>> 5. 2009-11-11 04:27:48,830 DEBUG
> > > org.apache.zookeeper.ClientCnxn:
> > > >> Got
> > > >> >> >>> ping response for sessionid:0x424dfd908c50009 after 0ms
> > > >> >> >>> 6. 2009-11-11 04:27:48,830 DEBUG
> > > org.apache.zookeeper.ClientCnxn:
> > > >> Got
> > > >> >> >>> ping response for sessionid:0x24dfd90c810002 after 0ms
> > > >> >> >>>
> > > >> >> >>> Which would indicate lag talking to zk too.
> > > >> >> >>>
> > > >> >> >>> Appending HLog also seems to get steadily worse.
> > > >> >> >>>
> > > >> >> >>> I also see these:
> > > >> >> >>>
> > > >> >> >>>
> > > >> >> >>> 1. 2009-11-11 04:27:01,233 WARN
> > > org.apache.zookeeper.ClientCnxn:
> > > >> >> >>> Exception closing session 0x24dfd90c810002 to
> > > >> >> >>> sun.nio.ch.SelectionKeyImpl@5e16a7e3
> > > >> >> >>> 2. java.io.IOException: Read error rc = -1
> > > >> >> >>> java.nio.DirectByteBuffer[pos=0 lim=4 cap=4]
> > > >> >> >>> 3. at
> > > >> >> >>>
> > > >>
> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
> > > >> >> >>> 4. at
> > > >> >> >>>
> > > >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
> > > >> >> >>>
> > > >> >> >>>
> > > >> >> >>> Were all 5 quorum members participating? Check by doing
> "echo
> > > >> stat|nc
> > > >> >> >>> QUORUM_HOST port" for each member of quorum.... the below
> might
> > > >> help:
> > > >> >> >>>
> > > >> >> >>> members="1,2,3,4,5"
> > > >> >> >>> while true; do date; for i in $members; do echo $i; echo stat
> |
> > > nc
> > > >> $i
> > > >> >> >>> 2181;
> > > >> >> >>> done; sleep 10; done
> > > >> >> >>>
> > > >> >> >>> We need to improve our zk_dump. It just reports quorum
> > > members...
> > > >> not
> > > >> >> if
> > > >> >> >>> they are actually participating or not. The above will tell
> > you
> > > if
> > > >> >> dead
> > > >> >> >>> or
> > > >> >> >>> alive.
> > > >> >> >>>
> > > >> >> >>> Please report back after checking above and following the
> > > >> Jean-Daniel
> > > >> >> >>> prescription.
> > > >> >> >>>
> > > >> >> >>> Thanks,
> > > >> >> >>> St.Ack
> > > >> >> >>>
> > > >> >> >>>
> > > >> >> >>>
> > > >> >> >>> On Wed, Nov 11, 2009 at 12:29 PM, Zhenyu Zhong <
> > > >> >> zhongresearch@gmail.com
> > > >> >> >>> >wrote:
> > > >> >> >>>
> > > >> >> >>> > Thanks a lot for the help.
> > > >> >> >>> >
> > > >> >> >>> > I understand that the change of disk configurations should
> be
> > > done
> > > >> >> >>> > progressively such that HDFS can balance and recover from a
> > > disk
> > > >> >> going
> > > >> >> >>> > away.
> > > >> >> >>> >
> > > >> >> >>> > The data we are trying to import into HBase is about
> > 200million
> > > >> rows,
> > > >> >> >>> each
> > > >> >> >>> > row is about hundreds of bytes. Basically, we read in a
> text
> > > file
> > > >> >> from
> > > >> >> >>> > HDFS,
> > > >> >> >>> > then for each row in the text file, we fetch data from
> HBase
> > > >> table,
> > > >> >> we
> > > >> >> >>> > compare the column value and update the latest value back
> > into
> > > >> HBase.
> > > >> >> I
> > > >> >> >>> was
> > > >> >> >>> > suggested to use CheckAndPut. Anyway this is the common
> > > usecase
> > > >> on
> > > >> >> my
> > > >> >> >>> > side.
> > > >> >> >>> >
> > > >> >> >>> >
> > > >> >> >>> > Many thanks
> > > >> >> >>> > zhenyu
> > > >> >> >>> >
> > > >> >> >>> >
> > > >> >> >>> >
> > > >> >> >>> > On Wed, Nov 11, 2009 at 3:03 PM, Jean-Daniel Cryans <
> > > >> >> >>> jdcryans@apache.org
> > > >> >> >>> > >wrote:
> > > >> >> >>> >
> > > >> >> >>> > > 6 disks, 19 machines and you see those kind of issues?
> I'm
> > > quite
> > > >> >> >>> > > surprised and I want to learn more about what you trying
> to
> > > to.
> > > >> >> >>> > >
> > > >> >> >>> > > But first, if hbase is managing ZK for you then you can
> > > change
> > > >> the
> > > >> >> >>> > > value of hbase.zookeeper.property.dataDir so that it
> points
> > > to
> > > >> its
> > > >> >> own
> > > >> >> >>> > > mount. Make sure you also change dfs.data.dir in
> > > >> conf/hdfs-site.xml
> > > >> >> >>> > > (hadoop stuff) so that it stops pointing there (you
> should
> > > have
> > > >> 6
> > > >> >> >>> > > values there at the moment if you configured it
> correctly).
> > > >> Doing
> > > >> >> this
> > > >> >> >>> > > could result in data loss if you do it all at once so do
> > the
> > > >> change
> > > >> >> >>> > > progressively. This also means that you may have to keep
> a
> > > >> seperate
> > > >> >> >>> > > HDFS configuration for those particular nodes. Please ask
> > > more
> > > >> >> >>> > > questions if it's not very clear.
> > > >> >> >>> > >
> > > >> >> >>> > > WRT to dedicating nodes to Zookeeeper, it makes sense
> only
> > if
> > > >> you
> > > >> >> have
> > > >> >> >>> > > a lot of nodes but in your case, unless there's also CPU
> > > >> >> contention,
> > > >> >> >>> > > you're good with only a dedicated disk.
> > > >> >> >>> > >
> > > >> >> >>> > > Now for what you are trying to do, is this a massive
> import
> > > that
> > > >> >> goes
> > > >> >> >>> > > over many hours? How big is each row? I'd love to get a
> lot
> > > of
> > > >> >> details
> > > >> >> >>> > > to help you optimize your cluster for your particular
> > > problem.
> > > >> >> >>> > >
> > > >> >> >>> > > Thx,
> > > >> >> >>> > >
> > > >> >> >>> > > J-D
> > > >> >> >>> > >
> > > >> >> >>> > > On Wed, Nov 11, 2009 at 11:51 AM, Zhenyu Zhong <
> > > >> >> >>> zhongresearch@gmail.com>
> > > >> >> >>> > > wrote:
> > > >> >> >>> > > > J-D,
> > > >> >> >>> > > >
> > > >> >> >>> > > > Thanks a lot for the insightful comments.
> > > >> >> >>> > > > We are running 5 zookeeper quorums now. On these 5
> quorum
> > > >> >> servers,
> > > >> >> >>> we
> > > >> >> >>> > > also
> > > >> >> >>> > > > run datanode, tasktracker and regionserver.
> > > >> >> >>> > > > Now we have 6 disks per machine.
> > > >> >> >>> > > > If it is the disk IO saturation, is there anyway to
> > > configure
> > > >> >> >>> zookeeper
> > > >> >> >>> > > to
> > > >> >> >>> > > > write to a seperate disk which is not the same one used
> > by
> > > >> HDFS?
> > > >> >> >>> > > > Or shall we run zookeeper quorums on different nodes
> > other
> > > >> than
> > > >> >> >>> HDFS?
> > > >> >> >>> > > >
> > > >> >> >>> > > > Are HLogs generated by Regionservers? Would it
> alleviate
> > > the
> > > >> >> issue
> > > >> >> >>> if
> > > >> >> >>> > we
> > > >> >> >>> > > use
> > > >> >> >>> > > > the first 14 for regionserver, and the rest of 5 for
> > > >> >> zookeeper?(In
> > > >> >> >>> that
> > > >> >> >>> > > > sense, the node running zookeeper quorum won't run
> > > >> regionserver)
> > > >> >> >>> > > >
> > > >> >> >>> > > > I am very appreciated for your comments.
> > > >> >> >>> > > >
> > > >> >> >>> > > > Best,
> > > >> >> >>> > > > zhenyu
> > > >> >> >>> > > >
> > > >> >> >>> > > >
> > > >> >> >>> > > >
> > > >> >> >>> > > > On Wed, Nov 11, 2009 at 2:30 PM, Jean-Daniel Cryans <
> > > >> >> >>> > jdcryans@apache.org
> > > >> >> >>> > > >wrote:
> > > >> >> >>> > > >
> > > >> >> >>> > > >> The problem seems to be that the IO is saturated. When
> > you
> > > >> see
> > > >> >> >>> this:
> > > >> >> >>> > > >>
> > > >> >> >>> > > >> #
> > > >> >> >>> > > >> 2009-11-11 04:28:10,308 WARN
> > > >> >> >>> > > >> org.apache.hadoop.hbase.regionserver.HLog: IPC Server
> > > handler
> > > >> 9
> > > >> >> on
> > > >> >> >>> > > >> 60021 took 6819ms appending an edit to hlog;
> > editcount=370
> > > >> >> >>> > > >> #
> > > >> >> >>> > > >> 2009-11-11 04:28:10,308 WARN
> > > >> >> >>> > > >> org.apache.hadoop.hbase.regionserver.HLog: IPC Server
> > > handler
> > > >> 2
> > > >> >> on
> > > >> >> >>> > > >> 60021 took 4412ms appending an edit to hlog;
> > editcount=371
> > > >> >> >>> > > >>
> > > >> >> >>> > > >> It's a sign that HDFS is slow. I guess that the
> > zookeeper
> > > >> quorum
> > > >> >> >>> > > >> members are on those same nodes using the same disks?
> If
> > > so,
> > > >> >> >>> there's
> > > >> >> >>> > > >> the problem because Zookeeper needs to be able to
> commit
> > > >> >> >>> transactions
> > > >> >> >>> > > >> very fast and if your disks are already hogged then
> they
> > > have
> > > >> to
> > > >> >> >>> wait
> > > >> >> >>> > > >> and you can get disconnections. Do you still have 19
> > > quorum
> > > >> >> >>> members?
> > > >> >> >>> > > >> How many disks do you have per machine?
> > > >> >> >>> > > >>
> > > >> >> >>> > > >> Thx
> > > >> >> >>> > > >>
> > > >> >> >>> > > >> J-D
> > > >> >> >>> > > >>
> > > >> >> >>> > > >> On Wed, Nov 11, 2009 at 11:22 AM, Zhenyu Zhong <
> > > >> >> >>> > zhongresearch@gmail.com
> > > >> >> >>> > > >
> > > >> >> >>> > > >> wrote:
> > > >> >> >>> > > >> > Please take a look at the following regionserver
> log.
> > > >> >> >>> > > >> > The error started to come out from 4:24am UTC to
> > 4:29am
> > > >> UTC.
> > > >> >> >>> > > >> >
> > > >> >> >>> > > >> > http://pastebin.com/m3717d064
> > > >> >> >>> > > >> >
> > > >> >> >>> > > >> > Thanks in advance.
> > > >> >> >>> > > >> >
> > > >> >> >>> > > >> > Best,
> > > >> >> >>> > > >> > zhenyu
> > > >> >> >>> > > >> >
> > > >> >> >>> > > >> >
> > > >> >> >>> > > >> > On Wed, Nov 11, 2009 at 2:01 PM, Jean-Daniel Cryans
> <
> > > >> >> >>> > > jdcryans@apache.org
> > > >> >> >>> > > >> >wrote:
> > > >> >> >>> > > >> >
> > > >> >> >>> > > >> >> Could you paste a bigger log trace (interested to
> see
> > > at
> > > >> >> least
> > > >> >> >>> all
> > > >> >> >>> > > 300
> > > >> >> >>> > > >> >> lines before that) in a pastebin?
> > > >> >> >>> > > >> >>
> > > >> >> >>> > > >> >> Thx,
> > > >> >> >>> > > >> >>
> > > >> >> >>> > > >> >> J-D
> > > >> >> >>> > > >> >>
> > > >> >> >>> > > >> >> On Wed, Nov 11, 2009 at 10:56 AM, Zhenyu Zhong <
> > > >> >> >>> > > zhongresearch@gmail.com
> > > >> >> >>> > > >> >
> > > >> >> >>> > > >> >> wrote:
> > > >> >> >>> > > >> >> > J-D,
> > > >> >> >>> > > >> >> >
> > > >> >> >>> > > >> >> > Thanks for the reply.
> > > >> >> >>> > > >> >> >
> > > >> >> >>> > > >> >> > The problem I have is one or more of the
> > > regionservers
> > > >> >> >>> disconnect
> > > >> >> >>> > > from
> > > >> >> >>> > > >> >> the
> > > >> >> >>> > > >> >> > cluster.
> > > >> >> >>> > > >> >> >
> > > >> >> >>> > > >> >> > I checked the resource usage including memory
> Heap
> > > usage
> > > >> >> for
> > > >> >> >>> > > >> >> regionserver,
> > > >> >> >>> > > >> >> > and found nothing suspicious. The regionserver
> heap
> > > >> usage
> > > >> >> is
> > > >> >> >>> way
> > > >> >> >>> > > below
> > > >> >> >>> > > >> >> the
> > > >> >> >>> > > >> >> > maxHeap defined in jvm.
> > > >> >> >>> > > >> >> >
> > > >> >> >>> > > >> >> > So I checked zookeeper log, then I found the
> > > following
> > > >> >> >>> > > >> >> > BufferUnderflowException, eventually it leads to
> > > >> >> >>> SocketException:
> > > >> >> >>> > > >> >> Transport
> > > >> >> >>> > > >> >> > endpoint is not connected.
> > > >> >> >>> > > >> >> >
> > > >> >> >>> > > >> >> >
> > > >> >> >>> > > >> >> > I also checked the regionserver log. It looks
> like
> > > the
> > > >> >> >>> > regionserver
> > > >> >> >>> > > >> tried
> > > >> >> >>> > > >> >> to
> > > >> >> >>> > > >> >> > connect to other zookeeper quorum server. It
> > connects
> > > >> >> >>> > successfully,
> > > >> >> >>> > > >> but
> > > >> >> >>> > > >> >> then
> > > >> >> >>> > > >> >> > it complained about SocketException: transport
> > > endpoint
> > > >> is
> > > >> >> not
> > > >> >> >>> > > >> connected.
> > > >> >> >>> > > >> >> >
> > > >> >> >>> > > >> >> > So both zookeeper log and regionserver log told
> me
> > > that
> > > >> the
> > > >> >> >>> > > transport
> > > >> >> >>> > > >> >> > endpoint is not connected. I don't know why.
> > > >> >> >>> > > >> >> >
> > > >> >> >>> > > >> >> > I would be very appreciated if you could point me
> > to
> > > the
> > > >> >> right
> > > >> >> >>> > > >> direction.
> > > >> >> >>> > > >> >> >
> > > >> >> >>> > > >> >> > Best,
> > > >> >> >>> > > >> >> >
> > > >> >> >>> > > >> >> >
> > > >> >> >>> > > >> >> > Regionserver log:
> > > >> >> >>> > > >> >> > 2009-11-11 04:24:53,718 INFO
> > > >> >> org.apache.zookeeper.ClientCnxn:
> > > >> >> >>> > > >> Attempting
> > > >> >> >>> > > >> >> > connection to server superpyxis0018/
> > > >> 192.168.100.132:2181
> > > >> >> >>> > > >> >> > 2009-11-11 04:24:53,718 INFO
> > > >> >> org.apache.zookeeper.ClientCnxn:
> > > >> >> >>> > > Priming
> > > >> >> >>> > > >> >> > connection to
> > > java.nio.channels.SocketChannel[connected
> > > >> >> >>> local=/
> > > >> >> >>> > > >> >> > 192.168.100.131:43065 remote=superpyxis0018/
> > > >> >> >>> 192.168.100.132:2181
> > > >> >> >>> > ]
> > > >> >> >>> > > >> >> > 2009-11-11 04:24:53,718 INFO
> > > >> >> org.apache.zookeeper.ClientCnxn:
> > > >> >> >>> > > Server
> > > >> >> >>> > > >> >> > connection successful
> > > >> >> >>> > > >> >> > 2009-11-11 04:24:53,768 INFO
> > > >> >> >>> > > >> >> >
> org.apache.hadoop.hbase.regionserver.HRegionServer:
> > > Got
> > > >> >> >>> ZooKeeper
> > > >> >> >>> > > >> event,
> > > >> >> >>> > > >> >> > state: Disconnected, type: None, path: null
> > > >> >> >>> > > >> >> > 2009-11-11 04:24:54,118 INFO
> > > >> >> org.apache.zookeeper.ClientCnxn:
> > > >> >> >>> > > >> Attempting
> > > >> >> >>> > > >> >> > connection to server superpyxis0017/
> > > >> 192.168.100.131:2181
> > > >> >> >>> > > >> >> > 2009-11-11 04:24:54,118 INFO
> > > >> >> org.apache.zookeeper.ClientCnxn:
> > > >> >> >>> > > Priming
> > > >> >> >>> > > >> >> > connection to
> > > java.nio.channels.SocketChannel[connected
> > > >> >> >>> local=/
> > > >> >> >>> > > >> >> > 192.168.100.131:46903 remote=superpyxis0017/
> > > >> >> >>> 192.168.100.131:2181
> > > >> >> >>> > ]
> > > >> >> >>> > > >> >> > 2009-11-11 04:24:54,118 INFO
> > > >> >> org.apache.zookeeper.ClientCnxn:
> > > >> >> >>> > > Server
> > > >> >> >>> > > >> >> > connection successful
> > > >> >> >>> > > >> >> > 2009-11-11 04:24:54,118 WARN
> > > >> >> org.apache.zookeeper.ClientCnxn:
> > > >> >> >>> > > >> Exception
> > > >> >> >>> > > >> >> > closing session 0x24dfd90c810002 to
> > > >> >> >>> > > >> sun.nio.ch.SelectionKeyImpl@2dd78ccf
> > > >> >> >>> > > >> >> > java.io.IOException: Read error rc = -1
> > > >> >> >>> > > >> java.nio.DirectByteBuffer[pos=0
> > > >> >> >>> > > >> >> > lim=4 cap=4]
> > > >> >> >>> > > >> >> > at
> > > >> >> >>> > > >> >> >
> > > >> >> >>> > >
> > > >> >>
> > org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
> > > >> >> >>> > > >> >> > at
> > > >> >> >>> > > >> >> >
> > > >> >> >>> >
> > > >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
> > > >> >> >>> > > >> >> > 2009-11-11 04:24:54,118 WARN
> > > >> >> org.apache.zookeeper.ClientCnxn:
> > > >> >> >>> > > Ignoring
> > > >> >> >>> > > >> >> > exception during shutdown input
> > > >> >> >>> > > >> >> > java.net.SocketException: Transport endpoint is
> not
> > > >> >> connected
> > > >> >> >>> > > >> >> > at
> > > sun.nio.ch.SocketChannelImpl.shutdown(Native
> > > >> >> Method)
> > > >> >> >>> > > >> >> > at
> > > >> >> >>> > > >> >> >
> > > >> >> >>> > >
> > > >> >>
> > > sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
> > > >> >> >>> > > >> >> > at
> > > >> >> >>> > > >>
> > > >> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> > > >> >> >>> > > >> >> > at
> > > >> >> >>> > > >> >> >
> > > >> >> >>> > > >>
> > > >> >> >>> >
> > > >> >>
> > > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> > > >> >> >>> > > >> >> > at
> > > >> >> >>> > > >> >> >
> > > >> >> >>> >
> > > >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> > > >> >> >>> > > >> >> >
> > > >> >> >>> > > >> >> >
> > > >> >> >>> > > >> >> >
> > > >> >> >>> > > >> >> >
> > > >> >> >>> > > >> >> >
> > > >> >> >>> > > >> >> > On Wed, Nov 11, 2009 at 1:40 PM, Jean-Daniel
> Cryans
> > <
> > > >> >> >>> > > >> jdcryans@apache.org
> > > >> >> >>> > > >> >> >wrote:
> > > >> >> >>> > > >> >> >
> > > >> >> >>> > > >> >> >> I'm really not sure exactly what the situation
> is
> > > with
> > > >> the
> > > >> >> >>> > > >> >> >> java.nio.BufferUnderflowException, but what do
> you
> > > mean
> > > >> by
> > > >> >> >>> > > >> >> >> "regionserver disconnection"? Could you give
> more
> > > >> details?
> > > >> >> >>> > > >> >> >>
> > > >> >> >>> > > >> >> >> Thx,
> > > >> >> >>> > > >> >> >>
> > > >> >> >>> > > >> >> >> J-D
> > > >> >> >>> > > >> >> >>
> > > >> >> >>> > > >> >> >> On Wed, Nov 11, 2009 at 10:34 AM, Zhenyu Zhong <
> > > >> >> >>> > > >> zhongresearch@gmail.com
> > > >> >> >>> > > >> >> >
> > > >> >> >>> > > >> >> >> wrote:
> > > >> >> >>> > > >> >> >> > I searched around and found HBASE-1645 "HBase
> > > could
> > > >> be a
> > > >> >> >>> > little
> > > >> >> >>> > > >> more
> > > >> >> >>> > > >> >> >> dainty
> > > >> >> >>> > > >> >> >> > when zk is pulled out from under it"
> > > >> >> >>> > > >> >> >> >
> > > >> >> >>> > > >> >> >> > I am wondering whether my problem is related
> to
> > > this
> > > >> >> >>> > HBase-1645.
> > > >> >> >>> > > Or
> > > >> >> >>> > > >> >> >> probably
> > > >> >> >>> > > >> >> >> > has nothing to do with it?
> > > >> >> >>> > > >> >> >> >
> > > >> >> >>> > > >> >> >> > thanks
> > > >> >> >>> > > >> >> >> >
> > > >> >> >>> > > >> >> >> >
> > > >> >> >>> > > >> >> >> > On Wed, Nov 11, 2009 at 12:02 PM, Zhenyu Zhong
> <
> > > >> >> >>> > > >> >> zhongresearch@gmail.com
> > > >> >> >>> > > >> >> >> >wrote:
> > > >> >> >>> > > >> >> >> >
> > > >> >> >>> > > >> >> >> >> Hi,
> > > >> >> >>> > > >> >> >> >>
> > > >> >> >>> > > >> >> >> >> I have been monitoring the system resources
> > such
> > > as
> > > >> >> memory
> > > >> >> >>> > Heap
> > > >> >> >>> > > >> usage
> > > >> >> >>> > > >> >> >> for
> > > >> >> >>> > > >> >> >> >> Regionserver, load average, network
> bandwidth,
> > > etc.
> > > >> >> >>> > > >> >> >> >>
> > > >> >> >>> > > >> >> >> >> I experienced a disconnection last night.
> When
> > I
> > > >> >> checked
> > > >> >> >>> the
> > > >> >> >>> > > >> memory
> > > >> >> >>> > > >> >> Heap
> > > >> >> >>> > > >> >> >> >> for Regionserver, it was 2336 which is way
> > below
> > > the
> > > >> >> max
> > > >> >> >>> Heap
> > > >> >> >>> > > >> 4096. I
> > > >> >> >>> > > >> >> >> don't
> > > >> >> >>> > > >> >> >> >> think it is Regionserver problem. The
> requests
> > > that
> > > >> the
> > > >> >> >>> > > >> regionserver
> > > >> >> >>> > > >> >> got
> > > >> >> >>> > > >> >> >> was
> > > >> >> >>> > > >> >> >> >> about 126 which doesn't seem to be high. The
> > load
> > > >> >> average
> > > >> >> >>> is
> > > >> >> >>> > > kind
> > > >> >> >>> > > >> of
> > > >> >> >>> > > >> >> >> high
> > > >> >> >>> > > >> >> >> >> >100. (but I saw disconnections while the
> load
> > > >> average
> > > >> >> is
> > > >> >> >>> > > around
> > > >> >> >>> > > >> 30).
> > > >> >> >>> > > >> >> >> The
> > > >> >> >>> > > >> >> >> >> bandwidth is not a problem.
> > > >> >> >>> > > >> >> >> >>
> > > >> >> >>> > > >> >> >> >> However, I found error messages from the
> > > zookeeper.
> > > >> >> Please
> > > >> >> >>> > see
> > > >> >> >>> > > the
> > > >> >> >>> > > >> >> >> >> following error messages. There are some
> > > >> >> >>> > > >> >> >> >> BufferUnderflowException error, IOException
> > > >> error,etc.
> > > >> >> >>> > > >> >> >> >> May I ask what might be the causes that
> > generate
> > > >> these
> > > >> >> >>> error
> > > >> >> >>> > > >> >> messages?
> > > >> >> >>> > > >> >> >> >>
> > > >> >> >>> > > >> >> >> >> Many thanks
> > > >> >> >>> > > >> >> >> >> zhenyu
> > > >> >> >>> > > >> >> >> >>
> > > >> >> >>> > > >> >> >> >>
> > > >> >> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
> > > >> >> >>> > > org.apache.zookeeper.server.Request:
> > > >> >> >>> > > >> >> >> Ignoring
> > > >> >> >>> > > >> >> >> >> exception during toString
> > > >> >> >>> > > >> >> >> >> java.nio.BufferUnderflowException
> > > >> >> >>> > > >> >> >> >> at
> > > >> >> >>> > java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
> > > >> >> >>> > > >> >> >> >> at
> > > >> java.nio.ByteBuffer.get(ByteBuffer.java:675)
> > > >> >> >>> > > >> >> >> >> at
> > > >> >> >>> > > >> >> >>
> > > >> >> >>>
> org.apache.zookeeper.server.Request.toString(Request.java:199)
> > > >> >> >>> > > >> >> >> >> at
> > > >> java.lang.String.valueOf(String.java:2827)
> > > >> >> >>> > > >> >> >> >> at
> > > >> >> >>> > > java.lang.StringBuilder.append(StringBuilder.java:115)
> > > >> >> >>> > > >> >> >> >> at
> > > >> >> >>> > > >> >> >> >>
> > > >> >> >>> > > >> >> >>
> > > >> >> >>> > > >> >>
> > > >> >> >>> > > >>
> > > >> >> >>> > >
> > > >> >> >>> >
> > > >> >> >>>
> > > >> >>
> > > >>
> > >
> >
> org.apache.zookeeper.server.quorum.CommitProcessor.processRequest(CommitProcessor.java:167)
> > > >> >> >>> > > >> >> >> >> at
> > > >> >> >>> > > >> >> >> >>
> > > >> >> >>> > > >> >> >>
> > > >> >> >>> > > >> >>
> > > >> >> >>> > > >>
> > > >> >> >>> > >
> > > >> >> >>> >
> > > >> >> >>>
> > > >> >>
> > > >>
> > >
> >
> org.apache.zookeeper.server.quorum.FollowerRequestProcessor.run(FollowerRequestProcessor.java:68)
> > > >> >> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
> > > >> >> >>> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
> > > >> >> >>> > > >> >> >> >> Exception causing close of session
> > > 0x424dfd908c50009
> > > >> >> >>> > > >> >> >> >> due to java.io.IOException: Read error
> > > >> >> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
> > > >> >> >>> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
> > > >> >> >>> > > >> >> >> >> IOException stack trace
> > > >> >> >>> > > >> >> >> >> java.io.IOException: Read error
> > > >> >> >>> > > >> >> >> >> at
> > > >> >> >>> > > >> >> >> >>
> > > >> >> >>> > > >> >>
> > > >> >> >>> > >
> > > >> >>
> > > org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:372)
> > > >> >> >>> > > >> >> >> >> at
> > > >> >> >>> > > >> >> >> >>
> > > >> >> >>> > > >> >> >>
> > > >> >> >>> > > >> >>
> > > >> >> >>> > > >>
> > > >> >> >>> > >
> > > >> >> >>> >
> > > >> >> >>>
> > > >> >>
> > > >>
> > >
> >
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:221)
> > > >> >> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
> > > >> >> >>> > > >> >> >> >>
> > > org.apache.zookeeper.server.quorum.CommitProcessor:
> > > >> >> >>> > Processing
> > > >> >> >>> > > >> >> request::
> > > >> >> >>> > > >> >> >> >> sessionid:0x424dfd908c50009 type:setWatches
> > > >> >> >>> > > >> cxid:0xfffffffffffffff8
> > > >> >> >>> > > >> >> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
> > > >> >> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
> > > >> >> >>> > > org.apache.zookeeper.server.Request:
> > > >> >> >>> > > >> >> >> Ignoring
> > > >> >> >>> > > >> >> >> >> exception during toString
> > > >> >> >>> > > >> >> >> >> java.nio.BufferUnderflowException
> > > >> >> >>> > > >> >> >> >> at
> > > >> >> >>> > java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
> > > >> >> >>> > > >> >> >> >> at
> > > >> java.nio.ByteBuffer.get(ByteBuffer.java:675)
> > > >> >> >>> > > >> >> >> >> at
> > > >> >> >>> > > >> >> >>
> > > >> >> >>>
> org.apache.zookeeper.server.Request.toString(Request.java:199)
> > > >> >> >>> > > >> >> >> >> at
> > > >> java.lang.String.valueOf(String.java:2827)
> > > >> >> >>> > > >> >> >> >> at
> > > >> >> >>> > > java.lang.StringBuilder.append(StringBuilder.java:115)
> > > >> >> >>> > > >> >> >> >> at
> > > >> >> >>> > > >> >> >> >>
> > > >> >> >>> > > >> >> >>
> > > >> >> >>> > > >> >>
> > > >> >> >>> > > >>
> > > >> >> >>> > >
> > > >> >> >>> >
> > > >> >> >>>
> > > >> >>
> > > >>
> > >
> >
> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:74)
> > > >> >> >>> > > >> >> >> >> at
> > > >> >> >>> > > >> >> >> >>
> > > >> >> >>> > > >> >> >>
> > > >> >> >>> > > >> >>
> > > >> >> >>> > > >>
> > > >> >> >>> > >
> > > >> >> >>> >
> > > >> >> >>>
> > > >> >>
> > > >>
> > >
> >
> org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)
> > > >> >> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
> > > >> >> >>> > > >> >> >> >>
> > > org.apache.zookeeper.server.FinalRequestProcessor:
> > > >> >> >>> Processing
> > > >> >> >>> > > >> >> request::
> > > >> >> >>> > > >> >> >> >> sessionid:0x424dfd908c50009 type:setWatches
> > > >> >> >>> > > >> cxid:0xfffffffffffffff8
> > > >> >> >>> > > >> >> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
> > > >> >> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
> > > >> >> >>> > > org.apache.zookeeper.server.Request:
> > > >> >> >>> > > >> >> >> Ignoring
> > > >> >> >>> > > >> >> >> >> exception during toString
> > > >> >> >>> > > >> >> >> >> java.nio.BufferUnderflowException
> > > >> >> >>> > > >> >> >> >> at
> > > >> >> >>> > java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
> > > >> >> >>> > > >> >> >> >> at
> > > >> java.nio.ByteBuffer.get(ByteBuffer.java:675)
> > > >> >> >>> > > >> >> >> >> at
> > > >> >> >>> > > >> >> >>
> > > >> >> >>>
> org.apache.zookeeper.server.Request.toString(Request.java:199)
> > > >> >> >>> > > >> >> >> >> at
> > > >> >> >>> > > >> >> >> >>
> > > >> >> >>> > > >>
> > > >> >> >>>
> > > >>
> org.apache.log4j.or.DefaultRenderer.doRender(DefaultRenderer.java:36)
> > > >> >> >>> > > >> >> >> >> at
> > > >> >> >>> > > >> >> >> >>
> > > >> >> >>> > >
> > > >> org.apache.log4j.or.RendererMap.findAndRender(RendererMap.java:80)
> > > >> >> >>> > > >> >> >> >> at
> > > >> >> >>> > > >> >> >> >>
> > > >> >> >>> > > >> >> >>
> > > >> >> >>> > > >> >>
> > > >> >> >>> > > >>
> > > >> >> >>> > >
> > > >> >> >>> >
> > > >> >> >>>
> > > >> >>
> > > >>
> > >
> >
> org.apache.log4j.spi.LoggingEvent.getRenderedMessage(LoggingEvent.java:362)
> > > >> >> >>> > > >> >> >> >> at
> > > >> >> >>> > > >> >> >> >>
> > > >> >> >>> > > >> >> >>
> > > >> >> >>> > > >> >>
> > > >> >> >>> > > >>
> > > >> >> >>> > >
> > > >> >> >>> >
> > > >> >> >>>
> > > >> >>
> > > >>
> > >
> >
> org.apache.log4j.helpers.PatternParser$BasicPatternConverter.convert(PatternParser.java:403)
> > > >> >> >>> > > >> >> >> >> at
> > > >> >> >>> > > >> >> >> >>
> > > >> >> >>> > > >> >> >>
> > > >> >> >>> > > >> >>
> > > >> >> >>> > > >>
> > > >> >> >>> > >
> > > >> >> >>> >
> > > >> >> >>>
> > > >> >>
> > > >>
> > >
> >
> org.apache.log4j.helpers.PatternConverter.format(PatternConverter.java:65)
> > > >> >> >>> > > >> >> >> >> at
> > > >> >> >>> > > >> >>
> > > >> org.apache.log4j.PatternLayout.format(PatternLayout.java:502)
> > > >> >> >>> > > >> >> >> >> at
> > > >> >> >>> > > >> >> >> >>
> > > >> >> >>> > >
> > > >> org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:302)
> > > >> >> >>> > > >> >> >> >> at
> > > >> >> >>> > > >> >> >> >>
> > > >> >> >>> > > >> >> >>
> > > >> >> >>> > > >> >>
> > > >> >> >>> > > >>
> > > >> >> >>> > >
> > > >> >> >>> >
> > > >> >> >>>
> > > >> >>
> > > >>
> > >
> >
> org.apache.log4j.DailyRollingFileAppender.subAppend(DailyRollingFileAppender.java:359)
> > > >> >> >>> > > >> >> >> >> at
> > > >> >> >>> > > >> >> >>
> > > >> >> >>>
> org.apache.log4j.WriterAppender.append(WriterAppender.java:160)
> > > >> >> >>> > > >> >> >> >> at
> > > >> >> >>> > > >> >> >> >>
> > > >> >> >>> > > >>
> > > >> >> >>>
> > > >>
> org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
> > > >> >> >>> > > >> >> >> >> at
> > > >> >> >>> > > >> >> >> >>
> > > >> >> >>> > > >> >> >>
> > > >> >> >>> > > >> >>
> > > >> >> >>> > > >>
> > > >> >> >>> > >
> > > >> >> >>> >
> > > >> >> >>>
> > > >> >>
> > > >>
> > >
> >
> org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
> > > >> >> >>> > > >> >> >> >> at
> > > >> >> >>> > > >>
> > org.apache.log4j.Category.callAppenders(Category.java:206)
> > > >> >> >>> > > >> >> >> >> at
> > > >> >> >>> > > org.apache.log4j.Category.forcedLog(Category.java:391)
> > > >> >> >>> > > >> >> >> >> at
> > > >> >> >>> org.apache.log4j.Category.debug(Category.java:260)
> > > >> >> >>> > > >> >> >> >> at
> > > >> >> >>> > > >> >> >> >>
> > > >> >> >>> > > >> >> >>
> > > >> >> >>> > > >> >>
> > > >> >> >>> > > >>
> > > >> >> >>> > >
> > > >> >> >>> >
> > > >> >> >>>
> > > >> >>
> > > >>
> > >
> >
> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:147)
> > > >> >> >>> > > >> >> >> >> at
> > > >> >> >>> > > >> >> >> >>
> > > >> >> >>> > > >> >> >>
> > > >> >> >>> > > >> >>
> > > >> >> >>> > > >>
> > > >> >> >>> > >
> > > >> >> >>> >
> > > >> >> >>>
> > > >> >>
> > > >>
> > >
> >
> org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)
> > > >> >> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
> > > >> >> >>> > > org.apache.zookeeper.server.Request:
> > > >> >> >>> > > >> >> >> Ignoring
> > > >> >> >>> > > >> >> >> >> exception during toString
> > > >> >> >>> > > >> >> >> >> sessionid:0x424dfd908c50009 type:setWatches
> > > >> >> >>> > > >> cxid:0xfffffffffffffff8
> > > >> >> >>> > > >> >> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
> > > >> >> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 INFO
> > > >> >> >>> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
> > > >> >> >>> > > >> >> >> >> closing session:0x424dfd908c50009
> > NIOServerCnxn:
> > > >> >> >>> > > >> >> >> >> java.nio.channels.SocketChannel[connected
> > local=/
> > > >> >> >>> > > >> >> 192.168.100.131:2181
> > > >> >> >>> > > >> >> >> remote=/
> > > >> >> >>> > > >> >> >> >> 192.168.100.131:35500]
> > > >> >> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
> > > >> >> >>> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
> > > >> >> >>> > > >> >> >> >> ignoring exception during input shutdown
> > > >> >> >>> > > >> >> >> >> java.net.SocketException: Transport endpoint
> is
> > > not
> > > >> >> >>> connected
> > > >> >> >>> > > >> >> >> >> at
> > > >> sun.nio.ch.SocketChannelImpl.shutdown(Native
> > > >> >> >>> > Method)
> > > >> >> >>> > > >> >> >> >> at
> > > >> >> >>> > > >> >> >> >>
> > > >> >> >>> > > >> >>
> > > >> >> >>> > >
> > > >> >>
> > > sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
> > > >> >> >>> > > >> >> >> >> at
> > > >> >> >>> > > >> >> >>
> > > >> >> >>>
> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> > > >> >> >>> > > >> >> >> >> at
> > > >> >> >>> > > >> >> >> >>
> > > >> >> >>> > > >> >>
> > > >> >> >>> > >
> > > >> >> >>>
> > > >>
> > org.apache.zookeeper.server.NIOServerCnxn.close(NIOServerCnxn.java:848)
> > > >> >> >>> > > >> >> >> >> at
> > > >> >> >>> > > >> >> >> >>
> > > >> >> >>> > > >> >>
> > > >> >> >>> > >
> > > >> >>
> > > org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:498)
> > > >> >> >>> > > >> >> >> >> at
> > > >> >> >>> > > >> >> >> >>
> > > >> >> >>> > > >> >> >>
> > > >> >> >>> > > >> >>
> > > >> >> >>> > > >>
> > > >> >> >>> > >
> > > >> >> >>> >
> > > >> >> >>>
> > > >> >>
> > > >>
> > >
> >
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:221)
> > > >> >> >>> > > >> >> >> >>
> > > >> >> >>> > > >> >> >> >
> > > >> >> >>> > > >> >> >>
> > > >> >> >>> > > >> >> >
> > > >> >> >>> > > >> >>
> > > >> >> >>> > > >> >
> > > >> >> >>> > > >>
> > > >> >> >>> > > >
> > > >> >> >>> > >
> > > >> >> >>> >
> > > >> >> >>>
> > > >> >> >>
> > > >> >> >>
> > > >> >> >
> > > >> >>
> > > >> >
> > > >>
> > > >
> > >
> >
>
Re: regionserver disconnection
Posted by stack <st...@duboce.net>.
superpyxis0002 is the name of the RS whose log you posted earlier?
Thanks,
St.Ack
On Thu, Nov 12, 2009 at 11:36 AM, Zhenyu Zhong <zh...@gmail.com>wrote:
> J-D,
>
> Please take a look at the master log during that time.
>
> http://pastebin.com/m502a3581
>
>
> We can see that during that time, one regionserver disconnected.
>
>
> Many thanks!
> zhenyu
>
>
> On Thu, Nov 12, 2009 at 2:14 PM, Jean-Daniel Cryans <jdcryans@apache.org
> >wrote:
>
> > Could we see the master log from around the same time the region
> > server got a session expired?
> >
> > Thx,
> >
> > J-D
> >
> > On Thu, Nov 12, 2009 at 10:54 AM, Zhenyu Zhong <zh...@gmail.com>
> > wrote:
> > > Sorry, there was a typo in my email. But it is 10 minutes that I set in
> > the
> > > hbase-site.xml.
> > >
> > > Please see the link.
> > > http://pastebin.com/m564c51bc
> > >
> > > Very much appreciated.
> > > zhenyu
> > >
> > > On Thu, Nov 12, 2009 at 1:42 PM, Jean-Daniel Cryans <
> jdcryans@apache.org
> > >wrote:
> > >
> > >> Usually if it was 10 minutes we would see sleepers complaining they
> > >> slept for way too long. In the logs I can see a 1 minute pause but not
> > >> 10 minutes because of the ZK debug. Are you sure you don't have a
> > >> typo? Like what you just wrote is not valid, you wrote
> > >> zookeeper.session.timout instead of zookeeper.session.timeout. Are you
> > >> sure it's distributed over the cluster? Could you pastebin
> > >> hbase-site.xml?
> > >>
> > >> Thx,
> > >>
> > >> J-D
> > >>
> > >> On Thu, Nov 12, 2009 at 10:28 AM, Zhenyu Zhong <
> zhongresearch@gmail.com
> > >
> > >> wrote:
> > >> > J-D,
> > >> >
> > >> > Yes, that is weird. But it is what I have in the log. Probably the
> log
> > is
> > >> > flushed by different threads, and one thread is blocked for a while?
> > >> >
> > >> > I set the zookeeper.session.timout=600000, which is 10 minutes.
> > However,
> > >> we
> > >> > still see this problem.
> > >> >
> > >> > Does GC pause for such a long time period?
> > >> > I will try different GC options to see the differences.
> > >> >
> > >> > thanks
> > >> > zhenyu
> > >> >
> > >> >
> > >> > On Thu, Nov 12, 2009 at 1:10 PM, Jean-Daniel Cryans <
> > jdcryans@apache.org
> > >> >wrote:
> > >> >
> > >> >> I see something weird
> > >> >>
> > >> >> 2009-11-12 15:04:36,370 WARN
> > >> >> org.apache.hadoop.hbase.regionserver.HRegionServer: Attempt=1
> > >> >> org.apache.hadoop.hbase.Leases$LeaseStillHeldException
> > >> >> at
> > sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> > >> >> Method)
> > >> >> at
> > >> >>
> > >>
> >
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> > >> >> at
> > >> >>
> > >>
> >
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> > >> >> ...
> > >> >> 2009-11-12 15:03:52,682 DEBUG org.apache.zookeeper.ClientCnxn: Got
> > >> >> ping response for sessionid:0x224e55436ad0004 after 0ms
> > >> >> 2009-11-12 15:04:52,480 WARN org.apache.zookeeper.ClientCnxn:
> > >> >> Exception closing session 0x24e55436a0007d to
> > >> >> sun.nio.ch.SelectionKeyImpl@136f6ed1
> > >> >>
> > >> >> Goes back in time? I does look better tho. Did you say you set the
> > >> >> zookeeper session timeout higher in a previous thread? I'd be
> > >> >> interested to see if a higher (2-3 minutes) value would solve the
> > >> >> problem until java 7.
> > >> >>
> > >> >> The only thing that happens after that LeaseStillHeldException is
> > that
> > >> >> for some reason (probably GC) the region server paused. This caused
> > >> >> the session to timeout and should be why the RS isn't connected to
> > the
> > >> >> ZK quorum member.
> > >> >>
> > >> >> BTW we worked with the Zookeeper dev team to have more meaningful
> > >> >> exceptions and it's already committed in their trunk.
> > >> >>
> > >> >> J-D
> > >> >>
> > >> >> On Thu, Nov 12, 2009 at 8:40 AM, Zhenyu Zhong <
> > zhongresearch@gmail.com>
> > >> >> wrote:
> > >> >> > Follow up on the regionserver disconnection:
> > >> >> >
> > >> >> > I made a few changes to the cluster setup.
> > >> >> > 1. split the regionserver and zookeeper quorum servers, such that
> > >> HRegion
> > >> >> > doesn't run on the same box as HQuorumPeer. I assume that there
> > should
> > >> be
> > >> >> no
> > >> >> > disk io contention between the regionserver and zookeeper.
> > >> >> > Now we have 14 regionservers, and 5 zookeeper quorum servers.
> > >> >> >
> > >> >> > Though I experienced 2 regionserver disconnection this morning,
> it
> > >> looks
> > >> >> > better from the regionserver log. (please see the following log)
> > >> >> >
> > >> >> >
> > >> >> > http://pastebin.com/m496dbfae
> > >> >> >
> > >> >> >
> > >> >> > I checked diskIO for one of the regionserver(192.168.100.116)
> > during
> > >> the
> > >> >> > time it disconnected.
> > >> >> >
> > >> >> > Time: 03:04:58 AM
> > >> >> > Device: tps Blk_read/s Blk_wrtn/s Blk_read
> > Blk_wrtn
> > >> >> > sda 105.31 5458.83 19503.64 9043873239
> > >> 32312473676
> > >> >> > sda1 2.90 6.64 99.25 10993934
> > 164433464
> > >> >> > sda2 1.72 23.76 30.25 39365817
> > 50124008
> > >> >> > sda3 0.30 0.38 3.58 624930
> > 5923000
> > >> >> > sda4 100.39 5428.06 19370.56 8992888270
> > >> 32091993204
> > >> >> >
> > >> >> >
> > >> >> > I also checked the zookeeper quorum server that the regionserver
> > tried
> > >> to
> > >> >> > connect according to the log. However, I don't see
> 192.168.100.116
> > in
> > >> the
> > >> >> > client list of the zookeeper quorum server that the regionserver
> > tried
> > >> to
> > >> >> > connect.
> > >> >> > Would that be a problem?
> > >> >> >
> > >> >> > Thu Nov 12 15:04:52 UTC 2009
> > >> >> > Zookeeper version: 3.2.1-808558, built on 08/27/2009 18:48 GMT
> > >> >> > Clients:
> > >> >> > /192.168.100.127:43045[1](queued=0,recved=26,sent=0)
> > >> >> > /192.168.100.131:39091[1](queued=0,recved=964,sent=0)
> > >> >> > /192.168.100.124:35961[1](queued=0,recved=3266,sent=0)
> > >> >> > /192.168.100.123:47935[1](queued=0,recved=233,sent=0)
> > >> >> > /192.168.100.125:46931[1](queued=0,recved=2,sent=0)
> > >> >> > /192.168.100.118:54924[1](queued=0,recved=295,sent=0)
> > >> >> > /192.168.100.118:41390[1](queued=0,recved=2290,sent=0)
> > >> >> > /192.168.100.136:42243[1](queued=0,recved=0,sent=0)
> > >> >> >
> > >> >> > Latency min/avg/max: 0/17/6333
> > >> >> > Received: 47111
> > >> >> > Sent: 0
> > >> >> > Outstanding: 0
> > >> >> > Zxid: 0x77000083f4
> > >> >> > Mode: leader
> > >> >> > Node count: 23
> > >> >> >
> > >> >> > Best,
> > >> >> > zhenyu
> > >> >> >
> > >> >> >
> > >> >> > On Wed, Nov 11, 2009 at 3:58 PM, Zhenyu Zhong <
> > >> zhongresearch@gmail.com
> > >> >> >wrote:
> > >> >> >
> > >> >> >> Stack
> > >> >> >>
> > >> >> >> I am very appreciated for your comments.
> > >> >> >> I will use the zookeeper monitoring script on my cluster and let
> > it
> > >> run
> > >> >> >> overnight to see the result.
> > >> >> >> I will follow up the thread when I get anything.
> > >> >> >>
> > >> >> >> thanks
> > >> >> >> zhenyu
> > >> >> >>
> > >> >> >>
> > >> >> >> On Wed, Nov 11, 2009 at 3:52 PM, stack <st...@duboce.net>
> wrote:
> > >> >> >>
> > >> >> >>> I see these in your log too:
> > >> >> >>>
> > >> >> >>>
> > >> >> >>> 1. 2009-11-11 04:27:19,018 DEBUG
> > org.apache.zookeeper.ClientCnxn:
> > >> Got
> > >> >> >>> ping response for sessionid:0x424dfd908c50009 after 4544ms
> > >> >> >>> 2. 2009-11-11 04:27:19,018 DEBUG
> > org.apache.zookeeper.ClientCnxn:
> > >> Got
> > >> >> >>> ping response for sessionid:0x24dfd90c810002 after 4548ms
> > >> >> >>> 3. 2009-11-11 04:27:43,960 DEBUG
> > org.apache.zookeeper.ClientCnxn:
> > >> Got
> > >> >> >>> ping response for sessionid:0x424dfd908c50009 after 9030ms
> > >> >> >>> 4. 2009-11-11 04:27:43,960 DEBUG
> > org.apache.zookeeper.ClientCnxn:
> > >> Got
> > >> >> >>> ping response for sessionid:0x24dfd90c810002 after 9030ms
> > >> >> >>> 5. 2009-11-11 04:27:48,830 DEBUG
> > org.apache.zookeeper.ClientCnxn:
> > >> Got
> > >> >> >>> ping response for sessionid:0x424dfd908c50009 after 0ms
> > >> >> >>> 6. 2009-11-11 04:27:48,830 DEBUG
> > org.apache.zookeeper.ClientCnxn:
> > >> Got
> > >> >> >>> ping response for sessionid:0x24dfd90c810002 after 0ms
> > >> >> >>>
> > >> >> >>> Which would indicate lag talking to zk too.
> > >> >> >>>
> > >> >> >>> Appending HLog also seems to get steadily worse.
> > >> >> >>>
> > >> >> >>> I also see these:
> > >> >> >>>
> > >> >> >>>
> > >> >> >>> 1. 2009-11-11 04:27:01,233 WARN
> > org.apache.zookeeper.ClientCnxn:
> > >> >> >>> Exception closing session 0x24dfd90c810002 to
> > >> >> >>> sun.nio.ch.SelectionKeyImpl@5e16a7e3
> > >> >> >>> 2. java.io.IOException: Read error rc = -1
> > >> >> >>> java.nio.DirectByteBuffer[pos=0 lim=4 cap=4]
> > >> >> >>> 3. at
> > >> >> >>>
> > >> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
> > >> >> >>> 4. at
> > >> >> >>>
> > >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
> > >> >> >>>
> > >> >> >>>
> > >> >> >>> Were all 5 quorum members participating? Check by doing "echo
> > >> stat|nc
> > >> >> >>> QUORUM_HOST port" for each member of quorum.... the below might
> > >> help:
> > >> >> >>>
> > >> >> >>> members="1,2,3,4,5"
> > >> >> >>> while true; do date; for i in $members; do echo $i; echo stat |
> > nc
> > >> $i
> > >> >> >>> 2181;
> > >> >> >>> done; sleep 10; done
> > >> >> >>>
> > >> >> >>> We need to improve our zk_dump. It just reports quorum
> > members...
> > >> not
> > >> >> if
> > >> >> >>> they are actually participating or not. The above will tell
> you
> > if
> > >> >> dead
> > >> >> >>> or
> > >> >> >>> alive.
> > >> >> >>>
> > >> >> >>> Please report back after checking above and following the
> > >> Jean-Daniel
> > >> >> >>> prescription.
> > >> >> >>>
> > >> >> >>> Thanks,
> > >> >> >>> St.Ack
> > >> >> >>>
> > >> >> >>>
> > >> >> >>>
> > >> >> >>> On Wed, Nov 11, 2009 at 12:29 PM, Zhenyu Zhong <
> > >> >> zhongresearch@gmail.com
> > >> >> >>> >wrote:
> > >> >> >>>
> > >> >> >>> > Thanks a lot for the help.
> > >> >> >>> >
> > >> >> >>> > I understand that the change of disk configurations should be
> > done
> > >> >> >>> > progressively such that HDFS can balance and recover from a
> > disk
> > >> >> going
> > >> >> >>> > away.
> > >> >> >>> >
> > >> >> >>> > The data we are trying to import into HBase is about
> 200million
> > >> rows,
> > >> >> >>> each
> > >> >> >>> > row is about hundreds of bytes. Basically, we read in a text
> > file
> > >> >> from
> > >> >> >>> > HDFS,
> > >> >> >>> > then for each row in the text file, we fetch data from HBase
> > >> table,
> > >> >> we
> > >> >> >>> > compare the column value and update the latest value back
> into
> > >> HBase.
> > >> >> I
> > >> >> >>> was
> > >> >> >>> > suggested to use CheckAndPut. Anyway this is the common
> > usecase
> > >> on
> > >> >> my
> > >> >> >>> > side.
> > >> >> >>> >
> > >> >> >>> >
> > >> >> >>> > Many thanks
> > >> >> >>> > zhenyu
> > >> >> >>> >
> > >> >> >>> >
> > >> >> >>> >
> > >> >> >>> > On Wed, Nov 11, 2009 at 3:03 PM, Jean-Daniel Cryans <
> > >> >> >>> jdcryans@apache.org
> > >> >> >>> > >wrote:
> > >> >> >>> >
> > >> >> >>> > > 6 disks, 19 machines and you see those kind of issues? I'm
> > quite
> > >> >> >>> > > surprised and I want to learn more about what you trying to
> > to.
> > >> >> >>> > >
> > >> >> >>> > > But first, if hbase is managing ZK for you then you can
> > change
> > >> the
> > >> >> >>> > > value of hbase.zookeeper.property.dataDir so that it points
> > to
> > >> its
> > >> >> own
> > >> >> >>> > > mount. Make sure you also change dfs.data.dir in
> > >> conf/hdfs-site.xml
> > >> >> >>> > > (hadoop stuff) so that it stops pointing there (you should
> > have
> > >> 6
> > >> >> >>> > > values there at the moment if you configured it correctly).
> > >> Doing
> > >> >> this
> > >> >> >>> > > could result in data loss if you do it all at once so do
> the
> > >> change
> > >> >> >>> > > progressively. This also means that you may have to keep a
> > >> seperate
> > >> >> >>> > > HDFS configuration for those particular nodes. Please ask
> > more
> > >> >> >>> > > questions if it's not very clear.
> > >> >> >>> > >
> > >> >> >>> > > WRT to dedicating nodes to Zookeeeper, it makes sense only
> if
> > >> you
> > >> >> have
> > >> >> >>> > > a lot of nodes but in your case, unless there's also CPU
> > >> >> contention,
> > >> >> >>> > > you're good with only a dedicated disk.
> > >> >> >>> > >
> > >> >> >>> > > Now for what you are trying to do, is this a massive import
> > that
> > >> >> goes
> > >> >> >>> > > over many hours? How big is each row? I'd love to get a lot
> > of
> > >> >> details
> > >> >> >>> > > to help you optimize your cluster for your particular
> > problem.
> > >> >> >>> > >
> > >> >> >>> > > Thx,
> > >> >> >>> > >
> > >> >> >>> > > J-D
> > >> >> >>> > >
> > >> >> >>> > > On Wed, Nov 11, 2009 at 11:51 AM, Zhenyu Zhong <
> > >> >> >>> zhongresearch@gmail.com>
> > >> >> >>> > > wrote:
> > >> >> >>> > > > J-D,
> > >> >> >>> > > >
> > >> >> >>> > > > Thanks a lot for the insightful comments.
> > >> >> >>> > > > We are running 5 zookeeper quorums now. On these 5 quorum
> > >> >> servers,
> > >> >> >>> we
> > >> >> >>> > > also
> > >> >> >>> > > > run datanode, tasktracker and regionserver.
> > >> >> >>> > > > Now we have 6 disks per machine.
> > >> >> >>> > > > If it is the disk IO saturation, is there anyway to
> > configure
> > >> >> >>> zookeeper
> > >> >> >>> > > to
> > >> >> >>> > > > write to a seperate disk which is not the same one used
> by
> > >> HDFS?
> > >> >> >>> > > > Or shall we run zookeeper quorums on different nodes
> other
> > >> than
> > >> >> >>> HDFS?
> > >> >> >>> > > >
> > >> >> >>> > > > Are HLogs generated by Regionservers? Would it alleviate
> > the
> > >> >> issue
> > >> >> >>> if
> > >> >> >>> > we
> > >> >> >>> > > use
> > >> >> >>> > > > the first 14 for regionserver, and the rest of 5 for
> > >> >> zookeeper?(In
> > >> >> >>> that
> > >> >> >>> > > > sense, the node running zookeeper quorum won't run
> > >> regionserver)
> > >> >> >>> > > >
> > >> >> >>> > > > I am very appreciated for your comments.
> > >> >> >>> > > >
> > >> >> >>> > > > Best,
> > >> >> >>> > > > zhenyu
> > >> >> >>> > > >
> > >> >> >>> > > >
> > >> >> >>> > > >
> > >> >> >>> > > > On Wed, Nov 11, 2009 at 2:30 PM, Jean-Daniel Cryans <
> > >> >> >>> > jdcryans@apache.org
> > >> >> >>> > > >wrote:
> > >> >> >>> > > >
> > >> >> >>> > > >> The problem seems to be that the IO is saturated. When
> you
> > >> see
> > >> >> >>> this:
> > >> >> >>> > > >>
> > >> >> >>> > > >> #
> > >> >> >>> > > >> 2009-11-11 04:28:10,308 WARN
> > >> >> >>> > > >> org.apache.hadoop.hbase.regionserver.HLog: IPC Server
> > handler
> > >> 9
> > >> >> on
> > >> >> >>> > > >> 60021 took 6819ms appending an edit to hlog;
> editcount=370
> > >> >> >>> > > >> #
> > >> >> >>> > > >> 2009-11-11 04:28:10,308 WARN
> > >> >> >>> > > >> org.apache.hadoop.hbase.regionserver.HLog: IPC Server
> > handler
> > >> 2
> > >> >> on
> > >> >> >>> > > >> 60021 took 4412ms appending an edit to hlog;
> editcount=371
> > >> >> >>> > > >>
> > >> >> >>> > > >> It's a sign that HDFS is slow. I guess that the
> zookeeper
> > >> quorum
> > >> >> >>> > > >> members are on those same nodes using the same disks? If
> > so,
> > >> >> >>> there's
> > >> >> >>> > > >> the problem because Zookeeper needs to be able to commit
> > >> >> >>> transactions
> > >> >> >>> > > >> very fast and if your disks are already hogged then they
> > have
> > >> to
> > >> >> >>> wait
> > >> >> >>> > > >> and you can get disconnections. Do you still have 19
> > quorum
> > >> >> >>> members?
> > >> >> >>> > > >> How many disks do you have per machine?
> > >> >> >>> > > >>
> > >> >> >>> > > >> Thx
> > >> >> >>> > > >>
> > >> >> >>> > > >> J-D
> > >> >> >>> > > >>
> > >> >> >>> > > >> On Wed, Nov 11, 2009 at 11:22 AM, Zhenyu Zhong <
> > >> >> >>> > zhongresearch@gmail.com
> > >> >> >>> > > >
> > >> >> >>> > > >> wrote:
> > >> >> >>> > > >> > Please take a look at the following regionserver log.
> > >> >> >>> > > >> > The error started to come out from 4:24am UTC to
> 4:29am
> > >> UTC.
> > >> >> >>> > > >> >
> > >> >> >>> > > >> > http://pastebin.com/m3717d064
> > >> >> >>> > > >> >
> > >> >> >>> > > >> > Thanks in advance.
> > >> >> >>> > > >> >
> > >> >> >>> > > >> > Best,
> > >> >> >>> > > >> > zhenyu
> > >> >> >>> > > >> >
> > >> >> >>> > > >> >
> > >> >> >>> > > >> > On Wed, Nov 11, 2009 at 2:01 PM, Jean-Daniel Cryans <
> > >> >> >>> > > jdcryans@apache.org
> > >> >> >>> > > >> >wrote:
> > >> >> >>> > > >> >
> > >> >> >>> > > >> >> Could you paste a bigger log trace (interested to see
> > at
> > >> >> least
> > >> >> >>> all
> > >> >> >>> > > 300
> > >> >> >>> > > >> >> lines before that) in a pastebin?
> > >> >> >>> > > >> >>
> > >> >> >>> > > >> >> Thx,
> > >> >> >>> > > >> >>
> > >> >> >>> > > >> >> J-D
> > >> >> >>> > > >> >>
> > >> >> >>> > > >> >> On Wed, Nov 11, 2009 at 10:56 AM, Zhenyu Zhong <
> > >> >> >>> > > zhongresearch@gmail.com
> > >> >> >>> > > >> >
> > >> >> >>> > > >> >> wrote:
> > >> >> >>> > > >> >> > J-D,
> > >> >> >>> > > >> >> >
> > >> >> >>> > > >> >> > Thanks for the reply.
> > >> >> >>> > > >> >> >
> > >> >> >>> > > >> >> > The problem I have is one or more of the
> > regionservers
> > >> >> >>> disconnect
> > >> >> >>> > > from
> > >> >> >>> > > >> >> the
> > >> >> >>> > > >> >> > cluster.
> > >> >> >>> > > >> >> >
> > >> >> >>> > > >> >> > I checked the resource usage including memory Heap
> > usage
> > >> >> for
> > >> >> >>> > > >> >> regionserver,
> > >> >> >>> > > >> >> > and found nothing suspicious. The regionserver heap
> > >> usage
> > >> >> is
> > >> >> >>> way
> > >> >> >>> > > below
> > >> >> >>> > > >> >> the
> > >> >> >>> > > >> >> > maxHeap defined in jvm.
> > >> >> >>> > > >> >> >
> > >> >> >>> > > >> >> > So I checked zookeeper log, then I found the
> > following
> > >> >> >>> > > >> >> > BufferUnderflowException, eventually it leads to
> > >> >> >>> SocketException:
> > >> >> >>> > > >> >> Transport
> > >> >> >>> > > >> >> > endpoint is not connected.
> > >> >> >>> > > >> >> >
> > >> >> >>> > > >> >> >
> > >> >> >>> > > >> >> > I also checked the regionserver log. It looks like
> > the
> > >> >> >>> > regionserver
> > >> >> >>> > > >> tried
> > >> >> >>> > > >> >> to
> > >> >> >>> > > >> >> > connect to other zookeeper quorum server. It
> connects
> > >> >> >>> > successfully,
> > >> >> >>> > > >> but
> > >> >> >>> > > >> >> then
> > >> >> >>> > > >> >> > it complained about SocketException: transport
> > endpoint
> > >> is
> > >> >> not
> > >> >> >>> > > >> connected.
> > >> >> >>> > > >> >> >
> > >> >> >>> > > >> >> > So both zookeeper log and regionserver log told me
> > that
> > >> the
> > >> >> >>> > > transport
> > >> >> >>> > > >> >> > endpoint is not connected. I don't know why.
> > >> >> >>> > > >> >> >
> > >> >> >>> > > >> >> > I would be very appreciated if you could point me
> to
> > the
> > >> >> right
> > >> >> >>> > > >> direction.
> > >> >> >>> > > >> >> >
> > >> >> >>> > > >> >> > Best,
> > >> >> >>> > > >> >> >
> > >> >> >>> > > >> >> >
> > >> >> >>> > > >> >> > Regionserver log:
> > >> >> >>> > > >> >> > 2009-11-11 04:24:53,718 INFO
> > >> >> org.apache.zookeeper.ClientCnxn:
> > >> >> >>> > > >> Attempting
> > >> >> >>> > > >> >> > connection to server superpyxis0018/
> > >> 192.168.100.132:2181
> > >> >> >>> > > >> >> > 2009-11-11 04:24:53,718 INFO
> > >> >> org.apache.zookeeper.ClientCnxn:
> > >> >> >>> > > Priming
> > >> >> >>> > > >> >> > connection to
> > java.nio.channels.SocketChannel[connected
> > >> >> >>> local=/
> > >> >> >>> > > >> >> > 192.168.100.131:43065 remote=superpyxis0018/
> > >> >> >>> 192.168.100.132:2181
> > >> >> >>> > ]
> > >> >> >>> > > >> >> > 2009-11-11 04:24:53,718 INFO
> > >> >> org.apache.zookeeper.ClientCnxn:
> > >> >> >>> > > Server
> > >> >> >>> > > >> >> > connection successful
> > >> >> >>> > > >> >> > 2009-11-11 04:24:53,768 INFO
> > >> >> >>> > > >> >> > org.apache.hadoop.hbase.regionserver.HRegionServer:
> > Got
> > >> >> >>> ZooKeeper
> > >> >> >>> > > >> event,
> > >> >> >>> > > >> >> > state: Disconnected, type: None, path: null
> > >> >> >>> > > >> >> > 2009-11-11 04:24:54,118 INFO
> > >> >> org.apache.zookeeper.ClientCnxn:
> > >> >> >>> > > >> Attempting
> > >> >> >>> > > >> >> > connection to server superpyxis0017/
> > >> 192.168.100.131:2181
> > >> >> >>> > > >> >> > 2009-11-11 04:24:54,118 INFO
> > >> >> org.apache.zookeeper.ClientCnxn:
> > >> >> >>> > > Priming
> > >> >> >>> > > >> >> > connection to
> > java.nio.channels.SocketChannel[connected
> > >> >> >>> local=/
> > >> >> >>> > > >> >> > 192.168.100.131:46903 remote=superpyxis0017/
> > >> >> >>> 192.168.100.131:2181
> > >> >> >>> > ]
> > >> >> >>> > > >> >> > 2009-11-11 04:24:54,118 INFO
> > >> >> org.apache.zookeeper.ClientCnxn:
> > >> >> >>> > > Server
> > >> >> >>> > > >> >> > connection successful
> > >> >> >>> > > >> >> > 2009-11-11 04:24:54,118 WARN
> > >> >> org.apache.zookeeper.ClientCnxn:
> > >> >> >>> > > >> Exception
> > >> >> >>> > > >> >> > closing session 0x24dfd90c810002 to
> > >> >> >>> > > >> sun.nio.ch.SelectionKeyImpl@2dd78ccf
> > >> >> >>> > > >> >> > java.io.IOException: Read error rc = -1
> > >> >> >>> > > >> java.nio.DirectByteBuffer[pos=0
> > >> >> >>> > > >> >> > lim=4 cap=4]
> > >> >> >>> > > >> >> > at
> > >> >> >>> > > >> >> >
> > >> >> >>> > >
> > >> >>
> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
> > >> >> >>> > > >> >> > at
> > >> >> >>> > > >> >> >
> > >> >> >>> >
> > >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
> > >> >> >>> > > >> >> > 2009-11-11 04:24:54,118 WARN
> > >> >> org.apache.zookeeper.ClientCnxn:
> > >> >> >>> > > Ignoring
> > >> >> >>> > > >> >> > exception during shutdown input
> > >> >> >>> > > >> >> > java.net.SocketException: Transport endpoint is not
> > >> >> connected
> > >> >> >>> > > >> >> > at
> > sun.nio.ch.SocketChannelImpl.shutdown(Native
> > >> >> Method)
> > >> >> >>> > > >> >> > at
> > >> >> >>> > > >> >> >
> > >> >> >>> > >
> > >> >>
> > sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
> > >> >> >>> > > >> >> > at
> > >> >> >>> > > >>
> > >> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> > >> >> >>> > > >> >> > at
> > >> >> >>> > > >> >> >
> > >> >> >>> > > >>
> > >> >> >>> >
> > >> >>
> > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> > >> >> >>> > > >> >> > at
> > >> >> >>> > > >> >> >
> > >> >> >>> >
> > >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> > >> >> >>> > > >> >> >
> > >> >> >>> > > >> >> >
> > >> >> >>> > > >> >> >
> > >> >> >>> > > >> >> >
> > >> >> >>> > > >> >> >
> > >> >> >>> > > >> >> > On Wed, Nov 11, 2009 at 1:40 PM, Jean-Daniel Cryans
> <
> > >> >> >>> > > >> jdcryans@apache.org
> > >> >> >>> > > >> >> >wrote:
> > >> >> >>> > > >> >> >
> > >> >> >>> > > >> >> >> I'm really not sure exactly what the situation is
> > with
> > >> the
> > >> >> >>> > > >> >> >> java.nio.BufferUnderflowException, but what do you
> > mean
> > >> by
> > >> >> >>> > > >> >> >> "regionserver disconnection"? Could you give more
> > >> details?
> > >> >> >>> > > >> >> >>
> > >> >> >>> > > >> >> >> Thx,
> > >> >> >>> > > >> >> >>
> > >> >> >>> > > >> >> >> J-D
> > >> >> >>> > > >> >> >>
> > >> >> >>> > > >> >> >> On Wed, Nov 11, 2009 at 10:34 AM, Zhenyu Zhong <
> > >> >> >>> > > >> zhongresearch@gmail.com
> > >> >> >>> > > >> >> >
> > >> >> >>> > > >> >> >> wrote:
> > >> >> >>> > > >> >> >> > I searched around and found HBASE-1645 "HBase
> > could
> > >> be a
> > >> >> >>> > little
> > >> >> >>> > > >> more
> > >> >> >>> > > >> >> >> dainty
> > >> >> >>> > > >> >> >> > when zk is pulled out from under it"
> > >> >> >>> > > >> >> >> >
> > >> >> >>> > > >> >> >> > I am wondering whether my problem is related to
> > this
> > >> >> >>> > HBase-1645.
> > >> >> >>> > > Or
> > >> >> >>> > > >> >> >> probably
> > >> >> >>> > > >> >> >> > has nothing to do with it?
> > >> >> >>> > > >> >> >> >
> > >> >> >>> > > >> >> >> > thanks
> > >> >> >>> > > >> >> >> >
> > >> >> >>> > > >> >> >> >
> > >> >> >>> > > >> >> >> > On Wed, Nov 11, 2009 at 12:02 PM, Zhenyu Zhong <
> > >> >> >>> > > >> >> zhongresearch@gmail.com
> > >> >> >>> > > >> >> >> >wrote:
> > >> >> >>> > > >> >> >> >
> > >> >> >>> > > >> >> >> >> Hi,
> > >> >> >>> > > >> >> >> >>
> > >> >> >>> > > >> >> >> >> I have been monitoring the system resources
> such
> > as
> > >> >> memory
> > >> >> >>> > Heap
> > >> >> >>> > > >> usage
> > >> >> >>> > > >> >> >> for
> > >> >> >>> > > >> >> >> >> Regionserver, load average, network bandwidth,
> > etc.
> > >> >> >>> > > >> >> >> >>
> > >> >> >>> > > >> >> >> >> I experienced a disconnection last night. When
> I
> > >> >> checked
> > >> >> >>> the
> > >> >> >>> > > >> memory
> > >> >> >>> > > >> >> Heap
> > >> >> >>> > > >> >> >> >> for Regionserver, it was 2336 which is way
> below
> > the
> > >> >> max
> > >> >> >>> Heap
> > >> >> >>> > > >> 4096. I
> > >> >> >>> > > >> >> >> don't
> > >> >> >>> > > >> >> >> >> think it is Regionserver problem. The requests
> > that
> > >> the
> > >> >> >>> > > >> regionserver
> > >> >> >>> > > >> >> got
> > >> >> >>> > > >> >> >> was
> > >> >> >>> > > >> >> >> >> about 126 which doesn't seem to be high. The
> load
> > >> >> average
> > >> >> >>> is
> > >> >> >>> > > kind
> > >> >> >>> > > >> of
> > >> >> >>> > > >> >> >> high
> > >> >> >>> > > >> >> >> >> >100. (but I saw disconnections while the load
> > >> average
> > >> >> is
> > >> >> >>> > > around
> > >> >> >>> > > >> 30).
> > >> >> >>> > > >> >> >> The
> > >> >> >>> > > >> >> >> >> bandwidth is not a problem.
> > >> >> >>> > > >> >> >> >>
> > >> >> >>> > > >> >> >> >> However, I found error messages from the
> > zookeeper.
> > >> >> Please
> > >> >> >>> > see
> > >> >> >>> > > the
> > >> >> >>> > > >> >> >> >> following error messages. There are some
> > >> >> >>> > > >> >> >> >> BufferUnderflowException error, IOException
> > >> error,etc.
> > >> >> >>> > > >> >> >> >> May I ask what might be the causes that
> generate
> > >> these
> > >> >> >>> error
> > >> >> >>> > > >> >> messages?
> > >> >> >>> > > >> >> >> >>
> > >> >> >>> > > >> >> >> >> Many thanks
> > >> >> >>> > > >> >> >> >> zhenyu
> > >> >> >>> > > >> >> >> >>
> > >> >> >>> > > >> >> >> >>
> > >> >> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
> > >> >> >>> > > org.apache.zookeeper.server.Request:
> > >> >> >>> > > >> >> >> Ignoring
> > >> >> >>> > > >> >> >> >> exception during toString
> > >> >> >>> > > >> >> >> >> java.nio.BufferUnderflowException
> > >> >> >>> > > >> >> >> >> at
> > >> >> >>> > java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
> > >> >> >>> > > >> >> >> >> at
> > >> java.nio.ByteBuffer.get(ByteBuffer.java:675)
> > >> >> >>> > > >> >> >> >> at
> > >> >> >>> > > >> >> >>
> > >> >> >>> org.apache.zookeeper.server.Request.toString(Request.java:199)
> > >> >> >>> > > >> >> >> >> at
> > >> java.lang.String.valueOf(String.java:2827)
> > >> >> >>> > > >> >> >> >> at
> > >> >> >>> > > java.lang.StringBuilder.append(StringBuilder.java:115)
> > >> >> >>> > > >> >> >> >> at
> > >> >> >>> > > >> >> >> >>
> > >> >> >>> > > >> >> >>
> > >> >> >>> > > >> >>
> > >> >> >>> > > >>
> > >> >> >>> > >
> > >> >> >>> >
> > >> >> >>>
> > >> >>
> > >>
> >
> org.apache.zookeeper.server.quorum.CommitProcessor.processRequest(CommitProcessor.java:167)
> > >> >> >>> > > >> >> >> >> at
> > >> >> >>> > > >> >> >> >>
> > >> >> >>> > > >> >> >>
> > >> >> >>> > > >> >>
> > >> >> >>> > > >>
> > >> >> >>> > >
> > >> >> >>> >
> > >> >> >>>
> > >> >>
> > >>
> >
> org.apache.zookeeper.server.quorum.FollowerRequestProcessor.run(FollowerRequestProcessor.java:68)
> > >> >> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
> > >> >> >>> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
> > >> >> >>> > > >> >> >> >> Exception causing close of session
> > 0x424dfd908c50009
> > >> >> >>> > > >> >> >> >> due to java.io.IOException: Read error
> > >> >> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
> > >> >> >>> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
> > >> >> >>> > > >> >> >> >> IOException stack trace
> > >> >> >>> > > >> >> >> >> java.io.IOException: Read error
> > >> >> >>> > > >> >> >> >> at
> > >> >> >>> > > >> >> >> >>
> > >> >> >>> > > >> >>
> > >> >> >>> > >
> > >> >>
> > org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:372)
> > >> >> >>> > > >> >> >> >> at
> > >> >> >>> > > >> >> >> >>
> > >> >> >>> > > >> >> >>
> > >> >> >>> > > >> >>
> > >> >> >>> > > >>
> > >> >> >>> > >
> > >> >> >>> >
> > >> >> >>>
> > >> >>
> > >>
> >
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:221)
> > >> >> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
> > >> >> >>> > > >> >> >> >>
> > org.apache.zookeeper.server.quorum.CommitProcessor:
> > >> >> >>> > Processing
> > >> >> >>> > > >> >> request::
> > >> >> >>> > > >> >> >> >> sessionid:0x424dfd908c50009 type:setWatches
> > >> >> >>> > > >> cxid:0xfffffffffffffff8
> > >> >> >>> > > >> >> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
> > >> >> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
> > >> >> >>> > > org.apache.zookeeper.server.Request:
> > >> >> >>> > > >> >> >> Ignoring
> > >> >> >>> > > >> >> >> >> exception during toString
> > >> >> >>> > > >> >> >> >> java.nio.BufferUnderflowException
> > >> >> >>> > > >> >> >> >> at
> > >> >> >>> > java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
> > >> >> >>> > > >> >> >> >> at
> > >> java.nio.ByteBuffer.get(ByteBuffer.java:675)
> > >> >> >>> > > >> >> >> >> at
> > >> >> >>> > > >> >> >>
> > >> >> >>> org.apache.zookeeper.server.Request.toString(Request.java:199)
> > >> >> >>> > > >> >> >> >> at
> > >> java.lang.String.valueOf(String.java:2827)
> > >> >> >>> > > >> >> >> >> at
> > >> >> >>> > > java.lang.StringBuilder.append(StringBuilder.java:115)
> > >> >> >>> > > >> >> >> >> at
> > >> >> >>> > > >> >> >> >>
> > >> >> >>> > > >> >> >>
> > >> >> >>> > > >> >>
> > >> >> >>> > > >>
> > >> >> >>> > >
> > >> >> >>> >
> > >> >> >>>
> > >> >>
> > >>
> >
> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:74)
> > >> >> >>> > > >> >> >> >> at
> > >> >> >>> > > >> >> >> >>
> > >> >> >>> > > >> >> >>
> > >> >> >>> > > >> >>
> > >> >> >>> > > >>
> > >> >> >>> > >
> > >> >> >>> >
> > >> >> >>>
> > >> >>
> > >>
> >
> org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)
> > >> >> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
> > >> >> >>> > > >> >> >> >>
> > org.apache.zookeeper.server.FinalRequestProcessor:
> > >> >> >>> Processing
> > >> >> >>> > > >> >> request::
> > >> >> >>> > > >> >> >> >> sessionid:0x424dfd908c50009 type:setWatches
> > >> >> >>> > > >> cxid:0xfffffffffffffff8
> > >> >> >>> > > >> >> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
> > >> >> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
> > >> >> >>> > > org.apache.zookeeper.server.Request:
> > >> >> >>> > > >> >> >> Ignoring
> > >> >> >>> > > >> >> >> >> exception during toString
> > >> >> >>> > > >> >> >> >> java.nio.BufferUnderflowException
> > >> >> >>> > > >> >> >> >> at
> > >> >> >>> > java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
> > >> >> >>> > > >> >> >> >> at
> > >> java.nio.ByteBuffer.get(ByteBuffer.java:675)
> > >> >> >>> > > >> >> >> >> at
> > >> >> >>> > > >> >> >>
> > >> >> >>> org.apache.zookeeper.server.Request.toString(Request.java:199)
> > >> >> >>> > > >> >> >> >> at
> > >> >> >>> > > >> >> >> >>
> > >> >> >>> > > >>
> > >> >> >>>
> > >> org.apache.log4j.or.DefaultRenderer.doRender(DefaultRenderer.java:36)
> > >> >> >>> > > >> >> >> >> at
> > >> >> >>> > > >> >> >> >>
> > >> >> >>> > >
> > >> org.apache.log4j.or.RendererMap.findAndRender(RendererMap.java:80)
> > >> >> >>> > > >> >> >> >> at
> > >> >> >>> > > >> >> >> >>
> > >> >> >>> > > >> >> >>
> > >> >> >>> > > >> >>
> > >> >> >>> > > >>
> > >> >> >>> > >
> > >> >> >>> >
> > >> >> >>>
> > >> >>
> > >>
> >
> org.apache.log4j.spi.LoggingEvent.getRenderedMessage(LoggingEvent.java:362)
> > >> >> >>> > > >> >> >> >> at
> > >> >> >>> > > >> >> >> >>
> > >> >> >>> > > >> >> >>
> > >> >> >>> > > >> >>
> > >> >> >>> > > >>
> > >> >> >>> > >
> > >> >> >>> >
> > >> >> >>>
> > >> >>
> > >>
> >
> org.apache.log4j.helpers.PatternParser$BasicPatternConverter.convert(PatternParser.java:403)
> > >> >> >>> > > >> >> >> >> at
> > >> >> >>> > > >> >> >> >>
> > >> >> >>> > > >> >> >>
> > >> >> >>> > > >> >>
> > >> >> >>> > > >>
> > >> >> >>> > >
> > >> >> >>> >
> > >> >> >>>
> > >> >>
> > >>
> >
> org.apache.log4j.helpers.PatternConverter.format(PatternConverter.java:65)
> > >> >> >>> > > >> >> >> >> at
> > >> >> >>> > > >> >>
> > >> org.apache.log4j.PatternLayout.format(PatternLayout.java:502)
> > >> >> >>> > > >> >> >> >> at
> > >> >> >>> > > >> >> >> >>
> > >> >> >>> > >
> > >> org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:302)
> > >> >> >>> > > >> >> >> >> at
> > >> >> >>> > > >> >> >> >>
> > >> >> >>> > > >> >> >>
> > >> >> >>> > > >> >>
> > >> >> >>> > > >>
> > >> >> >>> > >
> > >> >> >>> >
> > >> >> >>>
> > >> >>
> > >>
> >
> org.apache.log4j.DailyRollingFileAppender.subAppend(DailyRollingFileAppender.java:359)
> > >> >> >>> > > >> >> >> >> at
> > >> >> >>> > > >> >> >>
> > >> >> >>> org.apache.log4j.WriterAppender.append(WriterAppender.java:160)
> > >> >> >>> > > >> >> >> >> at
> > >> >> >>> > > >> >> >> >>
> > >> >> >>> > > >>
> > >> >> >>>
> > >> org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
> > >> >> >>> > > >> >> >> >> at
> > >> >> >>> > > >> >> >> >>
> > >> >> >>> > > >> >> >>
> > >> >> >>> > > >> >>
> > >> >> >>> > > >>
> > >> >> >>> > >
> > >> >> >>> >
> > >> >> >>>
> > >> >>
> > >>
> >
> org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
> > >> >> >>> > > >> >> >> >> at
> > >> >> >>> > > >>
> org.apache.log4j.Category.callAppenders(Category.java:206)
> > >> >> >>> > > >> >> >> >> at
> > >> >> >>> > > org.apache.log4j.Category.forcedLog(Category.java:391)
> > >> >> >>> > > >> >> >> >> at
> > >> >> >>> org.apache.log4j.Category.debug(Category.java:260)
> > >> >> >>> > > >> >> >> >> at
> > >> >> >>> > > >> >> >> >>
> > >> >> >>> > > >> >> >>
> > >> >> >>> > > >> >>
> > >> >> >>> > > >>
> > >> >> >>> > >
> > >> >> >>> >
> > >> >> >>>
> > >> >>
> > >>
> >
> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:147)
> > >> >> >>> > > >> >> >> >> at
> > >> >> >>> > > >> >> >> >>
> > >> >> >>> > > >> >> >>
> > >> >> >>> > > >> >>
> > >> >> >>> > > >>
> > >> >> >>> > >
> > >> >> >>> >
> > >> >> >>>
> > >> >>
> > >>
> >
> org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)
> > >> >> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
> > >> >> >>> > > org.apache.zookeeper.server.Request:
> > >> >> >>> > > >> >> >> Ignoring
> > >> >> >>> > > >> >> >> >> exception during toString
> > >> >> >>> > > >> >> >> >> sessionid:0x424dfd908c50009 type:setWatches
> > >> >> >>> > > >> cxid:0xfffffffffffffff8
> > >> >> >>> > > >> >> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
> > >> >> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 INFO
> > >> >> >>> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
> > >> >> >>> > > >> >> >> >> closing session:0x424dfd908c50009
> NIOServerCnxn:
> > >> >> >>> > > >> >> >> >> java.nio.channels.SocketChannel[connected
> local=/
> > >> >> >>> > > >> >> 192.168.100.131:2181
> > >> >> >>> > > >> >> >> remote=/
> > >> >> >>> > > >> >> >> >> 192.168.100.131:35500]
> > >> >> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
> > >> >> >>> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
> > >> >> >>> > > >> >> >> >> ignoring exception during input shutdown
> > >> >> >>> > > >> >> >> >> java.net.SocketException: Transport endpoint is
> > not
> > >> >> >>> connected
> > >> >> >>> > > >> >> >> >> at
> > >> sun.nio.ch.SocketChannelImpl.shutdown(Native
> > >> >> >>> > Method)
> > >> >> >>> > > >> >> >> >> at
> > >> >> >>> > > >> >> >> >>
> > >> >> >>> > > >> >>
> > >> >> >>> > >
> > >> >>
> > sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
> > >> >> >>> > > >> >> >> >> at
> > >> >> >>> > > >> >> >>
> > >> >> >>> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> > >> >> >>> > > >> >> >> >> at
> > >> >> >>> > > >> >> >> >>
> > >> >> >>> > > >> >>
> > >> >> >>> > >
> > >> >> >>>
> > >>
> org.apache.zookeeper.server.NIOServerCnxn.close(NIOServerCnxn.java:848)
> > >> >> >>> > > >> >> >> >> at
> > >> >> >>> > > >> >> >> >>
> > >> >> >>> > > >> >>
> > >> >> >>> > >
> > >> >>
> > org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:498)
> > >> >> >>> > > >> >> >> >> at
> > >> >> >>> > > >> >> >> >>
> > >> >> >>> > > >> >> >>
> > >> >> >>> > > >> >>
> > >> >> >>> > > >>
> > >> >> >>> > >
> > >> >> >>> >
> > >> >> >>>
> > >> >>
> > >>
> >
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:221)
> > >> >> >>> > > >> >> >> >>
> > >> >> >>> > > >> >> >> >
> > >> >> >>> > > >> >> >>
> > >> >> >>> > > >> >> >
> > >> >> >>> > > >> >>
> > >> >> >>> > > >> >
> > >> >> >>> > > >>
> > >> >> >>> > > >
> > >> >> >>> > >
> > >> >> >>> >
> > >> >> >>>
> > >> >> >>
> > >> >> >>
> > >> >> >
> > >> >>
> > >> >
> > >>
> > >
> >
>
Re: regionserver disconnection
Posted by Zhenyu Zhong <zh...@gmail.com>.
J-D,
Please take a look at the master log during that time.
http://pastebin.com/m502a3581
We can see that during that time, one regionserver disconnected.
Many thanks!
zhenyu
On Thu, Nov 12, 2009 at 2:14 PM, Jean-Daniel Cryans <jd...@apache.org>wrote:
> Could we see the master log from around the same time the region
> server got a session expired?
>
> Thx,
>
> J-D
>
> On Thu, Nov 12, 2009 at 10:54 AM, Zhenyu Zhong <zh...@gmail.com>
> wrote:
> > Sorry, there was a typo in my email. But it is 10 minutes that I set in
> the
> > hbase-site.xml.
> >
> > Please see the link.
> > http://pastebin.com/m564c51bc
> >
> > Very much appreciated.
> > zhenyu
> >
> > On Thu, Nov 12, 2009 at 1:42 PM, Jean-Daniel Cryans <jdcryans@apache.org
> >wrote:
> >
> >> Usually if it was 10 minutes we would see sleepers complaining they
> >> slept for way too long. In the logs I can see a 1 minute pause but not
> >> 10 minutes because of the ZK debug. Are you sure you don't have a
> >> typo? Like what you just wrote is not valid, you wrote
> >> zookeeper.session.timout instead of zookeeper.session.timeout. Are you
> >> sure it's distributed over the cluster? Could you pastebin
> >> hbase-site.xml?
> >>
> >> Thx,
> >>
> >> J-D
> >>
> >> On Thu, Nov 12, 2009 at 10:28 AM, Zhenyu Zhong <zhongresearch@gmail.com
> >
> >> wrote:
> >> > J-D,
> >> >
> >> > Yes, that is weird. But it is what I have in the log. Probably the log
> is
> >> > flushed by different threads, and one thread is blocked for a while?
> >> >
> >> > I set the zookeeper.session.timout=600000, which is 10 minutes.
> However,
> >> we
> >> > still see this problem.
> >> >
> >> > Does GC pause for such a long time period?
> >> > I will try different GC options to see the differences.
> >> >
> >> > thanks
> >> > zhenyu
> >> >
> >> >
> >> > On Thu, Nov 12, 2009 at 1:10 PM, Jean-Daniel Cryans <
> jdcryans@apache.org
> >> >wrote:
> >> >
> >> >> I see something weird
> >> >>
> >> >> 2009-11-12 15:04:36,370 WARN
> >> >> org.apache.hadoop.hbase.regionserver.HRegionServer: Attempt=1
> >> >> org.apache.hadoop.hbase.Leases$LeaseStillHeldException
> >> >> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> >> >> Method)
> >> >> at
> >> >>
> >>
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> >> >> at
> >> >>
> >>
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> >> >> ...
> >> >> 2009-11-12 15:03:52,682 DEBUG org.apache.zookeeper.ClientCnxn: Got
> >> >> ping response for sessionid:0x224e55436ad0004 after 0ms
> >> >> 2009-11-12 15:04:52,480 WARN org.apache.zookeeper.ClientCnxn:
> >> >> Exception closing session 0x24e55436a0007d to
> >> >> sun.nio.ch.SelectionKeyImpl@136f6ed1
> >> >>
> >> >> Goes back in time? I does look better tho. Did you say you set the
> >> >> zookeeper session timeout higher in a previous thread? I'd be
> >> >> interested to see if a higher (2-3 minutes) value would solve the
> >> >> problem until java 7.
> >> >>
> >> >> The only thing that happens after that LeaseStillHeldException is
> that
> >> >> for some reason (probably GC) the region server paused. This caused
> >> >> the session to timeout and should be why the RS isn't connected to
> the
> >> >> ZK quorum member.
> >> >>
> >> >> BTW we worked with the Zookeeper dev team to have more meaningful
> >> >> exceptions and it's already committed in their trunk.
> >> >>
> >> >> J-D
> >> >>
> >> >> On Thu, Nov 12, 2009 at 8:40 AM, Zhenyu Zhong <
> zhongresearch@gmail.com>
> >> >> wrote:
> >> >> > Follow up on the regionserver disconnection:
> >> >> >
> >> >> > I made a few changes to the cluster setup.
> >> >> > 1. split the regionserver and zookeeper quorum servers, such that
> >> HRegion
> >> >> > doesn't run on the same box as HQuorumPeer. I assume that there
> should
> >> be
> >> >> no
> >> >> > disk io contention between the regionserver and zookeeper.
> >> >> > Now we have 14 regionservers, and 5 zookeeper quorum servers.
> >> >> >
> >> >> > Though I experienced 2 regionserver disconnection this morning, it
> >> looks
> >> >> > better from the regionserver log. (please see the following log)
> >> >> >
> >> >> >
> >> >> > http://pastebin.com/m496dbfae
> >> >> >
> >> >> >
> >> >> > I checked diskIO for one of the regionserver(192.168.100.116)
> during
> >> the
> >> >> > time it disconnected.
> >> >> >
> >> >> > Time: 03:04:58 AM
> >> >> > Device: tps Blk_read/s Blk_wrtn/s Blk_read
> Blk_wrtn
> >> >> > sda 105.31 5458.83 19503.64 9043873239
> >> 32312473676
> >> >> > sda1 2.90 6.64 99.25 10993934
> 164433464
> >> >> > sda2 1.72 23.76 30.25 39365817
> 50124008
> >> >> > sda3 0.30 0.38 3.58 624930
> 5923000
> >> >> > sda4 100.39 5428.06 19370.56 8992888270
> >> 32091993204
> >> >> >
> >> >> >
> >> >> > I also checked the zookeeper quorum server that the regionserver
> tried
> >> to
> >> >> > connect according to the log. However, I don't see 192.168.100.116
> in
> >> the
> >> >> > client list of the zookeeper quorum server that the regionserver
> tried
> >> to
> >> >> > connect.
> >> >> > Would that be a problem?
> >> >> >
> >> >> > Thu Nov 12 15:04:52 UTC 2009
> >> >> > Zookeeper version: 3.2.1-808558, built on 08/27/2009 18:48 GMT
> >> >> > Clients:
> >> >> > /192.168.100.127:43045[1](queued=0,recved=26,sent=0)
> >> >> > /192.168.100.131:39091[1](queued=0,recved=964,sent=0)
> >> >> > /192.168.100.124:35961[1](queued=0,recved=3266,sent=0)
> >> >> > /192.168.100.123:47935[1](queued=0,recved=233,sent=0)
> >> >> > /192.168.100.125:46931[1](queued=0,recved=2,sent=0)
> >> >> > /192.168.100.118:54924[1](queued=0,recved=295,sent=0)
> >> >> > /192.168.100.118:41390[1](queued=0,recved=2290,sent=0)
> >> >> > /192.168.100.136:42243[1](queued=0,recved=0,sent=0)
> >> >> >
> >> >> > Latency min/avg/max: 0/17/6333
> >> >> > Received: 47111
> >> >> > Sent: 0
> >> >> > Outstanding: 0
> >> >> > Zxid: 0x77000083f4
> >> >> > Mode: leader
> >> >> > Node count: 23
> >> >> >
> >> >> > Best,
> >> >> > zhenyu
> >> >> >
> >> >> >
> >> >> > On Wed, Nov 11, 2009 at 3:58 PM, Zhenyu Zhong <
> >> zhongresearch@gmail.com
> >> >> >wrote:
> >> >> >
> >> >> >> Stack
> >> >> >>
> >> >> >> I am very appreciated for your comments.
> >> >> >> I will use the zookeeper monitoring script on my cluster and let
> it
> >> run
> >> >> >> overnight to see the result.
> >> >> >> I will follow up the thread when I get anything.
> >> >> >>
> >> >> >> thanks
> >> >> >> zhenyu
> >> >> >>
> >> >> >>
> >> >> >> On Wed, Nov 11, 2009 at 3:52 PM, stack <st...@duboce.net> wrote:
> >> >> >>
> >> >> >>> I see these in your log too:
> >> >> >>>
> >> >> >>>
> >> >> >>> 1. 2009-11-11 04:27:19,018 DEBUG
> org.apache.zookeeper.ClientCnxn:
> >> Got
> >> >> >>> ping response for sessionid:0x424dfd908c50009 after 4544ms
> >> >> >>> 2. 2009-11-11 04:27:19,018 DEBUG
> org.apache.zookeeper.ClientCnxn:
> >> Got
> >> >> >>> ping response for sessionid:0x24dfd90c810002 after 4548ms
> >> >> >>> 3. 2009-11-11 04:27:43,960 DEBUG
> org.apache.zookeeper.ClientCnxn:
> >> Got
> >> >> >>> ping response for sessionid:0x424dfd908c50009 after 9030ms
> >> >> >>> 4. 2009-11-11 04:27:43,960 DEBUG
> org.apache.zookeeper.ClientCnxn:
> >> Got
> >> >> >>> ping response for sessionid:0x24dfd90c810002 after 9030ms
> >> >> >>> 5. 2009-11-11 04:27:48,830 DEBUG
> org.apache.zookeeper.ClientCnxn:
> >> Got
> >> >> >>> ping response for sessionid:0x424dfd908c50009 after 0ms
> >> >> >>> 6. 2009-11-11 04:27:48,830 DEBUG
> org.apache.zookeeper.ClientCnxn:
> >> Got
> >> >> >>> ping response for sessionid:0x24dfd90c810002 after 0ms
> >> >> >>>
> >> >> >>> Which would indicate lag talking to zk too.
> >> >> >>>
> >> >> >>> Appending HLog also seems to get steadily worse.
> >> >> >>>
> >> >> >>> I also see these:
> >> >> >>>
> >> >> >>>
> >> >> >>> 1. 2009-11-11 04:27:01,233 WARN
> org.apache.zookeeper.ClientCnxn:
> >> >> >>> Exception closing session 0x24dfd90c810002 to
> >> >> >>> sun.nio.ch.SelectionKeyImpl@5e16a7e3
> >> >> >>> 2. java.io.IOException: Read error rc = -1
> >> >> >>> java.nio.DirectByteBuffer[pos=0 lim=4 cap=4]
> >> >> >>> 3. at
> >> >> >>>
> >> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
> >> >> >>> 4. at
> >> >> >>>
> >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
> >> >> >>>
> >> >> >>>
> >> >> >>> Were all 5 quorum members participating? Check by doing "echo
> >> stat|nc
> >> >> >>> QUORUM_HOST port" for each member of quorum.... the below might
> >> help:
> >> >> >>>
> >> >> >>> members="1,2,3,4,5"
> >> >> >>> while true; do date; for i in $members; do echo $i; echo stat |
> nc
> >> $i
> >> >> >>> 2181;
> >> >> >>> done; sleep 10; done
> >> >> >>>
> >> >> >>> We need to improve our zk_dump. It just reports quorum
> members...
> >> not
> >> >> if
> >> >> >>> they are actually participating or not. The above will tell you
> if
> >> >> dead
> >> >> >>> or
> >> >> >>> alive.
> >> >> >>>
> >> >> >>> Please report back after checking above and following the
> >> Jean-Daniel
> >> >> >>> prescription.
> >> >> >>>
> >> >> >>> Thanks,
> >> >> >>> St.Ack
> >> >> >>>
> >> >> >>>
> >> >> >>>
> >> >> >>> On Wed, Nov 11, 2009 at 12:29 PM, Zhenyu Zhong <
> >> >> zhongresearch@gmail.com
> >> >> >>> >wrote:
> >> >> >>>
> >> >> >>> > Thanks a lot for the help.
> >> >> >>> >
> >> >> >>> > I understand that the change of disk configurations should be
> done
> >> >> >>> > progressively such that HDFS can balance and recover from a
> disk
> >> >> going
> >> >> >>> > away.
> >> >> >>> >
> >> >> >>> > The data we are trying to import into HBase is about 200million
> >> rows,
> >> >> >>> each
> >> >> >>> > row is about hundreds of bytes. Basically, we read in a text
> file
> >> >> from
> >> >> >>> > HDFS,
> >> >> >>> > then for each row in the text file, we fetch data from HBase
> >> table,
> >> >> we
> >> >> >>> > compare the column value and update the latest value back into
> >> HBase.
> >> >> I
> >> >> >>> was
> >> >> >>> > suggested to use CheckAndPut. Anyway this is the common
> usecase
> >> on
> >> >> my
> >> >> >>> > side.
> >> >> >>> >
> >> >> >>> >
> >> >> >>> > Many thanks
> >> >> >>> > zhenyu
> >> >> >>> >
> >> >> >>> >
> >> >> >>> >
> >> >> >>> > On Wed, Nov 11, 2009 at 3:03 PM, Jean-Daniel Cryans <
> >> >> >>> jdcryans@apache.org
> >> >> >>> > >wrote:
> >> >> >>> >
> >> >> >>> > > 6 disks, 19 machines and you see those kind of issues? I'm
> quite
> >> >> >>> > > surprised and I want to learn more about what you trying to
> to.
> >> >> >>> > >
> >> >> >>> > > But first, if hbase is managing ZK for you then you can
> change
> >> the
> >> >> >>> > > value of hbase.zookeeper.property.dataDir so that it points
> to
> >> its
> >> >> own
> >> >> >>> > > mount. Make sure you also change dfs.data.dir in
> >> conf/hdfs-site.xml
> >> >> >>> > > (hadoop stuff) so that it stops pointing there (you should
> have
> >> 6
> >> >> >>> > > values there at the moment if you configured it correctly).
> >> Doing
> >> >> this
> >> >> >>> > > could result in data loss if you do it all at once so do the
> >> change
> >> >> >>> > > progressively. This also means that you may have to keep a
> >> seperate
> >> >> >>> > > HDFS configuration for those particular nodes. Please ask
> more
> >> >> >>> > > questions if it's not very clear.
> >> >> >>> > >
> >> >> >>> > > WRT to dedicating nodes to Zookeeeper, it makes sense only if
> >> you
> >> >> have
> >> >> >>> > > a lot of nodes but in your case, unless there's also CPU
> >> >> contention,
> >> >> >>> > > you're good with only a dedicated disk.
> >> >> >>> > >
> >> >> >>> > > Now for what you are trying to do, is this a massive import
> that
> >> >> goes
> >> >> >>> > > over many hours? How big is each row? I'd love to get a lot
> of
> >> >> details
> >> >> >>> > > to help you optimize your cluster for your particular
> problem.
> >> >> >>> > >
> >> >> >>> > > Thx,
> >> >> >>> > >
> >> >> >>> > > J-D
> >> >> >>> > >
> >> >> >>> > > On Wed, Nov 11, 2009 at 11:51 AM, Zhenyu Zhong <
> >> >> >>> zhongresearch@gmail.com>
> >> >> >>> > > wrote:
> >> >> >>> > > > J-D,
> >> >> >>> > > >
> >> >> >>> > > > Thanks a lot for the insightful comments.
> >> >> >>> > > > We are running 5 zookeeper quorums now. On these 5 quorum
> >> >> servers,
> >> >> >>> we
> >> >> >>> > > also
> >> >> >>> > > > run datanode, tasktracker and regionserver.
> >> >> >>> > > > Now we have 6 disks per machine.
> >> >> >>> > > > If it is the disk IO saturation, is there anyway to
> configure
> >> >> >>> zookeeper
> >> >> >>> > > to
> >> >> >>> > > > write to a seperate disk which is not the same one used by
> >> HDFS?
> >> >> >>> > > > Or shall we run zookeeper quorums on different nodes other
> >> than
> >> >> >>> HDFS?
> >> >> >>> > > >
> >> >> >>> > > > Are HLogs generated by Regionservers? Would it alleviate
> the
> >> >> issue
> >> >> >>> if
> >> >> >>> > we
> >> >> >>> > > use
> >> >> >>> > > > the first 14 for regionserver, and the rest of 5 for
> >> >> zookeeper?(In
> >> >> >>> that
> >> >> >>> > > > sense, the node running zookeeper quorum won't run
> >> regionserver)
> >> >> >>> > > >
> >> >> >>> > > > I am very appreciated for your comments.
> >> >> >>> > > >
> >> >> >>> > > > Best,
> >> >> >>> > > > zhenyu
> >> >> >>> > > >
> >> >> >>> > > >
> >> >> >>> > > >
> >> >> >>> > > > On Wed, Nov 11, 2009 at 2:30 PM, Jean-Daniel Cryans <
> >> >> >>> > jdcryans@apache.org
> >> >> >>> > > >wrote:
> >> >> >>> > > >
> >> >> >>> > > >> The problem seems to be that the IO is saturated. When you
> >> see
> >> >> >>> this:
> >> >> >>> > > >>
> >> >> >>> > > >> #
> >> >> >>> > > >> 2009-11-11 04:28:10,308 WARN
> >> >> >>> > > >> org.apache.hadoop.hbase.regionserver.HLog: IPC Server
> handler
> >> 9
> >> >> on
> >> >> >>> > > >> 60021 took 6819ms appending an edit to hlog; editcount=370
> >> >> >>> > > >> #
> >> >> >>> > > >> 2009-11-11 04:28:10,308 WARN
> >> >> >>> > > >> org.apache.hadoop.hbase.regionserver.HLog: IPC Server
> handler
> >> 2
> >> >> on
> >> >> >>> > > >> 60021 took 4412ms appending an edit to hlog; editcount=371
> >> >> >>> > > >>
> >> >> >>> > > >> It's a sign that HDFS is slow. I guess that the zookeeper
> >> quorum
> >> >> >>> > > >> members are on those same nodes using the same disks? If
> so,
> >> >> >>> there's
> >> >> >>> > > >> the problem because Zookeeper needs to be able to commit
> >> >> >>> transactions
> >> >> >>> > > >> very fast and if your disks are already hogged then they
> have
> >> to
> >> >> >>> wait
> >> >> >>> > > >> and you can get disconnections. Do you still have 19
> quorum
> >> >> >>> members?
> >> >> >>> > > >> How many disks do you have per machine?
> >> >> >>> > > >>
> >> >> >>> > > >> Thx
> >> >> >>> > > >>
> >> >> >>> > > >> J-D
> >> >> >>> > > >>
> >> >> >>> > > >> On Wed, Nov 11, 2009 at 11:22 AM, Zhenyu Zhong <
> >> >> >>> > zhongresearch@gmail.com
> >> >> >>> > > >
> >> >> >>> > > >> wrote:
> >> >> >>> > > >> > Please take a look at the following regionserver log.
> >> >> >>> > > >> > The error started to come out from 4:24am UTC to 4:29am
> >> UTC.
> >> >> >>> > > >> >
> >> >> >>> > > >> > http://pastebin.com/m3717d064
> >> >> >>> > > >> >
> >> >> >>> > > >> > Thanks in advance.
> >> >> >>> > > >> >
> >> >> >>> > > >> > Best,
> >> >> >>> > > >> > zhenyu
> >> >> >>> > > >> >
> >> >> >>> > > >> >
> >> >> >>> > > >> > On Wed, Nov 11, 2009 at 2:01 PM, Jean-Daniel Cryans <
> >> >> >>> > > jdcryans@apache.org
> >> >> >>> > > >> >wrote:
> >> >> >>> > > >> >
> >> >> >>> > > >> >> Could you paste a bigger log trace (interested to see
> at
> >> >> least
> >> >> >>> all
> >> >> >>> > > 300
> >> >> >>> > > >> >> lines before that) in a pastebin?
> >> >> >>> > > >> >>
> >> >> >>> > > >> >> Thx,
> >> >> >>> > > >> >>
> >> >> >>> > > >> >> J-D
> >> >> >>> > > >> >>
> >> >> >>> > > >> >> On Wed, Nov 11, 2009 at 10:56 AM, Zhenyu Zhong <
> >> >> >>> > > zhongresearch@gmail.com
> >> >> >>> > > >> >
> >> >> >>> > > >> >> wrote:
> >> >> >>> > > >> >> > J-D,
> >> >> >>> > > >> >> >
> >> >> >>> > > >> >> > Thanks for the reply.
> >> >> >>> > > >> >> >
> >> >> >>> > > >> >> > The problem I have is one or more of the
> regionservers
> >> >> >>> disconnect
> >> >> >>> > > from
> >> >> >>> > > >> >> the
> >> >> >>> > > >> >> > cluster.
> >> >> >>> > > >> >> >
> >> >> >>> > > >> >> > I checked the resource usage including memory Heap
> usage
> >> >> for
> >> >> >>> > > >> >> regionserver,
> >> >> >>> > > >> >> > and found nothing suspicious. The regionserver heap
> >> usage
> >> >> is
> >> >> >>> way
> >> >> >>> > > below
> >> >> >>> > > >> >> the
> >> >> >>> > > >> >> > maxHeap defined in jvm.
> >> >> >>> > > >> >> >
> >> >> >>> > > >> >> > So I checked zookeeper log, then I found the
> following
> >> >> >>> > > >> >> > BufferUnderflowException, eventually it leads to
> >> >> >>> SocketException:
> >> >> >>> > > >> >> Transport
> >> >> >>> > > >> >> > endpoint is not connected.
> >> >> >>> > > >> >> >
> >> >> >>> > > >> >> >
> >> >> >>> > > >> >> > I also checked the regionserver log. It looks like
> the
> >> >> >>> > regionserver
> >> >> >>> > > >> tried
> >> >> >>> > > >> >> to
> >> >> >>> > > >> >> > connect to other zookeeper quorum server. It connects
> >> >> >>> > successfully,
> >> >> >>> > > >> but
> >> >> >>> > > >> >> then
> >> >> >>> > > >> >> > it complained about SocketException: transport
> endpoint
> >> is
> >> >> not
> >> >> >>> > > >> connected.
> >> >> >>> > > >> >> >
> >> >> >>> > > >> >> > So both zookeeper log and regionserver log told me
> that
> >> the
> >> >> >>> > > transport
> >> >> >>> > > >> >> > endpoint is not connected. I don't know why.
> >> >> >>> > > >> >> >
> >> >> >>> > > >> >> > I would be very appreciated if you could point me to
> the
> >> >> right
> >> >> >>> > > >> direction.
> >> >> >>> > > >> >> >
> >> >> >>> > > >> >> > Best,
> >> >> >>> > > >> >> >
> >> >> >>> > > >> >> >
> >> >> >>> > > >> >> > Regionserver log:
> >> >> >>> > > >> >> > 2009-11-11 04:24:53,718 INFO
> >> >> org.apache.zookeeper.ClientCnxn:
> >> >> >>> > > >> Attempting
> >> >> >>> > > >> >> > connection to server superpyxis0018/
> >> 192.168.100.132:2181
> >> >> >>> > > >> >> > 2009-11-11 04:24:53,718 INFO
> >> >> org.apache.zookeeper.ClientCnxn:
> >> >> >>> > > Priming
> >> >> >>> > > >> >> > connection to
> java.nio.channels.SocketChannel[connected
> >> >> >>> local=/
> >> >> >>> > > >> >> > 192.168.100.131:43065 remote=superpyxis0018/
> >> >> >>> 192.168.100.132:2181
> >> >> >>> > ]
> >> >> >>> > > >> >> > 2009-11-11 04:24:53,718 INFO
> >> >> org.apache.zookeeper.ClientCnxn:
> >> >> >>> > > Server
> >> >> >>> > > >> >> > connection successful
> >> >> >>> > > >> >> > 2009-11-11 04:24:53,768 INFO
> >> >> >>> > > >> >> > org.apache.hadoop.hbase.regionserver.HRegionServer:
> Got
> >> >> >>> ZooKeeper
> >> >> >>> > > >> event,
> >> >> >>> > > >> >> > state: Disconnected, type: None, path: null
> >> >> >>> > > >> >> > 2009-11-11 04:24:54,118 INFO
> >> >> org.apache.zookeeper.ClientCnxn:
> >> >> >>> > > >> Attempting
> >> >> >>> > > >> >> > connection to server superpyxis0017/
> >> 192.168.100.131:2181
> >> >> >>> > > >> >> > 2009-11-11 04:24:54,118 INFO
> >> >> org.apache.zookeeper.ClientCnxn:
> >> >> >>> > > Priming
> >> >> >>> > > >> >> > connection to
> java.nio.channels.SocketChannel[connected
> >> >> >>> local=/
> >> >> >>> > > >> >> > 192.168.100.131:46903 remote=superpyxis0017/
> >> >> >>> 192.168.100.131:2181
> >> >> >>> > ]
> >> >> >>> > > >> >> > 2009-11-11 04:24:54,118 INFO
> >> >> org.apache.zookeeper.ClientCnxn:
> >> >> >>> > > Server
> >> >> >>> > > >> >> > connection successful
> >> >> >>> > > >> >> > 2009-11-11 04:24:54,118 WARN
> >> >> org.apache.zookeeper.ClientCnxn:
> >> >> >>> > > >> Exception
> >> >> >>> > > >> >> > closing session 0x24dfd90c810002 to
> >> >> >>> > > >> sun.nio.ch.SelectionKeyImpl@2dd78ccf
> >> >> >>> > > >> >> > java.io.IOException: Read error rc = -1
> >> >> >>> > > >> java.nio.DirectByteBuffer[pos=0
> >> >> >>> > > >> >> > lim=4 cap=4]
> >> >> >>> > > >> >> > at
> >> >> >>> > > >> >> >
> >> >> >>> > >
> >> >> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
> >> >> >>> > > >> >> > at
> >> >> >>> > > >> >> >
> >> >> >>> >
> >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
> >> >> >>> > > >> >> > 2009-11-11 04:24:54,118 WARN
> >> >> org.apache.zookeeper.ClientCnxn:
> >> >> >>> > > Ignoring
> >> >> >>> > > >> >> > exception during shutdown input
> >> >> >>> > > >> >> > java.net.SocketException: Transport endpoint is not
> >> >> connected
> >> >> >>> > > >> >> > at
> sun.nio.ch.SocketChannelImpl.shutdown(Native
> >> >> Method)
> >> >> >>> > > >> >> > at
> >> >> >>> > > >> >> >
> >> >> >>> > >
> >> >>
> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
> >> >> >>> > > >> >> > at
> >> >> >>> > > >>
> >> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> >> >> >>> > > >> >> > at
> >> >> >>> > > >> >> >
> >> >> >>> > > >>
> >> >> >>> >
> >> >>
> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> >> >> >>> > > >> >> > at
> >> >> >>> > > >> >> >
> >> >> >>> >
> >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> >> >> >>> > > >> >> >
> >> >> >>> > > >> >> >
> >> >> >>> > > >> >> >
> >> >> >>> > > >> >> >
> >> >> >>> > > >> >> >
> >> >> >>> > > >> >> > On Wed, Nov 11, 2009 at 1:40 PM, Jean-Daniel Cryans <
> >> >> >>> > > >> jdcryans@apache.org
> >> >> >>> > > >> >> >wrote:
> >> >> >>> > > >> >> >
> >> >> >>> > > >> >> >> I'm really not sure exactly what the situation is
> with
> >> the
> >> >> >>> > > >> >> >> java.nio.BufferUnderflowException, but what do you
> mean
> >> by
> >> >> >>> > > >> >> >> "regionserver disconnection"? Could you give more
> >> details?
> >> >> >>> > > >> >> >>
> >> >> >>> > > >> >> >> Thx,
> >> >> >>> > > >> >> >>
> >> >> >>> > > >> >> >> J-D
> >> >> >>> > > >> >> >>
> >> >> >>> > > >> >> >> On Wed, Nov 11, 2009 at 10:34 AM, Zhenyu Zhong <
> >> >> >>> > > >> zhongresearch@gmail.com
> >> >> >>> > > >> >> >
> >> >> >>> > > >> >> >> wrote:
> >> >> >>> > > >> >> >> > I searched around and found HBASE-1645 "HBase
> could
> >> be a
> >> >> >>> > little
> >> >> >>> > > >> more
> >> >> >>> > > >> >> >> dainty
> >> >> >>> > > >> >> >> > when zk is pulled out from under it"
> >> >> >>> > > >> >> >> >
> >> >> >>> > > >> >> >> > I am wondering whether my problem is related to
> this
> >> >> >>> > HBase-1645.
> >> >> >>> > > Or
> >> >> >>> > > >> >> >> probably
> >> >> >>> > > >> >> >> > has nothing to do with it?
> >> >> >>> > > >> >> >> >
> >> >> >>> > > >> >> >> > thanks
> >> >> >>> > > >> >> >> >
> >> >> >>> > > >> >> >> >
> >> >> >>> > > >> >> >> > On Wed, Nov 11, 2009 at 12:02 PM, Zhenyu Zhong <
> >> >> >>> > > >> >> zhongresearch@gmail.com
> >> >> >>> > > >> >> >> >wrote:
> >> >> >>> > > >> >> >> >
> >> >> >>> > > >> >> >> >> Hi,
> >> >> >>> > > >> >> >> >>
> >> >> >>> > > >> >> >> >> I have been monitoring the system resources such
> as
> >> >> memory
> >> >> >>> > Heap
> >> >> >>> > > >> usage
> >> >> >>> > > >> >> >> for
> >> >> >>> > > >> >> >> >> Regionserver, load average, network bandwidth,
> etc.
> >> >> >>> > > >> >> >> >>
> >> >> >>> > > >> >> >> >> I experienced a disconnection last night. When I
> >> >> checked
> >> >> >>> the
> >> >> >>> > > >> memory
> >> >> >>> > > >> >> Heap
> >> >> >>> > > >> >> >> >> for Regionserver, it was 2336 which is way below
> the
> >> >> max
> >> >> >>> Heap
> >> >> >>> > > >> 4096. I
> >> >> >>> > > >> >> >> don't
> >> >> >>> > > >> >> >> >> think it is Regionserver problem. The requests
> that
> >> the
> >> >> >>> > > >> regionserver
> >> >> >>> > > >> >> got
> >> >> >>> > > >> >> >> was
> >> >> >>> > > >> >> >> >> about 126 which doesn't seem to be high. The load
> >> >> average
> >> >> >>> is
> >> >> >>> > > kind
> >> >> >>> > > >> of
> >> >> >>> > > >> >> >> high
> >> >> >>> > > >> >> >> >> >100. (but I saw disconnections while the load
> >> average
> >> >> is
> >> >> >>> > > around
> >> >> >>> > > >> 30).
> >> >> >>> > > >> >> >> The
> >> >> >>> > > >> >> >> >> bandwidth is not a problem.
> >> >> >>> > > >> >> >> >>
> >> >> >>> > > >> >> >> >> However, I found error messages from the
> zookeeper.
> >> >> Please
> >> >> >>> > see
> >> >> >>> > > the
> >> >> >>> > > >> >> >> >> following error messages. There are some
> >> >> >>> > > >> >> >> >> BufferUnderflowException error, IOException
> >> error,etc.
> >> >> >>> > > >> >> >> >> May I ask what might be the causes that generate
> >> these
> >> >> >>> error
> >> >> >>> > > >> >> messages?
> >> >> >>> > > >> >> >> >>
> >> >> >>> > > >> >> >> >> Many thanks
> >> >> >>> > > >> >> >> >> zhenyu
> >> >> >>> > > >> >> >> >>
> >> >> >>> > > >> >> >> >>
> >> >> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
> >> >> >>> > > org.apache.zookeeper.server.Request:
> >> >> >>> > > >> >> >> Ignoring
> >> >> >>> > > >> >> >> >> exception during toString
> >> >> >>> > > >> >> >> >> java.nio.BufferUnderflowException
> >> >> >>> > > >> >> >> >> at
> >> >> >>> > java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
> >> >> >>> > > >> >> >> >> at
> >> java.nio.ByteBuffer.get(ByteBuffer.java:675)
> >> >> >>> > > >> >> >> >> at
> >> >> >>> > > >> >> >>
> >> >> >>> org.apache.zookeeper.server.Request.toString(Request.java:199)
> >> >> >>> > > >> >> >> >> at
> >> java.lang.String.valueOf(String.java:2827)
> >> >> >>> > > >> >> >> >> at
> >> >> >>> > > java.lang.StringBuilder.append(StringBuilder.java:115)
> >> >> >>> > > >> >> >> >> at
> >> >> >>> > > >> >> >> >>
> >> >> >>> > > >> >> >>
> >> >> >>> > > >> >>
> >> >> >>> > > >>
> >> >> >>> > >
> >> >> >>> >
> >> >> >>>
> >> >>
> >>
> org.apache.zookeeper.server.quorum.CommitProcessor.processRequest(CommitProcessor.java:167)
> >> >> >>> > > >> >> >> >> at
> >> >> >>> > > >> >> >> >>
> >> >> >>> > > >> >> >>
> >> >> >>> > > >> >>
> >> >> >>> > > >>
> >> >> >>> > >
> >> >> >>> >
> >> >> >>>
> >> >>
> >>
> org.apache.zookeeper.server.quorum.FollowerRequestProcessor.run(FollowerRequestProcessor.java:68)
> >> >> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
> >> >> >>> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
> >> >> >>> > > >> >> >> >> Exception causing close of session
> 0x424dfd908c50009
> >> >> >>> > > >> >> >> >> due to java.io.IOException: Read error
> >> >> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
> >> >> >>> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
> >> >> >>> > > >> >> >> >> IOException stack trace
> >> >> >>> > > >> >> >> >> java.io.IOException: Read error
> >> >> >>> > > >> >> >> >> at
> >> >> >>> > > >> >> >> >>
> >> >> >>> > > >> >>
> >> >> >>> > >
> >> >>
> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:372)
> >> >> >>> > > >> >> >> >> at
> >> >> >>> > > >> >> >> >>
> >> >> >>> > > >> >> >>
> >> >> >>> > > >> >>
> >> >> >>> > > >>
> >> >> >>> > >
> >> >> >>> >
> >> >> >>>
> >> >>
> >>
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:221)
> >> >> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
> >> >> >>> > > >> >> >> >>
> org.apache.zookeeper.server.quorum.CommitProcessor:
> >> >> >>> > Processing
> >> >> >>> > > >> >> request::
> >> >> >>> > > >> >> >> >> sessionid:0x424dfd908c50009 type:setWatches
> >> >> >>> > > >> cxid:0xfffffffffffffff8
> >> >> >>> > > >> >> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
> >> >> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
> >> >> >>> > > org.apache.zookeeper.server.Request:
> >> >> >>> > > >> >> >> Ignoring
> >> >> >>> > > >> >> >> >> exception during toString
> >> >> >>> > > >> >> >> >> java.nio.BufferUnderflowException
> >> >> >>> > > >> >> >> >> at
> >> >> >>> > java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
> >> >> >>> > > >> >> >> >> at
> >> java.nio.ByteBuffer.get(ByteBuffer.java:675)
> >> >> >>> > > >> >> >> >> at
> >> >> >>> > > >> >> >>
> >> >> >>> org.apache.zookeeper.server.Request.toString(Request.java:199)
> >> >> >>> > > >> >> >> >> at
> >> java.lang.String.valueOf(String.java:2827)
> >> >> >>> > > >> >> >> >> at
> >> >> >>> > > java.lang.StringBuilder.append(StringBuilder.java:115)
> >> >> >>> > > >> >> >> >> at
> >> >> >>> > > >> >> >> >>
> >> >> >>> > > >> >> >>
> >> >> >>> > > >> >>
> >> >> >>> > > >>
> >> >> >>> > >
> >> >> >>> >
> >> >> >>>
> >> >>
> >>
> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:74)
> >> >> >>> > > >> >> >> >> at
> >> >> >>> > > >> >> >> >>
> >> >> >>> > > >> >> >>
> >> >> >>> > > >> >>
> >> >> >>> > > >>
> >> >> >>> > >
> >> >> >>> >
> >> >> >>>
> >> >>
> >>
> org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)
> >> >> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
> >> >> >>> > > >> >> >> >>
> org.apache.zookeeper.server.FinalRequestProcessor:
> >> >> >>> Processing
> >> >> >>> > > >> >> request::
> >> >> >>> > > >> >> >> >> sessionid:0x424dfd908c50009 type:setWatches
> >> >> >>> > > >> cxid:0xfffffffffffffff8
> >> >> >>> > > >> >> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
> >> >> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
> >> >> >>> > > org.apache.zookeeper.server.Request:
> >> >> >>> > > >> >> >> Ignoring
> >> >> >>> > > >> >> >> >> exception during toString
> >> >> >>> > > >> >> >> >> java.nio.BufferUnderflowException
> >> >> >>> > > >> >> >> >> at
> >> >> >>> > java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
> >> >> >>> > > >> >> >> >> at
> >> java.nio.ByteBuffer.get(ByteBuffer.java:675)
> >> >> >>> > > >> >> >> >> at
> >> >> >>> > > >> >> >>
> >> >> >>> org.apache.zookeeper.server.Request.toString(Request.java:199)
> >> >> >>> > > >> >> >> >> at
> >> >> >>> > > >> >> >> >>
> >> >> >>> > > >>
> >> >> >>>
> >> org.apache.log4j.or.DefaultRenderer.doRender(DefaultRenderer.java:36)
> >> >> >>> > > >> >> >> >> at
> >> >> >>> > > >> >> >> >>
> >> >> >>> > >
> >> org.apache.log4j.or.RendererMap.findAndRender(RendererMap.java:80)
> >> >> >>> > > >> >> >> >> at
> >> >> >>> > > >> >> >> >>
> >> >> >>> > > >> >> >>
> >> >> >>> > > >> >>
> >> >> >>> > > >>
> >> >> >>> > >
> >> >> >>> >
> >> >> >>>
> >> >>
> >>
> org.apache.log4j.spi.LoggingEvent.getRenderedMessage(LoggingEvent.java:362)
> >> >> >>> > > >> >> >> >> at
> >> >> >>> > > >> >> >> >>
> >> >> >>> > > >> >> >>
> >> >> >>> > > >> >>
> >> >> >>> > > >>
> >> >> >>> > >
> >> >> >>> >
> >> >> >>>
> >> >>
> >>
> org.apache.log4j.helpers.PatternParser$BasicPatternConverter.convert(PatternParser.java:403)
> >> >> >>> > > >> >> >> >> at
> >> >> >>> > > >> >> >> >>
> >> >> >>> > > >> >> >>
> >> >> >>> > > >> >>
> >> >> >>> > > >>
> >> >> >>> > >
> >> >> >>> >
> >> >> >>>
> >> >>
> >>
> org.apache.log4j.helpers.PatternConverter.format(PatternConverter.java:65)
> >> >> >>> > > >> >> >> >> at
> >> >> >>> > > >> >>
> >> org.apache.log4j.PatternLayout.format(PatternLayout.java:502)
> >> >> >>> > > >> >> >> >> at
> >> >> >>> > > >> >> >> >>
> >> >> >>> > >
> >> org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:302)
> >> >> >>> > > >> >> >> >> at
> >> >> >>> > > >> >> >> >>
> >> >> >>> > > >> >> >>
> >> >> >>> > > >> >>
> >> >> >>> > > >>
> >> >> >>> > >
> >> >> >>> >
> >> >> >>>
> >> >>
> >>
> org.apache.log4j.DailyRollingFileAppender.subAppend(DailyRollingFileAppender.java:359)
> >> >> >>> > > >> >> >> >> at
> >> >> >>> > > >> >> >>
> >> >> >>> org.apache.log4j.WriterAppender.append(WriterAppender.java:160)
> >> >> >>> > > >> >> >> >> at
> >> >> >>> > > >> >> >> >>
> >> >> >>> > > >>
> >> >> >>>
> >> org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
> >> >> >>> > > >> >> >> >> at
> >> >> >>> > > >> >> >> >>
> >> >> >>> > > >> >> >>
> >> >> >>> > > >> >>
> >> >> >>> > > >>
> >> >> >>> > >
> >> >> >>> >
> >> >> >>>
> >> >>
> >>
> org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
> >> >> >>> > > >> >> >> >> at
> >> >> >>> > > >> org.apache.log4j.Category.callAppenders(Category.java:206)
> >> >> >>> > > >> >> >> >> at
> >> >> >>> > > org.apache.log4j.Category.forcedLog(Category.java:391)
> >> >> >>> > > >> >> >> >> at
> >> >> >>> org.apache.log4j.Category.debug(Category.java:260)
> >> >> >>> > > >> >> >> >> at
> >> >> >>> > > >> >> >> >>
> >> >> >>> > > >> >> >>
> >> >> >>> > > >> >>
> >> >> >>> > > >>
> >> >> >>> > >
> >> >> >>> >
> >> >> >>>
> >> >>
> >>
> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:147)
> >> >> >>> > > >> >> >> >> at
> >> >> >>> > > >> >> >> >>
> >> >> >>> > > >> >> >>
> >> >> >>> > > >> >>
> >> >> >>> > > >>
> >> >> >>> > >
> >> >> >>> >
> >> >> >>>
> >> >>
> >>
> org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)
> >> >> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
> >> >> >>> > > org.apache.zookeeper.server.Request:
> >> >> >>> > > >> >> >> Ignoring
> >> >> >>> > > >> >> >> >> exception during toString
> >> >> >>> > > >> >> >> >> sessionid:0x424dfd908c50009 type:setWatches
> >> >> >>> > > >> cxid:0xfffffffffffffff8
> >> >> >>> > > >> >> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
> >> >> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 INFO
> >> >> >>> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
> >> >> >>> > > >> >> >> >> closing session:0x424dfd908c50009 NIOServerCnxn:
> >> >> >>> > > >> >> >> >> java.nio.channels.SocketChannel[connected local=/
> >> >> >>> > > >> >> 192.168.100.131:2181
> >> >> >>> > > >> >> >> remote=/
> >> >> >>> > > >> >> >> >> 192.168.100.131:35500]
> >> >> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
> >> >> >>> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
> >> >> >>> > > >> >> >> >> ignoring exception during input shutdown
> >> >> >>> > > >> >> >> >> java.net.SocketException: Transport endpoint is
> not
> >> >> >>> connected
> >> >> >>> > > >> >> >> >> at
> >> sun.nio.ch.SocketChannelImpl.shutdown(Native
> >> >> >>> > Method)
> >> >> >>> > > >> >> >> >> at
> >> >> >>> > > >> >> >> >>
> >> >> >>> > > >> >>
> >> >> >>> > >
> >> >>
> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
> >> >> >>> > > >> >> >> >> at
> >> >> >>> > > >> >> >>
> >> >> >>> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> >> >> >>> > > >> >> >> >> at
> >> >> >>> > > >> >> >> >>
> >> >> >>> > > >> >>
> >> >> >>> > >
> >> >> >>>
> >> org.apache.zookeeper.server.NIOServerCnxn.close(NIOServerCnxn.java:848)
> >> >> >>> > > >> >> >> >> at
> >> >> >>> > > >> >> >> >>
> >> >> >>> > > >> >>
> >> >> >>> > >
> >> >>
> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:498)
> >> >> >>> > > >> >> >> >> at
> >> >> >>> > > >> >> >> >>
> >> >> >>> > > >> >> >>
> >> >> >>> > > >> >>
> >> >> >>> > > >>
> >> >> >>> > >
> >> >> >>> >
> >> >> >>>
> >> >>
> >>
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:221)
> >> >> >>> > > >> >> >> >>
> >> >> >>> > > >> >> >> >
> >> >> >>> > > >> >> >>
> >> >> >>> > > >> >> >
> >> >> >>> > > >> >>
> >> >> >>> > > >> >
> >> >> >>> > > >>
> >> >> >>> > > >
> >> >> >>> > >
> >> >> >>> >
> >> >> >>>
> >> >> >>
> >> >> >>
> >> >> >
> >> >>
> >> >
> >>
> >
>
Re: regionserver disconnection
Posted by Jean-Daniel Cryans <jd...@apache.org>.
Could we see the master log from around the same time the region
server got a session expired?
Thx,
J-D
On Thu, Nov 12, 2009 at 10:54 AM, Zhenyu Zhong <zh...@gmail.com> wrote:
> Sorry, there was a typo in my email. But it is 10 minutes that I set in the
> hbase-site.xml.
>
> Please see the link.
> http://pastebin.com/m564c51bc
>
> Very much appreciated.
> zhenyu
>
> On Thu, Nov 12, 2009 at 1:42 PM, Jean-Daniel Cryans <jd...@apache.org>wrote:
>
>> Usually if it was 10 minutes we would see sleepers complaining they
>> slept for way too long. In the logs I can see a 1 minute pause but not
>> 10 minutes because of the ZK debug. Are you sure you don't have a
>> typo? Like what you just wrote is not valid, you wrote
>> zookeeper.session.timout instead of zookeeper.session.timeout. Are you
>> sure it's distributed over the cluster? Could you pastebin
>> hbase-site.xml?
>>
>> Thx,
>>
>> J-D
>>
>> On Thu, Nov 12, 2009 at 10:28 AM, Zhenyu Zhong <zh...@gmail.com>
>> wrote:
>> > J-D,
>> >
>> > Yes, that is weird. But it is what I have in the log. Probably the log is
>> > flushed by different threads, and one thread is blocked for a while?
>> >
>> > I set the zookeeper.session.timout=600000, which is 10 minutes. However,
>> we
>> > still see this problem.
>> >
>> > Does GC pause for such a long time period?
>> > I will try different GC options to see the differences.
>> >
>> > thanks
>> > zhenyu
>> >
>> >
>> > On Thu, Nov 12, 2009 at 1:10 PM, Jean-Daniel Cryans <jdcryans@apache.org
>> >wrote:
>> >
>> >> I see something weird
>> >>
>> >> 2009-11-12 15:04:36,370 WARN
>> >> org.apache.hadoop.hbase.regionserver.HRegionServer: Attempt=1
>> >> org.apache.hadoop.hbase.Leases$LeaseStillHeldException
>> >> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>> >> Method)
>> >> at
>> >>
>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>> >> at
>> >>
>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>> >> ...
>> >> 2009-11-12 15:03:52,682 DEBUG org.apache.zookeeper.ClientCnxn: Got
>> >> ping response for sessionid:0x224e55436ad0004 after 0ms
>> >> 2009-11-12 15:04:52,480 WARN org.apache.zookeeper.ClientCnxn:
>> >> Exception closing session 0x24e55436a0007d to
>> >> sun.nio.ch.SelectionKeyImpl@136f6ed1
>> >>
>> >> Goes back in time? I does look better tho. Did you say you set the
>> >> zookeeper session timeout higher in a previous thread? I'd be
>> >> interested to see if a higher (2-3 minutes) value would solve the
>> >> problem until java 7.
>> >>
>> >> The only thing that happens after that LeaseStillHeldException is that
>> >> for some reason (probably GC) the region server paused. This caused
>> >> the session to timeout and should be why the RS isn't connected to the
>> >> ZK quorum member.
>> >>
>> >> BTW we worked with the Zookeeper dev team to have more meaningful
>> >> exceptions and it's already committed in their trunk.
>> >>
>> >> J-D
>> >>
>> >> On Thu, Nov 12, 2009 at 8:40 AM, Zhenyu Zhong <zh...@gmail.com>
>> >> wrote:
>> >> > Follow up on the regionserver disconnection:
>> >> >
>> >> > I made a few changes to the cluster setup.
>> >> > 1. split the regionserver and zookeeper quorum servers, such that
>> HRegion
>> >> > doesn't run on the same box as HQuorumPeer. I assume that there should
>> be
>> >> no
>> >> > disk io contention between the regionserver and zookeeper.
>> >> > Now we have 14 regionservers, and 5 zookeeper quorum servers.
>> >> >
>> >> > Though I experienced 2 regionserver disconnection this morning, it
>> looks
>> >> > better from the regionserver log. (please see the following log)
>> >> >
>> >> >
>> >> > http://pastebin.com/m496dbfae
>> >> >
>> >> >
>> >> > I checked diskIO for one of the regionserver(192.168.100.116) during
>> the
>> >> > time it disconnected.
>> >> >
>> >> > Time: 03:04:58 AM
>> >> > Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
>> >> > sda 105.31 5458.83 19503.64 9043873239
>> 32312473676
>> >> > sda1 2.90 6.64 99.25 10993934 164433464
>> >> > sda2 1.72 23.76 30.25 39365817 50124008
>> >> > sda3 0.30 0.38 3.58 624930 5923000
>> >> > sda4 100.39 5428.06 19370.56 8992888270
>> 32091993204
>> >> >
>> >> >
>> >> > I also checked the zookeeper quorum server that the regionserver tried
>> to
>> >> > connect according to the log. However, I don't see 192.168.100.116 in
>> the
>> >> > client list of the zookeeper quorum server that the regionserver tried
>> to
>> >> > connect.
>> >> > Would that be a problem?
>> >> >
>> >> > Thu Nov 12 15:04:52 UTC 2009
>> >> > Zookeeper version: 3.2.1-808558, built on 08/27/2009 18:48 GMT
>> >> > Clients:
>> >> > /192.168.100.127:43045[1](queued=0,recved=26,sent=0)
>> >> > /192.168.100.131:39091[1](queued=0,recved=964,sent=0)
>> >> > /192.168.100.124:35961[1](queued=0,recved=3266,sent=0)
>> >> > /192.168.100.123:47935[1](queued=0,recved=233,sent=0)
>> >> > /192.168.100.125:46931[1](queued=0,recved=2,sent=0)
>> >> > /192.168.100.118:54924[1](queued=0,recved=295,sent=0)
>> >> > /192.168.100.118:41390[1](queued=0,recved=2290,sent=0)
>> >> > /192.168.100.136:42243[1](queued=0,recved=0,sent=0)
>> >> >
>> >> > Latency min/avg/max: 0/17/6333
>> >> > Received: 47111
>> >> > Sent: 0
>> >> > Outstanding: 0
>> >> > Zxid: 0x77000083f4
>> >> > Mode: leader
>> >> > Node count: 23
>> >> >
>> >> > Best,
>> >> > zhenyu
>> >> >
>> >> >
>> >> > On Wed, Nov 11, 2009 at 3:58 PM, Zhenyu Zhong <
>> zhongresearch@gmail.com
>> >> >wrote:
>> >> >
>> >> >> Stack
>> >> >>
>> >> >> I am very appreciated for your comments.
>> >> >> I will use the zookeeper monitoring script on my cluster and let it
>> run
>> >> >> overnight to see the result.
>> >> >> I will follow up the thread when I get anything.
>> >> >>
>> >> >> thanks
>> >> >> zhenyu
>> >> >>
>> >> >>
>> >> >> On Wed, Nov 11, 2009 at 3:52 PM, stack <st...@duboce.net> wrote:
>> >> >>
>> >> >>> I see these in your log too:
>> >> >>>
>> >> >>>
>> >> >>> 1. 2009-11-11 04:27:19,018 DEBUG org.apache.zookeeper.ClientCnxn:
>> Got
>> >> >>> ping response for sessionid:0x424dfd908c50009 after 4544ms
>> >> >>> 2. 2009-11-11 04:27:19,018 DEBUG org.apache.zookeeper.ClientCnxn:
>> Got
>> >> >>> ping response for sessionid:0x24dfd90c810002 after 4548ms
>> >> >>> 3. 2009-11-11 04:27:43,960 DEBUG org.apache.zookeeper.ClientCnxn:
>> Got
>> >> >>> ping response for sessionid:0x424dfd908c50009 after 9030ms
>> >> >>> 4. 2009-11-11 04:27:43,960 DEBUG org.apache.zookeeper.ClientCnxn:
>> Got
>> >> >>> ping response for sessionid:0x24dfd90c810002 after 9030ms
>> >> >>> 5. 2009-11-11 04:27:48,830 DEBUG org.apache.zookeeper.ClientCnxn:
>> Got
>> >> >>> ping response for sessionid:0x424dfd908c50009 after 0ms
>> >> >>> 6. 2009-11-11 04:27:48,830 DEBUG org.apache.zookeeper.ClientCnxn:
>> Got
>> >> >>> ping response for sessionid:0x24dfd90c810002 after 0ms
>> >> >>>
>> >> >>> Which would indicate lag talking to zk too.
>> >> >>>
>> >> >>> Appending HLog also seems to get steadily worse.
>> >> >>>
>> >> >>> I also see these:
>> >> >>>
>> >> >>>
>> >> >>> 1. 2009-11-11 04:27:01,233 WARN org.apache.zookeeper.ClientCnxn:
>> >> >>> Exception closing session 0x24dfd90c810002 to
>> >> >>> sun.nio.ch.SelectionKeyImpl@5e16a7e3
>> >> >>> 2. java.io.IOException: Read error rc = -1
>> >> >>> java.nio.DirectByteBuffer[pos=0 lim=4 cap=4]
>> >> >>> 3. at
>> >> >>>
>> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
>> >> >>> 4. at
>> >> >>>
>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
>> >> >>>
>> >> >>>
>> >> >>> Were all 5 quorum members participating? Check by doing "echo
>> stat|nc
>> >> >>> QUORUM_HOST port" for each member of quorum.... the below might
>> help:
>> >> >>>
>> >> >>> members="1,2,3,4,5"
>> >> >>> while true; do date; for i in $members; do echo $i; echo stat | nc
>> $i
>> >> >>> 2181;
>> >> >>> done; sleep 10; done
>> >> >>>
>> >> >>> We need to improve our zk_dump. It just reports quorum members...
>> not
>> >> if
>> >> >>> they are actually participating or not. The above will tell you if
>> >> dead
>> >> >>> or
>> >> >>> alive.
>> >> >>>
>> >> >>> Please report back after checking above and following the
>> Jean-Daniel
>> >> >>> prescription.
>> >> >>>
>> >> >>> Thanks,
>> >> >>> St.Ack
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>> On Wed, Nov 11, 2009 at 12:29 PM, Zhenyu Zhong <
>> >> zhongresearch@gmail.com
>> >> >>> >wrote:
>> >> >>>
>> >> >>> > Thanks a lot for the help.
>> >> >>> >
>> >> >>> > I understand that the change of disk configurations should be done
>> >> >>> > progressively such that HDFS can balance and recover from a disk
>> >> going
>> >> >>> > away.
>> >> >>> >
>> >> >>> > The data we are trying to import into HBase is about 200million
>> rows,
>> >> >>> each
>> >> >>> > row is about hundreds of bytes. Basically, we read in a text file
>> >> from
>> >> >>> > HDFS,
>> >> >>> > then for each row in the text file, we fetch data from HBase
>> table,
>> >> we
>> >> >>> > compare the column value and update the latest value back into
>> HBase.
>> >> I
>> >> >>> was
>> >> >>> > suggested to use CheckAndPut. Anyway this is the common usecase
>> on
>> >> my
>> >> >>> > side.
>> >> >>> >
>> >> >>> >
>> >> >>> > Many thanks
>> >> >>> > zhenyu
>> >> >>> >
>> >> >>> >
>> >> >>> >
>> >> >>> > On Wed, Nov 11, 2009 at 3:03 PM, Jean-Daniel Cryans <
>> >> >>> jdcryans@apache.org
>> >> >>> > >wrote:
>> >> >>> >
>> >> >>> > > 6 disks, 19 machines and you see those kind of issues? I'm quite
>> >> >>> > > surprised and I want to learn more about what you trying to to.
>> >> >>> > >
>> >> >>> > > But first, if hbase is managing ZK for you then you can change
>> the
>> >> >>> > > value of hbase.zookeeper.property.dataDir so that it points to
>> its
>> >> own
>> >> >>> > > mount. Make sure you also change dfs.data.dir in
>> conf/hdfs-site.xml
>> >> >>> > > (hadoop stuff) so that it stops pointing there (you should have
>> 6
>> >> >>> > > values there at the moment if you configured it correctly).
>> Doing
>> >> this
>> >> >>> > > could result in data loss if you do it all at once so do the
>> change
>> >> >>> > > progressively. This also means that you may have to keep a
>> seperate
>> >> >>> > > HDFS configuration for those particular nodes. Please ask more
>> >> >>> > > questions if it's not very clear.
>> >> >>> > >
>> >> >>> > > WRT to dedicating nodes to Zookeeeper, it makes sense only if
>> you
>> >> have
>> >> >>> > > a lot of nodes but in your case, unless there's also CPU
>> >> contention,
>> >> >>> > > you're good with only a dedicated disk.
>> >> >>> > >
>> >> >>> > > Now for what you are trying to do, is this a massive import that
>> >> goes
>> >> >>> > > over many hours? How big is each row? I'd love to get a lot of
>> >> details
>> >> >>> > > to help you optimize your cluster for your particular problem.
>> >> >>> > >
>> >> >>> > > Thx,
>> >> >>> > >
>> >> >>> > > J-D
>> >> >>> > >
>> >> >>> > > On Wed, Nov 11, 2009 at 11:51 AM, Zhenyu Zhong <
>> >> >>> zhongresearch@gmail.com>
>> >> >>> > > wrote:
>> >> >>> > > > J-D,
>> >> >>> > > >
>> >> >>> > > > Thanks a lot for the insightful comments.
>> >> >>> > > > We are running 5 zookeeper quorums now. On these 5 quorum
>> >> servers,
>> >> >>> we
>> >> >>> > > also
>> >> >>> > > > run datanode, tasktracker and regionserver.
>> >> >>> > > > Now we have 6 disks per machine.
>> >> >>> > > > If it is the disk IO saturation, is there anyway to configure
>> >> >>> zookeeper
>> >> >>> > > to
>> >> >>> > > > write to a seperate disk which is not the same one used by
>> HDFS?
>> >> >>> > > > Or shall we run zookeeper quorums on different nodes other
>> than
>> >> >>> HDFS?
>> >> >>> > > >
>> >> >>> > > > Are HLogs generated by Regionservers? Would it alleviate the
>> >> issue
>> >> >>> if
>> >> >>> > we
>> >> >>> > > use
>> >> >>> > > > the first 14 for regionserver, and the rest of 5 for
>> >> zookeeper?(In
>> >> >>> that
>> >> >>> > > > sense, the node running zookeeper quorum won't run
>> regionserver)
>> >> >>> > > >
>> >> >>> > > > I am very appreciated for your comments.
>> >> >>> > > >
>> >> >>> > > > Best,
>> >> >>> > > > zhenyu
>> >> >>> > > >
>> >> >>> > > >
>> >> >>> > > >
>> >> >>> > > > On Wed, Nov 11, 2009 at 2:30 PM, Jean-Daniel Cryans <
>> >> >>> > jdcryans@apache.org
>> >> >>> > > >wrote:
>> >> >>> > > >
>> >> >>> > > >> The problem seems to be that the IO is saturated. When you
>> see
>> >> >>> this:
>> >> >>> > > >>
>> >> >>> > > >> #
>> >> >>> > > >> 2009-11-11 04:28:10,308 WARN
>> >> >>> > > >> org.apache.hadoop.hbase.regionserver.HLog: IPC Server handler
>> 9
>> >> on
>> >> >>> > > >> 60021 took 6819ms appending an edit to hlog; editcount=370
>> >> >>> > > >> #
>> >> >>> > > >> 2009-11-11 04:28:10,308 WARN
>> >> >>> > > >> org.apache.hadoop.hbase.regionserver.HLog: IPC Server handler
>> 2
>> >> on
>> >> >>> > > >> 60021 took 4412ms appending an edit to hlog; editcount=371
>> >> >>> > > >>
>> >> >>> > > >> It's a sign that HDFS is slow. I guess that the zookeeper
>> quorum
>> >> >>> > > >> members are on those same nodes using the same disks? If so,
>> >> >>> there's
>> >> >>> > > >> the problem because Zookeeper needs to be able to commit
>> >> >>> transactions
>> >> >>> > > >> very fast and if your disks are already hogged then they have
>> to
>> >> >>> wait
>> >> >>> > > >> and you can get disconnections. Do you still have 19 quorum
>> >> >>> members?
>> >> >>> > > >> How many disks do you have per machine?
>> >> >>> > > >>
>> >> >>> > > >> Thx
>> >> >>> > > >>
>> >> >>> > > >> J-D
>> >> >>> > > >>
>> >> >>> > > >> On Wed, Nov 11, 2009 at 11:22 AM, Zhenyu Zhong <
>> >> >>> > zhongresearch@gmail.com
>> >> >>> > > >
>> >> >>> > > >> wrote:
>> >> >>> > > >> > Please take a look at the following regionserver log.
>> >> >>> > > >> > The error started to come out from 4:24am UTC to 4:29am
>> UTC.
>> >> >>> > > >> >
>> >> >>> > > >> > http://pastebin.com/m3717d064
>> >> >>> > > >> >
>> >> >>> > > >> > Thanks in advance.
>> >> >>> > > >> >
>> >> >>> > > >> > Best,
>> >> >>> > > >> > zhenyu
>> >> >>> > > >> >
>> >> >>> > > >> >
>> >> >>> > > >> > On Wed, Nov 11, 2009 at 2:01 PM, Jean-Daniel Cryans <
>> >> >>> > > jdcryans@apache.org
>> >> >>> > > >> >wrote:
>> >> >>> > > >> >
>> >> >>> > > >> >> Could you paste a bigger log trace (interested to see at
>> >> least
>> >> >>> all
>> >> >>> > > 300
>> >> >>> > > >> >> lines before that) in a pastebin?
>> >> >>> > > >> >>
>> >> >>> > > >> >> Thx,
>> >> >>> > > >> >>
>> >> >>> > > >> >> J-D
>> >> >>> > > >> >>
>> >> >>> > > >> >> On Wed, Nov 11, 2009 at 10:56 AM, Zhenyu Zhong <
>> >> >>> > > zhongresearch@gmail.com
>> >> >>> > > >> >
>> >> >>> > > >> >> wrote:
>> >> >>> > > >> >> > J-D,
>> >> >>> > > >> >> >
>> >> >>> > > >> >> > Thanks for the reply.
>> >> >>> > > >> >> >
>> >> >>> > > >> >> > The problem I have is one or more of the regionservers
>> >> >>> disconnect
>> >> >>> > > from
>> >> >>> > > >> >> the
>> >> >>> > > >> >> > cluster.
>> >> >>> > > >> >> >
>> >> >>> > > >> >> > I checked the resource usage including memory Heap usage
>> >> for
>> >> >>> > > >> >> regionserver,
>> >> >>> > > >> >> > and found nothing suspicious. The regionserver heap
>> usage
>> >> is
>> >> >>> way
>> >> >>> > > below
>> >> >>> > > >> >> the
>> >> >>> > > >> >> > maxHeap defined in jvm.
>> >> >>> > > >> >> >
>> >> >>> > > >> >> > So I checked zookeeper log, then I found the following
>> >> >>> > > >> >> > BufferUnderflowException, eventually it leads to
>> >> >>> SocketException:
>> >> >>> > > >> >> Transport
>> >> >>> > > >> >> > endpoint is not connected.
>> >> >>> > > >> >> >
>> >> >>> > > >> >> >
>> >> >>> > > >> >> > I also checked the regionserver log. It looks like the
>> >> >>> > regionserver
>> >> >>> > > >> tried
>> >> >>> > > >> >> to
>> >> >>> > > >> >> > connect to other zookeeper quorum server. It connects
>> >> >>> > successfully,
>> >> >>> > > >> but
>> >> >>> > > >> >> then
>> >> >>> > > >> >> > it complained about SocketException: transport endpoint
>> is
>> >> not
>> >> >>> > > >> connected.
>> >> >>> > > >> >> >
>> >> >>> > > >> >> > So both zookeeper log and regionserver log told me that
>> the
>> >> >>> > > transport
>> >> >>> > > >> >> > endpoint is not connected. I don't know why.
>> >> >>> > > >> >> >
>> >> >>> > > >> >> > I would be very appreciated if you could point me to the
>> >> right
>> >> >>> > > >> direction.
>> >> >>> > > >> >> >
>> >> >>> > > >> >> > Best,
>> >> >>> > > >> >> >
>> >> >>> > > >> >> >
>> >> >>> > > >> >> > Regionserver log:
>> >> >>> > > >> >> > 2009-11-11 04:24:53,718 INFO
>> >> org.apache.zookeeper.ClientCnxn:
>> >> >>> > > >> Attempting
>> >> >>> > > >> >> > connection to server superpyxis0018/
>> 192.168.100.132:2181
>> >> >>> > > >> >> > 2009-11-11 04:24:53,718 INFO
>> >> org.apache.zookeeper.ClientCnxn:
>> >> >>> > > Priming
>> >> >>> > > >> >> > connection to java.nio.channels.SocketChannel[connected
>> >> >>> local=/
>> >> >>> > > >> >> > 192.168.100.131:43065 remote=superpyxis0018/
>> >> >>> 192.168.100.132:2181
>> >> >>> > ]
>> >> >>> > > >> >> > 2009-11-11 04:24:53,718 INFO
>> >> org.apache.zookeeper.ClientCnxn:
>> >> >>> > > Server
>> >> >>> > > >> >> > connection successful
>> >> >>> > > >> >> > 2009-11-11 04:24:53,768 INFO
>> >> >>> > > >> >> > org.apache.hadoop.hbase.regionserver.HRegionServer: Got
>> >> >>> ZooKeeper
>> >> >>> > > >> event,
>> >> >>> > > >> >> > state: Disconnected, type: None, path: null
>> >> >>> > > >> >> > 2009-11-11 04:24:54,118 INFO
>> >> org.apache.zookeeper.ClientCnxn:
>> >> >>> > > >> Attempting
>> >> >>> > > >> >> > connection to server superpyxis0017/
>> 192.168.100.131:2181
>> >> >>> > > >> >> > 2009-11-11 04:24:54,118 INFO
>> >> org.apache.zookeeper.ClientCnxn:
>> >> >>> > > Priming
>> >> >>> > > >> >> > connection to java.nio.channels.SocketChannel[connected
>> >> >>> local=/
>> >> >>> > > >> >> > 192.168.100.131:46903 remote=superpyxis0017/
>> >> >>> 192.168.100.131:2181
>> >> >>> > ]
>> >> >>> > > >> >> > 2009-11-11 04:24:54,118 INFO
>> >> org.apache.zookeeper.ClientCnxn:
>> >> >>> > > Server
>> >> >>> > > >> >> > connection successful
>> >> >>> > > >> >> > 2009-11-11 04:24:54,118 WARN
>> >> org.apache.zookeeper.ClientCnxn:
>> >> >>> > > >> Exception
>> >> >>> > > >> >> > closing session 0x24dfd90c810002 to
>> >> >>> > > >> sun.nio.ch.SelectionKeyImpl@2dd78ccf
>> >> >>> > > >> >> > java.io.IOException: Read error rc = -1
>> >> >>> > > >> java.nio.DirectByteBuffer[pos=0
>> >> >>> > > >> >> > lim=4 cap=4]
>> >> >>> > > >> >> > at
>> >> >>> > > >> >> >
>> >> >>> > >
>> >> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
>> >> >>> > > >> >> > at
>> >> >>> > > >> >> >
>> >> >>> >
>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
>> >> >>> > > >> >> > 2009-11-11 04:24:54,118 WARN
>> >> org.apache.zookeeper.ClientCnxn:
>> >> >>> > > Ignoring
>> >> >>> > > >> >> > exception during shutdown input
>> >> >>> > > >> >> > java.net.SocketException: Transport endpoint is not
>> >> connected
>> >> >>> > > >> >> > at sun.nio.ch.SocketChannelImpl.shutdown(Native
>> >> Method)
>> >> >>> > > >> >> > at
>> >> >>> > > >> >> >
>> >> >>> > >
>> >> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
>> >> >>> > > >> >> > at
>> >> >>> > > >>
>> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>> >> >>> > > >> >> > at
>> >> >>> > > >> >> >
>> >> >>> > > >>
>> >> >>> >
>> >> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
>> >> >>> > > >> >> > at
>> >> >>> > > >> >> >
>> >> >>> >
>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
>> >> >>> > > >> >> >
>> >> >>> > > >> >> >
>> >> >>> > > >> >> >
>> >> >>> > > >> >> >
>> >> >>> > > >> >> >
>> >> >>> > > >> >> > On Wed, Nov 11, 2009 at 1:40 PM, Jean-Daniel Cryans <
>> >> >>> > > >> jdcryans@apache.org
>> >> >>> > > >> >> >wrote:
>> >> >>> > > >> >> >
>> >> >>> > > >> >> >> I'm really not sure exactly what the situation is with
>> the
>> >> >>> > > >> >> >> java.nio.BufferUnderflowException, but what do you mean
>> by
>> >> >>> > > >> >> >> "regionserver disconnection"? Could you give more
>> details?
>> >> >>> > > >> >> >>
>> >> >>> > > >> >> >> Thx,
>> >> >>> > > >> >> >>
>> >> >>> > > >> >> >> J-D
>> >> >>> > > >> >> >>
>> >> >>> > > >> >> >> On Wed, Nov 11, 2009 at 10:34 AM, Zhenyu Zhong <
>> >> >>> > > >> zhongresearch@gmail.com
>> >> >>> > > >> >> >
>> >> >>> > > >> >> >> wrote:
>> >> >>> > > >> >> >> > I searched around and found HBASE-1645 "HBase could
>> be a
>> >> >>> > little
>> >> >>> > > >> more
>> >> >>> > > >> >> >> dainty
>> >> >>> > > >> >> >> > when zk is pulled out from under it"
>> >> >>> > > >> >> >> >
>> >> >>> > > >> >> >> > I am wondering whether my problem is related to this
>> >> >>> > HBase-1645.
>> >> >>> > > Or
>> >> >>> > > >> >> >> probably
>> >> >>> > > >> >> >> > has nothing to do with it?
>> >> >>> > > >> >> >> >
>> >> >>> > > >> >> >> > thanks
>> >> >>> > > >> >> >> >
>> >> >>> > > >> >> >> >
>> >> >>> > > >> >> >> > On Wed, Nov 11, 2009 at 12:02 PM, Zhenyu Zhong <
>> >> >>> > > >> >> zhongresearch@gmail.com
>> >> >>> > > >> >> >> >wrote:
>> >> >>> > > >> >> >> >
>> >> >>> > > >> >> >> >> Hi,
>> >> >>> > > >> >> >> >>
>> >> >>> > > >> >> >> >> I have been monitoring the system resources such as
>> >> memory
>> >> >>> > Heap
>> >> >>> > > >> usage
>> >> >>> > > >> >> >> for
>> >> >>> > > >> >> >> >> Regionserver, load average, network bandwidth, etc.
>> >> >>> > > >> >> >> >>
>> >> >>> > > >> >> >> >> I experienced a disconnection last night. When I
>> >> checked
>> >> >>> the
>> >> >>> > > >> memory
>> >> >>> > > >> >> Heap
>> >> >>> > > >> >> >> >> for Regionserver, it was 2336 which is way below the
>> >> max
>> >> >>> Heap
>> >> >>> > > >> 4096. I
>> >> >>> > > >> >> >> don't
>> >> >>> > > >> >> >> >> think it is Regionserver problem. The requests that
>> the
>> >> >>> > > >> regionserver
>> >> >>> > > >> >> got
>> >> >>> > > >> >> >> was
>> >> >>> > > >> >> >> >> about 126 which doesn't seem to be high. The load
>> >> average
>> >> >>> is
>> >> >>> > > kind
>> >> >>> > > >> of
>> >> >>> > > >> >> >> high
>> >> >>> > > >> >> >> >> >100. (but I saw disconnections while the load
>> average
>> >> is
>> >> >>> > > around
>> >> >>> > > >> 30).
>> >> >>> > > >> >> >> The
>> >> >>> > > >> >> >> >> bandwidth is not a problem.
>> >> >>> > > >> >> >> >>
>> >> >>> > > >> >> >> >> However, I found error messages from the zookeeper.
>> >> Please
>> >> >>> > see
>> >> >>> > > the
>> >> >>> > > >> >> >> >> following error messages. There are some
>> >> >>> > > >> >> >> >> BufferUnderflowException error, IOException
>> error,etc.
>> >> >>> > > >> >> >> >> May I ask what might be the causes that generate
>> these
>> >> >>> error
>> >> >>> > > >> >> messages?
>> >> >>> > > >> >> >> >>
>> >> >>> > > >> >> >> >> Many thanks
>> >> >>> > > >> >> >> >> zhenyu
>> >> >>> > > >> >> >> >>
>> >> >>> > > >> >> >> >>
>> >> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
>> >> >>> > > org.apache.zookeeper.server.Request:
>> >> >>> > > >> >> >> Ignoring
>> >> >>> > > >> >> >> >> exception during toString
>> >> >>> > > >> >> >> >> java.nio.BufferUnderflowException
>> >> >>> > > >> >> >> >> at
>> >> >>> > java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
>> >> >>> > > >> >> >> >> at
>> java.nio.ByteBuffer.get(ByteBuffer.java:675)
>> >> >>> > > >> >> >> >> at
>> >> >>> > > >> >> >>
>> >> >>> org.apache.zookeeper.server.Request.toString(Request.java:199)
>> >> >>> > > >> >> >> >> at
>> java.lang.String.valueOf(String.java:2827)
>> >> >>> > > >> >> >> >> at
>> >> >>> > > java.lang.StringBuilder.append(StringBuilder.java:115)
>> >> >>> > > >> >> >> >> at
>> >> >>> > > >> >> >> >>
>> >> >>> > > >> >> >>
>> >> >>> > > >> >>
>> >> >>> > > >>
>> >> >>> > >
>> >> >>> >
>> >> >>>
>> >>
>> org.apache.zookeeper.server.quorum.CommitProcessor.processRequest(CommitProcessor.java:167)
>> >> >>> > > >> >> >> >> at
>> >> >>> > > >> >> >> >>
>> >> >>> > > >> >> >>
>> >> >>> > > >> >>
>> >> >>> > > >>
>> >> >>> > >
>> >> >>> >
>> >> >>>
>> >>
>> org.apache.zookeeper.server.quorum.FollowerRequestProcessor.run(FollowerRequestProcessor.java:68)
>> >> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
>> >> >>> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
>> >> >>> > > >> >> >> >> Exception causing close of session 0x424dfd908c50009
>> >> >>> > > >> >> >> >> due to java.io.IOException: Read error
>> >> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
>> >> >>> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
>> >> >>> > > >> >> >> >> IOException stack trace
>> >> >>> > > >> >> >> >> java.io.IOException: Read error
>> >> >>> > > >> >> >> >> at
>> >> >>> > > >> >> >> >>
>> >> >>> > > >> >>
>> >> >>> > >
>> >> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:372)
>> >> >>> > > >> >> >> >> at
>> >> >>> > > >> >> >> >>
>> >> >>> > > >> >> >>
>> >> >>> > > >> >>
>> >> >>> > > >>
>> >> >>> > >
>> >> >>> >
>> >> >>>
>> >>
>> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:221)
>> >> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
>> >> >>> > > >> >> >> >> org.apache.zookeeper.server.quorum.CommitProcessor:
>> >> >>> > Processing
>> >> >>> > > >> >> request::
>> >> >>> > > >> >> >> >> sessionid:0x424dfd908c50009 type:setWatches
>> >> >>> > > >> cxid:0xfffffffffffffff8
>> >> >>> > > >> >> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
>> >> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
>> >> >>> > > org.apache.zookeeper.server.Request:
>> >> >>> > > >> >> >> Ignoring
>> >> >>> > > >> >> >> >> exception during toString
>> >> >>> > > >> >> >> >> java.nio.BufferUnderflowException
>> >> >>> > > >> >> >> >> at
>> >> >>> > java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
>> >> >>> > > >> >> >> >> at
>> java.nio.ByteBuffer.get(ByteBuffer.java:675)
>> >> >>> > > >> >> >> >> at
>> >> >>> > > >> >> >>
>> >> >>> org.apache.zookeeper.server.Request.toString(Request.java:199)
>> >> >>> > > >> >> >> >> at
>> java.lang.String.valueOf(String.java:2827)
>> >> >>> > > >> >> >> >> at
>> >> >>> > > java.lang.StringBuilder.append(StringBuilder.java:115)
>> >> >>> > > >> >> >> >> at
>> >> >>> > > >> >> >> >>
>> >> >>> > > >> >> >>
>> >> >>> > > >> >>
>> >> >>> > > >>
>> >> >>> > >
>> >> >>> >
>> >> >>>
>> >>
>> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:74)
>> >> >>> > > >> >> >> >> at
>> >> >>> > > >> >> >> >>
>> >> >>> > > >> >> >>
>> >> >>> > > >> >>
>> >> >>> > > >>
>> >> >>> > >
>> >> >>> >
>> >> >>>
>> >>
>> org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)
>> >> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
>> >> >>> > > >> >> >> >> org.apache.zookeeper.server.FinalRequestProcessor:
>> >> >>> Processing
>> >> >>> > > >> >> request::
>> >> >>> > > >> >> >> >> sessionid:0x424dfd908c50009 type:setWatches
>> >> >>> > > >> cxid:0xfffffffffffffff8
>> >> >>> > > >> >> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
>> >> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
>> >> >>> > > org.apache.zookeeper.server.Request:
>> >> >>> > > >> >> >> Ignoring
>> >> >>> > > >> >> >> >> exception during toString
>> >> >>> > > >> >> >> >> java.nio.BufferUnderflowException
>> >> >>> > > >> >> >> >> at
>> >> >>> > java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
>> >> >>> > > >> >> >> >> at
>> java.nio.ByteBuffer.get(ByteBuffer.java:675)
>> >> >>> > > >> >> >> >> at
>> >> >>> > > >> >> >>
>> >> >>> org.apache.zookeeper.server.Request.toString(Request.java:199)
>> >> >>> > > >> >> >> >> at
>> >> >>> > > >> >> >> >>
>> >> >>> > > >>
>> >> >>>
>> org.apache.log4j.or.DefaultRenderer.doRender(DefaultRenderer.java:36)
>> >> >>> > > >> >> >> >> at
>> >> >>> > > >> >> >> >>
>> >> >>> > >
>> org.apache.log4j.or.RendererMap.findAndRender(RendererMap.java:80)
>> >> >>> > > >> >> >> >> at
>> >> >>> > > >> >> >> >>
>> >> >>> > > >> >> >>
>> >> >>> > > >> >>
>> >> >>> > > >>
>> >> >>> > >
>> >> >>> >
>> >> >>>
>> >>
>> org.apache.log4j.spi.LoggingEvent.getRenderedMessage(LoggingEvent.java:362)
>> >> >>> > > >> >> >> >> at
>> >> >>> > > >> >> >> >>
>> >> >>> > > >> >> >>
>> >> >>> > > >> >>
>> >> >>> > > >>
>> >> >>> > >
>> >> >>> >
>> >> >>>
>> >>
>> org.apache.log4j.helpers.PatternParser$BasicPatternConverter.convert(PatternParser.java:403)
>> >> >>> > > >> >> >> >> at
>> >> >>> > > >> >> >> >>
>> >> >>> > > >> >> >>
>> >> >>> > > >> >>
>> >> >>> > > >>
>> >> >>> > >
>> >> >>> >
>> >> >>>
>> >>
>> org.apache.log4j.helpers.PatternConverter.format(PatternConverter.java:65)
>> >> >>> > > >> >> >> >> at
>> >> >>> > > >> >>
>> org.apache.log4j.PatternLayout.format(PatternLayout.java:502)
>> >> >>> > > >> >> >> >> at
>> >> >>> > > >> >> >> >>
>> >> >>> > >
>> org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:302)
>> >> >>> > > >> >> >> >> at
>> >> >>> > > >> >> >> >>
>> >> >>> > > >> >> >>
>> >> >>> > > >> >>
>> >> >>> > > >>
>> >> >>> > >
>> >> >>> >
>> >> >>>
>> >>
>> org.apache.log4j.DailyRollingFileAppender.subAppend(DailyRollingFileAppender.java:359)
>> >> >>> > > >> >> >> >> at
>> >> >>> > > >> >> >>
>> >> >>> org.apache.log4j.WriterAppender.append(WriterAppender.java:160)
>> >> >>> > > >> >> >> >> at
>> >> >>> > > >> >> >> >>
>> >> >>> > > >>
>> >> >>>
>> org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
>> >> >>> > > >> >> >> >> at
>> >> >>> > > >> >> >> >>
>> >> >>> > > >> >> >>
>> >> >>> > > >> >>
>> >> >>> > > >>
>> >> >>> > >
>> >> >>> >
>> >> >>>
>> >>
>> org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
>> >> >>> > > >> >> >> >> at
>> >> >>> > > >> org.apache.log4j.Category.callAppenders(Category.java:206)
>> >> >>> > > >> >> >> >> at
>> >> >>> > > org.apache.log4j.Category.forcedLog(Category.java:391)
>> >> >>> > > >> >> >> >> at
>> >> >>> org.apache.log4j.Category.debug(Category.java:260)
>> >> >>> > > >> >> >> >> at
>> >> >>> > > >> >> >> >>
>> >> >>> > > >> >> >>
>> >> >>> > > >> >>
>> >> >>> > > >>
>> >> >>> > >
>> >> >>> >
>> >> >>>
>> >>
>> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:147)
>> >> >>> > > >> >> >> >> at
>> >> >>> > > >> >> >> >>
>> >> >>> > > >> >> >>
>> >> >>> > > >> >>
>> >> >>> > > >>
>> >> >>> > >
>> >> >>> >
>> >> >>>
>> >>
>> org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)
>> >> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
>> >> >>> > > org.apache.zookeeper.server.Request:
>> >> >>> > > >> >> >> Ignoring
>> >> >>> > > >> >> >> >> exception during toString
>> >> >>> > > >> >> >> >> sessionid:0x424dfd908c50009 type:setWatches
>> >> >>> > > >> cxid:0xfffffffffffffff8
>> >> >>> > > >> >> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
>> >> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 INFO
>> >> >>> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
>> >> >>> > > >> >> >> >> closing session:0x424dfd908c50009 NIOServerCnxn:
>> >> >>> > > >> >> >> >> java.nio.channels.SocketChannel[connected local=/
>> >> >>> > > >> >> 192.168.100.131:2181
>> >> >>> > > >> >> >> remote=/
>> >> >>> > > >> >> >> >> 192.168.100.131:35500]
>> >> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
>> >> >>> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
>> >> >>> > > >> >> >> >> ignoring exception during input shutdown
>> >> >>> > > >> >> >> >> java.net.SocketException: Transport endpoint is not
>> >> >>> connected
>> >> >>> > > >> >> >> >> at
>> sun.nio.ch.SocketChannelImpl.shutdown(Native
>> >> >>> > Method)
>> >> >>> > > >> >> >> >> at
>> >> >>> > > >> >> >> >>
>> >> >>> > > >> >>
>> >> >>> > >
>> >> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
>> >> >>> > > >> >> >> >> at
>> >> >>> > > >> >> >>
>> >> >>> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>> >> >>> > > >> >> >> >> at
>> >> >>> > > >> >> >> >>
>> >> >>> > > >> >>
>> >> >>> > >
>> >> >>>
>> org.apache.zookeeper.server.NIOServerCnxn.close(NIOServerCnxn.java:848)
>> >> >>> > > >> >> >> >> at
>> >> >>> > > >> >> >> >>
>> >> >>> > > >> >>
>> >> >>> > >
>> >> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:498)
>> >> >>> > > >> >> >> >> at
>> >> >>> > > >> >> >> >>
>> >> >>> > > >> >> >>
>> >> >>> > > >> >>
>> >> >>> > > >>
>> >> >>> > >
>> >> >>> >
>> >> >>>
>> >>
>> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:221)
>> >> >>> > > >> >> >> >>
>> >> >>> > > >> >> >> >
>> >> >>> > > >> >> >>
>> >> >>> > > >> >> >
>> >> >>> > > >> >>
>> >> >>> > > >> >
>> >> >>> > > >>
>> >> >>> > > >
>> >> >>> > >
>> >> >>> >
>> >> >>>
>> >> >>
>> >> >>
>> >> >
>> >>
>> >
>>
>
Re: regionserver disconnection
Posted by stack <st...@duboce.net>.
On Thu, Nov 12, 2009 at 10:54 AM, Zhenyu Zhong <zh...@gmail.com>wrote:
> Sorry, there was a typo in my email. But it is 10 minutes that I set in the
> hbase-site.xml.
>
> Please see the link.
> http://pastebin.com/m564c51bc
>
> Very much appreciated.
> zhenyu
>
And you restarted after making the change?
St.Ack
>
> On Thu, Nov 12, 2009 at 1:42 PM, Jean-Daniel Cryans <jdcryans@apache.org
> >wrote:
>
> > Usually if it was 10 minutes we would see sleepers complaining they
> > slept for way too long. In the logs I can see a 1 minute pause but not
> > 10 minutes because of the ZK debug. Are you sure you don't have a
> > typo? Like what you just wrote is not valid, you wrote
> > zookeeper.session.timout instead of zookeeper.session.timeout. Are you
> > sure it's distributed over the cluster? Could you pastebin
> > hbase-site.xml?
> >
> > Thx,
> >
> > J-D
> >
> > On Thu, Nov 12, 2009 at 10:28 AM, Zhenyu Zhong <zh...@gmail.com>
> > wrote:
> > > J-D,
> > >
> > > Yes, that is weird. But it is what I have in the log. Probably the log
> is
> > > flushed by different threads, and one thread is blocked for a while?
> > >
> > > I set the zookeeper.session.timout=600000, which is 10 minutes.
> However,
> > we
> > > still see this problem.
> > >
> > > Does GC pause for such a long time period?
> > > I will try different GC options to see the differences.
> > >
> > > thanks
> > > zhenyu
> > >
> > >
> > > On Thu, Nov 12, 2009 at 1:10 PM, Jean-Daniel Cryans <
> jdcryans@apache.org
> > >wrote:
> > >
> > >> I see something weird
> > >>
> > >> 2009-11-12 15:04:36,370 WARN
> > >> org.apache.hadoop.hbase.regionserver.HRegionServer: Attempt=1
> > >> org.apache.hadoop.hbase.Leases$LeaseStillHeldException
> > >> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> > >> Method)
> > >> at
> > >>
> >
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> > >> at
> > >>
> >
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> > >> ...
> > >> 2009-11-12 15:03:52,682 DEBUG org.apache.zookeeper.ClientCnxn: Got
> > >> ping response for sessionid:0x224e55436ad0004 after 0ms
> > >> 2009-11-12 15:04:52,480 WARN org.apache.zookeeper.ClientCnxn:
> > >> Exception closing session 0x24e55436a0007d to
> > >> sun.nio.ch.SelectionKeyImpl@136f6ed1
> > >>
> > >> Goes back in time? I does look better tho. Did you say you set the
> > >> zookeeper session timeout higher in a previous thread? I'd be
> > >> interested to see if a higher (2-3 minutes) value would solve the
> > >> problem until java 7.
> > >>
> > >> The only thing that happens after that LeaseStillHeldException is that
> > >> for some reason (probably GC) the region server paused. This caused
> > >> the session to timeout and should be why the RS isn't connected to the
> > >> ZK quorum member.
> > >>
> > >> BTW we worked with the Zookeeper dev team to have more meaningful
> > >> exceptions and it's already committed in their trunk.
> > >>
> > >> J-D
> > >>
> > >> On Thu, Nov 12, 2009 at 8:40 AM, Zhenyu Zhong <
> zhongresearch@gmail.com>
> > >> wrote:
> > >> > Follow up on the regionserver disconnection:
> > >> >
> > >> > I made a few changes to the cluster setup.
> > >> > 1. split the regionserver and zookeeper quorum servers, such that
> > HRegion
> > >> > doesn't run on the same box as HQuorumPeer. I assume that there
> should
> > be
> > >> no
> > >> > disk io contention between the regionserver and zookeeper.
> > >> > Now we have 14 regionservers, and 5 zookeeper quorum servers.
> > >> >
> > >> > Though I experienced 2 regionserver disconnection this morning, it
> > looks
> > >> > better from the regionserver log. (please see the following log)
> > >> >
> > >> >
> > >> > http://pastebin.com/m496dbfae
> > >> >
> > >> >
> > >> > I checked diskIO for one of the regionserver(192.168.100.116) during
> > the
> > >> > time it disconnected.
> > >> >
> > >> > Time: 03:04:58 AM
> > >> > Device: tps Blk_read/s Blk_wrtn/s Blk_read
> Blk_wrtn
> > >> > sda 105.31 5458.83 19503.64 9043873239
> > 32312473676
> > >> > sda1 2.90 6.64 99.25 10993934
> 164433464
> > >> > sda2 1.72 23.76 30.25 39365817
> 50124008
> > >> > sda3 0.30 0.38 3.58 624930
> 5923000
> > >> > sda4 100.39 5428.06 19370.56 8992888270
> > 32091993204
> > >> >
> > >> >
> > >> > I also checked the zookeeper quorum server that the regionserver
> tried
> > to
> > >> > connect according to the log. However, I don't see 192.168.100.116
> in
> > the
> > >> > client list of the zookeeper quorum server that the regionserver
> tried
> > to
> > >> > connect.
> > >> > Would that be a problem?
> > >> >
> > >> > Thu Nov 12 15:04:52 UTC 2009
> > >> > Zookeeper version: 3.2.1-808558, built on 08/27/2009 18:48 GMT
> > >> > Clients:
> > >> > /192.168.100.127:43045[1](queued=0,recved=26,sent=0)
> > >> > /192.168.100.131:39091[1](queued=0,recved=964,sent=0)
> > >> > /192.168.100.124:35961[1](queued=0,recved=3266,sent=0)
> > >> > /192.168.100.123:47935[1](queued=0,recved=233,sent=0)
> > >> > /192.168.100.125:46931[1](queued=0,recved=2,sent=0)
> > >> > /192.168.100.118:54924[1](queued=0,recved=295,sent=0)
> > >> > /192.168.100.118:41390[1](queued=0,recved=2290,sent=0)
> > >> > /192.168.100.136:42243[1](queued=0,recved=0,sent=0)
> > >> >
> > >> > Latency min/avg/max: 0/17/6333
> > >> > Received: 47111
> > >> > Sent: 0
> > >> > Outstanding: 0
> > >> > Zxid: 0x77000083f4
> > >> > Mode: leader
> > >> > Node count: 23
> > >> >
> > >> > Best,
> > >> > zhenyu
> > >> >
> > >> >
> > >> > On Wed, Nov 11, 2009 at 3:58 PM, Zhenyu Zhong <
> > zhongresearch@gmail.com
> > >> >wrote:
> > >> >
> > >> >> Stack
> > >> >>
> > >> >> I am very appreciated for your comments.
> > >> >> I will use the zookeeper monitoring script on my cluster and let it
> > run
> > >> >> overnight to see the result.
> > >> >> I will follow up the thread when I get anything.
> > >> >>
> > >> >> thanks
> > >> >> zhenyu
> > >> >>
> > >> >>
> > >> >> On Wed, Nov 11, 2009 at 3:52 PM, stack <st...@duboce.net> wrote:
> > >> >>
> > >> >>> I see these in your log too:
> > >> >>>
> > >> >>>
> > >> >>> 1. 2009-11-11 04:27:19,018 DEBUG
> org.apache.zookeeper.ClientCnxn:
> > Got
> > >> >>> ping response for sessionid:0x424dfd908c50009 after 4544ms
> > >> >>> 2. 2009-11-11 04:27:19,018 DEBUG
> org.apache.zookeeper.ClientCnxn:
> > Got
> > >> >>> ping response for sessionid:0x24dfd90c810002 after 4548ms
> > >> >>> 3. 2009-11-11 04:27:43,960 DEBUG
> org.apache.zookeeper.ClientCnxn:
> > Got
> > >> >>> ping response for sessionid:0x424dfd908c50009 after 9030ms
> > >> >>> 4. 2009-11-11 04:27:43,960 DEBUG
> org.apache.zookeeper.ClientCnxn:
> > Got
> > >> >>> ping response for sessionid:0x24dfd90c810002 after 9030ms
> > >> >>> 5. 2009-11-11 04:27:48,830 DEBUG
> org.apache.zookeeper.ClientCnxn:
> > Got
> > >> >>> ping response for sessionid:0x424dfd908c50009 after 0ms
> > >> >>> 6. 2009-11-11 04:27:48,830 DEBUG
> org.apache.zookeeper.ClientCnxn:
> > Got
> > >> >>> ping response for sessionid:0x24dfd90c810002 after 0ms
> > >> >>>
> > >> >>> Which would indicate lag talking to zk too.
> > >> >>>
> > >> >>> Appending HLog also seems to get steadily worse.
> > >> >>>
> > >> >>> I also see these:
> > >> >>>
> > >> >>>
> > >> >>> 1. 2009-11-11 04:27:01,233 WARN org.apache.zookeeper.ClientCnxn:
> > >> >>> Exception closing session 0x24dfd90c810002 to
> > >> >>> sun.nio.ch.SelectionKeyImpl@5e16a7e3
> > >> >>> 2. java.io.IOException: Read error rc = -1
> > >> >>> java.nio.DirectByteBuffer[pos=0 lim=4 cap=4]
> > >> >>> 3. at
> > >> >>>
> > org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
> > >> >>> 4. at
> > >> >>>
> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
> > >> >>>
> > >> >>>
> > >> >>> Were all 5 quorum members participating? Check by doing "echo
> > stat|nc
> > >> >>> QUORUM_HOST port" for each member of quorum.... the below might
> > help:
> > >> >>>
> > >> >>> members="1,2,3,4,5"
> > >> >>> while true; do date; for i in $members; do echo $i; echo stat | nc
> > $i
> > >> >>> 2181;
> > >> >>> done; sleep 10; done
> > >> >>>
> > >> >>> We need to improve our zk_dump. It just reports quorum members...
> > not
> > >> if
> > >> >>> they are actually participating or not. The above will tell you
> if
> > >> dead
> > >> >>> or
> > >> >>> alive.
> > >> >>>
> > >> >>> Please report back after checking above and following the
> > Jean-Daniel
> > >> >>> prescription.
> > >> >>>
> > >> >>> Thanks,
> > >> >>> St.Ack
> > >> >>>
> > >> >>>
> > >> >>>
> > >> >>> On Wed, Nov 11, 2009 at 12:29 PM, Zhenyu Zhong <
> > >> zhongresearch@gmail.com
> > >> >>> >wrote:
> > >> >>>
> > >> >>> > Thanks a lot for the help.
> > >> >>> >
> > >> >>> > I understand that the change of disk configurations should be
> done
> > >> >>> > progressively such that HDFS can balance and recover from a disk
> > >> going
> > >> >>> > away.
> > >> >>> >
> > >> >>> > The data we are trying to import into HBase is about 200million
> > rows,
> > >> >>> each
> > >> >>> > row is about hundreds of bytes. Basically, we read in a text
> file
> > >> from
> > >> >>> > HDFS,
> > >> >>> > then for each row in the text file, we fetch data from HBase
> > table,
> > >> we
> > >> >>> > compare the column value and update the latest value back into
> > HBase.
> > >> I
> > >> >>> was
> > >> >>> > suggested to use CheckAndPut. Anyway this is the common usecase
> > on
> > >> my
> > >> >>> > side.
> > >> >>> >
> > >> >>> >
> > >> >>> > Many thanks
> > >> >>> > zhenyu
> > >> >>> >
> > >> >>> >
> > >> >>> >
> > >> >>> > On Wed, Nov 11, 2009 at 3:03 PM, Jean-Daniel Cryans <
> > >> >>> jdcryans@apache.org
> > >> >>> > >wrote:
> > >> >>> >
> > >> >>> > > 6 disks, 19 machines and you see those kind of issues? I'm
> quite
> > >> >>> > > surprised and I want to learn more about what you trying to
> to.
> > >> >>> > >
> > >> >>> > > But first, if hbase is managing ZK for you then you can change
> > the
> > >> >>> > > value of hbase.zookeeper.property.dataDir so that it points to
> > its
> > >> own
> > >> >>> > > mount. Make sure you also change dfs.data.dir in
> > conf/hdfs-site.xml
> > >> >>> > > (hadoop stuff) so that it stops pointing there (you should
> have
> > 6
> > >> >>> > > values there at the moment if you configured it correctly).
> > Doing
> > >> this
> > >> >>> > > could result in data loss if you do it all at once so do the
> > change
> > >> >>> > > progressively. This also means that you may have to keep a
> > seperate
> > >> >>> > > HDFS configuration for those particular nodes. Please ask more
> > >> >>> > > questions if it's not very clear.
> > >> >>> > >
> > >> >>> > > WRT to dedicating nodes to Zookeeeper, it makes sense only if
> > you
> > >> have
> > >> >>> > > a lot of nodes but in your case, unless there's also CPU
> > >> contention,
> > >> >>> > > you're good with only a dedicated disk.
> > >> >>> > >
> > >> >>> > > Now for what you are trying to do, is this a massive import
> that
> > >> goes
> > >> >>> > > over many hours? How big is each row? I'd love to get a lot of
> > >> details
> > >> >>> > > to help you optimize your cluster for your particular problem.
> > >> >>> > >
> > >> >>> > > Thx,
> > >> >>> > >
> > >> >>> > > J-D
> > >> >>> > >
> > >> >>> > > On Wed, Nov 11, 2009 at 11:51 AM, Zhenyu Zhong <
> > >> >>> zhongresearch@gmail.com>
> > >> >>> > > wrote:
> > >> >>> > > > J-D,
> > >> >>> > > >
> > >> >>> > > > Thanks a lot for the insightful comments.
> > >> >>> > > > We are running 5 zookeeper quorums now. On these 5 quorum
> > >> servers,
> > >> >>> we
> > >> >>> > > also
> > >> >>> > > > run datanode, tasktracker and regionserver.
> > >> >>> > > > Now we have 6 disks per machine.
> > >> >>> > > > If it is the disk IO saturation, is there anyway to
> configure
> > >> >>> zookeeper
> > >> >>> > > to
> > >> >>> > > > write to a seperate disk which is not the same one used by
> > HDFS?
> > >> >>> > > > Or shall we run zookeeper quorums on different nodes other
> > than
> > >> >>> HDFS?
> > >> >>> > > >
> > >> >>> > > > Are HLogs generated by Regionservers? Would it alleviate the
> > >> issue
> > >> >>> if
> > >> >>> > we
> > >> >>> > > use
> > >> >>> > > > the first 14 for regionserver, and the rest of 5 for
> > >> zookeeper?(In
> > >> >>> that
> > >> >>> > > > sense, the node running zookeeper quorum won't run
> > regionserver)
> > >> >>> > > >
> > >> >>> > > > I am very appreciated for your comments.
> > >> >>> > > >
> > >> >>> > > > Best,
> > >> >>> > > > zhenyu
> > >> >>> > > >
> > >> >>> > > >
> > >> >>> > > >
> > >> >>> > > > On Wed, Nov 11, 2009 at 2:30 PM, Jean-Daniel Cryans <
> > >> >>> > jdcryans@apache.org
> > >> >>> > > >wrote:
> > >> >>> > > >
> > >> >>> > > >> The problem seems to be that the IO is saturated. When you
> > see
> > >> >>> this:
> > >> >>> > > >>
> > >> >>> > > >> #
> > >> >>> > > >> 2009-11-11 04:28:10,308 WARN
> > >> >>> > > >> org.apache.hadoop.hbase.regionserver.HLog: IPC Server
> handler
> > 9
> > >> on
> > >> >>> > > >> 60021 took 6819ms appending an edit to hlog; editcount=370
> > >> >>> > > >> #
> > >> >>> > > >> 2009-11-11 04:28:10,308 WARN
> > >> >>> > > >> org.apache.hadoop.hbase.regionserver.HLog: IPC Server
> handler
> > 2
> > >> on
> > >> >>> > > >> 60021 took 4412ms appending an edit to hlog; editcount=371
> > >> >>> > > >>
> > >> >>> > > >> It's a sign that HDFS is slow. I guess that the zookeeper
> > quorum
> > >> >>> > > >> members are on those same nodes using the same disks? If
> so,
> > >> >>> there's
> > >> >>> > > >> the problem because Zookeeper needs to be able to commit
> > >> >>> transactions
> > >> >>> > > >> very fast and if your disks are already hogged then they
> have
> > to
> > >> >>> wait
> > >> >>> > > >> and you can get disconnections. Do you still have 19 quorum
> > >> >>> members?
> > >> >>> > > >> How many disks do you have per machine?
> > >> >>> > > >>
> > >> >>> > > >> Thx
> > >> >>> > > >>
> > >> >>> > > >> J-D
> > >> >>> > > >>
> > >> >>> > > >> On Wed, Nov 11, 2009 at 11:22 AM, Zhenyu Zhong <
> > >> >>> > zhongresearch@gmail.com
> > >> >>> > > >
> > >> >>> > > >> wrote:
> > >> >>> > > >> > Please take a look at the following regionserver log.
> > >> >>> > > >> > The error started to come out from 4:24am UTC to 4:29am
> > UTC.
> > >> >>> > > >> >
> > >> >>> > > >> > http://pastebin.com/m3717d064
> > >> >>> > > >> >
> > >> >>> > > >> > Thanks in advance.
> > >> >>> > > >> >
> > >> >>> > > >> > Best,
> > >> >>> > > >> > zhenyu
> > >> >>> > > >> >
> > >> >>> > > >> >
> > >> >>> > > >> > On Wed, Nov 11, 2009 at 2:01 PM, Jean-Daniel Cryans <
> > >> >>> > > jdcryans@apache.org
> > >> >>> > > >> >wrote:
> > >> >>> > > >> >
> > >> >>> > > >> >> Could you paste a bigger log trace (interested to see at
> > >> least
> > >> >>> all
> > >> >>> > > 300
> > >> >>> > > >> >> lines before that) in a pastebin?
> > >> >>> > > >> >>
> > >> >>> > > >> >> Thx,
> > >> >>> > > >> >>
> > >> >>> > > >> >> J-D
> > >> >>> > > >> >>
> > >> >>> > > >> >> On Wed, Nov 11, 2009 at 10:56 AM, Zhenyu Zhong <
> > >> >>> > > zhongresearch@gmail.com
> > >> >>> > > >> >
> > >> >>> > > >> >> wrote:
> > >> >>> > > >> >> > J-D,
> > >> >>> > > >> >> >
> > >> >>> > > >> >> > Thanks for the reply.
> > >> >>> > > >> >> >
> > >> >>> > > >> >> > The problem I have is one or more of the regionservers
> > >> >>> disconnect
> > >> >>> > > from
> > >> >>> > > >> >> the
> > >> >>> > > >> >> > cluster.
> > >> >>> > > >> >> >
> > >> >>> > > >> >> > I checked the resource usage including memory Heap
> usage
> > >> for
> > >> >>> > > >> >> regionserver,
> > >> >>> > > >> >> > and found nothing suspicious. The regionserver heap
> > usage
> > >> is
> > >> >>> way
> > >> >>> > > below
> > >> >>> > > >> >> the
> > >> >>> > > >> >> > maxHeap defined in jvm.
> > >> >>> > > >> >> >
> > >> >>> > > >> >> > So I checked zookeeper log, then I found the following
> > >> >>> > > >> >> > BufferUnderflowException, eventually it leads to
> > >> >>> SocketException:
> > >> >>> > > >> >> Transport
> > >> >>> > > >> >> > endpoint is not connected.
> > >> >>> > > >> >> >
> > >> >>> > > >> >> >
> > >> >>> > > >> >> > I also checked the regionserver log. It looks like the
> > >> >>> > regionserver
> > >> >>> > > >> tried
> > >> >>> > > >> >> to
> > >> >>> > > >> >> > connect to other zookeeper quorum server. It connects
> > >> >>> > successfully,
> > >> >>> > > >> but
> > >> >>> > > >> >> then
> > >> >>> > > >> >> > it complained about SocketException: transport
> endpoint
> > is
> > >> not
> > >> >>> > > >> connected.
> > >> >>> > > >> >> >
> > >> >>> > > >> >> > So both zookeeper log and regionserver log told me
> that
> > the
> > >> >>> > > transport
> > >> >>> > > >> >> > endpoint is not connected. I don't know why.
> > >> >>> > > >> >> >
> > >> >>> > > >> >> > I would be very appreciated if you could point me to
> the
> > >> right
> > >> >>> > > >> direction.
> > >> >>> > > >> >> >
> > >> >>> > > >> >> > Best,
> > >> >>> > > >> >> >
> > >> >>> > > >> >> >
> > >> >>> > > >> >> > Regionserver log:
> > >> >>> > > >> >> > 2009-11-11 04:24:53,718 INFO
> > >> org.apache.zookeeper.ClientCnxn:
> > >> >>> > > >> Attempting
> > >> >>> > > >> >> > connection to server superpyxis0018/
> > 192.168.100.132:2181
> > >> >>> > > >> >> > 2009-11-11 04:24:53,718 INFO
> > >> org.apache.zookeeper.ClientCnxn:
> > >> >>> > > Priming
> > >> >>> > > >> >> > connection to
> java.nio.channels.SocketChannel[connected
> > >> >>> local=/
> > >> >>> > > >> >> > 192.168.100.131:43065 remote=superpyxis0018/
> > >> >>> 192.168.100.132:2181
> > >> >>> > ]
> > >> >>> > > >> >> > 2009-11-11 04:24:53,718 INFO
> > >> org.apache.zookeeper.ClientCnxn:
> > >> >>> > > Server
> > >> >>> > > >> >> > connection successful
> > >> >>> > > >> >> > 2009-11-11 04:24:53,768 INFO
> > >> >>> > > >> >> > org.apache.hadoop.hbase.regionserver.HRegionServer:
> Got
> > >> >>> ZooKeeper
> > >> >>> > > >> event,
> > >> >>> > > >> >> > state: Disconnected, type: None, path: null
> > >> >>> > > >> >> > 2009-11-11 04:24:54,118 INFO
> > >> org.apache.zookeeper.ClientCnxn:
> > >> >>> > > >> Attempting
> > >> >>> > > >> >> > connection to server superpyxis0017/
> > 192.168.100.131:2181
> > >> >>> > > >> >> > 2009-11-11 04:24:54,118 INFO
> > >> org.apache.zookeeper.ClientCnxn:
> > >> >>> > > Priming
> > >> >>> > > >> >> > connection to
> java.nio.channels.SocketChannel[connected
> > >> >>> local=/
> > >> >>> > > >> >> > 192.168.100.131:46903 remote=superpyxis0017/
> > >> >>> 192.168.100.131:2181
> > >> >>> > ]
> > >> >>> > > >> >> > 2009-11-11 04:24:54,118 INFO
> > >> org.apache.zookeeper.ClientCnxn:
> > >> >>> > > Server
> > >> >>> > > >> >> > connection successful
> > >> >>> > > >> >> > 2009-11-11 04:24:54,118 WARN
> > >> org.apache.zookeeper.ClientCnxn:
> > >> >>> > > >> Exception
> > >> >>> > > >> >> > closing session 0x24dfd90c810002 to
> > >> >>> > > >> sun.nio.ch.SelectionKeyImpl@2dd78ccf
> > >> >>> > > >> >> > java.io.IOException: Read error rc = -1
> > >> >>> > > >> java.nio.DirectByteBuffer[pos=0
> > >> >>> > > >> >> > lim=4 cap=4]
> > >> >>> > > >> >> > at
> > >> >>> > > >> >> >
> > >> >>> > >
> > >> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
> > >> >>> > > >> >> > at
> > >> >>> > > >> >> >
> > >> >>> >
> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
> > >> >>> > > >> >> > 2009-11-11 04:24:54,118 WARN
> > >> org.apache.zookeeper.ClientCnxn:
> > >> >>> > > Ignoring
> > >> >>> > > >> >> > exception during shutdown input
> > >> >>> > > >> >> > java.net.SocketException: Transport endpoint is not
> > >> connected
> > >> >>> > > >> >> > at sun.nio.ch.SocketChannelImpl.shutdown(Native
> > >> Method)
> > >> >>> > > >> >> > at
> > >> >>> > > >> >> >
> > >> >>> > >
> > >> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
> > >> >>> > > >> >> > at
> > >> >>> > > >>
> > sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> > >> >>> > > >> >> > at
> > >> >>> > > >> >> >
> > >> >>> > > >>
> > >> >>> >
> > >>
> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> > >> >>> > > >> >> > at
> > >> >>> > > >> >> >
> > >> >>> >
> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> > >> >>> > > >> >> >
> > >> >>> > > >> >> >
> > >> >>> > > >> >> >
> > >> >>> > > >> >> >
> > >> >>> > > >> >> >
> > >> >>> > > >> >> > On Wed, Nov 11, 2009 at 1:40 PM, Jean-Daniel Cryans <
> > >> >>> > > >> jdcryans@apache.org
> > >> >>> > > >> >> >wrote:
> > >> >>> > > >> >> >
> > >> >>> > > >> >> >> I'm really not sure exactly what the situation is
> with
> > the
> > >> >>> > > >> >> >> java.nio.BufferUnderflowException, but what do you
> mean
> > by
> > >> >>> > > >> >> >> "regionserver disconnection"? Could you give more
> > details?
> > >> >>> > > >> >> >>
> > >> >>> > > >> >> >> Thx,
> > >> >>> > > >> >> >>
> > >> >>> > > >> >> >> J-D
> > >> >>> > > >> >> >>
> > >> >>> > > >> >> >> On Wed, Nov 11, 2009 at 10:34 AM, Zhenyu Zhong <
> > >> >>> > > >> zhongresearch@gmail.com
> > >> >>> > > >> >> >
> > >> >>> > > >> >> >> wrote:
> > >> >>> > > >> >> >> > I searched around and found HBASE-1645 "HBase could
> > be a
> > >> >>> > little
> > >> >>> > > >> more
> > >> >>> > > >> >> >> dainty
> > >> >>> > > >> >> >> > when zk is pulled out from under it"
> > >> >>> > > >> >> >> >
> > >> >>> > > >> >> >> > I am wondering whether my problem is related to
> this
> > >> >>> > HBase-1645.
> > >> >>> > > Or
> > >> >>> > > >> >> >> probably
> > >> >>> > > >> >> >> > has nothing to do with it?
> > >> >>> > > >> >> >> >
> > >> >>> > > >> >> >> > thanks
> > >> >>> > > >> >> >> >
> > >> >>> > > >> >> >> >
> > >> >>> > > >> >> >> > On Wed, Nov 11, 2009 at 12:02 PM, Zhenyu Zhong <
> > >> >>> > > >> >> zhongresearch@gmail.com
> > >> >>> > > >> >> >> >wrote:
> > >> >>> > > >> >> >> >
> > >> >>> > > >> >> >> >> Hi,
> > >> >>> > > >> >> >> >>
> > >> >>> > > >> >> >> >> I have been monitoring the system resources such
> as
> > >> memory
> > >> >>> > Heap
> > >> >>> > > >> usage
> > >> >>> > > >> >> >> for
> > >> >>> > > >> >> >> >> Regionserver, load average, network bandwidth,
> etc.
> > >> >>> > > >> >> >> >>
> > >> >>> > > >> >> >> >> I experienced a disconnection last night. When I
> > >> checked
> > >> >>> the
> > >> >>> > > >> memory
> > >> >>> > > >> >> Heap
> > >> >>> > > >> >> >> >> for Regionserver, it was 2336 which is way below
> the
> > >> max
> > >> >>> Heap
> > >> >>> > > >> 4096. I
> > >> >>> > > >> >> >> don't
> > >> >>> > > >> >> >> >> think it is Regionserver problem. The requests
> that
> > the
> > >> >>> > > >> regionserver
> > >> >>> > > >> >> got
> > >> >>> > > >> >> >> was
> > >> >>> > > >> >> >> >> about 126 which doesn't seem to be high. The load
> > >> average
> > >> >>> is
> > >> >>> > > kind
> > >> >>> > > >> of
> > >> >>> > > >> >> >> high
> > >> >>> > > >> >> >> >> >100. (but I saw disconnections while the load
> > average
> > >> is
> > >> >>> > > around
> > >> >>> > > >> 30).
> > >> >>> > > >> >> >> The
> > >> >>> > > >> >> >> >> bandwidth is not a problem.
> > >> >>> > > >> >> >> >>
> > >> >>> > > >> >> >> >> However, I found error messages from the
> zookeeper.
> > >> Please
> > >> >>> > see
> > >> >>> > > the
> > >> >>> > > >> >> >> >> following error messages. There are some
> > >> >>> > > >> >> >> >> BufferUnderflowException error, IOException
> > error,etc.
> > >> >>> > > >> >> >> >> May I ask what might be the causes that generate
> > these
> > >> >>> error
> > >> >>> > > >> >> messages?
> > >> >>> > > >> >> >> >>
> > >> >>> > > >> >> >> >> Many thanks
> > >> >>> > > >> >> >> >> zhenyu
> > >> >>> > > >> >> >> >>
> > >> >>> > > >> >> >> >>
> > >> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
> > >> >>> > > org.apache.zookeeper.server.Request:
> > >> >>> > > >> >> >> Ignoring
> > >> >>> > > >> >> >> >> exception during toString
> > >> >>> > > >> >> >> >> java.nio.BufferUnderflowException
> > >> >>> > > >> >> >> >> at
> > >> >>> > java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
> > >> >>> > > >> >> >> >> at
> > java.nio.ByteBuffer.get(ByteBuffer.java:675)
> > >> >>> > > >> >> >> >> at
> > >> >>> > > >> >> >>
> > >> >>> org.apache.zookeeper.server.Request.toString(Request.java:199)
> > >> >>> > > >> >> >> >> at
> > java.lang.String.valueOf(String.java:2827)
> > >> >>> > > >> >> >> >> at
> > >> >>> > > java.lang.StringBuilder.append(StringBuilder.java:115)
> > >> >>> > > >> >> >> >> at
> > >> >>> > > >> >> >> >>
> > >> >>> > > >> >> >>
> > >> >>> > > >> >>
> > >> >>> > > >>
> > >> >>> > >
> > >> >>> >
> > >> >>>
> > >>
> >
> org.apache.zookeeper.server.quorum.CommitProcessor.processRequest(CommitProcessor.java:167)
> > >> >>> > > >> >> >> >> at
> > >> >>> > > >> >> >> >>
> > >> >>> > > >> >> >>
> > >> >>> > > >> >>
> > >> >>> > > >>
> > >> >>> > >
> > >> >>> >
> > >> >>>
> > >>
> >
> org.apache.zookeeper.server.quorum.FollowerRequestProcessor.run(FollowerRequestProcessor.java:68)
> > >> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
> > >> >>> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
> > >> >>> > > >> >> >> >> Exception causing close of session
> 0x424dfd908c50009
> > >> >>> > > >> >> >> >> due to java.io.IOException: Read error
> > >> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
> > >> >>> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
> > >> >>> > > >> >> >> >> IOException stack trace
> > >> >>> > > >> >> >> >> java.io.IOException: Read error
> > >> >>> > > >> >> >> >> at
> > >> >>> > > >> >> >> >>
> > >> >>> > > >> >>
> > >> >>> > >
> > >> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:372)
> > >> >>> > > >> >> >> >> at
> > >> >>> > > >> >> >> >>
> > >> >>> > > >> >> >>
> > >> >>> > > >> >>
> > >> >>> > > >>
> > >> >>> > >
> > >> >>> >
> > >> >>>
> > >>
> >
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:221)
> > >> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
> > >> >>> > > >> >> >> >>
> org.apache.zookeeper.server.quorum.CommitProcessor:
> > >> >>> > Processing
> > >> >>> > > >> >> request::
> > >> >>> > > >> >> >> >> sessionid:0x424dfd908c50009 type:setWatches
> > >> >>> > > >> cxid:0xfffffffffffffff8
> > >> >>> > > >> >> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
> > >> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
> > >> >>> > > org.apache.zookeeper.server.Request:
> > >> >>> > > >> >> >> Ignoring
> > >> >>> > > >> >> >> >> exception during toString
> > >> >>> > > >> >> >> >> java.nio.BufferUnderflowException
> > >> >>> > > >> >> >> >> at
> > >> >>> > java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
> > >> >>> > > >> >> >> >> at
> > java.nio.ByteBuffer.get(ByteBuffer.java:675)
> > >> >>> > > >> >> >> >> at
> > >> >>> > > >> >> >>
> > >> >>> org.apache.zookeeper.server.Request.toString(Request.java:199)
> > >> >>> > > >> >> >> >> at
> > java.lang.String.valueOf(String.java:2827)
> > >> >>> > > >> >> >> >> at
> > >> >>> > > java.lang.StringBuilder.append(StringBuilder.java:115)
> > >> >>> > > >> >> >> >> at
> > >> >>> > > >> >> >> >>
> > >> >>> > > >> >> >>
> > >> >>> > > >> >>
> > >> >>> > > >>
> > >> >>> > >
> > >> >>> >
> > >> >>>
> > >>
> >
> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:74)
> > >> >>> > > >> >> >> >> at
> > >> >>> > > >> >> >> >>
> > >> >>> > > >> >> >>
> > >> >>> > > >> >>
> > >> >>> > > >>
> > >> >>> > >
> > >> >>> >
> > >> >>>
> > >>
> >
> org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)
> > >> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
> > >> >>> > > >> >> >> >> org.apache.zookeeper.server.FinalRequestProcessor:
> > >> >>> Processing
> > >> >>> > > >> >> request::
> > >> >>> > > >> >> >> >> sessionid:0x424dfd908c50009 type:setWatches
> > >> >>> > > >> cxid:0xfffffffffffffff8
> > >> >>> > > >> >> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
> > >> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
> > >> >>> > > org.apache.zookeeper.server.Request:
> > >> >>> > > >> >> >> Ignoring
> > >> >>> > > >> >> >> >> exception during toString
> > >> >>> > > >> >> >> >> java.nio.BufferUnderflowException
> > >> >>> > > >> >> >> >> at
> > >> >>> > java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
> > >> >>> > > >> >> >> >> at
> > java.nio.ByteBuffer.get(ByteBuffer.java:675)
> > >> >>> > > >> >> >> >> at
> > >> >>> > > >> >> >>
> > >> >>> org.apache.zookeeper.server.Request.toString(Request.java:199)
> > >> >>> > > >> >> >> >> at
> > >> >>> > > >> >> >> >>
> > >> >>> > > >>
> > >> >>>
> > org.apache.log4j.or.DefaultRenderer.doRender(DefaultRenderer.java:36)
> > >> >>> > > >> >> >> >> at
> > >> >>> > > >> >> >> >>
> > >> >>> > >
> > org.apache.log4j.or.RendererMap.findAndRender(RendererMap.java:80)
> > >> >>> > > >> >> >> >> at
> > >> >>> > > >> >> >> >>
> > >> >>> > > >> >> >>
> > >> >>> > > >> >>
> > >> >>> > > >>
> > >> >>> > >
> > >> >>> >
> > >> >>>
> > >>
> >
> org.apache.log4j.spi.LoggingEvent.getRenderedMessage(LoggingEvent.java:362)
> > >> >>> > > >> >> >> >> at
> > >> >>> > > >> >> >> >>
> > >> >>> > > >> >> >>
> > >> >>> > > >> >>
> > >> >>> > > >>
> > >> >>> > >
> > >> >>> >
> > >> >>>
> > >>
> >
> org.apache.log4j.helpers.PatternParser$BasicPatternConverter.convert(PatternParser.java:403)
> > >> >>> > > >> >> >> >> at
> > >> >>> > > >> >> >> >>
> > >> >>> > > >> >> >>
> > >> >>> > > >> >>
> > >> >>> > > >>
> > >> >>> > >
> > >> >>> >
> > >> >>>
> > >>
> >
> org.apache.log4j.helpers.PatternConverter.format(PatternConverter.java:65)
> > >> >>> > > >> >> >> >> at
> > >> >>> > > >> >>
> > org.apache.log4j.PatternLayout.format(PatternLayout.java:502)
> > >> >>> > > >> >> >> >> at
> > >> >>> > > >> >> >> >>
> > >> >>> > >
> > org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:302)
> > >> >>> > > >> >> >> >> at
> > >> >>> > > >> >> >> >>
> > >> >>> > > >> >> >>
> > >> >>> > > >> >>
> > >> >>> > > >>
> > >> >>> > >
> > >> >>> >
> > >> >>>
> > >>
> >
> org.apache.log4j.DailyRollingFileAppender.subAppend(DailyRollingFileAppender.java:359)
> > >> >>> > > >> >> >> >> at
> > >> >>> > > >> >> >>
> > >> >>> org.apache.log4j.WriterAppender.append(WriterAppender.java:160)
> > >> >>> > > >> >> >> >> at
> > >> >>> > > >> >> >> >>
> > >> >>> > > >>
> > >> >>>
> > org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
> > >> >>> > > >> >> >> >> at
> > >> >>> > > >> >> >> >>
> > >> >>> > > >> >> >>
> > >> >>> > > >> >>
> > >> >>> > > >>
> > >> >>> > >
> > >> >>> >
> > >> >>>
> > >>
> >
> org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
> > >> >>> > > >> >> >> >> at
> > >> >>> > > >> org.apache.log4j.Category.callAppenders(Category.java:206)
> > >> >>> > > >> >> >> >> at
> > >> >>> > > org.apache.log4j.Category.forcedLog(Category.java:391)
> > >> >>> > > >> >> >> >> at
> > >> >>> org.apache.log4j.Category.debug(Category.java:260)
> > >> >>> > > >> >> >> >> at
> > >> >>> > > >> >> >> >>
> > >> >>> > > >> >> >>
> > >> >>> > > >> >>
> > >> >>> > > >>
> > >> >>> > >
> > >> >>> >
> > >> >>>
> > >>
> >
> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:147)
> > >> >>> > > >> >> >> >> at
> > >> >>> > > >> >> >> >>
> > >> >>> > > >> >> >>
> > >> >>> > > >> >>
> > >> >>> > > >>
> > >> >>> > >
> > >> >>> >
> > >> >>>
> > >>
> >
> org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)
> > >> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
> > >> >>> > > org.apache.zookeeper.server.Request:
> > >> >>> > > >> >> >> Ignoring
> > >> >>> > > >> >> >> >> exception during toString
> > >> >>> > > >> >> >> >> sessionid:0x424dfd908c50009 type:setWatches
> > >> >>> > > >> cxid:0xfffffffffffffff8
> > >> >>> > > >> >> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
> > >> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 INFO
> > >> >>> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
> > >> >>> > > >> >> >> >> closing session:0x424dfd908c50009 NIOServerCnxn:
> > >> >>> > > >> >> >> >> java.nio.channels.SocketChannel[connected local=/
> > >> >>> > > >> >> 192.168.100.131:2181
> > >> >>> > > >> >> >> remote=/
> > >> >>> > > >> >> >> >> 192.168.100.131:35500]
> > >> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
> > >> >>> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
> > >> >>> > > >> >> >> >> ignoring exception during input shutdown
> > >> >>> > > >> >> >> >> java.net.SocketException: Transport endpoint is
> not
> > >> >>> connected
> > >> >>> > > >> >> >> >> at
> > sun.nio.ch.SocketChannelImpl.shutdown(Native
> > >> >>> > Method)
> > >> >>> > > >> >> >> >> at
> > >> >>> > > >> >> >> >>
> > >> >>> > > >> >>
> > >> >>> > >
> > >> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
> > >> >>> > > >> >> >> >> at
> > >> >>> > > >> >> >>
> > >> >>> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> > >> >>> > > >> >> >> >> at
> > >> >>> > > >> >> >> >>
> > >> >>> > > >> >>
> > >> >>> > >
> > >> >>>
> > org.apache.zookeeper.server.NIOServerCnxn.close(NIOServerCnxn.java:848)
> > >> >>> > > >> >> >> >> at
> > >> >>> > > >> >> >> >>
> > >> >>> > > >> >>
> > >> >>> > >
> > >> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:498)
> > >> >>> > > >> >> >> >> at
> > >> >>> > > >> >> >> >>
> > >> >>> > > >> >> >>
> > >> >>> > > >> >>
> > >> >>> > > >>
> > >> >>> > >
> > >> >>> >
> > >> >>>
> > >>
> >
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:221)
> > >> >>> > > >> >> >> >>
> > >> >>> > > >> >> >> >
> > >> >>> > > >> >> >>
> > >> >>> > > >> >> >
> > >> >>> > > >> >>
> > >> >>> > > >> >
> > >> >>> > > >>
> > >> >>> > > >
> > >> >>> > >
> > >> >>> >
> > >> >>>
> > >> >>
> > >> >>
> > >> >
> > >>
> > >
> >
>
Re: regionserver disconnection
Posted by Zhenyu Zhong <zh...@gmail.com>.
Sorry, there was a typo in my email. But it is 10 minutes that I set in the
hbase-site.xml.
Please see the link.
http://pastebin.com/m564c51bc
Very much appreciated.
zhenyu
On Thu, Nov 12, 2009 at 1:42 PM, Jean-Daniel Cryans <jd...@apache.org>wrote:
> Usually if it was 10 minutes we would see sleepers complaining they
> slept for way too long. In the logs I can see a 1 minute pause but not
> 10 minutes because of the ZK debug. Are you sure you don't have a
> typo? Like what you just wrote is not valid, you wrote
> zookeeper.session.timout instead of zookeeper.session.timeout. Are you
> sure it's distributed over the cluster? Could you pastebin
> hbase-site.xml?
>
> Thx,
>
> J-D
>
> On Thu, Nov 12, 2009 at 10:28 AM, Zhenyu Zhong <zh...@gmail.com>
> wrote:
> > J-D,
> >
> > Yes, that is weird. But it is what I have in the log. Probably the log is
> > flushed by different threads, and one thread is blocked for a while?
> >
> > I set the zookeeper.session.timout=600000, which is 10 minutes. However,
> we
> > still see this problem.
> >
> > Does GC pause for such a long time period?
> > I will try different GC options to see the differences.
> >
> > thanks
> > zhenyu
> >
> >
> > On Thu, Nov 12, 2009 at 1:10 PM, Jean-Daniel Cryans <jdcryans@apache.org
> >wrote:
> >
> >> I see something weird
> >>
> >> 2009-11-12 15:04:36,370 WARN
> >> org.apache.hadoop.hbase.regionserver.HRegionServer: Attempt=1
> >> org.apache.hadoop.hbase.Leases$LeaseStillHeldException
> >> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> >> Method)
> >> at
> >>
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> >> at
> >>
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> >> ...
> >> 2009-11-12 15:03:52,682 DEBUG org.apache.zookeeper.ClientCnxn: Got
> >> ping response for sessionid:0x224e55436ad0004 after 0ms
> >> 2009-11-12 15:04:52,480 WARN org.apache.zookeeper.ClientCnxn:
> >> Exception closing session 0x24e55436a0007d to
> >> sun.nio.ch.SelectionKeyImpl@136f6ed1
> >>
> >> Goes back in time? I does look better tho. Did you say you set the
> >> zookeeper session timeout higher in a previous thread? I'd be
> >> interested to see if a higher (2-3 minutes) value would solve the
> >> problem until java 7.
> >>
> >> The only thing that happens after that LeaseStillHeldException is that
> >> for some reason (probably GC) the region server paused. This caused
> >> the session to timeout and should be why the RS isn't connected to the
> >> ZK quorum member.
> >>
> >> BTW we worked with the Zookeeper dev team to have more meaningful
> >> exceptions and it's already committed in their trunk.
> >>
> >> J-D
> >>
> >> On Thu, Nov 12, 2009 at 8:40 AM, Zhenyu Zhong <zh...@gmail.com>
> >> wrote:
> >> > Follow up on the regionserver disconnection:
> >> >
> >> > I made a few changes to the cluster setup.
> >> > 1. split the regionserver and zookeeper quorum servers, such that
> HRegion
> >> > doesn't run on the same box as HQuorumPeer. I assume that there should
> be
> >> no
> >> > disk io contention between the regionserver and zookeeper.
> >> > Now we have 14 regionservers, and 5 zookeeper quorum servers.
> >> >
> >> > Though I experienced 2 regionserver disconnection this morning, it
> looks
> >> > better from the regionserver log. (please see the following log)
> >> >
> >> >
> >> > http://pastebin.com/m496dbfae
> >> >
> >> >
> >> > I checked diskIO for one of the regionserver(192.168.100.116) during
> the
> >> > time it disconnected.
> >> >
> >> > Time: 03:04:58 AM
> >> > Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
> >> > sda 105.31 5458.83 19503.64 9043873239
> 32312473676
> >> > sda1 2.90 6.64 99.25 10993934 164433464
> >> > sda2 1.72 23.76 30.25 39365817 50124008
> >> > sda3 0.30 0.38 3.58 624930 5923000
> >> > sda4 100.39 5428.06 19370.56 8992888270
> 32091993204
> >> >
> >> >
> >> > I also checked the zookeeper quorum server that the regionserver tried
> to
> >> > connect according to the log. However, I don't see 192.168.100.116 in
> the
> >> > client list of the zookeeper quorum server that the regionserver tried
> to
> >> > connect.
> >> > Would that be a problem?
> >> >
> >> > Thu Nov 12 15:04:52 UTC 2009
> >> > Zookeeper version: 3.2.1-808558, built on 08/27/2009 18:48 GMT
> >> > Clients:
> >> > /192.168.100.127:43045[1](queued=0,recved=26,sent=0)
> >> > /192.168.100.131:39091[1](queued=0,recved=964,sent=0)
> >> > /192.168.100.124:35961[1](queued=0,recved=3266,sent=0)
> >> > /192.168.100.123:47935[1](queued=0,recved=233,sent=0)
> >> > /192.168.100.125:46931[1](queued=0,recved=2,sent=0)
> >> > /192.168.100.118:54924[1](queued=0,recved=295,sent=0)
> >> > /192.168.100.118:41390[1](queued=0,recved=2290,sent=0)
> >> > /192.168.100.136:42243[1](queued=0,recved=0,sent=0)
> >> >
> >> > Latency min/avg/max: 0/17/6333
> >> > Received: 47111
> >> > Sent: 0
> >> > Outstanding: 0
> >> > Zxid: 0x77000083f4
> >> > Mode: leader
> >> > Node count: 23
> >> >
> >> > Best,
> >> > zhenyu
> >> >
> >> >
> >> > On Wed, Nov 11, 2009 at 3:58 PM, Zhenyu Zhong <
> zhongresearch@gmail.com
> >> >wrote:
> >> >
> >> >> Stack
> >> >>
> >> >> I am very appreciated for your comments.
> >> >> I will use the zookeeper monitoring script on my cluster and let it
> run
> >> >> overnight to see the result.
> >> >> I will follow up the thread when I get anything.
> >> >>
> >> >> thanks
> >> >> zhenyu
> >> >>
> >> >>
> >> >> On Wed, Nov 11, 2009 at 3:52 PM, stack <st...@duboce.net> wrote:
> >> >>
> >> >>> I see these in your log too:
> >> >>>
> >> >>>
> >> >>> 1. 2009-11-11 04:27:19,018 DEBUG org.apache.zookeeper.ClientCnxn:
> Got
> >> >>> ping response for sessionid:0x424dfd908c50009 after 4544ms
> >> >>> 2. 2009-11-11 04:27:19,018 DEBUG org.apache.zookeeper.ClientCnxn:
> Got
> >> >>> ping response for sessionid:0x24dfd90c810002 after 4548ms
> >> >>> 3. 2009-11-11 04:27:43,960 DEBUG org.apache.zookeeper.ClientCnxn:
> Got
> >> >>> ping response for sessionid:0x424dfd908c50009 after 9030ms
> >> >>> 4. 2009-11-11 04:27:43,960 DEBUG org.apache.zookeeper.ClientCnxn:
> Got
> >> >>> ping response for sessionid:0x24dfd90c810002 after 9030ms
> >> >>> 5. 2009-11-11 04:27:48,830 DEBUG org.apache.zookeeper.ClientCnxn:
> Got
> >> >>> ping response for sessionid:0x424dfd908c50009 after 0ms
> >> >>> 6. 2009-11-11 04:27:48,830 DEBUG org.apache.zookeeper.ClientCnxn:
> Got
> >> >>> ping response for sessionid:0x24dfd90c810002 after 0ms
> >> >>>
> >> >>> Which would indicate lag talking to zk too.
> >> >>>
> >> >>> Appending HLog also seems to get steadily worse.
> >> >>>
> >> >>> I also see these:
> >> >>>
> >> >>>
> >> >>> 1. 2009-11-11 04:27:01,233 WARN org.apache.zookeeper.ClientCnxn:
> >> >>> Exception closing session 0x24dfd90c810002 to
> >> >>> sun.nio.ch.SelectionKeyImpl@5e16a7e3
> >> >>> 2. java.io.IOException: Read error rc = -1
> >> >>> java.nio.DirectByteBuffer[pos=0 lim=4 cap=4]
> >> >>> 3. at
> >> >>>
> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
> >> >>> 4. at
> >> >>>
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
> >> >>>
> >> >>>
> >> >>> Were all 5 quorum members participating? Check by doing "echo
> stat|nc
> >> >>> QUORUM_HOST port" for each member of quorum.... the below might
> help:
> >> >>>
> >> >>> members="1,2,3,4,5"
> >> >>> while true; do date; for i in $members; do echo $i; echo stat | nc
> $i
> >> >>> 2181;
> >> >>> done; sleep 10; done
> >> >>>
> >> >>> We need to improve our zk_dump. It just reports quorum members...
> not
> >> if
> >> >>> they are actually participating or not. The above will tell you if
> >> dead
> >> >>> or
> >> >>> alive.
> >> >>>
> >> >>> Please report back after checking above and following the
> Jean-Daniel
> >> >>> prescription.
> >> >>>
> >> >>> Thanks,
> >> >>> St.Ack
> >> >>>
> >> >>>
> >> >>>
> >> >>> On Wed, Nov 11, 2009 at 12:29 PM, Zhenyu Zhong <
> >> zhongresearch@gmail.com
> >> >>> >wrote:
> >> >>>
> >> >>> > Thanks a lot for the help.
> >> >>> >
> >> >>> > I understand that the change of disk configurations should be done
> >> >>> > progressively such that HDFS can balance and recover from a disk
> >> going
> >> >>> > away.
> >> >>> >
> >> >>> > The data we are trying to import into HBase is about 200million
> rows,
> >> >>> each
> >> >>> > row is about hundreds of bytes. Basically, we read in a text file
> >> from
> >> >>> > HDFS,
> >> >>> > then for each row in the text file, we fetch data from HBase
> table,
> >> we
> >> >>> > compare the column value and update the latest value back into
> HBase.
> >> I
> >> >>> was
> >> >>> > suggested to use CheckAndPut. Anyway this is the common usecase
> on
> >> my
> >> >>> > side.
> >> >>> >
> >> >>> >
> >> >>> > Many thanks
> >> >>> > zhenyu
> >> >>> >
> >> >>> >
> >> >>> >
> >> >>> > On Wed, Nov 11, 2009 at 3:03 PM, Jean-Daniel Cryans <
> >> >>> jdcryans@apache.org
> >> >>> > >wrote:
> >> >>> >
> >> >>> > > 6 disks, 19 machines and you see those kind of issues? I'm quite
> >> >>> > > surprised and I want to learn more about what you trying to to.
> >> >>> > >
> >> >>> > > But first, if hbase is managing ZK for you then you can change
> the
> >> >>> > > value of hbase.zookeeper.property.dataDir so that it points to
> its
> >> own
> >> >>> > > mount. Make sure you also change dfs.data.dir in
> conf/hdfs-site.xml
> >> >>> > > (hadoop stuff) so that it stops pointing there (you should have
> 6
> >> >>> > > values there at the moment if you configured it correctly).
> Doing
> >> this
> >> >>> > > could result in data loss if you do it all at once so do the
> change
> >> >>> > > progressively. This also means that you may have to keep a
> seperate
> >> >>> > > HDFS configuration for those particular nodes. Please ask more
> >> >>> > > questions if it's not very clear.
> >> >>> > >
> >> >>> > > WRT to dedicating nodes to Zookeeeper, it makes sense only if
> you
> >> have
> >> >>> > > a lot of nodes but in your case, unless there's also CPU
> >> contention,
> >> >>> > > you're good with only a dedicated disk.
> >> >>> > >
> >> >>> > > Now for what you are trying to do, is this a massive import that
> >> goes
> >> >>> > > over many hours? How big is each row? I'd love to get a lot of
> >> details
> >> >>> > > to help you optimize your cluster for your particular problem.
> >> >>> > >
> >> >>> > > Thx,
> >> >>> > >
> >> >>> > > J-D
> >> >>> > >
> >> >>> > > On Wed, Nov 11, 2009 at 11:51 AM, Zhenyu Zhong <
> >> >>> zhongresearch@gmail.com>
> >> >>> > > wrote:
> >> >>> > > > J-D,
> >> >>> > > >
> >> >>> > > > Thanks a lot for the insightful comments.
> >> >>> > > > We are running 5 zookeeper quorums now. On these 5 quorum
> >> servers,
> >> >>> we
> >> >>> > > also
> >> >>> > > > run datanode, tasktracker and regionserver.
> >> >>> > > > Now we have 6 disks per machine.
> >> >>> > > > If it is the disk IO saturation, is there anyway to configure
> >> >>> zookeeper
> >> >>> > > to
> >> >>> > > > write to a seperate disk which is not the same one used by
> HDFS?
> >> >>> > > > Or shall we run zookeeper quorums on different nodes other
> than
> >> >>> HDFS?
> >> >>> > > >
> >> >>> > > > Are HLogs generated by Regionservers? Would it alleviate the
> >> issue
> >> >>> if
> >> >>> > we
> >> >>> > > use
> >> >>> > > > the first 14 for regionserver, and the rest of 5 for
> >> zookeeper?(In
> >> >>> that
> >> >>> > > > sense, the node running zookeeper quorum won't run
> regionserver)
> >> >>> > > >
> >> >>> > > > I am very appreciated for your comments.
> >> >>> > > >
> >> >>> > > > Best,
> >> >>> > > > zhenyu
> >> >>> > > >
> >> >>> > > >
> >> >>> > > >
> >> >>> > > > On Wed, Nov 11, 2009 at 2:30 PM, Jean-Daniel Cryans <
> >> >>> > jdcryans@apache.org
> >> >>> > > >wrote:
> >> >>> > > >
> >> >>> > > >> The problem seems to be that the IO is saturated. When you
> see
> >> >>> this:
> >> >>> > > >>
> >> >>> > > >> #
> >> >>> > > >> 2009-11-11 04:28:10,308 WARN
> >> >>> > > >> org.apache.hadoop.hbase.regionserver.HLog: IPC Server handler
> 9
> >> on
> >> >>> > > >> 60021 took 6819ms appending an edit to hlog; editcount=370
> >> >>> > > >> #
> >> >>> > > >> 2009-11-11 04:28:10,308 WARN
> >> >>> > > >> org.apache.hadoop.hbase.regionserver.HLog: IPC Server handler
> 2
> >> on
> >> >>> > > >> 60021 took 4412ms appending an edit to hlog; editcount=371
> >> >>> > > >>
> >> >>> > > >> It's a sign that HDFS is slow. I guess that the zookeeper
> quorum
> >> >>> > > >> members are on those same nodes using the same disks? If so,
> >> >>> there's
> >> >>> > > >> the problem because Zookeeper needs to be able to commit
> >> >>> transactions
> >> >>> > > >> very fast and if your disks are already hogged then they have
> to
> >> >>> wait
> >> >>> > > >> and you can get disconnections. Do you still have 19 quorum
> >> >>> members?
> >> >>> > > >> How many disks do you have per machine?
> >> >>> > > >>
> >> >>> > > >> Thx
> >> >>> > > >>
> >> >>> > > >> J-D
> >> >>> > > >>
> >> >>> > > >> On Wed, Nov 11, 2009 at 11:22 AM, Zhenyu Zhong <
> >> >>> > zhongresearch@gmail.com
> >> >>> > > >
> >> >>> > > >> wrote:
> >> >>> > > >> > Please take a look at the following regionserver log.
> >> >>> > > >> > The error started to come out from 4:24am UTC to 4:29am
> UTC.
> >> >>> > > >> >
> >> >>> > > >> > http://pastebin.com/m3717d064
> >> >>> > > >> >
> >> >>> > > >> > Thanks in advance.
> >> >>> > > >> >
> >> >>> > > >> > Best,
> >> >>> > > >> > zhenyu
> >> >>> > > >> >
> >> >>> > > >> >
> >> >>> > > >> > On Wed, Nov 11, 2009 at 2:01 PM, Jean-Daniel Cryans <
> >> >>> > > jdcryans@apache.org
> >> >>> > > >> >wrote:
> >> >>> > > >> >
> >> >>> > > >> >> Could you paste a bigger log trace (interested to see at
> >> least
> >> >>> all
> >> >>> > > 300
> >> >>> > > >> >> lines before that) in a pastebin?
> >> >>> > > >> >>
> >> >>> > > >> >> Thx,
> >> >>> > > >> >>
> >> >>> > > >> >> J-D
> >> >>> > > >> >>
> >> >>> > > >> >> On Wed, Nov 11, 2009 at 10:56 AM, Zhenyu Zhong <
> >> >>> > > zhongresearch@gmail.com
> >> >>> > > >> >
> >> >>> > > >> >> wrote:
> >> >>> > > >> >> > J-D,
> >> >>> > > >> >> >
> >> >>> > > >> >> > Thanks for the reply.
> >> >>> > > >> >> >
> >> >>> > > >> >> > The problem I have is one or more of the regionservers
> >> >>> disconnect
> >> >>> > > from
> >> >>> > > >> >> the
> >> >>> > > >> >> > cluster.
> >> >>> > > >> >> >
> >> >>> > > >> >> > I checked the resource usage including memory Heap usage
> >> for
> >> >>> > > >> >> regionserver,
> >> >>> > > >> >> > and found nothing suspicious. The regionserver heap
> usage
> >> is
> >> >>> way
> >> >>> > > below
> >> >>> > > >> >> the
> >> >>> > > >> >> > maxHeap defined in jvm.
> >> >>> > > >> >> >
> >> >>> > > >> >> > So I checked zookeeper log, then I found the following
> >> >>> > > >> >> > BufferUnderflowException, eventually it leads to
> >> >>> SocketException:
> >> >>> > > >> >> Transport
> >> >>> > > >> >> > endpoint is not connected.
> >> >>> > > >> >> >
> >> >>> > > >> >> >
> >> >>> > > >> >> > I also checked the regionserver log. It looks like the
> >> >>> > regionserver
> >> >>> > > >> tried
> >> >>> > > >> >> to
> >> >>> > > >> >> > connect to other zookeeper quorum server. It connects
> >> >>> > successfully,
> >> >>> > > >> but
> >> >>> > > >> >> then
> >> >>> > > >> >> > it complained about SocketException: transport endpoint
> is
> >> not
> >> >>> > > >> connected.
> >> >>> > > >> >> >
> >> >>> > > >> >> > So both zookeeper log and regionserver log told me that
> the
> >> >>> > > transport
> >> >>> > > >> >> > endpoint is not connected. I don't know why.
> >> >>> > > >> >> >
> >> >>> > > >> >> > I would be very appreciated if you could point me to the
> >> right
> >> >>> > > >> direction.
> >> >>> > > >> >> >
> >> >>> > > >> >> > Best,
> >> >>> > > >> >> >
> >> >>> > > >> >> >
> >> >>> > > >> >> > Regionserver log:
> >> >>> > > >> >> > 2009-11-11 04:24:53,718 INFO
> >> org.apache.zookeeper.ClientCnxn:
> >> >>> > > >> Attempting
> >> >>> > > >> >> > connection to server superpyxis0018/
> 192.168.100.132:2181
> >> >>> > > >> >> > 2009-11-11 04:24:53,718 INFO
> >> org.apache.zookeeper.ClientCnxn:
> >> >>> > > Priming
> >> >>> > > >> >> > connection to java.nio.channels.SocketChannel[connected
> >> >>> local=/
> >> >>> > > >> >> > 192.168.100.131:43065 remote=superpyxis0018/
> >> >>> 192.168.100.132:2181
> >> >>> > ]
> >> >>> > > >> >> > 2009-11-11 04:24:53,718 INFO
> >> org.apache.zookeeper.ClientCnxn:
> >> >>> > > Server
> >> >>> > > >> >> > connection successful
> >> >>> > > >> >> > 2009-11-11 04:24:53,768 INFO
> >> >>> > > >> >> > org.apache.hadoop.hbase.regionserver.HRegionServer: Got
> >> >>> ZooKeeper
> >> >>> > > >> event,
> >> >>> > > >> >> > state: Disconnected, type: None, path: null
> >> >>> > > >> >> > 2009-11-11 04:24:54,118 INFO
> >> org.apache.zookeeper.ClientCnxn:
> >> >>> > > >> Attempting
> >> >>> > > >> >> > connection to server superpyxis0017/
> 192.168.100.131:2181
> >> >>> > > >> >> > 2009-11-11 04:24:54,118 INFO
> >> org.apache.zookeeper.ClientCnxn:
> >> >>> > > Priming
> >> >>> > > >> >> > connection to java.nio.channels.SocketChannel[connected
> >> >>> local=/
> >> >>> > > >> >> > 192.168.100.131:46903 remote=superpyxis0017/
> >> >>> 192.168.100.131:2181
> >> >>> > ]
> >> >>> > > >> >> > 2009-11-11 04:24:54,118 INFO
> >> org.apache.zookeeper.ClientCnxn:
> >> >>> > > Server
> >> >>> > > >> >> > connection successful
> >> >>> > > >> >> > 2009-11-11 04:24:54,118 WARN
> >> org.apache.zookeeper.ClientCnxn:
> >> >>> > > >> Exception
> >> >>> > > >> >> > closing session 0x24dfd90c810002 to
> >> >>> > > >> sun.nio.ch.SelectionKeyImpl@2dd78ccf
> >> >>> > > >> >> > java.io.IOException: Read error rc = -1
> >> >>> > > >> java.nio.DirectByteBuffer[pos=0
> >> >>> > > >> >> > lim=4 cap=4]
> >> >>> > > >> >> > at
> >> >>> > > >> >> >
> >> >>> > >
> >> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
> >> >>> > > >> >> > at
> >> >>> > > >> >> >
> >> >>> >
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
> >> >>> > > >> >> > 2009-11-11 04:24:54,118 WARN
> >> org.apache.zookeeper.ClientCnxn:
> >> >>> > > Ignoring
> >> >>> > > >> >> > exception during shutdown input
> >> >>> > > >> >> > java.net.SocketException: Transport endpoint is not
> >> connected
> >> >>> > > >> >> > at sun.nio.ch.SocketChannelImpl.shutdown(Native
> >> Method)
> >> >>> > > >> >> > at
> >> >>> > > >> >> >
> >> >>> > >
> >> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
> >> >>> > > >> >> > at
> >> >>> > > >>
> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> >> >>> > > >> >> > at
> >> >>> > > >> >> >
> >> >>> > > >>
> >> >>> >
> >> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> >> >>> > > >> >> > at
> >> >>> > > >> >> >
> >> >>> >
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> >> >>> > > >> >> >
> >> >>> > > >> >> >
> >> >>> > > >> >> >
> >> >>> > > >> >> >
> >> >>> > > >> >> >
> >> >>> > > >> >> > On Wed, Nov 11, 2009 at 1:40 PM, Jean-Daniel Cryans <
> >> >>> > > >> jdcryans@apache.org
> >> >>> > > >> >> >wrote:
> >> >>> > > >> >> >
> >> >>> > > >> >> >> I'm really not sure exactly what the situation is with
> the
> >> >>> > > >> >> >> java.nio.BufferUnderflowException, but what do you mean
> by
> >> >>> > > >> >> >> "regionserver disconnection"? Could you give more
> details?
> >> >>> > > >> >> >>
> >> >>> > > >> >> >> Thx,
> >> >>> > > >> >> >>
> >> >>> > > >> >> >> J-D
> >> >>> > > >> >> >>
> >> >>> > > >> >> >> On Wed, Nov 11, 2009 at 10:34 AM, Zhenyu Zhong <
> >> >>> > > >> zhongresearch@gmail.com
> >> >>> > > >> >> >
> >> >>> > > >> >> >> wrote:
> >> >>> > > >> >> >> > I searched around and found HBASE-1645 "HBase could
> be a
> >> >>> > little
> >> >>> > > >> more
> >> >>> > > >> >> >> dainty
> >> >>> > > >> >> >> > when zk is pulled out from under it"
> >> >>> > > >> >> >> >
> >> >>> > > >> >> >> > I am wondering whether my problem is related to this
> >> >>> > HBase-1645.
> >> >>> > > Or
> >> >>> > > >> >> >> probably
> >> >>> > > >> >> >> > has nothing to do with it?
> >> >>> > > >> >> >> >
> >> >>> > > >> >> >> > thanks
> >> >>> > > >> >> >> >
> >> >>> > > >> >> >> >
> >> >>> > > >> >> >> > On Wed, Nov 11, 2009 at 12:02 PM, Zhenyu Zhong <
> >> >>> > > >> >> zhongresearch@gmail.com
> >> >>> > > >> >> >> >wrote:
> >> >>> > > >> >> >> >
> >> >>> > > >> >> >> >> Hi,
> >> >>> > > >> >> >> >>
> >> >>> > > >> >> >> >> I have been monitoring the system resources such as
> >> memory
> >> >>> > Heap
> >> >>> > > >> usage
> >> >>> > > >> >> >> for
> >> >>> > > >> >> >> >> Regionserver, load average, network bandwidth, etc.
> >> >>> > > >> >> >> >>
> >> >>> > > >> >> >> >> I experienced a disconnection last night. When I
> >> checked
> >> >>> the
> >> >>> > > >> memory
> >> >>> > > >> >> Heap
> >> >>> > > >> >> >> >> for Regionserver, it was 2336 which is way below the
> >> max
> >> >>> Heap
> >> >>> > > >> 4096. I
> >> >>> > > >> >> >> don't
> >> >>> > > >> >> >> >> think it is Regionserver problem. The requests that
> the
> >> >>> > > >> regionserver
> >> >>> > > >> >> got
> >> >>> > > >> >> >> was
> >> >>> > > >> >> >> >> about 126 which doesn't seem to be high. The load
> >> average
> >> >>> is
> >> >>> > > kind
> >> >>> > > >> of
> >> >>> > > >> >> >> high
> >> >>> > > >> >> >> >> >100. (but I saw disconnections while the load
> average
> >> is
> >> >>> > > around
> >> >>> > > >> 30).
> >> >>> > > >> >> >> The
> >> >>> > > >> >> >> >> bandwidth is not a problem.
> >> >>> > > >> >> >> >>
> >> >>> > > >> >> >> >> However, I found error messages from the zookeeper.
> >> Please
> >> >>> > see
> >> >>> > > the
> >> >>> > > >> >> >> >> following error messages. There are some
> >> >>> > > >> >> >> >> BufferUnderflowException error, IOException
> error,etc.
> >> >>> > > >> >> >> >> May I ask what might be the causes that generate
> these
> >> >>> error
> >> >>> > > >> >> messages?
> >> >>> > > >> >> >> >>
> >> >>> > > >> >> >> >> Many thanks
> >> >>> > > >> >> >> >> zhenyu
> >> >>> > > >> >> >> >>
> >> >>> > > >> >> >> >>
> >> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
> >> >>> > > org.apache.zookeeper.server.Request:
> >> >>> > > >> >> >> Ignoring
> >> >>> > > >> >> >> >> exception during toString
> >> >>> > > >> >> >> >> java.nio.BufferUnderflowException
> >> >>> > > >> >> >> >> at
> >> >>> > java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
> >> >>> > > >> >> >> >> at
> java.nio.ByteBuffer.get(ByteBuffer.java:675)
> >> >>> > > >> >> >> >> at
> >> >>> > > >> >> >>
> >> >>> org.apache.zookeeper.server.Request.toString(Request.java:199)
> >> >>> > > >> >> >> >> at
> java.lang.String.valueOf(String.java:2827)
> >> >>> > > >> >> >> >> at
> >> >>> > > java.lang.StringBuilder.append(StringBuilder.java:115)
> >> >>> > > >> >> >> >> at
> >> >>> > > >> >> >> >>
> >> >>> > > >> >> >>
> >> >>> > > >> >>
> >> >>> > > >>
> >> >>> > >
> >> >>> >
> >> >>>
> >>
> org.apache.zookeeper.server.quorum.CommitProcessor.processRequest(CommitProcessor.java:167)
> >> >>> > > >> >> >> >> at
> >> >>> > > >> >> >> >>
> >> >>> > > >> >> >>
> >> >>> > > >> >>
> >> >>> > > >>
> >> >>> > >
> >> >>> >
> >> >>>
> >>
> org.apache.zookeeper.server.quorum.FollowerRequestProcessor.run(FollowerRequestProcessor.java:68)
> >> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
> >> >>> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
> >> >>> > > >> >> >> >> Exception causing close of session 0x424dfd908c50009
> >> >>> > > >> >> >> >> due to java.io.IOException: Read error
> >> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
> >> >>> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
> >> >>> > > >> >> >> >> IOException stack trace
> >> >>> > > >> >> >> >> java.io.IOException: Read error
> >> >>> > > >> >> >> >> at
> >> >>> > > >> >> >> >>
> >> >>> > > >> >>
> >> >>> > >
> >> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:372)
> >> >>> > > >> >> >> >> at
> >> >>> > > >> >> >> >>
> >> >>> > > >> >> >>
> >> >>> > > >> >>
> >> >>> > > >>
> >> >>> > >
> >> >>> >
> >> >>>
> >>
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:221)
> >> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
> >> >>> > > >> >> >> >> org.apache.zookeeper.server.quorum.CommitProcessor:
> >> >>> > Processing
> >> >>> > > >> >> request::
> >> >>> > > >> >> >> >> sessionid:0x424dfd908c50009 type:setWatches
> >> >>> > > >> cxid:0xfffffffffffffff8
> >> >>> > > >> >> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
> >> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
> >> >>> > > org.apache.zookeeper.server.Request:
> >> >>> > > >> >> >> Ignoring
> >> >>> > > >> >> >> >> exception during toString
> >> >>> > > >> >> >> >> java.nio.BufferUnderflowException
> >> >>> > > >> >> >> >> at
> >> >>> > java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
> >> >>> > > >> >> >> >> at
> java.nio.ByteBuffer.get(ByteBuffer.java:675)
> >> >>> > > >> >> >> >> at
> >> >>> > > >> >> >>
> >> >>> org.apache.zookeeper.server.Request.toString(Request.java:199)
> >> >>> > > >> >> >> >> at
> java.lang.String.valueOf(String.java:2827)
> >> >>> > > >> >> >> >> at
> >> >>> > > java.lang.StringBuilder.append(StringBuilder.java:115)
> >> >>> > > >> >> >> >> at
> >> >>> > > >> >> >> >>
> >> >>> > > >> >> >>
> >> >>> > > >> >>
> >> >>> > > >>
> >> >>> > >
> >> >>> >
> >> >>>
> >>
> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:74)
> >> >>> > > >> >> >> >> at
> >> >>> > > >> >> >> >>
> >> >>> > > >> >> >>
> >> >>> > > >> >>
> >> >>> > > >>
> >> >>> > >
> >> >>> >
> >> >>>
> >>
> org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)
> >> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
> >> >>> > > >> >> >> >> org.apache.zookeeper.server.FinalRequestProcessor:
> >> >>> Processing
> >> >>> > > >> >> request::
> >> >>> > > >> >> >> >> sessionid:0x424dfd908c50009 type:setWatches
> >> >>> > > >> cxid:0xfffffffffffffff8
> >> >>> > > >> >> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
> >> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
> >> >>> > > org.apache.zookeeper.server.Request:
> >> >>> > > >> >> >> Ignoring
> >> >>> > > >> >> >> >> exception during toString
> >> >>> > > >> >> >> >> java.nio.BufferUnderflowException
> >> >>> > > >> >> >> >> at
> >> >>> > java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
> >> >>> > > >> >> >> >> at
> java.nio.ByteBuffer.get(ByteBuffer.java:675)
> >> >>> > > >> >> >> >> at
> >> >>> > > >> >> >>
> >> >>> org.apache.zookeeper.server.Request.toString(Request.java:199)
> >> >>> > > >> >> >> >> at
> >> >>> > > >> >> >> >>
> >> >>> > > >>
> >> >>>
> org.apache.log4j.or.DefaultRenderer.doRender(DefaultRenderer.java:36)
> >> >>> > > >> >> >> >> at
> >> >>> > > >> >> >> >>
> >> >>> > >
> org.apache.log4j.or.RendererMap.findAndRender(RendererMap.java:80)
> >> >>> > > >> >> >> >> at
> >> >>> > > >> >> >> >>
> >> >>> > > >> >> >>
> >> >>> > > >> >>
> >> >>> > > >>
> >> >>> > >
> >> >>> >
> >> >>>
> >>
> org.apache.log4j.spi.LoggingEvent.getRenderedMessage(LoggingEvent.java:362)
> >> >>> > > >> >> >> >> at
> >> >>> > > >> >> >> >>
> >> >>> > > >> >> >>
> >> >>> > > >> >>
> >> >>> > > >>
> >> >>> > >
> >> >>> >
> >> >>>
> >>
> org.apache.log4j.helpers.PatternParser$BasicPatternConverter.convert(PatternParser.java:403)
> >> >>> > > >> >> >> >> at
> >> >>> > > >> >> >> >>
> >> >>> > > >> >> >>
> >> >>> > > >> >>
> >> >>> > > >>
> >> >>> > >
> >> >>> >
> >> >>>
> >>
> org.apache.log4j.helpers.PatternConverter.format(PatternConverter.java:65)
> >> >>> > > >> >> >> >> at
> >> >>> > > >> >>
> org.apache.log4j.PatternLayout.format(PatternLayout.java:502)
> >> >>> > > >> >> >> >> at
> >> >>> > > >> >> >> >>
> >> >>> > >
> org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:302)
> >> >>> > > >> >> >> >> at
> >> >>> > > >> >> >> >>
> >> >>> > > >> >> >>
> >> >>> > > >> >>
> >> >>> > > >>
> >> >>> > >
> >> >>> >
> >> >>>
> >>
> org.apache.log4j.DailyRollingFileAppender.subAppend(DailyRollingFileAppender.java:359)
> >> >>> > > >> >> >> >> at
> >> >>> > > >> >> >>
> >> >>> org.apache.log4j.WriterAppender.append(WriterAppender.java:160)
> >> >>> > > >> >> >> >> at
> >> >>> > > >> >> >> >>
> >> >>> > > >>
> >> >>>
> org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
> >> >>> > > >> >> >> >> at
> >> >>> > > >> >> >> >>
> >> >>> > > >> >> >>
> >> >>> > > >> >>
> >> >>> > > >>
> >> >>> > >
> >> >>> >
> >> >>>
> >>
> org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
> >> >>> > > >> >> >> >> at
> >> >>> > > >> org.apache.log4j.Category.callAppenders(Category.java:206)
> >> >>> > > >> >> >> >> at
> >> >>> > > org.apache.log4j.Category.forcedLog(Category.java:391)
> >> >>> > > >> >> >> >> at
> >> >>> org.apache.log4j.Category.debug(Category.java:260)
> >> >>> > > >> >> >> >> at
> >> >>> > > >> >> >> >>
> >> >>> > > >> >> >>
> >> >>> > > >> >>
> >> >>> > > >>
> >> >>> > >
> >> >>> >
> >> >>>
> >>
> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:147)
> >> >>> > > >> >> >> >> at
> >> >>> > > >> >> >> >>
> >> >>> > > >> >> >>
> >> >>> > > >> >>
> >> >>> > > >>
> >> >>> > >
> >> >>> >
> >> >>>
> >>
> org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)
> >> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
> >> >>> > > org.apache.zookeeper.server.Request:
> >> >>> > > >> >> >> Ignoring
> >> >>> > > >> >> >> >> exception during toString
> >> >>> > > >> >> >> >> sessionid:0x424dfd908c50009 type:setWatches
> >> >>> > > >> cxid:0xfffffffffffffff8
> >> >>> > > >> >> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
> >> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 INFO
> >> >>> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
> >> >>> > > >> >> >> >> closing session:0x424dfd908c50009 NIOServerCnxn:
> >> >>> > > >> >> >> >> java.nio.channels.SocketChannel[connected local=/
> >> >>> > > >> >> 192.168.100.131:2181
> >> >>> > > >> >> >> remote=/
> >> >>> > > >> >> >> >> 192.168.100.131:35500]
> >> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
> >> >>> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
> >> >>> > > >> >> >> >> ignoring exception during input shutdown
> >> >>> > > >> >> >> >> java.net.SocketException: Transport endpoint is not
> >> >>> connected
> >> >>> > > >> >> >> >> at
> sun.nio.ch.SocketChannelImpl.shutdown(Native
> >> >>> > Method)
> >> >>> > > >> >> >> >> at
> >> >>> > > >> >> >> >>
> >> >>> > > >> >>
> >> >>> > >
> >> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
> >> >>> > > >> >> >> >> at
> >> >>> > > >> >> >>
> >> >>> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> >> >>> > > >> >> >> >> at
> >> >>> > > >> >> >> >>
> >> >>> > > >> >>
> >> >>> > >
> >> >>>
> org.apache.zookeeper.server.NIOServerCnxn.close(NIOServerCnxn.java:848)
> >> >>> > > >> >> >> >> at
> >> >>> > > >> >> >> >>
> >> >>> > > >> >>
> >> >>> > >
> >> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:498)
> >> >>> > > >> >> >> >> at
> >> >>> > > >> >> >> >>
> >> >>> > > >> >> >>
> >> >>> > > >> >>
> >> >>> > > >>
> >> >>> > >
> >> >>> >
> >> >>>
> >>
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:221)
> >> >>> > > >> >> >> >>
> >> >>> > > >> >> >> >
> >> >>> > > >> >> >>
> >> >>> > > >> >> >
> >> >>> > > >> >>
> >> >>> > > >> >
> >> >>> > > >>
> >> >>> > > >
> >> >>> > >
> >> >>> >
> >> >>>
> >> >>
> >> >>
> >> >
> >>
> >
>
Re: regionserver disconnection
Posted by Jean-Daniel Cryans <jd...@apache.org>.
Usually if it was 10 minutes we would see sleepers complaining they
slept for way too long. In the logs I can see a 1 minute pause but not
10 minutes because of the ZK debug. Are you sure you don't have a
typo? Like what you just wrote is not valid, you wrote
zookeeper.session.timout instead of zookeeper.session.timeout. Are you
sure it's distributed over the cluster? Could you pastebin
hbase-site.xml?
Thx,
J-D
On Thu, Nov 12, 2009 at 10:28 AM, Zhenyu Zhong <zh...@gmail.com> wrote:
> J-D,
>
> Yes, that is weird. But it is what I have in the log. Probably the log is
> flushed by different threads, and one thread is blocked for a while?
>
> I set the zookeeper.session.timout=600000, which is 10 minutes. However, we
> still see this problem.
>
> Does GC pause for such a long time period?
> I will try different GC options to see the differences.
>
> thanks
> zhenyu
>
>
> On Thu, Nov 12, 2009 at 1:10 PM, Jean-Daniel Cryans <jd...@apache.org>wrote:
>
>> I see something weird
>>
>> 2009-11-12 15:04:36,370 WARN
>> org.apache.hadoop.hbase.regionserver.HRegionServer: Attempt=1
>> org.apache.hadoop.hbase.Leases$LeaseStillHeldException
>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>> Method)
>> at
>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>> at
>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>> ...
>> 2009-11-12 15:03:52,682 DEBUG org.apache.zookeeper.ClientCnxn: Got
>> ping response for sessionid:0x224e55436ad0004 after 0ms
>> 2009-11-12 15:04:52,480 WARN org.apache.zookeeper.ClientCnxn:
>> Exception closing session 0x24e55436a0007d to
>> sun.nio.ch.SelectionKeyImpl@136f6ed1
>>
>> Goes back in time? I does look better tho. Did you say you set the
>> zookeeper session timeout higher in a previous thread? I'd be
>> interested to see if a higher (2-3 minutes) value would solve the
>> problem until java 7.
>>
>> The only thing that happens after that LeaseStillHeldException is that
>> for some reason (probably GC) the region server paused. This caused
>> the session to timeout and should be why the RS isn't connected to the
>> ZK quorum member.
>>
>> BTW we worked with the Zookeeper dev team to have more meaningful
>> exceptions and it's already committed in their trunk.
>>
>> J-D
>>
>> On Thu, Nov 12, 2009 at 8:40 AM, Zhenyu Zhong <zh...@gmail.com>
>> wrote:
>> > Follow up on the regionserver disconnection:
>> >
>> > I made a few changes to the cluster setup.
>> > 1. split the regionserver and zookeeper quorum servers, such that HRegion
>> > doesn't run on the same box as HQuorumPeer. I assume that there should be
>> no
>> > disk io contention between the regionserver and zookeeper.
>> > Now we have 14 regionservers, and 5 zookeeper quorum servers.
>> >
>> > Though I experienced 2 regionserver disconnection this morning, it looks
>> > better from the regionserver log. (please see the following log)
>> >
>> >
>> > http://pastebin.com/m496dbfae
>> >
>> >
>> > I checked diskIO for one of the regionserver(192.168.100.116) during the
>> > time it disconnected.
>> >
>> > Time: 03:04:58 AM
>> > Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
>> > sda 105.31 5458.83 19503.64 9043873239 32312473676
>> > sda1 2.90 6.64 99.25 10993934 164433464
>> > sda2 1.72 23.76 30.25 39365817 50124008
>> > sda3 0.30 0.38 3.58 624930 5923000
>> > sda4 100.39 5428.06 19370.56 8992888270 32091993204
>> >
>> >
>> > I also checked the zookeeper quorum server that the regionserver tried to
>> > connect according to the log. However, I don't see 192.168.100.116 in the
>> > client list of the zookeeper quorum server that the regionserver tried to
>> > connect.
>> > Would that be a problem?
>> >
>> > Thu Nov 12 15:04:52 UTC 2009
>> > Zookeeper version: 3.2.1-808558, built on 08/27/2009 18:48 GMT
>> > Clients:
>> > /192.168.100.127:43045[1](queued=0,recved=26,sent=0)
>> > /192.168.100.131:39091[1](queued=0,recved=964,sent=0)
>> > /192.168.100.124:35961[1](queued=0,recved=3266,sent=0)
>> > /192.168.100.123:47935[1](queued=0,recved=233,sent=0)
>> > /192.168.100.125:46931[1](queued=0,recved=2,sent=0)
>> > /192.168.100.118:54924[1](queued=0,recved=295,sent=0)
>> > /192.168.100.118:41390[1](queued=0,recved=2290,sent=0)
>> > /192.168.100.136:42243[1](queued=0,recved=0,sent=0)
>> >
>> > Latency min/avg/max: 0/17/6333
>> > Received: 47111
>> > Sent: 0
>> > Outstanding: 0
>> > Zxid: 0x77000083f4
>> > Mode: leader
>> > Node count: 23
>> >
>> > Best,
>> > zhenyu
>> >
>> >
>> > On Wed, Nov 11, 2009 at 3:58 PM, Zhenyu Zhong <zhongresearch@gmail.com
>> >wrote:
>> >
>> >> Stack
>> >>
>> >> I am very appreciated for your comments.
>> >> I will use the zookeeper monitoring script on my cluster and let it run
>> >> overnight to see the result.
>> >> I will follow up the thread when I get anything.
>> >>
>> >> thanks
>> >> zhenyu
>> >>
>> >>
>> >> On Wed, Nov 11, 2009 at 3:52 PM, stack <st...@duboce.net> wrote:
>> >>
>> >>> I see these in your log too:
>> >>>
>> >>>
>> >>> 1. 2009-11-11 04:27:19,018 DEBUG org.apache.zookeeper.ClientCnxn: Got
>> >>> ping response for sessionid:0x424dfd908c50009 after 4544ms
>> >>> 2. 2009-11-11 04:27:19,018 DEBUG org.apache.zookeeper.ClientCnxn: Got
>> >>> ping response for sessionid:0x24dfd90c810002 after 4548ms
>> >>> 3. 2009-11-11 04:27:43,960 DEBUG org.apache.zookeeper.ClientCnxn: Got
>> >>> ping response for sessionid:0x424dfd908c50009 after 9030ms
>> >>> 4. 2009-11-11 04:27:43,960 DEBUG org.apache.zookeeper.ClientCnxn: Got
>> >>> ping response for sessionid:0x24dfd90c810002 after 9030ms
>> >>> 5. 2009-11-11 04:27:48,830 DEBUG org.apache.zookeeper.ClientCnxn: Got
>> >>> ping response for sessionid:0x424dfd908c50009 after 0ms
>> >>> 6. 2009-11-11 04:27:48,830 DEBUG org.apache.zookeeper.ClientCnxn: Got
>> >>> ping response for sessionid:0x24dfd90c810002 after 0ms
>> >>>
>> >>> Which would indicate lag talking to zk too.
>> >>>
>> >>> Appending HLog also seems to get steadily worse.
>> >>>
>> >>> I also see these:
>> >>>
>> >>>
>> >>> 1. 2009-11-11 04:27:01,233 WARN org.apache.zookeeper.ClientCnxn:
>> >>> Exception closing session 0x24dfd90c810002 to
>> >>> sun.nio.ch.SelectionKeyImpl@5e16a7e3
>> >>> 2. java.io.IOException: Read error rc = -1
>> >>> java.nio.DirectByteBuffer[pos=0 lim=4 cap=4]
>> >>> 3. at
>> >>> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
>> >>> 4. at
>> >>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
>> >>>
>> >>>
>> >>> Were all 5 quorum members participating? Check by doing "echo stat|nc
>> >>> QUORUM_HOST port" for each member of quorum.... the below might help:
>> >>>
>> >>> members="1,2,3,4,5"
>> >>> while true; do date; for i in $members; do echo $i; echo stat | nc $i
>> >>> 2181;
>> >>> done; sleep 10; done
>> >>>
>> >>> We need to improve our zk_dump. It just reports quorum members... not
>> if
>> >>> they are actually participating or not. The above will tell you if
>> dead
>> >>> or
>> >>> alive.
>> >>>
>> >>> Please report back after checking above and following the Jean-Daniel
>> >>> prescription.
>> >>>
>> >>> Thanks,
>> >>> St.Ack
>> >>>
>> >>>
>> >>>
>> >>> On Wed, Nov 11, 2009 at 12:29 PM, Zhenyu Zhong <
>> zhongresearch@gmail.com
>> >>> >wrote:
>> >>>
>> >>> > Thanks a lot for the help.
>> >>> >
>> >>> > I understand that the change of disk configurations should be done
>> >>> > progressively such that HDFS can balance and recover from a disk
>> going
>> >>> > away.
>> >>> >
>> >>> > The data we are trying to import into HBase is about 200million rows,
>> >>> each
>> >>> > row is about hundreds of bytes. Basically, we read in a text file
>> from
>> >>> > HDFS,
>> >>> > then for each row in the text file, we fetch data from HBase table,
>> we
>> >>> > compare the column value and update the latest value back into HBase.
>> I
>> >>> was
>> >>> > suggested to use CheckAndPut. Anyway this is the common usecase on
>> my
>> >>> > side.
>> >>> >
>> >>> >
>> >>> > Many thanks
>> >>> > zhenyu
>> >>> >
>> >>> >
>> >>> >
>> >>> > On Wed, Nov 11, 2009 at 3:03 PM, Jean-Daniel Cryans <
>> >>> jdcryans@apache.org
>> >>> > >wrote:
>> >>> >
>> >>> > > 6 disks, 19 machines and you see those kind of issues? I'm quite
>> >>> > > surprised and I want to learn more about what you trying to to.
>> >>> > >
>> >>> > > But first, if hbase is managing ZK for you then you can change the
>> >>> > > value of hbase.zookeeper.property.dataDir so that it points to its
>> own
>> >>> > > mount. Make sure you also change dfs.data.dir in conf/hdfs-site.xml
>> >>> > > (hadoop stuff) so that it stops pointing there (you should have 6
>> >>> > > values there at the moment if you configured it correctly). Doing
>> this
>> >>> > > could result in data loss if you do it all at once so do the change
>> >>> > > progressively. This also means that you may have to keep a seperate
>> >>> > > HDFS configuration for those particular nodes. Please ask more
>> >>> > > questions if it's not very clear.
>> >>> > >
>> >>> > > WRT to dedicating nodes to Zookeeeper, it makes sense only if you
>> have
>> >>> > > a lot of nodes but in your case, unless there's also CPU
>> contention,
>> >>> > > you're good with only a dedicated disk.
>> >>> > >
>> >>> > > Now for what you are trying to do, is this a massive import that
>> goes
>> >>> > > over many hours? How big is each row? I'd love to get a lot of
>> details
>> >>> > > to help you optimize your cluster for your particular problem.
>> >>> > >
>> >>> > > Thx,
>> >>> > >
>> >>> > > J-D
>> >>> > >
>> >>> > > On Wed, Nov 11, 2009 at 11:51 AM, Zhenyu Zhong <
>> >>> zhongresearch@gmail.com>
>> >>> > > wrote:
>> >>> > > > J-D,
>> >>> > > >
>> >>> > > > Thanks a lot for the insightful comments.
>> >>> > > > We are running 5 zookeeper quorums now. On these 5 quorum
>> servers,
>> >>> we
>> >>> > > also
>> >>> > > > run datanode, tasktracker and regionserver.
>> >>> > > > Now we have 6 disks per machine.
>> >>> > > > If it is the disk IO saturation, is there anyway to configure
>> >>> zookeeper
>> >>> > > to
>> >>> > > > write to a seperate disk which is not the same one used by HDFS?
>> >>> > > > Or shall we run zookeeper quorums on different nodes other than
>> >>> HDFS?
>> >>> > > >
>> >>> > > > Are HLogs generated by Regionservers? Would it alleviate the
>> issue
>> >>> if
>> >>> > we
>> >>> > > use
>> >>> > > > the first 14 for regionserver, and the rest of 5 for
>> zookeeper?(In
>> >>> that
>> >>> > > > sense, the node running zookeeper quorum won't run regionserver)
>> >>> > > >
>> >>> > > > I am very appreciated for your comments.
>> >>> > > >
>> >>> > > > Best,
>> >>> > > > zhenyu
>> >>> > > >
>> >>> > > >
>> >>> > > >
>> >>> > > > On Wed, Nov 11, 2009 at 2:30 PM, Jean-Daniel Cryans <
>> >>> > jdcryans@apache.org
>> >>> > > >wrote:
>> >>> > > >
>> >>> > > >> The problem seems to be that the IO is saturated. When you see
>> >>> this:
>> >>> > > >>
>> >>> > > >> #
>> >>> > > >> 2009-11-11 04:28:10,308 WARN
>> >>> > > >> org.apache.hadoop.hbase.regionserver.HLog: IPC Server handler 9
>> on
>> >>> > > >> 60021 took 6819ms appending an edit to hlog; editcount=370
>> >>> > > >> #
>> >>> > > >> 2009-11-11 04:28:10,308 WARN
>> >>> > > >> org.apache.hadoop.hbase.regionserver.HLog: IPC Server handler 2
>> on
>> >>> > > >> 60021 took 4412ms appending an edit to hlog; editcount=371
>> >>> > > >>
>> >>> > > >> It's a sign that HDFS is slow. I guess that the zookeeper quorum
>> >>> > > >> members are on those same nodes using the same disks? If so,
>> >>> there's
>> >>> > > >> the problem because Zookeeper needs to be able to commit
>> >>> transactions
>> >>> > > >> very fast and if your disks are already hogged then they have to
>> >>> wait
>> >>> > > >> and you can get disconnections. Do you still have 19 quorum
>> >>> members?
>> >>> > > >> How many disks do you have per machine?
>> >>> > > >>
>> >>> > > >> Thx
>> >>> > > >>
>> >>> > > >> J-D
>> >>> > > >>
>> >>> > > >> On Wed, Nov 11, 2009 at 11:22 AM, Zhenyu Zhong <
>> >>> > zhongresearch@gmail.com
>> >>> > > >
>> >>> > > >> wrote:
>> >>> > > >> > Please take a look at the following regionserver log.
>> >>> > > >> > The error started to come out from 4:24am UTC to 4:29am UTC.
>> >>> > > >> >
>> >>> > > >> > http://pastebin.com/m3717d064
>> >>> > > >> >
>> >>> > > >> > Thanks in advance.
>> >>> > > >> >
>> >>> > > >> > Best,
>> >>> > > >> > zhenyu
>> >>> > > >> >
>> >>> > > >> >
>> >>> > > >> > On Wed, Nov 11, 2009 at 2:01 PM, Jean-Daniel Cryans <
>> >>> > > jdcryans@apache.org
>> >>> > > >> >wrote:
>> >>> > > >> >
>> >>> > > >> >> Could you paste a bigger log trace (interested to see at
>> least
>> >>> all
>> >>> > > 300
>> >>> > > >> >> lines before that) in a pastebin?
>> >>> > > >> >>
>> >>> > > >> >> Thx,
>> >>> > > >> >>
>> >>> > > >> >> J-D
>> >>> > > >> >>
>> >>> > > >> >> On Wed, Nov 11, 2009 at 10:56 AM, Zhenyu Zhong <
>> >>> > > zhongresearch@gmail.com
>> >>> > > >> >
>> >>> > > >> >> wrote:
>> >>> > > >> >> > J-D,
>> >>> > > >> >> >
>> >>> > > >> >> > Thanks for the reply.
>> >>> > > >> >> >
>> >>> > > >> >> > The problem I have is one or more of the regionservers
>> >>> disconnect
>> >>> > > from
>> >>> > > >> >> the
>> >>> > > >> >> > cluster.
>> >>> > > >> >> >
>> >>> > > >> >> > I checked the resource usage including memory Heap usage
>> for
>> >>> > > >> >> regionserver,
>> >>> > > >> >> > and found nothing suspicious. The regionserver heap usage
>> is
>> >>> way
>> >>> > > below
>> >>> > > >> >> the
>> >>> > > >> >> > maxHeap defined in jvm.
>> >>> > > >> >> >
>> >>> > > >> >> > So I checked zookeeper log, then I found the following
>> >>> > > >> >> > BufferUnderflowException, eventually it leads to
>> >>> SocketException:
>> >>> > > >> >> Transport
>> >>> > > >> >> > endpoint is not connected.
>> >>> > > >> >> >
>> >>> > > >> >> >
>> >>> > > >> >> > I also checked the regionserver log. It looks like the
>> >>> > regionserver
>> >>> > > >> tried
>> >>> > > >> >> to
>> >>> > > >> >> > connect to other zookeeper quorum server. It connects
>> >>> > successfully,
>> >>> > > >> but
>> >>> > > >> >> then
>> >>> > > >> >> > it complained about SocketException: transport endpoint is
>> not
>> >>> > > >> connected.
>> >>> > > >> >> >
>> >>> > > >> >> > So both zookeeper log and regionserver log told me that the
>> >>> > > transport
>> >>> > > >> >> > endpoint is not connected. I don't know why.
>> >>> > > >> >> >
>> >>> > > >> >> > I would be very appreciated if you could point me to the
>> right
>> >>> > > >> direction.
>> >>> > > >> >> >
>> >>> > > >> >> > Best,
>> >>> > > >> >> >
>> >>> > > >> >> >
>> >>> > > >> >> > Regionserver log:
>> >>> > > >> >> > 2009-11-11 04:24:53,718 INFO
>> org.apache.zookeeper.ClientCnxn:
>> >>> > > >> Attempting
>> >>> > > >> >> > connection to server superpyxis0018/192.168.100.132:2181
>> >>> > > >> >> > 2009-11-11 04:24:53,718 INFO
>> org.apache.zookeeper.ClientCnxn:
>> >>> > > Priming
>> >>> > > >> >> > connection to java.nio.channels.SocketChannel[connected
>> >>> local=/
>> >>> > > >> >> > 192.168.100.131:43065 remote=superpyxis0018/
>> >>> 192.168.100.132:2181
>> >>> > ]
>> >>> > > >> >> > 2009-11-11 04:24:53,718 INFO
>> org.apache.zookeeper.ClientCnxn:
>> >>> > > Server
>> >>> > > >> >> > connection successful
>> >>> > > >> >> > 2009-11-11 04:24:53,768 INFO
>> >>> > > >> >> > org.apache.hadoop.hbase.regionserver.HRegionServer: Got
>> >>> ZooKeeper
>> >>> > > >> event,
>> >>> > > >> >> > state: Disconnected, type: None, path: null
>> >>> > > >> >> > 2009-11-11 04:24:54,118 INFO
>> org.apache.zookeeper.ClientCnxn:
>> >>> > > >> Attempting
>> >>> > > >> >> > connection to server superpyxis0017/192.168.100.131:2181
>> >>> > > >> >> > 2009-11-11 04:24:54,118 INFO
>> org.apache.zookeeper.ClientCnxn:
>> >>> > > Priming
>> >>> > > >> >> > connection to java.nio.channels.SocketChannel[connected
>> >>> local=/
>> >>> > > >> >> > 192.168.100.131:46903 remote=superpyxis0017/
>> >>> 192.168.100.131:2181
>> >>> > ]
>> >>> > > >> >> > 2009-11-11 04:24:54,118 INFO
>> org.apache.zookeeper.ClientCnxn:
>> >>> > > Server
>> >>> > > >> >> > connection successful
>> >>> > > >> >> > 2009-11-11 04:24:54,118 WARN
>> org.apache.zookeeper.ClientCnxn:
>> >>> > > >> Exception
>> >>> > > >> >> > closing session 0x24dfd90c810002 to
>> >>> > > >> sun.nio.ch.SelectionKeyImpl@2dd78ccf
>> >>> > > >> >> > java.io.IOException: Read error rc = -1
>> >>> > > >> java.nio.DirectByteBuffer[pos=0
>> >>> > > >> >> > lim=4 cap=4]
>> >>> > > >> >> > at
>> >>> > > >> >> >
>> >>> > >
>> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
>> >>> > > >> >> > at
>> >>> > > >> >> >
>> >>> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
>> >>> > > >> >> > 2009-11-11 04:24:54,118 WARN
>> org.apache.zookeeper.ClientCnxn:
>> >>> > > Ignoring
>> >>> > > >> >> > exception during shutdown input
>> >>> > > >> >> > java.net.SocketException: Transport endpoint is not
>> connected
>> >>> > > >> >> > at sun.nio.ch.SocketChannelImpl.shutdown(Native
>> Method)
>> >>> > > >> >> > at
>> >>> > > >> >> >
>> >>> > >
>> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
>> >>> > > >> >> > at
>> >>> > > >> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>> >>> > > >> >> > at
>> >>> > > >> >> >
>> >>> > > >>
>> >>> >
>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
>> >>> > > >> >> > at
>> >>> > > >> >> >
>> >>> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
>> >>> > > >> >> >
>> >>> > > >> >> >
>> >>> > > >> >> >
>> >>> > > >> >> >
>> >>> > > >> >> >
>> >>> > > >> >> > On Wed, Nov 11, 2009 at 1:40 PM, Jean-Daniel Cryans <
>> >>> > > >> jdcryans@apache.org
>> >>> > > >> >> >wrote:
>> >>> > > >> >> >
>> >>> > > >> >> >> I'm really not sure exactly what the situation is with the
>> >>> > > >> >> >> java.nio.BufferUnderflowException, but what do you mean by
>> >>> > > >> >> >> "regionserver disconnection"? Could you give more details?
>> >>> > > >> >> >>
>> >>> > > >> >> >> Thx,
>> >>> > > >> >> >>
>> >>> > > >> >> >> J-D
>> >>> > > >> >> >>
>> >>> > > >> >> >> On Wed, Nov 11, 2009 at 10:34 AM, Zhenyu Zhong <
>> >>> > > >> zhongresearch@gmail.com
>> >>> > > >> >> >
>> >>> > > >> >> >> wrote:
>> >>> > > >> >> >> > I searched around and found HBASE-1645 "HBase could be a
>> >>> > little
>> >>> > > >> more
>> >>> > > >> >> >> dainty
>> >>> > > >> >> >> > when zk is pulled out from under it"
>> >>> > > >> >> >> >
>> >>> > > >> >> >> > I am wondering whether my problem is related to this
>> >>> > HBase-1645.
>> >>> > > Or
>> >>> > > >> >> >> probably
>> >>> > > >> >> >> > has nothing to do with it?
>> >>> > > >> >> >> >
>> >>> > > >> >> >> > thanks
>> >>> > > >> >> >> >
>> >>> > > >> >> >> >
>> >>> > > >> >> >> > On Wed, Nov 11, 2009 at 12:02 PM, Zhenyu Zhong <
>> >>> > > >> >> zhongresearch@gmail.com
>> >>> > > >> >> >> >wrote:
>> >>> > > >> >> >> >
>> >>> > > >> >> >> >> Hi,
>> >>> > > >> >> >> >>
>> >>> > > >> >> >> >> I have been monitoring the system resources such as
>> memory
>> >>> > Heap
>> >>> > > >> usage
>> >>> > > >> >> >> for
>> >>> > > >> >> >> >> Regionserver, load average, network bandwidth, etc.
>> >>> > > >> >> >> >>
>> >>> > > >> >> >> >> I experienced a disconnection last night. When I
>> checked
>> >>> the
>> >>> > > >> memory
>> >>> > > >> >> Heap
>> >>> > > >> >> >> >> for Regionserver, it was 2336 which is way below the
>> max
>> >>> Heap
>> >>> > > >> 4096. I
>> >>> > > >> >> >> don't
>> >>> > > >> >> >> >> think it is Regionserver problem. The requests that the
>> >>> > > >> regionserver
>> >>> > > >> >> got
>> >>> > > >> >> >> was
>> >>> > > >> >> >> >> about 126 which doesn't seem to be high. The load
>> average
>> >>> is
>> >>> > > kind
>> >>> > > >> of
>> >>> > > >> >> >> high
>> >>> > > >> >> >> >> >100. (but I saw disconnections while the load average
>> is
>> >>> > > around
>> >>> > > >> 30).
>> >>> > > >> >> >> The
>> >>> > > >> >> >> >> bandwidth is not a problem.
>> >>> > > >> >> >> >>
>> >>> > > >> >> >> >> However, I found error messages from the zookeeper.
>> Please
>> >>> > see
>> >>> > > the
>> >>> > > >> >> >> >> following error messages. There are some
>> >>> > > >> >> >> >> BufferUnderflowException error, IOException error,etc.
>> >>> > > >> >> >> >> May I ask what might be the causes that generate these
>> >>> error
>> >>> > > >> >> messages?
>> >>> > > >> >> >> >>
>> >>> > > >> >> >> >> Many thanks
>> >>> > > >> >> >> >> zhenyu
>> >>> > > >> >> >> >>
>> >>> > > >> >> >> >>
>> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
>> >>> > > org.apache.zookeeper.server.Request:
>> >>> > > >> >> >> Ignoring
>> >>> > > >> >> >> >> exception during toString
>> >>> > > >> >> >> >> java.nio.BufferUnderflowException
>> >>> > > >> >> >> >> at
>> >>> > java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
>> >>> > > >> >> >> >> at java.nio.ByteBuffer.get(ByteBuffer.java:675)
>> >>> > > >> >> >> >> at
>> >>> > > >> >> >>
>> >>> org.apache.zookeeper.server.Request.toString(Request.java:199)
>> >>> > > >> >> >> >> at java.lang.String.valueOf(String.java:2827)
>> >>> > > >> >> >> >> at
>> >>> > > java.lang.StringBuilder.append(StringBuilder.java:115)
>> >>> > > >> >> >> >> at
>> >>> > > >> >> >> >>
>> >>> > > >> >> >>
>> >>> > > >> >>
>> >>> > > >>
>> >>> > >
>> >>> >
>> >>>
>> org.apache.zookeeper.server.quorum.CommitProcessor.processRequest(CommitProcessor.java:167)
>> >>> > > >> >> >> >> at
>> >>> > > >> >> >> >>
>> >>> > > >> >> >>
>> >>> > > >> >>
>> >>> > > >>
>> >>> > >
>> >>> >
>> >>>
>> org.apache.zookeeper.server.quorum.FollowerRequestProcessor.run(FollowerRequestProcessor.java:68)
>> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
>> >>> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
>> >>> > > >> >> >> >> Exception causing close of session 0x424dfd908c50009
>> >>> > > >> >> >> >> due to java.io.IOException: Read error
>> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
>> >>> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
>> >>> > > >> >> >> >> IOException stack trace
>> >>> > > >> >> >> >> java.io.IOException: Read error
>> >>> > > >> >> >> >> at
>> >>> > > >> >> >> >>
>> >>> > > >> >>
>> >>> > >
>> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:372)
>> >>> > > >> >> >> >> at
>> >>> > > >> >> >> >>
>> >>> > > >> >> >>
>> >>> > > >> >>
>> >>> > > >>
>> >>> > >
>> >>> >
>> >>>
>> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:221)
>> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
>> >>> > > >> >> >> >> org.apache.zookeeper.server.quorum.CommitProcessor:
>> >>> > Processing
>> >>> > > >> >> request::
>> >>> > > >> >> >> >> sessionid:0x424dfd908c50009 type:setWatches
>> >>> > > >> cxid:0xfffffffffffffff8
>> >>> > > >> >> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
>> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
>> >>> > > org.apache.zookeeper.server.Request:
>> >>> > > >> >> >> Ignoring
>> >>> > > >> >> >> >> exception during toString
>> >>> > > >> >> >> >> java.nio.BufferUnderflowException
>> >>> > > >> >> >> >> at
>> >>> > java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
>> >>> > > >> >> >> >> at java.nio.ByteBuffer.get(ByteBuffer.java:675)
>> >>> > > >> >> >> >> at
>> >>> > > >> >> >>
>> >>> org.apache.zookeeper.server.Request.toString(Request.java:199)
>> >>> > > >> >> >> >> at java.lang.String.valueOf(String.java:2827)
>> >>> > > >> >> >> >> at
>> >>> > > java.lang.StringBuilder.append(StringBuilder.java:115)
>> >>> > > >> >> >> >> at
>> >>> > > >> >> >> >>
>> >>> > > >> >> >>
>> >>> > > >> >>
>> >>> > > >>
>> >>> > >
>> >>> >
>> >>>
>> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:74)
>> >>> > > >> >> >> >> at
>> >>> > > >> >> >> >>
>> >>> > > >> >> >>
>> >>> > > >> >>
>> >>> > > >>
>> >>> > >
>> >>> >
>> >>>
>> org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)
>> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
>> >>> > > >> >> >> >> org.apache.zookeeper.server.FinalRequestProcessor:
>> >>> Processing
>> >>> > > >> >> request::
>> >>> > > >> >> >> >> sessionid:0x424dfd908c50009 type:setWatches
>> >>> > > >> cxid:0xfffffffffffffff8
>> >>> > > >> >> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
>> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
>> >>> > > org.apache.zookeeper.server.Request:
>> >>> > > >> >> >> Ignoring
>> >>> > > >> >> >> >> exception during toString
>> >>> > > >> >> >> >> java.nio.BufferUnderflowException
>> >>> > > >> >> >> >> at
>> >>> > java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
>> >>> > > >> >> >> >> at java.nio.ByteBuffer.get(ByteBuffer.java:675)
>> >>> > > >> >> >> >> at
>> >>> > > >> >> >>
>> >>> org.apache.zookeeper.server.Request.toString(Request.java:199)
>> >>> > > >> >> >> >> at
>> >>> > > >> >> >> >>
>> >>> > > >>
>> >>> org.apache.log4j.or.DefaultRenderer.doRender(DefaultRenderer.java:36)
>> >>> > > >> >> >> >> at
>> >>> > > >> >> >> >>
>> >>> > > org.apache.log4j.or.RendererMap.findAndRender(RendererMap.java:80)
>> >>> > > >> >> >> >> at
>> >>> > > >> >> >> >>
>> >>> > > >> >> >>
>> >>> > > >> >>
>> >>> > > >>
>> >>> > >
>> >>> >
>> >>>
>> org.apache.log4j.spi.LoggingEvent.getRenderedMessage(LoggingEvent.java:362)
>> >>> > > >> >> >> >> at
>> >>> > > >> >> >> >>
>> >>> > > >> >> >>
>> >>> > > >> >>
>> >>> > > >>
>> >>> > >
>> >>> >
>> >>>
>> org.apache.log4j.helpers.PatternParser$BasicPatternConverter.convert(PatternParser.java:403)
>> >>> > > >> >> >> >> at
>> >>> > > >> >> >> >>
>> >>> > > >> >> >>
>> >>> > > >> >>
>> >>> > > >>
>> >>> > >
>> >>> >
>> >>>
>> org.apache.log4j.helpers.PatternConverter.format(PatternConverter.java:65)
>> >>> > > >> >> >> >> at
>> >>> > > >> >> org.apache.log4j.PatternLayout.format(PatternLayout.java:502)
>> >>> > > >> >> >> >> at
>> >>> > > >> >> >> >>
>> >>> > > org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:302)
>> >>> > > >> >> >> >> at
>> >>> > > >> >> >> >>
>> >>> > > >> >> >>
>> >>> > > >> >>
>> >>> > > >>
>> >>> > >
>> >>> >
>> >>>
>> org.apache.log4j.DailyRollingFileAppender.subAppend(DailyRollingFileAppender.java:359)
>> >>> > > >> >> >> >> at
>> >>> > > >> >> >>
>> >>> org.apache.log4j.WriterAppender.append(WriterAppender.java:160)
>> >>> > > >> >> >> >> at
>> >>> > > >> >> >> >>
>> >>> > > >>
>> >>> org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
>> >>> > > >> >> >> >> at
>> >>> > > >> >> >> >>
>> >>> > > >> >> >>
>> >>> > > >> >>
>> >>> > > >>
>> >>> > >
>> >>> >
>> >>>
>> org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
>> >>> > > >> >> >> >> at
>> >>> > > >> org.apache.log4j.Category.callAppenders(Category.java:206)
>> >>> > > >> >> >> >> at
>> >>> > > org.apache.log4j.Category.forcedLog(Category.java:391)
>> >>> > > >> >> >> >> at
>> >>> org.apache.log4j.Category.debug(Category.java:260)
>> >>> > > >> >> >> >> at
>> >>> > > >> >> >> >>
>> >>> > > >> >> >>
>> >>> > > >> >>
>> >>> > > >>
>> >>> > >
>> >>> >
>> >>>
>> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:147)
>> >>> > > >> >> >> >> at
>> >>> > > >> >> >> >>
>> >>> > > >> >> >>
>> >>> > > >> >>
>> >>> > > >>
>> >>> > >
>> >>> >
>> >>>
>> org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)
>> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
>> >>> > > org.apache.zookeeper.server.Request:
>> >>> > > >> >> >> Ignoring
>> >>> > > >> >> >> >> exception during toString
>> >>> > > >> >> >> >> sessionid:0x424dfd908c50009 type:setWatches
>> >>> > > >> cxid:0xfffffffffffffff8
>> >>> > > >> >> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
>> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 INFO
>> >>> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
>> >>> > > >> >> >> >> closing session:0x424dfd908c50009 NIOServerCnxn:
>> >>> > > >> >> >> >> java.nio.channels.SocketChannel[connected local=/
>> >>> > > >> >> 192.168.100.131:2181
>> >>> > > >> >> >> remote=/
>> >>> > > >> >> >> >> 192.168.100.131:35500]
>> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
>> >>> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
>> >>> > > >> >> >> >> ignoring exception during input shutdown
>> >>> > > >> >> >> >> java.net.SocketException: Transport endpoint is not
>> >>> connected
>> >>> > > >> >> >> >> at sun.nio.ch.SocketChannelImpl.shutdown(Native
>> >>> > Method)
>> >>> > > >> >> >> >> at
>> >>> > > >> >> >> >>
>> >>> > > >> >>
>> >>> > >
>> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
>> >>> > > >> >> >> >> at
>> >>> > > >> >> >>
>> >>> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>> >>> > > >> >> >> >> at
>> >>> > > >> >> >> >>
>> >>> > > >> >>
>> >>> > >
>> >>> org.apache.zookeeper.server.NIOServerCnxn.close(NIOServerCnxn.java:848)
>> >>> > > >> >> >> >> at
>> >>> > > >> >> >> >>
>> >>> > > >> >>
>> >>> > >
>> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:498)
>> >>> > > >> >> >> >> at
>> >>> > > >> >> >> >>
>> >>> > > >> >> >>
>> >>> > > >> >>
>> >>> > > >>
>> >>> > >
>> >>> >
>> >>>
>> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:221)
>> >>> > > >> >> >> >>
>> >>> > > >> >> >> >
>> >>> > > >> >> >>
>> >>> > > >> >> >
>> >>> > > >> >>
>> >>> > > >> >
>> >>> > > >>
>> >>> > > >
>> >>> > >
>> >>> >
>> >>>
>> >>
>> >>
>> >
>>
>
Re: regionserver disconnection
Posted by Zhenyu Zhong <zh...@gmail.com>.
J-D,
Yes, that is weird. But it is what I have in the log. Probably the log is
flushed by different threads, and one thread is blocked for a while?
I set the zookeeper.session.timout=600000, which is 10 minutes. However, we
still see this problem.
Does GC pause for such a long time period?
I will try different GC options to see the differences.
thanks
zhenyu
On Thu, Nov 12, 2009 at 1:10 PM, Jean-Daniel Cryans <jd...@apache.org>wrote:
> I see something weird
>
> 2009-11-12 15:04:36,370 WARN
> org.apache.hadoop.hbase.regionserver.HRegionServer: Attempt=1
> org.apache.hadoop.hbase.Leases$LeaseStillHeldException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> ...
> 2009-11-12 15:03:52,682 DEBUG org.apache.zookeeper.ClientCnxn: Got
> ping response for sessionid:0x224e55436ad0004 after 0ms
> 2009-11-12 15:04:52,480 WARN org.apache.zookeeper.ClientCnxn:
> Exception closing session 0x24e55436a0007d to
> sun.nio.ch.SelectionKeyImpl@136f6ed1
>
> Goes back in time? I does look better tho. Did you say you set the
> zookeeper session timeout higher in a previous thread? I'd be
> interested to see if a higher (2-3 minutes) value would solve the
> problem until java 7.
>
> The only thing that happens after that LeaseStillHeldException is that
> for some reason (probably GC) the region server paused. This caused
> the session to timeout and should be why the RS isn't connected to the
> ZK quorum member.
>
> BTW we worked with the Zookeeper dev team to have more meaningful
> exceptions and it's already committed in their trunk.
>
> J-D
>
> On Thu, Nov 12, 2009 at 8:40 AM, Zhenyu Zhong <zh...@gmail.com>
> wrote:
> > Follow up on the regionserver disconnection:
> >
> > I made a few changes to the cluster setup.
> > 1. split the regionserver and zookeeper quorum servers, such that HRegion
> > doesn't run on the same box as HQuorumPeer. I assume that there should be
> no
> > disk io contention between the regionserver and zookeeper.
> > Now we have 14 regionservers, and 5 zookeeper quorum servers.
> >
> > Though I experienced 2 regionserver disconnection this morning, it looks
> > better from the regionserver log. (please see the following log)
> >
> >
> > http://pastebin.com/m496dbfae
> >
> >
> > I checked diskIO for one of the regionserver(192.168.100.116) during the
> > time it disconnected.
> >
> > Time: 03:04:58 AM
> > Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
> > sda 105.31 5458.83 19503.64 9043873239 32312473676
> > sda1 2.90 6.64 99.25 10993934 164433464
> > sda2 1.72 23.76 30.25 39365817 50124008
> > sda3 0.30 0.38 3.58 624930 5923000
> > sda4 100.39 5428.06 19370.56 8992888270 32091993204
> >
> >
> > I also checked the zookeeper quorum server that the regionserver tried to
> > connect according to the log. However, I don't see 192.168.100.116 in the
> > client list of the zookeeper quorum server that the regionserver tried to
> > connect.
> > Would that be a problem?
> >
> > Thu Nov 12 15:04:52 UTC 2009
> > Zookeeper version: 3.2.1-808558, built on 08/27/2009 18:48 GMT
> > Clients:
> > /192.168.100.127:43045[1](queued=0,recved=26,sent=0)
> > /192.168.100.131:39091[1](queued=0,recved=964,sent=0)
> > /192.168.100.124:35961[1](queued=0,recved=3266,sent=0)
> > /192.168.100.123:47935[1](queued=0,recved=233,sent=0)
> > /192.168.100.125:46931[1](queued=0,recved=2,sent=0)
> > /192.168.100.118:54924[1](queued=0,recved=295,sent=0)
> > /192.168.100.118:41390[1](queued=0,recved=2290,sent=0)
> > /192.168.100.136:42243[1](queued=0,recved=0,sent=0)
> >
> > Latency min/avg/max: 0/17/6333
> > Received: 47111
> > Sent: 0
> > Outstanding: 0
> > Zxid: 0x77000083f4
> > Mode: leader
> > Node count: 23
> >
> > Best,
> > zhenyu
> >
> >
> > On Wed, Nov 11, 2009 at 3:58 PM, Zhenyu Zhong <zhongresearch@gmail.com
> >wrote:
> >
> >> Stack
> >>
> >> I am very appreciated for your comments.
> >> I will use the zookeeper monitoring script on my cluster and let it run
> >> overnight to see the result.
> >> I will follow up the thread when I get anything.
> >>
> >> thanks
> >> zhenyu
> >>
> >>
> >> On Wed, Nov 11, 2009 at 3:52 PM, stack <st...@duboce.net> wrote:
> >>
> >>> I see these in your log too:
> >>>
> >>>
> >>> 1. 2009-11-11 04:27:19,018 DEBUG org.apache.zookeeper.ClientCnxn: Got
> >>> ping response for sessionid:0x424dfd908c50009 after 4544ms
> >>> 2. 2009-11-11 04:27:19,018 DEBUG org.apache.zookeeper.ClientCnxn: Got
> >>> ping response for sessionid:0x24dfd90c810002 after 4548ms
> >>> 3. 2009-11-11 04:27:43,960 DEBUG org.apache.zookeeper.ClientCnxn: Got
> >>> ping response for sessionid:0x424dfd908c50009 after 9030ms
> >>> 4. 2009-11-11 04:27:43,960 DEBUG org.apache.zookeeper.ClientCnxn: Got
> >>> ping response for sessionid:0x24dfd90c810002 after 9030ms
> >>> 5. 2009-11-11 04:27:48,830 DEBUG org.apache.zookeeper.ClientCnxn: Got
> >>> ping response for sessionid:0x424dfd908c50009 after 0ms
> >>> 6. 2009-11-11 04:27:48,830 DEBUG org.apache.zookeeper.ClientCnxn: Got
> >>> ping response for sessionid:0x24dfd90c810002 after 0ms
> >>>
> >>> Which would indicate lag talking to zk too.
> >>>
> >>> Appending HLog also seems to get steadily worse.
> >>>
> >>> I also see these:
> >>>
> >>>
> >>> 1. 2009-11-11 04:27:01,233 WARN org.apache.zookeeper.ClientCnxn:
> >>> Exception closing session 0x24dfd90c810002 to
> >>> sun.nio.ch.SelectionKeyImpl@5e16a7e3
> >>> 2. java.io.IOException: Read error rc = -1
> >>> java.nio.DirectByteBuffer[pos=0 lim=4 cap=4]
> >>> 3. at
> >>> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
> >>> 4. at
> >>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
> >>>
> >>>
> >>> Were all 5 quorum members participating? Check by doing "echo stat|nc
> >>> QUORUM_HOST port" for each member of quorum.... the below might help:
> >>>
> >>> members="1,2,3,4,5"
> >>> while true; do date; for i in $members; do echo $i; echo stat | nc $i
> >>> 2181;
> >>> done; sleep 10; done
> >>>
> >>> We need to improve our zk_dump. It just reports quorum members... not
> if
> >>> they are actually participating or not. The above will tell you if
> dead
> >>> or
> >>> alive.
> >>>
> >>> Please report back after checking above and following the Jean-Daniel
> >>> prescription.
> >>>
> >>> Thanks,
> >>> St.Ack
> >>>
> >>>
> >>>
> >>> On Wed, Nov 11, 2009 at 12:29 PM, Zhenyu Zhong <
> zhongresearch@gmail.com
> >>> >wrote:
> >>>
> >>> > Thanks a lot for the help.
> >>> >
> >>> > I understand that the change of disk configurations should be done
> >>> > progressively such that HDFS can balance and recover from a disk
> going
> >>> > away.
> >>> >
> >>> > The data we are trying to import into HBase is about 200million rows,
> >>> each
> >>> > row is about hundreds of bytes. Basically, we read in a text file
> from
> >>> > HDFS,
> >>> > then for each row in the text file, we fetch data from HBase table,
> we
> >>> > compare the column value and update the latest value back into HBase.
> I
> >>> was
> >>> > suggested to use CheckAndPut. Anyway this is the common usecase on
> my
> >>> > side.
> >>> >
> >>> >
> >>> > Many thanks
> >>> > zhenyu
> >>> >
> >>> >
> >>> >
> >>> > On Wed, Nov 11, 2009 at 3:03 PM, Jean-Daniel Cryans <
> >>> jdcryans@apache.org
> >>> > >wrote:
> >>> >
> >>> > > 6 disks, 19 machines and you see those kind of issues? I'm quite
> >>> > > surprised and I want to learn more about what you trying to to.
> >>> > >
> >>> > > But first, if hbase is managing ZK for you then you can change the
> >>> > > value of hbase.zookeeper.property.dataDir so that it points to its
> own
> >>> > > mount. Make sure you also change dfs.data.dir in conf/hdfs-site.xml
> >>> > > (hadoop stuff) so that it stops pointing there (you should have 6
> >>> > > values there at the moment if you configured it correctly). Doing
> this
> >>> > > could result in data loss if you do it all at once so do the change
> >>> > > progressively. This also means that you may have to keep a seperate
> >>> > > HDFS configuration for those particular nodes. Please ask more
> >>> > > questions if it's not very clear.
> >>> > >
> >>> > > WRT to dedicating nodes to Zookeeeper, it makes sense only if you
> have
> >>> > > a lot of nodes but in your case, unless there's also CPU
> contention,
> >>> > > you're good with only a dedicated disk.
> >>> > >
> >>> > > Now for what you are trying to do, is this a massive import that
> goes
> >>> > > over many hours? How big is each row? I'd love to get a lot of
> details
> >>> > > to help you optimize your cluster for your particular problem.
> >>> > >
> >>> > > Thx,
> >>> > >
> >>> > > J-D
> >>> > >
> >>> > > On Wed, Nov 11, 2009 at 11:51 AM, Zhenyu Zhong <
> >>> zhongresearch@gmail.com>
> >>> > > wrote:
> >>> > > > J-D,
> >>> > > >
> >>> > > > Thanks a lot for the insightful comments.
> >>> > > > We are running 5 zookeeper quorums now. On these 5 quorum
> servers,
> >>> we
> >>> > > also
> >>> > > > run datanode, tasktracker and regionserver.
> >>> > > > Now we have 6 disks per machine.
> >>> > > > If it is the disk IO saturation, is there anyway to configure
> >>> zookeeper
> >>> > > to
> >>> > > > write to a seperate disk which is not the same one used by HDFS?
> >>> > > > Or shall we run zookeeper quorums on different nodes other than
> >>> HDFS?
> >>> > > >
> >>> > > > Are HLogs generated by Regionservers? Would it alleviate the
> issue
> >>> if
> >>> > we
> >>> > > use
> >>> > > > the first 14 for regionserver, and the rest of 5 for
> zookeeper?(In
> >>> that
> >>> > > > sense, the node running zookeeper quorum won't run regionserver)
> >>> > > >
> >>> > > > I am very appreciated for your comments.
> >>> > > >
> >>> > > > Best,
> >>> > > > zhenyu
> >>> > > >
> >>> > > >
> >>> > > >
> >>> > > > On Wed, Nov 11, 2009 at 2:30 PM, Jean-Daniel Cryans <
> >>> > jdcryans@apache.org
> >>> > > >wrote:
> >>> > > >
> >>> > > >> The problem seems to be that the IO is saturated. When you see
> >>> this:
> >>> > > >>
> >>> > > >> #
> >>> > > >> 2009-11-11 04:28:10,308 WARN
> >>> > > >> org.apache.hadoop.hbase.regionserver.HLog: IPC Server handler 9
> on
> >>> > > >> 60021 took 6819ms appending an edit to hlog; editcount=370
> >>> > > >> #
> >>> > > >> 2009-11-11 04:28:10,308 WARN
> >>> > > >> org.apache.hadoop.hbase.regionserver.HLog: IPC Server handler 2
> on
> >>> > > >> 60021 took 4412ms appending an edit to hlog; editcount=371
> >>> > > >>
> >>> > > >> It's a sign that HDFS is slow. I guess that the zookeeper quorum
> >>> > > >> members are on those same nodes using the same disks? If so,
> >>> there's
> >>> > > >> the problem because Zookeeper needs to be able to commit
> >>> transactions
> >>> > > >> very fast and if your disks are already hogged then they have to
> >>> wait
> >>> > > >> and you can get disconnections. Do you still have 19 quorum
> >>> members?
> >>> > > >> How many disks do you have per machine?
> >>> > > >>
> >>> > > >> Thx
> >>> > > >>
> >>> > > >> J-D
> >>> > > >>
> >>> > > >> On Wed, Nov 11, 2009 at 11:22 AM, Zhenyu Zhong <
> >>> > zhongresearch@gmail.com
> >>> > > >
> >>> > > >> wrote:
> >>> > > >> > Please take a look at the following regionserver log.
> >>> > > >> > The error started to come out from 4:24am UTC to 4:29am UTC.
> >>> > > >> >
> >>> > > >> > http://pastebin.com/m3717d064
> >>> > > >> >
> >>> > > >> > Thanks in advance.
> >>> > > >> >
> >>> > > >> > Best,
> >>> > > >> > zhenyu
> >>> > > >> >
> >>> > > >> >
> >>> > > >> > On Wed, Nov 11, 2009 at 2:01 PM, Jean-Daniel Cryans <
> >>> > > jdcryans@apache.org
> >>> > > >> >wrote:
> >>> > > >> >
> >>> > > >> >> Could you paste a bigger log trace (interested to see at
> least
> >>> all
> >>> > > 300
> >>> > > >> >> lines before that) in a pastebin?
> >>> > > >> >>
> >>> > > >> >> Thx,
> >>> > > >> >>
> >>> > > >> >> J-D
> >>> > > >> >>
> >>> > > >> >> On Wed, Nov 11, 2009 at 10:56 AM, Zhenyu Zhong <
> >>> > > zhongresearch@gmail.com
> >>> > > >> >
> >>> > > >> >> wrote:
> >>> > > >> >> > J-D,
> >>> > > >> >> >
> >>> > > >> >> > Thanks for the reply.
> >>> > > >> >> >
> >>> > > >> >> > The problem I have is one or more of the regionservers
> >>> disconnect
> >>> > > from
> >>> > > >> >> the
> >>> > > >> >> > cluster.
> >>> > > >> >> >
> >>> > > >> >> > I checked the resource usage including memory Heap usage
> for
> >>> > > >> >> regionserver,
> >>> > > >> >> > and found nothing suspicious. The regionserver heap usage
> is
> >>> way
> >>> > > below
> >>> > > >> >> the
> >>> > > >> >> > maxHeap defined in jvm.
> >>> > > >> >> >
> >>> > > >> >> > So I checked zookeeper log, then I found the following
> >>> > > >> >> > BufferUnderflowException, eventually it leads to
> >>> SocketException:
> >>> > > >> >> Transport
> >>> > > >> >> > endpoint is not connected.
> >>> > > >> >> >
> >>> > > >> >> >
> >>> > > >> >> > I also checked the regionserver log. It looks like the
> >>> > regionserver
> >>> > > >> tried
> >>> > > >> >> to
> >>> > > >> >> > connect to other zookeeper quorum server. It connects
> >>> > successfully,
> >>> > > >> but
> >>> > > >> >> then
> >>> > > >> >> > it complained about SocketException: transport endpoint is
> not
> >>> > > >> connected.
> >>> > > >> >> >
> >>> > > >> >> > So both zookeeper log and regionserver log told me that the
> >>> > > transport
> >>> > > >> >> > endpoint is not connected. I don't know why.
> >>> > > >> >> >
> >>> > > >> >> > I would be very appreciated if you could point me to the
> right
> >>> > > >> direction.
> >>> > > >> >> >
> >>> > > >> >> > Best,
> >>> > > >> >> >
> >>> > > >> >> >
> >>> > > >> >> > Regionserver log:
> >>> > > >> >> > 2009-11-11 04:24:53,718 INFO
> org.apache.zookeeper.ClientCnxn:
> >>> > > >> Attempting
> >>> > > >> >> > connection to server superpyxis0018/192.168.100.132:2181
> >>> > > >> >> > 2009-11-11 04:24:53,718 INFO
> org.apache.zookeeper.ClientCnxn:
> >>> > > Priming
> >>> > > >> >> > connection to java.nio.channels.SocketChannel[connected
> >>> local=/
> >>> > > >> >> > 192.168.100.131:43065 remote=superpyxis0018/
> >>> 192.168.100.132:2181
> >>> > ]
> >>> > > >> >> > 2009-11-11 04:24:53,718 INFO
> org.apache.zookeeper.ClientCnxn:
> >>> > > Server
> >>> > > >> >> > connection successful
> >>> > > >> >> > 2009-11-11 04:24:53,768 INFO
> >>> > > >> >> > org.apache.hadoop.hbase.regionserver.HRegionServer: Got
> >>> ZooKeeper
> >>> > > >> event,
> >>> > > >> >> > state: Disconnected, type: None, path: null
> >>> > > >> >> > 2009-11-11 04:24:54,118 INFO
> org.apache.zookeeper.ClientCnxn:
> >>> > > >> Attempting
> >>> > > >> >> > connection to server superpyxis0017/192.168.100.131:2181
> >>> > > >> >> > 2009-11-11 04:24:54,118 INFO
> org.apache.zookeeper.ClientCnxn:
> >>> > > Priming
> >>> > > >> >> > connection to java.nio.channels.SocketChannel[connected
> >>> local=/
> >>> > > >> >> > 192.168.100.131:46903 remote=superpyxis0017/
> >>> 192.168.100.131:2181
> >>> > ]
> >>> > > >> >> > 2009-11-11 04:24:54,118 INFO
> org.apache.zookeeper.ClientCnxn:
> >>> > > Server
> >>> > > >> >> > connection successful
> >>> > > >> >> > 2009-11-11 04:24:54,118 WARN
> org.apache.zookeeper.ClientCnxn:
> >>> > > >> Exception
> >>> > > >> >> > closing session 0x24dfd90c810002 to
> >>> > > >> sun.nio.ch.SelectionKeyImpl@2dd78ccf
> >>> > > >> >> > java.io.IOException: Read error rc = -1
> >>> > > >> java.nio.DirectByteBuffer[pos=0
> >>> > > >> >> > lim=4 cap=4]
> >>> > > >> >> > at
> >>> > > >> >> >
> >>> > >
> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
> >>> > > >> >> > at
> >>> > > >> >> >
> >>> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
> >>> > > >> >> > 2009-11-11 04:24:54,118 WARN
> org.apache.zookeeper.ClientCnxn:
> >>> > > Ignoring
> >>> > > >> >> > exception during shutdown input
> >>> > > >> >> > java.net.SocketException: Transport endpoint is not
> connected
> >>> > > >> >> > at sun.nio.ch.SocketChannelImpl.shutdown(Native
> Method)
> >>> > > >> >> > at
> >>> > > >> >> >
> >>> > >
> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
> >>> > > >> >> > at
> >>> > > >> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> >>> > > >> >> > at
> >>> > > >> >> >
> >>> > > >>
> >>> >
> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> >>> > > >> >> > at
> >>> > > >> >> >
> >>> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> >>> > > >> >> >
> >>> > > >> >> >
> >>> > > >> >> >
> >>> > > >> >> >
> >>> > > >> >> >
> >>> > > >> >> > On Wed, Nov 11, 2009 at 1:40 PM, Jean-Daniel Cryans <
> >>> > > >> jdcryans@apache.org
> >>> > > >> >> >wrote:
> >>> > > >> >> >
> >>> > > >> >> >> I'm really not sure exactly what the situation is with the
> >>> > > >> >> >> java.nio.BufferUnderflowException, but what do you mean by
> >>> > > >> >> >> "regionserver disconnection"? Could you give more details?
> >>> > > >> >> >>
> >>> > > >> >> >> Thx,
> >>> > > >> >> >>
> >>> > > >> >> >> J-D
> >>> > > >> >> >>
> >>> > > >> >> >> On Wed, Nov 11, 2009 at 10:34 AM, Zhenyu Zhong <
> >>> > > >> zhongresearch@gmail.com
> >>> > > >> >> >
> >>> > > >> >> >> wrote:
> >>> > > >> >> >> > I searched around and found HBASE-1645 "HBase could be a
> >>> > little
> >>> > > >> more
> >>> > > >> >> >> dainty
> >>> > > >> >> >> > when zk is pulled out from under it"
> >>> > > >> >> >> >
> >>> > > >> >> >> > I am wondering whether my problem is related to this
> >>> > HBase-1645.
> >>> > > Or
> >>> > > >> >> >> probably
> >>> > > >> >> >> > has nothing to do with it?
> >>> > > >> >> >> >
> >>> > > >> >> >> > thanks
> >>> > > >> >> >> >
> >>> > > >> >> >> >
> >>> > > >> >> >> > On Wed, Nov 11, 2009 at 12:02 PM, Zhenyu Zhong <
> >>> > > >> >> zhongresearch@gmail.com
> >>> > > >> >> >> >wrote:
> >>> > > >> >> >> >
> >>> > > >> >> >> >> Hi,
> >>> > > >> >> >> >>
> >>> > > >> >> >> >> I have been monitoring the system resources such as
> memory
> >>> > Heap
> >>> > > >> usage
> >>> > > >> >> >> for
> >>> > > >> >> >> >> Regionserver, load average, network bandwidth, etc.
> >>> > > >> >> >> >>
> >>> > > >> >> >> >> I experienced a disconnection last night. When I
> checked
> >>> the
> >>> > > >> memory
> >>> > > >> >> Heap
> >>> > > >> >> >> >> for Regionserver, it was 2336 which is way below the
> max
> >>> Heap
> >>> > > >> 4096. I
> >>> > > >> >> >> don't
> >>> > > >> >> >> >> think it is Regionserver problem. The requests that the
> >>> > > >> regionserver
> >>> > > >> >> got
> >>> > > >> >> >> was
> >>> > > >> >> >> >> about 126 which doesn't seem to be high. The load
> average
> >>> is
> >>> > > kind
> >>> > > >> of
> >>> > > >> >> >> high
> >>> > > >> >> >> >> >100. (but I saw disconnections while the load average
> is
> >>> > > around
> >>> > > >> 30).
> >>> > > >> >> >> The
> >>> > > >> >> >> >> bandwidth is not a problem.
> >>> > > >> >> >> >>
> >>> > > >> >> >> >> However, I found error messages from the zookeeper.
> Please
> >>> > see
> >>> > > the
> >>> > > >> >> >> >> following error messages. There are some
> >>> > > >> >> >> >> BufferUnderflowException error, IOException error,etc.
> >>> > > >> >> >> >> May I ask what might be the causes that generate these
> >>> error
> >>> > > >> >> messages?
> >>> > > >> >> >> >>
> >>> > > >> >> >> >> Many thanks
> >>> > > >> >> >> >> zhenyu
> >>> > > >> >> >> >>
> >>> > > >> >> >> >>
> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
> >>> > > org.apache.zookeeper.server.Request:
> >>> > > >> >> >> Ignoring
> >>> > > >> >> >> >> exception during toString
> >>> > > >> >> >> >> java.nio.BufferUnderflowException
> >>> > > >> >> >> >> at
> >>> > java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
> >>> > > >> >> >> >> at java.nio.ByteBuffer.get(ByteBuffer.java:675)
> >>> > > >> >> >> >> at
> >>> > > >> >> >>
> >>> org.apache.zookeeper.server.Request.toString(Request.java:199)
> >>> > > >> >> >> >> at java.lang.String.valueOf(String.java:2827)
> >>> > > >> >> >> >> at
> >>> > > java.lang.StringBuilder.append(StringBuilder.java:115)
> >>> > > >> >> >> >> at
> >>> > > >> >> >> >>
> >>> > > >> >> >>
> >>> > > >> >>
> >>> > > >>
> >>> > >
> >>> >
> >>>
> org.apache.zookeeper.server.quorum.CommitProcessor.processRequest(CommitProcessor.java:167)
> >>> > > >> >> >> >> at
> >>> > > >> >> >> >>
> >>> > > >> >> >>
> >>> > > >> >>
> >>> > > >>
> >>> > >
> >>> >
> >>>
> org.apache.zookeeper.server.quorum.FollowerRequestProcessor.run(FollowerRequestProcessor.java:68)
> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
> >>> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
> >>> > > >> >> >> >> Exception causing close of session 0x424dfd908c50009
> >>> > > >> >> >> >> due to java.io.IOException: Read error
> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
> >>> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
> >>> > > >> >> >> >> IOException stack trace
> >>> > > >> >> >> >> java.io.IOException: Read error
> >>> > > >> >> >> >> at
> >>> > > >> >> >> >>
> >>> > > >> >>
> >>> > >
> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:372)
> >>> > > >> >> >> >> at
> >>> > > >> >> >> >>
> >>> > > >> >> >>
> >>> > > >> >>
> >>> > > >>
> >>> > >
> >>> >
> >>>
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:221)
> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
> >>> > > >> >> >> >> org.apache.zookeeper.server.quorum.CommitProcessor:
> >>> > Processing
> >>> > > >> >> request::
> >>> > > >> >> >> >> sessionid:0x424dfd908c50009 type:setWatches
> >>> > > >> cxid:0xfffffffffffffff8
> >>> > > >> >> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
> >>> > > org.apache.zookeeper.server.Request:
> >>> > > >> >> >> Ignoring
> >>> > > >> >> >> >> exception during toString
> >>> > > >> >> >> >> java.nio.BufferUnderflowException
> >>> > > >> >> >> >> at
> >>> > java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
> >>> > > >> >> >> >> at java.nio.ByteBuffer.get(ByteBuffer.java:675)
> >>> > > >> >> >> >> at
> >>> > > >> >> >>
> >>> org.apache.zookeeper.server.Request.toString(Request.java:199)
> >>> > > >> >> >> >> at java.lang.String.valueOf(String.java:2827)
> >>> > > >> >> >> >> at
> >>> > > java.lang.StringBuilder.append(StringBuilder.java:115)
> >>> > > >> >> >> >> at
> >>> > > >> >> >> >>
> >>> > > >> >> >>
> >>> > > >> >>
> >>> > > >>
> >>> > >
> >>> >
> >>>
> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:74)
> >>> > > >> >> >> >> at
> >>> > > >> >> >> >>
> >>> > > >> >> >>
> >>> > > >> >>
> >>> > > >>
> >>> > >
> >>> >
> >>>
> org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)
> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
> >>> > > >> >> >> >> org.apache.zookeeper.server.FinalRequestProcessor:
> >>> Processing
> >>> > > >> >> request::
> >>> > > >> >> >> >> sessionid:0x424dfd908c50009 type:setWatches
> >>> > > >> cxid:0xfffffffffffffff8
> >>> > > >> >> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
> >>> > > org.apache.zookeeper.server.Request:
> >>> > > >> >> >> Ignoring
> >>> > > >> >> >> >> exception during toString
> >>> > > >> >> >> >> java.nio.BufferUnderflowException
> >>> > > >> >> >> >> at
> >>> > java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
> >>> > > >> >> >> >> at java.nio.ByteBuffer.get(ByteBuffer.java:675)
> >>> > > >> >> >> >> at
> >>> > > >> >> >>
> >>> org.apache.zookeeper.server.Request.toString(Request.java:199)
> >>> > > >> >> >> >> at
> >>> > > >> >> >> >>
> >>> > > >>
> >>> org.apache.log4j.or.DefaultRenderer.doRender(DefaultRenderer.java:36)
> >>> > > >> >> >> >> at
> >>> > > >> >> >> >>
> >>> > > org.apache.log4j.or.RendererMap.findAndRender(RendererMap.java:80)
> >>> > > >> >> >> >> at
> >>> > > >> >> >> >>
> >>> > > >> >> >>
> >>> > > >> >>
> >>> > > >>
> >>> > >
> >>> >
> >>>
> org.apache.log4j.spi.LoggingEvent.getRenderedMessage(LoggingEvent.java:362)
> >>> > > >> >> >> >> at
> >>> > > >> >> >> >>
> >>> > > >> >> >>
> >>> > > >> >>
> >>> > > >>
> >>> > >
> >>> >
> >>>
> org.apache.log4j.helpers.PatternParser$BasicPatternConverter.convert(PatternParser.java:403)
> >>> > > >> >> >> >> at
> >>> > > >> >> >> >>
> >>> > > >> >> >>
> >>> > > >> >>
> >>> > > >>
> >>> > >
> >>> >
> >>>
> org.apache.log4j.helpers.PatternConverter.format(PatternConverter.java:65)
> >>> > > >> >> >> >> at
> >>> > > >> >> org.apache.log4j.PatternLayout.format(PatternLayout.java:502)
> >>> > > >> >> >> >> at
> >>> > > >> >> >> >>
> >>> > > org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:302)
> >>> > > >> >> >> >> at
> >>> > > >> >> >> >>
> >>> > > >> >> >>
> >>> > > >> >>
> >>> > > >>
> >>> > >
> >>> >
> >>>
> org.apache.log4j.DailyRollingFileAppender.subAppend(DailyRollingFileAppender.java:359)
> >>> > > >> >> >> >> at
> >>> > > >> >> >>
> >>> org.apache.log4j.WriterAppender.append(WriterAppender.java:160)
> >>> > > >> >> >> >> at
> >>> > > >> >> >> >>
> >>> > > >>
> >>> org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
> >>> > > >> >> >> >> at
> >>> > > >> >> >> >>
> >>> > > >> >> >>
> >>> > > >> >>
> >>> > > >>
> >>> > >
> >>> >
> >>>
> org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
> >>> > > >> >> >> >> at
> >>> > > >> org.apache.log4j.Category.callAppenders(Category.java:206)
> >>> > > >> >> >> >> at
> >>> > > org.apache.log4j.Category.forcedLog(Category.java:391)
> >>> > > >> >> >> >> at
> >>> org.apache.log4j.Category.debug(Category.java:260)
> >>> > > >> >> >> >> at
> >>> > > >> >> >> >>
> >>> > > >> >> >>
> >>> > > >> >>
> >>> > > >>
> >>> > >
> >>> >
> >>>
> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:147)
> >>> > > >> >> >> >> at
> >>> > > >> >> >> >>
> >>> > > >> >> >>
> >>> > > >> >>
> >>> > > >>
> >>> > >
> >>> >
> >>>
> org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)
> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
> >>> > > org.apache.zookeeper.server.Request:
> >>> > > >> >> >> Ignoring
> >>> > > >> >> >> >> exception during toString
> >>> > > >> >> >> >> sessionid:0x424dfd908c50009 type:setWatches
> >>> > > >> cxid:0xfffffffffffffff8
> >>> > > >> >> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 INFO
> >>> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
> >>> > > >> >> >> >> closing session:0x424dfd908c50009 NIOServerCnxn:
> >>> > > >> >> >> >> java.nio.channels.SocketChannel[connected local=/
> >>> > > >> >> 192.168.100.131:2181
> >>> > > >> >> >> remote=/
> >>> > > >> >> >> >> 192.168.100.131:35500]
> >>> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
> >>> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
> >>> > > >> >> >> >> ignoring exception during input shutdown
> >>> > > >> >> >> >> java.net.SocketException: Transport endpoint is not
> >>> connected
> >>> > > >> >> >> >> at sun.nio.ch.SocketChannelImpl.shutdown(Native
> >>> > Method)
> >>> > > >> >> >> >> at
> >>> > > >> >> >> >>
> >>> > > >> >>
> >>> > >
> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
> >>> > > >> >> >> >> at
> >>> > > >> >> >>
> >>> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> >>> > > >> >> >> >> at
> >>> > > >> >> >> >>
> >>> > > >> >>
> >>> > >
> >>> org.apache.zookeeper.server.NIOServerCnxn.close(NIOServerCnxn.java:848)
> >>> > > >> >> >> >> at
> >>> > > >> >> >> >>
> >>> > > >> >>
> >>> > >
> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:498)
> >>> > > >> >> >> >> at
> >>> > > >> >> >> >>
> >>> > > >> >> >>
> >>> > > >> >>
> >>> > > >>
> >>> > >
> >>> >
> >>>
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:221)
> >>> > > >> >> >> >>
> >>> > > >> >> >> >
> >>> > > >> >> >>
> >>> > > >> >> >
> >>> > > >> >>
> >>> > > >> >
> >>> > > >>
> >>> > > >
> >>> > >
> >>> >
> >>>
> >>
> >>
> >
>
Re: regionserver disconnection
Posted by Jean-Daniel Cryans <jd...@apache.org>.
I see something weird
2009-11-12 15:04:36,370 WARN
org.apache.hadoop.hbase.regionserver.HRegionServer: Attempt=1
org.apache.hadoop.hbase.Leases$LeaseStillHeldException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
...
2009-11-12 15:03:52,682 DEBUG org.apache.zookeeper.ClientCnxn: Got
ping response for sessionid:0x224e55436ad0004 after 0ms
2009-11-12 15:04:52,480 WARN org.apache.zookeeper.ClientCnxn:
Exception closing session 0x24e55436a0007d to
sun.nio.ch.SelectionKeyImpl@136f6ed1
Goes back in time? I does look better tho. Did you say you set the
zookeeper session timeout higher in a previous thread? I'd be
interested to see if a higher (2-3 minutes) value would solve the
problem until java 7.
The only thing that happens after that LeaseStillHeldException is that
for some reason (probably GC) the region server paused. This caused
the session to timeout and should be why the RS isn't connected to the
ZK quorum member.
BTW we worked with the Zookeeper dev team to have more meaningful
exceptions and it's already committed in their trunk.
J-D
On Thu, Nov 12, 2009 at 8:40 AM, Zhenyu Zhong <zh...@gmail.com> wrote:
> Follow up on the regionserver disconnection:
>
> I made a few changes to the cluster setup.
> 1. split the regionserver and zookeeper quorum servers, such that HRegion
> doesn't run on the same box as HQuorumPeer. I assume that there should be no
> disk io contention between the regionserver and zookeeper.
> Now we have 14 regionservers, and 5 zookeeper quorum servers.
>
> Though I experienced 2 regionserver disconnection this morning, it looks
> better from the regionserver log. (please see the following log)
>
>
> http://pastebin.com/m496dbfae
>
>
> I checked diskIO for one of the regionserver(192.168.100.116) during the
> time it disconnected.
>
> Time: 03:04:58 AM
> Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
> sda 105.31 5458.83 19503.64 9043873239 32312473676
> sda1 2.90 6.64 99.25 10993934 164433464
> sda2 1.72 23.76 30.25 39365817 50124008
> sda3 0.30 0.38 3.58 624930 5923000
> sda4 100.39 5428.06 19370.56 8992888270 32091993204
>
>
> I also checked the zookeeper quorum server that the regionserver tried to
> connect according to the log. However, I don't see 192.168.100.116 in the
> client list of the zookeeper quorum server that the regionserver tried to
> connect.
> Would that be a problem?
>
> Thu Nov 12 15:04:52 UTC 2009
> Zookeeper version: 3.2.1-808558, built on 08/27/2009 18:48 GMT
> Clients:
> /192.168.100.127:43045[1](queued=0,recved=26,sent=0)
> /192.168.100.131:39091[1](queued=0,recved=964,sent=0)
> /192.168.100.124:35961[1](queued=0,recved=3266,sent=0)
> /192.168.100.123:47935[1](queued=0,recved=233,sent=0)
> /192.168.100.125:46931[1](queued=0,recved=2,sent=0)
> /192.168.100.118:54924[1](queued=0,recved=295,sent=0)
> /192.168.100.118:41390[1](queued=0,recved=2290,sent=0)
> /192.168.100.136:42243[1](queued=0,recved=0,sent=0)
>
> Latency min/avg/max: 0/17/6333
> Received: 47111
> Sent: 0
> Outstanding: 0
> Zxid: 0x77000083f4
> Mode: leader
> Node count: 23
>
> Best,
> zhenyu
>
>
> On Wed, Nov 11, 2009 at 3:58 PM, Zhenyu Zhong <zh...@gmail.com>wrote:
>
>> Stack
>>
>> I am very appreciated for your comments.
>> I will use the zookeeper monitoring script on my cluster and let it run
>> overnight to see the result.
>> I will follow up the thread when I get anything.
>>
>> thanks
>> zhenyu
>>
>>
>> On Wed, Nov 11, 2009 at 3:52 PM, stack <st...@duboce.net> wrote:
>>
>>> I see these in your log too:
>>>
>>>
>>> 1. 2009-11-11 04:27:19,018 DEBUG org.apache.zookeeper.ClientCnxn: Got
>>> ping response for sessionid:0x424dfd908c50009 after 4544ms
>>> 2. 2009-11-11 04:27:19,018 DEBUG org.apache.zookeeper.ClientCnxn: Got
>>> ping response for sessionid:0x24dfd90c810002 after 4548ms
>>> 3. 2009-11-11 04:27:43,960 DEBUG org.apache.zookeeper.ClientCnxn: Got
>>> ping response for sessionid:0x424dfd908c50009 after 9030ms
>>> 4. 2009-11-11 04:27:43,960 DEBUG org.apache.zookeeper.ClientCnxn: Got
>>> ping response for sessionid:0x24dfd90c810002 after 9030ms
>>> 5. 2009-11-11 04:27:48,830 DEBUG org.apache.zookeeper.ClientCnxn: Got
>>> ping response for sessionid:0x424dfd908c50009 after 0ms
>>> 6. 2009-11-11 04:27:48,830 DEBUG org.apache.zookeeper.ClientCnxn: Got
>>> ping response for sessionid:0x24dfd90c810002 after 0ms
>>>
>>> Which would indicate lag talking to zk too.
>>>
>>> Appending HLog also seems to get steadily worse.
>>>
>>> I also see these:
>>>
>>>
>>> 1. 2009-11-11 04:27:01,233 WARN org.apache.zookeeper.ClientCnxn:
>>> Exception closing session 0x24dfd90c810002 to
>>> sun.nio.ch.SelectionKeyImpl@5e16a7e3
>>> 2. java.io.IOException: Read error rc = -1
>>> java.nio.DirectByteBuffer[pos=0 lim=4 cap=4]
>>> 3. at
>>> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
>>> 4. at
>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
>>>
>>>
>>> Were all 5 quorum members participating? Check by doing "echo stat|nc
>>> QUORUM_HOST port" for each member of quorum.... the below might help:
>>>
>>> members="1,2,3,4,5"
>>> while true; do date; for i in $members; do echo $i; echo stat | nc $i
>>> 2181;
>>> done; sleep 10; done
>>>
>>> We need to improve our zk_dump. It just reports quorum members... not if
>>> they are actually participating or not. The above will tell you if dead
>>> or
>>> alive.
>>>
>>> Please report back after checking above and following the Jean-Daniel
>>> prescription.
>>>
>>> Thanks,
>>> St.Ack
>>>
>>>
>>>
>>> On Wed, Nov 11, 2009 at 12:29 PM, Zhenyu Zhong <zhongresearch@gmail.com
>>> >wrote:
>>>
>>> > Thanks a lot for the help.
>>> >
>>> > I understand that the change of disk configurations should be done
>>> > progressively such that HDFS can balance and recover from a disk going
>>> > away.
>>> >
>>> > The data we are trying to import into HBase is about 200million rows,
>>> each
>>> > row is about hundreds of bytes. Basically, we read in a text file from
>>> > HDFS,
>>> > then for each row in the text file, we fetch data from HBase table, we
>>> > compare the column value and update the latest value back into HBase. I
>>> was
>>> > suggested to use CheckAndPut. Anyway this is the common usecase on my
>>> > side.
>>> >
>>> >
>>> > Many thanks
>>> > zhenyu
>>> >
>>> >
>>> >
>>> > On Wed, Nov 11, 2009 at 3:03 PM, Jean-Daniel Cryans <
>>> jdcryans@apache.org
>>> > >wrote:
>>> >
>>> > > 6 disks, 19 machines and you see those kind of issues? I'm quite
>>> > > surprised and I want to learn more about what you trying to to.
>>> > >
>>> > > But first, if hbase is managing ZK for you then you can change the
>>> > > value of hbase.zookeeper.property.dataDir so that it points to its own
>>> > > mount. Make sure you also change dfs.data.dir in conf/hdfs-site.xml
>>> > > (hadoop stuff) so that it stops pointing there (you should have 6
>>> > > values there at the moment if you configured it correctly). Doing this
>>> > > could result in data loss if you do it all at once so do the change
>>> > > progressively. This also means that you may have to keep a seperate
>>> > > HDFS configuration for those particular nodes. Please ask more
>>> > > questions if it's not very clear.
>>> > >
>>> > > WRT to dedicating nodes to Zookeeeper, it makes sense only if you have
>>> > > a lot of nodes but in your case, unless there's also CPU contention,
>>> > > you're good with only a dedicated disk.
>>> > >
>>> > > Now for what you are trying to do, is this a massive import that goes
>>> > > over many hours? How big is each row? I'd love to get a lot of details
>>> > > to help you optimize your cluster for your particular problem.
>>> > >
>>> > > Thx,
>>> > >
>>> > > J-D
>>> > >
>>> > > On Wed, Nov 11, 2009 at 11:51 AM, Zhenyu Zhong <
>>> zhongresearch@gmail.com>
>>> > > wrote:
>>> > > > J-D,
>>> > > >
>>> > > > Thanks a lot for the insightful comments.
>>> > > > We are running 5 zookeeper quorums now. On these 5 quorum servers,
>>> we
>>> > > also
>>> > > > run datanode, tasktracker and regionserver.
>>> > > > Now we have 6 disks per machine.
>>> > > > If it is the disk IO saturation, is there anyway to configure
>>> zookeeper
>>> > > to
>>> > > > write to a seperate disk which is not the same one used by HDFS?
>>> > > > Or shall we run zookeeper quorums on different nodes other than
>>> HDFS?
>>> > > >
>>> > > > Are HLogs generated by Regionservers? Would it alleviate the issue
>>> if
>>> > we
>>> > > use
>>> > > > the first 14 for regionserver, and the rest of 5 for zookeeper?(In
>>> that
>>> > > > sense, the node running zookeeper quorum won't run regionserver)
>>> > > >
>>> > > > I am very appreciated for your comments.
>>> > > >
>>> > > > Best,
>>> > > > zhenyu
>>> > > >
>>> > > >
>>> > > >
>>> > > > On Wed, Nov 11, 2009 at 2:30 PM, Jean-Daniel Cryans <
>>> > jdcryans@apache.org
>>> > > >wrote:
>>> > > >
>>> > > >> The problem seems to be that the IO is saturated. When you see
>>> this:
>>> > > >>
>>> > > >> #
>>> > > >> 2009-11-11 04:28:10,308 WARN
>>> > > >> org.apache.hadoop.hbase.regionserver.HLog: IPC Server handler 9 on
>>> > > >> 60021 took 6819ms appending an edit to hlog; editcount=370
>>> > > >> #
>>> > > >> 2009-11-11 04:28:10,308 WARN
>>> > > >> org.apache.hadoop.hbase.regionserver.HLog: IPC Server handler 2 on
>>> > > >> 60021 took 4412ms appending an edit to hlog; editcount=371
>>> > > >>
>>> > > >> It's a sign that HDFS is slow. I guess that the zookeeper quorum
>>> > > >> members are on those same nodes using the same disks? If so,
>>> there's
>>> > > >> the problem because Zookeeper needs to be able to commit
>>> transactions
>>> > > >> very fast and if your disks are already hogged then they have to
>>> wait
>>> > > >> and you can get disconnections. Do you still have 19 quorum
>>> members?
>>> > > >> How many disks do you have per machine?
>>> > > >>
>>> > > >> Thx
>>> > > >>
>>> > > >> J-D
>>> > > >>
>>> > > >> On Wed, Nov 11, 2009 at 11:22 AM, Zhenyu Zhong <
>>> > zhongresearch@gmail.com
>>> > > >
>>> > > >> wrote:
>>> > > >> > Please take a look at the following regionserver log.
>>> > > >> > The error started to come out from 4:24am UTC to 4:29am UTC.
>>> > > >> >
>>> > > >> > http://pastebin.com/m3717d064
>>> > > >> >
>>> > > >> > Thanks in advance.
>>> > > >> >
>>> > > >> > Best,
>>> > > >> > zhenyu
>>> > > >> >
>>> > > >> >
>>> > > >> > On Wed, Nov 11, 2009 at 2:01 PM, Jean-Daniel Cryans <
>>> > > jdcryans@apache.org
>>> > > >> >wrote:
>>> > > >> >
>>> > > >> >> Could you paste a bigger log trace (interested to see at least
>>> all
>>> > > 300
>>> > > >> >> lines before that) in a pastebin?
>>> > > >> >>
>>> > > >> >> Thx,
>>> > > >> >>
>>> > > >> >> J-D
>>> > > >> >>
>>> > > >> >> On Wed, Nov 11, 2009 at 10:56 AM, Zhenyu Zhong <
>>> > > zhongresearch@gmail.com
>>> > > >> >
>>> > > >> >> wrote:
>>> > > >> >> > J-D,
>>> > > >> >> >
>>> > > >> >> > Thanks for the reply.
>>> > > >> >> >
>>> > > >> >> > The problem I have is one or more of the regionservers
>>> disconnect
>>> > > from
>>> > > >> >> the
>>> > > >> >> > cluster.
>>> > > >> >> >
>>> > > >> >> > I checked the resource usage including memory Heap usage for
>>> > > >> >> regionserver,
>>> > > >> >> > and found nothing suspicious. The regionserver heap usage is
>>> way
>>> > > below
>>> > > >> >> the
>>> > > >> >> > maxHeap defined in jvm.
>>> > > >> >> >
>>> > > >> >> > So I checked zookeeper log, then I found the following
>>> > > >> >> > BufferUnderflowException, eventually it leads to
>>> SocketException:
>>> > > >> >> Transport
>>> > > >> >> > endpoint is not connected.
>>> > > >> >> >
>>> > > >> >> >
>>> > > >> >> > I also checked the regionserver log. It looks like the
>>> > regionserver
>>> > > >> tried
>>> > > >> >> to
>>> > > >> >> > connect to other zookeeper quorum server. It connects
>>> > successfully,
>>> > > >> but
>>> > > >> >> then
>>> > > >> >> > it complained about SocketException: transport endpoint is not
>>> > > >> connected.
>>> > > >> >> >
>>> > > >> >> > So both zookeeper log and regionserver log told me that the
>>> > > transport
>>> > > >> >> > endpoint is not connected. I don't know why.
>>> > > >> >> >
>>> > > >> >> > I would be very appreciated if you could point me to the right
>>> > > >> direction.
>>> > > >> >> >
>>> > > >> >> > Best,
>>> > > >> >> >
>>> > > >> >> >
>>> > > >> >> > Regionserver log:
>>> > > >> >> > 2009-11-11 04:24:53,718 INFO org.apache.zookeeper.ClientCnxn:
>>> > > >> Attempting
>>> > > >> >> > connection to server superpyxis0018/192.168.100.132:2181
>>> > > >> >> > 2009-11-11 04:24:53,718 INFO org.apache.zookeeper.ClientCnxn:
>>> > > Priming
>>> > > >> >> > connection to java.nio.channels.SocketChannel[connected
>>> local=/
>>> > > >> >> > 192.168.100.131:43065 remote=superpyxis0018/
>>> 192.168.100.132:2181
>>> > ]
>>> > > >> >> > 2009-11-11 04:24:53,718 INFO org.apache.zookeeper.ClientCnxn:
>>> > > Server
>>> > > >> >> > connection successful
>>> > > >> >> > 2009-11-11 04:24:53,768 INFO
>>> > > >> >> > org.apache.hadoop.hbase.regionserver.HRegionServer: Got
>>> ZooKeeper
>>> > > >> event,
>>> > > >> >> > state: Disconnected, type: None, path: null
>>> > > >> >> > 2009-11-11 04:24:54,118 INFO org.apache.zookeeper.ClientCnxn:
>>> > > >> Attempting
>>> > > >> >> > connection to server superpyxis0017/192.168.100.131:2181
>>> > > >> >> > 2009-11-11 04:24:54,118 INFO org.apache.zookeeper.ClientCnxn:
>>> > > Priming
>>> > > >> >> > connection to java.nio.channels.SocketChannel[connected
>>> local=/
>>> > > >> >> > 192.168.100.131:46903 remote=superpyxis0017/
>>> 192.168.100.131:2181
>>> > ]
>>> > > >> >> > 2009-11-11 04:24:54,118 INFO org.apache.zookeeper.ClientCnxn:
>>> > > Server
>>> > > >> >> > connection successful
>>> > > >> >> > 2009-11-11 04:24:54,118 WARN org.apache.zookeeper.ClientCnxn:
>>> > > >> Exception
>>> > > >> >> > closing session 0x24dfd90c810002 to
>>> > > >> sun.nio.ch.SelectionKeyImpl@2dd78ccf
>>> > > >> >> > java.io.IOException: Read error rc = -1
>>> > > >> java.nio.DirectByteBuffer[pos=0
>>> > > >> >> > lim=4 cap=4]
>>> > > >> >> > at
>>> > > >> >> >
>>> > > org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
>>> > > >> >> > at
>>> > > >> >> >
>>> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
>>> > > >> >> > 2009-11-11 04:24:54,118 WARN org.apache.zookeeper.ClientCnxn:
>>> > > Ignoring
>>> > > >> >> > exception during shutdown input
>>> > > >> >> > java.net.SocketException: Transport endpoint is not connected
>>> > > >> >> > at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
>>> > > >> >> > at
>>> > > >> >> >
>>> > > sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
>>> > > >> >> > at
>>> > > >> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>>> > > >> >> > at
>>> > > >> >> >
>>> > > >>
>>> > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
>>> > > >> >> > at
>>> > > >> >> >
>>> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
>>> > > >> >> >
>>> > > >> >> >
>>> > > >> >> >
>>> > > >> >> >
>>> > > >> >> >
>>> > > >> >> > On Wed, Nov 11, 2009 at 1:40 PM, Jean-Daniel Cryans <
>>> > > >> jdcryans@apache.org
>>> > > >> >> >wrote:
>>> > > >> >> >
>>> > > >> >> >> I'm really not sure exactly what the situation is with the
>>> > > >> >> >> java.nio.BufferUnderflowException, but what do you mean by
>>> > > >> >> >> "regionserver disconnection"? Could you give more details?
>>> > > >> >> >>
>>> > > >> >> >> Thx,
>>> > > >> >> >>
>>> > > >> >> >> J-D
>>> > > >> >> >>
>>> > > >> >> >> On Wed, Nov 11, 2009 at 10:34 AM, Zhenyu Zhong <
>>> > > >> zhongresearch@gmail.com
>>> > > >> >> >
>>> > > >> >> >> wrote:
>>> > > >> >> >> > I searched around and found HBASE-1645 "HBase could be a
>>> > little
>>> > > >> more
>>> > > >> >> >> dainty
>>> > > >> >> >> > when zk is pulled out from under it"
>>> > > >> >> >> >
>>> > > >> >> >> > I am wondering whether my problem is related to this
>>> > HBase-1645.
>>> > > Or
>>> > > >> >> >> probably
>>> > > >> >> >> > has nothing to do with it?
>>> > > >> >> >> >
>>> > > >> >> >> > thanks
>>> > > >> >> >> >
>>> > > >> >> >> >
>>> > > >> >> >> > On Wed, Nov 11, 2009 at 12:02 PM, Zhenyu Zhong <
>>> > > >> >> zhongresearch@gmail.com
>>> > > >> >> >> >wrote:
>>> > > >> >> >> >
>>> > > >> >> >> >> Hi,
>>> > > >> >> >> >>
>>> > > >> >> >> >> I have been monitoring the system resources such as memory
>>> > Heap
>>> > > >> usage
>>> > > >> >> >> for
>>> > > >> >> >> >> Regionserver, load average, network bandwidth, etc.
>>> > > >> >> >> >>
>>> > > >> >> >> >> I experienced a disconnection last night. When I checked
>>> the
>>> > > >> memory
>>> > > >> >> Heap
>>> > > >> >> >> >> for Regionserver, it was 2336 which is way below the max
>>> Heap
>>> > > >> 4096. I
>>> > > >> >> >> don't
>>> > > >> >> >> >> think it is Regionserver problem. The requests that the
>>> > > >> regionserver
>>> > > >> >> got
>>> > > >> >> >> was
>>> > > >> >> >> >> about 126 which doesn't seem to be high. The load average
>>> is
>>> > > kind
>>> > > >> of
>>> > > >> >> >> high
>>> > > >> >> >> >> >100. (but I saw disconnections while the load average is
>>> > > around
>>> > > >> 30).
>>> > > >> >> >> The
>>> > > >> >> >> >> bandwidth is not a problem.
>>> > > >> >> >> >>
>>> > > >> >> >> >> However, I found error messages from the zookeeper. Please
>>> > see
>>> > > the
>>> > > >> >> >> >> following error messages. There are some
>>> > > >> >> >> >> BufferUnderflowException error, IOException error,etc.
>>> > > >> >> >> >> May I ask what might be the causes that generate these
>>> error
>>> > > >> >> messages?
>>> > > >> >> >> >>
>>> > > >> >> >> >> Many thanks
>>> > > >> >> >> >> zhenyu
>>> > > >> >> >> >>
>>> > > >> >> >> >>
>>> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
>>> > > org.apache.zookeeper.server.Request:
>>> > > >> >> >> Ignoring
>>> > > >> >> >> >> exception during toString
>>> > > >> >> >> >> java.nio.BufferUnderflowException
>>> > > >> >> >> >> at
>>> > java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
>>> > > >> >> >> >> at java.nio.ByteBuffer.get(ByteBuffer.java:675)
>>> > > >> >> >> >> at
>>> > > >> >> >>
>>> org.apache.zookeeper.server.Request.toString(Request.java:199)
>>> > > >> >> >> >> at java.lang.String.valueOf(String.java:2827)
>>> > > >> >> >> >> at
>>> > > java.lang.StringBuilder.append(StringBuilder.java:115)
>>> > > >> >> >> >> at
>>> > > >> >> >> >>
>>> > > >> >> >>
>>> > > >> >>
>>> > > >>
>>> > >
>>> >
>>> org.apache.zookeeper.server.quorum.CommitProcessor.processRequest(CommitProcessor.java:167)
>>> > > >> >> >> >> at
>>> > > >> >> >> >>
>>> > > >> >> >>
>>> > > >> >>
>>> > > >>
>>> > >
>>> >
>>> org.apache.zookeeper.server.quorum.FollowerRequestProcessor.run(FollowerRequestProcessor.java:68)
>>> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
>>> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
>>> > > >> >> >> >> Exception causing close of session 0x424dfd908c50009
>>> > > >> >> >> >> due to java.io.IOException: Read error
>>> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
>>> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
>>> > > >> >> >> >> IOException stack trace
>>> > > >> >> >> >> java.io.IOException: Read error
>>> > > >> >> >> >> at
>>> > > >> >> >> >>
>>> > > >> >>
>>> > > org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:372)
>>> > > >> >> >> >> at
>>> > > >> >> >> >>
>>> > > >> >> >>
>>> > > >> >>
>>> > > >>
>>> > >
>>> >
>>> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:221)
>>> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
>>> > > >> >> >> >> org.apache.zookeeper.server.quorum.CommitProcessor:
>>> > Processing
>>> > > >> >> request::
>>> > > >> >> >> >> sessionid:0x424dfd908c50009 type:setWatches
>>> > > >> cxid:0xfffffffffffffff8
>>> > > >> >> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
>>> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
>>> > > org.apache.zookeeper.server.Request:
>>> > > >> >> >> Ignoring
>>> > > >> >> >> >> exception during toString
>>> > > >> >> >> >> java.nio.BufferUnderflowException
>>> > > >> >> >> >> at
>>> > java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
>>> > > >> >> >> >> at java.nio.ByteBuffer.get(ByteBuffer.java:675)
>>> > > >> >> >> >> at
>>> > > >> >> >>
>>> org.apache.zookeeper.server.Request.toString(Request.java:199)
>>> > > >> >> >> >> at java.lang.String.valueOf(String.java:2827)
>>> > > >> >> >> >> at
>>> > > java.lang.StringBuilder.append(StringBuilder.java:115)
>>> > > >> >> >> >> at
>>> > > >> >> >> >>
>>> > > >> >> >>
>>> > > >> >>
>>> > > >>
>>> > >
>>> >
>>> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:74)
>>> > > >> >> >> >> at
>>> > > >> >> >> >>
>>> > > >> >> >>
>>> > > >> >>
>>> > > >>
>>> > >
>>> >
>>> org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)
>>> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
>>> > > >> >> >> >> org.apache.zookeeper.server.FinalRequestProcessor:
>>> Processing
>>> > > >> >> request::
>>> > > >> >> >> >> sessionid:0x424dfd908c50009 type:setWatches
>>> > > >> cxid:0xfffffffffffffff8
>>> > > >> >> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
>>> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
>>> > > org.apache.zookeeper.server.Request:
>>> > > >> >> >> Ignoring
>>> > > >> >> >> >> exception during toString
>>> > > >> >> >> >> java.nio.BufferUnderflowException
>>> > > >> >> >> >> at
>>> > java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
>>> > > >> >> >> >> at java.nio.ByteBuffer.get(ByteBuffer.java:675)
>>> > > >> >> >> >> at
>>> > > >> >> >>
>>> org.apache.zookeeper.server.Request.toString(Request.java:199)
>>> > > >> >> >> >> at
>>> > > >> >> >> >>
>>> > > >>
>>> org.apache.log4j.or.DefaultRenderer.doRender(DefaultRenderer.java:36)
>>> > > >> >> >> >> at
>>> > > >> >> >> >>
>>> > > org.apache.log4j.or.RendererMap.findAndRender(RendererMap.java:80)
>>> > > >> >> >> >> at
>>> > > >> >> >> >>
>>> > > >> >> >>
>>> > > >> >>
>>> > > >>
>>> > >
>>> >
>>> org.apache.log4j.spi.LoggingEvent.getRenderedMessage(LoggingEvent.java:362)
>>> > > >> >> >> >> at
>>> > > >> >> >> >>
>>> > > >> >> >>
>>> > > >> >>
>>> > > >>
>>> > >
>>> >
>>> org.apache.log4j.helpers.PatternParser$BasicPatternConverter.convert(PatternParser.java:403)
>>> > > >> >> >> >> at
>>> > > >> >> >> >>
>>> > > >> >> >>
>>> > > >> >>
>>> > > >>
>>> > >
>>> >
>>> org.apache.log4j.helpers.PatternConverter.format(PatternConverter.java:65)
>>> > > >> >> >> >> at
>>> > > >> >> org.apache.log4j.PatternLayout.format(PatternLayout.java:502)
>>> > > >> >> >> >> at
>>> > > >> >> >> >>
>>> > > org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:302)
>>> > > >> >> >> >> at
>>> > > >> >> >> >>
>>> > > >> >> >>
>>> > > >> >>
>>> > > >>
>>> > >
>>> >
>>> org.apache.log4j.DailyRollingFileAppender.subAppend(DailyRollingFileAppender.java:359)
>>> > > >> >> >> >> at
>>> > > >> >> >>
>>> org.apache.log4j.WriterAppender.append(WriterAppender.java:160)
>>> > > >> >> >> >> at
>>> > > >> >> >> >>
>>> > > >>
>>> org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
>>> > > >> >> >> >> at
>>> > > >> >> >> >>
>>> > > >> >> >>
>>> > > >> >>
>>> > > >>
>>> > >
>>> >
>>> org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
>>> > > >> >> >> >> at
>>> > > >> org.apache.log4j.Category.callAppenders(Category.java:206)
>>> > > >> >> >> >> at
>>> > > org.apache.log4j.Category.forcedLog(Category.java:391)
>>> > > >> >> >> >> at
>>> org.apache.log4j.Category.debug(Category.java:260)
>>> > > >> >> >> >> at
>>> > > >> >> >> >>
>>> > > >> >> >>
>>> > > >> >>
>>> > > >>
>>> > >
>>> >
>>> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:147)
>>> > > >> >> >> >> at
>>> > > >> >> >> >>
>>> > > >> >> >>
>>> > > >> >>
>>> > > >>
>>> > >
>>> >
>>> org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)
>>> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
>>> > > org.apache.zookeeper.server.Request:
>>> > > >> >> >> Ignoring
>>> > > >> >> >> >> exception during toString
>>> > > >> >> >> >> sessionid:0x424dfd908c50009 type:setWatches
>>> > > >> cxid:0xfffffffffffffff8
>>> > > >> >> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
>>> > > >> >> >> >> 2009-11-11 04:29:00,810 INFO
>>> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
>>> > > >> >> >> >> closing session:0x424dfd908c50009 NIOServerCnxn:
>>> > > >> >> >> >> java.nio.channels.SocketChannel[connected local=/
>>> > > >> >> 192.168.100.131:2181
>>> > > >> >> >> remote=/
>>> > > >> >> >> >> 192.168.100.131:35500]
>>> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
>>> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
>>> > > >> >> >> >> ignoring exception during input shutdown
>>> > > >> >> >> >> java.net.SocketException: Transport endpoint is not
>>> connected
>>> > > >> >> >> >> at sun.nio.ch.SocketChannelImpl.shutdown(Native
>>> > Method)
>>> > > >> >> >> >> at
>>> > > >> >> >> >>
>>> > > >> >>
>>> > > sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
>>> > > >> >> >> >> at
>>> > > >> >> >>
>>> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>>> > > >> >> >> >> at
>>> > > >> >> >> >>
>>> > > >> >>
>>> > >
>>> org.apache.zookeeper.server.NIOServerCnxn.close(NIOServerCnxn.java:848)
>>> > > >> >> >> >> at
>>> > > >> >> >> >>
>>> > > >> >>
>>> > > org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:498)
>>> > > >> >> >> >> at
>>> > > >> >> >> >>
>>> > > >> >> >>
>>> > > >> >>
>>> > > >>
>>> > >
>>> >
>>> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:221)
>>> > > >> >> >> >>
>>> > > >> >> >> >
>>> > > >> >> >>
>>> > > >> >> >
>>> > > >> >>
>>> > > >> >
>>> > > >>
>>> > > >
>>> > >
>>> >
>>>
>>
>>
>
Re: regionserver disconnection
Posted by Zhenyu Zhong <zh...@gmail.com>.
Follow up on the regionserver disconnection:
I made a few changes to the cluster setup.
1. split the regionserver and zookeeper quorum servers, such that HRegion
doesn't run on the same box as HQuorumPeer. I assume that there should be no
disk io contention between the regionserver and zookeeper.
Now we have 14 regionservers, and 5 zookeeper quorum servers.
Though I experienced 2 regionserver disconnection this morning, it looks
better from the regionserver log. (please see the following log)
http://pastebin.com/m496dbfae
I checked diskIO for one of the regionserver(192.168.100.116) during the
time it disconnected.
Time: 03:04:58 AM
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 105.31 5458.83 19503.64 9043873239 32312473676
sda1 2.90 6.64 99.25 10993934 164433464
sda2 1.72 23.76 30.25 39365817 50124008
sda3 0.30 0.38 3.58 624930 5923000
sda4 100.39 5428.06 19370.56 8992888270 32091993204
I also checked the zookeeper quorum server that the regionserver tried to
connect according to the log. However, I don't see 192.168.100.116 in the
client list of the zookeeper quorum server that the regionserver tried to
connect.
Would that be a problem?
Thu Nov 12 15:04:52 UTC 2009
Zookeeper version: 3.2.1-808558, built on 08/27/2009 18:48 GMT
Clients:
/192.168.100.127:43045[1](queued=0,recved=26,sent=0)
/192.168.100.131:39091[1](queued=0,recved=964,sent=0)
/192.168.100.124:35961[1](queued=0,recved=3266,sent=0)
/192.168.100.123:47935[1](queued=0,recved=233,sent=0)
/192.168.100.125:46931[1](queued=0,recved=2,sent=0)
/192.168.100.118:54924[1](queued=0,recved=295,sent=0)
/192.168.100.118:41390[1](queued=0,recved=2290,sent=0)
/192.168.100.136:42243[1](queued=0,recved=0,sent=0)
Latency min/avg/max: 0/17/6333
Received: 47111
Sent: 0
Outstanding: 0
Zxid: 0x77000083f4
Mode: leader
Node count: 23
Best,
zhenyu
On Wed, Nov 11, 2009 at 3:58 PM, Zhenyu Zhong <zh...@gmail.com>wrote:
> Stack
>
> I am very appreciated for your comments.
> I will use the zookeeper monitoring script on my cluster and let it run
> overnight to see the result.
> I will follow up the thread when I get anything.
>
> thanks
> zhenyu
>
>
> On Wed, Nov 11, 2009 at 3:52 PM, stack <st...@duboce.net> wrote:
>
>> I see these in your log too:
>>
>>
>> 1. 2009-11-11 04:27:19,018 DEBUG org.apache.zookeeper.ClientCnxn: Got
>> ping response for sessionid:0x424dfd908c50009 after 4544ms
>> 2. 2009-11-11 04:27:19,018 DEBUG org.apache.zookeeper.ClientCnxn: Got
>> ping response for sessionid:0x24dfd90c810002 after 4548ms
>> 3. 2009-11-11 04:27:43,960 DEBUG org.apache.zookeeper.ClientCnxn: Got
>> ping response for sessionid:0x424dfd908c50009 after 9030ms
>> 4. 2009-11-11 04:27:43,960 DEBUG org.apache.zookeeper.ClientCnxn: Got
>> ping response for sessionid:0x24dfd90c810002 after 9030ms
>> 5. 2009-11-11 04:27:48,830 DEBUG org.apache.zookeeper.ClientCnxn: Got
>> ping response for sessionid:0x424dfd908c50009 after 0ms
>> 6. 2009-11-11 04:27:48,830 DEBUG org.apache.zookeeper.ClientCnxn: Got
>> ping response for sessionid:0x24dfd90c810002 after 0ms
>>
>> Which would indicate lag talking to zk too.
>>
>> Appending HLog also seems to get steadily worse.
>>
>> I also see these:
>>
>>
>> 1. 2009-11-11 04:27:01,233 WARN org.apache.zookeeper.ClientCnxn:
>> Exception closing session 0x24dfd90c810002 to
>> sun.nio.ch.SelectionKeyImpl@5e16a7e3
>> 2. java.io.IOException: Read error rc = -1
>> java.nio.DirectByteBuffer[pos=0 lim=4 cap=4]
>> 3. at
>> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
>> 4. at
>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
>>
>>
>> Were all 5 quorum members participating? Check by doing "echo stat|nc
>> QUORUM_HOST port" for each member of quorum.... the below might help:
>>
>> members="1,2,3,4,5"
>> while true; do date; for i in $members; do echo $i; echo stat | nc $i
>> 2181;
>> done; sleep 10; done
>>
>> We need to improve our zk_dump. It just reports quorum members... not if
>> they are actually participating or not. The above will tell you if dead
>> or
>> alive.
>>
>> Please report back after checking above and following the Jean-Daniel
>> prescription.
>>
>> Thanks,
>> St.Ack
>>
>>
>>
>> On Wed, Nov 11, 2009 at 12:29 PM, Zhenyu Zhong <zhongresearch@gmail.com
>> >wrote:
>>
>> > Thanks a lot for the help.
>> >
>> > I understand that the change of disk configurations should be done
>> > progressively such that HDFS can balance and recover from a disk going
>> > away.
>> >
>> > The data we are trying to import into HBase is about 200million rows,
>> each
>> > row is about hundreds of bytes. Basically, we read in a text file from
>> > HDFS,
>> > then for each row in the text file, we fetch data from HBase table, we
>> > compare the column value and update the latest value back into HBase. I
>> was
>> > suggested to use CheckAndPut. Anyway this is the common usecase on my
>> > side.
>> >
>> >
>> > Many thanks
>> > zhenyu
>> >
>> >
>> >
>> > On Wed, Nov 11, 2009 at 3:03 PM, Jean-Daniel Cryans <
>> jdcryans@apache.org
>> > >wrote:
>> >
>> > > 6 disks, 19 machines and you see those kind of issues? I'm quite
>> > > surprised and I want to learn more about what you trying to to.
>> > >
>> > > But first, if hbase is managing ZK for you then you can change the
>> > > value of hbase.zookeeper.property.dataDir so that it points to its own
>> > > mount. Make sure you also change dfs.data.dir in conf/hdfs-site.xml
>> > > (hadoop stuff) so that it stops pointing there (you should have 6
>> > > values there at the moment if you configured it correctly). Doing this
>> > > could result in data loss if you do it all at once so do the change
>> > > progressively. This also means that you may have to keep a seperate
>> > > HDFS configuration for those particular nodes. Please ask more
>> > > questions if it's not very clear.
>> > >
>> > > WRT to dedicating nodes to Zookeeeper, it makes sense only if you have
>> > > a lot of nodes but in your case, unless there's also CPU contention,
>> > > you're good with only a dedicated disk.
>> > >
>> > > Now for what you are trying to do, is this a massive import that goes
>> > > over many hours? How big is each row? I'd love to get a lot of details
>> > > to help you optimize your cluster for your particular problem.
>> > >
>> > > Thx,
>> > >
>> > > J-D
>> > >
>> > > On Wed, Nov 11, 2009 at 11:51 AM, Zhenyu Zhong <
>> zhongresearch@gmail.com>
>> > > wrote:
>> > > > J-D,
>> > > >
>> > > > Thanks a lot for the insightful comments.
>> > > > We are running 5 zookeeper quorums now. On these 5 quorum servers,
>> we
>> > > also
>> > > > run datanode, tasktracker and regionserver.
>> > > > Now we have 6 disks per machine.
>> > > > If it is the disk IO saturation, is there anyway to configure
>> zookeeper
>> > > to
>> > > > write to a seperate disk which is not the same one used by HDFS?
>> > > > Or shall we run zookeeper quorums on different nodes other than
>> HDFS?
>> > > >
>> > > > Are HLogs generated by Regionservers? Would it alleviate the issue
>> if
>> > we
>> > > use
>> > > > the first 14 for regionserver, and the rest of 5 for zookeeper?(In
>> that
>> > > > sense, the node running zookeeper quorum won't run regionserver)
>> > > >
>> > > > I am very appreciated for your comments.
>> > > >
>> > > > Best,
>> > > > zhenyu
>> > > >
>> > > >
>> > > >
>> > > > On Wed, Nov 11, 2009 at 2:30 PM, Jean-Daniel Cryans <
>> > jdcryans@apache.org
>> > > >wrote:
>> > > >
>> > > >> The problem seems to be that the IO is saturated. When you see
>> this:
>> > > >>
>> > > >> #
>> > > >> 2009-11-11 04:28:10,308 WARN
>> > > >> org.apache.hadoop.hbase.regionserver.HLog: IPC Server handler 9 on
>> > > >> 60021 took 6819ms appending an edit to hlog; editcount=370
>> > > >> #
>> > > >> 2009-11-11 04:28:10,308 WARN
>> > > >> org.apache.hadoop.hbase.regionserver.HLog: IPC Server handler 2 on
>> > > >> 60021 took 4412ms appending an edit to hlog; editcount=371
>> > > >>
>> > > >> It's a sign that HDFS is slow. I guess that the zookeeper quorum
>> > > >> members are on those same nodes using the same disks? If so,
>> there's
>> > > >> the problem because Zookeeper needs to be able to commit
>> transactions
>> > > >> very fast and if your disks are already hogged then they have to
>> wait
>> > > >> and you can get disconnections. Do you still have 19 quorum
>> members?
>> > > >> How many disks do you have per machine?
>> > > >>
>> > > >> Thx
>> > > >>
>> > > >> J-D
>> > > >>
>> > > >> On Wed, Nov 11, 2009 at 11:22 AM, Zhenyu Zhong <
>> > zhongresearch@gmail.com
>> > > >
>> > > >> wrote:
>> > > >> > Please take a look at the following regionserver log.
>> > > >> > The error started to come out from 4:24am UTC to 4:29am UTC.
>> > > >> >
>> > > >> > http://pastebin.com/m3717d064
>> > > >> >
>> > > >> > Thanks in advance.
>> > > >> >
>> > > >> > Best,
>> > > >> > zhenyu
>> > > >> >
>> > > >> >
>> > > >> > On Wed, Nov 11, 2009 at 2:01 PM, Jean-Daniel Cryans <
>> > > jdcryans@apache.org
>> > > >> >wrote:
>> > > >> >
>> > > >> >> Could you paste a bigger log trace (interested to see at least
>> all
>> > > 300
>> > > >> >> lines before that) in a pastebin?
>> > > >> >>
>> > > >> >> Thx,
>> > > >> >>
>> > > >> >> J-D
>> > > >> >>
>> > > >> >> On Wed, Nov 11, 2009 at 10:56 AM, Zhenyu Zhong <
>> > > zhongresearch@gmail.com
>> > > >> >
>> > > >> >> wrote:
>> > > >> >> > J-D,
>> > > >> >> >
>> > > >> >> > Thanks for the reply.
>> > > >> >> >
>> > > >> >> > The problem I have is one or more of the regionservers
>> disconnect
>> > > from
>> > > >> >> the
>> > > >> >> > cluster.
>> > > >> >> >
>> > > >> >> > I checked the resource usage including memory Heap usage for
>> > > >> >> regionserver,
>> > > >> >> > and found nothing suspicious. The regionserver heap usage is
>> way
>> > > below
>> > > >> >> the
>> > > >> >> > maxHeap defined in jvm.
>> > > >> >> >
>> > > >> >> > So I checked zookeeper log, then I found the following
>> > > >> >> > BufferUnderflowException, eventually it leads to
>> SocketException:
>> > > >> >> Transport
>> > > >> >> > endpoint is not connected.
>> > > >> >> >
>> > > >> >> >
>> > > >> >> > I also checked the regionserver log. It looks like the
>> > regionserver
>> > > >> tried
>> > > >> >> to
>> > > >> >> > connect to other zookeeper quorum server. It connects
>> > successfully,
>> > > >> but
>> > > >> >> then
>> > > >> >> > it complained about SocketException: transport endpoint is not
>> > > >> connected.
>> > > >> >> >
>> > > >> >> > So both zookeeper log and regionserver log told me that the
>> > > transport
>> > > >> >> > endpoint is not connected. I don't know why.
>> > > >> >> >
>> > > >> >> > I would be very appreciated if you could point me to the right
>> > > >> direction.
>> > > >> >> >
>> > > >> >> > Best,
>> > > >> >> >
>> > > >> >> >
>> > > >> >> > Regionserver log:
>> > > >> >> > 2009-11-11 04:24:53,718 INFO org.apache.zookeeper.ClientCnxn:
>> > > >> Attempting
>> > > >> >> > connection to server superpyxis0018/192.168.100.132:2181
>> > > >> >> > 2009-11-11 04:24:53,718 INFO org.apache.zookeeper.ClientCnxn:
>> > > Priming
>> > > >> >> > connection to java.nio.channels.SocketChannel[connected
>> local=/
>> > > >> >> > 192.168.100.131:43065 remote=superpyxis0018/
>> 192.168.100.132:2181
>> > ]
>> > > >> >> > 2009-11-11 04:24:53,718 INFO org.apache.zookeeper.ClientCnxn:
>> > > Server
>> > > >> >> > connection successful
>> > > >> >> > 2009-11-11 04:24:53,768 INFO
>> > > >> >> > org.apache.hadoop.hbase.regionserver.HRegionServer: Got
>> ZooKeeper
>> > > >> event,
>> > > >> >> > state: Disconnected, type: None, path: null
>> > > >> >> > 2009-11-11 04:24:54,118 INFO org.apache.zookeeper.ClientCnxn:
>> > > >> Attempting
>> > > >> >> > connection to server superpyxis0017/192.168.100.131:2181
>> > > >> >> > 2009-11-11 04:24:54,118 INFO org.apache.zookeeper.ClientCnxn:
>> > > Priming
>> > > >> >> > connection to java.nio.channels.SocketChannel[connected
>> local=/
>> > > >> >> > 192.168.100.131:46903 remote=superpyxis0017/
>> 192.168.100.131:2181
>> > ]
>> > > >> >> > 2009-11-11 04:24:54,118 INFO org.apache.zookeeper.ClientCnxn:
>> > > Server
>> > > >> >> > connection successful
>> > > >> >> > 2009-11-11 04:24:54,118 WARN org.apache.zookeeper.ClientCnxn:
>> > > >> Exception
>> > > >> >> > closing session 0x24dfd90c810002 to
>> > > >> sun.nio.ch.SelectionKeyImpl@2dd78ccf
>> > > >> >> > java.io.IOException: Read error rc = -1
>> > > >> java.nio.DirectByteBuffer[pos=0
>> > > >> >> > lim=4 cap=4]
>> > > >> >> > at
>> > > >> >> >
>> > > org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
>> > > >> >> > at
>> > > >> >> >
>> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
>> > > >> >> > 2009-11-11 04:24:54,118 WARN org.apache.zookeeper.ClientCnxn:
>> > > Ignoring
>> > > >> >> > exception during shutdown input
>> > > >> >> > java.net.SocketException: Transport endpoint is not connected
>> > > >> >> > at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
>> > > >> >> > at
>> > > >> >> >
>> > > sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
>> > > >> >> > at
>> > > >> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>> > > >> >> > at
>> > > >> >> >
>> > > >>
>> > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
>> > > >> >> > at
>> > > >> >> >
>> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
>> > > >> >> >
>> > > >> >> >
>> > > >> >> >
>> > > >> >> >
>> > > >> >> >
>> > > >> >> > On Wed, Nov 11, 2009 at 1:40 PM, Jean-Daniel Cryans <
>> > > >> jdcryans@apache.org
>> > > >> >> >wrote:
>> > > >> >> >
>> > > >> >> >> I'm really not sure exactly what the situation is with the
>> > > >> >> >> java.nio.BufferUnderflowException, but what do you mean by
>> > > >> >> >> "regionserver disconnection"? Could you give more details?
>> > > >> >> >>
>> > > >> >> >> Thx,
>> > > >> >> >>
>> > > >> >> >> J-D
>> > > >> >> >>
>> > > >> >> >> On Wed, Nov 11, 2009 at 10:34 AM, Zhenyu Zhong <
>> > > >> zhongresearch@gmail.com
>> > > >> >> >
>> > > >> >> >> wrote:
>> > > >> >> >> > I searched around and found HBASE-1645 "HBase could be a
>> > little
>> > > >> more
>> > > >> >> >> dainty
>> > > >> >> >> > when zk is pulled out from under it"
>> > > >> >> >> >
>> > > >> >> >> > I am wondering whether my problem is related to this
>> > HBase-1645.
>> > > Or
>> > > >> >> >> probably
>> > > >> >> >> > has nothing to do with it?
>> > > >> >> >> >
>> > > >> >> >> > thanks
>> > > >> >> >> >
>> > > >> >> >> >
>> > > >> >> >> > On Wed, Nov 11, 2009 at 12:02 PM, Zhenyu Zhong <
>> > > >> >> zhongresearch@gmail.com
>> > > >> >> >> >wrote:
>> > > >> >> >> >
>> > > >> >> >> >> Hi,
>> > > >> >> >> >>
>> > > >> >> >> >> I have been monitoring the system resources such as memory
>> > Heap
>> > > >> usage
>> > > >> >> >> for
>> > > >> >> >> >> Regionserver, load average, network bandwidth, etc.
>> > > >> >> >> >>
>> > > >> >> >> >> I experienced a disconnection last night. When I checked
>> the
>> > > >> memory
>> > > >> >> Heap
>> > > >> >> >> >> for Regionserver, it was 2336 which is way below the max
>> Heap
>> > > >> 4096. I
>> > > >> >> >> don't
>> > > >> >> >> >> think it is Regionserver problem. The requests that the
>> > > >> regionserver
>> > > >> >> got
>> > > >> >> >> was
>> > > >> >> >> >> about 126 which doesn't seem to be high. The load average
>> is
>> > > kind
>> > > >> of
>> > > >> >> >> high
>> > > >> >> >> >> >100. (but I saw disconnections while the load average is
>> > > around
>> > > >> 30).
>> > > >> >> >> The
>> > > >> >> >> >> bandwidth is not a problem.
>> > > >> >> >> >>
>> > > >> >> >> >> However, I found error messages from the zookeeper. Please
>> > see
>> > > the
>> > > >> >> >> >> following error messages. There are some
>> > > >> >> >> >> BufferUnderflowException error, IOException error,etc.
>> > > >> >> >> >> May I ask what might be the causes that generate these
>> error
>> > > >> >> messages?
>> > > >> >> >> >>
>> > > >> >> >> >> Many thanks
>> > > >> >> >> >> zhenyu
>> > > >> >> >> >>
>> > > >> >> >> >>
>> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
>> > > org.apache.zookeeper.server.Request:
>> > > >> >> >> Ignoring
>> > > >> >> >> >> exception during toString
>> > > >> >> >> >> java.nio.BufferUnderflowException
>> > > >> >> >> >> at
>> > java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
>> > > >> >> >> >> at java.nio.ByteBuffer.get(ByteBuffer.java:675)
>> > > >> >> >> >> at
>> > > >> >> >>
>> org.apache.zookeeper.server.Request.toString(Request.java:199)
>> > > >> >> >> >> at java.lang.String.valueOf(String.java:2827)
>> > > >> >> >> >> at
>> > > java.lang.StringBuilder.append(StringBuilder.java:115)
>> > > >> >> >> >> at
>> > > >> >> >> >>
>> > > >> >> >>
>> > > >> >>
>> > > >>
>> > >
>> >
>> org.apache.zookeeper.server.quorum.CommitProcessor.processRequest(CommitProcessor.java:167)
>> > > >> >> >> >> at
>> > > >> >> >> >>
>> > > >> >> >>
>> > > >> >>
>> > > >>
>> > >
>> >
>> org.apache.zookeeper.server.quorum.FollowerRequestProcessor.run(FollowerRequestProcessor.java:68)
>> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
>> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
>> > > >> >> >> >> Exception causing close of session 0x424dfd908c50009
>> > > >> >> >> >> due to java.io.IOException: Read error
>> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
>> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
>> > > >> >> >> >> IOException stack trace
>> > > >> >> >> >> java.io.IOException: Read error
>> > > >> >> >> >> at
>> > > >> >> >> >>
>> > > >> >>
>> > > org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:372)
>> > > >> >> >> >> at
>> > > >> >> >> >>
>> > > >> >> >>
>> > > >> >>
>> > > >>
>> > >
>> >
>> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:221)
>> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
>> > > >> >> >> >> org.apache.zookeeper.server.quorum.CommitProcessor:
>> > Processing
>> > > >> >> request::
>> > > >> >> >> >> sessionid:0x424dfd908c50009 type:setWatches
>> > > >> cxid:0xfffffffffffffff8
>> > > >> >> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
>> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
>> > > org.apache.zookeeper.server.Request:
>> > > >> >> >> Ignoring
>> > > >> >> >> >> exception during toString
>> > > >> >> >> >> java.nio.BufferUnderflowException
>> > > >> >> >> >> at
>> > java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
>> > > >> >> >> >> at java.nio.ByteBuffer.get(ByteBuffer.java:675)
>> > > >> >> >> >> at
>> > > >> >> >>
>> org.apache.zookeeper.server.Request.toString(Request.java:199)
>> > > >> >> >> >> at java.lang.String.valueOf(String.java:2827)
>> > > >> >> >> >> at
>> > > java.lang.StringBuilder.append(StringBuilder.java:115)
>> > > >> >> >> >> at
>> > > >> >> >> >>
>> > > >> >> >>
>> > > >> >>
>> > > >>
>> > >
>> >
>> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:74)
>> > > >> >> >> >> at
>> > > >> >> >> >>
>> > > >> >> >>
>> > > >> >>
>> > > >>
>> > >
>> >
>> org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)
>> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
>> > > >> >> >> >> org.apache.zookeeper.server.FinalRequestProcessor:
>> Processing
>> > > >> >> request::
>> > > >> >> >> >> sessionid:0x424dfd908c50009 type:setWatches
>> > > >> cxid:0xfffffffffffffff8
>> > > >> >> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
>> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
>> > > org.apache.zookeeper.server.Request:
>> > > >> >> >> Ignoring
>> > > >> >> >> >> exception during toString
>> > > >> >> >> >> java.nio.BufferUnderflowException
>> > > >> >> >> >> at
>> > java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
>> > > >> >> >> >> at java.nio.ByteBuffer.get(ByteBuffer.java:675)
>> > > >> >> >> >> at
>> > > >> >> >>
>> org.apache.zookeeper.server.Request.toString(Request.java:199)
>> > > >> >> >> >> at
>> > > >> >> >> >>
>> > > >>
>> org.apache.log4j.or.DefaultRenderer.doRender(DefaultRenderer.java:36)
>> > > >> >> >> >> at
>> > > >> >> >> >>
>> > > org.apache.log4j.or.RendererMap.findAndRender(RendererMap.java:80)
>> > > >> >> >> >> at
>> > > >> >> >> >>
>> > > >> >> >>
>> > > >> >>
>> > > >>
>> > >
>> >
>> org.apache.log4j.spi.LoggingEvent.getRenderedMessage(LoggingEvent.java:362)
>> > > >> >> >> >> at
>> > > >> >> >> >>
>> > > >> >> >>
>> > > >> >>
>> > > >>
>> > >
>> >
>> org.apache.log4j.helpers.PatternParser$BasicPatternConverter.convert(PatternParser.java:403)
>> > > >> >> >> >> at
>> > > >> >> >> >>
>> > > >> >> >>
>> > > >> >>
>> > > >>
>> > >
>> >
>> org.apache.log4j.helpers.PatternConverter.format(PatternConverter.java:65)
>> > > >> >> >> >> at
>> > > >> >> org.apache.log4j.PatternLayout.format(PatternLayout.java:502)
>> > > >> >> >> >> at
>> > > >> >> >> >>
>> > > org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:302)
>> > > >> >> >> >> at
>> > > >> >> >> >>
>> > > >> >> >>
>> > > >> >>
>> > > >>
>> > >
>> >
>> org.apache.log4j.DailyRollingFileAppender.subAppend(DailyRollingFileAppender.java:359)
>> > > >> >> >> >> at
>> > > >> >> >>
>> org.apache.log4j.WriterAppender.append(WriterAppender.java:160)
>> > > >> >> >> >> at
>> > > >> >> >> >>
>> > > >>
>> org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
>> > > >> >> >> >> at
>> > > >> >> >> >>
>> > > >> >> >>
>> > > >> >>
>> > > >>
>> > >
>> >
>> org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
>> > > >> >> >> >> at
>> > > >> org.apache.log4j.Category.callAppenders(Category.java:206)
>> > > >> >> >> >> at
>> > > org.apache.log4j.Category.forcedLog(Category.java:391)
>> > > >> >> >> >> at
>> org.apache.log4j.Category.debug(Category.java:260)
>> > > >> >> >> >> at
>> > > >> >> >> >>
>> > > >> >> >>
>> > > >> >>
>> > > >>
>> > >
>> >
>> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:147)
>> > > >> >> >> >> at
>> > > >> >> >> >>
>> > > >> >> >>
>> > > >> >>
>> > > >>
>> > >
>> >
>> org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)
>> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
>> > > org.apache.zookeeper.server.Request:
>> > > >> >> >> Ignoring
>> > > >> >> >> >> exception during toString
>> > > >> >> >> >> sessionid:0x424dfd908c50009 type:setWatches
>> > > >> cxid:0xfffffffffffffff8
>> > > >> >> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
>> > > >> >> >> >> 2009-11-11 04:29:00,810 INFO
>> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
>> > > >> >> >> >> closing session:0x424dfd908c50009 NIOServerCnxn:
>> > > >> >> >> >> java.nio.channels.SocketChannel[connected local=/
>> > > >> >> 192.168.100.131:2181
>> > > >> >> >> remote=/
>> > > >> >> >> >> 192.168.100.131:35500]
>> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
>> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
>> > > >> >> >> >> ignoring exception during input shutdown
>> > > >> >> >> >> java.net.SocketException: Transport endpoint is not
>> connected
>> > > >> >> >> >> at sun.nio.ch.SocketChannelImpl.shutdown(Native
>> > Method)
>> > > >> >> >> >> at
>> > > >> >> >> >>
>> > > >> >>
>> > > sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
>> > > >> >> >> >> at
>> > > >> >> >>
>> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>> > > >> >> >> >> at
>> > > >> >> >> >>
>> > > >> >>
>> > >
>> org.apache.zookeeper.server.NIOServerCnxn.close(NIOServerCnxn.java:848)
>> > > >> >> >> >> at
>> > > >> >> >> >>
>> > > >> >>
>> > > org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:498)
>> > > >> >> >> >> at
>> > > >> >> >> >>
>> > > >> >> >>
>> > > >> >>
>> > > >>
>> > >
>> >
>> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:221)
>> > > >> >> >> >>
>> > > >> >> >> >
>> > > >> >> >>
>> > > >> >> >
>> > > >> >>
>> > > >> >
>> > > >>
>> > > >
>> > >
>> >
>>
>
>
Re: regionserver disconnection
Posted by Zhenyu Zhong <zh...@gmail.com>.
Stack
I am very appreciated for your comments.
I will use the zookeeper monitoring script on my cluster and let it run
overnight to see the result.
I will follow up the thread when I get anything.
thanks
zhenyu
On Wed, Nov 11, 2009 at 3:52 PM, stack <st...@duboce.net> wrote:
> I see these in your log too:
>
>
> 1. 2009-11-11 04:27:19,018 DEBUG org.apache.zookeeper.ClientCnxn: Got
> ping response for sessionid:0x424dfd908c50009 after 4544ms
> 2. 2009-11-11 04:27:19,018 DEBUG org.apache.zookeeper.ClientCnxn: Got
> ping response for sessionid:0x24dfd90c810002 after 4548ms
> 3. 2009-11-11 04:27:43,960 DEBUG org.apache.zookeeper.ClientCnxn: Got
> ping response for sessionid:0x424dfd908c50009 after 9030ms
> 4. 2009-11-11 04:27:43,960 DEBUG org.apache.zookeeper.ClientCnxn: Got
> ping response for sessionid:0x24dfd90c810002 after 9030ms
> 5. 2009-11-11 04:27:48,830 DEBUG org.apache.zookeeper.ClientCnxn: Got
> ping response for sessionid:0x424dfd908c50009 after 0ms
> 6. 2009-11-11 04:27:48,830 DEBUG org.apache.zookeeper.ClientCnxn: Got
> ping response for sessionid:0x24dfd90c810002 after 0ms
>
> Which would indicate lag talking to zk too.
>
> Appending HLog also seems to get steadily worse.
>
> I also see these:
>
>
> 1. 2009-11-11 04:27:01,233 WARN org.apache.zookeeper.ClientCnxn:
> Exception closing session 0x24dfd90c810002 to
> sun.nio.ch.SelectionKeyImpl@5e16a7e3
> 2. java.io.IOException: Read error rc = -1
> java.nio.DirectByteBuffer[pos=0 lim=4 cap=4]
> 3. at
> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
> 4. at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
>
>
> Were all 5 quorum members participating? Check by doing "echo stat|nc
> QUORUM_HOST port" for each member of quorum.... the below might help:
>
> members="1,2,3,4,5"
> while true; do date; for i in $members; do echo $i; echo stat | nc $i 2181;
> done; sleep 10; done
>
> We need to improve our zk_dump. It just reports quorum members... not if
> they are actually participating or not. The above will tell you if dead or
> alive.
>
> Please report back after checking above and following the Jean-Daniel
> prescription.
>
> Thanks,
> St.Ack
>
>
>
> On Wed, Nov 11, 2009 at 12:29 PM, Zhenyu Zhong <zhongresearch@gmail.com
> >wrote:
>
> > Thanks a lot for the help.
> >
> > I understand that the change of disk configurations should be done
> > progressively such that HDFS can balance and recover from a disk going
> > away.
> >
> > The data we are trying to import into HBase is about 200million rows,
> each
> > row is about hundreds of bytes. Basically, we read in a text file from
> > HDFS,
> > then for each row in the text file, we fetch data from HBase table, we
> > compare the column value and update the latest value back into HBase. I
> was
> > suggested to use CheckAndPut. Anyway this is the common usecase on my
> > side.
> >
> >
> > Many thanks
> > zhenyu
> >
> >
> >
> > On Wed, Nov 11, 2009 at 3:03 PM, Jean-Daniel Cryans <jdcryans@apache.org
> > >wrote:
> >
> > > 6 disks, 19 machines and you see those kind of issues? I'm quite
> > > surprised and I want to learn more about what you trying to to.
> > >
> > > But first, if hbase is managing ZK for you then you can change the
> > > value of hbase.zookeeper.property.dataDir so that it points to its own
> > > mount. Make sure you also change dfs.data.dir in conf/hdfs-site.xml
> > > (hadoop stuff) so that it stops pointing there (you should have 6
> > > values there at the moment if you configured it correctly). Doing this
> > > could result in data loss if you do it all at once so do the change
> > > progressively. This also means that you may have to keep a seperate
> > > HDFS configuration for those particular nodes. Please ask more
> > > questions if it's not very clear.
> > >
> > > WRT to dedicating nodes to Zookeeeper, it makes sense only if you have
> > > a lot of nodes but in your case, unless there's also CPU contention,
> > > you're good with only a dedicated disk.
> > >
> > > Now for what you are trying to do, is this a massive import that goes
> > > over many hours? How big is each row? I'd love to get a lot of details
> > > to help you optimize your cluster for your particular problem.
> > >
> > > Thx,
> > >
> > > J-D
> > >
> > > On Wed, Nov 11, 2009 at 11:51 AM, Zhenyu Zhong <
> zhongresearch@gmail.com>
> > > wrote:
> > > > J-D,
> > > >
> > > > Thanks a lot for the insightful comments.
> > > > We are running 5 zookeeper quorums now. On these 5 quorum servers, we
> > > also
> > > > run datanode, tasktracker and regionserver.
> > > > Now we have 6 disks per machine.
> > > > If it is the disk IO saturation, is there anyway to configure
> zookeeper
> > > to
> > > > write to a seperate disk which is not the same one used by HDFS?
> > > > Or shall we run zookeeper quorums on different nodes other than HDFS?
> > > >
> > > > Are HLogs generated by Regionservers? Would it alleviate the issue if
> > we
> > > use
> > > > the first 14 for regionserver, and the rest of 5 for zookeeper?(In
> that
> > > > sense, the node running zookeeper quorum won't run regionserver)
> > > >
> > > > I am very appreciated for your comments.
> > > >
> > > > Best,
> > > > zhenyu
> > > >
> > > >
> > > >
> > > > On Wed, Nov 11, 2009 at 2:30 PM, Jean-Daniel Cryans <
> > jdcryans@apache.org
> > > >wrote:
> > > >
> > > >> The problem seems to be that the IO is saturated. When you see this:
> > > >>
> > > >> #
> > > >> 2009-11-11 04:28:10,308 WARN
> > > >> org.apache.hadoop.hbase.regionserver.HLog: IPC Server handler 9 on
> > > >> 60021 took 6819ms appending an edit to hlog; editcount=370
> > > >> #
> > > >> 2009-11-11 04:28:10,308 WARN
> > > >> org.apache.hadoop.hbase.regionserver.HLog: IPC Server handler 2 on
> > > >> 60021 took 4412ms appending an edit to hlog; editcount=371
> > > >>
> > > >> It's a sign that HDFS is slow. I guess that the zookeeper quorum
> > > >> members are on those same nodes using the same disks? If so, there's
> > > >> the problem because Zookeeper needs to be able to commit
> transactions
> > > >> very fast and if your disks are already hogged then they have to
> wait
> > > >> and you can get disconnections. Do you still have 19 quorum members?
> > > >> How many disks do you have per machine?
> > > >>
> > > >> Thx
> > > >>
> > > >> J-D
> > > >>
> > > >> On Wed, Nov 11, 2009 at 11:22 AM, Zhenyu Zhong <
> > zhongresearch@gmail.com
> > > >
> > > >> wrote:
> > > >> > Please take a look at the following regionserver log.
> > > >> > The error started to come out from 4:24am UTC to 4:29am UTC.
> > > >> >
> > > >> > http://pastebin.com/m3717d064
> > > >> >
> > > >> > Thanks in advance.
> > > >> >
> > > >> > Best,
> > > >> > zhenyu
> > > >> >
> > > >> >
> > > >> > On Wed, Nov 11, 2009 at 2:01 PM, Jean-Daniel Cryans <
> > > jdcryans@apache.org
> > > >> >wrote:
> > > >> >
> > > >> >> Could you paste a bigger log trace (interested to see at least
> all
> > > 300
> > > >> >> lines before that) in a pastebin?
> > > >> >>
> > > >> >> Thx,
> > > >> >>
> > > >> >> J-D
> > > >> >>
> > > >> >> On Wed, Nov 11, 2009 at 10:56 AM, Zhenyu Zhong <
> > > zhongresearch@gmail.com
> > > >> >
> > > >> >> wrote:
> > > >> >> > J-D,
> > > >> >> >
> > > >> >> > Thanks for the reply.
> > > >> >> >
> > > >> >> > The problem I have is one or more of the regionservers
> disconnect
> > > from
> > > >> >> the
> > > >> >> > cluster.
> > > >> >> >
> > > >> >> > I checked the resource usage including memory Heap usage for
> > > >> >> regionserver,
> > > >> >> > and found nothing suspicious. The regionserver heap usage is
> way
> > > below
> > > >> >> the
> > > >> >> > maxHeap defined in jvm.
> > > >> >> >
> > > >> >> > So I checked zookeeper log, then I found the following
> > > >> >> > BufferUnderflowException, eventually it leads to
> SocketException:
> > > >> >> Transport
> > > >> >> > endpoint is not connected.
> > > >> >> >
> > > >> >> >
> > > >> >> > I also checked the regionserver log. It looks like the
> > regionserver
> > > >> tried
> > > >> >> to
> > > >> >> > connect to other zookeeper quorum server. It connects
> > successfully,
> > > >> but
> > > >> >> then
> > > >> >> > it complained about SocketException: transport endpoint is not
> > > >> connected.
> > > >> >> >
> > > >> >> > So both zookeeper log and regionserver log told me that the
> > > transport
> > > >> >> > endpoint is not connected. I don't know why.
> > > >> >> >
> > > >> >> > I would be very appreciated if you could point me to the right
> > > >> direction.
> > > >> >> >
> > > >> >> > Best,
> > > >> >> >
> > > >> >> >
> > > >> >> > Regionserver log:
> > > >> >> > 2009-11-11 04:24:53,718 INFO org.apache.zookeeper.ClientCnxn:
> > > >> Attempting
> > > >> >> > connection to server superpyxis0018/192.168.100.132:2181
> > > >> >> > 2009-11-11 04:24:53,718 INFO org.apache.zookeeper.ClientCnxn:
> > > Priming
> > > >> >> > connection to java.nio.channels.SocketChannel[connected local=/
> > > >> >> > 192.168.100.131:43065 remote=superpyxis0018/
> 192.168.100.132:2181
> > ]
> > > >> >> > 2009-11-11 04:24:53,718 INFO org.apache.zookeeper.ClientCnxn:
> > > Server
> > > >> >> > connection successful
> > > >> >> > 2009-11-11 04:24:53,768 INFO
> > > >> >> > org.apache.hadoop.hbase.regionserver.HRegionServer: Got
> ZooKeeper
> > > >> event,
> > > >> >> > state: Disconnected, type: None, path: null
> > > >> >> > 2009-11-11 04:24:54,118 INFO org.apache.zookeeper.ClientCnxn:
> > > >> Attempting
> > > >> >> > connection to server superpyxis0017/192.168.100.131:2181
> > > >> >> > 2009-11-11 04:24:54,118 INFO org.apache.zookeeper.ClientCnxn:
> > > Priming
> > > >> >> > connection to java.nio.channels.SocketChannel[connected local=/
> > > >> >> > 192.168.100.131:46903 remote=superpyxis0017/
> 192.168.100.131:2181
> > ]
> > > >> >> > 2009-11-11 04:24:54,118 INFO org.apache.zookeeper.ClientCnxn:
> > > Server
> > > >> >> > connection successful
> > > >> >> > 2009-11-11 04:24:54,118 WARN org.apache.zookeeper.ClientCnxn:
> > > >> Exception
> > > >> >> > closing session 0x24dfd90c810002 to
> > > >> sun.nio.ch.SelectionKeyImpl@2dd78ccf
> > > >> >> > java.io.IOException: Read error rc = -1
> > > >> java.nio.DirectByteBuffer[pos=0
> > > >> >> > lim=4 cap=4]
> > > >> >> > at
> > > >> >> >
> > > org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
> > > >> >> > at
> > > >> >> >
> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
> > > >> >> > 2009-11-11 04:24:54,118 WARN org.apache.zookeeper.ClientCnxn:
> > > Ignoring
> > > >> >> > exception during shutdown input
> > > >> >> > java.net.SocketException: Transport endpoint is not connected
> > > >> >> > at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
> > > >> >> > at
> > > >> >> >
> > > sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
> > > >> >> > at
> > > >> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> > > >> >> > at
> > > >> >> >
> > > >>
> > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> > > >> >> > at
> > > >> >> >
> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> > > >> >> >
> > > >> >> >
> > > >> >> >
> > > >> >> >
> > > >> >> >
> > > >> >> > On Wed, Nov 11, 2009 at 1:40 PM, Jean-Daniel Cryans <
> > > >> jdcryans@apache.org
> > > >> >> >wrote:
> > > >> >> >
> > > >> >> >> I'm really not sure exactly what the situation is with the
> > > >> >> >> java.nio.BufferUnderflowException, but what do you mean by
> > > >> >> >> "regionserver disconnection"? Could you give more details?
> > > >> >> >>
> > > >> >> >> Thx,
> > > >> >> >>
> > > >> >> >> J-D
> > > >> >> >>
> > > >> >> >> On Wed, Nov 11, 2009 at 10:34 AM, Zhenyu Zhong <
> > > >> zhongresearch@gmail.com
> > > >> >> >
> > > >> >> >> wrote:
> > > >> >> >> > I searched around and found HBASE-1645 "HBase could be a
> > little
> > > >> more
> > > >> >> >> dainty
> > > >> >> >> > when zk is pulled out from under it"
> > > >> >> >> >
> > > >> >> >> > I am wondering whether my problem is related to this
> > HBase-1645.
> > > Or
> > > >> >> >> probably
> > > >> >> >> > has nothing to do with it?
> > > >> >> >> >
> > > >> >> >> > thanks
> > > >> >> >> >
> > > >> >> >> >
> > > >> >> >> > On Wed, Nov 11, 2009 at 12:02 PM, Zhenyu Zhong <
> > > >> >> zhongresearch@gmail.com
> > > >> >> >> >wrote:
> > > >> >> >> >
> > > >> >> >> >> Hi,
> > > >> >> >> >>
> > > >> >> >> >> I have been monitoring the system resources such as memory
> > Heap
> > > >> usage
> > > >> >> >> for
> > > >> >> >> >> Regionserver, load average, network bandwidth, etc.
> > > >> >> >> >>
> > > >> >> >> >> I experienced a disconnection last night. When I checked
> the
> > > >> memory
> > > >> >> Heap
> > > >> >> >> >> for Regionserver, it was 2336 which is way below the max
> Heap
> > > >> 4096. I
> > > >> >> >> don't
> > > >> >> >> >> think it is Regionserver problem. The requests that the
> > > >> regionserver
> > > >> >> got
> > > >> >> >> was
> > > >> >> >> >> about 126 which doesn't seem to be high. The load average
> is
> > > kind
> > > >> of
> > > >> >> >> high
> > > >> >> >> >> >100. (but I saw disconnections while the load average is
> > > around
> > > >> 30).
> > > >> >> >> The
> > > >> >> >> >> bandwidth is not a problem.
> > > >> >> >> >>
> > > >> >> >> >> However, I found error messages from the zookeeper. Please
> > see
> > > the
> > > >> >> >> >> following error messages. There are some
> > > >> >> >> >> BufferUnderflowException error, IOException error,etc.
> > > >> >> >> >> May I ask what might be the causes that generate these
> error
> > > >> >> messages?
> > > >> >> >> >>
> > > >> >> >> >> Many thanks
> > > >> >> >> >> zhenyu
> > > >> >> >> >>
> > > >> >> >> >>
> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
> > > org.apache.zookeeper.server.Request:
> > > >> >> >> Ignoring
> > > >> >> >> >> exception during toString
> > > >> >> >> >> java.nio.BufferUnderflowException
> > > >> >> >> >> at
> > java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
> > > >> >> >> >> at java.nio.ByteBuffer.get(ByteBuffer.java:675)
> > > >> >> >> >> at
> > > >> >> >> org.apache.zookeeper.server.Request.toString(Request.java:199)
> > > >> >> >> >> at java.lang.String.valueOf(String.java:2827)
> > > >> >> >> >> at
> > > java.lang.StringBuilder.append(StringBuilder.java:115)
> > > >> >> >> >> at
> > > >> >> >> >>
> > > >> >> >>
> > > >> >>
> > > >>
> > >
> >
> org.apache.zookeeper.server.quorum.CommitProcessor.processRequest(CommitProcessor.java:167)
> > > >> >> >> >> at
> > > >> >> >> >>
> > > >> >> >>
> > > >> >>
> > > >>
> > >
> >
> org.apache.zookeeper.server.quorum.FollowerRequestProcessor.run(FollowerRequestProcessor.java:68)
> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
> > > >> >> >> >> Exception causing close of session 0x424dfd908c50009
> > > >> >> >> >> due to java.io.IOException: Read error
> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
> > > >> >> >> >> IOException stack trace
> > > >> >> >> >> java.io.IOException: Read error
> > > >> >> >> >> at
> > > >> >> >> >>
> > > >> >>
> > > org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:372)
> > > >> >> >> >> at
> > > >> >> >> >>
> > > >> >> >>
> > > >> >>
> > > >>
> > >
> >
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:221)
> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
> > > >> >> >> >> org.apache.zookeeper.server.quorum.CommitProcessor:
> > Processing
> > > >> >> request::
> > > >> >> >> >> sessionid:0x424dfd908c50009 type:setWatches
> > > >> cxid:0xfffffffffffffff8
> > > >> >> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
> > > org.apache.zookeeper.server.Request:
> > > >> >> >> Ignoring
> > > >> >> >> >> exception during toString
> > > >> >> >> >> java.nio.BufferUnderflowException
> > > >> >> >> >> at
> > java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
> > > >> >> >> >> at java.nio.ByteBuffer.get(ByteBuffer.java:675)
> > > >> >> >> >> at
> > > >> >> >> org.apache.zookeeper.server.Request.toString(Request.java:199)
> > > >> >> >> >> at java.lang.String.valueOf(String.java:2827)
> > > >> >> >> >> at
> > > java.lang.StringBuilder.append(StringBuilder.java:115)
> > > >> >> >> >> at
> > > >> >> >> >>
> > > >> >> >>
> > > >> >>
> > > >>
> > >
> >
> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:74)
> > > >> >> >> >> at
> > > >> >> >> >>
> > > >> >> >>
> > > >> >>
> > > >>
> > >
> >
> org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)
> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
> > > >> >> >> >> org.apache.zookeeper.server.FinalRequestProcessor:
> Processing
> > > >> >> request::
> > > >> >> >> >> sessionid:0x424dfd908c50009 type:setWatches
> > > >> cxid:0xfffffffffffffff8
> > > >> >> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
> > > org.apache.zookeeper.server.Request:
> > > >> >> >> Ignoring
> > > >> >> >> >> exception during toString
> > > >> >> >> >> java.nio.BufferUnderflowException
> > > >> >> >> >> at
> > java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
> > > >> >> >> >> at java.nio.ByteBuffer.get(ByteBuffer.java:675)
> > > >> >> >> >> at
> > > >> >> >> org.apache.zookeeper.server.Request.toString(Request.java:199)
> > > >> >> >> >> at
> > > >> >> >> >>
> > > >>
> org.apache.log4j.or.DefaultRenderer.doRender(DefaultRenderer.java:36)
> > > >> >> >> >> at
> > > >> >> >> >>
> > > org.apache.log4j.or.RendererMap.findAndRender(RendererMap.java:80)
> > > >> >> >> >> at
> > > >> >> >> >>
> > > >> >> >>
> > > >> >>
> > > >>
> > >
> >
> org.apache.log4j.spi.LoggingEvent.getRenderedMessage(LoggingEvent.java:362)
> > > >> >> >> >> at
> > > >> >> >> >>
> > > >> >> >>
> > > >> >>
> > > >>
> > >
> >
> org.apache.log4j.helpers.PatternParser$BasicPatternConverter.convert(PatternParser.java:403)
> > > >> >> >> >> at
> > > >> >> >> >>
> > > >> >> >>
> > > >> >>
> > > >>
> > >
> >
> org.apache.log4j.helpers.PatternConverter.format(PatternConverter.java:65)
> > > >> >> >> >> at
> > > >> >> org.apache.log4j.PatternLayout.format(PatternLayout.java:502)
> > > >> >> >> >> at
> > > >> >> >> >>
> > > org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:302)
> > > >> >> >> >> at
> > > >> >> >> >>
> > > >> >> >>
> > > >> >>
> > > >>
> > >
> >
> org.apache.log4j.DailyRollingFileAppender.subAppend(DailyRollingFileAppender.java:359)
> > > >> >> >> >> at
> > > >> >> >>
> org.apache.log4j.WriterAppender.append(WriterAppender.java:160)
> > > >> >> >> >> at
> > > >> >> >> >>
> > > >>
> org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
> > > >> >> >> >> at
> > > >> >> >> >>
> > > >> >> >>
> > > >> >>
> > > >>
> > >
> >
> org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
> > > >> >> >> >> at
> > > >> org.apache.log4j.Category.callAppenders(Category.java:206)
> > > >> >> >> >> at
> > > org.apache.log4j.Category.forcedLog(Category.java:391)
> > > >> >> >> >> at
> org.apache.log4j.Category.debug(Category.java:260)
> > > >> >> >> >> at
> > > >> >> >> >>
> > > >> >> >>
> > > >> >>
> > > >>
> > >
> >
> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:147)
> > > >> >> >> >> at
> > > >> >> >> >>
> > > >> >> >>
> > > >> >>
> > > >>
> > >
> >
> org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)
> > > >> >> >> >> 2009-11-11 04:29:00,810 WARN
> > > org.apache.zookeeper.server.Request:
> > > >> >> >> Ignoring
> > > >> >> >> >> exception during toString
> > > >> >> >> >> sessionid:0x424dfd908c50009 type:setWatches
> > > >> cxid:0xfffffffffffffff8
> > > >> >> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
> > > >> >> >> >> 2009-11-11 04:29:00,810 INFO
> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
> > > >> >> >> >> closing session:0x424dfd908c50009 NIOServerCnxn:
> > > >> >> >> >> java.nio.channels.SocketChannel[connected local=/
> > > >> >> 192.168.100.131:2181
> > > >> >> >> remote=/
> > > >> >> >> >> 192.168.100.131:35500]
> > > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
> > > >> >> org.apache.zookeeper.server.NIOServerCnxn:
> > > >> >> >> >> ignoring exception during input shutdown
> > > >> >> >> >> java.net.SocketException: Transport endpoint is not
> connected
> > > >> >> >> >> at sun.nio.ch.SocketChannelImpl.shutdown(Native
> > Method)
> > > >> >> >> >> at
> > > >> >> >> >>
> > > >> >>
> > > sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
> > > >> >> >> >> at
> > > >> >> >> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> > > >> >> >> >> at
> > > >> >> >> >>
> > > >> >>
> > > org.apache.zookeeper.server.NIOServerCnxn.close(NIOServerCnxn.java:848)
> > > >> >> >> >> at
> > > >> >> >> >>
> > > >> >>
> > > org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:498)
> > > >> >> >> >> at
> > > >> >> >> >>
> > > >> >> >>
> > > >> >>
> > > >>
> > >
> >
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:221)
> > > >> >> >> >>
> > > >> >> >> >
> > > >> >> >>
> > > >> >> >
> > > >> >>
> > > >> >
> > > >>
> > > >
> > >
> >
>
Re: regionserver disconnection
Posted by stack <st...@duboce.net>.
I see these in your log too:
1. 2009-11-11 04:27:19,018 DEBUG org.apache.zookeeper.ClientCnxn: Got
ping response for sessionid:0x424dfd908c50009 after 4544ms
2. 2009-11-11 04:27:19,018 DEBUG org.apache.zookeeper.ClientCnxn: Got
ping response for sessionid:0x24dfd90c810002 after 4548ms
3. 2009-11-11 04:27:43,960 DEBUG org.apache.zookeeper.ClientCnxn: Got
ping response for sessionid:0x424dfd908c50009 after 9030ms
4. 2009-11-11 04:27:43,960 DEBUG org.apache.zookeeper.ClientCnxn: Got
ping response for sessionid:0x24dfd90c810002 after 9030ms
5. 2009-11-11 04:27:48,830 DEBUG org.apache.zookeeper.ClientCnxn: Got
ping response for sessionid:0x424dfd908c50009 after 0ms
6. 2009-11-11 04:27:48,830 DEBUG org.apache.zookeeper.ClientCnxn: Got
ping response for sessionid:0x24dfd90c810002 after 0ms
Which would indicate lag talking to zk too.
Appending HLog also seems to get steadily worse.
I also see these:
1. 2009-11-11 04:27:01,233 WARN org.apache.zookeeper.ClientCnxn:
Exception closing session 0x24dfd90c810002 to
sun.nio.ch.SelectionKeyImpl@5e16a7e3
2. java.io.IOException: Read error rc = -1
java.nio.DirectByteBuffer[pos=0 lim=4 cap=4]
3. at
org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
4. at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
Were all 5 quorum members participating? Check by doing "echo stat|nc
QUORUM_HOST port" for each member of quorum.... the below might help:
members="1,2,3,4,5"
while true; do date; for i in $members; do echo $i; echo stat | nc $i 2181;
done; sleep 10; done
We need to improve our zk_dump. It just reports quorum members... not if
they are actually participating or not. The above will tell you if dead or
alive.
Please report back after checking above and following the Jean-Daniel
prescription.
Thanks,
St.Ack
On Wed, Nov 11, 2009 at 12:29 PM, Zhenyu Zhong <zh...@gmail.com>wrote:
> Thanks a lot for the help.
>
> I understand that the change of disk configurations should be done
> progressively such that HDFS can balance and recover from a disk going
> away.
>
> The data we are trying to import into HBase is about 200million rows, each
> row is about hundreds of bytes. Basically, we read in a text file from
> HDFS,
> then for each row in the text file, we fetch data from HBase table, we
> compare the column value and update the latest value back into HBase. I was
> suggested to use CheckAndPut. Anyway this is the common usecase on my
> side.
>
>
> Many thanks
> zhenyu
>
>
>
> On Wed, Nov 11, 2009 at 3:03 PM, Jean-Daniel Cryans <jdcryans@apache.org
> >wrote:
>
> > 6 disks, 19 machines and you see those kind of issues? I'm quite
> > surprised and I want to learn more about what you trying to to.
> >
> > But first, if hbase is managing ZK for you then you can change the
> > value of hbase.zookeeper.property.dataDir so that it points to its own
> > mount. Make sure you also change dfs.data.dir in conf/hdfs-site.xml
> > (hadoop stuff) so that it stops pointing there (you should have 6
> > values there at the moment if you configured it correctly). Doing this
> > could result in data loss if you do it all at once so do the change
> > progressively. This also means that you may have to keep a seperate
> > HDFS configuration for those particular nodes. Please ask more
> > questions if it's not very clear.
> >
> > WRT to dedicating nodes to Zookeeeper, it makes sense only if you have
> > a lot of nodes but in your case, unless there's also CPU contention,
> > you're good with only a dedicated disk.
> >
> > Now for what you are trying to do, is this a massive import that goes
> > over many hours? How big is each row? I'd love to get a lot of details
> > to help you optimize your cluster for your particular problem.
> >
> > Thx,
> >
> > J-D
> >
> > On Wed, Nov 11, 2009 at 11:51 AM, Zhenyu Zhong <zh...@gmail.com>
> > wrote:
> > > J-D,
> > >
> > > Thanks a lot for the insightful comments.
> > > We are running 5 zookeeper quorums now. On these 5 quorum servers, we
> > also
> > > run datanode, tasktracker and regionserver.
> > > Now we have 6 disks per machine.
> > > If it is the disk IO saturation, is there anyway to configure zookeeper
> > to
> > > write to a seperate disk which is not the same one used by HDFS?
> > > Or shall we run zookeeper quorums on different nodes other than HDFS?
> > >
> > > Are HLogs generated by Regionservers? Would it alleviate the issue if
> we
> > use
> > > the first 14 for regionserver, and the rest of 5 for zookeeper?(In that
> > > sense, the node running zookeeper quorum won't run regionserver)
> > >
> > > I am very appreciated for your comments.
> > >
> > > Best,
> > > zhenyu
> > >
> > >
> > >
> > > On Wed, Nov 11, 2009 at 2:30 PM, Jean-Daniel Cryans <
> jdcryans@apache.org
> > >wrote:
> > >
> > >> The problem seems to be that the IO is saturated. When you see this:
> > >>
> > >> #
> > >> 2009-11-11 04:28:10,308 WARN
> > >> org.apache.hadoop.hbase.regionserver.HLog: IPC Server handler 9 on
> > >> 60021 took 6819ms appending an edit to hlog; editcount=370
> > >> #
> > >> 2009-11-11 04:28:10,308 WARN
> > >> org.apache.hadoop.hbase.regionserver.HLog: IPC Server handler 2 on
> > >> 60021 took 4412ms appending an edit to hlog; editcount=371
> > >>
> > >> It's a sign that HDFS is slow. I guess that the zookeeper quorum
> > >> members are on those same nodes using the same disks? If so, there's
> > >> the problem because Zookeeper needs to be able to commit transactions
> > >> very fast and if your disks are already hogged then they have to wait
> > >> and you can get disconnections. Do you still have 19 quorum members?
> > >> How many disks do you have per machine?
> > >>
> > >> Thx
> > >>
> > >> J-D
> > >>
> > >> On Wed, Nov 11, 2009 at 11:22 AM, Zhenyu Zhong <
> zhongresearch@gmail.com
> > >
> > >> wrote:
> > >> > Please take a look at the following regionserver log.
> > >> > The error started to come out from 4:24am UTC to 4:29am UTC.
> > >> >
> > >> > http://pastebin.com/m3717d064
> > >> >
> > >> > Thanks in advance.
> > >> >
> > >> > Best,
> > >> > zhenyu
> > >> >
> > >> >
> > >> > On Wed, Nov 11, 2009 at 2:01 PM, Jean-Daniel Cryans <
> > jdcryans@apache.org
> > >> >wrote:
> > >> >
> > >> >> Could you paste a bigger log trace (interested to see at least all
> > 300
> > >> >> lines before that) in a pastebin?
> > >> >>
> > >> >> Thx,
> > >> >>
> > >> >> J-D
> > >> >>
> > >> >> On Wed, Nov 11, 2009 at 10:56 AM, Zhenyu Zhong <
> > zhongresearch@gmail.com
> > >> >
> > >> >> wrote:
> > >> >> > J-D,
> > >> >> >
> > >> >> > Thanks for the reply.
> > >> >> >
> > >> >> > The problem I have is one or more of the regionservers disconnect
> > from
> > >> >> the
> > >> >> > cluster.
> > >> >> >
> > >> >> > I checked the resource usage including memory Heap usage for
> > >> >> regionserver,
> > >> >> > and found nothing suspicious. The regionserver heap usage is way
> > below
> > >> >> the
> > >> >> > maxHeap defined in jvm.
> > >> >> >
> > >> >> > So I checked zookeeper log, then I found the following
> > >> >> > BufferUnderflowException, eventually it leads to SocketException:
> > >> >> Transport
> > >> >> > endpoint is not connected.
> > >> >> >
> > >> >> >
> > >> >> > I also checked the regionserver log. It looks like the
> regionserver
> > >> tried
> > >> >> to
> > >> >> > connect to other zookeeper quorum server. It connects
> successfully,
> > >> but
> > >> >> then
> > >> >> > it complained about SocketException: transport endpoint is not
> > >> connected.
> > >> >> >
> > >> >> > So both zookeeper log and regionserver log told me that the
> > transport
> > >> >> > endpoint is not connected. I don't know why.
> > >> >> >
> > >> >> > I would be very appreciated if you could point me to the right
> > >> direction.
> > >> >> >
> > >> >> > Best,
> > >> >> >
> > >> >> >
> > >> >> > Regionserver log:
> > >> >> > 2009-11-11 04:24:53,718 INFO org.apache.zookeeper.ClientCnxn:
> > >> Attempting
> > >> >> > connection to server superpyxis0018/192.168.100.132:2181
> > >> >> > 2009-11-11 04:24:53,718 INFO org.apache.zookeeper.ClientCnxn:
> > Priming
> > >> >> > connection to java.nio.channels.SocketChannel[connected local=/
> > >> >> > 192.168.100.131:43065 remote=superpyxis0018/192.168.100.132:2181
> ]
> > >> >> > 2009-11-11 04:24:53,718 INFO org.apache.zookeeper.ClientCnxn:
> > Server
> > >> >> > connection successful
> > >> >> > 2009-11-11 04:24:53,768 INFO
> > >> >> > org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper
> > >> event,
> > >> >> > state: Disconnected, type: None, path: null
> > >> >> > 2009-11-11 04:24:54,118 INFO org.apache.zookeeper.ClientCnxn:
> > >> Attempting
> > >> >> > connection to server superpyxis0017/192.168.100.131:2181
> > >> >> > 2009-11-11 04:24:54,118 INFO org.apache.zookeeper.ClientCnxn:
> > Priming
> > >> >> > connection to java.nio.channels.SocketChannel[connected local=/
> > >> >> > 192.168.100.131:46903 remote=superpyxis0017/192.168.100.131:2181
> ]
> > >> >> > 2009-11-11 04:24:54,118 INFO org.apache.zookeeper.ClientCnxn:
> > Server
> > >> >> > connection successful
> > >> >> > 2009-11-11 04:24:54,118 WARN org.apache.zookeeper.ClientCnxn:
> > >> Exception
> > >> >> > closing session 0x24dfd90c810002 to
> > >> sun.nio.ch.SelectionKeyImpl@2dd78ccf
> > >> >> > java.io.IOException: Read error rc = -1
> > >> java.nio.DirectByteBuffer[pos=0
> > >> >> > lim=4 cap=4]
> > >> >> > at
> > >> >> >
> > org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
> > >> >> > at
> > >> >> >
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
> > >> >> > 2009-11-11 04:24:54,118 WARN org.apache.zookeeper.ClientCnxn:
> > Ignoring
> > >> >> > exception during shutdown input
> > >> >> > java.net.SocketException: Transport endpoint is not connected
> > >> >> > at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
> > >> >> > at
> > >> >> >
> > sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
> > >> >> > at
> > >> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> > >> >> > at
> > >> >> >
> > >>
> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> > >> >> > at
> > >> >> >
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> > >> >> >
> > >> >> >
> > >> >> >
> > >> >> >
> > >> >> >
> > >> >> > On Wed, Nov 11, 2009 at 1:40 PM, Jean-Daniel Cryans <
> > >> jdcryans@apache.org
> > >> >> >wrote:
> > >> >> >
> > >> >> >> I'm really not sure exactly what the situation is with the
> > >> >> >> java.nio.BufferUnderflowException, but what do you mean by
> > >> >> >> "regionserver disconnection"? Could you give more details?
> > >> >> >>
> > >> >> >> Thx,
> > >> >> >>
> > >> >> >> J-D
> > >> >> >>
> > >> >> >> On Wed, Nov 11, 2009 at 10:34 AM, Zhenyu Zhong <
> > >> zhongresearch@gmail.com
> > >> >> >
> > >> >> >> wrote:
> > >> >> >> > I searched around and found HBASE-1645 "HBase could be a
> little
> > >> more
> > >> >> >> dainty
> > >> >> >> > when zk is pulled out from under it"
> > >> >> >> >
> > >> >> >> > I am wondering whether my problem is related to this
> HBase-1645.
> > Or
> > >> >> >> probably
> > >> >> >> > has nothing to do with it?
> > >> >> >> >
> > >> >> >> > thanks
> > >> >> >> >
> > >> >> >> >
> > >> >> >> > On Wed, Nov 11, 2009 at 12:02 PM, Zhenyu Zhong <
> > >> >> zhongresearch@gmail.com
> > >> >> >> >wrote:
> > >> >> >> >
> > >> >> >> >> Hi,
> > >> >> >> >>
> > >> >> >> >> I have been monitoring the system resources such as memory
> Heap
> > >> usage
> > >> >> >> for
> > >> >> >> >> Regionserver, load average, network bandwidth, etc.
> > >> >> >> >>
> > >> >> >> >> I experienced a disconnection last night. When I checked the
> > >> memory
> > >> >> Heap
> > >> >> >> >> for Regionserver, it was 2336 which is way below the max Heap
> > >> 4096. I
> > >> >> >> don't
> > >> >> >> >> think it is Regionserver problem. The requests that the
> > >> regionserver
> > >> >> got
> > >> >> >> was
> > >> >> >> >> about 126 which doesn't seem to be high. The load average is
> > kind
> > >> of
> > >> >> >> high
> > >> >> >> >> >100. (but I saw disconnections while the load average is
> > around
> > >> 30).
> > >> >> >> The
> > >> >> >> >> bandwidth is not a problem.
> > >> >> >> >>
> > >> >> >> >> However, I found error messages from the zookeeper. Please
> see
> > the
> > >> >> >> >> following error messages. There are some
> > >> >> >> >> BufferUnderflowException error, IOException error,etc.
> > >> >> >> >> May I ask what might be the causes that generate these error
> > >> >> messages?
> > >> >> >> >>
> > >> >> >> >> Many thanks
> > >> >> >> >> zhenyu
> > >> >> >> >>
> > >> >> >> >>
> > >> >> >> >> 2009-11-11 04:29:00,810 WARN
> > org.apache.zookeeper.server.Request:
> > >> >> >> Ignoring
> > >> >> >> >> exception during toString
> > >> >> >> >> java.nio.BufferUnderflowException
> > >> >> >> >> at
> java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
> > >> >> >> >> at java.nio.ByteBuffer.get(ByteBuffer.java:675)
> > >> >> >> >> at
> > >> >> >> org.apache.zookeeper.server.Request.toString(Request.java:199)
> > >> >> >> >> at java.lang.String.valueOf(String.java:2827)
> > >> >> >> >> at
> > java.lang.StringBuilder.append(StringBuilder.java:115)
> > >> >> >> >> at
> > >> >> >> >>
> > >> >> >>
> > >> >>
> > >>
> >
> org.apache.zookeeper.server.quorum.CommitProcessor.processRequest(CommitProcessor.java:167)
> > >> >> >> >> at
> > >> >> >> >>
> > >> >> >>
> > >> >>
> > >>
> >
> org.apache.zookeeper.server.quorum.FollowerRequestProcessor.run(FollowerRequestProcessor.java:68)
> > >> >> >> >> 2009-11-11 04:29:00,810 WARN
> > >> >> org.apache.zookeeper.server.NIOServerCnxn:
> > >> >> >> >> Exception causing close of session 0x424dfd908c50009
> > >> >> >> >> due to java.io.IOException: Read error
> > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
> > >> >> org.apache.zookeeper.server.NIOServerCnxn:
> > >> >> >> >> IOException stack trace
> > >> >> >> >> java.io.IOException: Read error
> > >> >> >> >> at
> > >> >> >> >>
> > >> >>
> > org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:372)
> > >> >> >> >> at
> > >> >> >> >>
> > >> >> >>
> > >> >>
> > >>
> >
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:221)
> > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
> > >> >> >> >> org.apache.zookeeper.server.quorum.CommitProcessor:
> Processing
> > >> >> request::
> > >> >> >> >> sessionid:0x424dfd908c50009 type:setWatches
> > >> cxid:0xfffffffffffffff8
> > >> >> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
> > >> >> >> >> 2009-11-11 04:29:00,810 WARN
> > org.apache.zookeeper.server.Request:
> > >> >> >> Ignoring
> > >> >> >> >> exception during toString
> > >> >> >> >> java.nio.BufferUnderflowException
> > >> >> >> >> at
> java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
> > >> >> >> >> at java.nio.ByteBuffer.get(ByteBuffer.java:675)
> > >> >> >> >> at
> > >> >> >> org.apache.zookeeper.server.Request.toString(Request.java:199)
> > >> >> >> >> at java.lang.String.valueOf(String.java:2827)
> > >> >> >> >> at
> > java.lang.StringBuilder.append(StringBuilder.java:115)
> > >> >> >> >> at
> > >> >> >> >>
> > >> >> >>
> > >> >>
> > >>
> >
> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:74)
> > >> >> >> >> at
> > >> >> >> >>
> > >> >> >>
> > >> >>
> > >>
> >
> org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)
> > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
> > >> >> >> >> org.apache.zookeeper.server.FinalRequestProcessor: Processing
> > >> >> request::
> > >> >> >> >> sessionid:0x424dfd908c50009 type:setWatches
> > >> cxid:0xfffffffffffffff8
> > >> >> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
> > >> >> >> >> 2009-11-11 04:29:00,810 WARN
> > org.apache.zookeeper.server.Request:
> > >> >> >> Ignoring
> > >> >> >> >> exception during toString
> > >> >> >> >> java.nio.BufferUnderflowException
> > >> >> >> >> at
> java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
> > >> >> >> >> at java.nio.ByteBuffer.get(ByteBuffer.java:675)
> > >> >> >> >> at
> > >> >> >> org.apache.zookeeper.server.Request.toString(Request.java:199)
> > >> >> >> >> at
> > >> >> >> >>
> > >> org.apache.log4j.or.DefaultRenderer.doRender(DefaultRenderer.java:36)
> > >> >> >> >> at
> > >> >> >> >>
> > org.apache.log4j.or.RendererMap.findAndRender(RendererMap.java:80)
> > >> >> >> >> at
> > >> >> >> >>
> > >> >> >>
> > >> >>
> > >>
> >
> org.apache.log4j.spi.LoggingEvent.getRenderedMessage(LoggingEvent.java:362)
> > >> >> >> >> at
> > >> >> >> >>
> > >> >> >>
> > >> >>
> > >>
> >
> org.apache.log4j.helpers.PatternParser$BasicPatternConverter.convert(PatternParser.java:403)
> > >> >> >> >> at
> > >> >> >> >>
> > >> >> >>
> > >> >>
> > >>
> >
> org.apache.log4j.helpers.PatternConverter.format(PatternConverter.java:65)
> > >> >> >> >> at
> > >> >> org.apache.log4j.PatternLayout.format(PatternLayout.java:502)
> > >> >> >> >> at
> > >> >> >> >>
> > org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:302)
> > >> >> >> >> at
> > >> >> >> >>
> > >> >> >>
> > >> >>
> > >>
> >
> org.apache.log4j.DailyRollingFileAppender.subAppend(DailyRollingFileAppender.java:359)
> > >> >> >> >> at
> > >> >> >> org.apache.log4j.WriterAppender.append(WriterAppender.java:160)
> > >> >> >> >> at
> > >> >> >> >>
> > >> org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
> > >> >> >> >> at
> > >> >> >> >>
> > >> >> >>
> > >> >>
> > >>
> >
> org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
> > >> >> >> >> at
> > >> org.apache.log4j.Category.callAppenders(Category.java:206)
> > >> >> >> >> at
> > org.apache.log4j.Category.forcedLog(Category.java:391)
> > >> >> >> >> at org.apache.log4j.Category.debug(Category.java:260)
> > >> >> >> >> at
> > >> >> >> >>
> > >> >> >>
> > >> >>
> > >>
> >
> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:147)
> > >> >> >> >> at
> > >> >> >> >>
> > >> >> >>
> > >> >>
> > >>
> >
> org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)
> > >> >> >> >> 2009-11-11 04:29:00,810 WARN
> > org.apache.zookeeper.server.Request:
> > >> >> >> Ignoring
> > >> >> >> >> exception during toString
> > >> >> >> >> sessionid:0x424dfd908c50009 type:setWatches
> > >> cxid:0xfffffffffffffff8
> > >> >> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
> > >> >> >> >> 2009-11-11 04:29:00,810 INFO
> > >> >> org.apache.zookeeper.server.NIOServerCnxn:
> > >> >> >> >> closing session:0x424dfd908c50009 NIOServerCnxn:
> > >> >> >> >> java.nio.channels.SocketChannel[connected local=/
> > >> >> 192.168.100.131:2181
> > >> >> >> remote=/
> > >> >> >> >> 192.168.100.131:35500]
> > >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
> > >> >> org.apache.zookeeper.server.NIOServerCnxn:
> > >> >> >> >> ignoring exception during input shutdown
> > >> >> >> >> java.net.SocketException: Transport endpoint is not connected
> > >> >> >> >> at sun.nio.ch.SocketChannelImpl.shutdown(Native
> Method)
> > >> >> >> >> at
> > >> >> >> >>
> > >> >>
> > sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
> > >> >> >> >> at
> > >> >> >> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> > >> >> >> >> at
> > >> >> >> >>
> > >> >>
> > org.apache.zookeeper.server.NIOServerCnxn.close(NIOServerCnxn.java:848)
> > >> >> >> >> at
> > >> >> >> >>
> > >> >>
> > org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:498)
> > >> >> >> >> at
> > >> >> >> >>
> > >> >> >>
> > >> >>
> > >>
> >
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:221)
> > >> >> >> >>
> > >> >> >> >
> > >> >> >>
> > >> >> >
> > >> >>
> > >> >
> > >>
> > >
> >
>
Re: regionserver disconnection
Posted by Zhenyu Zhong <zh...@gmail.com>.
Thanks a lot for the help.
I understand that the change of disk configurations should be done
progressively such that HDFS can balance and recover from a disk going away.
The data we are trying to import into HBase is about 200million rows, each
row is about hundreds of bytes. Basically, we read in a text file from HDFS,
then for each row in the text file, we fetch data from HBase table, we
compare the column value and update the latest value back into HBase. I was
suggested to use CheckAndPut. Anyway this is the common usecase on my side.
Many thanks
zhenyu
On Wed, Nov 11, 2009 at 3:03 PM, Jean-Daniel Cryans <jd...@apache.org>wrote:
> 6 disks, 19 machines and you see those kind of issues? I'm quite
> surprised and I want to learn more about what you trying to to.
>
> But first, if hbase is managing ZK for you then you can change the
> value of hbase.zookeeper.property.dataDir so that it points to its own
> mount. Make sure you also change dfs.data.dir in conf/hdfs-site.xml
> (hadoop stuff) so that it stops pointing there (you should have 6
> values there at the moment if you configured it correctly). Doing this
> could result in data loss if you do it all at once so do the change
> progressively. This also means that you may have to keep a seperate
> HDFS configuration for those particular nodes. Please ask more
> questions if it's not very clear.
>
> WRT to dedicating nodes to Zookeeeper, it makes sense only if you have
> a lot of nodes but in your case, unless there's also CPU contention,
> you're good with only a dedicated disk.
>
> Now for what you are trying to do, is this a massive import that goes
> over many hours? How big is each row? I'd love to get a lot of details
> to help you optimize your cluster for your particular problem.
>
> Thx,
>
> J-D
>
> On Wed, Nov 11, 2009 at 11:51 AM, Zhenyu Zhong <zh...@gmail.com>
> wrote:
> > J-D,
> >
> > Thanks a lot for the insightful comments.
> > We are running 5 zookeeper quorums now. On these 5 quorum servers, we
> also
> > run datanode, tasktracker and regionserver.
> > Now we have 6 disks per machine.
> > If it is the disk IO saturation, is there anyway to configure zookeeper
> to
> > write to a seperate disk which is not the same one used by HDFS?
> > Or shall we run zookeeper quorums on different nodes other than HDFS?
> >
> > Are HLogs generated by Regionservers? Would it alleviate the issue if we
> use
> > the first 14 for regionserver, and the rest of 5 for zookeeper?(In that
> > sense, the node running zookeeper quorum won't run regionserver)
> >
> > I am very appreciated for your comments.
> >
> > Best,
> > zhenyu
> >
> >
> >
> > On Wed, Nov 11, 2009 at 2:30 PM, Jean-Daniel Cryans <jdcryans@apache.org
> >wrote:
> >
> >> The problem seems to be that the IO is saturated. When you see this:
> >>
> >> #
> >> 2009-11-11 04:28:10,308 WARN
> >> org.apache.hadoop.hbase.regionserver.HLog: IPC Server handler 9 on
> >> 60021 took 6819ms appending an edit to hlog; editcount=370
> >> #
> >> 2009-11-11 04:28:10,308 WARN
> >> org.apache.hadoop.hbase.regionserver.HLog: IPC Server handler 2 on
> >> 60021 took 4412ms appending an edit to hlog; editcount=371
> >>
> >> It's a sign that HDFS is slow. I guess that the zookeeper quorum
> >> members are on those same nodes using the same disks? If so, there's
> >> the problem because Zookeeper needs to be able to commit transactions
> >> very fast and if your disks are already hogged then they have to wait
> >> and you can get disconnections. Do you still have 19 quorum members?
> >> How many disks do you have per machine?
> >>
> >> Thx
> >>
> >> J-D
> >>
> >> On Wed, Nov 11, 2009 at 11:22 AM, Zhenyu Zhong <zhongresearch@gmail.com
> >
> >> wrote:
> >> > Please take a look at the following regionserver log.
> >> > The error started to come out from 4:24am UTC to 4:29am UTC.
> >> >
> >> > http://pastebin.com/m3717d064
> >> >
> >> > Thanks in advance.
> >> >
> >> > Best,
> >> > zhenyu
> >> >
> >> >
> >> > On Wed, Nov 11, 2009 at 2:01 PM, Jean-Daniel Cryans <
> jdcryans@apache.org
> >> >wrote:
> >> >
> >> >> Could you paste a bigger log trace (interested to see at least all
> 300
> >> >> lines before that) in a pastebin?
> >> >>
> >> >> Thx,
> >> >>
> >> >> J-D
> >> >>
> >> >> On Wed, Nov 11, 2009 at 10:56 AM, Zhenyu Zhong <
> zhongresearch@gmail.com
> >> >
> >> >> wrote:
> >> >> > J-D,
> >> >> >
> >> >> > Thanks for the reply.
> >> >> >
> >> >> > The problem I have is one or more of the regionservers disconnect
> from
> >> >> the
> >> >> > cluster.
> >> >> >
> >> >> > I checked the resource usage including memory Heap usage for
> >> >> regionserver,
> >> >> > and found nothing suspicious. The regionserver heap usage is way
> below
> >> >> the
> >> >> > maxHeap defined in jvm.
> >> >> >
> >> >> > So I checked zookeeper log, then I found the following
> >> >> > BufferUnderflowException, eventually it leads to SocketException:
> >> >> Transport
> >> >> > endpoint is not connected.
> >> >> >
> >> >> >
> >> >> > I also checked the regionserver log. It looks like the regionserver
> >> tried
> >> >> to
> >> >> > connect to other zookeeper quorum server. It connects successfully,
> >> but
> >> >> then
> >> >> > it complained about SocketException: transport endpoint is not
> >> connected.
> >> >> >
> >> >> > So both zookeeper log and regionserver log told me that the
> transport
> >> >> > endpoint is not connected. I don't know why.
> >> >> >
> >> >> > I would be very appreciated if you could point me to the right
> >> direction.
> >> >> >
> >> >> > Best,
> >> >> >
> >> >> >
> >> >> > Regionserver log:
> >> >> > 2009-11-11 04:24:53,718 INFO org.apache.zookeeper.ClientCnxn:
> >> Attempting
> >> >> > connection to server superpyxis0018/192.168.100.132:2181
> >> >> > 2009-11-11 04:24:53,718 INFO org.apache.zookeeper.ClientCnxn:
> Priming
> >> >> > connection to java.nio.channels.SocketChannel[connected local=/
> >> >> > 192.168.100.131:43065 remote=superpyxis0018/192.168.100.132:2181]
> >> >> > 2009-11-11 04:24:53,718 INFO org.apache.zookeeper.ClientCnxn:
> Server
> >> >> > connection successful
> >> >> > 2009-11-11 04:24:53,768 INFO
> >> >> > org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper
> >> event,
> >> >> > state: Disconnected, type: None, path: null
> >> >> > 2009-11-11 04:24:54,118 INFO org.apache.zookeeper.ClientCnxn:
> >> Attempting
> >> >> > connection to server superpyxis0017/192.168.100.131:2181
> >> >> > 2009-11-11 04:24:54,118 INFO org.apache.zookeeper.ClientCnxn:
> Priming
> >> >> > connection to java.nio.channels.SocketChannel[connected local=/
> >> >> > 192.168.100.131:46903 remote=superpyxis0017/192.168.100.131:2181]
> >> >> > 2009-11-11 04:24:54,118 INFO org.apache.zookeeper.ClientCnxn:
> Server
> >> >> > connection successful
> >> >> > 2009-11-11 04:24:54,118 WARN org.apache.zookeeper.ClientCnxn:
> >> Exception
> >> >> > closing session 0x24dfd90c810002 to
> >> sun.nio.ch.SelectionKeyImpl@2dd78ccf
> >> >> > java.io.IOException: Read error rc = -1
> >> java.nio.DirectByteBuffer[pos=0
> >> >> > lim=4 cap=4]
> >> >> > at
> >> >> >
> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
> >> >> > at
> >> >> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
> >> >> > 2009-11-11 04:24:54,118 WARN org.apache.zookeeper.ClientCnxn:
> Ignoring
> >> >> > exception during shutdown input
> >> >> > java.net.SocketException: Transport endpoint is not connected
> >> >> > at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
> >> >> > at
> >> >> >
> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
> >> >> > at
> >> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> >> >> > at
> >> >> >
> >> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> >> >> > at
> >> >> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >> > On Wed, Nov 11, 2009 at 1:40 PM, Jean-Daniel Cryans <
> >> jdcryans@apache.org
> >> >> >wrote:
> >> >> >
> >> >> >> I'm really not sure exactly what the situation is with the
> >> >> >> java.nio.BufferUnderflowException, but what do you mean by
> >> >> >> "regionserver disconnection"? Could you give more details?
> >> >> >>
> >> >> >> Thx,
> >> >> >>
> >> >> >> J-D
> >> >> >>
> >> >> >> On Wed, Nov 11, 2009 at 10:34 AM, Zhenyu Zhong <
> >> zhongresearch@gmail.com
> >> >> >
> >> >> >> wrote:
> >> >> >> > I searched around and found HBASE-1645 "HBase could be a little
> >> more
> >> >> >> dainty
> >> >> >> > when zk is pulled out from under it"
> >> >> >> >
> >> >> >> > I am wondering whether my problem is related to this HBase-1645.
> Or
> >> >> >> probably
> >> >> >> > has nothing to do with it?
> >> >> >> >
> >> >> >> > thanks
> >> >> >> >
> >> >> >> >
> >> >> >> > On Wed, Nov 11, 2009 at 12:02 PM, Zhenyu Zhong <
> >> >> zhongresearch@gmail.com
> >> >> >> >wrote:
> >> >> >> >
> >> >> >> >> Hi,
> >> >> >> >>
> >> >> >> >> I have been monitoring the system resources such as memory Heap
> >> usage
> >> >> >> for
> >> >> >> >> Regionserver, load average, network bandwidth, etc.
> >> >> >> >>
> >> >> >> >> I experienced a disconnection last night. When I checked the
> >> memory
> >> >> Heap
> >> >> >> >> for Regionserver, it was 2336 which is way below the max Heap
> >> 4096. I
> >> >> >> don't
> >> >> >> >> think it is Regionserver problem. The requests that the
> >> regionserver
> >> >> got
> >> >> >> was
> >> >> >> >> about 126 which doesn't seem to be high. The load average is
> kind
> >> of
> >> >> >> high
> >> >> >> >> >100. (but I saw disconnections while the load average is
> around
> >> 30).
> >> >> >> The
> >> >> >> >> bandwidth is not a problem.
> >> >> >> >>
> >> >> >> >> However, I found error messages from the zookeeper. Please see
> the
> >> >> >> >> following error messages. There are some
> >> >> >> >> BufferUnderflowException error, IOException error,etc.
> >> >> >> >> May I ask what might be the causes that generate these error
> >> >> messages?
> >> >> >> >>
> >> >> >> >> Many thanks
> >> >> >> >> zhenyu
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> 2009-11-11 04:29:00,810 WARN
> org.apache.zookeeper.server.Request:
> >> >> >> Ignoring
> >> >> >> >> exception during toString
> >> >> >> >> java.nio.BufferUnderflowException
> >> >> >> >> at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
> >> >> >> >> at java.nio.ByteBuffer.get(ByteBuffer.java:675)
> >> >> >> >> at
> >> >> >> org.apache.zookeeper.server.Request.toString(Request.java:199)
> >> >> >> >> at java.lang.String.valueOf(String.java:2827)
> >> >> >> >> at
> java.lang.StringBuilder.append(StringBuilder.java:115)
> >> >> >> >> at
> >> >> >> >>
> >> >> >>
> >> >>
> >>
> org.apache.zookeeper.server.quorum.CommitProcessor.processRequest(CommitProcessor.java:167)
> >> >> >> >> at
> >> >> >> >>
> >> >> >>
> >> >>
> >>
> org.apache.zookeeper.server.quorum.FollowerRequestProcessor.run(FollowerRequestProcessor.java:68)
> >> >> >> >> 2009-11-11 04:29:00,810 WARN
> >> >> org.apache.zookeeper.server.NIOServerCnxn:
> >> >> >> >> Exception causing close of session 0x424dfd908c50009
> >> >> >> >> due to java.io.IOException: Read error
> >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
> >> >> org.apache.zookeeper.server.NIOServerCnxn:
> >> >> >> >> IOException stack trace
> >> >> >> >> java.io.IOException: Read error
> >> >> >> >> at
> >> >> >> >>
> >> >>
> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:372)
> >> >> >> >> at
> >> >> >> >>
> >> >> >>
> >> >>
> >>
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:221)
> >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
> >> >> >> >> org.apache.zookeeper.server.quorum.CommitProcessor: Processing
> >> >> request::
> >> >> >> >> sessionid:0x424dfd908c50009 type:setWatches
> >> cxid:0xfffffffffffffff8
> >> >> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
> >> >> >> >> 2009-11-11 04:29:00,810 WARN
> org.apache.zookeeper.server.Request:
> >> >> >> Ignoring
> >> >> >> >> exception during toString
> >> >> >> >> java.nio.BufferUnderflowException
> >> >> >> >> at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
> >> >> >> >> at java.nio.ByteBuffer.get(ByteBuffer.java:675)
> >> >> >> >> at
> >> >> >> org.apache.zookeeper.server.Request.toString(Request.java:199)
> >> >> >> >> at java.lang.String.valueOf(String.java:2827)
> >> >> >> >> at
> java.lang.StringBuilder.append(StringBuilder.java:115)
> >> >> >> >> at
> >> >> >> >>
> >> >> >>
> >> >>
> >>
> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:74)
> >> >> >> >> at
> >> >> >> >>
> >> >> >>
> >> >>
> >>
> org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)
> >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
> >> >> >> >> org.apache.zookeeper.server.FinalRequestProcessor: Processing
> >> >> request::
> >> >> >> >> sessionid:0x424dfd908c50009 type:setWatches
> >> cxid:0xfffffffffffffff8
> >> >> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
> >> >> >> >> 2009-11-11 04:29:00,810 WARN
> org.apache.zookeeper.server.Request:
> >> >> >> Ignoring
> >> >> >> >> exception during toString
> >> >> >> >> java.nio.BufferUnderflowException
> >> >> >> >> at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
> >> >> >> >> at java.nio.ByteBuffer.get(ByteBuffer.java:675)
> >> >> >> >> at
> >> >> >> org.apache.zookeeper.server.Request.toString(Request.java:199)
> >> >> >> >> at
> >> >> >> >>
> >> org.apache.log4j.or.DefaultRenderer.doRender(DefaultRenderer.java:36)
> >> >> >> >> at
> >> >> >> >>
> org.apache.log4j.or.RendererMap.findAndRender(RendererMap.java:80)
> >> >> >> >> at
> >> >> >> >>
> >> >> >>
> >> >>
> >>
> org.apache.log4j.spi.LoggingEvent.getRenderedMessage(LoggingEvent.java:362)
> >> >> >> >> at
> >> >> >> >>
> >> >> >>
> >> >>
> >>
> org.apache.log4j.helpers.PatternParser$BasicPatternConverter.convert(PatternParser.java:403)
> >> >> >> >> at
> >> >> >> >>
> >> >> >>
> >> >>
> >>
> org.apache.log4j.helpers.PatternConverter.format(PatternConverter.java:65)
> >> >> >> >> at
> >> >> org.apache.log4j.PatternLayout.format(PatternLayout.java:502)
> >> >> >> >> at
> >> >> >> >>
> org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:302)
> >> >> >> >> at
> >> >> >> >>
> >> >> >>
> >> >>
> >>
> org.apache.log4j.DailyRollingFileAppender.subAppend(DailyRollingFileAppender.java:359)
> >> >> >> >> at
> >> >> >> org.apache.log4j.WriterAppender.append(WriterAppender.java:160)
> >> >> >> >> at
> >> >> >> >>
> >> org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
> >> >> >> >> at
> >> >> >> >>
> >> >> >>
> >> >>
> >>
> org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
> >> >> >> >> at
> >> org.apache.log4j.Category.callAppenders(Category.java:206)
> >> >> >> >> at
> org.apache.log4j.Category.forcedLog(Category.java:391)
> >> >> >> >> at org.apache.log4j.Category.debug(Category.java:260)
> >> >> >> >> at
> >> >> >> >>
> >> >> >>
> >> >>
> >>
> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:147)
> >> >> >> >> at
> >> >> >> >>
> >> >> >>
> >> >>
> >>
> org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)
> >> >> >> >> 2009-11-11 04:29:00,810 WARN
> org.apache.zookeeper.server.Request:
> >> >> >> Ignoring
> >> >> >> >> exception during toString
> >> >> >> >> sessionid:0x424dfd908c50009 type:setWatches
> >> cxid:0xfffffffffffffff8
> >> >> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
> >> >> >> >> 2009-11-11 04:29:00,810 INFO
> >> >> org.apache.zookeeper.server.NIOServerCnxn:
> >> >> >> >> closing session:0x424dfd908c50009 NIOServerCnxn:
> >> >> >> >> java.nio.channels.SocketChannel[connected local=/
> >> >> 192.168.100.131:2181
> >> >> >> remote=/
> >> >> >> >> 192.168.100.131:35500]
> >> >> >> >> 2009-11-11 04:29:00,810 DEBUG
> >> >> org.apache.zookeeper.server.NIOServerCnxn:
> >> >> >> >> ignoring exception during input shutdown
> >> >> >> >> java.net.SocketException: Transport endpoint is not connected
> >> >> >> >> at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
> >> >> >> >> at
> >> >> >> >>
> >> >>
> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
> >> >> >> >> at
> >> >> >> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> >> >> >> >> at
> >> >> >> >>
> >> >>
> org.apache.zookeeper.server.NIOServerCnxn.close(NIOServerCnxn.java:848)
> >> >> >> >> at
> >> >> >> >>
> >> >>
> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:498)
> >> >> >> >> at
> >> >> >> >>
> >> >> >>
> >> >>
> >>
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:221)
> >> >> >> >>
> >> >> >> >
> >> >> >>
> >> >> >
> >> >>
> >> >
> >>
> >
>
Re: regionserver disconnection
Posted by Jean-Daniel Cryans <jd...@apache.org>.
6 disks, 19 machines and you see those kind of issues? I'm quite
surprised and I want to learn more about what you trying to to.
But first, if hbase is managing ZK for you then you can change the
value of hbase.zookeeper.property.dataDir so that it points to its own
mount. Make sure you also change dfs.data.dir in conf/hdfs-site.xml
(hadoop stuff) so that it stops pointing there (you should have 6
values there at the moment if you configured it correctly). Doing this
could result in data loss if you do it all at once so do the change
progressively. This also means that you may have to keep a seperate
HDFS configuration for those particular nodes. Please ask more
questions if it's not very clear.
WRT to dedicating nodes to Zookeeeper, it makes sense only if you have
a lot of nodes but in your case, unless there's also CPU contention,
you're good with only a dedicated disk.
Now for what you are trying to do, is this a massive import that goes
over many hours? How big is each row? I'd love to get a lot of details
to help you optimize your cluster for your particular problem.
Thx,
J-D
On Wed, Nov 11, 2009 at 11:51 AM, Zhenyu Zhong <zh...@gmail.com> wrote:
> J-D,
>
> Thanks a lot for the insightful comments.
> We are running 5 zookeeper quorums now. On these 5 quorum servers, we also
> run datanode, tasktracker and regionserver.
> Now we have 6 disks per machine.
> If it is the disk IO saturation, is there anyway to configure zookeeper to
> write to a seperate disk which is not the same one used by HDFS?
> Or shall we run zookeeper quorums on different nodes other than HDFS?
>
> Are HLogs generated by Regionservers? Would it alleviate the issue if we use
> the first 14 for regionserver, and the rest of 5 for zookeeper?(In that
> sense, the node running zookeeper quorum won't run regionserver)
>
> I am very appreciated for your comments.
>
> Best,
> zhenyu
>
>
>
> On Wed, Nov 11, 2009 at 2:30 PM, Jean-Daniel Cryans <jd...@apache.org>wrote:
>
>> The problem seems to be that the IO is saturated. When you see this:
>>
>> #
>> 2009-11-11 04:28:10,308 WARN
>> org.apache.hadoop.hbase.regionserver.HLog: IPC Server handler 9 on
>> 60021 took 6819ms appending an edit to hlog; editcount=370
>> #
>> 2009-11-11 04:28:10,308 WARN
>> org.apache.hadoop.hbase.regionserver.HLog: IPC Server handler 2 on
>> 60021 took 4412ms appending an edit to hlog; editcount=371
>>
>> It's a sign that HDFS is slow. I guess that the zookeeper quorum
>> members are on those same nodes using the same disks? If so, there's
>> the problem because Zookeeper needs to be able to commit transactions
>> very fast and if your disks are already hogged then they have to wait
>> and you can get disconnections. Do you still have 19 quorum members?
>> How many disks do you have per machine?
>>
>> Thx
>>
>> J-D
>>
>> On Wed, Nov 11, 2009 at 11:22 AM, Zhenyu Zhong <zh...@gmail.com>
>> wrote:
>> > Please take a look at the following regionserver log.
>> > The error started to come out from 4:24am UTC to 4:29am UTC.
>> >
>> > http://pastebin.com/m3717d064
>> >
>> > Thanks in advance.
>> >
>> > Best,
>> > zhenyu
>> >
>> >
>> > On Wed, Nov 11, 2009 at 2:01 PM, Jean-Daniel Cryans <jdcryans@apache.org
>> >wrote:
>> >
>> >> Could you paste a bigger log trace (interested to see at least all 300
>> >> lines before that) in a pastebin?
>> >>
>> >> Thx,
>> >>
>> >> J-D
>> >>
>> >> On Wed, Nov 11, 2009 at 10:56 AM, Zhenyu Zhong <zhongresearch@gmail.com
>> >
>> >> wrote:
>> >> > J-D,
>> >> >
>> >> > Thanks for the reply.
>> >> >
>> >> > The problem I have is one or more of the regionservers disconnect from
>> >> the
>> >> > cluster.
>> >> >
>> >> > I checked the resource usage including memory Heap usage for
>> >> regionserver,
>> >> > and found nothing suspicious. The regionserver heap usage is way below
>> >> the
>> >> > maxHeap defined in jvm.
>> >> >
>> >> > So I checked zookeeper log, then I found the following
>> >> > BufferUnderflowException, eventually it leads to SocketException:
>> >> Transport
>> >> > endpoint is not connected.
>> >> >
>> >> >
>> >> > I also checked the regionserver log. It looks like the regionserver
>> tried
>> >> to
>> >> > connect to other zookeeper quorum server. It connects successfully,
>> but
>> >> then
>> >> > it complained about SocketException: transport endpoint is not
>> connected.
>> >> >
>> >> > So both zookeeper log and regionserver log told me that the transport
>> >> > endpoint is not connected. I don't know why.
>> >> >
>> >> > I would be very appreciated if you could point me to the right
>> direction.
>> >> >
>> >> > Best,
>> >> >
>> >> >
>> >> > Regionserver log:
>> >> > 2009-11-11 04:24:53,718 INFO org.apache.zookeeper.ClientCnxn:
>> Attempting
>> >> > connection to server superpyxis0018/192.168.100.132:2181
>> >> > 2009-11-11 04:24:53,718 INFO org.apache.zookeeper.ClientCnxn: Priming
>> >> > connection to java.nio.channels.SocketChannel[connected local=/
>> >> > 192.168.100.131:43065 remote=superpyxis0018/192.168.100.132:2181]
>> >> > 2009-11-11 04:24:53,718 INFO org.apache.zookeeper.ClientCnxn: Server
>> >> > connection successful
>> >> > 2009-11-11 04:24:53,768 INFO
>> >> > org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper
>> event,
>> >> > state: Disconnected, type: None, path: null
>> >> > 2009-11-11 04:24:54,118 INFO org.apache.zookeeper.ClientCnxn:
>> Attempting
>> >> > connection to server superpyxis0017/192.168.100.131:2181
>> >> > 2009-11-11 04:24:54,118 INFO org.apache.zookeeper.ClientCnxn: Priming
>> >> > connection to java.nio.channels.SocketChannel[connected local=/
>> >> > 192.168.100.131:46903 remote=superpyxis0017/192.168.100.131:2181]
>> >> > 2009-11-11 04:24:54,118 INFO org.apache.zookeeper.ClientCnxn: Server
>> >> > connection successful
>> >> > 2009-11-11 04:24:54,118 WARN org.apache.zookeeper.ClientCnxn:
>> Exception
>> >> > closing session 0x24dfd90c810002 to
>> sun.nio.ch.SelectionKeyImpl@2dd78ccf
>> >> > java.io.IOException: Read error rc = -1
>> java.nio.DirectByteBuffer[pos=0
>> >> > lim=4 cap=4]
>> >> > at
>> >> > org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
>> >> > at
>> >> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
>> >> > 2009-11-11 04:24:54,118 WARN org.apache.zookeeper.ClientCnxn: Ignoring
>> >> > exception during shutdown input
>> >> > java.net.SocketException: Transport endpoint is not connected
>> >> > at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
>> >> > at
>> >> > sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
>> >> > at
>> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>> >> > at
>> >> >
>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
>> >> > at
>> >> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > On Wed, Nov 11, 2009 at 1:40 PM, Jean-Daniel Cryans <
>> jdcryans@apache.org
>> >> >wrote:
>> >> >
>> >> >> I'm really not sure exactly what the situation is with the
>> >> >> java.nio.BufferUnderflowException, but what do you mean by
>> >> >> "regionserver disconnection"? Could you give more details?
>> >> >>
>> >> >> Thx,
>> >> >>
>> >> >> J-D
>> >> >>
>> >> >> On Wed, Nov 11, 2009 at 10:34 AM, Zhenyu Zhong <
>> zhongresearch@gmail.com
>> >> >
>> >> >> wrote:
>> >> >> > I searched around and found HBASE-1645 "HBase could be a little
>> more
>> >> >> dainty
>> >> >> > when zk is pulled out from under it"
>> >> >> >
>> >> >> > I am wondering whether my problem is related to this HBase-1645. Or
>> >> >> probably
>> >> >> > has nothing to do with it?
>> >> >> >
>> >> >> > thanks
>> >> >> >
>> >> >> >
>> >> >> > On Wed, Nov 11, 2009 at 12:02 PM, Zhenyu Zhong <
>> >> zhongresearch@gmail.com
>> >> >> >wrote:
>> >> >> >
>> >> >> >> Hi,
>> >> >> >>
>> >> >> >> I have been monitoring the system resources such as memory Heap
>> usage
>> >> >> for
>> >> >> >> Regionserver, load average, network bandwidth, etc.
>> >> >> >>
>> >> >> >> I experienced a disconnection last night. When I checked the
>> memory
>> >> Heap
>> >> >> >> for Regionserver, it was 2336 which is way below the max Heap
>> 4096. I
>> >> >> don't
>> >> >> >> think it is Regionserver problem. The requests that the
>> regionserver
>> >> got
>> >> >> was
>> >> >> >> about 126 which doesn't seem to be high. The load average is kind
>> of
>> >> >> high
>> >> >> >> >100. (but I saw disconnections while the load average is around
>> 30).
>> >> >> The
>> >> >> >> bandwidth is not a problem.
>> >> >> >>
>> >> >> >> However, I found error messages from the zookeeper. Please see the
>> >> >> >> following error messages. There are some
>> >> >> >> BufferUnderflowException error, IOException error,etc.
>> >> >> >> May I ask what might be the causes that generate these error
>> >> messages?
>> >> >> >>
>> >> >> >> Many thanks
>> >> >> >> zhenyu
>> >> >> >>
>> >> >> >>
>> >> >> >> 2009-11-11 04:29:00,810 WARN org.apache.zookeeper.server.Request:
>> >> >> Ignoring
>> >> >> >> exception during toString
>> >> >> >> java.nio.BufferUnderflowException
>> >> >> >> at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
>> >> >> >> at java.nio.ByteBuffer.get(ByteBuffer.java:675)
>> >> >> >> at
>> >> >> org.apache.zookeeper.server.Request.toString(Request.java:199)
>> >> >> >> at java.lang.String.valueOf(String.java:2827)
>> >> >> >> at java.lang.StringBuilder.append(StringBuilder.java:115)
>> >> >> >> at
>> >> >> >>
>> >> >>
>> >>
>> org.apache.zookeeper.server.quorum.CommitProcessor.processRequest(CommitProcessor.java:167)
>> >> >> >> at
>> >> >> >>
>> >> >>
>> >>
>> org.apache.zookeeper.server.quorum.FollowerRequestProcessor.run(FollowerRequestProcessor.java:68)
>> >> >> >> 2009-11-11 04:29:00,810 WARN
>> >> org.apache.zookeeper.server.NIOServerCnxn:
>> >> >> >> Exception causing close of session 0x424dfd908c50009
>> >> >> >> due to java.io.IOException: Read error
>> >> >> >> 2009-11-11 04:29:00,810 DEBUG
>> >> org.apache.zookeeper.server.NIOServerCnxn:
>> >> >> >> IOException stack trace
>> >> >> >> java.io.IOException: Read error
>> >> >> >> at
>> >> >> >>
>> >> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:372)
>> >> >> >> at
>> >> >> >>
>> >> >>
>> >>
>> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:221)
>> >> >> >> 2009-11-11 04:29:00,810 DEBUG
>> >> >> >> org.apache.zookeeper.server.quorum.CommitProcessor: Processing
>> >> request::
>> >> >> >> sessionid:0x424dfd908c50009 type:setWatches
>> cxid:0xfffffffffffffff8
>> >> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
>> >> >> >> 2009-11-11 04:29:00,810 WARN org.apache.zookeeper.server.Request:
>> >> >> Ignoring
>> >> >> >> exception during toString
>> >> >> >> java.nio.BufferUnderflowException
>> >> >> >> at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
>> >> >> >> at java.nio.ByteBuffer.get(ByteBuffer.java:675)
>> >> >> >> at
>> >> >> org.apache.zookeeper.server.Request.toString(Request.java:199)
>> >> >> >> at java.lang.String.valueOf(String.java:2827)
>> >> >> >> at java.lang.StringBuilder.append(StringBuilder.java:115)
>> >> >> >> at
>> >> >> >>
>> >> >>
>> >>
>> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:74)
>> >> >> >> at
>> >> >> >>
>> >> >>
>> >>
>> org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)
>> >> >> >> 2009-11-11 04:29:00,810 DEBUG
>> >> >> >> org.apache.zookeeper.server.FinalRequestProcessor: Processing
>> >> request::
>> >> >> >> sessionid:0x424dfd908c50009 type:setWatches
>> cxid:0xfffffffffffffff8
>> >> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
>> >> >> >> 2009-11-11 04:29:00,810 WARN org.apache.zookeeper.server.Request:
>> >> >> Ignoring
>> >> >> >> exception during toString
>> >> >> >> java.nio.BufferUnderflowException
>> >> >> >> at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
>> >> >> >> at java.nio.ByteBuffer.get(ByteBuffer.java:675)
>> >> >> >> at
>> >> >> org.apache.zookeeper.server.Request.toString(Request.java:199)
>> >> >> >> at
>> >> >> >>
>> org.apache.log4j.or.DefaultRenderer.doRender(DefaultRenderer.java:36)
>> >> >> >> at
>> >> >> >> org.apache.log4j.or.RendererMap.findAndRender(RendererMap.java:80)
>> >> >> >> at
>> >> >> >>
>> >> >>
>> >>
>> org.apache.log4j.spi.LoggingEvent.getRenderedMessage(LoggingEvent.java:362)
>> >> >> >> at
>> >> >> >>
>> >> >>
>> >>
>> org.apache.log4j.helpers.PatternParser$BasicPatternConverter.convert(PatternParser.java:403)
>> >> >> >> at
>> >> >> >>
>> >> >>
>> >>
>> org.apache.log4j.helpers.PatternConverter.format(PatternConverter.java:65)
>> >> >> >> at
>> >> org.apache.log4j.PatternLayout.format(PatternLayout.java:502)
>> >> >> >> at
>> >> >> >> org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:302)
>> >> >> >> at
>> >> >> >>
>> >> >>
>> >>
>> org.apache.log4j.DailyRollingFileAppender.subAppend(DailyRollingFileAppender.java:359)
>> >> >> >> at
>> >> >> org.apache.log4j.WriterAppender.append(WriterAppender.java:160)
>> >> >> >> at
>> >> >> >>
>> org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
>> >> >> >> at
>> >> >> >>
>> >> >>
>> >>
>> org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
>> >> >> >> at
>> org.apache.log4j.Category.callAppenders(Category.java:206)
>> >> >> >> at org.apache.log4j.Category.forcedLog(Category.java:391)
>> >> >> >> at org.apache.log4j.Category.debug(Category.java:260)
>> >> >> >> at
>> >> >> >>
>> >> >>
>> >>
>> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:147)
>> >> >> >> at
>> >> >> >>
>> >> >>
>> >>
>> org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)
>> >> >> >> 2009-11-11 04:29:00,810 WARN org.apache.zookeeper.server.Request:
>> >> >> Ignoring
>> >> >> >> exception during toString
>> >> >> >> sessionid:0x424dfd908c50009 type:setWatches
>> cxid:0xfffffffffffffff8
>> >> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
>> >> >> >> 2009-11-11 04:29:00,810 INFO
>> >> org.apache.zookeeper.server.NIOServerCnxn:
>> >> >> >> closing session:0x424dfd908c50009 NIOServerCnxn:
>> >> >> >> java.nio.channels.SocketChannel[connected local=/
>> >> 192.168.100.131:2181
>> >> >> remote=/
>> >> >> >> 192.168.100.131:35500]
>> >> >> >> 2009-11-11 04:29:00,810 DEBUG
>> >> org.apache.zookeeper.server.NIOServerCnxn:
>> >> >> >> ignoring exception during input shutdown
>> >> >> >> java.net.SocketException: Transport endpoint is not connected
>> >> >> >> at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
>> >> >> >> at
>> >> >> >>
>> >> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
>> >> >> >> at
>> >> >> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>> >> >> >> at
>> >> >> >>
>> >> org.apache.zookeeper.server.NIOServerCnxn.close(NIOServerCnxn.java:848)
>> >> >> >> at
>> >> >> >>
>> >> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:498)
>> >> >> >> at
>> >> >> >>
>> >> >>
>> >>
>> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:221)
>> >> >> >>
>> >> >> >
>> >> >>
>> >> >
>> >>
>> >
>>
>
Re: regionserver disconnection
Posted by Zhenyu Zhong <zh...@gmail.com>.
J-D,
Thanks a lot for the insightful comments.
We are running 5 zookeeper quorums now. On these 5 quorum servers, we also
run datanode, tasktracker and regionserver.
Now we have 6 disks per machine.
If it is the disk IO saturation, is there anyway to configure zookeeper to
write to a seperate disk which is not the same one used by HDFS?
Or shall we run zookeeper quorums on different nodes other than HDFS?
Are HLogs generated by Regionservers? Would it alleviate the issue if we use
the first 14 for regionserver, and the rest of 5 for zookeeper?(In that
sense, the node running zookeeper quorum won't run regionserver)
I am very appreciated for your comments.
Best,
zhenyu
On Wed, Nov 11, 2009 at 2:30 PM, Jean-Daniel Cryans <jd...@apache.org>wrote:
> The problem seems to be that the IO is saturated. When you see this:
>
> #
> 2009-11-11 04:28:10,308 WARN
> org.apache.hadoop.hbase.regionserver.HLog: IPC Server handler 9 on
> 60021 took 6819ms appending an edit to hlog; editcount=370
> #
> 2009-11-11 04:28:10,308 WARN
> org.apache.hadoop.hbase.regionserver.HLog: IPC Server handler 2 on
> 60021 took 4412ms appending an edit to hlog; editcount=371
>
> It's a sign that HDFS is slow. I guess that the zookeeper quorum
> members are on those same nodes using the same disks? If so, there's
> the problem because Zookeeper needs to be able to commit transactions
> very fast and if your disks are already hogged then they have to wait
> and you can get disconnections. Do you still have 19 quorum members?
> How many disks do you have per machine?
>
> Thx
>
> J-D
>
> On Wed, Nov 11, 2009 at 11:22 AM, Zhenyu Zhong <zh...@gmail.com>
> wrote:
> > Please take a look at the following regionserver log.
> > The error started to come out from 4:24am UTC to 4:29am UTC.
> >
> > http://pastebin.com/m3717d064
> >
> > Thanks in advance.
> >
> > Best,
> > zhenyu
> >
> >
> > On Wed, Nov 11, 2009 at 2:01 PM, Jean-Daniel Cryans <jdcryans@apache.org
> >wrote:
> >
> >> Could you paste a bigger log trace (interested to see at least all 300
> >> lines before that) in a pastebin?
> >>
> >> Thx,
> >>
> >> J-D
> >>
> >> On Wed, Nov 11, 2009 at 10:56 AM, Zhenyu Zhong <zhongresearch@gmail.com
> >
> >> wrote:
> >> > J-D,
> >> >
> >> > Thanks for the reply.
> >> >
> >> > The problem I have is one or more of the regionservers disconnect from
> >> the
> >> > cluster.
> >> >
> >> > I checked the resource usage including memory Heap usage for
> >> regionserver,
> >> > and found nothing suspicious. The regionserver heap usage is way below
> >> the
> >> > maxHeap defined in jvm.
> >> >
> >> > So I checked zookeeper log, then I found the following
> >> > BufferUnderflowException, eventually it leads to SocketException:
> >> Transport
> >> > endpoint is not connected.
> >> >
> >> >
> >> > I also checked the regionserver log. It looks like the regionserver
> tried
> >> to
> >> > connect to other zookeeper quorum server. It connects successfully,
> but
> >> then
> >> > it complained about SocketException: transport endpoint is not
> connected.
> >> >
> >> > So both zookeeper log and regionserver log told me that the transport
> >> > endpoint is not connected. I don't know why.
> >> >
> >> > I would be very appreciated if you could point me to the right
> direction.
> >> >
> >> > Best,
> >> >
> >> >
> >> > Regionserver log:
> >> > 2009-11-11 04:24:53,718 INFO org.apache.zookeeper.ClientCnxn:
> Attempting
> >> > connection to server superpyxis0018/192.168.100.132:2181
> >> > 2009-11-11 04:24:53,718 INFO org.apache.zookeeper.ClientCnxn: Priming
> >> > connection to java.nio.channels.SocketChannel[connected local=/
> >> > 192.168.100.131:43065 remote=superpyxis0018/192.168.100.132:2181]
> >> > 2009-11-11 04:24:53,718 INFO org.apache.zookeeper.ClientCnxn: Server
> >> > connection successful
> >> > 2009-11-11 04:24:53,768 INFO
> >> > org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper
> event,
> >> > state: Disconnected, type: None, path: null
> >> > 2009-11-11 04:24:54,118 INFO org.apache.zookeeper.ClientCnxn:
> Attempting
> >> > connection to server superpyxis0017/192.168.100.131:2181
> >> > 2009-11-11 04:24:54,118 INFO org.apache.zookeeper.ClientCnxn: Priming
> >> > connection to java.nio.channels.SocketChannel[connected local=/
> >> > 192.168.100.131:46903 remote=superpyxis0017/192.168.100.131:2181]
> >> > 2009-11-11 04:24:54,118 INFO org.apache.zookeeper.ClientCnxn: Server
> >> > connection successful
> >> > 2009-11-11 04:24:54,118 WARN org.apache.zookeeper.ClientCnxn:
> Exception
> >> > closing session 0x24dfd90c810002 to
> sun.nio.ch.SelectionKeyImpl@2dd78ccf
> >> > java.io.IOException: Read error rc = -1
> java.nio.DirectByteBuffer[pos=0
> >> > lim=4 cap=4]
> >> > at
> >> > org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
> >> > at
> >> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
> >> > 2009-11-11 04:24:54,118 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> >> > exception during shutdown input
> >> > java.net.SocketException: Transport endpoint is not connected
> >> > at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
> >> > at
> >> > sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
> >> > at
> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> >> > at
> >> >
> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> >> > at
> >> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > On Wed, Nov 11, 2009 at 1:40 PM, Jean-Daniel Cryans <
> jdcryans@apache.org
> >> >wrote:
> >> >
> >> >> I'm really not sure exactly what the situation is with the
> >> >> java.nio.BufferUnderflowException, but what do you mean by
> >> >> "regionserver disconnection"? Could you give more details?
> >> >>
> >> >> Thx,
> >> >>
> >> >> J-D
> >> >>
> >> >> On Wed, Nov 11, 2009 at 10:34 AM, Zhenyu Zhong <
> zhongresearch@gmail.com
> >> >
> >> >> wrote:
> >> >> > I searched around and found HBASE-1645 "HBase could be a little
> more
> >> >> dainty
> >> >> > when zk is pulled out from under it"
> >> >> >
> >> >> > I am wondering whether my problem is related to this HBase-1645. Or
> >> >> probably
> >> >> > has nothing to do with it?
> >> >> >
> >> >> > thanks
> >> >> >
> >> >> >
> >> >> > On Wed, Nov 11, 2009 at 12:02 PM, Zhenyu Zhong <
> >> zhongresearch@gmail.com
> >> >> >wrote:
> >> >> >
> >> >> >> Hi,
> >> >> >>
> >> >> >> I have been monitoring the system resources such as memory Heap
> usage
> >> >> for
> >> >> >> Regionserver, load average, network bandwidth, etc.
> >> >> >>
> >> >> >> I experienced a disconnection last night. When I checked the
> memory
> >> Heap
> >> >> >> for Regionserver, it was 2336 which is way below the max Heap
> 4096. I
> >> >> don't
> >> >> >> think it is Regionserver problem. The requests that the
> regionserver
> >> got
> >> >> was
> >> >> >> about 126 which doesn't seem to be high. The load average is kind
> of
> >> >> high
> >> >> >> >100. (but I saw disconnections while the load average is around
> 30).
> >> >> The
> >> >> >> bandwidth is not a problem.
> >> >> >>
> >> >> >> However, I found error messages from the zookeeper. Please see the
> >> >> >> following error messages. There are some
> >> >> >> BufferUnderflowException error, IOException error,etc.
> >> >> >> May I ask what might be the causes that generate these error
> >> messages?
> >> >> >>
> >> >> >> Many thanks
> >> >> >> zhenyu
> >> >> >>
> >> >> >>
> >> >> >> 2009-11-11 04:29:00,810 WARN org.apache.zookeeper.server.Request:
> >> >> Ignoring
> >> >> >> exception during toString
> >> >> >> java.nio.BufferUnderflowException
> >> >> >> at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
> >> >> >> at java.nio.ByteBuffer.get(ByteBuffer.java:675)
> >> >> >> at
> >> >> org.apache.zookeeper.server.Request.toString(Request.java:199)
> >> >> >> at java.lang.String.valueOf(String.java:2827)
> >> >> >> at java.lang.StringBuilder.append(StringBuilder.java:115)
> >> >> >> at
> >> >> >>
> >> >>
> >>
> org.apache.zookeeper.server.quorum.CommitProcessor.processRequest(CommitProcessor.java:167)
> >> >> >> at
> >> >> >>
> >> >>
> >>
> org.apache.zookeeper.server.quorum.FollowerRequestProcessor.run(FollowerRequestProcessor.java:68)
> >> >> >> 2009-11-11 04:29:00,810 WARN
> >> org.apache.zookeeper.server.NIOServerCnxn:
> >> >> >> Exception causing close of session 0x424dfd908c50009
> >> >> >> due to java.io.IOException: Read error
> >> >> >> 2009-11-11 04:29:00,810 DEBUG
> >> org.apache.zookeeper.server.NIOServerCnxn:
> >> >> >> IOException stack trace
> >> >> >> java.io.IOException: Read error
> >> >> >> at
> >> >> >>
> >> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:372)
> >> >> >> at
> >> >> >>
> >> >>
> >>
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:221)
> >> >> >> 2009-11-11 04:29:00,810 DEBUG
> >> >> >> org.apache.zookeeper.server.quorum.CommitProcessor: Processing
> >> request::
> >> >> >> sessionid:0x424dfd908c50009 type:setWatches
> cxid:0xfffffffffffffff8
> >> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
> >> >> >> 2009-11-11 04:29:00,810 WARN org.apache.zookeeper.server.Request:
> >> >> Ignoring
> >> >> >> exception during toString
> >> >> >> java.nio.BufferUnderflowException
> >> >> >> at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
> >> >> >> at java.nio.ByteBuffer.get(ByteBuffer.java:675)
> >> >> >> at
> >> >> org.apache.zookeeper.server.Request.toString(Request.java:199)
> >> >> >> at java.lang.String.valueOf(String.java:2827)
> >> >> >> at java.lang.StringBuilder.append(StringBuilder.java:115)
> >> >> >> at
> >> >> >>
> >> >>
> >>
> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:74)
> >> >> >> at
> >> >> >>
> >> >>
> >>
> org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)
> >> >> >> 2009-11-11 04:29:00,810 DEBUG
> >> >> >> org.apache.zookeeper.server.FinalRequestProcessor: Processing
> >> request::
> >> >> >> sessionid:0x424dfd908c50009 type:setWatches
> cxid:0xfffffffffffffff8
> >> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
> >> >> >> 2009-11-11 04:29:00,810 WARN org.apache.zookeeper.server.Request:
> >> >> Ignoring
> >> >> >> exception during toString
> >> >> >> java.nio.BufferUnderflowException
> >> >> >> at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
> >> >> >> at java.nio.ByteBuffer.get(ByteBuffer.java:675)
> >> >> >> at
> >> >> org.apache.zookeeper.server.Request.toString(Request.java:199)
> >> >> >> at
> >> >> >>
> org.apache.log4j.or.DefaultRenderer.doRender(DefaultRenderer.java:36)
> >> >> >> at
> >> >> >> org.apache.log4j.or.RendererMap.findAndRender(RendererMap.java:80)
> >> >> >> at
> >> >> >>
> >> >>
> >>
> org.apache.log4j.spi.LoggingEvent.getRenderedMessage(LoggingEvent.java:362)
> >> >> >> at
> >> >> >>
> >> >>
> >>
> org.apache.log4j.helpers.PatternParser$BasicPatternConverter.convert(PatternParser.java:403)
> >> >> >> at
> >> >> >>
> >> >>
> >>
> org.apache.log4j.helpers.PatternConverter.format(PatternConverter.java:65)
> >> >> >> at
> >> org.apache.log4j.PatternLayout.format(PatternLayout.java:502)
> >> >> >> at
> >> >> >> org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:302)
> >> >> >> at
> >> >> >>
> >> >>
> >>
> org.apache.log4j.DailyRollingFileAppender.subAppend(DailyRollingFileAppender.java:359)
> >> >> >> at
> >> >> org.apache.log4j.WriterAppender.append(WriterAppender.java:160)
> >> >> >> at
> >> >> >>
> org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
> >> >> >> at
> >> >> >>
> >> >>
> >>
> org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
> >> >> >> at
> org.apache.log4j.Category.callAppenders(Category.java:206)
> >> >> >> at org.apache.log4j.Category.forcedLog(Category.java:391)
> >> >> >> at org.apache.log4j.Category.debug(Category.java:260)
> >> >> >> at
> >> >> >>
> >> >>
> >>
> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:147)
> >> >> >> at
> >> >> >>
> >> >>
> >>
> org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)
> >> >> >> 2009-11-11 04:29:00,810 WARN org.apache.zookeeper.server.Request:
> >> >> Ignoring
> >> >> >> exception during toString
> >> >> >> sessionid:0x424dfd908c50009 type:setWatches
> cxid:0xfffffffffffffff8
> >> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
> >> >> >> 2009-11-11 04:29:00,810 INFO
> >> org.apache.zookeeper.server.NIOServerCnxn:
> >> >> >> closing session:0x424dfd908c50009 NIOServerCnxn:
> >> >> >> java.nio.channels.SocketChannel[connected local=/
> >> 192.168.100.131:2181
> >> >> remote=/
> >> >> >> 192.168.100.131:35500]
> >> >> >> 2009-11-11 04:29:00,810 DEBUG
> >> org.apache.zookeeper.server.NIOServerCnxn:
> >> >> >> ignoring exception during input shutdown
> >> >> >> java.net.SocketException: Transport endpoint is not connected
> >> >> >> at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
> >> >> >> at
> >> >> >>
> >> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
> >> >> >> at
> >> >> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> >> >> >> at
> >> >> >>
> >> org.apache.zookeeper.server.NIOServerCnxn.close(NIOServerCnxn.java:848)
> >> >> >> at
> >> >> >>
> >> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:498)
> >> >> >> at
> >> >> >>
> >> >>
> >>
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:221)
> >> >> >>
> >> >> >
> >> >>
> >> >
> >>
> >
>
Re: regionserver disconnection
Posted by Jean-Daniel Cryans <jd...@apache.org>.
The problem seems to be that the IO is saturated. When you see this:
#
2009-11-11 04:28:10,308 WARN
org.apache.hadoop.hbase.regionserver.HLog: IPC Server handler 9 on
60021 took 6819ms appending an edit to hlog; editcount=370
#
2009-11-11 04:28:10,308 WARN
org.apache.hadoop.hbase.regionserver.HLog: IPC Server handler 2 on
60021 took 4412ms appending an edit to hlog; editcount=371
It's a sign that HDFS is slow. I guess that the zookeeper quorum
members are on those same nodes using the same disks? If so, there's
the problem because Zookeeper needs to be able to commit transactions
very fast and if your disks are already hogged then they have to wait
and you can get disconnections. Do you still have 19 quorum members?
How many disks do you have per machine?
Thx
J-D
On Wed, Nov 11, 2009 at 11:22 AM, Zhenyu Zhong <zh...@gmail.com> wrote:
> Please take a look at the following regionserver log.
> The error started to come out from 4:24am UTC to 4:29am UTC.
>
> http://pastebin.com/m3717d064
>
> Thanks in advance.
>
> Best,
> zhenyu
>
>
> On Wed, Nov 11, 2009 at 2:01 PM, Jean-Daniel Cryans <jd...@apache.org>wrote:
>
>> Could you paste a bigger log trace (interested to see at least all 300
>> lines before that) in a pastebin?
>>
>> Thx,
>>
>> J-D
>>
>> On Wed, Nov 11, 2009 at 10:56 AM, Zhenyu Zhong <zh...@gmail.com>
>> wrote:
>> > J-D,
>> >
>> > Thanks for the reply.
>> >
>> > The problem I have is one or more of the regionservers disconnect from
>> the
>> > cluster.
>> >
>> > I checked the resource usage including memory Heap usage for
>> regionserver,
>> > and found nothing suspicious. The regionserver heap usage is way below
>> the
>> > maxHeap defined in jvm.
>> >
>> > So I checked zookeeper log, then I found the following
>> > BufferUnderflowException, eventually it leads to SocketException:
>> Transport
>> > endpoint is not connected.
>> >
>> >
>> > I also checked the regionserver log. It looks like the regionserver tried
>> to
>> > connect to other zookeeper quorum server. It connects successfully, but
>> then
>> > it complained about SocketException: transport endpoint is not connected.
>> >
>> > So both zookeeper log and regionserver log told me that the transport
>> > endpoint is not connected. I don't know why.
>> >
>> > I would be very appreciated if you could point me to the right direction.
>> >
>> > Best,
>> >
>> >
>> > Regionserver log:
>> > 2009-11-11 04:24:53,718 INFO org.apache.zookeeper.ClientCnxn: Attempting
>> > connection to server superpyxis0018/192.168.100.132:2181
>> > 2009-11-11 04:24:53,718 INFO org.apache.zookeeper.ClientCnxn: Priming
>> > connection to java.nio.channels.SocketChannel[connected local=/
>> > 192.168.100.131:43065 remote=superpyxis0018/192.168.100.132:2181]
>> > 2009-11-11 04:24:53,718 INFO org.apache.zookeeper.ClientCnxn: Server
>> > connection successful
>> > 2009-11-11 04:24:53,768 INFO
>> > org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
>> > state: Disconnected, type: None, path: null
>> > 2009-11-11 04:24:54,118 INFO org.apache.zookeeper.ClientCnxn: Attempting
>> > connection to server superpyxis0017/192.168.100.131:2181
>> > 2009-11-11 04:24:54,118 INFO org.apache.zookeeper.ClientCnxn: Priming
>> > connection to java.nio.channels.SocketChannel[connected local=/
>> > 192.168.100.131:46903 remote=superpyxis0017/192.168.100.131:2181]
>> > 2009-11-11 04:24:54,118 INFO org.apache.zookeeper.ClientCnxn: Server
>> > connection successful
>> > 2009-11-11 04:24:54,118 WARN org.apache.zookeeper.ClientCnxn: Exception
>> > closing session 0x24dfd90c810002 to sun.nio.ch.SelectionKeyImpl@2dd78ccf
>> > java.io.IOException: Read error rc = -1 java.nio.DirectByteBuffer[pos=0
>> > lim=4 cap=4]
>> > at
>> > org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
>> > at
>> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
>> > 2009-11-11 04:24:54,118 WARN org.apache.zookeeper.ClientCnxn: Ignoring
>> > exception during shutdown input
>> > java.net.SocketException: Transport endpoint is not connected
>> > at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
>> > at
>> > sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
>> > at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>> > at
>> > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
>> > at
>> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
>> >
>> >
>> >
>> >
>> >
>> > On Wed, Nov 11, 2009 at 1:40 PM, Jean-Daniel Cryans <jdcryans@apache.org
>> >wrote:
>> >
>> >> I'm really not sure exactly what the situation is with the
>> >> java.nio.BufferUnderflowException, but what do you mean by
>> >> "regionserver disconnection"? Could you give more details?
>> >>
>> >> Thx,
>> >>
>> >> J-D
>> >>
>> >> On Wed, Nov 11, 2009 at 10:34 AM, Zhenyu Zhong <zhongresearch@gmail.com
>> >
>> >> wrote:
>> >> > I searched around and found HBASE-1645 "HBase could be a little more
>> >> dainty
>> >> > when zk is pulled out from under it"
>> >> >
>> >> > I am wondering whether my problem is related to this HBase-1645. Or
>> >> probably
>> >> > has nothing to do with it?
>> >> >
>> >> > thanks
>> >> >
>> >> >
>> >> > On Wed, Nov 11, 2009 at 12:02 PM, Zhenyu Zhong <
>> zhongresearch@gmail.com
>> >> >wrote:
>> >> >
>> >> >> Hi,
>> >> >>
>> >> >> I have been monitoring the system resources such as memory Heap usage
>> >> for
>> >> >> Regionserver, load average, network bandwidth, etc.
>> >> >>
>> >> >> I experienced a disconnection last night. When I checked the memory
>> Heap
>> >> >> for Regionserver, it was 2336 which is way below the max Heap 4096. I
>> >> don't
>> >> >> think it is Regionserver problem. The requests that the regionserver
>> got
>> >> was
>> >> >> about 126 which doesn't seem to be high. The load average is kind of
>> >> high
>> >> >> >100. (but I saw disconnections while the load average is around 30).
>> >> The
>> >> >> bandwidth is not a problem.
>> >> >>
>> >> >> However, I found error messages from the zookeeper. Please see the
>> >> >> following error messages. There are some
>> >> >> BufferUnderflowException error, IOException error,etc.
>> >> >> May I ask what might be the causes that generate these error
>> messages?
>> >> >>
>> >> >> Many thanks
>> >> >> zhenyu
>> >> >>
>> >> >>
>> >> >> 2009-11-11 04:29:00,810 WARN org.apache.zookeeper.server.Request:
>> >> Ignoring
>> >> >> exception during toString
>> >> >> java.nio.BufferUnderflowException
>> >> >> at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
>> >> >> at java.nio.ByteBuffer.get(ByteBuffer.java:675)
>> >> >> at
>> >> org.apache.zookeeper.server.Request.toString(Request.java:199)
>> >> >> at java.lang.String.valueOf(String.java:2827)
>> >> >> at java.lang.StringBuilder.append(StringBuilder.java:115)
>> >> >> at
>> >> >>
>> >>
>> org.apache.zookeeper.server.quorum.CommitProcessor.processRequest(CommitProcessor.java:167)
>> >> >> at
>> >> >>
>> >>
>> org.apache.zookeeper.server.quorum.FollowerRequestProcessor.run(FollowerRequestProcessor.java:68)
>> >> >> 2009-11-11 04:29:00,810 WARN
>> org.apache.zookeeper.server.NIOServerCnxn:
>> >> >> Exception causing close of session 0x424dfd908c50009
>> >> >> due to java.io.IOException: Read error
>> >> >> 2009-11-11 04:29:00,810 DEBUG
>> org.apache.zookeeper.server.NIOServerCnxn:
>> >> >> IOException stack trace
>> >> >> java.io.IOException: Read error
>> >> >> at
>> >> >>
>> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:372)
>> >> >> at
>> >> >>
>> >>
>> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:221)
>> >> >> 2009-11-11 04:29:00,810 DEBUG
>> >> >> org.apache.zookeeper.server.quorum.CommitProcessor: Processing
>> request::
>> >> >> sessionid:0x424dfd908c50009 type:setWatches cxid:0xfffffffffffffff8
>> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
>> >> >> 2009-11-11 04:29:00,810 WARN org.apache.zookeeper.server.Request:
>> >> Ignoring
>> >> >> exception during toString
>> >> >> java.nio.BufferUnderflowException
>> >> >> at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
>> >> >> at java.nio.ByteBuffer.get(ByteBuffer.java:675)
>> >> >> at
>> >> org.apache.zookeeper.server.Request.toString(Request.java:199)
>> >> >> at java.lang.String.valueOf(String.java:2827)
>> >> >> at java.lang.StringBuilder.append(StringBuilder.java:115)
>> >> >> at
>> >> >>
>> >>
>> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:74)
>> >> >> at
>> >> >>
>> >>
>> org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)
>> >> >> 2009-11-11 04:29:00,810 DEBUG
>> >> >> org.apache.zookeeper.server.FinalRequestProcessor: Processing
>> request::
>> >> >> sessionid:0x424dfd908c50009 type:setWatches cxid:0xfffffffffffffff8
>> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
>> >> >> 2009-11-11 04:29:00,810 WARN org.apache.zookeeper.server.Request:
>> >> Ignoring
>> >> >> exception during toString
>> >> >> java.nio.BufferUnderflowException
>> >> >> at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
>> >> >> at java.nio.ByteBuffer.get(ByteBuffer.java:675)
>> >> >> at
>> >> org.apache.zookeeper.server.Request.toString(Request.java:199)
>> >> >> at
>> >> >> org.apache.log4j.or.DefaultRenderer.doRender(DefaultRenderer.java:36)
>> >> >> at
>> >> >> org.apache.log4j.or.RendererMap.findAndRender(RendererMap.java:80)
>> >> >> at
>> >> >>
>> >>
>> org.apache.log4j.spi.LoggingEvent.getRenderedMessage(LoggingEvent.java:362)
>> >> >> at
>> >> >>
>> >>
>> org.apache.log4j.helpers.PatternParser$BasicPatternConverter.convert(PatternParser.java:403)
>> >> >> at
>> >> >>
>> >>
>> org.apache.log4j.helpers.PatternConverter.format(PatternConverter.java:65)
>> >> >> at
>> org.apache.log4j.PatternLayout.format(PatternLayout.java:502)
>> >> >> at
>> >> >> org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:302)
>> >> >> at
>> >> >>
>> >>
>> org.apache.log4j.DailyRollingFileAppender.subAppend(DailyRollingFileAppender.java:359)
>> >> >> at
>> >> org.apache.log4j.WriterAppender.append(WriterAppender.java:160)
>> >> >> at
>> >> >> org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
>> >> >> at
>> >> >>
>> >>
>> org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
>> >> >> at org.apache.log4j.Category.callAppenders(Category.java:206)
>> >> >> at org.apache.log4j.Category.forcedLog(Category.java:391)
>> >> >> at org.apache.log4j.Category.debug(Category.java:260)
>> >> >> at
>> >> >>
>> >>
>> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:147)
>> >> >> at
>> >> >>
>> >>
>> org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)
>> >> >> 2009-11-11 04:29:00,810 WARN org.apache.zookeeper.server.Request:
>> >> Ignoring
>> >> >> exception during toString
>> >> >> sessionid:0x424dfd908c50009 type:setWatches cxid:0xfffffffffffffff8
>> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
>> >> >> 2009-11-11 04:29:00,810 INFO
>> org.apache.zookeeper.server.NIOServerCnxn:
>> >> >> closing session:0x424dfd908c50009 NIOServerCnxn:
>> >> >> java.nio.channels.SocketChannel[connected local=/
>> 192.168.100.131:2181
>> >> remote=/
>> >> >> 192.168.100.131:35500]
>> >> >> 2009-11-11 04:29:00,810 DEBUG
>> org.apache.zookeeper.server.NIOServerCnxn:
>> >> >> ignoring exception during input shutdown
>> >> >> java.net.SocketException: Transport endpoint is not connected
>> >> >> at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
>> >> >> at
>> >> >>
>> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
>> >> >> at
>> >> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>> >> >> at
>> >> >>
>> org.apache.zookeeper.server.NIOServerCnxn.close(NIOServerCnxn.java:848)
>> >> >> at
>> >> >>
>> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:498)
>> >> >> at
>> >> >>
>> >>
>> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:221)
>> >> >>
>> >> >
>> >>
>> >
>>
>
Re: regionserver disconnection
Posted by Zhenyu Zhong <zh...@gmail.com>.
Please take a look at the following regionserver log.
The error started to come out from 4:24am UTC to 4:29am UTC.
http://pastebin.com/m3717d064
Thanks in advance.
Best,
zhenyu
On Wed, Nov 11, 2009 at 2:01 PM, Jean-Daniel Cryans <jd...@apache.org>wrote:
> Could you paste a bigger log trace (interested to see at least all 300
> lines before that) in a pastebin?
>
> Thx,
>
> J-D
>
> On Wed, Nov 11, 2009 at 10:56 AM, Zhenyu Zhong <zh...@gmail.com>
> wrote:
> > J-D,
> >
> > Thanks for the reply.
> >
> > The problem I have is one or more of the regionservers disconnect from
> the
> > cluster.
> >
> > I checked the resource usage including memory Heap usage for
> regionserver,
> > and found nothing suspicious. The regionserver heap usage is way below
> the
> > maxHeap defined in jvm.
> >
> > So I checked zookeeper log, then I found the following
> > BufferUnderflowException, eventually it leads to SocketException:
> Transport
> > endpoint is not connected.
> >
> >
> > I also checked the regionserver log. It looks like the regionserver tried
> to
> > connect to other zookeeper quorum server. It connects successfully, but
> then
> > it complained about SocketException: transport endpoint is not connected.
> >
> > So both zookeeper log and regionserver log told me that the transport
> > endpoint is not connected. I don't know why.
> >
> > I would be very appreciated if you could point me to the right direction.
> >
> > Best,
> >
> >
> > Regionserver log:
> > 2009-11-11 04:24:53,718 INFO org.apache.zookeeper.ClientCnxn: Attempting
> > connection to server superpyxis0018/192.168.100.132:2181
> > 2009-11-11 04:24:53,718 INFO org.apache.zookeeper.ClientCnxn: Priming
> > connection to java.nio.channels.SocketChannel[connected local=/
> > 192.168.100.131:43065 remote=superpyxis0018/192.168.100.132:2181]
> > 2009-11-11 04:24:53,718 INFO org.apache.zookeeper.ClientCnxn: Server
> > connection successful
> > 2009-11-11 04:24:53,768 INFO
> > org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
> > state: Disconnected, type: None, path: null
> > 2009-11-11 04:24:54,118 INFO org.apache.zookeeper.ClientCnxn: Attempting
> > connection to server superpyxis0017/192.168.100.131:2181
> > 2009-11-11 04:24:54,118 INFO org.apache.zookeeper.ClientCnxn: Priming
> > connection to java.nio.channels.SocketChannel[connected local=/
> > 192.168.100.131:46903 remote=superpyxis0017/192.168.100.131:2181]
> > 2009-11-11 04:24:54,118 INFO org.apache.zookeeper.ClientCnxn: Server
> > connection successful
> > 2009-11-11 04:24:54,118 WARN org.apache.zookeeper.ClientCnxn: Exception
> > closing session 0x24dfd90c810002 to sun.nio.ch.SelectionKeyImpl@2dd78ccf
> > java.io.IOException: Read error rc = -1 java.nio.DirectByteBuffer[pos=0
> > lim=4 cap=4]
> > at
> > org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
> > at
> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
> > 2009-11-11 04:24:54,118 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> > exception during shutdown input
> > java.net.SocketException: Transport endpoint is not connected
> > at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
> > at
> > sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
> > at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> > at
> > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> > at
> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> >
> >
> >
> >
> >
> > On Wed, Nov 11, 2009 at 1:40 PM, Jean-Daniel Cryans <jdcryans@apache.org
> >wrote:
> >
> >> I'm really not sure exactly what the situation is with the
> >> java.nio.BufferUnderflowException, but what do you mean by
> >> "regionserver disconnection"? Could you give more details?
> >>
> >> Thx,
> >>
> >> J-D
> >>
> >> On Wed, Nov 11, 2009 at 10:34 AM, Zhenyu Zhong <zhongresearch@gmail.com
> >
> >> wrote:
> >> > I searched around and found HBASE-1645 "HBase could be a little more
> >> dainty
> >> > when zk is pulled out from under it"
> >> >
> >> > I am wondering whether my problem is related to this HBase-1645. Or
> >> probably
> >> > has nothing to do with it?
> >> >
> >> > thanks
> >> >
> >> >
> >> > On Wed, Nov 11, 2009 at 12:02 PM, Zhenyu Zhong <
> zhongresearch@gmail.com
> >> >wrote:
> >> >
> >> >> Hi,
> >> >>
> >> >> I have been monitoring the system resources such as memory Heap usage
> >> for
> >> >> Regionserver, load average, network bandwidth, etc.
> >> >>
> >> >> I experienced a disconnection last night. When I checked the memory
> Heap
> >> >> for Regionserver, it was 2336 which is way below the max Heap 4096. I
> >> don't
> >> >> think it is Regionserver problem. The requests that the regionserver
> got
> >> was
> >> >> about 126 which doesn't seem to be high. The load average is kind of
> >> high
> >> >> >100. (but I saw disconnections while the load average is around 30).
> >> The
> >> >> bandwidth is not a problem.
> >> >>
> >> >> However, I found error messages from the zookeeper. Please see the
> >> >> following error messages. There are some
> >> >> BufferUnderflowException error, IOException error,etc.
> >> >> May I ask what might be the causes that generate these error
> messages?
> >> >>
> >> >> Many thanks
> >> >> zhenyu
> >> >>
> >> >>
> >> >> 2009-11-11 04:29:00,810 WARN org.apache.zookeeper.server.Request:
> >> Ignoring
> >> >> exception during toString
> >> >> java.nio.BufferUnderflowException
> >> >> at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
> >> >> at java.nio.ByteBuffer.get(ByteBuffer.java:675)
> >> >> at
> >> org.apache.zookeeper.server.Request.toString(Request.java:199)
> >> >> at java.lang.String.valueOf(String.java:2827)
> >> >> at java.lang.StringBuilder.append(StringBuilder.java:115)
> >> >> at
> >> >>
> >>
> org.apache.zookeeper.server.quorum.CommitProcessor.processRequest(CommitProcessor.java:167)
> >> >> at
> >> >>
> >>
> org.apache.zookeeper.server.quorum.FollowerRequestProcessor.run(FollowerRequestProcessor.java:68)
> >> >> 2009-11-11 04:29:00,810 WARN
> org.apache.zookeeper.server.NIOServerCnxn:
> >> >> Exception causing close of session 0x424dfd908c50009
> >> >> due to java.io.IOException: Read error
> >> >> 2009-11-11 04:29:00,810 DEBUG
> org.apache.zookeeper.server.NIOServerCnxn:
> >> >> IOException stack trace
> >> >> java.io.IOException: Read error
> >> >> at
> >> >>
> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:372)
> >> >> at
> >> >>
> >>
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:221)
> >> >> 2009-11-11 04:29:00,810 DEBUG
> >> >> org.apache.zookeeper.server.quorum.CommitProcessor: Processing
> request::
> >> >> sessionid:0x424dfd908c50009 type:setWatches cxid:0xfffffffffffffff8
> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
> >> >> 2009-11-11 04:29:00,810 WARN org.apache.zookeeper.server.Request:
> >> Ignoring
> >> >> exception during toString
> >> >> java.nio.BufferUnderflowException
> >> >> at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
> >> >> at java.nio.ByteBuffer.get(ByteBuffer.java:675)
> >> >> at
> >> org.apache.zookeeper.server.Request.toString(Request.java:199)
> >> >> at java.lang.String.valueOf(String.java:2827)
> >> >> at java.lang.StringBuilder.append(StringBuilder.java:115)
> >> >> at
> >> >>
> >>
> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:74)
> >> >> at
> >> >>
> >>
> org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)
> >> >> 2009-11-11 04:29:00,810 DEBUG
> >> >> org.apache.zookeeper.server.FinalRequestProcessor: Processing
> request::
> >> >> sessionid:0x424dfd908c50009 type:setWatches cxid:0xfffffffffffffff8
> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
> >> >> 2009-11-11 04:29:00,810 WARN org.apache.zookeeper.server.Request:
> >> Ignoring
> >> >> exception during toString
> >> >> java.nio.BufferUnderflowException
> >> >> at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
> >> >> at java.nio.ByteBuffer.get(ByteBuffer.java:675)
> >> >> at
> >> org.apache.zookeeper.server.Request.toString(Request.java:199)
> >> >> at
> >> >> org.apache.log4j.or.DefaultRenderer.doRender(DefaultRenderer.java:36)
> >> >> at
> >> >> org.apache.log4j.or.RendererMap.findAndRender(RendererMap.java:80)
> >> >> at
> >> >>
> >>
> org.apache.log4j.spi.LoggingEvent.getRenderedMessage(LoggingEvent.java:362)
> >> >> at
> >> >>
> >>
> org.apache.log4j.helpers.PatternParser$BasicPatternConverter.convert(PatternParser.java:403)
> >> >> at
> >> >>
> >>
> org.apache.log4j.helpers.PatternConverter.format(PatternConverter.java:65)
> >> >> at
> org.apache.log4j.PatternLayout.format(PatternLayout.java:502)
> >> >> at
> >> >> org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:302)
> >> >> at
> >> >>
> >>
> org.apache.log4j.DailyRollingFileAppender.subAppend(DailyRollingFileAppender.java:359)
> >> >> at
> >> org.apache.log4j.WriterAppender.append(WriterAppender.java:160)
> >> >> at
> >> >> org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
> >> >> at
> >> >>
> >>
> org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
> >> >> at org.apache.log4j.Category.callAppenders(Category.java:206)
> >> >> at org.apache.log4j.Category.forcedLog(Category.java:391)
> >> >> at org.apache.log4j.Category.debug(Category.java:260)
> >> >> at
> >> >>
> >>
> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:147)
> >> >> at
> >> >>
> >>
> org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)
> >> >> 2009-11-11 04:29:00,810 WARN org.apache.zookeeper.server.Request:
> >> Ignoring
> >> >> exception during toString
> >> >> sessionid:0x424dfd908c50009 type:setWatches cxid:0xfffffffffffffff8
> >> >> zxid:0xfffffffffffffffe txntype:unknown n/a
> >> >> 2009-11-11 04:29:00,810 INFO
> org.apache.zookeeper.server.NIOServerCnxn:
> >> >> closing session:0x424dfd908c50009 NIOServerCnxn:
> >> >> java.nio.channels.SocketChannel[connected local=/
> 192.168.100.131:2181
> >> remote=/
> >> >> 192.168.100.131:35500]
> >> >> 2009-11-11 04:29:00,810 DEBUG
> org.apache.zookeeper.server.NIOServerCnxn:
> >> >> ignoring exception during input shutdown
> >> >> java.net.SocketException: Transport endpoint is not connected
> >> >> at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
> >> >> at
> >> >>
> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
> >> >> at
> >> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> >> >> at
> >> >>
> org.apache.zookeeper.server.NIOServerCnxn.close(NIOServerCnxn.java:848)
> >> >> at
> >> >>
> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:498)
> >> >> at
> >> >>
> >>
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:221)
> >> >>
> >> >
> >>
> >
>
Re: regionserver disconnection
Posted by Jean-Daniel Cryans <jd...@apache.org>.
Could you paste a bigger log trace (interested to see at least all 300
lines before that) in a pastebin?
Thx,
J-D
On Wed, Nov 11, 2009 at 10:56 AM, Zhenyu Zhong <zh...@gmail.com> wrote:
> J-D,
>
> Thanks for the reply.
>
> The problem I have is one or more of the regionservers disconnect from the
> cluster.
>
> I checked the resource usage including memory Heap usage for regionserver,
> and found nothing suspicious. The regionserver heap usage is way below the
> maxHeap defined in jvm.
>
> So I checked zookeeper log, then I found the following
> BufferUnderflowException, eventually it leads to SocketException: Transport
> endpoint is not connected.
>
>
> I also checked the regionserver log. It looks like the regionserver tried to
> connect to other zookeeper quorum server. It connects successfully, but then
> it complained about SocketException: transport endpoint is not connected.
>
> So both zookeeper log and regionserver log told me that the transport
> endpoint is not connected. I don't know why.
>
> I would be very appreciated if you could point me to the right direction.
>
> Best,
>
>
> Regionserver log:
> 2009-11-11 04:24:53,718 INFO org.apache.zookeeper.ClientCnxn: Attempting
> connection to server superpyxis0018/192.168.100.132:2181
> 2009-11-11 04:24:53,718 INFO org.apache.zookeeper.ClientCnxn: Priming
> connection to java.nio.channels.SocketChannel[connected local=/
> 192.168.100.131:43065 remote=superpyxis0018/192.168.100.132:2181]
> 2009-11-11 04:24:53,718 INFO org.apache.zookeeper.ClientCnxn: Server
> connection successful
> 2009-11-11 04:24:53,768 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
> state: Disconnected, type: None, path: null
> 2009-11-11 04:24:54,118 INFO org.apache.zookeeper.ClientCnxn: Attempting
> connection to server superpyxis0017/192.168.100.131:2181
> 2009-11-11 04:24:54,118 INFO org.apache.zookeeper.ClientCnxn: Priming
> connection to java.nio.channels.SocketChannel[connected local=/
> 192.168.100.131:46903 remote=superpyxis0017/192.168.100.131:2181]
> 2009-11-11 04:24:54,118 INFO org.apache.zookeeper.ClientCnxn: Server
> connection successful
> 2009-11-11 04:24:54,118 WARN org.apache.zookeeper.ClientCnxn: Exception
> closing session 0x24dfd90c810002 to sun.nio.ch.SelectionKeyImpl@2dd78ccf
> java.io.IOException: Read error rc = -1 java.nio.DirectByteBuffer[pos=0
> lim=4 cap=4]
> at
> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
> 2009-11-11 04:24:54,118 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> exception during shutdown input
> java.net.SocketException: Transport endpoint is not connected
> at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
>
>
>
>
>
> On Wed, Nov 11, 2009 at 1:40 PM, Jean-Daniel Cryans <jd...@apache.org>wrote:
>
>> I'm really not sure exactly what the situation is with the
>> java.nio.BufferUnderflowException, but what do you mean by
>> "regionserver disconnection"? Could you give more details?
>>
>> Thx,
>>
>> J-D
>>
>> On Wed, Nov 11, 2009 at 10:34 AM, Zhenyu Zhong <zh...@gmail.com>
>> wrote:
>> > I searched around and found HBASE-1645 "HBase could be a little more
>> dainty
>> > when zk is pulled out from under it"
>> >
>> > I am wondering whether my problem is related to this HBase-1645. Or
>> probably
>> > has nothing to do with it?
>> >
>> > thanks
>> >
>> >
>> > On Wed, Nov 11, 2009 at 12:02 PM, Zhenyu Zhong <zhongresearch@gmail.com
>> >wrote:
>> >
>> >> Hi,
>> >>
>> >> I have been monitoring the system resources such as memory Heap usage
>> for
>> >> Regionserver, load average, network bandwidth, etc.
>> >>
>> >> I experienced a disconnection last night. When I checked the memory Heap
>> >> for Regionserver, it was 2336 which is way below the max Heap 4096. I
>> don't
>> >> think it is Regionserver problem. The requests that the regionserver got
>> was
>> >> about 126 which doesn't seem to be high. The load average is kind of
>> high
>> >> >100. (but I saw disconnections while the load average is around 30).
>> The
>> >> bandwidth is not a problem.
>> >>
>> >> However, I found error messages from the zookeeper. Please see the
>> >> following error messages. There are some
>> >> BufferUnderflowException error, IOException error,etc.
>> >> May I ask what might be the causes that generate these error messages?
>> >>
>> >> Many thanks
>> >> zhenyu
>> >>
>> >>
>> >> 2009-11-11 04:29:00,810 WARN org.apache.zookeeper.server.Request:
>> Ignoring
>> >> exception during toString
>> >> java.nio.BufferUnderflowException
>> >> at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
>> >> at java.nio.ByteBuffer.get(ByteBuffer.java:675)
>> >> at
>> org.apache.zookeeper.server.Request.toString(Request.java:199)
>> >> at java.lang.String.valueOf(String.java:2827)
>> >> at java.lang.StringBuilder.append(StringBuilder.java:115)
>> >> at
>> >>
>> org.apache.zookeeper.server.quorum.CommitProcessor.processRequest(CommitProcessor.java:167)
>> >> at
>> >>
>> org.apache.zookeeper.server.quorum.FollowerRequestProcessor.run(FollowerRequestProcessor.java:68)
>> >> 2009-11-11 04:29:00,810 WARN org.apache.zookeeper.server.NIOServerCnxn:
>> >> Exception causing close of session 0x424dfd908c50009
>> >> due to java.io.IOException: Read error
>> >> 2009-11-11 04:29:00,810 DEBUG org.apache.zookeeper.server.NIOServerCnxn:
>> >> IOException stack trace
>> >> java.io.IOException: Read error
>> >> at
>> >> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:372)
>> >> at
>> >>
>> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:221)
>> >> 2009-11-11 04:29:00,810 DEBUG
>> >> org.apache.zookeeper.server.quorum.CommitProcessor: Processing request::
>> >> sessionid:0x424dfd908c50009 type:setWatches cxid:0xfffffffffffffff8
>> >> zxid:0xfffffffffffffffe txntype:unknown n/a
>> >> 2009-11-11 04:29:00,810 WARN org.apache.zookeeper.server.Request:
>> Ignoring
>> >> exception during toString
>> >> java.nio.BufferUnderflowException
>> >> at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
>> >> at java.nio.ByteBuffer.get(ByteBuffer.java:675)
>> >> at
>> org.apache.zookeeper.server.Request.toString(Request.java:199)
>> >> at java.lang.String.valueOf(String.java:2827)
>> >> at java.lang.StringBuilder.append(StringBuilder.java:115)
>> >> at
>> >>
>> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:74)
>> >> at
>> >>
>> org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)
>> >> 2009-11-11 04:29:00,810 DEBUG
>> >> org.apache.zookeeper.server.FinalRequestProcessor: Processing request::
>> >> sessionid:0x424dfd908c50009 type:setWatches cxid:0xfffffffffffffff8
>> >> zxid:0xfffffffffffffffe txntype:unknown n/a
>> >> 2009-11-11 04:29:00,810 WARN org.apache.zookeeper.server.Request:
>> Ignoring
>> >> exception during toString
>> >> java.nio.BufferUnderflowException
>> >> at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
>> >> at java.nio.ByteBuffer.get(ByteBuffer.java:675)
>> >> at
>> org.apache.zookeeper.server.Request.toString(Request.java:199)
>> >> at
>> >> org.apache.log4j.or.DefaultRenderer.doRender(DefaultRenderer.java:36)
>> >> at
>> >> org.apache.log4j.or.RendererMap.findAndRender(RendererMap.java:80)
>> >> at
>> >>
>> org.apache.log4j.spi.LoggingEvent.getRenderedMessage(LoggingEvent.java:362)
>> >> at
>> >>
>> org.apache.log4j.helpers.PatternParser$BasicPatternConverter.convert(PatternParser.java:403)
>> >> at
>> >>
>> org.apache.log4j.helpers.PatternConverter.format(PatternConverter.java:65)
>> >> at org.apache.log4j.PatternLayout.format(PatternLayout.java:502)
>> >> at
>> >> org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:302)
>> >> at
>> >>
>> org.apache.log4j.DailyRollingFileAppender.subAppend(DailyRollingFileAppender.java:359)
>> >> at
>> org.apache.log4j.WriterAppender.append(WriterAppender.java:160)
>> >> at
>> >> org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
>> >> at
>> >>
>> org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
>> >> at org.apache.log4j.Category.callAppenders(Category.java:206)
>> >> at org.apache.log4j.Category.forcedLog(Category.java:391)
>> >> at org.apache.log4j.Category.debug(Category.java:260)
>> >> at
>> >>
>> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:147)
>> >> at
>> >>
>> org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)
>> >> 2009-11-11 04:29:00,810 WARN org.apache.zookeeper.server.Request:
>> Ignoring
>> >> exception during toString
>> >> sessionid:0x424dfd908c50009 type:setWatches cxid:0xfffffffffffffff8
>> >> zxid:0xfffffffffffffffe txntype:unknown n/a
>> >> 2009-11-11 04:29:00,810 INFO org.apache.zookeeper.server.NIOServerCnxn:
>> >> closing session:0x424dfd908c50009 NIOServerCnxn:
>> >> java.nio.channels.SocketChannel[connected local=/192.168.100.131:2181
>> remote=/
>> >> 192.168.100.131:35500]
>> >> 2009-11-11 04:29:00,810 DEBUG org.apache.zookeeper.server.NIOServerCnxn:
>> >> ignoring exception during input shutdown
>> >> java.net.SocketException: Transport endpoint is not connected
>> >> at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
>> >> at
>> >> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
>> >> at
>> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>> >> at
>> >> org.apache.zookeeper.server.NIOServerCnxn.close(NIOServerCnxn.java:848)
>> >> at
>> >> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:498)
>> >> at
>> >>
>> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:221)
>> >>
>> >
>>
>
Re: regionserver disconnection
Posted by Zhenyu Zhong <zh...@gmail.com>.
J-D,
Thanks for the reply.
The problem I have is one or more of the regionservers disconnect from the
cluster.
I checked the resource usage including memory Heap usage for regionserver,
and found nothing suspicious. The regionserver heap usage is way below the
maxHeap defined in jvm.
So I checked zookeeper log, then I found the following
BufferUnderflowException, eventually it leads to SocketException: Transport
endpoint is not connected.
I also checked the regionserver log. It looks like the regionserver tried to
connect to other zookeeper quorum server. It connects successfully, but then
it complained about SocketException: transport endpoint is not connected.
So both zookeeper log and regionserver log told me that the transport
endpoint is not connected. I don't know why.
I would be very appreciated if you could point me to the right direction.
Best,
Regionserver log:
2009-11-11 04:24:53,718 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server superpyxis0018/192.168.100.132:2181
2009-11-11 04:24:53,718 INFO org.apache.zookeeper.ClientCnxn: Priming
connection to java.nio.channels.SocketChannel[connected local=/
192.168.100.131:43065 remote=superpyxis0018/192.168.100.132:2181]
2009-11-11 04:24:53,718 INFO org.apache.zookeeper.ClientCnxn: Server
connection successful
2009-11-11 04:24:53,768 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
state: Disconnected, type: None, path: null
2009-11-11 04:24:54,118 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server superpyxis0017/192.168.100.131:2181
2009-11-11 04:24:54,118 INFO org.apache.zookeeper.ClientCnxn: Priming
connection to java.nio.channels.SocketChannel[connected local=/
192.168.100.131:46903 remote=superpyxis0017/192.168.100.131:2181]
2009-11-11 04:24:54,118 INFO org.apache.zookeeper.ClientCnxn: Server
connection successful
2009-11-11 04:24:54,118 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x24dfd90c810002 to sun.nio.ch.SelectionKeyImpl@2dd78ccf
java.io.IOException: Read error rc = -1 java.nio.DirectByteBuffer[pos=0
lim=4 cap=4]
at
org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
2009-11-11 04:24:54,118 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown input
java.net.SocketException: Transport endpoint is not connected
at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
at
sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
at
org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
On Wed, Nov 11, 2009 at 1:40 PM, Jean-Daniel Cryans <jd...@apache.org>wrote:
> I'm really not sure exactly what the situation is with the
> java.nio.BufferUnderflowException, but what do you mean by
> "regionserver disconnection"? Could you give more details?
>
> Thx,
>
> J-D
>
> On Wed, Nov 11, 2009 at 10:34 AM, Zhenyu Zhong <zh...@gmail.com>
> wrote:
> > I searched around and found HBASE-1645 "HBase could be a little more
> dainty
> > when zk is pulled out from under it"
> >
> > I am wondering whether my problem is related to this HBase-1645. Or
> probably
> > has nothing to do with it?
> >
> > thanks
> >
> >
> > On Wed, Nov 11, 2009 at 12:02 PM, Zhenyu Zhong <zhongresearch@gmail.com
> >wrote:
> >
> >> Hi,
> >>
> >> I have been monitoring the system resources such as memory Heap usage
> for
> >> Regionserver, load average, network bandwidth, etc.
> >>
> >> I experienced a disconnection last night. When I checked the memory Heap
> >> for Regionserver, it was 2336 which is way below the max Heap 4096. I
> don't
> >> think it is Regionserver problem. The requests that the regionserver got
> was
> >> about 126 which doesn't seem to be high. The load average is kind of
> high
> >> >100. (but I saw disconnections while the load average is around 30).
> The
> >> bandwidth is not a problem.
> >>
> >> However, I found error messages from the zookeeper. Please see the
> >> following error messages. There are some
> >> BufferUnderflowException error, IOException error,etc.
> >> May I ask what might be the causes that generate these error messages?
> >>
> >> Many thanks
> >> zhenyu
> >>
> >>
> >> 2009-11-11 04:29:00,810 WARN org.apache.zookeeper.server.Request:
> Ignoring
> >> exception during toString
> >> java.nio.BufferUnderflowException
> >> at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
> >> at java.nio.ByteBuffer.get(ByteBuffer.java:675)
> >> at
> org.apache.zookeeper.server.Request.toString(Request.java:199)
> >> at java.lang.String.valueOf(String.java:2827)
> >> at java.lang.StringBuilder.append(StringBuilder.java:115)
> >> at
> >>
> org.apache.zookeeper.server.quorum.CommitProcessor.processRequest(CommitProcessor.java:167)
> >> at
> >>
> org.apache.zookeeper.server.quorum.FollowerRequestProcessor.run(FollowerRequestProcessor.java:68)
> >> 2009-11-11 04:29:00,810 WARN org.apache.zookeeper.server.NIOServerCnxn:
> >> Exception causing close of session 0x424dfd908c50009
> >> due to java.io.IOException: Read error
> >> 2009-11-11 04:29:00,810 DEBUG org.apache.zookeeper.server.NIOServerCnxn:
> >> IOException stack trace
> >> java.io.IOException: Read error
> >> at
> >> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:372)
> >> at
> >>
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:221)
> >> 2009-11-11 04:29:00,810 DEBUG
> >> org.apache.zookeeper.server.quorum.CommitProcessor: Processing request::
> >> sessionid:0x424dfd908c50009 type:setWatches cxid:0xfffffffffffffff8
> >> zxid:0xfffffffffffffffe txntype:unknown n/a
> >> 2009-11-11 04:29:00,810 WARN org.apache.zookeeper.server.Request:
> Ignoring
> >> exception during toString
> >> java.nio.BufferUnderflowException
> >> at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
> >> at java.nio.ByteBuffer.get(ByteBuffer.java:675)
> >> at
> org.apache.zookeeper.server.Request.toString(Request.java:199)
> >> at java.lang.String.valueOf(String.java:2827)
> >> at java.lang.StringBuilder.append(StringBuilder.java:115)
> >> at
> >>
> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:74)
> >> at
> >>
> org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)
> >> 2009-11-11 04:29:00,810 DEBUG
> >> org.apache.zookeeper.server.FinalRequestProcessor: Processing request::
> >> sessionid:0x424dfd908c50009 type:setWatches cxid:0xfffffffffffffff8
> >> zxid:0xfffffffffffffffe txntype:unknown n/a
> >> 2009-11-11 04:29:00,810 WARN org.apache.zookeeper.server.Request:
> Ignoring
> >> exception during toString
> >> java.nio.BufferUnderflowException
> >> at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
> >> at java.nio.ByteBuffer.get(ByteBuffer.java:675)
> >> at
> org.apache.zookeeper.server.Request.toString(Request.java:199)
> >> at
> >> org.apache.log4j.or.DefaultRenderer.doRender(DefaultRenderer.java:36)
> >> at
> >> org.apache.log4j.or.RendererMap.findAndRender(RendererMap.java:80)
> >> at
> >>
> org.apache.log4j.spi.LoggingEvent.getRenderedMessage(LoggingEvent.java:362)
> >> at
> >>
> org.apache.log4j.helpers.PatternParser$BasicPatternConverter.convert(PatternParser.java:403)
> >> at
> >>
> org.apache.log4j.helpers.PatternConverter.format(PatternConverter.java:65)
> >> at org.apache.log4j.PatternLayout.format(PatternLayout.java:502)
> >> at
> >> org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:302)
> >> at
> >>
> org.apache.log4j.DailyRollingFileAppender.subAppend(DailyRollingFileAppender.java:359)
> >> at
> org.apache.log4j.WriterAppender.append(WriterAppender.java:160)
> >> at
> >> org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
> >> at
> >>
> org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
> >> at org.apache.log4j.Category.callAppenders(Category.java:206)
> >> at org.apache.log4j.Category.forcedLog(Category.java:391)
> >> at org.apache.log4j.Category.debug(Category.java:260)
> >> at
> >>
> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:147)
> >> at
> >>
> org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)
> >> 2009-11-11 04:29:00,810 WARN org.apache.zookeeper.server.Request:
> Ignoring
> >> exception during toString
> >> sessionid:0x424dfd908c50009 type:setWatches cxid:0xfffffffffffffff8
> >> zxid:0xfffffffffffffffe txntype:unknown n/a
> >> 2009-11-11 04:29:00,810 INFO org.apache.zookeeper.server.NIOServerCnxn:
> >> closing session:0x424dfd908c50009 NIOServerCnxn:
> >> java.nio.channels.SocketChannel[connected local=/192.168.100.131:2181
> remote=/
> >> 192.168.100.131:35500]
> >> 2009-11-11 04:29:00,810 DEBUG org.apache.zookeeper.server.NIOServerCnxn:
> >> ignoring exception during input shutdown
> >> java.net.SocketException: Transport endpoint is not connected
> >> at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
> >> at
> >> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
> >> at
> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> >> at
> >> org.apache.zookeeper.server.NIOServerCnxn.close(NIOServerCnxn.java:848)
> >> at
> >> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:498)
> >> at
> >>
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:221)
> >>
> >
>
Re: regionserver disconnection
Posted by Jean-Daniel Cryans <jd...@apache.org>.
I'm really not sure exactly what the situation is with the
java.nio.BufferUnderflowException, but what do you mean by
"regionserver disconnection"? Could you give more details?
Thx,
J-D
On Wed, Nov 11, 2009 at 10:34 AM, Zhenyu Zhong <zh...@gmail.com> wrote:
> I searched around and found HBASE-1645 "HBase could be a little more dainty
> when zk is pulled out from under it"
>
> I am wondering whether my problem is related to this HBase-1645. Or probably
> has nothing to do with it?
>
> thanks
>
>
> On Wed, Nov 11, 2009 at 12:02 PM, Zhenyu Zhong <zh...@gmail.com>wrote:
>
>> Hi,
>>
>> I have been monitoring the system resources such as memory Heap usage for
>> Regionserver, load average, network bandwidth, etc.
>>
>> I experienced a disconnection last night. When I checked the memory Heap
>> for Regionserver, it was 2336 which is way below the max Heap 4096. I don't
>> think it is Regionserver problem. The requests that the regionserver got was
>> about 126 which doesn't seem to be high. The load average is kind of high
>> >100. (but I saw disconnections while the load average is around 30). The
>> bandwidth is not a problem.
>>
>> However, I found error messages from the zookeeper. Please see the
>> following error messages. There are some
>> BufferUnderflowException error, IOException error,etc.
>> May I ask what might be the causes that generate these error messages?
>>
>> Many thanks
>> zhenyu
>>
>>
>> 2009-11-11 04:29:00,810 WARN org.apache.zookeeper.server.Request: Ignoring
>> exception during toString
>> java.nio.BufferUnderflowException
>> at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
>> at java.nio.ByteBuffer.get(ByteBuffer.java:675)
>> at org.apache.zookeeper.server.Request.toString(Request.java:199)
>> at java.lang.String.valueOf(String.java:2827)
>> at java.lang.StringBuilder.append(StringBuilder.java:115)
>> at
>> org.apache.zookeeper.server.quorum.CommitProcessor.processRequest(CommitProcessor.java:167)
>> at
>> org.apache.zookeeper.server.quorum.FollowerRequestProcessor.run(FollowerRequestProcessor.java:68)
>> 2009-11-11 04:29:00,810 WARN org.apache.zookeeper.server.NIOServerCnxn:
>> Exception causing close of session 0x424dfd908c50009
>> due to java.io.IOException: Read error
>> 2009-11-11 04:29:00,810 DEBUG org.apache.zookeeper.server.NIOServerCnxn:
>> IOException stack trace
>> java.io.IOException: Read error
>> at
>> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:372)
>> at
>> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:221)
>> 2009-11-11 04:29:00,810 DEBUG
>> org.apache.zookeeper.server.quorum.CommitProcessor: Processing request::
>> sessionid:0x424dfd908c50009 type:setWatches cxid:0xfffffffffffffff8
>> zxid:0xfffffffffffffffe txntype:unknown n/a
>> 2009-11-11 04:29:00,810 WARN org.apache.zookeeper.server.Request: Ignoring
>> exception during toString
>> java.nio.BufferUnderflowException
>> at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
>> at java.nio.ByteBuffer.get(ByteBuffer.java:675)
>> at org.apache.zookeeper.server.Request.toString(Request.java:199)
>> at java.lang.String.valueOf(String.java:2827)
>> at java.lang.StringBuilder.append(StringBuilder.java:115)
>> at
>> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:74)
>> at
>> org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)
>> 2009-11-11 04:29:00,810 DEBUG
>> org.apache.zookeeper.server.FinalRequestProcessor: Processing request::
>> sessionid:0x424dfd908c50009 type:setWatches cxid:0xfffffffffffffff8
>> zxid:0xfffffffffffffffe txntype:unknown n/a
>> 2009-11-11 04:29:00,810 WARN org.apache.zookeeper.server.Request: Ignoring
>> exception during toString
>> java.nio.BufferUnderflowException
>> at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
>> at java.nio.ByteBuffer.get(ByteBuffer.java:675)
>> at org.apache.zookeeper.server.Request.toString(Request.java:199)
>> at
>> org.apache.log4j.or.DefaultRenderer.doRender(DefaultRenderer.java:36)
>> at
>> org.apache.log4j.or.RendererMap.findAndRender(RendererMap.java:80)
>> at
>> org.apache.log4j.spi.LoggingEvent.getRenderedMessage(LoggingEvent.java:362)
>> at
>> org.apache.log4j.helpers.PatternParser$BasicPatternConverter.convert(PatternParser.java:403)
>> at
>> org.apache.log4j.helpers.PatternConverter.format(PatternConverter.java:65)
>> at org.apache.log4j.PatternLayout.format(PatternLayout.java:502)
>> at
>> org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:302)
>> at
>> org.apache.log4j.DailyRollingFileAppender.subAppend(DailyRollingFileAppender.java:359)
>> at org.apache.log4j.WriterAppender.append(WriterAppender.java:160)
>> at
>> org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
>> at
>> org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
>> at org.apache.log4j.Category.callAppenders(Category.java:206)
>> at org.apache.log4j.Category.forcedLog(Category.java:391)
>> at org.apache.log4j.Category.debug(Category.java:260)
>> at
>> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:147)
>> at
>> org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)
>> 2009-11-11 04:29:00,810 WARN org.apache.zookeeper.server.Request: Ignoring
>> exception during toString
>> sessionid:0x424dfd908c50009 type:setWatches cxid:0xfffffffffffffff8
>> zxid:0xfffffffffffffffe txntype:unknown n/a
>> 2009-11-11 04:29:00,810 INFO org.apache.zookeeper.server.NIOServerCnxn:
>> closing session:0x424dfd908c50009 NIOServerCnxn:
>> java.nio.channels.SocketChannel[connected local=/192.168.100.131:2181remote=/
>> 192.168.100.131:35500]
>> 2009-11-11 04:29:00,810 DEBUG org.apache.zookeeper.server.NIOServerCnxn:
>> ignoring exception during input shutdown
>> java.net.SocketException: Transport endpoint is not connected
>> at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
>> at
>> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
>> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>> at
>> org.apache.zookeeper.server.NIOServerCnxn.close(NIOServerCnxn.java:848)
>> at
>> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:498)
>> at
>> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:221)
>>
>
Re: regionserver disconnection
Posted by Zhenyu Zhong <zh...@gmail.com>.
I searched around and found HBASE-1645 "HBase could be a little more dainty
when zk is pulled out from under it"
I am wondering whether my problem is related to this HBase-1645. Or probably
has nothing to do with it?
thanks
On Wed, Nov 11, 2009 at 12:02 PM, Zhenyu Zhong <zh...@gmail.com>wrote:
> Hi,
>
> I have been monitoring the system resources such as memory Heap usage for
> Regionserver, load average, network bandwidth, etc.
>
> I experienced a disconnection last night. When I checked the memory Heap
> for Regionserver, it was 2336 which is way below the max Heap 4096. I don't
> think it is Regionserver problem. The requests that the regionserver got was
> about 126 which doesn't seem to be high. The load average is kind of high
> >100. (but I saw disconnections while the load average is around 30). The
> bandwidth is not a problem.
>
> However, I found error messages from the zookeeper. Please see the
> following error messages. There are some
> BufferUnderflowException error, IOException error,etc.
> May I ask what might be the causes that generate these error messages?
>
> Many thanks
> zhenyu
>
>
> 2009-11-11 04:29:00,810 WARN org.apache.zookeeper.server.Request: Ignoring
> exception during toString
> java.nio.BufferUnderflowException
> at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
> at java.nio.ByteBuffer.get(ByteBuffer.java:675)
> at org.apache.zookeeper.server.Request.toString(Request.java:199)
> at java.lang.String.valueOf(String.java:2827)
> at java.lang.StringBuilder.append(StringBuilder.java:115)
> at
> org.apache.zookeeper.server.quorum.CommitProcessor.processRequest(CommitProcessor.java:167)
> at
> org.apache.zookeeper.server.quorum.FollowerRequestProcessor.run(FollowerRequestProcessor.java:68)
> 2009-11-11 04:29:00,810 WARN org.apache.zookeeper.server.NIOServerCnxn:
> Exception causing close of session 0x424dfd908c50009
> due to java.io.IOException: Read error
> 2009-11-11 04:29:00,810 DEBUG org.apache.zookeeper.server.NIOServerCnxn:
> IOException stack trace
> java.io.IOException: Read error
> at
> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:372)
> at
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:221)
> 2009-11-11 04:29:00,810 DEBUG
> org.apache.zookeeper.server.quorum.CommitProcessor: Processing request::
> sessionid:0x424dfd908c50009 type:setWatches cxid:0xfffffffffffffff8
> zxid:0xfffffffffffffffe txntype:unknown n/a
> 2009-11-11 04:29:00,810 WARN org.apache.zookeeper.server.Request: Ignoring
> exception during toString
> java.nio.BufferUnderflowException
> at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
> at java.nio.ByteBuffer.get(ByteBuffer.java:675)
> at org.apache.zookeeper.server.Request.toString(Request.java:199)
> at java.lang.String.valueOf(String.java:2827)
> at java.lang.StringBuilder.append(StringBuilder.java:115)
> at
> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:74)
> at
> org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)
> 2009-11-11 04:29:00,810 DEBUG
> org.apache.zookeeper.server.FinalRequestProcessor: Processing request::
> sessionid:0x424dfd908c50009 type:setWatches cxid:0xfffffffffffffff8
> zxid:0xfffffffffffffffe txntype:unknown n/a
> 2009-11-11 04:29:00,810 WARN org.apache.zookeeper.server.Request: Ignoring
> exception during toString
> java.nio.BufferUnderflowException
> at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
> at java.nio.ByteBuffer.get(ByteBuffer.java:675)
> at org.apache.zookeeper.server.Request.toString(Request.java:199)
> at
> org.apache.log4j.or.DefaultRenderer.doRender(DefaultRenderer.java:36)
> at
> org.apache.log4j.or.RendererMap.findAndRender(RendererMap.java:80)
> at
> org.apache.log4j.spi.LoggingEvent.getRenderedMessage(LoggingEvent.java:362)
> at
> org.apache.log4j.helpers.PatternParser$BasicPatternConverter.convert(PatternParser.java:403)
> at
> org.apache.log4j.helpers.PatternConverter.format(PatternConverter.java:65)
> at org.apache.log4j.PatternLayout.format(PatternLayout.java:502)
> at
> org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:302)
> at
> org.apache.log4j.DailyRollingFileAppender.subAppend(DailyRollingFileAppender.java:359)
> at org.apache.log4j.WriterAppender.append(WriterAppender.java:160)
> at
> org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
> at
> org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
> at org.apache.log4j.Category.callAppenders(Category.java:206)
> at org.apache.log4j.Category.forcedLog(Category.java:391)
> at org.apache.log4j.Category.debug(Category.java:260)
> at
> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:147)
> at
> org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)
> 2009-11-11 04:29:00,810 WARN org.apache.zookeeper.server.Request: Ignoring
> exception during toString
> sessionid:0x424dfd908c50009 type:setWatches cxid:0xfffffffffffffff8
> zxid:0xfffffffffffffffe txntype:unknown n/a
> 2009-11-11 04:29:00,810 INFO org.apache.zookeeper.server.NIOServerCnxn:
> closing session:0x424dfd908c50009 NIOServerCnxn:
> java.nio.channels.SocketChannel[connected local=/192.168.100.131:2181remote=/
> 192.168.100.131:35500]
> 2009-11-11 04:29:00,810 DEBUG org.apache.zookeeper.server.NIOServerCnxn:
> ignoring exception during input shutdown
> java.net.SocketException: Transport endpoint is not connected
> at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at
> org.apache.zookeeper.server.NIOServerCnxn.close(NIOServerCnxn.java:848)
> at
> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:498)
> at
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:221)
>