You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zookeeper.apache.org by "Patrick Hunt (JIRA)" <ji...@apache.org> on 2010/11/24 09:24:16 UTC
[jira] Commented: (ZOOKEEPER-939) the threads number of a zookeeper
is increased all the time
[ https://issues.apache.org/jira/browse/ZOOKEEPER-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12935252#action_12935252 ]
Patrick Hunt commented on ZOOKEEPER-939:
----------------------------------------
Looks like a dupe of ZOOKEEPER-880 to me. Can you take a look and give that patch a try (success has been reported by other users).
> the threads number of a zookeeper is increased all the time
> -----------------------------------------------------------
>
> Key: ZOOKEEPER-939
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-939
> Project: ZooKeeper
> Issue Type: Bug
> Components: server
> Affects Versions: 3.3.0
> Environment: Linux 2.6.9-52bs #2 SMP Fri Jan 26 13:34:38 CST 2007 x86_64 x86_64 x86_64 GNU/Linux
> Reporter: Qian Ye
>
> I have a group of zookeeper servers, there are three servers in this group.
> server.0=10.81.4.11:2888:3888
> server.1=10.23.240.93:2888:3888
> server.2=10.23.244.224:2888:3888
> At first, the cluster ran well. About several days ago, I shut down the zookeeper process on one of servers(server.2)., and today, I find that the other two servers run in wired status(the network is fine). The zookeeper process take pretty much resource on the two servers:
> on server.1 (it's the leader)
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 26836 work 18 0 12.8g 803m 8724 S 3.7 10.1 195:56.56 java
> $ ll /proc/26836/fd/ | wc -l
> 3586
> [work@tc-test-aos03.tc.baidu.com conf]$ ll /proc/26836/task/ | wc -l
> 10510
> some warning log:
> 2010-11-24 15:37:48,705 - WARN [Thread-37409:QuorumCnxManager$SendWorker@589] - Send worker leaving thread
> 2010-11-24 15:39:48,626 - WARN [Thread-37414:QuorumCnxManager$RecvWorker@658] - Connection broken:
> java.nio.channels.AsynchronousCloseException
> at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:185)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:263)
> at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:629)
> 2010-11-24 15:39:48,656 - WARN [Thread-37413:QuorumCnxManager$SendWorker@581] - Interrupted while waiting for message on queue
> java.lang.InterruptedException
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1899)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1976)
> at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:342)
> at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:570)
> 2010-11-24 15:39:48,657 - WARN [Thread-37413:QuorumCnxManager$SendWorker@589] - Send worker leaving thread
> 2010-11-24 15:41:48,614 - WARN [Thread-37417:QuorumCnxManager$SendWorker@581] - Interrupted while waiting for message on queue
> java.lang.InterruptedException
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1899)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1976)
> at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:342)
> at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:570)
> 2010-11-24 15:41:48,643 - WARN [Thread-37418:QuorumCnxManager$RecvWorker@658] - Connection broken:
> java.nio.channels.AsynchronousCloseException
> at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:185)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:263)
> at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:629)
> 2010-11-24 15:41:48,662 - WARN [Thread-37417:QuorumCnxManager$SendWorker@589] - Send worker leaving thread
> 2010-11-24 15:43:48,627 - WARN [Thread-37421:QuorumCnxManager$SendWorker@581] - Interrupted while waiting for message on queue
> java.lang.InterruptedException
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1899)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1976)
> at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:342)
> at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:570)
> 2010-11-24 15:43:48,627 - WARN [Thread-37422:QuorumCnxManager$RecvWorker@658] - Connection broken:
> java.nio.channels.AsynchronousCloseException
> at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:185)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:263)
> at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:629)
> 2010-11-24 15:43:48,654 - WARN [Thread-37421:QuorumCnxManager$SendWorker@589] - Send worker leaving thread
> 2010-11-24 15:44:48,622 - WARN [Thread-37424:QuorumCnxManager$RecvWorker@658] - Connection broken:
> java.nio.channels.AsynchronousCloseException
> at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:185)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:263)
> at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:629)
> 2010-11-24 15:44:48,652 - WARN [Thread-37423:QuorumCnxManager$SendWorker@581] - Interrupted while waiting for message on queue
> java.lang.InterruptedException
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1899)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1976)
> at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:342)
> at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:570)
> 2010-11-24 15:44:48,653 - WARN [Thread-37423:QuorumCnxManager$SendWorker@589] - Send worker leaving thread
> 2010-11-24 15:45:48,668 - WARN [Thread-37426:QuorumCnxManager$RecvWorker@658] - Connection broken:
> java.io.IOException: Channel eof
> at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:630)
> 2010-11-24 15:46:48,647 - WARN [Thread-37427:QuorumCnxManager$SendWorker@581] - Interrupted while waiting for message on queue
> java.lang.InterruptedException
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1899)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1976)
> at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:342)
> at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:570)
> 2010-11-24 15:46:48,722 - WARN [Thread-37428:QuorumCnxManager$RecvWorker@658] - Connection broken:
> java.nio.channels.AsynchronousCloseException
> at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:185)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:263)
> at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:629)
> 2010-11-24 15:46:48,736 - WARN [Thread-37427:QuorumCnxManager$SendWorker@589] - Send worker leaving thread
> 2010-11-24 15:47:48,687 - WARN [Thread-37430:QuorumCnxManager$RecvWorker@658] - Connection broken:
> java.io.IOException: Channel eof
> at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:630)
> on server.0
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 27322 work 19 0 15.2g 943m 9140 S 38.6 11.8 1396:51 java
> $ ll /proc/27322/fd/ | wc -l
> 3587
> $ ll /proc/27322/task/ | wc -l
> 12938
> 2010-11-24 15:37:49,269 - WARN [Thread-37407:QuorumCnxManager$SendWorker@589] - Send worker leaving thread
> 2010-11-24 15:39:49,235 - WARN [Thread-37412:QuorumCnxManager$RecvWorker@658] - Connection broken:
> java.io.IOException: Channel eof
> at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:630)
> 2010-11-24 15:39:49,410 - WARN [Thread-37411:QuorumCnxManager$SendWorker@581] - Interrupted while waiting for message on queue
> java.lang.InterruptedException
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1899)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1976)
> at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:342)
> at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:570)
> 2010-11-24 15:39:49,411 - WARN [Thread-37411:QuorumCnxManager$SendWorker@589] - Send worker leaving thread
> 2010-11-24 15:41:49,314 - WARN [Thread-37416:QuorumCnxManager$RecvWorker@658] - Connection broken:
> java.io.IOException: Channel eof
> at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:630)
> 2010-11-24 15:41:49,383 - WARN [Thread-37415:QuorumCnxManager$SendWorker@581] - Interrupted while waiting for message on queue
> java.lang.InterruptedException
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1899)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1976)
> at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:342)
> at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:570)
> 2010-11-24 15:41:49,405 - WARN [Thread-37415:QuorumCnxManager$SendWorker@589] - Send worker leaving thread
> 2010-11-24 15:43:49,372 - WARN [Thread-37420:QuorumCnxManager$RecvWorker@658] - Connection broken:
> java.io.IOException: Channel eof
> at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:630)
> 2010-11-24 15:43:49,512 - WARN [Thread-37419:QuorumCnxManager$SendWorker@581] - Interrupted while waiting for message on queue
> java.lang.InterruptedException
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1899)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1976)
> at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:342)
> at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:570)
> 2010-11-24 15:43:49,513 - WARN [Thread-37419:QuorumCnxManager$SendWorker@589] - Send worker leaving thread
> 2010-11-24 15:44:49,407 - WARN [Thread-37422:QuorumCnxManager$RecvWorker@658] - Connection broken:
> java.io.IOException: Channel eof
> at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:630)
> 2010-11-24 15:45:49,645 - WARN [Thread-37424:QuorumCnxManager$RecvWorker@658] - Connection broken:
> java.nio.channels.AsynchronousCloseException
> at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:185)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:263)
> at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:629)
> 2010-11-24 15:45:49,781 - WARN [Thread-37423:QuorumCnxManager$SendWorker@581] - Interrupted while waiting for message on queue
> java.lang.InterruptedException
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1899)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1976)
> at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:342)
> at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:570)
> 2010-11-24 15:45:49,799 - WARN [Thread-37423:QuorumCnxManager$SendWorker@589] - Send worker leaving thread
> 2010-11-24 15:46:49,495 - WARN [Thread-37427:QuorumCnxManager$RecvWorker@658] - Connection broken:
> java.io.IOException: Channel eof
> at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:630)
> 2010-11-24 15:47:49,541 - WARN [Thread-37429:QuorumCnxManager$RecvWorker@658] - Connection broken:
> java.nio.channels.AsynchronousCloseException
> at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:185)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:263)
> at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:629)
> 2010-11-24 15:47:49,622 - WARN [Thread-37428:QuorumCnxManager$SendWorker@581] - Interrupted while waiting for message on queue
> java.lang.InterruptedException
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1899)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1976)
> at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:342)
> at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:570)
> 2010-11-24 15:47:49,622 - WARN [Thread-37428:QuorumCnxManager$SendWorker@589] - Send worker leaving thread
> 2010-11-24 15:48:48,827 - WARN [Thread-37431:QuorumCnxManager$RecvWorker@658] - Connection broken:
> java.io.IOException: Channel eof
> at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:630)
> What's more, the number of threads under the zookeeper process is still increasing time by time. It seems that , something is wrong in communication of the two servers. Have anyone met such problem before?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.