You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zookeeper.apache.org by "yeshuangshuang (JIRA)" <ji...@apache.org> on 2019/03/06 02:40:00 UTC
[jira] [Commented] (ZOOKEEPER-3211) zookeeper standalone mode，found a high level bug in kernel of centos7.0 ，zookeeper Server's tcp/ip socket connections(default 60 ) are CLOSE_WAIT ，this lead to zk can't work for client any more
[ https://issues.apache.org/jira/browse/ZOOKEEPER-3211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16785170#comment-16785170 ]
yeshuangshuang commented on ZOOKEEPER-3211:
-------------------------------------------
at java.lang.Thread.run(Thread.java:748)
2019-03-05 20:11:13,214 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:ZooKeeperServer@793] - Connection request from old client /127.0.0.1:43911; will be dropped if server is in r-o mode
2019-03-05 20:11:18,817 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:NIOServerCnxn@349] - caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x1694f35e3bc0058, likely client has closed socket
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
at java.lang.Thread.run(Thread.java:748)
2019-03-05 20:11:19,818 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:ZooKeeperServer@793] - Connection request from old client /127.0.0.1:44131; will be dropped if server is in r-o mode
2019-03-05 20:11:19,881 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:NIOServerCnxn@349] - caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x1694f35e3bc0000, likely client has closed socket
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
at java.lang.Thread.run(Thread.java:748)
2019-03-05 20:11:19,882 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:ZooKeeperServer@793] - Connection request from old client /127.0.0.1:44135; will be dropped if server is in r-o mode
2019-03-05 20:11:20,079 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:NIOServerCnxn@349] - caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x1694f35e3bc0057, likely client has closed socket
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
at java.lang.Thread.run(Thread.java:748)
2019-03-05 20:11:21,079 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:ZooKeeperServer@793] - Connection request from old client /127.0.0.1:44179; will be dropped if server is in r-o mode
2019-03-05 20:11:24,517 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:ZooKeeperServer@793] - Connection request from old client /127.0.0.1:44284; will be dropped if server is in r-o mode
2019-03-05 20:11:26,485 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:NIOServerCnxn@349] - caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x1694f35e3bc0058, likely client has closed socket
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
at java.lang.Thread.run(Thread.java:748)
2019-03-05 20:11:26,550 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:ZooKeeperServer@793] - Connection request from old client /127.0.0.1:44316; will be dropped if server is in r-o mode
2019-03-05 20:11:27,485 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:ZooKeeperServer@793] - Connection request from old client /127.0.0.1:44358; will be dropped if server is in r-o mode
2019-03-05 20:11:33,221 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:ZooKeeperServer@793] - Connection request from old client /127.0.0.1:44490; will be dropped if server is in r-o mode
2019-03-05 20:11:33,746 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:ZooKeeperServer@793] - Connection request from old client /127.0.0.1:44512; will be dropped if server is in r-o mode
2019-03-05 20:11:35,153 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:ZooKeeperServer@793] - Connection request from old client /127.0.0.1:44554; will be dropped if server is in r-o mode
2019-03-05 20:11:39,888 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:ZooKeeperServer@793] - Connection request from old client /127.0.0.1:44670; will be dropped if server is in r-o mode
2019-03-05 20:11:40,625 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:ZooKeeperServer@793] - Connection request from old client /127.0.0.1:44691; will be dropped if server is in r-o mode
2019-03-05 20:11:42,821 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:ZooKeeperServer@793] - Connection request from old client /127.0.0.1:44763; will be dropped if server is in r-o mode
2019-03-05 20:11:46,414 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:ZooKeeperServer@793] - Connection request from old client /127.0.0.1:44889; will be dropped if server is in r-o mode
2019-03-05 20:11:46,556 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:ZooKeeperServer@793] - Connection request from old client /127.0.0.1:44893; will be dropped if server is in r-o mode
2019-03-05 20:11:50,489 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:ZooKeeperServer@793] - Connection request from old client /127.0.0.1:44952; will be dropped if server is in r-o mode
2019-03-05 20:11:53,228 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:ZooKeeperServer@793] - Connection request from old client /127.0.0.1:45007; will be dropped if server is in r-o mode
2019-03-05 20:11:58,156 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:ZooKeeperServer@793] - Connection request from old client /127.0.0.1:45151; will be dropped if server is in r-o mode
2019-03-05 20:11:59,081 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:ZooKeeperServer@793] - Connection request from old client /127.0.0.1:45180; will be dropped if server is in r-o mode
2019-03-05 20:11:59,897 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:ZooKeeperServer@793] - Connection request from old client /127.0.0.1:45206; will be dropped if server is in r-o mode
2019-03-05 20:12:04,244 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:ZooKeeperServer@793] - Connection request from old client /127.0.0.1:45270; will be dropped if server is in r-o mode
2019-03-05 20:12:05,824 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:ZooKeeperServer@793] - Connection request from old client /127.0.0.1:45300; will be dropped if server is in r-o mode
2019-03-05 20:12:06,567 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:ZooKeeperServer@793] - Connection request from old client /127.0.0.1:45310; will be dropped if server is in r-o mode
2019-03-05 20:12:10,373 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:ZooKeeperServer@793] - Connection request from old client /127.0.0.1:45422; will be dropped if server is in r-o mode
2019-03-05 20:12:11,749 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:ZooKeeperServer@793] - Connection request from old client /127.0.0.1:45448; will be dropped if server is in r-o mode
2019-03-05 20:12:13,237 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:ZooKeeperServer@793] - Connection request from old client /127.0.0.1:45491; will be dropped if server is in r-o mode
2019-03-05 20:12:13,492 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:ZooKeeperServer@793] - Connection request from old client /127.0.0.1:45498; will be dropped if server is in r-o mode
2019-03-05 20:12:19,905 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:ZooKeeperServer@793] - Connection request from old client /127.0.0.1:45582; will be dropped if server is in r-o mode
2019-03-05 20:12:21,161 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:ZooKeeperServer@793] - Connection request from old client /127.0.0.1:45592; will be dropped if server is in r-o mode
2019-03-05 20:12:24,367 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:ZooKeeperServer@793] - Connection request from old client /127.0.0.1:45657; will be dropped if server is in r-o mode
2019-03-05 20:12:24,417 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:ZooKeeperServer@793] - Connection request from old client /127.0.0.1:45658; will be dropped if server is in r-o mode
2019-03-05 20:12:26,574 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2900:ZooKeeperServer@793] - Connection request from old client /127.0.0.1:45700; will be dropped if server is
zookeeper.log
> zookeeper standalone mode，found a high level bug in kernel of centos7.0 ，zookeeper Server's tcp/ip socket connections(default 60 ) are CLOSE_WAIT ，this lead to zk can't work for client any more
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: ZOOKEEPER-3211
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3211
> Project: ZooKeeper
> Issue Type: Bug
> Components: server
> Affects Versions: 3.4.5
> Environment: 1.zoo.cfg
> server.1=127.0.0.1:2902:2903
> 2.kernel
> kernel：Linux localhost.localdomain 3.10.0-123.el7.x86_64 #1 SMP Tue Feb 12 19:44:50 EST 2019 x86_64 x86_64 x86_64 GNU/Linux
> JDK：
> java version "1.7.0_181"
> OpenJDK Runtime Environment (rhel-2.6.14.5.el7-x86_64 u181-b00)
> OpenJDK 64-Bit Server VM (build 24.181-b00, mixed mode)
> zk: 3.4.5
> Reporter: yeshuangshuang
> Priority: Blocker
> Fix For: 3.4.5
>
> Attachments: 1.log, 2018-12-09_124131.png, 2018-12-09_124210.png, 2018-12-09_132854.png, 2018-12-09_133017.png, 2018-12-09_133049.png, 2018-12-09_133111.png, 2018-12-09_133131.png, 2018-12-09_133150.png, 2018-12-09_133210.png, 2018-12-09_133229.png, 2018-12-09_133248.png, 2018-12-09_133320.png
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> 1.config--zoo.cfg
> server.1=127.0.0.1:2902:2903
> 2.kernel version
> version：Linux localhost.localdomain 3.10.0-123.el7.x86_64 #1 SMP Tue Feb 12 19:44:50 EST 2019 x86_64 x86_64 x86_64 GNU/Linux
> JDK：
> java version "1.7.0_181"
> OpenJDK Runtime Environment (rhel-2.6.14.5.el7-x86_64 u181-b00)
> OpenJDK 64-Bit Server VM (build 24.181-b00, mixed mode)
> zk: 3.4.5
> 3.bug details：
> Occasionally，But the recurrence probability is extremely high. At first, the read-write timeout takes about 6s, and after a few minutes, all connections (including long ones) will be CLOSE_WAIT state.
> 4.:Circumvention scheme: it is found that all connections become close_wait to restart the zookeeper server side actively
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)