You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by ashish soni <ai...@gmail.com> on 2020/05/14 23:20:25 UTC
Error in zookeeper 3.6.1 logs with Apache Storm 1.22: Unable to read
additional data from client, it probably closed the socket
Hi,
I am running a containerized zookeeper 3.6.1 version and it is getting
filled with below WARN logs,
[myid:2] - WARN [NIOWorkerThread-2:NIOServerCnxn@364] - Unexpected
exception EndOfStreamException: Unable to read additional data from client,
it probably closed the socket: address = /10.162.53.62:48988, session =
0x200866346830367 at
org.apache.zookeeper.server.NIOServerCnxn.handleFailedRead(NIOServerCnxn.java:163)
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:326)
at
org.apache.zookeeper.server.NIOServerCnxnFactory$IOWorkRequest.doWork(NIOServerCnxnFactory.java:522)
at
org.apache.zookeeper.server.WorkerService$ScheduledWorkRequest.run(WorkerService.java:154)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[myid:2] - INFO [NIOWorkerThread-1:Learner@137] - Revalidating client:
0x3008663464f03ac [myid:2] - INFO [NIOWorkerThread-1:Learner@137] -
Revalidating client: 0x100866346320382 INFO [NIOWorkerThread-1:Learner@137]
- Revalidating client: 0x1008663463202d9 INFO [NIOWorkerThread-1:Learner@137]
- Revalidating client: 0x1008663463203b2 INFO [NIOWorkerThread-1:Learner@137]
- Revalidating client: 0x2008663468302d5 INFO [NIOWorkerThread-1:Learner@137]
- Revalidating client: 0x4008663438f03ca INFO [NIOWorkerThread-1:Learner@137]
- Revalidating client: 0x1008663463203bf and so on
I have multiple instances of Apache Storm 1.22 running as containerized
service which is running on 10.162.53.62 (IP address in the above WARN
logs). The storm is using a 3.4.14 zookeeper version and the curator is at
4.0.1 (which is compatible with 3.5+ versions). Here is the pool file from
the storm 1.1.x branch
https://github.com/apache/storm/blob/1.1.x-branch/pom.xml for reference.
Even the latest storm is using the same configs as far as the zookeeper is
concerned.
So, is something wrong with the compatibility of the storm's zookeeper
version and mine? Or is anything missing from my zookeeper configs point of
view? Any details on what might be happening would really be helpful.
Here is my zoo.cfg,
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=20
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=15
# HA (multi-node) setup
standaloneEnabled=false
dynamicConfigFile=/opt/zookeeper/conf/zoo.cfg.dynamic
# location of in-memory database snapshots
dataDir=/data
# location of the transaction-log
dataLogDir=/data/xnlogs
# enable regular purging of old data
autopurge.purgeInterval=24
autopurge.snapRetainCount=5
Re: Error in zookeeper 3.6.1 logs with Apache Storm 1.22: Unable to
read additional data from client, it probably closed the socket
Posted by Szalay-Bekő Máté <sz...@gmail.com>.
In theory, the 3.4 client can connect / work with the 3.6 server without
any problem. I am not sure how Curator complicates this issue, but based on
the logs I don't see any ZooKeeper issue just yet.
The logs you pasted here shows that the Storm/Curator node closed the
ZooKeeper client session, maybe due to some timeout. The ZooKeeper Client
will terminate the session if the 2/3 time of SessionTimeout elapsed, then
it tries to reconnect (I guess to an other ZooKeeper node). The
"Revalidating client" appears because of the reconnection. Maybe the
ZooKeeper server is overloaded / slow to respond? Or maybe the negotiated
session timeout is too low? Can you see the session timeout for these
sessions (e.g. for 0x200866346830367) in the Storm logs? Or can you see
anything else suspicious in the ZooKeeper client logs (aka. Storm/Curator
logs) around this time?
On Fri, May 15, 2020 at 1:22 AM ashish soni <ai...@gmail.com>
wrote:
> An old issue for reference:
> https://issues.apache.org/jira/browse/ZOOKEEPER-1582
>
> On Thu, May 14, 2020 at 4:20 PM ashish soni <ai...@gmail.com>
> wrote:
>
> > Hi,
> >
> > I am running a containerized zookeeper 3.6.1 version and it is getting
> > filled with below WARN logs,
> >
> > [myid:2] - WARN [NIOWorkerThread-2:NIOServerCnxn@364] - Unexpected
> > exception EndOfStreamException: Unable to read additional data from
> client,
> > it probably closed the socket: address = /10.162.53.62:48988, session =
> > 0x200866346830367 at
> >
> org.apache.zookeeper.server.NIOServerCnxn.handleFailedRead(NIOServerCnxn.java:163)
> > at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:326)
> > at
> >
> org.apache.zookeeper.server.NIOServerCnxnFactory$IOWorkRequest.doWork(NIOServerCnxnFactory.java:522)
> > at
> >
> org.apache.zookeeper.server.WorkerService$ScheduledWorkRequest.run(WorkerService.java:154)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > at java.lang.Thread.run(Thread.java:748)
> > [myid:2] - INFO [NIOWorkerThread-1:Learner@137] - Revalidating client:
> > 0x3008663464f03ac [myid:2] - INFO [NIOWorkerThread-1:Learner@137] -
> > Revalidating client: 0x100866346320382 INFO
> [NIOWorkerThread-1:Learner@137]
> > - Revalidating client: 0x1008663463202d9 INFO
> [NIOWorkerThread-1:Learner@137]
> > - Revalidating client: 0x1008663463203b2 INFO
> [NIOWorkerThread-1:Learner@137]
> > - Revalidating client: 0x2008663468302d5 INFO
> [NIOWorkerThread-1:Learner@137]
> > - Revalidating client: 0x4008663438f03ca INFO
> [NIOWorkerThread-1:Learner@137]
> > - Revalidating client: 0x1008663463203bf and so on
> >
> > I have multiple instances of Apache Storm 1.22 running as containerized
> > service which is running on 10.162.53.62 (IP address in the above WARN
> > logs). The storm is using a 3.4.14 zookeeper version and the curator is
> at
> > 4.0.1 (which is compatible with 3.5+ versions). Here is the pool file
> from
> > the storm 1.1.x branch
> > https://github.com/apache/storm/blob/1.1.x-branch/pom.xml for reference.
> > Even the latest storm is using the same configs as far as the zookeeper
> is
> > concerned.
> >
> > So, is something wrong with the compatibility of the storm's zookeeper
> > version and mine? Or is anything missing from my zookeeper configs point
> of
> > view? Any details on what might be happening would really be helpful.
> >
> > Here is my zoo.cfg,
> >
> > tickTime=2000
> > # The number of ticks that the initial
> > # synchronization phase can take
> > initLimit=20
> > # The number of ticks that can pass between
> > # sending a request and getting an acknowledgement
> > syncLimit=15
> > # HA (multi-node) setup
> > standaloneEnabled=false
> > dynamicConfigFile=/opt/zookeeper/conf/zoo.cfg.dynamic
> >
> > # location of in-memory database snapshots
> > dataDir=/data
> > # location of the transaction-log
> > dataLogDir=/data/xnlogs
> >
> > # enable regular purging of old data
> > autopurge.purgeInterval=24
> > autopurge.snapRetainCount=5
> >
> >
>
Re: Error in zookeeper 3.6.1 logs with Apache Storm 1.22: Unable to
read additional data from client, it probably closed the socket
Posted by ashish soni <ai...@gmail.com>.
An old issue for reference:
https://issues.apache.org/jira/browse/ZOOKEEPER-1582
On Thu, May 14, 2020 at 4:20 PM ashish soni <ai...@gmail.com>
wrote:
> Hi,
>
> I am running a containerized zookeeper 3.6.1 version and it is getting
> filled with below WARN logs,
>
> [myid:2] - WARN [NIOWorkerThread-2:NIOServerCnxn@364] - Unexpected
> exception EndOfStreamException: Unable to read additional data from client,
> it probably closed the socket: address = /10.162.53.62:48988, session =
> 0x200866346830367 at
> org.apache.zookeeper.server.NIOServerCnxn.handleFailedRead(NIOServerCnxn.java:163)
> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:326)
> at
> org.apache.zookeeper.server.NIOServerCnxnFactory$IOWorkRequest.doWork(NIOServerCnxnFactory.java:522)
> at
> org.apache.zookeeper.server.WorkerService$ScheduledWorkRequest.run(WorkerService.java:154)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> [myid:2] - INFO [NIOWorkerThread-1:Learner@137] - Revalidating client:
> 0x3008663464f03ac [myid:2] - INFO [NIOWorkerThread-1:Learner@137] -
> Revalidating client: 0x100866346320382 INFO [NIOWorkerThread-1:Learner@137]
> - Revalidating client: 0x1008663463202d9 INFO [NIOWorkerThread-1:Learner@137]
> - Revalidating client: 0x1008663463203b2 INFO [NIOWorkerThread-1:Learner@137]
> - Revalidating client: 0x2008663468302d5 INFO [NIOWorkerThread-1:Learner@137]
> - Revalidating client: 0x4008663438f03ca INFO [NIOWorkerThread-1:Learner@137]
> - Revalidating client: 0x1008663463203bf and so on
>
> I have multiple instances of Apache Storm 1.22 running as containerized
> service which is running on 10.162.53.62 (IP address in the above WARN
> logs). The storm is using a 3.4.14 zookeeper version and the curator is at
> 4.0.1 (which is compatible with 3.5+ versions). Here is the pool file from
> the storm 1.1.x branch
> https://github.com/apache/storm/blob/1.1.x-branch/pom.xml for reference.
> Even the latest storm is using the same configs as far as the zookeeper is
> concerned.
>
> So, is something wrong with the compatibility of the storm's zookeeper
> version and mine? Or is anything missing from my zookeeper configs point of
> view? Any details on what might be happening would really be helpful.
>
> Here is my zoo.cfg,
>
> tickTime=2000
> # The number of ticks that the initial
> # synchronization phase can take
> initLimit=20
> # The number of ticks that can pass between
> # sending a request and getting an acknowledgement
> syncLimit=15
> # HA (multi-node) setup
> standaloneEnabled=false
> dynamicConfigFile=/opt/zookeeper/conf/zoo.cfg.dynamic
>
> # location of in-memory database snapshots
> dataDir=/data
> # location of the transaction-log
> dataLogDir=/data/xnlogs
>
> # enable regular purging of old data
> autopurge.purgeInterval=24
> autopurge.snapRetainCount=5
>
>