Too many logs recorded in zookeeper.out

[Noriyuki TAKEI <nt...@sios.com>]

Hi.All.

I use Solr Cloud with 3 Zoo Keepers and 2 Solr Servers,
having 3 shards and 2 replicas.

These servers are running as virtual machine on VMWare and
virtual machines are stored in the iSCSI storage attached to VMWare.

One day,iSCSI storage failure suddenly occurred and then 1 Solr Server and
2 Zoo Keepers were inaccessible via SSH.But indexing and searching
seemed to work properly.

In order to recover, I powered down and started up virtual machines
inaccessible via SSH.
For a few minutes after Zoo Keeper starting up,too many logs as below were
recorded in
zookeeper.out

　　ERROR[LearnerHandler-/XXX.XXX.XXX.XXX:36524:LearnerHandler@631]
　　Unexpected exception causing shutdown while sock still open
 
　　java.io.EOFEception
　　　at java.io.DataInputStream.readInt(DataInputStream.java:392)
　　　at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)


The logs shown above were recorded in zookeeper.out once per 3 millisecond.

Why were too many logs recorded?




--
View this message in context: http://lucene.472066.n3.nabble.com/Too-many-logs-recorded-in-zookeeper-out-tp4335238.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Too many logs recorded in zookeeper.out

[Noriyuki TAKEI <nt...@sios.com>]

Hi

Tahnks for your reply.I,ll try to join zookeeper mailg list!! 




--
View this message in context: http://lucene.472066.n3.nabble.com/Too-many-logs-recorded-in-zookeeper-out-tp4335238p4336914.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Too many logs recorded in zookeeper.out

[Shawn Heisey <ap...@elyograg.org>]

On 5/16/2017 3:07 AM, Noriyuki TAKEI wrote:
> I use Solr Cloud with 3 Zoo Keepers and 2 Solr Servers,
> having 3 shards and 2 replicas.
>
> These servers are running as virtual machine on VMWare and
> virtual machines are stored in the iSCSI storage attached to VMWare.
>
> One day,iSCSI storage failure suddenly occurred and then 1 Solr Server and
> 2 Zoo Keepers were inaccessible via SSH.But indexing and searching
> seemed to work properly.

Indexing should not have worked with the loss of two ZK servers.  With
three total ZK servers, you must have at least two of them operational
to maintain quorum.  When quorum is lost, Solr will go read-only.

> In order to recover, I powered down and started up virtual machines
> inaccessible via SSH. For a few minutes after Zoo Keeper starting up,too many logs as below were recorded in zookeeper.out

SolrCloud uses zookeeper, but it's a completely separate software
project, and we aren't experts on it.  The problem you're experiencing
looks like it might require expert help.  You're going to need to go to
the zookeeper mailing list, or one of their other support avenues like
their IRC channel.  We can try to help you with the Solr side, and I
have a question about that.

You'll need to have ZK experts confirm this, but usually EOFException
with a TCP-based protocol (like the one that zookeeper uses) means that
the other side (client in this case) disconnected the TCP connection
before the side logging the exception (server) had sent its response. 
Solr is the client here.  What's in the Solr logfile?

Thanks,
Shawn