You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Chris Rogers <ch...@bodleian.ox.ac.uk> on 2016/08/22 12:20:33 UTC

SolrCloud: Collection creation failed due to timeout

Hi there,

I'm struggling with setting up a test implementation of SolrCloud using Vagrant.

I've got two vagrant boxes on a private network:

* One Ubuntu machine running Zookeeper
* One Ubuntu machine running Solr

My zookeeper config is as follows:


tickTime=4000

dataDir=/home/vagrant/zookeeper-data

clientPort=2181

server.1=localhost:2888:3888


Zookeeper is running as expected. I’m starting it in foreground mode with the command below:


bin/zkServer.sh start-foreground



I’m then starting Solr in cloud mode:


 bin/solr start -c -z 172.28.128.3:2181


At this point, all seems well. Nothing bad shows up in the Solr logs, and no alarming messages coming through in the console for Zookeeper.

I then try and create a collection with the command below:

  bin/solr create -c mycollection -d data_driven_schema_configs


It’s then that I start seeing lots of errors in the Solr logs, and lots of repetitive messages appearing in Zookeeper:

These two Solr errors over and over:

java.io.IOException: Xid out of order. Got Xid 1299 with err 0 expected Xid 1298 for a packet with details: clientPath:null serverPath:null finished:false header:: 1298,14  replyHeader:: 0,0,-4 request:: org.apache.zookeeper.MultiTransactionRecord@95acc4f3 response:: org.apache.zookeeper.MultiResponse@0

There was a problem trying to register as the leader:org.apache.solr.common.SolrException: Could not register as the leader because creating the ephemeral registration node in ZooKeeper failed


And this from Zookeeper:

[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251] - Accepted socket connection from /172.28.128.4:54548
2016-08-21 21:33:39,137 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@770] - Client attempting to renew session 0x156aefeba2d0004 at /172.28.128.4:54548
2016-08-21 21:33:39,145 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1580] - Established session 0x156aefeba2d0004 with negotiated timeout 15000 for client /172.28.128.4:54548
2016-08-21 21:33:39,147 - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@593] - Dropping packet at server of type 14
2016-08-21 21:33:39,154 - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@634] - EndOfStreamException: Unable to read additional data from client sessionid 0x156aefeba2d0004, likely client has closed socket
2016-08-21 21:33:39,155 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1435] - Closed socket connection for client /172.28.128.4:54548 which had sessionid 0x156aefeba2d0004
2016-08-21 21:33:40,698 - INFO


Finally, the collection creation fails with the following message:

ERROR: Failed to create collection 'mycollection' due to: create the collection time out:180s


I can't find any useful references to these errors in SO or in the docs. Can anyone help me diagnose the problem?

Many thanks,
Chris

Chris Rogers
Digital Projects Manager
Bodleian Digital Library Systems and Services
chris.rogers@bodleian.ox.ac.uk<ma...@bodleian.ox.ac.uk>

Re: SolrCloud: Collection creation failed due to timeout

Posted by Chris Rogers <ch...@bodleian.ox.ac.uk>.
Thanks Shawn, that was indeed the issue. Upgraded and everything is working as expected.

Much appreciated.
Chris

On 22/08/2016, 14:50, "Shawn Heisey" <ap...@elyograg.org> wrote:

    On 8/22/2016 7:27 AM, Chris Rogers wrote:
    > Solr version: 6.1.0 Zookeeper: 3.3.6 
    
    This is probably the root of the issue.
    
    Solr includes Zookeeper 3.4.6, for the client.  The server you're
    running is significantly older than the client version.  The WARN
    messages I quoted previously indicates that the server dropped a packet
    type (number 14) that it didn't understand, which seems to indicate that
    the client code in Solr is trying to use functionality that it knows
    about, but is not provided by the 3.3.6 server version.
    
    You should upgrade your zookeeper server, probably to the latest
    version, which is 3.4.8.  Don't use 3.4.7 if you manage to find it -- it
    has an extremely serious bug, so it was removed from the mirror system.
    
    Thanks,
    Shawn
    
    


Re: SolrCloud: Collection creation failed due to timeout

Posted by Shawn Heisey <ap...@elyograg.org>.
On 8/22/2016 7:27 AM, Chris Rogers wrote:
> Solr version: 6.1.0 Zookeeper: 3.3.6 

This is probably the root of the issue.

Solr includes Zookeeper 3.4.6, for the client.  The server you're
running is significantly older than the client version.  The WARN
messages I quoted previously indicates that the server dropped a packet
type (number 14) that it didn't understand, which seems to indicate that
the client code in Solr is trying to use functionality that it knows
about, but is not provided by the 3.3.6 server version.

You should upgrade your zookeeper server, probably to the latest
version, which is 3.4.8.  Don't use 3.4.7 if you manage to find it -- it
has an extremely serious bug, so it was removed from the mirror system.

Thanks,
Shawn


Re: SolrCloud: Collection creation failed due to timeout

Posted by Chris Rogers <ch...@bodleian.ox.ac.uk>.
Hi Shawn,

Thanks for replying.

Solr version: 6.1.0
Zookeeper: 3.3.6

Solr log errors below are from around the 21:33:39 timestamp:


2016-08-21 21:33:37.135 WARN  (main-SendThread(172.28.128.3:2181)) [   ] o.a.z.ClientCnxn Session 0x156aefeba2d0004 for server 172.28.128.3/172.28.128.3:2181, unexpected error, closing socket connection and attempting reconnect
java.io.IOException: Xid out of order. Got Xid 1438 with err 0 expected Xid 1437 for a packet with details: clientPath:null serverPath:null finished:false header:: 1437,14  replyHeader:: 0,0,-4  request:: org.apache.zookeeper.MultiTransactionRecord@95ad06de response:: org.apache.zookeeper.MultiResponse@0
	at org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:798)
	at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:94)
	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366)
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
2016-08-21 21:33:37.144 ERROR (recoveryExecutor-3-thread-1-processing-n:10.0.2.15:8983_solr x:mycollection_shard1_replica1 s:shard1 c:mycollection r:core_node1) [c:mycollection s:shard1 r:core_node1 x:mycollection_shard1_replica1] o.a.s.c.RecoveryStrategy Error while trying to recover. core=mycollection_shard1_replica1:org.apache.solr.common.SolrException: No registered leader was found after waiting for 4000ms , collection: mycollection slice: shard1
	at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:718)
	at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:704)
	at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:305)
	at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:221)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$22(ExecutorUtil.java:229)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

2016-08-21 21:33:37.144 INFO  (recoveryExecutor-3-thread-1-processing-n:10.0.2.15:8983_solr x:mycollection_shard1_replica1 s:shard1 c:mycollection r:core_node1) [c:mycollection s:shard1 r:core_node1 x:mycollection_shard1_replica1] o.a.s.c.RecoveryStrategy Replay not started, or was not successful... still buffering updates.
2016-08-21 21:33:37.144 INFO  (recoveryExecutor-3-thread-1-processing-n:10.0.2.15:8983_solr x:mycollection_shard1_replica1 s:shard1 c:mycollection r:core_node1) [c:mycollection s:shard1 r:core_node1 x:mycollection_shard1_replica1] o.a.s.c.RecoveryStrategy RecoveryStrategy has been closed
2016-08-21 21:33:37.149 INFO  (recoveryExecutor-3-thread-1-processing-n:10.0.2.15:8983_solr x:mycollection_shard1_replica1 s:shard1 c:mycollection r:core_node1) [c:mycollection s:shard1 r:core_node1 x:mycollection_shard1_replica1] o.a.s.c.RecoveryStrategy Finished recovery process, successful=[false]
2016-08-21 21:33:37.237 INFO  (zkCallback-4-thread-168-processing-n:10.0.2.15:8983_solr) [   ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7d41da47 name:ZooKeeperConnection Watcher:172.28.128.3:2181 got event WatchedEvent state:Disconnected type:None path:null path:null type:None
2016-08-21 21:33:37.237 INFO  (zkCallback-4-thread-168-processing-n:10.0.2.15:8983_solr) [   ] o.a.s.c.c.ConnectionManager zkClient has disconnected
2016-08-21 21:33:39.062 WARN  (main-SendThread(172.28.128.3:2181)) [   ] o.a.z.ClientCnxnSocket Connected to an old server; r-o mode will be unavailable
2016-08-21 21:33:39.063 INFO  (zkCallback-4-thread-171-processing-n:10.0.2.15:8983_solr) [   ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7d41da47 name:ZooKeeperConnection Watcher:172.28.128.3:2181 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
2016-08-21 21:33:39.071 WARN  (main-SendThread(172.28.128.3:2181)) [   ] o.a.z.ClientCnxn Session 0x156aefeba2d0004 for server 172.28.128.3/172.28.128.3:2181, unexpected error, closing socket connection and attempting reconnect
java.io.IOException: Xid out of order. Got Xid 1441 with err 0 expected Xid 1440 for a packet with details: clientPath:null serverPath:null finished:false header:: 1440,14  replyHeader:: 0,0,-4  request:: org.apache.zookeeper.MultiTransactionRecord@95ad06de response:: org.apache.zookeeper.MultiResponse@0
	at org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:798)
	at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:94)
	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366)
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
2016-08-21 21:33:39.174 INFO  (zkCallback-4-thread-168-processing-n:10.0.2.15:8983_solr) [   ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7d41da47 name:ZooKeeperConnection Watcher:172.28.128.3:2181 got event WatchedEvent state:Disconnected type:None path:null path:null type:None
2016-08-21 21:33:39.175 INFO  (zkCallback-4-thread-168-processing-n:10.0.2.15:8983_solr) [   ] o.a.s.c.c.ConnectionManager zkClient has disconnected
2016-08-21 21:33:40.023 INFO  (qtp110456297-14) [   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/logging params={wt=json&_=1471815019974&since=0} status=0 QTime=0
2016-08-21 21:33:40.622 WARN  (main-SendThread(172.28.128.3:2181)) [   ] o.a.z.ClientCnxnSocket Connected to an old server; r-o mode will be unavailable
2016-08-21 21:33:40.623 INFO  (zkCallback-4-thread-171-processing-n:10.0.2.15:8983_solr) [   ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7d41da47 name:ZooKeeperConnection Watcher:172.28.128.3:2181 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
2016-08-21 21:33:42.177 WARN  (main-SendThread(172.28.128.3:2181)) [   ] o.a.z.ClientCnxn Session 0x156aefeba2d0004 for server 172.28.128.3/172.28.128.3:2181, unexpected error, closing socket connection and attempting reconnect
java.io.IOException: Xid out of order. Got Xid 1444 with err 0 expected Xid 1443 for a packet with details: clientPath:null serverPath:null finished:false header:: 1443,14  replyHeader:: 0,0,-4  request:: org.apache.zookeeper.MultiTransactionRecord@95ad06de response:: org.apache.zookeeper.MultiResponse@0
	at org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:798)
	at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:94)
	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366)
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
2016-08-21 21:33:42.280 INFO  (zkCallback-4-thread-168-processing-n:10.0.2.15:8983_solr) [   ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7d41da47 name:ZooKeeperConnection Watcher:172.28.128.3:2181 got event WatchedEvent state:Disconnected type:None path:null path:null type:None
2016-08-21 21:33:42.280 INFO  (zkCallback-4-thread-168-processing-n:10.0.2.15:8983_solr) [   ] o.a.s.c.c.ConnectionManager zkClient has disconnected
2016-08-21 21:33:43.911 WARN  (main-SendThread(172.28.128.3:2181)) [   ] o.a.z.ClientCnxnSocket Connected to an old server; r-o mode will be unavailable
2016-08-21 21:33:43.912 INFO  (zkCallback-4-thread-171-processing-n:10.0.2.15:8983_solr) [   ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7d41da47 name:ZooKeeperConnection Watcher:172.28.128.3:2181 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
2016-08-21 21:33:46.782 WARN  (main-SendThread(172.28.128.3:2181)) [   ] o.a.z.ClientCnxn Session 0x156aefeba2d0004 for server 172.28.128.3/172.28.128.3:2181, unexpected error, closing socket connection and attempting reconnect
java.io.IOException: Xid out of order. Got Xid 1447 with err 0 expected Xid 1446 for a packet with details: clientPath:null serverPath:null finished:false header:: 1446,14  replyHeader:: 0,0,-4  request:: org.apache.zookeeper.MultiTransactionRecord@95ad06de response:: org.apache.zookeeper.MultiResponse@0
	at org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:798)
	at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:94)
	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366)
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
2016-08-21 21:33:46.885 INFO  (zkCallback-4-thread-168-processing-n:10.0.2.15:8983_solr) [   ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7d41da47 name:ZooKeeperConnection Watcher:172.28.128.3:2181 got event WatchedEvent state:Disconnected type:None path:null path:null type:None
2016-08-21 21:33:46.885 INFO  (zkCallback-4-thread-168-processing-n:10.0.2.15:8983_solr) [   ] o.a.s.c.c.ConnectionManager zkClient has disconnected
2016-08-21 21:33:48.760 WARN  (main-SendThread(172.28.128.3:2181)) [   ] o.a.z.ClientCnxnSocket Connected to an old server; r-o mode will be unavailable
2016-08-21 21:33:48.761 INFO  (zkCallback-4-thread-171-processing-n:10.0.2.15:8983_solr) [   ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7d41da47 name:ZooKeeperConnection Watcher:172.28.128.3:2181 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
2016-08-21 21:33:50.027 INFO  (qtp110456297-16) [   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/logging params={wt=json&_=1471815019974&since=0} status=0 QTime=0
2016-08-21 21:33:52.887 WARN  (main-SendThread(172.28.128.3:2181)) [   ] o.a.z.ClientCnxn Session 0x156aefeba2d0004 for server 172.28.128.3/172.28.128.3:2181, unexpected error, closing socket connection and attempting reconnect
java.io.IOException: Xid out of order. Got Xid 1450 with err 0 expected Xid 1449 for a packet with details: clientPath:null serverPath:null finished:false header:: 1449,14  replyHeader:: 0,0,-4  request:: org.apache.zookeeper.MultiTransactionRecord@95ad06de response:: org.apache.zookeeper.MultiResponse@0
	at org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:798)
	at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:94)
	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366)
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
2016-08-21 21:33:52.990 INFO  (zkCallback-4-thread-168-processing-n:10.0.2.15:8983_solr) [   ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7d41da47 name:ZooKeeperConnection Watcher:172.28.128.3:2181 got event WatchedEvent state:Disconnected type:None path:null path:null type:None
2016-08-21 21:33:52.990 INFO  (zkCallback-4-thread-168-processing-n:10.0.2.15:8983_solr) [   ] o.a.s.c.c.ConnectionManager zkClient has disconnected
2016-08-21 21:33:54.718 WARN  (main-SendThread(172.28.128.3:2181)) [   ] o.a.z.ClientCnxnSocket Connected to an old server; r-o mode will be unavailable
2016-08-21 21:33:54.719 INFO  (zkCallback-4-thread-171-processing-n:10.0.2.15:8983_solr) [   ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7d41da47 name:ZooKeeperConnection Watcher:172.28.128.3:2181 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
2016-08-21 21:34:00.028 INFO  (qtp110456297-220) [   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/logging params={wt=json&_=1471815019974&since=0} status=0 QTime=0
2016-08-21 21:34:00.493 ERROR (qtp110456297-11) [c:mycollection s:shard1 r:core_node1 x:mycollection_shard1_replica1] o.a.s.c.ShardLeaderElectionContext There was a problem trying to register as the leader:org.apache.solr.common.SolrException: Could not register as the leader because creating the ephemeral registration node in ZooKeeper failed
	at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:218)
	at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:417)
	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)
	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)
	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)
	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:216)
	at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:670)
	at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:441)
	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)
	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)
	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)
	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:216)
	at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:670)
	at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:441)
	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)
	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)
	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)
	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:216)
	at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:670)
	at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:441)
	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)
	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)
	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)
	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:216)
	at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:670)
	at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:441)
	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)
	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)
	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)
	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:216)
	at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:670)
	at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:441)
	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)
	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)
	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)
	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:216)
	at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:670)
	at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:441)
	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)
	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)
	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)
	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:216)
	at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:670)
	at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:441)
	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)
	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)
	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)
	at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1040)
	at org.apache.solr.cloud.ZkController.register(ZkController.java:851)
	at org.apache.solr.cloud.ZkController.register(ZkController.java:806)
	at org.apache.solr.core.ZkContainer$2.run(ZkContainer.java:183)
	at org.apache.solr.core.ZkContainer.registerInZk(ZkContainer.java:212)
	at org.apache.solr.core.CoreContainer.registerCore(CoreContainer.java:695)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:819)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:749)
	at org.apache.solr.handler.admin.CoreAdminOperation$1.call(CoreAdminOperation.java:119)
	at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:367)
	at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:158)
	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:156)
	at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:663)
	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:445)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
	at org.eclipse.jetty.server.Server.handle(Server.java:518)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
	at org.apache.zookeeper.ZooKeeper.multiInternal(ZooKeeper.java:935)
	at org.apache.zookeeper.ZooKeeper.multi(ZooKeeper.java:915)
	at org.apache.solr.common.cloud.SolrZkClient$11.execute(SolrZkClient.java:572)
	at org.apache.solr.common.cloud.SolrZkClient$11.execute(SolrZkClient.java:569)
	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
	at org.apache.solr.common.cloud.SolrZkClient.multi(SolrZkClient.java:569)
	at org.apache.solr.cloud.ShardLeaderElectionContextBase$1.execute(ElectionContext.java:201)
	at org.apache.solr.common.util.RetryUtil.retryOnThrowable(RetryUtil.java:49)
	at org.apache.solr.common.util.RetryUtil.retryOnThrowable(RetryUtil.java:42)
	at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:183)
	... 86 more

2016-08-21 21:34:00.493 INFO  (qtp110456297-11) [c:mycollection s:shard1 r:core_node1 x:mycollection_shard1_replica1] o.a.s.c.ShardLeaderElectionContext There may be a better leader candidate than us - going back into recovery
2016-08-21 21:34:00.494 INFO  (qtp110456297-11) [c:mycollection s:shard1 r:core_node1 x:mycollection_shard1_replica1] o.a.s.c.ElectionContext Canceling election /collections/mycollection/leader_elect/shard1/election/96456851103481860-core_node1-n_0000000014
2016-08-21 21:34:00.494 WARN  (main-SendThread(172.28.128.3:2181)) [   ] o.a.z.ClientCnxn Session 0x156aefeba2d0004 for server 172.28.128.3/172.28.128.3:2181, unexpected error, closing socket connection and attempting reconnect

Thanks again,
Chris


On 22/08/2016, 14:11, "Shawn Heisey" <ap...@elyograg.org> wrote:

    On 8/22/2016 6:20 AM, Chris Rogers wrote:
    > It’s then that I start seeing lots of errors in the Solr logs, and lots of repetitive messages appearing in Zookeeper:
    >
    > These two Solr errors over and over:
    >
    > java.io.IOException: Xid out of order. Got Xid 1299 with err 0 expected Xid 1298 for a packet with details: clientPath:null serverPath:null finished:false header:: 1298,14  replyHeader:: 0,0,-4 request:: org.apache.zookeeper.MultiTransactionRecord@95acc4f3 response:: org.apache.zookeeper.MultiResponse@0
    
    That appears to be one log message, but you said there were two.  Also,
    this message is incomplete.  It is missing the timestamp at the
    beginning and appears to have been cut off at the end too.  I think the
    message probably had *many* more lines of output that weren't included.
    
    > And this from Zookeeper:
    <snip>
    > 2016-08-21 21:33:39,147 - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@593] - Dropping packet at server of type 14
    > 2016-08-21 21:33:39,154 - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@634] - EndOfStreamException: Unable to read additional data from client sessionid 0x156aefeba2d0004, likely client has closed socket
    
    That seems to be saying that Solr closed the connection to zookeeper.  I
    have no idea what might be wrong, based just on what's been provided
    here.  This section of logging seems to contain everything related to
    the specific connection from port 54548, and if that's true, then it
    does not appear to have been a timeout.
    
    Is there anything in the solr.log file at the timestamp at or near
    21:33:39.154(when zookeeper thought the connection was closed)?
    
    What version of Solr?  What version of zookeeper did you install on the
    other node?
    
    Thanks,
    Shawn
    
    


Re: SolrCloud: Collection creation failed due to timeout

Posted by Shawn Heisey <ap...@elyograg.org>.
On 8/22/2016 6:20 AM, Chris Rogers wrote:
> It\u2019s then that I start seeing lots of errors in the Solr logs, and lots of repetitive messages appearing in Zookeeper:
>
> These two Solr errors over and over:
>
> java.io.IOException: Xid out of order. Got Xid 1299 with err 0 expected Xid 1298 for a packet with details: clientPath:null serverPath:null finished:false header:: 1298,14  replyHeader:: 0,0,-4 request:: org.apache.zookeeper.MultiTransactionRecord@95acc4f3 response:: org.apache.zookeeper.MultiResponse@0

That appears to be one log message, but you said there were two.  Also,
this message is incomplete.  It is missing the timestamp at the
beginning and appears to have been cut off at the end too.  I think the
message probably had *many* more lines of output that weren't included.

> And this from Zookeeper:
<snip>
> 2016-08-21 21:33:39,147 - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@593] - Dropping packet at server of type 14
> 2016-08-21 21:33:39,154 - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@634] - EndOfStreamException: Unable to read additional data from client sessionid 0x156aefeba2d0004, likely client has closed socket

That seems to be saying that Solr closed the connection to zookeeper.  I
have no idea what might be wrong, based just on what's been provided
here.  This section of logging seems to contain everything related to
the specific connection from port 54548, and if that's true, then it
does not appear to have been a timeout.

Is there anything in the solr.log file at the timestamp at or near
21:33:39.154(when zookeeper thought the connection was closed)?

What version of Solr?  What version of zookeeper did you install on the
other node?

Thanks,
Shawn