Issue on server restarts with Solr 4.6.0 Cloud

[andrew jenner <g1...@gmail.com>]

Using Solr 4.6.0 on linux with Java 6 (Oracle JRockit 1.6.0_75
R28.3.2-14-160877-1.6.0_75)


We are seeing these issues when doing a restart on a Solr cloud
configuration.After restarting each server in sequence none of them
will come up. The servers start up after a long time but the cloud
status shows the Solr as being down.


java.nio.channels.ClosedChannelException
	at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:87)
	at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:603)
	at org.apache.solr.update.ChannelFastInputStream.readWrappedStream(TransactionLog.java:778)
	at org.apache.solr.common.util.FastInputStream.refill(FastInputStream.java:89)
	at org.apache.solr.common.util.FastInputStream.readUnsignedByte(FastInputStream.java:71)
	at org.apache.solr.common.util.FastInputStream.readInt(FastInputStream.java:216)
	at org.apache.solr.update.TransactionLog$FSReverseReader.<init>(TransactionLog.java:696)
	at org.apache.solr.update.TransactionLog.getReverseReader(TransactionLog.java:575)
	at org.apache.solr.update.UpdateLog$RecentUpdates.update(UpdateLog.java:942)
	at org.apache.solr.update.UpdateLog$RecentUpdates.access$000(UpdateLog.java:885)
	at org.apache.solr.update.UpdateLog.getRecentUpdates(UpdateLog.java:1043)
	at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:280)
	at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:244)


SnapPull failed :org.apache.lucene.store.AlreadyClosedException: Already closed
at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:340)
at org.apache.solr.handler.ReplicationHandler.loadReplicationProperties(ReplicationHandler.java:811)
at org.apache.solr.handler.SnapPuller.logReplicationTimeAndConfFiles(SnapPuller.java:564)
at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:506)
at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:322)
at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:156)
at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:433)
at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:244)


Error while trying to recover.
core=[REDACTED]:org.apache.solr.common.SolrException: No registered
leader was found, collection:[REDACTED] slice:shard1
	at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:484)
	at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:467)
	at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:342)
	at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:244)