this IndexWriter is closed

[Jay Potharaju <js...@gmail.com>]

Hi,
I am getting Indexwriter is closed error only on some of my shards in the
collection. This seems to be happening on leader shards only. There is are
other shards on the box and they are not throwing any error. Also there is
enough disc space on the box available at this time.

Solr: 5.3.0.

Any recommendations on how to address this issue??

null:org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
	at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:719)
	at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:733)
	at org.apache.lucene.index.IndexWriter.deleteDocuments(IndexWriter.java:1438)
	at org.apache.solr.update.DirectUpdateHandler2.deleteByQuery(DirectUpdateHandler2.java:408)
	at org.apache.solr.update.processor.RunUpdateProcessor.processDelete(RunUpdateProcessorFactory.java:80)
	at org.apache.solr.update.processor.UpdateRequestProcessor.processDelete(UpdateRequestProcessor.java:55)
	at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalDelete(DistributedUpdateProcessor.java:960)
	at org.apache.solr.update.processor.DistributedUpdateProcessor.doDeleteByQuery(DistributedUpdateProcessor.java:1360)
	at org.apache.solr.update.processor.DistributedUpdateProcessor.processDelete(DistributedUpdateProcessor.java:1154)
	at org.apache.solr.handler.loader.JavabinLoader.delete(JavabinLoader.java:163)
	at org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoader.java:116)
	at org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:58)
	at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:98)
	at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
	at org.apache.solr.core.SolrCore.execute(SolrCore.java:2068)
	at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:669)
	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:462)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:210)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:179)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
	at org.eclipse.jetty.server.Server.handle(Server.java:499)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
	at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
	at java.lang.Thread.run(Unknown Source)
Caused by: java.io.IOException: No space left on device
	at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
	at sun.nio.ch.FileDispatcherImpl.write(Unknown Source)
	at sun.nio.ch.IOUtil.writeFromNativeBuffer(Unknown Source)
	at sun.nio.ch.IOUtil.write(Unknown Source)
	at sun.nio.ch.FileChannelImpl.write(Unknown Source)
	at java.nio.channels.Channels.writeFullyImpl(Unknown Source)
	at java.nio.channels.Channels.writeFully(Unknown Source)
	at java.nio.channels.Channels.access$000(Unknown Source)
	at java.nio.channels.Channels$1.write(Unknown Source)
	at org.apache.lucene.store.FSDirectory$FSIndexOutput$1.write(FSDirectory.java:271)
	at java.util.zip.CheckedOutputStream.write(Unknown Source)
	at java.io.BufferedOutputStream.flushBuffer(Unknown Source)
	at java.io.BufferedOutputStream.write(Unknown Source)
	at org.apache.lucene.store.OutputStreamIndexOutput.writeByte(OutputStreamIndexOutput.java:47)
	at org.apache.lucene.store.RateLimitedIndexOutput.writeByte(RateLimitedIndexOutput.java:66)
	at org.apache.lucene.codecs.lucene53.Lucene53NormsConsumer.addByte1(Lucene53NormsConsumer.java:101)
	at org.apache.lucene.codecs.lucene53.Lucene53NormsConsumer.addNormsField(Lucene53NormsConsumer.java:81)
	at org.apache.lucene.codecs.NormsConsumer.mergeNormsField(NormsConsumer.java:110)
	at org.apache.lucene.codecs.NormsConsumer.merge(NormsConsumer.java:96)
	at org.apache.lucene.index.SegmentMerger.mergeNorms(SegmentMerger.java:156)
	at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:116)
	at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4089)
	at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3664)
	at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:588)
	at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:626)
	Suppressed: java.io.IOException: No space left on device
		at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
		at sun.nio.ch.FileDispatcherImpl.write(Unknown Source)
		at sun.nio.ch.IOUtil.writeFromNativeBuffer(Unknown Source)
		at sun.nio.ch.IOUtil.write(Unknown Source)
		at sun.nio.ch.FileChannelImpl.write(Unknown Source)
		at java.nio.channels.Channels.writeFullyImpl(Unknown Source)
		at java.nio.channels.Channels.writeFully(Unknown Source)
		at java.nio.channels.Channels.access$000(Unknown Source)
		at java.nio.channels.Channels$1.write(Unknown Source)
		at org.apache.lucene.store.FSDirectory$FSIndexOutput$1.write(FSDirectory.java:271)
		at java.util.zip.CheckedOutputStream.write(Unknown Source)
		at java.io.BufferedOutputStream.flushBuffer(Unknown Source)
		at java.io.BufferedOutputStream.flush(Unknown Source)
		at org.apache.lucene.store.OutputStreamIndexOutput.getChecksum(OutputStreamIndexOutput.java:80)
		at org.apache.lucene.store.RateLimitedIndexOutput.getChecksum(RateLimitedIndexOutput.java:59)
		at org.apache.lucene.codecs.CodecUtil.writeCRC(CodecUtil.java:471)
		at org.apache.lucene.codecs.CodecUtil.writeFooter(CodecUtil.java:310)
		at org.apache.lucene.codecs.lucene53.Lucene53NormsConsumer.close(Lucene53NormsConsumer.java:138)
		at org.apache.lucene.index.SegmentMerger.mergeNorms(SegmentMerger.java:157)


Thanks
Jay

Re: this IndexWriter is closed

[Shawn Heisey <ap...@elyograg.org>]

On 4/9/2018 12:31 PM, Jay Potharaju wrote:
> I am getting Indexwriter is closed error only on some of my shards in the
> collection. This seems to be happening on leader shards only. There is are
> other shards on the box and they are not throwing any error. Also there is
> enough disc space on the box available at this time.
<snip>
> Caused by: java.io.IOException: No space left on device

Lucene (used by Solr) is reporting information given to it by Java,
which got its information from the operating system.  The OS said that
there's no space left.

There are a few possibilities here that I can think of:

1) The user that is running the application has a quota on the storage
and has reached that quota, so that specific user is being told there's
no space, while another user can see lots of space.

2) You've actually run out of some other resource besides literal
storage.  One example is that the storage volume has reached the maximum
number of inodes that it can store.  I don't know if this can result in
the same error message, but  it wouldn't surprise me.

http://blog.scoutapp.com/articles/2014/10/08/understanding-disk-inodes

3) You're looking at the wrong storage volume to see the free space.

Thanks,
Shawn