You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Erick Erickson (JIRA)" <ji...@apache.org> on 2016/12/06 16:16:58 UTC
[jira] [Resolved] (SOLR-9829) Solr cannot provide index service
after a large GC pause but core state in ZK is still active
[ https://issues.apache.org/jira/browse/SOLR-9829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Erick Erickson resolved SOLR-9829.
----------------------------------
Resolution: Invalid
Please bring things like this up on the user's list before opening a JIRA.
1> You have to look both at the live_nodes list as well as the state.json to know if a node is really up or not.
2> The error you're asking about has been seen many times as something that gets reported that isn't actually the root cause, you _must_ look in the Solr logs.
> Solr cannot provide index service after a large GC pause but core state in ZK is still active
> ---------------------------------------------------------------------------------------------
>
> Key: SOLR-9829
> URL: https://issues.apache.org/jira/browse/SOLR-9829
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: update
> Affects Versions: 5.3.2
> Environment: Redhat enterprise server 64bit
> Reporter: Forest Soup
>
> When Solr meets a large GC pause like https://issues.apache.org/jira/browse/SOLR-9828 , the collections on it cannot provide service and never come back until restart.
> But in the ZooKeeper, the cores on that server still shows active.
> Some /update requests got http 500 due to "IndexWriter is closed". Some gots http 400 due to "possible analysis error." whose root cause is still "IndexWriter is closed", which we think it should return 500 instead(documented in https://issues.apache.org/jira/browse/SOLR-9825).
> Our questions in this JIRA are:
> 1, should solr mark cores as down in zk when it cannot provide index service?
> 2, Is it possible solr re-open the IndexWriter to provide index service again?
> solr log snippets:
> 2016-11-22 20:47:37.274 ERROR (qtp2011912080-76) [c:collection12 s:shard1 r:core_node1 x:collection12_shard1_replica1] o.a.s.c.SolrCore org.apache.solr.common.SolrException: Exception writing document id Q049dXMxYjMtbWFpbDg4L089bGxuX3VzMQ==20841350!270CE4F9C032EC26002580730061473C to the index; possible analysis error.
> at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:167)
> at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:69)
> at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:51)
> at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:955)
> at org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1110)
> at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:706)
> at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:51)
> at org.apache.solr.update.processor.LanguageIdentifierUpdateProcessor.processAdd(LanguageIdentifierUpdateProcessor.java:207)
> at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:51)
> at org.apache.solr.update.processor.CloneFieldUpdateProcessorFactory$1.processAdd(CloneFieldUpdateProcessorFactory.java:231)
> at org.apache.solr.handler.loader.JsonLoader$SingleThreadedJsonLoader.processUpdate(JsonLoader.java:143)
> at org.apache.solr.handler.loader.JsonLoader$SingleThreadedJsonLoader.load(JsonLoader.java:113)
> at org.apache.solr.handler.loader.JsonLoader.load(JsonLoader.java:76)
> at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:98)
> at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2068)
> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:672)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:463)
> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:235)
> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:199)
> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
> at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
> at org.eclipse.jetty.server.Server.handle(Server.java:499)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
> at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
> at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:719)
> at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:733)
> at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1471)
> at org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:239)
> at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:163)
> ... 40 more
> Caused by: java.nio.channels.ClosedByInterruptException
> at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
> at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:315)
> at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:170)
> at org.apache.lucene.store.LockValidatingDirectoryWrapper.createOutput(LockValidatingDirectoryWrapper.java:43)
> at org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:43)
> at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.<init>(BlockTreeTermsWriter.java:328)
> at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.<init>(BlockTreeTermsWriter.java:280)
> at org.apache.lucene.codecs.lucene50.Lucene50PostingsFormat.fieldsConsumer(Lucene50PostingsFormat.java:428)
> at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.write(PerFieldPostingsFormat.java:196)
> at org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:107)
> at org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:112)
> at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:422)
> at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:503)
> at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:615)
> at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:423)
> at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:273)
> at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:203)
> at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1581)
> at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1773)
> at org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:609)
> at org.apache.solr.update.CommitTracker.run(CommitTracker.java:216)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> ... 1 more
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org