Re: Error opening new searcher due to LockObtainFailedException

[Erick Erickson <er...@gmail.com>]

Hmmm. oddly another poster was seeing this due to permissions issues,
although I don't know why that would clear up after a while. But it's
something to check.

Erick

On Wed, Aug 30, 2017 at 3:24 PM, Sundeep T <su...@gmail.com> wrote:
> Hello,
>
> Occasionally we are seeing errors opening new searcher for certain solr
> cores. Whenever this happens, we are unable to query or ingest new data
> into these cores. It seems to clear up after some time though. The root
> cause seems to be - *"org.apache.lucene.store.LockObtainFailedException:
> Lock held by this virtual machine:
> /opt/solr/volumes/data9/7d50b38e114af075-core-24/data/index/write.lock"*
>
> Below is the full stack trace. Any ideas on what could be going on that
> causes such an exception and how to mitigate this? thanks a lot for your
> help!
>
> Unable to create core
> [7d50b38e114af075-core-24],trace=org.apache.solr.common.SolrException:
> Unable to create core [7d50b38e114af075-core-24]
> at org.apache.solr.core.CoreContainer.create(CoreContainer.java:903)
> at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:1167)
> at org.apache.solr.servlet.HttpSolrCall.init(HttpSolrCall.java:252)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:418)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:345)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:296)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> at org.eclipse.jetty.server.Server.handle(Server.java:534)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
> at
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
> at
> org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.solr.common.SolrException: Error opening new searcher
> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:952)
> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:816)
> at org.apache.solr.core.CoreContainer.create(CoreContainer.java:890)
> ... 30 more
> Caused by: org.apache.solr.common.SolrException: Error opening new searcher
> at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1891)
> at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:2011)
> at org.apache.solr.core.SolrCore.initSearcher(SolrCore.java:1041)
> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:925)
> ... 32 more
> Caused by: org.apache.lucene.store.LockObtainFailedException: Lock held by
> this virtual machine:
> /opt/solr/volumes/data9/7d50b38e114af075-core-24/data/index/write.lock
> at
> org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:127)
> at org.apache.lucene.store.FSLockFactory.obtainLock(FSLockFactory.java:41)
> at org.apache.lucene.store.BaseDirectory.obtainLock(BaseDirectory.java:45)
> at
> org.apache.lucene.store.FilterDirectory.obtainLock(FilterDirectory.java:104)
> at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:804)
> at org.apache.solr.update.SolrIndexWriter.<init>(SolrIndexWriter.java:125)
> at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:100)
> at
> org.apache.solr.update.DefaultSolrCoreState.createMainIndexWriter(DefaultSolrCoreState.java:240)
> at
> org.apache.solr.update.DefaultSolrCoreState.getIndexWriter(DefaultSolrCoreState.java:114)
> at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1852)
> ... 35 more
> ,code=500}```