You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Shashank Pedamallu <sp...@vmware.com> on 2017/09/22 01:19:00 UTC
Error Opening new IndexSearcher - LockObtainFailedException
Hi,
I’m seeing the following exception in Solr that gets automatically resolved eventually.
2017-09-22 00:18:17.243 ERROR (qtp1702660825-17) [ x:spedamallu1-core-1] o.a.s.c.CoreContainer Error creating core [spedamallu1-core-1]: Error opening new searcher
org.apache.solr.common.SolrException: Error opening new searcher
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:952)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:816)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:890)
at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:1167)
at org.apache.solr.servlet.HttpSolrCall.init(HttpSolrCall.java:252)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:418)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:345)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:296)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:534)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.solr.common.SolrException: Error opening new searcher
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1891)
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:2011)
at org.apache.solr.core.SolrCore.initSearcher(SolrCore.java:1041)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:925)
... 32 more
Caused by: org.apache.lucene.store.LockObtainFailedException: Lock held by this virtual machine: /Users/spedamallu/Desktop/mount-1/spedamallu1-core-1/data/index/write.lock
at org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:127)
at org.apache.lucene.store.FSLockFactory.obtainLock(FSLockFactory.java:41)
at org.apache.lucene.store.BaseDirectory.obtainLock(BaseDirectory.java:45)
at org.apache.lucene.store.FilterDirectory.obtainLock(FilterDirectory.java:104)
at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:804)
at org.apache.solr.update.SolrIndexWriter.<init>(SolrIndexWriter.java:125)
at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:100)
at org.apache.solr.update.DefaultSolrCoreState.createMainIndexWriter(DefaultSolrCoreState.java:240)
at org.apache.solr.update.DefaultSolrCoreState.getIndexWriter(DefaultSolrCoreState.java:114)
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1852)
I kind of have a theory of why this is happening. Can someone please confirm if this is indeed the case.
* I was trying to run a long running operation on a transient core and it was getting evicted because of LRU
* So, to ensure my operation completion, I have added a solrCore.open() call before the beginning of the operation that would increment the core refCount and decrementing the refCount again in the finally block.
* Now, I was successful in completing the operation, but that caused above Exception.
* Based on the logs and the point that this exception disappears when my operation is completed, my theory is that:
* Incrementing the refCount does not prevent or delay the solrCore.close() call for eviction based on LRU. However, while the refCount is decreased and Solr assumes that this core has been evicted, it is still indeed loaded performing the long-running operation.
* So, now when a new request comes to this core, Solr assumes that it has to be loaded anew and tries to obtain the lock which is actually held by the long running operation and throws this error.
* Can someone, please confirm if this could be the possibility. Is there a way I can get out of it?
Thanks in advance,
Shashank
Re: Error Opening new IndexSearcher - LockObtainFailedException
Posted by Erick Erickson <er...@gmail.com>.
Hmmm, 6.4 was considerably before the refactoring that this patch
addresses so it's not a surprise that it doesn't apply.
On Thu, Sep 21, 2017 at 10:28 PM, Shashank Pedamallu
<sp...@vmware.com> wrote:
> Hi Luiz,
>
> Unfortunately, I’m on version Solr-6.4.2 and the patch does not apply straight away.
>
> Thanks,
> Shashank
>
> On 9/21/17, 8:35 PM, "Luiz Armesto" <lu...@gmail.com> wrote:
>
> Hi Shashank,
>
> There is an open issue about this exception [1]. Can you take a look and
> test the patch to see if it works in your case?
>
> [1] https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_SOLR-2D11297&d=DwIFaQ&c=uilaK90D4TOVoH58JNXRgQ&r=blJD2pBapH3dDkoajIf9mT9SSbbs19wRbChNde1ErNI&m=EBLEhJ6TlQpK4rJngNBxBwypGpdbAuhnuqmgiRGcxZg&s=j69wKZOK2Ve9oeIPl92iyiQLSZS38Qe-ZLj-2OeN-u0&e=
>
> On Sep 21, 2017 10:19 PM, "Shashank Pedamallu" <sp...@vmware.com>
> wrote:
>
> Hi,
>
> I’m seeing the following exception in Solr that gets automatically resolved
> eventually.
> 2017-09-22 00:18:17.243 ERROR (qtp1702660825-17) [ x:spedamallu1-core-1]
> o.a.s.c.CoreContainer Error creating core [spedamallu1-core-1]: Error
> opening new searcher
> org.apache.solr.common.SolrException: Error opening new searcher
> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:952)
> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:816)
> at org.apache.solr.core.CoreContainer.create(CoreContainer.java:890)
> at org.apache.solr.core.CoreContainer.getCore(
> CoreContainer.java:1167)
> at org.apache.solr.servlet.HttpSolrCall.init(HttpSolrCall.java:252)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:418)
> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(
> SolrDispatchFilter.java:345)
> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(
> SolrDispatchFilter.java:296)
> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.
> doFilter(ServletHandler.java:1691)
> at org.eclipse.jetty.servlet.ServletHandler.doHandle(
> ServletHandler.java:582)
> at org.eclipse.jetty.server.handler.ScopedHandler.handle(
> ScopedHandler.java:143)
> at org.eclipse.jetty.security.SecurityHandler.handle(
> SecurityHandler.java:548)
> at org.eclipse.jetty.server.session.SessionHandler.
> doHandle(SessionHandler.java:226)
> at org.eclipse.jetty.server.handler.ContextHandler.
> doHandle(ContextHandler.java:1180)
> at org.eclipse.jetty.servlet.ServletHandler.doScope(
> ServletHandler.java:512)
> at org.eclipse.jetty.server.session.SessionHandler.
> doScope(SessionHandler.java:185)
> at org.eclipse.jetty.server.handler.ContextHandler.
> doScope(ContextHandler.java:1112)
> at org.eclipse.jetty.server.handler.ScopedHandler.handle(
> ScopedHandler.java:141)
> at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(
> ContextHandlerCollection.java:213)
> at org.eclipse.jetty.server.handler.HandlerCollection.
> handle(HandlerCollection.java:119)
> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(
> HandlerWrapper.java:134)
> at org.eclipse.jetty.server.Server.handle(Server.java:534)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
> at org.eclipse.jetty.server.HttpConnection.onFillable(
> HttpConnection.java:251)
> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(
> AbstractConnection.java:273)
> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
> at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(
> SelectChannelEndPoint.java:93)
> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.
> executeProduceConsume(ExecuteProduceConsume.java:303)
> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.
> produceConsume(ExecuteProduceConsume.java:148)
> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(
> ExecuteProduceConsume.java:136)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(
> QueuedThreadPool.java:671)
> at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(
> QueuedThreadPool.java:589)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.solr.common.SolrException: Error opening new searcher
> at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1891)
> at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:2011)
> at org.apache.solr.core.SolrCore.initSearcher(SolrCore.java:1041)
> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:925)
> ... 32 more
> Caused by: org.apache.lucene.store.LockObtainFailedException: Lock held by
> this virtual machine: /Users/spedamallu/Desktop/mount-1/spedamallu1-core-1/
> data/index/write.lock
> at org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(
> NativeFSLockFactory.java:127)
> at org.apache.lucene.store.FSLockFactory.obtainLock(
> FSLockFactory.java:41)
> at org.apache.lucene.store.BaseDirectory.obtainLock(
> BaseDirectory.java:45)
> at org.apache.lucene.store.FilterDirectory.obtainLock(
> FilterDirectory.java:104)
> at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:804)
> at org.apache.solr.update.SolrIndexWriter.<init>(
> SolrIndexWriter.java:125)
> at org.apache.solr.update.SolrIndexWriter.create(
> SolrIndexWriter.java:100)
> at org.apache.solr.update.DefaultSolrCoreState.
> createMainIndexWriter(DefaultSolrCoreState.java:240)
> at org.apache.solr.update.DefaultSolrCoreState.getIndexWriter(
> DefaultSolrCoreState.java:114)
> at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1852)
>
> I kind of have a theory of why this is happening. Can someone please
> confirm if this is indeed the case.
>
> * I was trying to run a long running operation on a transient core and
> it was getting evicted because of LRU
> * So, to ensure my operation completion, I have added a solrCore.open()
> call before the beginning of the operation that would increment the core
> refCount and decrementing the refCount again in the finally block.
> * Now, I was successful in completing the operation, but that caused
> above Exception.
> * Based on the logs and the point that this exception disappears when
> my operation is completed, my theory is that:
> * Incrementing the refCount does not prevent or delay the
> solrCore.close() call for eviction based on LRU. However, while the
> refCount is decreased and Solr assumes that this core has been evicted, it
> is still indeed loaded performing the long-running operation.
> * So, now when a new request comes to this core, Solr assumes that
> it has to be loaded anew and tries to obtain the lock which is actually
> held by the long running operation and throws this error.
> * Can someone, please confirm if this could be the possibility. Is
> there a way I can get out of it?
>
> Thanks in advance,
> Shashank
>
>
Re: Error Opening new IndexSearcher - LockObtainFailedException
Posted by Shashank Pedamallu <sp...@vmware.com>.
Hi Luiz,
Unfortunately, I’m on version Solr-6.4.2 and the patch does not apply straight away.
Thanks,
Shashank
On 9/21/17, 8:35 PM, "Luiz Armesto" <lu...@gmail.com> wrote:
Hi Shashank,
There is an open issue about this exception [1]. Can you take a look and
test the patch to see if it works in your case?
[1] https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_SOLR-2D11297&d=DwIFaQ&c=uilaK90D4TOVoH58JNXRgQ&r=blJD2pBapH3dDkoajIf9mT9SSbbs19wRbChNde1ErNI&m=EBLEhJ6TlQpK4rJngNBxBwypGpdbAuhnuqmgiRGcxZg&s=j69wKZOK2Ve9oeIPl92iyiQLSZS38Qe-ZLj-2OeN-u0&e=
On Sep 21, 2017 10:19 PM, "Shashank Pedamallu" <sp...@vmware.com>
wrote:
Hi,
I’m seeing the following exception in Solr that gets automatically resolved
eventually.
2017-09-22 00:18:17.243 ERROR (qtp1702660825-17) [ x:spedamallu1-core-1]
o.a.s.c.CoreContainer Error creating core [spedamallu1-core-1]: Error
opening new searcher
org.apache.solr.common.SolrException: Error opening new searcher
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:952)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:816)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:890)
at org.apache.solr.core.CoreContainer.getCore(
CoreContainer.java:1167)
at org.apache.solr.servlet.HttpSolrCall.init(HttpSolrCall.java:252)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:418)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(
SolrDispatchFilter.java:345)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(
SolrDispatchFilter.java:296)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.
doFilter(ServletHandler.java:1691)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(
ServletHandler.java:582)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(
ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(
SecurityHandler.java:548)
at org.eclipse.jetty.server.session.SessionHandler.
doHandle(SessionHandler.java:226)
at org.eclipse.jetty.server.handler.ContextHandler.
doHandle(ContextHandler.java:1180)
at org.eclipse.jetty.servlet.ServletHandler.doScope(
ServletHandler.java:512)
at org.eclipse.jetty.server.session.SessionHandler.
doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.
doScope(ContextHandler.java:1112)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(
ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(
ContextHandlerCollection.java:213)
at org.eclipse.jetty.server.handler.HandlerCollection.
handle(HandlerCollection.java:119)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(
HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:534)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
at org.eclipse.jetty.server.HttpConnection.onFillable(
HttpConnection.java:251)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(
AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(
SelectChannelEndPoint.java:93)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.
executeProduceConsume(ExecuteProduceConsume.java:303)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.
produceConsume(ExecuteProduceConsume.java:148)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(
ExecuteProduceConsume.java:136)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(
QueuedThreadPool.java:671)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(
QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.solr.common.SolrException: Error opening new searcher
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1891)
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:2011)
at org.apache.solr.core.SolrCore.initSearcher(SolrCore.java:1041)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:925)
... 32 more
Caused by: org.apache.lucene.store.LockObtainFailedException: Lock held by
this virtual machine: /Users/spedamallu/Desktop/mount-1/spedamallu1-core-1/
data/index/write.lock
at org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(
NativeFSLockFactory.java:127)
at org.apache.lucene.store.FSLockFactory.obtainLock(
FSLockFactory.java:41)
at org.apache.lucene.store.BaseDirectory.obtainLock(
BaseDirectory.java:45)
at org.apache.lucene.store.FilterDirectory.obtainLock(
FilterDirectory.java:104)
at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:804)
at org.apache.solr.update.SolrIndexWriter.<init>(
SolrIndexWriter.java:125)
at org.apache.solr.update.SolrIndexWriter.create(
SolrIndexWriter.java:100)
at org.apache.solr.update.DefaultSolrCoreState.
createMainIndexWriter(DefaultSolrCoreState.java:240)
at org.apache.solr.update.DefaultSolrCoreState.getIndexWriter(
DefaultSolrCoreState.java:114)
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1852)
I kind of have a theory of why this is happening. Can someone please
confirm if this is indeed the case.
* I was trying to run a long running operation on a transient core and
it was getting evicted because of LRU
* So, to ensure my operation completion, I have added a solrCore.open()
call before the beginning of the operation that would increment the core
refCount and decrementing the refCount again in the finally block.
* Now, I was successful in completing the operation, but that caused
above Exception.
* Based on the logs and the point that this exception disappears when
my operation is completed, my theory is that:
* Incrementing the refCount does not prevent or delay the
solrCore.close() call for eviction based on LRU. However, while the
refCount is decreased and Solr assumes that this core has been evicted, it
is still indeed loaded performing the long-running operation.
* So, now when a new request comes to this core, Solr assumes that
it has to be loaded anew and tries to obtain the lock which is actually
held by the long running operation and throws this error.
* Can someone, please confirm if this could be the possibility. Is
there a way I can get out of it?
Thanks in advance,
Shashank
Re: Error Opening new IndexSearcher - LockObtainFailedException
Posted by Luiz Armesto <lu...@gmail.com>.
Hi Shashank,
There is an open issue about this exception [1]. Can you take a look and
test the patch to see if it works in your case?
[1] https://issues.apache.org/jira/browse/SOLR-11297
On Sep 21, 2017 10:19 PM, "Shashank Pedamallu" <sp...@vmware.com>
wrote:
Hi,
I’m seeing the following exception in Solr that gets automatically resolved
eventually.
2017-09-22 00:18:17.243 ERROR (qtp1702660825-17) [ x:spedamallu1-core-1]
o.a.s.c.CoreContainer Error creating core [spedamallu1-core-1]: Error
opening new searcher
org.apache.solr.common.SolrException: Error opening new searcher
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:952)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:816)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:890)
at org.apache.solr.core.CoreContainer.getCore(
CoreContainer.java:1167)
at org.apache.solr.servlet.HttpSolrCall.init(HttpSolrCall.java:252)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:418)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(
SolrDispatchFilter.java:345)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(
SolrDispatchFilter.java:296)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.
doFilter(ServletHandler.java:1691)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(
ServletHandler.java:582)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(
ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(
SecurityHandler.java:548)
at org.eclipse.jetty.server.session.SessionHandler.
doHandle(SessionHandler.java:226)
at org.eclipse.jetty.server.handler.ContextHandler.
doHandle(ContextHandler.java:1180)
at org.eclipse.jetty.servlet.ServletHandler.doScope(
ServletHandler.java:512)
at org.eclipse.jetty.server.session.SessionHandler.
doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.
doScope(ContextHandler.java:1112)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(
ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(
ContextHandlerCollection.java:213)
at org.eclipse.jetty.server.handler.HandlerCollection.
handle(HandlerCollection.java:119)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(
HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:534)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
at org.eclipse.jetty.server.HttpConnection.onFillable(
HttpConnection.java:251)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(
AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(
SelectChannelEndPoint.java:93)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.
executeProduceConsume(ExecuteProduceConsume.java:303)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.
produceConsume(ExecuteProduceConsume.java:148)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(
ExecuteProduceConsume.java:136)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(
QueuedThreadPool.java:671)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(
QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.solr.common.SolrException: Error opening new searcher
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1891)
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:2011)
at org.apache.solr.core.SolrCore.initSearcher(SolrCore.java:1041)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:925)
... 32 more
Caused by: org.apache.lucene.store.LockObtainFailedException: Lock held by
this virtual machine: /Users/spedamallu/Desktop/mount-1/spedamallu1-core-1/
data/index/write.lock
at org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(
NativeFSLockFactory.java:127)
at org.apache.lucene.store.FSLockFactory.obtainLock(
FSLockFactory.java:41)
at org.apache.lucene.store.BaseDirectory.obtainLock(
BaseDirectory.java:45)
at org.apache.lucene.store.FilterDirectory.obtainLock(
FilterDirectory.java:104)
at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:804)
at org.apache.solr.update.SolrIndexWriter.<init>(
SolrIndexWriter.java:125)
at org.apache.solr.update.SolrIndexWriter.create(
SolrIndexWriter.java:100)
at org.apache.solr.update.DefaultSolrCoreState.
createMainIndexWriter(DefaultSolrCoreState.java:240)
at org.apache.solr.update.DefaultSolrCoreState.getIndexWriter(
DefaultSolrCoreState.java:114)
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1852)
I kind of have a theory of why this is happening. Can someone please
confirm if this is indeed the case.
* I was trying to run a long running operation on a transient core and
it was getting evicted because of LRU
* So, to ensure my operation completion, I have added a solrCore.open()
call before the beginning of the operation that would increment the core
refCount and decrementing the refCount again in the finally block.
* Now, I was successful in completing the operation, but that caused
above Exception.
* Based on the logs and the point that this exception disappears when
my operation is completed, my theory is that:
* Incrementing the refCount does not prevent or delay the
solrCore.close() call for eviction based on LRU. However, while the
refCount is decreased and Solr assumes that this core has been evicted, it
is still indeed loaded performing the long-running operation.
* So, now when a new request comes to this core, Solr assumes that
it has to be loaded anew and tries to obtain the lock which is actually
held by the long running operation and throws this error.
* Can someone, please confirm if this could be the possibility. Is
there a way I can get out of it?
Thanks in advance,
Shashank