You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by eks dev <ek...@yahoo.co.uk> on 2012/02/22 16:56:58 UTC
SnapPull failed :org.apache.solr.common.SolrException: Error opening
new searcher
We started observing strange failures from ReplicationHandler when we
commit on master.... trunk version 4-5 days old.
It works sometimes, and sometimes not.... didn't dig deeper yet.
Looks like the real culprit hides behind:
org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
Looks familiar to somebody?
120222 154959 SEVERE SnapPull failed
:org.apache.solr.common.SolrException: Error opening new searcher
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1138)
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1251)
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1043)
at org.apache.solr.update.DirectUpdateHandler2.commit(Unknown Source)
at org.apache.solr.handler.SnapPuller.doCommit(SnapPuller.java:503)
at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:348)
at org.apache.solr.handler.ReplicationHandler.doFetch(Unknown Source)
at org.apache.solr.handler.SnapPuller$1.run(SnapPuller.java:163)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
Caused by: org.apache.lucene.store.AlreadyClosedException: this
IndexWriter is closed
at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:810)
at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:815)
at org.apache.lucene.index.IndexWriter.nrtIsCurrent(IndexWriter.java:3984)
at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:254)
at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:233)
at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:223)
at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:170)
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1095)
... 15 more
Re: SnapPull failed :org.apache.solr.common.SolrException: Error
opening new searcher
Posted by eks dev <ek...@yahoo.co.uk>.
it loos like it works, with patch, after a couple of hours of testing
under same conditions didn't see it happen (without it, approx. every
15 minutes).
I do not think it will happen again with this patch.
Thanks again and my respect to your debugging capacity, my bug report
was really thin.
On Thu, Feb 23, 2012 at 8:47 AM, eks dev <ek...@yahoo.co.uk> wrote:
> thanks Mark, I will give it a go and report back...
>
> On Thu, Feb 23, 2012 at 1:31 AM, Mark Miller <ma...@gmail.com> wrote:
>> Looks like an issue around replication IndexWriter reboot, soft commits and hard commits.
>>
>> I think I've got a workaround for it:
>>
>> Index: solr/core/src/java/org/apache/solr/handler/SnapPuller.java
>> ===================================================================
>> --- solr/core/src/java/org/apache/solr/handler/SnapPuller.java (revision 1292344)
>> +++ solr/core/src/java/org/apache/solr/handler/SnapPuller.java (working copy)
>> @@ -499,6 +499,17 @@
>>
>> // reboot the writer on the new index and get a new searcher
>> solrCore.getUpdateHandler().newIndexWriter();
>> + Future[] waitSearcher = new Future[1];
>> + solrCore.getSearcher(true, false, waitSearcher, true);
>> + if (waitSearcher[0] != null) {
>> + try {
>> + waitSearcher[0].get();
>> + } catch (InterruptedException e) {
>> + SolrException.log(LOG,e);
>> + } catch (ExecutionException e) {
>> + SolrException.log(LOG,e);
>> + }
>> + }
>> // update our commit point to the right dir
>> solrCore.getUpdateHandler().commit(new CommitUpdateCommand(req, false));
>>
>> That should allow the searcher that the following commit command prompts to see the *new* IndexWriter.
>>
>> On Feb 22, 2012, at 10:56 AM, eks dev wrote:
>>
>>> We started observing strange failures from ReplicationHandler when we
>>> commit on master.... trunk version 4-5 days old.
>>> It works sometimes, and sometimes not.... didn't dig deeper yet.
>>>
>>> Looks like the real culprit hides behind:
>>> org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
>>>
>>> Looks familiar to somebody?
>>>
>>>
>>> 120222 154959 SEVERE SnapPull failed
>>> :org.apache.solr.common.SolrException: Error opening new searcher
>>> at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1138)
>>> at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1251)
>>> at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1043)
>>> at org.apache.solr.update.DirectUpdateHandler2.commit(Unknown Source)
>>> at org.apache.solr.handler.SnapPuller.doCommit(SnapPuller.java:503)
>>> at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:348)
>>> at org.apache.solr.handler.ReplicationHandler.doFetch(Unknown Source)
>>> at org.apache.solr.handler.SnapPuller$1.run(SnapPuller.java:163)
>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>> at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
>>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>>> at java.lang.Thread.run(Thread.java:722)
>>> Caused by: org.apache.lucene.store.AlreadyClosedException: this
>>> IndexWriter is closed
>>> at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:810)
>>> at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:815)
>>> at org.apache.lucene.index.IndexWriter.nrtIsCurrent(IndexWriter.java:3984)
>>> at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:254)
>>> at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:233)
>>> at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:223)
>>> at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:170)
>>> at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1095)
>>> ... 15 more
>>
>> - Mark Miller
>> lucidimagination.com
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
Re: SnapPull failed :org.apache.solr.common.SolrException: Error
opening new searcher
Posted by eks dev <ek...@yahoo.co.uk>.
thanks Mark, I will give it a go and report back...
On Thu, Feb 23, 2012 at 1:31 AM, Mark Miller <ma...@gmail.com> wrote:
> Looks like an issue around replication IndexWriter reboot, soft commits and hard commits.
>
> I think I've got a workaround for it:
>
> Index: solr/core/src/java/org/apache/solr/handler/SnapPuller.java
> ===================================================================
> --- solr/core/src/java/org/apache/solr/handler/SnapPuller.java (revision 1292344)
> +++ solr/core/src/java/org/apache/solr/handler/SnapPuller.java (working copy)
> @@ -499,6 +499,17 @@
>
> // reboot the writer on the new index and get a new searcher
> solrCore.getUpdateHandler().newIndexWriter();
> + Future[] waitSearcher = new Future[1];
> + solrCore.getSearcher(true, false, waitSearcher, true);
> + if (waitSearcher[0] != null) {
> + try {
> + waitSearcher[0].get();
> + } catch (InterruptedException e) {
> + SolrException.log(LOG,e);
> + } catch (ExecutionException e) {
> + SolrException.log(LOG,e);
> + }
> + }
> // update our commit point to the right dir
> solrCore.getUpdateHandler().commit(new CommitUpdateCommand(req, false));
>
> That should allow the searcher that the following commit command prompts to see the *new* IndexWriter.
>
> On Feb 22, 2012, at 10:56 AM, eks dev wrote:
>
>> We started observing strange failures from ReplicationHandler when we
>> commit on master.... trunk version 4-5 days old.
>> It works sometimes, and sometimes not.... didn't dig deeper yet.
>>
>> Looks like the real culprit hides behind:
>> org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
>>
>> Looks familiar to somebody?
>>
>>
>> 120222 154959 SEVERE SnapPull failed
>> :org.apache.solr.common.SolrException: Error opening new searcher
>> at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1138)
>> at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1251)
>> at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1043)
>> at org.apache.solr.update.DirectUpdateHandler2.commit(Unknown Source)
>> at org.apache.solr.handler.SnapPuller.doCommit(SnapPuller.java:503)
>> at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:348)
>> at org.apache.solr.handler.ReplicationHandler.doFetch(Unknown Source)
>> at org.apache.solr.handler.SnapPuller$1.run(SnapPuller.java:163)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>> at java.lang.Thread.run(Thread.java:722)
>> Caused by: org.apache.lucene.store.AlreadyClosedException: this
>> IndexWriter is closed
>> at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:810)
>> at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:815)
>> at org.apache.lucene.index.IndexWriter.nrtIsCurrent(IndexWriter.java:3984)
>> at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:254)
>> at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:233)
>> at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:223)
>> at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:170)
>> at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1095)
>> ... 15 more
>
> - Mark Miller
> lucidimagination.com
>
>
>
>
>
>
>
>
>
>
>
Re: SnapPull failed :org.apache.solr.common.SolrException: Error opening new searcher
Posted by Mark Miller <ma...@gmail.com>.
Looks like an issue around replication IndexWriter reboot, soft commits and hard commits.
I think I've got a workaround for it:
Index: solr/core/src/java/org/apache/solr/handler/SnapPuller.java
===================================================================
--- solr/core/src/java/org/apache/solr/handler/SnapPuller.java (revision 1292344)
+++ solr/core/src/java/org/apache/solr/handler/SnapPuller.java (working copy)
@@ -499,6 +499,17 @@
// reboot the writer on the new index and get a new searcher
solrCore.getUpdateHandler().newIndexWriter();
+ Future[] waitSearcher = new Future[1];
+ solrCore.getSearcher(true, false, waitSearcher, true);
+ if (waitSearcher[0] != null) {
+ try {
+ waitSearcher[0].get();
+ } catch (InterruptedException e) {
+ SolrException.log(LOG,e);
+ } catch (ExecutionException e) {
+ SolrException.log(LOG,e);
+ }
+ }
// update our commit point to the right dir
solrCore.getUpdateHandler().commit(new CommitUpdateCommand(req, false));
That should allow the searcher that the following commit command prompts to see the *new* IndexWriter.
On Feb 22, 2012, at 10:56 AM, eks dev wrote:
> We started observing strange failures from ReplicationHandler when we
> commit on master.... trunk version 4-5 days old.
> It works sometimes, and sometimes not.... didn't dig deeper yet.
>
> Looks like the real culprit hides behind:
> org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
>
> Looks familiar to somebody?
>
>
> 120222 154959 SEVERE SnapPull failed
> :org.apache.solr.common.SolrException: Error opening new searcher
> at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1138)
> at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1251)
> at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1043)
> at org.apache.solr.update.DirectUpdateHandler2.commit(Unknown Source)
> at org.apache.solr.handler.SnapPuller.doCommit(SnapPuller.java:503)
> at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:348)
> at org.apache.solr.handler.ReplicationHandler.doFetch(Unknown Source)
> at org.apache.solr.handler.SnapPuller$1.run(SnapPuller.java:163)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: org.apache.lucene.store.AlreadyClosedException: this
> IndexWriter is closed
> at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:810)
> at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:815)
> at org.apache.lucene.index.IndexWriter.nrtIsCurrent(IndexWriter.java:3984)
> at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:254)
> at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:233)
> at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:223)
> at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:170)
> at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1095)
> ... 15 more
- Mark Miller
lucidimagination.com