You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Osborn Chan <oc...@shutterfly.com> on 2010/01/15 21:23:21 UTC
Index Courruption after replication by new Solr 1.4 Replication
Hi all,
I have migrated new Solr 1.4 Replication feature with multicore support from Solr 1.2 with NFS mounting recently. The following exceptions are in catalina.log from time to time, and there are some EOF exceptions which I believe the slave index files are corrupted after replication from index server. I have following configuration with Solr 1.4, please correct me if it is configured incorrectly.
(The index files are not corrupted in master servers, but it is corrupted in slave servers. Usually only one of the slave servers are corrupted with EOF exception, but not all.)
1 Master Server: (Index Server)
- 8 indexes with multicore configuration.
- All indexes are configured to "replicateAfter" optimize only.
- The size of index data are vary. The smallest index only have 2.5 MB. The biggest index have ~ 100 MB.
- There would be infrequent optimize calls to indexes. (a optimize call every ~30 mins to 6 hours depending on indexes).
- There are many commit calls to all indexes. (But there is no concurrent commit and optimize for all indexes.)
- Did not configure "commitReserveDuration" in ReplicationHandler - Using default values.
4 Slave Servers (Search Server)
- 8 indexes with multicore configuration.
- All indexes are configured to poll for every ~15 minutes.
- All update handler configuration are removed in solrconfig-slave.xml (solrconfig.xml) in order to prevent add/commit/optimize calls.
- (Search Slave Servers are only responsible for search operation.)
- <updateHandler class="solr.DirectUpdateHandler2"> removed.
- <requestHandler name="/update" class="solr.XmlUpdateRequestHandler" /> removed.
- <requestHandler name="/update/javabin" class="solr.BinaryUpdateRequestHandler" /> removed.
A) FileNotFoundException
INFO: Total time taken for download : 1 secs
Jan 15, 2010 10:34:16 AM org.apache.solr.handler.ReplicationHandler doFetch
SEVERE: SnapPull failed
org.apache.solr.common.SolrException: Index fetch failed :
at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:329)
at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:264)
at org.apache.solr.handler.SnapPuller$1.run(SnapPuller.java:159)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:280)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:135)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:65)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:142)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:166)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
at java.lang.Thread.run(Thread.java:595)
Caused by: java.io.FileNotFoundException: File does not exist /slaveIndexData/publicGalleryTagDef/index.20100115103415/_al.fdx
at org.apache.solr.common.util.FileUtils.sync(FileUtils.java:55)
at org.apache.solr.handler.SnapPuller$FileFetcher$1.run(SnapPuller.java:911)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
at java.util.concurrent.FutureTask.run(FutureTask.java:123)
... 3 more
Jan 15, 2010 10:34:17 AM org.apache.solr.core.SolrCore execute
INFO: [publicGalleryPostMaster] webapp=/multicore path=/select params={wt=javabin&rows=10&start=0&sort=createTime_dt+desc&q=%2B(profileId_s:/community/sfly/publicprofile/0AcM27Nw3aNWLi4)+%2Bstate_s:A&version=1} hits=1 status=0 QTime=1
B) LockReleaseFailedException
SEVERE: SnapPull failed
org.apache.solr.common.SolrException: Index fetch failed :
at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:329)
at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:264)
at org.apache.solr.handler.SnapPuller$1.run(SnapPuller.java:159)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:280)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:135)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:65)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:142)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:166)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
at java.lang.Thread.run(Thread.java:595)
Caused by: org.apache.lucene.store.LockReleaseFailedException: failed to delete /slaveIndexData/publicGalleryTagDefAggregate/index/lucene-fb30bdbbdc6927666873dd616884ba29-write.lock
at org.apache.lucene.store.NativeFSLock.release(NativeFSLockFactory.java:298)
at org.apache.lucene.index.IndexWriter.closeInternal(IndexWriter.java:2225)
at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:2153)
at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:2117)
at org.apache.solr.update.SolrIndexWriter.close(SolrIndexWriter.java:229)
at org.apache.solr.update.DirectUpdateHandler2.closeWriter(DirectUpdateHandler2.java:181)
at org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:409)
at org.apache.solr.handler.SnapPuller.doCommit(SnapPuller.java:467)
at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:319)
... 11 more
Jan 15, 2010 12:21:18 AM org.apache.solr.handler.SnapPuller fetchLatestIndex
INFO: Slave in sync with master.
C) EOF Exception
INFO: [publicGalleryPostMaster] webapp=/multicore path=/select params={wt=javabin&rows=1&start=0&sort=createTime_dt+desc&q=%2B(profileId_s:/community/sfly/publicprofile/0AbOWLNszaOWTiw)+%2B(lastBookmarked_dt:[2010-01-08T08:49:38.271Z+TO+2010-01-15T08:49:38.271Z]+lastCommented_dt:[2010-01-08T08:49:38.271Z+TO+2010-01-15T08:49:38.271Z])+%2Bstate_s:A&version=1} hits=0 status=0 QTime=2
Jan 15, 2010 12:49:42 AM org.apache.solr.common.SolrException log
SEVERE: java.io.IOException: read past EOF
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:151)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:38)
at org.apache.lucene.store.IndexInput.readVInt(IndexInput.java:80)
at org.apache.lucene.index.TermBuffer.read(TermBuffer.java:64)
at org.apache.lucene.index.SegmentTermEnum.next(SegmentTermEnum.java:129)
at org.apache.lucene.index.SegmentTermEnum.scanTo(SegmentTermEnum.java:160)
at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:232)
at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:179)
at org.apache.lucene.index.SegmentReader.docFreq(SegmentReader.java:975)
at org.apache.lucene.index.DirectoryReader.docFreq(DirectoryReader.java:627)
at org.apache.solr.search.SolrIndexReader.docFreq(SolrIndexReader.java:308)
at org.apache.lucene.search.IndexSearcher.docFreq(IndexSearcher.java:147)
at org.apache.lucene.search.Similarity.idfExplain(Similarity.java:833)
at org.apache.lucene.search.PhraseQuery$PhraseWeight.<init>(PhraseQuery.java:122)
at org.apache.lucene.search.PhraseQuery.createWeight(PhraseQuery.java:250)
at org.apache.lucene.search.BooleanQuery$BooleanWeight.<init>(BooleanQuery.java:184)
at org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:415)
at org.apache.lucene.search.Query.weight(Query.java:99)
at org.apache.lucene.search.Searcher.createWeight(Searcher.java:230)
at org.apache.lucene.search.Searcher.search(Searcher.java:171)
at org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:988)
at org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:884)
at org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:341)
at org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:182)
at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:195)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316)
at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:541)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)
at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664)
at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:80)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
at java.lang.Thread.run(Thread.java:595)
Thanks a lot!
Osborn
Re: Index Courruption after replication by new Solr 1.4 Replication
Posted by Chris Hostetter <ho...@fucit.org>.
: Subject: Index Courruption after replication by new Solr 1.4 Replication
: References: <3C...@mail02.tveyes.com>
: <66...@web52905.mail.re2.yahoo.com>
: <3C...@mail02.tveyes.com>
: <35...@mail.gmail.com>
: <3C...@mail02.tveyes.com>
: <35...@mail.gmail.com>
: <3C...@mail02.tveyes.com>
: In-Reply-To: <3C...@mail02.tveyes.com>
http://people.apache.org/~hossman/#threadhijack
Thread Hijacking on Mailing Lists
When starting a new discussion on a mailing list, please do not reply to
an existing message, instead start a fresh email. Even if you change the
subject line of your email, other mail headers still track which thread
you replied to and your question is "hidden" in that thread and gets less
attention. It makes following discussions in the mailing list archives
particularly difficult.
See Also: http://en.wikipedia.org/wiki/User:DonDiego/Thread_hijacking
-Hoss
RE: Index Courruption after replication by new Solr 1.4 Replication
Posted by bramsreddy <br...@gmail.com>.
Hi,
Is this already registered as bug.Is their any fix to this issue(i want use
EmbeddedSolrServer server only).
Regards,
Ram.
--
View this message in context: http://lucene.472066.n3.nabble.com/1-3-help-with-update-timeout-issue-tp505766p3310048.html
Sent from the Solr - User mailing list archive at Nabble.com.
RE: Index Courruption after replication by new Solr 1.4 Replication
Posted by Osborn Chan <oc...@shutterfly.com>.
Hi All,
I found out there is file corruption issue by using both "EmbeddedSolrServer" & "Solr 1.4 Java based replication" together in slave server.
In my slave server, I have 2 webapps in a tomcat instance.
1) "multicore" webapp with slave config
2) "my custom" webapp using EmbeddedSolrServer while queries Solr Index Data.
Both webapps were set up according to the instruction from Solr wiki.
However, I found out there are multi-threading issue which cause index file corruption.
The following is the root case:
EmbeddedSolrServer requires to have a CoreContainer object as parameter. However, during the creation of CoreContainer object, the process load the slave solr configuration which silently creates an Extra ReplcationHandler (SnapPuller) in background. However, there is a ReplcationHandler (SnapPuller) already created by multicore webapp because of the slave configuration.
As a result, there are 2 threads doing file replication as same time. It causes index corruption with different IOExceptions.
After I replaced the usage of EmbeddedSolrServer with CommonsHttpSolrServer (Stop creating CoreContainer object in slave server), Solr 1.4 Java based replication work perfectly without having any file corruption issue.
In other to use EmbeddedSolrServer in slave server, I think we need to have a way to create CoreContainer object with slave configuration without creating extra thread to replicate files.
Should I file a bug?
Thanks,
Osborn
-----Original Message-----
From: Osborn Chan [mailto:ochan@shutterfly.com]
Sent: Friday, January 15, 2010 12:35 PM
To: solr-user@lucene.apache.org
Subject: RE: Index Courruption after replication by new Solr 1.4 Replication
Hi Otis,
Thanks. There is no NFS anymore, and all index files are local. We migrated to new Solr 1.4 new Replication in order to avoid all the NSF Stale Exception.
Thanks,
Osborn
-----Original Message-----
From: Otis Gospodnetic [mailto:otis_gospodnetic@yahoo.com]
Sent: Friday, January 15, 2010 12:31 PM
To: solr-user@lucene.apache.org
Subject: Re: Index Courruption after replication by new Solr 1.4 Replication
This is not a direct answer to your question, but can you avoid NFS? My first guess would be that NFS somehow causes this problem. If you check the ML archives for: NFS lock , you will see what I mean.
Otis
--
Sematext -- http://sematext.com/ -- Solr - Lucene - Nutch
----- Original Message ----
> From: Osborn Chan <oc...@shutterfly.com>
> To: "solr-user@lucene.apache.org" <so...@lucene.apache.org>
> Sent: Fri, January 15, 2010 3:23:21 PM
> Subject: Index Courruption after replication by new Solr 1.4 Replication
>
> Hi all,
>
> I have migrated new Solr 1.4 Replication feature with multicore support from
> Solr 1.2 with NFS mounting recently. The following exceptions are in
> catalina.log from time to time, and there are some EOF exceptions which I
> believe the slave index files are corrupted after replication from index server.
> I have following configuration with Solr 1.4, please correct me if it is
> configured incorrectly.
>
> (The index files are not corrupted in master servers, but it is corrupted in
> slave servers. Usually only one of the slave servers are corrupted with EOF
> exception, but not all.)
>
> 1 Master Server: (Index Server)
> - 8 indexes with multicore configuration.
> - All indexes are configured to "replicateAfter" optimize only.
> - The size of index data are vary. The smallest index only have 2.5 MB. The
> biggest index have ~ 100 MB.
> - There would be infrequent optimize calls to indexes. (a optimize call
> every ~30 mins to 6 hours depending on indexes).
> - There are many commit calls to all indexes. (But there is no concurrent
> commit and optimize for all indexes.)
> - Did not configure "commitReserveDuration" in ReplicationHandler - Using
> default values.
>
> 4 Slave Servers (Search Server)
> - 8 indexes with multicore configuration.
> - All indexes are configured to poll for every ~15 minutes.
> - All update handler configuration are removed in solrconfig-slave.xml
> (solrconfig.xml) in order to prevent add/commit/optimize calls.
> - (Search Slave Servers are only responsible for search operation.)
> - removed.
> -
> removed.
> -
> class="solr.BinaryUpdateRequestHandler" /> removed.
>
> A) FileNotFoundException
>
> INFO: Total time taken for download : 1 secs
> Jan 15, 2010 10:34:16 AM org.apache.solr.handler.ReplicationHandler doFetch
> SEVERE: SnapPull failed
> org.apache.solr.common.SolrException: Index fetch failed :
> at
> org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:329)
> at
> org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:264)
> at org.apache.solr.handler.SnapPuller$1.run(SnapPuller.java:159)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417)
> at
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:280)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:135)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:65)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:142)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:166)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
> at java.lang.Thread.run(Thread.java:595)
> Caused by: java.io.FileNotFoundException: File does not exist
> /slaveIndexData/publicGalleryTagDef/index.20100115103415/_al.fdx
> at org.apache.solr.common.util.FileUtils.sync(FileUtils.java:55)
> at
> org.apache.solr.handler.SnapPuller$FileFetcher$1.run(SnapPuller.java:911)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
> at java.util.concurrent.FutureTask.run(FutureTask.java:123)
> ... 3 more
> Jan 15, 2010 10:34:17 AM org.apache.solr.core.SolrCore execute
> INFO: [publicGalleryPostMaster] webapp=/multicore path=/select
> params={wt=javabin&rows=10&start=0&sort=createTime_dt+desc&q=%2B(profileId_s:/community/sfly/publicprofile/0AcM27Nw3aNWLi4)+%2Bstate_s:A&version=1}
> hits=1 status=0 QTime=1
>
> B) LockReleaseFailedException
>
> SEVERE: SnapPull failed
> org.apache.solr.common.SolrException: Index fetch failed :
> at
> org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:329)
> at
> org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:264)
> at org.apache.solr.handler.SnapPuller$1.run(SnapPuller.java:159)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417)
> at
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:280)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:135)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:65)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:142)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:166)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
> at java.lang.Thread.run(Thread.java:595)
> Caused by: org.apache.lucene.store.LockReleaseFailedException: failed to delete
> /slaveIndexData/publicGalleryTagDefAggregate/index/lucene-fb30bdbbdc6927666873dd616884ba29-write.lock
> at
> org.apache.lucene.store.NativeFSLock.release(NativeFSLockFactory.java:298)
> at
> org.apache.lucene.index.IndexWriter.closeInternal(IndexWriter.java:2225)
> at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:2153)
> at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:2117)
> at
> org.apache.solr.update.SolrIndexWriter.close(SolrIndexWriter.java:229)
> at
> org.apache.solr.update.DirectUpdateHandler2.closeWriter(DirectUpdateHandler2.java:181)
> at
> org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:409)
> at org.apache.solr.handler.SnapPuller.doCommit(SnapPuller.java:467)
> at
> org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:319)
> ... 11 more
> Jan 15, 2010 12:21:18 AM org.apache.solr.handler.SnapPuller fetchLatestIndex
> INFO: Slave in sync with master.
>
> C) EOF Exception
> INFO: [publicGalleryPostMaster] webapp=/multicore path=/select
> params={wt=javabin&rows=1&start=0&sort=createTime_dt+desc&q=%2B(profileId_s:/community/sfly/publicprofile/0AbOWLNszaOWTiw)+%2B(lastBookmarked_dt:[2010-01-08T08:49:38.271Z+TO+2010-01-15T08:49:38.271Z]+lastCommented_dt:[2010-01-08T08:49:38.271Z+TO+2010-01-15T08:49:38.271Z])+%2Bstate_s:A&version=1}
> hits=0 status=0 QTime=2
> Jan 15, 2010 12:49:42 AM org.apache.solr.common.SolrException log
> SEVERE: java.io.IOException: read past EOF
> at
> org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:151)
> at
> org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:38)
> at org.apache.lucene.store.IndexInput.readVInt(IndexInput.java:80)
> at org.apache.lucene.index.TermBuffer.read(TermBuffer.java:64)
> at
> org.apache.lucene.index.SegmentTermEnum.next(SegmentTermEnum.java:129)
> at
> org.apache.lucene.index.SegmentTermEnum.scanTo(SegmentTermEnum.java:160)
> at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:232)
> at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:179)
> at org.apache.lucene.index.SegmentReader.docFreq(SegmentReader.java:975)
> at
> org.apache.lucene.index.DirectoryReader.docFreq(DirectoryReader.java:627)
> at
> org.apache.solr.search.SolrIndexReader.docFreq(SolrIndexReader.java:308)
> at
> org.apache.lucene.search.IndexSearcher.docFreq(IndexSearcher.java:147)
> at org.apache.lucene.search.Similarity.idfExplain(Similarity.java:833)
> at
> org.apache.lucene.search.PhraseQuery$PhraseWeight.(PhraseQuery.java:122)
> at
> org.apache.lucene.search.PhraseQuery.createWeight(PhraseQuery.java:250)
> at
> org.apache.lucene.search.BooleanQuery$BooleanWeight.(BooleanQuery.java:184)
> at
> org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:415)
> at org.apache.lucene.search.Query.weight(Query.java:99)
> at org.apache.lucene.search.Searcher.createWeight(Searcher.java:230)
> at org.apache.lucene.search.Searcher.search(Searcher.java:171)
> at
> org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:988)
> at
> org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:884)
> at
> org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:341)
> at
> org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:182)
> at
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:195)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316)
> at
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
> at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
> at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
> at
> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:541)
> at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
> at
> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)
> at
> org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664)
> at
> org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
> at
> org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:80)
> at
> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
> at java.lang.Thread.run(Thread.java:595)
>
> Thanks a lot!
>
> Osborn
RE: Index Courruption after replication by new Solr 1.4 Replication
Posted by Osborn Chan <oc...@shutterfly.com>.
Hi Otis,
Thanks. There is no NFS anymore, and all index files are local. We migrated to new Solr 1.4 new Replication in order to avoid all the NSF Stale Exception.
Thanks,
Osborn
-----Original Message-----
From: Otis Gospodnetic [mailto:otis_gospodnetic@yahoo.com]
Sent: Friday, January 15, 2010 12:31 PM
To: solr-user@lucene.apache.org
Subject: Re: Index Courruption after replication by new Solr 1.4 Replication
This is not a direct answer to your question, but can you avoid NFS? My first guess would be that NFS somehow causes this problem. If you check the ML archives for: NFS lock , you will see what I mean.
Otis
--
Sematext -- http://sematext.com/ -- Solr - Lucene - Nutch
----- Original Message ----
> From: Osborn Chan <oc...@shutterfly.com>
> To: "solr-user@lucene.apache.org" <so...@lucene.apache.org>
> Sent: Fri, January 15, 2010 3:23:21 PM
> Subject: Index Courruption after replication by new Solr 1.4 Replication
>
> Hi all,
>
> I have migrated new Solr 1.4 Replication feature with multicore support from
> Solr 1.2 with NFS mounting recently. The following exceptions are in
> catalina.log from time to time, and there are some EOF exceptions which I
> believe the slave index files are corrupted after replication from index server.
> I have following configuration with Solr 1.4, please correct me if it is
> configured incorrectly.
>
> (The index files are not corrupted in master servers, but it is corrupted in
> slave servers. Usually only one of the slave servers are corrupted with EOF
> exception, but not all.)
>
> 1 Master Server: (Index Server)
> - 8 indexes with multicore configuration.
> - All indexes are configured to "replicateAfter" optimize only.
> - The size of index data are vary. The smallest index only have 2.5 MB. The
> biggest index have ~ 100 MB.
> - There would be infrequent optimize calls to indexes. (a optimize call
> every ~30 mins to 6 hours depending on indexes).
> - There are many commit calls to all indexes. (But there is no concurrent
> commit and optimize for all indexes.)
> - Did not configure "commitReserveDuration" in ReplicationHandler - Using
> default values.
>
> 4 Slave Servers (Search Server)
> - 8 indexes with multicore configuration.
> - All indexes are configured to poll for every ~15 minutes.
> - All update handler configuration are removed in solrconfig-slave.xml
> (solrconfig.xml) in order to prevent add/commit/optimize calls.
> - (Search Slave Servers are only responsible for search operation.)
> - removed.
> -
> removed.
> -
> class="solr.BinaryUpdateRequestHandler" /> removed.
>
> A) FileNotFoundException
>
> INFO: Total time taken for download : 1 secs
> Jan 15, 2010 10:34:16 AM org.apache.solr.handler.ReplicationHandler doFetch
> SEVERE: SnapPull failed
> org.apache.solr.common.SolrException: Index fetch failed :
> at
> org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:329)
> at
> org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:264)
> at org.apache.solr.handler.SnapPuller$1.run(SnapPuller.java:159)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417)
> at
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:280)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:135)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:65)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:142)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:166)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
> at java.lang.Thread.run(Thread.java:595)
> Caused by: java.io.FileNotFoundException: File does not exist
> /slaveIndexData/publicGalleryTagDef/index.20100115103415/_al.fdx
> at org.apache.solr.common.util.FileUtils.sync(FileUtils.java:55)
> at
> org.apache.solr.handler.SnapPuller$FileFetcher$1.run(SnapPuller.java:911)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
> at java.util.concurrent.FutureTask.run(FutureTask.java:123)
> ... 3 more
> Jan 15, 2010 10:34:17 AM org.apache.solr.core.SolrCore execute
> INFO: [publicGalleryPostMaster] webapp=/multicore path=/select
> params={wt=javabin&rows=10&start=0&sort=createTime_dt+desc&q=%2B(profileId_s:/community/sfly/publicprofile/0AcM27Nw3aNWLi4)+%2Bstate_s:A&version=1}
> hits=1 status=0 QTime=1
>
> B) LockReleaseFailedException
>
> SEVERE: SnapPull failed
> org.apache.solr.common.SolrException: Index fetch failed :
> at
> org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:329)
> at
> org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:264)
> at org.apache.solr.handler.SnapPuller$1.run(SnapPuller.java:159)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417)
> at
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:280)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:135)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:65)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:142)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:166)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
> at java.lang.Thread.run(Thread.java:595)
> Caused by: org.apache.lucene.store.LockReleaseFailedException: failed to delete
> /slaveIndexData/publicGalleryTagDefAggregate/index/lucene-fb30bdbbdc6927666873dd616884ba29-write.lock
> at
> org.apache.lucene.store.NativeFSLock.release(NativeFSLockFactory.java:298)
> at
> org.apache.lucene.index.IndexWriter.closeInternal(IndexWriter.java:2225)
> at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:2153)
> at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:2117)
> at
> org.apache.solr.update.SolrIndexWriter.close(SolrIndexWriter.java:229)
> at
> org.apache.solr.update.DirectUpdateHandler2.closeWriter(DirectUpdateHandler2.java:181)
> at
> org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:409)
> at org.apache.solr.handler.SnapPuller.doCommit(SnapPuller.java:467)
> at
> org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:319)
> ... 11 more
> Jan 15, 2010 12:21:18 AM org.apache.solr.handler.SnapPuller fetchLatestIndex
> INFO: Slave in sync with master.
>
> C) EOF Exception
> INFO: [publicGalleryPostMaster] webapp=/multicore path=/select
> params={wt=javabin&rows=1&start=0&sort=createTime_dt+desc&q=%2B(profileId_s:/community/sfly/publicprofile/0AbOWLNszaOWTiw)+%2B(lastBookmarked_dt:[2010-01-08T08:49:38.271Z+TO+2010-01-15T08:49:38.271Z]+lastCommented_dt:[2010-01-08T08:49:38.271Z+TO+2010-01-15T08:49:38.271Z])+%2Bstate_s:A&version=1}
> hits=0 status=0 QTime=2
> Jan 15, 2010 12:49:42 AM org.apache.solr.common.SolrException log
> SEVERE: java.io.IOException: read past EOF
> at
> org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:151)
> at
> org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:38)
> at org.apache.lucene.store.IndexInput.readVInt(IndexInput.java:80)
> at org.apache.lucene.index.TermBuffer.read(TermBuffer.java:64)
> at
> org.apache.lucene.index.SegmentTermEnum.next(SegmentTermEnum.java:129)
> at
> org.apache.lucene.index.SegmentTermEnum.scanTo(SegmentTermEnum.java:160)
> at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:232)
> at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:179)
> at org.apache.lucene.index.SegmentReader.docFreq(SegmentReader.java:975)
> at
> org.apache.lucene.index.DirectoryReader.docFreq(DirectoryReader.java:627)
> at
> org.apache.solr.search.SolrIndexReader.docFreq(SolrIndexReader.java:308)
> at
> org.apache.lucene.search.IndexSearcher.docFreq(IndexSearcher.java:147)
> at org.apache.lucene.search.Similarity.idfExplain(Similarity.java:833)
> at
> org.apache.lucene.search.PhraseQuery$PhraseWeight.(PhraseQuery.java:122)
> at
> org.apache.lucene.search.PhraseQuery.createWeight(PhraseQuery.java:250)
> at
> org.apache.lucene.search.BooleanQuery$BooleanWeight.(BooleanQuery.java:184)
> at
> org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:415)
> at org.apache.lucene.search.Query.weight(Query.java:99)
> at org.apache.lucene.search.Searcher.createWeight(Searcher.java:230)
> at org.apache.lucene.search.Searcher.search(Searcher.java:171)
> at
> org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:988)
> at
> org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:884)
> at
> org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:341)
> at
> org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:182)
> at
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:195)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316)
> at
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
> at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
> at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
> at
> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:541)
> at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
> at
> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)
> at
> org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664)
> at
> org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
> at
> org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:80)
> at
> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
> at java.lang.Thread.run(Thread.java:595)
>
> Thanks a lot!
>
> Osborn
Re: Index Courruption after replication by new Solr 1.4 Replication
Posted by Otis Gospodnetic <ot...@yahoo.com>.
This is not a direct answer to your question, but can you avoid NFS? My first guess would be that NFS somehow causes this problem. If you check the ML archives for: NFS lock , you will see what I mean.
Otis
--
Sematext -- http://sematext.com/ -- Solr - Lucene - Nutch
----- Original Message ----
> From: Osborn Chan <oc...@shutterfly.com>
> To: "solr-user@lucene.apache.org" <so...@lucene.apache.org>
> Sent: Fri, January 15, 2010 3:23:21 PM
> Subject: Index Courruption after replication by new Solr 1.4 Replication
>
> Hi all,
>
> I have migrated new Solr 1.4 Replication feature with multicore support from
> Solr 1.2 with NFS mounting recently. The following exceptions are in
> catalina.log from time to time, and there are some EOF exceptions which I
> believe the slave index files are corrupted after replication from index server.
> I have following configuration with Solr 1.4, please correct me if it is
> configured incorrectly.
>
> (The index files are not corrupted in master servers, but it is corrupted in
> slave servers. Usually only one of the slave servers are corrupted with EOF
> exception, but not all.)
>
> 1 Master Server: (Index Server)
> - 8 indexes with multicore configuration.
> - All indexes are configured to "replicateAfter" optimize only.
> - The size of index data are vary. The smallest index only have 2.5 MB. The
> biggest index have ~ 100 MB.
> - There would be infrequent optimize calls to indexes. (a optimize call
> every ~30 mins to 6 hours depending on indexes).
> - There are many commit calls to all indexes. (But there is no concurrent
> commit and optimize for all indexes.)
> - Did not configure "commitReserveDuration" in ReplicationHandler - Using
> default values.
>
> 4 Slave Servers (Search Server)
> - 8 indexes with multicore configuration.
> - All indexes are configured to poll for every ~15 minutes.
> - All update handler configuration are removed in solrconfig-slave.xml
> (solrconfig.xml) in order to prevent add/commit/optimize calls.
> - (Search Slave Servers are only responsible for search operation.)
> - removed.
> -
> removed.
> -
> class="solr.BinaryUpdateRequestHandler" /> removed.
>
> A) FileNotFoundException
>
> INFO: Total time taken for download : 1 secs
> Jan 15, 2010 10:34:16 AM org.apache.solr.handler.ReplicationHandler doFetch
> SEVERE: SnapPull failed
> org.apache.solr.common.SolrException: Index fetch failed :
> at
> org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:329)
> at
> org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:264)
> at org.apache.solr.handler.SnapPuller$1.run(SnapPuller.java:159)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417)
> at
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:280)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:135)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:65)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:142)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:166)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
> at java.lang.Thread.run(Thread.java:595)
> Caused by: java.io.FileNotFoundException: File does not exist
> /slaveIndexData/publicGalleryTagDef/index.20100115103415/_al.fdx
> at org.apache.solr.common.util.FileUtils.sync(FileUtils.java:55)
> at
> org.apache.solr.handler.SnapPuller$FileFetcher$1.run(SnapPuller.java:911)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
> at java.util.concurrent.FutureTask.run(FutureTask.java:123)
> ... 3 more
> Jan 15, 2010 10:34:17 AM org.apache.solr.core.SolrCore execute
> INFO: [publicGalleryPostMaster] webapp=/multicore path=/select
> params={wt=javabin&rows=10&start=0&sort=createTime_dt+desc&q=%2B(profileId_s:/community/sfly/publicprofile/0AcM27Nw3aNWLi4)+%2Bstate_s:A&version=1}
> hits=1 status=0 QTime=1
>
> B) LockReleaseFailedException
>
> SEVERE: SnapPull failed
> org.apache.solr.common.SolrException: Index fetch failed :
> at
> org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:329)
> at
> org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:264)
> at org.apache.solr.handler.SnapPuller$1.run(SnapPuller.java:159)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417)
> at
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:280)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:135)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:65)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:142)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:166)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
> at java.lang.Thread.run(Thread.java:595)
> Caused by: org.apache.lucene.store.LockReleaseFailedException: failed to delete
> /slaveIndexData/publicGalleryTagDefAggregate/index/lucene-fb30bdbbdc6927666873dd616884ba29-write.lock
> at
> org.apache.lucene.store.NativeFSLock.release(NativeFSLockFactory.java:298)
> at
> org.apache.lucene.index.IndexWriter.closeInternal(IndexWriter.java:2225)
> at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:2153)
> at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:2117)
> at
> org.apache.solr.update.SolrIndexWriter.close(SolrIndexWriter.java:229)
> at
> org.apache.solr.update.DirectUpdateHandler2.closeWriter(DirectUpdateHandler2.java:181)
> at
> org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:409)
> at org.apache.solr.handler.SnapPuller.doCommit(SnapPuller.java:467)
> at
> org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:319)
> ... 11 more
> Jan 15, 2010 12:21:18 AM org.apache.solr.handler.SnapPuller fetchLatestIndex
> INFO: Slave in sync with master.
>
> C) EOF Exception
> INFO: [publicGalleryPostMaster] webapp=/multicore path=/select
> params={wt=javabin&rows=1&start=0&sort=createTime_dt+desc&q=%2B(profileId_s:/community/sfly/publicprofile/0AbOWLNszaOWTiw)+%2B(lastBookmarked_dt:[2010-01-08T08:49:38.271Z+TO+2010-01-15T08:49:38.271Z]+lastCommented_dt:[2010-01-08T08:49:38.271Z+TO+2010-01-15T08:49:38.271Z])+%2Bstate_s:A&version=1}
> hits=0 status=0 QTime=2
> Jan 15, 2010 12:49:42 AM org.apache.solr.common.SolrException log
> SEVERE: java.io.IOException: read past EOF
> at
> org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:151)
> at
> org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:38)
> at org.apache.lucene.store.IndexInput.readVInt(IndexInput.java:80)
> at org.apache.lucene.index.TermBuffer.read(TermBuffer.java:64)
> at
> org.apache.lucene.index.SegmentTermEnum.next(SegmentTermEnum.java:129)
> at
> org.apache.lucene.index.SegmentTermEnum.scanTo(SegmentTermEnum.java:160)
> at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:232)
> at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:179)
> at org.apache.lucene.index.SegmentReader.docFreq(SegmentReader.java:975)
> at
> org.apache.lucene.index.DirectoryReader.docFreq(DirectoryReader.java:627)
> at
> org.apache.solr.search.SolrIndexReader.docFreq(SolrIndexReader.java:308)
> at
> org.apache.lucene.search.IndexSearcher.docFreq(IndexSearcher.java:147)
> at org.apache.lucene.search.Similarity.idfExplain(Similarity.java:833)
> at
> org.apache.lucene.search.PhraseQuery$PhraseWeight.(PhraseQuery.java:122)
> at
> org.apache.lucene.search.PhraseQuery.createWeight(PhraseQuery.java:250)
> at
> org.apache.lucene.search.BooleanQuery$BooleanWeight.(BooleanQuery.java:184)
> at
> org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:415)
> at org.apache.lucene.search.Query.weight(Query.java:99)
> at org.apache.lucene.search.Searcher.createWeight(Searcher.java:230)
> at org.apache.lucene.search.Searcher.search(Searcher.java:171)
> at
> org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:988)
> at
> org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:884)
> at
> org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:341)
> at
> org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:182)
> at
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:195)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316)
> at
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
> at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
> at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
> at
> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:541)
> at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
> at
> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)
> at
> org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664)
> at
> org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
> at
> org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:80)
> at
> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
> at java.lang.Thread.run(Thread.java:595)
>
> Thanks a lot!
>
> Osborn