You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Bill Graham <em...@yahoo.com> on 2008/10/30 07:06:21 UTC
corrupt solr index on ec2
Hi,
I've been running solr 1.3 on an ec2 instance for a couple of weeks and I've had some stability issues. It seems like I need to bounce the app once a day. That I could live with and ultimately maybe troubleshoot, but what's more disturbing is that three times in the last 2 weeks my index has been corrupted when FileNotFoundExceptions started to appear.
I'm running in jetty and had my index on the local file system until I lost the index the first time. Then I moved it to my mounted ebs volume so I could restore from a snapshot if needed. I'm wondering if perhaps there are issues with the locking mechanize on either the local directory (which is really a virual instance), or the mounted xfs volume. Has anyone seem this, or have suggestions re the cause? I'm using the single lockType.
I'm running a single solr instance that gets frequent updates from multiple threads, and commits about every hour.
A few things I see in the logs:
- From time to time I see write lock timeouts:
SEVERE: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: SingleInstanceLock: write.lock
- I've seen OOM exceptions during warming. I've changed maxWarmingSearchers=1, which I suspect will do he trick
- The finally, this is what I fond in the logs today when the index got corrupt:
Oct 29, 2008 12:18:39 AM org.apache.solr.update.DirectUpdateHandler2 commit
INFO: start commit(optimize=false,waitFlush=true,waitSearcher=true)
Oct 29, 2008 12:18:41 AM org.apache.solr.common.SolrException log
SEVERE: java.lang.RuntimeException: java.io.FileNotFoundException: /var/local/solr/data/production/index/_2rv.fdt (No such file or directory)
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:960)
at org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:368)
at org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:77)
at org.apache.solr.handler.XmlUpdateRequestHandler.processUpdate(XmlUpdateRequestHandler.java:226)
at org.apache.solr.handler.XmlUpdateRequestHandler.handleRequestBody(XmlUpdateRequestHandler.java:123)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1204)
at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:303)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:232)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1089)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:365)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:712)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:211)
at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:139)
at org.mortbay.jetty.Server.handle(Server.java:285)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:502)
at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:835)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:641)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:202)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:378)
at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:226)
at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442)
Caused by: java.io.FileNotFoundException: /var/local/solr/data/production/index/_2rv.fdt (No such file or directory)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
at org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.<init>(FSDirectory.java:552)
at org.apache.lucene.store.FSDirectory$FSIndexInput.<init>(FSDirectory.java:582)
at org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:488)
at org.apache.lucene.index.FieldsReader.<init>(FieldsReader.java:77)
at org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:355)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:304)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:226)
at org.apache.lucene.index.MultiSegmentReader.<init>(MultiSegmentReader.java:56)
at org.apache.lucene.index.ReadOnlyMultiSegmentReader.<init>(ReadOnlyMultiSegmentReader.java:27)
at org.apache.lucene.index.DirectoryIndexReader$1.doBody(DirectoryIndexReader.java:101)
at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:653)
at org.apache.lucene.index.DirectoryIndexReader.open(DirectoryIndexReader.java:111)
at org.apache.lucene.index.IndexReader.open(IndexReader.java:316)
at org.apache.lucene.index.IndexReader.open(IndexReader.java:237)
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:951)
... 25 more
Oct 29, 2008 12:18:41 AM org.apache.solr.core.SolrCore execute
INFO: [] webapp=/solr path=/update params={} status=500 QTime=1149
Oct 29, 2008 12:18:41 AM org.apache.solr.common.SolrException log
SEVERE: java.lang.RuntimeException: java.io.FileNotFoundException: /var/local/solr/data/production/index/_2rv.fdt (No such file or directory)
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:960)
at org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:368)
at org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:77)
at org.apache.solr.handler.XmlUpdateRequestHandler.processUpdate(XmlUpdateRequestHandler.java:226)
at org.apache.solr.handler.XmlUpdateRequestHandler.handleRequestBody(XmlUpdateRequestHandler.java:123)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1204)
at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:303)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:232)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1089)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:365)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:712)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:211)
at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:139)
at org.mortbay.jetty.Server.handle(Server.java:285)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:502)
at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:835)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:641)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:202)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:378)
at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:226)
at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442)
Caused by: java.io.FileNotFoundException: /var/local/solr/data/production/index/_2rv.fdt (No such file or directory)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
at org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.<init>(FSDirectory.java:552)
at org.apache.lucene.store.FSDirectory$FSIndexInput.<init>(FSDirectory.java:582)
at org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:488)
at org.apache.lucene.index.FieldsReader.<init>(FieldsReader.java:77)
at org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:355)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:304)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:226)
at org.apache.lucene.index.MultiSegmentReader.<init>(MultiSegmentReader.java:56)
at org.apache.lucene.index.ReadOnlyMultiSegmentReader.<init>(ReadOnlyMultiSegmentReader.java:27)
at org.apache.lucene.index.DirectoryIndexReader$1.doBody(DirectoryIndexReader.java:101)
at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:653)
at org.apache.lucene.index.DirectoryIndexReader.open(DirectoryIndexReader.java:111)
at org.apache.lucene.index.IndexReader.open(IndexReader.java:316)
at org.apache.lucene.index.IndexReader.open(IndexReader.java:237)
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:951)
... 25 more
[ then things look ok for a while, but these exceptions start to appear ]
Oct 29, 2008 12:41:45 AM org.apache.solr.core.SolrCore execute
INFO: [] webapp=/solr path=/update params={} status=0 QTime=1291
Exception in thread "Lucene Merge Thread #0" org.apache.lucene.index.MergePolicy$MergeException: java.io.FileNotFoundException: /var/local/solr/data/production/index/_2rv.fdt (No such file or directory)
at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:314)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:291)
Caused by: java.io.FileNotFoundException: /var/local/solr/data/production/index/_2rv.fdt (No such file or directory)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
at org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.<init>(FSDirectory.java:552)
at org.apache.lucene.store.FSDirectory$FSIndexInput.<init>(FSDirectory.java:582)
at org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:488)
at org.apache.lucene.index.FieldsReader.<init>(FieldsReader.java:77)
at org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:355)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:304)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:258)
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4280)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3932)
at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:205)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:260)
Any ideas?
thanks,
Bill
Re: corrupt solr index on ec2
Posted by Michael McCandless <lu...@mikemccandless.com>.
One small correction below:
Yonik Seeley wrote:
>> - I've seen OOM exceptions during warming. I've changed
>> maxWarmingSearchers=1, which I suspect will do he trick
>
> OOM errors are really tricky - if they happen in the wrong place, it's
> hard to recover gracefully from. Correctly cleaning up after an OOM
> error in the IndexWriter recently had some little fixes in lucene
> trunk - you might want to try the latest dev version of Lucene and see
> if it helps.
This change (to not commit index changes after IndexWriter hits OOME)
went in Feb 2008. Solr 1.3 should already have it.
(I'm working now on adding javadocs to IW explaining this).
Mike
Re: corrupt solr index on ec2
Posted by Yonik Seeley <yo...@apache.org>.
On Thu, Oct 30, 2008 at 2:06 AM, Bill Graham <em...@yahoo.com> wrote:
> I've been running solr 1.3 on an ec2 instance for a couple of weeks and I've had some stability issues. It seems like I need to bounce the app once a day. That I could live with and ultimately maybe troubleshoot, but what's more disturbing is that three times in the last 2 weeks my index has been corrupted when FileNotFoundExceptions started to appear.
>
> I'm running in jetty and had my index on the local file system until I lost the index the first time. Then I moved it to my mounted ebs volume so I could restore from a snapshot if needed. I'm wondering if perhaps there are issues with the locking mechanize on either the local directory (which is really a virual instance), or the mounted xfs volume. Has anyone seem this, or have suggestions re the cause? I'm using the single lockType.
>
> I'm running a single solr instance that gets frequent updates from multiple threads, and commits about every hour.
>
> A few things I see in the logs:
>
> - From time to time I see write lock timeouts:
> SEVERE: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: SingleInstanceLock: write.lock
This is really strange. It suggests that there is another in-process
writer that is holding the lock. That should be impossible, unless
it's caused by a previous exception trying to open an IndexWriter and
the lock is simply stale. What seems to be the first exception that
occurs?
Also, you might try changing the lock type from single to simple to
make it visible cross-process. That would rule out trying to start
another solr instance on the same index directory.... opening two
writers on the same directory is one way to get missing files like you
appear to have.
> - I've seen OOM exceptions during warming. I've changed maxWarmingSearchers=1, which I suspect will do he trick
OOM errors are really tricky - if they happen in the wrong place, it's
hard to recover gracefully from. Correctly cleaning up after an OOM
error in the IndexWriter recently had some little fixes in lucene
trunk - you might want to try the latest dev version of Lucene and see
if it helps.
-Yonik
> - The finally, this is what I fond in the logs today when the index got corrupt:
>
> Oct 29, 2008 12:18:39 AM org.apache.solr.update.DirectUpdateHandler2 commit
> INFO: start commit(optimize=false,waitFlush=true,waitSearcher=true)
> Oct 29, 2008 12:18:41 AM org.apache.solr.common.SolrException log
> SEVERE: java.lang.RuntimeException: java.io.FileNotFoundException: /var/local/solr/data/production/index/_2rv.fdt (No such file or directory)
> at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:960)
> at org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:368)
> at org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:77)
> at org.apache.solr.handler.XmlUpdateRequestHandler.processUpdate(XmlUpdateRequestHandler.java:226)
> at org.apache.solr.handler.XmlUpdateRequestHandler.handleRequestBody(XmlUpdateRequestHandler.java:123)
> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1204)
> at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:303)
> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:232)
> at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1089)
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:365)
> at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
> at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:712)
> at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405)
> at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:211)
> at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
> at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:139)
> at org.mortbay.jetty.Server.handle(Server.java:285)
> at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:502)
> at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:835)
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:641)
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:202)
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:378)
> at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:226)
> at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442)
> Caused by: java.io.FileNotFoundException: /var/local/solr/data/production/index/_2rv.fdt (No such file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
> at org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.<init>(FSDirectory.java:552)
> at org.apache.lucene.store.FSDirectory$FSIndexInput.<init>(FSDirectory.java:582)
> at org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:488)
> at org.apache.lucene.index.FieldsReader.<init>(FieldsReader.java:77)
> at org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:355)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:304)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:226)
> at org.apache.lucene.index.MultiSegmentReader.<init>(MultiSegmentReader.java:56)
> at org.apache.lucene.index.ReadOnlyMultiSegmentReader.<init>(ReadOnlyMultiSegmentReader.java:27)
> at org.apache.lucene.index.DirectoryIndexReader$1.doBody(DirectoryIndexReader.java:101)
> at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:653)
> at org.apache.lucene.index.DirectoryIndexReader.open(DirectoryIndexReader.java:111)
> at org.apache.lucene.index.IndexReader.open(IndexReader.java:316)
> at org.apache.lucene.index.IndexReader.open(IndexReader.java:237)
> at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:951)
> ... 25 more
>
>
> Oct 29, 2008 12:18:41 AM org.apache.solr.core.SolrCore execute
> INFO: [] webapp=/solr path=/update params={} status=500 QTime=1149
> Oct 29, 2008 12:18:41 AM org.apache.solr.common.SolrException log
> SEVERE: java.lang.RuntimeException: java.io.FileNotFoundException: /var/local/solr/data/production/index/_2rv.fdt (No such file or directory)
> at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:960)
> at org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:368)
> at org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:77)
> at org.apache.solr.handler.XmlUpdateRequestHandler.processUpdate(XmlUpdateRequestHandler.java:226)
> at org.apache.solr.handler.XmlUpdateRequestHandler.handleRequestBody(XmlUpdateRequestHandler.java:123)
> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1204)
> at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:303)
> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:232)
> at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1089)
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:365)
> at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
> at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:712)
> at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405)
> at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:211)
> at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
> at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:139)
> at org.mortbay.jetty.Server.handle(Server.java:285)
> at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:502)
> at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:835)
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:641)
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:202)
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:378)
> at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:226)
> at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442)
> Caused by: java.io.FileNotFoundException: /var/local/solr/data/production/index/_2rv.fdt (No such file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
> at org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.<init>(FSDirectory.java:552)
> at org.apache.lucene.store.FSDirectory$FSIndexInput.<init>(FSDirectory.java:582)
> at org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:488)
> at org.apache.lucene.index.FieldsReader.<init>(FieldsReader.java:77)
> at org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:355)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:304)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:226)
> at org.apache.lucene.index.MultiSegmentReader.<init>(MultiSegmentReader.java:56)
> at org.apache.lucene.index.ReadOnlyMultiSegmentReader.<init>(ReadOnlyMultiSegmentReader.java:27)
> at org.apache.lucene.index.DirectoryIndexReader$1.doBody(DirectoryIndexReader.java:101)
> at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:653)
> at org.apache.lucene.index.DirectoryIndexReader.open(DirectoryIndexReader.java:111)
> at org.apache.lucene.index.IndexReader.open(IndexReader.java:316)
> at org.apache.lucene.index.IndexReader.open(IndexReader.java:237)
> at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:951)
> ... 25 more
>
> [ then things look ok for a while, but these exceptions start to appear ]
>
> Oct 29, 2008 12:41:45 AM org.apache.solr.core.SolrCore execute
> INFO: [] webapp=/solr path=/update params={} status=0 QTime=1291
> Exception in thread "Lucene Merge Thread #0" org.apache.lucene.index.MergePolicy$MergeException: java.io.FileNotFoundException: /var/local/solr/data/production/index/_2rv.fdt (No such file or directory)
> at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:314)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:291)
> Caused by: java.io.FileNotFoundException: /var/local/solr/data/production/index/_2rv.fdt (No such file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
> at org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.<init>(FSDirectory.java:552)
> at org.apache.lucene.store.FSDirectory$FSIndexInput.<init>(FSDirectory.java:582)
> at org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:488)
> at org.apache.lucene.index.FieldsReader.<init>(FieldsReader.java:77)
> at org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:355)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:304)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:258)
> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4280)
> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3932)
> at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:205)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:260)
>
> Any ideas?
>
> thanks,
> Bill