You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Prakhar Birla <pr...@gmail.com> on 2013/02/23 09:54:27 UTC
Solr 4.0 committing on an index built by another instance
Hi,
We use Solr 4.0 for our main searcher so it is a very vital part. We have
set up a process called Index reassurance which assures that all documents
are available in Solr by comparing to our database. In short this is
achieved as: The production server (read/write) is a slave while another
server (write only) is the master where the indexes are built and
replicated to the slave.
We are adding a backup/restore feature to this process which means that the
backup can originate from either server (while it is running) and will be
applied to the master after which the indexes are built and replicated.
The backup is a tar.gz copy of the core.
The problem we are facing is when a commit is done on data loaded from the
backup. Following is the stack trace:
SEVERE: null:java.io.FileNotFoundException: /var/www/locationapp7078/solr/
> collection1/data/index.20130222154157971/_18.fnm (No such file or
> directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
> at org.apache.lucene.store.MMapDirectory.openInput(
> MMapDirectory.java:222)
> at org.apache.lucene.store.NRTCachingDirectory.openInput(
> NRTCachingDirectory.java:232)
> at org.apache.lucene.codecs.lucene40.Lucene40FieldInfosReader.read(
> Lucene40FieldInfosReader.java:52)
> at org.apache.lucene.index.SegmentCoreReaders.<init>(
> SegmentCoreReaders.java:101)
> at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:57)
> at org.apache.lucene.index.ReadersAndLiveDocs.getReader(
> ReadersAndLiveDocs.java:120)
> at org.apache.lucene.index.BufferedDeletesStream.applyDeletes(
> BufferedDeletesStream.java:267)
> at org.apache.lucene.index.IndexWriter.applyAllDeletes(
> IndexWriter.java:3010)
> at org.apache.lucene.index.IndexWriter.maybeApplyDeletes(
> IndexWriter.java:3001)
> at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:2974)
> at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:2943)
> at org.apache.lucene.index.IndexWriter.forceMerge(
> IndexWriter.java:1606)
> at org.apache.lucene.index.IndexWriter.forceMerge(
> IndexWriter.java:1582)
> at org.apache.solr.update.DirectUpdateHandler2.commit(
> DirectUpdateHandler2.java:515)
> at org.apache.solr.update.processor.RunUpdateProcessor.processCommit(
> RunUpdateProcessorFactory.java:87)
> at org.apache.solr.update.processor.UpdateRequestProcessor.
> processCommit(UpdateRequestProcessor.java:64)
> at org.apache.solr.update.processor.DistributedUpdateProcessor.
> processCommit(DistributedUpdateProcessor.java:1007)
> at org.apache.solr.update.processor.LogUpdateProcessor.processCommit(
> LogUpdateProcessorFactory.java:157)
> at org.apache.solr.handler.loader.XMLLoader.
> processUpdate(XMLLoader.java:250)
> at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:157)
> at org.apache.solr.handler.UpdateRequestHandler$1.load(
> UpdateRequestHandler.java:92)
> at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(
> ContentStreamHandlerBase.java:74)
> at org.apache.solr.handler.RequestHandlerBase.handleRequest(
> RequestHandlerBase.java:129)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1699)
> at org.apache.solr.servlet.SolrDispatchFilter.execute(
> SolrDispatchFilter.java:455)
> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(
> SolrDispatchFilter.java:276)
> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.
> doFilter(ServletHandler.java:1337)
> at org.eclipse.jetty.servlet.ServletHandler.doHandle(
> ServletHandler.java:484)
> at org.eclipse.jetty.server.handler.ScopedHandler.handle(
> ScopedHandler.java:119)
> at org.eclipse.jetty.security.SecurityHandler.handle(
> SecurityHandler.java:524)
> at org.eclipse.jetty.server.session.SessionHandler.
> doHandle(SessionHandler.java:233)
> at org.eclipse.jetty.server.handler.ContextHandler.
> doHandle(ContextHandler.java:1065)
> at org.eclipse.jetty.servlet.ServletHandler.doScope(
> ServletHandler.java:413)
> at org.eclipse.jetty.server.session.SessionHandler.
> doScope(SessionHandler.java:192)
> at org.eclipse.jetty.server.handler.ContextHandler.
> doScope(ContextHandler.java:999)
> at org.eclipse.jetty.server.handler.ScopedHandler.handle(
> ScopedHandler.java:117)
> at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(
> ContextHandlerCollection.java:250)
> at org.eclipse.jetty.server.handler.HandlerCollection.
> handle(HandlerCollection.java:149)
> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(
> HandlerWrapper.java:111)
> at org.eclipse.jetty.server.Server.handle(Server.java:351)
> at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(
> AbstractHttpConnection.java:454)
> at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(
> BlockingHttpConnection.java:47)
> at org.eclipse.jetty.server.AbstractHttpConnection.content(
> AbstractHttpConnection.java:900)
> at org.eclipse.jetty.server.AbstractHttpConnection$
> RequestHandler.content(AbstractHttpConnection.java:954)
> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:857)
> at org.eclipse.jetty.http.HttpParser.parseAvailable(
> HttpParser.java:235)
> at org.eclipse.jetty.server.BlockingHttpConnection.handle(
> BlockingHttpConnection.java:66)
> at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(
> SocketConnector.java:254)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(
> QueuedThreadPool.java:599)
> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(
> QueuedThreadPool.java:534)
> at java.lang.Thread.run(Thread.java:679)
Thanks!
--
Regards,
Prakhar Birla
Re: Solr 4.0 committing on an index built by another instance
Posted by Prakhar Birla <pr...@gmail.com>.
Hi Mark,
Thanks for the reply. The issue happened because the server kill/restart
process failed which meant the old instance was still running, and it had
the data related to old files loaded into memory. And when commit was
issued, it was trying to locate those old files and failing. Once this was
fixed, it's running like a charm!
On 23 February 2013 20:18, Mark Miller <ma...@gmail.com> wrote:
> How are you doing the backup? You have to coordinate with Solr - files may
> be changing when you try and copy it, leaving to an inconsistent index. If
> you want to do a live backup, you have to use the backup feature of the
> replication handler.
>
> - Mark
>
> On Feb 23, 2013, at 3:54 AM, Prakhar Birla <pr...@gmail.com> wrote:
>
> > Hi,
> >
> > We use Solr 4.0 for our main searcher so it is a very vital part. We have
> > set up a process called Index reassurance which assures that all
> documents
> > are available in Solr by comparing to our database. In short this is
> > achieved as: The production server (read/write) is a slave while another
> > server (write only) is the master where the indexes are built and
> > replicated to the slave.
> >
> > We are adding a backup/restore feature to this process which means that
> the
> > backup can originate from either server (while it is running) and will be
> > applied to the master after which the indexes are built and replicated.
> >
> > The backup is a tar.gz copy of the core.
> >
> > The problem we are facing is when a commit is done on data loaded from
> the
> > backup. Following is the stack trace:
> >
> > SEVERE: null:java.io.FileNotFoundException:
> /var/www/locationapp7078/solr/
> >> collection1/data/index.20130222154157971/_18.fnm (No such file or
> >> directory)
> >> at java.io.RandomAccessFile.open(Native Method)
> >> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
> >> at org.apache.lucene.store.MMapDirectory.openInput(
> >> MMapDirectory.java:222)
> >> at org.apache.lucene.store.NRTCachingDirectory.openInput(
> >> NRTCachingDirectory.java:232)
> >> at org.apache.lucene.codecs.lucene40.Lucene40FieldInfosReader.read(
> >> Lucene40FieldInfosReader.java:52)
> >> at org.apache.lucene.index.SegmentCoreReaders.<init>(
> >> SegmentCoreReaders.java:101)
> >> at
> org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:57)
> >> at org.apache.lucene.index.ReadersAndLiveDocs.getReader(
> >> ReadersAndLiveDocs.java:120)
> >> at org.apache.lucene.index.BufferedDeletesStream.applyDeletes(
> >> BufferedDeletesStream.java:267)
> >> at org.apache.lucene.index.IndexWriter.applyAllDeletes(
> >> IndexWriter.java:3010)
> >> at org.apache.lucene.index.IndexWriter.maybeApplyDeletes(
> >> IndexWriter.java:3001)
> >> at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:2974)
> >> at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:2943)
> >> at org.apache.lucene.index.IndexWriter.forceMerge(
> >> IndexWriter.java:1606)
> >> at org.apache.lucene.index.IndexWriter.forceMerge(
> >> IndexWriter.java:1582)
> >> at org.apache.solr.update.DirectUpdateHandler2.commit(
> >> DirectUpdateHandler2.java:515)
> >> at org.apache.solr.update.processor.RunUpdateProcessor.processCommit(
> >> RunUpdateProcessorFactory.java:87)
> >> at org.apache.solr.update.processor.UpdateRequestProcessor.
> >> processCommit(UpdateRequestProcessor.java:64)
> >> at org.apache.solr.update.processor.DistributedUpdateProcessor.
> >> processCommit(DistributedUpdateProcessor.java:1007)
> >> at org.apache.solr.update.processor.LogUpdateProcessor.processCommit(
> >> LogUpdateProcessorFactory.java:157)
> >> at org.apache.solr.handler.loader.XMLLoader.
> >> processUpdate(XMLLoader.java:250)
> >> at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:157)
> >> at org.apache.solr.handler.UpdateRequestHandler$1.load(
> >> UpdateRequestHandler.java:92)
> >> at
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(
> >> ContentStreamHandlerBase.java:74)
> >> at org.apache.solr.handler.RequestHandlerBase.handleRequest(
> >> RequestHandlerBase.java:129)
> >> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1699)
> >> at org.apache.solr.servlet.SolrDispatchFilter.execute(
> >> SolrDispatchFilter.java:455)
> >> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(
> >> SolrDispatchFilter.java:276)
> >> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.
> >> doFilter(ServletHandler.java:1337)
> >> at org.eclipse.jetty.servlet.ServletHandler.doHandle(
> >> ServletHandler.java:484)
> >> at org.eclipse.jetty.server.handler.ScopedHandler.handle(
> >> ScopedHandler.java:119)
> >> at org.eclipse.jetty.security.SecurityHandler.handle(
> >> SecurityHandler.java:524)
> >> at org.eclipse.jetty.server.session.SessionHandler.
> >> doHandle(SessionHandler.java:233)
> >> at org.eclipse.jetty.server.handler.ContextHandler.
> >> doHandle(ContextHandler.java:1065)
> >> at org.eclipse.jetty.servlet.ServletHandler.doScope(
> >> ServletHandler.java:413)
> >> at org.eclipse.jetty.server.session.SessionHandler.
> >> doScope(SessionHandler.java:192)
> >> at org.eclipse.jetty.server.handler.ContextHandler.
> >> doScope(ContextHandler.java:999)
> >> at org.eclipse.jetty.server.handler.ScopedHandler.handle(
> >> ScopedHandler.java:117)
> >> at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(
> >> ContextHandlerCollection.java:250)
> >> at org.eclipse.jetty.server.handler.HandlerCollection.
> >> handle(HandlerCollection.java:149)
> >> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(
> >> HandlerWrapper.java:111)
> >> at org.eclipse.jetty.server.Server.handle(Server.java:351)
> >> at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(
> >> AbstractHttpConnection.java:454)
> >> at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(
> >> BlockingHttpConnection.java:47)
> >> at org.eclipse.jetty.server.AbstractHttpConnection.content(
> >> AbstractHttpConnection.java:900)
> >> at org.eclipse.jetty.server.AbstractHttpConnection$
> >> RequestHandler.content(AbstractHttpConnection.java:954)
> >> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:857)
> >> at org.eclipse.jetty.http.HttpParser.parseAvailable(
> >> HttpParser.java:235)
> >> at org.eclipse.jetty.server.BlockingHttpConnection.handle(
> >> BlockingHttpConnection.java:66)
> >> at
> org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(
> >> SocketConnector.java:254)
> >> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(
> >> QueuedThreadPool.java:599)
> >> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(
> >> QueuedThreadPool.java:534)
> >> at java.lang.Thread.run(Thread.java:679)
> >
> >
> > Thanks!
> >
> > --
> > Regards,
> > Prakhar Birla
>
>
--
Regards,
Prakhar Birla
+91 9739868086
Re: Solr 4.0 committing on an index built by another instance
Posted by Mark Miller <ma...@gmail.com>.
How are you doing the backup? You have to coordinate with Solr - files may be changing when you try and copy it, leaving to an inconsistent index. If you want to do a live backup, you have to use the backup feature of the replication handler.
- Mark
On Feb 23, 2013, at 3:54 AM, Prakhar Birla <pr...@gmail.com> wrote:
> Hi,
>
> We use Solr 4.0 for our main searcher so it is a very vital part. We have
> set up a process called Index reassurance which assures that all documents
> are available in Solr by comparing to our database. In short this is
> achieved as: The production server (read/write) is a slave while another
> server (write only) is the master where the indexes are built and
> replicated to the slave.
>
> We are adding a backup/restore feature to this process which means that the
> backup can originate from either server (while it is running) and will be
> applied to the master after which the indexes are built and replicated.
>
> The backup is a tar.gz copy of the core.
>
> The problem we are facing is when a commit is done on data loaded from the
> backup. Following is the stack trace:
>
> SEVERE: null:java.io.FileNotFoundException: /var/www/locationapp7078/solr/
>> collection1/data/index.20130222154157971/_18.fnm (No such file or
>> directory)
>> at java.io.RandomAccessFile.open(Native Method)
>> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
>> at org.apache.lucene.store.MMapDirectory.openInput(
>> MMapDirectory.java:222)
>> at org.apache.lucene.store.NRTCachingDirectory.openInput(
>> NRTCachingDirectory.java:232)
>> at org.apache.lucene.codecs.lucene40.Lucene40FieldInfosReader.read(
>> Lucene40FieldInfosReader.java:52)
>> at org.apache.lucene.index.SegmentCoreReaders.<init>(
>> SegmentCoreReaders.java:101)
>> at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:57)
>> at org.apache.lucene.index.ReadersAndLiveDocs.getReader(
>> ReadersAndLiveDocs.java:120)
>> at org.apache.lucene.index.BufferedDeletesStream.applyDeletes(
>> BufferedDeletesStream.java:267)
>> at org.apache.lucene.index.IndexWriter.applyAllDeletes(
>> IndexWriter.java:3010)
>> at org.apache.lucene.index.IndexWriter.maybeApplyDeletes(
>> IndexWriter.java:3001)
>> at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:2974)
>> at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:2943)
>> at org.apache.lucene.index.IndexWriter.forceMerge(
>> IndexWriter.java:1606)
>> at org.apache.lucene.index.IndexWriter.forceMerge(
>> IndexWriter.java:1582)
>> at org.apache.solr.update.DirectUpdateHandler2.commit(
>> DirectUpdateHandler2.java:515)
>> at org.apache.solr.update.processor.RunUpdateProcessor.processCommit(
>> RunUpdateProcessorFactory.java:87)
>> at org.apache.solr.update.processor.UpdateRequestProcessor.
>> processCommit(UpdateRequestProcessor.java:64)
>> at org.apache.solr.update.processor.DistributedUpdateProcessor.
>> processCommit(DistributedUpdateProcessor.java:1007)
>> at org.apache.solr.update.processor.LogUpdateProcessor.processCommit(
>> LogUpdateProcessorFactory.java:157)
>> at org.apache.solr.handler.loader.XMLLoader.
>> processUpdate(XMLLoader.java:250)
>> at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:157)
>> at org.apache.solr.handler.UpdateRequestHandler$1.load(
>> UpdateRequestHandler.java:92)
>> at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(
>> ContentStreamHandlerBase.java:74)
>> at org.apache.solr.handler.RequestHandlerBase.handleRequest(
>> RequestHandlerBase.java:129)
>> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1699)
>> at org.apache.solr.servlet.SolrDispatchFilter.execute(
>> SolrDispatchFilter.java:455)
>> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(
>> SolrDispatchFilter.java:276)
>> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.
>> doFilter(ServletHandler.java:1337)
>> at org.eclipse.jetty.servlet.ServletHandler.doHandle(
>> ServletHandler.java:484)
>> at org.eclipse.jetty.server.handler.ScopedHandler.handle(
>> ScopedHandler.java:119)
>> at org.eclipse.jetty.security.SecurityHandler.handle(
>> SecurityHandler.java:524)
>> at org.eclipse.jetty.server.session.SessionHandler.
>> doHandle(SessionHandler.java:233)
>> at org.eclipse.jetty.server.handler.ContextHandler.
>> doHandle(ContextHandler.java:1065)
>> at org.eclipse.jetty.servlet.ServletHandler.doScope(
>> ServletHandler.java:413)
>> at org.eclipse.jetty.server.session.SessionHandler.
>> doScope(SessionHandler.java:192)
>> at org.eclipse.jetty.server.handler.ContextHandler.
>> doScope(ContextHandler.java:999)
>> at org.eclipse.jetty.server.handler.ScopedHandler.handle(
>> ScopedHandler.java:117)
>> at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(
>> ContextHandlerCollection.java:250)
>> at org.eclipse.jetty.server.handler.HandlerCollection.
>> handle(HandlerCollection.java:149)
>> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(
>> HandlerWrapper.java:111)
>> at org.eclipse.jetty.server.Server.handle(Server.java:351)
>> at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(
>> AbstractHttpConnection.java:454)
>> at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(
>> BlockingHttpConnection.java:47)
>> at org.eclipse.jetty.server.AbstractHttpConnection.content(
>> AbstractHttpConnection.java:900)
>> at org.eclipse.jetty.server.AbstractHttpConnection$
>> RequestHandler.content(AbstractHttpConnection.java:954)
>> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:857)
>> at org.eclipse.jetty.http.HttpParser.parseAvailable(
>> HttpParser.java:235)
>> at org.eclipse.jetty.server.BlockingHttpConnection.handle(
>> BlockingHttpConnection.java:66)
>> at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(
>> SocketConnector.java:254)
>> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(
>> QueuedThreadPool.java:599)
>> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(
>> QueuedThreadPool.java:534)
>> at java.lang.Thread.run(Thread.java:679)
>
>
> Thanks!
>
> --
> Regards,
> Prakhar Birla