You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by "Danzig, Scott" <Sc...@nymag.com> on 2013/03/13 17:03:23 UTC
Handling a closed IndexWriter in Solr
Hey all,
We're using a Solr 4 core to handle our article data. When someone in our CMS publishes an article, we have a listener that indexes it straight to solr. We use the previously instantiated HttpSolrServer, build the solr document, add it with server.add(doc) .. then do a server.commit() right away. This works fine, except that meanwhile, another process is importing new data from an SQL db into the same core. For some reason, sometimes this process errors:
Feb 26, 2013 3:05:03 PM org.apache.solr.handler.dataimport.DataImporter doDeltaImport
SEVERE: Delta Import Failed
java.lang.RuntimeException: java.lang.IllegalArgumentException: deltaQuery has no column to resolve to declared primary key pk='id'
at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:273)
at org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:412)
at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:450)
at org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:429)
Caused by: java.lang.IllegalArgumentException: deltaQuery has no column to resolve to declared primary key pk='id'
at org.apache.solr.handler.dataimport.DocBuilder.findMatchingPkColumn(DocBuilder.java:725)
at org.apache.solr.handler.dataimport.DocBuilder.collectDelta(DocBuilder.java:773)
at org.apache.solr.handler.dataimport.DocBuilder.doDelta(DocBuilder.java:341)
at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:226)
... 3 more
Not really relevant to this problem, because I basically want my real time indexing to keep working in spite of this. However, when it's happening, I am pretty sure this is causing my real-time indexing commit to throw this exception:
Feb 26, 2013 5:07:51 PM org.apache.solr.common.SolrException log
SEVERE: null:org.apache.solr.common.SolrException: Error opening new searcher
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1310)
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1422)
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1200)
at org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:560)
at org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:87)
at org.apache.solr.update.processor.UpdateRequestProcessor.processCommit(UpdateRequestProcessor.java:64)
at org.apache.solr.update.processor.DistributedUpdateProcessor.processCommit(DistributedUpdateProcessor.java:1007)
at org.apache.solr.update.processor.LogUpdateProcessor.processCommit(LogUpdateProcessorFactory.java:157)
at org.apache.solr.handler.RequestHandlerUtils.handleCommit(RequestHandlerUtils.java:69)
at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1699)
at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:455)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:276)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:225)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:169)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:927)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:999)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:565)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:309)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:550)
at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:563)
at org.apache.lucene.index.IndexWriter.nrtIsCurrent(IndexWriter.java:4196)
at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:266)
at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:245)
at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:235)
at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:169)
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1256)
... 28 more
Usually, I can get around this by basically creating a new HttpSolrServer and trying again. But if the problem is very frequent, one more try doesn't work. I don't like the idea of continually trying new solr connections until it works. What's the proper way to fix this? I'm assuming adding a doc with a commitWithMs of "0" is not any different than committing right away, but that suggestion was raised as well.
RE: Handling a closed IndexWriter in Solr
Posted by "Dyer, James" <Ja...@ingramcontent.com>.
Scott,
I have no idea how these problems might be related. However, I think you can solve the problem with DIH not being able to find the primary key if you explicitly declare it by specifying the "pk" property on the parent entity.
<entity name="parent" pk="keyName" ... />
You might get more and better resonses to your solr-related question by posting to the solr-user list rather than the java-user list.
James Dyer
Ingram Content Group
(615) 213-4311
-----Original Message-----
From: Danzig, Scott [mailto:Scott.Danzig@nymag.com]
Sent: Wednesday, March 13, 2013 11:03 AM
To: java-user@lucene.apache.org
Subject: Handling a closed IndexWriter in Solr
Importance: Low
Hey all,
We're using a Solr 4 core to handle our article data. When someone in our CMS publishes an article, we have a listener that indexes it straight to solr. We use the previously instantiated HttpSolrServer, build the solr document, add it with server.add(doc) .. then do a server.commit() right away. This works fine, except that meanwhile, another process is importing new data from an SQL db into the same core. For some reason, sometimes this process errors:
Feb 26, 2013 3:05:03 PM org.apache.solr.handler.dataimport.DataImporter doDeltaImport
SEVERE: Delta Import Failed
java.lang.RuntimeException: java.lang.IllegalArgumentException: deltaQuery has no column to resolve to declared primary key pk='id'
at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:273)
at org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:412)
at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:450)
at org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:429)
Caused by: java.lang.IllegalArgumentException: deltaQuery has no column to resolve to declared primary key pk='id'
at org.apache.solr.handler.dataimport.DocBuilder.findMatchingPkColumn(DocBuilder.java:725)
at org.apache.solr.handler.dataimport.DocBuilder.collectDelta(DocBuilder.java:773)
at org.apache.solr.handler.dataimport.DocBuilder.doDelta(DocBuilder.java:341)
at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:226)
... 3 more
Not really relevant to this problem, because I basically want my real time indexing to keep working in spite of this. However, when it's happening, I am pretty sure this is causing my real-time indexing commit to throw this exception:
Feb 26, 2013 5:07:51 PM org.apache.solr.common.SolrException log
SEVERE: null:org.apache.solr.common.SolrException: Error opening new searcher
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1310)
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1422)
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1200)
at org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:560)
at org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:87)
at org.apache.solr.update.processor.UpdateRequestProcessor.processCommit(UpdateRequestProcessor.java:64)
at org.apache.solr.update.processor.DistributedUpdateProcessor.processCommit(DistributedUpdateProcessor.java:1007)
at org.apache.solr.update.processor.LogUpdateProcessor.processCommit(LogUpdateProcessorFactory.java:157)
at org.apache.solr.handler.RequestHandlerUtils.handleCommit(RequestHandlerUtils.java:69)
at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1699)
at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:455)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:276)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:225)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:169)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:927)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:999)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:565)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:309)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:550)
at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:563)
at org.apache.lucene.index.IndexWriter.nrtIsCurrent(IndexWriter.java:4196)
at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:266)
at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:245)
at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:235)
at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:169)
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1256)
... 28 more
Usually, I can get around this by basically creating a new HttpSolrServer and trying again. But if the problem is very frequent, one more try doesn't work. I don't like the idea of continually trying new solr connections until it works. What's the proper way to fix this? I'm assuming adding a doc with a commitWithMs of "0" is not any different than committing right away, but that suggestion was raised as well.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org