You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Jason Rutherglen <ja...@gmail.com> on 2009/09/11 01:19:20 UTC
Index docstore flush problem
I'm seeing a strange exception when indexing using the latest Solr rev on EC2.
org.apache.solr.client.solrj.SolrServerException:
org.apache.solr.client.solrj.SolrServerException:
java.lang.RuntimeException: after flush: fdx size mismatch: 468 docs
vs 298404 length in bytes of _0.fdx
at org.apache.solr.client.solrj.embedded.EmbeddedSolrServer.request(EmbeddedSolrServer.java:153)
at org.apache.solr.client.solrj.request.UpdateRequest.process(UpdateRequest.java:268)
at org.apache.solr.client.solrj.SolrServer.commit(SolrServer.java:86)
at org.apache.solr.hadoop.SolrRecordWriter$1.run(SolrRecordWriter.java:239)
Caused by: org.apache.solr.client.solrj.SolrServerException:
java.lang.RuntimeException: after flush: fdx size mismatch: 468 docs
vs 298404 length in bytes of _0.fdx
at org.apache.solr.client.solrj.embedded.EmbeddedSolrServer.request(EmbeddedSolrServer.java:141)
... 3 more
Caused by: java.lang.RuntimeException: after flush: fdx size mismatch:
468 docs vs 298404 length in bytes of _0.fdx
at org.apache.lucene.index.StoredFieldsWriter.closeDocStore(StoredFieldsWriter.java:95)
at org.apache.lucene.index.DocFieldProcessor.closeDocStore(DocFieldProcessor.java:50)
at org.apache.lucene.index.DocumentsWriter.closeDocStore(DocumentsWriter.java:380)
at org.apache.lucene.index.DocumentsWriter.flush(DocumentsWriter.java:574)
at org.apache.lucene.index.IndexWriter.doFlushInternal(IndexWriter.java:4212)
at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:4110)
at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:4101)
at org.apache.lucene.index.IndexWriter.closeInternal(IndexWriter.java:2108)
at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:2071)
at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:2035)
at org.apache.solr.update.SolrIndexWriter.close(SolrIndexWriter.java:215)
at org.apache.solr.update.DirectUpdateHandler2.closeWriter(DirectUpdateHandler2.java:180)
at org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:404)
at org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:85)
at org.apache.solr.handler.RequestHandlerUtils.handleCommit(RequestHandlerUtils.java:105)
at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:48)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1299)
at org.apache.solr.client.solrj.embedded.EmbeddedSolrServer.request(EmbeddedSolrServer.java:139)
... 3 more
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Index docstore flush problem
Posted by Michael McCandless <lu...@mikemccandless.com>.
Phew :)
Mike
On Thu, Sep 10, 2009 at 8:14 PM, Jason Rutherglen
<ja...@gmail.com> wrote:
> Indexing locking was off, there was a bug higher up clobbering the
> index. Sorry and thanks!
>
> On Thu, Sep 10, 2009 at 4:49 PM, Michael McCandless
> <lu...@mikemccandless.com> wrote:
>> That's an odd exception. It means IndexWriter thinks 468 docs have
>> been written to the stored fields file, which should mean the fdx file
>> size is 3748 (= 4 + 468*8), yet the file size is far larger than that
>> (298404).
>>
>> How repeatable is it? Can you turn on infoStream, get the exception
>> to happen, then post the resulting output?
>>
>> Mike
>>
>> On Thu, Sep 10, 2009 at 7:19 PM, Jason Rutherglen
>> <ja...@gmail.com> wrote:
>>> I'm seeing a strange exception when indexing using the latest Solr rev on EC2.
>>>
>>> org.apache.solr.client.solrj.SolrServerException:
>>> org.apache.solr.client.solrj.SolrServerException:
>>> java.lang.RuntimeException: after flush: fdx size mismatch: 468 docs
>>> vs 298404 length in bytes of _0.fdx
>>> at org.apache.solr.client.solrj.embedded.EmbeddedSolrServer.request(EmbeddedSolrServer.java:153)
>>> at org.apache.solr.client.solrj.request.UpdateRequest.process(UpdateRequest.java:268)
>>> at org.apache.solr.client.solrj.SolrServer.commit(SolrServer.java:86)
>>> at org.apache.solr.hadoop.SolrRecordWriter$1.run(SolrRecordWriter.java:239)
>>> Caused by: org.apache.solr.client.solrj.SolrServerException:
>>> java.lang.RuntimeException: after flush: fdx size mismatch: 468 docs
>>> vs 298404 length in bytes of _0.fdx
>>> at org.apache.solr.client.solrj.embedded.EmbeddedSolrServer.request(EmbeddedSolrServer.java:141)
>>> ... 3 more
>>> Caused by: java.lang.RuntimeException: after flush: fdx size mismatch:
>>> 468 docs vs 298404 length in bytes of _0.fdx
>>> at org.apache.lucene.index.StoredFieldsWriter.closeDocStore(StoredFieldsWriter.java:95)
>>> at org.apache.lucene.index.DocFieldProcessor.closeDocStore(DocFieldProcessor.java:50)
>>> at org.apache.lucene.index.DocumentsWriter.closeDocStore(DocumentsWriter.java:380)
>>> at org.apache.lucene.index.DocumentsWriter.flush(DocumentsWriter.java:574)
>>> at org.apache.lucene.index.IndexWriter.doFlushInternal(IndexWriter.java:4212)
>>> at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:4110)
>>> at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:4101)
>>> at org.apache.lucene.index.IndexWriter.closeInternal(IndexWriter.java:2108)
>>> at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:2071)
>>> at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:2035)
>>> at org.apache.solr.update.SolrIndexWriter.close(SolrIndexWriter.java:215)
>>> at org.apache.solr.update.DirectUpdateHandler2.closeWriter(DirectUpdateHandler2.java:180)
>>> at org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:404)
>>> at org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:85)
>>> at org.apache.solr.handler.RequestHandlerUtils.handleCommit(RequestHandlerUtils.java:105)
>>> at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:48)
>>> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
>>> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1299)
>>> at org.apache.solr.client.solrj.embedded.EmbeddedSolrServer.request(EmbeddedSolrServer.java:139)
>>> ... 3 more
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Index docstore flush problem
Posted by Jason Rutherglen <ja...@gmail.com>.
Indexing locking was off, there was a bug higher up clobbering the
index. Sorry and thanks!
On Thu, Sep 10, 2009 at 4:49 PM, Michael McCandless
<lu...@mikemccandless.com> wrote:
> That's an odd exception. It means IndexWriter thinks 468 docs have
> been written to the stored fields file, which should mean the fdx file
> size is 3748 (= 4 + 468*8), yet the file size is far larger than that
> (298404).
>
> How repeatable is it? Can you turn on infoStream, get the exception
> to happen, then post the resulting output?
>
> Mike
>
> On Thu, Sep 10, 2009 at 7:19 PM, Jason Rutherglen
> <ja...@gmail.com> wrote:
>> I'm seeing a strange exception when indexing using the latest Solr rev on EC2.
>>
>> org.apache.solr.client.solrj.SolrServerException:
>> org.apache.solr.client.solrj.SolrServerException:
>> java.lang.RuntimeException: after flush: fdx size mismatch: 468 docs
>> vs 298404 length in bytes of _0.fdx
>> at org.apache.solr.client.solrj.embedded.EmbeddedSolrServer.request(EmbeddedSolrServer.java:153)
>> at org.apache.solr.client.solrj.request.UpdateRequest.process(UpdateRequest.java:268)
>> at org.apache.solr.client.solrj.SolrServer.commit(SolrServer.java:86)
>> at org.apache.solr.hadoop.SolrRecordWriter$1.run(SolrRecordWriter.java:239)
>> Caused by: org.apache.solr.client.solrj.SolrServerException:
>> java.lang.RuntimeException: after flush: fdx size mismatch: 468 docs
>> vs 298404 length in bytes of _0.fdx
>> at org.apache.solr.client.solrj.embedded.EmbeddedSolrServer.request(EmbeddedSolrServer.java:141)
>> ... 3 more
>> Caused by: java.lang.RuntimeException: after flush: fdx size mismatch:
>> 468 docs vs 298404 length in bytes of _0.fdx
>> at org.apache.lucene.index.StoredFieldsWriter.closeDocStore(StoredFieldsWriter.java:95)
>> at org.apache.lucene.index.DocFieldProcessor.closeDocStore(DocFieldProcessor.java:50)
>> at org.apache.lucene.index.DocumentsWriter.closeDocStore(DocumentsWriter.java:380)
>> at org.apache.lucene.index.DocumentsWriter.flush(DocumentsWriter.java:574)
>> at org.apache.lucene.index.IndexWriter.doFlushInternal(IndexWriter.java:4212)
>> at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:4110)
>> at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:4101)
>> at org.apache.lucene.index.IndexWriter.closeInternal(IndexWriter.java:2108)
>> at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:2071)
>> at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:2035)
>> at org.apache.solr.update.SolrIndexWriter.close(SolrIndexWriter.java:215)
>> at org.apache.solr.update.DirectUpdateHandler2.closeWriter(DirectUpdateHandler2.java:180)
>> at org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:404)
>> at org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:85)
>> at org.apache.solr.handler.RequestHandlerUtils.handleCommit(RequestHandlerUtils.java:105)
>> at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:48)
>> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
>> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1299)
>> at org.apache.solr.client.solrj.embedded.EmbeddedSolrServer.request(EmbeddedSolrServer.java:139)
>> ... 3 more
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Index docstore flush problem
Posted by Michael McCandless <lu...@mikemccandless.com>.
That's an odd exception. It means IndexWriter thinks 468 docs have
been written to the stored fields file, which should mean the fdx file
size is 3748 (= 4 + 468*8), yet the file size is far larger than that
(298404).
How repeatable is it? Can you turn on infoStream, get the exception
to happen, then post the resulting output?
Mike
On Thu, Sep 10, 2009 at 7:19 PM, Jason Rutherglen
<ja...@gmail.com> wrote:
> I'm seeing a strange exception when indexing using the latest Solr rev on EC2.
>
> org.apache.solr.client.solrj.SolrServerException:
> org.apache.solr.client.solrj.SolrServerException:
> java.lang.RuntimeException: after flush: fdx size mismatch: 468 docs
> vs 298404 length in bytes of _0.fdx
> at org.apache.solr.client.solrj.embedded.EmbeddedSolrServer.request(EmbeddedSolrServer.java:153)
> at org.apache.solr.client.solrj.request.UpdateRequest.process(UpdateRequest.java:268)
> at org.apache.solr.client.solrj.SolrServer.commit(SolrServer.java:86)
> at org.apache.solr.hadoop.SolrRecordWriter$1.run(SolrRecordWriter.java:239)
> Caused by: org.apache.solr.client.solrj.SolrServerException:
> java.lang.RuntimeException: after flush: fdx size mismatch: 468 docs
> vs 298404 length in bytes of _0.fdx
> at org.apache.solr.client.solrj.embedded.EmbeddedSolrServer.request(EmbeddedSolrServer.java:141)
> ... 3 more
> Caused by: java.lang.RuntimeException: after flush: fdx size mismatch:
> 468 docs vs 298404 length in bytes of _0.fdx
> at org.apache.lucene.index.StoredFieldsWriter.closeDocStore(StoredFieldsWriter.java:95)
> at org.apache.lucene.index.DocFieldProcessor.closeDocStore(DocFieldProcessor.java:50)
> at org.apache.lucene.index.DocumentsWriter.closeDocStore(DocumentsWriter.java:380)
> at org.apache.lucene.index.DocumentsWriter.flush(DocumentsWriter.java:574)
> at org.apache.lucene.index.IndexWriter.doFlushInternal(IndexWriter.java:4212)
> at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:4110)
> at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:4101)
> at org.apache.lucene.index.IndexWriter.closeInternal(IndexWriter.java:2108)
> at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:2071)
> at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:2035)
> at org.apache.solr.update.SolrIndexWriter.close(SolrIndexWriter.java:215)
> at org.apache.solr.update.DirectUpdateHandler2.closeWriter(DirectUpdateHandler2.java:180)
> at org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:404)
> at org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:85)
> at org.apache.solr.handler.RequestHandlerUtils.handleCommit(RequestHandlerUtils.java:105)
> at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:48)
> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1299)
> at org.apache.solr.client.solrj.embedded.EmbeddedSolrServer.request(EmbeddedSolrServer.java:139)
> ... 3 more
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org