You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Steven Bethard (JIRA)" <ji...@apache.org> on 2010/04/29 03:36:52 UTC
[jira] Created: (LUCENE-2420) "fdx size mismatch" overflow causes
RuntimeException
"fdx size mismatch" overflow causes RuntimeException
----------------------------------------------------
Key: LUCENE-2420
URL: https://issues.apache.org/jira/browse/LUCENE-2420
Project: Lucene - Java
Issue Type: Bug
Components: Index
Affects Versions: 3.0.1
Environment: CentOS 5.4
Reporter: Steven Bethard
I just saw the following error:
java.lang.RuntimeException: after flush: fdx size mismatch: -512764976 docs vs 30257618564 length in bytes of _0.fdx file exists?=true
at org.apache.lucene.index.StoredFieldsWriter.closeDocStore(StoredFieldsWriter.java:97)
at org.apache.lucene.index.DocFieldProcessor.closeDocStore(DocFieldProcessor.java:51)
at org.apache.lucene.index.DocumentsWriter.closeDocStore(DocumentsWriter.java:371)
at org.apache.lucene.index.IndexWriter.flushDocStores(IndexWriter.java:1724)
at org.apache.lucene.index.IndexWriter.doFlushInternal(IndexWriter.java:3565)
at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3491)
at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3482)
at org.apache.lucene.index.IndexWriter.closeInternal(IndexWriter.java:1658)
at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1621)
at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1585)
Note the negative SegmentWriteState.numDocsInStore. I assume this is because Lucene has a limit of 2 ^ 31 - 1 = 2147483647 (sizeof(int)) documents per index, though I couldn't find this documented clearly anywhere. It would have been nice to get this error earlier, back when I exceeded the limit, rather than now, after a bunch of indexing that was apparently doomed to fail.
Hence, two suggestions:
* State clearly somewhere that the maximum number of documents in a Lucene index is sizeof(int).
* Throw an exception when an IndexWriter first exceeds this number rather than only on close.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2420) "fdx size mismatch" overflow causes
RuntimeException
Posted by "Shai Erera (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12862312#action_12862312 ]
Shai Erera commented on LUCENE-2420:
------------------------------------
I remember that the Integer.MAX_VAL is documented somewhere. I can try to look it up later. But a lot of places in the API use int as the doc Id (IndexReader, ScoreDoc, even IndexWriter.max/numDocs()), and so I think there's a strong hint about that limitation.
As for throwing the exception sooner, I don't think it will be correct. IndexWriter implements transaction semantics. Until you call commit() or close(), whatever operations you've made are not *officially* in the index yet. If your JVM dies before that, they will get lost. Therefore throwing the exception earlier would be wrong. Also think that you intend to index 1000 docs and delete 100,000. Would you want to get the exception while adding the docs, knowing that you are about to delete much more soon?
> "fdx size mismatch" overflow causes RuntimeException
> ----------------------------------------------------
>
> Key: LUCENE-2420
> URL: https://issues.apache.org/jira/browse/LUCENE-2420
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: 3.0.1
> Environment: CentOS 5.4
> Reporter: Steven Bethard
>
> I just saw the following error:
> java.lang.RuntimeException: after flush: fdx size mismatch: -512764976 docs vs 30257618564 length in bytes of _0.fdx file exists?=true
> at org.apache.lucene.index.StoredFieldsWriter.closeDocStore(StoredFieldsWriter.java:97)
> at org.apache.lucene.index.DocFieldProcessor.closeDocStore(DocFieldProcessor.java:51)
> at org.apache.lucene.index.DocumentsWriter.closeDocStore(DocumentsWriter.java:371)
> at org.apache.lucene.index.IndexWriter.flushDocStores(IndexWriter.java:1724)
> at org.apache.lucene.index.IndexWriter.doFlushInternal(IndexWriter.java:3565)
> at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3491)
> at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3482)
> at org.apache.lucene.index.IndexWriter.closeInternal(IndexWriter.java:1658)
> at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1621)
> at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1585)
> Note the negative SegmentWriteState.numDocsInStore. I assume this is because Lucene has a limit of 2 ^ 31 - 1 = 2147483647 (sizeof(int)) documents per index, though I couldn't find this documented clearly anywhere. It would have been nice to get this error earlier, back when I exceeded the limit, rather than now, after a bunch of indexing that was apparently doomed to fail.
> Hence, two suggestions:
> * State clearly somewhere that the maximum number of documents in a Lucene index is sizeof(int).
> * Throw an exception when an IndexWriter first exceeds this number rather than only on close.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2420) "fdx size mismatch" overflow causes
RuntimeException
Posted by "Steven Bethard (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12862571#action_12862571 ]
Steven Bethard commented on LUCENE-2420:
----------------------------------------
Thanks, yeah, something more explicit and easier to find with Google would be good.
Yeah, I mean that code should throw the exception. Actually, anywhere that checks the value of numDocsInStore could/should throw the exception, e.g.
synchronized public void flush(SegmentWriteState state) throws IOException {
if (state.numDocsInStore > 0) {
synchronized public void closeDocStore(SegmentWriteState state) throws IOException {
final int inc = state.numDocsInStore - lastDocID;
if (inc > 0) {
Don't know if SegmentWriteState.numDocsInStore is used anywhere else (I haven't loaded it into an IDE to look at files other than StoredFieldsWriter), but in at least these two cases it would be easy to give an exception explaining that the maximum number of documents had been exceeded.
Alternatively, you could try to fix the code to work correctly after integer overflow (to support the transaction use case you described above) though it's less obvious to me how to do that correctly everywhere. Probably involves changing some "> 0"s into "!= 0"s and being careful in a few other ways.
> "fdx size mismatch" overflow causes RuntimeException
> ----------------------------------------------------
>
> Key: LUCENE-2420
> URL: https://issues.apache.org/jira/browse/LUCENE-2420
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: 3.0.1
> Environment: CentOS 5.4
> Reporter: Steven Bethard
>
> I just saw the following error:
> java.lang.RuntimeException: after flush: fdx size mismatch: -512764976 docs vs 30257618564 length in bytes of _0.fdx file exists?=true
> at org.apache.lucene.index.StoredFieldsWriter.closeDocStore(StoredFieldsWriter.java:97)
> at org.apache.lucene.index.DocFieldProcessor.closeDocStore(DocFieldProcessor.java:51)
> at org.apache.lucene.index.DocumentsWriter.closeDocStore(DocumentsWriter.java:371)
> at org.apache.lucene.index.IndexWriter.flushDocStores(IndexWriter.java:1724)
> at org.apache.lucene.index.IndexWriter.doFlushInternal(IndexWriter.java:3565)
> at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3491)
> at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3482)
> at org.apache.lucene.index.IndexWriter.closeInternal(IndexWriter.java:1658)
> at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1621)
> at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1585)
> Note the negative SegmentWriteState.numDocsInStore. I assume this is because Lucene has a limit of 2 ^ 31 - 1 = 2147483647 (sizeof(int)) documents per index, though I couldn't find this documented clearly anywhere. It would have been nice to get this error earlier, back when I exceeded the limit, rather than now, after a bunch of indexing that was apparently doomed to fail.
> Hence, two suggestions:
> * State clearly somewhere that the maximum number of documents in a Lucene index is sizeof(int).
> * Throw an exception when an IndexWriter first exceeds this number rather than only on close.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2420) "fdx size mismatch" overflow causes
RuntimeException
Posted by "Steven Bethard (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12862463#action_12862463 ]
Steven Bethard commented on LUCENE-2420:
----------------------------------------
I finally found the documentation saying that the maximum number of documents is ~274 billion:
http://lucene.apache.org/java/3_0_1/fileformats.html
Google queries that failed to find this:
lucene index maximum documents
lucene document limit
lucene max docs
Maybe a bullet could be added to the FAQ (which does turn up for most of these queries)?
http://wiki.apache.org/lucene-java/LuceneFAQ
As far as the exception goes, regardless of the transaction semantics, I really don't think the code works correctly after numeric overflow. Once SegmentWriteState.numDocsInStore is negative, I would expect code like StoredFieldsWriter.flush to fail:
synchronized public void flush(SegmentWriteState state) throws IOException {
if (state.numDocsInStore > 0) {
...
Perhaps I'm wrong, but it seems like this is going to do the wrong thing when SegmentWriteState.numDocsInStore is negative. If I'm not wrong, then it seems sensible to me to raise an exception on numeric overflow.
> "fdx size mismatch" overflow causes RuntimeException
> ----------------------------------------------------
>
> Key: LUCENE-2420
> URL: https://issues.apache.org/jira/browse/LUCENE-2420
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: 3.0.1
> Environment: CentOS 5.4
> Reporter: Steven Bethard
>
> I just saw the following error:
> java.lang.RuntimeException: after flush: fdx size mismatch: -512764976 docs vs 30257618564 length in bytes of _0.fdx file exists?=true
> at org.apache.lucene.index.StoredFieldsWriter.closeDocStore(StoredFieldsWriter.java:97)
> at org.apache.lucene.index.DocFieldProcessor.closeDocStore(DocFieldProcessor.java:51)
> at org.apache.lucene.index.DocumentsWriter.closeDocStore(DocumentsWriter.java:371)
> at org.apache.lucene.index.IndexWriter.flushDocStores(IndexWriter.java:1724)
> at org.apache.lucene.index.IndexWriter.doFlushInternal(IndexWriter.java:3565)
> at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3491)
> at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3482)
> at org.apache.lucene.index.IndexWriter.closeInternal(IndexWriter.java:1658)
> at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1621)
> at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1585)
> Note the negative SegmentWriteState.numDocsInStore. I assume this is because Lucene has a limit of 2 ^ 31 - 1 = 2147483647 (sizeof(int)) documents per index, though I couldn't find this documented clearly anywhere. It would have been nice to get this error earlier, back when I exceeded the limit, rather than now, after a bunch of indexing that was apparently doomed to fail.
> Hence, two suggestions:
> * State clearly somewhere that the maximum number of documents in a Lucene index is sizeof(int).
> * Throw an exception when an IndexWriter first exceeds this number rather than only on close.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2420) "fdx size mismatch" overflow causes
RuntimeException
Posted by "Shai Erera (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12862528#action_12862528 ]
Shai Erera commented on LUCENE-2420:
------------------------------------
That documentation discusses the limitation around the number of unique terms Lucene can handle, which sums up to ~274 billion: "_which means the maximum number of *unique terms* in any single index segment is ~2.1 billion times the term index interval (default 128) = ~274 billion_".
One line below this you can find this documentation: "_Similarly, Lucene uses a Java int to refer to document numbers, and the index file format uses an Int32 on-disk to store document numbers_" which suggests the Integer.MAX_VAL limitation ... but I agree it could have been spelled out more clearly.
If numDocsInStore is negative, SegmentFieldsWriter.flush() doesn't do anything ... you mean that that code should throw the exception?
> "fdx size mismatch" overflow causes RuntimeException
> ----------------------------------------------------
>
> Key: LUCENE-2420
> URL: https://issues.apache.org/jira/browse/LUCENE-2420
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: 3.0.1
> Environment: CentOS 5.4
> Reporter: Steven Bethard
>
> I just saw the following error:
> java.lang.RuntimeException: after flush: fdx size mismatch: -512764976 docs vs 30257618564 length in bytes of _0.fdx file exists?=true
> at org.apache.lucene.index.StoredFieldsWriter.closeDocStore(StoredFieldsWriter.java:97)
> at org.apache.lucene.index.DocFieldProcessor.closeDocStore(DocFieldProcessor.java:51)
> at org.apache.lucene.index.DocumentsWriter.closeDocStore(DocumentsWriter.java:371)
> at org.apache.lucene.index.IndexWriter.flushDocStores(IndexWriter.java:1724)
> at org.apache.lucene.index.IndexWriter.doFlushInternal(IndexWriter.java:3565)
> at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3491)
> at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3482)
> at org.apache.lucene.index.IndexWriter.closeInternal(IndexWriter.java:1658)
> at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1621)
> at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1585)
> Note the negative SegmentWriteState.numDocsInStore. I assume this is because Lucene has a limit of 2 ^ 31 - 1 = 2147483647 (sizeof(int)) documents per index, though I couldn't find this documented clearly anywhere. It would have been nice to get this error earlier, back when I exceeded the limit, rather than now, after a bunch of indexing that was apparently doomed to fail.
> Hence, two suggestions:
> * State clearly somewhere that the maximum number of documents in a Lucene index is sizeof(int).
> * Throw an exception when an IndexWriter first exceeds this number rather than only on close.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org