You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Michael McCandless (JIRA)" <ji...@apache.org> on 2010/02/09 20:15:27 UTC
[jira] Created: (LUCENE-2257) relax the per-segment max unique term
limit
relax the per-segment max unique term limit
-------------------------------------------
Key: LUCENE-2257
URL: https://issues.apache.org/jira/browse/LUCENE-2257
Project: Lucene - Java
Issue Type: Improvement
Reporter: Michael McCandless
Assignee: Michael McCandless
Priority: Minor
Fix For: 2.9.2, 3.0.1, 3.1
Lucene can't handle more than 2.1B (limit of signed 32 bit int) unique terms in a single segment.
But I think we can improve this to termIndexInterval (default 128) * 2.1B. There is one place (internal API only) where Lucene uses an int but should use a long.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Resolved: (LUCENE-2257) relax the per-segment max unique
term limit
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless resolved LUCENE-2257.
----------------------------------------
Resolution: Fixed
Committed for 2.9.2, 3.0.1, 3.1.
> relax the per-segment max unique term limit
> -------------------------------------------
>
> Key: LUCENE-2257
> URL: https://issues.apache.org/jira/browse/LUCENE-2257
> Project: Lucene - Java
> Issue Type: Improvement
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Priority: Minor
> Fix For: 2.9.2, 3.0.1, 3.1
>
> Attachments: LUCENE-2257.patch, LUCENE-2257.patch
>
>
> Lucene can't handle more than 2.1B (limit of signed 32 bit int) unique terms in a single segment.
> But I think we can improve this to termIndexInterval (default 128) * 2.1B. There is one place (internal API only) where Lucene uses an int but should use a long.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2257) relax the per-segment max unique
term limit
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12832698#action_12832698 ]
Robert Muir commented on LUCENE-2257:
-------------------------------------
bq. (I discovered that queries containing Korean characters would consistently trigger the bug).
this makes sense because Hangul is sorted towards the end of the term dictionary
you can see this visually here: http://unicode.org/roadmaps/bmp/
> relax the per-segment max unique term limit
> -------------------------------------------
>
> Key: LUCENE-2257
> URL: https://issues.apache.org/jira/browse/LUCENE-2257
> Project: Lucene - Java
> Issue Type: Improvement
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Priority: Minor
> Fix For: 2.9.2, 3.0.1, 3.1
>
> Attachments: LUCENE-2257.patch
>
>
> Lucene can't handle more than 2.1B (limit of signed 32 bit int) unique terms in a single segment.
> But I think we can improve this to termIndexInterval (default 128) * 2.1B. There is one place (internal API only) where Lucene uses an int but should use a long.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2257) relax the per-segment max unique
term limit
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12832729#action_12832729 ]
Michael McCandless commented on LUCENE-2257:
--------------------------------------------
bq. With the patch, we don't see any ArrayIndexOutOfBounds exceptions.
Great! And the results look correct?
bq. Other than walking though the code in the debugger, is there some systematic way of looking for any other places where an int is used that might also have problems when we have over 2.1x billion terms?
Not that I know of! The code that handles the term dict lookup is
fairly contained, in TermInfosReader and SegmentTermEnum. I think
scrutinizing the code and testing (as you're doing) is the only way.
I just looked again -- there are a few places where int is still being used.
First is two methods (get(int position) and scanEnum), in
TermInfosReader, that are actually dead code (package private &
unused). Second is int SegmentTermEnum.scanTo, but this is fine
because it's never asked to can more than termIndexInterval terms.
I'll attach patch that additionally just removes that dead code.
> relax the per-segment max unique term limit
> -------------------------------------------
>
> Key: LUCENE-2257
> URL: https://issues.apache.org/jira/browse/LUCENE-2257
> Project: Lucene - Java
> Issue Type: Improvement
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Priority: Minor
> Fix For: 2.9.2, 3.0.1, 3.1
>
> Attachments: LUCENE-2257.patch, LUCENE-2257.patch
>
>
> Lucene can't handle more than 2.1B (limit of signed 32 bit int) unique terms in a single segment.
> But I think we can improve this to termIndexInterval (default 128) * 2.1B. There is one place (internal API only) where Lucene uses an int but should use a long.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2257) relax the per-segment max unique term
limit
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless updated LUCENE-2257:
---------------------------------------
Attachment: LUCENE-2257.patch
Possible patch fixing the issue. I'm not yet certain there is no other place where we use an int...
> relax the per-segment max unique term limit
> -------------------------------------------
>
> Key: LUCENE-2257
> URL: https://issues.apache.org/jira/browse/LUCENE-2257
> Project: Lucene - Java
> Issue Type: Improvement
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Priority: Minor
> Fix For: 2.9.2, 3.0.1, 3.1
>
> Attachments: LUCENE-2257.patch
>
>
> Lucene can't handle more than 2.1B (limit of signed 32 bit int) unique terms in a single segment.
> But I think we can improve this to termIndexInterval (default 128) * 2.1B. There is one place (internal API only) where Lucene uses an int but should use a long.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2257) relax the per-segment max unique
term limit
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12832076#action_12832076 ]
Michael McCandless commented on LUCENE-2257:
--------------------------------------------
OK, I'm glad to hear that.
The attached patch applies to 2.9, and I think should apply fine to the revision of Lucene you're using (779312) that you're using within Solr. I'd recommend checking out that exact revision of Lucene (svn co -r779312 ...), applying this patch, building a JAR, and replacing Solr's Lucene JAR with this one.
It's only queries that contain terms above the 2.1B mark (your last ~390 M terms) that will hit the exception. Once you find such a query it should always hit the exception on this large segment.
> relax the per-segment max unique term limit
> -------------------------------------------
>
> Key: LUCENE-2257
> URL: https://issues.apache.org/jira/browse/LUCENE-2257
> Project: Lucene - Java
> Issue Type: Improvement
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Priority: Minor
> Fix For: 2.9.2, 3.0.1, 3.1
>
> Attachments: LUCENE-2257.patch
>
>
> Lucene can't handle more than 2.1B (limit of signed 32 bit int) unique terms in a single segment.
> But I think we can improve this to termIndexInterval (default 128) * 2.1B. There is one place (internal API only) where Lucene uses an int but should use a long.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2257) relax the per-segment max unique term
limit
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless updated LUCENE-2257:
---------------------------------------
Attachment: LUCENE-2257.patch
> relax the per-segment max unique term limit
> -------------------------------------------
>
> Key: LUCENE-2257
> URL: https://issues.apache.org/jira/browse/LUCENE-2257
> Project: Lucene - Java
> Issue Type: Improvement
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Priority: Minor
> Fix For: 2.9.2, 3.0.1, 3.1
>
> Attachments: LUCENE-2257.patch, LUCENE-2257.patch
>
>
> Lucene can't handle more than 2.1B (limit of signed 32 bit int) unique terms in a single segment.
> But I think we can improve this to termIndexInterval (default 128) * 2.1B. There is one place (internal API only) where Lucene uses an int but should use a long.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2257) relax the per-segment max unique
term limit
Posted by "Tom Burton-West (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12832054#action_12832054 ]
Tom Burton-West commented on LUCENE-2257:
-----------------------------------------
Thanks for the patch Michael,
The patch worked fine with CheckIndex. Checkindex worked with an index with 2.49 billion terms.
I added commas to the output below:
test: terms, freq, prox...OK [2,487,224,745 terms; 23,573,976,855 terms/docs pairs; 97,223,318,067 tokens]
We are working on determining how to test it with Solr. The ArrayIndexOutOfBounds exception appears in the logs about for about 1 in every 100 queries. We haven't been able to determine which queries trigger the problem.
We are using an older version of Solr with lucene 2.9-dev 779312 - 2009-05-27 17:19:55 . I'm not sure if we can just drop in a later version of lucene with the patch or if I need to patch the older 2.9 dev lucene version that came with our Solr. What do you suggest?
What I'm thinking of is to run 10,000 queries against our dev server pointing at one of the large segment indexes with and without the patch.
Tom
> relax the per-segment max unique term limit
> -------------------------------------------
>
> Key: LUCENE-2257
> URL: https://issues.apache.org/jira/browse/LUCENE-2257
> Project: Lucene - Java
> Issue Type: Improvement
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Priority: Minor
> Fix For: 2.9.2, 3.0.1, 3.1
>
> Attachments: LUCENE-2257.patch
>
>
> Lucene can't handle more than 2.1B (limit of signed 32 bit int) unique terms in a single segment.
> But I think we can improve this to termIndexInterval (default 128) * 2.1B. There is one place (internal API only) where Lucene uses an int but should use a long.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2257) relax the per-segment max unique
term limit
Posted by "Tom Burton-West (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12832689#action_12832689 ]
Tom Burton-West commented on LUCENE-2257:
-----------------------------------------
Hi Michael,
Thanks for your help. We mounted one of the indexes with 2.4 billion terms on our dev server and tested with and without the patch. (I discovered that queries containing Korean characters would consistently trigger the bug). With the patch, we don't see any ArrayIndexOutOfBounds exceptions. We are going to do a bit more testing before we put this into production. (We rolled back our production indexes temporarily to indexes that split the terms over 2 segments and therefore didn't trigger the bug).
Other than walking though the code in the debugger, is there some systematic way of looking for any other places where an int is used that might also have problems when we have over 2.1x billion terms?
Tom
> relax the per-segment max unique term limit
> -------------------------------------------
>
> Key: LUCENE-2257
> URL: https://issues.apache.org/jira/browse/LUCENE-2257
> Project: Lucene - Java
> Issue Type: Improvement
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Priority: Minor
> Fix For: 2.9.2, 3.0.1, 3.1
>
> Attachments: LUCENE-2257.patch
>
>
> Lucene can't handle more than 2.1B (limit of signed 32 bit int) unique terms in a single segment.
> But I think we can improve this to termIndexInterval (default 128) * 2.1B. There is one place (internal API only) where Lucene uses an int but should use a long.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org