You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Shai Erera (JIRA)" <ji...@apache.org> on 2011/01/26 15:26:43 UTC
[jira] Created: (LUCENE-2891) IndexWriterConfig does not allow
termInfosDivisor to be -1, while the latest indicates TermInfos should not
be loaded
IndexWriterConfig does not allow termInfosDivisor to be -1, while the latest indicates TermInfos should not be loaded
---------------------------------------------------------------------------------------------------------------------
Key: LUCENE-2891
URL: https://issues.apache.org/jira/browse/LUCENE-2891
Project: Lucene - Java
Issue Type: Bug
Components: Index
Reporter: Shai Erera
Assignee: Shai Erera
Fix For: 3.1, 4.0
While you can pass -1 to IR.open(), and it's documented, you cannot do the same for IndexWriter's readers (b/c IWC blocks it). Need to allow this setting as well as add support for it in our tests, e.g. we should randomly set it to -1. Robert also suggested RandomIW use -1 randomly when it opens readers.
I'll work on a patch
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2891) IndexWriterConfig does not allow
readerTermsIndexDivisor to be -1, while the latest indicates the terms
index should not be loaded
Posted by "Shai Erera (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Shai Erera updated LUCENE-2891:
-------------------------------
Summary: IndexWriterConfig does not allow readerTermsIndexDivisor to be -1, while the latest indicates the terms index should not be loaded (was: IndexWriterConfig does not allow termInfosDivisor to be -1, while the latest indicates TermInfos should not be loaded)
Fixed issue subject (replaced termInfos with terms index)
> IndexWriterConfig does not allow readerTermsIndexDivisor to be -1, while the latest indicates the terms index should not be loaded
> ----------------------------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-2891
> URL: https://issues.apache.org/jira/browse/LUCENE-2891
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Reporter: Shai Erera
> Assignee: Shai Erera
> Fix For: 3.1, 4.0
>
> Attachments: LUCENE-2891.patch, LUCENE-2891.patch, LUCENE-2891.patch
>
>
> While you can pass -1 to IR.open(), and it's documented, you cannot do the same for IndexWriter's readers (b/c IWC blocks it). Need to allow this setting as well as add support for it in our tests, e.g. we should randomly set it to -1. Robert also suggested RandomIW use -1 randomly when it opens readers.
> I'll work on a patch
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2891) IndexWriterConfig does not allow
termInfosDivisor to be -1, while the latest indicates TermInfos should not
be loaded
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12987124#action_12987124 ]
Robert Muir commented on LUCENE-2891:
-------------------------------------
bq. -1 means "don't load terms index because I will never, ever seek-by-Term". As far as I know, it's only segment merging that does this...
but it allows next()'ing through all the terms only?
I suggested the testing because I had seen some code for -1 but I didn't know how tested it is,
its a tad confusing in the javadoc which just says "Set this to -1 to skip loading the terms index entirely."
This makes me think (based on the language "skip loading"), that if you do this everything will still work,
but seek() will just be brute-force or something aweful...
> IndexWriterConfig does not allow termInfosDivisor to be -1, while the latest indicates TermInfos should not be loaded
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-2891
> URL: https://issues.apache.org/jira/browse/LUCENE-2891
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Reporter: Shai Erera
> Assignee: Shai Erera
> Fix For: 3.1, 4.0
>
> Attachments: LUCENE-2891.patch, LUCENE-2891.patch
>
>
> While you can pass -1 to IR.open(), and it's documented, you cannot do the same for IndexWriter's readers (b/c IWC blocks it). Need to allow this setting as well as add support for it in our tests, e.g. we should randomly set it to -1. Robert also suggested RandomIW use -1 randomly when it opens readers.
> I'll work on a patch
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2891) IndexWriterConfig does not allow
termInfosDivisor to be -1, while the latest indicates TermInfos should not
be loaded
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12987254#action_12987254 ]
Michael McCandless commented on LUCENE-2891:
--------------------------------------------
bq. but it allows next()'ing through all the terms only?
Right.
{quote}
I suggested the testing because I had seen some code for -1 but I didn't know how tested it is,
{quote}
It's tested inside IW -- for merging, it opens all SRs w/ -1.
bq. its a tad confusing in the javadoc which just says "Set this to -1 to skip loading the terms index entirely."
I agree -- I think we should sharpen the jdocs to state that this is only useful in very narrow situations, ie when you are certain you just need to next() through all terms. (And probably mark this value as @lucene.experimental... maybe internal?).
> IndexWriterConfig does not allow termInfosDivisor to be -1, while the latest indicates TermInfos should not be loaded
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-2891
> URL: https://issues.apache.org/jira/browse/LUCENE-2891
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Reporter: Shai Erera
> Assignee: Shai Erera
> Fix For: 3.1, 4.0
>
> Attachments: LUCENE-2891.patch, LUCENE-2891.patch
>
>
> While you can pass -1 to IR.open(), and it's documented, you cannot do the same for IndexWriter's readers (b/c IWC blocks it). Need to allow this setting as well as add support for it in our tests, e.g. we should randomly set it to -1. Robert also suggested RandomIW use -1 randomly when it opens readers.
> I'll work on a patch
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2891) IndexWriterConfig does not allow
termInfosDivisor to be -1, while the latest indicates TermInfos should not
be loaded
Posted by "Shai Erera (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Shai Erera updated LUCENE-2891:
-------------------------------
Attachment: LUCENE-2891.patch
Removed the random setting in tests and added a test TestIndexWriterReader.testNoTermsIndex() which ensure IR.open(IW) and seek-by-term throws an exception when the setting is set to -1 on IWC.
> IndexWriterConfig does not allow termInfosDivisor to be -1, while the latest indicates TermInfos should not be loaded
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-2891
> URL: https://issues.apache.org/jira/browse/LUCENE-2891
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Reporter: Shai Erera
> Assignee: Shai Erera
> Fix For: 3.1, 4.0
>
> Attachments: LUCENE-2891.patch, LUCENE-2891.patch
>
>
> While you can pass -1 to IR.open(), and it's documented, you cannot do the same for IndexWriter's readers (b/c IWC blocks it). Need to allow this setting as well as add support for it in our tests, e.g. we should randomly set it to -1. Robert also suggested RandomIW use -1 randomly when it opens readers.
> I'll work on a patch
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2891) IndexWriterConfig does not allow
termInfosDivisor to be -1, while the latest indicates TermInfos should not
be loaded
Posted by "Shai Erera (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Shai Erera updated LUCENE-2891:
-------------------------------
Attachment: LUCENE-2891.patch
Patch supports -1 passed to IWC.setReaderTermIndexDivisor and randomly sets it in LuceneTestCase.newIndexWriterConfig() and RandomIndexWriter.
Many tests fail with exceptions like this:
{noformat}
[junit] java.lang.IllegalStateException: terms index was not loaded when this reader was createde=29178856,total=88990720
[junit] at org.apache.lucene.index.TermInfosReader.ensureIndexIsRead(TermInfosReader.java:309)
[junit] at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:215)
[junit] at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:208)
[junit] at org.apache.lucene.index.SegmentTermDocs.seek(SegmentTermDocs.java:57) Caused an ERROR
[junit] at org.apache.lucene.index.BufferedDeletes.applyDeletes(BufferedDeletes.java:301)
[junit] at org.apache.lucene.index.BufferedDeletes.applyDeletes(BufferedDeletes.java:289)ted
[junit] at org.apache.lucene.index.BufferedDeletes.applyDeletes(BufferedDeletes.java:191):309)
{noformat}
So maybe IWC not supporting -1 is not a bug? I didn't see anywhere, even in IR.open() what are the consequences of not loading the terms index. Thought it just means slower term seeks, but apparently it's not supported?
If so, I think we should mention IR.open() what does it mean to not load the terms index.
> IndexWriterConfig does not allow termInfosDivisor to be -1, while the latest indicates TermInfos should not be loaded
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-2891
> URL: https://issues.apache.org/jira/browse/LUCENE-2891
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Reporter: Shai Erera
> Assignee: Shai Erera
> Fix For: 3.1, 4.0
>
> Attachments: LUCENE-2891.patch
>
>
> While you can pass -1 to IR.open(), and it's documented, you cannot do the same for IndexWriter's readers (b/c IWC blocks it). Need to allow this setting as well as add support for it in our tests, e.g. we should randomly set it to -1. Robert also suggested RandomIW use -1 randomly when it opens readers.
> I'll work on a patch
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Resolved: (LUCENE-2891) IndexWriterConfig does not allow
readerTermsIndexDivisor to be -1, while the latest indicates the terms
index should not be loaded
Posted by "Shai Erera (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Shai Erera resolved LUCENE-2891.
--------------------------------
Resolution: Fixed
Committed revision 1064255 (3x).
Committed revision 1064285 (trunk).
Merge to trunk was not so easy. testNoTermsIndex had to use different API than in 3x (which is ok), but the it turns out few Codecs don't honor the -1 setting, like PreFlex and SimpleText. Surprisingly, MockRandom also fails. I put all of them into an illegalCodecs Set and if that's what the test draws, the test does not continue and succeeds.
I'll open a separate issue to check why MockRandomCodec fails the test (i.e., why doesn't it honor the -1 setting).
> IndexWriterConfig does not allow readerTermsIndexDivisor to be -1, while the latest indicates the terms index should not be loaded
> ----------------------------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-2891
> URL: https://issues.apache.org/jira/browse/LUCENE-2891
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Reporter: Shai Erera
> Assignee: Shai Erera
> Fix For: 3.1, 4.0
>
> Attachments: LUCENE-2891.patch, LUCENE-2891.patch, LUCENE-2891.patch
>
>
> While you can pass -1 to IR.open(), and it's documented, you cannot do the same for IndexWriter's readers (b/c IWC blocks it). Need to allow this setting as well as add support for it in our tests, e.g. we should randomly set it to -1. Robert also suggested RandomIW use -1 randomly when it opens readers.
> I'll work on a patch
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2891) IndexWriterConfig does not allow
termInfosDivisor to be -1, while the latest indicates TermInfos should not
be loaded
Posted by "Shai Erera (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Shai Erera updated LUCENE-2891:
-------------------------------
Attachment: LUCENE-2891.patch
Patch fixes javadocs and renames term infos to terms index. I plan to commit this later today or tomorrow morning.
> IndexWriterConfig does not allow termInfosDivisor to be -1, while the latest indicates TermInfos should not be loaded
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-2891
> URL: https://issues.apache.org/jira/browse/LUCENE-2891
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Reporter: Shai Erera
> Assignee: Shai Erera
> Fix For: 3.1, 4.0
>
> Attachments: LUCENE-2891.patch, LUCENE-2891.patch, LUCENE-2891.patch
>
>
> While you can pass -1 to IR.open(), and it's documented, you cannot do the same for IndexWriter's readers (b/c IWC blocks it). Need to allow this setting as well as add support for it in our tests, e.g. we should randomly set it to -1. Robert also suggested RandomIW use -1 randomly when it opens readers.
> I'll work on a patch
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2891) IndexWriterConfig does not allow
termInfosDivisor to be -1, while the latest indicates TermInfos should not
be loaded
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12987058#action_12987058 ]
Michael McCandless commented on LUCENE-2891:
--------------------------------------------
-1 means "don't load terms index because I will never, ever seek-by-Term". As far as I know, it's only segment merging that does this...
Maybe we shouldn't allow it for IR.open? Though, eg if you want to pass the reader to addIndexes (say), opening with -1 should work...
> IndexWriterConfig does not allow termInfosDivisor to be -1, while the latest indicates TermInfos should not be loaded
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-2891
> URL: https://issues.apache.org/jira/browse/LUCENE-2891
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Reporter: Shai Erera
> Assignee: Shai Erera
> Fix For: 3.1, 4.0
>
> Attachments: LUCENE-2891.patch
>
>
> While you can pass -1 to IR.open(), and it's documented, you cannot do the same for IndexWriter's readers (b/c IWC blocks it). Need to allow this setting as well as add support for it in our tests, e.g. we should randomly set it to -1. Robert also suggested RandomIW use -1 randomly when it opens readers.
> I'll work on a patch
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2891) IndexWriterConfig does not allow
termInfosDivisor to be -1, while the latest indicates TermInfos should not
be loaded
Posted by "Shai Erera (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12987408#action_12987408 ]
Shai Erera commented on LUCENE-2891:
------------------------------------
Did you guys review my latest patch, which includes a clarification in the jdocs?
> IndexWriterConfig does not allow termInfosDivisor to be -1, while the latest indicates TermInfos should not be loaded
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-2891
> URL: https://issues.apache.org/jira/browse/LUCENE-2891
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Reporter: Shai Erera
> Assignee: Shai Erera
> Fix For: 3.1, 4.0
>
> Attachments: LUCENE-2891.patch, LUCENE-2891.patch
>
>
> While you can pass -1 to IR.open(), and it's documented, you cannot do the same for IndexWriter's readers (b/c IWC blocks it). Need to allow this setting as well as add support for it in our tests, e.g. we should randomly set it to -1. Robert also suggested RandomIW use -1 randomly when it opens readers.
> I'll work on a patch
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org