You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "John Wang (JIRA)" <ji...@apache.org> on 2009/04/25 05:01:30 UTC
[jira] Created: (LUCENE-1612) expose lastDocId in the posting from
the TermEnum API
expose lastDocId in the posting from the TermEnum API
-----------------------------------------------------
Key: LUCENE-1612
URL: https://issues.apache.org/jira/browse/LUCENE-1612
Project: Lucene - Java
Issue Type: Improvement
Components: Index
Affects Versions: 2.4
Reporter: John Wang
We currently have on the TermEnum api: docFreq() which gives the number docs in the posting.
It would be good to also have the max docid in the posting. That information is useful when construction a custom DocIdSet, .e.g determine sparseness of the doc list to decide whether or not to use a BitSet.
I have written a patch to do this, the problem with it is the TermInfosWriter encodes values in VInt/VLong, there is very little flexibility to add in lastDocId while making the index backward compatible. (If simple int is used for say, docFreq, a bit can be used to flag reading of a new piece of information)
output.writeVInt(ti.docFreq); // write doc freq
output.writeVLong(ti.freqPointer - lastTi.freqPointer); // write pointers
output.writeVLong(ti.proxPointer - lastTi.proxPointer);
Anyway, patch is attached with:TestSegmentTermEnum modified to test this. TestBackwardsCompatibility fails due to reasons described above.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1612) expose lastDocId in the posting
from the TermEnum API
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12702706#action_12702706 ]
Michael McCandless commented on LUCENE-1612:
--------------------------------------------
Well... a couple problems with always doing this:
* The in-memory terms index now consumes another 4 bytes per indexed term
* The tii/tis files got larger
One way to optimize it might be to only record it for terms whose freq is > X (and for the long tail of low-freq terms you iterate its postings to get the last docID).
Also, most apps don't need this information. So I don't think we should turn this on, always.
So maybe we wait for LUCENE-1458 and then build this as an alternate codec?
> expose lastDocId in the posting from the TermEnum API
> -----------------------------------------------------
>
> Key: LUCENE-1612
> URL: https://issues.apache.org/jira/browse/LUCENE-1612
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Affects Versions: 2.4
> Reporter: John Wang
> Attachments: lucene-1612-patch.txt
>
>
> We currently have on the TermEnum api: docFreq() which gives the number docs in the posting.
> It would be good to also have the max docid in the posting. That information is useful when construction a custom DocIdSet, .e.g determine sparseness of the doc list to decide whether or not to use a BitSet.
> I have written a patch to do this, the problem with it is the TermInfosWriter encodes values in VInt/VLong, there is very little flexibility to add in lastDocId while making the index backward compatible. (If simple int is used for say, docFreq, a bit can be used to flag reading of a new piece of information)
> output.writeVInt(ti.docFreq); // write doc freq
> output.writeVLong(ti.freqPointer - lastTi.freqPointer); // write pointers
> output.writeVLong(ti.proxPointer - lastTi.proxPointer);
> Anyway, patch is attached with:TestSegmentTermEnum modified to test this. TestBackwardsCompatibility fails due to reasons described above.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1612) expose lastDocId in the posting
from the TermEnum API
Posted by "John Wang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12702827#action_12702827 ]
John Wang commented on LUCENE-1612:
-----------------------------------
I am fine with waiting for LUCENE-1458. But Michael, then how would it help the merge of postings you described? Merging would be outside of the codec, no?
> expose lastDocId in the posting from the TermEnum API
> -----------------------------------------------------
>
> Key: LUCENE-1612
> URL: https://issues.apache.org/jira/browse/LUCENE-1612
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Affects Versions: 2.4
> Reporter: John Wang
> Attachments: lucene-1612-patch.txt
>
>
> We currently have on the TermEnum api: docFreq() which gives the number docs in the posting.
> It would be good to also have the max docid in the posting. That information is useful when construction a custom DocIdSet, .e.g determine sparseness of the doc list to decide whether or not to use a BitSet.
> I have written a patch to do this, the problem with it is the TermInfosWriter encodes values in VInt/VLong, there is very little flexibility to add in lastDocId while making the index backward compatible. (If simple int is used for say, docFreq, a bit can be used to flag reading of a new piece of information)
> output.writeVInt(ti.docFreq); // write doc freq
> output.writeVLong(ti.freqPointer - lastTi.freqPointer); // write pointers
> output.writeVLong(ti.proxPointer - lastTi.proxPointer);
> Anyway, patch is attached with:TestSegmentTermEnum modified to test this. TestBackwardsCompatibility fails due to reasons described above.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1612) expose lastDocId in the posting
from the TermEnum API
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12702669#action_12702669 ]
Michael McCandless commented on LUCENE-1612:
--------------------------------------------
This would be a good test (custom codec) for flexible indexing (LUCENE-1458), ie, allowing you to write whatever you want per-term.
Also, if lastDocID is always available, this could make merging of postings much faster than it is today, because you could bulk-copy the doc/freq posting bytes while just "fixing up" the boundary between them, because docIDs are delta coded.
> expose lastDocId in the posting from the TermEnum API
> -----------------------------------------------------
>
> Key: LUCENE-1612
> URL: https://issues.apache.org/jira/browse/LUCENE-1612
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Affects Versions: 2.4
> Reporter: John Wang
> Attachments: lucene-1612-patch.txt
>
>
> We currently have on the TermEnum api: docFreq() which gives the number docs in the posting.
> It would be good to also have the max docid in the posting. That information is useful when construction a custom DocIdSet, .e.g determine sparseness of the doc list to decide whether or not to use a BitSet.
> I have written a patch to do this, the problem with it is the TermInfosWriter encodes values in VInt/VLong, there is very little flexibility to add in lastDocId while making the index backward compatible. (If simple int is used for say, docFreq, a bit can be used to flag reading of a new piece of information)
> output.writeVInt(ti.docFreq); // write doc freq
> output.writeVLong(ti.freqPointer - lastTi.freqPointer); // write pointers
> output.writeVLong(ti.proxPointer - lastTi.proxPointer);
> Anyway, patch is attached with:TestSegmentTermEnum modified to test this. TestBackwardsCompatibility fails due to reasons described above.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1612) expose lastDocId in the posting
from the TermEnum API
Posted by "Yonik Seeley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12702710#action_12702710 ]
Yonik Seeley commented on LUCENE-1612:
--------------------------------------
bq. So maybe we wait for LUCENE-1458 and then build this as an alternate codec?
+1, seems pretty specialized.
> expose lastDocId in the posting from the TermEnum API
> -----------------------------------------------------
>
> Key: LUCENE-1612
> URL: https://issues.apache.org/jira/browse/LUCENE-1612
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Affects Versions: 2.4
> Reporter: John Wang
> Attachments: lucene-1612-patch.txt
>
>
> We currently have on the TermEnum api: docFreq() which gives the number docs in the posting.
> It would be good to also have the max docid in the posting. That information is useful when construction a custom DocIdSet, .e.g determine sparseness of the doc list to decide whether or not to use a BitSet.
> I have written a patch to do this, the problem with it is the TermInfosWriter encodes values in VInt/VLong, there is very little flexibility to add in lastDocId while making the index backward compatible. (If simple int is used for say, docFreq, a bit can be used to flag reading of a new piece of information)
> output.writeVInt(ti.docFreq); // write doc freq
> output.writeVLong(ti.freqPointer - lastTi.freqPointer); // write pointers
> output.writeVLong(ti.proxPointer - lastTi.proxPointer);
> Anyway, patch is attached with:TestSegmentTermEnum modified to test this. TestBackwardsCompatibility fails due to reasons described above.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1612) expose lastDocId in the posting
from the TermEnum API
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12702829#action_12702829 ]
Michael McCandless commented on LUCENE-1612:
--------------------------------------------
Actually I think the codec will handle merging (this was recently proposed for payloads), so it should be able to do that optimization within itself.
> expose lastDocId in the posting from the TermEnum API
> -----------------------------------------------------
>
> Key: LUCENE-1612
> URL: https://issues.apache.org/jira/browse/LUCENE-1612
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Affects Versions: 2.4
> Reporter: John Wang
> Attachments: lucene-1612-patch.txt
>
>
> We currently have on the TermEnum api: docFreq() which gives the number docs in the posting.
> It would be good to also have the max docid in the posting. That information is useful when construction a custom DocIdSet, .e.g determine sparseness of the doc list to decide whether or not to use a BitSet.
> I have written a patch to do this, the problem with it is the TermInfosWriter encodes values in VInt/VLong, there is very little flexibility to add in lastDocId while making the index backward compatible. (If simple int is used for say, docFreq, a bit can be used to flag reading of a new piece of information)
> output.writeVInt(ti.docFreq); // write doc freq
> output.writeVLong(ti.freqPointer - lastTi.freqPointer); // write pointers
> output.writeVLong(ti.proxPointer - lastTi.proxPointer);
> Anyway, patch is attached with:TestSegmentTermEnum modified to test this. TestBackwardsCompatibility fails due to reasons described above.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-1612) expose lastDocId in the posting from
the TermEnum API
Posted by "John Wang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John Wang updated LUCENE-1612:
------------------------------
Attachment: lucene-1612-patch.txt
Patch attach with test. Index is not backwards compatible.
> expose lastDocId in the posting from the TermEnum API
> -----------------------------------------------------
>
> Key: LUCENE-1612
> URL: https://issues.apache.org/jira/browse/LUCENE-1612
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Affects Versions: 2.4
> Reporter: John Wang
> Attachments: lucene-1612-patch.txt
>
>
> We currently have on the TermEnum api: docFreq() which gives the number docs in the posting.
> It would be good to also have the max docid in the posting. That information is useful when construction a custom DocIdSet, .e.g determine sparseness of the doc list to decide whether or not to use a BitSet.
> I have written a patch to do this, the problem with it is the TermInfosWriter encodes values in VInt/VLong, there is very little flexibility to add in lastDocId while making the index backward compatible. (If simple int is used for say, docFreq, a bit can be used to flag reading of a new piece of information)
> output.writeVInt(ti.docFreq); // write doc freq
> output.writeVLong(ti.freqPointer - lastTi.freqPointer); // write pointers
> output.writeVLong(ti.proxPointer - lastTi.proxPointer);
> Anyway, patch is attached with:TestSegmentTermEnum modified to test this. TestBackwardsCompatibility fails due to reasons described above.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1612) expose lastDocId in the posting
from the TermEnum API
Posted by "John Wang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12702692#action_12702692 ]
John Wang commented on LUCENE-1612:
-----------------------------------
Excellent point Michael! What do you suggest on how to move forward with this?
> expose lastDocId in the posting from the TermEnum API
> -----------------------------------------------------
>
> Key: LUCENE-1612
> URL: https://issues.apache.org/jira/browse/LUCENE-1612
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Affects Versions: 2.4
> Reporter: John Wang
> Attachments: lucene-1612-patch.txt
>
>
> We currently have on the TermEnum api: docFreq() which gives the number docs in the posting.
> It would be good to also have the max docid in the posting. That information is useful when construction a custom DocIdSet, .e.g determine sparseness of the doc list to decide whether or not to use a BitSet.
> I have written a patch to do this, the problem with it is the TermInfosWriter encodes values in VInt/VLong, there is very little flexibility to add in lastDocId while making the index backward compatible. (If simple int is used for say, docFreq, a bit can be used to flag reading of a new piece of information)
> output.writeVInt(ti.docFreq); // write doc freq
> output.writeVLong(ti.freqPointer - lastTi.freqPointer); // write pointers
> output.writeVLong(ti.proxPointer - lastTi.proxPointer);
> Anyway, patch is attached with:TestSegmentTermEnum modified to test this. TestBackwardsCompatibility fails due to reasons described above.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org