You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Mark Harwood (JIRA)" <ji...@apache.org> on 2008/11/03 15:27:44 UTC
[jira] Created: (LUCENE-1437) Faster skipTo on SegmentTermEnum
Faster skipTo on SegmentTermEnum
--------------------------------
Key: LUCENE-1437
URL: https://issues.apache.org/jira/browse/LUCENE-1437
Project: Lucene - Java
Issue Type: Improvement
Components: Index
Affects Versions: 2.4
Reporter: Mark Harwood
Priority: Minor
Attachments: fasterSkip.patch
I've been trying to speed up function I have that involves calling TermEnum.skipTo a lot on a very large index with many terms.
This patch avoids excessive object creation of Term objects as part of the default implementation in the TermEnum base class.
My tests output with this patch was as follows:
Took 1894 ms for 42304 calls to skipTo on index with 182693176 docs
and unmodified Lucene 2.4 results were:
Took 2438 ms for 42304 calls to skipTo on index with 182693176 docs
The logic is based on the existing code in SegmentTermEnum. scanTo(Term term) and avoids the object construction code found in the TermEnum.skipTo code which repeatedly calls next() and term() to create Term objects.
Anyone see any negative side effects in changing this?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-1437) Faster skipTo on SegmentTermEnum
Posted by "Mark Harwood (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mark Harwood updated LUCENE-1437:
---------------------------------
Attachment: fasterSkip.patch
Patch to SegmentTermEnum to override slow TermEnum.skipTo
> Faster skipTo on SegmentTermEnum
> --------------------------------
>
> Key: LUCENE-1437
> URL: https://issues.apache.org/jira/browse/LUCENE-1437
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Affects Versions: 2.4
> Reporter: Mark Harwood
> Priority: Minor
> Attachments: fasterSkip.patch
>
>
> I've been trying to speed up function I have that involves calling TermEnum.skipTo a lot on a very large index with many terms.
> This patch avoids excessive object creation of Term objects as part of the default implementation in the TermEnum base class.
> My tests output with this patch was as follows:
> Took 1894 ms for 42304 calls to skipTo on index with 182693176 docs
> and unmodified Lucene 2.4 results were:
> Took 2438 ms for 42304 calls to skipTo on index with 182693176 docs
> The logic is based on the existing code in SegmentTermEnum. scanTo(Term term) and avoids the object construction code found in the TermEnum.skipTo code which repeatedly calls next() and term() to create Term objects.
> Anyone see any negative side effects in changing this?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1437) Faster skipTo on SegmentTermEnum
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12644946#action_12644946 ]
Michael McCandless commented on LUCENE-1437:
--------------------------------------------
Also, I'm [trying to] refactor this code in working on flexible indexing. So I think for now at least it's low priority to fix this.
> Faster skipTo on SegmentTermEnum
> --------------------------------
>
> Key: LUCENE-1437
> URL: https://issues.apache.org/jira/browse/LUCENE-1437
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Affects Versions: 2.4
> Reporter: Mark Harwood
> Priority: Minor
> Attachments: fasterSkip.patch
>
>
> I've been trying to speed up function I have that involves calling TermEnum.skipTo a lot on a very large index with many terms.
> This patch avoids excessive object creation of Term objects as part of the default implementation in the TermEnum base class.
> My tests output with this patch was as follows:
> Took 1894 ms for 42304 calls to skipTo on index with 182693176 docs
> and unmodified Lucene 2.4 results were:
> Took 2438 ms for 42304 calls to skipTo on index with 182693176 docs
> The logic is based on the existing code in SegmentTermEnum. scanTo(Term term) and avoids the object construction code found in the TermEnum.skipTo code which repeatedly calls next() and term() to create Term objects.
> Anyone see any negative side effects in changing this?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1437) Faster skipTo on SegmentTermEnum
Posted by "Yonik Seeley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12645033#action_12645033 ]
Yonik Seeley commented on LUCENE-1437:
--------------------------------------
bq. I can't find anywhere else that uses TermEnum.skipTo so this optimisation now seems somewhat redundant.
Right, it does a linear scan, so it's normally best avoided. It would probably be nice if it could somehow use the index as IndexReader.terms(term) does when it returns a new TermEnum.
> Faster skipTo on SegmentTermEnum
> --------------------------------
>
> Key: LUCENE-1437
> URL: https://issues.apache.org/jira/browse/LUCENE-1437
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Affects Versions: 2.4
> Reporter: Mark Harwood
> Priority: Minor
> Attachments: fasterSkip.patch
>
>
> I've been trying to speed up function I have that involves calling TermEnum.skipTo a lot on a very large index with many terms.
> This patch avoids excessive object creation of Term objects as part of the default implementation in the TermEnum base class.
> My tests output with this patch was as follows:
> Took 1894 ms for 42304 calls to skipTo on index with 182693176 docs
> and unmodified Lucene 2.4 results were:
> Took 2438 ms for 42304 calls to skipTo on index with 182693176 docs
> The logic is based on the existing code in SegmentTermEnum. scanTo(Term term) and avoids the object construction code found in the TermEnum.skipTo code which repeatedly calls next() and term() to create Term objects.
> Anyone see any negative side effects in changing this?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1437) Faster skipTo on SegmentTermEnum
Posted by "Mark Harwood (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12644938#action_12644938 ]
Mark Harwood commented on LUCENE-1437:
--------------------------------------
This may now be a valid optimisation but without a use case.
I have subsequently found that I can get significantly better performance repeatedly calling IndexReader.docFreq(term) for a large ordered list of terms than I can calling TermEnum.skipTo(term) and then TermEnum.docFreq().
I can't find anywhere else that uses TermEnum.skipTo so this optimisation now seems somewhat redundant.
> Faster skipTo on SegmentTermEnum
> --------------------------------
>
> Key: LUCENE-1437
> URL: https://issues.apache.org/jira/browse/LUCENE-1437
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Affects Versions: 2.4
> Reporter: Mark Harwood
> Priority: Minor
> Attachments: fasterSkip.patch
>
>
> I've been trying to speed up function I have that involves calling TermEnum.skipTo a lot on a very large index with many terms.
> This patch avoids excessive object creation of Term objects as part of the default implementation in the TermEnum base class.
> My tests output with this patch was as follows:
> Took 1894 ms for 42304 calls to skipTo on index with 182693176 docs
> and unmodified Lucene 2.4 results were:
> Took 2438 ms for 42304 calls to skipTo on index with 182693176 docs
> The logic is based on the existing code in SegmentTermEnum. scanTo(Term term) and avoids the object construction code found in the TermEnum.skipTo code which repeatedly calls next() and term() to create Term objects.
> Anyone see any negative side effects in changing this?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org