You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Michael McCandless (JIRA)" <ji...@apache.org> on 2011/02/15 17:11:57 UTC
[jira] Created: (LUCENE-2922) Optimize BlockTermsReader.seek
Optimize BlockTermsReader.seek
------------------------------
Key: LUCENE-2922
URL: https://issues.apache.org/jira/browse/LUCENE-2922
Project: Lucene - Java
Issue Type: Improvement
Components: Index
Reporter: Michael McCandless
Assignee: Michael McCandless
Priority: Minor
Fix For: 4.0
When we seek, we first consult the terms index to find the right block
of 32 (default) terms that may hold the target term. Then, we scan
that block looking for an exact match.
The scanning just uses next() and then compares the full term, but
this is actually rather wasteful. First off, since all terms in the
block share a common prefix, we should compare the target against that
common prefix once, and then only compare the new suffix of each
term. Second, since the term suffixes have already been read up front
into a byte[], we should do a no-copy comparison (vs today, where we
first read a copy into the local BytesRef and then compare).
With this opto, I removed the ability for BlockTermsWriter/Reader to
support arbitrary term sort order -- it's now hardwired to
BytesRef.utf8SortedAsUnicode.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2922) Optimize BlockTermsReader.seek
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12994865#comment-12994865 ]
Michael McCandless commented on LUCENE-2922:
--------------------------------------------
The opto is a big win for FuzzyQuery (and, automaton respeller):
||Query||QPS base||QPS opto||Pct diff||||
|"united states"|13.92|13.81|{color:red}-0.8%{color}|
|+united +states|20.59|20.55|{color:red}-0.2%{color}|
|united states|20.06|20.03|{color:red}-0.1%{color}|
|states|56.67|56.68|{color:green}0.0%{color}|
|"united states"~3|9.55|9.55|{color:green}0.0%{color}|
|uni*|17.67|17.71|{color:green}0.2%{color}|
|spanNear([unit, state], 10, true)|65.84|66.03|{color:green}0.3%{color}|
|unit*|31.50|31.62|{color:green}0.4%{color}|
|timesecnum:[10000 TO 60000]|10.88|10.93|{color:green}0.4%{color}|
|un*d|19.64|19.74|{color:green}0.5%{color}|
|title:.*[Uu]nited.*|1.48|1.49|{color:green}0.9%{color}|
|u*d|8.52|8.63|{color:green}1.3%{color}|
|+nebraska +states|230.99|235.15|{color:green}1.8%{color}|
|spanFirst(unit, 5)|289.74|300.65|{color:green}3.8%{color}|
|united~0.75|18.01|19.26|{color:green}7.0%{color}|
|unit~0.7|36.39|40.33|{color:green}10.8%{color}|
|united~0.6|14.15|15.73|{color:green}11.1%{color}|
|unit~0.5|24.99|29.82|{color:green}19.3%{color}|
> Optimize BlockTermsReader.seek
> ------------------------------
>
> Key: LUCENE-2922
> URL: https://issues.apache.org/jira/browse/LUCENE-2922
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Priority: Minor
> Fix For: 4.0
>
> Attachments: LUCENE-2922.patch
>
>
> When we seek, we first consult the terms index to find the right block
> of 32 (default) terms that may hold the target term. Then, we scan
> that block looking for an exact match.
> The scanning just uses next() and then compares the full term, but
> this is actually rather wasteful. First off, since all terms in the
> block share a common prefix, we should compare the target against that
> common prefix once, and then only compare the new suffix of each
> term. Second, since the term suffixes have already been read up front
> into a byte[], we should do a no-copy comparison (vs today, where we
> first read a copy into the local BytesRef and then compare).
> With this opto, I removed the ability for BlockTermsWriter/Reader to
> support arbitrary term sort order -- it's now hardwired to
> BytesRef.utf8SortedAsUnicode.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Resolved: (LUCENE-2922) Optimize BlockTermsReader.seek
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless resolved LUCENE-2922.
----------------------------------------
Resolution: Fixed
> Optimize BlockTermsReader.seek
> ------------------------------
>
> Key: LUCENE-2922
> URL: https://issues.apache.org/jira/browse/LUCENE-2922
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Priority: Minor
> Fix For: 4.0
>
> Attachments: LUCENE-2922.patch
>
>
> When we seek, we first consult the terms index to find the right block
> of 32 (default) terms that may hold the target term. Then, we scan
> that block looking for an exact match.
> The scanning just uses next() and then compares the full term, but
> this is actually rather wasteful. First off, since all terms in the
> block share a common prefix, we should compare the target against that
> common prefix once, and then only compare the new suffix of each
> term. Second, since the term suffixes have already been read up front
> into a byte[], we should do a no-copy comparison (vs today, where we
> first read a copy into the local BytesRef and then compare).
> With this opto, I removed the ability for BlockTermsWriter/Reader to
> support arbitrary term sort order -- it's now hardwired to
> BytesRef.utf8SortedAsUnicode.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2922) Optimize BlockTermsReader.seek
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless updated LUCENE-2922:
---------------------------------------
Attachment: LUCENE-2922.patch
Patch.
> Optimize BlockTermsReader.seek
> ------------------------------
>
> Key: LUCENE-2922
> URL: https://issues.apache.org/jira/browse/LUCENE-2922
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Priority: Minor
> Fix For: 4.0
>
> Attachments: LUCENE-2922.patch
>
>
> When we seek, we first consult the terms index to find the right block
> of 32 (default) terms that may hold the target term. Then, we scan
> that block looking for an exact match.
> The scanning just uses next() and then compares the full term, but
> this is actually rather wasteful. First off, since all terms in the
> block share a common prefix, we should compare the target against that
> common prefix once, and then only compare the new suffix of each
> term. Second, since the term suffixes have already been read up front
> into a byte[], we should do a no-copy comparison (vs today, where we
> first read a copy into the local BytesRef and then compare).
> With this opto, I removed the ability for BlockTermsWriter/Reader to
> support arbitrary term sort order -- it's now hardwired to
> BytesRef.utf8SortedAsUnicode.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org