You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Koji Sekiguchi (JIRA)" <ji...@apache.org> on 2011/09/10 16:33:08 UTC
[jira] [Created] (LUCENE-3426) optimizer for n-gram PhraseQuery
optimizer for n-gram PhraseQuery
--------------------------------
Key: LUCENE-3426
URL: https://issues.apache.org/jira/browse/LUCENE-3426
Project: Lucene - Java
Issue Type: Improvement
Components: core/search
Reporter: Koji Sekiguchi
Priority: Trivial
If 2-gram is used and the length of query string is 4, for example q="ABCD", QueryParser generates (when autoGeneratePhraseQueries is true) PhraseQuery("AB BC CD") with slop 0. But it can be optimized PhraseQuery("AB CD") with appropriate positions.
The idea came from the Japanese paper "N.M-gram: Implementation of Inverted Index Using N-gram with Hash Values" by Mikio Hirabayashi, et al. (The main theme of the paper is different from the idea that I'm using here, though)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3426) optimizer for n-gram PhraseQuery
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13103775#comment-13103775 ]
Michael McCandless commented on LUCENE-3426:
--------------------------------------------
Patch looks great! What a nice opto :)
> optimizer for n-gram PhraseQuery
> --------------------------------
>
> Key: LUCENE-3426
> URL: https://issues.apache.org/jira/browse/LUCENE-3426
> Project: Lucene - Java
> Issue Type: Improvement
> Components: core/search
> Reporter: Koji Sekiguchi
> Priority: Trivial
> Attachments: LUCENE-3426.patch, LUCENE-3426.patch, LUCENE-3426.patch, LUCENE-3426.patch, LUCENE-3426.patch, PerfTest.java, PerfTest.java
>
>
> If 2-gram is used and the length of query string is 4, for example q="ABCD", QueryParser generates (when autoGeneratePhraseQueries is true) PhraseQuery("AB BC CD") with slop 0. But it can be optimized PhraseQuery("AB CD") with appropriate positions.
> The idea came from the Japanese paper "N.M-gram: Implementation of Inverted Index Using N-gram with Hash Values" by Mikio Hirabayashi, et al. (The main theme of the paper is different from the idea that I'm using here, though)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3426) optimizer for n-gram PhraseQuery
Posted by "Koji Sekiguchi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Koji Sekiguchi updated LUCENE-3426:
-----------------------------------
Attachment: PerfTest.java
> optimizer for n-gram PhraseQuery
> --------------------------------
>
> Key: LUCENE-3426
> URL: https://issues.apache.org/jira/browse/LUCENE-3426
> Project: Lucene - Java
> Issue Type: Improvement
> Components: core/search
> Reporter: Koji Sekiguchi
> Priority: Trivial
> Attachments: LUCENE-3426.patch, LUCENE-3426.patch, LUCENE-3426.patch, LUCENE-3426.patch, PerfTest.java, PerfTest.java
>
>
> If 2-gram is used and the length of query string is 4, for example q="ABCD", QueryParser generates (when autoGeneratePhraseQueries is true) PhraseQuery("AB BC CD") with slop 0. But it can be optimized PhraseQuery("AB CD") with appropriate positions.
> The idea came from the Japanese paper "N.M-gram: Implementation of Inverted Index Using N-gram with Hash Values" by Mikio Hirabayashi, et al. (The main theme of the paper is different from the idea that I'm using here, though)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3426) optimizer for n-gram PhraseQuery
Posted by "Koji Sekiguchi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Koji Sekiguchi updated LUCENE-3426:
-----------------------------------
Attachment: LUCENE-3426.patch
First draft. The patch doesn't include test yet.
I added optimizeForNgram() methods in the patch that will remove "redundant" terms and positions.
Optimizing PhraseQuery will change score.
> optimizer for n-gram PhraseQuery
> --------------------------------
>
> Key: LUCENE-3426
> URL: https://issues.apache.org/jira/browse/LUCENE-3426
> Project: Lucene - Java
> Issue Type: Improvement
> Components: core/search
> Reporter: Koji Sekiguchi
> Priority: Trivial
> Attachments: LUCENE-3426.patch
>
>
> If 2-gram is used and the length of query string is 4, for example q="ABCD", QueryParser generates (when autoGeneratePhraseQueries is true) PhraseQuery("AB BC CD") with slop 0. But it can be optimized PhraseQuery("AB CD") with appropriate positions.
> The idea came from the Japanese paper "N.M-gram: Implementation of Inverted Index Using N-gram with Hash Values" by Mikio Hirabayashi, et al. (The main theme of the paper is different from the idea that I'm using here, though)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3426) optimizer for n-gram PhraseQuery
Posted by "Koji Sekiguchi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13102205#comment-13102205 ]
Koji Sekiguchi commented on LUCENE-3426:
----------------------------------------
The result of speed up is:
||n-gram/query length||normal(ms)||optimized(ms)||speed up||
|bi-gram/4|21,766|16,641|30.8%|
|bi-gram/6|29,865|21,518|38.8%|
|tri-gram/5|8,188|6,140|33.4%|
|tri-gram/6|9,001|5,925|51.9%|
> optimizer for n-gram PhraseQuery
> --------------------------------
>
> Key: LUCENE-3426
> URL: https://issues.apache.org/jira/browse/LUCENE-3426
> Project: Lucene - Java
> Issue Type: Improvement
> Components: core/search
> Reporter: Koji Sekiguchi
> Priority: Trivial
> Attachments: LUCENE-3426.patch, LUCENE-3426.patch
>
>
> If 2-gram is used and the length of query string is 4, for example q="ABCD", QueryParser generates (when autoGeneratePhraseQueries is true) PhraseQuery("AB BC CD") with slop 0. But it can be optimized PhraseQuery("AB CD") with appropriate positions.
> The idea came from the Japanese paper "N.M-gram: Implementation of Inverted Index Using N-gram with Hash Values" by Mikio Hirabayashi, et al. (The main theme of the paper is different from the idea that I'm using here, though)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3426) optimizer for n-gram PhraseQuery
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13102393#comment-13102393 ]
Robert Muir commented on LUCENE-3426:
-------------------------------------
I think I like it better too... though I wonder if its possible to keep the original NGramPhraseQuery unmodified?
this way its not changed by Query.rewrite(), and if a user reuses the query (which we document they can do), they could then call add() again and everything works.
Also, somewhat related to the issue might be SOLR-2660. We don't have to commit that patch, but we could separate
out the queryparser refactoring to make it easier for such an optimization to be "automatic" in solr, because it allows
SolrQueryParser to delegate creation of Phrase/MultiPhraseQuery to the FieldType.
> optimizer for n-gram PhraseQuery
> --------------------------------
>
> Key: LUCENE-3426
> URL: https://issues.apache.org/jira/browse/LUCENE-3426
> Project: Lucene - Java
> Issue Type: Improvement
> Components: core/search
> Reporter: Koji Sekiguchi
> Priority: Trivial
> Attachments: LUCENE-3426.patch, LUCENE-3426.patch, LUCENE-3426.patch, PerfTest.java
>
>
> If 2-gram is used and the length of query string is 4, for example q="ABCD", QueryParser generates (when autoGeneratePhraseQueries is true) PhraseQuery("AB BC CD") with slop 0. But it can be optimized PhraseQuery("AB CD") with appropriate positions.
> The idea came from the Japanese paper "N.M-gram: Implementation of Inverted Index Using N-gram with Hash Values" by Mikio Hirabayashi, et al. (The main theme of the paper is different from the idea that I'm using here, though)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3426) optimizer for n-gram PhraseQuery
Posted by "Koji Sekiguchi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Koji Sekiguchi updated LUCENE-3426:
-----------------------------------
Attachment: LUCENE-3426.patch
I like the idea of introducing the newly created class! Here is the new patch.
> optimizer for n-gram PhraseQuery
> --------------------------------
>
> Key: LUCENE-3426
> URL: https://issues.apache.org/jira/browse/LUCENE-3426
> Project: Lucene - Java
> Issue Type: Improvement
> Components: core/search
> Reporter: Koji Sekiguchi
> Priority: Trivial
> Attachments: LUCENE-3426.patch, LUCENE-3426.patch, LUCENE-3426.patch, PerfTest.java
>
>
> If 2-gram is used and the length of query string is 4, for example q="ABCD", QueryParser generates (when autoGeneratePhraseQueries is true) PhraseQuery("AB BC CD") with slop 0. But it can be optimized PhraseQuery("AB CD") with appropriate positions.
> The idea came from the Japanese paper "N.M-gram: Implementation of Inverted Index Using N-gram with Hash Values" by Mikio Hirabayashi, et al. (The main theme of the paper is different from the idea that I'm using here, though)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3426) optimizer for n-gram PhraseQuery
Posted by "Koji Sekiguchi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Koji Sekiguchi updated LUCENE-3426:
-----------------------------------
Attachment: PerfTest.java
> optimizer for n-gram PhraseQuery
> --------------------------------
>
> Key: LUCENE-3426
> URL: https://issues.apache.org/jira/browse/LUCENE-3426
> Project: Lucene - Java
> Issue Type: Improvement
> Components: core/search
> Reporter: Koji Sekiguchi
> Priority: Trivial
> Attachments: LUCENE-3426.patch, LUCENE-3426.patch, PerfTest.java
>
>
> If 2-gram is used and the length of query string is 4, for example q="ABCD", QueryParser generates (when autoGeneratePhraseQueries is true) PhraseQuery("AB BC CD") with slop 0. But it can be optimized PhraseQuery("AB CD") with appropriate positions.
> The idea came from the Japanese paper "N.M-gram: Implementation of Inverted Index Using N-gram with Hash Values" by Mikio Hirabayashi, et al. (The main theme of the paper is different from the idea that I'm using here, though)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3426) optimizer for n-gram PhraseQuery
Posted by "Bill Bell (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13190550#comment-13190550 ]
Bill Bell commented on LUCENE-3426:
-----------------------------------
Is this automatic in SOLR? Or do we need to add a feature to support his in SOLR?
> optimizer for n-gram PhraseQuery
> --------------------------------
>
> Key: LUCENE-3426
> URL: https://issues.apache.org/jira/browse/LUCENE-3426
> Project: Lucene - Java
> Issue Type: Improvement
> Components: core/search
> Affects Versions: 2.9.4, 3.0.3, 3.1, 3.2, 3.3, 3.4, 4.0
> Reporter: Koji Sekiguchi
> Assignee: Koji Sekiguchi
> Priority: Trivial
> Fix For: 3.5, 4.0
>
> Attachments: LUCENE-3426.patch, LUCENE-3426.patch, LUCENE-3426.patch, LUCENE-3426.patch, LUCENE-3426.patch, LUCENE-3426.patch, PerfTest.java, PerfTest.java
>
>
> If 2-gram is used and the length of query string is 4, for example q="ABCD", QueryParser generates (when autoGeneratePhraseQueries is true) PhraseQuery("AB BC CD") with slop 0. But it can be optimized PhraseQuery("AB CD") with appropriate positions.
> The idea came from the Japanese paper "N.M-gram: Implementation of Inverted Index Using N-gram with Hash Values" by Mikio Hirabayashi, et al. (The main theme of the paper is different from the idea that I'm using here, though)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3426) optimizer for n-gram PhraseQuery
Posted by "Koji Sekiguchi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13102405#comment-13102405 ]
Koji Sekiguchi commented on LUCENE-3426:
----------------------------------------
For automatic in Solr, I wonder if we could move the feature to n-gram tokenizers, and we could have something like:
{code}
<fieldType name="text_cjk" class="solr.TextField" positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.CJKTokenizerFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.CJKTokenizerFactory" optimizePhraseQuery="true"/>
</analyzer>
</fieldType>
{code}
> optimizer for n-gram PhraseQuery
> --------------------------------
>
> Key: LUCENE-3426
> URL: https://issues.apache.org/jira/browse/LUCENE-3426
> Project: Lucene - Java
> Issue Type: Improvement
> Components: core/search
> Reporter: Koji Sekiguchi
> Priority: Trivial
> Attachments: LUCENE-3426.patch, LUCENE-3426.patch, LUCENE-3426.patch, LUCENE-3426.patch, PerfTest.java, PerfTest.java
>
>
> If 2-gram is used and the length of query string is 4, for example q="ABCD", QueryParser generates (when autoGeneratePhraseQueries is true) PhraseQuery("AB BC CD") with slop 0. But it can be optimized PhraseQuery("AB CD") with appropriate positions.
> The idea came from the Japanese paper "N.M-gram: Implementation of Inverted Index Using N-gram with Hash Values" by Mikio Hirabayashi, et al. (The main theme of the paper is different from the idea that I'm using here, though)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3426) optimizer for n-gram PhraseQuery
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13102406#comment-13102406 ]
Robert Muir commented on LUCENE-3426:
-------------------------------------
Well if we apply the refactoring part of SOLR-2660 (we can split out into a separate issue), we could add such a thing as an attribute to the fieldType?
I like the way your patch looks now! A couple more questions:
* doesn't the optimization also apply to MultiPhraseQuery? If so, NGramPhraseQuery could extend MultiPhraseQuery and just rewrite to the correct one (MultiPhrase or Phrase depending upon the situation after optimization)
* what about hashCode/equals? Although the same results will be returned, scoring will differ, maybe it NGramPhraseQuery should implement these?
> optimizer for n-gram PhraseQuery
> --------------------------------
>
> Key: LUCENE-3426
> URL: https://issues.apache.org/jira/browse/LUCENE-3426
> Project: Lucene - Java
> Issue Type: Improvement
> Components: core/search
> Reporter: Koji Sekiguchi
> Priority: Trivial
> Attachments: LUCENE-3426.patch, LUCENE-3426.patch, LUCENE-3426.patch, LUCENE-3426.patch, PerfTest.java, PerfTest.java
>
>
> If 2-gram is used and the length of query string is 4, for example q="ABCD", QueryParser generates (when autoGeneratePhraseQueries is true) PhraseQuery("AB BC CD") with slop 0. But it can be optimized PhraseQuery("AB CD") with appropriate positions.
> The idea came from the Japanese paper "N.M-gram: Implementation of Inverted Index Using N-gram with Hash Values" by Mikio Hirabayashi, et al. (The main theme of the paper is different from the idea that I'm using here, though)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3426) optimizer for n-gram PhraseQuery
Posted by "Koji Sekiguchi (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13190560#comment-13190560 ]
Koji Sekiguchi commented on LUCENE-3426:
----------------------------------------
bq. Is this automatic in SOLR?
No. I've opened SOLR-3055.
> optimizer for n-gram PhraseQuery
> --------------------------------
>
> Key: LUCENE-3426
> URL: https://issues.apache.org/jira/browse/LUCENE-3426
> Project: Lucene - Java
> Issue Type: Improvement
> Components: core/search
> Affects Versions: 2.9.4, 3.0.3, 3.1, 3.2, 3.3, 3.4, 4.0
> Reporter: Koji Sekiguchi
> Assignee: Koji Sekiguchi
> Priority: Trivial
> Fix For: 3.5, 4.0
>
> Attachments: LUCENE-3426.patch, LUCENE-3426.patch, LUCENE-3426.patch, LUCENE-3426.patch, LUCENE-3426.patch, LUCENE-3426.patch, PerfTest.java, PerfTest.java
>
>
> If 2-gram is used and the length of query string is 4, for example q="ABCD", QueryParser generates (when autoGeneratePhraseQueries is true) PhraseQuery("AB BC CD") with slop 0. But it can be optimized PhraseQuery("AB CD") with appropriate positions.
> The idea came from the Japanese paper "N.M-gram: Implementation of Inverted Index Using N-gram with Hash Values" by Mikio Hirabayashi, et al. (The main theme of the paper is different from the idea that I'm using here, though)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3426) optimizer for n-gram PhraseQuery
Posted by "Koji Sekiguchi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Koji Sekiguchi updated LUCENE-3426:
-----------------------------------
Attachment: LUCENE-3426.patch
New patch. I added equals/hashCode in the patch.
I think it is too complex to apply optimization to MultiPhraseQuery, so I'd like to stick with PhraseQuery in the patch.
> optimizer for n-gram PhraseQuery
> --------------------------------
>
> Key: LUCENE-3426
> URL: https://issues.apache.org/jira/browse/LUCENE-3426
> Project: Lucene - Java
> Issue Type: Improvement
> Components: core/search
> Reporter: Koji Sekiguchi
> Priority: Trivial
> Attachments: LUCENE-3426.patch, LUCENE-3426.patch, LUCENE-3426.patch, LUCENE-3426.patch, LUCENE-3426.patch, PerfTest.java, PerfTest.java
>
>
> If 2-gram is used and the length of query string is 4, for example q="ABCD", QueryParser generates (when autoGeneratePhraseQueries is true) PhraseQuery("AB BC CD") with slop 0. But it can be optimized PhraseQuery("AB CD") with appropriate positions.
> The idea came from the Japanese paper "N.M-gram: Implementation of Inverted Index Using N-gram with Hash Values" by Mikio Hirabayashi, et al. (The main theme of the paper is different from the idea that I'm using here, though)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3426) optimizer for n-gram PhraseQuery
Posted by "Koji Sekiguchi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Koji Sekiguchi updated LUCENE-3426:
-----------------------------------
Affects Version/s: 4.0
3.4
2.9.4
3.0.3
3.1
3.2
3.3
Fix Version/s: 4.0
3.5
Assignee: Koji Sekiguchi
> optimizer for n-gram PhraseQuery
> --------------------------------
>
> Key: LUCENE-3426
> URL: https://issues.apache.org/jira/browse/LUCENE-3426
> Project: Lucene - Java
> Issue Type: Improvement
> Components: core/search
> Affects Versions: 2.9.4, 3.0.3, 3.1, 3.2, 3.3, 3.4, 4.0
> Reporter: Koji Sekiguchi
> Assignee: Koji Sekiguchi
> Priority: Trivial
> Fix For: 3.5, 4.0
>
> Attachments: LUCENE-3426.patch, LUCENE-3426.patch, LUCENE-3426.patch, LUCENE-3426.patch, LUCENE-3426.patch, LUCENE-3426.patch, PerfTest.java, PerfTest.java
>
>
> If 2-gram is used and the length of query string is 4, for example q="ABCD", QueryParser generates (when autoGeneratePhraseQueries is true) PhraseQuery("AB BC CD") with slop 0. But it can be optimized PhraseQuery("AB CD") with appropriate positions.
> The idea came from the Japanese paper "N.M-gram: Implementation of Inverted Index Using N-gram with Hash Values" by Mikio Hirabayashi, et al. (The main theme of the paper is different from the idea that I'm using here, though)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3426) optimizer for n-gram PhraseQuery
Posted by "Koji Sekiguchi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Koji Sekiguchi updated LUCENE-3426:
-----------------------------------
Attachment: LUCENE-3426.patch
I added test code.
> optimizer for n-gram PhraseQuery
> --------------------------------
>
> Key: LUCENE-3426
> URL: https://issues.apache.org/jira/browse/LUCENE-3426
> Project: Lucene - Java
> Issue Type: Improvement
> Components: core/search
> Reporter: Koji Sekiguchi
> Priority: Trivial
> Attachments: LUCENE-3426.patch, LUCENE-3426.patch
>
>
> If 2-gram is used and the length of query string is 4, for example q="ABCD", QueryParser generates (when autoGeneratePhraseQueries is true) PhraseQuery("AB BC CD") with slop 0. But it can be optimized PhraseQuery("AB CD") with appropriate positions.
> The idea came from the Japanese paper "N.M-gram: Implementation of Inverted Index Using N-gram with Hash Values" by Mikio Hirabayashi, et al. (The main theme of the paper is different from the idea that I'm using here, though)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3426) optimizer for n-gram PhraseQuery
Posted by "Koji Sekiguchi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13102408#comment-13102408 ]
Koji Sekiguchi commented on LUCENE-3426:
----------------------------------------
I'm not sure it could apply MutiPhraseQuery. Let me take more time.
Considering hashCode/equals is good point. I'll see.
> optimizer for n-gram PhraseQuery
> --------------------------------
>
> Key: LUCENE-3426
> URL: https://issues.apache.org/jira/browse/LUCENE-3426
> Project: Lucene - Java
> Issue Type: Improvement
> Components: core/search
> Reporter: Koji Sekiguchi
> Priority: Trivial
> Attachments: LUCENE-3426.patch, LUCENE-3426.patch, LUCENE-3426.patch, LUCENE-3426.patch, PerfTest.java, PerfTest.java
>
>
> If 2-gram is used and the length of query string is 4, for example q="ABCD", QueryParser generates (when autoGeneratePhraseQueries is true) PhraseQuery("AB BC CD") with slop 0. But it can be optimized PhraseQuery("AB CD") with appropriate positions.
> The idea came from the Japanese paper "N.M-gram: Implementation of Inverted Index Using N-gram with Hash Values" by Mikio Hirabayashi, et al. (The main theme of the paper is different from the idea that I'm using here, though)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3426) optimizer for n-gram PhraseQuery
Posted by "Koji Sekiguchi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Koji Sekiguchi updated LUCENE-3426:
-----------------------------------
Attachment: LUCENE-3426.patch
{quote}
I think I like it better too... though I wonder if its possible to keep the original NGramPhraseQuery unmodified?
this way its not changed by Query.rewrite(), and if a user reuses the query (which we document they can do), they could then call add() again and everything works.
{quote}
I wonder it that too. Here is the new patch. This time I added assertSame()/NotSame() to check the rewritten Query to test code.
> optimizer for n-gram PhraseQuery
> --------------------------------
>
> Key: LUCENE-3426
> URL: https://issues.apache.org/jira/browse/LUCENE-3426
> Project: Lucene - Java
> Issue Type: Improvement
> Components: core/search
> Reporter: Koji Sekiguchi
> Priority: Trivial
> Attachments: LUCENE-3426.patch, LUCENE-3426.patch, LUCENE-3426.patch, LUCENE-3426.patch, PerfTest.java
>
>
> If 2-gram is used and the length of query string is 4, for example q="ABCD", QueryParser generates (when autoGeneratePhraseQueries is true) PhraseQuery("AB BC CD") with slop 0. But it can be optimized PhraseQuery("AB CD") with appropriate positions.
> The idea came from the Japanese paper "N.M-gram: Implementation of Inverted Index Using N-gram with Hash Values" by Mikio Hirabayashi, et al. (The main theme of the paper is different from the idea that I'm using here, though)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Issue Comment Edited] (LUCENE-3426) optimizer for n-gram
PhraseQuery
Posted by "Koji Sekiguchi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13102405#comment-13102405 ]
Koji Sekiguchi edited comment on LUCENE-3426 at 9/12/11 2:02 AM:
-----------------------------------------------------------------
For automatic in Solr, I wonder if we could move the feature to n-gram tokenizers, and we could have something like:
{code}
<fieldType name="text_cjk" class="solr.TextField" positionIncrementGap="100"
autoGeneratePhraseQueries="true">
<analyzer type="index">
<tokenizer class="solr.CJKTokenizerFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.CJKTokenizerFactory" optimizePhraseQuery="true"/>
</analyzer>
</fieldType>
{code}
was (Author: koji):
For automatic in Solr, I wonder if we could move the feature to n-gram tokenizers, and we could have something like:
{code}
<fieldType name="text_cjk" class="solr.TextField" positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.CJKTokenizerFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.CJKTokenizerFactory" optimizePhraseQuery="true"/>
</analyzer>
</fieldType>
{code}
> optimizer for n-gram PhraseQuery
> --------------------------------
>
> Key: LUCENE-3426
> URL: https://issues.apache.org/jira/browse/LUCENE-3426
> Project: Lucene - Java
> Issue Type: Improvement
> Components: core/search
> Reporter: Koji Sekiguchi
> Priority: Trivial
> Attachments: LUCENE-3426.patch, LUCENE-3426.patch, LUCENE-3426.patch, LUCENE-3426.patch, PerfTest.java, PerfTest.java
>
>
> If 2-gram is used and the length of query string is 4, for example q="ABCD", QueryParser generates (when autoGeneratePhraseQueries is true) PhraseQuery("AB BC CD") with slop 0. But it can be optimized PhraseQuery("AB CD") with appropriate positions.
> The idea came from the Japanese paper "N.M-gram: Implementation of Inverted Index Using N-gram with Hash Values" by Mikio Hirabayashi, et al. (The main theme of the paper is different from the idea that I'm using here, though)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Resolved] (LUCENE-3426) optimizer for n-gram PhraseQuery
Posted by "Koji Sekiguchi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Koji Sekiguchi resolved LUCENE-3426.
------------------------------------
Resolution: Fixed
trunk: Committed revision 1170586.
3x: Committed revision 1170593.
> optimizer for n-gram PhraseQuery
> --------------------------------
>
> Key: LUCENE-3426
> URL: https://issues.apache.org/jira/browse/LUCENE-3426
> Project: Lucene - Java
> Issue Type: Improvement
> Components: core/search
> Affects Versions: 2.9.4, 3.0.3, 3.1, 3.2, 3.3, 3.4, 4.0
> Reporter: Koji Sekiguchi
> Assignee: Koji Sekiguchi
> Priority: Trivial
> Fix For: 3.5, 4.0
>
> Attachments: LUCENE-3426.patch, LUCENE-3426.patch, LUCENE-3426.patch, LUCENE-3426.patch, LUCENE-3426.patch, LUCENE-3426.patch, PerfTest.java, PerfTest.java
>
>
> If 2-gram is used and the length of query string is 4, for example q="ABCD", QueryParser generates (when autoGeneratePhraseQueries is true) PhraseQuery("AB BC CD") with slop 0. But it can be optimized PhraseQuery("AB CD") with appropriate positions.
> The idea came from the Japanese paper "N.M-gram: Implementation of Inverted Index Using N-gram with Hash Values" by Mikio Hirabayashi, et al. (The main theme of the paper is different from the idea that I'm using here, though)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3426) optimizer for n-gram PhraseQuery
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13103750#comment-13103750 ]
Robert Muir commented on LUCENE-3426:
-------------------------------------
I think I agree Koji, the patch looks good.
Though we should be able to keep PhraseQuery's internal members 'private' since NGramPhraseQuery now uses getter methods to access positions, terms, slop ?
> optimizer for n-gram PhraseQuery
> --------------------------------
>
> Key: LUCENE-3426
> URL: https://issues.apache.org/jira/browse/LUCENE-3426
> Project: Lucene - Java
> Issue Type: Improvement
> Components: core/search
> Reporter: Koji Sekiguchi
> Priority: Trivial
> Attachments: LUCENE-3426.patch, LUCENE-3426.patch, LUCENE-3426.patch, LUCENE-3426.patch, LUCENE-3426.patch, PerfTest.java, PerfTest.java
>
>
> If 2-gram is used and the length of query string is 4, for example q="ABCD", QueryParser generates (when autoGeneratePhraseQueries is true) PhraseQuery("AB BC CD") with slop 0. But it can be optimized PhraseQuery("AB CD") with appropriate positions.
> The idea came from the Japanese paper "N.M-gram: Implementation of Inverted Index Using N-gram with Hash Values" by Mikio Hirabayashi, et al. (The main theme of the paper is different from the idea that I'm using here, though)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3426) optimizer for n-gram PhraseQuery
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13102271#comment-13102271 ]
Robert Muir commented on LUCENE-3426:
-------------------------------------
Hi Koji, I wonder if instead it would be cleaner as a subclass of PhraseQuery (NGramPhraseQuery or similar),
that rewrites to the (possibly optimized) PhraseQuery in rewrite(). For example, it would build an optimized
PhraseQuery when slop = 0, and there are enough terms to optimize, otherwise it would build a "normal" phrasequery.
Then the optimization would be easy to apply, the user just uses NGramPhraseQuery instead of PhraseQuery.
for example, from QueryParser:
{noformat}
@Override
protected PhraseQuery newPhraseQuery() {
return new NGramPhraseQuery();
}
{noformat}
> optimizer for n-gram PhraseQuery
> --------------------------------
>
> Key: LUCENE-3426
> URL: https://issues.apache.org/jira/browse/LUCENE-3426
> Project: Lucene - Java
> Issue Type: Improvement
> Components: core/search
> Reporter: Koji Sekiguchi
> Priority: Trivial
> Attachments: LUCENE-3426.patch, LUCENE-3426.patch, PerfTest.java
>
>
> If 2-gram is used and the length of query string is 4, for example q="ABCD", QueryParser generates (when autoGeneratePhraseQueries is true) PhraseQuery("AB BC CD") with slop 0. But it can be optimized PhraseQuery("AB CD") with appropriate positions.
> The idea came from the Japanese paper "N.M-gram: Implementation of Inverted Index Using N-gram with Hash Values" by Mikio Hirabayashi, et al. (The main theme of the paper is different from the idea that I'm using here, though)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3426) optimizer for n-gram PhraseQuery
Posted by "Koji Sekiguchi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Koji Sekiguchi updated LUCENE-3426:
-----------------------------------
Attachment: LUCENE-3426.patch
Thank you for your continuous review the patch, Robert! Here is the new patch. Now I don't touch the existing PhraseQuery as I use getter methods.
I think this is ready to commit.
> optimizer for n-gram PhraseQuery
> --------------------------------
>
> Key: LUCENE-3426
> URL: https://issues.apache.org/jira/browse/LUCENE-3426
> Project: Lucene - Java
> Issue Type: Improvement
> Components: core/search
> Reporter: Koji Sekiguchi
> Priority: Trivial
> Attachments: LUCENE-3426.patch, LUCENE-3426.patch, LUCENE-3426.patch, LUCENE-3426.patch, LUCENE-3426.patch, LUCENE-3426.patch, PerfTest.java, PerfTest.java
>
>
> If 2-gram is used and the length of query string is 4, for example q="ABCD", QueryParser generates (when autoGeneratePhraseQueries is true) PhraseQuery("AB BC CD") with slop 0. But it can be optimized PhraseQuery("AB CD") with appropriate positions.
> The idea came from the Japanese paper "N.M-gram: Implementation of Inverted Index Using N-gram with Hash Values" by Mikio Hirabayashi, et al. (The main theme of the paper is different from the idea that I'm using here, though)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org