You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-dev@lucene.apache.org by "Todd Feak (JIRA)" <ji...@apache.org> on 2008/10/17 19:59:44 UTC
[jira] Created: (SOLR-813) Add new DoubleMetaphone Filter and
Factory
Add new DoubleMetaphone Filter and Factory
------------------------------------------
Key: SOLR-813
URL: https://issues.apache.org/jira/browse/SOLR-813
Project: Solr
Issue Type: New Feature
Components: search
Affects Versions: 1.3
Reporter: Todd Feak
Priority: Minor
The existing PhoneticFilter allows for use of the DoubleMetaphone encoder. However, it doesn't expose the maxCodeLength() setting, and it ignores the alternate encodings that the encoder provides for some words. This new filter is not as generic as the PhoneticFilter, but allows more detailed control over the DoubleMetaphone encoder.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-813) Add new DoubleMetaphone Filter and
Factory
Posted by "Ryan McKinley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ryan McKinley updated SOLR-813:
-------------------------------
Attachment: SOLR-813.patch
oops, last patch had a bug if the stream ended in a non-phonetic value:
{code:java}
if( !isPhonetic ) {
t = next(in);
if( t != null ) {
t.setPositionIncrement( t.getPositionIncrement()+1 );
}
return t;
}
{code}
> Add new DoubleMetaphone Filter and Factory
> ------------------------------------------
>
> Key: SOLR-813
> URL: https://issues.apache.org/jira/browse/SOLR-813
> Project: Solr
> Issue Type: New Feature
> Components: search
> Affects Versions: 1.3
> Reporter: Todd Feak
> Assignee: Ryan McKinley
> Priority: Minor
> Attachments: SOLR-813.patch, SOLR-813.patch, SOLR-813.patch
>
>
> The existing PhoneticFilter allows for use of the DoubleMetaphone encoder. However, it doesn't expose the maxCodeLength() setting, and it ignores the alternate encodings that the encoder provides for some words. This new filter is not as generic as the PhoneticFilter, but allows more detailed control over the DoubleMetaphone encoder.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (SOLR-813) Add new DoubleMetaphone Filter and
Factory
Posted by "Ryan McKinley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ryan McKinley reassigned SOLR-813:
----------------------------------
Assignee: Ryan McKinley
> Add new DoubleMetaphone Filter and Factory
> ------------------------------------------
>
> Key: SOLR-813
> URL: https://issues.apache.org/jira/browse/SOLR-813
> Project: Solr
> Issue Type: New Feature
> Components: search
> Affects Versions: 1.3
> Reporter: Todd Feak
> Assignee: Ryan McKinley
> Priority: Minor
> Attachments: SOLR-813.patch, SOLR-813.patch
>
>
> The existing PhoneticFilter allows for use of the DoubleMetaphone encoder. However, it doesn't expose the maxCodeLength() setting, and it ignores the alternate encodings that the encoder provides for some words. This new filter is not as generic as the PhoneticFilter, but allows more detailed control over the DoubleMetaphone encoder.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-813) Add new DoubleMetaphone Filter and
Factory
Posted by "Todd Feak (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12640640#action_12640640 ]
Todd Feak commented on SOLR-813:
--------------------------------
Other DoubleMetaphone specific calls are made in the filter. Specifically, it checks to see if there is an alternate encoding for the token, and adds that to the stream as well, if it's different then the default encoding. This is part of the strength of the DoubleMetaphone implementation that the Phonetic filter doesn't leverage.
An additional change that I made that *could* be done in the Phonetic filter is blocking empty tokens that are generated on non-alpha strings. This just wraps them all.
> Add new DoubleMetaphone Filter and Factory
> ------------------------------------------
>
> Key: SOLR-813
> URL: https://issues.apache.org/jira/browse/SOLR-813
> Project: Solr
> Issue Type: New Feature
> Components: search
> Affects Versions: 1.3
> Reporter: Todd Feak
> Priority: Minor
> Attachments: SOLR-813.patch
>
>
> The existing PhoneticFilter allows for use of the DoubleMetaphone encoder. However, it doesn't expose the maxCodeLength() setting, and it ignores the alternate encodings that the encoder provides for some words. This new filter is not as generic as the PhoneticFilter, but allows more detailed control over the DoubleMetaphone encoder.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-813) Add new DoubleMetaphone Filter and
Factory
Posted by "Ryan McKinley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ryan McKinley updated SOLR-813:
-------------------------------
Attachment: SOLR-813.patch
Here is an update that adresses two concerns:
1. position increments -- this keeps the tokens in sync with the input
2. previous version would stop processing after a number. That is: "aaa 1234 bbb" would not process "bbb"
3. Token types... this changes it to "DoubleMetaphone" rather then "ALPHANUM"
here is the key part:
{code:java}
boolean isPhonetic = false;
String v = new String(t.termBuffer(), 0, t.termLength());
String primaryPhoneticValue = encoder.doubleMetaphone(v);
if (primaryPhoneticValue.length() > 0) {
Token token = (Token) t.clone();
if( inject ) {
token.setPositionIncrement( 0 );
}
token.setType( TOKEN_TYPE );
token.setTermBuffer(primaryPhoneticValue);
remainingTokens.addLast(token);
isPhonetic = true;
}
String alternatePhoneticValue = encoder.doubleMetaphone(v, true);
if (alternatePhoneticValue.length() > 0
&& !primaryPhoneticValue.equals(alternatePhoneticValue)) {
Token token = (Token) t.clone();
token.setPositionIncrement( 0 );
token.setType( TOKEN_TYPE );
token.setTermBuffer(alternatePhoneticValue);
remainingTokens.addLast(token);
isPhonetic = true;
}
// If we did not add something, then go to the next one...
if( !isPhonetic ) {
t = next(in);
t.setPositionIncrement( t.getPositionIncrement()+1 );
return t;
}
{code}
> Add new DoubleMetaphone Filter and Factory
> ------------------------------------------
>
> Key: SOLR-813
> URL: https://issues.apache.org/jira/browse/SOLR-813
> Project: Solr
> Issue Type: New Feature
> Components: search
> Affects Versions: 1.3
> Reporter: Todd Feak
> Priority: Minor
> Attachments: SOLR-813.patch, SOLR-813.patch
>
>
> The existing PhoneticFilter allows for use of the DoubleMetaphone encoder. However, it doesn't expose the maxCodeLength() setting, and it ignores the alternate encodings that the encoder provides for some words. This new filter is not as generic as the PhoneticFilter, but allows more detailed control over the DoubleMetaphone encoder.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-813) Add new DoubleMetaphone Filter and
Factory
Posted by "Todd Feak (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Feak updated SOLR-813:
---------------------------
Attachment: SOLR-813.patch
Added patch containing Filter, Factor, and Unit Tests for both.
> Add new DoubleMetaphone Filter and Factory
> ------------------------------------------
>
> Key: SOLR-813
> URL: https://issues.apache.org/jira/browse/SOLR-813
> Project: Solr
> Issue Type: New Feature
> Components: search
> Affects Versions: 1.3
> Reporter: Todd Feak
> Priority: Minor
> Attachments: SOLR-813.patch
>
>
> The existing PhoneticFilter allows for use of the DoubleMetaphone encoder. However, it doesn't expose the maxCodeLength() setting, and it ignores the alternate encodings that the encoder provides for some words. This new filter is not as generic as the PhoneticFilter, but allows more detailed control over the DoubleMetaphone encoder.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (SOLR-813) Add new DoubleMetaphone Filter and
Factory
Posted by "Ryan McKinley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ryan McKinley resolved SOLR-813.
--------------------------------
Resolution: Fixed
Fix Version/s: 1.4
Added in rev: 705903
Thanks Todd!
> Add new DoubleMetaphone Filter and Factory
> ------------------------------------------
>
> Key: SOLR-813
> URL: https://issues.apache.org/jira/browse/SOLR-813
> Project: Solr
> Issue Type: New Feature
> Components: search
> Affects Versions: 1.3
> Reporter: Todd Feak
> Assignee: Ryan McKinley
> Priority: Minor
> Fix For: 1.4
>
> Attachments: SOLR-813.patch, SOLR-813.patch, SOLR-813.patch
>
>
> The existing PhoneticFilter allows for use of the DoubleMetaphone encoder. However, it doesn't expose the maxCodeLength() setting, and it ignores the alternate encodings that the encoder provides for some words. This new filter is not as generic as the PhoneticFilter, but allows more detailed control over the DoubleMetaphone encoder.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-813) Add new DoubleMetaphone Filter and
Factory
Posted by "Ryan McKinley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12640637#action_12640637 ]
Ryan McKinley commented on SOLR-813:
------------------------------------
rather then create a new Filter for DoubleMetaphone, why not just extend PhoneticFilter to support maxCodeLength?
Here is a quick untested bit that uses reflection to set the maxCodeLength -- the advantage is that it would also work for 'Metaphone' (though i'm not sure anyone uses that).
Since the reflection only happens once at starup, it is not a big deal.
{code:java}
Index: src/java/org/apache/solr/analysis/PhoneticFilterFactory.java
===================================================================
--- src/java/org/apache/solr/analysis/PhoneticFilterFactory.java (revision 704289)
+++ src/java/org/apache/solr/analysis/PhoneticFilterFactory.java (working copy)
@@ -17,10 +17,10 @@
package org.apache.solr.analysis;
+import java.lang.reflect.Method;
import java.util.HashMap;
import java.util.Map;
-import org.apache.solr.core.SolrConfig;
import org.apache.commons.codec.Encoder;
import org.apache.commons.codec.language.DoubleMetaphone;
import org.apache.commons.codec.language.Metaphone;
@@ -80,6 +80,13 @@
try {
encoder = clazz.newInstance();
+
+ // Try to set the maxCodeLength
+ String v = args.get( "maxCodeLength" );
+ if( v != null ) {
+ Method setter = encoder.getClass().getMethod( "setMaxCodeLength", Integer.class );
+ setter.invoke( encoder, Integer.parseInt( v ) );
+ }
}
catch (Exception e) {
throw new SolrException( SolrException.ErrorCode.SERVER_ERROR, "Error initializing: "+name + "/"+clazz, e );
{code}
> Add new DoubleMetaphone Filter and Factory
> ------------------------------------------
>
> Key: SOLR-813
> URL: https://issues.apache.org/jira/browse/SOLR-813
> Project: Solr
> Issue Type: New Feature
> Components: search
> Affects Versions: 1.3
> Reporter: Todd Feak
> Priority: Minor
> Attachments: SOLR-813.patch
>
>
> The existing PhoneticFilter allows for use of the DoubleMetaphone encoder. However, it doesn't expose the maxCodeLength() setting, and it ignores the alternate encodings that the encoder provides for some words. This new filter is not as generic as the PhoneticFilter, but allows more detailed control over the DoubleMetaphone encoder.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-813) Add new DoubleMetaphone Filter and
Factory
Posted by "Todd Feak (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12641032#action_12641032 ]
Todd Feak commented on SOLR-813:
--------------------------------
Good catch on that bug and enhancements. I put them in my current implementation. Thank you.
> Add new DoubleMetaphone Filter and Factory
> ------------------------------------------
>
> Key: SOLR-813
> URL: https://issues.apache.org/jira/browse/SOLR-813
> Project: Solr
> Issue Type: New Feature
> Components: search
> Affects Versions: 1.3
> Reporter: Todd Feak
> Assignee: Ryan McKinley
> Priority: Minor
> Fix For: 1.4
>
> Attachments: SOLR-813.patch, SOLR-813.patch, SOLR-813.patch
>
>
> The existing PhoneticFilter allows for use of the DoubleMetaphone encoder. However, it doesn't expose the maxCodeLength() setting, and it ignores the alternate encodings that the encoder provides for some words. This new filter is not as generic as the PhoneticFilter, but allows more detailed control over the DoubleMetaphone encoder.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.