You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Jamie Johnson (Created) (JIRA)" <ji...@apache.org> on 2012/03/11 03:34:57 UTC
[jira] [Created] (SOLR-3231) Add the ability to KStemmer to
preserve the original token when stemming
Add the ability to KStemmer to preserve the original token when stemming
------------------------------------------------------------------------
Key: SOLR-3231
URL: https://issues.apache.org/jira/browse/SOLR-3231
Project: Solr
Issue Type: Improvement
Components: Schema and Analysis
Affects Versions: 4.0
Reporter: Jamie Johnson
Attachments: KStemFilter.patch
While using the PorterStemmer, I found that there were often times that it was far to aggressive in it's stemming. In my particular case it is unrealistic to provide a protected word list which captures all possible words which should not be stemmed. To avoid this I proposed a solution whereby we store the original token as well as the stemmed token so exact searches would always work. Based on discussions on the mailing list Ahmet Arslan, I believe the attached patch to KStemmer provides the desired capabilities through a configuration parameter. This largely is a copy of the org.apache.lucene.wordnet.SynonymTokenFilter.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3231) Add the ability to KStemmer to
preserve the original token when stemming
Posted by "Jamie Johnson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13227484#comment-13227484 ]
Jamie Johnson commented on SOLR-3231:
-------------------------------------
this should (unless I messed it up which is possible) also produce a token for the original term. For instance if the term was "bricks" it should produce tokens for "bricks" and "brick". If that's not the case please let me know.
{code:title=TestKStemFilterFactory.java|borderStyle=solid}
package org.apache.solr.analysis;
import java.io.Reader;
import java.io.StringReader;
import org.apache.lucene.analysis.MockTokenizer;
import org.apache.lucene.analysis.TokenStream;
import org.apache.lucene.analysis.tokenattributes.CharTermAttribute;
/**
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
/**
* Simple tests to ensure the kstem filter factory is working.
*/
public class TestKStemFilterFactory extends BaseTokenTestCase {
public void testStemming() throws Exception {
Reader reader = new StringReader("bricks");
KStemFilterFactory factory = new KStemFilterFactory();
TokenStream stream = factory.create(new MockTokenizer(reader, MockTokenizer.WHITESPACE, false));
assertTokenStreamContents(stream, new String[] { "bricks", "brick" }, new int[]{1, 0});
}
}
{code}
That is what this tests right?
> Add the ability to KStemmer to preserve the original token when stemming
> ------------------------------------------------------------------------
>
> Key: SOLR-3231
> URL: https://issues.apache.org/jira/browse/SOLR-3231
> Project: Solr
> Issue Type: Improvement
> Components: Schema and Analysis
> Affects Versions: 4.0
> Reporter: Jamie Johnson
> Attachments: KStemFilter.patch
>
>
> While using the PorterStemmer, I found that there were often times that it was far to aggressive in it's stemming. In my particular case it is unrealistic to provide a protected word list which captures all possible words which should not be stemmed. To avoid this I proposed a solution whereby we store the original token as well as the stemmed token so exact searches would always work. Based on discussions on the mailing list Ahmet Arslan, I believe the attached patch to KStemmer provides the desired capabilities through a configuration parameter. This largely is a copy of the org.apache.lucene.wordnet.SynonymTokenFilter.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (SOLR-3231) Add the ability to KStemmer to
preserve the original token when stemming
Posted by "Jamie Johnson (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jamie Johnson updated SOLR-3231:
--------------------------------
Attachment: (was: KStemFilter.patch)
> Add the ability to KStemmer to preserve the original token when stemming
> ------------------------------------------------------------------------
>
> Key: SOLR-3231
> URL: https://issues.apache.org/jira/browse/SOLR-3231
> Project: Solr
> Issue Type: Improvement
> Components: Schema and Analysis
> Affects Versions: 4.0
> Reporter: Jamie Johnson
> Attachments: KStemFilter.patch
>
>
> While using the PorterStemmer, I found that there were often times that it was far to aggressive in it's stemming. In my particular case it is unrealistic to provide a protected word list which captures all possible words which should not be stemmed. To avoid this I proposed a solution whereby we store the original token as well as the stemmed token so exact searches would always work. Based on discussions on the mailing list Ahmet Arslan, I believe the attached patch to KStemmer provides the desired capabilities through a configuration parameter. This largely is a copy of the org.apache.lucene.wordnet.SynonymTokenFilter.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3231) Add the ability to KStemmer to
preserve the original token when stemming
Posted by "Robert Muir (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13238557#comment-13238557 ]
Robert Muir commented on SOLR-3231:
-----------------------------------
I don't think we should approach the problem this way: this is the
same discussion as LUCENE-3415
> Add the ability to KStemmer to preserve the original token when stemming
> ------------------------------------------------------------------------
>
> Key: SOLR-3231
> URL: https://issues.apache.org/jira/browse/SOLR-3231
> Project: Solr
> Issue Type: Improvement
> Components: Schema and Analysis
> Reporter: Jamie Johnson
> Fix For: 4.0
>
> Attachments: KStemFilter.patch
>
>
> While using the PorterStemmer, I found that there were often times that it was far to aggressive in it's stemming. In my particular case it is unrealistic to provide a protected word list which captures all possible words which should not be stemmed. To avoid this I proposed a solution whereby we store the original token as well as the stemmed token so exact searches would always work. Based on discussions on the mailing list Ahmet Arslan, I believe the attached patch to KStemmer provides the desired capabilities through a configuration parameter. This largely is a copy of the org.apache.lucene.wordnet.SynonymTokenFilter.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3231) Add the ability to KStemmer to
preserve the original token when stemming
Posted by "Ryan McKinley (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13227257#comment-13227257 ]
Ryan McKinley commented on SOLR-3231:
-------------------------------------
If I understand the patch, this patch just sets the tokenType attribute to "STEM" right?
> Add the ability to KStemmer to preserve the original token when stemming
> ------------------------------------------------------------------------
>
> Key: SOLR-3231
> URL: https://issues.apache.org/jira/browse/SOLR-3231
> Project: Solr
> Issue Type: Improvement
> Components: Schema and Analysis
> Affects Versions: 4.0
> Reporter: Jamie Johnson
> Attachments: KStemFilter.patch
>
>
> While using the PorterStemmer, I found that there were often times that it was far to aggressive in it's stemming. In my particular case it is unrealistic to provide a protected word list which captures all possible words which should not be stemmed. To avoid this I proposed a solution whereby we store the original token as well as the stemmed token so exact searches would always work. Based on discussions on the mailing list Ahmet Arslan, I believe the attached patch to KStemmer provides the desired capabilities through a configuration parameter. This largely is a copy of the org.apache.lucene.wordnet.SynonymTokenFilter.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (SOLR-3231) Add the ability to KStemmer to
preserve the original token when stemming
Posted by "Jamie Johnson (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jamie Johnson updated SOLR-3231:
--------------------------------
Attachment: KStemFilter.patch
> Add the ability to KStemmer to preserve the original token when stemming
> ------------------------------------------------------------------------
>
> Key: SOLR-3231
> URL: https://issues.apache.org/jira/browse/SOLR-3231
> Project: Solr
> Issue Type: Improvement
> Components: Schema and Analysis
> Affects Versions: 4.0
> Reporter: Jamie Johnson
> Attachments: KStemFilter.patch
>
>
> While using the PorterStemmer, I found that there were often times that it was far to aggressive in it's stemming. In my particular case it is unrealistic to provide a protected word list which captures all possible words which should not be stemmed. To avoid this I proposed a solution whereby we store the original token as well as the stemmed token so exact searches would always work. Based on discussions on the mailing list Ahmet Arslan, I believe the attached patch to KStemmer provides the desired capabilities through a configuration parameter. This largely is a copy of the org.apache.lucene.wordnet.SynonymTokenFilter.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3231) Add the ability to KStemmer to
preserve the original token when stemming
Posted by "Jamie Johnson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13238564#comment-13238564 ]
Jamie Johnson commented on SOLR-3231:
-------------------------------------
Thanks Robert. I just read LUCENE-3415 and understand the approach. My biggest issue is I don't like having to create a separate field to do an exact search, this of course is based on the fact that I was burned by this so perhaps I am biased. It feels like the right thing to do from a user of the API would be to do the least destructive thing, but again I have a specific use case in mind and am not considering all other implications.
> Add the ability to KStemmer to preserve the original token when stemming
> ------------------------------------------------------------------------
>
> Key: SOLR-3231
> URL: https://issues.apache.org/jira/browse/SOLR-3231
> Project: Solr
> Issue Type: Improvement
> Components: Schema and Analysis
> Reporter: Jamie Johnson
> Fix For: 4.0
>
> Attachments: KStemFilter.patch
>
>
> While using the PorterStemmer, I found that there were often times that it was far to aggressive in it's stemming. In my particular case it is unrealistic to provide a protected word list which captures all possible words which should not be stemmed. To avoid this I proposed a solution whereby we store the original token as well as the stemmed token so exact searches would always work. Based on discussions on the mailing list Ahmet Arslan, I believe the attached patch to KStemmer provides the desired capabilities through a configuration parameter. This largely is a copy of the org.apache.lucene.wordnet.SynonymTokenFilter.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (SOLR-3231) Add the ability to KStemmer to
preserve the original token when stemming
Posted by "Jamie Johnson (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jamie Johnson updated SOLR-3231:
--------------------------------
Attachment: KStemFilter.patch
> Add the ability to KStemmer to preserve the original token when stemming
> ------------------------------------------------------------------------
>
> Key: SOLR-3231
> URL: https://issues.apache.org/jira/browse/SOLR-3231
> Project: Solr
> Issue Type: Improvement
> Components: Schema and Analysis
> Affects Versions: 4.0
> Reporter: Jamie Johnson
> Attachments: KStemFilter.patch
>
>
> While using the PorterStemmer, I found that there were often times that it was far to aggressive in it's stemming. In my particular case it is unrealistic to provide a protected word list which captures all possible words which should not be stemmed. To avoid this I proposed a solution whereby we store the original token as well as the stemmed token so exact searches would always work. Based on discussions on the mailing list Ahmet Arslan, I believe the attached patch to KStemmer provides the desired capabilities through a configuration parameter. This largely is a copy of the org.apache.lucene.wordnet.SynonymTokenFilter.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (SOLR-3231) Add the ability to KStemmer to
preserve the original token when stemming
Posted by "Mark Miller (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mark Miller updated SOLR-3231:
------------------------------
Affects Version/s: (was: 4.0)
Fix Version/s: 4.0
> Add the ability to KStemmer to preserve the original token when stemming
> ------------------------------------------------------------------------
>
> Key: SOLR-3231
> URL: https://issues.apache.org/jira/browse/SOLR-3231
> Project: Solr
> Issue Type: Improvement
> Components: Schema and Analysis
> Reporter: Jamie Johnson
> Fix For: 4.0
>
> Attachments: KStemFilter.patch
>
>
> While using the PorterStemmer, I found that there were often times that it was far to aggressive in it's stemming. In my particular case it is unrealistic to provide a protected word list which captures all possible words which should not be stemmed. To avoid this I proposed a solution whereby we store the original token as well as the stemmed token so exact searches would always work. Based on discussions on the mailing list Ahmet Arslan, I believe the attached patch to KStemmer provides the desired capabilities through a configuration parameter. This largely is a copy of the org.apache.lucene.wordnet.SynonymTokenFilter.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org