You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Jamie <ja...@mailarchiva.com> on 2014/06/09 11:57:10 UTC
searching with stemming
Greetings
Our app currently uses language specific analysers (e.g.
EnglishAnalyzer, GermanAnalyzer, etc.). We need an option to disable
stemming. What's the recommended way to do this? These analyzers do not
include an option to disable stemming, only a parameter to specify a
list words for which stemming should not apply. Furthermore, my
understanding is that the StandardAnalyzer is tied to English
specifically. I am trying to avoid having to override each of these
analyzers with an option to disable stemming. Is there a better alternative?
Much appreciate your consideration.
Jamie
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: searching with stemming
Posted by Jamie <ja...@mailarchiva.com>.
Done. https://issues.apache.org/jira/browse/LUCENE-5749
On 2014/06/10, 1:18 AM, Jack Krupansky wrote:
> Please do file a Jira. I'm sure the discussion will be interesting.
>
> -- Jack Krupansky
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: searching with stemming
Posted by Jack Krupansky <ja...@basetechnology.com>.
Please do file a Jira. I'm sure the discussion will be interesting.
-- Jack Krupansky
-----Original Message-----
From: Jamie
Sent: Monday, June 9, 2014 9:33 AM
To: java-user@lucene.apache.org
Subject: Re: searching with stemming
Jack
Thanks. I figured as much.
I'm modifying each analyzer with constructors that take a Stem argument:
public enum Stem { AGGRESSIVE, LIGHT, NONE };
This is obviously, not ideal, 20 or more Lucene classes must be
updated. I now need to maintain each analyzer.
Regards
Jamie
On 2014/06/09, 3:15 PM, Jack Krupansky wrote:
> I find the weak Javadoc even more troubling:
> http://lucene.apache.org/core/4_8_0/analyzers-common/org/apache/lucene/analysis/en/EnglishAnalyzer.html
>
> The real bottom line is that you are expected to "roll your own" in this
> area.
>
> Feel free to file a Jira suggesting your suggested improvement.
>
> -- Jack Krupansky
>
> -----Original Message----- From: Jamie
> Sent: Monday, June 9, 2014 6:56 AM
> To: java-user@lucene.apache.org
> Subject: Re: searching with stemming
>
> To me, it seems strange that these default analyzers, don't provide
> constructors that enable one to override stemming, etc?
>
> On 2014/06/09, 12:39 PM, Trejkaz wrote:
>> On Mon, Jun 9, 2014 at 7:57 PM, Jamie <ja...@mailarchiva.com> wrote:
>>> Greetings
>>>
>>> Our app currently uses language specific analysers (e.g.
>>> EnglishAnalyzer,
>>> GermanAnalyzer, etc.). We need an option to disable stemming. What's the
>>> recommended way to do this? These analyzers do not include an option to
>>> disable stemming, only a parameter to specify a list words for which
>>> stemming should not apply.
>>> Furthermore, my understanding is that the StandardAnalyzer is tied to
>>> English specifically.
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: searching with stemming
Posted by Jamie <ja...@mailarchiva.com>.
Jack
Thanks. I figured as much.
I'm modifying each analyzer with constructors that take a Stem argument:
public enum Stem { AGGRESSIVE, LIGHT, NONE };
This is obviously, not ideal, 20 or more Lucene classes must be
updated. I now need to maintain each analyzer.
Regards
Jamie
On 2014/06/09, 3:15 PM, Jack Krupansky wrote:
> I find the weak Javadoc even more troubling:
> http://lucene.apache.org/core/4_8_0/analyzers-common/org/apache/lucene/analysis/en/EnglishAnalyzer.html
>
>
> The real bottom line is that you are expected to "roll your own" in
> this area.
>
> Feel free to file a Jira suggesting your suggested improvement.
>
> -- Jack Krupansky
>
> -----Original Message----- From: Jamie
> Sent: Monday, June 9, 2014 6:56 AM
> To: java-user@lucene.apache.org
> Subject: Re: searching with stemming
>
> To me, it seems strange that these default analyzers, don't provide
> constructors that enable one to override stemming, etc?
>
> On 2014/06/09, 12:39 PM, Trejkaz wrote:
>> On Mon, Jun 9, 2014 at 7:57 PM, Jamie <ja...@mailarchiva.com> wrote:
>>> Greetings
>>>
>>> Our app currently uses language specific analysers (e.g.
>>> EnglishAnalyzer,
>>> GermanAnalyzer, etc.). We need an option to disable stemming. What's
>>> the
>>> recommended way to do this? These analyzers do not include an option to
>>> disable stemming, only a parameter to specify a list words for which
>>> stemming should not apply.
>>> Furthermore, my understanding is that the StandardAnalyzer is tied to
>>> English specifically.
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: searching with stemming
Posted by Jack Krupansky <ja...@basetechnology.com>.
I find the weak Javadoc even more troubling:
http://lucene.apache.org/core/4_8_0/analyzers-common/org/apache/lucene/analysis/en/EnglishAnalyzer.html
The real bottom line is that you are expected to "roll your own" in this
area.
Feel free to file a Jira suggesting your suggested improvement.
-- Jack Krupansky
-----Original Message-----
From: Jamie
Sent: Monday, June 9, 2014 6:56 AM
To: java-user@lucene.apache.org
Subject: Re: searching with stemming
To me, it seems strange that these default analyzers, don't provide
constructors that enable one to override stemming, etc?
On 2014/06/09, 12:39 PM, Trejkaz wrote:
> On Mon, Jun 9, 2014 at 7:57 PM, Jamie <ja...@mailarchiva.com> wrote:
>> Greetings
>>
>> Our app currently uses language specific analysers (e.g. EnglishAnalyzer,
>> GermanAnalyzer, etc.). We need an option to disable stemming. What's the
>> recommended way to do this? These analyzers do not include an option to
>> disable stemming, only a parameter to specify a list words for which
>> stemming should not apply.
>> Furthermore, my understanding is that the StandardAnalyzer is tied to
>> English specifically.
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: searching with stemming
Posted by Jamie <ja...@mailarchiva.com>.
Benson. Thanks. I was just hoping to avoid a whole bunch of boilerplate.
On 2014/06/09, 1:07 PM, Benson Margulies wrote:
> Analyzer classes are optional; an analyzer is just a factory for a set of
> token stream components. you can usually do just fine with an anonymous
> class. Or in your case, the only thing different for each language will be
> the stop words, so you can have one analyzer class with a language
> parameter.
> On Jun 9, 2014 7:02 AM, "Jamie" <ja...@mailarchiva.com> wrote:
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: searching with stemming
Posted by Benson Margulies <be...@basistech.com>.
Analyzer classes are optional; an analyzer is just a factory for a set of
token stream components. you can usually do just fine with an anonymous
class. Or in your case, the only thing different for each language will be
the stop words, so you can have one analyzer class with a language
parameter.
On Jun 9, 2014 7:02 AM, "Jamie" <ja...@mailarchiva.com> wrote:
> I am not using Solr. I am using the default analyzers...
>
> On 2014/06/09, 12:59 PM, Benson Margulies wrote:
>
>> Are you using Solr? If so you are on the wrong mailing list. If not, why
>> do
>> you need a non-
>> -anonymous analyzer at all.
>> On Jun 9, 2014 6:55 AM, "Jamie" <ja...@mailarchiva.com> wrote:
>>
>> To me, it seems strange that these default analyzers, don't provide
>>> constructors that enable one to override stemming, etc?
>>>
>>> On 2014/06/09, 12:39 PM, Trejkaz wrote:
>>>
>>> On Mon, Jun 9, 2014 at 7:57 PM, Jamie <ja...@mailarchiva.com> wrote:
>>>>
>>>> Greetings
>>>>>
>>>>> Our app currently uses language specific analysers (e.g.
>>>>> EnglishAnalyzer,
>>>>> GermanAnalyzer, etc.). We need an option to disable stemming. What's
>>>>> the
>>>>> recommended way to do this? These analyzers do not include an option to
>>>>> disable stemming, only a parameter to specify a list words for which
>>>>> stemming should not apply.
>>>>> Furthermore, my understanding is that the StandardAnalyzer is tied to
>>>>> English specifically.
>>>>>
>>>>>
>>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: searching with stemming
Posted by Jamie <ja...@mailarchiva.com>.
I am not using Solr. I am using the default analyzers...
On 2014/06/09, 12:59 PM, Benson Margulies wrote:
> Are you using Solr? If so you are on the wrong mailing list. If not, why do
> you need a non-
> -anonymous analyzer at all.
> On Jun 9, 2014 6:55 AM, "Jamie" <ja...@mailarchiva.com> wrote:
>
>> To me, it seems strange that these default analyzers, don't provide
>> constructors that enable one to override stemming, etc?
>>
>> On 2014/06/09, 12:39 PM, Trejkaz wrote:
>>
>>> On Mon, Jun 9, 2014 at 7:57 PM, Jamie <ja...@mailarchiva.com> wrote:
>>>
>>>> Greetings
>>>>
>>>> Our app currently uses language specific analysers (e.g. EnglishAnalyzer,
>>>> GermanAnalyzer, etc.). We need an option to disable stemming. What's the
>>>> recommended way to do this? These analyzers do not include an option to
>>>> disable stemming, only a parameter to specify a list words for which
>>>> stemming should not apply.
>>>> Furthermore, my understanding is that the StandardAnalyzer is tied to
>>>> English specifically.
>>>>
>>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: searching with stemming
Posted by Benson Margulies <be...@basistech.com>.
Are you using Solr? If so you are on the wrong mailing list. If not, why do
you need a non-
-anonymous analyzer at all.
On Jun 9, 2014 6:55 AM, "Jamie" <ja...@mailarchiva.com> wrote:
> To me, it seems strange that these default analyzers, don't provide
> constructors that enable one to override stemming, etc?
>
> On 2014/06/09, 12:39 PM, Trejkaz wrote:
>
>> On Mon, Jun 9, 2014 at 7:57 PM, Jamie <ja...@mailarchiva.com> wrote:
>>
>>> Greetings
>>>
>>> Our app currently uses language specific analysers (e.g. EnglishAnalyzer,
>>> GermanAnalyzer, etc.). We need an option to disable stemming. What's the
>>> recommended way to do this? These analyzers do not include an option to
>>> disable stemming, only a parameter to specify a list words for which
>>> stemming should not apply.
>>> Furthermore, my understanding is that the StandardAnalyzer is tied to
>>> English specifically.
>>>
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: searching with stemming
Posted by Jamie <ja...@mailarchiva.com>.
To me, it seems strange that these default analyzers, don't provide
constructors that enable one to override stemming, etc?
On 2014/06/09, 12:39 PM, Trejkaz wrote:
> On Mon, Jun 9, 2014 at 7:57 PM, Jamie <ja...@mailarchiva.com> wrote:
>> Greetings
>>
>> Our app currently uses language specific analysers (e.g. EnglishAnalyzer,
>> GermanAnalyzer, etc.). We need an option to disable stemming. What's the
>> recommended way to do this? These analyzers do not include an option to
>> disable stemming, only a parameter to specify a list words for which
>> stemming should not apply.
>> Furthermore, my understanding is that the StandardAnalyzer is tied to
>> English specifically.
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: searching with stemming
Posted by Trejkaz <tr...@trypticon.org>.
On Mon, Jun 9, 2014 at 7:57 PM, Jamie <ja...@mailarchiva.com> wrote:
> Greetings
>
> Our app currently uses language specific analysers (e.g. EnglishAnalyzer,
> GermanAnalyzer, etc.). We need an option to disable stemming. What's the
> recommended way to do this? These analyzers do not include an option to
> disable stemming, only a parameter to specify a list words for which
> stemming should not apply.
> Furthermore, my understanding is that the StandardAnalyzer is tied to
> English specifically.
I would say that StandardAnalyzer is actually a weird mix. UAX#29
(what StandardTokenizer is implementing) has rules which are not
convenient for analysing English (e.g. it doesn't break on colons nor
underscores) and ultimately if you want English-friendly tokenisation,
you should be using additional filters or customising the analyser
itself to work around these shortcomings.
Presumably EnglishAnalyzer is already working around these (or if it
isn't, it should. I don't know, because we don't use it.)
> I am trying to avoid having to override each of these analyzers with an option
> to disable stemming. Is there a better alternative?
Rather than using the Analyzer classes, we use the TokeniserFactory
and TokenFilterFactory (actually our own alternatives with the same
names - we're still on an older version of Lucene) and a single
Analyzer class which is configured by passing in the appropriate
factories.
Then there is a separate abstraction of analysis language, which takes
the stemming setting and whatever other settings you might have, and
creates the appropriate list of factories.
This way, you still get the reuse, but also gain an additional form of
backwards compatibility, since even if you change what filters are
used from version to version, you can store off what specific filters
were used to create each index.
TX
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: searching with stemming
Posted by Jamie <ja...@mailarchiva.com>.
Benson
Yes, I can of course do this, as far I can see I would have to override
each analyzer. This is a pain.
Regards
Jamie
On 2014/06/09, 12:29 PM, Benson Margulies wrote:
> You should construct an analysis chain that does what you need. Read the
> source of the relevant analyzer and pick the tokenizer and filter(s) that
> you need, and don't include stemming.
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: searching with stemming
Posted by Benson Margulies <be...@basistech.com>.
You should construct an analysis chain that does what you need. Read the
source of the relevant analyzer and pick the tokenizer and filter(s) that
you need, and don't include stemming.
On Mon, Jun 9, 2014 at 5:57 AM, Jamie <ja...@mailarchiva.com> wrote:
> Greetings
>
> Our app currently uses language specific analysers (e.g. EnglishAnalyzer,
> GermanAnalyzer, etc.). We need an option to disable stemming. What's the
> recommended way to do this? These analyzers do not include an option to
> disable stemming, only a parameter to specify a list words for which
> stemming should not apply. Furthermore, my understanding is that the
> StandardAnalyzer is tied to English specifically. I am trying to avoid
> having to override each of these analyzers with an option to disable
> stemming. Is there a better alternative?
>
> Much appreciate your consideration.
>
> Jamie
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>