You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by "Wagner,Harry" <wa...@oclc.org> on 2007/08/28 22:08:44 UTC

Solr and KStem

There is a version of the KStem stemmer
(http://ciir.cs.umass.edu/cgi-bin/downloads/downloads.cgi) that has been
adapted for Lucene.  What would be the simplest way to implement this in
Solr?  As a plug-in?  Has anyone already done this?

Thanks... harry

Re: Solr and KStem

Posted by Yonik Seeley <yo...@apache.org>.
Some other notes:
I just read the license... it's nice and short, and appears to be ASL
compatible to me.
We could either include the source in Solr and build it, or add it as
a pre-compiled jar into lib.
The FilterFactory should probably have it's package changed to
org.apache.solr.analysis (definitely if it will be included in source
form in our repository).


-Yonik

On 9/10/07, Mike Klaas <mi...@gmail.com> wrote:
> Hi Harry,
>
> Thanks for your contribution!  Unfortunately, we can't include it in
> Solr unless the necessary legal hurdles are cleared.
>
> An issue needs to be opened on http://issues.apache.org/jira/browse/
> SOLR and you have to attach the file and check the "Grant License to
> ASF" button.  It is also important to verify that you have the legal
> right to grant the code to ASF (since it is probably your employer's
> intellectual property).
>
> Legal issues are a hassle, but are unavoidable, I'm afraid.
>
> Thanks again,
> -Mike
>
> On 10-Sep-07, at 10:22 AM, Wagner,Harry wrote:
>
> > Hi Yonik,
> > The modified KStemmer source is attached. The original KStemFilter is
> > now wrapped (and replaced) by KStemFilterFactory.  I also changed the
> > path to avoid any naming collisions with existing Lucene code.
> >
> > I included the jar file also, for anyone who wants to just drop and
> > play:
> >
> > - put KStem2.jar in your solr/lib directory.
> > - change your schema to use: <filter
> > class="org.oclc.solr.analysis.KStemFilterFactory" cacheSize="20000"/>
> > - restart your app server
> >
> > I don't know if you credit contributions, but if so please include
> > OCLC.
> > Seems only fair since I did this on their dime :)
> >
> > Cheers!
> > harry
> >
> >
> > -----Original Message-----
> > From: yseeley@gmail.com [mailto:yseeley@gmail.com] On Behalf Of Yonik
> > Seeley
> > Sent: Friday, September 07, 2007 3:59 PM
> > To: solr-user@lucene.apache.org
> > Subject: Re: Solr and KStem
> >
> > On 9/7/07, Wagner,Harry <wa...@oclc.org> wrote:
> >> I've implemented a Solr plug-in that wraps KStem for Solr use.  KStem
> > is
> >> considered to be more appropriate for library usage since it is much
> >> less aggressive than Porter (i.e., searches for organization do NOT
> >> match on organ!). If there is any interest in feeding this back into
> >> Solr I would be happy to contribute it.
> >
> > Absolutely.
> > We need to make sure that the license for that k-stemmer is ASL
> > compatible of course.
> >
> > -Yonik
> > <kstem_solr.tar.gz>
>
>

RE: Solr and KStem

Posted by "Wagner,Harry" <wa...@oclc.org>.
Hi Yonik and Mike,
No problem regarding my employer.  I've checked and they are happy to
contribute it.  I'm not sure what to do about the KStem code though.  It
was originally written by Bob Krovetz and then modified for Lucene by
Sergio Guzman-Lara (both from UMASS Amherst).  I modified the Guzman
version for Solr.  Perhaps I should contribute only what I modified,
with instructions for making it work?

Let me know... harry

-----Original Message-----
From: Mike Klaas [mailto:mike.klaas@gmail.com] 
Sent: Monday, September 10, 2007 2:49 PM
To: solr-user@lucene.apache.org
Subject: Re: Solr and KStem

Hi Harry,

Thanks for your contribution!  Unfortunately, we can't include it in  
Solr unless the necessary legal hurdles are cleared.

An issue needs to be opened on http://issues.apache.org/jira/browse/ 
SOLR and you have to attach the file and check the "Grant License to  
ASF" button.  It is also important to verify that you have the legal  
right to grant the code to ASF (since it is probably your employer's  
intellectual property).

Legal issues are a hassle, but are unavoidable, I'm afraid.

Thanks again,
-Mike

On 10-Sep-07, at 10:22 AM, Wagner,Harry wrote:

> Hi Yonik,
> The modified KStemmer source is attached. The original KStemFilter is
> now wrapped (and replaced) by KStemFilterFactory.  I also changed the
> path to avoid any naming collisions with existing Lucene code.
>
> I included the jar file also, for anyone who wants to just drop and
> play:
>
> - put KStem2.jar in your solr/lib directory.
> - change your schema to use: <filter
> class="org.oclc.solr.analysis.KStemFilterFactory" cacheSize="20000"/>
> - restart your app server
>
> I don't know if you credit contributions, but if so please include  
> OCLC.
> Seems only fair since I did this on their dime :)
>
> Cheers!
> harry
>
>
> -----Original Message-----
> From: yseeley@gmail.com [mailto:yseeley@gmail.com] On Behalf Of Yonik
> Seeley
> Sent: Friday, September 07, 2007 3:59 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Solr and KStem
>
> On 9/7/07, Wagner,Harry <wa...@oclc.org> wrote:
>> I've implemented a Solr plug-in that wraps KStem for Solr use.  KStem
> is
>> considered to be more appropriate for library usage since it is much
>> less aggressive than Porter (i.e., searches for organization do NOT
>> match on organ!). If there is any interest in feeding this back into
>> Solr I would be happy to contribute it.
>
> Absolutely.
> We need to make sure that the license for that k-stemmer is ASL
> compatible of course.
>
> -Yonik
> <kstem_solr.tar.gz>


Re: Solr and KStem

Posted by Mike Klaas <mi...@gmail.com>.
Hi Harry,

Thanks for your contribution!  Unfortunately, we can't include it in  
Solr unless the necessary legal hurdles are cleared.

An issue needs to be opened on http://issues.apache.org/jira/browse/ 
SOLR and you have to attach the file and check the "Grant License to  
ASF" button.  It is also important to verify that you have the legal  
right to grant the code to ASF (since it is probably your employer's  
intellectual property).

Legal issues are a hassle, but are unavoidable, I'm afraid.

Thanks again,
-Mike

On 10-Sep-07, at 10:22 AM, Wagner,Harry wrote:

> Hi Yonik,
> The modified KStemmer source is attached. The original KStemFilter is
> now wrapped (and replaced) by KStemFilterFactory.  I also changed the
> path to avoid any naming collisions with existing Lucene code.
>
> I included the jar file also, for anyone who wants to just drop and
> play:
>
> - put KStem2.jar in your solr/lib directory.
> - change your schema to use: <filter
> class="org.oclc.solr.analysis.KStemFilterFactory" cacheSize="20000"/>
> - restart your app server
>
> I don't know if you credit contributions, but if so please include  
> OCLC.
> Seems only fair since I did this on their dime :)
>
> Cheers!
> harry
>
>
> -----Original Message-----
> From: yseeley@gmail.com [mailto:yseeley@gmail.com] On Behalf Of Yonik
> Seeley
> Sent: Friday, September 07, 2007 3:59 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Solr and KStem
>
> On 9/7/07, Wagner,Harry <wa...@oclc.org> wrote:
>> I've implemented a Solr plug-in that wraps KStem for Solr use.  KStem
> is
>> considered to be more appropriate for library usage since it is much
>> less aggressive than Porter (i.e., searches for organization do NOT
>> match on organ!). If there is any interest in feeding this back into
>> Solr I would be happy to contribute it.
>
> Absolutely.
> We need to make sure that the license for that k-stemmer is ASL
> compatible of course.
>
> -Yonik
> <kstem_solr.tar.gz>


RE: Solr and KStem

Posted by "Wagner,Harry" <wa...@oclc.org>.
Hi Piete,
Good idea. Thanks.  One other change that should probably be made is to
change the package statement from org.oclc.solr.analysis to
org.apache.solr.analysis.  Thanks again.

Cheers!
harry

-----Original Message-----
From: Pieter Berkel [mailto:pieter.berkel@gmail.com] 
Sent: Tuesday, October 09, 2007 9:10 PM
To: solr-user@lucene.apache.org
Subject: Re: Solr and KStem

Hi Harry,

I re-discovered this thread last week and have made some minor changes
to
the code (remove deprication warnings) so that it compiles with trunk.
I
think it would be quite useful to get this stemmer into Solr once all
the
legal / licensing issues are resolved.  If there are no objections, I'll
open a JIRA ticket and upload my changes so we can make sure we're all
working with the same code.

cheers,
Piete



On 11/09/2007, Wagner,Harry <wa...@oclc.org> wrote:
>
> Bill,
> Currently it is a plug-in.  Put the lower case filter ahead of kstem,
> just as for porter (example below).  You can use it with porter, but I
> can't imagine why you would want to.  At least not in the same
analyzer.
> Hope this helps.
>
> <fieldtype name="text_kstem" class="solr.TextField">
>       <analyzer type="index">
>         <tokenizer class="solr.WhitespaceTokenizerFactory "/>
>         <filter class="solr.StopFilterFactory" ignoreCase="true"
> words="stopwords.txt"/>
>         <filter class="solr.WordDelimiterFilterFactory"
> generateWordParts="1" generateNumberParts="1" catenateWords="1"
> catenateNumbers="1" catenateAll="0"/>
>         <filter class="solr.LowerCaseFilterFactory"/>
>         <filter class="org.oclc.solr.analysis.KStemFilterFactory"
> cacheSize="20000"/>
>         <filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
>       </analyzer>
>       <analyzer type="query">
>         <tokenizer class=" solr.WhitespaceTokenizerFactory"/>
>         <filter class="solr.SynonymFilterFactory"
> synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
>         <filter class=" solr.StopFilterFactory" ignoreCase="true"
> words="stopwords.txt"/>
>         <filter class="solr.WordDelimiterFilterFactory"
> generateWordParts="1" generateNumberParts="1" catenateWords="0"
> catenateNumbers="0" catenateAll="0"/>
>         <filter class="solr.LowerCaseFilterFactory"/>
>         <filter class="org.oclc.solr.analysis.KStemFilterFactory"
> cacheSize="20000"/>
>         <filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
>       </analyzer>
>     </fieldtype>
>
> Cheers... harry
>
>

Re: Solr and KStem

Posted by Pieter Berkel <pi...@gmail.com>.
Hi Harry,

I re-discovered this thread last week and have made some minor changes to
the code (remove deprication warnings) so that it compiles with trunk.  I
think it would be quite useful to get this stemmer into Solr once all the
legal / licensing issues are resolved.  If there are no objections, I'll
open a JIRA ticket and upload my changes so we can make sure we're all
working with the same code.

cheers,
Piete



On 11/09/2007, Wagner,Harry <wa...@oclc.org> wrote:
>
> Bill,
> Currently it is a plug-in.  Put the lower case filter ahead of kstem,
> just as for porter (example below).  You can use it with porter, but I
> can't imagine why you would want to.  At least not in the same analyzer.
> Hope this helps.
>
> <fieldtype name="text_kstem" class="solr.TextField">
>       <analyzer type="index">
>         <tokenizer class="solr.WhitespaceTokenizerFactory "/>
>         <filter class="solr.StopFilterFactory" ignoreCase="true"
> words="stopwords.txt"/>
>         <filter class="solr.WordDelimiterFilterFactory"
> generateWordParts="1" generateNumberParts="1" catenateWords="1"
> catenateNumbers="1" catenateAll="0"/>
>         <filter class="solr.LowerCaseFilterFactory"/>
>         <filter class="org.oclc.solr.analysis.KStemFilterFactory"
> cacheSize="20000"/>
>         <filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
>       </analyzer>
>       <analyzer type="query">
>         <tokenizer class=" solr.WhitespaceTokenizerFactory"/>
>         <filter class="solr.SynonymFilterFactory"
> synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
>         <filter class=" solr.StopFilterFactory" ignoreCase="true"
> words="stopwords.txt"/>
>         <filter class="solr.WordDelimiterFilterFactory"
> generateWordParts="1" generateNumberParts="1" catenateWords="0"
> catenateNumbers="0" catenateAll="0"/>
>         <filter class="solr.LowerCaseFilterFactory"/>
>         <filter class="org.oclc.solr.analysis.KStemFilterFactory"
> cacheSize="20000"/>
>         <filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
>       </analyzer>
>     </fieldtype>
>
> Cheers... harry
>
>

RE: Solr and KStem

Posted by "Wagner,Harry" <wa...@oclc.org>.
Bill,
Currently it is a plug-in.  Put the lower case filter ahead of kstem,
just as for porter (example below).  You can use it with porter, but I
can't imagine why you would want to.  At least not in the same analyzer.
Hope this helps.

<fieldtype name="text_kstem" class="solr.TextField">
      <analyzer type="index">
        <tokenizer class="solr.WhitespaceTokenizerFactory"/>
        <filter class="solr.StopFilterFactory" ignoreCase="true"
words="stopwords.txt"/>
        <filter class="solr.WordDelimiterFilterFactory"
generateWordParts="1" generateNumberParts="1" catenateWords="1"
catenateNumbers="1" catenateAll="0"/>
        <filter class="solr.LowerCaseFilterFactory"/>
	<filter class="org.oclc.solr.analysis.KStemFilterFactory"
cacheSize="20000"/>
        <filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
      </analyzer>
      <analyzer type="query">
        <tokenizer class="solr.WhitespaceTokenizerFactory"/>
        <filter class="solr.SynonymFilterFactory"
synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
        <filter class="solr.StopFilterFactory" ignoreCase="true"
words="stopwords.txt"/>
        <filter class="solr.WordDelimiterFilterFactory"
generateWordParts="1" generateNumberParts="1" catenateWords="0"
catenateNumbers="0" catenateAll="0"/>
        <filter class="solr.LowerCaseFilterFactory"/>
        <filter class="org.oclc.solr.analysis.KStemFilterFactory"
cacheSize="20000"/>
        <filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
      </analyzer>
    </fieldtype>

Cheers... harry

-----Original Message-----
From: Bill Fowler [mailto:wwfowler@gmail.com] 
Sent: Monday, September 10, 2007 8:33 PM
To: solr-user@lucene.apache.org
Subject: Re: Solr and KStem

Hello,

I would like to test this and have a few questions (please excuse what
may
seem naive questions).

I would like to verify that this is purely a configuration feature --
since
the schema.xml defines the analysis/tokerizer chain no other changes are
required.  Also, the source seems to say that a lower case factory needs
to
be "farther down" the tokenizer chain.  So does this mean that the KStem
factory appears before the lower case filter factory in the schema.xml.
Is
there a recommended (required?) tokenizer factory.  I am using the
WhiteSpaceFactory which seems OK.  Finally, I take it that I need to
remove
the EnglishPorterFilterFactory item in the schema.xml -- or no?

Thanks,

Bill



On 9/10/07, Wagner,Harry <wa...@oclc.org> wrote:
>
> Hi Yonik,
> The modified KStemmer source is attached. The original KStemFilter is
> now wrapped (and replaced) by KStemFilterFactory.  I also changed the
> path to avoid any naming collisions with existing Lucene code.
>
> I included the jar file also, for anyone who wants to just drop and
> play:
>
> - put KStem2.jar in your solr/lib directory.
> - change your schema to use: <filter
> class="org.oclc.solr.analysis.KStemFilterFactory" cacheSize="20000"/>
> - restart your app server
>
> I don't know if you credit contributions, but if so please include
OCLC.
> Seems only fair since I did this on their dime :)
>
> Cheers!
> harry
>
>
> -----Original Message-----
> From: yseeley@gmail.com [mailto:yseeley@gmail.com] On Behalf Of Yonik
> Seeley
> Sent: Friday, September 07, 2007 3:59 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Solr and KStem
>
> On 9/7/07, Wagner,Harry <wa...@oclc.org> wrote:
> > I've implemented a Solr plug-in that wraps KStem for Solr use.
KStem
> is
> > considered to be more appropriate for library usage since it is much
> > less aggressive than Porter (i.e., searches for organization do NOT
> > match on organ!). If there is any interest in feeding this back into
> > Solr I would be happy to contribute it.
>
> Absolutely.
> We need to make sure that the license for that k-stemmer is ASL
> compatible of course.
>
> -Yonik
>
>

Re: Solr and KStem

Posted by Bill Fowler <ww...@gmail.com>.
Hello,

I would like to test this and have a few questions (please excuse what may
seem naive questions).

I would like to verify that this is purely a configuration feature -- since
the schema.xml defines the analysis/tokerizer chain no other changes are
required.  Also, the source seems to say that a lower case factory needs to
be "farther down" the tokenizer chain.  So does this mean that the KStem
factory appears before the lower case filter factory in the schema.xml.  Is
there a recommended (required?) tokenizer factory.  I am using the
WhiteSpaceFactory which seems OK.  Finally, I take it that I need to remove
the EnglishPorterFilterFactory item in the schema.xml -- or no?

Thanks,

Bill



On 9/10/07, Wagner,Harry <wa...@oclc.org> wrote:
>
> Hi Yonik,
> The modified KStemmer source is attached. The original KStemFilter is
> now wrapped (and replaced) by KStemFilterFactory.  I also changed the
> path to avoid any naming collisions with existing Lucene code.
>
> I included the jar file also, for anyone who wants to just drop and
> play:
>
> - put KStem2.jar in your solr/lib directory.
> - change your schema to use: <filter
> class="org.oclc.solr.analysis.KStemFilterFactory" cacheSize="20000"/>
> - restart your app server
>
> I don't know if you credit contributions, but if so please include OCLC.
> Seems only fair since I did this on their dime :)
>
> Cheers!
> harry
>
>
> -----Original Message-----
> From: yseeley@gmail.com [mailto:yseeley@gmail.com] On Behalf Of Yonik
> Seeley
> Sent: Friday, September 07, 2007 3:59 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Solr and KStem
>
> On 9/7/07, Wagner,Harry <wa...@oclc.org> wrote:
> > I've implemented a Solr plug-in that wraps KStem for Solr use.  KStem
> is
> > considered to be more appropriate for library usage since it is much
> > less aggressive than Porter (i.e., searches for organization do NOT
> > match on organ!). If there is any interest in feeding this back into
> > Solr I would be happy to contribute it.
>
> Absolutely.
> We need to make sure that the license for that k-stemmer is ASL
> compatible of course.
>
> -Yonik
>
>

RE: Solr and KStem

Posted by "Wagner,Harry" <wa...@oclc.org>.
Hi Yonik,
The modified KStemmer source is attached. The original KStemFilter is
now wrapped (and replaced) by KStemFilterFactory.  I also changed the
path to avoid any naming collisions with existing Lucene code.

I included the jar file also, for anyone who wants to just drop and
play:

- put KStem2.jar in your solr/lib directory.
- change your schema to use: <filter
class="org.oclc.solr.analysis.KStemFilterFactory" cacheSize="20000"/>
- restart your app server

I don't know if you credit contributions, but if so please include OCLC.
Seems only fair since I did this on their dime :)

Cheers!
harry
  

-----Original Message-----
From: yseeley@gmail.com [mailto:yseeley@gmail.com] On Behalf Of Yonik
Seeley
Sent: Friday, September 07, 2007 3:59 PM
To: solr-user@lucene.apache.org
Subject: Re: Solr and KStem

On 9/7/07, Wagner,Harry <wa...@oclc.org> wrote:
> I've implemented a Solr plug-in that wraps KStem for Solr use.  KStem
is
> considered to be more appropriate for library usage since it is much
> less aggressive than Porter (i.e., searches for organization do NOT
> match on organ!). If there is any interest in feeding this back into
> Solr I would be happy to contribute it.

Absolutely.
We need to make sure that the license for that k-stemmer is ASL
compatible of course.

-Yonik

Re: Solr and KStem

Posted by Walter Underwood <wu...@netflix.com>.
Even if KStem isn't ASL, we could include the plug-in code
with notes about how to get the stemmer. Or, the Solr plug-in
could be contributed to the group that manages the KStem
distribution:

  http://ciir.cs.umass.edu/cgi-bin/downloads/downloads.cgi

wunder

On 9/7/07 12:59 PM, "Yonik Seeley" <yo...@apache.org> wrote:

> On 9/7/07, Wagner,Harry <wa...@oclc.org> wrote:
>> I've implemented a Solr plug-in that wraps KStem for Solr use.  KStem is
>> considered to be more appropriate for library usage since it is much
>> less aggressive than Porter (i.e., searches for organization do NOT
>> match on organ!). If there is any interest in feeding this back into
>> Solr I would be happy to contribute it.
> 
> Absolutely.
> We need to make sure that the license for that k-stemmer is ASL
> compatible of course.
> 
> -Yonik


Re: Solr and KStem

Posted by Yonik Seeley <yo...@apache.org>.
On 9/7/07, Wagner,Harry <wa...@oclc.org> wrote:
> I've implemented a Solr plug-in that wraps KStem for Solr use.  KStem is
> considered to be more appropriate for library usage since it is much
> less aggressive than Porter (i.e., searches for organization do NOT
> match on organ!). If there is any interest in feeding this back into
> Solr I would be happy to contribute it.

Absolutely.
We need to make sure that the license for that k-stemmer is ASL
compatible of course.

-Yonik

RE: Solr and KStem

Posted by "Wagner,Harry" <wa...@oclc.org>.
I've implemented a Solr plug-in that wraps KStem for Solr use.  KStem is
considered to be more appropriate for library usage since it is much
less aggressive than Porter (i.e., searches for organization do NOT
match on organ!). If there is any interest in feeding this back into
Solr I would be happy to contribute it.

Cheers!
harry

-----Original Message-----
From: Wagner,Harry 
Sent: Tuesday, August 28, 2007 4:09 PM
To: 'solr-user@lucene.apache.org'
Subject: Solr and KStem

There is a version of the KStem stemmer
(http://ciir.cs.umass.edu/cgi-bin/downloads/downloads.cgi) that has been
adapted for Lucene.  What would be the simplest way to implement this in
Solr?  As a plug-in?  Has anyone already done this?

Thanks... harry

Re: Solr and KStem

Posted by Mike Klaas <mi...@gmail.com>.
On 28-Aug-07, at 1:08 PM, Wagner,Harry wrote:

> There is a version of the KStem stemmer
> (http://ciir.cs.umass.edu/cgi-bin/downloads/downloads.cgi) that has  
> been
> adapted for Lucene.  What would be the simplest way to implement  
> this in
> Solr?  As a plug-in?  Has anyone already done this?

You should be able to drop it in lib/ and use it via something like  
this in schema.xml:

     <!-- One can also specify an existing Analyzer class that has a
          default constructor via the class attribute on the analyzer  
element
     <fieldType name="text_greek" class="solr.TextField">
       <analyzer class="org.apache.lucene.analysis.el.GreekAnalyzer"/>
     </fieldType>
     -->

If it is a tokenfilter (rather than an Analyzer), you can write a  
little wrapper Factory class (see examples in org/apache/solr/ 
analysis, then use it as follows:

     <fieldType name="text_ws" class="solr.TextField"  
positionIncrementGap="100">
       <analyzer>
         <tokenizer class="solr.WhitespaceTokenizerFactory"/>
       </analyzer>
     </fieldType>

best,
-Mike