You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Howie Wang <ho...@hotmail.com> on 2005/06/08 06:18:18 UTC
Stemming in Nutch
Hi,
It seems that stemming is not working for me in nutch. If a document
has the word "kittens" in it, when I search for "kitten" it is not
being returned. Is there something I need to do to enable or install
support for stemming in English?
Howie
Re: Stemming in Nutch
Posted by kanimesh <ka...@sapient.com>.
howie - can you please send me the code for the same?
Howie Wang wrote:
>
>> > Any tips on how to stem the search query text?
>>
>>Take a look at
>>org.apache.nutch.searcher.Query
>
> Thanks for the tips. I think I have something that works
> for me. I ended up creating a QueryFilter called
> StemmerQueryFilter. It's based off of BasicQueryFilter.
> I just run the stemmer on the phrases/terms of the query
> by passing a StringReader to PorterStemFilter. I also
> had to add PorterStemFilter to the TokenStream in
> CommonGrams.optimizePhrase. The other change
> I mentioned previously was adding the PorterStemFilter
> to NutchDocumentAnalysis.tokenStream.
>
> If anyone is interested in the changes, let me know
> and I'll send them to you. Or maybe it's worth slapping
> onto the Wiki.
>
> Howie
>
>
>
>
--
View this message in context: http://old.nabble.com/Stemming-in-Nutch-tp155112p27849297.html
Sent from the Nutch - User mailing list archive at Nabble.com.
Re: Stemming in Nutch
Posted by Howie Wang <ho...@hotmail.com>.
> > Any tips on how to stem the search query text?
>
>Take a look at
>org.apache.nutch.searcher.Query
Thanks for the tips. I think I have something that works
for me. I ended up creating a QueryFilter called
StemmerQueryFilter. It's based off of BasicQueryFilter.
I just run the stemmer on the phrases/terms of the query
by passing a StringReader to PorterStemFilter. I also
had to add PorterStemFilter to the TokenStream in
CommonGrams.optimizePhrase. The other change
I mentioned previously was adding the PorterStemFilter
to NutchDocumentAnalysis.tokenStream.
If anyone is interested in the changes, let me know
and I'll send them to you. Or maybe it's worth slapping
onto the Wiki.
Howie
Re: Stemming in Nutch
Posted by Jérôme Charron <je...@gmail.com>.
> Any tips on how to stem the search query text?
Take a look at
org.apache.nutch.searcher.Query
--
http://motrech.free.fr/
http://frutch.free.fr/
"Explain" shows no matches...
Posted by J B <be...@hotmail.com>.
Hi,
When doing a query containing a special character like "ö", or "ä", the
explanation of the score (as normally showed when clicking "Explain") fails.
For example, when searching for göteborg, I get a number of matches as
expected. However, the Explain-output for every match only shows:
score for query: g teborg
0.0 = match required
As you can see, the special character disappears, creating a "new" search
word, which Nutch of course doesn't find.
Does anyone know why this happens and how it can be solved? Any hints are
greatly appreciated!
Best regards,
Jon
_________________________________________________________________
Auktioner: Tjäna en hacka på gamla prylar http://tradera.msn.se
Re: Stemming in Nutch
Posted by Howie Wang <ho...@hotmail.com>.
I just tried the following code in NutchDocumentAnalysis.java:
public TokenStream tokenStream(String field, Reader reader) {
TokenStream ts = CommonGrams.getFilter(new
NutchDocumentTokenizer(reader), field);
TokenStream ts2 = new LowerCaseFilter(ts);
return new PorterStemFilter(ts2);
It seems to work when parsing documents, but I get problems
when trying to do a search. It stores the stemmed phrases in
the index, but it doesn't do it on search terms. So if "kittens"
is saved as "kitten" in the index, I have to search for "kitten"
to get it returned. Searching for "kittens" doesn't yield any results.
Any tips on how to stem the search query text?
Howie
>Thanks for the info! Is the way to approach it just to call
>PorterStemFilter in NutchDocumentAnalysis.java? Something
>like this:
>
> /** Analyzer used to index textual content. */
> private static class ContentAnalyzer extends Analyzer {
> /** Constructs a {@link NutchDocumentTokenizer}. */
> public TokenStream tokenStream(String field, Reader reader) {
> TokenStream ts = CommonGrams.getFilter(new
>NutchDocumentTokenizer(reader), field);
> return new PorterStemFilter(ts);
> }
> }
>
>Am I completely off-base?
>
>Howie
>
>>From: Andy Liu <an...@gmail.com>
>>
>>There's a couple that have been developed for Lucene. You'd have to
>>modify the Nutch code to use your new stemming analyzer.
>>
>>On 6/8/05, Jérôme Charron <je...@gmail.com> wrote:
>> > > It seems that stemming is not working for me in nutch. If a document
>> > > has the word "kittens" in it, when I search for "kitten" it is not
>> > > being returned. Is there something I need to do to enable or install
>> > > support for stemming in English?
>> >
>> > As far as I know, it does not seem to me that the Nutch Analyzer
>>performs
>> > stemming.
>> > I planned for the next release to write a proposal for integrating
>> > multi-language analyzers in Nutch (like in Lucene).
>> > But for now, as far as I know, there is nothing done on this area.
>> >
>> > Jerome
>> >
>> > --
>> > http://motrech.free.fr/
>> > http://frutch.free.fr/
>> >
>> >
>
>
Re: Stemming in Nutch
Posted by Howie Wang <ho...@hotmail.com>.
Thanks for the info! Is the way to approach it just to call
PorterStemFilter in NutchDocumentAnalysis.java? Something
like this:
/** Analyzer used to index textual content. */
private static class ContentAnalyzer extends Analyzer {
/** Constructs a {@link NutchDocumentTokenizer}. */
public TokenStream tokenStream(String field, Reader reader) {
TokenStream ts = CommonGrams.getFilter(new
NutchDocumentTokenizer(reader), field);
return new PorterStemFilter(ts);
}
}
Am I completely off-base?
Howie
>From: Andy Liu <an...@gmail.com>
>
>There's a couple that have been developed for Lucene. You'd have to
>modify the Nutch code to use your new stemming analyzer.
>
>On 6/8/05, Jérôme Charron <je...@gmail.com> wrote:
> > > It seems that stemming is not working for me in nutch. If a document
> > > has the word "kittens" in it, when I search for "kitten" it is not
> > > being returned. Is there something I need to do to enable or install
> > > support for stemming in English?
> >
> > As far as I know, it does not seem to me that the Nutch Analyzer
>performs
> > stemming.
> > I planned for the next release to write a proposal for integrating
> > multi-language analyzers in Nutch (like in Lucene).
> > But for now, as far as I know, there is nothing done on this area.
> >
> > Jerome
> >
> > --
> > http://motrech.free.fr/
> > http://frutch.free.fr/
> >
> >
Re: Stemming in Nutch
Posted by Andy Liu <an...@gmail.com>.
There's a couple that have been developed for Lucene. You'd have to
modify the Nutch code to use your new stemming analyzer.
On 6/8/05, Jérôme Charron <je...@gmail.com> wrote:
> > It seems that stemming is not working for me in nutch. If a document
> > has the word "kittens" in it, when I search for "kitten" it is not
> > being returned. Is there something I need to do to enable or install
> > support for stemming in English?
>
> As far as I know, it does not seem to me that the Nutch Analyzer performs
> stemming.
> I planned for the next release to write a proposal for integrating
> multi-language analyzers in Nutch (like in Lucene).
> But for now, as far as I know, there is nothing done on this area.
>
> Jerome
>
> --
> http://motrech.free.fr/
> http://frutch.free.fr/
>
>
Re: Stemming in Nutch
Posted by Jérôme Charron <je...@gmail.com>.
> It seems that stemming is not working for me in nutch. If a document
> has the word "kittens" in it, when I search for "kitten" it is not
> being returned. Is there something I need to do to enable or install
> support for stemming in English?
As far as I know, it does not seem to me that the Nutch Analyzer performs
stemming.
I planned for the next release to write a proposal for integrating
multi-language analyzers in Nutch (like in Lucene).
But for now, as far as I know, there is nothing done on this area.
Jerome
--
http://motrech.free.fr/
http://frutch.free.fr/
Re: Stemming in Nutch
Posted by Matthias Jaekle <ja...@eventax.de>.
Hi,
we once did it for German textes:
http://nutch.sourceforge.net/cgi-bin/twiki/view/Main/German
But I think, syntax higlighting does not work correct.
Matthias
Howie Wang schrieb:
> Hi,
>
> It seems that stemming is not working for me in nutch. If a document
> has the word "kittens" in it, when I search for "kitten" it is not
> being returned. Is there something I need to do to enable or install
> support for stemming in English?
>
> Howie
>
>
>
--
http://www.eventax.com - eventax GmbH
http://www.umkreisfinder.de - Die Suchmaschine für Lokales und Events