You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Otis Gospodnetic <ot...@yahoo.com> on 2010/07/30 22:41:55 UTC
Good list of English words that get "butchered" by Porter Stemmer
Hello,
I'm looking for a list of English words that, when stemmed by Porter stemmer,
end up in the same stem as some similar, but unrelated words. Below are some
examples:
# this gets stemmed to "iron", so if you search for "ironic", you'll get "iron"
matches
ironic
# same stem as animal
anime
animated
animation
animations
I imagine such a list could be added to the example protwords.txt
Thanks,
Otis
----
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
Lucene ecosystem search :: http://search-lucene.com/
Re: Good list of English words that get "butchered" by Porter Stemmer
Posted by Yonik Seeley <yo...@lucidimagination.com>.
On Fri, Jul 30, 2010 at 4:41 PM, Otis Gospodnetic
<ot...@yahoo.com> wrote:
> I'm looking for a list of English words that, when stemmed by Porter stemmer,
> end up in the same stem as some similar, but unrelated words. Below are some
> examples:
>
> # this gets stemmed to "iron", so if you search for "ironic", you'll get "iron"
> matches
> ironic
>
> # same stem as animal
> anime
> animated
> animation
> animations
>
> I imagine such a list could be added to the example protwords.txt
+1
No reason to make everyone come up with their own list.
Unless a good list already exists out there... we could semi-automate
it by running a large corpus through the stemmer and then for each
stem, list the original words. The manual part would be looking at
the output to see the collisions (unless someone has a better idea).
-Yonik
http://www.lucidimagination.com
RE: Good list of English words that get "butchered" by Porter
Stemmer
Posted by "Burton-West, Tom" <tb...@umich.edu>.
A good starting place might be the list of stemming errors for the original Porter stemmer in this article that describes k-stem:
Krovetz, R. (1993). Viewing morphology as an inference process. In Proceedings of the 16th annual international ACM SIGIR conference on Research and development in information retrieval (pp. 191-202). Pittsburgh, Pennsylvania, United States: ACM. doi:10.1145/160688.160718
I don't know if the current porter stemmer is different. I do see that on the snowball page there is a porter and a porter2 stemmer and this explanation is linked from the porter2 stemmer page: http://snowball.tartarus.org/algorithms/english/stemmer.html
Tom Burton-West
http://www.hathitrust.org/blogs/large-scale-search
-----Original Message-----
From: Otis Gospodnetic [mailto:otis_gospodnetic@yahoo.com]
Sent: Friday, July 30, 2010 4:42 PM
To: solr-user@lucene.apache.org
Subject: Good list of English words that get "butchered" by Porter Stemmer
Hello,
I'm looking for a list of English words that, when stemmed by Porter stemmer,
end up in the same stem as some similar, but unrelated words. Below are some
examples:
# this gets stemmed to "iron", so if you search for "ironic", you'll get "iron"
matches
ironic
# same stem as animal
anime
animated
animation
animations
I imagine such a list could be added to the example protwords.txt
Thanks,
Otis
----
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
Lucene ecosystem search :: http://search-lucene.com/
Re: Good list of English words that get "butchered" by Porter Stemmer
Posted by Robert Muir <rc...@gmail.com>.
Otis,
I think this is a great idea.
you could also go even further by making a better example for
StemmerOverrideFilter (stemdict.txt)
(
http://wiki.apache.org/solr/LanguageAnalysis#solr.StemmerOverrideFilterFactory
)
for example:
animated <tab> animate
animation <tab> animation
animations <tab> animation
this might be a bit better (but more work!) than protected words since then
you could let animation and animations conflate, rather than just forcing
them to be all unchanged. i wouldnt go crazy and worry about animator
matching animation etc, but would at least let plural forms match the
singular, without screwing other things up.
On Fri, Jul 30, 2010 at 4:41 PM, Otis Gospodnetic <
otis_gospodnetic@yahoo.com> wrote:
> Hello,
>
> I'm looking for a list of English words that, when stemmed by Porter
> stemmer,
> end up in the same stem as some similar, but unrelated words. Below are
> some
> examples:
>
> # this gets stemmed to "iron", so if you search for "ironic", you'll get
> "iron"
> matches
> ironic
>
> # same stem as animal
> anime
> animated
> animation
> animations
>
> I imagine such a list could be added to the example protwords.txt
>
> Thanks,
> Otis
> ----
> Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
> Lucene ecosystem search :: http://search-lucene.com/
>
>
--
Robert Muir
rcmuir@gmail.com
Re: Good list of English words that get "butchered" by Porter Stemmer
Posted by Walter Underwood <wu...@wunderwood.org>.
Some collisions are listed here:
http://www.attivio.com/blog/34-attivio-blog/333-doing-things-with-words-part-three-stemming-and-lemmatization.html
Have you asked Martin Porter? You can find his e-mail here: http://tartarus.org/~martin/
wunder
On Jul 30, 2010, at 1:41 PM, Otis Gospodnetic wrote:
> Hello,
>
> I'm looking for a list of English words that, when stemmed by Porter stemmer,
> end up in the same stem as some similar, but unrelated words. Below are some
> examples:
>
> # this gets stemmed to "iron", so if you search for "ironic", you'll get "iron"
> matches
> ironic
>
> # same stem as animal
> anime
> animated
> animation
> animations
>
> I imagine such a list could be added to the example protwords.txt
>
> Thanks,
> Otis
> ----
> Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
> Lucene ecosystem search :: http://search-lucene.com/