You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@madlib.apache.org by "Frank McQuillan (JIRA)" <ji...@apache.org> on 2016/03/09 01:21:40 UTC
[jira] [Updated] (MADLIB-902) Stemming
[ https://issues.apache.org/jira/browse/MADLIB-902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Frank McQuillan updated MADLIB-902:
-----------------------------------
Attachment: MADlib_ Stemming.pdf
> Stemming
> --------
>
> Key: MADLIB-902
> URL: https://issues.apache.org/jira/browse/MADLIB-902
> Project: Apache MADlib
> Issue Type: New Feature
> Components: Module: Utilities
> Reporter: Frank McQuillan
> Assignee: Rahul Iyer
> Fix For: v1.9
>
> Attachments: MADlib_ Stemming.pdf
>
>
> Story
> As a data scientist, I want to perform stemming on my data set to reduce words to their word stem, so that I can prepare for input to predictive analytics algorithms.
> Details
> 1) Choice of stemming algorithm. Suggest use the Porter2 stemmer [2]. The ANSI C code for the Porter2 stemmer is available at [3].
> As Porter reports: “after twenty years of use of the Porter stemmer, certain improvements did suggest themselves, and a new algorithm for English is therefore offered here. (It could be called the ‘Porter2’ stemmer to distinguish it from the Porter stemmer, from which it derives.)”
> Both Natural Language Toolkit (NLTK) uses Porter2.
> 2) Language - English only. No other languages required for now.
> 3) Interface should support both individual tokens and arrays of tokens, e.g.:
> stem_token(token text)
> stem_token_arr(token text[])
> References
> [1] Original Porter stemmer
> http://snowball.tartarus.org/algorithms/porter/stemmer.html
> [2] Improved Porter2 stemmer
> http://snowball.tartarus.org/algorithms/english/stemmer.html
> [3] ANSI C stemmer and header file from reference [3] above
> http://snowball.tartarus.org/algorithms/english/stem.c
> http://snowball.tartarus.org/algorithms/english/stem.h
> [4] General information on stemming
> https://en.wikipedia.org/wiki/Stemming
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)