[jira] [Closed] (MADLIB-902) Stemming

["Frank McQuillan (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/MADLIB-902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Frank McQuillan closed MADLIB-902.
----------------------------------

> Stemming
> --------
>
>                 Key: MADLIB-902
>                 URL: https://issues.apache.org/jira/browse/MADLIB-902
>             Project: Apache MADlib
>          Issue Type: New Feature
>          Components: Module: Utilities
>            Reporter: Frank McQuillan
>            Assignee: Rahul Iyer
>             Fix For: v1.9
>
>         Attachments: MADlib_ Stemming.pdf
>
>
> Story
> As a data scientist, I want to perform stemming on my data set to reduce words to their word stem, so that I can prepare for input to predictive analytics algorithms.
> Details
> 1) Choice of stemming algorithm.  Suggest use the Porter2 stemmer [2].  The ANSI C code for the Porter2 stemmer is available at [3].
> As Porter reports:  “after twenty years of use of the Porter stemmer, certain improvements did suggest themselves, and a new algorithm for English is therefore offered here. (It could be called the ‘Porter2’ stemmer to distinguish it from the Porter stemmer, from which it derives.)”
> Both Natural Language Toolkit (NLTK) uses Porter2.
> 2) Language - English only. No other languages required for now.
> 3) Interface should support both individual tokens and arrays of tokens, e.g.:
> stem_token(token text)
> stem_token_arr(token text[])
> References
> [1] Original Porter stemmer
> http://snowball.tartarus.org/algorithms/porter/stemmer.html
> [2] Improved Porter2 stemmer
> http://snowball.tartarus.org/algorithms/english/stemmer.html
> [3] ANSI C stemmer and header file from reference [3] above
> http://snowball.tartarus.org/algorithms/english/stem.c
> http://snowball.tartarus.org/algorithms/english/stem.h
> [4] General information on stemming
> https://en.wikipedia.org/wiki/Stemming



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)