[jira] Commented: (MAHOUT-442) Simple feature reduction options for Bayes classifiers

["Drew Farris (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MAHOUT-442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12890979#action_12890979 ] 

Drew Farris commented on MAHOUT-442:
------------------------------------

setting -minDf and -minSupport to 2 for all wikipedia chunks and countries generated using the commands above causes a drop from 9.5m features to 2.7m features and 22.3M label/feature pairs to 15.6M label/feature pairs. 


> Simple feature reduction options for Bayes classifiers
> ------------------------------------------------------
>
>                 Key: MAHOUT-442
>                 URL: https://issues.apache.org/jira/browse/MAHOUT-442
>             Project: Mahout
>          Issue Type: Improvement
>          Components: Classification
>    Affects Versions: 0.3
>            Reporter: Drew Farris
>            Assignee: Drew Farris
>         Attachments: MAHOUT-442.patch
>
>
> Adding options to the Bayes TrainClassifier driver to filter features using minimum df or tf. Features that only appear in a handful of documents or less than X times within the entire input set will be removed from the training feature set entirely. This will allow the Bayes classifiers to scale to larger corpora.
> More background: 
> When running the wikipedia example, I discovered that the number of features produced with -ng 1 was pretty outstanding: 9,500,000 using the default settings after running the following commands:
> {code}
> ./bin/mahout org.apache.mahout.classifier.bayes.WikipediaXmlSplitter -d wikipedia/enwiki-20100622-pages-articles.xml.bz2 -owikipedia/chunks -c 64
> ./bin/mahout org.apache.mahout.classifier.bayes.WikipediaDatasetCreatorDriver -i wikipedia/chunks -o wikipedia/bayes-input -c examples/src/test/resources/country.txt
> ./bin/mahout org.apache.mahout.classifier.bayes.TrainClassifier -i wikipedia/bayes-input -o wikipedia/bayes-model -type cbayes -ng 1  -source hdfs
> {code}
> This if course makes testing the classifier tricky on machines of modest means because TestClassifier attempts to load all features into memory on the machines the mapper is running on.
> It appears that Grant ran into a similar issue last year: 
> http://www.lucidimagination.com/search/document/7fff9bc0b3350370/getting_started_with_classification#ba6838a9c8b9090c
> This patch will add --minDf and --minSupport options to TrainClassifier. Also --skipCleanup to prevent the deletion of the output of the BayesFeatureDriver, which can be useful in order to allow inspection the resulting feature set in order to tune rules for feature production.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.