[jira] [Resolved] (MAHOUT-808) logical error with term counting in org.apache.mahout.vectorizer.DictionaryVectorizer

["Sean Owen (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/MAHOUT-808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sean Owen resolved MAHOUT-808.
------------------------------

       Resolution: Fixed
    Fix Version/s: 0.6
         Assignee: Sean Owen

Bleh, easy enough I just fixed it by separating out the combiner.

> logical error with term counting in org.apache.mahout.vectorizer.DictionaryVectorizer
> -------------------------------------------------------------------------------------
>
>                 Key: MAHOUT-808
>                 URL: https://issues.apache.org/jira/browse/MAHOUT-808
>             Project: Mahout
>          Issue Type: Bug
>    Affects Versions: 0.5
>            Reporter: Phil
>            Assignee: Sean Owen
>            Priority: Critical
>              Labels: DictionaryVectorizer, TermCountReducer
>             Fix For: 0.6
>
>   Original Estimate: 0.25h
>  Remaining Estimate: 0.25h
>
> when using mahout lda for topic modeling, creating vectors from SequenceFile is essential, (refer to https://cwiki.apache.org/confluence/display/MAHOUT/Creating+Vectors+from+Text) but when the --minSupport was set a little bit larger, I found the term counting not right --- there is a logical error at org.apache.mahout.vectorizer.DictionaryVectorizer.java:line 335
>     job.setCombinerClass(TermCountReducer.class);
> Now turn to line 41 at org.apache.mahout.vectorizer.term.TermCountReducer.java
>     if (sum >= minSupport) {
>       context.write(key, new LongWritable(sum));
>     }
> so some terms would be filtered at Combiner even though they actually could pass through, absolutely this is not what we've expected.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira