[jira] [Commented] (MAHOUT-738) Collocation driver has long being statically cast to an int

["Sean Owen (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MAHOUT-738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13052630#comment-13052630 ] 

Sean Owen commented on MAHOUT-738:
----------------------------------

How about we just make this method operate on longs? seems like the easiest and most straightforward thing.

> Collocation driver has long being statically cast to an int
> -----------------------------------------------------------
>
>                 Key: MAHOUT-738
>                 URL: https://issues.apache.org/jira/browse/MAHOUT-738
>             Project: Mahout
>          Issue Type: Bug
>          Components: Math
>    Affects Versions: 0.5
>            Reporter: peter andrews
>            Priority: Minor
>
> org.apache.mahout.vectorizer.collocations.llr.LLRReducer, which is part of the collocation driver, statically casts a long to an int.
> private long ngramTotal;
> ...
> int k11 = ngram.getFrequency(); /* a&b */
> int k12 = gramFreq[0] - ngram.getFrequency(); /* a&!b */
> int k21 = gramFreq[1] - ngram.getFrequency(); /* !b&a */
> int k22 = (int) (ngramTotal - (gramFreq[0] + gramFreq[1] - ngram.getFrequency())); /* !a&!b */
> These numbers are then fed into 
> org.apache.mahout.math.stats.LogLikelihood
> specifically the function below.
> public static double logLikelihoodRatio(int k11, int k12, int k21, int k22) {
>   // note that we have counts here, not probabilities, and that the entropy is not normalized.
>   double rowEntropy = entropy(k11, k12) + entropy(k21, k22);
>   double columnEntropy = entropy(k11, k21) + entropy(k12, k22);
>   double matrixEntropy = entropy(k11, k12, k21, k22);
>   if (rowEntropy + columnEntropy > matrixEntropy) {
>     // round off error
>     return 0.0;
>   }
>   return 2.0 * (matrixEntropy - rowEntropy - columnEntropy);
> }
> In short if the long ngramTotal is larger than Integer.MAX_VALUE (which will happen in large datasets), then the driver will either crash or in the case that it casts to a negative int, will continue as usual but produce no output due to error checking.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira