[jira] [Resolved] (SPARK-14886) RankingMetrics.ndcgAt throw java.lang.ArrayIndexOutOfBoundsException

["Nick Pentreath (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-14886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nick Pentreath resolved SPARK-14886.
------------------------------------
       Resolution: Fixed
    Fix Version/s: 2.0.0

Issue resolved by pull request 12756
[https://github.com/apache/spark/pull/12756]

> RankingMetrics.ndcgAt  throw  java.lang.ArrayIndexOutOfBoundsException
> ----------------------------------------------------------------------
>
>                 Key: SPARK-14886
>                 URL: https://issues.apache.org/jira/browse/SPARK-14886
>             Project: Spark
>          Issue Type: Bug
>          Components: MLlib
>    Affects Versions: 1.6.1
>            Reporter: lichenglin
>            Priority: Minor
>             Fix For: 2.0.0
>
>
> {code} 
> @Since("1.2.0")
>   def ndcgAt(k: Int): Double = {
>     require(k > 0, "ranking position k should be positive")
>     predictionAndLabels.map { case (pred, lab) =>
>       val labSet = lab.toSet
>       if (labSet.nonEmpty) {
>         val labSetSize = labSet.size
>         val n = math.min(math.max(pred.length, labSetSize), k)
>         var maxDcg = 0.0
>         var dcg = 0.0
>         var i = 0
>         while (i < n) {
>           val gain = 1.0 / math.log(i + 2)
>           if (labSet.contains(pred(i))) {
>             dcg += gain
>           }
>           if (i < labSetSize) {
>             maxDcg += gain
>           }
>           i += 1
>         }
>         dcg / maxDcg
>       } else {
>         logWarning("Empty ground truth set, check input data")
>         0.0
>       }
>     }.mean()
>   }
> {code}
> "if (labSet.contains(pred(i)))" will throw ArrayIndexOutOfBoundsException when pred's size less then k.
> That meas the true relevant documents has less size then the param k.
> just try this with sample_movielens_data.txt
> for example set pred.size to 5,labSetSize to 10,k to 20,then the n is 10. pred[10] not exists;
> precisionAt is ok just because it has         
> val n = math.min(pred.length, k)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org