[jira] [Closed] (FLINK-4713) Implementing ranking evaluation scores for recommender systems

["Chesnay Schepler (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-4713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chesnay Schepler closed FLINK-4713.
-----------------------------------
    Resolution: Won't Do

Closing since flink-ml is effectively frozen.

> Implementing ranking evaluation scores for recommender systems
> --------------------------------------------------------------
>
>                 Key: FLINK-4713
>                 URL: https://issues.apache.org/jira/browse/FLINK-4713
>             Project: Flink
>          Issue Type: New Feature
>          Components: Library / Machine Learning
>            Reporter: Domokos Miklós Kelen
>            Assignee: Gábor Hermann
>            Priority: Major
>
> Follow up work to [4712|https://issues.apache.org/jira/browse/FLINK-4712] includes implementing ranking recommendation evaluation metrics (such as precision@k, recall@k, ndcg@k), [similar to Spark's implementations|https://spark.apache.org/docs/1.5.0/mllib-evaluation-metrics.html#ranking-systems]. It would be beneficial if we were able to design the API such that it could be included in the proposed evaluation framework (see [2157|https://issues.apache.org/jira/browse/FLINK-2157]).
> In it's current form, this would mean generalizing the PredictionType type parameter of the Score class to allow for {{Array[Int]}} or {{Array[(Int, Double)]}}, and outputting the recommendations in the form {{DataSet[(Int, Array[Int])]}} or {{DataSet[(Int, Array[(Int,Double)])]}} meaning (user, array of items), possibly including the predicted scores as well. 
> However, calculating for example nDCG for a given user u requires us to be able to access all of the (u, item, relevance) records in the test dataset, which means we would need to put this information in the second element of the {{DataSet[(PredictionType, PredictionType)]}} input of the scorer function as PredictionType={{Array[(Int, Double)]}}. This is problematic, as this Array could be arbitrarily long.
> Another option is to further rework the proposed evaluation framework to allow us to implement this properly, with inputs in the form of {{recommendations : DataSet[(Int,Int,Int)]}} (user, item, rank) and {{test : DataSet[(Int,Int,Double)]}} (user, item relevance). This way, the scores could be implemented such that they can be calculated in a distributed way.
> The third option is to implement the scorer functions outside the evaluation framework.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)