[jira] [Commented] (SPARK-10802) Let ALS recommend for subset of data

["jinglei.chen (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-10802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15046264#comment-15046264 ] 

jinglei.chen commented on SPARK-10802:
--------------------------------------

recommendProductsForUsers(num: Int): RDD[(Int, Array[Rating])] spent much long time, in final, it take error:User class threw exception: org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory hdfs://recomm-30-1.liepin.inc:9000/recommend/result/job/newb already exists

it run error on 1517459 users and 1190671 products, Every time an error, but not error when the dataset is small 

> Let ALS recommend for subset of data
> ------------------------------------
>
>                 Key: SPARK-10802
>                 URL: https://issues.apache.org/jira/browse/SPARK-10802
>             Project: Spark
>          Issue Type: Improvement
>          Components: MLlib
>    Affects Versions: 1.5.0
>            Reporter: Tomasz Bartczak
>            Priority: Minor
>
> Currently MatrixFactorizationModel allows to get recommendations for
> - single user 
> - single product 
> - all users
> - all products
> recommendation for all users/products do a cartesian join inside.
> It would be useful in some cases to get recommendations for subset of users/products by providing an RDD with which MatrixFactorizationModel could do an intersection before doing a cartesian join. This would make it much faster in situation where recommendations are needed only for subset of users/products, and when the subset is still too large to make it feasible to recommend one-by-one.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org