[GitHub] [spark] zhengruifeng commented on pull request #30468: [SPARK-33518][ML][WIP] Improve performance of ML ALS recommendForAll by GEMV

[GitBox <gi...@apache.org>]

zhengruifeng commented on pull request #30468:
URL: https://github.com/apache/spark/pull/30468#issuecomment-732089234


   dataset: [ml-latest/ratings.csv](https://grouplens.org/datasets/movielens/)
   number of users: 283228
   number of items: 53889
   number of ratings: 27753444
   
   env: Ubuntu 20.04
   blas: f2jBLAS
   cmd: bin/spark-shell --driver-memory=64G --conf spark.driver.maxResultSize=10g
   
   train (in 2.4.7):
   ```
   import org.apache.spark.ml.recommendation._
   sc.setLogLevel("OFF")
   
   val df = spark.read.option("header", true).option("inferSchema", "true").csv("/d1/Datasets/ml-latest/ratings.csv")
   
   df.select(countDistinct("userId"), countDistinct("movieId"), count("rating")).head
   org.apache.spark.sql.Row = [283228,53889,27753444]
   
   val als = new ALS().setMaxIter(1).setUserCol("userId").setItemCol("movieId").setRatingCol("rating")
   
   val model = als.fit(df)
   
   model.save("/d0/tmp/ml-latest/als-model")
   ```
   
   
   test code:
   ```
   import org.apache.spark.ml.recommendation._
   sc.setLogLevel("OFF")
   
   val model = ALSModel.load("/d0/tmp/ml-latest/als-model")
   
   
   val start = System.currentTimeMillis;
   model.recommendForAllUsers(10).count
   model.recommendForAllItems(10).count
   val end = System.currentTimeMillis;
   end - start
   ```


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org