You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Erlend Hamnaberg <er...@hamnaberg.net> on 2015/02/23 13:53:07 UTC
Efficient way of scoring all items and users in an ALS model
Hi.
We are using the ALS model, and would like to get all users and items
scored.
currently we have these methods.
https://gist.github.com/hamnis/e396854f4654bd46ebe0
We want to be able to distribute the calculations to the slaves so we dont
have to do this on the master.
Is there an efficient and distributed way of doing this?
I suppose we could collect all items in the product features and send that
into a broadcast, but that needs all items on the master, and we want to
avoid that.
Regards
Erlend
Re: Efficient way of scoring all items and users in an ALS model
Posted by Xiangrui Meng <me...@gmail.com>.
You can use rdd.cartesian then find top-k by key to distribute the
work to executors. There is a trick to boost the performance: you need
to blockify user/product features and then use native matrix-matrix
multiplication. There is a relevant PR from Deb:
https://github.com/apache/spark/pull/3098 . -Xiangrui
On Mon, Feb 23, 2015 at 4:53 AM, Erlend Hamnaberg <er...@hamnaberg.net> wrote:
> Hi.
>
> We are using the ALS model, and would like to get all users and items
> scored.
>
> currently we have these methods.
>
> https://gist.github.com/hamnis/e396854f4654bd46ebe0
>
> We want to be able to distribute the calculations to the slaves so we dont
> have to do this on the master.
>
> Is there an efficient and distributed way of doing this?
>
>
> I suppose we could collect all items in the product features and send that
> into a broadcast, but that needs all items on the master, and we want to
> avoid that.
>
> Regards
>
> Erlend
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org