You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Erlend Hamnaberg <er...@hamnaberg.net> on 2015/02/23 13:53:07 UTC

Efficient way of scoring all items and users in an ALS model

Hi.

We are using the ALS model, and would like to get all users and items
scored.

currently we have these methods.

https://gist.github.com/hamnis/e396854f4654bd46ebe0

We want to be able to distribute the calculations to the slaves so we dont
have to do this on the master.

Is there an efficient and distributed way of doing this?


I suppose we could collect all items in the product features and send that
into a broadcast, but that needs all items on the master, and we want to
avoid that.

Regards

Erlend

Re: Efficient way of scoring all items and users in an ALS model

Posted by Xiangrui Meng <me...@gmail.com>.
You can use rdd.cartesian then find top-k by key to distribute the
work to executors. There is a trick to boost the performance: you need
to blockify user/product features and then use native matrix-matrix
multiplication. There is a relevant PR from Deb:
https://github.com/apache/spark/pull/3098 . -Xiangrui

On Mon, Feb 23, 2015 at 4:53 AM, Erlend Hamnaberg <er...@hamnaberg.net> wrote:
> Hi.
>
> We are using the ALS model, and would like to get all users and items
> scored.
>
> currently we have these methods.
>
> https://gist.github.com/hamnis/e396854f4654bd46ebe0
>
> We want to be able to distribute the calculations to the slaves so we dont
> have to do this on the master.
>
> Is there an efficient and distributed way of doing this?
>
>
> I suppose we could collect all items in the product features and send that
> into a broadcast, but that needs all items on the master, and we want to
> avoid that.
>
> Regards
>
> Erlend

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org