You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Antony Mayi <an...@yahoo.com.INVALID> on 2015/02/19 11:15:38 UTC

storing MatrixFactorizationModel (pyspark)

Hi,
when getting the model out of ALS.train it would be beneficial to store it (to disk) so the model can be reused later for any following predictions. I am using pyspark and I had no luck pickling it either using standard pickle module or even dill.
does anyone have a solution for this (note it is pyspark)?
thank you,Antony.

Re: storing MatrixFactorizationModel (pyspark)

Posted by Antony Mayi <an...@yahoo.com.INVALID>.
well, I understand the math (having two vectors) but the python MatrixFactorizationModel object seems to be just a wrapper around java class so not sure how to extract the two RDDs?thx,Antony. 

     On Thursday, 19 February 2015, 16:32, Ilya Ganelin <il...@gmail.com> wrote:
   
 

 Yep. the matrix model had two RDD vectors representing the decomposed matrix. You can save these to disk and re use them. 
On Thu, Feb 19, 2015 at 2:19 AM Antony Mayi <an...@yahoo.com.invalid> wrote:

Hi,
when getting the model out of ALS.train it would be beneficial to store it (to disk) so the model can be reused later for any following predictions. I am using pyspark and I had no luck pickling it either using standard pickle module or even dill.
does anyone have a solution for this (note it is pyspark)?
thank you,Antony.


 
   

Re: storing MatrixFactorizationModel (pyspark)

Posted by Ilya Ganelin <il...@gmail.com>.
Yep. the matrix model had two RDD vectors representing the decomposed
matrix. You can save these to disk and re use them.
On Thu, Feb 19, 2015 at 2:19 AM Antony Mayi <an...@yahoo.com.invalid>
wrote:

> Hi,
>
> when getting the model out of ALS.train it would be beneficial to store it
> (to disk) so the model can be reused later for any following predictions. I
> am using pyspark and I had no luck pickling it either using standard pickle
> module or even dill.
>
> does anyone have a solution for this (note it is pyspark)?
>
> thank you,
> Antony.
>