You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by "Fernando O." <fo...@gmail.com> on 2015/01/06 21:57:01 UTC
[MLLib] storageLevel in ALS
Hi,
I was doing a tests with ALS and I noticed that if I persist the inner
RDDs from a MatrixFactorizationModel the RDD is not replicated, it seems
like the storagelevel is hardcoded to MEMORY_AND_DISK, do you think it
makes sense to make that configurable?
[image: Inline image 1]
Re: [MLLib] storageLevel in ALS
Posted by "Fernando O." <fo...@gmail.com>.
1.2
I run() you have
usersOut.setName("usersOut").persist(StorageLevel.MEMORY_AND_DISK)
productsOut.setName("productsOut").persist(StorageLevel.MEMORY_AND_DISK)
On Wed, Jan 7, 2015, 02:11 Xiangrui Meng <me...@gmail.com> wrote:
> Which Spark version are you using? We made this configurable in 1.1:
>
>
> https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/recommendation/ALS.scala#L202
>
> -Xiangrui
>
> On Tue, Jan 6, 2015 at 12:57 PM, Fernando O. <fo...@gmail.com> wrote:
>
>> Hi,
>> I was doing a tests with ALS and I noticed that if I persist the inner
>> RDDs from a MatrixFactorizationModel the RDD is not replicated, it seems
>> like the storagelevel is hardcoded to MEMORY_AND_DISK, do you think it
>> makes sense to make that configurable?
>> [image: Inline image 1]
>>
>
>
Re: [MLLib] storageLevel in ALS
Posted by "Fernando O." <fo...@gmail.com>.
I guess I can but it would be nicer if that is made a configuration, I can
create the issue, test and PR if you guys think its appropiate
On Wed, Jan 7, 2015 at 1:41 PM, Sean Owen <so...@cloudera.com> wrote:
> Ah, Fernando means the usersOut / productsOut RDDs, not the intermediate
> links RDDs.
> Can you unpersist() them, and persist() again at the desired level? the
> downside is that this might mean recomputing and repersisting the RDDs.
>
> On Wed, Jan 7, 2015 at 5:11 AM, Xiangrui Meng <me...@gmail.com> wrote:
>
>> Which Spark version are you using? We made this configurable in 1.1:
>>
>>
>> https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/recommendation/ALS.scala#L202
>>
>> -Xiangrui
>>
>> On Tue, Jan 6, 2015 at 12:57 PM, Fernando O. <fo...@gmail.com> wrote:
>>
>>> Hi,
>>> I was doing a tests with ALS and I noticed that if I persist the
>>> inner RDDs from a MatrixFactorizationModel the RDD is not replicated, it
>>> seems like the storagelevel is hardcoded to MEMORY_AND_DISK, do you think
>>> it makes sense to make that configurable?
>>> [image: Inline image 1]
>>>
>>
>>
>
Re: [MLLib] storageLevel in ALS
Posted by Sean Owen <so...@cloudera.com>.
Ah, Fernando means the usersOut / productsOut RDDs, not the intermediate
links RDDs.
Can you unpersist() them, and persist() again at the desired level? the
downside is that this might mean recomputing and repersisting the RDDs.
On Wed, Jan 7, 2015 at 5:11 AM, Xiangrui Meng <me...@gmail.com> wrote:
> Which Spark version are you using? We made this configurable in 1.1:
>
>
> https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/recommendation/ALS.scala#L202
>
> -Xiangrui
>
> On Tue, Jan 6, 2015 at 12:57 PM, Fernando O. <fo...@gmail.com> wrote:
>
>> Hi,
>> I was doing a tests with ALS and I noticed that if I persist the inner
>> RDDs from a MatrixFactorizationModel the RDD is not replicated, it seems
>> like the storagelevel is hardcoded to MEMORY_AND_DISK, do you think it
>> makes sense to make that configurable?
>> [image: Inline image 1]
>>
>
>
Re: [MLLib] storageLevel in ALS
Posted by Xiangrui Meng <me...@gmail.com>.
Which Spark version are you using? We made this configurable in 1.1:
https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/recommendation/ALS.scala#L202
-Xiangrui
On Tue, Jan 6, 2015 at 12:57 PM, Fernando O. <fo...@gmail.com> wrote:
> Hi,
> I was doing a tests with ALS and I noticed that if I persist the inner
> RDDs from a MatrixFactorizationModel the RDD is not replicated, it seems
> like the storagelevel is hardcoded to MEMORY_AND_DISK, do you think it
> makes sense to make that configurable?
> [image: Inline image 1]
>