You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mahout.apache.org by Tamas Jambor <ja...@gmail.com> on 2010/06/18 12:33:14 UTC
out for memory
hi,
i am trying to run an SVD recommender with the netflix dataset, first I
am trying to read the data to a GenericDataModel object, for some reason
I always run out of memory, even if I assign quite a lot of memory to
the task (e.g 10gb). is there a more efficient way to run this task?
thanks,
Tamas
Re: out for memory
Posted by Tamas Jambor <ja...@gmail.com>.
Thanks. In the end, I got away with 7.3GB, I guess this is as good as it
gets for now.
On 18/06/2010 14:52, Sean Owen wrote:
> GDM needs roughly 28 bytes per preference, all told. I'd expect it
> alone consumes "just" 2.8GB of heap on the Netflix data set. The rest
> may be consumed by SVD matrices and your application storage and other
> JVM stuff.
>
> 10GB seems very large. Are you setting the new generation size to be
> relatively small? otherwise the defaults will probably waste a couple
> gigabytes of heap.
>
> Do what works best for you though.
>
> On Fri, Jun 18, 2010 at 2:41 PM, Tamas Jambor<ja...@gmail.com> wrote:
>
>> I mean I used to run SVD with my implementation storing only three arrays
>> (user, item, rating), that I was able to fit in 3GB memory. I guess
>> GenericDataModel takes up quite a lot of memory, because the data is indexed
>> by users and by items, which is not necessary for SVD
>>
>> On 18/06/2010 14:21, Sean Owen wrote:
>>
>>> Memory requirements may be much higher for this algorithm as it builds
>>> large intermediate data structures to compute the SVD. Yes I think the
>>> simple data fits in 3GB or so. Sounds like you have solved your
>>> problem by supplying more memory.
>>>
>>>
>>>
>>
Re: out for memory
Posted by Sean Owen <sr...@gmail.com>.
GDM needs roughly 28 bytes per preference, all told. I'd expect it
alone consumes "just" 2.8GB of heap on the Netflix data set. The rest
may be consumed by SVD matrices and your application storage and other
JVM stuff.
10GB seems very large. Are you setting the new generation size to be
relatively small? otherwise the defaults will probably waste a couple
gigabytes of heap.
Do what works best for you though.
On Fri, Jun 18, 2010 at 2:41 PM, Tamas Jambor <ja...@gmail.com> wrote:
> I mean I used to run SVD with my implementation storing only three arrays
> (user, item, rating), that I was able to fit in 3GB memory. I guess
> GenericDataModel takes up quite a lot of memory, because the data is indexed
> by users and by items, which is not necessary for SVD
>
> On 18/06/2010 14:21, Sean Owen wrote:
>>
>> Memory requirements may be much higher for this algorithm as it builds
>> large intermediate data structures to compute the SVD. Yes I think the
>> simple data fits in 3GB or so. Sounds like you have solved your
>> problem by supplying more memory.
>>
>>
>
Re: out for memory
Posted by Tamas Jambor <ja...@gmail.com>.
I mean I used to run SVD with my implementation storing only three
arrays (user, item, rating), that I was able to fit in 3GB memory. I
guess GenericDataModel takes up quite a lot of memory, because the data
is indexed by users and by items, which is not necessary for SVD
On 18/06/2010 14:21, Sean Owen wrote:
> Memory requirements may be much higher for this algorithm as it builds
> large intermediate data structures to compute the SVD. Yes I think the
> simple data fits in 3GB or so. Sounds like you have solved your
> problem by supplying more memory.
>
>
Re: out for memory
Posted by Sean Owen <sr...@gmail.com>.
Memory requirements may be much higher for this algorithm as it builds
large intermediate data structures to compute the SVD. Yes I think the
simple data fits in 3GB or so. Sounds like you have solved your
problem by supplying more memory.
On Fri, Jun 18, 2010 at 2:10 PM, Tamas Jambor <ja...@gmail.com> wrote:
> it's the standard netflix dataset around 100m ratings. It seems it is
> working OK with around 10gb memory. if I remember correctly this could be
> run using around 3-4gb.
Re: out for memory
Posted by Tamas Jambor <ja...@gmail.com>.
it's the standard netflix dataset around 100m ratings. It seems it is
working OK with around 10gb memory. if I remember correctly this could
be run using around 3-4gb.
On 18/06/2010 13:49, Sean Owen wrote:
> How big is your input?
> I am not sure this necessarily scales to tens of millions of data points, no.
> A distributed implementation is being created which could be more appropriate.
>
> On Fri, Jun 18, 2010 at 11:33 AM, Tamas Jambor<ja...@gmail.com> wrote:
>
>> hi,
>>
>> i am trying to run an SVD recommender with the netflix dataset, first I am
>> trying to read the data to a GenericDataModel object, for some reason I
>> always run out of memory, even if I assign quite a lot of memory to the task
>> (e.g 10gb). is there a more efficient way to run this task?
>>
>> thanks,
>> Tamas
>>
>>
Re: out for memory
Posted by Sean Owen <sr...@gmail.com>.
How big is your input?
I am not sure this necessarily scales to tens of millions of data points, no.
A distributed implementation is being created which could be more appropriate.
On Fri, Jun 18, 2010 at 11:33 AM, Tamas Jambor <ja...@gmail.com> wrote:
> hi,
>
> i am trying to run an SVD recommender with the netflix dataset, first I am
> trying to read the data to a GenericDataModel object, for some reason I
> always run out of memory, even if I assign quite a lot of memory to the task
> (e.g 10gb). is there a more efficient way to run this task?
>
> thanks,
> Tamas
>