You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mahout.apache.org by Otis Gospodnetic <ot...@yahoo.com> on 2009/11/12 20:40:26 UTC
Taste and -XX:+NewRatio
Hello,
Something on http://lucene.apache.org/mahout/taste.html#performance caught my attention:
-XX:+NewRatio=9: Increase heap allocated to 'old' objects, which is most of them in this framework
So I looked up this java parameter on http://java.sun.com/javase/technologies/hotspot/vmoptions.jsp :
-XX:NewRatio=2
Ratio of new/old generation sizes. [Sparc -client: 8; x86 -server: 8; x86 -client: 12.]-client: 4 (1.3) 8 (1.3.1+), x86: 12]
If I'm reading this correctly, the larger the number, the bigger portion of the heap is used for "old" objects.
Also, it looks like different platforms have different defaults, so maybe advising a single "9" value is not the right thing to do?
For example, it looks like on x86 the default is 12, so using NewRatio=9 would actually decrease the space for old objects.
I'd love to hear from anyone who actually knows what this means, how it should be used, and if anyone has actually used different values with a large Taste instance.
Thanks,
Otis
--
Sematext is hiring -- http://sematext.com/about/jobs.html?mls
Lucene, Solr, Nutch, Katta, Hadoop, HBase, UIMA, NLP, NER, IR
Re: Taste and -XX:+NewRatio
Posted by Ted Dunning <te...@gmail.com>.
If you are using the concurrent collector on an active system, you should be
getting parallel collections pretty often. Each of these will increment the
generation for the objects.
Thus, long-lived means t > tenuring-age x new-space-collection-interval. My
guess is that this typically reduces to t > a few minutes.
The scary part, though, is whether refreshing the model will invoke a stop
and collect operation on the entire heap. Load testing is very important
here because you need to know that if you starve the parallel collector of
CPU time that it will still succeed without long pauses.
On Thu, Nov 12, 2009 at 12:55 PM, Sean Owen <sr...@gmail.com> wrote:
> It should, yes. Almost all the memory is long-lived (well, relative
> not-short-lived) even during the transition.
>
> On Thu, Nov 12, 2009 at 8:48 PM, Otis Gospodnetic
> <ot...@yahoo.com> wrote:
> > Does this hold true even if you refresh the whole model fairly
> frequently, say every 5, 10, 15, or 60 minutes?
>
--
Ted Dunning, CTO
DeepDyve
Re: Taste and -XX:+NewRatio
Posted by Sean Owen <sr...@gmail.com>.
It should, yes. Almost all the memory is long-lived (well, relative
not-short-lived) even during the transition.
On Thu, Nov 12, 2009 at 8:48 PM, Otis Gospodnetic
<ot...@yahoo.com> wrote:
> Does this hold true even if you refresh the whole model fairly frequently, say every 5, 10, 15, or 60 minutes?
Re: Taste and -XX:+NewRatio
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Hello,
----- Original Message ----
> From: Sean Owen <sr...@gmail.com>
>
> Yes, the advice is intended to be "make it larger than the default".
> As I recall the default was a lot lower, like 4. This was on a Mac
> though. I am happy to make the default advice more like 12. The point
> is that almost all the memory is long-lived data structures so almost
You are referring to objects mostly references from DataModel? Things like preferences and preference arrays, sets with user and item iDs and such? Or some other ones that I'm forgetting?
> all the heap should be allocated to this. It ought to be tuned since
> the ratio of long-lived to short-lived objects is unusually high. YMMV
> but I'm sure the right value is at least 9.
Does this hold true even if you refresh the whole model fairly frequently, say every 5, 10, 15, or 60 minutes?
Thanks,
Otis
> On Thu, Nov 12, 2009 at 7:40 PM, Otis Gospodnetic
> wrote:
> > Hello,
> >
> > Something on http://lucene.apache.org/mahout/taste.html#performance caught my
> attention:
> >
> > -XX:+NewRatio=9: Increase heap allocated to 'old' objects, which is most of
> them in this framework
> >
> > So I looked up this java parameter on
> http://java.sun.com/javase/technologies/hotspot/vmoptions.jsp :
> >
> >
> > -XX:NewRatio=2
> > Ratio of new/old generation sizes. [Sparc -client: 8; x86 -server: 8; x86
> -client: 12.]-client: 4 (1.3) 8 (1.3.1+), x86: 12]
> >
> > If I'm reading this correctly, the larger the number, the bigger portion of
> the heap is used for "old" objects.
> > Also, it looks like different platforms have different defaults, so maybe
> advising a single "9" value is not the right thing to do?
> > For example, it looks like on x86 the default is 12, so using NewRatio=9 would
> actually decrease the space for old objects.
> >
> > I'd love to hear from anyone who actually knows what this means, how it should
> be used, and if anyone has actually used different values with a large Taste
> instance.
> >
> > Thanks,
> > Otis
> > --
> > Sematext is hiring -- http://sematext.com/about/jobs.html?mls
> > Lucene, Solr, Nutch, Katta, Hadoop, HBase, UIMA, NLP, NER, IR
> >
> >
Re: Taste and -XX:+NewRatio
Posted by Sean Owen <sr...@gmail.com>.
Yes, the advice is intended to be "make it larger than the default".
As I recall the default was a lot lower, like 4. This was on a Mac
though. I am happy to make the default advice more like 12. The point
is that almost all the memory is long-lived data structures so almost
all the heap should be allocated to this. It ought to be tuned since
the ratio of long-lived to short-lived objects is unusually high. YMMV
but I'm sure the right value is at least 9.
On Thu, Nov 12, 2009 at 7:40 PM, Otis Gospodnetic
<ot...@yahoo.com> wrote:
> Hello,
>
> Something on http://lucene.apache.org/mahout/taste.html#performance caught my attention:
>
> -XX:+NewRatio=9: Increase heap allocated to 'old' objects, which is most of them in this framework
>
> So I looked up this java parameter on http://java.sun.com/javase/technologies/hotspot/vmoptions.jsp :
>
>
> -XX:NewRatio=2
> Ratio of new/old generation sizes. [Sparc -client: 8; x86 -server: 8; x86 -client: 12.]-client: 4 (1.3) 8 (1.3.1+), x86: 12]
>
> If I'm reading this correctly, the larger the number, the bigger portion of the heap is used for "old" objects.
> Also, it looks like different platforms have different defaults, so maybe advising a single "9" value is not the right thing to do?
> For example, it looks like on x86 the default is 12, so using NewRatio=9 would actually decrease the space for old objects.
>
> I'd love to hear from anyone who actually knows what this means, how it should be used, and if anyone has actually used different values with a large Taste instance.
>
> Thanks,
> Otis
> --
> Sematext is hiring -- http://sematext.com/about/jobs.html?mls
> Lucene, Solr, Nutch, Katta, Hadoop, HBase, UIMA, NLP, NER, IR
>
>
Re: Taste and -XX:+NewRatio
Posted by Jason Rennie <jr...@gmail.com>.
Note, especially, that there is no override for amd64, which is
the architecture you are likely using if you have 4+ gigs of ram, so you're
stuck with the idiotic value of 2 unless you set the option explicitly.
I.e. if you set a 1 gig heap, only 683m of that can be used for "old"
allocation. And, for doing this kind of work, just about all your data is
"old", so your max heap needs to be set about 50% higher than what you
actually need (unless you set NewRatio to a sane value). Any value between
8 and 12 should be fine for mahout as these are in the right ballpark and
aren't very different from a practical perspective (8 corresponds to 11%
new; 12 corresponds to 7.7% new).
More grumblings on java gc/memory:
http://javaquirks.blogspot.com/2008/08/garbage-collection-or-outofmemoryerror.html
http://javaquirks.blogspot.com/2009/04/usecompressedoops.html
Cheers,
Jason
--
Jason Rennie
Research Scientist, ITA Software
617-714-2645
http://www.itasoftware.com/