You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Steve Loughran <st...@apache.org> on 2009/08/20 12:40:27 UTC
NN memory consumption on 0.20/0.21 with compressed pointers/
does anyone have any up to date data on the memory consumption per
block/file on the NN on a 64-bit JVM with compressed pointers?
The best documentation on consumption is
http://issues.apache.org/jira/browse/HADOOP-1687 -I'm just wondering if
anyone has looked at the memory footprint on the latest Hadoop releases,
on those latest JVMs? -and which JVM the numbers from HADOOP-1687 came from?
Those compressed pointers (which BEA JRockit had for a while) save RAM
when the pointer references are within a couple of GB of the other refs,
and which are discussed in some papers
http://rappist.elis.ugent.be/~leeckhou/papers/cgo06.pdf
http://www.elis.ugent.be/~kvenster/papers/VenstermansKris_ORA.pdf
sun's commentary is up here
http://wikis.sun.com/display/HotSpotInternals/CompressedOops
I'm just not sure what it means for the NameNode, and as there is no
sizeof() operator in Java, something that will take a bit of effort to
work out. From what I read of the Sun wiki, when you go compressed,
while your heap is <3-4GB, there is no decompress operation; once you go
above that there is a shift and an add, which is probably faster than
fetching another 32 bits from $L2 or main RAM. The result could be
-could be- that your NN takes up much less space on 64 bit JVMs than it
did before, but is no slower.
Has anyone worked out the numbers yet?
-steve
Re: NN memory consumption on 0.20/0.21 with compressed pointers/
Posted by Aaron Kimball <aa...@cloudera.com>.
Compressed OOPs are available now in 1.6.0u14:
https://jdk6.dev.java.net/6uNea.html
- Aaron
On Thu, Aug 20, 2009 at 10:51 AM, Raghu Angadi <ra...@yahoo-inc.com>wrote:
>
> Suresh had made an spreadsheet for memory consumption.. will check.
>
> A large portion of NN memory is taken by references. I would expect memory
> savings to be very substantial (same as going from 64bit to 32bit), could be
> on the order of 40%.
>
> The last I heard from Sun was that compressed pointers will be in very near
> future JVM (certainly JDK 1.6_x). It can use compressed pointers upto 32GB
> of heap.
>
> I would expect runtime over head on NN would be minimal in practice.
>
> Raghu.
>
>
> Steve Loughran wrote:
>
>>
>> does anyone have any up to date data on the memory consumption per
>> block/file on the NN on a 64-bit JVM with compressed pointers?
>>
>> The best documentation on consumption is
>> http://issues.apache.org/jira/browse/HADOOP-1687 -I'm just wondering if
>> anyone has looked at the memory footprint on the latest Hadoop releases, on
>> those latest JVMs? -and which JVM the numbers from HADOOP-1687 came from?
>>
>> Those compressed pointers (which BEA JRockit had for a while) save RAM
>> when the pointer references are within a couple of GB of the other refs, and
>> which are discussed in some papers
>> http://rappist.elis.ugent.be/~leeckhou/papers/cgo06.pdf<http://rappist.elis.ugent.be/%7Eleeckhou/papers/cgo06.pdf>
>> http://www.elis.ugent.be/~kvenster/papers/VenstermansKris_ORA.pdf<http://www.elis.ugent.be/%7Ekvenster/papers/VenstermansKris_ORA.pdf>
>>
>> sun's commentary is up here
>> http://wikis.sun.com/display/HotSpotInternals/CompressedOops
>>
>> I'm just not sure what it means for the NameNode, and as there is no
>> sizeof() operator in Java, something that will take a bit of effort to work
>> out. From what I read of the Sun wiki, when you go compressed, while your
>> heap is <3-4GB, there is no decompress operation; once you go above that
>> there is a shift and an add, which is probably faster than fetching another
>> 32 bits from $L2 or main RAM. The result could be -could be- that your NN
>> takes up much less space on 64 bit JVMs than it did before, but is no
>> slower.
>>
>> Has anyone worked out the numbers yet?
>>
>> -steve
>>
>
>
Re: NN memory consumption on 0.20/0.21 with compressed pointers/
Posted by Edward Capriolo <ed...@gmail.com>.
On Tue, Aug 25, 2009 at 11:59 AM, Ted Dunning<te...@gmail.com> wrote:
> On Tue, Aug 25, 2009 at 2:57 AM, Steve Loughran <st...@apache.org> wrote:
>
>> On petascale-level computers, the application codes' CPU instructions are
>>> about 10% floating point (that is, in scientific applications, there are
>>> less floating point instructions than in most floating point benchmarks).
>>> Of the remaining instructions, about 1/3 are memory-related and 2/3 are
>>> integer. Of the integer instructions, 40% are computing memory locations.
>>>
>>
>> cool. I wonder what percentage is vtable/function lookup in OO code versus
>> data retrieval? After all, every time you read or write an instance field in
>> a class instance, there is the this+offset maths before the actual
>> retrieval, though there is a fair amount of CPU support for such offset
>> operations
>
>
> Since the codes running on these machines tend to be matrix oriented
> Fortran, I would expect almost all of this is array index computation.
>
>
>>
>>
>>> So, on the biggest DOE computers, about 50% of the CPU time is spent on
>>> memory-related computations. I found this pretty mind-boggling when I
>>> learned this. It seems to me that the "central" part of the computer is
>>> becoming the bus, not the CPU.
>>>
>>
>> welcome to the new bottlenecks.
>>
>
> Comm ACM had a recent article in which the author measured (informally) the
> throughput available for Disk, SSD and main memory with sequential and
> random access patterns. Sequential disk was slightly faster than random
> access to main memory.
>
> --
> Ted Dunning, CTO
> DeepDyve
>
Warning!
If you decide to switch to 1.6.0u15 with 18.3 you are going to get a
task tracker crash. An odd silent error that appears in logs but does
not manifest itself in a failed status with hadoop. Logs will show:
Exception in thread "main" java.lang.NoClassDefFoundError:
Could_not_reserve_enough_space_for_object_heap
Caused by: java.lang.ClassNotFoundException:
Could_not_reserve_enough_space_for_object_heap
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
Could not find the main class:
Could_not_reserve_enough_space_for_object_heap. Program will exit.
Issue is : HADOOP-5564
Fix is here: http://www.koopman.me/2009/04/hadoop-0183-could-not-create-the-java-virtual-machine/
Re: NN memory consumption on 0.20/0.21 with compressed pointers/
Posted by Ted Dunning <te...@gmail.com>.
On Tue, Aug 25, 2009 at 2:57 AM, Steve Loughran <st...@apache.org> wrote:
> On petascale-level computers, the application codes' CPU instructions are
>> about 10% floating point (that is, in scientific applications, there are
>> less floating point instructions than in most floating point benchmarks).
>> Of the remaining instructions, about 1/3 are memory-related and 2/3 are
>> integer. Of the integer instructions, 40% are computing memory locations.
>>
>
> cool. I wonder what percentage is vtable/function lookup in OO code versus
> data retrieval? After all, every time you read or write an instance field in
> a class instance, there is the this+offset maths before the actual
> retrieval, though there is a fair amount of CPU support for such offset
> operations
Since the codes running on these machines tend to be matrix oriented
Fortran, I would expect almost all of this is array index computation.
>
>
>> So, on the biggest DOE computers, about 50% of the CPU time is spent on
>> memory-related computations. I found this pretty mind-boggling when I
>> learned this. It seems to me that the "central" part of the computer is
>> becoming the bus, not the CPU.
>>
>
> welcome to the new bottlenecks.
>
Comm ACM had a recent article in which the author measured (informally) the
throughput available for Disk, SSD and main memory with sequential and
random access patterns. Sequential disk was slightly faster than random
access to main memory.
--
Ted Dunning, CTO
DeepDyve
Re: NN memory consumption on 0.20/0.21 with compressed pointers/
Posted by Steve Loughran <st...@apache.org>.
Brian Bockelman wrote:
>
> On Aug 24, 2009, at 5:42 AM, Steve Loughran wrote:
>
>> Raghu Angadi wrote:
>>> Suresh had made an spreadsheet for memory consumption.. will check.
>>> A large portion of NN memory is taken by references. I would expect
>>> memory savings to be very substantial (same as going from 64bit to
>>> 32bit), could be on the order of 40%.
>>> The last I heard from Sun was that compressed pointers will be in
>>> very near future JVM (certainly JDK 1.6_x). It can use compressed
>>> pointers upto 32GB of heap.
>>
>> It's in JDK 1.6u14. Looking at the source and reading the specs
>> implies there is savings, but we need to experiment to see. I now know
>> how to do sizeof() in java, (in the instrumentation API), so these
>> experiments are possible
>>
>
> Hey Steve,
>
> I'm a bit dumb with Java (so this might be something you already know),
> but last week I discovered the "jhat" tool. You can dump the Java stack
> with JMX to a file, then use jhat to build a little webserver that
> allows you to explore your heap.
>
> One page it provides is a table histogram of the instance counts and #
> of bytes per class.
I've worked out how to do sizeof, got my first results during last
night's london HUG event,
all the data is going into https://issues.apache.org/jira/browse/HDFS-559
>
> This helped me a lot when I was trying to track memory leaks in libhdfs.
>
>>> I would expect runtime over head on NN would be minimal in practice.
>>
>>
>> I think there's a small extra deref cost, but its very minimal; one 8
>> bit logical shift left, possibly also an addition. Both of which run
>> at CPU-speeds, not main memory bus rates
>
> One interesting tidbit (from my memory of a presentation 2 months ago...
> I might have the numbers wrong, but the general message is the same):
>
> On petascale-level computers, the application codes' CPU instructions
> are about 10% floating point (that is, in scientific applications, there
> are less floating point instructions than in most floating point
> benchmarks). Of the remaining instructions, about 1/3 are
> memory-related and 2/3 are integer. Of the integer instructions, 40%
> are computing memory locations.
cool. I wonder what percentage is vtable/function lookup in OO code
versus data retrieval? After all, every time you read or write an
instance field in a class instance, there is the this+offset maths
before the actual retrieval, though there is a fair amount of CPU
support for such offset operations
>
> So, on the biggest DOE computers, about 50% of the CPU time is spent on
> memory-related computations. I found this pretty mind-boggling when I
> learned this. It seems to me that the "central" part of the computer is
> becoming the bus, not the CPU.
welcome to the new bottlenecks.
Re: NN memory consumption on 0.20/0.21 with compressed pointers/
Posted by Brian Bockelman <bb...@cse.unl.edu>.
On Aug 24, 2009, at 5:42 AM, Steve Loughran wrote:
> Raghu Angadi wrote:
>> Suresh had made an spreadsheet for memory consumption.. will check.
>> A large portion of NN memory is taken by references. I would expect
>> memory savings to be very substantial (same as going from 64bit to
>> 32bit), could be on the order of 40%.
>> The last I heard from Sun was that compressed pointers will be in
>> very near future JVM (certainly JDK 1.6_x). It can use compressed
>> pointers upto 32GB of heap.
>
> It's in JDK 1.6u14. Looking at the source and reading the specs
> implies there is savings, but we need to experiment to see. I now
> know how to do sizeof() in java, (in the instrumentation API), so
> these experiments are possible
>
Hey Steve,
I'm a bit dumb with Java (so this might be something you already
know), but last week I discovered the "jhat" tool. You can dump the
Java stack with JMX to a file, then use jhat to build a little
webserver that allows you to explore your heap.
One page it provides is a table histogram of the instance counts and #
of bytes per class.
This helped me a lot when I was trying to track memory leaks in libhdfs.
>> I would expect runtime over head on NN would be minimal in practice.
>
>
> I think there's a small extra deref cost, but its very minimal; one
> 8 bit logical shift left, possibly also an addition. Both of which
> run at CPU-speeds, not main memory bus rates
One interesting tidbit (from my memory of a presentation 2 months
ago... I might have the numbers wrong, but the general message is the
same):
On petascale-level computers, the application codes' CPU instructions
are about 10% floating point (that is, in scientific applications,
there are less floating point instructions than in most floating point
benchmarks). Of the remaining instructions, about 1/3 are memory-
related and 2/3 are integer. Of the integer instructions, 40% are
computing memory locations.
So, on the biggest DOE computers, about 50% of the CPU time is spent
on memory-related computations. I found this pretty mind-boggling
when I learned this. It seems to me that the "central" part of the
computer is becoming the bus, not the CPU.
Brian
Re: NN memory consumption on 0.20/0.21 with compressed pointers/
Posted by Steve Loughran <st...@apache.org>.
Raghu Angadi wrote:
>
> Suresh had made an spreadsheet for memory consumption.. will check.
>
> A large portion of NN memory is taken by references. I would expect
> memory savings to be very substantial (same as going from 64bit to
> 32bit), could be on the order of 40%.
>
> The last I heard from Sun was that compressed pointers will be in very
> near future JVM (certainly JDK 1.6_x). It can use compressed pointers
> upto 32GB of heap.
It's in JDK 1.6u14. Looking at the source and reading the specs implies
there is savings, but we need to experiment to see. I now know how to do
sizeof() in java, (in the instrumentation API), so these experiments are
possible
>
> I would expect runtime over head on NN would be minimal in practice.
I think there's a small extra deref cost, but its very minimal; one 8
bit logical shift left, possibly also an addition. Both of which run at
CPU-speeds, not main memory bus rates
Re: NN memory consumption on 0.20/0.21 with compressed pointers/
Posted by Raghu Angadi <ra...@yahoo-inc.com>.
Suresh had made an spreadsheet for memory consumption.. will check.
A large portion of NN memory is taken by references. I would expect
memory savings to be very substantial (same as going from 64bit to
32bit), could be on the order of 40%.
The last I heard from Sun was that compressed pointers will be in very
near future JVM (certainly JDK 1.6_x). It can use compressed pointers
upto 32GB of heap.
I would expect runtime over head on NN would be minimal in practice.
Raghu.
Steve Loughran wrote:
>
> does anyone have any up to date data on the memory consumption per
> block/file on the NN on a 64-bit JVM with compressed pointers?
>
> The best documentation on consumption is
> http://issues.apache.org/jira/browse/HADOOP-1687 -I'm just wondering if
> anyone has looked at the memory footprint on the latest Hadoop releases,
> on those latest JVMs? -and which JVM the numbers from HADOOP-1687 came
> from?
>
> Those compressed pointers (which BEA JRockit had for a while) save RAM
> when the pointer references are within a couple of GB of the other refs,
> and which are discussed in some papers
> http://rappist.elis.ugent.be/~leeckhou/papers/cgo06.pdf
> http://www.elis.ugent.be/~kvenster/papers/VenstermansKris_ORA.pdf
>
> sun's commentary is up here
> http://wikis.sun.com/display/HotSpotInternals/CompressedOops
>
> I'm just not sure what it means for the NameNode, and as there is no
> sizeof() operator in Java, something that will take a bit of effort to
> work out. From what I read of the Sun wiki, when you go compressed,
> while your heap is <3-4GB, there is no decompress operation; once you go
> above that there is a shift and an add, which is probably faster than
> fetching another 32 bits from $L2 or main RAM. The result could be
> -could be- that your NN takes up much less space on 64 bit JVMs than it
> did before, but is no slower.
>
> Has anyone worked out the numbers yet?
>
> -steve
Re: NN memory consumption on 0.20/0.21 with compressed pointers/
Posted by Steve Loughran <st...@apache.org>.
Scott Carey wrote:
> The implementation in JRE 6u14 uses a shift for all heap sizes, the
> optimization to remove that for heaps less than 4GB is not in the hotspot
> version there (but will be later).
OK. I've been using JRockit 64 bit for a while, and it did a check on
every pointer to see if it was real or relative, then an add, so I
suspect its computation was more complex. The sun approach seems better
> The size advantage is there either way.
>
> I have not tested an app myself that was not faster using
> -XX:+UseCompressedOops on a 64 bit JVM.
> The extra bit shifting is overshadowed by how much faster and less frequent
> GC is with a smaller dataset.
Excellent.
You get better cache efficiency too -less cache misses, and save on
memory bandwidth
Re: NN memory consumption on 0.20/0.21 with compressed pointers/
Posted by Scott Carey <sc...@richrelevance.com>.
On 8/20/09 3:40 AM, "Steve Loughran" <st...@apache.org> wrote:
>
>
> does anyone have any up to date data on the memory consumption per
> block/file on the NN on a 64-bit JVM with compressed pointers?
>
> The best documentation on consumption is
> http://issues.apache.org/jira/browse/HADOOP-1687 -I'm just wondering if
> anyone has looked at the memory footprint on the latest Hadoop releases,
> on those latest JVMs? -and which JVM the numbers from HADOOP-1687 came from?
>
> Those compressed pointers (which BEA JRockit had for a while) save RAM
> when the pointer references are within a couple of GB of the other refs,
> and which are discussed in some papers
> http://rappist.elis.ugent.be/~leeckhou/papers/cgo06.pdf
> http://www.elis.ugent.be/~kvenster/papers/VenstermansKris_ORA.pdf
>
> sun's commentary is up here
> http://wikis.sun.com/display/HotSpotInternals/CompressedOops
>
> I'm just not sure what it means for the NameNode, and as there is no
> sizeof() operator in Java, something that will take a bit of effort to
> work out. From what I read of the Sun wiki, when you go compressed,
> while your heap is <3-4GB, there is no decompress operation; once you go
> above that there is a shift and an add, which is probably faster than
> fetching another 32 bits from $L2 or main RAM. The result could be
> -could be- that your NN takes up much less space on 64 bit JVMs than it
> did before, but is no slower.
The implementation in JRE 6u14 uses a shift for all heap sizes, the
optimization to remove that for heaps less than 4GB is not in the hotspot
version there (but will be later).
The size advantage is there either way.
I have not tested an app myself that was not faster using
-XX:+UseCompressedOops on a 64 bit JVM.
The extra bit shifting is overshadowed by how much faster and less frequent
GC is with a smaller dataset.
>
> Has anyone worked out the numbers yet?
>
> -steve
>
Every Java reference is 4 bytes instead of 8, and for several types --
arrays in particular -- the object is also 4 bytes smaller. Given that the
NN data structures have plenty of references, a 30% reduction in memory used
would not be a surprise.
Collection classes in particular are near half the size.