You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Casper Wandahl Schmidt <ka...@gmail.com> on 2011/12/01 09:39:42 UTC
[OT]RE: Maximum memory that can be assigned to Tomcat on windows platform
See below. I hope MS Outlook does some decent indend so my response is clear -.-
-----Original Message-----
From: Christopher Schultz [mailto:chris@christopherschultz.net]
Sent: 30. november 2011 18:51
To: Tomcat Users List
Subject: Re: Maximum memory that can be assigned to Tomcat on windows platform
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Casper,
On 11/30/11 3:37 AM, Casper Wandahl Schmidt wrote:
> Another question to ask is, why do you have 8GB memory when running
> 32bit? That is just stupid since 32bit cannot address more than 4GB of
> memory no matter what you do. Any sysadmin should know that right?
That's per process. All reasonably recent 32-bit OSs can address way more than 4GiB internally.
For example:
http://msdn.microsoft.com/en-us/library/windows/desktop/aa366778%28v=vs.85%29.aspx#memory_limits
This is generally done through PAE
(http://en.wikipedia.org/wiki/Physical_Address_Extension) which allows 32-bit OSs to access more than 4GiB at the kernel level, though each process is still limited to 4GiB.
Aha so I learned something new today :) I'm still puzzled as to how a 32 bit CPU can compute and fetch a memory cell with address above 4GB since it cannot hold this large value. Anyway that is just too much low-level computer science for me, all I ever had was a seven week course on architecture and networking (a single week out of the seven) :)
-Casper
Running a machine with more than 4GiB in 32-bit mode isn't stupid at all IMO. If you have relatively small processes, there's no need for the overhead of 64-bit even if you have 16GiB or more.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk7WbQYACgkQ9CaO5/Lv0PBsWwCgnifhHtqrLUBi7K4PeDjp4hnC
JMkAn0gilsNy2hv3zu3nzUkrmrzxoYWF
=AZpI
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: [OT]RE: Maximum memory that can be assigned to Tomcat on windows
platform
Posted by "Mark H. Wood" <mw...@IUPUI.Edu>.
The OS has little to do with the calculation. The CPU hardware is
doing it. The processor's address logic uses registers which are
wider than 32 bits. Just as you can add a 1-digit number to a 3-digit
number and get a 3-digit result, the widget that maps a process'
virtual address space to the hardware's physical address space can add
the content of a 32-bit register to the content of a 36-bit register
and get a 36-bit result.
(I'm ignoring the possibility of overflow, like adding 1 to 999 in
a 3-digit field. With good management they can be avoided.)
Only a tiny bit of the OS kernel, and nothing in any process, needs to
know about physical memory. The hardware is set up by that bit and
makes processes, and the rest of the kernel, think they each live in a
block of memory that starts at 0 and ends at, say, 3GB. In physical
memory they live side-by-side (to oversimplify a bit).
For how it does that, track down a little story called The Paging Game.
--
Mark H. Wood, Lead System Programmer mwood@IUPUI.Edu
Asking whether markets are efficient is like asking whether people are smart.
RE: [OT]RE: Maximum memory that can be assigned to Tomcat on windows platform
Posted by Casper Wandahl Schmidt <ka...@gmail.com>.
-----Original Message-----
From: Francis GALIEGUE [mailto:fge@one2team.com]
Sent: 1. december 2011 12:33
To: Tomcat Users List
Subject: Re: [OT]RE: Maximum memory that can be assigned to Tomcat on windows platform
On Thu, Dec 1, 2011 at 12:29, Casper Wandahl Schmidt <ka...@gmail.com> wrote:
[...]
>
> That didn't quite help me understand, because how can the OS map from ie.
> 0-4GB to 4-8GB (the window is moved) when it can only use a 32bit
> register to tell the machine where to look in the psysical memory,
> that is where my knowledge ends :) So I read about PAE and found out
> that it uses 2 registers
> (36 bits due to some bits being used as flags) and that makes good
> sense, but how can the cpu calculate an address without overflow and
> send a command to the bus containing a 36bit address (or whatever
> fetches the bits from RAM)? That is where I'm puzzled but I guess it
> is because I'm not at all into ISA-level and below :)
>
It is the role of the MMU to do that. At any one time, it can map a "virtual", 32-bit wide, address to a "real", 36-bit wide address. It uses TLBs (Translation Lookaside Buffers) for that, and it is the OS'
role to have the correct TLB in place at any time.
Nice to know :) That explained it all :)
-Casper
--
Francis Galiegue
ONE2TEAM
Ingénieur système
Mob : +33 (0) 683 877 875
Tel : +33 (0) 178 945 552
fge@one2team.com
40 avenue Raymond Poincaré
75116 Paris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: [OT]RE: Maximum memory that can be assigned to Tomcat on windows platform
Posted by Francis GALIEGUE <fg...@one2team.com>.
On Thu, Dec 1, 2011 at 12:29, Casper Wandahl Schmidt
<ka...@gmail.com> wrote:
[...]
>
> That didn't quite help me understand, because how can the OS map from ie.
> 0-4GB to 4-8GB (the window is moved) when it can only use a 32bit register
> to tell the machine where to look in the psysical memory, that is where my
> knowledge ends :) So I read about PAE and found out that it uses 2 registers
> (36 bits due to some bits being used as flags) and that makes good sense,
> but how can the cpu calculate an address without overflow and send a command
> to the bus containing a 36bit address (or whatever fetches the bits from
> RAM)? That is where I'm puzzled but I guess it is because I'm not at all
> into ISA-level and below :)
>
It is the role of the MMU to do that. At any one time, it can map a
"virtual", 32-bit wide, address to a "real", 36-bit wide address. It
uses TLBs (Translation Lookaside Buffers) for that, and it is the OS'
role to have the correct TLB in place at any time.
--
Francis Galiegue
ONE2TEAM
Ingénieur système
Mob : +33 (0) 683 877 875
Tel : +33 (0) 178 945 552
fge@one2team.com
40 avenue Raymond Poincaré
75116 Paris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: [OT]RE: Maximum memory that can be assigned to Tomcat on windows
platform
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Mikolaj,
On 12/1/11 6:38 AM, Mikolaj Rydzewski wrote:
> On Thu, 1 Dec 2011 12:29:14 +0100, Casper Wandahl Schmidt wrote:
>
>> That didn't quite help me understand, because how can the OS map
>> from ie. 0-4GB to 4-8GB (the window is moved) when it can only
>> use a 32bit register to tell the machine where to look in the
>> psysical memory, that is where my knowledge ends :) So I read
>> about PAE and found out that it uses 2 registers (36 bits due to
>> some bits being used as flags) and that makes good sense, but how
>> can the cpu calculate an address without overflow and send a
>> command to the bus containing a 36bit address (or whatever
>> fetches the bits from RAM)? That is where I'm puzzled but I guess
>> it is because I'm not at all into ISA-level and below :)
>
> Well, it's rather out of the scope of this list.
>
> On the other hand, increasing java heap size is not always the
> best option. It heavily depends on memory usage pattern in your
> application. In general: the bigger heap, the longer GC will run.
That's a rather sweeping generalization. The heap size doesn't matter
directly.. it's the number of objects being managed within that heap
that matters. Of course, with a larger heap, you can fit more objects
into it before a major collection is required. Generational heap
strategies are fairly efficient, and performance depends upon the
number of LIVE objects, not just the total number of objects. Oddly
enough, most "garbage collection" is really collecting non-garbage and
ignoring the actual garbage.
It's a bit like moving to a different house when yours gets too
cluttered: you just take the things you want to keep and leave
everything else behind.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk7YK88ACgkQ9CaO5/Lv0PDzdACgmYJEuWHFNkFyEVWRcucJo4Yu
6uwAoK2JWcjX0SRY6PPIWwd1m7Fhx+f8
=XY04
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: [OT]RE: Maximum memory that can be assigned to Tomcat on windows
platform
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Mark,
On 12/1/11 9:50 AM, Mark H. Wood wrote:
> On Thu, Dec 01, 2011 at 12:38:01PM +0100, Mikolaj Rydzewski wrote:
>> On the other hand, increasing java heap size is not always the
>> best option. It heavily depends on memory usage pattern in your
>> application. In general: the bigger heap, the longer GC will
>> run.
>
> I was thinking that someone should bring this up. When a program
> uses unexpectedly huge amounts of memory in practice, the *first*
> thing to consider is:
>
> 1. does it actually need that much?
+1 !!
> 2. ...or is it leaking dynamically created objects? 3. ...or has
> cheap allocation and garbage collection lured me into doing
> something suboptimal, like sucking down an entire database table
> into an array or list and then walking it sequentially, when I
> could have used an iterator and let the DBMS code work out
> near-optimal buffering?
>
> IOW "is my problem fundamentally this big, or is something else
> going on?"
The 2 times our production servers have suffered OOMEs, it's been
because we were running with fairly small, (intentionally) restricted
heaps (64MiB at first, then 192MiB) and our traffic simply increased
beyond our heap size: we had a legitimate reason to increase the heap
size (and plenty of physical RAM available to do it).
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk7YLEwACgkQ9CaO5/Lv0PCGTQCfSwBVBLSKIW2OMjYZWVobxrKY
JzkAoJQmi4JK2CHqo23DCuMRGE5Fzq/0
=Qte1
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: [OT]RE: Maximum memory that can be assigned to Tomcat on windows
platform
Posted by "Mark H. Wood" <mw...@IUPUI.Edu>.
On Thu, Dec 01, 2011 at 12:38:01PM +0100, Mikolaj Rydzewski wrote:
> On the other hand, increasing java heap size is not always the best
> option. It heavily depends on memory usage pattern in your application.
> In general: the bigger heap, the longer GC will run.
I was thinking that someone should bring this up. When a program uses
unexpectedly huge amounts of memory in practice, the *first* thing to
consider is:
1. does it actually need that much?
2. ...or is it leaking dynamically created objects?
3. ...or has cheap allocation and garbage collection lured me into
doing something suboptimal, like sucking down an entire database
table into an array or list and then walking it sequentially, when
I could have used an iterator and let the DBMS code work out
near-optimal buffering?
IOW "is my problem fundamentally this big, or is something else going on?"
--
Mark H. Wood, Lead System Programmer mwood@IUPUI.Edu
Asking whether markets are efficient is like asking whether people are smart.
RE: [OT]RE: Maximum memory that can be assigned to Tomcat on
windows platform
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: André Warnier [mailto:aw@ice-sa.com]
> Subject: Re: [OT]RE: Maximum memory that can be assigned to Tomcat on windows platform
> > In general: the bigger heap, the longer GC will run.
Not strictly true, and hasn't been true for many years. GC time is proportional to the number of live (reachable) objects, not the size of the heap. If the app is making heavy use of weak references, this may allow more live objects to persist in a larger heap until GC gets fed up with the mess and throws them all away.
> Why do I feel that a comment from Chuck is going to follow
> that one later on ?
Just had to wake up first.
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
Re: [OT]RE: Maximum memory that can be assigned to Tomcat on windows
platform
Posted by André Warnier <aw...@ice-sa.com>.
Mikolaj Rydzewski wrote:
> On Thu, 1 Dec 2011 12:29:14 +0100, Casper Wandahl Schmidt wrote:
>
>> That didn't quite help me understand, because how can the OS map from ie.
>> 0-4GB to 4-8GB (the window is moved) when it can only use a 32bit
>> register
>> to tell the machine where to look in the psysical memory, that is
>> where my
>> knowledge ends :) So I read about PAE and found out that it uses 2
>> registers
>> (36 bits due to some bits being used as flags) and that makes good sense,
>> but how can the cpu calculate an address without overflow and send a
>> command
>> to the bus containing a 36bit address (or whatever fetches the bits from
>> RAM)? That is where I'm puzzled but I guess it is because I'm not at all
>> into ISA-level and below :)
>
> Well, it's rather out of the scope of this list.
>
> On the other hand, increasing java heap size is not always the best
> option. It heavily depends on memory usage pattern in your application.
> In general: the bigger heap, the longer GC will run.
>
Why do I feel that a comment from Chuck is going to follow that one later on ?
;-)
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: [OT]RE: Maximum memory that can be assigned to Tomcat on
windows platform
Posted by Mikolaj Rydzewski <mi...@ceti.pl>.
On Thu, 1 Dec 2011 12:29:14 +0100, Casper Wandahl Schmidt wrote:
> That didn't quite help me understand, because how can the OS map from
> ie.
> 0-4GB to 4-8GB (the window is moved) when it can only use a 32bit
> register
> to tell the machine where to look in the psysical memory, that is
> where my
> knowledge ends :) So I read about PAE and found out that it uses 2
> registers
> (36 bits due to some bits being used as flags) and that makes good
> sense,
> but how can the cpu calculate an address without overflow and send a
> command
> to the bus containing a 36bit address (or whatever fetches the bits
> from
> RAM)? That is where I'm puzzled but I guess it is because I'm not at
> all
> into ISA-level and below :)
Well, it's rather out of the scope of this list.
On the other hand, increasing java heap size is not always the best
option. It heavily depends on memory usage pattern in your application.
In general: the bigger heap, the longer GC will run.
--
Mikolaj Rydzewski <mi...@ceti.pl>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: [OT]RE: Maximum memory that can be assigned to Tomcat on windows platform
Posted by Casper Wandahl Schmidt <ka...@gmail.com>.
-----Original Message-----
From: Ronald Klop (Mailing List) [mailto:ronald-mailinglist@base.nl]
Sent: 1. december 2011 12:06
To: Tomcat Users List
Subject: Re: [OT]RE: Maximum memory that can be assigned to Tomcat on
windows platform
Op donderdag, 1 december 2011 09:39 schreef Casper Wandahl Schmidt
<ka...@gmail.com>:
>
>
> See below. I hope MS Outlook does some decent indend so my response
> is clear -.-
>
> -----Original Message-----
> From: Christopher Schultz [mailto:chris@christopherschultz.net]
> Sent: 30. november 2011 18:51
> To: Tomcat Users List
> Subject: Re: Maximum memory that can be assigned to Tomcat on windows
> platform
>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Casper,
>
> On 11/30/11 3:37 AM, Casper Wandahl Schmidt wrote:
> > Another question to ask is, why do you have 8GB memory when running
> > 32bit? That is just stupid since 32bit cannot address more than 4GB
> of > memory no matter what you do. Any sysadmin should know that right?
>
> That's per process. All reasonably recent 32-bit OSs can address way more
than 4GiB internally.
>
> For example:
>
> http://msdn.microsoft.com/en-us/library/windows/desktop/aa366778%28v=v
> s.85%29.aspx#memory_limits
>
> This is generally done through PAE
> (http://en.wikipedia.org/wiki/Physical_Address_Extension) which allows
32-bit OSs to access more than 4GiB at the kernel level, though each process
is still limited to 4GiB.
>
> Aha so I learned something new today :) I'm still puzzled as to how a
> 32 bit CPU can compute and fetch a memory cell with address above 4GB
> since it cannot hold this large value. Anyway that is just too much
> low-level computer science for me, all I ever had was a seven week
> course on architecture and networking (a single week out of the seven)
> :)
>
> -Casper
>
> Running a machine with more than 4GiB in 32-bit mode isn't stupid at all
IMO. If you have relatively small processes, there's no need for the
overhead of 64-bit even if you have 16GiB or more.
>
> - -chris
>
>
>
>
>
I have an analogy for you.
If you look out of your window you only see a small part of the world. If
you move your window you wil see another part of the world.
This is what the OS does with PAE. It moves the window on your RAM
frequently. That is why a 32 bits application only sees max. 4GB. That is
the size of its window.
Ronald.
That didn't quite help me understand, because how can the OS map from ie.
0-4GB to 4-8GB (the window is moved) when it can only use a 32bit register
to tell the machine where to look in the psysical memory, that is where my
knowledge ends :) So I read about PAE and found out that it uses 2 registers
(36 bits due to some bits being used as flags) and that makes good sense,
but how can the cpu calculate an address without overflow and send a command
to the bus containing a 36bit address (or whatever fetches the bits from
RAM)? That is where I'm puzzled but I guess it is because I'm not at all
into ISA-level and below :)
-Casper
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: [OT]RE: Maximum memory that can be assigned to Tomcat on
windows platform
Posted by "Ronald Klop (Mailing List)" <ro...@base.nl>.
Op donderdag, 1 december 2011 09:39 schreef Casper Wandahl Schmidt <ka...@gmail.com>:
>
>
> See below. I hope MS Outlook does some decent indend so my response is clear -.-
>
> -----Original Message-----
> From: Christopher Schultz [mailto:chris@christopherschultz.net]
> Sent: 30. november 2011 18:51
> To: Tomcat Users List
> Subject: Re: Maximum memory that can be assigned to Tomcat on windows platform
>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Casper,
>
> On 11/30/11 3:37 AM, Casper Wandahl Schmidt wrote:
> > Another question to ask is, why do you have 8GB memory when running
> > 32bit? That is just stupid since 32bit cannot address more than 4GB of
> > memory no matter what you do. Any sysadmin should know that right?
>
> That's per process. All reasonably recent 32-bit OSs can address way more than 4GiB internally.
>
> For example:
> http://msdn.microsoft.com/en-us/library/windows/desktop/aa366778%28v=vs.85%29.aspx#memory_limits
>
> This is generally done through PAE
> (http://en.wikipedia.org/wiki/Physical_Address_Extension) which allows 32-bit OSs to access more than 4GiB at the kernel level, though each process is still limited to 4GiB.
>
> Aha so I learned something new today :) I'm still puzzled as to how a 32 bit CPU can compute and fetch a memory cell with address above 4GB since it cannot hold this large value. Anyway that is just too much low-level computer science for me, all I ever had was a seven week course on architecture and networking (a single week out of the seven) :)
>
> -Casper
>
> Running a machine with more than 4GiB in 32-bit mode isn't stupid at all IMO. If you have relatively small processes, there's no need for the overhead of 64-bit even if you have 16GiB or more.
>
> - -chris
>
>
>
>
>
I have an analogy for you.
If you look out of your window you only see a small part of the world. If you move your window you wil see another part of the world.
This is what the OS does with PAE. It moves the window on your RAM frequently. That is why a 32 bits application only sees max. 4GB. That is the size of its window.
Ronald.
Re: [OT]RE: Maximum memory that can be assigned to Tomcat on windows
platform
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Casper,
On 12/1/11 3:39 AM, Casper Wandahl Schmidt wrote:
> Aha so I learned something new today :) I'm still puzzled as to
> how a 32 bit CPU can compute and fetch a memory cell with address
> above 4GB since it cannot hold this large value.
OS != CPU
Also, OS != process
While the chips and OSs are officially 32-bit, both are able to handle
integers that don't fit into 32-bit registers in various ways. Usually,
CPUs have registers that are larger than their architecture would
suggest, and uses them even to perform computations on 32-bit data.
The real issue here is that in a 32-bit environment, word-sized
pointers are 32-bits and therefore an individual process gets a 4GiB
maximum process space, which can be mapped-into a much larger space by
the kernel, and even by the underlying hardware if it's in on the deal.
> Anyway that is just too much low-level computer science for me, all
> I ever had was a seven week course on architecture and networking
> (a single week out of the seven) :)
It never hurts to learn more. Unless your brain is full. Then it
*really* hurts.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk7YKp0ACgkQ9CaO5/Lv0PDpDgCgwNXVZ1k43CrOFDjcDryl3JTw
dSkAoK5XWk47MjE+fbsNnOS3CbGBdjxb
=nuE/
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org