You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by "Robinson, Eric" <er...@psmnv.com> on 2011/12/19 19:51:26 UTC
Do I Have Java Memory Fragmentation?
We have a Windows 2003 R2 x64 virtual machine running as a guest under
Oracle VirtualBox on 64-bit Linux. The Windows VM has 16GB of RAM and is
running approximately 80 separate instances of tomcat. (These instances
do a specific small function and do not require much power or cause much
load.) With all services running, the server still has 3-4GB of free
RAM.
When we only had about 40-50 tomcat instances, everything was fine. But
as we add more instances, we have a problem where tomcat services just
spontaneously die. If we check the server after about 12 hours of
uptime, we find that 5-6 services have stopped. A little later it might
be 10-12 instances, and so on. It gets worse over time. There are no OOM
errors in the logs, but attempting to restart the stopped services
produces Windows error 1067, and the java logs contain the message
"unable to allocate enough java heap space."
By default, my Java instances are configured with -Xms128M -Xmx384M.
Raising these numbers does not help, but if I lower them to, say 64MB,
then Java will start and the services will come up (but then I have
other problems with the application).
All of this makes me think we have a problem with Java memory
fragmentation because of having so many instances of Java, and the
problem gets worse as the hours go by.
Can someone recommend a way to diagnose and/or fix this problem?
--
Eric Robinson
Director of Information Technology
Physician Select Management, LLC
775-885-2211 x 111
Disclaimer - December 19, 2011
This email and any files transmitted with it are confidential and intended solely for users@tomcat.apache.org. If you are not the named addressee you should not disseminate, distribute, copy or alter this email. Any views or opinions presented in this email are solely those of the author and might not represent those of Physicians' Managed Care or Physician Select Management. Warning: Although Physicians' Managed Care or Physician Select Management has taken reasonable precautions to ensure no viruses are present in this email, the company cannot accept responsibility for any loss or damage arising from the use of this email or attachments.
This disclaimer was added by Policy Patrol: http://www.policypatrol.com/
RE: Do I Have Java Memory Fragmentation?
Posted by Jeff Sturm <je...@eprize.com>.
> -----Original Message-----
> From: Robinson, Eric [mailto:eric.robinson@psmnv.com]
> Sent: Monday, December 19, 2011 1:51 PM
>
> By default, my Java instances are configured with -Xms128M -Xmx384M.
80 JVM instances allocating 384M of heap each will need about 30GB total virtual memory for heap alone.
You said your server has 16GB of real memory. How large is the paging file? I doubt 16GB is enough based on your heap parameters.
-Jeff
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Do I Have Java Memory Fragmentation?
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
André,
On 12/19/11 5:19 PM, André Warnier wrote:
> On a 32-bit Linux system, this is what "top" has to say about the
> memory usage of one tomcat 5.5 instance, running one single small
> app, with -Xms128M -Xmx128M, and totally idle for the time being :
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>
> 2871 tomcat55 20 0 539m 137m 4560 S 0.0 9.1 129:40.19 jsvc
>
> If one multiplies this by 60, one gets : RES : +- 8 GB VIRT : +- 32
> GB
The virtual size gets all muddied by recent JVMs that use shared
memory between JVM processes to share the built-in types, etc.
> There may be some overhead due to jsvc, but that would be the same
> under Windows with tomcat55.exe.
Not much.
> Note also that on a 64-bit system like the OP, the numbers above
> may be somewhat larger.
Good thought: Eric, are you running 32-bit or 64-bit JVMs? You
probably ought to run under 32-bit JVMs since you have no need for
large heaps (if you intend to continue to run all several-dozen of
these webapps in separate JVMs) and all that extra pointer room can
really burn-through RAM quickly. If you must use a 64-bit JVM (why!?),
you might want to look at using "compressed OOPS" which will shrink
pointer sizes for Java objects. This may improve performance a bit too.
> Which make me muse also that since the OP obviously doesn't need a
> very large heap, he may benefit, maybe greatly, from running a
> 32-bit JVM for his Tomcat instances, if that is allowed under his
> version of Windows. He may gain even more by running his VM with a
> 32-bit Windows in the first place.
At least he's running his VM situation properly: with Linux as the
host and Microsoft Windows as the guest. I've seen so many going he
other way around and it baffles me.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk7vwIwACgkQ9CaO5/Lv0PD+2gCeIiWeAMkL4DeAgxKBIa0BUhKD
GWMAnjgvwCQ876uytY1e+jdBfUtC3YZ3
=InPi
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Do I Have Java Memory Fragmentation?
Posted by André Warnier <aw...@ice-sa.com>.
Pid wrote:
> On 19/12/2011 19:41, Leon Rosenberg wrote:
>> Do a simple math. If you give each VM up to 384M heap and if you add
>> some 100 MB for permspace and overhead, you have a consumption of
>> 500MB per VM.
>
> Agreed. Object heap != total process usage.
>
>> Take 1GB for OS needs, you have 15, and 15.000/500 makes
>> it save to run 30 tomcat instances max.
>> Since you experience the problems after some running time, I would
>> assume that you VMs are starting with 128 but slowly take up all the
>> space, and the later VMs just have nothing to get.
>
> Is the OP monitoring the JVM memory consumption (via a JMX connection)?
>
>> Advice: Calculate your needs exactly by playing with settings
>> (remember always set Ms=Mx) and watching gc logs. You will probably
>> need to reduce the number of instances or buy another machine (or put
>> more memory into this one).
>
> It would be useful to establish what is consuming memory once Tomcat
> instances have failed. Is it failed JVM instances that have hung and
> are still consuming memory, or is it something more exotic to do with
> the hypervisor?
>
> Fixing memory size will offer some gains, but you need to understand
> what each instance is actually doing, rather than using a standard.
>
>> Also, do you really need a separate instance for each webapp? You have
>> 60x catalina code in the heap and everything, which probably takes
>> half of your memory.
>
> I don't think it's safe to say "catalina code will take half your
> memory", given that we don't know anything about the OPs apps, but it
> may be a reasonable gain to collapse some apps into one Tomcat.
>
On a 32-bit Linux system, this is what "top" has to say about the memory usage of one
tomcat 5.5 instance, running one single small app, with -Xms128M -Xmx128M, and totally
idle for the time being :
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2871 tomcat55 20 0 539m 137m 4560 S 0.0 9.1 129:40.19 jsvc
If one multiplies this by 60, one gets :
RES : +- 8 GB
VIRT : +- 32 GB
There may be some overhead due to jsvc, but that would be the same under Windows with
tomcat55.exe.
Note also that on a 64-bit system like the OP, the numbers above may be somewhat larger.
Which make me muse also that since the OP obviously doesn't need a very large heap, he
may benefit, maybe greatly, from running a 32-bit JVM for his Tomcat instances, if that is
allowed under his version of Windows. He may gain even more by running his VM with a
32-bit Windows in the first place.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Do I Have Java Memory Fragmentation?
Posted by Pid <pi...@pidster.com>.
On 19/12/2011 19:41, Leon Rosenberg wrote:
> Do a simple math. If you give each VM up to 384M heap and if you add
> some 100 MB for permspace and overhead, you have a consumption of
> 500MB per VM.
Agreed. Object heap != total process usage.
> Take 1GB for OS needs, you have 15, and 15.000/500 makes
> it save to run 30 tomcat instances max.
> Since you experience the problems after some running time, I would
> assume that you VMs are starting with 128 but slowly take up all the
> space, and the later VMs just have nothing to get.
Is the OP monitoring the JVM memory consumption (via a JMX connection)?
> Advice: Calculate your needs exactly by playing with settings
> (remember always set Ms=Mx) and watching gc logs. You will probably
> need to reduce the number of instances or buy another machine (or put
> more memory into this one).
It would be useful to establish what is consuming memory once Tomcat
instances have failed. Is it failed JVM instances that have hung and
are still consuming memory, or is it something more exotic to do with
the hypervisor?
Fixing memory size will offer some gains, but you need to understand
what each instance is actually doing, rather than using a standard.
> Also, do you really need a separate instance for each webapp? You have
> 60x catalina code in the heap and everything, which probably takes
> half of your memory.
I don't think it's safe to say "catalina code will take half your
memory", given that we don't know anything about the OPs apps, but it
may be a reasonable gain to collapse some apps into one Tomcat.
p
> regards
> Leon
>
>
> On Mon, Dec 19, 2011 at 8:24 PM, Caldarale, Charles R
> <Ch...@unisys.com> wrote:
>>> From: Robinson, Eric [mailto:eric.robinson@psmnv.com]
>>> Subject: Do I Have Java Memory Fragmentation?
>>
>>> All of this makes me think we have a problem with Java memory
>>> fragmentation because of having so many instances of Java
>>
>> Fragmentation doesn't really exist anymore, thanks to paging and garbage collection. (Except in very unusual circumstances involving badly coded JNI methods that pin objects unnecessarily; are you doing that?)
>>
>>> Can someone recommend a way to diagnose and/or fix this problem?
>>
>> Look in the Windows logs (event viewer), not the Tomcat ones. Windows may think you're running out of memory + swap space. Raising the -Xmx value for each Tomcat instance will only make it worse, not better.
>>
>> - Chuck
>>
>>
>> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
--
[key:62590808]
Re: Do I Have Java Memory Fragmentation?
Posted by Leon Rosenberg <ro...@gmail.com>.
Do a simple math. If you give each VM up to 384M heap and if you add
some 100 MB for permspace and overhead, you have a consumption of
500MB per VM. Take 1GB for OS needs, you have 15, and 15.000/500 makes
it save to run 30 tomcat instances max.
Since you experience the problems after some running time, I would
assume that you VMs are starting with 128 but slowly take up all the
space, and the later VMs just have nothing to get.
Advice: Calculate your needs exactly by playing with settings
(remember always set Ms=Mx) and watching gc logs. You will probably
need to reduce the number of instances or buy another machine (or put
more memory into this one).
Also, do you really need a separate instance for each webapp? You have
60x catalina code in the heap and everything, which probably takes
half of your memory.
regards
Leon
On Mon, Dec 19, 2011 at 8:24 PM, Caldarale, Charles R
<Ch...@unisys.com> wrote:
>> From: Robinson, Eric [mailto:eric.robinson@psmnv.com]
>> Subject: Do I Have Java Memory Fragmentation?
>
>> All of this makes me think we have a problem with Java memory
>> fragmentation because of having so many instances of Java
>
> Fragmentation doesn't really exist anymore, thanks to paging and garbage collection. (Except in very unusual circumstances involving badly coded JNI methods that pin objects unnecessarily; are you doing that?)
>
>> Can someone recommend a way to diagnose and/or fix this problem?
>
> Look in the Windows logs (event viewer), not the Tomcat ones. Windows may think you're running out of memory + swap space. Raising the -Xmx value for each Tomcat instance will only make it worse, not better.
>
> - Chuck
>
>
> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Do I Have Java Memory Fragmentation?
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: Robinson, Eric [mailto:eric.robinson@psmnv.com]
> Subject: Do I Have Java Memory Fragmentation?
> All of this makes me think we have a problem with Java memory
> fragmentation because of having so many instances of Java
Fragmentation doesn't really exist anymore, thanks to paging and garbage collection. (Except in very unusual circumstances involving badly coded JNI methods that pin objects unnecessarily; are you doing that?)
> Can someone recommend a way to diagnose and/or fix this problem?
Look in the Windows logs (event viewer), not the Tomcat ones. Windows may think you're running out of memory + swap space. Raising the -Xmx value for each Tomcat instance will only make it worse, not better.
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org