You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by "Dan D." <da...@dubomail.com> on 2008/02/05 00:46:01 UTC
Out of Memory Errors when upgrading from 5.0.27 to 6.0.14
Hello all,
I'm currently working on upgrading our systems from using Tomcat 5.0.27 to 6.0.14.
The upgrades are not being done in place (ie, we will have both systems out there, and will be doing migrations to the new version). I have been able to resolve most of the issues we have run into thus far, however, I am now somewhat stuck, and am hoping someone here may be able to help.
About the environment:
Shared hosting, with approximately 300-600 users on a server, with an average of ~1500 applications (many users have a few applications). Not all users are utilizing any java at all, however. Apache 1.3.33 is running and configured to serve static content.
The older servers with TC5 on them have ~6GB of available RAM, new servers for TC6 have 8GB of RAM.
Customer content is stored on a remote NAS server.
While I understand that having the <Host> and <Context> declaration for all users listed in server.xml, that is how we have it currently configured.
About the problem:
When tomcat is started and running on our existing 5.0.27 server, it uses approximately 1GB of RAM, and successfully runs all applications without issue.
When starting tomcat on the 6.0.14 server with the _exact_ same content, however, it runs out of heap space. This is the issue I'm running into right now that I can't seem to find any answers about. Attempting to increase the available heap results in an error allocating memory upon startup.
The output from jmap seems to show that the majority of the heap/memory is being consumed by character arrays.
When viewing the {tomcat.home}/work/{engine}/{application} directory, it appears there is a fair difference on how these are loaded into tomcat.
On Tomcat5, this directory consists of only directories that contain a WEB-INF directory in them, and each of those have a tldCache.ser file.
On Tomcat6, this directory consists of all directories located inside a directory that contains a WEB-INF directory.
For example:
Tomcat 6:
[root@servername ]# pwd ; find .
/web/tomcat/work/hosting/sitename.com
.
./_
./cgi
./stats
./php_uploads
./images
./ad
./Video
./Templates
./Flv
./Songs
./newwebsite
./old files
./manager
Tomcat5:
[root@servername ]# pwd; find .
/web/tomcat/work/hosting/sitename.com
.
./_
./_/tldCache.ser
And the contents of the actual home directory are:
[root@servername html]# find . -maxdepth 1 -type d
.
./cgi
./stats
./php_uploads
./WEB-INF
./images
./ad
./Video
./Templates
./Flv
./Songs
./newwebsite
./old files
I have read http://wiki.apache.org/tomcat/FAQ/Memory and attempted all the suggestions there, to no avail.
Thus, it appears that on Tomcat6, it is attempting to load all of these paths into memory, and eventually runs out. Tomcat5 did not load all of the paths to directories without applications, and thus uses less of the heap.
Does it seem like I am on the right track here? Have others encountered this in the past? Is there a way to prevent this behavior within Tomcat 6?
I thank you for any insight you can provide on this issue.
Sincerely,
Dan D.
Re: Out of Memory Errors when upgrading from 5.0.27 to 6.0.14
Posted by "Dan D." <da...@dubomail.com>.
This worked perfectly.
Thank you very much for your patience and help.
----- Original Message -----
From: "Caldarale, Charles R" <Ch...@unisys.com>
To: "Tomcat Users List" <us...@tomcat.apache.org>
Sent: Monday, February 11, 2008 5:09 PM
Subject: RE: Out of Memory Errors when upgrading from 5.0.27 to 6.0.14
> From: Dan D. [mailto:dan@dubomail.com]
> Subject: Re: Out of Memory Errors when upgrading from 5.0.27 to 6.0.14
>
> Would it work to then change the appBase to
> "/home/user/html/ROOT", or would
> it be better to have a secondary directory
> (/home/user/html/webapps) with a
> ROOT directory inside of it?
The latter, since you must not ever have appBase == docBase.
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
MATERIAL and is thus for use only by the intended recipient. If you
received this in error, please contact the sender and delete the e-mail
and its attachments from all computers.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Out of Memory Errors when upgrading from 5.0.27 to 6.0.14
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: Dan D. [mailto:dan@dubomail.com]
> Subject: Re: Out of Memory Errors when upgrading from 5.0.27 to 6.0.14
>
> Would it work to then change the appBase to
> "/home/user/html/ROOT", or would
> it be better to have a secondary directory
> (/home/user/html/webapps) with a
> ROOT directory inside of it?
The latter, since you must not ever have appBase == docBase.
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
MATERIAL and is thus for use only by the intended recipient. If you
received this in error, please contact the sender and delete the e-mail
and its attachments from all computers.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Out of Memory Errors when upgrading from 5.0.27 to 6.0.14
Posted by "Dan D." <da...@dubomail.com>.
>Since it's the declared appBase, *any* directories immediately under it
>are going to be treated as webapps, and will create entries in Tomcat's
>work directory.
conf/Catalina/www.domain.com doesn't have any xml files, so that isn't it.
However, /home/user/html is the users home directory, so is the root of
where they upload files. _Everything_ is in there. This then is the root
of my problem.
Would it work to then change the appBase to "/home/user/html/ROOT", or would
it be better to have a secondary directory (/home/user/html/webapps) with a
ROOT directory inside of it?
----- Original Message -----
From: "Caldarale, Charles R" <Ch...@unisys.com>
Subject: RE: Out of Memory Errors when upgrading from 5.0.27 to 6.0.14
> From: Dan D. [mailto:dan@dubomail.com]
> Subject: Re: Out of Memory Errors when upgrading from 5.0.27 to 6.0.14
>
> I added a ROOT directory in /home/user/html, with a META-INF
> and WEB-INF directory in it.
>
> When starting tomcat, however, the work directory is still full of
> directories that have nothing to do with the actual application.
What you did sounds good, so far. What else is under /home/user/html?
Since it's the declared appBase, *any* directories immediately under it
are going to be treated as webapps, and will create entries in Tomcat's
work directory.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Out of Memory Errors when upgrading from 5.0.27 to 6.0.14
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: Dan D. [mailto:dan@dubomail.com]
> Subject: Re: Out of Memory Errors when upgrading from 5.0.27 to 6.0.14
>
> I added a ROOT directory in /home/user/html, with a META-INF
> and WEB-INF directory in it.
>
> When starting tomcat, however, the work directory is still full of
> directories that have nothing to do with the actual application.
What you did sounds good, so far. What else is under /home/user/html?
Since it's the declared appBase, *any* directories immediately under it
are going to be treated as webapps, and will create entries in Tomcat's
work directory.
Also check what's under conf/Catalina/www.domain.com; any files
immediately under that directory with an extension of .xml will be
considered as context descriptors.
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
MATERIAL and is thus for use only by the intended recipient. If you
received this in error, please contact the sender and delete the e-mail
and its attachments from all computers.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Out of Memory Errors when upgrading from 5.0.27 to 6.0.14
Posted by "Dan D." <da...@dubomail.com>.
Chuck,
I updated the server.xml for this user to be:
<Host name='www.domain.com'
debug='0'
appBase='/home/user/html'
unpackWARs='true'
autoDeploy='false'>
<Alias>domain.com</Alias>
<Valve
className='org.apache.catalina.authenticator.SingleSignOn'
debug='0'/>
<Valve
className='org.apache.catalina.valves.AccessLogValve'
directory='/var/log/user'
prefix='tomcat_access.'
suffix='.log'
pattern='common'
resolveHosts='false'/>
</Host>
I added a ROOT directory in /home/user/html, with a META-INF and WEB-INF
directory in it. Added context.xml with the contents of:
<Context debug='0' reloadable='false'/>
inside META-INF.
Removed WEB-INF from /home/user/html.
Removed all contents from the Tomcat work directory as well.
When starting tomcat, however, the work directory is still full of
directories that have nothing to do with the actual application.
If I read your response correctly (there's a good chance I didn't), the
steps I took should be the right ones. However, if that were the case,
should all of the random directories still have a place in the Tomcat work
directory?
Thanks!
Dan.
----- Original Message -----
From: "Caldarale, Charles R" <Ch...@unisys.com>
To: "Tomcat Users List" <us...@tomcat.apache.org>
Sent: Thursday, February 07, 2008 6:09 PM
Subject: RE: Out of Memory Errors when upgrading from 5.0.27 to 6.0.14
> From: Dan D. [mailto:dan@dubomail.com]
> Subject: Re: Out of Memory Errors when upgrading from 5.0.27 to 6.0.14
>
> <Host name='www.domain.com'
> appBase='/home/user/html'
> <Context path='' docBase='' debug='0' reloadable='false'/>
> </Host>
Right there is the first problem: docBase and appBase must never be the
same - lots of things go wrong internally when they both refer to the
same directory. You need to remove the <Context> entry for the default
webapp from server.xml and place the default webapp files under
/home/user/html/ROOT. If you want to specify attributes for the default
webapp, put the <Context> element in
/home/user/html/ROOT/META-INF/context.xml, without the path or docBase
attributes. This works on all levels from 5.0 on up.
When that's straightened out, delete everything under the work directory
before restarting Tomcat.
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
MATERIAL and is thus for use only by the intended recipient. If you
received this in error, please contact the sender and delete the e-mail
and its attachments from all computers.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Out of Memory Errors when upgrading from 5.0.27 to 6.0.14
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: Dan D. [mailto:dan@dubomail.com]
> Subject: Re: Out of Memory Errors when upgrading from 5.0.27 to 6.0.14
>
> <Host name='www.domain.com'
> appBase='/home/user/html'
> <Context path='' docBase='' debug='0' reloadable='false'/>
> </Host>
Right there is the first problem: docBase and appBase must never be the
same - lots of things go wrong internally when they both refer to the
same directory. You need to remove the <Context> entry for the default
webapp from server.xml and place the default webapp files under
/home/user/html/ROOT. If you want to specify attributes for the default
webapp, put the <Context> element in
/home/user/html/ROOT/META-INF/context.xml, without the path or docBase
attributes. This works on all levels from 5.0 on up.
When that's straightened out, delete everything under the work directory
before restarting Tomcat.
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
MATERIAL and is thus for use only by the intended recipient. If you
received this in error, please contact the sender and delete the e-mail
and its attachments from all computers.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Out of Memory Errors when upgrading from 5.0.27 to 6.0.14
Posted by "Dan D." <da...@dubomail.com>.
It is entirely possible that this environment isn't setup in a optimal way,
and also that I may be messing up some of the proper names.
To provide a real world example (names and paths have been changed, but all
information is the same), here is the Host entry for a domain (same in both
TC5 and 6 currently) that we are using, as well as the work directories for
each:
server.xml entry:
<Host name='www.domain.com'
debug='0'
appBase='/home/user/html'
unpackWARs='true'
autoDeploy='false'>
<Alias>domain.com</Alias>
<Context path='' docBase='' debug='0' reloadable='false'/>
<Valve className='org.apache.catalina.authenticator.SingleSignOn'
debug='0'/>
<Valve className='org.apache.catalina.valves.AccessLogValve'
directory='/var/log/user'
prefix='tomcat_access.'
suffix='.log'
pattern='common'
resolveHosts='false'/>
</Host>
Work directory under Tomcat 5:
[root@javaserver www.domain.com]# pwd; find .
/web/tomcat/work/hosting/www.domain.com
.
./_
./_/tldCache.ser
Work directory under Tomcat 6:
[root@javaserver www.domain.com]# pwd ; find .
/web/tomcat/work/hosting/www.domain.com
.
./_
./stats
./files
./includes
./misc
./modules
./profiles
./scripts
./sites
./themes
./forum
./lists
./members
./staging
As for why the WEB-INF is at the same level as a war file, it is a shared
hosting environment, and that is their document root. It allows the user to
have an application at http://domain.com as well as upload a war file for
additional applications they wish to run from say, http://domain.com/blog.
It sounds as though this should be done differently?
I'm more or less rebuilding the environment with the upgrade, so any
recommendations are very welcome.
Thanks,
Dan
----- Original Message -----
From: "Caldarale, Charles R" <Ch...@unisys.com>
To: "Tomcat Users List" <us...@tomcat.apache.org>
Sent: Thursday, February 07, 2008 2:46 PM
Subject: RE: Out of Memory Errors when upgrading from 5.0.27 to 6.0.14
> From: Dan D. [mailto:dan@dubomail.com]
> Subject: Re: Out of Memory Errors when upgrading from 5.0.27 to 6.0.14
>
> Customer has the following directory structure:
>
> /home/user/html
> \_ WEB-INF/
> | \_ classes/
> | \_ lib/
> \_ images/
> \_ movies/
> \_ mp3s/
> \_ blog/
> \_ new_application.war
That looks very wrong. Why is there a .war file at the same level as
WEB-INF?
> Now, in Tomcat 5, they would have a work directory structure of the
> following:
> /{tomcat.home}/work/{engine.name}/
> \_ ./application_name
> \_ _ (the directory name is simply "_")
> | \_ tldCache.ser
> \_ new_application
> \_ tldCache.ser
That also looks wrong; all tldCache.ser files should be at the same
level - immediately under the respective work/Catalina/[host]/[appname]
directories. Even if you're confusing "application_name" with the name
on a <Host> element, something seems very broken with your appBase
and/or docBase settings. Until that's straightened out, all bets are
off.
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
MATERIAL and is thus for use only by the intended recipient. If you
received this in error, please contact the sender and delete the e-mail
and its attachments from all computers.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Out of Memory Errors when upgrading from 5.0.27 to 6.0.14
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: Dan D. [mailto:dan@dubomail.com]
> Subject: Re: Out of Memory Errors when upgrading from 5.0.27 to 6.0.14
>
> Customer has the following directory structure:
>
> /home/user/html
> \_ WEB-INF/
> | \_ classes/
> | \_ lib/
> \_ images/
> \_ movies/
> \_ mp3s/
> \_ blog/
> \_ new_application.war
That looks very wrong. Why is there a .war file at the same level as
WEB-INF?
> Now, in Tomcat 5, they would have a work directory structure of the
> following:
> /{tomcat.home}/work/{engine.name}/
> \_ ./application_name
> \_ _ (the directory name is simply "_")
> | \_ tldCache.ser
> \_ new_application
> \_ tldCache.ser
That also looks wrong; all tldCache.ser files should be at the same
level - immediately under the respective work/Catalina/[host]/[appname]
directories. Even if you're confusing "application_name" with the name
on a <Host> element, something seems very broken with your appBase
and/or docBase settings. Until that's straightened out, all bets are
off.
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
MATERIAL and is thus for use only by the intended recipient. If you
received this in error, please contact the sender and delete the e-mail
and its attachments from all computers.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Out of Memory Errors when upgrading from 5.0.27 to 6.0.14
Posted by "Dan D." <da...@dubomail.com>.
Most of the advice that I received on this topic was pretty much the same:
Use a profiler to determine what is using the heap.
I went out and got a demo copy of yourkit as Filip suggested, and ran it
against a 512MB heap dump (lowered max heap to that, and
used the heap that was dumped on OOME).
Well over half of the heap was consumed by
java.util.HashMap(.HashMap$Entry[]).
This lead me back to my suspicion that the issue may lie with the number of
directories that are created in the tomcat work/engine/application
directory.
The following example explains the difference in behavior between Tomcat 5
and Tomcat 6 that I am seeing:
Customer has the following directory structure:
/home/user/html
\_ WEB-INF/
| \_ classes/
| \_ lib/
\_ images/
\_ movies/
\_ mp3s/
\_ blog/
\_ new_application.war
Now, in Tomcat 5, they would have a work directory structure of the
following:
/{tomcat.home}/work/{engine.name}/
\_ ./application_name
\_ _ (the directory name is simply “_”)
| \_ tldCache.ser
\_ new_application
\_ tldCache.ser
Tomcat 6, however, would set it up as follows:
./application_name
\_ _
\_ new_application
\_ images/
\_ movies/
\_ mp3s/
\_ blog/
Add in a few more directories for flair (say, for subdomains / aliases /
other applications), and we can see that the number of these directories
that are created gets out of hand quickly.
To put numbers on it, for Tomcat 5, we have a total of ~11,000 files and
directories in the {tomcat.home}/work/{engine.name} directory and
subdirectories. In Tomcat 6, it fails on OOME at around 27,000 directories.
So, I suppose my question is, is this behavior intended? Is there a config
option I can use to disable it at all, or do we simply have to reduce the
number of applications that are being run under the new version?
As a side note, I was able to eventually get Tomcat to start, once I removed
a number of applications that had the highest count of directories that were
created in the work directory structure.
Also, we do hope to move to 64bit versions soon, but they aren't here yet,
so I only have the 32 bit to work with.
Thanks for your help,
Dan
From: "Christopher Schultz" <ch...@christopherschultz.net>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Dan,
>
> (While others are handling your primary question, I thought I'd mention
> a couple of things)
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Out of Memory Errors when upgrading from 5.0.27 to 6.0.14
Posted by prakash shanmugam <pr...@gmail.com>.
hai dan,
If I am not wrong, try to close all the connection string and make the user
defined variables as null.
(Eg)
1) (Statement object)--à st.close();
2) (String)às=null;
Hope it will solve your problem to some extend, even then if your problem is
not solved. Then you have to change the heap size in the tomcat server (that
may be relative to RAM size of the machine where the tomcat is running).
--
with Regards
S.Prakash
Re: Out of Memory Errors when upgrading from 5.0.27 to 6.0.14
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Dan,
(While others are handling your primary question, I thought I'd mention
a couple of things)
Dan D. wrote:
| I'm currently working on upgrading our systems from using Tomcat
| 5.0.27 to 6.0.14.
While you're at it, you might want to:
* If you're running a 64-bit OS, upgrade to a 64-bit JVM.
~ Without this, your 6GB -> 8GB memory upgrade isn't going
~ to give your Java process any more memory, since the 32-bit
~ JVMs can only give you about 2GB of useful heap.
* Consider running fewer application per JVM. Are you really running
~ ~1500 applications per Tomcat instance? That seems like quite a lot
~ (though by no means should it be considered impossible!)
| The output from jmap seems to show that the majority of the
| heap/memory is being consumed by character arrays.
This is not surprising. Java uses a lot of String objects internally for
lots of stuff (like package, class, and method names, for instance) and
every single one of them is backed by a character array.
Don't read too much into the fact that character arrays take up a lot of
space, because they can be allocated in all kinds of ways that are not
related to the "character array" itself.
What you really want to do is run a profiler to find out where many of
the character arrays were created in the first place.
Good luck,
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEARECAAYFAkeogdsACgkQ9CaO5/Lv0PBATwCfaOcnTrgBCr/Dfm8e/R+/1xpk
fRUAnRL+Pe4mRmUSBsK3Ts+JPGZ3o9N8
=CkWb
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Out of Memory Errors when upgrading from 5.0.27 to 6.0.14
Posted by Filip Hanik - Dev Lists <de...@hanik.com>.
ok, so there is a leak in the Java Heap, one way to figure out what is
going would be to add
-XX:+*HeapDumpOnOutOfMemoryError
to the command line options.
This will dump the heap to a file that you can analyze. You need some
serious RAM on a box to analyze a thread dump like that, but it will
tell you everything.
I use www.yourkit.com when I analyze heapdumps
Filip
*
Dan D. wrote:
> Following is the exception that is thrown, along with a few
> surrounding lines from the catalina.out log file:
>
> [Full GC 1982271K->1982271K(1999424K), 7.5088700 secs]
> [Full GC 1982271K->1982269K(1999424K), 7.5053160 secs]
> java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [Full GC 1982271K->1982271K(1999424K), 7.4678950 secs]
> [Full GC 1982271K->1982246K(1999424K), 9.5055930 secs]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)
> at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at java.util.Arrays.copyOf(Arrays.java:2882)
> at
> java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
>
> at
> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
> at java.lang.StringBuffer.append(StringBuffer.java:224)
> at java.io.StringWriter.write(StringWriter.java:95)
> at java.io.PrintWriter.write(PrintWriter.java:412)
> at java.io.PrintWriter.write(PrintWriter.java:429)
> at java.io.PrintWriter.print(PrintWriter.java:559)
> at java.io.PrintWriter.println(PrintWriter.java:695)
> at java.lang.Throwable.printStackTrace(Throwable.java:512)
> at
> java.util.logging.SimpleFormatter.format(SimpleFormatter.java:72)
> at java.util.logging.StreamHandler.publish(StreamHandler.java:179)
> at
> java.util.logging.ConsoleHandler.publish(ConsoleHandler.java:88)
> at java.util.logging.Logger.log(Logger.java:472)
> at java.util.logging.Logger.doLog(Logger.java:494)
> at java.util.logging.Logger.logp(Logger.java:694)
> [Full GC 1982271K->1982271K(1999424K), 7.5629480 secs]
> [Full GC 1982271K->1982247K(1999424K), 7.4791920 secs]
> at org.apache.juli.logging.DirectJDKLog.log(DirectJDKLog.java:167)
> at
> org.apache.juli.logging.DirectJDKLog.error(DirectJDKLog.java:135)
> at
> org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:929)
>
> at
> org.apache.catalina.startup.HostConfig.deployDirectories(HostConfig.java:883)
>
> at
> org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:492)
> at
> org.apache.catalina.startup.HostConfig.start(HostConfig.java:1138)
> at
> org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:311)
>
> at
> org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:117)
>
> at
> org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1053)
> at
> org.apache.catalina.core.StandardHost.start(StandardHost.java:719)
> at
> org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045)
> at
> org.apache.catalina.core.StandardEngine.start(StandardEngine.java:443)
> at
> org.apache.catalina.core.StandardService.start(StandardService.java:516)
> at
> org.apache.catalina.core.StandardServer.start(StandardServer.java:710)
> at org.apache.catalina.startup.Catalina.start(Catalina.java:566)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [Full GC 1982271K->1982271K(1999424K), 7.4982800 secs]
> [Full GC 1982271K->1982271K(1999424K), 7.4976100 secs]
> [Full GC 1982271K->1982244K(1999424K), 9.5026360 secs]
> [Full GC 1982272K->1982244K(1999424K), 7.4494750 secs]
> [Full GC 1982272K->1982272K(1999424K), 7.4763190 secs]
> [Full GC 1982272K->1982234K(1999424K), 9.3931880 secs]
> [Full GC[Unloading class
> com.atlassian.confluence.event.events.plugin.PluginDisableEvent]
> [Unloading class org.apache.log4j.spi.RepositorySelector]
> [Unloading class org.apache.xerces.xni.grammars.Grammar]
> [Unloading class org.apache.xerces.dom.DocumentTypeImpl]
> [Unloading class
> org.springframework.web.util.IntrospectorCleanupListener]
> [Unloading class org.apache.xerces.impl.io.UTF8Reader]
> [Unloading class org.apache.xerces.impl.dtd.XMLDTDDescription]
>
> <snip> (many more classes are unloaded here)
>
> [Unloading class org.springframework.web.context.ContextLoaderListener]
> [Unloading class com.atlassian.core.util.XMLUtils]
> [Unloading class
> org.springframework.beans.factory.support.ReplaceOverride]
> 1982271K->1982271K(1999424K), 7.4622510 secs]
> [Full GC 1982271K->1981801K(1999424K), 8.9731420 secs]
>
> And the Full GC's continue for a little while, and eventually the
> process just dies off.
>
>
> ----- Original Message ----- From: "Filip Hanik - Dev Lists"
> <de...@hanik.com>
> To: "Tomcat Users List" <us...@tomcat.apache.org>
> Sent: Monday, February 04, 2008 5:22 PM
> Subject: Re: Out of Memory Errors when upgrading from 5.0.27 to 6.0.14
>
>
>> what's the message after
>>
>> java.lang.OutOfMemoryError
>>
>> as there can be many different types of errors.
>>
>> Filip
>>
>> Dan D. wrote:
>>> Hello all,
>>>
>>> I'm currently working on upgrading our systems from using Tomcat
>>> 5.0.27 to 6.0.14.
>>>
>>> The upgrades are not being done in place (ie, we will have both
>>> systems out there, and will be doing migrations to the new
>>> version). I have been able to resolve most of the issues we have
>>> run into thus far, however, I am now somewhat stuck, and am hoping
>>> someone here may be able to help.
>>>
>>> About the environment:
>>> Shared hosting, with approximately 300-600 users on a server, with
>>> an average of ~1500 applications (many users have a few
>>> applications). Not all users are utilizing any java at all,
>>> however. Apache 1.3.33 is running and configured to serve static
>>> content.
>>>
>>> The older servers with TC5 on them have ~6GB of available RAM, new
>>> servers for TC6 have 8GB of RAM.
>>> Customer content is stored on a remote NAS server.
>>>
>>> While I understand that having the <Host> and <Context> declaration
>>> for all users listed in server.xml, that is how we have it currently
>>> configured.
>>> About the problem:
>>> When tomcat is started and running on our existing 5.0.27 server, it
>>> uses approximately 1GB of RAM, and successfully runs all
>>> applications without issue.
>>>
>>> When starting tomcat on the 6.0.14 server with the _exact_ same
>>> content, however, it runs out of heap space. This is the issue I'm
>>> running into right now that I can't seem to find any answers about.
>>> Attempting to increase the available heap results in an error
>>> allocating memory upon startup.
>>>
>>>
>>> The output from jmap seems to show that the majority of the
>>> heap/memory is being consumed by character arrays.
>>>
>>> When viewing the {tomcat.home}/work/{engine}/{application}
>>> directory, it appears there is a fair difference on how these are
>>> loaded into tomcat.
>>>
>>> On Tomcat5, this directory consists of only directories that contain
>>> a WEB-INF directory in them, and each of those have a tldCache.ser
>>> file.
>>>
>>> On Tomcat6, this directory consists of all directories located
>>> inside a directory that contains a WEB-INF directory.
>>>
>>> For example:
>>> Tomcat 6:
>>> [root@servername ]# pwd ; find .
>>> /web/tomcat/work/hosting/sitename.com
>>> .
>>> ./_
>>> ./cgi
>>> ./stats
>>> ./php_uploads
>>> ./images
>>> ./ad
>>> ./Video
>>> ./Templates
>>> ./Flv
>>> ./Songs
>>> ./newwebsite
>>> ./old files
>>> ./manager
>>>
>>> Tomcat5:
>>> [root@servername ]# pwd; find .
>>> /web/tomcat/work/hosting/sitename.com
>>> .
>>> ./_
>>> ./_/tldCache.ser
>>>
>>> And the contents of the actual home directory are:
>>> [root@servername html]# find . -maxdepth 1 -type d
>>> .
>>> ./cgi
>>> ./stats
>>> ./php_uploads
>>> ./WEB-INF
>>> ./images
>>> ./ad
>>> ./Video
>>> ./Templates
>>> ./Flv
>>> ./Songs
>>> ./newwebsite
>>> ./old files
>>>
>>> I have read http://wiki.apache.org/tomcat/FAQ/Memory and attempted
>>> all the suggestions there, to no avail.
>>>
>>> Thus, it appears that on Tomcat6, it is attempting to load all of
>>> these paths into memory, and eventually runs out. Tomcat5 did not
>>> load all of the paths to directories without applications, and thus
>>> uses less of the heap.
>>>
>>> Does it seem like I am on the right track here? Have others
>>> encountered this in the past? Is there a way to prevent this
>>> behavior within Tomcat 6?
>>>
>>> I thank you for any insight you can provide on this issue.
>>>
>>> Sincerely,
>>> Dan D.
>>>
>>>
>>> ------------------------------------------------------------------------
>>>
>>>
>>> No virus found in this incoming message.
>>> Checked by AVG Free Edition. Version: 7.5.516 / Virus Database:
>>> 269.19.19/1257 - Release Date: 2/3/2008 5:49 PM
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To start a new topic, e-mail: users@tomcat.apache.org
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
>
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Out of Memory Errors when upgrading from 5.0.27 to 6.0.14
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: Dan D. [mailto:dan@dubomail.com]
> Subject: Re: Out of Memory Errors when upgrading from 5.0.27 to 6.0.14
>
> [Full GC 1982271K->1982271K(1999424K), 7.5088700 secs]
You obviously have a serious memory leak - the 2 GB heap is filled with
live objects, so the GC doesn't really accomplish anything. Try using a
profiler to find out not only what objects are taking up the space but
also who's allocating them. Do this with a much smaller heap so you can
get to the full condition more quickly.
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
MATERIAL and is thus for use only by the intended recipient. If you
received this in error, please contact the sender and delete the e-mail
and its attachments from all computers.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Out of Memory Errors when upgrading from 5.0.27 to 6.0.14
Posted by "Dan D." <da...@dubomail.com>.
Following is the exception that is thrown, along with a few surrounding
lines from the catalina.out log file:
[Full GC 1982271K->1982271K(1999424K), 7.5088700 secs]
[Full GC 1982271K->1982269K(1999424K), 7.5053160 secs]
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[Full GC 1982271K->1982271K(1999424K), 7.4678950 secs]
[Full GC 1982271K->1982246K(1999424K), 9.5055930 secs]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)
Caused by: java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2882)
at
java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
at
java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
at java.lang.StringBuffer.append(StringBuffer.java:224)
at java.io.StringWriter.write(StringWriter.java:95)
at java.io.PrintWriter.write(PrintWriter.java:412)
at java.io.PrintWriter.write(PrintWriter.java:429)
at java.io.PrintWriter.print(PrintWriter.java:559)
at java.io.PrintWriter.println(PrintWriter.java:695)
at java.lang.Throwable.printStackTrace(Throwable.java:512)
at java.util.logging.SimpleFormatter.format(SimpleFormatter.java:72)
at java.util.logging.StreamHandler.publish(StreamHandler.java:179)
at java.util.logging.ConsoleHandler.publish(ConsoleHandler.java:88)
at java.util.logging.Logger.log(Logger.java:472)
at java.util.logging.Logger.doLog(Logger.java:494)
at java.util.logging.Logger.logp(Logger.java:694)
[Full GC 1982271K->1982271K(1999424K), 7.5629480 secs]
[Full GC 1982271K->1982247K(1999424K), 7.4791920 secs]
at org.apache.juli.logging.DirectJDKLog.log(DirectJDKLog.java:167)
at org.apache.juli.logging.DirectJDKLog.error(DirectJDKLog.java:135)
at
org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:929)
at
org.apache.catalina.startup.HostConfig.deployDirectories(HostConfig.java:883)
at
org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:492)
at
org.apache.catalina.startup.HostConfig.start(HostConfig.java:1138)
at
org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:311)
at
org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:117)
at
org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1053)
at
org.apache.catalina.core.StandardHost.start(StandardHost.java:719)
at
org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045)
at
org.apache.catalina.core.StandardEngine.start(StandardEngine.java:443)
at
org.apache.catalina.core.StandardService.start(StandardService.java:516)
at
org.apache.catalina.core.StandardServer.start(StandardServer.java:710)
at org.apache.catalina.startup.Catalina.start(Catalina.java:566)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[Full GC 1982271K->1982271K(1999424K), 7.4982800 secs]
[Full GC 1982271K->1982271K(1999424K), 7.4976100 secs]
[Full GC 1982271K->1982244K(1999424K), 9.5026360 secs]
[Full GC 1982272K->1982244K(1999424K), 7.4494750 secs]
[Full GC 1982272K->1982272K(1999424K), 7.4763190 secs]
[Full GC 1982272K->1982234K(1999424K), 9.3931880 secs]
[Full GC[Unloading class
com.atlassian.confluence.event.events.plugin.PluginDisableEvent]
[Unloading class org.apache.log4j.spi.RepositorySelector]
[Unloading class org.apache.xerces.xni.grammars.Grammar]
[Unloading class org.apache.xerces.dom.DocumentTypeImpl]
[Unloading class org.springframework.web.util.IntrospectorCleanupListener]
[Unloading class org.apache.xerces.impl.io.UTF8Reader]
[Unloading class org.apache.xerces.impl.dtd.XMLDTDDescription]
<snip> (many more classes are unloaded here)
[Unloading class org.springframework.web.context.ContextLoaderListener]
[Unloading class com.atlassian.core.util.XMLUtils]
[Unloading class org.springframework.beans.factory.support.ReplaceOverride]
1982271K->1982271K(1999424K), 7.4622510 secs]
[Full GC 1982271K->1981801K(1999424K), 8.9731420 secs]
And the Full GC's continue for a little while, and eventually the process
just dies off.
----- Original Message -----
From: "Filip Hanik - Dev Lists" <de...@hanik.com>
To: "Tomcat Users List" <us...@tomcat.apache.org>
Sent: Monday, February 04, 2008 5:22 PM
Subject: Re: Out of Memory Errors when upgrading from 5.0.27 to 6.0.14
> what's the message after
>
> java.lang.OutOfMemoryError
>
> as there can be many different types of errors.
>
> Filip
>
> Dan D. wrote:
>> Hello all,
>>
>> I'm currently working on upgrading our systems from using Tomcat 5.0.27
>> to 6.0.14.
>>
>> The upgrades are not being done in place (ie, we will have both systems
>> out there, and will be doing migrations to the new version). I have been
>> able to resolve most of the issues we have run into thus far, however, I
>> am now somewhat stuck, and am hoping someone here may be able to help.
>>
>> About the environment:
>> Shared hosting, with approximately 300-600 users on a server, with an
>> average of ~1500 applications (many users have a few applications). Not
>> all users are utilizing any java at all, however. Apache 1.3.33 is
>> running and configured to serve static content.
>>
>> The older servers with TC5 on them have ~6GB of available RAM, new
>> servers for TC6 have 8GB of RAM.
>> Customer content is stored on a remote NAS server.
>>
>> While I understand that having the <Host> and <Context> declaration for
>> all users listed in server.xml, that is how we have it currently
>> configured.
>> About the problem:
>> When tomcat is started and running on our existing 5.0.27 server, it uses
>> approximately 1GB of RAM, and successfully runs all applications without
>> issue.
>>
>> When starting tomcat on the 6.0.14 server with the _exact_ same content,
>> however, it runs out of heap space. This is the issue I'm running into
>> right now that I can't seem to find any answers about. Attempting to
>> increase the available heap results in an error allocating memory upon
>> startup.
>>
>>
>> The output from jmap seems to show that the majority of the heap/memory
>> is being consumed by character arrays.
>>
>> When viewing the {tomcat.home}/work/{engine}/{application} directory, it
>> appears there is a fair difference on how these are loaded into tomcat.
>>
>> On Tomcat5, this directory consists of only directories that contain a
>> WEB-INF directory in them, and each of those have a tldCache.ser file.
>>
>> On Tomcat6, this directory consists of all directories located inside a
>> directory that contains a WEB-INF directory.
>>
>> For example:
>> Tomcat 6:
>> [root@servername ]# pwd ; find .
>> /web/tomcat/work/hosting/sitename.com
>> .
>> ./_
>> ./cgi
>> ./stats
>> ./php_uploads
>> ./images
>> ./ad
>> ./Video
>> ./Templates
>> ./Flv
>> ./Songs
>> ./newwebsite
>> ./old files
>> ./manager
>>
>> Tomcat5:
>> [root@servername ]# pwd; find .
>> /web/tomcat/work/hosting/sitename.com
>> .
>> ./_
>> ./_/tldCache.ser
>>
>> And the contents of the actual home directory are:
>> [root@servername html]# find . -maxdepth 1 -type d
>> .
>> ./cgi
>> ./stats
>> ./php_uploads
>> ./WEB-INF
>> ./images
>> ./ad
>> ./Video
>> ./Templates
>> ./Flv
>> ./Songs
>> ./newwebsite
>> ./old files
>>
>> I have read http://wiki.apache.org/tomcat/FAQ/Memory and attempted all
>> the suggestions there, to no avail.
>>
>> Thus, it appears that on Tomcat6, it is attempting to load all of these
>> paths into memory, and eventually runs out. Tomcat5 did not load all of
>> the paths to directories without applications, and thus uses less of the
>> heap.
>>
>> Does it seem like I am on the right track here? Have others encountered
>> this in the past? Is there a way to prevent this behavior within Tomcat
>> 6?
>>
>> I thank you for any insight you can provide on this issue.
>>
>> Sincerely,
>> Dan D.
>>
>> ------------------------------------------------------------------------
>>
>> No virus found in this incoming message.
>> Checked by AVG Free Edition. Version: 7.5.516 / Virus Database:
>> 269.19.19/1257 - Release Date: 2/3/2008 5:49 PM
>>
>
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Out of Memory Errors when upgrading from 5.0.27 to 6.0.14
Posted by Filip Hanik - Dev Lists <de...@hanik.com>.
what's the message after
java.lang.OutOfMemoryError
as there can be many different types of errors.
Filip
Dan D. wrote:
> Hello all,
>
> I'm currently working on upgrading our systems from using Tomcat 5.0.27 to 6.0.14.
>
> The upgrades are not being done in place (ie, we will have both systems out there, and will be doing migrations to the new version). I have been able to resolve most of the issues we have run into thus far, however, I am now somewhat stuck, and am hoping someone here may be able to help.
>
> About the environment:
> Shared hosting, with approximately 300-600 users on a server, with an average of ~1500 applications (many users have a few applications). Not all users are utilizing any java at all, however. Apache 1.3.33 is running and configured to serve static content.
>
> The older servers with TC5 on them have ~6GB of available RAM, new servers for TC6 have 8GB of RAM.
>
> Customer content is stored on a remote NAS server.
>
> While I understand that having the <Host> and <Context> declaration for all users listed in server.xml, that is how we have it currently configured.
>
> About the problem:
> When tomcat is started and running on our existing 5.0.27 server, it uses approximately 1GB of RAM, and successfully runs all applications without issue.
>
> When starting tomcat on the 6.0.14 server with the _exact_ same content, however, it runs out of heap space. This is the issue I'm running into right now that I can't seem to find any answers about. Attempting to increase the available heap results in an error allocating memory upon startup.
>
>
> The output from jmap seems to show that the majority of the heap/memory is being consumed by character arrays.
>
> When viewing the {tomcat.home}/work/{engine}/{application} directory, it appears there is a fair difference on how these are loaded into tomcat.
>
> On Tomcat5, this directory consists of only directories that contain a WEB-INF directory in them, and each of those have a tldCache.ser file.
>
> On Tomcat6, this directory consists of all directories located inside a directory that contains a WEB-INF directory.
>
> For example:
> Tomcat 6:
> [root@servername ]# pwd ; find .
> /web/tomcat/work/hosting/sitename.com
> .
> ./_
> ./cgi
> ./stats
> ./php_uploads
> ./images
> ./ad
> ./Video
> ./Templates
> ./Flv
> ./Songs
> ./newwebsite
> ./old files
> ./manager
>
> Tomcat5:
> [root@servername ]# pwd; find .
> /web/tomcat/work/hosting/sitename.com
> .
> ./_
> ./_/tldCache.ser
>
> And the contents of the actual home directory are:
> [root@servername html]# find . -maxdepth 1 -type d
> .
> ./cgi
> ./stats
> ./php_uploads
> ./WEB-INF
> ./images
> ./ad
> ./Video
> ./Templates
> ./Flv
> ./Songs
> ./newwebsite
> ./old files
>
> I have read http://wiki.apache.org/tomcat/FAQ/Memory and attempted all the suggestions there, to no avail.
>
> Thus, it appears that on Tomcat6, it is attempting to load all of these paths into memory, and eventually runs out. Tomcat5 did not load all of the paths to directories without applications, and thus uses less of the heap.
>
> Does it seem like I am on the right track here? Have others encountered this in the past? Is there a way to prevent this behavior within Tomcat 6?
>
> I thank you for any insight you can provide on this issue.
>
> Sincerely,
> Dan D.
>
>
> ------------------------------------------------------------------------
>
> No virus found in this incoming message.
> Checked by AVG Free Edition.
> Version: 7.5.516 / Virus Database: 269.19.19/1257 - Release Date: 2/3/2008 5:49 PM
>
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org