You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by S Arvind <ar...@gmail.com> on 2009/04/29 14:20:55 UTC
GC Problem
Tomcat after running for couple of hours , somthing is getting wrong and it
is keep on doing Garbage Collection(as it dispaly in console) . And tomcat
also becoming dump slow.. and we cant able to do any, until it restarts...
JAVA_HOME=/usr/java/jdk1.5.0_13
JRE_HOME=/usr/java/jdk1.5.0_13/jre
export CATALINA_OPTS="-verbose:gc -Xloggc:gc.log -XX:+PrintGCDetails
-XX:MaxPermSize=512M -Xms1024M -Xmx1024M -Djava.awt.headless=true"
server is 4 gb ram with two tomcat instance (5 & 6).
Thanks,
Arvind S
"Many of lifes failure are people who did not realize how close they were to
success when they gave up."
-Thomas Edison
Re: GC Problem
Posted by S Arvind <ar...@gmail.com>.
in console its keep on displaying this every second..
12945.607: [Full GC [PSYoungGen: 340288K->39757K(343552K)] [PSOldGen:
699071K->699071K(699072K)] 1039359K->738829K(1042624K) [PSPermGen:
346754K->346754K(346816K)], 2.6522010 secs]
sessionCreated----->C63974AACCC2800C89576C0C693467CB
12948.363: [Full GC [PSYoungGen: 340288K->24050K(343552K)] [PSOldGen:
699071K->699071K(699072K)] 1039359K->723122K(1042624K) [PSPermGen:
346754K->346754K(346816K)], 2.6768820 secs]
sessionCreated----->8B39EA776552C44F126958B979E36684
12951.149: [Full GC [PSYoungGen: 340288K->37166K(343552K)] [PSOldGen:
699071K->699071K(699072K)] 1039359K->736238K(1042624K) [PSPermGen:
346755K->346755K(346816K)], 2.6909590 secs]
sessionCreated----->D2E9E1A9B8C77500D6BACD40A3194F1D
Apr 29, 2009 8:39:11 AM org.apache.tomcat.util.http.Parameters
processParameters
WARNING: Parameters: Invalid chunk ignored.
12953.947: [Full GC [PSYoungGen: 340288K->37218K(343552K)] [PSOldGen:
699071K->699071K(699072K)] 1039359K->736290K(1042624K) [PSPermGen:
346756K->346756K(346816K)], 2.6655880 secs]
sessionCreated----->E33C0E94BB3C4AE3A767FC2F5025BDFF
12956.716: [Full GC [PSYoungGen: 340288K->37348K(343552K)] [PSOldGen:
699071K->699071K(699072K)] 1039359K->736420K(1042624K) [PSPermGen:
346756K->346756K(346816K)], 2.6625880 secs]
sessionCreated----->93623EE1D595BD38ED5CE42BF15165CB
12959.483: [Full GC [PSYoungGen: 340288K->24075K(343552K)] [PSOldGen:
699071K->699071K(699072K)] 1039359K->723147K(1042624K) [PSPermGen:
346756K->346755K(346816K)], 2.6804850 secs]
sessionCreated----->DC1BBF75947F0F52CDBA4461B33FD0E1
Apr 29, 2009 8:39:19 AM org.apache.tomcat.util.http.Parameters
processParameters
WARNING: Parameters: Invalid chunk ignored.
12962.272: [Full GC [PSYoungGen: 340288K->30321K(343552K)] [PSOldGen:
699071K->699071K(699072K)] 1039359K->729393K(1042624K) [PSPermGen:
346756K->346756K(346816K)], 2.6328120 secs]
Arvind
*
"Many of lifes failure are people who did not realize how close they were to
success when they gave up."
-Thomas Edison
*
On Wed, Apr 29, 2009 at 5:50 PM, S Arvind <ar...@gmail.com> wrote:
> Tomcat after running for couple of hours , somthing is getting wrong and it
> is keep on doing Garbage Collection(as it dispaly in console) . And tomcat
> also becoming dump slow.. and we cant able to do any, until it restarts...
>
>
> JAVA_HOME=/usr/java/jdk1.5.0_13
> JRE_HOME=/usr/java/jdk1.5.0_13/jre
> export CATALINA_OPTS="-verbose:gc -Xloggc:gc.log -XX:+PrintGCDetails
> -XX:MaxPermSize=512M -Xms1024M -Xmx1024M -Djava.awt.headless=true"
>
> server is 4 gb ram with two tomcat instance (5 & 6).
>
> Thanks,
> Arvind S
>
>
>
>
>
> "Many of lifes failure are people who did not realize how close they were
> to success when they gave up."
> -Thomas Edison
>
Re: GC Problem
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Pid,
On 4/29/2009 9:55 AM, Pid wrote:
> S Arvind wrote:
>> Is there any good settings for GC for tomcat running in server with 4GB and
>> Quad Core processor. I bascially need GC parameter for RESOURCE HUNGRY WEB
>> APPLICATION
>
> Why not try fixing the memory leak instead?
>
> Reconfiguring the JVM params, memory or GC will only delay the
> inevitable OutOfMemory error.
+1
> The "YourKit" profiler is frequently mentioned on this list, but there
> are others.
You can also use the simple scripts below to help you out. We were
getting OOME errors after several years of smooth sailing, and it turned
out that our load had simply increased to the point that we needed to
allocate more than 64MB heap to our webapp. We used the scripts below to
monitor everything for a few weeks afterward.
Enjoy,
- -chris
HealthCheck.jsp
- ---------------
<?xml version="1.0" encoding="UTF-8" ?>
<%@ page
contentType="text/xml; charset=UTF-8"
session="false"
language="java"
import="
java.lang.Runtime,
java.sql.Connection,
java.sql.PreparedStatement,
java.sql.ResultSet,
java.sql.SQLException,
javax.sql.DataSource,
javax.naming.Context,
javax.naming.InitialContext,
javax.naming.NamingException,
org.apache.tomcat.dbcp.dbcp.BasicDataSource
"
%><%!
private DataSource getDataSource()
throws NamingException
{
Context ctx = new InitialContext();
DataSource ds =
(DataSource)ctx.lookup("java:/comp/env/jdbc/YOUR-DATASOURCE-NAME");
if(null == ds)
throw new NamingException("Cannot obtain DataSource");
return ds;
}
private String escapeQuotes(Object o)
{
String s;
if(o instanceof String)
s = (String)o;
else if(null == o)
s = "null";
else
s = o.toString();
s.replaceAll("\"", """); // "
return s;
}
%><health-report time="<%= System.currentTimeMillis() %>">
<%
response.setCharacterEncoding("UTF-8"); // Just in case
Runtime runtime = Runtime.getRuntime();
long maxMemory = runtime.maxMemory();
long totalMemory = runtime.totalMemory();
long freeMemory = runtime.freeMemory();
int maxActiveConnections = 0;
int maxIdleConnections = 0;
int minIdleConnections = 0;
int activeConnections = 0;
int idleConnections = 0;
long checkoutDelay = 0;
boolean connectionOkay = false;
Connection conn = null;
PreparedStatement ps = null;
ResultSet rs = null;
try
{
DataSource ds = getDataSource();
if(ds instanceof BasicDataSource)
{
BasicDataSource bds = (BasicDataSource)ds;
maxActiveConnections = bds.getMaxActive();
maxIdleConnections = bds.getMaxIdle();
minIdleConnections = bds.getMinIdle();
activeConnections = bds.getNumActive();
idleConnections = bds.getNumIdle();
}
long elapsed = System.currentTimeMillis();
conn = ds.getConnection();
checkoutDelay = System.currentTimeMillis() - elapsed;
ps = conn.prepareStatement("SELECT 1"); // Simple query
rs = ps.executeQuery();
connectionOkay = rs.next();
}
catch (Throwable t)
{
while(null != t)
{
%> <error type="<%= t.getClass().getName() %>">
<% if(null != t.getMessage() && !"".equals(t.getMessage().trim())) { %>
<message><%= escapeQuotes(t.getMessage()) %></message>
<% }
%> <stack-trace>
<%
StackTraceElement[] st = t.getStackTrace();
for(int i=0; i<st.length; ++i)
{
StackTraceElement ste = st[i];
%> at <%= ste.getClassName() %>.<%= ste.getMethodName() %>(<%=
ste.getFileName() %>:<%= (ste.isNativeMethod() ? "native" :
ste.getLineNumber()) %>)
<%
}
%>
</stack-trace>
</error>
<%
t = t.getCause();
}
}
finally
{
try { if(null != rs) rs.close(); } catch (SQLException sqle)
{ sqle.printStackTrace(); }
try { if(null != ps) ps.close(); } catch (SQLException sqle)
{ sqle.printStackTrace(); }
try { if(null != conn) conn.close(); } catch (SQLException sqle)
{ sqle.printStackTrace(); }
}
%>
<vm-info vendor="<%=
escapeQuotes(System.getProperty("java.vm.vendor")) %>" name="<%=
escapeQuotes(System.getProperty("java.vm.name")) %>" version="<%=
escapeQuotes(System.getProperty("java.vm.version")) %>" info="<%=
escapeQuotes(System.getProperty("java.vm.info")) %>"/>
<memory-info max="<%= maxMemory %>" total="<%= totalMemory %>"
free="<%= freeMemory %>" />
<jdbc-datasource-info maxActive="<%= maxActiveConnections %>"
active="<%= activeConnections %>" maxIdle="<%= maxIdleConnections %>"
minIdle="<%= minIdleConnections %>" idle="<%= idleConnections %>"
okay="<%= connectionOkay %>" checkoutTime="<%= checkoutDelay %>" />
</health-report>
health-report-memory-parser.pl
- ------------------------------
#!/usr/bin/perl
#
# Parses output from the HealthReport.jsp script to generate
# a single, tab-separated line of text showing the current time
# (in seconds) and the max, total, and free memory (in bytes).
#
# This can then be used to easily graph memory usage in a running
# Java VM.
#
my($timestamp);
my($maxMem);
my($totalMem);
my($freeMem);
while(<>) {
if ( /<health-report.*time="([0-9]+)"/ ) {
$timestamp = $1;
# Convert timestamp from milliseconds to seconds
$timestamp = int($timestamp / 1000);
}
elsif ( /<memory-info/ ) {
($maxMem) = /max="([0-9]+)"/;
($totalMem) = /total="([0-9]+)"/;
($freeMem) = /free="([0-9]+)"/;
}
elsif ( /<error/ ) {
$maxMem = "ERROR";
}
}
Then, write a script like this and run it every minute or so:
#!/bin/sh
wget -qO http://host/path/to/HealthCheck.jsp \
| health-report-memory-parser.pl \
>> memory-usage.txt
Graph that baby using your favorite tool and see if the memory usage
graph looks good. It should look like /\/\/\/\/\/\/\/\/\/\/\/\/\/\
and not like
/
/
/
/\/
/
/
/\/
/
/
/
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAkn7J1gACgkQ9CaO5/Lv0PA5HQCeILlscpfNZXs8VXnG8xdPznXp
vFEAnjkRwD5G/LEIofWhJMaFmG+ZtY1b
=LYyX
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: GC Problem
Posted by Pid <p...@pidster.com>.
S Arvind wrote:
> Is there any good settings for GC for tomcat running in server with 4GB and
> Quad Core processor. I bascially need GC parameter for RESOURCE HUNGRY WEB
> APPLICATION
Why not try fixing the memory leak instead?
Reconfiguring the JVM params, memory or GC will only delay the
inevitable OutOfMemory error.
The "YourKit" profiler is frequently mentioned on this list, but there
are others.
p
> -Arvind S
>
>
> *
> "Many of lifes failure are people who did not realize how close they were to
> success when they gave up."
> -Thomas Edison
> *
>
> On Wed, Apr 29, 2009 at 6:19 PM, Peter Crowther <Peter.Crowther@melandra.com
>> wrote:
>
>>> From: S Arvind [mailto:arvindwill@gmail.com]
>>> Tomcat after running for couple of hours , somthing is
>>> getting wrong and it
>>> is keep on doing Garbage Collection(as it dispaly in console)
>>> . And tomcat
>>> also becoming dump slow.. and we cant able to do any, until
>>> it restarts...
>> I suspect your application has a memory leak somewhere - and it is almost
>> always the application, not Tomcat :-). Use a profiler to find out where
>> all the memory is going.
>>
>> Do you get out-of-memory errors in the Tomcat log files if you leave the
>> application running even once it has become slow?
>>
>> - Peter
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: GC Problem
Posted by Peter Crowther <Pe...@melandra.com>.
> From: S Arvind [mailto:arvindwill@gmail.com]
> Is there any good settings for GC for tomcat running in
> server with 4GB and Quad Core processor.
"As much RAM as the OS doesn't need for other purposes". There's no straiht answer for this, as it depends what else is running on the box. You might try initial and maximum heap sizes of 3 Gbytes, but that's a finger-in-the-air guess. Try it; measure it; and adjust it if your application spends a lot of time collecting garbage, or if the OS starts using its swap file.
> I bascially need GC parameter for
> RESOURCE HUNGRY WEB APPLICATION
I'm going to make a prediction: it doesn't matter how much RAM you allocate, you won't have enough. You will still run out of memory and have to restart Tomcat. I think this, because I think you have a memory leak in your application.
- Peter
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: GC Problem
Posted by S Arvind <ar...@gmail.com>.
Is there any good settings for GC for tomcat running in server with 4GB and
Quad Core processor. I bascially need GC parameter for RESOURCE HUNGRY WEB
APPLICATION
-Arvind S
*
"Many of lifes failure are people who did not realize how close they were to
success when they gave up."
-Thomas Edison
*
On Wed, Apr 29, 2009 at 6:19 PM, Peter Crowther <Peter.Crowther@melandra.com
> wrote:
> > From: S Arvind [mailto:arvindwill@gmail.com]
> > Tomcat after running for couple of hours , somthing is
> > getting wrong and it
> > is keep on doing Garbage Collection(as it dispaly in console)
> > . And tomcat
> > also becoming dump slow.. and we cant able to do any, until
> > it restarts...
>
> I suspect your application has a memory leak somewhere - and it is almost
> always the application, not Tomcat :-). Use a profiler to find out where
> all the memory is going.
>
> Do you get out-of-memory errors in the Tomcat log files if you leave the
> application running even once it has become slow?
>
> - Peter
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
RE: GC Problem
Posted by Peter Crowther <Pe...@melandra.com>.
> From: S Arvind [mailto:arvindwill@gmail.com]
> Yeah daily atleast once Out-of-memory will raise and we will
> restart the tomcat.
As the amount of free memory goes down, the JVM will have to collect garbage more and more frequently, until you get the symptoms you see where it is spending most of its time in GC.
You can prove it's a memory leak by increasing the amount of RAM available to the JVM. If you still run out of memory - but a little later - it's almost certainly a leak.
- Peter
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: GC Problem
Posted by Juha Laiho <Ju...@iki.fi>.
S Arvind wrote:
> Yeah daily atleast once Out-of-memory will raise and we will restart the
> tomcat.
As others have already said, this has all the symptoms of a memory leak.
It could be Tomcat, but more likely is your application. The continuous
GC is a sign of the Java VM running out of memory assigned for it, and
desperately attempting to find something to throw away.
In an ideal case you'll have almost linear correlation of amount of
Java heap memory assigned and amount of time your application remains
working properly.
And I second the notion that YourKit is a good tool for finding out which
objects are eating up the memory (and also, what is the reference chain
through different objects to the ones accumulating -- so you'll have
easier time to find out where the references could and should be cleaned up).
--
..Juha
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: GC Problem
Posted by S Arvind <ar...@gmail.com>.
Yeah daily atleast once Out-of-memory will raise and we will restart the
tomcat.
Arvind S
*
"Many of lifes failure are people who did not realize how close they were to
success when they gave up."
-Thomas Edison*
On Wed, Apr 29, 2009 at 6:19 PM, Peter Crowther <Peter.Crowther@melandra.com
> wrote:
> > From: S Arvind [mailto:arvindwill@gmail.com]
> > Tomcat after running for couple of hours , somthing is
> > getting wrong and it
> > is keep on doing Garbage Collection(as it dispaly in console)
> > . And tomcat
> > also becoming dump slow.. and we cant able to do any, until
> > it restarts...
>
> I suspect your application has a memory leak somewhere - and it is almost
> always the application, not Tomcat :-). Use a profiler to find out where
> all the memory is going.
>
> Do you get out-of-memory errors in the Tomcat log files if you leave the
> application running even once it has become slow?
>
> - Peter
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
RE: GC Problem
Posted by Peter Crowther <Pe...@melandra.com>.
> From: S Arvind [mailto:arvindwill@gmail.com]
> Tomcat after running for couple of hours , somthing is
> getting wrong and it
> is keep on doing Garbage Collection(as it dispaly in console)
> . And tomcat
> also becoming dump slow.. and we cant able to do any, until
> it restarts...
I suspect your application has a memory leak somewhere - and it is almost always the application, not Tomcat :-). Use a profiler to find out where all the memory is going.
Do you get out-of-memory errors in the Tomcat log files if you leave the application running even once it has become slow?
- Peter
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org