Re: JSP Page "hangs"... clues? (SOLVED)

[Jordan Michaels <jo...@viviotech.net>]

For the sake of record-keeping, I believe we found the fix for this.

The client got in touch with us today stating that the entire site was 
now unresponsive. We reviewed the log files and miraculously this time 
there WAS an error message (lots of them).

The error was as follows:
-----------
SEVERE: Error allocating socket processor
java.lang.OutOfMemoryError: unable to create new native thread
-----------

Googling this led me to the following:
http://java-monitor.com/forum/showthread.php?t=570
and
http://jroller.com/rreyelts/date/20040909

So, looking at my setenv.sh values, it turns out that Xss256m is 
extremely massive. It means each time a new thread was created, 256 
*MEGS* would get allocated to that thread (even if it wasn't used). So 
we would look at top and see that there was plenty of free memory, but 
not realize that it had all been allocated to the threads. I reduced the 
value to 512k, and things are now working beautifully.

I can only guess at why there were no logs when this happened previously 
(maybe no memory left to spawn a new logging thread? no idea...), but 
the page that was hanging now runs very quickly even when I hammer my 
refresh button on it. =D

Thanks to those of you who tried to help with this issue. Hopefully this 
post can help someone in the future!

Warm regards,
Jordan Michaels
Vivio Technologies
http://www.viviotech.net/
Open BlueDragon Steering Committee
Railo Community Distributions


Jordan Michaels wrote:
> We have an interesting situation and I'm wondering if anyone has any 
> ideas on where we could look for possible solutions.
> 
> We have a JSP page that performs a large mysql query of product titles 
> (just the names). The list is then spit out on the page with drill-down 
> links. Pretty simple really. The problem is that just after a tomcat 
> restart, the first and occasionally the second and third requests will 
> process all the way just fine, but most subsequent requests will "hang" 
> on a random record. Occasionally one will go all the way through, but 
> most will hang.
> 
> Let me clarify, we restart tomcat, hit the page, and the page loads 
> fine. We hit refresh once or twice, and the page will fail to load 
> fully. The page will hang after a random product title and just stop 
> responding. You can click on any other page on the site, but this one 
> page with the large product listing query hangs in random spots.
> 
> We've studied this intently. The server this is running on is a CentOS 
> 4, 32-bit box with 1.5 GB RAM, Apache 2, and mod_jk.
> 
> Our first thought was memory issues. We increased Tomcat's memory using 
>  the setenv.sh file, which now contains the following line:
> 
> JAVA_OPTS="-Xms128m -Xmx1024m -Xss256m -XX:MaxPermSize=256m "; export 
> JAVA_OPTS
> 
> We've verified that the settings are in place by using the "ps ax" 
> command and making sure the java process includes the updated settings. 
> Unfortunately, this seems to have no effect at all.
> 
> We then reviewed the logs: catalina.out, mod_jk.log, and apache's
> error_log, among others. We monitored these logs using "tail" so we
> could see if any errors were thrown as we hit the page. No errors were
> thrown if the request went through good or it failed. The logs were 
> completely quiet.
> 
> We've updated Tomcat, mod_jk, the JVM, and the MySQL driver to the 
> absolute latest versions available as of this writing.
> 
> Does anyone have any clues as to where we might look for what is 
> happening to make this one page hang? It still feels like a memory 
> issue, but with nothing in the logs and no indicators other then the 
> problem itself, we'd appreciate any suggestions on what to look for at 
> this point.
> 
> Thank you in advance!
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

Re: JSP Page "hangs"... clues? (SOLVED)

[Christopher Schultz <ch...@christopherschultz.net>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Jordan,

On 2/15/2010 6:24 PM, Jordan Michaels wrote:
> The client got in touch with us today stating that the entire site was
> now unresponsive. We reviewed the log files and miraculously this time
> there WAS an error message (lots of them).
> 
> The error was as follows:
> -----------
> SEVERE: Error allocating socket processor
> java.lang.OutOfMemoryError: unable to create new native thread

That'll do it.

> So, looking at my setenv.sh values, it turns out that Xss256m is
> extremely massive.

Hah, yeah. I didn't notice that on first reading.

> I reduced the
> value to 512k, and things are now working beautifully.

I would leave the stack size to the default, unless you see a need to
increase it.

You might also want to review your nominal thread count in the JVM: it's
possible that you are getting close to the limit of threads for a
process in your environment.

> I can only guess at why there were no logs when this happened previously
> (maybe no memory left to spawn a new logging thread? no idea...), but
> the page that was hanging now runs very quickly even when I hammer my
> refresh button on it. =D

Try ApacheBench... it's a much better hammer.

Good luck,
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkt7FYYACgkQ9CaO5/Lv0PB8dQCcDNkl0DCH05Jic0IhnpTUHzKv
prIAnRN1eBhNiWYvqUgsDnfXbK8Ckevq
=3v5F
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org