You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by qi zhang <qi...@CS.WM.EDU> on 2003/12/06 17:24:33 UTC
Urgent! Tomcat4 closed giving Out of Memory Error
Hi,
I have stunk on this problem for almost 2 weeks. :-(
In these two weeks, I have tried many methods, but it still doesn't work.
I built a system to run TPCW benchmark. One web and application server,
with tomcat4.0 running, one database with mysql4.0 running on it and
several client machines. The Jave version is j2sdk1.4.0_03. The system
worked perfect under light load. But when the number of clients
exceeded some number (like 600), the tomcat4 didn't work anymore.
I checked catalina.out. Sometimes no error message appear, while in some
time it appeared the error:
SEVERE: Caught exception executing
org.apache.tomcat.util.net.TcpWorkerThread@a2f02e, terminating thread
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start(Native Method)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.<init>(ThreadPool.java:497)
at
org.apache.tomcat.util.threads.ThreadPool.openThreads(ThreadPool.java:387)
at
org.apache.tomcat.util.threads.ThreadPool.runIt(ThreadPool.java:226)
at
org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:503)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:530)
at java.lang.Thread.run(Thread.java:536)
But my tomcat4 is running with option -server -Xmx1536M. (I also tried to
change -Xss and -Xms, which didn't help.) And the actual memory
utilization was less than 40%. (I have 2GB memory in this machine.)
I also changed 'ulimit -n 8192' and server.xml :
<!-- Define a non-SSL Coyote HTTP/1.1 Connector on port 8080 -->
<Connector className="org.apache.coyote.tomcat4.CoyoteConnector"
port="8080" minProcessors="5" maxProcessors="1750"
enableLookups="true" redirectPort="8443"
acceptCount="1000" debug="0" connectionTimeout="-1"
useURIValidationHack="false" disableUploadTimeout="true" />
<!-- Note : To disable connection timeouts, set connectionTimeout
value to -1 -->
Why this error happened? I was trying to use strace -f -p to trace the
system call, but it only gave me one line "accept(13, " and waited at that
point. Did anyone meet this problem before? Can I get some detailed error
log besided catalina.out?
I really appreciate your kind help.
Qi
--
Qi Zhang, Ph.D Student
Department of Computer Science,
College of William and Mary
P.O. Box 8795
Williamsburg, VA 23187-8795
---------------------------------------------------------------------
To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: tomcat-user-help@jakarta.apache.org
Re: Urgent! Tomcat4 closed giving Out of Memory Error
Posted by qi zhang <qi...@CS.WM.EDU>.
I got "Open too many files" error when I set the max open files 4096, so
that I changed it to 9182. But I think no matter how large it is, it won't
affect the performace, right?
I'm not good at Java. But I don't think I reloaded the classes in Tomcat
manager. I didn't change web.xml. Is it the problem?
Now the problem is more and more confusing. I used "-verbosegc" to log GC
behavior and the memory usage. If I set CATALINA_OPTS="-server -Xmx1200M
-Xms1200M -Xss256k", I got OutOfMemory error quickly as
################################
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start(Native Method)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.<init>(ThreadPool.java:497)
at
org.apache.tomcat.util.threads.ThreadPool.openThreads(ThreadPool.java:387)
at
org.apache.tomcat.util.threads.ThreadPool.runIt(ThreadPool.java:226)
at
org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:503)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:530)
at java.lang.Thread.run(Thread.java:536)
[GC 338409K->282510K(1215168K), 0.3190985 secs]
################################
While if I set CATALINA_OPTS="-server -Xmx768M -Xms768M -Xss256k", I didn't
get any error message in catalina.out. But tomcat stopped, and I got "Time
Out" in browser. The final line in catalina.out was
########################################
[GC 412489K->360388K(777728K), 0.1470509 secs]
########################################
Then I stopped tomcat4, and got the following error message in catalina.out.
#########################################
Java HotSpot(TM) Server VM warning: Attempt to unguard stack red zone failed.
An irrecoverable stack overflow has occurred.
An unexpected exception has been detected in native code outside the VM.
Unexpected Signal : 11 occurred at PC=0x40039203
Function=(null)+0x40039203
Library=/lib/libpthread.so.0
NOTE: We are unable to locate the function name symbol for the error
just occurred. Please refer to release documentation for possible
reason and solutions.
#########################################
I'm wondering if it is not good to set too large heap file size, which may
limit other required space. I am really confused about why tomcat stopped
without any error message. How can I find the error? Did anybody meet this
problem before?
Thank you very much.
Qi
>Look at the open files - it is 9182. That IS already pretty high, but
>you may be hitting that limit since in Unix, everything is a file.
>Sockets, etc.
>On Linux, is there a way in the JVM to tell the difference between
>memory given to the new generation, old generation, and permanent
>generation in the JVM? There were users on this list posting weird
>OutOfMemory errors last week that were doing class reloading, and the
>permanent generation was filling up even though there was plenty of
>space in the old and new gen. Of course, I think these comments only
>apply to a 1.4.x JVM. Does your code do funky stuff with the
>classloaders, or are you reloading your application in Tomcat's manager?
> (or changing web.xml, which would cause reloads??)
>Have you profiled your application to find any memory leaks?
> I would hunt the archives, I did a search for outofmemoryerror native
> and saw this - I am sure there are more similar threads with this
> issue:
>
> http://marc.theaimsgroup.com/?l=tomcat-user&m=105535455830487&w=2
>
> If _POSIX_THREAD_THREADS_MAX=64 - then you would have problems since
> that means tomcat can only have 64 threads at one time. Tomcat is a
> single process living under the JVM. (But then again - this is Linux
> and the process vs thread model is sometimes a little strange)
>
> (I don't use tomcat on linux too much so my trouble shooting skills
> there are iffy)
>
>
> -Tim
>
> qi zhang wrote:
>
>> I am using Linux2.4.20-smp. I think the limits for the entire system
>> are not problem:
>>
>>>cat /proc/sys/kernel/threads-max
>>
>> 14336
>>
>>>cat /proc/sys/fs/file-max
>>
>> 209708
>>
>>>cat /proc/sys/fs/file-nr
>>
>> 2231 1366 209708
>>
>>>ulimit -a
>>
>> core file size (blocks, -c) 0
>> data seg size (kbytes, -d) unlimited
>> file size (blocks, -f) unlimited
>> max locked memory (kbytes, -l) unlimited
>> max memory size (kbytes, -m) unlimited
>> open files (-n) 9182
>> pipe size (512 bytes, -p) 8
>> stack size (kbytes, -s) 8192
>> cpu time (seconds, -t) unlimited
>> max user processes (-u) 7168
>> virtual memory (kbytes, -v) unlimited
>>
>> But I can't find where the limits for each process/user are. One
>> message found from google said PTHREAD_THREADS_MAX could be the max
>> thread count per process. In my /usr/include/bits/local_lim.h, I have
>>
>> /* The number of threads per process. */
>> #define _POSIX_THREAD_THREADS_MAX 64
>> /* This is the value this implementation supports. */
>> #define PTHREAD_THREADS_MAX 16384
>>
>> 16384 is large enough, I believe. Then what could be the reason of the
>> error?
>>
>> By the way, the "Out of Memory" error didn't appear all the time.
>> Sometimes there was just no response without any error message. Do you
>> know where I can get more detailed bug files?
>>
>> Thank you very much.
>>
>> Qi
>>
>>
>> On Sat, 6 Dec 2003, Tim Funk wrote:
>>
>>
>>>When you get "java.lang.OutOfMemoryError: unable to create new native
>>> thread" it means you are running into an operating system limit. You
>>> can give the JVM 1 Exabyte of memory and you'll still get this error.
>>>
>>>Look at the kernel parameters for your machine and the following
>>> limits: 1) Threads allowable on the system
>>>2) Threads run by a user
>>>3) Threads run by a process
>>>4) File handles per process
>>>5) File handles per user
>>>
>>>Use your OS manual or Google to tweak these parameters.
>>>
>>>http://jakarta.apache.org/tomcat/faq/memory.html#why
>>>
>>>-Tim
>>>
>>>qi zhang wrote:
>>>
>>>>Hi,
>>>>
>>>>I have stunk on this problem for almost 2 weeks. :-(
>>>>In these two weeks, I have tried many methods, but it still doesn't
>>>> work.
>>>>
>>>>I built a system to run TPCW benchmark. One web and application
>>>> server, with tomcat4.0 running, one database with mysql4.0 running
>>>> on it and several client machines. The Jave version is
>>>> j2sdk1.4.0_03. The system worked perfect under light load. But when
>>>> the number of clients exceeded some number (like 600), the tomcat4
>>>> didn't work anymore. I checked catalina.out. Sometimes no error
>>>> message appear, while in some time it appeared the error:
>>>>
>>>>SEVERE: Caught exception executing
>>>>org.apache.tomcat.util.net.TcpWorkerThread@a2f02e, terminating thread
>>>> java.lang.OutOfMemoryError: unable to create new native thread
>>>> at java.lang.Thread.start(Native Method)
>>>> at
>>>>org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.<init>(ThreadPool.java:497)
>>>> at
>>>
>
>
> --------------------------------------------------------------------- To
> unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org For
> additional commands, e-mail: tomcat-user-help@jakarta.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: tomcat-user-help@jakarta.apache.org
Re: Urgent! Tomcat4 closed giving Out of Memory Error
Posted by Tim Funk <fu...@joedog.org>.
I would hunt the archives, I did a search for outofmemoryerror native and saw
this - I am sure there are more similar threads with this issue:
http://marc.theaimsgroup.com/?l=tomcat-user&m=105535455830487&w=2
If _POSIX_THREAD_THREADS_MAX=64 - then you would have problems since that
means tomcat can only have 64 threads at one time. Tomcat is a single process
living under the JVM. (But then again - this is Linux and the process vs
thread model is sometimes a little strange)
(I don't use tomcat on linux too much so my trouble shooting skills there are
iffy)
-Tim
qi zhang wrote:
> I am using Linux2.4.20-smp. I think the limits for the entire system are
> not problem:
>
>>cat /proc/sys/kernel/threads-max
>
> 14336
>
>>cat /proc/sys/fs/file-max
>
> 209708
>
>>cat /proc/sys/fs/file-nr
>
> 2231 1366 209708
>
>>ulimit -a
>
> core file size (blocks, -c) 0
> data seg size (kbytes, -d) unlimited
> file size (blocks, -f) unlimited
> max locked memory (kbytes, -l) unlimited
> max memory size (kbytes, -m) unlimited
> open files (-n) 9182
> pipe size (512 bytes, -p) 8
> stack size (kbytes, -s) 8192
> cpu time (seconds, -t) unlimited
> max user processes (-u) 7168
> virtual memory (kbytes, -v) unlimited
>
> But I can't find where the limits for each process/user are. One message
> found from google said PTHREAD_THREADS_MAX could be the max thread count
> per process. In my /usr/include/bits/local_lim.h, I have
>
> /* The number of threads per process. */
> #define _POSIX_THREAD_THREADS_MAX 64
> /* This is the value this implementation supports. */
> #define PTHREAD_THREADS_MAX 16384
>
> 16384 is large enough, I believe. Then what could be the reason of the
> error?
>
> By the way, the "Out of Memory" error didn't appear all the time.
> Sometimes there was just no response without any error message. Do you
> know where I can get more detailed bug files?
>
> Thank you very much.
>
> Qi
>
>
> On Sat, 6 Dec 2003, Tim Funk wrote:
>
>
>>When you get "java.lang.OutOfMemoryError: unable to create new native thread"
>>it means you are running into an operating system limit. You can give the JVM
>>1 Exabyte of memory and you'll still get this error.
>>
>>Look at the kernel parameters for your machine and the following limits:
>>1) Threads allowable on the system
>>2) Threads run by a user
>>3) Threads run by a process
>>4) File handles per process
>>5) File handles per user
>>
>>Use your OS manual or Google to tweak these parameters.
>>
>>http://jakarta.apache.org/tomcat/faq/memory.html#why
>>
>>-Tim
>>
>>qi zhang wrote:
>>
>>>Hi,
>>>
>>>I have stunk on this problem for almost 2 weeks. :-(
>>>In these two weeks, I have tried many methods, but it still doesn't work.
>>>
>>>I built a system to run TPCW benchmark. One web and application server,
>>>with tomcat4.0 running, one database with mysql4.0 running on it and
>>>several client machines. The Jave version is j2sdk1.4.0_03. The system
>>>worked perfect under light load. But when the number of clients
>>>exceeded some number (like 600), the tomcat4 didn't work anymore.
>>>I checked catalina.out. Sometimes no error message appear, while in some
>>>time it appeared the error:
>>>
>>>SEVERE: Caught exception executing
>>>org.apache.tomcat.util.net.TcpWorkerThread@a2f02e, terminating thread
>>>java.lang.OutOfMemoryError: unable to create new native thread
>>> at java.lang.Thread.start(Native Method)
>>> at
>>>org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.<init>(ThreadPool.java:497)
>>> at
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: tomcat-user-help@jakarta.apache.org
Re: Urgent! Tomcat4 closed giving Out of Memory Error
Posted by qi zhang <qi...@CS.WM.EDU>.
I am using Linux2.4.20-smp. I think the limits for the entire system are
not problem:
>cat /proc/sys/kernel/threads-max
14336
>cat /proc/sys/fs/file-max
209708
>cat /proc/sys/fs/file-nr
2231 1366 209708
>ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 9182
pipe size (512 bytes, -p) 8
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 7168
virtual memory (kbytes, -v) unlimited
But I can't find where the limits for each process/user are. One message
found from google said PTHREAD_THREADS_MAX could be the max thread count
per process. In my /usr/include/bits/local_lim.h, I have
/* The number of threads per process. */
#define _POSIX_THREAD_THREADS_MAX 64
/* This is the value this implementation supports. */
#define PTHREAD_THREADS_MAX 16384
16384 is large enough, I believe. Then what could be the reason of the
error?
By the way, the "Out of Memory" error didn't appear all the time.
Sometimes there was just no response without any error message. Do you
know where I can get more detailed bug files?
Thank you very much.
Qi
On Sat, 6 Dec 2003, Tim Funk wrote:
> When you get "java.lang.OutOfMemoryError: unable to create new native thread"
> it means you are running into an operating system limit. You can give the JVM
> 1 Exabyte of memory and you'll still get this error.
>
> Look at the kernel parameters for your machine and the following limits:
> 1) Threads allowable on the system
> 2) Threads run by a user
> 3) Threads run by a process
> 4) File handles per process
> 5) File handles per user
>
> Use your OS manual or Google to tweak these parameters.
>
> http://jakarta.apache.org/tomcat/faq/memory.html#why
>
> -Tim
>
> qi zhang wrote:
> > Hi,
> >
> > I have stunk on this problem for almost 2 weeks. :-(
> > In these two weeks, I have tried many methods, but it still doesn't work.
> >
> > I built a system to run TPCW benchmark. One web and application server,
> > with tomcat4.0 running, one database with mysql4.0 running on it and
> > several client machines. The Jave version is j2sdk1.4.0_03. The system
> > worked perfect under light load. But when the number of clients
> > exceeded some number (like 600), the tomcat4 didn't work anymore.
> > I checked catalina.out. Sometimes no error message appear, while in some
> > time it appeared the error:
> >
> > SEVERE: Caught exception executing
> > org.apache.tomcat.util.net.TcpWorkerThread@a2f02e, terminating thread
> > java.lang.OutOfMemoryError: unable to create new native thread
> > at java.lang.Thread.start(Native Method)
> > at
> > org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.<init>(ThreadPool.java:497)
> > at
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: tomcat-user-help@jakarta.apache.org
>
--
Qi Zhang, Ph.D Student
Department of Computer Science,
College of William and Mary
P.O. Box 8795
Williamsburg, VA 23187-8795
---------------------------------------------------------------------
To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: tomcat-user-help@jakarta.apache.org
Re: Urgent! Tomcat4 closed giving Out of Memory Error
Posted by Tim Funk <fu...@joedog.org>.
When you get "java.lang.OutOfMemoryError: unable to create new native thread"
it means you are running into an operating system limit. You can give the JVM
1 Exabyte of memory and you'll still get this error.
Look at the kernel parameters for your machine and the following limits:
1) Threads allowable on the system
2) Threads run by a user
3) Threads run by a process
4) File handles per process
5) File handles per user
Use your OS manual or Google to tweak these parameters.
http://jakarta.apache.org/tomcat/faq/memory.html#why
-Tim
qi zhang wrote:
> Hi,
>
> I have stunk on this problem for almost 2 weeks. :-(
> In these two weeks, I have tried many methods, but it still doesn't work.
>
> I built a system to run TPCW benchmark. One web and application server,
> with tomcat4.0 running, one database with mysql4.0 running on it and
> several client machines. The Jave version is j2sdk1.4.0_03. The system
> worked perfect under light load. But when the number of clients
> exceeded some number (like 600), the tomcat4 didn't work anymore.
> I checked catalina.out. Sometimes no error message appear, while in some
> time it appeared the error:
>
> SEVERE: Caught exception executing
> org.apache.tomcat.util.net.TcpWorkerThread@a2f02e, terminating thread
> java.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start(Native Method)
> at
> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.<init>(ThreadPool.java:497)
> at
---------------------------------------------------------------------
To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: tomcat-user-help@jakarta.apache.org