You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Greg Huber <gr...@gmail.com> on 2017/11/10 09:31:00 UTC
Service killed by signal 9
Since switching to jsvc, randomly I am getting tomcat restarting, looking
at the logs I see that the jsvc is using alot of memory total-vm: and
being killed by the system.
Nov 9 13:11:11 prodbox kernel: Out of memory: Kill process 1287 (jsvc)
score 121 or sacrifice child
Nov 9 13:11:11 prodbox kernel: Killed process 1287 (jsvc)
total-vm:3453120kB, anon-rss:378280kB, file-rss:0kB, shmem-rss:0kB
I am using the below on properties on the startup (from previous version
statup scripts):
-Xms256M -Xmx768m -Xss1280k -XX:+UseParallelGC -XX:MaxGCPauseMillis=1500
-XX:GCTimeRatio=9 -server -XX:+DisableExplicitGC
(I replaced -Xss256k with -Xss1280k to stop crashing see
https://issues.apache.org/jira/browse/DAEMON-365)
Is there a way to limit the total-vm or find out why its got so big?
-XX:+DisableExplicitGC ??
Standard tomcat install with apache/modjk.
Cheers Greg
Re: Service killed by signal 9
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Konstantin,
On 11/10/17 8:46 AM, Konstantin Kolinko wrote:
> 2017-11-10 12:31 GMT+03:00 Greg Huber <gr...@gmail.com>:
>> Since switching to jsvc, randomly I am getting tomcat restarting,
>> looking at the logs I see that the jsvc is using alot of memory
>> total-vm: and being killed by the system.
>>
>> Nov 9 13:11:11 prodbox kernel: Out of memory: Kill process 1287
>> (jsvc) score 121 or sacrifice child Nov 9 13:11:11 prodbox
>> kernel: Killed process 1287 (jsvc) total-vm:3453120kB,
>> anon-rss:378280kB, file-rss:0kB, shmem-rss:0kB
>>
>> I am using the below on properties on the startup (from previous
>> version statup scripts):
>>
>> -Xms256M -Xmx768m -Xss1280k -XX:+UseParallelGC
>> -XX:MaxGCPauseMillis=1500 -XX:GCTimeRatio=9 -server
>> -XX:+DisableExplicitGC
>>
>> (I replaced -Xss256k with -Xss1280k to stop crashing see
>> https://issues.apache.org/jira/browse/DAEMON-365)
>
> The links in that issue go to a thread that explains that the issue
> is actually caused a kernel bug. It was reported & diagnosed in
> June 2017, thus should it have already been fixed?
>
> 1,2 Mb of stack for each thread mean that for 200 threads you will
> need 240 Mb of memory just for thread stacks. Most of that is
> wasted memory: You do not need to increase -Xss unless you are
> hitting StackOverflowError.
>
>> Is there a way to limit the total-vm or find out why its got so
>> big? -XX:+DisableExplicitGC ??
>>
>> Standard tomcat install with apache/modjk.
>
> https://wiki.apache.org/tomcat/FAQ/Linux_Unix#Q5
>
> In Tomcat Manager web application there is "Server Status" page.
> It has a table that shows current memory consumption, as reported
> by Java. The values can also be queried via JMX (e.g. with
> jconsole application).
Linux oom will use the actual process's used memory to compute its
killability, so the Java-reported memory usage won't tell the whole stor
y.
Greg, how much total memory does the server actually have? Do you have
other services running on it as well? If you run low on memory, Linux
will simply kill your process. There are ways to minimize the
likelihood that your process will be killed, but if it's the only way
to free-up memory that the kernel is convinced it needs, well, then
your process is simply going to die.
One way to reduce your memory usage would be to reduce the heap size
you are using. Do you need that much heap? It's also possible that you
have a native memory leak somewhere, possibly caused by code outside
of your control. Do you have any native components involved that don't
ship with the JVM (or jsvc)? I'm unaware of any memory issues with
jsvc, but I'm not super familiar with it.
- -chris
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAloFvOIACgkQHPApP6U8
pFhPdBAAyvgvwEovYQXGpDDVZ2c3JwkzyKjQrWboW9wYjKsPTjy+xGiDqVfhxCLC
ukbQL6spVYlqc1y5N+5i2iyqX+JRKa7OA3unvFRDrpxJDOfJmDqwBavJ8xlpl9mX
v/E/WMPXAgZLYK05aPCsul7/ewavahADi2vwt2SQgUE9ARrV/qN7QXYPaa6XwNjM
SSb35dWK0k9pDCl2OrKfFOEgm3nsY+P6FKx29k1z/8sAYBYQJRC/slBboqsj7uzj
X4DFThtKvkiPGRSjzEeDmkthMN7CFz1QFHa3NaPDkj5Zjv7mUfLZRdHB0ETlQdvq
Zx2md/OZBw/EoE5j+K/C9yfK6qwcdgHX2zsHdwbweC/QZJrFVaf9VxfaXmeZavLX
KXYRhk3W1R93/QigXVjPxMod1ToiLjSzGPG34F1jkC/FBj7X359dZokL6pLl6hDw
h+6M5rUxWevLVdKg9gWX8peT/jxk4P7KInWjdH2oI4Rl40vMbSPA23qWkaVTbHfG
WWsdDMWrmxNW2e+zzNRCikjMT6ksWas18tzgn9KVWy8cau8yy208d0opSUloUZht
2LwUhrEKmbZjQywyyq/7MyXTIwMeAg6y3xJ04al6ZBp7YnrmW/zhay4yaT2ZBJ8R
E1PnVf1MPnOAIHIev562CadG/FiveQ0n94/7KMiQf81YgIiFQeo=
=AXyl
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Service killed by signal 9
Posted by Konstantin Kolinko <kn...@gmail.com>.
2017-11-10 12:31 GMT+03:00 Greg Huber <gr...@gmail.com>:
> Since switching to jsvc, randomly I am getting tomcat restarting, looking
> at the logs I see that the jsvc is using alot of memory total-vm: and
> being killed by the system.
>
> Nov 9 13:11:11 prodbox kernel: Out of memory: Kill process 1287 (jsvc)
> score 121 or sacrifice child
> Nov 9 13:11:11 prodbox kernel: Killed process 1287 (jsvc)
> total-vm:3453120kB, anon-rss:378280kB, file-rss:0kB, shmem-rss:0kB
>
> I am using the below on properties on the startup (from previous version
> statup scripts):
>
> -Xms256M -Xmx768m -Xss1280k -XX:+UseParallelGC -XX:MaxGCPauseMillis=1500
> -XX:GCTimeRatio=9 -server -XX:+DisableExplicitGC
>
> (I replaced -Xss256k with -Xss1280k to stop crashing see
> https://issues.apache.org/jira/browse/DAEMON-365)
The links in that issue go to a thread that explains that the issue is
actually caused a kernel bug.
It was reported & diagnosed in June 2017, thus should it have already
been fixed?
1,2 Mb of stack for each thread mean that for 200 threads you will
need 240 Mb of memory just for thread stacks. Most of that is wasted
memory: You do not need to increase -Xss unless you are hitting
StackOverflowError.
> Is there a way to limit the total-vm or find out why its got so big?
> -XX:+DisableExplicitGC ??
>
> Standard tomcat install with apache/modjk.
https://wiki.apache.org/tomcat/FAQ/Linux_Unix#Q5
In Tomcat Manager web application there is "Server Status" page. It
has a table that shows current memory consumption, as reported by
Java. The values can also be queried via JMX (e.g. with jconsole
application).
Best regards,
Konstantin Kolinko
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Service killed by signal 9
Posted by Greg Huber <gr...@gmail.com>.
Coty,
I will try and set it back to 256k, I am using centos 7 on the latest
kernel.
It is an older PC and has 4gb of memory but only reports 3gb.
I can see from top that the memory is not good (gkreallm reports 800mb free)
top - 08:35:40 up 2:25, 1 user, load average: 0.50, 0.18, 0.14
Tasks: 258 total, 1 running, 257 sleeping, 0 stopped, 0 zombie
%Cpu(s): 13.4 us, 2.2 sy, 0.0 ni, 84.1 id, 0.2 wa, 0.0 hi, 0.2 si,
0.0 st
KiB Mem : 3036956 total, 236104 free, 2216104 used, 584748 buff/cache
KiB Swap: 2113532 total, 1912644 free, 200888 used. 516460 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1292 user 20 0 3435632 476088 6560 S 0.3 15.7
4:43.96 jsvc
Cheers Greg
On 10 November 2017 at 15:06, Coty Sutherland <cs...@redhat.com> wrote:
> On Fri, Nov 10, 2017 at 4:31 AM, Greg Huber <gr...@gmail.com> wrote:
> > Since switching to jsvc, randomly I am getting tomcat restarting, looking
> > at the logs I see that the jsvc is using alot of memory total-vm: and
> > being killed by the system.
> >
> > Nov 9 13:11:11 prodbox kernel: Out of memory: Kill process 1287 (jsvc)
> > score 121 or sacrifice child
> > Nov 9 13:11:11 prodbox kernel: Killed process 1287 (jsvc)
> > total-vm:3453120kB, anon-rss:378280kB, file-rss:0kB, shmem-rss:0kB
> >
> > I am using the below on properties on the startup (from previous version
> > statup scripts):
> >
> > -Xms256M -Xmx768m -Xss1280k -XX:+UseParallelGC -XX:MaxGCPauseMillis=1500
> > -XX:GCTimeRatio=9 -server -XX:+DisableExplicitGC
> >
> > (I replaced -Xss256k with -Xss1280k to stop crashing see
> > https://issues.apache.org/jira/browse/DAEMON-365)
>
> The root cause for that broke a lot of things, so I imagine your OS
> has fixed the kernel problem by now. If you tell me which OS I can try
> and see if it's fixed for you. I'd suggest that you try reducing your
> Xss back to 256k and see if that shrinks your process size enough to
> keep it off of oom-killer's radar.
>
> > Is there a way to limit the total-vm or find out why its got so big?
> > -XX:+DisableExplicitGC ??
> >
> > Standard tomcat install with apache/modjk.
> >
> > Cheers Greg
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: Service killed by signal 9
Posted by Coty Sutherland <cs...@redhat.com>.
On Fri, Nov 10, 2017 at 4:31 AM, Greg Huber <gr...@gmail.com> wrote:
> Since switching to jsvc, randomly I am getting tomcat restarting, looking
> at the logs I see that the jsvc is using alot of memory total-vm: and
> being killed by the system.
>
> Nov 9 13:11:11 prodbox kernel: Out of memory: Kill process 1287 (jsvc)
> score 121 or sacrifice child
> Nov 9 13:11:11 prodbox kernel: Killed process 1287 (jsvc)
> total-vm:3453120kB, anon-rss:378280kB, file-rss:0kB, shmem-rss:0kB
>
> I am using the below on properties on the startup (from previous version
> statup scripts):
>
> -Xms256M -Xmx768m -Xss1280k -XX:+UseParallelGC -XX:MaxGCPauseMillis=1500
> -XX:GCTimeRatio=9 -server -XX:+DisableExplicitGC
>
> (I replaced -Xss256k with -Xss1280k to stop crashing see
> https://issues.apache.org/jira/browse/DAEMON-365)
The root cause for that broke a lot of things, so I imagine your OS
has fixed the kernel problem by now. If you tell me which OS I can try
and see if it's fixed for you. I'd suggest that you try reducing your
Xss back to 256k and see if that shrinks your process size enough to
keep it off of oom-killer's radar.
> Is there a way to limit the total-vm or find out why its got so big?
> -XX:+DisableExplicitGC ??
>
> Standard tomcat install with apache/modjk.
>
> Cheers Greg
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org