You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cocoon.apache.org by Johan Cwiklinski <jo...@ajlsm.com> on 2010/12/22 08:51:51 UTC
Tomcat6/Cocoon 2.1.10 using 100% CPU on windows
Hello,
I have a problem with a cocoon 2.1.10 webapp running under tomcat 6.0.26
under windows 2003 server 64 bits with oracle's JDK 1.6.0_21.
This application is installed on a 'background' server, an application
on another server request it via AJP using apache mod_proxy_ajp.
For some reasons, the application will often eats 100% of the CPU, we
then need to kill and restart tomcat. Logs says absolutely nothing :(
I was not able yet to reproduce the issue on my dev environment.
This application mainly use some classes we've developed on the top of
cocoon that will:
- search for image file in some directories on different hard disks
(mainly by testing each directory + image path and looking if the file
exists),
- retrieve and show the image,
- additionally use ImageMagick to resize, rotate, etc.
The 'main' class extends cocoon's ResourceReader.
Using the jvisualvm tool provided with Oracle's JDK, I can observe that:
- ajp threads are sometimes running, and sometimes waiting; ok, that
seems normal,
- when the 100% cpu issue occurs, some ajp threads keeps running (never
get back to waiting state). At the beginning, only one or two threads
are affected, many more will be if we wait. I can also observe that some
threads (a few ones unfortunately) will still having the normal behavior.
All running threads using our class (ImageMagickReader) seems to be kind
of blocked on super.setup or super.generate methods:
"ajp-9009-9" - Thread t@65
java.lang.Thread.State: RUNNABLE
at java.util.HashMap.get(HashMap.java:303)
at
org.apache.cocoon.reading.ResourceReader.getLastModified(ResourceReader.java:242)
at
org.apache.cocoon.reading.ResourceReader.setupHeaders(ResourceReader.java:177)
at org.apache.cocoon.reading.ResourceReader.setup(ResourceReader.java:157)
at org.pleade.reading.ImageMagickReader.setup(ImageMagickReader.java:272)
[...]
Line 242 of ResourceReader.java is:
final String systemId = (String) documents.get(request.getRequestURI());
"ajp-9009-8" - Thread t@102
java.lang.Thread.State: RUNNABLE
at java.util.HashMap.transfer(HashMap.java:484)
at java.util.HashMap.resize(HashMap.java:463)
at java.util.HashMap.addEntry(HashMap.java:755)
at java.util.HashMap.put(HashMap.java:385)
at
org.apache.cocoon.reading.ResourceReader.generate(ResourceReader.java:346)
at
org.pleade.reading.ImageMagickReader.generate(ImageMagickReader.java:584)
[...]
Line 346 of ResourceReader.java is:
documents.put(request.getRequestURI(), inputSource.getURI());
Those two examples are based on the first threads that will never release.
I do not know if it is possible for a HashMap to be sort of corrupted ;
of maybe HTTP headers? I'm not sure even if what we're seeing is is the
cause or the consequence of the issue :(
The same issue has been observed in the past on another server which is
now running under GNU/Linux, and now seems to be ok (about two weeks
under Linux, and no longer 100% CPU!).
We've trying several tomcat and java versions, that changes anything.
The issue can occurs after several uptime hours, or only a few minutes!
If there are many connections, the issue will occurs more often; but is
still present with just a few connections.
I really do not know where the problem should be... Is it our code? Is
it cocoon? Is it tomcat? Or more probably something one of them is doing
that windows dislikes?
It's difficult to know when exactly the problem happens (we've asked
system administrators but get no answer) ; so I've not yet tried to log
in debug mode (well, I've tried it once, but this is really verbose...).
Any ideas? I do not know what to try or where to look at now :/
Many similar issues I can show over the web were related to a bug in
tcnative under tomcat 5.5 ; I guess that is resolved now, I did not
found any similar bug under tomcat 6. I also found a few ones with
tomcat 6, but some were related to the apps, and the others were not
resolved (at least there is no information on the forums/mailing lists
saying it is resolved and what was the issue).
You could take a look at the whole thread dump after about 10-15 minutes
of 100% cpu usage:
http://ouessant2.ajlsm.com/cocoon_app_cpu_issue
The two threads I gave in example (ajp-9009-8 and ajp-9009-9) are ones
that were started approximately when the server runs out of CPU ; and
are still in the same state 10-15 minutes after.
Thank you.
Regards,
--
Johan Cwiklinski
AJLSM
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
For additional commands, e-mail: users-help@cocoon.apache.org
Re: Tomcat6/Cocoon 2.1.10 using 100% CPU on windows
Posted by Johan Cwiklinski <jo...@ajlsm.com>.
Hello,
Le 22/12/2010 11:53, Laurent Medioni a écrit :
> Hi,
> have you tried to upgrade to the latest JDK fix available ?
Not yet, I'll try that this afternoon. I have already tested that with
at least two jdk versions (_17 and _21).
> Laurent
>
>
Thank you,
Johan
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
For additional commands, e-mail: users-help@cocoon.apache.org
RE: Tomcat6/Cocoon 2.1.10 using 100% CPU on windows
Posted by Laurent Medioni <lm...@odyssey-group.com>.
Hi,
have you tried to upgrade to the latest JDK fix available ?
Laurent
____________________________________________________________
• This email and any files transmitted with it are CONFIDENTIAL and intended
solely for the use of the individual or entity to which they are addressed.
• Any unauthorized copying, disclosure, or distribution of the material within
this email is strictly forbidden.
• Any views or opinions presented within this e-mail are solely those of the
author and do not necessarily represent those of Odyssey Financial
Technologies SA unless otherwise specifically stated.
• An electronic message is not binding on its sender. Any message referring to
a binding engagement must be confirmed in writing and duly signed.
• If you have received this email in error, please notify the sender immediately
and delete the original.
Re: Tomcat6/Cocoon 2.1.10 using 100% CPU on windows
Posted by Johan Cwiklinski <jo...@ajlsm.com>.
Hello and best wishes for 2011 :)
Le 23/12/2010 18:38, Johan Cwiklinski a écrit :
> Hi,
>
> Le 23/12/2010 10:49, Johan Cwiklinski a écrit :
>> I've just put the 2.1.11 RessourceReader class into my WEB-INF/classes
>> directory ; I'm currently testing that way on the server (would be great
>> if I can solve this problem until I can spend some time for 2.1.11
>> upgrade ;)).
>
> Well... It seems that solved the issue :)
>
> Server is up and running for a few hours now, without having any more
> CPU issues.
After more than a week in pa production environment, I can confirm that
the issue is now solved.
Again, thank you!
Regards,
Johan
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
For additional commands, e-mail: users-help@cocoon.apache.org
Re: Tomcat6/Cocoon 2.1.10 using 100% CPU on windows
Posted by Johan Cwiklinski <jo...@ajlsm.com>.
Hi,
Le 23/12/2010 10:49, Johan Cwiklinski a écrit :
> I've just put the 2.1.11 RessourceReader class into my WEB-INF/classes
> directory ; I'm currently testing that way on the server (would be great
> if I can solve this problem until I can spend some time for 2.1.11
> upgrade ;)).
Well... It seems that solved the issue :)
Server is up and running for a few hours now, without having any more
CPU issues.
Thank you very much Laurent!
Merry Xmas and happy new year everyone,
Johan
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
For additional commands, e-mail: users-help@cocoon.apache.org
Re: Tomcat6/Cocoon 2.1.10 using 100% CPU on windows
Posted by Johan Cwiklinski <jo...@ajlsm.com>.
Hello,
Le 22/12/2010 17:12, Johan Cwiklinski a écrit :
> Le 22/12/2010 15:14, Laurent Medioni a écrit :
>> Time to swich to 2.1.11 then ;)
>>
>> ResourceReader in 2.1.10:
>> private static final Map documents = new HashMap();
>>
>> ResourceReader in 2.1.11:
>> private static final Map documents = Collections.synchronizedMap(new HashMap());
>>
>> See https://issues.apache.org/jira/browse/COCOON-1977
>
> Oh! I was not aware of that...
>
> Thank you for the tip! :-)
JDK 1.6.0_23 did not solve the issue :(
I've just put the 2.1.11 RessourceReader class into my WEB-INF/classes
directory ; I'm currently testing that way on the server (would be great
if I can solve this problem until I can spend some time for 2.1.11
upgrade ;)).
>
>>
>> Laurent
>>
>
> Johan
Johan
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
For additional commands, e-mail: users-help@cocoon.apache.org
Re: Tomcat6/Cocoon 2.1.10 using 100% CPU on windows
Posted by Johan Cwiklinski <jo...@ajlsm.com>.
Le 22/12/2010 15:14, Laurent Medioni a écrit :
> Time to swich to 2.1.11 then ;)
>
> ResourceReader in 2.1.10:
> private static final Map documents = new HashMap();
>
> ResourceReader in 2.1.11:
> private static final Map documents = Collections.synchronizedMap(new HashMap());
>
> See https://issues.apache.org/jira/browse/COCOON-1977
Oh! I was not aware of that...
Thank you for the tip! :-)
>
> Laurent
>
Johan
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
For additional commands, e-mail: users-help@cocoon.apache.org
RE: Tomcat6/Cocoon 2.1.10 using 100% CPU on windows
Posted by Laurent Medioni <lm...@odyssey-group.com>.
Time to swich to 2.1.11 then ;)
ResourceReader in 2.1.10:
private static final Map documents = new HashMap();
ResourceReader in 2.1.11:
private static final Map documents = Collections.synchronizedMap(new HashMap());
See https://issues.apache.org/jira/browse/COCOON-1977
Laurent
____________________________________________________________
• This email and any files transmitted with it are CONFIDENTIAL and intended
solely for the use of the individual or entity to which they are addressed.
• Any unauthorized copying, disclosure, or distribution of the material within
this email is strictly forbidden.
• Any views or opinions presented within this e-mail are solely those of the
author and do not necessarily represent those of Odyssey Financial
Technologies SA unless otherwise specifically stated.
• An electronic message is not binding on its sender. Any message referring to
a binding engagement must be confirmed in writing and duly signed.
• If you have received this email in error, please notify the sender immediately
and delete the original.
Re: Tomcat6/Cocoon 2.1.10 using 100% CPU on windows
Posted by Johan Cwiklinski <jo...@ajlsm.com>.
Hello André,
Le 22/12/2010 13:23, andre.davignon@free.fr a écrit :
> Hi Johan,
>
> Does the same problem occur with mod_proxy (full http) instead of mod_proxy_ajp.
>
> We have encountered some problems with mod_proxy_ajp that were solved by using simple mod_proxy.
Yes, I also have tested with http proxy... Issue is still the same, even
if the tomcat seems to load less in normal usage.
I have also tried to set up a load balancer (with ajp then with http),
without more luck :(
>
> Cheers, buddy.
>
> André
Thank you :)
Johan
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
For additional commands, e-mail: users-help@cocoon.apache.org
Re: Tomcat6/Cocoon 2.1.10 using 100% CPU on windows
Posted by an...@free.fr.
Hi Johan,
Does the same problem occur with mod_proxy (full http) instead of mod_proxy_ajp.
We have encountered some problems with mod_proxy_ajp that were solved by using simple mod_proxy.
Cheers, buddy.
André
-----Original Message-----
From: Johan Cwiklinski <jo...@ajlsm.com>
Date: Wed, 22 Dec 2010 08:51:51
To: <us...@cocoon.apache.org>
Reply-To: users@cocoon.apache.org
Subject: Tomcat6/Cocoon 2.1.10 using 100% CPU on windows
Hello,
I have a problem with a cocoon 2.1.10 webapp running under tomcat 6.0.26
under windows 2003 server 64 bits with oracle's JDK 1.6.0_21.
This application is installed on a 'background' server, an application
on another server request it via AJP using apache mod_proxy_ajp.
For some reasons, the application will often eats 100% of the CPU, we
then need to kill and restart tomcat. Logs says absolutely nothing :(
I was not able yet to reproduce the issue on my dev environment.
This application mainly use some classes we've developed on the top of
cocoon that will:
- search for image file in some directories on different hard disks
(mainly by testing each directory + image path and looking if the file
exists),
- retrieve and show the image,
- additionally use ImageMagick to resize, rotate, etc.
The 'main' class extends cocoon's ResourceReader.
Using the jvisualvm tool provided with Oracle's JDK, I can observe that:
- ajp threads are sometimes running, and sometimes waiting; ok, that
seems normal,
- when the 100% cpu issue occurs, some ajp threads keeps running (never
get back to waiting state). At the beginning, only one or two threads
are affected, many more will be if we wait. I can also observe that some
threads (a few ones unfortunately) will still having the normal behavior.
All running threads using our class (ImageMagickReader) seems to be kind
of blocked on super.setup or super.generate methods:
"ajp-9009-9" - Thread t@65
java.lang.Thread.State: RUNNABLE
at java.util.HashMap.get(HashMap.java:303)
at
org.apache.cocoon.reading.ResourceReader.getLastModified(ResourceReader.java:242)
at
org.apache.cocoon.reading.ResourceReader.setupHeaders(ResourceReader.java:177)
at org.apache.cocoon.reading.ResourceReader.setup(ResourceReader.java:157)
at org.pleade.reading.ImageMagickReader.setup(ImageMagickReader.java:272)
[...]
Line 242 of ResourceReader.java is:
final String systemId = (String) documents.get(request.getRequestURI());
"ajp-9009-8" - Thread t@102
java.lang.Thread.State: RUNNABLE
at java.util.HashMap.transfer(HashMap.java:484)
at java.util.HashMap.resize(HashMap.java:463)
at java.util.HashMap.addEntry(HashMap.java:755)
at java.util.HashMap.put(HashMap.java:385)
at
org.apache.cocoon.reading.ResourceReader.generate(ResourceReader.java:346)
at
org.pleade.reading.ImageMagickReader.generate(ImageMagickReader.java:584)
[...]
Line 346 of ResourceReader.java is:
documents.put(request.getRequestURI(), inputSource.getURI());
Those two examples are based on the first threads that will never release.
I do not know if it is possible for a HashMap to be sort of corrupted ;
of maybe HTTP headers? I'm not sure even if what we're seeing is is the
cause or the consequence of the issue :(
The same issue has been observed in the past on another server which is
now running under GNU/Linux, and now seems to be ok (about two weeks
under Linux, and no longer 100% CPU!).
We've trying several tomcat and java versions, that changes anything.
The issue can occurs after several uptime hours, or only a few minutes!
If there are many connections, the issue will occurs more often; but is
still present with just a few connections.
I really do not know where the problem should be... Is it our code? Is
it cocoon? Is it tomcat? Or more probably something one of them is doing
that windows dislikes?
It's difficult to know when exactly the problem happens (we've asked
system administrators but get no answer) ; so I've not yet tried to log
in debug mode (well, I've tried it once, but this is really verbose...).
Any ideas? I do not know what to try or where to look at now :/
Many similar issues I can show over the web were related to a bug in
tcnative under tomcat 5.5 ; I guess that is resolved now, I did not
found any similar bug under tomcat 6. I also found a few ones with
tomcat 6, but some were related to the apps, and the others were not
resolved (at least there is no information on the forums/mailing lists
saying it is resolved and what was the issue).
You could take a look at the whole thread dump after about 10-15 minutes
of 100% cpu usage:
http://ouessant2.ajlsm.com/cocoon_app_cpu_issue
The two threads I gave in example (ajp-9009-8 and ajp-9009-9) are ones
that were started approximately when the server runs out of CPU ; and
are still in the same state 10-15 minutes after.
Thank you.
Regards,
--
Johan Cwiklinski
AJLSM
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
For additional commands, e-mail: users-help@cocoon.apache.org