You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by "RAY, DAVID" <DA...@tccd.edu> on 2015/07/01 22:00:18 UTC
[users@httpd] RE: AIX OS Patch Breaks Apache/Tomcat
>-----Original Message-----
>From: RAY, DAVID [mailto:DAVID.RAY@tccd.edu]
>Sent: Tuesday, June 30, 2015 12:10 PM
>To: users@tomcat.apache.org
>Subject: AIX OS Patch Breaks Apache/Tomcat
>
>Hello.
>
>I am running Apache 2.2.29 and Tomcat 7.0.59 with tomcat connector(mod_jk) version 1.2.40 on AIX version 7.1 server. Started having problems this morning after AIX OS was patched to AIX 7.1 TL 03 SP 04 and openssh to >6.0.0.6108. WebAdvisor runs fine immediately after apache is started or restarted. However its response slowing down. AIX server CPU steadily increases and approaches 100% after running for 5 or 10 minutes under heavy >load.. Seeing http process accumulate. Not seeing much traffic at all in Tomcat server status. Recompiled apache, tomcat connector, and Tomcat native. Still no luck. Later determined there is a bug with openssh 6.0.0.6.108 >and tried 6.0.0.6109. Recompiled again. Still no improvement. Just booted from Clone backup taken before the AIX patch and everything runs fine, no problems. Anyone else have this experience with AIX >patches/updates? Not really much to see in error logs. Tomcat is configured to run with java6 32 bit:
>
>IBM J9 VM (build 2.4, JRE 1.6.0 IBM J9 2.4 AIX ppc-32 jvmap3260sr16-20141216_227499 (JIT enabled, AOT enabled)
>J9VM - 20141216_227499
>JIT - r9_20140523_64469ifx3
>GC - GA24_Java6_SR16_20141216_1020_B227499)
>JCL - 20141216_01
>
>
>Thank you.
>David
We have determined that that this problem is triggered by Apache server-status. I have always used server-status to monitor our production system, no problem. I rarely use server-status to monitor our test system because there is nothing to see there due to extremely light traffic. But after the AIX patch, server-status gradually causes the sever CPU to be maxed out. This problem was just duplicated on our test server. The CPU drops back to normal for that server when server-status is stopped. Example of server-status URL I use in my IE 11 browser: https://website.com/server-status?refresh=5
Has anyone else had this experience? I guess for now, I will just have to stop using server-status to monitor Apache so that we can patch our AIX system.
Interesting how server-status is actually showing the gradual CPU increase. When this snapshot was taken, server-status had just moved from line 9 to line 10, where the W is. The 53.55 at line 9 is what the CPU usage was when this snapshot was taken:
0-0 17039448 0/13/13 _ 48.65 52 371 0.0 0.09 0.09 xxx.xxx.xx.x website.com GET /server-status?refresh=5 HTTP/1.1
0-0 17039448 0/13/13 _ 49.14 47 305 0.0 0.10 0.10 xxx.xxx.xx.x website.com GET /server-status?refresh=5 HTTP/1.1
0-0 17039448 0/13/13 _ 49.64 41 297 0.0 0.11 0.11 xxx.xxx.xx.x website.com GET /server-status?refresh=5 HTTP/1.1
0-0 17039448 0/13/13 _ 50.46 35 554 0.0 0.11 0.11 xxx.xxx.xx.x website.com GET /server-status?refresh=5 HTTP/1.1
0-0 17039448 0/13/13 _ 51.58 23 380 0.0 0.12 0.12 xxx.xxx.xx.x website.com GET /server-status?refresh=5 HTTP/1.1
0-0 17039448 0/13/13 _ 51.04 29 373 0.0 0.09 0.09 xxx.xxx.xx.x website.com GET /server-status?refresh=5 HTTP/1.1
0-0 17039448 0/13/13 _ 52.19 17 399 0.0 0.14 0.14 xxx.xxx.xx.x website.com GET /server-status?refresh=5 HTTP/1.1
0-0 17039448 0/13/13 _ 52.93 11 528 0.0 0.11 0.11 xxx.xxx.xx.x website.com GET /server-status?refresh=5 HTTP/1.1
0-0 17039448 0/13/13 _ 53.55 5 438 0.0 0.11 0.11 xxx.xxx.xx.x website.com GET /server-status?refresh=5 HTTP/1.1
0-0 17039448 0/12/12 W 40.99 0 0 0.0 0.09 0.09 xxx.xxx.xx.x website.com GET /server-status?refresh=5 HTTP/1.1
0-0 17039448 0/12/12 _ 43.98 104 256 0.0 0.09 0.09 xxx.xxx.xx.x website.com GET /server-status?refresh=5 HTTP/1.1
0-0 17039448 0/12/12 _ 41.50 133 355 0.0 0.09 0.09 xxx.xxx.xx.x website.com GET /server-status?refresh=5 HTTP/1.1
0-0 17039448 0/12/12 _ 42.01 127 338 0.0 0.10 0.10 xxx.xxx.xx.x website.com GET /server-status?refresh=5 HTTP/1.1
0-0 17039448 0/11/11 _ 42.47 122 237 0.0 0.11 0.11 xxx.xxx.xx.x website.com GET /server-status?refresh=5 HTTP/1.1
0-0 17039448 0/12/12 _ 42.93 116 325 0.0 0.10 0.10 xxx.xxx.xx.x website.com GET /server-status?refresh=5 HTTP/1.1
0-0 17039448 0/12/12 _ 43.45 110 320 0.0 0.11 0.11 xxx.xxx.xx.x website.com GET /server-status?refresh=5 HTTP/1.1
0-0 17039448 0/12/12 _ 44.41 99 264 0.0 0.11 0.11 xxx.xxx.xx.x website.com GET /server-status?refresh=5 HTTP/1.1
0-0 17039448 0/12/12 _ 44.91 93 368 0.0 0.12 0.12 xxx.xxx.xx.x website.com GET /server-status?refresh=5 HTTP/1.1
0-0 17039448 0/12/12 _ 46.99 70 378 0.0 0.08 0.08 xxx.xxx.xx.x website.com GET /server-status?refresh=5 HTTP/1.1
0-0 17039448 0/12/12 _ 45.48 87 365 0.0 0.11 0.11 xxx.xxx.xx.x website.com GET /server-status?refresh=5 HTTP/1.1
0-0 17039448 0/12/12 _ 45.91 82 256 0.0 0.10 0.10 xxx.xxx.xx.x website.com GET /server-status?refresh=5 HTTP/1.1
0-0 17039448 0/12/12 _ 48.06 58 254 0.0 0.09 0.09 xxx.xxx.xx.x website.com GET /server-status?refresh=5 HTTP/1.1
0-0 17039448 0/12/12 _ 46.41 76 294 0.0 0.09 0.09 xxx.xxx.xx.x website.com GET /server-status?refresh=5 HTTP/1.1
0-0 17039448 0/12/12 _ 47.53 64 368 0.0 0.09 0.09 xxx.xxx.xx.x website.com GET /server-status?refresh=5 HTTP/1.1
David
[users@httpd] RE: AIX OS Patch Breaks Apache/Tomcat
Posted by "RAY, DAVID" <DA...@tccd.edu>.
>>Hello.
>>
>>I am running Apache 2.2.29 and Tomcat 7.0.59 with tomcat connector(mod_jk) version 1.2.40 on AIX version 7.1 server. Started having problems this morning after AIX OS was patched to AIX 7.1 TL 03 SP 04 and openssh to >>6.0.0.6108. WebAdvisor runs fine immediately after apache is started or restarted. However its response slowing down. AIX server CPU steadily increases and approaches 100% after running for 5 or 10 minutes under heavy >>load.. Seeing http process accumulate. Not seeing much traffic at all in Tomcat server status. Recompiled apache, tomcat connector, and Tomcat native. Still no luck. Later determined there is a bug with openssh >>6.0.0.6.108 >and tried 6.0.0.6109. Recompiled again. Still no improvement. Just booted from Clone backup taken before the AIX patch and everything runs fine, no problems. Anyone else have this experience with AIX >>patches/updates? Not really much to see in error logs. Tomcat is configured to run with java6 32 bit:
>>
>>IBM J9 VM (build 2.4, JRE 1.6.0 IBM J9 2.4 AIX ppc-32 jvmap3260sr16-20141216_227499 (JIT enabled, AOT enabled)
>>J9VM - 20141216_227499
>>JIT - r9_20140523_64469ifx3
>>GC - GA24_Java6_SR16_20141216_1020_B227499)
>>JCL - 20141216_01
>>
>>
>>Thank you.
>>David
>
>Just an update on this problem. Now this appears to be an issue meant for the Apache list, which I just sent this to.
>
>We have determined that that this problem is triggered by Apache server-status. I have always used server-status to monitor our production system, no problem. I rarely use server-status to monitor our test system because >there is nothing to see there due to extremely light traffic. But after the AIX patch, server-status gradually causes the sever CPU to be maxed out. This problem was just duplicated on our test server. The CPU drops back to >normal for that server when server-status is stopped. Example of server-status URL I use in my IE 11 browser: https://website.com/server-status?refresh=5
>
>Has anyone else had this experience? I guess for now, I will just have to stop using server-status to monitor Apache so that we can patch our AIX system.
RESOLVED:
Solved this problem by downloading and compiling the latest openssl, currently 1.0.2c, from openssl.org. Recompiled Apache with openssl 1.0.2c. Recompiled Tomcat Connector and Tomcat Native too just to make sure all bases are covered. I'm guessing that the problem was caused by the AIX 7.1 TL 03 SP 04 update patching the server openssl. This triggered the Apache server-status/mod_status high CPU usage issue when used to monitor our https site.
David