You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by Abhi Auradkar <au...@yahoo.com> on 2011/09/26 23:35:21 UTC
[users@httpd] Apache freezes after running for 2 days.
Hi All,
I have been running the apache server to load test a application.
The apache 2.2.20 is running on CentOS 5.6
-bash-3.2$ uname -a
Linux hdm102 2.6.18-238.19.1.el5.centos.plus #1 SMP Mon Jul 18 10:05:09 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux
-bash-3.2$ cat /etc/issue
CentOS release 5.6 (Final)
Kernel \r on an \m
-bash-3.2$ ./httpd -V
Server version: Apache/2.2.20 (Unix)
Server built: Sep 8 2011 23:41:28
Server's Module Magic Number: 20051115:28
Server loaded: APR 1.4.5, APR-Util 1.3.12
Compiled using: APR 1.4.5, APR-Util 1.3.12
Architecture: 64-bit
Server MPM: Worker
threaded: yes (fixed thread count)
forked: yes (variable process count)
Server compiled with....
-D APACHE_MPM_DIR="server/mpm/worker"
-D APR_HAS_SENDFILE
-D APR_HAS_MMAP
-D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
-D APR_USE_SYSVSEM_SERIALIZE
-D APR_USE_PTHREAD_SERIALIZE
-D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
-D APR_HAS_OTHER_CHILD
-D AP_HAVE_RELIABLE_PIPED_LOGS
-D DYNAMIC_MODULE_LIMIT=128
-D HTTPD_ROOT="/usr/local/apache2"
-D SUEXEC_BIN="/usr/local/apache2/bin/suexec"
-D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
-D DEFAULT_ERRORLOG="logs/error_log"
-D AP_TYPES_CONFIG_FILE="conf/mime.types"
-D SERVER_CONFIG_FILE="conf/httpd.conf"
I have a custom module(mod_XXXX.so) and oracle weblogic apache plugin(mod_wl_22.so) loaded via LoadModule directive into apache.
The load that is being tested is 2500 req/second.
Under this load apache continues to run for 2-3 days but suddnely stops responding. Has been happening consistently.
Once this condition occurs apache stops serving even static pages exa: index.html or does not display server-status(mod_status).
And the only way to recover is to restart the server.
The errors that I see in the error log are the following ones, which keep showing up frequently of which the first one when the weblogic server refuses connections dues to overload condition.
1. [Tue Sep 20 08:15:11 2011] [error] [client 135.115.142.108] ap_proxy: trying POST /cwmp-allow/cwmpWeb/CPEMgt at backend host '64.186.185.155/7003; got exception 'CONNECTION_REFUSED [os error=0, line 1715 of ../nsapi/URL.cpp]: Error connecting to host 64.186.185.155:7003'
2. [Tue Sep 20 08:15:12 2011] [error] [client 135.115.142.100] (34)Numerical result out of range: mod_XXXX: XXXX_throttle_handler(): failed to release the mutex lock
Had a look at the errno of apr and (34)Numerical result out of range doesn't seem to be coming from APR.
Has some come across such a issue? Any kind of help/advice will really helpfull.
Re: [users@httpd] Apache freezes after running for 2 days.
Posted by Rainer Jung <ra...@kippdata.de>.
On 26.09.2011 23:35, Abhi Auradkar wrote:
> I have been running the apache server to load test a application.
>
> The apache 2.2.20 is running on CentOS 5.6
>
> -bash-3.2$ uname -a
> Linux hdm102 2.6.18-238.19.1.el5.centos.plus #1 SMP Mon Jul 18 10:05:09
> EDT 2011 x86_64 x86_64 x86_64 GNU/Linux
> -bash-3.2$ cat /etc/issue
> CentOS release 5.6 (Final)
> Kernel \r on an \m
>
>
> -bash-3.2$ ./httpd -V
> Server version: Apache/2.2.20 (Unix)
> Server built: Sep 8 2011 23:41:28
> Server's Module Magic Number: 20051115:28
> Server loaded: APR 1.4.5, APR-Util 1.3.12
> Compiled using: APR 1.4.5, APR-Util 1.3.12
> Architecture: 64-bit
> Server MPM: Worker
> threaded: yes (fixed thread count)
> forked: yes (variable process count)
> Server compiled with....
> -D APACHE_MPM_DIR="server/mpm/worker"
> -D APR_HAS_SENDFILE
> -D APR_HAS_MMAP
> -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
> -D APR_USE_SYSVSEM_SERIALIZE
> -D APR_USE_PTHREAD_SERIALIZE
> -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
> -D APR_HAS_OTHER_CHILD
> -D AP_HAVE_RELIABLE_PIPED_LOGS
> -D DYNAMIC_MODULE_LIMIT=128
> -D HTTPD_ROOT="/usr/local/apache2"
> -D SUEXEC_BIN="/usr/local/apache2/bin/suexec"
> -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
> -D DEFAULT_ERRORLOG="logs/error_log"
> -D AP_TYPES_CONFIG_FILE="conf/mime.types"
> -D SERVER_CONFIG_FILE="conf/httpd.conf"
>
>
> I have a custom module(mod_XXXX.so) and oracle weblogic apache
> plugin(mod_wl_22.so) loaded via LoadModule directive into apache.
>
> The load that is being tested is 2500 req/second.
>
> Under this load apache continues to run for 2-3 days but suddnely stops
> responding. Has been happening consistently.
> Once this condition occurs apache stops serving even static pages exa:
> index.html or does not display server-status(mod_status).
>
> And the only way to recover is to restart the server.
>
> The errors that I see in the error log are the following ones, which
> keep showing up frequently of which the first one when the weblogic
> server refuses connections dues to overload condition.
>
> *1.* [Tue Sep 20 08:15:11 2011] [error] [client 135.115.142.108]
> ap_proxy: trying POST /cwmp-allow/cwmpWeb/CPEMgt at backend host
> '64.186.185.155/7003; got exception 'CONNECTION_REFUSED [os error=0,
> line 1715 of ../nsapi/URL.cpp]: Error connecting to host
> 64.186.185.155:7003'
> *2.* [Tue Sep 20 08:15:12 2011] [error] [client 135.115.142.100]
> *(34)Numerical result out of range: *mod_XXXX: XXXX_throttle_handler():
> failed to release the mutex lock
>
> Had a look at the errno of apr and *(34)Numerical result out of range
> *doesn't seem to be coming from APR.
>
> Has some come across such a issue? Any kind of help/advice will really
> helpfull.
Use gstack or similar against the hangingApache processes to inspect
what they are doing. My guess: since your modules logs "failed to
release the mutex lock", the worker threads micht block in trying to
acquire the unreleased lock.
Regards,
Rainer
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Apache freezes after running for 2 days.
Posted by Tom Evans <te...@googlemail.com>.
On Mon, Sep 26, 2011 at 10:35 PM, Abhi Auradkar <au...@yahoo.com> wrote:
> Hi All,
>
> I have been running the apache server to load test a application.
> The apache 2.2.20 is running on CentOS 5.6
> -bash-3.2$ uname -a
> Linux hdm102 2.6.18-238.19.1.el5.centos.plus #1 SMP Mon Jul 18 10:05:09 EDT
> 2011 x86_64 x86_64 x86_64 GNU/Linux
> -bash-3.2$ cat /etc/issue
> CentOS release 5.6 (Final)
> Kernel \r on an \m
>
> -bash-3.2$ ./httpd -V
> Server version: Apache/2.2.20 (Unix)
> Server built: Sep 8 2011 23:41:28
> Server's Module Magic Number: 20051115:28
> Server loaded: APR 1.4.5, APR-Util 1.3.12
> Compiled using: APR 1.4.5, APR-Util 1.3.12
> Architecture: 64-bit
> Server MPM: Worker
> threaded: yes (fixed thread count)
> forked: yes (variable process count)
> Server compiled with....
> -D APACHE_MPM_DIR="server/mpm/worker"
> -D APR_HAS_SENDFILE
> -D APR_HAS_MMAP
> -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
> -D APR_USE_SYSVSEM_SERIALIZE
> -D APR_USE_PTHREAD_SERIALIZE
> -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
> -D APR_HAS_OTHER_CHILD
> -D AP_HAVE_RELIABLE_PIPED_LOGS
> -D DYNAMIC_MODULE_LIMIT=128
> -D HTTPD_ROOT="/usr/local/apache2"
> -D SUEXEC_BIN="/usr/local/apache2/bin/suexec"
> -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
> -D DEFAULT_ERRORLOG="logs/error_log"
> -D AP_TYPES_CONFIG_FILE="conf/mime.types"
> -D SERVER_CONFIG_FILE="conf/httpd.conf"
>
> I have a custom module(mod_XXXX.so) and oracle weblogic apache
> plugin(mod_wl_22.so) loaded via LoadModule directive into apache.
> The load that is being tested is 2500 req/second.
> Under this load apache continues to run for 2-3 days but suddnely stops
> responding. Has been happening consistently.
> Once this condition occurs apache stops serving even static pages exa:
> index.html or does not display server-status(mod_status).
> And the only way to recover is to restart the server.
> The errors that I see in the error log are the following ones, which keep
> showing up frequently of which the first one when the weblogic server
> refuses connections dues to overload condition.
> 1. [Tue Sep 20 08:15:11 2011] [error] [client 135.115.142.108] ap_proxy:
> trying POST /cwmp-allow/cwmpWeb/CPEMgt at backend host '64.186.185.155/7003;
> got exception 'CONNECTION_REFUSED [os error=0, line 1715 of
> ../nsapi/URL.cpp]: Error connecting to host 64.186.185.155:7003'
> 2. [Tue Sep 20 08:15:12 2011] [error] [client 135.115.142.100] (34)Numerical
> result out of range: mod_XXXX: XXXX_throttle_handler(): failed to release
> the mutex lock
>
> Had a look at the errno of apr and (34)Numerical result out of range doesn't
> seem to be coming from APR.
> Has some come across such a issue? Any kind of help/advice will really
> helpfull.
errno 34 is ERANGE. Your custom module is trying to parse a number
from a string, probably strtol() or variant, and fails. It then
reports a problem releasing a mutex, and your entire server locks up.
I'd suggest these things are not unrelated.
Given that the failing code is in your custom module, I guess you are
on your own. The proxy error comes from mod_wl_22; you should contact
Oracle to resolve that.
Cheers
Tom
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org