You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by Marcin Wanat <ma...@gmail.com> on 2013/03/27 14:42:27 UTC
[users@httpd] Apache httpd 2.4.4 many connections in Reading Request state
Hi,
i have weird problem with newly configured server.
I have running Apache 2.2.16(prefork) with php(module) 5.2.14 running
on CentOS5 for two years on my old server and last week i decided to
move to new server (CentOS6) and upgrade Apache to 2.4.4(prefork) and
php to 5.4.13.
On old server everything is working fine but when i route this traffic
(A DNS record of my domain) to new server then strange things are
happening.
For the first 30 seconds after httpd start everything is normal, and
then after 30 seconds about 25% of active connections are in "R"
(Reading Request) state. These connections are freezed in R state for
10-25secs and then dissapear.
They are not logged in error_log even in trace6 level.
server-status for such connection looks like:
PID Acc M CPU SS Req Conn Child Slot Client VHost
22182 0/90/90 R 0.73 3 5 0.0 0.03 0.03 81.96.86.224
netstat for this IP shows:
tcp 0 0 ::ffff:95.211.131.XXX:80
::ffff:81.96.86.224:52168 TIME_WAIT -
tcp 0 0 ::ffff:95.211.131.XXX:80
::ffff:81.96.86.224:52202 TIME_WAIT -
tcp 0 0 ::ffff:95.211.131.XXX:80
::ffff:81.96.86.224:52254 TIME_WAIT -
tcp 0 0 ::ffff:95.211.131.XXX:80
::ffff:81.96.86.224:52201 TIME_WAIT -
tcp 0 0 ::ffff:95.211.131.XXX:80
::ffff:81.96.86.224:52255 ESTABLISHED 22182/httpd
tcp 0 0 ::ffff:95.211.131.XXX:80
::ffff:81.96.86.224:52169 TIME_WAIT -
tcp 0 0 ::ffff:95.211.131.XXX:80
::ffff:81.96.86.224:52203 TIME_WAIT -
Server load is about 0.5 and total CPU usage ~5%. Server is 16core,
32gb ram, 4xSAS hdd so it is not overloaded.
Network is 100mbps and usage of about 4mbps (only html content). I
have tested connection to few IPs which freeze in R state with mtr,
but no single packet loss were found.
Server is handling about 100requests/sec in peek hours and 50req/sec
during night.
Regardless of req/sec problem always occurs after about 30secs after
httpd start.
apachectl restart does not change anything. Only manually killing all
httpd processes and starting them from scratch can give 30seconds free
of R state connections.
I was also trying to recompile httpd with event mpm - effect is the
same. Even when i remove all htdocs content and every request is 404,
Reading Request connections appears after 30secs. Was trying httpd
2.2.24 and the effect was the same.
I was even trying different kernels (Centos6 2.6.32-358.2.1.el6 and
newest one 3.8.4) but with no luck.
These Reading Request connections are comming from different networks
all over the world.
I was also trying to do some stress tests with ab -n 50000 -c 20 from
different servers but each time these tests completed without any
freeze.
Some details:
[root@srv logs]# httpd -V
Server version: Apache/2.4.4 (Unix)
Server built: Mar 24 2013 01:43:25
Server's Module Magic Number: 20120211:11
Server loaded: APR 1.4.6, APR-UTIL 1.5.1
Compiled using: APR 1.4.6, APR-UTIL 1.5.1
Architecture: 64-bit
Server MPM: prefork
threaded: no
forked: yes (variable process count)
Server compiled with....
-D APR_HAS_SENDFILE
-D APR_HAS_MMAP
-D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
-D APR_USE_SYSVSEM_SERIALIZE
-D APR_USE_PTHREAD_SERIALIZE
-D APR_HAS_OTHER_CHILD
-D AP_HAVE_RELIABLE_PIPED_LOGS
-D DYNAMIC_MODULE_LIMIT=256
-D HTTPD_ROOT="/usr/local/httpd"
-D SUEXEC_BIN="/usr/local/httpd/bin/suexec"
-D DEFAULT_PIDLOG="logs/httpd.pid"
-D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
-D DEFAULT_ERRORLOG="logs/error_log"
-D AP_TYPES_CONFIG_FILE="conf/mime.types"
-D SERVER_CONFIG_FILE="conf/httpd.conf"
[root@srv logs]# httpd -l
Compiled in modules:
core.c
mod_so.c
http_core.c
prefork.c
Loaded modules:
LoadModule php5_module modules/libphp5.so
LoadModule unixd_module modules/mod_unixd.so
LoadModule authz_host_module modules/mod_authz_host.so
LoadModule authz_core_module modules/mod_authz_core.so
LoadModule access_compat_module modules/mod_access_compat.so
LoadModule rewrite_module modules/mod_rewrite.so
LoadModule mime_module modules/mod_mime.so
LoadModule dir_module modules/mod_dir.so
LoadModule authn_core_module modules/mod_authn_core.so
LoadModule authn_file_module modules/mod_authn_file.so
LoadModule status_module modules/mod_status.so
LoadModule authz_user_module modules/mod_authz_user.so
LoadModule auth_basic_module modules/mod_auth_basic.so
Prefork config:
StartServers 256
MinSpareServers 256
MaxSpareServers 512
MaxRequestWorkers 1024
MaxConnectionsPerChild 0
Currently i am out of ideas what else could i check to trace this problem.
Regards,
Marcin Wanat
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Apache httpd 2.4.4 many connections in Reading
Request state
Posted by Marcin Wanat <ma...@gmail.com>.
2013/3/27 Marcin Wanat <ma...@gmail.com>:
>>
>> What are the equivalent settings in your 2.2 setup?
>>
> # ./httpd -V
> Server version: Apache/2.2.16 (Unix)
> Server built: Aug 2 2010 19:26:08
> Server's Module Magic Number: 20051115:24
> Server loaded: APR 1.4.2, APR-Util 1.3.9
> Compiled using: APR 1.4.2, APR-Util 1.3.9
> Architecture: 64-bit
> Server MPM: Prefork
> threaded: no
> forked: yes (variable process count)
I was also able to find some logs for these ReadingRequest
connections. All these freezed requests results in such logs:
[Wed Mar 27 19:27:11.751005 2013] [core:trace6] [pid 23345]
core_filters.c(525): [client 223.29.232.XXX:43313] core_output_filter:
flushing because of FLUSH bucket
but i still don't know what to do to fix it.
Very odd is also 30 sec delay after httpd start after which
ReadingRequest connections appear in server status. For first 30
seconds server can handle ~3000 requests without any problem and after
exactly 30 seconds ReadingRequest connections start showing in server
status.
Regards,
Marcin.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Apache httpd 2.4.4 many connections in Reading
Request state
Posted by Marcin Wanat <ma...@gmail.com>.
>
> What are the equivalent settings in your 2.2 setup?
>
# ./httpd -V
Server version: Apache/2.2.16 (Unix)
Server built: Aug 2 2010 19:26:08
Server's Module Magic Number: 20051115:24
Server loaded: APR 1.4.2, APR-Util 1.3.9
Compiled using: APR 1.4.2, APR-Util 1.3.9
Architecture: 64-bit
Server MPM: Prefork
threaded: no
forked: yes (variable process count)
Server compiled with....
-D APACHE_MPM_DIR="server/mpm/prefork"
-D APR_HAS_SENDFILE
-D APR_HAS_MMAP
-D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
-D APR_USE_SYSVSEM_SERIALIZE
-D APR_USE_PTHREAD_SERIALIZE
-D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
-D APR_HAS_OTHER_CHILD
-D AP_HAVE_RELIABLE_PIPED_LOGS
-D DYNAMIC_MODULE_LIMIT=128
-D HTTPD_ROOT="/usr/local/httpd"
-D SUEXEC_BIN="/usr/local/httpd/bin/suexec"
-D DEFAULT_PIDLOG="logs/httpd.pid"
-D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
-D DEFAULT_LOCKFILE="logs/accept.lock"
-D DEFAULT_ERRORLOG="logs/error_log"
-D AP_TYPES_CONFIG_FILE="conf/mime.types"
-D SERVER_CONFIG_FILE="conf/httpd.conf"
# ./httpd -l
Compiled in modules:
core.c
mod_authn_file.c
mod_authn_default.c
mod_authz_host.c
mod_authz_groupfile.c
mod_authz_user.c
mod_authz_default.c
mod_auth_basic.c
mod_reqtimeout.c
mod_include.c
mod_filter.c
mod_deflate.c
mod_log_config.c
mod_expires.c
mod_headers.c
mod_usertrack.c
mod_unique_id.c
mod_setenvif.c
mod_version.c
prefork.c
http_core.c
mod_mime.c
mod_status.c
mod_autoindex.c
mod_asis.c
mod_cgi.c
mod_vhost_alias.c
mod_negotiation.c
mod_dir.c
mod_actions.c
mod_userdir.c
mod_alias.c
mod_rewrite.c
mod_so.c
Prefork config:
StartServers 128
MinSpareServers 128
MaxSpareServers 256
MaxClients 400
MaxRequestsPerChild 0
Regards,
Marcin
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Apache httpd 2.4.4 many connections in Reading
Request state
Posted by Jeff Trawick <tr...@gmail.com>.
On Wed, Mar 27, 2013 at 9:42 AM, Marcin Wanat <ma...@gmail.com> wrote:
> Hi,
>
> i have weird problem with newly configured server.
> I have running Apache 2.2.16(prefork) with php(module) 5.2.14 running
> on CentOS5 for two years on my old server and last week i decided to
> move to new server (CentOS6) and upgrade Apache to 2.4.4(prefork) and
> php to 5.4.13.
> On old server everything is working fine but when i route this traffic
> (A DNS record of my domain) to new server then strange things are
> happening.
>
> For the first 30 seconds after httpd start everything is normal, and
> then after 30 seconds about 25% of active connections are in "R"
> (Reading Request) state. These connections are freezed in R state for
> 10-25secs and then dissapear.
> They are not logged in error_log even in trace6 level.
>
> server-status for such connection looks like:
> PID Acc M CPU SS Req Conn Child Slot Client VHost
> 22182 0/90/90 R 0.73 3 5 0.0 0.03 0.03 81.96.86.224
>
> netstat for this IP shows:
> tcp 0 0 ::ffff:95.211.131.XXX:80
> ::ffff:81.96.86.224:52168 TIME_WAIT -
> tcp 0 0 ::ffff:95.211.131.XXX:80
> ::ffff:81.96.86.224:52202 TIME_WAIT -
> tcp 0 0 ::ffff:95.211.131.XXX:80
> ::ffff:81.96.86.224:52254 TIME_WAIT -
> tcp 0 0 ::ffff:95.211.131.XXX:80
> ::ffff:81.96.86.224:52201 TIME_WAIT -
> tcp 0 0 ::ffff:95.211.131.XXX:80
> ::ffff:81.96.86.224:52255 ESTABLISHED 22182/httpd
> tcp 0 0 ::ffff:95.211.131.XXX:80
> ::ffff:81.96.86.224:52169 TIME_WAIT -
> tcp 0 0 ::ffff:95.211.131.XXX:80
> ::ffff:81.96.86.224:52203 TIME_WAIT -
>
> Server load is about 0.5 and total CPU usage ~5%. Server is 16core,
> 32gb ram, 4xSAS hdd so it is not overloaded.
> Network is 100mbps and usage of about 4mbps (only html content). I
> have tested connection to few IPs which freeze in R state with mtr,
> but no single packet loss were found.
>
> Server is handling about 100requests/sec in peek hours and 50req/sec
> during night.
>
> Regardless of req/sec problem always occurs after about 30secs after
> httpd start.
> apachectl restart does not change anything. Only manually killing all
> httpd processes and starting them from scratch can give 30seconds free
> of R state connections.
>
> I was also trying to recompile httpd with event mpm - effect is the
> same. Even when i remove all htdocs content and every request is 404,
> Reading Request connections appears after 30secs. Was trying httpd
> 2.2.24 and the effect was the same.
>
> I was even trying different kernels (Centos6 2.6.32-358.2.1.el6 and
> newest one 3.8.4) but with no luck.
>
> These Reading Request connections are comming from different networks
> all over the world.
>
> I was also trying to do some stress tests with ab -n 50000 -c 20 from
> different servers but each time these tests completed without any
> freeze.
>
> Some details:
> [root@srv logs]# httpd -V
> Server version: Apache/2.4.4 (Unix)
> Server built: Mar 24 2013 01:43:25
> Server's Module Magic Number: 20120211:11
> Server loaded: APR 1.4.6, APR-UTIL 1.5.1
> Compiled using: APR 1.4.6, APR-UTIL 1.5.1
> Architecture: 64-bit
> Server MPM: prefork
> threaded: no
> forked: yes (variable process count)
> Server compiled with....
> -D APR_HAS_SENDFILE
> -D APR_HAS_MMAP
> -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
> -D APR_USE_SYSVSEM_SERIALIZE
> -D APR_USE_PTHREAD_SERIALIZE
> -D APR_HAS_OTHER_CHILD
> -D AP_HAVE_RELIABLE_PIPED_LOGS
> -D DYNAMIC_MODULE_LIMIT=256
> -D HTTPD_ROOT="/usr/local/httpd"
> -D SUEXEC_BIN="/usr/local/httpd/bin/suexec"
> -D DEFAULT_PIDLOG="logs/httpd.pid"
> -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
> -D DEFAULT_ERRORLOG="logs/error_log"
> -D AP_TYPES_CONFIG_FILE="conf/mime.types"
> -D SERVER_CONFIG_FILE="conf/httpd.conf"
> [root@srv logs]# httpd -l
> Compiled in modules:
> core.c
> mod_so.c
> http_core.c
> prefork.c
>
> Loaded modules:
> LoadModule php5_module modules/libphp5.so
> LoadModule unixd_module modules/mod_unixd.so
> LoadModule authz_host_module modules/mod_authz_host.so
> LoadModule authz_core_module modules/mod_authz_core.so
> LoadModule access_compat_module modules/mod_access_compat.so
> LoadModule rewrite_module modules/mod_rewrite.so
> LoadModule mime_module modules/mod_mime.so
> LoadModule dir_module modules/mod_dir.so
> LoadModule authn_core_module modules/mod_authn_core.so
> LoadModule authn_file_module modules/mod_authn_file.so
> LoadModule status_module modules/mod_status.so
> LoadModule authz_user_module modules/mod_authz_user.so
> LoadModule auth_basic_module modules/mod_auth_basic.so
>
> Prefork config:
> StartServers 256
> MinSpareServers 256
> MaxSpareServers 512
> MaxRequestWorkers 1024
> MaxConnectionsPerChild 0
What are the equivalent settings in your 2.2 setup?
>
>
>
> Currently i am out of ideas what else could i check to trace this problem.
>
> Regards,
> Marcin Wanat
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
--
Born in Roswell... married an alien...
http://emptyhammock.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org