You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by Neil Martin <ne...@4js.com> on 2007/02/08 12:08:55 UTC
[users@httpd] Timeouts with Threaded Apache 2.2.3
Hi,
Our company has been trying to benchmark(using ab) a large number of connections
to the web server(THREADED) on Solaris. The problem is we are often getting timeouts
(apr_poll: The timeout specified has expired (70007))
Originally we suspected the database connection in our module or in mod_perl was causing the problems,
so we retested with a dummy module call mod_foo ( attached ) but we still get timeouts.
Is this a know issue is the module api in threaded Apache ?
Reason for using threaded is we are implimenting connection pooling in our database, this gives MUCH
faster connection and through put times. This can't be done to the same level using preforked apache.
Full version information details of the benchmark are below:
% uname -a
SunOS sun642 5.9 Generic_112233-12 sun4u sparc SUNW,Ultra-4
Apache installation with the mpm worker threads:
% ./configure --with-mpm=worker --prefix=/usr/local/apache22
same behavior with the following option for Solaris/SPARC platform: --enable-nonportable-atomics=yes
% httpd -V
Server version: Apache/2.2.3
Server built: Jan 31 2007 01:00:53
Server's Module Magic Number: 20051115:3
Server loaded: APR 1.2.7, APR-Util 1.2.7
Compiled using: APR 1.2.7, APR-Util 1.2.7
Architecture: 32-bit
Server MPM: Worker
threaded: yes (fixed thread count)
forked: yes (variable process count)
Server compiled with....
-D APACHE_MPM_DIR="server/mpm/worker"
-D APR_HAS_SENDFILE
-D APR_HAS_MMAP
-D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
-D APR_USE_FCNTL_SERIALIZE
-D APR_USE_PTHREAD_SERIALIZE
-D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
-D APR_HAS_OTHER_CHILD
-D AP_HAVE_RELIABLE_PIPED_LOGS
-D DYNAMIC_MODULE_LIMIT=128
-D HTTPD_ROOT="/usr/local/apache22"
-D SUEXEC_BIN="/usr/local/apache22/bin/suexec"
-D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
-D DEFAULT_ERRORLOG="logs/error_log"
-D AP_TYPES_CONFIG_FILE="conf/mime.types"
-D SERVER_CONFIG_FILE="conf/httpd.conf"
% httpd -l
Compiled in modules:
core.c
mod_authn_file.c
mod_authn_default.c
mod_authz_host.c
mod_authz_groupfile.c
mod_authz_user.c
mod_authz_default.c
mod_auth_basic.c
mod_include.c
mod_filter.c
mod_log_config.c
mod_env.c
mod_setenvif.c
worker.c
http_core.c
mod_mime.c
mod_status.c
mod_autoindex.c
mod_asis.c
mod_cgid.c
mod_negotiation.c
mod_dir.c
mod_actions.c
mod_userdir.c
mod_alias.c
mod_so.c
Same behavior with the default or other values in the config files, so it's not necessary to provide them.
So, in the /usr/local/apache22/conf/extra/httpd-mpm.conf the following lines are still commented:
#include /usr/local/apache22/conf/extra/http-worker.conf
#include /usr/local/apache22/conf/extra/http-default.conf
The default ab.c timeout is 30 seconds:
apr_interval_time_t aprtimeout = apr_time_from_sec(30); /* timeout value */
mod_foo provided installation:
% cd /home/antslab/fourjs_apachebenchmark/httpd-2.2.3/modules/experimental/mod_foo.c
% apxs -c mod_foo.c
% apxs -i -a -n foo mod_foo.la
mod_foo bench test:
% ab -v 1 -n 2000 -c 200 http://192.168.11.87/foo-info
% sar 1 1000
03:06:44 80 20 0 0
03:06:45 77 23 0 0
03:06:46 78 22 0 0
03:06:47 78 18 0 4
03:06:48 78 7 0 16 <<< Something happens here!
03:06:49 77 3 0 20
03:06:50 75 1 0 24
03:06:51 75 2 0 23
03:06:52 77 6 0 17
...
(the Timeout expires after 30s)
Result:
Completed 1800 requests
apr_poll: The timeout specified has expired (70007)
Total of 1921 requests completed
Only one thread is still running out..
% ps -eLf | grep httpd
...
daemon 22326 22255 7 27 0 04:03:01 ? 0:00 /usr/local/apache22/bin/httpd -k start
daemon 22326 22255 8 27 20 04:03:01 ? 0:33 /usr/local/apache22/bin/httpd -k start <<< !
daemon 22326 22255 9 27 0 04:03:01 ? 0:00 /usr/local/apache22/bin/httpd -k start
...
% prstat
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
...
22326 daemon 26M 6096K cpu0 30 0 1:23:49 25% httpd/27
...
% pstack 22326
(normal thread)
----------------- lwp# 7 / thread# 7 --------------------
ff0d58f4 lwp_park (0, 0, 0)
ff0d166c mutex_lock_queue (ff0e8b44, 0, 17a738, ff0e8000, 81010100, ff00) + 104
ff0d206c slow_lock (17a738, feed2800, 4fbda0, fffffff8, 0, 4fbef5) + 58
ff29669c apr_thread_mutex_lock (17a730, 51b119, 0, 4, 0, 51b130) + 34
ff299758 allocator_alloc (179288, 1fe8, ff04283c, 1, 0, 51b0a0) + c0
ff29a8a4 apr_pool_create_ex (fc4fbb90, 1d26e0, 0, 0, 1f9610, fee74cf4) + bc
fee822d8 trace_add (1871d0, 0, 1f8df8, fee86cb0, 519e48, 51adf8) + c0
fee836fc x_quick_handler (519e80, 0, 519e80, 4fbe90, 51a018, 51a020) + 64
0007d488 ap_run_quick_handler (519e80, 0, 4, 519e80, 0, 51a031) + 78
000aa778 ap_process_request (519e80, 4, 519e80, 4fbe90, 81010100, ff00) + 40
000a60c8 ap_process_http_connection (4fbe90, 4fbda0, 4fbda0, fffffff8, 0, 4fbef5) + 80
000899ec ap_run_process_connection (4fbe90, 4fbda0, 4fbda0, 11a6, 4fbe88, 511e28) + 6c
0008a088 ap_process_connection (4fbe90, 4fbda0, 4fbda0, 11a6, 4fbe88, 511e28) + 98
000ce62c process_socket (4fbd60, 4fbda0, 9, 12, 511e28, 0) + c4
000cf270 worker_thread (184ba0, 1693e0, 0, 0, 0, 0) + 300
ff2acde8 dummy_worker (184ba0, 0, 0, 0, 0, 0) + 48
ff0d57b4 _lwp_start (0, 0, 0, 0, 0, 0)
(blocked thread)
----------------- lwp# 8 / thread# 8 --------------------
ff299bb4 allocator_free (179288, 2118a8, 0, 7efefeff, 0, 2118f8) + 8c
ff29a798 apr_pool_destroy (2118c0, 2118f8, 0, 0, 0, fee97f5c) + 170
fee61d60 trace_add (1871d0, 0, 1f95f8, fee64294, 746f746f, 297fc5) + 140
fee63078 x_insert_filter (29c848, 2a2e00, 188fc8, 0, 666f0070, 666f0049) + 60
00075908 ap_run_insert_filter (29c848, 0, 29c848, 276638, 0, 0) + 70
0007dd88 ap_invoke_handler (29c848, 0, 4, 29c848, 0, 29c9f9) + 18
000aa7b8 ap_process_request (29c848, 4, 29c848, 276638, 81010100, ff00) + 80
000a60c8 ap_process_http_connection (276638, 276548, 276548, fffffff8, 0, 27669d) + 80
000899ec ap_run_process_connection (276638, 276548, 276548, 11a7, 276630, 2845b0) + 6c
0008a088 ap_process_connection (276638, 276548, 276548, 11a7, 276630, 2845b0) + 98
000ce62c process_socket (276508, 276548, 9, 13, 2845b0, 0) + c4
000cf270 worker_thread (184bc0, 1693f8, 0, 0, 0, 0) + 300
ff2acde8 dummy_worker (184bc0, 0, 0, 0, 0, 0) + 48
ff0d57b4 _lwp_start (0, 0, 0, 0, 0, 0)
Regards,
--------------------------------------------------------------------------------
Neil J MARTIN [neilm@4js.com] Tel +44 (0)208 757 5817 Fax +44 (0)208 757 5827
Product Support Engineer - Four J's Development Tools (UK) [www.4js.com]
--
Regards,
--------------------------------------------------------------------------------
Neil J MARTIN [neilm@4js.com] Tel +44 (0)208 757 5817 Fax +44 (0)208 757 5827
Product Support Engineer - Four J's Development Tools (UK) [www.4js.com]
Re: [users@httpd] Timeouts with Threaded Apache 2.2.3
Posted by Neil Martin <ne...@4js.com>.
Hi,
Sorry, my question was worded badly:
The real question is about thread safty of DBI in mod_perl.
On Solaris we were getting core dumps from this combination using
our ODBC driver and an Informix ODBC driver. Don't get the core dumps
when using preforked apache.
Regards,
Neil
Issac Goldstand wrote:
> Neil Martin wrote:
>
>>On the same subject of 'thread safe' does anyone know if mod_perl
>>is being made thread safe? ( I believe there was a project for a threaded
>>version but it died, so just wondering is the main mod_perl branch was
>>being considered for making thread safe. )
>>
>
>
> AFAIK mod_perl *is* thread-safe.
>
> http://perl.apache.org/docs/2.0/user/intro/overview.html#Thread_environment_Issues
>
> Issac
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>
>
--
Regards,
--------------------------------------------------------------------------------
Neil J MARTIN [neilm@4js.com] Tel +44 (0)208 757 5817 Fax +44 (0)208 757 5827
Product Support Engineer - Four J's Development Tools (UK) [www.4js.com]
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Timeouts with Threaded Apache 2.2.3
Posted by Issac Goldstand <ma...@beamartyr.net>.
Neil Martin wrote:
>
> On the same subject of 'thread safe' does anyone know if mod_perl
> is being made thread safe? ( I believe there was a project for a threaded
> version but it died, so just wondering is the main mod_perl branch was
> being considered for making thread safe. )
>
AFAIK mod_perl *is* thread-safe.
http://perl.apache.org/docs/2.0/user/intro/overview.html#Thread_environment_Issues
Issac
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Timeouts with Threaded Apache 2.2.3
Posted by Neil Martin <ne...@4js.com>.
Hi Sander,
Thanks for the information.
I'll make sure the mod_foo is thread safe, stripped down and retest.
On the same subject of 'thread safe' does anyone know if mod_perl
is being made thread safe? ( I believe there was a project for a threaded
version but it died, so just wondering is the main mod_perl branch was
being considered for making thread safe. )
Regards,
Neil
Sander Temme wrote:
>
> On Feb 8, 2007, at 3:08 AM, Neil Martin wrote:
>
>> so we retested with a dummy module call mod_foo ( attached ) but we
>> still get timeouts.
>> Is this a know issue is the module api in threaded Apache ?
>
>
> You clearly based your mod_foo on mod_example, and just about the only
> thing you deleted was the comment atop the file that warns the
> mod_example code is not thread-safe. You should at least remove the
> call to trace_add() from the handler function, because that uses global
> variables and can't be used in a threaded server.
>
> I'm trying to clean this up for the mod_example.c in our development
> trunk, but this has not been done for 2.2.x.
>
> In a typical module (I think most if not all of your magic is in the
> handler function?), you don't need to implement any of the handlers
> unless you're actually using them to do something. You might override
> child_init to set up your database connection pool, but won't need
> stuff like post_read_request or http_scheme.
>
> You can generate a very small, functional sample module by calling apxs
> -g -n foo
>
> When running your benchmarks, you should make sure that you tune Apache
> in relation to the load you are sending it. The default worker mpm
> configuration tops out at 150 concurrent requests, and you are running
> ab at 200... that may not be a problem given the connection backlog in
> the kernel, but especially if your module takes some time to do its
> database thing, you may run out of resources and render ab confused.
> See conf/extra/httpd-mpm.conf to get an idea of the tunables for the
> worker MPM.
>
> S.
>
--
Regards,
--------------------------------------------------------------------------------
Neil J MARTIN [neilm@4js.com] Tel +44 (0)208 757 5817 Fax +44 (0)208 757 5827
Product Support Engineer - Four J's Development Tools (UK) [www.4js.com]
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Timeouts with Threaded Apache 2.2.3
Posted by Sander Temme <sc...@apache.org>.
On Feb 8, 2007, at 3:08 AM, Neil Martin wrote:
> so we retested with a dummy module call mod_foo ( attached ) but we
> still get timeouts.
> Is this a know issue is the module api in threaded Apache ?
You clearly based your mod_foo on mod_example, and just about the
only thing you deleted was the comment atop the file that warns the
mod_example code is not thread-safe. You should at least remove the
call to trace_add() from the handler function, because that uses
global variables and can't be used in a threaded server.
I'm trying to clean this up for the mod_example.c in our development
trunk, but this has not been done for 2.2.x.
In a typical module (I think most if not all of your magic is in the
handler function?), you don't need to implement any of the handlers
unless you're actually using them to do something. You might override
child_init to set up your database connection pool, but won't need
stuff like post_read_request or http_scheme.
You can generate a very small, functional sample module by calling
apxs -g -n foo
When running your benchmarks, you should make sure that you tune
Apache in relation to the load you are sending it. The default worker
mpm configuration tops out at 150 concurrent requests, and you are
running ab at 200... that may not be a problem given the connection
backlog in the kernel, but especially if your module takes some time
to do its database thing, you may run out of resources and render ab
confused. See conf/extra/httpd-mpm.conf to get an idea of the
tunables for the worker MPM.
S.
--
sctemme@apache.org http://www.temme.net/sander/
PGP FP: 51B4 8727 466A 0BC3 69F4 B7B8 B2BE BC40 1529 24AF