Investigating an httpd deadlock

[John Gotts <jg...@gmail.com>]

Hi,

[NOTE: All sensitive data has been sanitized.]

I am running into an httpd deadlock, where all POSTs are wedging Apache
httpd 2.2.21/mod_perl 2.0.5/Mason 1.47. I recompiled mod_perl with
MP_TRACE=1 on my development host and on our staging host and the error log
on the development host ends with the following:

modperl_perl_ids_get: pid=12399, uid=1000, euid=1000, gid=1000, egid=1000
modperl_callback_run_handlers: no PerlChildInitHandler handlers configured
()

I have a known working staging host running httpd 2.2.11, mod_perl 2.0.4,
and Mason 1.40 which does not deadlock (the POST succeeds). Instead of the
two above lines I see:

modperl_global_lock: locking anon_cnt
modperl_global_unlock: unlocking anon_cnt
modperl_handler_anon_next: [15177] new anon handler: 'anon4'
modperl_handler_anon_add: anonsub 'anon4' added
modperl_env_magic_clear: [15177/0x9ef7340] r->subprocess_env unset:
DBI_AUTOPROXY
modperl_callback_run_handlers: callback 'HTML::Mason::ApacheHandler'
returned 0
modperl_io_perlio_restore_stdin: start
PerlIOApache_close: done with request_rec obj: 0xbfce6e8
PerlIOApache_popped: done
modperl_io_perlio_restore_stdin: restoring STDIN
modperl_io_perlio_restore_stdin: end
modperl_io_perlio_restore_stdout: start
PerlIOApache_flush:    0b []
PerlIOApache_flush:    0b []
PerlIOApache_close: done with request_rec obj: 0xbfce6e8
PerlIOApache_popped: done

and so on.

When I kill the httpd processes on my development host I see a flurry of
activity:

modperl_interp_select: using parent 0x2f01690 for non-threaded mpm (
xxx.xxx.com:0)
modperl_callback_run_handlers: no PerlCleanupHandler handlers configured
(/xxx/xxx.mhtml)
modperl_env_request_unpopulate:
    [12336/0x52878a0/xxx.xxx.com:0/xxx/xxx.mhtml]
    delete @ENV{keys r->subprocess_env};
modperl_env_table_unpopulate: delete $ENV{HTTPS};
modperl_env_table_unpopulate: delete $ENV{SSL_TLS_SNI};
modperl_env_table_unpopulate: delete $ENV{HTTP_HOST};
modperl_env_table_unpopulate: delete $ENV{HTTP_ACCEPT};
modperl_env_table_unpopulate: delete $ENV{HTTP_COOKIE};
modperl_env_table_unpopulate: delete $ENV{CONTENT_LENGTH};
modperl_env_table_unpopulate: delete $ENV{CONTENT_TYPE};
modperl_env_table_unpopulate: delete $ENV{PATH};
modperl_env_table_unpopulate: delete $ENV{SERVER_SIGNATURE};
modperl_env_table_unpopulate: modperl_interp_select: using parent 0x2f01690
for non-threaded mpm (xxx.xxx.com:0)
modperl_callback_run_handlers: no PerlCleanupHandler handlers configured
(/xxx/xxx.mhtml)
modperl_env_request_unpopulate:
    [12335/0x52878a0/xxx.xxx.com:0/xxx/xxx.mhtml]
    delete @ENV{keys r->subprocess_env};
modperl_env_table_unpopulate: delete $ENV{HTTPS};
modperl_env_table_unpopulate: delete $ENV{SSL_TLS_SNI};

and so on.

All configuration files are the same on both hosts. In order to eliminate
any browser interaction, I am performing the POST with the attached php
script.

Any idea where to go from here to figure out this issue?

Thanks for any and all help! This is the first time I've ever witnessed
Apache httpd freeze. Operating system on both hosts is fairly recent Linux
(Ubuntu and Fedora).

John