You are viewing a plain text version of this content. The canonical link for it is here.
Posted to modperl@perl.apache.org by da...@ubs.com on 2004/05/19 13:05:39 UTC
FW: Possible Bug: mod_perl version 1 conflict with IBM Websphere Apache plugin
Hi
Perl Version: Any version after 5.005 (Tested on 5.8.0)
mod_perl Version: 1.27
Apache Version: 1.3.27
With the assistance of IBM we may have highlighted a possible bug within mod_perl/perl.
The problem started as we could not load the IBM Websphere 5 Apache plugin if mod_perl was enabled. (On Solaris or Reh-Hat Linux) We narrowed this down to any mod_perl built against perl after 5.005. IBM support assisted us and we managed to enable a truss report that clearly shows the following.
-----------------------------------------------------
Apache server calls WAS plug-in as_init() function twice. In
as_init(), plug-in first checks if a WAS environment variable has been
set. In order to avoid being initialized twice, when as_init() is
first called, plug-in sets the environment variable, and skips the rest
initialization. When as_init() is called again, WAS plug-in will be
initialized.
In this customer's case, the WAS environment set by WAS plug-in was
wiped out by mod_perl. Therefore, when as_init() was called for the
second time, plug-in initialization was skipped again. Since plug-in
was not initialized correctly, pthread mutex was not created, plug-in
log was not opened, and later crash happened when request was passed to
plug-in. If mod_perl had not incorrectly freed the storage that WAS
plug-in used for an environment variable, WAS plug-in initialization
would have completed successfully and the crash on first request would
not have occurred.
Following are related information from truss output:
-> ap_init_modules(0x7f4a0, 0x7f4c8, 0x7c64c, 0xff23c004)
-> mod_app_server_http_eapi_new:as_init(0x7f4c8, 0x7f4a0, 0xfe02c9e0,
0x0)
-> libc:getenv(0xfe072e6c, 0x0, 0x0, 0x0)
<- libc:getenv() = 0
-> libc:putenv(0xd26a0, 0xd26a0, 0x0, 0xc6ebc)
-> libc:malloc(0x94, 0xff23f0e8, 0x0, 0xc6e30)
-> libthread:mutex_lock(0xff240600, 0x7a398, 0x0, 0xff23c004)
<- libthread:mutex_lock() = 0
-> libc:_malloc_unlocked(0x94, 0x0, 0x0, 0xff23c004)
-> libc:cleanfree(0x0, 0xff242844, 0x0, 0xff2427c0)
<- libc:cleanfree() = 0
-> libc:t_delete(0xcf7c0, 0x0, 0xff2427c4, 0xff242844)
-> libc:t_splay(0xcf7c0, 0x798c8, 0x0, 0x0)
<- libc:t_splay() = 0
<- libc:t_delete() = 0xcf7c0
<- libc:_malloc_unlocked() = 0xcf7c8
-> libthread:mutex_unlock(0xff240600, 0xcf7c0, 0xff23c004, 0x98)
<- libthread:mutex_unlock() = 0
<- libc:malloc() = 0xcf7c8
-> libc_psr:memcpy(0xcf7c8, 0xc6e30, 0x94, 0xc6e30)
<- libc_psr:memcpy() = 0xcf7c8
<- libc:putenv() = 0
<- mod_app_server_http_eapi_new:as_init() = 0x7f4c8
-> libperl:perl_module_init(0x7f4c8, 0x7f4a0, 0xfe1730c0, 0x0)
... ...
-> libperl:Perl_safesysfree(0xcf7c8, 0xcf7c8, 0x95600, 0x95600)
<- libperl:Perl_safesysfree() = 0xcf7c8
... ...
<- libperl:perl_module_init() = 0x7f4c8
... ...
<- ap_init_modules() = 0x7f4a0
So cause of segfault on plug-in request is that mod_perl has freed
storage that was allocated by WAS plug-in to act as flag for whether or
not it has initialized so far. We do not know why mod_perl freed the
storage. However, we do see the memory used by WAS plug-in has been
used and freed by mod_perl. It is possible that there is a bug where
mod_perl retained a reference to the storage after freeing it, and
subsequently freed it again.
-----------------------------------------------------
There is an addition part to the truss that might shed light on the problem. To spare you a 500Mb truss file, basically within the mod_perl_boot routines it assigns itself the memory block 0xcf7c8. It then uses this within other modperl routines, but the final line before that block of memory is used again by Websphere is :-
-----------------------------------------------------
-> libperl:Perl_safesysfree(0xcf7c8, 0xcf7c8, 0x4840104, 0x0)
-> libc:free(0xcf7c8, 0x0, 0x0, 0x0)
-> libthread:mutex_lock(0xff240600, 0x79504, 0x0, 0xff23c004)
<- libthread:mutex_lock() = 0
-> libc:_free_unlocked(0xcf7c8, 0x0, 0x0, 0xff23c004)
<- libc:_free_unlocked() = 0xff2427c4
-> libthread:mutex_unlock(0xff240600, 0x0, 0x0, 0x0)
<- libc:free() = 0
<- libperl:Perl_safesysfree() = 0xcf7c8
-----------------------------------------------------
You can see that the Perl_safesysfree reference's 0xcf7c8 with a potential "free" statement. Without knowing the full in's and out's of that routine, it could be that this statement is releasing the memory to the operating system. After that Websphere requests a free memory block and we see the problem that IBM have documented.
*** mod_perl compile info
Build Details
o-----------o
O-S:
SunOS ssyd0045dsc 5.8 Generic_108528-23 sun4u sparc SUNW,Ultra-2
Compiler:
gcc 3.1
Built against:
apache 1.3.27
perl 5.8.0
perl Makefile.PL \
PERL_USELARGEFILES=0 \
INSTALLDIRS=perl \
PREFIX=/sbcimp/run/pd/apache_modules/1.3.27/mod_perl/1.29+perl-
5.8.0 \
USE_APXS=1 WITH_APXS=/sbcimp/run/pd/apache/1.3.27/bin/apxs \
EVERYTHING=1
*** Perl compile info
Summary of my perl5 (revision 5.0 version 8 subversion 0) configuration:
Platform:
osname=solaris, osvers=2.8, archname=sun4-solaris-thread-multi
uname='sunos sldn0037dsc 5.8 generic_108528-12 sun4u sparc sunw,
ultra-enterprise '
config_args='-Dprefix=/sbcimp/build/pd/perl/perl-5.8.0-test -
Dprivlib=/sbcimp/build/pd/perl/perl-5.8.0-test/lib -
Darchlib=/sbcimp/build/pd/perl/perl-5.8.0-test/lib -
Dsitelib=/sbcimp/build/pd/perl/perl-5.8.0-test/lib -
Dsitearch=/sbcimp/build/pd/perl/perl-5.8.0-test/lib -
Dlocincpth=/sbcimp/run/pd/gdbm/1.8.0/lib -Dcc=gcc -Dusethreads -
Uinstallusrbinperl'
hint=recommended, useposix=true, d_sigaction=define
usethreads=define use5005threads=undef useithreads=define
usemultiplicity=define
useperlio=define d_sfio=undef uselargefiles=undef usesocks=undef
use64bitint=undef use64bitall=undef uselongdouble=undef
usemymalloc=n, bincompat5005=undef
Compiler:
cc='gcc', ccflags ='-D_REENTRANT -fno-strict-aliasing -
I/sbcimp/run/pd/gdbm/1.8.0/lib',
optimize='-O',
cppflags='-D_REENTRANT -fno-strict-aliasing -I/sbcimp/run/pd/gdbm/1.
8.0/lib'
ccversion='', gccversion='3.1', gccosandvers='solaris2.8'
intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=4321
d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t',
lseeksize=4
alignbytes=8, prototype=define
Linker and Libraries:
ld='gcc', ldflags =''
libpth=/usr/lib /usr/ccs/lib
libs=-lsocket -lnsl -ldl -lm -lrt -lpthread -lc
perllibs=-lsocket -lnsl -ldl -lm -lrt -lpthread -lc
libc=/lib/libc.so, so=so, useshrplib=false, libperl=libperl.a
gnulibc_version=''
Dynamic Linking:
dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags=' '
cccdlflags='-fPIC', lddlflags='-G'
Rgds
Darren Heather
UBS Invesment Bank
Visit our website at http://www.ubs.com
This message contains confidential information and is intended only
for the individual named. If you are not the named addressee you
should not disseminate, distribute or copy this e-mail. Please
notify the sender immediately by e-mail if you have received this
e-mail by mistake and delete this e-mail from your system.
E-mail transmission cannot be guaranteed to be secure or error-free
as information could be intercepted, corrupted, lost, destroyed,
arrive late or incomplete, or contain viruses. The sender therefore
does not accept liability for any errors or omissions in the contents
of this message which arise as a result of e-mail transmission. If
verification is required please request a hard-copy version. This
message is provided for informational purposes and should not be
construed as a solicitation or offer to buy or sell any securities or
related financial instruments.
--
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html
Re: FW: Possible Bug: mod_perl version 1 conflict with IBM Websphere
Apache plugin
Posted by Stas Bekman <st...@stason.org>.
darren.heather@ubs.com wrote:
> Hi
>
> Perl Version: Any version after 5.005 (Tested on 5.8.0)
> mod_perl Version: 1.27
> Apache Version: 1.3.27
>
> With the assistance of IBM we may have highlighted a possible bug within mod_perl/perl.
>
> The problem started as we could not load the IBM Websphere 5 Apache plugin if mod_perl was enabled. (On Solaris or Reh-Hat Linux) We narrowed this down to any mod_perl built against perl after 5.005. IBM support assisted us and we managed to enable a truss report that clearly shows the following.
>
> -----------------------------------------------------
> Apache server calls WAS plug-in as_init() function twice. In
> as_init(), plug-in first checks if a WAS environment variable has been
> set. In order to avoid being initialized twice, when as_init() is
> first called, plug-in sets the environment variable, and skips the rest
> initialization. When as_init() is called again, WAS plug-in will be
> initialized.
Unrelated to the bug report, how does it deal with restart? It'll run
as_init() with full initialization again, since env var will be set. Why not
doing the init on the very first start, if you really want to call it once.
> In this customer's case, the WAS environment set by WAS plug-in was
> wiped out by mod_perl. Therefore, when as_init() was called for the
> second time, plug-in initialization was skipped again. Since plug-in
> was not initialized correctly, pthread mutex was not created, plug-in
> log was not opened, and later crash happened when request was passed to
> plug-in. If mod_perl had not incorrectly freed the storage that WAS
> plug-in used for an environment variable, WAS plug-in initialization
> would have completed successfully and the crash on first request would
> not have occurred.
>
> Following are related information from truss output:
>
> -> ap_init_modules(0x7f4a0, 0x7f4c8, 0x7c64c, 0xff23c004)
> -> mod_app_server_http_eapi_new:as_init(0x7f4c8, 0x7f4a0, 0xfe02c9e0,
> 0x0)
>
>
>
> -> libc:getenv(0xfe072e6c, 0x0, 0x0, 0x0)
> <- libc:getenv() = 0
> -> libc:putenv(0xd26a0, 0xd26a0, 0x0, 0xc6ebc)
>
> -> libc:malloc(0x94, 0xff23f0e8, 0x0, 0xc6e30)
> -> libthread:mutex_lock(0xff240600, 0x7a398, 0x0, 0xff23c004)
> <- libthread:mutex_lock() = 0
> -> libc:_malloc_unlocked(0x94, 0x0, 0x0, 0xff23c004)
> -> libc:cleanfree(0x0, 0xff242844, 0x0, 0xff2427c0)
> <- libc:cleanfree() = 0
> -> libc:t_delete(0xcf7c0, 0x0, 0xff2427c4, 0xff242844)
> -> libc:t_splay(0xcf7c0, 0x798c8, 0x0, 0x0)
> <- libc:t_splay() = 0
> <- libc:t_delete() = 0xcf7c0
> <- libc:_malloc_unlocked() = 0xcf7c8
> -> libthread:mutex_unlock(0xff240600, 0xcf7c0, 0xff23c004, 0x98)
> <- libthread:mutex_unlock() = 0
> <- libc:malloc() = 0xcf7c8
> -> libc_psr:memcpy(0xcf7c8, 0xc6e30, 0x94, 0xc6e30)
> <- libc_psr:memcpy() = 0xcf7c8
> <- libc:putenv() = 0
> <- mod_app_server_http_eapi_new:as_init() = 0x7f4c8
> -> libperl:perl_module_init(0x7f4c8, 0x7f4a0, 0xfe1730c0, 0x0)
> ... ...
> -> libperl:Perl_safesysfree(0xcf7c8, 0xcf7c8, 0x95600, 0x95600)
> <- libperl:Perl_safesysfree() = 0xcf7c8
> ... ...
> <- libperl:perl_module_init() = 0x7f4c8
> ... ...
> <- ap_init_modules() = 0x7f4a0
Confusingly mod_perl 1's shared lib has the same name as perl's shared lib
libperl.so. So libperl:Perl_safesysfree() is not a mod_perl call but a perl's
one. mod_perl.c has this code:
/* Force the environment to be copied out of its original location
above argv[]. This fixes a crash caused when a module called putenv()
before any Perl modified the environment - environ would change to a
new value, and the check in my_setenv() to duplicate the environment
would fail, and then setting some environment value which had a previous
value would cause perl to try to free() something from the original env.
This crashed free(). */
my_setenv("MODPERL_ENV_FIXUP", "0");
my_setenv("MODPERL_ENV_FIXUP", NULL);
my_setenv is Perl_my_setenv which you can find in util.c in the perl source.
It performs a full copy of the environ data the first time it's used unless it
thinks that it's safe to use putenv. So may be the problem is coming from there.
BTW, I don't remember such a problem being ever reported. I could be wrong though.
--
__________________________________________________________________
Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide ---> http://perl.apache.org
mailto:stas@stason.org http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org http://ticketmaster.com
--
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html