You are viewing a plain text version of this content. The canonical link for it is here.
Posted to modperl@perl.apache.org by Marc Gracia <mg...@oasyssoft.com> on 2004/05/14 14:07:31 UTC
Segmentation Fault problem
Hi,
I have some problem that makes me mad for some time.
We just setted up a web farm to support our application that runs
entirely using mod_perl.
Until now we used a traditional apache+vhosts to serve our customers,
but as it became so unadministrable, we started this new sistem to serve
better.
The basic structure is a reverse proxy as a frontend that redirects
requests to a bunch of different machines, each one with a bunch of
"apaches" on differents ports for each customer.
All this little apaches are running as non-root users on ports > 50000,
to protect better one customer of the other.
Well, once all setted up all seemed to go well. But on one page that
uses the Mail::Sendmail module to send an e-mail, the server crashed
with a segmentation fault.
After tracing all we could into all the perl modules, we found that the
server crashed
when Mail::Sendmail tried to open the network socket.
Then we did a little test and setted up a program that just opened a
socket, and once the page are called, the server segfaults...
The same test, works perfect outside mod_perl...
The server is an Fully Updated RedHat 9
custom "WOLK" 2.4 kernel ( 2.4.20-wolk4.9s )
perl-5.8 (first tried with stock redhat. Then I recompiled my own rpm
with no threads, with the same results)
apache-1.3.29 (EAPI+no EXPAT options, tested activating and deactivating
those options, with no success)
mod_ssl
mod_perl 1.29 (I've tested 1.27 and 1.28 also, with the same results)
Then I start debugging apache, to see what would be happening...
gdb httpd
(gdb) run -X
<Click on the fatal page with the Mail::Sendmail>
Program received signal SIGSEGV, Segmentation fault.
0x1555ef8e in do_lookup_versioned () from /lib/ld-linux.so.2
(gdb) where
#0 0x1555ef8e in do_lookup_versioned () from /lib/ld-linux.so.2
#1 0x1555e156 in _dl_lookup_versioned_symbol_internal () from
/lib/ld-linux.so.2
#2 0x15561e03 in fixup () from /lib/ld-linux.so.2
#3 0x15561cc0 in _dl_runtime_resolve () from /lib/ld-linux.so.2
#4 0x156e60a8 in getprotobyname_r@@GLIBC_2.1.2 () from /lib/libc.so.6
#5 0x156e5f5f in getprotobyname () from /lib/libc.so.6
#6 0x15d234eb in Perl_pp_gprotoent () at pp_sys.c:4856
#7 0x15d23299 in Perl_pp_gpbyname () at pp_sys.c:4823
#8 0x15cd08d2 in Perl_runops_debug () at dump.c:1414
#9 0x15c8c54e in S_call_body (myop=0x3fffdc40, is_eval=0) at
perl.c:2069
#10 0x15c8c1fd in Perl_call_sv (sv=0x15d72d54, flags=4) at perl.c:1987
#11 0x157c82ad in perl_call_handler (sv=0x914d048, r=0x985fffc,
args=0x0) at mod_perl.c:1661
#12 0x157c7a90 in perl_run_stacked_handlers (hook=0x914d048
"4?\024\t\001", r=0x985fffc, handlers=0x9121560)
at mod_perl.c:1374
#13 0x157c60da in perl_handler (r=0x9121560) at mod_perl.c:914
#14 0x08054d0c in ap_invoke_handler ()
#15 0x0806b2aa in process_request_internal ()
#16 0x0806b307 in ap_process_request ()
#17 0x08061b5d in child_main ()
#18 0x08061d30 in make_child ()
#19 0x08061eaf in startup_children ()
#20 0x080625a8 in standalone_main ()
#21 0x08062e61 in main ()
#22 0x15602917 in __libc_start_main () from /lib/libc.so.6
(gdb)quit
As the program fails in getprotobyname glibc function, I suppose the
problem is the infamous buggy glibc's of RedHat. or also a
incompatibility with my current WOLK kernel...
The strace confirms that the problem seems related to the /etc/protocols
file (Used by getprotobyname)
# strace -X
...
...
brk(0) = 0x9bdc000
brk(0x9bdd000) = 0x9bdd000
brk(0) = 0x9bdd000
brk(0x9bde000) = 0x9bde000
brk(0) = 0x9bde000
brk(0x9bdf000) = 0x9bdf000
time(NULL) = 1084532294
open("/etc/protocols", O_RDONLY) = 9
fcntl64(9, F_GETFD) = 0
fcntl64(9, F_SETFD, FD_CLOEXEC) = 0
fstat64(9, {st_mode=S_IFREG|0744, st_size=2168, ...}) = 0
old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS,
-1, 0) = 0x1556d000
read(9, "ip\t0\tIP\nicmp\t1\tICMP\t\t\nigmp\t2\tIGM"..., 4096) = 2168
--- SIGSEGV (Segmentation fault) @ 0 (0) ---
+++ killed by SIGSEGV +++
First I supposed a permission problem, as all our apaches runs as normal
users, but changing the permissions of that file did'nt help. Seeing the
strace, seems to open OK the file in all cases.
To discart silly parsing problems i removed all comments and void lines
on that file, with no difference.
I also downgraded glibc to the RedHat default (glibc-2.3.2-11.9), from
the upgraded glibc-2.3.2-27. No success...
I'm stuck on this... I suspect a permissions problem, but due to the
mature of the system (httpd.conf heavily customized realtime based on
username) I can't test it easily. I also suspect on a DNS (Red Hat
caused me some strange problems in previous versions) or multiple
interface problem, (every machine has a public IP and a private IP).
Also can be the custom kernel....
While I'll do some more tests, someone can help me on this?
I'm really desperate...
Re: Segmentation Fault problem
Posted by Stas Bekman <st...@stason.org>.
Marc Gracia (Oasyssoft) wrote:
> On Fri, 2004-05-14 at 20:56, Stas Bekman wrote:
>
>
>
>>There something else you can try though. Force an early resolution of all
>>symbols when the program loads (which is a default behavior for MacOSX and a
>>few other platforms). For perl xs modules you do that by setting env var
>>RTLD_NOW=1. Though that won't work for perl itself. For perl itself (or any
>>other app that links to shared libs) you will need to set env var
>>LD_BIND_NOW=1. Let us know whether that trick has worked.
>
>
> GREAT!!!!! IT WORKED!!!
> I've setted up both variables and VOILA!!
>
> Many many thanks, you've made the most happy man in my town, and maybe
> saved me for being fired (Those machines would be on production 1 month
> ago...:)
Glad to hear that it worked, Marc :) Es muy magnifico ;)
--
__________________________________________________________________
Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide ---> http://perl.apache.org
mailto:stas@stason.org http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org http://ticketmaster.com
--
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html
Re: Segmentation Fault problem
Posted by "Marc Gracia (Oasyssoft)" <mg...@oasyssoft.com>.
On Fri, 2004-05-14 at 20:56, Stas Bekman wrote:
> There something else you can try though. Force an early resolution of all
> symbols when the program loads (which is a default behavior for MacOSX and a
> few other platforms). For perl xs modules you do that by setting env var
> RTLD_NOW=1. Though that won't work for perl itself. For perl itself (or any
> other app that links to shared libs) you will need to set env var
> LD_BIND_NOW=1. Let us know whether that trick has worked.
GREAT!!!!! IT WORKED!!!
I've setted up both variables and VOILA!!
Many many thanks, you've made the most happy man in my town, and maybe
saved me for being fired (Those machines would be on production 1 month
ago...:)
--
Marc Gracia (Oasyssoft) <mg...@oasyssoft.com>
Re: Segmentation Fault problem
Posted by "Marc Gracia (Oasyssoft)" <mg...@oasyssoft.com>.
On Fri, 2004-05-14 at 20:56, Stas Bekman wrote:
> Marc Gracia (Oasyssoft) wrote:
> > Opps.. Sorry after all gdb an strace I forgot to send the perl -V
> >
> > Summary of my perl5 (revision 5.0 version 8 subversion 0) configuration:
> [...]
>
> Thanks.
>
> Have you tried a more recent 5.8.x perl? 5.8.4 is out for quite some time
> already. I'm not sure if it's going to be of help, since it seems to be a
> glibc bug, but if you are already building from scratch, why not go with the
> best available version.
>
I'll try it... But I'll preffer not to modify this system more by now.
> > The staticaly linked perl it's a good idea... but will this hit the
> > memory space of the running httpd proceses? There will be a lot on those
> > machines....
>
> You mean you will have several independent Apache/mod_perl instances running
> on the same machine? If so, yes, then you will use more memory as there will
> be no perl's libperl.so sharing. Though I'm thinking - if you use a static
> perl's libperl.a, but dynamic modperl's libperl.so (which is confusingly
> called in the same way in mp1), you may not be affected at all. I guess that
> doesn't work, since modperl's shared lib gets loaded by Apache, and not
> through ld.so, so the system will not know to reuse the already loaded instance.
> There something else you can try though. Force an early resolution of all
> symbols when the program loads (which is a default behavior for MacOSX and a
> few other platforms). For perl xs modules you do that by setting env var
> RTLD_NOW=1. Though that won't work for perl itself. For perl itself (or any
> other app that links to shared libs) you will need to set env var
> LD_BIND_NOW=1. Let us know whether that trick has worked.
>
I'll try it now, get back to you with the results...
> Also this seems to be an interesting util, which I haven't tried yet.
> http://www.linuxforum.com/man/prelink.8.php
Aha!! You thinked like me, I'm also a fanatic gentoo user, and gentoo
makes use of this utility intensively.
When I suspected a dinamic link resoultion problem I've tried it
(RedHat9 includes the utility, and the latest rpm seems to use it... I
did'nt know this).
Perl libperl and mod_perl libperl gets prelinked OK, but definitively,
is something wrong with the glibc, because prelink "aborts" trying to
prelink it....
And also, I can't recompile the glibc srpms. I'm sure it's because I've
changed the default RedHat kernel for the aggresive "Wolk" kernel, and
there are some incompatibility....
Many thanks for the help, and sorry for this now off-topic post :)
Marc Gracia
Promotion Manager
e-mail: mgracia@bubblesoulmusic.com
tel: +34 675 508 820
fax: +34 938 721 549
http://www.bubblesoulmusic.com
C/Muralla del Carme, 10
08240 Manresa
BARCELONA
SPAIN
Re: Segmentation Fault problem
Posted by Stas Bekman <st...@stason.org>.
Marc Gracia (Oasyssoft) wrote:
> Opps.. Sorry after all gdb an strace I forgot to send the perl -V
>
> Summary of my perl5 (revision 5.0 version 8 subversion 0) configuration:
[...]
Thanks.
Have you tried a more recent 5.8.x perl? 5.8.4 is out for quite some time
already. I'm not sure if it's going to be of help, since it seems to be a
glibc bug, but if you are already building from scratch, why not go with the
best available version.
> The staticaly linked perl it's a good idea... but will this hit the
> memory space of the running httpd proceses? There will be a lot on those
> machines....
You mean you will have several independent Apache/mod_perl instances running
on the same machine? If so, yes, then you will use more memory as there will
be no perl's libperl.so sharing. Though I'm thinking - if you use a static
perl's libperl.a, but dynamic modperl's libperl.so (which is confusingly
called in the same way in mp1), you may not be affected at all. I guess that
doesn't work, since modperl's shared lib gets loaded by Apache, and not
through ld.so, so the system will not know to reuse the already loaded instance.
There something else you can try though. Force an early resolution of all
symbols when the program loads (which is a default behavior for MacOSX and a
few other platforms). For perl xs modules you do that by setting env var
RTLD_NOW=1. Though that won't work for perl itself. For perl itself (or any
other app that links to shared libs) you will need to set env var
LD_BIND_NOW=1. Let us know whether that trick has worked.
Also this seems to be an interesting util, which I haven't tried yet.
http://www.linuxforum.com/man/prelink.8.php
--
__________________________________________________________________
Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide ---> http://perl.apache.org
mailto:stas@stason.org http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org http://ticketmaster.com
--
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html
Re: Segmentation Fault problem
Posted by "Marc Gracia (Oasyssoft)" <mg...@oasyssoft.com>.
Opps.. Sorry after all gdb an strace I forgot to send the perl -V
Summary of my perl5 (revision 5.0 version 8 subversion 0) configuration:
Platform:
osname=linux, osvers=2.4.20-wolk4.9s, archname=i386-linux
uname='linux massiel.oasysasp.loc 2.4.20-wolk4.9s #27 jue abr 8
12:28:33 cest 2004 i686 i686 i386 gnulinux '
config_args='-des -Doptimize=-O2 -g -pipe -march=i686
-Dmyhostname=localhost -Dperladmin=root@localhost -Dcc=gcc -Dcf_by=Red
Hat, Inc. -Dinstallprefix=/usr -Dprefix=/usr -Darchname=i386-linux
-Dvendorprefix=/usr -Dsiteprefix=/usr
-Dotherlibdirs=/usr/lib/perl5/5.8.0 -Duseshrplib -Uusethreads
-Uuseithreads -Duselargefiles -Dd_dosuid -Dd_semctl_semun -Di_db
-Ui_ndbm -Di_gdbm -Di_shadow -Di_syslog -Dman3ext=3pm -Duseperlio
-Dinstallusrbinperl -Ubincompat5005 -Uversiononly -Dpager=/usr/bin/less
-isr'
hint=recommended, useposix=true, d_sigaction=define
usethreads=undef use5005threads=undef useithreads=undef
usemultiplicity=undef
useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
use64bitint=undef use64bitall=undef uselongdouble=undef
usemymalloc=n, bincompat5005=undef
Compiler:
cc='gcc', ccflags ='-DDEBUGGING -fno-strict-aliasing
-I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64
-I/usr/include/gdbm',
optimize='-O2 -g -pipe -march=i686',
cppflags='-DDEBUGGING -fno-strict-aliasing -I/usr/local/include
-I/usr/include/gdbm'
ccversion='', gccversion='3.2.2 20030222 (Red Hat Linux 3.2.2-5)',
gccosandvers=''
intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t',
lseeksize=8
alignbytes=4, prototype=define
Linker and Libraries:
ld='gcc', ldflags =' -L/usr/local/lib'
libpth=/usr/local/lib /lib /usr/lib
libs=-lnsl -lgdbm -ldb -ldl -lm -lc -lcrypt -lutil
perllibs=-lnsl -ldl -lm -lc -lcrypt -lutil
libc=/lib/libc-2.3.2.so, so=so, useshrplib=true, libperl=libperl.so
gnulibc_version='2.3.2'
Dynamic Linking:
dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-rdynamic
-Wl,-rpath,/usr/lib/perl5/5.8.0/i386-linux/CORE'
cccdlflags='-fPIC', lddlflags='-shared -L/usr/local/lib'
Characteristics of this binary (from libperl):
Compile-time options: DEBUGGING USE_LARGE_FILES
Locally applied patches:
MAINT18379
Built under linux
Compiled at May 13 2004 11:19:51
@INC:
/usr/lib/perl5/5.8.0/i386-linux
/usr/lib/perl5/5.8.0
/usr/lib/perl5/site_perl/5.8.0/i386-linux
/usr/lib/perl5/site_perl/5.8.0
/usr/lib/perl5/site_perl
/usr/lib/perl5/vendor_perl/5.8.0/i386-linux
/usr/lib/perl5/vendor_perl/5.8.0
/usr/lib/perl5/vendor_perl
/usr/lib/perl5/5.8.0/i386-linux
/usr/lib/perl5/5.8.0
The staticaly linked perl it's a good idea... but will this hit the
memory space of the running httpd proceses? There will be a lot on those
machines....
Many many thanks,
On Fri, 2004-05-14 at 20:03, Stas Bekman wrote:
> Marc Gracia wrote:
> > Hi,
> > I have some problem that makes me mad for some time.
> > We just setted up a web farm to support our application that runs
> > entirely using mod_perl.
> > Until now we used a traditional apache+vhosts to serve our customers,
> > but as it became so unadministrable, we started this new sistem to serve
> > better.
> > The basic structure is a reverse proxy as a frontend that redirects
> > requests to a bunch of different machines, each one with a bunch of
> > "apaches" on differents ports for each customer.
> > All this little apaches are running as non-root users on ports > 50000,
> > to protect better one customer of the other.
> >
> > Well, once all setted up all seemed to go well. But on one page that
> > uses the Mail::Sendmail module to send an e-mail, the server crashed
> > with a segmentation fault.
> > After tracing all we could into all the perl modules, we found that the
> > server crashed
> > when Mail::Sendmail tried to open the network socket.
> > Then we did a little test and setted up a program that just opened a
> > socket, and once the page are called, the server segfaults...
> > The same test, works perfect outside mod_perl...
> >
> > The server is an Fully Updated RedHat 9
> > custom "WOLK" 2.4 kernel ( 2.4.20-wolk4.9s )
> > perl-5.8 (first tried with stock redhat. Then I recompiled my own rpm
> > with no threads, with the same results)
> > apache-1.3.29 (EAPI+no EXPAT options, tested activating and deactivating
> > those options, with no success)
> > mod_ssl
> > mod_perl 1.29 (I've tested 1.27 and 1.28 also, with the same results)
> >
> > Then I start debugging apache, to see what would be happening...
> >
> > gdb httpd
> > (gdb) run -X
> > <Click on the fatal page with the Mail::Sendmail>
> >
> > Program received signal SIGSEGV, Segmentation fault.
> > 0x1555ef8e in do_lookup_versioned () from /lib/ld-linux.so.2
> > (gdb) where
> > #0 0x1555ef8e in do_lookup_versioned () from /lib/ld-linux.so.2
> > #1 0x1555e156 in _dl_lookup_versioned_symbol_internal () from
> > /lib/ld-linux.so.2
> > #2 0x15561e03 in fixup () from /lib/ld-linux.so.2
> > #3 0x15561cc0 in _dl_runtime_resolve () from /lib/ld-linux.so.2
> > #4 0x156e60a8 in getprotobyname_r@@GLIBC_2.1.2 () from /lib/libc.so.6
> > #5 0x156e5f5f in getprotobyname () from /lib/libc.so.6
> > #6 0x15d234eb in Perl_pp_gprotoent () at pp_sys.c:4856
> > #7 0x15d23299 in Perl_pp_gpbyname () at pp_sys.c:4823
> > #8 0x15cd08d2 in Perl_runops_debug () at dump.c:1414
> > #9 0x15c8c54e in S_call_body (myop=0x3fffdc40, is_eval=0) at
> > perl.c:2069
> > #10 0x15c8c1fd in Perl_call_sv (sv=0x15d72d54, flags=4) at perl.c:1987
> > #11 0x157c82ad in perl_call_handler (sv=0x914d048, r=0x985fffc,
> > args=0x0) at mod_perl.c:1661
>
> As you can see, the segfault doesn't happen in the mod_perl land. So I'm
> afraid we can't do much about it. I'm not sure why it happens only under
> mod_perl. Perhaps because it's running under Apache -- as you can see the
> segfault is happening when it tries to dynamically resolve the symbol to use
> the thread-safe implementation of getprotobyname (that's the _r suffix). Try a
> static build of perl, it should have all the symbols resolved at the linking
> time. You didn't show us your 'perl -V' so I can't tell what kind of perl
> build you are using.
>
> I've googled a bit, you can see similar reports in other projects:
>
> http://groups.google.ca/groups?selm=bel0bm%241hbe%241%40FreeBSD.csie.NCTU.edu.tw&output=gplain
> http://zebra.fh-weingarten.de/~maxi/html/transcode-users/2003-09/msg00030.html
Marc Gracia
Promotion Manager
e-mail: mgracia@bubblesoulmusic.com
tel: +34 675 508 820
fax: +34 938 721 549
http://www.bubblesoulmusic.com
C/Muralla del Carme, 10
08240 Manresa
BARCELONA
SPAIN
Re: Segmentation Fault problem
Posted by Stas Bekman <st...@stason.org>.
Marc Gracia wrote:
> Hi,
> I have some problem that makes me mad for some time.
> We just setted up a web farm to support our application that runs
> entirely using mod_perl.
> Until now we used a traditional apache+vhosts to serve our customers,
> but as it became so unadministrable, we started this new sistem to serve
> better.
> The basic structure is a reverse proxy as a frontend that redirects
> requests to a bunch of different machines, each one with a bunch of
> "apaches" on differents ports for each customer.
> All this little apaches are running as non-root users on ports > 50000,
> to protect better one customer of the other.
>
> Well, once all setted up all seemed to go well. But on one page that
> uses the Mail::Sendmail module to send an e-mail, the server crashed
> with a segmentation fault.
> After tracing all we could into all the perl modules, we found that the
> server crashed
> when Mail::Sendmail tried to open the network socket.
> Then we did a little test and setted up a program that just opened a
> socket, and once the page are called, the server segfaults...
> The same test, works perfect outside mod_perl...
>
> The server is an Fully Updated RedHat 9
> custom "WOLK" 2.4 kernel ( 2.4.20-wolk4.9s )
> perl-5.8 (first tried with stock redhat. Then I recompiled my own rpm
> with no threads, with the same results)
> apache-1.3.29 (EAPI+no EXPAT options, tested activating and deactivating
> those options, with no success)
> mod_ssl
> mod_perl 1.29 (I've tested 1.27 and 1.28 also, with the same results)
>
> Then I start debugging apache, to see what would be happening...
>
> gdb httpd
> (gdb) run -X
> <Click on the fatal page with the Mail::Sendmail>
>
> Program received signal SIGSEGV, Segmentation fault.
> 0x1555ef8e in do_lookup_versioned () from /lib/ld-linux.so.2
> (gdb) where
> #0 0x1555ef8e in do_lookup_versioned () from /lib/ld-linux.so.2
> #1 0x1555e156 in _dl_lookup_versioned_symbol_internal () from
> /lib/ld-linux.so.2
> #2 0x15561e03 in fixup () from /lib/ld-linux.so.2
> #3 0x15561cc0 in _dl_runtime_resolve () from /lib/ld-linux.so.2
> #4 0x156e60a8 in getprotobyname_r@@GLIBC_2.1.2 () from /lib/libc.so.6
> #5 0x156e5f5f in getprotobyname () from /lib/libc.so.6
> #6 0x15d234eb in Perl_pp_gprotoent () at pp_sys.c:4856
> #7 0x15d23299 in Perl_pp_gpbyname () at pp_sys.c:4823
> #8 0x15cd08d2 in Perl_runops_debug () at dump.c:1414
> #9 0x15c8c54e in S_call_body (myop=0x3fffdc40, is_eval=0) at
> perl.c:2069
> #10 0x15c8c1fd in Perl_call_sv (sv=0x15d72d54, flags=4) at perl.c:1987
> #11 0x157c82ad in perl_call_handler (sv=0x914d048, r=0x985fffc,
> args=0x0) at mod_perl.c:1661
As you can see, the segfault doesn't happen in the mod_perl land. So I'm
afraid we can't do much about it. I'm not sure why it happens only under
mod_perl. Perhaps because it's running under Apache -- as you can see the
segfault is happening when it tries to dynamically resolve the symbol to use
the thread-safe implementation of getprotobyname (that's the _r suffix). Try a
static build of perl, it should have all the symbols resolved at the linking
time. You didn't show us your 'perl -V' so I can't tell what kind of perl
build you are using.
I've googled a bit, you can see similar reports in other projects:
http://groups.google.ca/groups?selm=bel0bm%241hbe%241%40FreeBSD.csie.NCTU.edu.tw&output=gplain
http://zebra.fh-weingarten.de/~maxi/html/transcode-users/2003-09/msg00030.html
--
__________________________________________________________________
Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide ---> http://perl.apache.org
mailto:stas@stason.org http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org http://ticketmaster.com
--
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html