You are viewing a plain text version of this content. The canonical link for it is here.
Posted to modperl@perl.apache.org by Jason Aubrey <au...@gmail.com> on 2012/10/03 15:45:51 UTC
reloading perl modules under heavy load
Hi All,
We have modperl application running on 64bit (2 cores), freebsd 8.2, perl
1.14.1, modperl 2.0.5, apache 2.2.21, 24GB of ram and about 5.5k users.
Additional system details are below. Under heavy load we are seeing errors
of the form
Failed to evaluate module Parser::Legacy: Attempt to reload
> Parser/Legacy.pm aborted.\nCompilation failed in require at (eval 15635)
> line 1....
I vaguely suspect this is a concurrency issue. Requests that generate such
errors often, but not always, call to GD.pm to build dynamically generated
images. The errors often, but not always, refer to Parser::Legacy a module
which is part of our application, but I think this is a red-herring since
it only occurs under load, and I've seen such errors refer to other
modules, for example DateTime.
My server is the largest production freebsd machine running this
application; most servers of this size run Red Hat or Ubuntu and as far as
I can tell nobody has seen this error before.
I honestly have no idea if this is a modperl issue, an apache issue, a
freebsd issue, etc. But I thought I would start here since this is where
the rubber (requests) meet the road (perl). Has anyone see such an error
before? Or know of a similar case that might give me some additional clues?
Thanks very much,
Jason Aubrey
OS:
FreeBSD 8.2-RELEASE-p10
Perl:
modperl 2.0.5
Platform:
osname=freebsd, osvers=8.2-release-p2, archname=amd64-freebsd
uname='freebsd ww64 8.2-release-p2 freebsd 8.2-release-p2 #0: sat sep
17 13:09:25 cdt 2011 root@ww64:usrobjusrsrcsysvmware amd64 '
config_args='-sde -Dprefix=/usr/local
-Darchlib=/usr/local/lib/perl5/5.14.1/mach
-Dprivlib=/usr/local/lib/perl5/5.14.1
-Dman3dir=/usr/local/lib/perl5/5.14.1/perl/man/man3
-Dman1dir=/usr/local/man/man1
-Dsitearch=/usr/local/lib/perl5/site_perl/5.14.1/mach
-Dsitelib=/usr/local/lib/perl5/site_perl/5.14.1 -Dscriptdir=/usr/local/bin
-Dsiteman3dir=/usr/local/lib/perl5/5.14.1/man/man3
-Dsiteman1dir=/usr/local/man/man1 -Ui_malloc -Ui_iconv -Uinstallusrbinperl
-Dcc=cc -Duseshrplib -Dinc_version_list=none
-Dccflags=-DAPPLLIB_EXP="/usr/local/lib/perl5/5.14.1/BSDPAN" -Doptimize=-O2
-fno-strict-aliasing -pipe -march=native -Ui_gdbm -Dusethreads=n
-Dusemymalloc=y -Duse64bitint -Dusesitecustomize'
hint=recommended, useposix=true, d_sigaction=define
useithreads=undef, usemultiplicity=undef
useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
use64bitint=define, use64bitall=define, uselongdouble=undef
usemymalloc=y, bincompat5005=undef
Compiler:
cc='cc', ccflags ='-DAPPLLIB_EXP="/usr/local/lib/perl5/5.14.1/BSDPAN"
-DHAS_FPSETMASK -DHAS_FLOATINGPOINT_H -fno-strict-aliasing -pipe
-fstack-protector -I/usr/local/include',
optimize='-O2 -fno-strict-aliasing -pipe -march=native',
cppflags='-DAPPLLIB_EXP="/usr/local/lib/perl5/5.14.1/BSDPAN"
-DHAS_FPSETMASK -DHAS_FLOATINGPOINT_H -fno-strict-aliasing -pipe
-fstack-protector -I/usr/local/include'
ccversion='', gccversion='4.2.1 20070719 [FreeBSD]', gccosandvers=''
intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t',
lseeksize=8
alignbytes=8, prototype=define
Linker and Libraries:
ld='cc', ldflags =' -Wl,-E -fstack-protector -L/usr/local/lib'
libpth=/usr/lib /usr/local/lib
libs=-lm -lcrypt -lutil
perllibs=-lm -lcrypt -lutil
libc=, so=so, useshrplib=true, libperl=libperl.so
gnulibc_version=''
Dynamic Linking:
dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='
-Wl,-R/usr/local/lib/perl5/5.14.1/mach/CORE'
cccdlflags='-DPIC -fPIC', lddlflags='-shared -L/usr/local/lib
-fstack-protector'
Characteristics of this binary (from libperl):
Compile-time options: MYMALLOC PERL_DONT_CREATE_GVSV PERL_MALLOC_WRAP
PERL_PRESERVE_IVUV USE_64_BIT_ALL USE_64_BIT_INT
USE_LARGE_FILES USE_PERLIO USE_PERL_ATOF
USE_SITECUSTOMIZE
Built under freebsd
Compiled at Sep 17 2011 19:35:32
Apache:
Server version: Apache/2.2.21 (FreeBSD)
Server built: Sep 17 2011 20:41:36
Server's Module Magic Number: 20051115:30
Server loaded: APR 1.4.5, APR-Util 1.3.12
Compiled using: APR 1.4.5, APR-Util 1.3.12
Architecture: 64-bit
Server MPM: Prefork
threaded: no
forked: yes (variable process count)
Server compiled with....
-D APACHE_MPM_DIR="server/mpm/prefork"
-D APR_HAS_SENDFILE
-D APR_HAS_MMAP
-D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
-D APR_USE_FLOCK_SERIALIZE
-D APR_USE_PTHREAD_SERIALIZE
-D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
-D APR_HAS_OTHER_CHILD
-D AP_HAVE_RELIABLE_PIPED_LOGS
-D DYNAMIC_MODULE_LIMIT=128
-D HTTPD_ROOT="/usr/local"
-D SUEXEC_BIN="/usr/local/bin/suexec"
-D DEFAULT_PIDLOG="/var/run/httpd.pid"
-D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
-D DEFAULT_LOCKFILE="/var/run/accept.lock"
-D DEFAULT_ERRORLOG="logs/error_log"
-D AP_TYPES_CONFIG_FILE="etc/apache22/mime.types"
-D SERVER_CONFIG_FILE="etc/apache22/httpd.conf"
RE: reloading perl modules under heavy load
Posted by j....@jobtarget.com.
Hi - Are you loading that at startup? Is it necessary to reload the module while running, or can you load it once? We had something similar that looked like random loading problems on Debian about 3 years back. When I switched to put all of the use statements in a startup.pl and configured apache to run that at launch, the problem went away.
Jonathan L Brown
Senior Software Engineer
JOBTARGET
225 State Street, Suite 300
New London, CT 06320
Ph: (810) 938-7115
Email:j.brown@jobtarget.com
Web: www.JobTarget.com<http://www.jobtarget.com/>
[image001]
As I usually am offsite, please call my mobile number in the event of emergency.
From: Jason Aubrey [mailto:aubreyja@gmail.com]
Sent: Wednesday, October 03, 2012 9:46 AM
To: modperl@perl.apache.org
Subject: reloading perl modules under heavy load
Hi All,
We have modperl application running on 64bit (2 cores), freebsd 8.2, perl 1.14.1, modperl 2.0.5, apache 2.2.21, 24GB of ram and about 5.5k users. Additional system details are below. Under heavy load we are seeing errors of the form
Failed to evaluate module Parser::Legacy: Attempt to reload Parser/Legacy.pm aborted.\nCompilation failed in require at (eval 15635) line 1....
I vaguely suspect this is a concurrency issue. Requests that generate such errors often, but not always, call to GD.pm to build dynamically generated images. The errors often, but not always, refer to Parser::Legacy a module which is part of our application, but I think this is a red-herring since it only occurs under load, and I've seen such errors refer to other modules, for example DateTime.
My server is the largest production freebsd machine running this application; most servers of this size run Red Hat or Ubuntu and as far as I can tell nobody has seen this error before.
I honestly have no idea if this is a modperl issue, an apache issue, a freebsd issue, etc. But I thought I would start here since this is where the rubber (requests) meet the road (perl). Has anyone see such an error before? Or know of a similar case that might give me some additional clues?
Thanks very much,
Jason Aubrey
OS:
FreeBSD 8.2-RELEASE-p10
Perl:
modperl 2.0.5
Platform:
osname=freebsd, osvers=8.2-release-p2, archname=amd64-freebsd
uname='freebsd ww64 8.2-release-p2 freebsd 8.2-release-p2 #0: sat sep 17 13:09:25 cdt 2011 root@ww64:usrobjusrsrcsysvmware amd64 '
config_args='-sde -Dprefix=/usr/local -Darchlib=/usr/local/lib/perl5/5.14.1/mach -Dprivlib=/usr/local/lib/perl5/5.14.1 -Dman3dir=/usr/local/lib/perl5/5.14.1/perl/man/man3 -Dman1dir=/usr/local/man/man1 -Dsitearch=/usr/local/lib/perl5/site_perl/5.14.1/mach -Dsitelib=/usr/local/lib/perl5/site_perl/5.14.1 -Dscriptdir=/usr/local/bin -Dsiteman3dir=/usr/local/lib/perl5/5.14.1/man/man3 -Dsiteman1dir=/usr/local/man/man1 -Ui_malloc -Ui_iconv -Uinstallusrbinperl -Dcc=cc -Duseshrplib -Dinc_version_list=none -Dccflags=-DAPPLLIB_EXP="/usr/local/lib/perl5/5.14.1/BSDPAN" -Doptimize=-O2 -fno-strict-aliasing -pipe -march=native -Ui_gdbm -Dusethreads=n -Dusemymalloc=y -Duse64bitint -Dusesitecustomize'
hint=recommended, useposix=true, d_sigaction=define
useithreads=undef, usemultiplicity=undef
useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
use64bitint=define, use64bitall=define, uselongdouble=undef
usemymalloc=y, bincompat5005=undef
Compiler:
cc='cc', ccflags ='-DAPPLLIB_EXP="/usr/local/lib/perl5/5.14.1/BSDPAN" -DHAS_FPSETMASK -DHAS_FLOATINGPOINT_H -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include',
optimize='-O2 -fno-strict-aliasing -pipe -march=native',
cppflags='-DAPPLLIB_EXP="/usr/local/lib/perl5/5.14.1/BSDPAN" -DHAS_FPSETMASK -DHAS_FLOATINGPOINT_H -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include'
ccversion='', gccversion='4.2.1 20070719 [FreeBSD]', gccosandvers=''
intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
alignbytes=8, prototype=define
Linker and Libraries:
ld='cc', ldflags =' -Wl,-E -fstack-protector -L/usr/local/lib'
libpth=/usr/lib /usr/local/lib
libs=-lm -lcrypt -lutil
perllibs=-lm -lcrypt -lutil
libc=, so=so, useshrplib=true, libperl=libperl.so
gnulibc_version=''
Dynamic Linking:
dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags=' -Wl,-R/usr/local/lib/perl5/5.14.1/mach/CORE'
cccdlflags='-DPIC -fPIC', lddlflags='-shared -L/usr/local/lib -fstack-protector'
Characteristics of this binary (from libperl):
Compile-time options: MYMALLOC PERL_DONT_CREATE_GVSV PERL_MALLOC_WRAP
PERL_PRESERVE_IVUV USE_64_BIT_ALL USE_64_BIT_INT
USE_LARGE_FILES USE_PERLIO USE_PERL_ATOF
USE_SITECUSTOMIZE
Built under freebsd
Compiled at Sep 17 2011 19:35:32
Apache:
Server version: Apache/2.2.21 (FreeBSD)
Server built: Sep 17 2011 20:41:36
Server's Module Magic Number: 20051115:30
Server loaded: APR 1.4.5, APR-Util 1.3.12
Compiled using: APR 1.4.5, APR-Util 1.3.12
Architecture: 64-bit
Server MPM: Prefork
threaded: no
forked: yes (variable process count)
Server compiled with....
-D APACHE_MPM_DIR="server/mpm/prefork"
-D APR_HAS_SENDFILE
-D APR_HAS_MMAP
-D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
-D APR_USE_FLOCK_SERIALIZE
-D APR_USE_PTHREAD_SERIALIZE
-D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
-D APR_HAS_OTHER_CHILD
-D AP_HAVE_RELIABLE_PIPED_LOGS
-D DYNAMIC_MODULE_LIMIT=128
-D HTTPD_ROOT="/usr/local"
-D SUEXEC_BIN="/usr/local/bin/suexec"
-D DEFAULT_PIDLOG="/var/run/httpd.pid"
-D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
-D DEFAULT_LOCKFILE="/var/run/accept.lock"
-D DEFAULT_ERRORLOG="logs/error_log"
-D AP_TYPES_CONFIG_FILE="etc/apache22/mime.types"
-D SERVER_CONFIG_FILE="etc/apache22/httpd.conf"
Re: reloading perl modules under heavy load
Posted by Randolf Richardson <ra...@modperl.pl>.
> Hi All,
>
> We have modperl application running on 64bit (2 cores), freebsd 8.2, perl
> 1.14.1, modperl 2.0.5, apache 2.2.21, 24GB of ram and about 5.5k users.
> Additional system details are below. Under heavy load we are seeing errors
> of the form
>
> Failed to evaluate module Parser::Legacy: Attempt to reload
> > Parser/Legacy.pm aborted.\nCompilation failed in require at (eval 15635)
> > line 1....
>
>
> I vaguely suspect this is a concurrency issue. Requests that generate such
> errors often, but not always, call to GD.pm to build dynamically generated
> images. The errors often, but not always, refer to Parser::Legacy a module
> which is part of our application, but I think this is a red-herring since
> it only occurs under load, and I've seen such errors refer to other
> modules, for example DateTime.
>
> My server is the largest production freebsd machine running this
> application; most servers of this size run Red Hat or Ubuntu and as far as
> I can tell nobody has seen this error before.
>
> I honestly have no idea if this is a modperl issue, an apache issue, a
> freebsd issue, etc. But I thought I would start here since this is where
> the rubber (requests) meet the road (perl). Has anyone see such an error
> before? Or know of a similar case that might give me some additional clues?
I normally use Reload on non-production/test servers where there
isn't a heavy load as I've encountered problems like this in the past
as well. On a test server I can reproduce this error by updating
scripts and then reloading the page many times quickly (this is most
easily reproduced where the Windows OS acts as a server as it tends
to get lost and even crash to the point of requiring a full OS
reboot; I never have this problem with Unix or Linux which seem to
isolate processes from the inner workings of the OS, although I have
been able to reproduce the script reload problem on NetBSD which is
the OS that I use for my development and production servers).
I suspect that you're right about it being a concurrency issue, and
when more scripts/modules have to be reloaded there also seems to be
a slightly greater instance of this problem (perhaps, I assume,
because more time is required to dynamically compile more code which
opens further opportunity for the [suspected] concurrency issue).
Since development/test servers are far more likely to see regular
updates to code than production servers, it seems reasonable to me
that issuing a simple "/etc/rc.d/apache restart" command is a fitting
solution for production systems which also appeals to administrators
who desire/need more control. Although having the automatic
reloading even on production servers could also be convenient, there
is a slight advantage with optimization in not having scripts
constantly checked for changes to determine if they need to be
reloaded, particularly in high volume scenarios (e.g., when the web
site gets slash-dotted).
[End of reply.]
> Thanks very much,
> Jason Aubrey
>
> OS:
>
> FreeBSD 8.2-RELEASE-p10
>
>
> Perl:
> modperl 2.0.5
>
> Platform:
> osname=freebsd, osvers=8.2-release-p2, archname=amd64-freebsd
> uname='freebsd ww64 8.2-release-p2 freebsd 8.2-release-p2 #0: sat sep
> 17 13:09:25 cdt 2011 root@ww64:usrobjusrsrcsysvmware amd64 '
> config_args='-sde -Dprefix=/usr/local
> -Darchlib=/usr/local/lib/perl5/5.14.1/mach
> -Dprivlib=/usr/local/lib/perl5/5.14.1
> -Dman3dir=/usr/local/lib/perl5/5.14.1/perl/man/man3
> -Dman1dir=/usr/local/man/man1
> -Dsitearch=/usr/local/lib/perl5/site_perl/5.14.1/mach
> -Dsitelib=/usr/local/lib/perl5/site_perl/5.14.1 -Dscriptdir=/usr/local/bin
> -Dsiteman3dir=/usr/local/lib/perl5/5.14.1/man/man3
> -Dsiteman1dir=/usr/local/man/man1 -Ui_malloc -Ui_iconv -Uinstallusrbinperl
> -Dcc=cc -Duseshrplib -Dinc_version_list=none
> -Dccflags=-DAPPLLIB_EXP="/usr/local/lib/perl5/5.14.1/BSDPAN" -Doptimize=-O2
> -fno-strict-aliasing -pipe -march=native -Ui_gdbm -Dusethreads=n
> -Dusemymalloc=y -Duse64bitint -Dusesitecustomize'
> hint=recommended, useposix=true, d_sigaction=define
> useithreads=undef, usemultiplicity=undef
> useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
> use64bitint=define, use64bitall=define, uselongdouble=undef
> usemymalloc=y, bincompat5005=undef
> Compiler:
> cc='cc', ccflags ='-DAPPLLIB_EXP="/usr/local/lib/perl5/5.14.1/BSDPAN"
> -DHAS_FPSETMASK -DHAS_FLOATINGPOINT_H -fno-strict-aliasing -pipe
> -fstack-protector -I/usr/local/include',
> optimize='-O2 -fno-strict-aliasing -pipe -march=native',
> cppflags='-DAPPLLIB_EXP="/usr/local/lib/perl5/5.14.1/BSDPAN"
> -DHAS_FPSETMASK -DHAS_FLOATINGPOINT_H -fno-strict-aliasing -pipe
> -fstack-protector -I/usr/local/include'
> ccversion='', gccversion='4.2.1 20070719 [FreeBSD]', gccosandvers=''
> intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
> d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
> ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t',
> lseeksize=8
> alignbytes=8, prototype=define
> Linker and Libraries:
> ld='cc', ldflags =' -Wl,-E -fstack-protector -L/usr/local/lib'
> libpth=/usr/lib /usr/local/lib
> libs=-lm -lcrypt -lutil
> perllibs=-lm -lcrypt -lutil
> libc=, so=so, useshrplib=true, libperl=libperl.so
> gnulibc_version=''
> Dynamic Linking:
> dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='
> -Wl,-R/usr/local/lib/perl5/5.14.1/mach/CORE'
> cccdlflags='-DPIC -fPIC', lddlflags='-shared -L/usr/local/lib
> -fstack-protector'
>
>
> Characteristics of this binary (from libperl):
> Compile-time options: MYMALLOC PERL_DONT_CREATE_GVSV PERL_MALLOC_WRAP
> PERL_PRESERVE_IVUV USE_64_BIT_ALL USE_64_BIT_INT
> USE_LARGE_FILES USE_PERLIO USE_PERL_ATOF
> USE_SITECUSTOMIZE
> Built under freebsd
> Compiled at Sep 17 2011 19:35:32
>
> Apache:
>
> Server version: Apache/2.2.21 (FreeBSD)
> Server built: Sep 17 2011 20:41:36
> Server's Module Magic Number: 20051115:30
> Server loaded: APR 1.4.5, APR-Util 1.3.12
> Compiled using: APR 1.4.5, APR-Util 1.3.12
> Architecture: 64-bit
> Server MPM: Prefork
> threaded: no
> forked: yes (variable process count)
> Server compiled with....
> -D APACHE_MPM_DIR="server/mpm/prefork"
> -D APR_HAS_SENDFILE
> -D APR_HAS_MMAP
> -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
> -D APR_USE_FLOCK_SERIALIZE
> -D APR_USE_PTHREAD_SERIALIZE
> -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
> -D APR_HAS_OTHER_CHILD
> -D AP_HAVE_RELIABLE_PIPED_LOGS
> -D DYNAMIC_MODULE_LIMIT=128
> -D HTTPD_ROOT="/usr/local"
> -D SUEXEC_BIN="/usr/local/bin/suexec"
> -D DEFAULT_PIDLOG="/var/run/httpd.pid"
> -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
> -D DEFAULT_LOCKFILE="/var/run/accept.lock"
> -D DEFAULT_ERRORLOG="logs/error_log"
> -D AP_TYPES_CONFIG_FILE="etc/apache22/mime.types"
> -D SERVER_CONFIG_FILE="etc/apache22/httpd.conf"
>
Randolf Richardson - randolf@inter-corporate.com
Inter-Corporate Computer & Network Services, Inc.
Beautiful British Columbia, Canada
http://www.inter-corporate.com/
Re: reloading perl modules under heavy load
Posted by Andy Colson <an...@squeakycode.net>.
On 10/3/2012 8:45 AM, Jason Aubrey wrote:
> Hi All,
>
> We have modperl application running on 64bit (2 cores), freebsd 8.2,
> perl 1.14.1, modperl 2.0.5, apache 2.2.21, 24GB of ram and about 5.5k
> users. Additional system details are below. Under heavy load we are
> seeing errors of the form
>
> Failed to evaluate module Parser::Legacy: Attempt to reload
> Parser/Legacy.pm aborted.\nCompilation failed in require at (eval
> 15635) line 1....
>
>
> I vaguely suspect this is a concurrency issue. Requests that generate
> such errors often, but not always, call to GD.pm to build dynamically
> generated images. The errors often, but not always, refer to
> Parser::Legacy a module which is part of our application, but I think
> this is a red-herring since it only occurs under load, and I've seen
> such errors refer to other modules, for example DateTime.
>
> My server is the largest production freebsd machine running this
> application; most servers of this size run Red Hat or Ubuntu and as far
> as I can tell nobody has seen this error before.
>
> I honestly have no idea if this is a modperl issue, an apache issue, a
> freebsd issue, etc. But I thought I would start here since this is where
> the rubber (requests) meet the road (perl). Has anyone see such an
> error before? Or know of a similar case that might give me some
> additional clues?
>
> Thanks very much,
> Jason Aubrey
>
Are you including Apache2::Reload anyplace? I use it on my dev box and
get messages like that quite often. I don't use it on my production
box. To update production, I 'svn up; apache restart' sort of thing.
-Andy
Re: Modperl support for Apache 2.4
Posted by "Brett @Google" <br...@gmail.com>.
There is already a fork for this :
http://svn.apache.org/repos/asf/perl/modperl/branches/httpd24
I have not tried it, personally.. but many people have.
On Sat, Oct 6, 2012 at 2:02 AM, Hong Ye <ho...@cornell.edu> wrote:
> Hi,
>
> Does anyone know when mod_perl will be available to download for Apache
> 2.4?
>
> Thanks,
>
> Hong
>
>
--
*The only thing that interferes with my learning is my education.*
*
Albert Einstein*
Modperl support for Apache 2.4
Posted by Hong Ye <ho...@cornell.edu>.
Hi,
Does anyone know when mod_perl will be available to download for Apache 2.4?
Thanks,
Hong
Re: reloading perl modules under heavy load
Posted by Perrin Harkins <pe...@elem.com>.
On Thu, Oct 4, 2012 at 10:36 AM, Jason Aubrey <au...@gmail.com> wrote:
> Now, we have MaxClients set super high
I hope you don't have it set so high that if you hit it you would run
out of memory and start swapping...
> Also possibly relevant is
> kern.ipc.somaxconn which "limits the size of the listen queue for accepting
> new TCP connections". We had that at the default of 128. I also don't
> understand the interaction between this setting and apache's MaxClients
> since when we had kern.ipc.somaxconn = 128 and MaxClients 150 we quickly
> maxed out max clients.
The gist of it is that when you do hit MaxClients, the people trying
to get in are waiting in the listen queue. It's like overflow for
MaxClients. People don't just get sent away when MaxClients is hit.
I can't tell you what a sane value for FreeBSD on your hardware is
though.
- Perrin
Re: reloading perl modules under heavy load
Posted by Jason Aubrey <au...@gmail.com>.
On Thu, Oct 4, 2012 at 9:53 AM, demerphq <de...@gmail.com> wrote:
> On 4 October 2012 16:36, Jason Aubrey <au...@gmail.com> wrote:
> > Thanks all for your replies to my question.
> >
> > Because of the nature of our application, we can't really load
> everything at
> > start up, but I did some digging and there are clearly some
> inefficiencies
> > here and the situation would indeed be improved by cleaning these up.
> > However, it does look like it may be the interaction between these
> > inefficiencies and freebsd in particular that is causing the error.
> >
> > So, here's what I did: on a linux development box I wrote a script using
> > Linux::Inotify2 to watch the directories containing the code for our
> > application that we (think we) have to load at run time. A typical
> request
> > generated an average of 225 IN_OPEN inotify events. (FWIW, it also
> > generated around 850 IN_ACCESS events. I don't understand the
> difference,
> > but IN_OPEN events seem more relevant to what I'm getting to.)
> >
> > Now, we have MaxClients set super high, but with just 150 concurrent
> > requests that's 33,750 open files and that's not taking into account
> files
> > opened for other processes on the machine, such as the kernel or the web
> > server, etc.
> >
> > But, here's where freebsd comes in. The freebsd kernel has a bunch of
> > tunables, one of which is kern.maxfiles. The freebsd manual says "This
> > variable indicates the maximum number of file descriptors on your
> system...
> > Each open file, socket, or fifo uses one file descriptor. A large-scale
> > production server may easily require many thousands of file descriptors,
> > depending on the kind and number of services running concurrently"
> >
> > Well, we had kern.maxfiles = 12328. Also possibly relevant is
> > kern.ipc.somaxconn which "limits the size of the listen queue for
> accepting
> > new TCP connections". We had that at the default of 128. I also don't
> > understand the interaction between this setting and apache's MaxClients
> > since when we had kern.ipc.somaxconn = 128 and MaxClients 150 we quickly
> > maxed out max clients.
> >
> > So, we bumped up kern.maxfiles to 50000 and kern.ipc.somaxconn to 8192
> and
> > things appeared to be better last night under heavy load. But, as I said,
> > there are clearly some in efficiencies too in the way our application
> loads
> > code at run time, and cleaning these up would certainly help.
>
> You would also see reduced memory pressure and performance
> improvements. Perl is actually pretty slow at compiling code (in
> comparison to executing code), and code that is compiled prefork is
> going to be shared, whereas code compiled post-fork is going to be
> duplicated per process.
>
> By preloading you will both win performance improvements and reduce
> memory pressure on your application.
>
> I strongly recommend you approach this subject aggressively.
>
> yves
>
Hi all,
Here's another update which you will be glad to hear because it proves you
are awesome (or at least that you were right). The freebsd tuning I
mentioned below helped but didn't fix the problem.
We still got a bunch of such errors after this tuning (fewer, but still a
lot). Well, I added just one file to the list of code compiled in at start
up and since doing so we have had exactly zero such errors. Admittedly,
that file pulls in a large package, but this single 8-character change to
one line of code seems to have had a tremendously positive impact on the
performance of our application.
Thanks again for all of of your help,
Jason
Re: reloading perl modules under heavy load
Posted by demerphq <de...@gmail.com>.
On 4 October 2012 16:36, Jason Aubrey <au...@gmail.com> wrote:
> Thanks all for your replies to my question.
>
> Because of the nature of our application, we can't really load everything at
> start up, but I did some digging and there are clearly some inefficiencies
> here and the situation would indeed be improved by cleaning these up.
> However, it does look like it may be the interaction between these
> inefficiencies and freebsd in particular that is causing the error.
>
> So, here's what I did: on a linux development box I wrote a script using
> Linux::Inotify2 to watch the directories containing the code for our
> application that we (think we) have to load at run time. A typical request
> generated an average of 225 IN_OPEN inotify events. (FWIW, it also
> generated around 850 IN_ACCESS events. I don't understand the difference,
> but IN_OPEN events seem more relevant to what I'm getting to.)
>
> Now, we have MaxClients set super high, but with just 150 concurrent
> requests that's 33,750 open files and that's not taking into account files
> opened for other processes on the machine, such as the kernel or the web
> server, etc.
>
> But, here's where freebsd comes in. The freebsd kernel has a bunch of
> tunables, one of which is kern.maxfiles. The freebsd manual says "This
> variable indicates the maximum number of file descriptors on your system...
> Each open file, socket, or fifo uses one file descriptor. A large-scale
> production server may easily require many thousands of file descriptors,
> depending on the kind and number of services running concurrently"
>
> Well, we had kern.maxfiles = 12328. Also possibly relevant is
> kern.ipc.somaxconn which "limits the size of the listen queue for accepting
> new TCP connections". We had that at the default of 128. I also don't
> understand the interaction between this setting and apache's MaxClients
> since when we had kern.ipc.somaxconn = 128 and MaxClients 150 we quickly
> maxed out max clients.
>
> So, we bumped up kern.maxfiles to 50000 and kern.ipc.somaxconn to 8192 and
> things appeared to be better last night under heavy load. But, as I said,
> there are clearly some in efficiencies too in the way our application loads
> code at run time, and cleaning these up would certainly help.
You would also see reduced memory pressure and performance
improvements. Perl is actually pretty slow at compiling code (in
comparison to executing code), and code that is compiled prefork is
going to be shared, whereas code compiled post-fork is going to be
duplicated per process.
By preloading you will both win performance improvements and reduce
memory pressure on your application.
I strongly recommend you approach this subject aggressively.
yves
--
perl -Mre=debug -e "/just|another|perl|hacker/"
Re: reloading perl modules under heavy load
Posted by Jason Aubrey <au...@gmail.com>.
Thanks all for your replies to my question.
Because of the nature of our application, we can't really load everything
at start up, but I did some digging and there are clearly some
inefficiencies here and the situation would indeed be improved by cleaning
these up. However, it does look like it may be the interaction between
these inefficiencies and freebsd in particular that is causing the error.
So, here's what I did: on a linux development box I wrote a script using
Linux::Inotify2 to watch the directories containing the code for our
application that we (think we) have to load at run time. A typical request
generated an average of 225 IN_OPEN inotify events. (FWIW, it also
generated around 850 IN_ACCESS events. I don't understand the difference,
but IN_OPEN events seem more relevant to what I'm getting to.)
Now, we have MaxClients set super high, but with just 150 concurrent
requests that's 33,750 open files and that's not taking into account files
opened for other processes on the machine, such as the kernel or the web
server, etc.
But, here's where freebsd comes in. The freebsd kernel has a bunch of
tunables, one of which is kern.maxfiles. The freebsd manual says "This
variable indicates the maximum number of file descriptors on your system...
Each open file, socket, or fifo uses one file descriptor. A large-scale
production server may easily require many thousands of file descriptors,
depending on the kind and number of services running concurrently"
Well, we had kern.maxfiles = 12328. Also possibly relevant is
kern.ipc.somaxconn
which "limits the size of the listen queue for accepting new TCP
connections". We had that at the default of 128. I also don't understand
the interaction between this setting and apache's MaxClients since when we
had kern.ipc.somaxconn = 128 and MaxClients 150 we quickly maxed out max
clients.
So, we bumped up kern.maxfiles to 50000 and kern.ipc.somaxconn to 8192 and
things appeared to be better last night under heavy load. But, as I said,
there are clearly some in efficiencies too in the way our application loads
code at run time, and cleaning these up would certainly help.
If you're interested, here is a discussion from a free-bsd mailing list
yesterday about low kernel limits that a colleague forwarded to me:
http://goo.gl/Dpzre
Thanks again for all of your replies,
Jason
On Wed, Oct 3, 2012 at 12:48 PM, Perrin Harkins <pe...@elem.com> wrote:
> On Wed, Oct 3, 2012 at 9:45 AM, Jason Aubrey <au...@gmail.com> wrote:
> > Hi All,
> >
> > We have modperl application running on 64bit (2 cores), freebsd 8.2,
> perl
> > 1.14.1, modperl 2.0.5, apache 2.2.21, 24GB of ram and about 5.5k users.
> > Additional system details are below. Under heavy load we are seeing
> errors
> > of the form
> >
> >> Failed to evaluate module Parser::Legacy: Attempt to reload
> >> Parser/Legacy.pm aborted.\nCompilation failed in require at (eval 15635)
> >> line 1....
>
> I remember seeing this kind of thing when I was trying to use
> ActiveState's mod_perl-ish thing with IIS years ago. It would fail
> to find things like strict.pm when it was under load. I guessed the
> same thing: some sort of concurrency issue. I don't have a solution
> to offer though, since I gave up on that one and used CGI.
>
> - Perrin
>
Re: reloading perl modules under heavy load
Posted by Perrin Harkins <pe...@elem.com>.
On Wed, Oct 3, 2012 at 9:45 AM, Jason Aubrey <au...@gmail.com> wrote:
> Hi All,
>
> We have modperl application running on 64bit (2 cores), freebsd 8.2, perl
> 1.14.1, modperl 2.0.5, apache 2.2.21, 24GB of ram and about 5.5k users.
> Additional system details are below. Under heavy load we are seeing errors
> of the form
>
>> Failed to evaluate module Parser::Legacy: Attempt to reload
>> Parser/Legacy.pm aborted.\nCompilation failed in require at (eval 15635)
>> line 1....
I remember seeing this kind of thing when I was trying to use
ActiveState's mod_perl-ish thing with IIS years ago. It would fail
to find things like strict.pm when it was under load. I guessed the
same thing: some sort of concurrency issue. I don't have a solution
to offer though, since I gave up on that one and used CGI.
- Perrin