You are viewing a plain text version of this content. The canonical link for it is here.
Posted to embperl@perl.apache.org by Jarrod Stenberg <sl...@yahoo.com> on 2002/01/08 08:08:56 UTC
Unexplained stalls.
I ask this question only after trying many different
attempts to resolve. On my EmbperlObj driven site,
apache stalls sometimes, resulting in the spawning of
numerous other instances (probably due to the fact
that each client winds up with a stalled httpd server
on this busy site... the two just wait and wait as new
clients spawn their own infinite wait sessions).
The setup:
This began with a simple, standard FreeBSD ports
install, which I think was Apache/1.3.20, mod_perl
1.2.6 and perl 5.005_03. I read some stuff about a
memory leak in pre 5.005_04 perl, and I was seeing
memory leak-like problems so I went ahead and upped
everything. It now runs Apache/1.3.22, mod_perl 1.2.6
and perl v5.6.1. Additionally, I (re)installed all
the latest stuff that was required from CPAN. The
Embperl version is 1.3.4. This runs a database heavy
site using mysql 3.23.42. The server itself is a Soyo
Dragon K7V Athalon 1ghz, 512M DDR memory with 3 7200
RPM ATA drives (one accidently throttled to UDMA 33
due to "non-ATA66 compliant cable" (to be fixed soon).
The original install used a DSO apache, the new one
has everything compiled in.
The upgrades did not really help anything. The memory
climbs all day long from about 8M to 40M for each
instance of apache. And the lockups still occur.
Basically, you get about halfway through a page load,
and then you wait for a long time. If you wait long
enough, it might go away, in fact it often does.
Initially, I thought it was just an efficiency
problem. I tweaked the heck out of mySQL tables and I
believe they are quite tuned. I upped some of the
memory allocations to mysql in addition to the table
tuning. I studied my Embperl code a great deal and
made sure, for example, that every database connection
was definitely very neatly opened, used and closed. I
put some subroutines into a module and load it as an
object so that the code is more lightweight
(consolidating common tasks), e.g. in the init.htm
file: use commonTools; $req->{'ct'} = new commonTools.
In the close.htm file: undef $req->{'ct'}.
Still thought it could be the database. I took a
great deal of the local SQL load off and moved it to a
very nice, dual proc, high memory separate server. I
got into caching just about everything (as I planned
to do, but it had urgency now) so that data is most
often fetched from a file rather than from the
database. I put these cache files in sub-folders by
date and by numeric ids, whichever was appropriate for
the task. So none of these caches are ever found in a
folder with more than about 200 files, often just a
few files or even one. I took these cache files and
divvied them across 3 drives after watching iostat all
day long. This and other stuff increased
responsiveness, but still the lockups remain.
I watch the processlist in mysql when this happens, I
see nothing wrong, no waiting or locking issues. It
may be important to note that I lock while updating a
table. There is no reason these locks wouldn't be
brief, AND the demand for a lock does cause delays on
these update submits as they wait for all reads to
terminate, but these are expected delays. There are
no untidy locks, I'm sure they are all done correctly.
I use "MySQL Semaphore" for EMBPERL_SESSION_CLASSES.
I'm actually suspicious of using Semaphore locking a
bit, but I guess I don't exactly get IPC right now and
so I don't know where to look for trouble. A moment
ago, I moved this to mysql locking. We'll see if this
does it.
I have also had it so that Apache loads most of the
modules ahead of time. I have used Apache::DBI and
have done without (btw, it seems to me that
Apache::DBI and Apache::SessionX are a deadly combo
not to be used together at this point, but that's a
different topic). I switched to
Apache-SessionX-2.00b3 (very handy changes).
I have used Apache::Status to try to find anything
obvious (not sure if I get how this can help so maybe
someone can give advice).
BTW, none of this generates errors. Aside from the
runaway processes it can and does cause, it is a user
perception issue. But it's definitely a real issue
and one that will keep my site from scaling much
further.
So it's two problems: stalls and memory leak-like
behavior. They don't exactly seem related but could
be.
Please, someone, give me any advice you can think of
that may help. I am losing hair.
-Jarrod
P.S. Thanks Gerald and Ed for your previous help.
__________________________________________________
Do You Yahoo!?
Send FREE video emails in Yahoo! Mail!
http://promo.yahoo.com/videomail/
---------------------------------------------------------------------
To unsubscribe, e-mail: embperl-unsubscribe@perl.apache.org
For additional commands, e-mail: embperl-help@perl.apache.org
Re: Unexplained stalls.
Posted by Jarrod Stenberg <sl...@yahoo.com>.
> Maybe somebodyelse on the list has more knowlegde
> here
FTR, I may have solved the problem. We'll see when
the traffic peaks tomorrow. I found some kernel
advice related to IPC and big databases on the FreeBSD
4+ OS:
options MAXDSIZ="(1024*1024*1024)"
options DFLDSIZ="(1024*1024*1024)" # System V
options needed.
options SYSVSHM #SYSV-style shared memory
options SHMMAXPGS=262144 #max amount of shared
mem. pages
options SHMMNI=256 #max number of shared memory
ident if.
options SHMSEG=100 #max shared mem.segs per
process
options SYSVMSG #SYSV-style message queues
options MSGSEG=32767 #max num. of mes.segments in
system
options MSGSSZ=32 #size of msg-seg. MUST be power
of 2
options MSGMNB=65535 #max char. per message queue
options MSGTQL=2046 #max amount of msgs in system
options SYSVSEM #SYSV-style semaphores
options SEMMNU=256 #number of semaphore UNDO
structures
options SEMMNS=1024 #number of semaphores in
system
options SEMMNI=520 #number of semaphore
indentifiers
options SEMUME=100 #number of UNDO keys
These are the semaphore related kernel options (from
LINT):
# System V semaphores and tunable parameters
options SYSVSEM # include support for
semaphores
options SEMMAP=31 # amount of entries in
semaphore map
options SEMMNI=11 # number of semaphore
identifiers in the system
options SEMMNS=61 # number of semaphores
in the system
options SEMMNU=31 # number of undo
structures in the system
options SEMMSL=61 # max number of
semaphores per id
options SEMOPM=101 # max number of
operations per semop call
options SEMUME=11 # max number of undo
entries per process
I still have not raised my NSem setting since I
neglected to put this into the kernel as well:
options SEMMSL=64
which I believe is what I need to do in order to allow
NSems to be anything more than 16. Even so, it seems
to be a lot faster.
Also, I wonder if, since the apache processes can grow
much larger, if these:
options MAXDSIZ="(1024*1024*1024)"
options DFLDSIZ="(1024*1024*1024)"
might fall under highly recommended as FreeBSD will,
by default, cap processes at 128M. I know that's
huge, but so is 40M and I imagine you can get there.
Anyway, thanks again for the attention you gave this
one.
Regards,
-Jarrod
__________________________________________________
Do You Yahoo!?
Send FREE video emails in Yahoo! Mail!
http://promo.yahoo.com/videomail/
---------------------------------------------------------------------
To unsubscribe, e-mail: embperl-unsubscribe@perl.apache.org
For additional commands, e-mail: embperl-help@perl.apache.org
Re: Unexplained stalls.
Posted by Gerald Richter <ri...@ecos.de>.
>
> But I'm also wondering about the relationship with
> jail, or that last setting. I'm looking into this
> independently, but I was wondering if you knew
> anything about this off the top of your head.
>
Sorry, here I have to less knowledge of OS to say anything out of my head.
My Linux 2.4.10 doesn't output anything for
sysctl -a | grep ipc
but has
sysctl -a | grep sem
kernel.sem = 250 256000 32 1024
whatever this means...., but you can try
ipcs -l
this should output the limits in clear text:
------ Shared Memory Limits --------
max number of segments = 4096
max seg size (kbytes) = 32768
max total shared memory (kbytes) = 8388608
min seg size (bytes) = 1
------ Semaphore Limits --------
max number of arrays = 1024
max semaphores per array = 250
max semaphores system wide = 256000
max ops per semop call = 32
semaphore max value = 32767
------ Messages: Limits --------
max queues system wide = 128
max size of message (bytes) = 8192
default max size of queue (bytes) = 16384
Maybe somebodyelse on the list has more knowlegde here
Gerald
-------------------------------------------------------------
Gerald Richter ecos electronic communication services gmbh
Internetconnect * Webserver/-design/-datenbanken * Consulting
Post: Tulpenstrasse 5 D-55276 Dienheim b. Mainz
E-Mail: richter@ecos.de Voice: +49 6133 925131
WWW: http://www.ecos.de Fax: +49 6133 925152
-------------------------------------------------------------
---------------------------------------------------------------------
To unsubscribe, e-mail: embperl-unsubscribe@perl.apache.org
For additional commands, e-mail: embperl-help@perl.apache.org
Re: Unexplained stalls.
Posted by Jarrod Stenberg <sl...@yahoo.com>.
> If it went already away by using mysql locker, you
> could try to increase the
> number of semaphores Apache::Session uses by setting
> NSems=64 or even higher
> in EMBPERL_SESSION_ARGS.
It seems the maximum nsems is 16 on my system. This
caused me to look for where these limits are made:
sysctl -a | grep ipc
kern.ipc.maxsockbuf: 262144
kern.ipc.sockbuf_waste_factor: 8
kern.ipc.somaxconn: 4096
kern.ipc.max_linkhdr: 16
kern.ipc.max_protohdr: 40
kern.ipc.max_hdr: 56
kern.ipc.max_datalen: 156
kern.ipc.nmbclusters: 65536
kern.ipc.semmap: 30
kern.ipc.semmni: 10
kern.ipc.semmns: 60
kern.ipc.semmnu: 30
kern.ipc.semmsl: 60
kern.ipc.semopm: 100
kern.ipc.semume: 10
kern.ipc.semusz: 92
kern.ipc.semvmx: 32767
kern.ipc.semaem: 16384
kern.ipc.shmmax: 33554432
kern.ipc.shmmin: 1
kern.ipc.shmmni: 192
kern.ipc.shmseg: 128
kern.ipc.shmall: 8192
kern.ipc.shm_use_phys: 0
kern.ipc.mbuf_wait: 32
kern.ipc.mbtypes: 829 66 1 0 0 0 0 0 0 0 0 0 0 0 0 0
kern.ipc.nmbufs: 262144
kern.ipc.maxsockets: 16424
ripcb: 192, 16424, 0, 42,
6
jail.sysvipc_allowed: 0
Th obvious setting to choose is kern.ipc.max_linkhdr:
16 since it says 16, after all.
But I'm also wondering about the relationship with
jail, or that last setting. I'm looking into this
independently, but I was wondering if you knew
anything about this off the top of your head.
-Jarrod
__________________________________________________
Do You Yahoo!?
Send FREE video emails in Yahoo! Mail!
http://promo.yahoo.com/videomail/
---------------------------------------------------------------------
To unsubscribe, e-mail: embperl-unsubscribe@perl.apache.org
For additional commands, e-mail: embperl-help@perl.apache.org
Re: Unexplained stalls.
Posted by Gerald Richter <ri...@ecos.de>.
> This began with a simple, standard FreeBSD ports
> install, which I think was Apache/1.3.20, mod_perl
> 1.2.6 and perl 5.005_03. I read some stuff about a
> memory leak in pre 5.005_04 perl,
If you have this from the Embperl docs, then 5.004_04 is meant (not
5.005_04), but installing 5.6.1 is ok anyway.
>
> The upgrades did not really help anything. The memory
> climbs all day long from about 8M to 40M for each
> instance of apache.
If you have many Embperl pages/Perl modules then 40M per childs is ok. All
the Perl code is precompiled and held in memory, so the processes grow huge,
but when every page has been hit once and every Perl module is loaded in a
child, then the childs memory should not grow anymore. Because you have
preloaded Perl modules some of this memory is shared between childs.
> And the lockups still occur.
Since you are sure your database locking is correct, I would suspect the
session locking.
You could try to use the Null locker. If the lockups went away, we know it's
the session handling and we have to track this down further. If the lockups
still occur it has nothing to do with session handling and we have to search
another direction.
If it went already away by using mysql locker, you could try to increase the
number of semaphores Apache::Session uses by setting NSems=64 or even higher
in EMBPERL_SESSION_ARGS.
Gerald
-------------------------------------------------------------
Gerald Richter ecos electronic communication services gmbh
Internetconnect * Webserver/-design/-datenbanken * Consulting
Post: Tulpenstrasse 5 D-55276 Dienheim b. Mainz
E-Mail: richter@ecos.de Voice: +49 6133 925131
WWW: http://www.ecos.de Fax: +49 6133 925152
-------------------------------------------------------------
---------------------------------------------------------------------
To unsubscribe, e-mail: embperl-unsubscribe@perl.apache.org
For additional commands, e-mail: embperl-help@perl.apache.org
Re: Unexplained stalls.
Posted by Gerald Richter <ri...@ecos.de>.
>
> while debugging lockups in a file/semaphore setup, I was very
> suspicious of the semaphore hashing done by
> Apache::Session::Lock::Semaphore.
>
> since there are a limited number of semaphores, Lock::Semaphore does a
> hash on the session id to work out which semaphore to use.
> (if you want to use > 16 (32 on linux) semaphores, you'll need to pass
> a larger NSems arg to Lock::Semaphore btw - see the code).
>
> in the lockup cases i was getting, the %udat session id and the
> %mdat file name both hashed to the same semaphore. so when Embperl
> requests a write lock on both sessions ...
>
This really makes sense. There is a possibility that both id's become the
same hash value for the semaphore array and in this case we get a deadlock
This will happen very seldom, so you see it only on a highly loaded site
sometimes and only if you use %mdat and %udat. If you don't use both, you
don't will get this effect, because no locking is done for the unused one.
That's I guess is the reason why nobody reported this problem so far.
I will fix it by using different semaphores in the next release
Gerald
-------------------------------------------------------------
Gerald Richter ecos electronic communication services gmbh
Internetconnect * Webserver/-design/-datenbanken * Consulting
Post: Tulpenstrasse 5 D-55276 Dienheim b. Mainz
E-Mail: richter@ecos.de Voice: +49 6133 925131
WWW: http://www.ecos.de Fax: +49 6133 925152
-------------------------------------------------------------
---------------------------------------------------------------------
To unsubscribe, e-mail: embperl-unsubscribe@perl.apache.org
For additional commands, e-mail: embperl-help@perl.apache.org
Re: Unexplained stalls.
Posted by Angus Lees <gu...@switchonline.com.au>.
On Mon, Jan 07, 2002 at 11:08:56PM -0800, Jarrod Stenberg wrote:
> I use "MySQL Semaphore" for EMBPERL_SESSION_CLASSES.
> I'm actually suspicious of using Semaphore locking a
> bit, but I guess I don't exactly get IPC right now and
> so I don't know where to look for trouble. A moment
> ago, I moved this to mysql locking. We'll see if this
> does it.
if it does fix it, i'd be very interested.
while debugging lockups in a file/semaphore setup, I was very
suspicious of the semaphore hashing done by
Apache::Session::Lock::Semaphore.
since there are a limited number of semaphores, Lock::Semaphore does a
hash on the session id to work out which semaphore to use.
(if you want to use > 16 (32 on linux) semaphores, you'll need to pass
a larger NSems arg to Lock::Semaphore btw - see the code).
in the lockup cases i was getting, the %udat session id and the
%mdat file name both hashed to the same semaphore. so when Embperl
requests a write lock on both sessions ...
i was in a hurry, switched back to file locking (only a devel setup)
and stopped thinking about it at that point.
i presume its ok, otherwise there'd be lots more reports of lockups
using semaphore locking. guess i have to think about it harder ;)
> BTW, none of this generates errors.
(aside)
if its an os lockup issue (or a livelock thats making syscalls),
strace(1)ing the apache process is usually very useful. eg: a
semaphore lockup will show clearly as a process blocking on semop(2).
found a great tip for debugging perl server spins in the section on
"spinning httpds" in mod_perl(3pm) (only use SIGUSR2, not SIGUSR1 -
apache uses SIGUSR1 for graceful restarts, iirc):
spinning httpds
To see where an httpd is "spinning", try adding this
to your script or a startup file:
use Carp ();
$SIG{'USR2'} = sub {
Carp::confess("caught SIGUSR2!");
};
Then issue the command line:
kill -USR2 <spinning_httpd_pid>
> So it's two problems: stalls and memory leak-like
> behavior. They don't exactly seem related but could
> be.
for a graceful way to deal with Embperl's unbounded caching, see
Apache::SizeLimit.
i use this:
<Perl>
use Apache::SizeLimit;
$Apache::SizeLimit::MAX_PROCESS_SIZE = 15*1024; # kb -> 15Mb
$Apache::SizeLimit::CHECK_EVERY_N_REQUESTS = 5;
$PerlCleanupHandler = 'Apache::SizeLimit';
</Perl>
you will probably want to raise MAX_PROCESS_SIZE.
--
- Gus
---------------------------------------------------------------------
To unsubscribe, e-mail: embperl-unsubscribe@perl.apache.org
For additional commands, e-mail: embperl-help@perl.apache.org