You are viewing a plain text version of this content. The canonical link for it is here.
Posted to modperl@perl.apache.org by Stefan Cars <st...@snowfall.se> on 2004/02/20 14:51:27 UTC
MPM worker and ithreads
Hi!
Anyone that has any good links or any good input on using the worker MPM
and ithreads with mod_perl 2 and Apache 2
Kind Regards,
Stefan Cars
--
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html
Re: MPM worker and ithreads
Posted by Stas Bekman <st...@stason.org>.
Richard F. Rebel wrote:
> Hello,
>
> I am using ap2 + mp2 with the worker mpm and ithreads to serve about 12
> million dynamically generated pages per day. This includes RPC calls
> via HTTP and database queries. I am currently experimenting with
> creating pools of threads inside the perl interpreter for handling
> communicating with multiple remote machines simultaniously.
>
> My project would not be possible using prefork without a huge increase
> in hardware expense (the reason we tried using worker mpm).
>
> Performance is good on decent hardware, sub second page generation
> times. I have noticed that for my app, prefork was a smidgen faster.
Thanks Richard. We are always looking for success stories, if you wish to
contribute one. Please see:
http://perl.apache.org/outstanding/index.html
__________________________________________________________________
Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide ---> http://perl.apache.org
mailto:stas@stason.org http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org http://ticketmaster.com
--
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html
poor man's interpreter accounting (Was Re: MPM worker and ithreads)
Posted by Stas Bekman <st...@stason.org>.
Stas Bekman wrote:
[...]
>> Great hack Stas, but alas, I suspect(all right, I *know*) that there are
>> interpreters that are getting reaped in my situation. Is there a
>> similar magic BLOCK for gc like the 'CLONE' convention?
>
>
> I think the END block is run for every clone, I'll write a test later.
I was wrong, it doesn't. It runs only by the main interpreter (I'm talking
pure perl here, it's not special to mod_perl).
Here is a package that will do the accounting for you. I've used a dummy
object to use its DESTROY method to emulate END for cloned interpreters. All
you need is to load this package at the server startup and look at your
error_log during the server life and at its very end.
Here is the package:
# My/InterpreterCounter.pm
package My::InterpreterCounter;
use strict;
use warnings FATAL => 'all';
use threads;
use threads::shared;
use subs qw(say);
# a special object created in the parent interpreter which will call
# DESTROY when each interpreter goes down, providing the END
# equivalent for spawned ithreads
#
# we also use it to track the thread id, while we have it
my $obj = My::InterpreterCounter->new();
my $ctr : shared = &share({});
# 1 is the parent interpreter which already exists
$ctr->{cnt} = 1;
$ctr->{max} = 1;
$ctr->{tot} = 1;
sub new {
my $class = shift;
my $self = 0;
return bless \$self, $class;
}
sub CLONE {
my $tid = threads->self->tid;
say "a cloned interpreter #$tid was spawned";
$$obj = $tid;
lock $ctr;
$ctr->{tot}++;
$ctr->{cnt}++;
$ctr->{max}++ if $ctr->{cnt} > $ctr->{max};
status();
}
sub DESTROY {
my $self = shift;
my $tid = $$self;
lock $ctr;
$ctr->{cnt}--;
say "a cloned interpreter #$tid went down";
status();
}
sub END {
say "the main interpreter goes down";
status();
}
sub status {
lock $ctr;
printf STDERR " " x 9 .
"total: $ctr->{tot}, count $ctr->{cnt}, at most $ctr->{max}\n";
}
sub say {
(my $caller = (caller(1))[3]) =~ s/.*:://;
printf STDERR "%-7s: %s\n", $caller, join '', @_;
}
1;
here is a standalone program that requires no mod_perl. As you can see it does
nothing to the package besides loading it *before* it spawns any new threads:
#test.pl
use My::InterpreterCounter;
use threads;
use threads::shared;
for (0..1) {
my $thr1 = threads->new(\&worker);
my $thr2 = threads->new(\&worker);
$thr1->join;
$thr2->join;
}
sub worker {
my $tid = threads->self->tid;
#print STDERR "TID is $tid\n";
}
Running it:
% perl -I. test.pl
CLONE : a cloned interpreter #1 was spawned
total: 2, count 2, at most 2
CLONE : a cloned interpreter #2 was spawned
total: 3, count 3, at most 3
DESTROY: a cloned interpreter #2 went down
total: 3, count 2, at most 3
DESTROY: a cloned interpreter #1 went down
total: 3, count 1, at most 3
CLONE : a cloned interpreter #3 was spawned
total: 4, count 2, at most 3
CLONE : a cloned interpreter #4 was spawned
total: 5, count 3, at most 3
DESTROY: a cloned interpreter #4 went down
total: 5, count 2, at most 3
DESTROY: a cloned interpreter #3 went down
total: 5, count 1, at most 3
END : the main interpreter goes down
total: 5, count 1, at most 3
DESTROY: a cloned interpreter #0 went down
total: 5, count 0, at most 3
So you can see that during the program life, at any given time there were at
most 3 perl interpreters running (1 parent + 2 clones). And you can see that
there were a total of 4 clones started (plus one parent perl).
So I loaded this module from modperl-2.0/t/conf/modperl_extra.pl and run the
threads tests (this is just a preforked mpm which spawn ithreads from the test):
t/TEST -v perl/ithreads
the error_log had:
CLONE : a cloned interpreter #1 was spawned
total: 2, count 2, at most 2
Attempt to free unreferenced scalar: SV 0x9842900 during global destruction.
DESTROY: a cloned interpreter #1 went down
total: 2, count 1, at most 2
CLONE : a cloned interpreter #2 was spawned
total: 3, count 2, at most 2
Attempt to free unreferenced scalar: SV 0x99de3f4 during global destruction.
DESTROY: a cloned interpreter #2 went down
total: 3, count 1, at most 2
So, as you can see the test has spawned two clones, but only one was active at
any given time. Indeed looking at modperl-2.0/t/response/TestPerl/ithreads.pm
you can see that the first ithread was joined before a new one was spawned.
Give it a try under a worker mpm and see if you get a nice report in a nice
progress and the totals at the shutdown.
__________________________________________________________________
Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide ---> http://perl.apache.org
mailto:stas@stason.org http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org http://ticketmaster.com
--
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html
Re: MPM worker and ithreads
Posted by Stas Bekman <st...@stason.org>.
Richard F. Rebel wrote:
> On Mon, 2004-02-23 at 20:34, Stas Bekman wrote:
>
>>Richard F. Rebel wrote:
>>
>>
>>>Speaking of, does anyone know of a way to tell exactly how many
>>>interpreters are running in a given process?
>>
>>Sounds like a job for Apache::Status. Though it will need an API to query the
>>interpreter pools which AFAIK don't exist yet. Or may be it'll better suite
>>Apache::VMonitor http://search.cpan.org/dist/Apache-VMonitor/. At the moment
>>it will show you all the active threads for each process, though it won't tell
>>you which one is perl and which is not.
>>
>>Meanwhile if your interpreters pool is always growing and never get its items
>>killed you can cheat and count the number of started interpreters by defining
>>a function CLONE in some package, e.g. in httpd.conf:
>
>
> Great hack Stas, but alas, I suspect(all right, I *know*) that there are
> interpreters that are getting reaped in my situation. Is there a
> similar magic BLOCK for gc like the 'CLONE' convention?
I think the END block is run for every clone, I'll write a test later.
Really, the API to query the status of interpreter pools would be the best.
> The CLONE special block should be documented somewhere, who shall I rag
> at for that??
It's mentioned in perlmod:
Ithreads work by cloning the data tree so that no data is shared
between different threads. These threads can be used by using the
"threads" module or by doing fork() on win32 (fake fork() support).
When a thread is cloned all Perl data is cloned, however non-Perl data
cannot be cloned automatically. Perl after 5.7.2 has support for the
"CLONE" special subroutine . In "CLONE" you can do whatever you need
to do, like for example handle the cloning of non-Perl data, if neces-
sary. "CLONE" will be executed once for every package that has it
defined (or inherits it). It will be called in the context of the new
thread, so all modifications are made in the new area.
If you want to CLONE all objects you will need to keep track of them
per package. This is simply done using a hash and
Scalar::Util::weaken().
I'm working on a full blown example and docs of where it's indispensible:
http://apache.org/~stas/Example-CLONE-0.02.tar.gz
> And if I( haven't said it b4, thanks for the hard work on mp2...
Thanks for using it! ;)
__________________________________________________________________
Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide ---> http://perl.apache.org
mailto:stas@stason.org http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org http://ticketmaster.com
--
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html
Re: MPM worker and ithreads
Posted by "Richard F. Rebel" <rr...@whenu.com>.
On Mon, 2004-02-23 at 20:34, Stas Bekman wrote:
> Richard F. Rebel wrote:
>
> > Speaking of, does anyone know of a way to tell exactly how many
> > interpreters are running in a given process?
>
> Sounds like a job for Apache::Status. Though it will need an API to query the
> interpreter pools which AFAIK don't exist yet. Or may be it'll better suite
> Apache::VMonitor http://search.cpan.org/dist/Apache-VMonitor/. At the moment
> it will show you all the active threads for each process, though it won't tell
> you which one is perl and which is not.
>
> Meanwhile if your interpreters pool is always growing and never get its items
> killed you can cheat and count the number of started interpreters by defining
> a function CLONE in some package, e.g. in httpd.conf:
Great hack Stas, but alas, I suspect(all right, I *know*) that there are
interpreters that are getting reaped in my situation. Is there a
similar magic BLOCK for gc like the 'CLONE' convention?
The CLONE special block should be documented somewhere, who shall I rag
at for that??
And if I( haven't said it b4, thanks for the hard work on mp2...
> <Perl>
> package My::InterpreterCounter;
> use threads;
> use threads::shared;
> my $counter : shared = 0;
> sub CLONE {
> lock $counter;
> $counter++;
> }
> sub END {
> print STDERR "$counter interpreters were started";
> }
> </Perl>
>
> this is untested... the special function CLONE is called every time a new perl
> interpreter is created (via perl_clone()). It can exist in any package.
>
> __________________________________________________________________
> Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
> http://stason.org/ mod_perl Guide ---> http://perl.apache.org
> mailto:stas@stason.org http://use.perl.org http://apacheweek.com
> http://modperlbook.org http://apache.org http://ticketmaster.com
--
Richard F. Rebel <rr...@whenu.com>
WhenU.com
Re: MPM worker and ithreads
Posted by Stas Bekman <st...@stason.org>.
Richard F. Rebel wrote:
> Speaking of, does anyone know of a way to tell exactly how many
> interpreters are running in a given process?
Sounds like a job for Apache::Status. Though it will need an API to query the
interpreter pools which AFAIK don't exist yet. Or may be it'll better suite
Apache::VMonitor http://search.cpan.org/dist/Apache-VMonitor/. At the moment
it will show you all the active threads for each process, though it won't tell
you which one is perl and which is not.
Meanwhile if your interpreters pool is always growing and never get its items
killed you can cheat and count the number of started interpreters by defining
a function CLONE in some package, e.g. in httpd.conf:
<Perl>
package My::InterpreterCounter;
use threads;
use threads::shared;
my $counter : shared = 0;
sub CLONE {
lock $counter;
$counter++;
}
sub END {
print STDERR "$counter interpreters were started";
}
</Perl>
this is untested... the special function CLONE is called every time a new perl
interpreter is created (via perl_clone()). It can exist in any package.
__________________________________________________________________
Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide ---> http://perl.apache.org
mailto:stas@stason.org http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org http://ticketmaster.com
--
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html
Re: MPM worker and ithreads
Posted by "Richard F. Rebel" <rr...@whenu.com>.
On Mon, 2004-02-23 at 19:48, Perrin Harkins wrote:
> On Mon, 2004-02-23 at 19:37, Richard F. Rebel wrote:
> > Eh well, do I get points for making a prod
> > system run with mp2 and mpm-worker?
>
> Certainly. We are all eager for this kind of info.
Yay, points.
> > Most of our clients are *slow*, so perhaps this is why things seem to
> > work so well.
>
> Actually, if your clients are slow you would be better off with a
> reverse proxy to do buffering.
This makes sense but if the perl interpreters are tied up as you
mentioned below, I'd need a separate proxy. We do this on our general
purpose app servers already so I can do some experimentation and report
back.
> > Now, does this sort of explain why I was seeing more memory usage with
> > prefork, slow clients?
>
> That was because you were running 500 interpreters instead of 50.
Speaking of, does anyone know of a way to tell exactly how many
interpreters are running in a given process?
Again, Thanks!
> > When exactly is the perl interpreter put back
> > into the 'free' list?
>
> More of a question for someone closer to the code than me, but I thought
> that the interpreter was tied up until the server finishes the request,
> meaning that slow clients will keep an interpreter tied up for a long
> time.
>
> - Perrin
--
Richard F. Rebel <rr...@whenu.com>
WhenU.com
Re: MPM worker and ithreads
Posted by Perrin Harkins <pe...@elem.com>.
On Mon, 2004-02-23 at 19:37, Richard F. Rebel wrote:
> Eh well, do I get points for making a prod
> system run with mp2 and mpm-worker?
Certainly. We are all eager for this kind of info.
> Most of our clients are *slow*, so perhaps this is why things seem to
> work so well.
Actually, if your clients are slow you would be better off with a
reverse proxy to do buffering.
> I have increased the number of interpreters and have
> noticed a decrease in cpu usage and an improvement in performance in
> general.
Cool.
> Now, does this sort of explain why I was seeing more memory usage with
> prefork, slow clients?
That was because you were running 500 interpreters instead of 50.
> When exactly is the perl interpreter put back
> into the 'free' list?
More of a question for someone closer to the code than me, but I thought
that the interpreter was tied up until the server finishes the request,
meaning that slow clients will keep an interpreter tied up for a long
time.
- Perrin
--
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html
Re: MPM worker and ithreads
Posted by "Richard F. Rebel" <rr...@whenu.com>.
On Mon, 2004-02-23 at 18:01, Perrin Harkins wrote:
> On Mon, 2004-02-23 at 17:16, Richard F. Rebel wrote:
> > Interesting. I just noticed documentation on the web site about these
> > directives. Were these docs here about 1 year ago (when I wrote this
> > app???).
>
> Those have been there for years. I remember that stuff being in Doug's
> slides before the first alpha release of mp2.
I feel like king dork. Eh well, do I get points for making a prod
system run with mp2 and mpm-worker?
> > I have *no* perlInterp* configuration directives.
>
> Judging by the defaults, it sounds like you have a total of 50 perl
> interpreters then, and all other threads are either serving static files
> or queuing in line for use of an interpreter.
This explains some of the sporadic slow results I get on occasion that
don't seem to be related to anything external. Apache was having to on
occasion start new interpreter threads.
> You would get basically the same effect by running prefork with
> MaxClients set to 50, and a reverse proxy in front. Actually, you could
> probably run a lot more interpreters with prefork, because it is
> currently MORE memory efficient than threads. I guess you gain a bit of
> performance by not going through the proxy, but lose some scalability by
> not having the proxy buffer your output to slow clients.
Most of our clients are *slow*, so perhaps this is why things seem to
work so well. I have increased the number of interpreters and have
noticed a decrease in cpu usage and an improvement in performance in
general. Thanks.
Now, does this sort of explain why I was seeing more memory usage with
prefork, slow clients? When exactly is the perl interpreter put back
into the 'free' list? Does apache spoon feed the slow clients freeing
up the interpreter for other requests after you return from your
handler?
> - Perrin
--
Richard F. Rebel <rr...@whenu.com>
WhenU.com
Re: MPM worker and ithreads
Posted by Perrin Harkins <pe...@elem.com>.
On Mon, 2004-02-23 at 17:16, Richard F. Rebel wrote:
> Interesting. I just noticed documentation on the web site about these
> directives. Were these docs here about 1 year ago (when I wrote this
> app???).
Those have been there for years. I remember that stuff being in Doug's
slides before the first alpha release of mp2.
> I have *no* perlInterp* configuration directives.
Judging by the defaults, it sounds like you have a total of 50 perl
interpreters then, and all other threads are either serving static files
or queuing in line for use of an interpreter.
You would get basically the same effect by running prefork with
MaxClients set to 50, and a reverse proxy in front. Actually, you could
probably run a lot more interpreters with prefork, because it is
currently MORE memory efficient than threads. I guess you gain a bit of
performance by not going through the proxy, but lose some scalability by
not having the proxy buffer your output to slow clients.
- Perrin
--
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html
Re: MPM worker and ithreads
Posted by "Richard F. Rebel" <rr...@whenu.com>.
On Mon, 2004-02-23 at 16:49, Perrin Harkins wrote:
> On Mon, 2004-02-23 at 16:40, Richard F. Rebel wrote:
> > Umm, maybe we are talking about different things. If I run my
> > application with 500+ httpd's in the process list using prefork, it uses
> > a lot more memory than running 10 httpds with 64 threads each using
> > worker. It also gets worse over time (as shared pages get altered).
> >
> > Did I miss something or did I do something wrong?
>
> I'm not sure you did anything wrong, but your results are very different
> from what other people have reported. In general, because perl threads
> don't actually share anything (except opcode trees) unless you tell them
> to, all the data gets copied to each thread, which ends up taking up
> more memory than the prefork model with COW.
Interesting. I just noticed documentation on the web site about these
directives. Were these docs here about 1 year ago (when I wrote this
app???).
> Maybe you have the PerlInterp* settings set low so that you actually
> have fewer interpreters running than you did when you were running
> prefork? If so, that's similar to using prefork with a reverse proxy in
> front.
I have *no* perlInterp* configuration directives.
> - Perrin
--
Richard F. Rebel
rrebel@whenu.com
t. 212.239.0000
Re: MPM worker and ithreads
Posted by Perrin Harkins <pe...@elem.com>.
On Mon, 2004-02-23 at 16:40, Richard F. Rebel wrote:
> Umm, maybe we are talking about different things. If I run my
> application with 500+ httpd's in the process list using prefork, it uses
> a lot more memory than running 10 httpds with 64 threads each using
> worker. It also gets worse over time (as shared pages get altered).
>
> Did I miss something or did I do something wrong?
I'm not sure you did anything wrong, but your results are very different
from what other people have reported. In general, because perl threads
don't actually share anything (except opcode trees) unless you tell them
to, all the data gets copied to each thread, which ends up taking up
more memory than the prefork model with COW.
Maybe you have the PerlInterp* settings set low so that you actually
have fewer interpreters running than you did when you were running
prefork? If so, that's similar to using prefork with a reverse proxy in
front.
- Perrin
--
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html
Re: MPM worker and ithreads
Posted by Nigel Hamilton <ni...@turbo10.com>.
>
> Umm, maybe we are talking about different things. If I run my
> application with 500+ httpd's in the process list using prefork, it uses
> a lot more memory than running 10 httpds with 64 threads each using
> worker. It also gets worse over time (as shared pages get altered).
>
Hi Richard,
If you don't mind me asking how much RAM do you need to run 500
preforked httpds?
Nige
--
Nigel Hamilton
Turbo10 Metasearch Engine
email: nigel@turbo10.com
tel: +44 (0) 207 987 5460
fax: +44 (0) 207 987 5468
________________________________________________________________________________
http://turbo10.com Search Deeper. Browse Faster.
--
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html
Re: MPM worker and ithreads
Posted by "Richard F. Rebel" <rr...@whenu.com>.
On Mon, 2004-02-23 at 16:04, Perrin Harkins wrote:
> On Mon, 2004-02-23 at 10:03, Richard F. Rebel wrote:
> > I am using ap2 + mp2 with the worker mpm and ithreads to serve about 12
> > million dynamically generated pages per day. This includes RPC calls
> > via HTTP and database queries. I am currently experimenting with
> > creating pools of threads inside the perl interpreter for handling
> > communicating with multiple remote machines simultaniously.
> >
> > My project would not be possible using prefork without a huge increase
> > in hardware expense (the reason we tried using worker mpm).
>
> What's the reason for this? Are you saving memory by using the worker
> MPM? I thought the consensus was that perl threads actually use more
> memory than pre-fork at this point.
Hello Perrin,
Umm, maybe we are talking about different things. If I run my
application with 500+ httpd's in the process list using prefork, it uses
a lot more memory than running 10 httpds with 64 threads each using
worker. It also gets worse over time (as shared pages get altered).
Did I miss something or did I do something wrong?
> - Perrin
--
Richard F. Rebel
rrebel@whenu.com
t. 212.239.0000
Re: MPM worker and ithreads
Posted by Perrin Harkins <pe...@elem.com>.
On Mon, 2004-02-23 at 10:03, Richard F. Rebel wrote:
> I am using ap2 + mp2 with the worker mpm and ithreads to serve about 12
> million dynamically generated pages per day. This includes RPC calls
> via HTTP and database queries. I am currently experimenting with
> creating pools of threads inside the perl interpreter for handling
> communicating with multiple remote machines simultaniously.
>
> My project would not be possible using prefork without a huge increase
> in hardware expense (the reason we tried using worker mpm).
What's the reason for this? Are you saving memory by using the worker
MPM? I thought the consensus was that perl threads actually use more
memory than pre-fork at this point.
- Perrin
--
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html
Re: MPM worker and ithreads
Posted by "Richard F. Rebel" <rr...@whenu.com>.
On Mon, 2004-02-23 at 14:05, Stas Bekman wrote:
> Ged Haywood wrote:
> >
> > I see your experience with the -er- people at P5P was similar to mine.
>
> That's very untrue, guys. p5p cares a lot for mod_perl and helps a lot to fix
> things so that mod_perl will be happy. Not once a major release was delayed
> because of problems with mod_perl. I wish httpd-dev cared about mod_perl just
> a tiny fraction of what p5p does.
There is a general attitude to posters on that list, we all know it. I
don't think anyone meant to say they didn't care about mp but rather
that owning up to problems in perl itself is often low on the priority
list. Eg, broken select().
> The issue with ithreads support/maintenance is different: At the moment the
> ithreads package has no maintainer, so any fixes that you see at all are
> spotty. Arthur Bergman has left to work on Ponie (perl5 on parrot
> http://www.poniecode.org/) and noone has stepped up to replace him.
No one mentioned this during my posts.
> So if someone feels like picking up an important sub-project, by all means do
> so. It requires understanding of threads, XS and Perl guts.
Well, that would be the problem... XS + Perl Guts. Talk about arcane.
--
Richard F. Rebel
rrebel@whenu.com
t. 212.239.0000
Re: MPM worker and ithreads
Posted by Stas Bekman <st...@stason.org>.
Ged Haywood wrote:
> Hi there,
>
> On Mon, 23 Feb 2004, Richard F. Rebel wrote:
>
>
>>I am using ap2 + mp2 with the worker mpm and ithreads to serve about 12
>>million dynamically generated pages per day. ... expense ... performance
>>... bugs ... ithreads ... snots on P5P
>
>
> Thanks very much for that report, it's really useful.
>
> I see your experience with the -er- people at P5P was similar to mine.
That's very untrue, guys. p5p cares a lot for mod_perl and helps a lot to fix
things so that mod_perl will be happy. Not once a major release was delayed
because of problems with mod_perl. I wish httpd-dev cared about mod_perl just
a tiny fraction of what p5p does.
The issue with ithreads support/maintenance is different: At the moment the
ithreads package has no maintainer, so any fixes that you see at all are
spotty. Arthur Bergman has left to work on Ponie (perl5 on parrot
http://www.poniecode.org/) and noone has stepped up to replace him.
So if someone feels like picking up an important sub-project, by all means do
so. It requires understanding of threads, XS and Perl guts.
__________________________________________________________________
Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide ---> http://perl.apache.org
mailto:stas@stason.org http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org http://ticketmaster.com
--
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html
Re: MPM worker and ithreads
Posted by Ged Haywood <ge...@jubileegroup.co.uk>.
Hi there,
On Mon, 23 Feb 2004, Richard F. Rebel wrote:
> I am using ap2 + mp2 with the worker mpm and ithreads to serve about 12
> million dynamically generated pages per day. ... expense ... performance
> ... bugs ... ithreads ... snots on P5P
Thanks very much for that report, it's really useful.
I see your experience with the -er- people at P5P was similar to mine.
73,
Ged.
--
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html
Re: MPM worker and ithreads
Posted by "Richard F. Rebel" <rr...@whenu.com>.
Hello,
I am using ap2 + mp2 with the worker mpm and ithreads to serve about 12
million dynamically generated pages per day. This includes RPC calls
via HTTP and database queries. I am currently experimenting with
creating pools of threads inside the perl interpreter for handling
communicating with multiple remote machines simultaniously.
My project would not be possible using prefork without a huge increase
in hardware expense (the reason we tried using worker mpm).
Performance is good on decent hardware, sub second page generation
times. I have noticed that for my app, prefork was a smidgen faster.
That said the following should be said as well:
Make sure (atleast on Linux) that you have set your number of available
threads high enough that requests do not have to wait for a new thread
to be created. Your performance will suck otherwise as new thread
creation is slower than one would expect. This is true for apache2
without mod_perl as well btw but not quite as bad..
Apache2 still has bugs, so does mp2, be careful. I have identified at
at least one run away memory leak that I can't track down and apache2
segfaults on occasion. Isolating such things in busy threaded apps is
*very* difficult.
ithreads in perl are very slow to start and the implementation is a bit
buggy with spotty documentation. Eg, you *can NOT* return a list from a
thread, you may return exactly one scalar ref to whatever, the docs are
*wrong* and the snots on P5P have yet to even admit (atleast to me, I
stopped following the issue a while ago, may be fixed now) that this is
even a bug or a problem.
Best,
Richard F. Rebel
On Sat, 2004-02-21 at 06:07, Stefan Cars wrote:
> Hi!
>
> I don't have any problems, I'm just interested in what people think of
> the performance vs. preforking and if there is any problems using worker
> and ithreads ?
>
> On Fri, 20 Feb 2004, Stas Bekman wrote:
>
> > Stefan Cars wrote:
> > > Hi!
> > >
> > > Anyone that has any good links or any good input on using the worker MPM
> > > and ithreads with mod_perl 2 and Apache 2
> >
> > You could be a bit more specific, Stafan. What kind of problems do you have?
> > You also have to tell us more about your environment, which is done the best
> > by calling mp2bug or t/REPORT
> > http://perl.apache.org/docs/2.0/user/help/help.html#Important_Information
> >
> > __________________________________________________________________
> > Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
> > http://stason.org/ mod_perl Guide ---> http://perl.apache.org
> > mailto:stas@stason.org http://use.perl.org http://apacheweek.com
> > http://modperlbook.org http://apache.org http://ticketmaster.com
> >
>
> --
> Stefan Cars
> Snowfall Communications
> Tel: +46 (0)18 430 80 50 - Direct: +46 (0)18 430 80 51
> Mobile: +46 (0)708 44 36 00 - Fax: +46 (0)708 44 36 04
>
>
>
> ______________________________________________________________________
> SNOWFALL DISCLAIMER:
> The information contained in this email and in any
> attachments is confidential and may be privileged. If you are not the
> intended recipient, please destroy this message and notify the sender
> immediately. You should not retain, copy or use this email for any
> purpose, nor disclose all or any part of its content to any other person.
> Any views expressed in this message are those of the individual sender,
> except where the sender specifically states them to be the views of
> Snowfall Communications.
>
> Snowfall Communications monitors the content of emails sent and received
> via its network for unauthorised use and for other lawful business
> purposes. The contents of an attachment to this email may contain viruses
> which could damage your computer system. While Snowfall Communications has
> taken every reasonable precaution to minimise this risk, we cannot accept
> liability for any damage which you sustain as a result of software
> viruses. You should carry out your own virus checks before opening the
> attachment.
--
Richard F. Rebel
rrebel@whenu.com
t. 212.239.0000
Re: MPM worker and ithreads
Posted by Stefan Cars <st...@snowfall.se>.
Hi!
I don't have any problems, I'm just interested in what people think of
the performance vs. preforking and if there is any problems using worker
and ithreads ?
On Fri, 20 Feb 2004, Stas Bekman wrote:
> Stefan Cars wrote:
> > Hi!
> >
> > Anyone that has any good links or any good input on using the worker MPM
> > and ithreads with mod_perl 2 and Apache 2
>
> You could be a bit more specific, Stafan. What kind of problems do you have?
> You also have to tell us more about your environment, which is done the best
> by calling mp2bug or t/REPORT
> http://perl.apache.org/docs/2.0/user/help/help.html#Important_Information
>
> __________________________________________________________________
> Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
> http://stason.org/ mod_perl Guide ---> http://perl.apache.org
> mailto:stas@stason.org http://use.perl.org http://apacheweek.com
> http://modperlbook.org http://apache.org http://ticketmaster.com
>
--
Stefan Cars
Snowfall Communications
Tel: +46 (0)18 430 80 50 - Direct: +46 (0)18 430 80 51
Mobile: +46 (0)708 44 36 00 - Fax: +46 (0)708 44 36 04
______________________________________________________________________
SNOWFALL DISCLAIMER:
The information contained in this email and in any
attachments is confidential and may be privileged. If you are not the
intended recipient, please destroy this message and notify the sender
immediately. You should not retain, copy or use this email for any
purpose, nor disclose all or any part of its content to any other person.
Any views expressed in this message are those of the individual sender,
except where the sender specifically states them to be the views of
Snowfall Communications.
Snowfall Communications monitors the content of emails sent and received
via its network for unauthorised use and for other lawful business
purposes. The contents of an attachment to this email may contain viruses
which could damage your computer system. While Snowfall Communications has
taken every reasonable precaution to minimise this risk, we cannot accept
liability for any damage which you sustain as a result of software
viruses. You should carry out your own virus checks before opening the
attachment.
--
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html
Re: MPM worker and ithreads
Posted by Stas Bekman <st...@stason.org>.
Stefan Cars wrote:
> Hi!
>
> Anyone that has any good links or any good input on using the worker MPM
> and ithreads with mod_perl 2 and Apache 2
You could be a bit more specific, Stafan. What kind of problems do you have?
You also have to tell us more about your environment, which is done the best
by calling mp2bug or t/REPORT
http://perl.apache.org/docs/2.0/user/help/help.html#Important_Information
__________________________________________________________________
Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide ---> http://perl.apache.org
mailto:stas@stason.org http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org http://ticketmaster.com
--
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html