You are viewing a plain text version of this content. The canonical link for it is here.
Posted to modperl@perl.apache.org by Michael Lackhoff <mi...@lackhoff.de> on 2007/11/14 16:07:14 UTC
Parallel processing within mod_perl
Hello,
I have a mod_perl2 application that searches some databases (custom
socket connections, no DBI).
Now I would like to run the searches in parallel, collect all the
results and return them after some post-processing.
I searched the net and asked on perlmonks
(http://www.perlmonks.org/?node_id=650263) and came to this not yet
working code (taken mostly from an example in practical mod_perl):
sub metasearch {
my $self = shift;
my $db_ref = $self->db_defs;
my @dbs = @{$self->dbs};
my $logger = $self->logger;
$SIG{CHLD} = 'IGNORE';
my @result;
foreach my $db (@dbs) {
defined (my $kid = fork) or die "Cannot fork: $!\n";
$logger->debug("Processing $db in process $kid");
if ($kid) {
$logger->debug("Parent $$ has finished, kid's PID: $kid");
}
else {
# $r->cleanup_for_exec(); # untie the socket
open STDIN, '/dev/null' or die "Can't read /dev/null: $!";
open STDOUT, '>/dev/null' or die "Can't write: $!";
open STDERR, '>/tmp/log' or die "Can't write: $!";
# setsid or die "Can't start a new session: $!";
my $oldfh = select STDERR; local $| = 1; select $oldfh;
warn "started\n";
sleep 1, warn "$_\n" for 1..20;
warn "completed\n";
# The search would be here, for tests just add a string:
push @result, "simulate result for $db in $kid";
CORE::exit(0); # terminate the process
}
$logger->debug("End of $db");
}
$logger->debug(join(', ', @result));
return @result; # this is what I am after!
}
The problem with this code is that I don't get @result filled.
There seem to be two problems:
1. I have to use waitpid somehow and don't know how (all examples I
found just deal with one child).
2. There doesn't seem to be a way back from the forked children to
the parent because the children get their own copy of @result as
soon as they write to it (correct?)
If my analysis is correct I would like to know:
1. How to properly wait for the children and
2. How to get the results back to the main process
Is there some common storage in apache/mod_perl where the children
can write and the parent can read the results when the children are
finished? Can I use pnotes for this?
Thanks,
-Michael
Re: Parallel processing within mod_perl
Posted by Andy Armstrong <an...@hexten.net>.
On 15 Nov 2007, at 12:30, Michael Lackhoff wrote:
> Yes.
> http://perl.apache.org/docs/1.0/guide/performance.html#Forking_a_New_Process
>
> gives an explanation:
>> you must use CORE::exit() and not exit(), which would be
>> automatically
>> overridden by Apache::exit() if used in conjunction with
>> Apache::Registry and similar modules.
>
> It should be safe to replace exit with CORE::exit even for non-
> mod_perl
> uses, so it would be nice to have this in the official version of the
> module. Then it is easy to move the application to a new machine where
> all modules are installed from CPAN.
Done, thanks. New release on its way to CPAN.
--
Andy Armstrong, Hexten
Re: Parallel processing within mod_perl
Posted by Michael Lackhoff <mi...@lackhoff.de>.
On 15.11.2007 13:06 Andy Armstrong wrote:
> On 15 Nov 2007, at 08:35, Michael Lackhoff wrote:
>> Thanks, it really works (more tests to come but it looks good so far)!
>> I had only some easy to fix problems:
>> 1. in line 434 of Iterator.pm I had to replace 'exit' with 'CORE::exit'
>
> Ah. I assume that's a mod_perl thing?
Yes.
http://perl.apache.org/docs/1.0/guide/performance.html#Forking_a_New_Process
gives an explanation:
> you must use CORE::exit() and not exit(), which would be automatically
> overridden by Apache::exit() if used in conjunction with
> Apache::Registry and similar modules.
It should be safe to replace exit with CORE::exit even for non-mod_perl
uses, so it would be nice to have this in the official version of the
module. Then it is easy to move the application to a new machine where
all modules are installed from CPAN.
-Michael
Re: Parallel processing within mod_perl
Posted by Andy Armstrong <an...@hexten.net>.
On 15 Nov 2007, at 08:35, Michael Lackhoff wrote:
> Thanks, it really works (more tests to come but it looks good so far)!
> I had only some easy to fix problems:
> 1. in line 434 of Iterator.pm I had to replace 'exit' with
> 'CORE::exit'
Ah. I assume that's a mod_perl thing?
> 2. A minor doc error/misunderstanding: From the example in the POD it
> looks as if the code_ref gets the value as the first parameter:
> sub fetch { my $url = shift ...
> but I get ($index, $url), so the value is the second parameter, not
> the first.
Yeah I know... I read the documentation myself after suggesting the
module to you and could barely make sense of it :)
> 3. My original code returns objects Storable has problems to digest. I
> will have to partly rewrite my code to only return data that is
> easyly serializable and (re)build the objects later.
Yup.
> Thanks for this fine module!
You're welcome. Let me know if you hit any other problems.
--
Andy Armstrong, Hexten
Re: Parallel processing within mod_perl
Posted by Michael Lackhoff <mi...@lackhoff.de>.
On 14.11.2007 17:03 Andy Armstrong wrote:
> I've never tried it under MP or MP2 but Parallel::Iterator[1] is
> designed to make this kind of thing easy. It hides the details of
> forking, waiting and getting results back.
>
> I'd be interested to know if you find it useful. And if you give it a go
> and hit problems let me know - if nothing else I should be able to
> improve the doc / examples as a result :)
Thanks, it really works (more tests to come but it looks good so far)!
I had only some easy to fix problems:
1. in line 434 of Iterator.pm I had to replace 'exit' with 'CORE::exit'
2. A minor doc error/misunderstanding: From the example in the POD it
looks as if the code_ref gets the value as the first parameter:
sub fetch { my $url = shift ...
but I get ($index, $url), so the value is the second parameter, not
the first.
3. My original code returns objects Storable has problems to digest. I
will have to partly rewrite my code to only return data that is
easyly serializable and (re)build the objects later.
Thanks for this fine module!
- Michael
Re: Parallel processing within mod_perl
Posted by Andy Armstrong <an...@hexten.net>.
On 14 Nov 2007, at 15:07, Michael Lackhoff wrote:
> If my analysis is correct I would like to know:
> 1. How to properly wait for the children and
> 2. How to get the results back to the main process
> Is there some common storage in apache/mod_perl where the children
> can write and the parent can read the results when the children are
> finished? Can I use pnotes for this?
I've never tried it under MP or MP2 but Parallel::Iterator[1] is
designed to make this kind of thing easy. It hides the details of
forking, waiting and getting results back.
I'd be interested to know if you find it useful. And if you give it a
go and hit problems let me know - if nothing else I should be able to
improve the doc / examples as a result :)
[1] http://search.cpan.org/~andya/Parallel-Iterator/
--
Andy Armstrong, Hexten