No image creation in mod_perl (was RE: Apache2/MP2 Segfaults ...)

[Frank Maas <fr...@cheiron-it.nl>]

Tom,

> As a sidenote often it is not really desired/dangerous to run image
> creation as a mod_perl handler because of the nature of perl, memory
> allocated once is never freed until the process shutdowns or is killed
> (by your Apache::SizeLimit handler).

Ah, you got me worried there... How in your opinion should one do creation of images in a mod_perl environment. Think of captcha's, on-the-fly images, etc.?

I am very interested to hear opinions about this (just before I am going to use this heavily ;-).

Regards,
Frank

Re: No image creation in mod_perl (was RE: Apache2/MP2 Segfaults ...)

[Tom Schindl <to...@gmx.at>]

Well image creation in mod_perl is not a bad idea if you ensure that the
process is killed after it succeeded a certain memory. The problem in
case of mod-perl/httpd is that that if every httpd-process is eating 20
MB of space only because you have created once an image. You must ensure
that only very view processes of your apache are used to create images
and /me thinks this could be accomplished best using the following setup.

1. light-weight httpd to server static content
2. mod-perl enabled httpd to serve dynamic content beside images
3. mod-perl enabled httpd to server images with few processes (e.g.
MaxChild 5). This ensures that you can answer 5 image creation request
simultaneously but your system will never use more than 100 MB to create
images.

If your system is not stressed heavily 1 and 3 could be run in one server.

Tom

Frank Maas wrote:
> Tom,
> 
> 
>>As a sidenote often it is not really desired/dangerous to run image
>>creation as a mod_perl handler because of the nature of perl, memory
>>allocated once is never freed until the process shutdowns or is killed
>>(by your Apache::SizeLimit handler).
> 
> 
> Ah, you got me worried there... How in your opinion should one do creation of images in a mod_perl environment. Think of captcha's, on-the-fly images, etc.?
> 
> I am very interested to hear opinions about this (just before I am going to use this heavily ;-).
> 
> Regards,
> Frank
> 
> 

Re: No image creation in mod_perl (Or Perl Memory Management is a beast)

[Tom Schindl <to...@gmx.at>]

And even "worse":

-----------------8<-----------------
#!/usr/bin/perl

use Benchmark;

$t0 = new Benchmark;
&bla();
$t1 = new Benchmark;
# Memory has grown on my machine to 110 MB
#sleep 20;
$t2 = new Benchmark;
&bla();
$t3 = new Benchmark;
# Memory has resides on my machine on 110 MB

print "First run took: " . timestr(timediff($t1, $t0)) . "\n";
print "Second run took: " . timestr(timediff($t3, $t2)) . "\n";

sleep;

sub bla {
   my $var = &blo($ARGV[0]);
#   undef $var;
}

sub blo {
   my $var = "";
      for( 1 .. 20_000_000  ) {
            $var .= "xxx";
      }

      #   undef $var;
      if( ! defined $_[0] ) {
         return $var;
      } else {
         return \$var; ## This will also result in the fact that perl
                       ## releases the memory after the sub has been
                       ## ended
      }
}
-----------------8<-----------------

Just to quote perl gurus from the p5p you'll get to here something like
this:

Yitzchak Scott-Thoennes:
...
Memory is *not* released back to perl.  Lexicals(*) hold on to their
storage space, including the string buffer, under the presumption that
they will be reused.  Perl's memory management strategy can be
described as allocate as early as possible and for as long as
possible.
...

Please note the fact that Lexicals!!! hold their values until they are
undef-ed or they are also GCed when returned as references. You should
also note the time used in the second run (especially sys-time) which is
much lower because memory allocation has not to be done.


>>>Can I confirm that in the Windows implementation, if a thread allocates
>>>requires 100MB of memory and then releases it, the next thread can reuse
>>>that memory?
>>>
>>>Thanks.
>>>
>>>
>>
>>
> 

Because the above said applies to any perl you won't gain much on win32
because memory allocated belongs exactly to the sub it was allocated in ;-)

Tom

Re: No image creation in mod_perl

[Tom Schindl <to...@gmx.at>]

Well this example does not demonstrate the problem it demonstrates the
solution to let perl free memory allocated once by setting the variable
to undef ;-)

---------------8<---------------
#!/usr/bin/perl


&bla();
print "Done";
my $bla = <STDIN>;

sub bla {
   my $var = "";
   for( 1 .. 10_000_000  ) {
      $var .= "xxx";
   }

#   undef $var; # The solution to wipe out allocated memory
                # by explicitly setting vars to undef
}
---------------8<---------------


Tom

Tom Schindl wrote:
> You can try memory management yourself and see that the memory allocated
> is not wiped until the script is finished.
> 
> --------------------8<--------------------
> #!/usr/bin/perl
> 
> &bla();
> print "Done";
> 
> sub bla {
>    my $var = "";
>    for( 1 .. 10_000_000  ) {
>       $var .= "xxx";
>    }
> 
>    my $bla = <STDIN>;
>    undef $var;
> }
> --------------------8<--------------------
> 
> You can modify this example to use threads and see what's happeing to
> your memory.
> 
> Tom
> 
> 
> Foo Ji-Haw wrote:
> 
>>Hello Carl,
>>
>>
>>>Nope that's right, so you load up one image. The perl process
>>>allocates itself 100MB of memory for it from the OS. Then doesn't
>>>release it back to the OS once it's finished with.
>>>
>>>The perl process will re-use this memory, so if you process another
>>>image you don't grab another 100MB, it's just not available at the OS
>>>level or for other processes.
>>>
>>>This isn't completely bad as long as your OS has good memory
>>>management. The unused memory in the perl process will just be swapped
>>>out to disk and left there until that process uses it again or exits.
>>
>>Can I confirm that in the Windows implementation, if a thread allocates
>>requires 100MB of memory and then releases it, the next thread can reuse
>>that memory?
>>
>>Thanks.
>>
>>
> 
> 

Re: No image creation in mod_perl

[Tom Schindl <to...@gmx.at>]

You can try memory management yourself and see that the memory allocated
is not wiped until the script is finished.

--------------------8<--------------------
#!/usr/bin/perl

&bla();
print "Done";

sub bla {
   my $var = "";
   for( 1 .. 10_000_000  ) {
      $var .= "xxx";
   }

   my $bla = <STDIN>;
   undef $var;
}
--------------------8<--------------------

You can modify this example to use threads and see what's happeing to
your memory.

Tom


Foo Ji-Haw wrote:
> Hello Carl,
> 
>>
>> Nope that's right, so you load up one image. The perl process
>> allocates itself 100MB of memory for it from the OS. Then doesn't
>> release it back to the OS once it's finished with.
>>
>> The perl process will re-use this memory, so if you process another
>> image you don't grab another 100MB, it's just not available at the OS
>> level or for other processes.
>>
>> This isn't completely bad as long as your OS has good memory
>> management. The unused memory in the perl process will just be swapped
>> out to disk and left there until that process uses it again or exits.
> 
> Can I confirm that in the Windows implementation, if a thread allocates
> requires 100MB of memory and then releases it, the next thread can reuse
> that memory?
> 
> Thanks.
> 
> 

Re: No image creation in mod_perl

[Foo Ji-Haw <jh...@extracktor.com>]

Hello Carl,
>
> Nope that's right, so you load up one image. The perl process 
> allocates itself 100MB of memory for it from the OS. Then doesn't 
> release it back to the OS once it's finished with.
>
> The perl process will re-use this memory, so if you process another 
> image you don't grab another 100MB, it's just not available at the OS 
> level or for other processes.
>
> This isn't completely bad as long as your OS has good memory 
> management. The unused memory in the perl process will just be swapped 
> out to disk and left there until that process uses it again or exits.
Can I confirm that in the Windows implementation, if a thread allocates 
requires 100MB of memory and then releases it, the next thread can reuse 
that memory?

Thanks.

Re: No image creation in mod_perl

[Carl Johnstone <mo...@fadetoblack.me.uk>]

> In (eg) the worker MPM, each process contains its own perl interpreter,
> so if each process handles one image once in its lifetime, there is a
> lot of memory that has been grabbed by perl which is not available to
> create more perl processes.
>
> ... is what makes sense to me but may be utterly meaningless.

Nope that's right, so you load up one image. The perl process allocates 
itself 100MB of memory for it from the OS. Then doesn't release it back to 
the OS once it's finished with.

The perl process will re-use this memory, so if you process another image 
you don't grab another 100MB, it's just not available at the OS level or for 
other processes.

This isn't completely bad as long as your OS has good memory management. The 
unused memory in the perl process will just be swapped out to disk and left 
there until that process uses it again or exits.

Carl

Re: No image creation in mod_perl (was RE: Apache2/MP2 Segfaults ...)

[Clinton Gormley <cl...@traveljury.com>]

> This revelation of how Perl does not free up memory it allocates is 
> worrying, especially as I do process large documents regularly.
> 
> If I read you right, you are saying that $r->child_terminate will force 
> the current thread to terminate, causing Apache to create a new thread. 
> Is that right? I use 'threads' in place of 'processes' as I refer to the 
> Windows implementation of Apache.

No, this only works with processes, not with threaded MPMs:
http://perl.apache.org/docs/2.0/api/Apache2/RequestUtil.html#C_child_terminate_

But that makes sense to me, if I have understood it correctly.

Once a perl process has allocated memory, it doesn't return it to the
system until the perl interpreter exits. It is, however, available to
the process for reuse.

So in a threaded MPM, a single process may grow in size, but all the
threads will have access to the allocated memory.

In (eg) the worker MPM, each process contains its own perl interpreter,
so if each process handles one image once in its lifetime, there is a
lot of memory that has been grabbed by perl which is not available to
create more perl processes.

... is what makes sense to me but may be utterly meaningless.

clint

Re: No image creation in mod_perl (was RE: Apache2/MP2 Segfaults ...)

[Foo Ji-Haw <jh...@extracktor.com>]

Hey Carl,

> The only place where forking is useful is where you want something to
> continue processing after sending the response back to the client.
>
> You can achieve the same effective result by calling 
> $r->child_terminate()
> (assuming your using pre-fork). The currently running child exits at 
> the end
> of the request freeing any memory it's allocated, and the main apache 
> server
> process will fork a new child to replace it if needed. 
This revelation of how Perl does not free up memory it allocates is 
worrying, especially as I do process large documents regularly.

If I read you right, you are saying that $r->child_terminate will force 
the current thread to terminate, causing Apache to create a new thread. 
Is that right? I use 'threads' in place of 'processes' as I refer to the 
Windows implementation of Apache.

Thanks.

Re: No image creation in mod_perl (was RE: Apache2/MP2 Segfaults ...)

[Carl Johnstone <mo...@fadetoblack.me.uk>]

> Can you fork off a separate process to do this (that will die once it is
> completed)?

The only place where forking is useful is where you want something to
continue processing after sending the response back to the client.

You can achieve the same effective result by calling $r->child_terminate()
(assuming your using pre-fork). The currently running child exits at the end
of the request freeing any memory it's allocated, and the main apache server
process will fork a new child to replace it if needed.

Carl

Re: No image creation in mod_perl (was RE: Apache2/MP2 Segfaults ...)

[Sean Davis <sd...@mail.nih.gov>]

On 3/27/06 6:21 AM, "Tom Schindl" <to...@gmx.at> wrote:

> Please note that this is not only true for Image-Creation but also if
> you are restoring large string contents in a variable (e.g. after
> processing a file-upload).
> 
> Tom
> 
> Frank Maas wrote:
>> Tom,
>> 
>> 
>>> As a sidenote often it is not really desired/dangerous to run image
>>> creation as a mod_perl handler because of the nature of perl, memory
>>> allocated once is never freed until the process shutdowns or is killed
>>> (by your Apache::SizeLimit handler).
>> 
>> 
>> Ah, you got me worried there... How in your opinion should one do creation of
>> images in a mod_perl environment. Think of captcha's, on-the-fly images,
>> etc.?
>> 
>> I am very interested to hear opinions about this (just before I am going to
>> use this heavily ;-).

Can you fork off a separate process to do this (that will die once it is
completed)?  

Sean

Re: No image creation in mod_perl (was RE: Apache2/MP2 Segfaults ...)

[Tom Schindl <to...@gmx.at>]

Please note that this is not only true for Image-Creation but also if
you are restoring large string contents in a variable (e.g. after
processing a file-upload).

Tom

Frank Maas wrote:
> Tom,
> 
> 
>>As a sidenote often it is not really desired/dangerous to run image
>>creation as a mod_perl handler because of the nature of perl, memory
>>allocated once is never freed until the process shutdowns or is killed
>>(by your Apache::SizeLimit handler).
> 
> 
> Ah, you got me worried there... How in your opinion should one do creation of images in a mod_perl environment. Think of captcha's, on-the-fly images, etc.?
> 
> I am very interested to hear opinions about this (just before I am going to use this heavily ;-).
> 
> Regards,
> Frank
> 
> 

Re: No image creation in mod_perl (was RE: Apache2/MP2 Segfaults ...)

[Jonathan Vanasco <mo...@2xlp.com>]

i generally don't like to do that in modperl unless i have enough  
webservers.  i think its a bit of a waste of the mp resource.

i do this:
	on upload, generate a thumbnail w/Image:Epeg (which is a super fast  
implementation of epeg.  previously i compiled my own epeg tools and  
the perl mod is WAY faster ) and save the original
	have a cronjob that passes over new uploads and redoes the thumbnail  
and creates sized versions using the python imaging library ( much  
faster and way more powerful than the perl modules or image magick  
and way easier to code )

captchas - you can pregenerate a bunch then use rewrite to alias  
them.  to me, its silly to generate them on the fly - its a rather  
tedious task.  you can even have a cronjob create a new pool of  
captchas and a new map of filenames -> text in image every few hours  
if you want.

i'm ok w/using image:epeg, because it just scans the image for info  
and doesn't read the whole thing into memory.

i really hate doing image manipulation stuff in mod_perl or perl in  
general.