You are viewing a plain text version of this content. The canonical link for it is here.
Posted to embperl@perl.apache.org by Hans de Groot <ha...@dandy.nl> on 2003/12/18 14:48:31 UTC
memory usage
Hello,
I use Embperl for a few years now and I am sure I am not using it in the most
efficient way.
I only use the %udat,%fdat features.
I usally create pages that look like this:
[*
some perl here
store all output in $ouput
use http_headers_out followed with and exit() if I need to redirect.
*]
[+$output+]
This works great and really do not need all the extra's Embperl offers.
I do have one problem. My apache processes take a lot of memory.
Every request of a page it grows in size. I limited the "max request per child"
to 200 to prevent them from becoming to big, but spawning of a new child is not
really fast.
How do I prevent, or minimize the loss of memory?
I tried clearing all variables but it does not seem to help.
I could be my lack of knowlege about perl.
I usually clear vars like this:
%myhash=(); undef %myhash;
$template=''; undef $template;
But it seems this does not help.
People warned me about using the [* *] why is that?
Anyway some hints how to prevent apache processes from growing every request
would be nice. (or how to minimize this)
Thank you.
Hans de Groot
---------------------------------------------------------------------
To unsubscribe, e-mail: embperl-unsubscribe@perl.apache.org
For additional commands, e-mail: embperl-help@perl.apache.org
Re: memory usage / session cleanup
Posted by Wim Kerkhoff <wi...@nyetwork.org>.
Pierre Etchemaite wrote:
>Le Fri, 19 Dec 2003 10:38:40 -0800, Wim Kerkhoff <wi...@nyetwork.org> a écrit
>:
>
>
>
>>You can't do this with MySQL, since it won't let you set a default value
>>to a function, only a constant (null, 1, 'foo', etc).
>>
>>
>
>For MySQL, it seems that the way to go are TIMESTAMP fields:
>
>http://www.mysql.com/doc/en/DATETIME.html
>
>Those will be updated each time the record is, but in this case this looks
>appropriate.
>
>
Ok, thanks for the clarification. Pg's handling of default values,
triggers, etc still seems way better, IMO :-)
Wim
---------------------------------------------------------------------
To unsubscribe, e-mail: embperl-unsubscribe@perl.apache.org
For additional commands, e-mail: embperl-help@perl.apache.org
Re: memory usage / session cleanup
Posted by Pierre Etchemaite <pe...@concept-micro.com>.
Le Fri, 19 Dec 2003 10:38:40 -0800, Wim Kerkhoff <wi...@nyetwork.org> a écrit
:
> You can't do this with MySQL, since it won't let you set a default value
> to a function, only a constant (null, 1, 'foo', etc).
For MySQL, it seems that the way to go are TIMESTAMP fields:
http://www.mysql.com/doc/en/DATETIME.html
Those will be updated each time the record is, but in this case this looks
appropriate.
---------------------------------------------------------------------
To unsubscribe, e-mail: embperl-unsubscribe@perl.apache.org
For additional commands, e-mail: embperl-help@perl.apache.org
Re: memory usage / session cleanup
Posted by Wim Kerkhoff <wi...@nyetwork.org>.
Hans de Groot wrote:
>
>Than an other question
>I installed embperl a while ago, with storable and apache session and made the
>session database.
>
>What is the properway to cleanup the session database? Right now I just delete
>the content of the session table but this also deletes the sessions of the
>current users, which is a it rude.
>
>
>
Here's what I do...
I'm using PostgreSQL to store the sessions.
In the sessions table, I added an extra column for stamp_inserted with
type timestamp, in addition to the id and a_session fields.
I set the default on this timestamp column to be now(). Here's the DDL
from pg_dump:
CREATE TABLE sessions (
id character(32) NOT NULL,
a_session text,
stamp_inserted timestamp without time zone DEFAULT now()
);
ALTER TABLE ONLY sessions
ADD CONSTRAINT sessions_pkey PRIMARY KEY (id);
Then, have a cronjob that runs a DBI script, with something like:
$dbh->do("delete from sessions where stamp_inserted < now() - interval
'2 days'")
You can't do this with MySQL, since it won't let you set a default value
to a function, only a constant (null, 1, 'foo', etc).
HTH,
Wim
---------------------------------------------------------------------
To unsubscribe, e-mail: embperl-unsubscribe@perl.apache.org
For additional commands, e-mail: embperl-help@perl.apache.org
Re: memory usage
Posted by Gerald Richter <ri...@ecos.de>.
Hi,
>
> I am using 1.3.6. and apache 1.3.27
>
> Wel it keeps getting bigger with one page. When I reload it or click
> through it it keeps growing.
>
Then there must be something inside the page, that grows.
> To ask a beginners question:
>
> if I make a silly script like this:
> ------
> [*
> open F,"</tmp/largetextfile.txt";
> while(<F>)
> {
> $data.=$_;
> }
> close(<F>);
this should be
close F ;
> $counter=0;
> while ($data =~ s/image.jpg/$counter.jpg/)
> {
> $counter++;
> }
> *]
> <html>
> <body>
> [+$data+]
> </body>
> </html>
> --------
>
Not sure if the close syntax error has memory side effects, but the rest of
the page is ok
> Do I need to empty the vars used
No, EMbperl does this for you
>or if every thing works okay this
> should not cause my apache process to grow all the time? (if this is
> the only page in the server)
>
Yes, it should not grow
>
> Than an other question
> I installed embperl a while ago, with storable and apache session and
> made the session database.
>
> What is the properway to cleanup the session database? Right now I
> just delete the content of the session table but this also deletes
> the sessions of the current users, which is a it rude.
>
You might add a datetime column and select on that what to delete. Search
the mailing listarchive, there should be some more ideas
Gerald
--------------------------------------------------------------
Gerald Richter ecos electronic communication services gmbh
IT-Securitylösungen * dynamische Webapplikationen * Consulting
Post: Tulpenstrasse 5 D-55276 Dienheim b. Mainz
E-Mail: richter@ecos.de Voice: +49 6133 939-122
WWW: http://www.ecos.de/ Fax: +49 6133 939-333
--------------------------------------------------------------
|
| ECOS BB-5000 Firewall- und IT-Security Appliance: www.bb-5000.info
|
+-------------------------------------------------------------
---------------------------------------------------------------------
To unsubscribe, e-mail: embperl-unsubscribe@perl.apache.org
For additional commands, e-mail: embperl-help@perl.apache.org
Re: memory usage
Posted by Kee Hinckley <na...@somewhere.com>.
At 1:44 PM +0100 12/19/03, Hans de Groot wrote:
>if I make a silly script like this:
Just so you know, that while is *real* inefficient. What you really want is something like:
s/foo.jpg/++$counter . ".jpg"/ge
See the below results. I did them in 1 iteration and 10 separately because the while version was too slow for me to wait for the results if I tried it 10 times.
Benchmark: timing 1 iterations of per-line g, slurp g, while data, while inline...
per-line g: 0 wallclock secs ( 0.14 usr + 0.00 sys = 0.14 CPU) @ 7.14/s (n=1)
(warning: too few iterations for a reliable count)
slurp g: 0 wallclock secs ( 0.06 usr + 0.00 sys = 0.06 CPU) @ 16.67/s (n=1)
(warning: too few iterations for a reliable count)
while data: 44 wallclock secs (35.37 usr + 0.00 sys = 35.37 CPU) @ 0.03/s (n=1)
(warning: too few iterations for a reliable count)
while inline: 44 wallclock secs (35.20 usr + 0.00 sys = 35.20 CPU) @ 0.03/s (n=1)
(warning: too few iterations for a reliable count)
s/iter while data while inline per-line g slurp g
while data 35.4 -- -0% -100% -100%
while inline 35.2 0% -- -100% -100%
per-line g 0.140 25164% 25043% -- -57%
slurp g 6.00e-02 58850% 58567% 133% --
at 10 iterations (with the slow two turned off)
Benchmark: timing 10 iterations of per-line g, slurp g, while data, while inline...
per-line g: 2 wallclock secs ( 1.41 usr + 0.00 sys = 1.41 CPU) @ 7.09/s (n=10)
slurp g: 1 wallclock secs ( 0.64 usr + 0.00 sys = 0.64 CPU) @ 15.62/s (n=10)
while data: 0 wallclock secs ( 0.00 usr + 0.00 sys = 0.00 CPU)
(warning: too few iterations for a reliable count)
while inline: 0 wallclock secs ( 0.00 usr + 0.00 sys = 0.00 CPU)
(warning: too few iterations for a reliable count)
Rate per-line g slurp g while data while inline
per-line g 7.09/s -- -55% -100% -100%
slurp g 15.6/s 120% -- -100% -100%
while data 10000000000000000/s 141000000000000000% 64000000000000016% -- 0%
while inline 10000000000000000/s 141000000000000000% 64000000000000016% 0% --
#!/usr/bin/perl
#
use strict;
use warnings;
use Benchmark qw(cmpthese);
cmpthese($ARGV[0] || 10, {
'while data' => q{
return if ($ARGV[0] > 1); # too slow
#print STDERR "while data: ";
my ($data, $counter);
$data = '';
open(T, "</etc/magic");
while (<T>) {
$data .= $_;
}
close(T);
$counter = 0;
while ($data =~ s/e/$counter/) { ++$counter; }
#print STDERR "$counter\n";
},
'while inline' => q{
return if ($ARGV[0] > 1); # too slow
#print STDERR "while inline: ";
my ($data, $counter);
$counter = 0;
open(T, "</etc/magic");
while (<T>) {
while (s/e/$counter/) { ++$counter; }
}
close(T);
#print STDERR "$counter\n";
},
'per-line g' => q{
#print STDERR "per-line g: ";
my ($data, $counter);
$counter = 0;
open(T, "</etc/magic");
while (<T>) {
s/e/++$counter/ge;
}
close(T);
#print STDERR "$counter\n";
},
'slurp g' => q{
#print STDERR "slurp g: ";
my ($data, $counter);
$counter = 0;
open(T, "</etc/magic");
select((select(T), $/ = undef)[0]); # no eol
$data = <T>;
close(T);
$data =~ s/e/++$counter/ge;
#print STDERR "$counter\n";
},
});
--
Kee Hinckley
http://www.messagefire.com/ Next Generation Spam Defense
http://commons.somewhere.com/buzz/ Writings on Technology and Society
I'm not sure which upsets me more: that people are so unwilling to accept
responsibility for their own actions, or that they are so eager to regulate
everyone else's.
---------------------------------------------------------------------
To unsubscribe, e-mail: embperl-unsubscribe@perl.apache.org
For additional commands, e-mail: embperl-help@perl.apache.org
Re: memory usage
Posted by Hans de Groot <ha...@dandy.nl>.
Hi,
I am using 1.3.6. and apache 1.3.27
Wel it keeps getting bigger with one page. When I reload it or click through it
it keeps growing.
To ask a beginners question:
if I make a silly script like this:
------
[*
open F,"</tmp/largetextfile.txt";
while(<F>)
{
$data.=$_;
}
close(<F>);
$counter=0;
while ($data =~ s/image.jpg/$counter.jpg/)
{
$counter++;
}
*]
<html>
<body>
[+$data+]
</body>
</html>
--------
Do I need to empty the vars used or if every thing works okay this should not
cause my apache process to grow all the time? (if this is the only page in the
server)
Than an other question
I installed embperl a while ago, with storable and apache session and made the
session database.
What is the properway to cleanup the session database? Right now I just delete
the content of the session table but this also deletes the sessions of the
current users, which is a it rude.
Regards
Hans de Groot
Met vriendelijke groet,
Hans de Groot
Quoting Gerald Richter <ri...@ecos.de>:
> >
> > I do have one problem. My apache processes take a lot of memory.
> > Every request of a page it grows in size.
>
> mod_perl httpd takes a lot of memory, that can't be changed. Every new
> request will eat up more memory, because Perl code is compiled. If you
> request only one page more than once memory useage should stay the same,
> that's how you can test if any of your code has a memory leak.
>
> BTW Which Embperl version do you are using?
>
> Gerald
>
> --------------------------------------------------------------
> Gerald Richter ecos electronic communication services gmbh
> IT-Securitylösungen * dynamische Webapplikationen * Consulting
>
> Post: Tulpenstrasse 5 D-55276 Dienheim b. Mainz
> E-Mail: richter@ecos.de Voice: +49 6133 939-122
> WWW: http://www.ecos.de/ Fax: +49 6133 939-333
> --------------------------------------------------------------
> |
> | ECOS BB-5000 Firewall- und IT-Security Appliance: www.bb-5000.info
> |
> +-------------------------------------------------------------
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: embperl-unsubscribe@perl.apache.org
> For additional commands, e-mail: embperl-help@perl.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: embperl-unsubscribe@perl.apache.org
For additional commands, e-mail: embperl-help@perl.apache.org
Re: memory usage
Posted by Gerald Richter <ri...@ecos.de>.
>
> I do have one problem. My apache processes take a lot of memory.
> Every request of a page it grows in size.
mod_perl httpd takes a lot of memory, that can't be changed. Every new
request will eat up more memory, because Perl code is compiled. If you
request only one page more than once memory useage should stay the same,
that's how you can test if any of your code has a memory leak.
BTW Which Embperl version do you are using?
Gerald
--------------------------------------------------------------
Gerald Richter ecos electronic communication services gmbh
IT-Securitylösungen * dynamische Webapplikationen * Consulting
Post: Tulpenstrasse 5 D-55276 Dienheim b. Mainz
E-Mail: richter@ecos.de Voice: +49 6133 939-122
WWW: http://www.ecos.de/ Fax: +49 6133 939-333
--------------------------------------------------------------
|
| ECOS BB-5000 Firewall- und IT-Security Appliance: www.bb-5000.info
|
+-------------------------------------------------------------
---------------------------------------------------------------------
To unsubscribe, e-mail: embperl-unsubscribe@perl.apache.org
For additional commands, e-mail: embperl-help@perl.apache.org
Re: memory usage
Posted by Donovan Allen <al...@votehere.net>.
Hans de Groot wrote:
>I usally create pages that look like this:
>[*
>some perl here
>store all output in $ouput
>use http_headers_out followed with and exit() if I need to redirect.
>*]
>
>[+$output+]
>
>
>
>
Unless you need the "local" scope of the [* *] blocks, you might try the
[- -] instead and see if that fixes your problem. Change it on one page
and repeatedly call that page while watching the memory use.
I have never had memory problems using [* *], but that may be because I
only use them for loops around html output. I generally stick with [-
-] blocks.
>I do have one problem. My apache processes take a lot of memory.
>
>
Apache + any mod_perl tend to take a lot of memory anyway, however some
of it is shared between processes.
>Every request of a page it grows in size. How do I prevent, or minimize the loss of memory?
>
>
One other thing to be aware of is that if you are using modules, the
variables in the module need to have proper scope (such as my() in
functions and methods), otherwise you will get what "appears" to be a
memory leak.
Getting all the "use" lines from your code and putting them into one
document where you can comment them out, refresh the page, and check
memory is a dirty but quick was to isolate a module that has not
declared proper variable scope.
It is possible that things have changed, but as I recall default perl
installs rarely (if ever?) use versions of malloc() that allow
deallocation of used memory while running (providing that your OS even
supplies one). Your best tactic in general is to use memory carefully
in practice and find what is causing the memory consumption and fixing it.
---------------------------------------------------------------------
To unsubscribe, e-mail: embperl-unsubscribe@perl.apache.org
For additional commands, e-mail: embperl-help@perl.apache.org