You are viewing a plain text version of this content. The canonical link for it is here.
Posted to modperl@perl.apache.org by Dan Wilga <dw...@MtHolyoke.edu> on 2002/03/21 18:47:54 UTC
Berkeley DB 4.0.14 not releasing lockers under mod_perl
Hello,
I have run into a problem specific to Berkeley 4.0.14 that I hope you
can help me to diagnose. There are enough other products involved
that I'm not sure if it's DB's fault or mod_perl's. Here's the setup:
Linux 7.2 i386
perl 5.6.1
apache 1.3.19 with mod_perl 1.25
DB 4.0.14
BerkeleyDB 0.18
I'm testing with the Perl script below, with the filename ending
".mperl" (which, in my configuration, causes it to run as a mod_perl
registry script).
The problem: Under DB 4.0.14, five lockers get allocated, one during
each iteration of the loop. The test program will show "Current
number of lockers" and "Maximum number of lockers so far" both to be
5. In a production environment, this behavior eventually leads to an
error, because the maximum number of lockers is reached.
If I either use DB 3.x or even run this from the commandline
(bypassing mod_perl) under DB 4 the problem goes away: only one
locker is allocated per loop, and therefore the total number used
does not increase unexpectedly.
I have already contacted Sleepycat (the makers of DB), but they
cannot help much, since their experience with mod_perl is limited.
--------- TEST SCRIPT ---------
#!/usr/bin/perl
use strict;
use BerkeleyDB qw( DB_CREATE DB_INIT_MPOOL DB_INIT_CDB );
# Change me to something appropriate for your system
my $dir='/home/httpd/some/directory';
system( "rm $dir/__db* $dir/TESTdb" );
foreach( 1..5 ) {
my $env = open_env($dir);
my %hash;
my $db = open_db( "TESTdb", \%hash, $env );
untie %hash;
undef $db;
undef $env;
}
print "HTTP/1.1 200\nContent-type: text/plain\n\n";
print `db_stat -c -h $dir`;
print "\n";
sub open_env {
my $env = new BerkeleyDB::Env(
-Flags=>DB_INIT_MPOOL|DB_INIT_CDB|DB_CREATE,
-Home=> $_[0],
);
die "Could not create env: $! ".$BerkeleyDB::Error. "\n" if !$env;
return $env;
}
sub open_db {
my( $file, $Rhash, $env ) = @_;
my $db_key = tie( %{$Rhash}, 'BerkeleyDB::Btree',
-Flags=>DB_CREATE,
-Filename=>$file,
-Env=>$env );
die "Can't open $file: $! ".$BerkeleyDB::Error."\n" if !$db_key;
return $db_key;
}
Dan Wilga dwilga@mtholyoke.edu
Web Technology Specialist http://www.mtholyoke.edu
Mount Holyoke College Tel: 413-538-3027
South Hadley, MA 01075 "I have a bold and cunning plan" - Baldric
Re: Berkeley DB 4.0.14 not releasing lockers under mod_perl
Posted by Dan Wilga <dw...@MtHolyoke.edu>.
At 1:55 PM -0500 3/21/02, Perrin Harkins wrote:
>Dan Wilga wrote:
>>What surprises me is that all I have to do to introduce the problem
>>is run it under mod_perl. It acts normally when run from the
>>commandline.
>
>Well, let's see, what would be different...
>
>Is it possible that the problem is concurrency from multiple
>mod_perl processes? What happens when you run it with httpd -X ?
The same (bad) results.
>Could there be a permissions issue involved?
Nope, all the files get deleted and re-created in the script (and
they do get created correctly). Permissions also wouldn't explain why
the same thing works in 3.x.
>Are you sure you're only invoking the script once under mod_perl?
Yes, I'm doing it by a direct URL.
> If you raise the number of repetitions, does the command-line
>version exhibit the problem?
No, the commandline version never does. When run under mod_perl, the
"Current number of lockers" reported by db_stat is always equal to
the number of iterations, no matter how many are used.
>I don't see any scoping issues in your script, but are you running
>it under Apache::Registry? Maybe the nested subroutines that
>creates are causing problems.
Ack. Ptooey. I just renamed the file to end in ".cgi" and the problem
persists. I even removed mod_perl from my configuration files, and it
still fails. I thought I had tested the before, but I guess not.
This can only mean mod_perl isn't at fault. Sorry for the false
alarm. I'll have to go back to Sleepycat with this new information.
Dan Wilga dwilga@mtholyoke.edu
Web Technology Specialist http://www.mtholyoke.edu
Mount Holyoke College Tel: 413-538-3027
South Hadley, MA 01075 "Who left the cake out in the rain?"
Re: Berkeley DB 4.0.14 not releasing lockers under mod_perl
Posted by Dan Wilga <dw...@MtHolyoke.edu>.
OK, I'm an idiot who can't read. I was reading the wrong part of
db_stat's output when I thought that running it with the .cgi ending
also caused the problem. Turns out it doesn't.
So I have not absolved mod_perl after all. It's definitely an
interaction between it and DB 4.
This is one of those times I really wish I could go back and edit my
posts :-). Anyone reading this thread will now be seriously confused.
So I'll try Aaron's suggestion of writing it as a handler. Any other
suggestions are welcome.
Dan Wilga dwilga@mtholyoke.edu
Web Technology Specialist http://www.mtholyoke.edu
Mount Holyoke College Tel: 413-538-3027
South Hadley, MA 01075 "I have a bold and cunning plan" - Baldric
Re: Berkeley DB 4.0.14 not releasing lockers under mod_perl
Posted by Perrin Harkins <pe...@elem.com>.
Dan Wilga wrote:
> What surprises me is that all I have to do to introduce the problem is
> run it under mod_perl. It acts normally when run from the commandline.
Well, let's see, what would be different...
Is it possible that the problem is concurrency from multiple mod_perl
processes? What happens when you run it with httpd -X ?
Could there be a permissions issue involved?
Are you sure you're only invoking the script once under mod_perl? If
you raise the number of repetitions, does the command-line version
exhibit the problem?
I don't see any scoping issues in your script, but are you running it
under Apache::Registry? Maybe the nested subroutines that creates are
causing problems.
- Perrin
Re: Berkeley DB 4.0.14 not releasing lockers under mod_perl
Posted by Dan Wilga <dw...@MtHolyoke.edu>.
At 1:32 PM -0500 3/21/02, Perrin Harkins wrote:
>Dan Wilga wrote:
>>If I either use DB 3.x or even run this from the commandline
>>(bypassing mod_perl) under DB 4 the problem goes away: only one
>>locker is allocated per loop, and therefore the total number used
>>does not increase unexpectedly.
>
>This sort of begs the question: why not use DB 3.x? Is there some
>new feature you need in DB 4?
I'm having problems with minor DB corruption in the last release of
3.x (3.3.11), so I wanted to try 4.
>It's been a little while since I messed with this stuff, but I don't
>think you need to tear down the Env each time. I think you can just
>create it the first time and reuse it after that. Maybe that will
>help.
I tried that, believe me :-). I only did it this way in the test
script to show that the problem occurs despite that bit of extra
caution. (Even the "undefs" are probably not needed, but are there
for clarity.)
What surprises me is that all I have to do to introduce the problem
is run it under mod_perl. It acts normally when run from the
commandline.
>(Ah, Mount Holyoke. As a Five College alumn, I remember it well.)
:-)
Dan Wilga dwilga@mtholyoke.edu
Web Technology Specialist http://www.mtholyoke.edu
Mount Holyoke College Tel: 413-538-3027
South Hadley, MA 01075 "Seduced by the chocolate side of the Force"
Re: Berkeley DB 4.0.14 not releasing lockers under mod_perl
Posted by Jim Smith <jg...@moya.tamu.edu>.
On Thu, Mar 21, 2002 at 02:01:49PM -0500, Michael Alan Dorman wrote:
> Perrin Harkins <pe...@elem.com> writes:
> > This sort of begs the question: why not use DB 3.x? Is there some new
> > feature you need in DB 4?
>
> Anecdotaly, I believe the OpenLDAP and Cyrus projects have both found
> DB4 to be more reliable under load than DB3.
To add to the anecdotes...
We have been using the predecessor of Sendmail, Inc's SAMS product
(ultimately derived from Cyrus, I believe) and have seen some
problems with locking and sleepycat db's resulting in hanging IMAP
servers or database corruption. I would not blame mod_perl first --
it's not responsible for the database.
I'm not certain that the problem is in the IMAP server either -- but
the environment consists of the IMAP server, POP server, tcl scripts,
and web interface all trying to access a sleepycat database.
Sendmail, Inc. has mentioned (I believe privately) that they feel the
problems have been resolved in the newer versions of the sleepycat
db. I'm not trying to put words in their mouth, so I could be
mistaken, but this is afaik.
Before blaming mod_perl, I would run the server with the -X option to
make sure the script is only being called once within the mod_perl
environment (just being paranoid). Multiple (almost) simultaneous
invocations might be causing some of the locking problems. They
wouldn't show up then when being run from the command line.
my $0.02
--jim
Re: Berkeley DB 4.0.14 not releasing lockers under mod_perl
Posted by Michael Alan Dorman <md...@debian.org>.
Perrin Harkins <pe...@elem.com> writes:
> This sort of begs the question: why not use DB 3.x? Is there some new
> feature you need in DB 4?
Anecdotaly, I believe the OpenLDAP and Cyrus projects have both found
DB4 to be more reliable under load than DB3.
Mike.
Re: Berkeley DB 4.0.14 not releasing lockers under mod_perl
Posted by Perrin Harkins <pe...@elem.com>.
Dan Wilga wrote:
> If I either use DB 3.x or even run this from the commandline (bypassing
> mod_perl) under DB 4 the problem goes away: only one locker is allocated
> per loop, and therefore the total number used does not increase
> unexpectedly.
This sort of begs the question: why not use DB 3.x? Is there some new
feature you need in DB 4?
It's been a little while since I messed with this stuff, but I don't
think you need to tear down the Env each time. I think you can just
create it the first time and reuse it after that. Maybe that will help.
(Ah, Mount Holyoke. As a Five College alumn, I remember it well.)
- Perrin
Re: Berkeley DB 4.0.14 not releasing lockers under mod_perl
Posted by Dan Wilga <dw...@MtHolyoke.edu>.
At 2:03 PM -0500 3/21/02, Aaron Ross wrote:
>
>> I'm testing with the Perl script below, with the filename ending
>> ".mperl" (which, in my configuration, causes it to run as a mod_perl
>> registry script).
>
> I would re-write it as a handler and see if Apache::Registry is partly
>to blame.
Sorry, I figured out that the problem persists even when mod_perl is
completely disabled, so the problem must lie in either Apache or
Berkeley DB. (As a diagnostic, I think I'll try disabling
mod_auth_db.* and mod_ssl, since they both use DB.)
Thanks for the quick responses, folks!
Dan Wilga dwilga@mtholyoke.edu
Web Technology Specialist http://www.mtholyoke.edu
Mount Holyoke College Tel: 413-538-3027
South Hadley, MA 01075 "I have a bold and cunning plan" - Baldric
Re: Berkeley DB 4.0.14 not releasing lockers under mod_perl
Posted by Dan Wilga <dw...@MtHolyoke.edu>.
Based on others' comments, I found the solution to the problem. I
just added a db_close(), and the problem is really, truly gone. For
good measure, I put a db_sync() in as well.
Why db_close isn't necessary with 3.x, but is with 4.0 is the weird
part. Maybe this suggests a bug in 4.0? Or is it just bad style to
have left it out in the first place? I was under the impression that
the untie/undef should have accomplished this implicitly within
BerkeleyDB.pm.
Here is the modified test script:
--------- TEST SCRIPT ---------
#!/usr/bin/perl
use strict;
use BerkeleyDB qw( DB_CREATE DB_INIT_MPOOL DB_INIT_CDB );
# Change me to something appropriate for your system
my $dir='/home/httpd/some/directory';
system( "rm $dir/__db* $dir/TESTdb" );
foreach( 1..5 ) {
my $env = open_env($dir);
my %hash;
my $db = open_db( "TESTdb", \%hash, $env );
$db->db_sync(); ### NEW
$db->db_close(); ### NEW
untie %hash;
undef $db;
undef $env;
}
print "HTTP/1.1 200\nContent-type: text/plain\n\n";
print `db_stat -c -h $dir`;
print "\n";
sub open_env {
my $env = new BerkeleyDB::Env(
-Flags=>DB_INIT_MPOOL|DB_INIT_CDB|DB_CREATE,
-Home=> $_[0],
);
die "Could not create env: $! ".$BerkeleyDB::Error. "\n" if !$env;
return $env;
}
sub open_db {
my( $file, $Rhash, $env ) = @_;
my $db_key = tie( %{$Rhash}, 'BerkeleyDB::Btree',
-Flags=>DB_CREATE,
-Filename=>$file,
-Env=>$env );
die "Can't open $file: $! ".$BerkeleyDB::Error."\n" if !$db_key;
return $db_key;
}
Dan Wilga dwilga@mtholyoke.edu
Web Technology Specialist http://www.mtholyoke.edu
Mount Holyoke College Tel: 413-538-3027
South Hadley, MA 01075 "Seduced by the chocolate side of the Force"
Re: Berkeley DB 4.0.14 not releasing lockers under mod_perl
Posted by Dan Wilga <dw...@MtHolyoke.edu>.
Aaron Ross wrote:
> > > my $db_key = tie( %{$Rhash}, 'BerkeleyDB::Btree',
> > > -Flags=>DB_CREATE,
> > > -Filename=>$file,
> > > -Env=>$env );
> > > die "Can't open $file: $! ".$BerkeleyDB::Error."\n" if !$db_key;
> > > return $db_key;
>
>I was wondering if using tie() instead of just the OO interface was
>causing a problem. Maybe rewriting the code to just use new() would
>help? Of course that means losing the hash interface.
Of course, that would avoid the problem, because I wouldn't be using
DB's built-in locking anymore :-) Using tie() means losing the
environment, which is needed for DB's internal locking.
In a production environment with thousands of hits per hour coming in
on multiple threads, I really don't trust flock() to do the job.
Dan Wilga dwilga@mtholyoke.edu
Web Technology Specialist http://www.mtholyoke.edu
Mount Holyoke College Tel: 413-538-3027
South Hadley, MA 01075 "Who left the cake out in the rain?"
Re: Berkeley DB 4.0.14 not releasing lockers under mod_perl
Posted by Aaron Ross <ro...@alias-i.com>.
> >> I'm testing with the Perl script below, with the filename ending
> >> ".mperl" (which, in my configuration, causes it to run as a mod_perl
> >> registry script).
> >
> > I would re-write it as a handler and see if Apache::Registry is partly
> >to blame.
>
> I tried doing it as a handler, using the configuration below (and the
> appropriate changes in the source) and the problem persists. So it
> doesn't seem to be Registry's fault.
Bummer. I wish I had a machine to test this code on, but i won't until the
end of next week. it's ugly, but you could use Devel::Peek to look at why
garbage collection isn't working, maybe there is a circular reference in
BerkeleyDB.
Ugh. Please keep me informed.
-- Aaron
Re: Berkeley DB 4.0.14 not releasing lockers under mod_perl
Posted by Aaron Ross <ro...@alias-i.com>.
> (As an aside, I'm not sure I'll ever get over undef being proper closing
> of a database connection; it seems so synonomous to free([23]). I
> expect something like $db->db_close() or something.)
You mean like $db->db_close()? :)
http://sleepycat.com/docs/api_c/db_close.html
> > my $db_key = tie( %{$Rhash}, 'BerkeleyDB::Btree',
> > -Flags=>DB_CREATE,
> > -Filename=>$file,
> > -Env=>$env );
> > die "Can't open $file: $! ".$BerkeleyDB::Error."\n" if !$db_key;
> > return $db_key;
I was wondering if using tie() instead of just the OO interface was
causing a problem. Maybe rewriting the code to just use new() would
help? Of course that means losing the hash interface.
grasping at straws, aaron
Re: Berkeley DB 4.0.14 not releasing lockers under mod_perl
Posted by Dan Wilga <dw...@MtHolyoke.edu>.
At 12:27 AM -0500 3/22/02, Ed Grimm wrote:
>Does shutting down apache free up your locks?
Nope. They are held open in the environment files (__db00*). When the
server is restarted, the count increases where it left off.
Just to be clear, the items left allocated are what DB calls
"lockers", which are different from actual "locks". A locker is just
a slot for data concerning an active lock (think of it as an array
index).
Normally, DB will grow the array by one, store info about the lock in
it, and then when the lock is freed, shrink the array. Instead,
what's happening is that the lock itself is freed, but the array
doesn't shrink (the locker isn't freed). So the next time a new lock
is required, it allocates another locker. If you do this enough (1000
times), the environment file runs out of room, and opens fail.
>(As an aside, I'm not sure I'll ever get over undef being proper closing
>of a database connection; it seems so synonomous to free([23]). I
>expect something like $db->db_close() or something.)
Agreed!
Dan Wilga dwilga@mtholyoke.edu
Web Technology Specialist http://www.mtholyoke.edu
Mount Holyoke College Tel: 413-538-3027
South Hadley, MA 01075 "I have a bold and cunning plan" - Baldric
Re: Berkeley DB 4.0.14 not releasing lockers under mod_perl
Posted by Ed Grimm <ed...@tgape.org>.
Does shutting down apache free up your locks?
(As an aside, I'm not sure I'll ever get over undef being proper closing
of a database connection; it seems so synonomous to free([23]). I
expect something like $db->db_close() or something.)
Ed
On Thu, 21 Mar 2002, Dan Wilga wrote:
> At 2:03 PM -0500 3/21/02, Aaron Ross wrote:
>>
>>> I'm testing with the Perl script below, with the filename ending
>>> ".mperl" (which, in my configuration, causes it to run as a mod_perl
>>> registry script).
>>
>> I would re-write it as a handler and see if Apache::Registry is partly
>>to blame.
>
> I tried doing it as a handler, using the configuration below (and the
> appropriate changes in the source) and the problem persists. So it
> doesn't seem to be Registry's fault.
>
> <Location /dan>
> SetHandler perl-script
> PerlHandler DanTest
> </Location>
>
> ---- source code ----
>
> #!/usr/bin/perl
>
> package DanTest;
>
> use strict;
> use BerkeleyDB qw( DB_CREATE DB_INIT_MPOOL DB_INIT_CDB );
>
> my $dir='/home/httpd/some/path';
>
> sub handler {
> system( "rm $dir/__db* $dir/TESTdb" );
>
> foreach( 1..5 ) {
> my $env = open_env($dir);
> my %hash;
> my $db = open_db( "TESTdb", \%hash, $env );
> untie %hash;
> undef $db;
> undef $env;
> }
> print "HTTP/1.1 200\nContent-type: text/plain\n\n";
> print `db_stat -c -h $dir`;
> print "\n";
> }
>
> sub open_env {
> my $env = new BerkeleyDB::Env(
> -Flags=>DB_INIT_MPOOL|DB_INIT_CDB|DB_CREATE,
> -Home=> $_[0],
> );
> die "Could not create env: $! ".$BerkeleyDB::Error. "\n" if !$env;
> return $env;
> }
>
> sub open_db {
> my( $file, $Rhash, $env ) = @_;
> my $db_key = tie( %{$Rhash}, 'BerkeleyDB::Btree',
> -Flags=>DB_CREATE,
> -Filename=>$file,
> -Env=>$env );
> die "Can't open $file: $! ".$BerkeleyDB::Error."\n" if !$db_key;
> return $db_key;
> }
>
> 1;
>
>
> Dan Wilga dwilga@mtholyoke.edu
> Web Technology Specialist http://www.mtholyoke.edu
> Mount Holyoke College Tel: 413-538-3027
> South Hadley, MA 01075 "Seduced by the chocolate side of the Force"
>
Re: Berkeley DB 4.0.14 not releasing lockers under mod_perl
Posted by Dan Wilga <dw...@MtHolyoke.edu>.
At 2:03 PM -0500 3/21/02, Aaron Ross wrote:
>
>> I'm testing with the Perl script below, with the filename ending
>> ".mperl" (which, in my configuration, causes it to run as a mod_perl
>> registry script).
>
> I would re-write it as a handler and see if Apache::Registry is partly
>to blame.
I tried doing it as a handler, using the configuration below (and the
appropriate changes in the source) and the problem persists. So it
doesn't seem to be Registry's fault.
<Location /dan>
SetHandler perl-script
PerlHandler DanTest
</Location>
---- source code ----
#!/usr/bin/perl
package DanTest;
use strict;
use BerkeleyDB qw( DB_CREATE DB_INIT_MPOOL DB_INIT_CDB );
my $dir='/home/httpd/some/path';
sub handler {
system( "rm $dir/__db* $dir/TESTdb" );
foreach( 1..5 ) {
my $env = open_env($dir);
my %hash;
my $db = open_db( "TESTdb", \%hash, $env );
untie %hash;
undef $db;
undef $env;
}
print "HTTP/1.1 200\nContent-type: text/plain\n\n";
print `db_stat -c -h $dir`;
print "\n";
}
sub open_env {
my $env = new BerkeleyDB::Env(
-Flags=>DB_INIT_MPOOL|DB_INIT_CDB|DB_CREATE,
-Home=> $_[0],
);
die "Could not create env: $! ".$BerkeleyDB::Error. "\n" if !$env;
return $env;
}
sub open_db {
my( $file, $Rhash, $env ) = @_;
my $db_key = tie( %{$Rhash}, 'BerkeleyDB::Btree',
-Flags=>DB_CREATE,
-Filename=>$file,
-Env=>$env );
die "Can't open $file: $! ".$BerkeleyDB::Error."\n" if !$db_key;
return $db_key;
}
1;
Dan Wilga dwilga@mtholyoke.edu
Web Technology Specialist http://www.mtholyoke.edu
Mount Holyoke College Tel: 413-538-3027
South Hadley, MA 01075 "Seduced by the chocolate side of the Force"
Re: Berkeley DB 4.0.14 not releasing lockers under mod_perl
Posted by Aaron Ross <ro...@alias-i.com>.
> I'm testing with the Perl script below, with the filename ending
> ".mperl" (which, in my configuration, causes it to run as a mod_perl
> registry script).
I would re-write it as a handler and see if Apache::Registry is partly
to blame.
hth, aaron