You are viewing a plain text version of this content. The canonical link for it is here.
Posted to modperl@perl.apache.org by Chris Ochs <ch...@paymentonline.net> on 2004/04/22 21:55:47 UTC

BerkeleyDB with mod perl

I know this isn't exactly mod perl specific, but since there isn't a mailing
list for BerkeleyDB and this is an issue I know people using BerkeleyDB and
mod perl have probably run accross I thought I would post it here.

I'm using the latest BerkeleyDB perl module on freebsd 5.2 with mod perl
1.xx.  Following is the BerkeleyDB code I am using.  When I start running
lots of concurrent processes that run db_put I get the error 'Locker does
not exist' in $BerkeleyDB::Error and db_put returns 'Invalid argument'.   I
must be setting up the environment incorrectly somehow.  I am trying to use
the BerkeleyDB concurrent data store mode which I assumed would handle all
the locking for me.


my %Cache;
my $env = new BerkeleyDB::Env(
      -Home  => '/tmp/bdb',
      -Flags => DB_INIT_CDB | DB_CREATE | DB_INIT_MPOOL,
      )
      or die "can't create BerkelyDB::Env: $!";

  my $bdb = tie %Cache, 'BerkeleyDB::Hash',
     -Filename => '/tmp/bdb/bdbfile',
     -Flags    => DB_CREATE,
     -Mode     => 0640,
     -Env      => $env
     or die ("Can't tie to /tmp/bdb/bdbdfile: $!");


My db_put looks like this:
my $status = $bdb->db_put($key,$value);

Chris


-- 
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html

[OT] Re: BerkeleyDB with mod perl

Posted by William McKee <wi...@knowmad.com>.
Hi Chris,

I sympathize with you. I've been through my share of problems with
BerkeleyDB as well. So far, I haven't had problems using it with
mod_perl. I wrote a stress test tool[1] to prove to myself that
BerkeleyDB was indeed safe to use in concurrent processes. You can set
it to run as many processes doing as many inserts and reads as you would
like. You may want to run that against a copy of your database to see if
it exposes any errors.

My mod_perl setup is on a Windows 2000 system and seems to be stable. I
setup my environment in the same way that you are doing except that I
pass in -ErrFile to the BerkeleyDB::Env setup. You may want to try that
to see if it provides any more details for you.

The only other suggestion I would make is to try without the tied
interface if you don't have to change too much code. Since you're using
db_put, you could setup just the handle via the following:

    my $bdb  = new BerkeleyDB::Btree
        -Flags    => DB_CREATE,
        -Filename => $dbfile,
        -Env      => $env,
        or die "Cannot open file $dbfile: $! - '$BerkeleyDB::Error'\n";


Good luck and let us know if you find a solution,
William

[1] http://perlmonks.org/index.pl?node_id=330510

-- 
Knowmad Services Inc.
http://www.knowmad.com

-- 
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html

Re: BerkeleyDB with mod perl

Posted by Chris Ochs <ch...@paymentonline.net>.
Yep sticking it in a global and handling it like Apache::DBI works.


----- Original Message ----- 
From: "Perrin Harkins" <pe...@elem.com>
To: "Chris Ochs" <ch...@paymentonline.net>
Cc: "Modperl List" <mo...@perl.apache.org>
Sent: Thursday, April 22, 2004 1:36 PM
Subject: Re: BerkeleyDB with mod perl


> On Thu, 2004-04-22 at 16:34, Chris Ochs wrote:
> > If I set the environment in startup.pl but open the db
> > handle in my perl handler every time the page is accessed the
performance
> > drops by about 300%, but it does make the errors go away.
>
> There is no need to open the handle every time.  You can open it once
> and put it in a global.  Just do it after the fork.  It's the same idea
> as Apache::DBI.
>
> - Perrin
>
>
> -- 
> Report problems: http://perl.apache.org/bugs/
> Mail list info: http://perl.apache.org/maillist/modperl.html
> List etiquette: http://perl.apache.org/maillist/email-etiquette.html
>
>


-- 
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html

Re: BerkeleyDB with mod perl

Posted by Perrin Harkins <pe...@elem.com>.
On Thu, 2004-04-22 at 16:34, Chris Ochs wrote:
> If I set the environment in startup.pl but open the db
> handle in my perl handler every time the page is accessed the performance
> drops by about 300%, but it does make the errors go away.

There is no need to open the handle every time.  You can open it once
and put it in a global.  Just do it after the fork.  It's the same idea
as Apache::DBI.

- Perrin


-- 
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html

Re: BerkeleyDB with mod perl

Posted by Chris Ochs <ch...@paymentonline.net>.
Sheesh yes I forgot to say that I set the environment and open the db handle
in startup.pl.  If I set the environment in startup.pl but open the db
handle in my perl handler every time the page is accessed the performance
drops by about 300%, but it does make the errors go away.

Chris


----- Original Message ----- 
From: "Perrin Harkins" <pe...@elem.com>
To: "Chris Ochs" <ch...@paymentonline.net>
Cc: "Modperl List" <mo...@perl.apache.org>
Sent: Thursday, April 22, 2004 1:12 PM
Subject: Re: BerkeleyDB with mod perl


> On Thu, 2004-04-22 at 15:55, Chris Ochs wrote:
> > I'm using the latest BerkeleyDB perl module on freebsd 5.2 with mod perl
> > 1.xx.  Following is the BerkeleyDB code I am using.  When I start
running
> > lots of concurrent processes that run db_put I get the error 'Locker
does
> > not exist' in $BerkeleyDB::Error and db_put returns 'Invalid argument'.
>
> Any chance you might be opening the db handle before forking, i.e. in
> your startup.pl?  I suspect that would cause problems.
>
> - Perrin
>
>


-- 
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html

Re: BerkeleyDB with mod perl

Posted by Perrin Harkins <pe...@elem.com>.
On Thu, 2004-04-22 at 16:27, William McKee wrote:
> My gut reaction is "Yes, this makes sense." However, I'm scratching my
> head trying to understand why.

There are file handles involved.  You can't fork with an open file and
expect to be able to do things with it from each process, at least not
without awareness and coordination between them.  I don't know enough
about BDB internals to say for sure what would happen, but this sort of
thing is often a problem with forking servers.

- Perrin


-- 
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html

Re: BerkeleyDB with mod perl

Posted by William McKee <wi...@knowmad.com>.
On Thu, Apr 22, 2004 at 04:12:51PM -0400, Perrin Harkins wrote:
> Any chance you might be opening the db handle before forking, i.e. in
> your startup.pl?  I suspect that would cause problems.

Hi Perrin,

My gut reaction is "Yes, this makes sense." However, I'm scratching my
head trying to understand why. It seems that all the children would
share the same locked environment. But perhaps, they shouldn't be
sharing the same handle (is this the equivalent of a connection?) to
that environment. Is that where the problem lies?  Does each child need
to have a separate handle?

Thanks,
William

-- 
Knowmad Services Inc.
http://www.knowmad.com

-- 
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html

Re: BerkeleyDB with mod perl

Posted by Perrin Harkins <pe...@elem.com>.
On Thu, 2004-04-22 at 15:55, Chris Ochs wrote:
> I'm using the latest BerkeleyDB perl module on freebsd 5.2 with mod perl
> 1.xx.  Following is the BerkeleyDB code I am using.  When I start running
> lots of concurrent processes that run db_put I get the error 'Locker does
> not exist' in $BerkeleyDB::Error and db_put returns 'Invalid argument'.

Any chance you might be opening the db handle before forking, i.e. in
your startup.pl?  I suspect that would cause problems.

- Perrin


-- 
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html