You are viewing a plain text version of this content. The canonical link for it is here.
Posted to modperl@perl.apache.org by Adi Fairbank <ad...@certsite.com> on 2001/02/28 05:55:32 UTC
mod_perl shared memory with MM
I am trying to squeeze more performance out of my persistent session cache. In
my application, the Storable image size of my sessions can grow upwards of
100-200K. It can take on the order of 200ms for Storable to deserialize and
serialize this on my (lousy) hardware.
I'm looking at RSE's MM and the Perl module IPC::MM as a persistent session
cache. Right now IPC::MM doesn't support multi-dimensional Perl data
structures, nor blessed references, so I will have to extend it to support
these.
My question is: is anyone else using IPC::MM under mod_perl? .. would you if it
supported multi-dimensional Perl data?
My other question is: since this will be somewhat moot once Apache 2.0 +
mod_perl 2.0 are stable, is it worth the effort? What's the ETA on mod_perl
2.0? Should I spend my effort helping with that instead?
Any comments appreciated,
-Adi
Re: mod_perl shared memory with MM
Posted by Joshua Chamas <jo...@chamas.com>.
Adi Fairbank wrote:
>
> I am trying to squeeze more performance out of my persistent session cache. In
> my application, the Storable image size of my sessions can grow upwards of
> 100-200K. It can take on the order of 200ms for Storable to deserialize and
> serialize this on my (lousy) hardware.
>
Its a different approach, but I use simple MLDBM + SDBM_File
when possible, as its really fast for small records, but it has
that 1024 byte limit per record! I am releasing a wrapper
to CPAN ( on its way now ) called MLDBM::Sync that handles
concurrent locking & i/o flushing for you. One advantage
of this approach is that your session state will persist
through a server reboot if its written to disk.
I also wrote a wrapper for SDBM_File called MLDBM::Sync::SDBM_File
that overcomes the 1024 byte limit per record. The below
numbers were for a benchmark on my dual PIII 450, linux 2.2.14,
SCSI raid-1 ext2 fs mounted async. The benchmark can be found
in the MLDBM::Sync package in the bench directory once it makes
it to CPAN.
With MLDBM ( perldoc MLDBM ) you can use Storable or
XS Data::Dumper method for serialization as well as
various DBMs.
--Josh
=== INSERT OF 50 BYTE RECORDS ===
Time for 100 write/read's for SDBM_File 0.12 seconds 12288 bytes
Time for 100 write/read's for MLDBM::Sync::SDBM_File 0.14 seconds 12288 bytes
Time for 100 write/read's for GDBM_File 2.07 seconds 18066 bytes
Time for 100 write/read's for DB_File 2.48 seconds 20480 bytes
=== INSERT OF 500 BYTE RECORDS ===
Time for 100 write/read's for SDBM_File 0.21 seconds 658432 bytes
Time for 100 write/read's for MLDBM::Sync::SDBM_File 0.51 seconds 135168 bytes
Time for 100 write/read's for GDBM_File 2.29 seconds 63472 bytes
Time for 100 write/read's for DB_File 2.44 seconds 114688 bytes
=== INSERT OF 5000 BYTE RECORDS ===
(skipping test for SDBM_File 1024 byte limit)
Time for 100 write/read's for MLDBM::Sync::SDBM_File 1.30 seconds 2101248 bytes
Time for 100 write/read's for GDBM_File 2.55 seconds 832400 bytes
Time for 100 write/read's for DB_File 3.27 seconds 839680 bytes
=== INSERT OF 20000 BYTE RECORDS ===
(skipping test for SDBM_File 1024 byte limit)
Time for 100 write/read's for MLDBM::Sync::SDBM_File 4.54 seconds 13162496 bytes
Time for 100 write/read's for GDBM_File 5.39 seconds 2063912 bytes
Time for 100 write/read's for DB_File 4.79 seconds 2068480 bytes
Re: [OT] Re: mod_perl shared memory with MM
Posted by Perrin Harkins <pe...@primenet.com>.
> I'm very intrigued by your thinking on locking. I had never
> considered the transaction based approach to caching you are referring
> to. I'll take this up privately with you, because we've strayed far
> off the mod_perl topic, although I find it fascinating.
One more suggestion before you take this off the list: it's nice to have
both. There are uses for explicit locking (I remember Randal saying he
wished File::Cache had some locking support), but most people will be happy
with atomic updates, and that's usually faster. Gunther's eXtropia stuff
supports various locking options, and you can read some of the reasoning
behind it in the docs at
http://new.extropia.com/development/webware2/webware2.html. (See chapters
13 and 18.)
> > - why don't you use 'real' constants for $SUCCESS and the like? (use
> > constant)
>
> Two reasons, mostly historical, and not necessarily good ones.
>
> One, I benchmarked some code once that required high performance, and
> the use of constants was just slightly slower.
Ick.
> Two, I like the syntax $hash{$CONSTANT}. If I remember correctly,
> $hash{CONSTANT} didn't work. This may have changed in newer versions
> of Perl.
No, the use of constants as hash keys or in interpolated strings still
doesn't work. I tried the constants module in my last project, and I found
it to be more trouble than it was worth. It's annoying to have to write
things like $hash{&CONSTANT} or "string @{[&CONSTANT]}".
> Do you know if Storeable is definitely faster?
It is, and it's now part of the standard distribution.
http://www.astray.com/pipermail/foo/2000-August/000169.html
- Perrin
Re: [OT] Re: mod_perl shared memory with MM
Posted by Greg Cope <gr...@rubberplant.freeserve.co.uk>.
Perrin Harkins wrote:
>
> > Can I ask why you are not useing IPC::Sharedlight (as its pure C and
> > apparently much faster than IPC::Shareable - I've never benchmarked it
> > as I've also used IPC::Sharedlight).
>
> Full circle back to the original topic...
> IPC::MM is implemented in C and offers an actual hash interface backed by a
> BTree in shared memory. IPC::ShareLite only works for individual scalars.
>
Not tried that one !
I'ce used the obvious Sharedlight plus Storable to serialise hashes.
> It wouldn't surprise me if a file system approach was faster than either of
> these on Linux, because of the agressive caching.
I would be an interesting benchmark ... Althought it may only be a
performance win on a lightly loaded machine,the assymption being that
the stat'ing is fast on a lowly loaded system with fast understressed
disks. I could be completly wrong here tho ;-).
Has anyone used the file system approach on a RAM disk ?
Greg
>
> - Perrin
Re: [OT] Re: mod_perl shared memory with MM
Posted by Perrin Harkins <pe...@primenet.com>.
> Can I ask why you are not useing IPC::Sharedlight (as its pure C and
> apparently much faster than IPC::Shareable - I've never benchmarked it
> as I've also used IPC::Sharedlight).
Full circle back to the original topic...
IPC::MM is implemented in C and offers an actual hash interface backed by a
BTree in shared memory. IPC::ShareLite only works for individual scalars.
It wouldn't surprise me if a file system approach was faster than either of
these on Linux, because of the agressive caching.
- Perrin
Re: [OT] Re: mod_perl shared memory with MM
Posted by Greg Cope <gr...@rubberplant.freeserve.co.uk>.
DeWitt Clinton wrote:
>
> On Sun, Mar 11, 2001 at 03:33:12PM +0100, Christian Jaeger wrote:
>
> > I've looked at Cache::FileCache now and think it's (currently) not
> > possible to use for IPC::FsSharevars:
> >
> > I really miss locking capabilities. Imagine a script that reads a
> > value at the beginning of a request and writes it back at the end of
> > the request. If it's not locked during this time, another instance
> > can read the same value and then write another change back which is
> > then overwritten by the first instance.
>
> I'm very intrigued by your thinking on locking. I had never
> considered the transaction based approach to caching you are referring
> to. I'll take this up privately with you, because we've strayed far
> off the mod_perl topic, although I find it fascinating.
>
> > - why don't you use 'real' constants for $SUCCESS and the like? (use
> > constant)
>
> Two reasons, mostly historical, and not necessarily good ones.
>
> One, I benchmarked some code once that required high performance, and
> the use of constants was just slightly slower.
>
> Two, I like the syntax $hash{$CONSTANT}. If I remember correctly,
> $hash{CONSTANT} didn't work. This may have changed in newer versions
> of Perl.
>
> Obviously those are *very* small issues, and so it is mostly by habit
> that I don't use constant. I would consider changing, but it would
> mean asking everyone using the code to change too, because they
> currently import and use the constants as Exported scalars.
>
> Do you know of a very important reason to break compatibility and
> force the switch? I'm not opposed to switching if I have to, but I'd
> like to minimize the impact on the users.
>
> > - you probably should either append the userid of the process to
> > /tmp/FileCache or make this folder globally writeable (and set the
> > sticky flag). Otherwise other users get a permission error.
>
> As of version 0.03, the cache directories, but not the cache entries,
> are globally writable by default. Users can override this by changing
> the 'directory_umask' option, or keep data private altogether by
> changing the 'cache_root'. What version did you test with? There may
> be a bug in there.
>
> > - why don't you use Storable.pm? It should be much faster than Data::Dumper
>
> The TODO contains "Replace Data::Dumper with Storable (maybe)". :)
>
> The old File::Cache module used Storable, btw.
>
> It will be trivial to port the new Cache::FileCache to use Storable.
> I simply wanted to wait until I had the benchmarking code so I could
> be sure that Storeable was faster. Actually, I'm not 100% sure that I
> expect Storeable to be faster than Data::Dumper. If Data::Dumper
> turns out to be about equally fast, then I'll stay with it, because it
> is available on all Perl installations, I believe.
>
> Do you know if Storeable is definitely faster? If you have benchmarks
> then I am more than happy to switch now. Or, do you know of a reason,
> feature wise, that I should switch? Again, it is trivial to do so.
I've found it to be arround 5 - 10 % faster - on simple stuff on some
benchmarking I did arround a year ago.
Can I ask why you are not useing IPC::Sharedlight (as its pure C and
apparently much faster than IPC::Shareable - I've never benchmarked it
as I've also used IPC::Sharedlight).
Greg
>
> > >I have some preliminary benchmark code -- only good for relative
> > >benchmarking, but it is a start. I'd be happy to post the results
> > >here if people are interested.
> >
> > Could you send me the code?, then I'll look into benchmarking my
> > module too.
>
> I checked it in as Cache::CacheBenchmark. It isn't good code, nor
> does it necessarily work just yet. I simply checked it in while I was
> in the middle of working on it. I'm turning it into a real
> benchmarking class for the cache, and hopefully that will help you a
> little bit.
>
> Cheers,
>
> -DeWitt
[OT] Re: mod_perl shared memory with MM
Posted by DeWitt Clinton <de...@unto.net>.
On Sun, Mar 11, 2001 at 03:33:12PM +0100, Christian Jaeger wrote:
> I've looked at Cache::FileCache now and think it's (currently) not
> possible to use for IPC::FsSharevars:
>
> I really miss locking capabilities. Imagine a script that reads a
> value at the beginning of a request and writes it back at the end of
> the request. If it's not locked during this time, another instance
> can read the same value and then write another change back which is
> then overwritten by the first instance.
I'm very intrigued by your thinking on locking. I had never
considered the transaction based approach to caching you are referring
to. I'll take this up privately with you, because we've strayed far
off the mod_perl topic, although I find it fascinating.
> - why don't you use 'real' constants for $SUCCESS and the like? (use
> constant)
Two reasons, mostly historical, and not necessarily good ones.
One, I benchmarked some code once that required high performance, and
the use of constants was just slightly slower.
Two, I like the syntax $hash{$CONSTANT}. If I remember correctly,
$hash{CONSTANT} didn't work. This may have changed in newer versions
of Perl.
Obviously those are *very* small issues, and so it is mostly by habit
that I don't use constant. I would consider changing, but it would
mean asking everyone using the code to change too, because they
currently import and use the constants as Exported scalars.
Do you know of a very important reason to break compatibility and
force the switch? I'm not opposed to switching if I have to, but I'd
like to minimize the impact on the users.
> - you probably should either append the userid of the process to
> /tmp/FileCache or make this folder globally writeable (and set the
> sticky flag). Otherwise other users get a permission error.
As of version 0.03, the cache directories, but not the cache entries,
are globally writable by default. Users can override this by changing
the 'directory_umask' option, or keep data private altogether by
changing the 'cache_root'. What version did you test with? There may
be a bug in there.
> - why don't you use Storable.pm? It should be much faster than Data::Dumper
The TODO contains "Replace Data::Dumper with Storable (maybe)". :)
The old File::Cache module used Storable, btw.
It will be trivial to port the new Cache::FileCache to use Storable.
I simply wanted to wait until I had the benchmarking code so I could
be sure that Storeable was faster. Actually, I'm not 100% sure that I
expect Storeable to be faster than Data::Dumper. If Data::Dumper
turns out to be about equally fast, then I'll stay with it, because it
is available on all Perl installations, I believe.
Do you know if Storeable is definitely faster? If you have benchmarks
then I am more than happy to switch now. Or, do you know of a reason,
feature wise, that I should switch? Again, it is trivial to do so.
> >I have some preliminary benchmark code -- only good for relative
> >benchmarking, but it is a start. I'd be happy to post the results
> >here if people are interested.
>
> Could you send me the code?, then I'll look into benchmarking my
> module too.
I checked it in as Cache::CacheBenchmark. It isn't good code, nor
does it necessarily work just yet. I simply checked it in while I was
in the middle of working on it. I'm turning it into a real
benchmarking class for the cache, and hopefully that will help you a
little bit.
Cheers,
-DeWitt
Re: mod_perl shared memory with MM
Posted by Christian Jaeger <ch...@sl.ethz.ch>.
At 22:23 Uhr -0500 10.3.2001, DeWitt Clinton wrote:
>On Sat, Mar 10, 2001 at 04:35:02PM -0800, Perrin Harkins wrote:
> > Christian Jaeger wrote:
>> > Yes, it uses a separate file for each variable. This way also locking
>> > is solved, each variable has it's own file lock.
>>
>> You should take a look at DeWitt Clinton's Cache::FileCache module,
>> announced on this list. It might make sense to merge your work into
>> that module, which is the next generation of the popular File::Cache
>> module.
>
>Yes! I'm actively looking for additional developers for the Perl
>Cache project. I'd love new implementations of the Cache interface.
>Cache::BerkeleyDBCache would be wonderful. Check out:
>
> http://sourceforge.net/projects/perl-cache/
>
>For what it is worth, I don't explicitly lock. I do atomic writes
>instead, and have yet to hear anyone report a problem in the year the
>code has been public.
I've looked at Cache::FileCache now and think it's (currently) not
possible to use for IPC::FsSharevars:
I really miss locking capabilities. Imagine a script that reads a
value at the beginning of a request and writes it back at the end of
the request. If it's not locked during this time, another instance
can read the same value and then write another change back which is
then overwritten by the first instance.
IPC::FsSharevars even goes one step further: instead of locking
everything for a particular session, it only locks individual
variables. So you can say "I use the variables $foo and %bar from
session 12345 and will write %bar back", in which case %bar of
session 12345 is locked until it is written back, while $foo and @baz
are still unlocked and may be read (and written) by other instances.
:-) Such behaviour is useful if you have framesets where a browser
may request several frames of the same session in parallel (you can
see an example on http://testwww.ethz.ch, click on 'Suche' then on
the submit button, the two appearing frames are executed in parallel
and both access different session variables), or for handling session
independant (global) data.
One thing to be careful about in such situations is dead locking.
IPC::FsSharevars prevents dead locks by getting all needed locks at
the same time (this is done by first requesting a general session
lock and then trying to lock all needed variable container files - if
it fails, the session lock is freed again and the process waits for a
unix signal indicating a change in the locking states). Getting all
locks at the same time is more efficient than getting locks always in
the same order.
BTW some questions/suggestions for DeWitt:
- why don't you use 'real' constants for $SUCCESS and the like? (use constant)
- you probably should either append the userid of the process to
/tmp/FileCache or make this folder globally writeable (and set the
sticky flag). Otherwise other users get a permission error.
- why don't you use Storable.pm? It should be much faster than Data::Dumper
>I have some preliminary benchmark code -- only good for relative
>benchmarking, but it is a start. I'd be happy to post the results
>here if people are interested.
Could you send me the code?, then I'll look into benchmarking my module too.
Re: mod_perl shared memory with MM
Posted by Perrin Harkins <pe...@primenet.com>.
> I have some preliminary benchmark code -- only good for relative
> benchmarking, but it is a start. I'd be happy to post the results
> here if people are interested.
Please do.
- Perrin
Re: mod_perl shared memory with MM
Posted by DeWitt Clinton <de...@unto.net>.
On Sat, Mar 10, 2001 at 04:35:02PM -0800, Perrin Harkins wrote:
> Christian Jaeger wrote:
> > Yes, it uses a separate file for each variable. This way also locking
> > is solved, each variable has it's own file lock.
>
> You should take a look at DeWitt Clinton's Cache::FileCache module,
> announced on this list. It might make sense to merge your work into
> that module, which is the next generation of the popular File::Cache
> module.
Yes! I'm actively looking for additional developers for the Perl
Cache project. I'd love new implementations of the Cache interface.
Cache::BerkeleyDBCache would be wonderful. Check out:
http://sourceforge.net/projects/perl-cache/
For what it is worth, I don't explicitly lock. I do atomic writes
instead, and have yet to hear anyone report a problem in the year the
code has been public.
> > It's a bit difficult to write a realworld benchmark.
>
> It certainly is. Benchmarking all of the options is something that I've
> always wanted to do and never find enough time for.
I have some preliminary benchmark code -- only good for relative
benchmarking, but it is a start. I'd be happy to post the results
here if people are interested.
-DeWitt
Re: mod_perl shared memory with MM
Posted by Perrin Harkins <pe...@primenet.com>.
Christian Jaeger wrote:
> Yes, it uses a separate file for each variable. This way also locking
> is solved, each variable has it's own file lock.
You should take a look at DeWitt Clinton's Cache::FileCache module,
announced on this list. It might make sense to merge your work into
that module, which is the next generation of the popular File::Cache
module.
> It's a bit difficult to write a realworld benchmark.
It certainly is. Benchmarking all of the options is something that I've
always wanted to do and never find enough time for.
> I've tried to
> use DB_File before but it was very slow when doing a sync after every
> write as is recommended in various documentation to make it
> multiprocess safe. What do you mean with BerkeleyDB, something
> different than DB_File?
BerkeleyDB.pm is an interface to later versions of the Berkeley DB
library. It has a shared memory cache, and does not require syncing or
opening and closing of files on every access. It has built-in locking,
which can be configured to work at a page level, allowing mutiple
simultaneous writers.
> Currently I don't use Mmap (are there no cross platform issues using
> that?), that might speed it up a bit more.
That would be a nice option. Take a look at Cache::Mmap before you
start.
- Perrin
Re: mod_perl shared memory with MM
Posted by Christian Jaeger <ch...@sl.ethz.ch>.
At 0:23 Uhr -0800 10.3.2001, Perrin Harkins wrote:
>On Sat, 10 Mar 2001, Christian Jaeger wrote:
>> For all of you trying to share session information efficently my
>> IPC::FsSharevars module might be the right thing. I wrote it after
>> having considered all the other solutions. It uses the file system
>> directly (no BDB/etc. overhead) and provides sophisticated locking
>> (even different variables from the same session can be written at the
>> same time).
>
>Sounds very interesting. Does it use a multi-file approach like
>File::Cache? Have you actually benchmarked it against BerkeleyDB? It's
>hard to beat BDB because it uses a shared memory buffer, but theoretically
>the file system buffer could do it since that's managed by the kernel.
Yes, it uses a separate file for each variable. This way also locking
is solved, each variable has it's own file lock.
It's a bit difficult to write a realworld benchmark. I've tried to
use DB_File before but it was very slow when doing a sync after every
write as is recommended in various documentation to make it
multiprocess safe. What do you mean with BerkeleyDB, something
different than DB_File?
Currently I don't use Mmap (are there no cross platform issues using
that?), that might speed it up a bit more.
Christian.
Re: mod_perl shared memory with MM
Posted by Perrin Harkins <pe...@primenet.com>.
On Sat, 10 Mar 2001, Christian Jaeger wrote:
> For all of you trying to share session information efficently my
> IPC::FsSharevars module might be the right thing. I wrote it after
> having considered all the other solutions. It uses the file system
> directly (no BDB/etc. overhead) and provides sophisticated locking
> (even different variables from the same session can be written at the
> same time).
Sounds very interesting. Does it use a multi-file approach like
File::Cache? Have you actually benchmarked it against BerkeleyDB? It's
hard to beat BDB because it uses a shared memory buffer, but theoretically
the file system buffer could do it since that's managed by the kernel.
- Perrin
Re: mod_perl shared memory with MM
Posted by Christian Jaeger <ch...@sl.ethz.ch>.
For all of you trying to share session information efficently my
IPC::FsSharevars module might be the right thing. I wrote it after
having considered all the other solutions. It uses the file system
directly (no BDB/etc. overhead) and provides sophisticated locking
(even different variables from the same session can be written at the
same time). I wrote it for my fastcgi based web app framework (Eile)
but it should be useable for mod_perl things as well (I'm awaiting
patches and suggestions in case it is not). It has not seen very much
real world testing yet.
You may find the manpage on
http://testwww.ethz.ch/perldoc/IPC/FsSharevars.pm and the module (no
Makefile.PL yet) under http://testwww.ethz.ch/eile/download/ .
Cheers
Christian.
Re: mod_perl shared memory with MM
Posted by "Alexander Farber (EED)" <ee...@eed.ericsson.se>.
Adi Fairbank wrote:
> Yeah, I was thinking about something like that at first, but I've never played
> with named pipes, and it didn't sound too safe after reading the perlipc man
> page. What do you use, Perl open() calls, IPC::Open2/3, IPC::ChildSafe, or
IPC:ChildSafe is a good module, I use it here to access ClearCase, but
it probably won't help you to exchange any data between Apache children
Re: mod_perl shared memory with MM
Posted by barries <ba...@slaysys.com>.
On Mon, Mar 12, 2001 at 12:14:26PM -0800, Sean Chittenden wrote:
> Sorry for taking a while to get back to this, road trips
> can be good at interrupting the flow of life.
>
> It depends on the application. I typically use a few
> instances of open() for the sake of simplicity, but I have also had
> decent luck with IPC::Open(2|3). The only problems I've had with
> either was an OS specific bug with Linux (the pipe was newline
> buffering and dropping all characters over 1023, moved to FreeBSD and
> the problem went away).
>
> Words of wisdom: start slow because debugging over a pipe can
> be a headache (understatement). Simple additions + simple debugging =
> good thing(tm). I've spent too many afternoons/nights ripping apart
> these kinds of programs only to find a small type-o and then
> reconstructing a much larger query/response set of programs. -sc
<plug>
If you're working with Open{2,3}, you might want to take a gander at
IPC::Run, it's like Open{2,3,...}+select loop+expect, but with a
shell-like API and the ability to trace events in both the parent and
child processes (via a separate debugging pipe). Just turn debugging on
and you can see what's sent and received on each pipe. It avoids the
deadlocks warned about in the perlipc manpages (as you mention), which
can be quite tricky in the face of different failure modes. IPC::Run
establishes or manages pipes, PTYs, plain ol' filehandles and timers in
such a wayavoid deadlocks (though if you try hard engugh...).
It's a bit bloated to do all that, but the debugging feature can make it
worth the bloat in many cases, and you can certainly rename it and carve
off great hunks of unneeded features (like param parsing or say ptys)
pretty easily if you want to whittle it down after debugging.
That being said, open2 and open3 are for forking around, not for
communicating over named pipes to server processes.
</plug>
> PS You also want to attach the program listening to the named
> pipe to something like DJB's daemon tools
> (http://cr.yp.to/daemontools.html) to prevent new requests from
> blocking if the listener dies: bad thing(tm).
Neat, thanks for that link.
- Barrie
Re: mod_perl shared memory with MM
Posted by Sean Chittenden <se...@chittenden.org>.
Sorry for taking a while to get back to this, road trips
can be good at interrupting the flow of life.
It depends on the application. I typically use a few
instances of open() for the sake of simplicity, but I have also had
decent luck with IPC::Open(2|3). The only problems I've had with
either was an OS specific bug with Linux (the pipe was newline
buffering and dropping all characters over 1023, moved to FreeBSD and
the problem went away).
Words of wisdom: start slow because debugging over a pipe can
be a headache (understatement). Simple additions + simple debugging =
good thing(tm). I've spent too many afternoons/nights ripping apart
these kinds of programs only to find a small type-o and then
reconstructing a much larger query/response set of programs. -sc
PS You also want to attach the program listening to the named
pipe to something like DJB's daemon tools
(http://cr.yp.to/daemontools.html) to prevent new requests from
blocking if the listener dies: bad thing(tm).
On Wed, Feb 28, 2001 at 10:23:06PM -0500, Adi Fairbank wrote:
> Delivered-To: sean@chittenden.org
> Date: Wed, 28 Feb 2001 22:23:06 -0500
> From: Adi Fairbank <ad...@certsite.com>
> X-Mailer: Mozilla 4.75 [en] (X11; U; Linux 2.2.14-5.0 i586)
> X-Accept-Language: en
> To: Sean Chittenden <se...@chittenden.org>
> CC: modperl@apache.org
> Subject: Re: mod_perl shared memory with MM
>
> Sean,
>
> Yeah, I was thinking about something like that at first, but I've never played
> with named pipes, and it didn't sound too safe after reading the perlipc man
> page. What do you use, Perl open() calls, IPC::Open2/3, IPC::ChildSafe, or
> something else? How stable has it been for you? I just didn't like all those
> warnings in the IPC::Open2 and perlipc man pages.
>
> -Adi
>
> Sean Chittenden wrote:
> >
> > The night of Fat Tuesday no less... that didn't help any
> > either. ::sigh::
> >
> > Here's one possibility that I've done in the past becuase I
> > needed mod_perl sessions to be able to talk with non-mod_perl
> > programs. I setup a named bi-directional pipe that let you write a
> > query to it for session information, and it wrote back with whatever
> > you were looking for. Given that this needed to support perl, java,
> > and c, it worked _very_ well and was extremely fast. Something you
> > may also want to consider because it keeps your session information
> > outside of apache (incase of restart of apache, or desire to
> > synchronize session information across multiple hosts).
> >
> > -sc
> >
>
--
Sean Chittenden sean@chittenden.org
Re: mod_perl shared memory with MM
Posted by Adi Fairbank <ad...@certsite.com>.
Sean,
Yeah, I was thinking about something like that at first, but I've never played
with named pipes, and it didn't sound too safe after reading the perlipc man
page. What do you use, Perl open() calls, IPC::Open2/3, IPC::ChildSafe, or
something else? How stable has it been for you? I just didn't like all those
warnings in the IPC::Open2 and perlipc man pages.
-Adi
Sean Chittenden wrote:
>
> The night of Fat Tuesday no less... that didn't help any
> either. ::sigh::
>
> Here's one possibility that I've done in the past becuase I
> needed mod_perl sessions to be able to talk with non-mod_perl
> programs. I setup a named bi-directional pipe that let you write a
> query to it for session information, and it wrote back with whatever
> you were looking for. Given that this needed to support perl, java,
> and c, it worked _very_ well and was extremely fast. Something you
> may also want to consider because it keeps your session information
> outside of apache (incase of restart of apache, or desire to
> synchronize session information across multiple hosts).
>
> -sc
>
Re: mod_perl shared memory with MM
Posted by Sean Chittenden <se...@chittenden.org>.
The night of Fat Tuesday no less... that didn't help any
either. ::sigh::
Here's one possibility that I've done in the past becuase I
needed mod_perl sessions to be able to talk with non-mod_perl
programs. I setup a named bi-directional pipe that let you write a
query to it for session information, and it wrote back with whatever
you were looking for. Given that this needed to support perl, java,
and c, it worked _very_ well and was extremely fast. Something you
may also want to consider because it keeps your session information
outside of apache (incase of restart of apache, or desire to
synchronize session information across multiple hosts).
-sc
On Wed, Feb 28, 2001 at 09:25:45PM -0500, Adi Fairbank wrote:
> Delivered-To: sean@chittenden.org
> Date: Wed, 28 Feb 2001 21:25:45 -0500
> From: Adi Fairbank <ad...@certsite.com>
> X-Mailer: Mozilla 4.75 [en] (X11; U; Linux 2.2.14-5.0 i586)
> X-Accept-Language: en
> To: Sean Chittenden <se...@chittenden.org>
> Subject: Re: mod_perl shared memory with MM
>
> It's ok, I do that a lot, too. Usually right after I click "Send" is when I
> realize I forgot something or didn't think it through all the way. :)
>
> Sean Chittenden wrote:
> >
> > Hmm... yeah, whoops. I suppose that's what I get for sending
> > email that late. <:~) -sc
> >
>
--
Sean Chittenden sean@chittenden.org
C665 A17F 9A56 286C 5CFB 1DEA 9F4F 5CEF 1EDD FAAD
Re: mod_perl shared memory with MM
Posted by Adi Fairbank <ad...@certsite.com>.
Sean Chittenden wrote:
>
> > > Is there a way you can do that without using Storable?
> >
> > Right after I sent the message, I was thinking to myself that same
> > question... If I extended IPC::MM, how could I get it to be any
> > faster than Storable already is?
>
> You can also read in the data you want in a startup.pl file
> and put the info in a hash in a global memory space
> (MyApp::datastruct{}) that gets shared through forking (copy on write,
> not read, right?). If the data is read only, and only a certain size,
> this option has worked _very_ well for me in the past. -sc
>
Yeah, I do use that method for all my read-only data, but by definition the
persistent session cache is *not* read-only... it gets changed on pretty much
every request.
-Adi
Re: mod_perl shared memory with MM
Posted by Sean Chittenden <se...@chittenden.org>.
> > Is there a way you can do that without using Storable?
>
> Right after I sent the message, I was thinking to myself that same
> question... If I extended IPC::MM, how could I get it to be any
> faster than Storable already is?
You can also read in the data you want in a startup.pl file
and put the info in a hash in a global memory space
(MyApp::datastruct{}) that gets shared through forking (copy on write,
not read, right?). If the data is read only, and only a certain size,
this option has worked _very_ well for me in the past. -sc
--
Sean Chittenden sean@chittenden.org
C665 A17F 9A56 286C 5CFB 1DEA 9F4F 5CEF 1EDD FAAD
Re: mod_perl shared memory with MM
Posted by Adi Fairbank <ad...@certsite.com>.
Perrin Harkins wrote:
>
> Adi Fairbank wrote:
> >
> > I am trying to squeeze more performance out of my persistent session cache. In
> > my application, the Storable image size of my sessions can grow upwards of
> > 100-200K. It can take on the order of 200ms for Storable to deserialize and
> > serialize this on my (lousy) hardware.
> >
> > I'm looking at RSE's MM and the Perl module IPC::MM as a persistent session
> > cache. Right now IPC::MM doesn't support multi-dimensional Perl data
> > structures, nor blessed references, so I will have to extend it to support
> > these.
>
> Is there a way you can do that without using Storable?
Right after I sent the message, I was thinking to myself that same question...
If I extended IPC::MM, how could I get it to be any faster than Storable already
is?
Basically what I came up with off the top of my head was to try to map each Perl
hash to a mm_hash and each Perl array to a mm_btree_table, all the way down
through the multi-level data structure. Every time you add a hashref to your
tied IPC::MM hash, it would create a new mm_hash and store the reference to that
child in the parent. Ditto for arrayrefs, but use mm_btree_table.
If this is possible, then you could operate on the guts of a deep data structure
without completely serializing and deserializing it every time.
> If not, maybe
> you should look at partitioning your data more, so that only the parts
> you really need for a given request are loaded and saved.
Good idea! That would save a lot of speed, and would be easy to do with my
design. Silly I didn't think of that.
>
> I'm pleased to see people using IPC::MM, since I bugged Arthur to put it
> on CPAN. However, if it doesn't work for you there are other options
> such as BerkeleyDB (not DB_File) which should provide a similar level of
> performance.
Thanks.. I'll look at BerkeleyDB.
-Adi
Re: mod_perl shared memory with MM
Posted by Perrin Harkins <pe...@primenet.com>.
Adi Fairbank wrote:
>
> I am trying to squeeze more performance out of my persistent session cache. In
> my application, the Storable image size of my sessions can grow upwards of
> 100-200K. It can take on the order of 200ms for Storable to deserialize and
> serialize this on my (lousy) hardware.
>
> I'm looking at RSE's MM and the Perl module IPC::MM as a persistent session
> cache. Right now IPC::MM doesn't support multi-dimensional Perl data
> structures, nor blessed references, so I will have to extend it to support
> these.
Is there a way you can do that without using Storable? If not, maybe
you should look at partitioning your data more, so that only the parts
you really need for a given request are loaded and saved.
I'm pleased to see people using IPC::MM, since I bugged Arthur to put it
on CPAN. However, if it doesn't work for you there are other options
such as BerkeleyDB (not DB_File) which should provide a similar level of
performance.
- Perrin