cdb/Storable (Was: Storable Freeze/thaw ....)

[Bill Whillers <mo...@whohasit.com>]

> Why wrap the stored object in a database?

Thanks Matthew -- Our data is somewhat volatile but since our frozen objects 
can get pretty big, like others we're always looking for an all-around better 
"shared" solution for volatile data which is required during almost every 
session.

From what I've learned (mostly from the generous people on this list), our 
local mysql, Storable and other usages does a great job at meeting those 
needs.  In reality, the ideal case would be if our data were non-changing and 
could simply be loaded at server start up.  So, anything that emulates that 
non-lossy effect across multiple machines and changing data, comparable 
efficiency, and perhaps elimination of the freeze/thaw step altogether would 
be great. 

Bill



On Monday 19 September 2005 06:10, Matthew Berk wrote:
> Bill,
>
> I see that Peter's helped out here. Just a comment, though. Why wrap the
> stored object in a database? I used to do this, but have elminated the
> database completely. Instead, I used storable in combination with cdb for
> persistence, and then build custom indexes on top of the data. Our data is
> not volatile, so we optimixe for quick access.
>
> Matthew
>
> -----Original Message-----
> From: Bill Whillers <mo...@whohasit.com>
> Date: Sun, 18 Sep 2005 22:43:09
> To:"mod_perl list" <mo...@perl.apache.org>
> Subject: Storable Freeze/thaw - Size Limits - 65535 (?)
>
> When using Storable freeze/thaw to work with basic object (hash) storage
> within a mysql db (blob column), I can't seem to get any more than 65535
> bytes stored.
>
> Since 65535 appears to be a "magic" number, I've been digging for where my
> problem might reside (Storable, Perl, etc).  but knowing many on this list
> use Storable to do lots of cool things,  can someone provide a clue?
>
> Any other comments are much appreciated.
>
> Thanks in advance,
> Bill

Re: cdb/Storable (Was: Storable Freeze/thaw ....)

[Perrin Harkins <pe...@elem.com>]

On Mon, 2005-09-19 at 13:09 -0700, Bill Whillers wrote:
> So at what point (1k, 10k, 100k) might the overhead of a decompress on a 
> frozen chunk make real sense?

You have to benchmark it yourself with your own data and network to see.

- Perrin

Re: cdb/Storable (Was: Storable Freeze/thaw ....)

[Bill Whillers <mo...@whohasit.com>]

Thanks Perrin,

> Compression (using zlib) tends to speed things up a bit.

So at what point (1k, 10k, 100k) might the overhead of a decompress on a 
frozen chunk make real sense?  -- If we compressed every frozen 1k item 
(requiring decompress everytime), might this only add unnecessary overhead?



On Monday 19 September 2005 12:19, Perrin Harkins wrote:
> If it has to work across multiple machines, you will need to use a
> daemon like MySQL.  If it's local to one machine, BerkeleyDB or
> Cache::FastMmap can beat it.  Compression (using zlib) tends to speed
> things up a bit when pushing huge amounts of data into MySQL across a
> socket connection, so you might want to add that in as well.
>
> - Perrin

Re: cdb/Storable (Was: Storable Freeze/thaw ....)

[Perrin Harkins <pe...@elem.com>]

On Mon, 2005-09-19 at 12:02 -0700, Bill Whillers wrote:
> From what I've learned (mostly from the generous people on this list), our 
> local mysql, Storable and other usages does a great job at meeting those 
> needs.  In reality, the ideal case would be if our data were non-changing and 
> could simply be loaded at server start up.  So, anything that emulates that 
> non-lossy effect across multiple machines and changing data, comparable 
> efficiency, and perhaps elimination of the freeze/thaw step altogether would 
> be great. 

If it has to work across multiple machines, you will need to use a
daemon like MySQL.  If it's local to one machine, BerkeleyDB or
Cache::FastMmap can beat it.  Compression (using zlib) tends to speed
things up a bit when pushing huge amounts of data into MySQL across a
socket connection, so you might want to add that in as well.

- Perrin