Hash Cache using Require

[Bill Whillers <mo...@whohasit.com>]

I recently ported a set of old CGI applications for mod_perl. Each script 
loads a few hashes from flat files (using "open") filled with plain rows of 
delimited data that rarely changes (~50k each). The hashes are built by 
iterating thru each file, row by row, assigning values to keys.

All the file data is permanently stored in a non-local sql database (mysql) 
and anytime the hash data is updated in the database, the flat file content 
is simply refreshed using results from a simple query that joins a couple 
tables.

QUESTION:

Am I more or less efficient than just creating the hashes directly from 
hitting the database everytime the application needs it? The database is 
*not* under heavy load, at all.

I'd like to efficiently cache this same data (somehow) now that I've ported 
the applications for a mod_perl environment for future scaling; Is this 
pointless (or more expensive doing file access). What if the data in each 
file increases from 50k to 500k or even 1-2 MB giving larger hashes?

I've considered writing the cache file as a require-able file knowing that it 
won't be reloaded unless updated; would this be more reasonable or more 
pointless? Can anyone suggest a better solution?


Thanks for your suggestions!


-- 
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html

Re: Hash Cache using Require

[Tom Schindl <to...@gmx.at>]

Perrin Harkins wrote:
> On Fri, 2004-12-03 at 11:17 -0800, Bill Whillers wrote:
> 
>>Am I more or less efficient than just creating the hashes directly from 
>>hitting the database everytime the application needs it? The database is 
>>*not* under heavy load, at all.
> 
> 
> If the data is small enough, loading it into memory should work fine,
> and will be faster than loading from the database.  If you only need
> access to specific items at any given time, I'd read those individually
> from the database or use a local dbm file (try MLDBM::Sync or
> BerkeleyDB) because it will save you memory over loading the full hash
> in each process.

I'd suggest reading the data from the mysql database directly and 
keeping it in memory. You've many possibilities to check whether the 
data in the database has been modified. e.g. by setting a 
update-timestamp which you request.

Tom


-- 
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html

Re: Hash Cache using Require

[Perrin Harkins <pe...@elem.com>]

On Fri, 2004-12-03 at 11:17 -0800, Bill Whillers wrote:
> Am I more or less efficient than just creating the hashes directly from 
> hitting the database everytime the application needs it? The database is 
> *not* under heavy load, at all.

If the data is small enough, loading it into memory should work fine,
and will be faster than loading from the database.  If you only need
access to specific items at any given time, I'd read those individually
from the database or use a local dbm file (try MLDBM::Sync or
BerkeleyDB) because it will save you memory over loading the full hash
in each process.

> I've considered writing the cache file as a require-able file knowing that it 
> won't be reloaded unless updated; would this be more reasonable or more 
> pointless?

That would work, but you would have to use something like Apache::Reload
to make it refresh when the file changes.  This is not an issue with the
dbm approach.

- Perrin


-- 
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html