Memory usage

[Will Fould <wi...@gmail.com>]

Hi.  I'm trying to estimate (forecast) perl memory requirements for storing
simple lists (hashes) in memory.

If my lists use integers as keys and all values are either integers or
simple scalar strings (of maximum size each), can the total memory size be
estimated by simply factoring the total number of key=value pairs when
considering the maximum byte size of each value? What other overhead is
required for each list?

Any links or discussion would be great.

~Will

Re: Memory usage

[Raymond Wan <rw...@kuicr.kyoto-u.ac.jp>]

Michael Peters wrote:
>> Any links or discussion would be great.
>>     
>
> You really need to benchmark it for yourself using the version you are
> targetting and the OS. You can do it outside of apache and just have a simple
> Perl script that does nothing but sleep after it's created the large data
> structure you want. Then use something like top (or whatever the best tool on
> your OS is) to see how much memory it takes us.
>
>   

I'm not sure if the numbers reported are accurate, but if Unix/Linux is 
used, then you can run the program under "valgrind" or the Unix "time" 
command (with appropriate flags).

I don't know the details of perl, but I have written various data 
structures such as hashes in Perl and then C and the C version takes a 
lot less memory.  I'm sure there is overhead because the perl script is 
interpreted, but there is probably a lot more going on.  If you want to 
see how much memory the data structure (alone) is using, perhaps you 
(Will) can write two versions of the script...one with and one without 
the data structure...take the difference, as reported by valgrind, time, 
or top and you'll get the space used by the data structure.

Ray

Re: Memory usage

[Michael Peters <mp...@plusthree.com>]

Will Fould wrote:
> Hi.  I'm trying to estimate (forecast) perl memory requirements for
> storing simple lists (hashes) in memory.
> 
> If my lists use integers as keys and all values are either integers or
> simple scalar strings (of maximum size each), can the total memory size
> be estimated by simply factoring the total number of key=value pairs
> when considering the maximum byte size of each value? What other
> overhead is required for each list?

This will change depending on which version of Perl (the upcoming 5.10 is
supposed to be much nicer memory wise) and which OS you are using, so even if
you found some rule-of-thumb somewhere it wouldn't necessarily match what you'll
get.

> Any links or discussion would be great.

You really need to benchmark it for yourself using the version you are
targetting and the OS. You can do it outside of apache and just have a simple
Perl script that does nothing but sleep after it's created the large data
structure you want. Then use something like top (or whatever the best tool on
your OS is) to see how much memory it takes us.

-- 
Michael Peters
Developer
Plus Three, LP

Re: Memory usage

[Aaron Trevena <aa...@gmail.com>]

Hi Will,

How many keys will your hashes contain?

The main memory wastage with hashes isn't due to the size of what you
keep in them but the number of keys.

If you are using integers as keys, why not just use an array?

Bare in mind that the 'hashspace' is shared between your hashes, so if
you have a lot of  hashes with only a relatively small number of
entries (i.e. hundreds rather than thousands) you might want to look
at using Binary::Tree::Dictionary - something I wrote when I needed a
lot of low memory hash/dictionarys for relatively small data sets in a
perl daemon.

A.

-- 
http://www.aarontrevena.co.uk
LAMP System Integration, Development and Hosting