You are viewing a plain text version of this content. The canonical link for it is here.
Posted to modperl@perl.apache.org by Ernest Lergon <er...@virtualitas.net> on 2002/04/28 17:18:24 UTC
Memory explodes loading CSV into hash
Hi,
in a mod_perl package I load a CSV file on apache startup into a simple
hash as read-only class data to be shared by all childs.
A loading routine reads the file line by line and uses one numeric field
as hash entry (error checks etc. omitted):
package Data;
my $class_data = {};
ReadFile ( 'data.txt', $class_data, 4 );
sub ReadFile
{
my $filename = shift; # path string
my $data = shift; # ref to hash
my $ndx_field = shift; # field number
my ( @record, $num_key );
local $_;
open ( INFILE, "<$filename" );
while ( <INFILE> )
{
chomp;
@record = split "\t";
$num_key = $record[$ndx_field];
$data->{$num_key} = [ @record ];
}
close ( INFILE );
}
sub new...
creates an object for searching the data, last result, errors etc.
sub find...
method with something like:
if exists $class_data->{$key} return...
etc.
Now I'm scared about the memory consumption:
The CSV file has 14.000 records with 18 fields and a size of 2 MB
(approx. 150 Bytes per record).
Omitting the loading, top shows, that each httpd instance has 10 MB (all
shared as it should be).
But reading in the file explodes the memory to 36 MB (ok, shared as
well)!
So, how comes, that 2 MB data need 26 MB of memory, if it is stored as a
hash?
Reading perldebguts.pod I did not expect such an increasing:
Description (avg.) CSV Perl
4 string fields (4 chars) 16 bytes (32 bytes) 128 bytes
9 float fields (5 chars) 45 bytes (24 bytes) 216 bytes
5 string fields (rest) 89 bytes (32 bytes) 160 bytes
the integer key (20 bytes) 20 bytes
150 bytes 524 bytes
That will give 14.000 x 524 = approx. 7 MB, but not 26 MB !?
Lost in space...
Ernest
P.S.:
Apache/1.3.19 (Unix) (SuSE/Linux) mod_perl/1.25
This is perl, v5.6.0 built for i586-linux
--
*********************************************************************
* VIRTUALITAS Inc. * *
* * *
* European Consultant Office * http://www.virtualitas.net *
* Internationales Handelszentrum * contact:Ernest Lergon *
* Friedrichstraße 95 * mailto:Ernest@virtualitas.net *
* 10117 Berlin / Germany * ums:+49180528132130266 *
*********************************************************************
PGP-Key http://www.virtualitas.net/Ernest_Lergon.asc
Re: Memory explodes loading CSV into hash
Posted by Ernest Lergon <er...@virtualitas.net>.
Perrin Harkins wrote:
>
> [snip]
>
> Incidentally, that map statement in your script isn't doing
> anything that I can see.
>
It simulates different values for each record - e.g.:
$line = "AAAA\tBBBB\t1000\t10.99";
@record = split "\t", $line;
for ( $i = 0; $i < 14000; $i++ )
{
map { $_++ } @record;
# $i=0 @record=('AAAB','BBBC',1001,11.99);
# $i=1 @record=('AAAC','BBBD',1002,12.99);
# $i=2 @record=('AAAD','BBBE',1003,13.99);
# etc.
}
> [snip]
>
Thanks for your explanations about perl's memory usage.
> At any rate, mod_perl doesn't seem to be at fault here. It's just a
> general perl issue.
>
I think so, too.
Ernest
--
*********************************************************************
* VIRTUALITAS Inc. * *
* * *
* European Consultant Office * http://www.virtualitas.net *
* Internationales Handelszentrum * contact:Ernest Lergon *
* Friedrichstraße 95 * mailto:Ernest@virtualitas.net *
* 10117 Berlin / Germany * ums:+49180528132130266 *
*********************************************************************
PGP-Key http://www.virtualitas.net/Ernest_Lergon.asc
Re: Memory explodes loading CSV into hash
Posted by Perrin Harkins <pe...@elem.com>.
Ernest Lergon wrote:
> So I turned it around:
>
> $col holds now 18 arrays with 14000 entries each and prints the correct
> results:
...
> and gives:
>
> SIZE RSS SHARE
> 12364 12M 1044
>
> Wow, 2 MB saved ;-))
That's pretty good, but obviously not what you were after.
I tried using the pre-size array syntax ($#array = 14000), but it didn't
help any. Incidentally, that map statement in your script isn't doing
anything that I can see.
> I think, a reference is a pointer of 8 Bytes, so:
>
> 14.000 * 8 = approx. 112 KBytes - right?
Probably more. Perl data types are complex. They hold a lot of meta
data (is the ref blessed, for example).
> This doesn't explain the difference of 7 MB calculated and 14 MB
> measured.
The explanation of this is that perl uses a lot of memory. For one
thing, it allocates RAM in buckets. When you hit the limit of the
allocated memory, it grabs more, and I believe it grabs an amount in
proportion to what you've already used. That means that as your
structures get bigger, it grabs bigger chunks. The whole 12MB may not
be in use, although perl has reserved it for possible use. (Grabbing
memory byte by byte would be less wasteful, but much too slow.)
The stuff in perldebguts is the best reference on this, and you've
already looked at that. I think your original calculation failed to
account for the fact that the minimum numbers given there for scalars
are minimums (i.e. scalars with something in them won't be that small)
and that you are accessing many of these in more than one way (i.e. as
string, float, and integer), which increases their size. You can try
playing with compile options (your choice of malloc affects this a
little), but at this point it's probably not worth it. There's nothing
wrong with 12MB of shared memory, as long as it stays shared. If that
doesn't work for you, your only choice will be to trade some speed for
reduced memory useage, by using a disk-based structure.
At any rate, mod_perl doesn't seem to be at fault here. It's just a
general perl issue.
- Perrin
Re: Memory explodes loading CSV into hash
Posted by Ernest Lergon <er...@virtualitas.net>.
Perrin Harkins wrote:
>
> > $foo->{$i} = [ @record ];
>
> You're creating 14000 arrays, and references to them (refs take up space
> too!). That's where the memory is going.
>
> See if you can use a more efficient data structure. For example, it
> takes less space to make 4 arrays with 14000 entries in each than to
> make 14000 arrays with 4 entries each.
>
So I turned it around:
$col holds now 18 arrays with 14000 entries each and prints the correct
results:
#!/usr/bin/perl -w
$col = {};
$line = "AAAA\tBBBB\tCCCC\tDDDD"; # 4 string fields (4 chars)
$line .= "\t10.99"x9; # 9 float fields (5 chars)
$line .= "\t" . 'A'x17; # 5 string fields (rest)
$line .= "\t" . 'B'x17; #
$line .= "\t" . 'C'x17; #
$line .= "\t" . 'D'x17; #
$line .= "\t" . 'E'x17; #
@record = split "\t", $line;
foreach $j ( 0 .. $#record )
{
$col->{$j} = [];
}
for ( $i = 0; $i < 14000; $i++ )
{
map { $_++ } @record;
foreach $j ( 0 .. $#record )
{
push @ { $col->{$j} }, $record [$j];
}
print "$i\t$col->{0}->[$i],$col->{5}->[$i]\n" unless $i % 1000;
}
<>;
1;
and gives:
SIZE RSS SHARE
12364 12M 1044
Wow, 2 MB saved ;-))
I think, a reference is a pointer of 8 Bytes, so:
14.000 * 8 = approx. 112 KBytes - right?
This doesn't explain the difference of 7 MB calculated and 14 MB
measured.
Ernest
--
*********************************************************************
* VIRTUALITAS Inc. * *
* * *
* European Consultant Office * http://www.virtualitas.net *
* Internationales Handelszentrum * contact:Ernest Lergon *
* Friedrichstraße 95 * mailto:Ernest@virtualitas.net *
* 10117 Berlin / Germany * ums:+49180528132130266 *
*********************************************************************
PGP-Key http://www.virtualitas.net/Ernest_Lergon.asc
Re: Memory explodes loading CSV into hash
Posted by Perrin Harkins <pe...@elem.com>.
> $foo->{$i} = [ @record ];
You're creating 14000 arrays, and references to them (refs take up space
too!). That's where the memory is going.
See if you can use a more efficient data structure. For example, it
takes less space to make 4 arrays with 14000 entries in each than to
make 14000 arrays with 4 entries each. There is some discussion of this
in the Advanced Perl Programming book, and probably some CPAN modules
that can help.
- Perrin
Re: Memory explodes loading CSV into hash
Posted by Ernest Lergon <er...@virtualitas.net>.
Jeffrey Baker wrote:
>
> I tried this program in Perl (outside of modperl) and the memory
> consumption is only 4.5MB:
>
> #!/usr/bin/perl -w
>
> $foo = {};
>
> for ($i = 0; $i < 14000; $i++) {
> $foo->{sprintf('%020d', $i)} = 'A'x150;
> }
>
> <>;
>
> 1;
>
> So I suggest something else might be going on causing your memory
> problems.
>
Hi Jeffrey,
good idea to boil it down.
Yes, your prog gave me only:
SIZE RSS SHARE
4696 4696 964
Running my code snippet outside mod_perl (with real data) still gives:
SIZE RSS SHARE
14932 14M 1012
A simulation like this:
#!/usr/bin/perl -w
$foo = {};
$line = "AAAA\tBBBB\tCCCC\tDDDD"; # 4 string fields (4 chars)
$line .= "\t10.99"x9; # 9 float fields (5 chars)
$line .= "\t" . 'A'x17; # 5 string fields (rest)
$line .= "\t" . 'B'x17; #
$line .= "\t" . 'C'x17; #
$line .= "\t" . 'D'x17; #
$line .= "\t" . 'E'x17; #
@record = split "\t", $line;
for ($i = 0; $i < 14000; $i++)
{
map { $_++ } @record;
$foo->{$i} = [ @record ];
print "$i\t$foo->{$i}->[0],$foo->{$i}->[5]\n" unless $i % 1000;
}
<>;
1;
prints:
0 AAAB,11.99
1000 ABMN,1011.99
2000 ACYZ,2011.99
3000 AELL,3011.99
4000 AFXX,4011.99
5000 AHKJ,5011.99
6000 AIWV,6011.99
7000 AKJH,7011.99
8000 ALVT,8011.99
9000 ANIF,9011.99
10000 AOUR,10011.99
11000 AQHD,11011.99
12000 ARTP,12011.99
13000 ATGB,13011.99
and gives:
SIZE RSS SHARE
14060 13M 1036
There is no difference between real and random data.
But I think, there is an optimization mechanism in perl concerning
strings, so you need less memory for your code.
So what is going on? 2 MB -> 14 MB ?
Still lost in space ;-))
Ernest
--
*********************************************************************
* VIRTUALITAS Inc. * *
* * *
* European Consultant Office * http://www.virtualitas.net *
* Internationales Handelszentrum * contact:Ernest Lergon *
* Friedrichstraße 95 * mailto:Ernest@virtualitas.net *
* 10117 Berlin / Germany * ums:+49180528132130266 *
*********************************************************************
PGP-Key http://www.virtualitas.net/Ernest_Lergon.asc
Re: Memory explodes loading CSV into hash
Posted by Jeffrey Baker <jw...@acm.org>.
On Sun, Apr 28, 2002 at 05:18:24PM +0200, Ernest Lergon wrote:
> Hi,
>
> in a mod_perl package I load a CSV file on apache startup into a simple
> hash as read-only class data to be shared by all childs.
>
> A loading routine reads the file line by line and uses one numeric field
> as hash entry (error checks etc. omitted):
>
> package Data;
>
> my $class_data = {};
>
> ReadFile ( 'data.txt', $class_data, 4 );
>
> sub ReadFile
> {
> my $filename = shift; # path string
> my $data = shift; # ref to hash
> my $ndx_field = shift; # field number
>
> my ( @record, $num_key );
> local $_;
>
> open ( INFILE, "<$filename" );
> while ( <INFILE> )
> {
> chomp;
> @record = split "\t";
> $num_key = $record[$ndx_field];
> $data->{$num_key} = [ @record ];
> }
> close ( INFILE );
> }
>
> sub new...
> creates an object for searching the data, last result, errors etc.
>
> sub find...
> method with something like:
> if exists $class_data->{$key} return...
> etc.
>
> Now I'm scared about the memory consumption:
>
> The CSV file has 14.000 records with 18 fields and a size of 2 MB
> (approx. 150 Bytes per record).
>
> Omitting the loading, top shows, that each httpd instance has 10 MB (all
> shared as it should be).
>
> But reading in the file explodes the memory to 36 MB (ok, shared as
> well)!
>
> So, how comes, that 2 MB data need 26 MB of memory, if it is stored as a
> hash?
>
> Reading perldebguts.pod I did not expect such an increasing:
>
> Description (avg.) CSV Perl
> 4 string fields (4 chars) 16 bytes (32 bytes) 128 bytes
> 9 float fields (5 chars) 45 bytes (24 bytes) 216 bytes
> 5 string fields (rest) 89 bytes (32 bytes) 160 bytes
> the integer key (20 bytes) 20 bytes
> 150 bytes 524 bytes
>
> That will give 14.000 x 524 = approx. 7 MB, but not 26 MB !?
I tried this program in Perl (outside of modperl) and the memory
consumption is only 4.5MB:
#!/usr/bin/perl -w
$foo = {};
for ($i = 0; $i < 14000; $i++) {
$foo->{sprintf('%020d', $i)} = 'A'x150;
}
<>;
1;
So I suggest something else might be going on causing your memory
problems.
-jwb
Re: Memory explodes loading CSV into hash
Posted by Ernest Lergon <er...@virtualitas.net>.
Stas Bekman wrote:
>
> Ideally when such a
> situation happens, and you must load all the data into the memory, which
> is at short, your best bet is to rewrite the datastorage layer in XS/C,
> and use a tie interface to make it transparent to your perl code. So you
> will still use the hash but the refs to arrays will be actually C arrays.
>
Sorry, I'm not familiar with C(hinese) - but if someone could develop a
XS/Pascal interface ;-))
> Ernest Lergon wrote:
>
> > Another thing I found is, that Apache::Status seems not always report
> > complete values. Therefore I recorded the sizes from top, too.
>
> Were you running a single process? If you aren't Apache::Status could
> have shown you a different process.
>
Running httpd -X shows the same results.
I will use the named %index structure for now. Thanks to the modular OO
perl I can re-code my data package later, if the "memory explosion" hits
me again ;-))
Ernest
--
*********************************************************************
* VIRTUALITAS Inc. * *
* * *
* European Consultant Office * http://www.virtualitas.net *
* Internationales Handelszentrum * contact:Ernest Lergon *
* Friedrichstraße 95 * mailto:Ernest@virtualitas.net *
* 10117 Berlin / Germany * ums:+49180528132130266 *
*********************************************************************
PGP-Key http://www.virtualitas.net/Ernest_Lergon.asc
Re: Memory explodes loading CSV into hash
Posted by Stas Bekman <st...@stason.org>.
Ernest Lergon wrote:
> having a look at Apache::Status and playing around with your tips on
>
> http://www.apacheweek.com/features/mod_perl11
>
> I found some interesting results and a compromising solution:
Glad to hear that Apache::Status was of help to you. Ideally when such a
situation happens, and you must load all the data into the memory, which
is at short, your best bet is to rewrite the datastorage layer in XS/C,
and use a tie interface to make it transparent to your perl code. So you
will still use the hash but the refs to arrays will be actually C arrays.
> Another thing I found is, that Apache::Status seems not always report
> complete values. Therefore I recorded the sizes from top, too.
Were you running a single process? If you aren't Apache::Status could
have shown you a different process. Also you can use GTop, if you have
libgtop on your system, which gives you a perl interface to the proc's
memory usage. See the guide for many examples.
> Success: A reduction from 26 MB to 7 MB - what I estimated in my first
> mail.
:)
__________________________________________________________________
Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide ---> http://perl.apache.org
mailto:stas@stason.org http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org http://ticketmaster.com
Re: Memory explodes loading CSV into hash
Posted by Ernest Lergon <er...@virtualitas.net>.
Hi Stas,
having a look at Apache::Status and playing around with your tips on
http://www.apacheweek.com/features/mod_perl11
I found some interesting results and a compromising solution:
In a module I load a CSV file as class data into different structures
and compared the output of Apache::Status with top.
Enclosed you'll find a test report.
The code below 'building' shows, how the lines are put into the
structures.
The lines below 'perl-status' show the output of Apache::Status.
The line below 'top' shows the output of top.
Examples for the tested structures are:
$buffer = '1\tr1v1\tr1v2\tr1v3\n2\tr2v1\tr2v2\tr2v3\n' ...
@lines = (
'1\tr1v1\tr1v2\tr1v3',
'2\tr2v1\tr2v2\tr2v3',
... )
%data = (
1 => [ 1, 'r1v1' , 'r1v2' , 'r1v3' ],
2 => [ 2, 'r2v1' , 'r2v2' , 'r2v3' ],
... )
$pack = {
1 => [ 1, 'r1v1' , 'r1v2' , 'r1v3' ],
2 => [ 2, 'r2v1' , 'r2v2' , 'r2v3' ],
... }
%index = (
1 => '1\tr1v1\tr1v2\tr1v3',
2 => '2\tr2v1\tr2v2\tr2v3',
... )
One thing I realized using Devel::Peek is, that using a hash of
array-ref, each item in the array has the full blown perl flags etc.
That seems to be the reason for the 'memory explosion'.
Another thing I found is, that Apache::Status seems not always report
complete values. Therefore I recorded the sizes from top, too.
Especially for the the hash of array-refs (%data) and the hash-ref of
array-refs ($pack) perl-status reports only a part of the used memory
size: for $pack only the pointer (16 bytes), for %data only the keys
(?).
As compromise I'll use the %index structure. It is small enough while
providing a fast access. A further optimization will be to remove the
redundant key-field from each line.
Success: A reduction from 26 MB to 7 MB - what I estimated in my first
mail.
A last word from perldebguts.pod:
|| Perl is a profligate wastrel when it comes to memory use. There is a
|| saying that to estimate memory usage of Perl, assume a reasonable
|| algorithm for memory allocation, multiply that estimate by 10, and
|| while you still may miss the mark, at least you won't be quite so
|| astonished. This is not absolutely true, but may prvide a good grasp
|| of what happens.
||
|| [...]
||
|| Anecdotal estimates of source-to-compiled code bloat suggest an
|| eightfold increase.
Perhaps my experiences could be added to the long row of anecdotes ;-))
Thank you all again for escorting me on this deep dive.
Ernest
--
*********************************************************************
* VIRTUALITAS Inc. * *
* * *
* European Consultant Office * http://www.virtualitas.net *
* Internationales Handelszentrum * contact:Ernest Lergon *
* Friedrichstraße 95 * mailto:Ernest@virtualitas.net *
* 10117 Berlin / Germany * ums:+49180528132130266 *
*********************************************************************
PGP-Key http://www.virtualitas.net/Ernest_Lergon.asc
TEST REPORT
===========
CSV file:
14350 records
CSV 2151045 bytes = 2101 Kbytes
CSV_2 2136695 bytes = 2086 Kbytes (w/o CR)
1 all empty
=================
building:
none
perl-status:
*buffer{SCALAR} 25 bytes
*lines{ARRAY} 56 bytes
*data{HASH} 228 bytes
*pack{SCALAR} 16 bytes
*index{HASH} 228 bytes
top:
12992 12M 12844 base
2 buffer
==============
building:
$buffer .= $_ . "\n";
perl-status:
*buffer{SCALAR} 2151069 bytes = CSV + 24 bytes
*lines{ARRAY} 56 bytes
*data{HASH} 228 bytes
*pack{SCALAR} 16 bytes
*index{HASH} 228 bytes
top:
17200 16M 17040 base + 4208 Kbytes = CSV + 2107 KBytes
3 lines
=============
building:
push @lines, $_;
perl-status:
*buffer{SCALAR} 25 bytes
*lines{ARRAY} 2519860 bytes = CSV_2 + 383165 bytes
(approx. 27 * 14350 )
*data{HASH} 228 bytes
*pack{SCALAR} 16 bytes
*index{HASH} 228 bytes
top:
18220 17M 18076 base + 5228 Kbytes = CSV_2 + 3142 Kbytes
4 data
============
building:
@record = split ( "\t", $_ );
$key = 0 + $record[0];
$data{$key} = [ @record ];
perl-status:
*buffer{SCALAR} 25 bytes
*lines{ARRAY} 56 bytes
*data{HASH} 723302 bytes = approx. 50 * 14350 ( key +
ref )
(where is the data?)
*pack{SCALAR} 16 bytes
*index{HASH} 228 bytes
top:
40488 38M 39208 base + 27566 Kbytes = CSV_2 + 25480 Kbytes
(!)
5 pack
============
building:
@record = split ( "\t", $_ );
$key = 0 + $record[0];
$pack->{$key} = [ @record ];
perl-status:
*buffer{SCALAR} 25 bytes
*lines{ARRAY} 56 bytes
*data{HASH} 228 bytes
*pack{SCALAR} 16 bytes (where is the data?)
*index{HASH} 228 bytes
top:
40492 39M 40340 base + 27570 Kbytes = CSV_2 + 25484 Kbytes
(!)
6 index
=============
building:
@record = split ( "\t", $_ );
$key = 0 + $record[0];
$index->{$key} = $_; # !!!
perl-status:
*buffer{SCALAR} 25 bytes
*lines{ARRAY} 56 bytes
*data{HASH} 228 bytes
*pack{SCALAR} 16 bytes
*index{HASH} 2989146 bytes = CSV_2 + 852448 bytes
( approx. 59 * 14350 )
top:
19988 19M 19824 base + 6996 Kbytes = CSV_2 + 4910 Kbytes
EOF
Re: Memory explodes loading CSV into hash
Posted by Stas Bekman <st...@stason.org>.
Ernest Lergon wrote:
> Hi,
>
> thank you all for your hints, BUT (with capital letters ;-)
>
> I think, it's a question of speed: If I hold my data in a hash in
> memory, access should be faster than using any kind of external
> database.
>
> What makes me wonder is the extremely blown up size (mod)perl uses for
> datastructures.
Looks like you've skipped over my suggestion to use Apache::Status. It
uses B::Size and B::TerseSize to show you *exactly* how much memory each
variable, opcode and what not uses. No need to guess.
You can use B:: modules directly, but since you say that outside of
mod_perl the memory usage pattern is different I'd suggest using
Apache::Status.
__________________________________________________________________
Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide ---> http://perl.apache.org
mailto:stas@stason.org http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org http://ticketmaster.com
Re: Memory explodes loading CSV into hash
Posted by Ged Haywood <ge...@www2.jubileegroup.co.uk>.
Hi there,
On Mon, 29 Apr 2002, Ernest Lergon wrote:
> I think, it's a question of speed: If I hold my data in a hash in
> memory, access should be faster than using any kind of external
> database.
Test it. You may be surprised what the OS will do for you by caching.
> What makes me wonder is the extremely blown up size (mod)perl uses for
> datastructures.
I think it just grew, like Topsy. :)
73,
Ged.
Re: Memory explodes loading CSV into hash
Posted by Ernest Lergon <er...@virtualitas.net>.
Hi,
thank you all for your hints, BUT (with capital letters ;-)
I think, it's a question of speed: If I hold my data in a hash in
memory, access should be faster than using any kind of external
database.
What makes me wonder is the extremely blown up size (mod)perl uses for
datastructures.
Ernest
--
*********************************************************************
* VIRTUALITAS Inc. * *
* * *
* European Consultant Office * http://www.virtualitas.net *
* Internationales Handelszentrum * contact:Ernest Lergon *
* Friedrichstraße 95 * mailto:Ernest@virtualitas.net *
* 10117 Berlin / Germany * ums:+49180528132130266 *
*********************************************************************
PGP-Key http://www.virtualitas.net/Ernest_Lergon.asc
Re: Memory explodes loading CSV into hash
Posted by Stas Bekman <st...@stason.org>.
Ernest Lergon wrote:
> Hi,
>
> in a mod_perl package I load a CSV file on apache startup into a simple
> hash as read-only class data to be shared by all childs.
>
> A loading routine reads the file line by line and uses one numeric field
> as hash entry (error checks etc. omitted):
>
> package Data;
>
> my $class_data = {};
>
> ReadFile ( 'data.txt', $class_data, 4 );
>
> sub ReadFile
> {
> my $filename = shift; # path string
> my $data = shift; # ref to hash
> my $ndx_field = shift; # field number
>
> my ( @record, $num_key );
> local $_;
>
> open ( INFILE, "<$filename" );
> while ( <INFILE> )
> {
> chomp;
> @record = split "\t";
> $num_key = $record[$ndx_field];
> $data->{$num_key} = [ @record ];
> }
> close ( INFILE );
> }
>
> sub new...
> creates an object for searching the data, last result, errors etc.
>
> sub find...
> method with something like:
> if exists $class_data->{$key} return...
> etc.
>
> Now I'm scared about the memory consumption:
>
> The CSV file has 14.000 records with 18 fields and a size of 2 MB
> (approx. 150 Bytes per record).
>
> Omitting the loading, top shows, that each httpd instance has 10 MB (all
> shared as it should be).
>
> But reading in the file explodes the memory to 36 MB (ok, shared as
> well)!
>
> So, how comes, that 2 MB data need 26 MB of memory, if it is stored as a
> hash?
>
> Reading perldebguts.pod I did not expect such an increasing:
>
> Description (avg.) CSV Perl
> 4 string fields (4 chars) 16 bytes (32 bytes) 128 bytes
> 9 float fields (5 chars) 45 bytes (24 bytes) 216 bytes
> 5 string fields (rest) 89 bytes (32 bytes) 160 bytes
> the integer key (20 bytes) 20 bytes
> 150 bytes 524 bytes
> That will give 14.000 x 524 = approx. 7 MB, but not 26 MB !?
>
> Lost in space...
Use Apache::Status, which can show you exactly where all the bytes go.
See the guide or its manpage for more info. I suggest that you
experiment with a very small data set and look at how much memory each
record takes.
__________________________________________________________________
Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide ---> http://perl.apache.org
mailto:stas@stason.org http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org http://ticketmaster.com
Re: Memory explodes loading CSV into hash
Posted by Andrew McNaughton <an...@scoop.co.nz>.
On Sun, 28 Apr 2002, Per Einar Ellefsen wrote:
> At 17:18 28.04.2002, Ernest Lergon wrote:
> >Now I'm scared about the memory consumption:
> >
> >The CSV file has 14.000 records with 18 fields and a size of 2 MB
> >(approx. 150 Bytes per record).
>
> Now a question I would like to ask: do you *need* to read the whole CSV
> info into memory? There are ways to overcome this. For example, looking at
> your data I suppose you might want to look for specific IDs, in that case
> it would be much more efficient to read one line at a time and check if
> it's the correct one. Otherwise you might want to make the move th a
> relational database, this is the kind of thing RDMSes excel at.
You might also want to look at loading your CSV data into a MLDBM file,
and then having your apache processes access it from there. That way
most of your data stays on disk, and you access it in much the same way as
before through a hash of arrays.
Andrew McNaughton
Re: Memory explodes loading CSV into hash
Posted by Per Einar Ellefsen <pe...@skynet.be>.
At 17:18 28.04.2002, Ernest Lergon wrote:
>Now I'm scared about the memory consumption:
>
>The CSV file has 14.000 records with 18 fields and a size of 2 MB
>(approx. 150 Bytes per record).
Now a question I would like to ask: do you *need* to read the whole CSV
info into memory? There are ways to overcome this. For example, looking at
your data I suppose you might want to look for specific IDs, in that case
it would be much more efficient to read one line at a time and check if
it's the correct one. Otherwise you might want to make the move th a
relational database, this is the kind of thing RDMSes excel at.
Just some tips.
--
Per Einar Ellefsen
per.einar@skynet.be