You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spamassassin.apache.org by bu...@bugzilla.spamassassin.org on 2006/04/07 20:32:35 UTC
[Bug 4859] New: memory bloat in resolver
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4859
Summary: memory bloat in resolver
Product: Spamassassin
Version: SVN Trunk (Latest Devel Version)
Platform: Other
OS/Version: other
Status: NEW
Severity: critical
Priority: P1
Component: spamassassin
AssignedTo: dev@spamassassin.apache.org
ReportedBy: dallase@nmgi.com
i had slow memory leak that i've been fighting for a while now... so today i
tackled it.
this effects all versions of SA that use URIBLDNS via plugin...
initial testing of looping through a corpus and running each file through M::SA
see memory creep from 23MB starting out to over 50MB after 6000 messages unless
i create a new SA instance after so many messages processed. spamd's
max-conn-per-child is why many of you dont notice this problem.
So I ran some testing, and did some DataDump's and here are the results. As you
can see, the number of items returned from a Dumper \$sa increases with each scan.
# perl samemtest.pl
Run #1, $sa length = 84881
Run #2, $sa length = 1383448
Run #3, $sa length = 1386779
Run #4, $sa length = 1390087
Run #5, $sa length = 1393439
Run #6, $sa length = 1396768
Run #7, $sa length = 1400074
Run #8, $sa length = 1403419
Run #9, $sa length = 1406740
Run #10, $sa length = 1410038
Run #11, $sa length = 1413383
Run #12, $sa length = 1416706
Run #13, $sa length = 1420009
Run #14, $sa length = 1423363
Run #15, $sa length = 1426694
Run #16, $sa length = 1430002
Run #17, $sa length = 1433351
Run #18, $sa length = 1436677
Run #19, $sa length = 1439979
Run #20, $sa length = 1443323
Run #21, $sa length = 1446644
Run #22, $sa length = 1449965
Run #23, $sa length = 1453312
Run #24, $sa length = 1456639
create new instance...
Run #25, $sa length = 84881
Run #26, $sa length = 1382241
Run #27, $sa length = 1385571
Run #28, $sa length = 1388878
Run #29, $sa length = 1392224
Run #30, $sa length = 1395546
Run #31, $sa length = 1398844
Run #32, $sa length = 1402188
Run #33, $sa length = 1405510
Run #34, $sa length = 1408810
Run #35, $sa length = 1412161
Run #36, $sa length = 1415492
So, i decided to write Dumper \$sa to dump$count.txt for each iteration and
compare the changes. What I have found is that the bloat comes from the
resolver. seeing that the $self->{resolver} is created M::SA->new(), the
memory bloat (not a leak) comes from there.
the problem exists in the storage of $self->{resolver}->{id_to_callback} area.
in dump2 we have 27 instances in id_to_callback. in dump19, we have 486
instances. if i let it loop through a few thousand messages, it just keeps on
growing. poll_responses() is supposed to delete the instances with
my $cb = delete $self->{id_to_callback}->{$id};
but, its failing to get to that point on messages that abort the dns queries due
to deadline being reached (aka uridnsbl_timeout).
i added the following 3 lines to the bottom of
M::SA::Plugin::URIDNSBL::abort_remaining_lookups(); and now i have no more
memory bloat (thank god, those process restarts every 2 days sucked), datadump
per run as shown....
-rw-r--r-- 1 root root 84995 Apr 7 19:16 dump1.txt
-rw-r--r-- 1 root root 1324687 Apr 7 19:16 dump2.txt
-rw-r--r-- 1 root root 1324687 Apr 7 19:16 dump3.txt
-rw-r--r-- 1 root root 1324687 Apr 7 19:16 dump4.txt
-rw-r--r-- 1 root root 1324687 Apr 7 19:16 dump5.txt
-rw-r--r-- 1 root root 1324687 Apr 7 19:16 dump6.txt
-rw-r--r-- 1 root root 1324687 Apr 7 19:16 dump7.txt
attaching patch shortly...
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 4859] [Review] memory bloat in resolver
Posted by bu...@bugzilla.spamassassin.org.
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4859
felicity@apache.org changed:
What |Removed |Added
----------------------------------------------------------------------------
Status Whiteboard|Needs 1 votes for 3.1 branch|ready to commit
------- Additional Comments From felicity@apache.org 2006-04-08 23:34 -------
+1
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 4859] memory bloat in resolver
Posted by bu...@bugzilla.spamassassin.org.
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4859
------- Additional Comments From jm@jmason.org 2006-04-08 15:08 -------
+1. thanks for tracking this down Dallas!
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 4859] memory bloat in resolver
Posted by bu...@bugzilla.spamassassin.org.
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4859
felicity@apache.org changed:
What |Removed |Added
----------------------------------------------------------------------------
Target Milestone|Undefined |3.1.2
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 4859] [Review] memory bloat in resolver
Posted by bu...@bugzilla.spamassassin.org.
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4859
sidney@sidney.com changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |RESOLVED
Resolution| |FIXED
Status Whiteboard|ready to commit |
------- Additional Comments From sidney@sidney.com 2006-04-09 01:26 -------
Committed revision 392640.
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 4859] [Review] memory bloat in resolver
Posted by bu...@bugzilla.spamassassin.org.
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4859
------- Additional Comments From dallase@nmgi.com 2006-06-05 13:55 -------
did this not make 3.1.2?
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 4859] memory bloat in resolver
Posted by bu...@bugzilla.spamassassin.org.
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4859
------- Additional Comments From dallase@nmgi.com 2006-04-07 19:34 -------
Created an attachment (id=3461)
--> (http://issues.apache.org/SpamAssassin/attachment.cgi?id=3461&action=view)
patch to URIDNSBL.pm
enjoy! ;)
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 4859] [Review] memory bloat in resolver
Posted by bu...@bugzilla.spamassassin.org.
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4859
sidney@sidney.com changed:
What |Removed |Added
----------------------------------------------------------------------------
Summary|memory bloat in resolver |[Review] memory bloat in
| |resolver
Status Whiteboard|Needs 2 votes for 3.1 branch|Needs 1 votes for 3.1 branch
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 4859] memory bloat in resolver
Posted by bu...@bugzilla.spamassassin.org.
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4859
------- Additional Comments From felicity@apache.org 2006-04-08 03:09 -------
(In reply to comment #2)
> Perl experts: Is this equivalent to but smaller and faster than Dallas' patch?
> $self->{main}->{resolver}->{id_to_callback} = ();
It should be:
$self->{main}->{resolver}->{id_to_callback} = {};
Both work, but the first sets the scalar to a blank array reference, and the second sets the scalar to a blank
hash reference. :)
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 4859] memory bloat in resolver
Posted by bu...@bugzilla.spamassassin.org.
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4859
------- Additional Comments From sidney@sidney.com 2006-04-08 01:35 -------
Perl experts: Is this equivalent to but smaller and faster than Dallas' patch?
$self->{main}->{resolver}->{id_to_callback} = ();
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 4859] memory bloat in resolver
Posted by bu...@bugzilla.spamassassin.org.
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4859
------- Additional Comments From felicity@apache.org 2006-04-08 16:59 -------
(In reply to comment #3)
> Both work, but the first sets the scalar to a blank array reference, and the
second sets the scalar to a blank
> hash reference. :)
It occurred to me that I was wrong about this. "$var = ()" seems to set $var to
undef (I would have thought it would set it to "0", but testing it shows undef).
"$var = []" sets it to an array reference, of course. and "$var = {}" as
previously mentioned, sets $var to a hash reference.
just wanted to correct myself. :)
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 4859] memory bloat in resolver
Posted by bu...@bugzilla.spamassassin.org.
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4859
------- Additional Comments From sidney@sidney.com 2006-04-08 03:53 -------
I don't like exposing id_to_callback as an external component of resolver that
URIDNSBL has to know to clear. How about adding a sub to
Mail::SpamAssassin::DnsResolver->bgabort() that does it and call that from URIDNSBL?
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 4859] memory bloat in resolver
Posted by bu...@bugzilla.spamassassin.org.
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4859
sidney@sidney.com changed:
What |Removed |Added
----------------------------------------------------------------------------
Status Whiteboard| |Needs 2 votes for 3.1 branch
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 4859] memory bloat in resolver
Posted by bu...@bugzilla.spamassassin.org.
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4859
sidney@sidney.com changed:
What |Removed |Added
----------------------------------------------------------------------------
Attachment #3461 is|0 |1
obsolete| |
------- Additional Comments From sidney@sidney.com 2006-04-08 06:01 -------
Created an attachment (id=3462)
--> (http://issues.apache.org/SpamAssassin/attachment.cgi?id=3462&action=view)
patch against 3.1 branch
Committed to trunk revision 392480.
This patch puts the code in an advertised sub of DnsResolver.pm so that
URIDNSBL.pm doesn't have to know about and touch an internal element of the
DnsResolver object.
Please vote on this for 3.1 branch
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 4859] [Review] memory bloat in resolver
Posted by bu...@bugzilla.spamassassin.org.
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4859
------- Additional Comments From schulz@adi.com 2006-06-05 14:12 -------
> did this not make 3.1.2?
It is the 3.1.2 that I got on May 30.
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.