You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by Paul Griffith <pa...@cse.yorku.ca> on 2007/09/10 18:48:20 UTC
Handling Spam Surges
Greetings,
How do you handle Spam surges/DoS attacks? We just had a Spam surge/DoS
and are looking at ways to better withstand (as best as we can) another
surge
Here is how we start SA:
-c -d -r $PIDFILE -s /var/log/spamd --socketpath=$SOCKET
--max-children=150 --min-children=10
Our (1) mail server is configured like this:
CentOS 4.5
Exim 4.67
SpamAssassin version 3.2.3 running on Perl version 5.8.8
ClamAV 0.91.2 (saneSecurity updates)
- handles incoming/outgoing mail
- handles imap/pop/webmail request
Intel D Cpu 3.00Ghz with 2GB of Mem
80GB SATA root disk
200GB SATA mail disk (softraid mirror)
2xIntel e1000
Our mail server was taking a pounding on Friday,
Fri Sep 7 16:17:09 2007 [26914] info: prefork: child states: BBBBB
Fri Sep 7 16:17:09 2007 [26914] info: prefork: child states: BBBBBB
Fri Sep 7 16:17:09 2007 [26914] info: prefork: child states: BBBBBBB
..snip...
..snip...
..snip...
Fri Sep 7 16:17:17 2007 [26914] info: prefork: child states:
BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB
Fri Sep 7 16:17:17 2007 [26914] info: prefork: child states:
BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBIBBB
Fri Sep 7 16:17:19 2007 [26914] info: prefork: child states:
BBBBBBBBBBBBBBBBBBBBBBBBBIBBBBBBBBBBBBBBBBBBBBBBIBBB
..snip..
..snip..
Fri Sep 7 16:19:22 2007 [26914] info: prefork: child states:
BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBSBBSBB
Fri Sep 7 16:19:23 2007 [26914] info: prefork: child states:
BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBIBBISBB
At the mist of the surge we had 95 child processess running, all busy!
Here are the sar memory stats...
kbmemfree kbmemused %memused kbbuffers kbcached kbswpfree
kbswpused %swpused kbswpcad
16:10:02 16804 2056424 99.19 2900 1310880
2040036 208 0.01 0
16:20:10 37676 2035552 98.18 1872 237376 1736152
304092 14.90 78992
16:30:51 13924 2059304 99.33 1292 308944 1044160
996084 48.82 357444
16:40:02 76652 1996576 96.30 8208 1280796 1756236
284008 13.92 178696
Average: 26403 2046825 98.73 5880 1364057
2024199 16045 0.79 6152
Here are the warnings we saw in the spamd log...
Fri Sep 7 16:20:39 2007 [26914] info: prefork: child states:
IBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB
Fri Sep 7 16:20:40 2007 [25431] warn: spf: lookup failed: Can't locate
object method "new" via package "Net::DNS::RR::TXT" at
/xsys/lib/perl5/site_perl/5.8.8/i686-linux/Net/
DNS/RR.pm line 312.
Fri Sep 7 16:20:41 2007 [25428] warn: spf: lookup failed: Can't locate
object method "new" via package "Net::DNS::RR::TXT" at
/xsys/lib/perl5/site_perl/5.8.8/i686-linux/Net/
DNS/RR.pm line 312.
Fri Sep 7 16:22:18 2007 [24684] warn: plugin: eval failed: child
processing timeout at /xsys/sbin//spamd line 1246, <GEN683> line 3398.
Fri Sep 7 16:22:20 2007 [24711] warn: Use of uninitialized value in
pattern match (m//) at /xsys/lib/perl5/5.8.8/utf8_heavy.pl line 211,
<GEN749> line 3398.
Fri Sep 7 16:22:20 2007 [24711] warn: Use of uninitialized value in
scalar assignment at /xsys/lib/perl5/5.8.8/utf8_heavy.pl line 227,
<GEN749> line 3398.
Fri Sep 7 16:26:15 2007 [25227] info: spamd: clean message (1.5/5.0) for
cs242027:9190 in 406.1 seconds, 243776 bytes.
Fri Sep 7 16:26:19 2007 [24688] warn: spamd: copy_config timeout,
respawning child process after 1 messages at /xsys/sbin//spamd line 1103.
Fri Sep 7 16:26:24 2007 [25046] warn: spamd: copy_config timeout,
respawning child process after 1 messages at /xsys/sbin//spamd line 1103.
Fri Sep 7 16:26:28 2007 [24692] warn: spamd: copy_config timeout,
respawning child process after 1 messages at /xsys/sbin//spamd line 1103.
Fri Sep 7 16:30:35 2007 [26914] info: spamd: server successfully spawned
child process, pid 26312
Fri Sep 7 16:30:37 2007 [24685] warn: spamd: copy_config timeout,
respawning child process after 1 messages at /xsys/sbin//spamd line 1103.
Fri Sep 7 16:30:39 2007 [26914] warn: prefork: cannot ping 24702, file
handle not defined, child likely to still be processing SIGCHLD handler
after killing itself
Fri Sep 7 16:30:39 2007 [26914] warn: prefork: killing failed child 24702
fd=undefined at
/xsys/lib/perl5/site_perl/5.8.8/Mail/SpamAssassin/SpamdForkScaling.pm line
171.
Fri Sep 7 16:30:39 2007 [26914] warn: prefork: killed child 24702
Fri Sep 7 16:30:41 2007 [26914] info: spamd: handled cleanup of child pid
24702 due to SIGCHLD
Fri Sep 7 16:30:41 2007 [26914] warn: prefork: cannot ping 24687, file
handle not defined, child likely to still be processing SIGCHLD handler
after killing itself
Fri Sep 7 16:30:41 2007 [26914] warn: prefork: killing failed child 24687
fd=undefined at
/xsys/lib/perl5/site_perl/5.8.8/Mail/SpamAssassin/SpamdForkScaling.pm line
171.
Fri Sep 7 16:30:41 2007 [26914] warn: prefork: killed child 24687
Looking at the swap usage, I was thinking I would be better if I reduced
the number of children processes and let thing queue up. I know I will
also have to look at exim and it's ratelimit command. Any other idea's on
handling spam surges/DoS?
Thanks
Paul
Re: Handling Spam Surges
Posted by Aaron Wolfe <aa...@gmail.com>.
On 9/10/07, Paul Griffith <pa...@cse.yorku.ca> wrote:
>
> Greetings,
>
> How do you handle Spam surges/DoS attacks? We just had a Spam surge/DoS
> and are looking at ways to better withstand (as best as we can) another
> surge
>
>
> Here is how we start SA:
>
> -c -d -r $PIDFILE -s /var/log/spamd --socketpath=$SOCKET
> --max-children=150 --min-children=10
>
> Our (1) mail server is configured like this:
>
> CentOS 4.5
> Exim 4.67
> SpamAssassin version 3.2.3 running on Perl version 5.8.8
> ClamAV 0.91.2 (saneSecurity updates)
> - handles incoming/outgoing mail
> - handles imap/pop/webmail request
>
> Intel D Cpu 3.00Ghz with 2GB of Mem
> 80GB SATA root disk
> 200GB SATA mail disk (softraid mirror)
> 2xIntel e1000
>
> Our mail server was taking a pounding on Friday,
>
> Fri Sep 7 16:17:09 2007 [26914] info: prefork: child states: BBBBB
> Fri Sep 7 16:17:09 2007 [26914] info: prefork: child states: BBBBBB
> Fri Sep 7 16:17:09 2007 [26914] info: prefork: child states: BBBBBBB
> ..snip...
> ..snip...
> ..snip...
> Fri Sep 7 16:17:17 2007 [26914] info: prefork: child states:
> BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB
> Fri Sep 7 16:17:17 2007 [26914] info: prefork: child states:
> BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBIBBB
> Fri Sep 7 16:17:19 2007 [26914] info: prefork: child states:
> BBBBBBBBBBBBBBBBBBBBBBBBBIBBBBBBBBBBBBBBBBBBBBBBIBBB
> ..snip..
> ..snip..
> Fri Sep 7 16:19:22 2007 [26914] info: prefork: child states:
>
> BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBSBBSBB
> Fri Sep 7 16:19:23 2007 [26914] info: prefork: child states:
>
> BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBIBBISBB
>
> At the mist of the surge we had 95 child processess running, all busy!
>
> Here are the sar memory stats...
>
> kbmemfree kbmemused %memused kbbuffers kbcached kbswpfree
> kbswpused %swpused kbswpcad
> 16:10:02 16804 2056424 99.19 2900 1310880
> 2040036 208 0.01 0
> 16:20:10 37676 2035552 98.18 1872 237376 1736152
> 304092 14.90 78992
> 16:30:51 13924 2059304 99.33 1292 308944 1044160
> 996084 48.82 357444
> 16:40:02 76652 1996576 96.30 8208 1280796 1756236
> 284008 13.92 178696
> Average: 26403 2046825 98.73 5880 1364057
> 2024199 16045 0.79 6152
>
>
> Here are the warnings we saw in the spamd log...
>
> Fri Sep 7 16:20:39 2007 [26914] info: prefork: child states:
>
> IBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB
> Fri Sep 7 16:20:40 2007 [25431] warn: spf: lookup failed: Can't locate
> object method "new" via package "Net::DNS::RR::TXT" at
> /xsys/lib/perl5/site_perl/5.8.8/i686-linux/Net/
> DNS/RR.pm line 312.
> Fri Sep 7 16:20:41 2007 [25428] warn: spf: lookup failed: Can't locate
> object method "new" via package "Net::DNS::RR::TXT" at
> /xsys/lib/perl5/site_perl/5.8.8/i686-linux/Net/
> DNS/RR.pm line 312.
>
> Fri Sep 7 16:22:18 2007 [24684] warn: plugin: eval failed: child
> processing timeout at /xsys/sbin//spamd line 1246, <GEN683> line 3398.
> Fri Sep 7 16:22:20 2007 [24711] warn: Use of uninitialized value in
> pattern match (m//) at /xsys/lib/perl5/5.8.8/utf8_heavy.pl line 211,
> <GEN749> line 3398.
> Fri Sep 7 16:22:20 2007 [24711] warn: Use of uninitialized value in
> scalar assignment at /xsys/lib/perl5/5.8.8/utf8_heavy.pl line 227,
> <GEN749> line 3398.
>
> Fri Sep 7 16:26:15 2007 [25227] info: spamd: clean message (1.5/5.0) for
> cs242027:9190 in 406.1 seconds, 243776 bytes.
> Fri Sep 7 16:26:19 2007 [24688] warn: spamd: copy_config timeout,
> respawning child process after 1 messages at /xsys/sbin//spamd line 1103.
> Fri Sep 7 16:26:24 2007 [25046] warn: spamd: copy_config timeout,
> respawning child process after 1 messages at /xsys/sbin//spamd line 1103.
> Fri Sep 7 16:26:28 2007 [24692] warn: spamd: copy_config timeout,
> respawning child process after 1 messages at /xsys/sbin//spamd line 1103.
>
> Fri Sep 7 16:30:35 2007 [26914] info: spamd: server successfully spawned
> child process, pid 26312
> Fri Sep 7 16:30:37 2007 [24685] warn: spamd: copy_config timeout,
> respawning child process after 1 messages at /xsys/sbin//spamd line 1103.
> Fri Sep 7 16:30:39 2007 [26914] warn: prefork: cannot ping 24702, file
> handle not defined, child likely to still be processing SIGCHLD handler
> after killing itself
> Fri Sep 7 16:30:39 2007 [26914] warn: prefork: killing failed child 24702
> fd=undefined at
> /xsys/lib/perl5/site_perl/5.8.8/Mail/SpamAssassin/SpamdForkScaling.pm line
> 171.
> Fri Sep 7 16:30:39 2007 [26914] warn: prefork: killed child 24702
> Fri Sep 7 16:30:41 2007 [26914] info: spamd: handled cleanup of child pid
> 24702 due to SIGCHLD
> Fri Sep 7 16:30:41 2007 [26914] warn: prefork: cannot ping 24687, file
> handle not defined, child likely to still be processing SIGCHLD handler
> after killing itself
> Fri Sep 7 16:30:41 2007 [26914] warn: prefork: killing failed child 24687
> fd=undefined at
> /xsys/lib/perl5/site_perl/5.8.8/Mail/SpamAssassin/SpamdForkScaling.pm line
> 171.
> Fri Sep 7 16:30:41 2007 [26914] warn: prefork: killed child 24687
>
>
> Looking at the swap usage, I was thinking I would be better if I reduced
> the number of children processes and let thing queue up. I know I will
> also have to look at exim and it's ratelimit command. Any other idea's on
> handling spam surges/DoS?
>
> Thanks
> Paul
>
At my site we operate under the presumption that SpamAssassin should be
avoided if at all possible because it is so expensive on our resources
compared to some other easy checks. This helps us to deal with DoS and
"surges" from retarded bots quite well (so far at least).
We reduce the messages bound for SA to less than 10% of our traffic by a
combination of postfix UCE checks, a couple very accurate RBLs, selective
greylisting and our own whitelist. When the surges/DOS happen, they tend to
increase the number of messages thrown away but rarely effect the volume
running through SA.
-Aaron
Summary - Handling Spam Surges
Posted by Paul Griffith <pa...@cse.yorku.ca>.
Here is summary of all the responses. Thanks to all who resonded, your
suggestions have been very helpful.
We will
- reduce the number of SA max-children
- look at ratelimit in exim
- only spam scan messages under a certain size
====================
Aaron Wolfe
We reduce the messages bound for SA to less than 10% of our traffic by
a combination of postfix UCE checks, a couple very accurate RBLs,
selective greylisting and our own whitelist. When the surges/DOS happen,
they tend to increase the number of messages thrown away but rarely effect
the volume running through SA.
Dave Funk
With only 2GB of memory you could die in swapping hell with
max-children=150. Each SA process will take 30~60Mbyes of RSS
(depending upon addition of optional rules & plugins).
This means that 150 children could take 5GB of ram, thus hitting
your swap hard. Either add more RAM or reduce that max-children.
To prevent melt-down from surges/DoS attacks some kind of incoming
SMTP rate limiting is the way to go (with that small a setup).
This would be done by your Exim config, ask the Exim list for
suggestions on this.
Michael Scheidell
Handle it in the MTA.
Best to block all unknown recipients at least.
The rest of what to do in the MTA depends on what MTA you have.
Once the MTA is finished with it then pass it to SA. If under attack,
only thing you can do to help SA is disable network tests till its done.
Visit the mailing list or FAQ's of the MTA you are using for more help
on this.
(example: smtp connection limiting, session tarpiting, even some
firewall rules to limit concurrent connections might help)
===========================
Thanks
Paul
Re: Handling Spam Surges
Posted by David B Funk <db...@engineering.uiowa.edu>.
On Mon, 10 Sep 2007, Paul Griffith wrote:
> Greetings,
>
> How do you handle Spam surges/DoS attacks? We just had a Spam surge/DoS
> and are looking at ways to better withstand (as best as we can) another
> surge
>
>
> Here is how we start SA:
>
> -c -d -r $PIDFILE -s /var/log/spamd --socketpath=$SOCKET
> --max-children=150 --min-children=10
>
> Our (1) mail server is configured like this:
>
> CentOS 4.5
> Exim 4.67
> SpamAssassin version 3.2.3 running on Perl version 5.8.8
> ClamAV 0.91.2 (saneSecurity updates)
> - handles incoming/outgoing mail
> - handles imap/pop/webmail request
>
> Intel D Cpu 3.00Ghz with 2GB of Mem
> 80GB SATA root disk
> 200GB SATA mail disk (softraid mirror)
> 2xIntel e1000
With only 2GB of memory you could die in swapping hell with
max-children=150. Each SA process will take 30~60Mbyes of RSS
(depending upon addition of optional rules & plugins).
This means that 150 children could take 5GB of ram, thus hitting
your swap hard. Either add more RAM or reduce that max-children.
To prevent melt-down from surges/DoS attacks some kind of incoming
SMTP rate limiting is the way to go (with that small a setup).
This would be done by your Exim config, ask the Exim list for
suggestions on this.
--
Dave Funk University of Iowa
<dbfunk (at) engineering.uiowa.edu> College of Engineering
319/335-5751 FAX: 319/384-0549 1256 Seamans Center
Sys_admin/Postmaster/cell_admin Iowa City, IA 52242-1527
#include <std_disclaimer.h>
Better is not better, 'standard' is better. B{