You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by Jean-Paul Natola <jn...@familycareintl.org> on 2010/05/21 21:29:19 UTC
remmonded max children settings
Hi all,
I am constantly getting the server reached --max children setting entries in my log
I started with 10 max children and have been raising it by 2. I am now at 40 , but still getting the messages (though not as often) how high can I go given these specs:
sa 3.3 on freebsd , hardware is a PIV 1.3 ghz with 1 gig of ram 20 gig 5400 rpm PATA drive, and processing an average of 8000 messages a day.
When running top I have seen swap usage go as high as ~500M
TIA,
JP
Re: remmonded max children settings
Posted by Karsten Bräckelmann <gu...@rudersport.de>.
On Fri, 2010-05-21 at 15:37 -0400, Bowie Bailey wrote:
> Jean-Paul Natola wrote:
> > I am constantly getting the server reached --max children setting
> > entries in my log
> >
> > I started with 10 max children and have been raising it by 2. I am
> > now at 40 , but still getting the messages (though not as often) how
> > high can I go given these specs:
> >
> > sa 3.3 on freebsd , hardware is a PIV 1.3 ghz with 1 gig of ram
Lower than 40. Your RAM can't handle it.
> > 20 gig 5400 rpm PATA drive, and processing an average of 8000
> > messages a day.
Ahem. So every mail can take 10 seconds, and you would need only a
single child for that...
Your problem definitely is *not* lack of yet more SA children.
> > When running top I have seen swap usage go as high as ~500M
>
> Lower it until you see the swap usage go away. Having messages waiting
> for an available child is MUCH better than having the system using swap.
Any chance mail is piling up galore at times, maybe even as a result of
going into swap? Appears to me throttling of the MTA isn't working as it
should, and instead loves to hammer SA with all requests at once.
I definitely agree with Bowie. Avoid swap.
--
char *t="\10pse\0r\0dtu\0.@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4";
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i<l;i++){ i%8? c<<=1:
(c=*++x); c&128 && (s+=h); if (!(h>>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}
RE: remmonded max children settings
Posted by Per Jessen <pe...@computer.org>.
David B Funk wrote:
> On Fri, 21 May 2010, Jean-Paul Natola wrote:
>
>> These are the stats for the week
>> Total number of emails processed by the spam filter : 58249
>> Number of spams : 54479 ( 93.53%)
>> Number of clean messages : 3770 ( 6.47%)
>> Average message analysis time : 10.98 seconds
>> Average spam analysis time : 6.76 seconds
>> Average clean message analysis time : 23.35 seconds
>> Average message score : 11.49
>> Average spam score : 20.10
>> Average clean message score : -13.76
>> Total spam volume : 677 Mbytes
>> Total clean volume : 579 Mbytes
>
> That box is rather elderly and under-powered by modern standards.
> New improved versions of SA and other apps are only going to get
> -more- resource hungry.
Whilst that is true, todays SA (3.2.5) does very well handling about
6000msgs/day on my test system consisting of 4 x PII 400MHz with about
400MB RAM each. Each system runs max 5 spamd children, 3 for regular
mail, 2 for honeypotted.
> Let's run the numbers;
> 1 week = 604800 seconds, 604800/58249 messages in a week
> = 10.38 seconds per message.
> Your average analysis time is 10.98 seconds.
>
> Thus you could almost run your system in single threaded mode
> and be OK. two threads should be plenty, if the incoming arrivals
> were evenly distributed. Should be no need for 40 children
> ever, unless you're getting hit with a spam flood.
The MTA queue will deal with a flood far better than spamd - my test
system boxes regularly have queues of up to 300 emails, occasionally
more, but they're mostly honeypotted and rarely last very long.
> So you are either going to need to upgrade it or protect it by
> reducing the number of concurrent messages being processed.
+1
/Per Jessen, Zürich
RE: remmonded max children settings
Posted by David B Funk <db...@engineering.uiowa.edu>.
On Fri, 21 May 2010, Jean-Paul Natola wrote:
>
> On Fri, 21 May 2010, Bowie Bailey wrote:
>
> > Jean-Paul Natola wrote:
> > > Hi all,
> > >
> > > I am constantly getting the server reached --max children setting entries in my log
> > >
> > > I started with 10 max children and have been raising it by 2. I am now at 40 , but still getting the messages (though not as often) how high can I go given these specs:
> > >
> > > sa 3.3 on freebsd , hardware is a PIV 1.3 ghz with 1 gig of ram 20 gig 5400 rpm PATA drive, and processing an average of 8000 messages a day.
> > >
> > > When running top I have seen swap usage go as high as ~500M
> >Can you tune your MTA to limit the number of incoming SMTP connections?
>
> These are my mta settings
> smtp_accept_max = 10
> smtp_accept_max_per_host = 5
> smtp_accept_reserve = 0
>
> These are the stats for the week
> Total number of emails processed by the spam filter : 58249
> Number of spams : 54479 ( 93.53%)
> Number of clean messages : 3770 ( 6.47%)
> Average message analysis time : 10.98 seconds
> Average spam analysis time : 6.76 seconds
> Average clean message analysis time : 23.35 seconds
> Average message score : 11.49
> Average spam score : 20.10
> Average clean message score : -13.76
> Total spam volume : 677 Mbytes
> Total clean volume : 579 Mbytes
That box is rather elderly and under-powered by modern standards.
New improved versions of SA and other apps are only going to get
-more- resource hungry. (I'm old enough to remember when putting
a 128KB memory board in a PDP-11/34 was a big upgrade ;).
Let's run the numbers;
1 week = 604800 seconds, 604800/58249 messages in a week
= 10.38 seconds per message.
Your average analysis time is 10.98 seconds.
Thus you could almost run your system in single threaded mode
and be OK. two threads should be plenty, if the incoming arrivals
were evenly distributed. Should be no need for 40 children
ever, unless you're getting hit with a spam flood.
So you are either going to need to upgrade it or protect it by
reducing the number of concurrent messages being processed.
I don't know exim (we use sendmail & postfix) but it looks like
you need to adjust those mta settings to throttle down that
incoming rate.
One other possible problem source, network timeouts. SA makes heavy
use of DNS lookups for various network tests (RBLS, SPF, DKIM, botnet,
etc) as well as optional things like DCC, Razor, etc.
So network (particularly DNS) issues can cause SA to take too long
and timeout. A DNS cache run on your SA box will help that issue
(but of course makes demands on RAM & CPU).
--
Dave Funk University of Iowa
<dbfunk (at) engineering.uiowa.edu> College of Engineering
319/335-5751 FAX: 319/384-0549 1256 Seamans Center
Sys_admin/Postmaster/cell_admin Iowa City, IA 52242-1527
#include <std_disclaimer.h>
Better is not better, 'standard' is better. B{
RE: remmonded max children settings
Posted by Jean-Paul Natola <jn...@familycareintl.org>.
On Fri, 21 May 2010, Bowie Bailey wrote:
> Jean-Paul Natola wrote:
> > Hi all,
> >
> > I am constantly getting the server reached --max children setting entries in my log
> >
> > I started with 10 max children and have been raising it by 2. I am now at 40 , but still getting the messages (though not as often) how high can I go given these specs:
> >
> > sa 3.3 on freebsd , hardware is a PIV 1.3 ghz with 1 gig of ram 20 gig 5400 rpm PATA drive, and processing an average of 8000 messages a day.
> >
> > When running top I have seen swap usage go as high as ~500M
> >
>
> Lower it until you see the swap usage go away. Having messages waiting
> for an available child is MUCH better than having the system using swap.
>Can you tune your MTA to limit the number of incoming SMTP connections?
These are my mta settings
smtp_accept_max = 10
smtp_accept_max_per_host = 5
smtp_accept_reserve = 0
>8000 messages a day works out to about 5 per minute, so on average you
>shouldn't have more than 2 or 3 simultaneous messages in the queue. This
>probably means that you're getting hit with sporadic spam/dictionary
>attack floods that may peak at multiple messages/second.
>Throttle those at the incoming MTA and your SA should be much happier.
>One other question, that 8000 messages a day, are those total incoming
>messages or 8000 ham messages? Assuming a 90% spam rate, to get 8000 hams
>a day you need to process 80,000 total incoming messages a day.
These are the stats for the week
Total number of emails processed by the spam filter : 58249
Number of spams : 54479 ( 93.53%)
Number of clean messages : 3770 ( 6.47%)
Average message analysis time : 10.98 seconds
Average spam analysis time : 6.76 seconds
Average clean message analysis time : 23.35 seconds
Average message score : 11.49
Average spam score : 20.10
Average clean message score : -13.76
Total spam volume : 677 Mbytes
Total clean volume : 579 Mbytes
I don't know if or how the "spamd timeouts" are categorized
Re: remmonded max children settings
Posted by David B Funk <db...@engineering.uiowa.edu>.
On Fri, 21 May 2010, Bowie Bailey wrote:
> Jean-Paul Natola wrote:
> > Hi all,
> >
> > I am constantly getting the server reached --max children setting entries in my log
> >
> > I started with 10 max children and have been raising it by 2. I am now at 40 , but still getting the messages (though not as often) how high can I go given these specs:
> >
> > sa 3.3 on freebsd , hardware is a PIV 1.3 ghz with 1 gig of ram 20 gig 5400 rpm PATA drive, and processing an average of 8000 messages a day.
> >
> > When running top I have seen swap usage go as high as ~500M
> >
>
> Lower it until you see the swap usage go away. Having messages waiting
> for an available child is MUCH better than having the system using swap.
Can you tune your MTA to limit the number of incoming SMTP connections?
8000 messages a day works out to about 5 per minute, so on average you
shouldn't have more than 2 or 3 simultaneous messages in the queue. This
probably means that you're getting hit with sporadic spam/dictionary
attack floods that may peak at multiple messages/second.
Throttle those at the incoming MTA and your SA should be much happier.
One other question, that 8000 messages a day, are those total incoming
messages or 8000 ham messages? Assuming a 90% spam rate, to get 8000 hams
a day you need to process 80,000 total incoming messages a day.
That's probably too big a work load for that little P4 box unless
you have serious MTA filtering in front of it (RBLs, graylisting, etc).
--
Dave Funk University of Iowa
<dbfunk (at) engineering.uiowa.edu> College of Engineering
319/335-5751 FAX: 319/384-0549 1256 Seamans Center
Sys_admin/Postmaster/cell_admin Iowa City, IA 52242-1527
#include <std_disclaimer.h>
Better is not better, 'standard' is better. B{
Re: remmonded max children settings
Posted by Bowie Bailey <Bo...@BUC.com>.
Jean-Paul Natola wrote:
> Hi all,
>
>
>
> I am constantly getting the server reached --max children setting entries in my log
>
> I started with 10 max children and have been raising it by 2. I am now at 40 , but still getting the messages (though not as often) how high can I go given these specs:
>
> sa 3.3 on freebsd , hardware is a PIV 1.3 ghz with 1 gig of ram 20 gig 5400 rpm PATA drive, and processing an average of 8000 messages a day.
>
> When running top I have seen swap usage go as high as ~500M
>
Lower it until you see the swap usage go away. Having messages waiting
for an available child is MUCH better than having the system using swap.
--
Bowie
Re: remmonded max children settings
Posted by John Hardin <jh...@impsec.org>.
On Sat, 22 May 2010, John GALLET wrote:
> This stopped dead "spam outbursts": your 8000 mails per day are NOT
> received in a linear way, but everytime a spammer sends you a "batch"
> and you just can not keep up: who would drink from a fire hose ?
Question: are you utilizing any SMTP-time DNSBLs? Many admins trust the
Spamhaus Zen DNSBL enough to reject at SMTP time based on it. That would
also tend to reduce your load (though, granted, not as much as firewall
rules!)
--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
Users mistake widespread adoption of Microsoft Office for the
development of a document format standard.
Re: remmonded max children settings
Posted by John GALLET <sp...@saphirtech.fr>.
Hi,
FWIW: the way I solved it was limiting the number of concurrently incoming
spam because my box serves only few different domains, so I limited the
number of connections from the same smtp client to 5 using iptables (the
connn-limit module). This might or not be possible for others.
This stopped dead "spam outbursts": your 8000 mails per day are NOT
received in a linear way, but everytime a spammer sends you a "batch" and
you just can not keep up: who would drink from a fire hose ?
HTH
JGA
Re: remmonded max children settings
Posted by RW <rw...@googlemail.com>.
On Fri, 21 May 2010 16:50:32 -0400
Bowie Bailey <Bo...@BUC.com> wrote:
> > The reason I was trying to tweak it, maybe I'm going about it
> > wrong, but spamd winds up timing out a lot. Then people call and
> > say there messages were not delivered or delayed etc...
>
> That may be a consequence of the swap usage. Lower the number of
> children until you do not see any swap usage.
OTOH if you do want to push the system to its limits make sure you are
using the default spamd prefork algorithm, and not round-robin.
Re: remmonded max children settings
Posted by Bowie Bailey <Bo...@BUC.com>.
Jean-Paul Natola wrote:
>> Just accept the fact that mail gets into a queue when using processes like
>> SA. If outgoing spam is not a concern, set your system so that outbound
>> mail is not passed thru SA.
>>
>
> I actually only use the exim/sa as incoming filter, I do not send through it.
>
> The reason I was trying to tweak it, maybe I'm going about it wrong, but spamd winds up timing out a lot. Then people call and say there messages were not delivered or delayed etc...
>
That may be a consequence of the swap usage. Lower the number of
children until you do not see any swap usage. Alternately, add some ram
so you can support more children.
Adding ram is always a good idea if you can manage it. Linux will use
any excess ram as disk cache, which can also help speed up the system.
I always spec out systems with 4GB of ram these days.
--
Bowie
RE: remmonded max children settings
Posted by Jean-Paul Natola <jn...@familycareintl.org>.
>Just accept the fact that mail gets into a queue when using processes like
>SA. If outgoing spam is not a concern, set your system so that outbound
>mail is not passed thru SA.
I actually only use the exim/sa as incoming filter, I do not send through it.
The reason I was trying to tweak it, maybe I'm going about it wrong, but spamd winds up timing out a lot. Then people call and say there messages were not delivered or delayed etc...
Re: remmonded max children settings
Posted by Jari Fredriksson <ja...@iki.fi>.
> Hi all,
>
>
>
> I am constantly getting the server reached --max children setting entries
> in my log
>
> I started with 10 max children and have been raising it by 2. I am now at
> 40 , but still getting the messages (though not as often) how high can I
> go given these specs:
>
> sa 3.3 on freebsd , hardware is a PIV 1.3 ghz with 1 gig of ram 20 gig
> 5400 rpm PATA drive, and processing an average of 8000 messages a day.
>
What they said, it only generates swapping if you try to get rid of that
(IMO dummy message).
Processing SA will always take longer than basic delivery, and no matter
how you tune you will always see that message.
Just accept the fact that mail gets into a queue when using processes like
SA. If outgoing spam is not a concern, set your system so that outbound
mail is not passed thru SA.