You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by Matias Lopez Bergero <ml...@udesa.edu.ar> on 2006/10/09 17:19:29 UTC
Mail server performance problems. Possible SA slow down?
Hello!
I was very happy using SpamAssassin at my email server (Xeon 2.8GHz, 1.5
GB memory, Dual Ultra SCSI HD 73.4GB in RAID 1, Linux 2.4.33)
The last few weeks I have noted (angry users calling me by phone) that
the server is really slow. The loadav goes from 1.5 to 12.5; normally is
about 3.00.
There are only 2500 email boxes at the server. The server is running:
Sendmail, SpamAssassin 3.1.5 (using milter-spamc), ClamAV (using
clamav-milter), Apache 1.3.x, SquirrelMail, pop3, etc.
I have seeing some king of bursts of incoming emails (spam mostly), that
it is producing a DoS effect.
The server shows a table of ~1700 processes and about ~800 tcp sessions
(sendmail and milter-spamc most) during this bursts. This seems to
prevent other users from connecting to the server in order to use pop3
or smtp services.
I have increased the child processes of spamd, but I was unsuccessfully
to reduce this effect(I have seing in the logs a message about the need
of increase the spamd childs). Also I tweak the sendmail.cf to easy the
connection, but the problem persist.
Looks to me that SpamAssassin is taking to long to process the incoming
emails, and as result, it is slowing down the server, and finally
causing the DoS.
Can anyone help me with some ideas to solve this? or to see were exactly
is the problem? Do I need to improve my hardware?
Thanks.
BR,
Matias.
Re: Mail server performance problems. Possible SA slow down?
Posted by "Jack L. Stone" <ja...@sage-american.com>.
At 01:09 PM 10.9.2006 -0500, David B Funk wrote:
>On Mon, 9 Oct 2006, Matias Lopez Bergero wrote:
>
>> Hello!
>>
>> I was very happy using SpamAssassin at my email server (Xeon 2.8GHz, 1.5
>> GB memory, Dual Ultra SCSI HD 73.4GB in RAID 1, Linux 2.4.33)
>>
>> There are only 2500 email boxes at the server. The server is running:
>> Sendmail, SpamAssassin 3.1.5 (using milter-spamc), ClamAV (using
>> clamav-milter), Apache 1.3.x, SquirrelMail, pop3, etc.
>>
>
>> The server shows a table of ~1700 processes and about ~800 tcp sessions
>> (sendmail and milter-spamc most) during this bursts. This seems to
>> prevent other users from connecting to the server in order to use pop3
>> or smtp services.
>
>With all those services running on that box your 1.5GB isn't enough
>to service 800 incoming simultanious connects. Either throw more RAM
>at it or edit your sendmail config to limit the number of incoming
>connections. Use the sendmail 'MaxDaemonChildren' and
>'ConnectionRateThrottle' options to limit incoming connects.
>
>--
>Dave Funk University of Iowa
I haven't seen mention of adding more mail queues for sendmail's use. I run
several busy mail discussion lists + some fairly large customer mail lists
-- in additon to my user traffic.
I increased the number of queues (x10 then x20) and saw a blazing jump in
speed.
Something to consider....???
(^_^)
Happy trails,
Jack L. Stone
System Admin
Sage-american
Re: Mail server performance problems. Possible SA slow down?
Posted by David B Funk <db...@engineering.uiowa.edu>.
On Mon, 9 Oct 2006, Matias Lopez Bergero wrote:
> Hello!
>
> I was very happy using SpamAssassin at my email server (Xeon 2.8GHz, 1.5
> GB memory, Dual Ultra SCSI HD 73.4GB in RAID 1, Linux 2.4.33)
>
> There are only 2500 email boxes at the server. The server is running:
> Sendmail, SpamAssassin 3.1.5 (using milter-spamc), ClamAV (using
> clamav-milter), Apache 1.3.x, SquirrelMail, pop3, etc.
>
> The server shows a table of ~1700 processes and about ~800 tcp sessions
> (sendmail and milter-spamc most) during this bursts. This seems to
> prevent other users from connecting to the server in order to use pop3
> or smtp services.
With all those services running on that box your 1.5GB isn't enough
to service 800 incoming simultanious connects. Either throw more RAM
at it or edit your sendmail config to limit the number of incoming
connections. Use the sendmail 'MaxDaemonChildren' and
'ConnectionRateThrottle' options to limit incoming connects.
--
Dave Funk University of Iowa
<dbfunk (at) engineering.uiowa.edu> College of Engineering
319/335-5751 FAX: 319/384-0549 1256 Seamans Center
Sys_admin/Postmaster/cell_admin Iowa City, IA 52242-1527
#include <std_disclaimer.h>
Better is not better, 'standard' is better. B{
Re: Mail server performance problems. Possible SA slow down?
Posted by Matias Lopez Bergero <ml...@udesa.edu.ar>.
Chris St. Pierre wrote:
> On Mon, 9 Oct 2006, Craig Baird wrote:
>
>
>>I have an old Redhat box that started doing this a while back. After a lot of
>>hair pulling, I finally figured out that the problem was related to spam
>>floods, but seemed to be caused by the syslog daemon.
>
>
> That's actually not a bad suggestion; the OP should check to make sure
> that whatever log SpamAssassin is going to (probably the maillog) is
> asynchronous (is prefixed with a "-"), like this:
>
> mail.* -/var/log/maillog
>
> That's the default on most Linuxes, but if you've changed that, or if
> you're filtering SpamAssassin stuff to a different log, that could
> cause the problems.
That is actually very interesting. One of my ideas was that the server
was getting slow due to the disc usage, and I have checked my
syslog.conf and I did not hove the asynchronous logging mode set for
maillog.
Thanks for the tip.
BR,
Matias.
Re: Mail server performance problems. Possible SA slow down?
Posted by "Chris St. Pierre" <st...@NebrWesleyan.edu>.
On Mon, 9 Oct 2006, Craig Baird wrote:
> I have an old Redhat box that started doing this a while back. After a lot of
> hair pulling, I finally figured out that the problem was related to spam
> floods, but seemed to be caused by the syslog daemon.
That's actually not a bad suggestion; the OP should check to make sure
that whatever log SpamAssassin is going to (probably the maillog) is
asynchronous (is prefixed with a "-"), like this:
mail.* -/var/log/maillog
That's the default on most Linuxes, but if you've changed that, or if
you're filtering SpamAssassin stuff to a different log, that could
cause the problems.
Chris St. Pierre
Unix Systems Administrator
Nebraska Wesleyan University
Re: Mail server performance problems. Possible SA slow down?
Posted by Craig Baird <cr...@xpressweb.com>.
I have an old Redhat box that started doing this a while back. After a lot of
hair pulling, I finally figured out that the problem was related to spam
floods, but seemed to be caused by the syslog daemon. I shut down syslogd
for a few days, and the problem went away completely. After those few days,
I turned syslogd back on, but configured it to log to a separate syslog
server (thinking that perhaps the problem was disk I/O related). However,
the problem began happening again. I finally downgraded my syslogd to the
previous version, and haven't seen any problems since.
Note that this is very likely not what is causing your problem. But it
sometimes pays to consider the non-obvious. Your problem may be caused by
something relatively unrelated to (but affected by) mail.
Craig
Quoting Matias Lopez Bergero <ml...@udesa.edu.ar>:
> Hello!
>
> I was very happy using SpamAssassin at my email server (Xeon 2.8GHz, 1.5
> GB memory, Dual Ultra SCSI HD 73.4GB in RAID 1, Linux 2.4.33)
>
> The last few weeks I have noted (angry users calling me by phone) that
> the server is really slow. The loadav goes from 1.5 to 12.5; normally is
> about 3.00.
>
> There are only 2500 email boxes at the server. The server is running:
> Sendmail, SpamAssassin 3.1.5 (using milter-spamc), ClamAV (using
> clamav-milter), Apache 1.3.x, SquirrelMail, pop3, etc.
>
> I have seeing some king of bursts of incoming emails (spam mostly), that
> it is producing a DoS effect.
>
> The server shows a table of ~1700 processes and about ~800 tcp sessions
> (sendmail and milter-spamc most) during this bursts. This seems to
> prevent other users from connecting to the server in order to use pop3
> or smtp services.
>
> I have increased the child processes of spamd, but I was unsuccessfully
> to reduce this effect(I have seing in the logs a message about the need
> of increase the spamd childs). Also I tweak the sendmail.cf to easy the
> connection, but the problem persist.
>
> Looks to me that SpamAssassin is taking to long to process the incoming
> emails, and as result, it is slowing down the server, and finally
> causing the DoS.
>
> Can anyone help me with some ideas to solve this? or to see were exactly
> is the problem? Do I need to improve my hardware?
>
> Thanks.
>
> BR,
> Matias.
>
>
Re: Mail server performance problems. Possible SA slow down?
Posted by "Chris St. Pierre" <st...@NebrWesleyan.edu>.
On Mon, 9 Oct 2006, Matias Lopez Bergero wrote:
>Thanks Robert,
>
>I have requested more memory and an additional processor for the server,
>hope they give me something...
>
>I will be working with the Greylisting and validrcptto, and let you know
>how it goes.
>
>Thanks again,
>
>BR,
>Matias.
You might also consider using DNSBL at the MTA level, which could
significantly reduce the amount of scanning SA does. We run lots of
fairly intensive tests on our mail -- ImageInfo, FuzzyOcr, several
network tests, etc. -- so we use greylisting, DNSBL, and a number of
header checks within Postfix to prevent the lion's share of spam from
ever reaching SpamAssassin. If the temporary spikes in spam you're
seeing are originating from a single IP/class C/etc., blacklisting at
the MTA could be especially effective.
Here's a graph of the activity on one of our mail servers over the
past 24 hours:
http://www.nebrwesleyan.edu/people/stpierre/mailgraph.png
Notice that we reject *lots* of messages -- more than we mark as
spam. Lots of messages are greylisted, too, although it's
prohibitively difficult to graph how many are eventually accepted.
Still, you can see that we're rejecting over 9 messages per minute due
to bad HELOs, bogus recipients, etc., and only marking 3 messages per
minute as spam.
Chris St. Pierre
Unix Systems Administrator
Nebraska Wesleyan University
RE: Mail server performance problems. Possible SA slow down?
Posted by Email Lists <li...@abbacomm.net>.
->
-> I have requested more memory and an additional processor for the server,
-> hope they give me something...
->
-> I will be working with the Greylisting and validrcptto, and let you know
-> how it goes.
->
-> Thanks again,
->
-> BR,
-> Matias.
Welcome. :-)
Remember validrcptto is a qmail patch... http://qmail.jms1.net
The *theory* behind it is what I was getting at.
I don't know how to do that in sendmail or whatever you are using as I am
not as well versed in sendmail as I used to be years ago when I knew it
inside out and it was my main MTA
It is hopefully really easy flick of a switch thing in sendmail.* or
something
Thanks
- rh
--
Robert - Abba Communications
Computer & Internet Services
(509) 624-7159 - www.abbacomm.net
Re: Mail server performance problems. Possible SA slow down?
Posted by Matias Lopez Bergero <ml...@udesa.edu.ar>.
Email Lists wrote:
> -> [...]
> -> Looks to me that SpamAssassin is taking to long to process the
> -> incoming emails, and as result, it is slowing down the server, and
> -> finally causing the DoS.
> [...]
>
> The first thing to do is relieve some of the load. To do this we need better
> mail gate keeping.
>
> Find and run the proper and right greylisting function for your server. That
> will swat away some of the bursty sp*mmers.
>
> Then the next thing to do is implement a function we call validrcptto which,
> rejects emails to nonexistent email addresses at the smtp level.
>
> Those two things alone will help a lot.
>
> If you do not do those two things, start there.
>
> Increasing the amount of spamd child processes may not be the correct
> initial answer... that can eat up the RAM and put you into heavy swap.
>
> Ummmmm if you are in a position to, add a bunch more RAM to the server.
>
> Then later, when you are ready and if it will accept it, add a second proper
> mate of a processor and go DUAL.
>
> Don't forget to check all your logs, you will get other valuable hints to
> help deal with issues.
>
> Let us know how it turns out.
Thanks Robert,
I have requested more memory and an additional processor for the server,
hope they give me something...
I will be working with the Greylisting and validrcptto, and let you know
how it goes.
Thanks again,
BR,
Matias.
RE: Mail server performance problems. Possible SA slow down?
Posted by Email Lists <li...@abbacomm.net>.
->
-> I was very happy using SpamAssassin at my email server (Xeon 2.8GHz,
-> 1.5 GB memory, Dual Ultra SCSI HD 73.4GB in RAID 1, Linux 2.4.33)
->
-> The last few weeks I have noted (angry users calling me by phone)
-> that the server is really slow. The loadav goes from 1.5 to 12.5;
-> normally is about 3.00.
->
-> There are only 2500 email boxes at the server. The server is running:
-> Sendmail, SpamAssassin 3.1.5 (using milter-spamc), ClamAV (using
-> clamav-milter), Apache 1.3.x, SquirrelMail, pop3, etc.
->
-> I have seeing some king of bursts of incoming emails (spam mostly),
-> that it is producing a DoS effect.
->
-> The server shows a table of ~1700 processes and about ~800 tcp
-> sessions (sendmail and milter-spamc most) during this bursts. This
-> seems to prevent other users from connecting to the server in order
-> to use pop3 or smtp services.
->
-> I have increased the child processes of spamd, but I was
-> unsuccessfully to reduce this effect(I have seing in the logs a
-> message about the need of increase the spamd childs). Also I tweak
-> the sendmail.cf to easy the connection, but the problem persist.
->
-> Looks to me that SpamAssassin is taking to long to process the
-> incoming emails, and as result, it is slowing down the server, and
-> finally causing the DoS.
->
-> Can anyone help me with some ideas to solve this? or to see were
-> exactly is the problem? Do I need to improve my hardware?
->
-> Thanks.
->
-> BR,
-> Matias.
The first thing to do is relieve some of the load. To do this we need better
mail gate keeping.
Find and run the proper and right greylisting function for your server. That
will swat away some of the bursty sp*mmers.
Then the next thing to do is implement a function we call validrcptto which,
rejects emails to nonexistent email addresses at the smtp level.
Those two things alone will help a lot.
If you do not do those two things, start there.
Increasing the amount of spamd child processes may not be the correct
initial answer... that can eat up the RAM and put you into heavy swap.
Ummmmm if you are in a position to, add a bunch more RAM to the server.
Then later, when you are ready and if it will accept it, add a second proper
mate of a processor and go DUAL.
Don't forget to check all your logs, you will get other valuable hints to
help deal with issues.
Let us know how it turns out.
- rh
--
Robert - Abba Communications
Computer & Internet Services
(509) 624-7159 - www.abbacomm.net