You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by Debbie D <we...@beautytech.com> on 2006/10/16 20:28:55 UTC
I'm getting killed with spammers
I am a learn as I go type of hosting.. my server with cpanel exim SA and
ClamAV does a good job for the most part but since last Monday I have been
getting major issues.. I do read this list when I have time or remember to
do so but more importantly when issues crop up, sometimes I get it,
sometimes you guys are so far over my head I want to run screaming from the
PC..
I need some help here..
Last Mon, Tues & Wed I had severe inflow of spam, always at 12.30p EST, Wed
it didn't stop till almost 5p. The server seems to not be very cooperative
when the queue grows over 200 or so.
I have max child set to 15 (up from 5) and not sure what else I can offer in
the way of what you need to know to help me, but if you tell me where to
look I can spout what you need.
The install is out of the box with few if any mods except exim does have the
dictionary attack, I run BFD and APF
I do not believe I have been hacked into.. I DO read the logwatch daily and
do poke around looking for dropped files on a semi regular basis..
this high amount of spam, (BTW scoring at 20-well over 1000) is killing the
loads and I have screaming clients..
Just this afternoon (again around 12.30) it loaded up again with 312 mails..
the web based control panel was reacting so slow I would get 3 new ones for
every one I managed to delete or deliver (I could not just delete the queue
because some were actually valid mails in there) Server loads rose to well
over 30, I shut exim - but cpanel was so kind to automagically restart it
every time.. tried a reboot from ssh but that just hung.. the tech peeps did
it from their end it it worked and brought the loads down so I could delete
faster than they came in and now we're back to normal loads and queue
I did upgrade to SA 3.1.7 last week - Wed night after a long day of battling
the loads.. and that seemed to go well
suggestions? Offers of help???
thanks
Re: I'm getting killed with spammers
Posted by Logan Shaw <ls...@emitinc.com>.
On Mon, 16 Oct 2006, Debbie D wrote:
> I have max child set to 15 (up from 5) and not sure what else I can offer in
> the way of what you need to know to help me, but if you tell me where to
> look I can spout what you need.
:
:
> Just this afternoon (again around 12.30) it loaded up again with 312 mails..
> the web based control panel was reacting so slow I would get 3 new ones for
> every one I managed to delete or deliver (I could not just delete the queue
> because some were actually valid mails in there) Server loads rose to well
> over 30, I shut exim
You probably have max children set too high. When a big
bunch of messages come in, they all run, you don't have
enough memory, and your system starts swapping like crazy.
That brings everything on your server to a near halt.
It reduces throughput, which means you get a backlog, which
means you get stuck in this state because all the children
stay active hogging RAM and trying to process the backlog.
The solution is to either expand the RAM so the system can
really handle that many active children at once, or set the
maximum number of children to something much lower. Try 2
or 3 even. It seems like more children would mean more work
getting done, and that's true, but it's only true up to a point,
and you've passed that point.
- Logan
R: I'm getting killed with spammers
Posted by Giampaolo Tomassoni <g....@libero.it>.
> Sorry, I was being a bit vague, I've got a stateful firewall between my
> mailserver and the external world, and I kept seeing that there were
> session timeouts "no_connection_for_this_packet" from a lot of different
> places.
>
> There's absolutely no problems with my connection or my mailserver load,
> and it was something that was leaving me a bit confused.
This seems to be a sort of address-spoofing: someone sends you a SYN packet directed to the smtp port of your server, but the source address of the packet itself is fake.
This kind of attack would attempt to cause a DoS or, even worse, may be used to attempt issue a short message apparently caming from 127.0.0.1 or your intranet. However, apart the fact you got your firewall, this kind of attack may eventually have success on quite old OSes: linux, in example, has a lot of ways to discover that these packets are not valid (by syn_cookies or simply since they came from the "wrong" interface).
However, this matter fits better on a firewall list or maybe on the linux-networking list.
giampaolo
>
> I don't use anvil, at least not at the moment.
>
> A sort of moment of clarity.
>
> Nick
Re: I'm getting killed with spammers
Posted by nick <ni...@mobilia.it>.
John Andersen wrote:
> On Wednesday 18 October 2006 00:50, nick wrote:
>
>> So that's what my firewall has been killing.
>>
>> I kept noticing timeout sessions with my mailserver (in the firewall
>> log), and wondered why that was happening.
>
> You should see anvil messages in mail log, but from the man page
> it is not at all clear that the firewall would be involved. It seems to
> handle this at the smtp server.
>
Sorry, I was being a bit vague, I've got a stateful firewall between my
mailserver and the external world, and I kept seeing that there were
session timeouts "no_connection_for_this_packet" from a lot of different
places.
There's absolutely no problems with my connection or my mailserver load,
and it was something that was leaving me a bit confused.
I don't use anvil, at least not at the moment.
A sort of moment of clarity.
Nick
Re: I'm getting killed with spammers
Posted by John Andersen <js...@pen.homeip.net>.
On Wednesday 18 October 2006 00:50, nick wrote:
> So that's what my firewall has been killing.
>
> I kept noticing timeout sessions with my mailserver (in the firewall
> log), and wondered why that was happening.
You should see anvil messages in mail log, but from the man page
it is not at all clear that the firewall would be involved. It seems to
handle this at the smtp server.
--
_____________________________________
John Andersen
Re: I'm getting killed with spammers
Posted by nick <ni...@mobilia.it>.
John Andersen wrote:
> On Tuesday 17 October 2006 23:09, Bill Taroli wrote:
>> Debbie D wrote:
>>> Last Mon, Tues & Wed I had severe inflow of spam, always at 12.30p EST,
>>> Wed it didn't stop till almost 5p. The server seems to not be very
>>> cooperative when the queue grows over 200 or so.
>>> ...
>>> this high amount of spam, (BTW scoring at 20-well over 1000) is killing
>>> the loads and I have screaming clients..
>> I don't know that you're alone in seeing this increased traffic. For
>> another domain I help manage, they were seeing a large influx of
>> connections. For the most part, sender verification and RBL's were
>> blocking them. But then they threw in a little twist... opening SMTP
>> sessions and letting them sit. Open enough of these and processes build
>> up (awaiting timeout) doing nothing and new connections fail -- a crude
>> but effective DOS.
>>
>
> Isn't this something Anvil is designed to handle? It seems SuSE installs
> this by default for postfix. I see log entries where is rate limits some
> IPs, usually when it looks like they are doing a dictionary job on me.
>
> The Postfix anvil(8) server maintains short-term statistics to defend
> against clients that hammer a server with either too many simultaneous
> sessions, or with too many successive requests within a configurable
> time interval.
>
>
So that's what my firewall has been killing.
I kept noticing timeout sessions with my mailserver (in the firewall
log), and wondered why that was happening.
Re: I'm getting killed with spammers
Posted by John Andersen <js...@pen.homeip.net>.
On Tuesday 17 October 2006 23:09, Bill Taroli wrote:
> Debbie D wrote:
> > Last Mon, Tues & Wed I had severe inflow of spam, always at 12.30p EST,
> > Wed it didn't stop till almost 5p. The server seems to not be very
> > cooperative when the queue grows over 200 or so.
> > ...
> > this high amount of spam, (BTW scoring at 20-well over 1000) is killing
> > the loads and I have screaming clients..
>
> I don't know that you're alone in seeing this increased traffic. For
> another domain I help manage, they were seeing a large influx of
> connections. For the most part, sender verification and RBL's were
> blocking them. But then they threw in a little twist... opening SMTP
> sessions and letting them sit. Open enough of these and processes build
> up (awaiting timeout) doing nothing and new connections fail -- a crude
> but effective DOS.
>
Isn't this something Anvil is designed to handle? It seems SuSE installs
this by default for postfix. I see log entries where is rate limits some
IPs, usually when it looks like they are doing a dictionary job on me.
The Postfix anvil(8) server maintains short-term statistics to defend
against clients that hammer a server with either too many simultaneous
sessions, or with too many successive requests within a configurable
time interval.
--
_____________________________________
John Andersen
Re: I'm getting killed with spammers
Posted by Bill Taroli <bi...@billsden.org>.
Debbie D wrote:
> Last Mon, Tues & Wed I had severe inflow of spam, always at 12.30p EST, Wed
> it didn't stop till almost 5p. The server seems to not be very cooperative
> when the queue grows over 200 or so.
> ...
> this high amount of spam, (BTW scoring at 20-well over 1000) is killing the
> loads and I have screaming clients..
>
I don't know that you're alone in seeing this increased traffic. For
another domain I help manage, they were seeing a large influx of
connections. For the most part, sender verification and RBL's were
blocking them. But then they threw in a little twist... opening SMTP
sessions and letting them sit. Open enough of these and processes build
up (awaiting timeout) doing nothing and new connections fail -- a crude
but effective DOS.
In my case, I now have a job running there that frequently scans the
logs to check for messages resulting in these kinds of connections and
adds them to a block list. Not perfect, but it has proved very
effective. In this case, Courier is being used... so sender
verification, RBL, SPF, etc checks happen directly in the SMTP daemon
even before spamassassin gets it's hooks on the message. I don't know
what options exist for your stack, but it's well worth looking into to
help filter out significant noise. RBL checks alone can do wonders.
Bill
Re: I'm getting killed with spammers
Posted by Debbie D <we...@beautytech.com>.
> On Mon, October 16, 2006 2:28 pm, Debbie D said:
>
>> this high amount of spam, (BTW scoring at 20-well over 1000) is killing
>> the loads and I have screaming clients..
>>
>> Just this afternoon (again around 12.30) it loaded up again with 312
>> mails.. the web based control panel was reacting so slow I would get 3
>> new ones for every one I managed to delete or deliver (I could not just
>> delete the queue because some were actually valid mails in there) Server
>> loads rose to well over 30, I shut exim - but cpanel was so kind to
>> automagically restart it every time.. tried a reboot from ssh but that
>> just hung.. the tech peeps did it from their end it it worked and brought
>> the loads down so I could delete faster than they came in and now we're
>> back to normal loads and queue
>>
>> I did upgrade to SA 3.1.7 last week - Wed night after a long day of
>> battling the loads.. and that seemed to go well
>>
>> suggestions? Offers of help???
> At this point, you probably need to find some way to blacklist part of
> that load, to keep your server from dealing with it. It may be possible
> to improve SA performance so that you can survive the onslaught, but SA
> does mean that your server has to do something with each email it scans.
>
> A 'quick fix' would actually be to turn SA off. The (spam) messages will
> all go through, but it should mean less load on your system.
>
> Look through the spam sent in those bursts and see if there is any way you
> can identify them *quickly*, preferably by IP addresses. Then block them
> so your server doesn't have to deal with them.
>
> Daniel T. Staal
Daniel I have tried that but apparently they are coming from everywhere all
at once.. I did find one that was really bad and blocked it with IPtables..
but that one continues to show up in my log watch where I would think it
would go away with the entry..
client 12.130.132.229 error sending response: host unreachable: 853
Time(s)
and that is a LOW number for this guy.. it some days its up to 2000 I traced
this and it is a an AT&T IP for some kind of business service they offer
>
> You probably have max children set too high. When a big
> bunch of messages come in, they all run, you don't have
> enough memory, and your system starts swapping like crazy.
> That brings everything on your server to a near halt.
> It reduces throughput, which means you get a backlog, which
> means you get stuck in this state because all the children
> stay active hogging RAM and trying to process the backlog.
>
> The solution is to either expand the RAM so the system can
> really handle that many active children at once, or set the
> maximum number of children to something much lower. Try 2
> or 3 even. It seems like more children would mean more work
> getting done, and that's true, but it's only true up to a point,
> and you've passed that point.
>
> - Logan
OK Logan I will investigate the RAM and see if it needs to be up'd and kick
the maxchild back down to 10 in the mean time.. the other thing I did last
week was
Number of minutes between mail server queue runs (default is 60).:
I lowered it to 90 minutes from 4 hours but obviously that didn't help one
bit
> Is the mail legitimate email?
>
> Meaning does the email come from wherever to *valid email addresses* on
> the
> server or do you have a system that will catch everything at the smtp
> level
> and then sort it out later?
>
> If your server catches everything, the smtp gate should probably be
> fortified with greylisting and invalid email address rejection first.
>
> There is not enough other info for me to recommend further...
>
> Thanks and kind regards,
>
> - rh
99% of the 300+ mails today and last week were addressed to valid users but
I'd say 60%+ was truly spam.. today as I manually delivered from Cpanel's
WHM individually, I tailed the maillog and many of them were scored and
trashed.. but with that said there was several very valid mails to very
valid users.. I have the whole machine set to fail for invalid users which
everyone on the cpanel forums say is much more efficient than blackhole
Re: I'm getting killed with spammers
Posted by "Daniel T. Staal" <DS...@usa.net>.
On Mon, October 16, 2006 2:28 pm, Debbie D said:
> this high amount of spam, (BTW scoring at 20-well over 1000) is killing
> the loads and I have screaming clients..
>
> Just this afternoon (again around 12.30) it loaded up again with 312
> mails.. the web based control panel was reacting so slow I would get 3
> new ones for every one I managed to delete or deliver (I could not just
> delete the queue because some were actually valid mails in there) Server
> loads rose to well over 30, I shut exim - but cpanel was so kind to
> automagically restart it every time.. tried a reboot from ssh but that
> just hung.. the tech peeps did it from their end it it worked and brought
> the loads down so I could delete faster than they came in and now we're
> back to normal loads and queue
>
> I did upgrade to SA 3.1.7 last week - Wed night after a long day of
> battling the loads.. and that seemed to go well
>
> suggestions? Offers of help???
At this point, you probably need to find some way to blacklist part of
that load, to keep your server from dealing with it. It may be possible
to improve SA performance so that you can survive the onslaught, but SA
does mean that your server has to do something with each email it scans.
A 'quick fix' would actually be to turn SA off. The (spam) messages will
all go through, but it should mean less load on your system.
Look through the spam sent in those bursts and see if there is any way you
can identify them *quickly*, preferably by IP addresses. Then block them
so your server doesn't have to deal with them.
Daniel T. Staal
---------------------------------------------------------------
This email copyright the author. Unless otherwise noted, you
are expressly allowed to retransmit, quote, or otherwise use
the contents for non-commercial purposes. This copyright will
expire 5 years after the author's death, or in 30 years,
whichever is longer, unless such a period is in excess of
local copyright law.
---------------------------------------------------------------
RE: I'm getting killed with spammers
Posted by R Lists06 <li...@abbacomm.net>.
>
> I need some help here..
>
> Last Mon, Tues & Wed I had severe inflow of spam, always at 12.30p EST,
> Wed
> it didn't stop till almost 5p. The server seems to not be very cooperative
> when the queue grows over 200 or so.
>
> I have max child set to 15 (up from 5) and not sure what else I can offer
> in
> the way of what you need to know to help me, but if you tell me where to
> look I can spout what you need.
>
> The install is out of the box with few if any mods except exim does have
> the
> dictionary attack, I run BFD and APF
>
> I do not believe I have been hacked into.. I DO read the logwatch daily
> and
> do poke around looking for dropped files on a semi regular basis..
>
> this high amount of spam, (BTW scoring at 20-well over 1000) is killing
> the
> loads and I have screaming clients..
>
> Just this afternoon (again around 12.30) it loaded up again with 312
> mails..
> the web based control panel was reacting so slow I would get 3 new ones
> for
> every one I managed to delete or deliver (I could not just delete the
> queue
> because some were actually valid mails in there) Server loads rose to well
> over 30, I shut exim - but cpanel was so kind to automagically restart it
> every time.. tried a reboot from ssh but that just hung.. the tech peeps
> did
> it from their end it it worked and brought the loads down so I could
> delete
> faster than they came in and now we're back to normal loads and queue
>
> I did upgrade to SA 3.1.7 last week - Wed night after a long day of
> battling
> the loads.. and that seemed to go well
>
> suggestions? Offers of help???
>
> thanks
Debbie,
Is the mail legitimate email?
Meaning does the email come from wherever to *valid email addresses* on the
server or do you have a system that will catch everything at the smtp level
and then sort it out later?
If your server catches everything, the smtp gate should probably be
fortified with greylisting and invalid email address rejection first.
There is not enough other info for me to recommend further...
Thanks and kind regards,
- rh
--
Robert - Abba Communications
Computer & Internet Services
(509) 624-7159 - www.abbacomm.net