You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by Mark London <mr...@psfc.mit.edu> on 2016/06/08 17:10:43 UTC
Email with attachment caused 100% CPU usage.
Hi - We received an email with several large postscript attachments,
and the content type was "text/plain". This caused our spamassassin
server to use up 100% CPU, parsing the attachments as text. I
temporarily disabled spam scanning to allow the message to go through.
How can I prevent this in the future? I know about the time limit
feature, but this doesn't prevent the server from running 100% of the
time, before the time limit is reached. Any suggestions? Thanks. - Mark
Content-Transfer-Encoding: base64
Content-Type: text/plain;
name=OTBW_3D_256_ngtot100_de03_coll_dissip_1248.ps
Content-Disposition: attachment;
filename=OTBW_3D_256_ngtot100_de03_coll_dissip_1248.ps
Re: Email with attachment caused 100% CPU usage.
Posted by Bill Cole <sa...@billmail.scconsult.com>.
On 10 Jun 2016, at 3:40, Merijn van den Kroonenberg wrote:
>> On 9 Jun 2016, at 0:53, Henrik K wrote:
>>
>>> Garbage text/plain is known problem..
>>
>> text/html too. From GMail.
>>
>> Last week I had a *perfectly legitimate* message with a 151KB logical
>> single line of HTML (QP encoded of course) freeze up a server scaled
>> for
>> 10k users.
>> [snip]
>
> Are there publically available some mails which might cause these kind
> of
> problems? It would be interesting for testing set-ups.
Not that I'm aware of. This particular case involved a customer
organization who I wouldn't ever consider asking to share their mail for
any reason, as they specialize in a field where the privacy issues would
be insurmountable.
One could mock up such messages by hand rather easily: find & copy or
create (maybe Word would be a choice tool for this...) a bloated HTML
page, replace all the line breaks with spaces in a proper text editor.
Attach that to an email message or just make it the sole body part of a
pure text/html message.
It MAY be that this was a case where someone wrote something big-ish
and highly-formatted inside GMail or Google Docs and mailed it so the
structure is GMail's fault. It could also be that some MUA or document
generation tool constructed the mail and it came through GMail via SMTP
submission. I do not know which it is but I would bet on the latter,
given Google's core expertise in working with HTML. My reflexes in these
sorts of cases where I have to handle customer non-spam mail are to not
look too closely at anything non-critical to solving the problem at hand
and swiftly forget anything unimportant I happen to see (which gets
easier every year...)
Re: Email with attachment caused 100% CPU usage.
Posted by Merijn van den Kroonenberg <me...@web2all.nl>.
> On 9 Jun 2016, at 0:53, Henrik K wrote:
>
>> Garbage text/plain is known problem..
>
> text/html too. From GMail.
>
> Last week I had a *perfectly legitimate* message with a 151KB logical
> single line of HTML (QP encoded of course) freeze up a server scaled for
> 10k users.
> [snip]
Are there publically available some mails which might cause these kind of
problems? It would be interesting for testing set-ups.
Re: Email with attachment caused 100% CPU usage.
Posted by Martin Gregorie <ma...@gregorie.org>.
On Fri, 2016-06-10 at 18:26 -0400, Bill Cole wrote:
> It will be interesting to see the stats on scantimes this week to see
> if�my tightening up on sloppy rules has an impact. I expect it will,
> since�I now have a concrete theory to explain that long tail out to 2
> minutes,�which before now I've ignored as pure noise.
>
Thanks for the heads-up.�
It prodded me into scanning my local rule set for unguarded '.*' in
local rules. I found a few - more or less what I expected and virtually
all in my older rules - and limited them all to runs of up to 32
characters before running my spam corpus against them. This showed only
two instances of a rule now failing to fire. Finding the rule and
changing it to allow up to 64 characters to match in the 'don't care'
parts of the regex fixed that.�
FWIW this is a rule that looks for two consecutive URLs in body text. I
knew that some of these can be quite long, but since, in all the spam
I've inspected they were used as a more or less neat final line in the
message, they probably don't exceed 32 chars by much. So yes, while I
know that 64 chars is probably overkill, its not so much overkill that
its worth further fiddling.
���
Martin
Re: Email with attachment caused 100% CPU usage.
Posted by Bill Cole <sa...@billmail.scconsult.com>.
On 10 Jun 2016, at 6:17, Merijn van den Kroonenberg wrote:
[...]
> From the manual:
>
> This is a best-effort advisory setting, processing will not be
> abruptly
> aborted at an arbitrary point in processing when the time limit is
> exceeded, but only on reaching one of locations in the program flow
> equipped with a time test. Currently equipped with the test are the
> main
> checking loop, asynchronous DNS lookups, plugins which are calling
> external programs. Rule evaluation is guarded by starting a timer
> (alarm)
> on each set of compiled rules.
The last line is critical. The alarm isn't even on individual rules but
rather on precompiled sets of rules. I already knew it was soft but that
particular detail had not stuck in any of the many times I've read that
man page. Thanks for quoting it.
> What does this mean, can still a single operation take more than this
> time_limit? But I guess the timer on the rules means the rules at
> least
> cannot take more than time_limit, right?
Nope: time_limit is in seconds and I had it set to 270. The default is
300 which is 1/2 the canonical SMTP EOD timeout but matches the
canonical "server timeout" (how long a server should wait for a client
command. See RFC 5321) and I'm sure there's misunderstanding on that
distinction. Since under peak loads the plumbing of that particular
system can add whole seconds and some clients can be a bit impatient, I
gave it what I thought was very generous additional headroom although it
really didn't need it. The actual normal spamd scantime distribution
there roughly 2/3 under 1 second, 90% under 2 seconds, 99% under 10
seconds, 99.9% under a minute. There's a long thin tail out to ~2
minutes, but until last week I'd never had anything actually hit the
timeout that I can recall, and the 1st bad rule was hardly fresh. I
think it is >8 years old and the others I found had not caused trouble
in their multiple (3?) years of existence.
I figured out which rule was the proximal cause by running the message
through the spamassassin script with rule debugging on so I could narrow
down the bad rule based on what matched right before
TIME_LIMIT_EXCEEDED. I didn't dtrace the process to nail it down, but my
hypothesis is that ultimately Perl is calling its low-level internal
equivalent of regexec() which, like POSIX regexec(), has no timeout
facility: it runs until it matches or exhausts the input of starting
points. At the Perl level, I've never encountered any way to limit the
time '=~' takes to operate, but I'm no Larry Wall so maybe there's some
arcane way to do that. It seems clear that if Perl has such a feature to
break out of an operator routine that is taking too much clock time, it
has not been used in SpamAssassin. I'd bet on there NOT being such a
feature and further, that the process doing the match might not even die
immediately with a SIGKILL while inside that call.
What most annoys me about this is that the potential for blowing up
systems with REs is the first thing one learns about them in a formal
setting (rather than just by reading man pages above the BUGS section.)
Back when I first got that warning the emphasis was on an ability of REs
to compile to disastrous scale but back then that meant a few megabytes,
and who cares about that today. However, I also got the warning decades
ago that '.*' could cause a RE to take a long time if you didn't take
care to limit your input size and write the RE to rule out most starting
points fast, but again absolute sizes matter and until last week I'd not
envisioned "optimizing" HTML by removing all formally unnecessary
whitespace including line breaks. This is obviously somewhat rare, but
it's apparently A Thing HTML Parsers Like and this was a big hunk of
HTML, so I guess optimizing parsing was important...
It will be interesting to see the stats on scantimes this week to see if
my tightening up on sloppy rules has an impact. I expect it will, since
I now have a concrete theory to explain that long tail out to 2 minutes,
which before now I've ignored as pure noise.
Re: Email with attachment caused 100% CPU usage.
Posted by Sidney Markowitz <si...@sidney.com>.
Merijn van den Kroonenberg wrote on 10/06/16 10:17 PM:
> What does this mean, can still a single operation take more than this
> time_limit?
There is a fundamental difficulty in perl that the built-in timer alarm
facility cannot always interrupt the built-in regular expression matching
facility. That means there is not a good way of protecting against a rule
pattern that can take a very long time to process certain input being given
that input. There is some effort to do that with the time_limit setting and
spawning of individual child processes in spamd that can be individually
killed if one of them gets hung up, but that only goes so far.
More technical discussion of how to better deal with this probably would be
better in the dev mailing list.
There have been a number of bug reports having to do with some "toxic" email
bogging down SpamAssassin which were tracked down to some use of, for example,
.* in a rule pattern. It can be tricky to fix a pattern to keep this from
happening. When you write your own rules for local use you are more likely to
run into this problem because you probably have less experience in writing
patterns to avoid it and don't run your rules through the many test cases of
our mass check system.
Sidney
Re: Email with attachment caused 100% CPU usage.
Posted by Merijn van den Kroonenberg <me...@web2all.nl>.
>
>
> Am 10.06.2016 um 04:49 schrieb Bill Cole:
>> On 9 Jun 2016, at 0:53, Henrik K wrote:
>>
>>> Garbage text/plain is known problem..
>>
>> text/html too. From GMail.
>>
>> Last week I had a *perfectly legitimate* message with a 151KB logical
>> single line of HTML (QP encoded of course) freeze up a server scaled for
>> 10k users. It did it slowly over a day, because it took a spamd child
>> ~20 minutes to scan
>
> why in the world do you allow a single spamd child to scan 20 minutes
> for a message and what happens if all your childs have such mails to
> proceed - that's hardly scaled for 10k users on rainy days
>
> time_limit 20
>
> read the manual, it works like shortcircuit meaning all other rules
> already finished (RBL/URIBL in any case) will give their score and so
> you don't open the machine widely while stop easy DOS attacks with
> handcrafted mails
From the manual:
This is a best-effort advisory setting, processing will not be abruptly
aborted at an arbitrary point in processing when the time limit is
exceeded, but only on reaching one of locations in the program flow
equipped with a time test. Currently equipped with the test are the main
checking loop, asynchronous DNS lookups, plugins which are calling
external programs. Rule evaluation is guarded by starting a timer (alarm)
on each set of compiled rules.
What does this mean, can still a single operation take more than this
time_limit? But I guess the timer on the rules means the rules at least
cannot take more than time_limit, right?
>
> if the server is not a feature-phone when you don't have a result within
> 20 seconds you hardly get one 5 minutes later (besides that in a proper
> setup rejecting based on teh result the client don't wait that long and
> comes again and again)
>
>
Re: Email with attachment caused 100% CPU usage.
Posted by Reindl Harald <h....@thelounge.net>.
Am 10.06.2016 um 04:49 schrieb Bill Cole:
> On 9 Jun 2016, at 0:53, Henrik K wrote:
>
>> Garbage text/plain is known problem..
>
> text/html too. From GMail.
>
> Last week I had a *perfectly legitimate* message with a 151KB logical
> single line of HTML (QP encoded of course) freeze up a server scaled for
> 10k users. It did it slowly over a day, because it took a spamd child
> ~20 minutes to scan
why in the world do you allow a single spamd child to scan 20 minutes
for a message and what happens if all your childs have such mails to
proceed - that's hardly scaled for 10k users on rainy days
time_limit 20
read the manual, it works like shortcircuit meaning all other rules
already finished (RBL/URIBL in any case) will give their score and so
you don't open the machine widely while stop easy DOS attacks with
handcrafted mails
if the server is not a feature-phone when you don't have a result within
20 seconds you hardly get one 5 minutes later (besides that in a proper
setup rejecting based on teh result the client don't wait that long and
comes again and again)
Re: Email with attachment caused 100% CPU usage.
Posted by Bill Cole <sa...@billmail.scconsult.com>.
On 9 Jun 2016, at 0:53, Henrik K wrote:
> Garbage text/plain is known problem..
text/html too. From GMail.
Last week I had a *perfectly legitimate* message with a 151KB logical
single line of HTML (QP encoded of course) freeze up a server scaled for
10k users. It did it slowly over a day, because it took a spamd child
~20 minutes to scan and the way the system is plumbed and how GMail
times out at EOD, the message was re-queued for scanning and GMail kept
coming back to retry it, then the sender and recipient figured out it
hadn't arrive so it was re-sent ands o eventually all the spamd children
were scanning copies of the message plus were spares in the queue, and
it all got ugly. SA could not use its own timeout because the 20 minutes
was spent checking it against a single (local, rawbody) rule (matching
idiosyncratic spammer HTML) with an incautious '.*' such that the regex
matching code would be of n! (or maybe n^n ?) order where n=[number of
'div' open tags in the line]. I'm a bit surprised it ever finished...
However, it did, and SA timed out right after that rule failed to match
after ~20 minutes (longer, once the busy spamd children outnumbered the
cores...) As it turned out, there were actually 3 rules (out of over a
hundred local rules) that broke when fed that message. I found 2 more
that could hypothetically have trouble with plain text in the
crlf-per-paragraph mis-format used by older versions of Outlook and
newer versions of Apple Mail. I ended up replacing every occurrence of
'.*' and '.+' many uses of '(pattern)*' in the local rules. It is *Perl*
after all, not classic grep...
In short, the lesson is: IF you create local rules that match against
idiosyncratic but loose styles of HTML, spacing, or punctuation, be
EXTREMELY cautious with '*' and especially with the classic lazy '.*'
combination. It's especially a risk with 'rawbody' and multiline 'full'
rules, but given the junk tools people use to generate "valid" email,
rules of any of the body types that are too fuzzy could run into problem
of lines of unusual length.
Re: Re: Email with attachment caused 100% CPU usage.
Posted by Henrik K <he...@hege.li>.
On Thu, Jun 09, 2016 at 12:16:11AM -0400, Mark London wrote:
> On 6/8/2016 1:20 PM, John Hardin wrote:
> >On Wed, 8 Jun 2016, Mark London wrote:
> >>Hi - We received an email with several large postscript
> >>attachments, and the content type was "text/plain". This
> >>caused our spamassassin server to use up 100% CPU, parsing the
> >>attachments as text. I temporarily disabled spam scanning to
> >>allow the message to go through. How can I prevent this in the
> >>future? I know about the time limit feature, but this doesn't
> >>prevent the server from running 100% of the time, before the
> >>time limit is reached. Any suggestions? Thanks. - Mark
> >>
> >>Content-Transfer-Encoding: base64
> >>Content-Type: text/plain;
> >>name=OTBW_3D_256_ngtot100_de03_coll_dissip_1248.ps
> >>Content-Disposition: attachment;
> >> filename=OTBW_3D_256_ngtot100_de03_coll_dissip_1248.ps
> >Do you have something that could catch text/plain + *.ps before SA
> >get handed the message (e.g. a regex milter or other test)?
>
> I'm using MIMEDefang. I haven't looked to see what I could do with
> that. I've been running spamassassin for more years than I
> remember, and this is the first time I've encountered this
> situation.
>
> Someone else asked about my file size limit. I know that for a 512K
> postscript file as text/plain, that it takes up 100% of the CPU of
> one process, for about 1 minute. But I have a much larger file size
> limit, which I've increased over the years, in response to spam that
> we've received here.
>
> I believe the problem has always been there, but it's rarely been
> abused like this. I can't think of a proper solution. I guess
> maybe I'll just hope it never happens again. :) - Mark
Garbage text/plain is known problem.. 3.4.2 should have a workaround if
someone manages to release it.. :-)
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=6582
Re: Re: Email with attachment caused 100% CPU usage.
Posted by Mark London <mr...@psfc.mit.edu>.
On 6/8/2016 1:20 PM, John Hardin wrote:
> On Wed, 8 Jun 2016, Mark London wrote:
>> Hi - We received an email with several large postscript attachments,
>> and the content type was "text/plain". This caused our spamassassin
>> server to use up 100% CPU, parsing the attachments as text. I
>> temporarily disabled spam scanning to allow the message to go
>> through. How can I prevent this in the future? I know about the
>> time limit feature, but this doesn't prevent the server from running
>> 100% of the time, before the time limit is reached. Any suggestions?
>> Thanks. - Mark
>>
>> Content-Transfer-Encoding: base64
>> Content-Type: text/plain;
>> name=OTBW_3D_256_ngtot100_de03_coll_dissip_1248.ps
>> Content-Disposition: attachment;
>> filename=OTBW_3D_256_ngtot100_de03_coll_dissip_1248.ps
> Do you have something that could catch text/plain + *.ps before SA get
> handed the message (e.g. a regex milter or other test)?
I'm using MIMEDefang. I haven't looked to see what I could do with
that. I've been running spamassassin for more years than I remember,
and this is the first time I've encountered this situation.
Someone else asked about my file size limit. I know that for a 512K
postscript file as text/plain, that it takes up 100% of the CPU of one
process, for about 1 minute. But I have a much larger file size limit,
which I've increased over the years, in response to spam that we've
received here.
I believe the problem has always been there, but it's rarely been abused
like this. I can't think of a proper solution. I guess maybe I'll
just hope it never happens again. :) - Mark
Re: Email with attachment caused 100% CPU usage.
Posted by Joe Quinn <jq...@pccc.com>.
On 6/8/2016 1:20 PM, John Hardin wrote:
> On Wed, 8 Jun 2016, Mark London wrote:
>
>> Hi - We received an email with several large postscript attachments,
>> and the content type was "text/plain". This caused our spamassassin
>> server to use up 100% CPU, parsing the attachments as text. I
>> temporarily disabled spam scanning to allow the message to go
>> through. How can I prevent this in the future? I know about the
>> time limit feature, but this doesn't prevent the server from running
>> 100% of the time, before the time limit is reached. Any suggestions?
>> Thanks. - Mark
>>
>> Content-Transfer-Encoding: base64
>> Content-Type: text/plain;
>> name=OTBW_3D_256_ngtot100_de03_coll_dissip_1248.ps
>> Content-Disposition: attachment;
>> filename=OTBW_3D_256_ngtot100_de03_coll_dissip_1248.ps
>
> Ouch. Please file a bug for that.
>
> Do you have something that could catch text/plain + *.ps before SA get
> handed the message (e.g. a regex milter or other test)?
>
Before you file a bug, do you have a size limit set? How large is the
attachment?
Re: Email with attachment caused 100% CPU usage.
Posted by John Hardin <jh...@impsec.org>.
On Wed, 8 Jun 2016, Mark London wrote:
> Hi - We received an email with several large postscript attachments, and the
> content type was "text/plain". This caused our spamassassin server to use
> up 100% CPU, parsing the attachments as text. I temporarily disabled spam
> scanning to allow the message to go through. How can I prevent this in the
> future? I know about the time limit feature, but this doesn't prevent the
> server from running 100% of the time, before the time limit is reached. Any
> suggestions? Thanks. - Mark
>
> Content-Transfer-Encoding: base64
> Content-Type: text/plain;
> name=OTBW_3D_256_ngtot100_de03_coll_dissip_1248.ps
> Content-Disposition: attachment;
> filename=OTBW_3D_256_ngtot100_de03_coll_dissip_1248.ps
Ouch. Please file a bug for that.
Do you have something that could catch text/plain + *.ps before SA get
handed the message (e.g. a regex milter or other test)?
--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
Rights can only ever be individual, which means that you cannot
gain a right by joining a mob, no matter how shiny the issued
badges are, or how many of your neighbors are part of it. -- Marko
-----------------------------------------------------------------------
169 days since the first successful real return to launch site (SpaceX)