You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by Jason Haar <Ja...@trimble.co.nz> on 2009/09/08 03:50:25 UTC
how to speed up scans of really large text-only emails?
Hi there
We're having problems with a particular class of email. >400K in size,
text-only. spamd takes 40-80sec to process it, and spamc is set with a
30sec timeout. The long processing time isn't network-related: it's all
those "body" searches that are causing the hang.
Obviously there are several things that could be done. Increase spamc
timeout or reduce the limit of the maxsize handed off to spamc. However,
they are both already non-default for good reasons, so I'd rather not
fiddle with them for such a corner-case.
So are there any other options? Allowing spamd to only scan the first
50KB of text attachments would do the trick. I can't think of a way that
could be misused by spammers? (ie they aren't going to send text-spam
where the first 50KB is "bayes killer" and the final bit is the spam -
potential customers won't scroll past the first couple of screens to
find the spam). Or a spamd-based "max-runtime" setting?
--
Cheers
Jason Haar
Information Security Manager, Trimble Navigation Ltd.
Phone: +64 3 9635 377 Fax: +64 3 9635 417
PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1
Re: how to speed up scans of really large text-only emails?
Posted by John Hardin <jh...@impsec.org>.
On Wed, 9 Sep 2009, Jason Haar wrote:
> On 09/09/2009 04:07 AM, John Hardin wrote:
>>
>> Do you have any stats on how spammy this class of mail is? Is it pure
>> ham that you can detect using other methods, e.g. it's sent from a
>> trusted source?
>>
>
> Actually, it's HAM - not spam.
My point. If those messages are 100% ham, and they have indicators that
are easy for the MTA to detect - e.g., from a known correspondent MTA IP
address - then put some rules in at the MTA glue layer to say "don't pass
these messages to SA at all".
--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
It is criminal to teach a man not to defend himself when he is the
constant victim of brutal attacks. -- Malcolm X (1964)
-----------------------------------------------------------------------
9 days until the 222nd anniversary of the signing of the U.S. Constitution
Re: how to speed up scans of really large text-only emails?
Posted by Jason Haar <Ja...@trimble.co.nz>.
On 09/09/2009 12:53 PM, Karsten Bräckelmann wrote:
>
> Ah, good point, Mark -- that reminds me of the infamous issue of
> un-bound or nested quantifiers in RE rules. In some pathological cases,
> I've even debugged these to be the culprit of bringing SA down to its
> knees.
>
> Any custom rules? Do you still see the same timing when disabling them
> temporarily? Might be worth a shot.
>
No really. Removing all .cf files and running it again knocked around 4
seconds off - so that isn't the cause by itself.
I also got SA-3.3dev up, but it appears sa-update doesn't support it
yet, and it complained about there being no 3.3 "rules" yet, and so I
copied in the last 3.2.5 rules to give it something to chew on - and it
complained about that, yadda, yadda. After all that it appeared to run
~10% faster - but given the sad state 3.3 said it was in, I don't trust
the result.
I'm afraid I'm not up to that level of testing yet - a bit early
meethinks ;-)
--
Cheers
Jason Haar
Information Security Manager, Trimble Navigation Ltd.
Phone: +64 3 9635 377 Fax: +64 3 9635 417
PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1
Re: how to speed up scans of really large text-only emails?
Posted by Karsten Bräckelmann <gu...@rudersport.de>.
On Wed, 2009-09-09 at 02:21 +0200, Mark Martinec wrote:
> On Tuesday September 8 2009 21:23:42 Jason Haar wrote:
> > Actually, it's HAM - not spam. In the end it's really become clear it
> > shows limitations in perl's parsing power - so either we get gruntier
> > boxes - or increase the timeout. We've gone with the latter.
>
> Some regexps do perform terribly when given a large chunk of text
> with multiple matching opportunities. Some monolithic HTML with
Ah, good point, Mark -- that reminds me of the infamous issue of
un-bound or nested quantifiers in RE rules. In some pathological cases,
I've even debugged these to be the culprit of bringing SA down to its
knees.
Any custom rules? Do you still see the same timing when disabling them
temporarily? Might be worth a shot.
> nested tables is one such example. If you still have the sample message,
> it would be interesting to try it on the current 3.3 code. The main
> difference could come from the fact that the current code splits
> mail text into smaller chunks and does not allow a rule regexp to
> work on an entire mail block. In some corner cases this brings a
> significant speedup, while on most of the rest it makes no difference.
--
char *t="\10pse\0r\0dtu\0.@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4";
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i<l;i++){ i%8? c<<=1:
(c=*++x); c&128 && (s+=h); if (!(h>>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}
Re: how to speed up scans of really large text-only emails?
Posted by Mark Martinec <Ma...@ijs.si>.
On Tuesday September 8 2009 21:23:42 Jason Haar wrote:
> Actually, it's HAM - not spam. In the end it's really become clear it
> shows limitations in perl's parsing power - so either we get gruntier
> boxes - or increase the timeout. We've gone with the latter.
Some regexps do perform terribly when given a large chunk of text
with multiple matching opportunities. Some monolithic HTML with
nested tables is one such example. If you still have the sample message,
it would be interesting to try it on the current 3.3 code. The main
difference could come from the fact that the current code splits
mail text into smaller chunks and does not allow a rule regexp to
work on an entire mail block. In some corner cases this brings a
significant speedup, while on most of the rest it makes no difference.
http://people.apache.org/~jm/mcsnapshot.tgz
Mark
Re: how to speed up scans of really large text-only emails?
Posted by Jason Haar <Ja...@trimble.co.nz>.
On 09/09/2009 04:07 AM, John Hardin wrote:
>
> Do you have any stats on how spammy this class of mail is? Is it pure
> ham that you can detect using other methods, e.g. it's sent from a
> trusted source?
>
Actually, it's HAM - not spam. In the end it's really become clear it
shows limitations in perl's parsing power - so either we get gruntier
boxes - or increase the timeout. We've gone with the latter.
--
Cheers
Jason Haar
Information Security Manager, Trimble Navigation Ltd.
Phone: +64 3 9635 377 Fax: +64 3 9635 417
PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1
Re: how to speed up scans of really large text-only emails?
Posted by John Hardin <jh...@impsec.org>.
On Tue, 8 Sep 2009, Jason Haar wrote:
> We're having problems with a particular class of email. >400K in size,
> text-only.
Do you have any stats on how spammy this class of mail is? Is it pure ham
that you can detect using other methods, e.g. it's sent from a trusted
source?
If so, you may be able to develop some rules to detect these messages and
use those rules at the MTA glue level to tell it to skip SA processing for
those messages.
--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
Homeland Security: Specializing in Tactical Band-aids for Strategic
Problems. -- Eric K. in Bruce Schneier's blog
-----------------------------------------------------------------------
9 days until the 222nd anniversary of the signing of the U.S. Constitution
Re: how to speed up scans of really large text-only emails?
Posted by Jason Haar <Ja...@trimble.co.nz>.
On 09/08/2009 07:54 PM, Matus UHLAR - fantomas wrote:
>
> It would also make spamd more complicated for no good reason. Simply use
> spamc -t 120 or 180, I think up to 240 is safe at SMTP level unless you are
> using other time-consuming test (data phase should end in 5 minutes
> otherwise client may assumt the connection hanged and close it).
>
>
As spamc normally runs in 1-5 sec, 30sec was considered by us to be a
sensible timeout. However, a longer timeout of 120s should totally get
around this sort of problem without really altering anything else. The
end of DATA phase was of concern to me - but I think the RFCs states the
client has to wait around for either 10 or 20 minutes - either way it'd
be fine.
So yup - will do! :-)
Thanks
--
Cheers
Jason Haar
Information Security Manager, Trimble Navigation Ltd.
Phone: +64 3 9635 377 Fax: +64 3 9635 417
PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1
Re: how to speed up scans of really large text-only emails?
Posted by Matus UHLAR - fantomas <uh...@fantomas.sk>.
> On 09/08/2009 01:50 PM, Jason Haar wrote:
> > We're having problems with a particular class of email. >400K in size,
> > text-only. spamd takes 40-80sec to process it, and spamc is set with a
> > 30sec timeout. The long processing time isn't network-related: it's
> > all those "body" searches that are causing the hang.
On 08.09.09 16:27, Jason Haar wrote:
> Just to follow myself up, it looks like "spamc -t 30" means if spamd
> doesn't return in 30sec, spamc will simply output the email to stdout
> and exit. spamd carries on processing the email and I can see the final
> spamd syslog report when it finishes in >30sec. Unfortunately we
> re-invoke spamc and it just goes through the same problem again :-( It's
> a pity spamd can't keep a small cache of checksum'ed previous messages
> and their scores, so that if it sees the same message again within (say)
> 10-30 min, it just throws up the cached score?
That could cause other troubles. Note that at second scan of the same
message the result can be much different mostlu because of network checks.
It would also make spamd more complicated for no good reason. Simply use
spamc -t 120 or 180, I think up to 240 is safe at SMTP level unless you are
using other time-consuming test (data phase should end in 5 minutes
otherwise client may assumt the connection hanged and close it).
--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Silvester Stallone: Father of the RISC concept.
Re: how to speed up scans of really large text-only emails?
Posted by Jason Haar <Ja...@trimble.co.nz>.
Just to follow myself up, it looks like "spamc -t 30" means if spamd
doesn't return in 30sec, spamc will simply output the email to stdout
and exit. spamd carries on processing the email and I can see the final
spamd syslog report when it finishes in >30sec. Unfortunately we
re-invoke spamc and it just goes through the same problem again :-( It's
a pity spamd can't keep a small cache of checksum'ed previous messages
and their scores, so that if it sees the same message again within (say)
10-30 min, it just throws up the cached score?
Jason
On 09/08/2009 01:50 PM, Jason Haar wrote:
> Hi there
>
> We're having problems with a particular class of email. >400K in size,
> text-only. spamd takes 40-80sec to process it, and spamc is set with a
> 30sec timeout. The long processing time isn't network-related: it's
> all those "body" searches that are causing the hang.
>
> Obviously there are several things that could be done. Increase spamc
> timeout or reduce the limit of the maxsize handed off to spamc.
> However, they are both already non-default for good reasons, so I'd
> rather not fiddle with them for such a corner-case.
>
> So are there any other options? Allowing spamd to only scan the first
> 50KB of text attachments would do the trick. I can't think of a way
> that could be misused by spammers? (ie they aren't going to send
> text-spam where the first 50KB is "bayes killer" and the final bit is
> the spam - potential customers won't scroll past the first couple of
> screens to find the spam). Or a spamd-based "max-runtime" setting?
> --
> Cheers
>
> Jason Haar
> Information Security Manager, Trimble Navigation Ltd.
> Phone: +64 3 9635 377 Fax: +64 3 9635 417
> PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1
>
--
Cheers
Jason Haar
Information Security Manager, Trimble Navigation Ltd.
Phone: +64 3 9635 377 Fax: +64 3 9635 417
PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1
Re: how to speed up scans of really large text-only emails?
Posted by Karsten Bräckelmann <gu...@rudersport.de>.
On Tue, 2009-09-08 at 13:50 +1200, Jason Haar wrote:
> [...] Allowing spamd to only scan the first 50KB of text attachments
> would do the trick. I can't think of a way that could be misused by
> spammers? (ie they aren't going to send text-spam where the first 50KB
> is "bayes killer" and the final bit is the spam - potential customers
> won't scroll past the first couple of screens to find the spam).
I can.
It is indeed trivial to construct large messages, where the actual
payload is way down the end -- and the user will never see the bulk
above.
--
char *t="\10pse\0r\0dtu\0.@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4";
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i<l;i++){ i%8? c<<=1:
(c=*++x); c&128 && (s+=h); if (!(h>>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}