You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by Alex <my...@gmail.com> on 2014/05/11 02:19:08 UTC
Help with short bodys with URLs
Hi,
I'm having trouble catching spam with short body's, typically just a URL,
sometimes with also some junk text. I've been using a rule developed with
KAM some time ago, but not sure if something's changed, or the rule never
worked as I expected, but it's having problems, and I hoped someone could
help.
body __RB_GT_200 /^.{201}/s
meta __BODY_LE_200 (__RB_LE_200 == 1) && !__RB_GT_200
meta __RB_LE_200 !__RB_GT_200 # less or equal IFF not greater
mimeheader __MIME_IMAGE Content-Type =~ /^image\/./
mimeheader __MIME_ATTACH Content-Disposition =~ /^attachment/
meta LOC_SHORT ((__BODY_LE_200 && __HAS_HTTP_URI) &&
(!(__MIME_IMAGE || __MIME_ATTACH )))
describe LOC_SHORT Has URI and short body
score LOC_SHORT 1.1
Here's a pastebin example, which clearly isn't a short body. It's catching
a ton of false-positives. Is it possible the rule is not properly
distinguishing the "multi-part message in MIME format" short body with the
actual message? There are other cases where there isn't the "multi-part
MIME" section and it's still being tagged.
http://pastebin.com/JnCHUtqV
Thanks,
Alex
Re: Help with short bodys with URLs
Posted by Alex <my...@gmail.com>.
Hi,
On Mon, May 12, 2014 at 7:08 PM, Karsten Bräckelmann <guenther@rudersport.de
> wrote:
> On Mon, 2014-05-12 at 13:46 -0400, Alex wrote:
> > On Sun, May 11, 2014 at 9:32 PM, Karsten Bräckelmann <
> guenther@rudersport.de> wrote:
>
> > > This is supposed to be a rawbody rule. I know, because I've discussed
> > > and partly developed the rule(set) in question with you before, back in
> > > Oct 2013. And the RB prefix is a hint as well. ;)
> > >
> > > http://markmail.org/message/ebrm6snglxipj6wx
> >
> > Oh, I remember this thread very well. I referenced your helping me
> > with it in the beginning of this post.
>
> You mentioned KAM. ;)
>
Heh, my apologies.
> > It wasn't a case of not understanding the difference between body and
> > rawbody, at least. I plan to experiment further with the body version
> > you've just created, and see if there's usefulness with that in other
> > cases.
>
> While potentially useful in other cases, it is required to make your
> rule apply as intended to the sample provided.
>
You mean 'it' is rawbody here, right? I've changed it, and it's much
better. I do see other cases where body would be more effective, so I'd
like to work on that as a separate rule too.
> The actual text sure is less than 200 chars, but with the amount of HTML
> markup, the rawbody payload doesn't count as short and easily exceeds
> the 200 char threshold. With the __RB_GT_200 sub-rule fixed to a rawbody
> rule, the overall rule LOC_SHORT will not match the sample.
>
Yes, I think you're just summarizing what we've already discussed here, and
I really appreciate your help.
Thanks,
Alex
>
>
> --
> char *t="\10pse\0r\0dtu\0.@ghno
> \x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4";
> main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i<l;i++){ i%8?
> c<<=1:
> (c=*++x); c&128 && (s+=h); if (!(h>>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0;
> }}}
>
>
Re: Help with short bodys with URLs
Posted by Karsten Bräckelmann <gu...@rudersport.de>.
On Mon, 2014-05-12 at 13:46 -0400, Alex wrote:
> On Sun, May 11, 2014 at 9:32 PM, Karsten Bräckelmann <gu...@rudersport.de> wrote:
> > This is supposed to be a rawbody rule. I know, because I've discussed
> > and partly developed the rule(set) in question with you before, back in
> > Oct 2013. And the RB prefix is a hint as well. ;)
> >
> > http://markmail.org/message/ebrm6snglxipj6wx
>
> Oh, I remember this thread very well. I referenced your helping me
> with it in the beginning of this post.
You mentioned KAM. ;)
> I don't think it was broken all this time, as I know I tested it quite
> extensively, but I don't know how it became a body instead of rawbody
> rule. Since I thought for sure it was once working, I didn't even
> think it was something I did.
>
>
> It wasn't a case of not understanding the difference between body and
> rawbody, at least. I plan to experiment further with the body version
> you've just created, and see if there's usefulness with that in other
> cases.
While potentially useful in other cases, it is required to make your
rule apply as intended to the sample provided.
The actual text sure is less than 200 chars, but with the amount of HTML
markup, the rawbody payload doesn't count as short and easily exceeds
the 200 char threshold. With the __RB_GT_200 sub-rule fixed to a rawbody
rule, the overall rule LOC_SHORT will not match the sample.
--
char *t="\10pse\0r\0dtu\0.@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4";
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i<l;i++){ i%8? c<<=1:
(c=*++x); c&128 && (s+=h); if (!(h>>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}
Re: Help with short bodys with URLs
Posted by Alex <my...@gmail.com>.
Hi,
On Sun, May 11, 2014 at 9:32 PM, Karsten Bräckelmann <guenther@rudersport.de
> wrote:
> On Sat, 2014-05-10 at 20:19 -0400, Alex wrote:
> > [...] not sure if something's changed, or the rule never worked as I
> > expected, but it's having problems, and I hoped someone could help.
>
> Something changed indeed -- you broke the __RB_GT_200 sub-rule.
>
> > body __RB_GT_200 /^.{201}/s
>
> This is supposed to be a rawbody rule. I know, because I've discussed
> and partly developed the rule(set) in question with you before, back in
> Oct 2013. And the RB prefix is a hint as well. ;)
>
> http://markmail.org/message/ebrm6snglxipj6wx
>
Oh, I remember this thread very well. I referenced your helping me with it
in the beginning of this post. I don't think it was broken all this time,
as I know I tested it quite extensively, but I don't know how it became a
body instead of rawbody rule. Since I thought for sure it was once working,
I didn't even think it was something I did.
It wasn't a case of not understanding the difference between body and
rawbody, at least. I plan to experiment further with the body version
you've just created, and see if there's usefulness with that in other cases.
Thanks again,
Alex
> --
> char *t="\10pse\0r\0dtu\0.@ghno
> \x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4";
> main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i<l;i++){ i%8?
> c<<=1:
> (c=*++x); c&128 && (s+=h); if (!(h>>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0;
> }}}
>
>
Re: Help with short bodys with URLs
Posted by Karsten Bräckelmann <gu...@rudersport.de>.
On Sat, 2014-05-10 at 20:19 -0400, Alex wrote:
> [...] not sure if something's changed, or the rule never worked as I
> expected, but it's having problems, and I hoped someone could help.
Something changed indeed -- you broke the __RB_GT_200 sub-rule.
> body __RB_GT_200 /^.{201}/s
This is supposed to be a rawbody rule. I know, because I've discussed
and partly developed the rule(set) in question with you before, back in
Oct 2013. And the RB prefix is a hint as well. ;)
http://markmail.org/message/ebrm6snglxipj6wx
Body rules are applied against the rendered, normalized textual parts of
a message, one paragraph (delimited by double newline) at a time.
Rawbody rules are applied against the raw (merely decoded) textual parts
(split up into 1-2 kB chunks).
Thus, the above RE used in a rawbody rule translates into the desired
"contains more than 200 chars", whereas used in a body rule means "at
least one paragraph of 200 chars". The difference in meaning can be seen
in your sample: The textual parts (regardless of with or without HTML
markup) are clearly more than 200 chars. However, with the amount of
double newlines (paragraphs!), no paragraph exceeds the limit.
So that's what broke the rule(set) and causes FPs. Any chance for a fix?
If you really want to count chars in the body, rendered and normalized
without HTML markup, the simple RE-based solution of counting does not
work. Instead, you need a more complex variant of counting:
body __BODY_COUNT_CHAR /./
tflags __BODY_COUNT_CHAR multiple maxhits=201 # maxhits is SA 3.4 only
meta __BODY_LE_200 __BODY_COUNT_CHAR <= 200
Beware, that is just briefly tested and modified afterward. Use with
care and test before going into production.
The __BODY_LE_200 sub-rule is meant as a replacement for its rawbody
counterpart __RB_LE_200. The body sub-rule counts chars in (all) textual
parts, rendered and normalized without HTML markup. The meta rule is
used to get a single threshold rule.
I did not test how much counting chars like that, abusing tflags
multiple for a single char level, might affect runtime. This holds in
particular without the maxhits argument, available since SA 3.4.
> meta __BODY_LE_200 (__RB_LE_200 == 1) && !__RB_GT_200
> meta __RB_LE_200 !__RB_GT_200 # less or equal IFF not greater
The simple rule __RB_LE_200 supersedes the __BODY_LE_200 rule. As I
pointed out before, please drop the latter and its broken logic.
Generally, I suggest to re-read the thread of Oct 2013, since I
explained quite a few issues contributing to this recurring question.
> meta LOC_SHORT ((__BODY_LE_200 && __HAS_HTTP_URI) && (!(__MIME_IMAGE || __MIME_ATTACH )))
Also use __RB_LE_200 here.
--
char *t="\10pse\0r\0dtu\0.@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4";
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i<l;i++){ i%8? c<<=1:
(c=*++x); c&128 && (s+=h); if (!(h>>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}