You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by "Lawrence @ Rogers" <la...@nl.rogers.com> on 2010/11/04 19:25:09 UTC
new headers rule
Hi,
I've noticed a bunch of spams coming in recently that have no To: and
Subject: and have cobbled together the following rule to combat them.
Any feedback would be appreciated.
# Message has empty To: and Subject: headers
# Likely spam
header __LW_EMPTY_SUBJECT Subject =~ /[[:space:]]$/
meta LW_EMPTY_SUBJECT_TO (__LW_EMPTY_SUBJECT && MISSING_HEADERS)
describe LW_EMPTY_SUBJECT_TO Message has empty To and Subject headers
score LW_EMPTY_SUBJECT_TO 2.5
If anyone would like to test this as part of the mass corpus, please
feel free to do so. I am curious to know how it performs.
Regards,
Lawrence Williams
LCWSoft
www.lcwsoft.com
Re: new headers rule
Posted by Randy Ramsdell <rr...@activedg.com>.
Lawrence @ Rogers wrote:
> On 04/11/2010 6:35 PM, Randy Ramsdell wrote:
>> Are the Subject lines blank or missing from the body? And that goes
>> for the "To" also.
> In the spam I am seeing, there are both present and empty.
>
> Example
>
> To:
> Subject:
I ran a email through spamc and it hits missing and empty Subject.
I have not tested the "To"
Re: new headers rule
Posted by "Lawrence @ Rogers" <la...@nl.rogers.com>.
On 04/11/2010 6:35 PM, Randy Ramsdell wrote:
> Are the Subject lines blank or missing from the body? And that goes
> for the "To" also.
In the spam I am seeing, there are both present and empty.
Example
To:
Subject:
Re: new headers rule
Posted by Randy Ramsdell <rr...@activedg.com>.
Lawrence @ Rogers wrote:
> Hi,
>
> I've noticed a bunch of spams coming in recently that have no To: and
> Subject: and have cobbled together the following rule to combat them.
> Any feedback would be appreciated.
>
> # Message has empty To: and Subject: headers
> # Likely spam
> header __LW_EMPTY_SUBJECT Subject =~ /[[:space:]]$/
> meta LW_EMPTY_SUBJECT_TO (__LW_EMPTY_SUBJECT && MISSING_HEADERS)
> describe LW_EMPTY_SUBJECT_TO Message has empty To and Subject headers
> score LW_EMPTY_SUBJECT_TO 2.5
>
> If anyone would like to test this as part of the mass corpus, please
> feel free to do so. I am curious to know how it performs.
>
> Regards,
>
> Lawrence Williams
> LCWSoft
> www.lcwsoft.com
Are the Subject lines blank or missing from the body? And that goes for
the "To" also.
Re: new headers rule
Posted by Randy Ramsdell <rr...@activedg.com>.
Lawrence @ Rogers wrote:
> On 05/11/2010 6:00 PM, Randy Ramsdell wrote:
>> Lawrence @ Rogers wrote:
>>> On 05/11/2010 10:58 AM, Randy Ramsdell wrote:
>>>> X-MB-Message-Source: WebUI
>>> You appear to have records of the same spam influencing your bayes
>>> results (it hits BAYES_99, which is good). What are your Bayes
>>> threshold settings?
>>>
>>> Cheers,
>>> Lawrence
>>
>> I am not sure what you are asking me. Our spam cutoff is around 5.
>> Note that the above example was from a s"subject" modified message
>> that made it through spamassassin. I simply removed the Subject.
>>
> In your SpamAssassin configuration, what you you have the following
> options set to:
>
> bayes_auto_learn_threshold_nonspam
> bayes_auto_learn_threshold_spam
>
> Cheers,
> Lawrence
Default
Oh an when you reply-all, just remove the To: $myemailaddress and change
the Cc: to a To:users@spamassassin.apache.org cuz you are sending 2
messages.
Re: new headers rule
Posted by "Lawrence @ Rogers" <la...@nl.rogers.com>.
On 05/11/2010 6:00 PM, Randy Ramsdell wrote:
> Lawrence @ Rogers wrote:
>> On 05/11/2010 10:58 AM, Randy Ramsdell wrote:
>>> X-MB-Message-Source: WebUI
>> You appear to have records of the same spam influencing your bayes
>> results (it hits BAYES_99, which is good). What are your Bayes
>> threshold settings?
>>
>> Cheers,
>> Lawrence
>
> I am not sure what you are asking me. Our spam cutoff is around 5.
> Note that the above example was from a s"subject" modified message
> that made it through spamassassin. I simply removed the Subject.
>
In your SpamAssassin configuration, what you you have the following
options set to:
bayes_auto_learn_threshold_nonspam
bayes_auto_learn_threshold_spam
Cheers,
Lawrence
Re: new headers rule
Posted by Randy Ramsdell <rr...@activedg.com>.
Lawrence @ Rogers wrote:
> On 05/11/2010 10:58 AM, Randy Ramsdell wrote:
>> X-MB-Message-Source: WebUI
> You appear to have records of the same spam influencing your bayes
> results (it hits BAYES_99, which is good). What are your Bayes threshold
> settings?
>
> Cheers,
> Lawrence
I am not sure what you are asking me. Our spam cutoff is around 5. Note
that the above example was from a s"subject" modified message that made
it through spamassassin. I simply removed the Subject.
Re: new headers rule
Posted by "Lawrence @ Rogers" <la...@nl.rogers.com>.
On 05/11/2010 10:58 AM, Randy Ramsdell wrote:
> X-MB-Message-Source: WebUI
You appear to have records of the same spam influencing your bayes
results (it hits BAYES_99, which is good). What are your Bayes threshold
settings?
Cheers,
Lawrence
Re: new headers rule
Posted by Randy Ramsdell <rr...@activedg.com>.
Lawrence @ Rogers wrote:
> On 04/11/2010 8:11 PM, Karsten Bräckelmann wrote:
>> Moving back on-list, since it doesn't appear to be personally directed
>> at me.
>>
>> On Thu, 2010-11-04 at 19:22 -0230, Lawrence @ Rogers wrote:
>>> On 04/11/2010 7:13 PM, Karsten Bräckelmann wrote:
>>>> No, that requires the Subject to consist of exactly one whitespace.
>>>>
>>>> Read it out load. The ^ beginning of the string, followed by exactly
>>>> one
>>>> whitespace char [2]. Followed by the $ end of the string.
>>> No offense, but I am a C and PHP programmer and Perl's documentation is
>>> lacking, to put it politely. Too much theory and far too few actual real
>>> world examples.
>> This is not about Perl, but Regular Expressions. The much more feature-
>> rich (and widely adopted) Perl flavor, out of all the existing variants.
>> But that's actually irrelevant in this case, cause you would need a very
>> limited sub-set only, pretty much available in any tool sporting REs.
>>
>> Any introduction to REs would do, no need to tend to the Perl docs you
>> don't like. Though it sounds like you didn't even had a look at the docs
>> I pointed you to.
>>
>>
>>> That is exactly what I am trying to match, and according to my tests, it
>>> works as expected. When the To and Subject are empty, all that's there
>>> (before the newline) is one whitespace.
>> Are you referring to the whitespace delimiter between the Header: and
>> its content? It's not part of the content.
>>
>>> What I am looking to check is a situation where both the To: and
>>> Subject: headers contain nothing at all, but are set (I've seen this in
>>> several spam e-mails recently)
>> Now you're confusing me. Do you want to match a single whitespace, or a
>> completely empty header?
>>
>>
>>> If there's a better way of doing this, I would appreciate you providing
>>> an example.
>> Well, better way... One that does what you just described.
>>
>> Assuming you want to match "headers containing nothing at all", as per
>> your previous paragraph. That would be nothing between the beginning and
>> end.
>> header __FOO Foo =~ /^$/
>>
>> Or, negated, not anything.
>> header __FOO Foo !~ /./
>>
>> Now, since you specifically constrained this, you might want to check
>> for the header's existence. Probably not worth it, though. The following
>> is copied from stock 20_head_tests.cf, and documented in SA Conf.
>> header __HAS_SUBJECT exists:Subject
>>
>>
>> Anyway, in cases like these it's best to provide a *raw* sample, showing
>> the headers in question completely un-munged and exactly as seen by SA.
>> (Otherwise our help often is limited to guessing and an informal
>> description.) This prohibits copy-n-paste from your MUA, which too often
>> changes subtle but important details.
>>
>> One easy way to come to a conclusion whether you want to match
>> whitespace or not, is the following ad-hoc header rule with spamassassin
>> debug. The matching header's contents are shown in double quotes.
>>
>> spamassassin -D --cf="header FOO To =~ /^.*/"< msg 2>&1 | grep FOO
>>
>> And just for reference, 'grep' uses REs...
>>
>>
> Thanks Karsten,
>
> One of these days when I get some free time, I will be sitting down and
> reading up on REs :)
>
> Using your examples, and some hackery, I came up with this. It checks
> for the existence of the To header as well, as SA doesn't seem to have a
> rule for doing this on it's own (a grep -r "exists:To" * on the rules
> pulled in from updates.spamassassin.org produced nothing).
>
> # Message has empty To: and Subject: headers
> # Likely spam
> header __LW_HAS_TO exists:To
> header __LW_EMPTY_TO To =~ /^$/
> header __LW_EMPTY_SUBJECT Subject =~ /^$/
> meta LW_EMPTY_SUBJECT_TO (__HAS_SUBJECT && __LW_HAS_TO &&
> __LW_EMPTY_SUBJECT && __LW_EMPTY_TO)
> describe LW_EMPTY_SUBJECT_TO Message has empty To and Subject headers
> score LW_EMPTY_SUBJECT_TO 2.5
>
> I added this to my custom .cf rules file and ran spamassassin --lint and
> got no complaints. I ran it over a sample spam, and it hit. I took
> another spam where both headers had information in them, and it didn't
> hit. Guess it works as expected :)
>
> Cheers,
> Lawrence
Am I missing something?'
[29480] dbg: check: tests=AWL,BAYES_99,MISSING_SUBJECT
<snip>
Content-Transfer-Encoding: quoted-printable
Subject:
X-MB-Message-Source: WebUI
</snip>
Re: new headers rule
Posted by "Lawrence @ Rogers" <la...@nl.rogers.com>.
On 04/11/2010 8:11 PM, Karsten Bräckelmann wrote:
> Moving back on-list, since it doesn't appear to be personally directed
> at me.
>
> On Thu, 2010-11-04 at 19:22 -0230, Lawrence @ Rogers wrote:
>> On 04/11/2010 7:13 PM, Karsten Bräckelmann wrote:
>>> No, that requires the Subject to consist of exactly one whitespace.
>>>
>>> Read it out load. The ^ beginning of the string, followed by exactly one
>>> whitespace char [2]. Followed by the $ end of the string.
>> No offense, but I am a C and PHP programmer and Perl's documentation is
>> lacking, to put it politely. Too much theory and far too few actual real
>> world examples.
> This is not about Perl, but Regular Expressions. The much more feature-
> rich (and widely adopted) Perl flavor, out of all the existing variants.
> But that's actually irrelevant in this case, cause you would need a very
> limited sub-set only, pretty much available in any tool sporting REs.
>
> Any introduction to REs would do, no need to tend to the Perl docs you
> don't like. Though it sounds like you didn't even had a look at the docs
> I pointed you to.
>
>
>> That is exactly what I am trying to match, and according to my tests, it
>> works as expected. When the To and Subject are empty, all that's there
>> (before the newline) is one whitespace.
> Are you referring to the whitespace delimiter between the Header: and
> its content? It's not part of the content.
>
>> What I am looking to check is a situation where both the To: and
>> Subject: headers contain nothing at all, but are set (I've seen this in
>> several spam e-mails recently)
> Now you're confusing me. Do you want to match a single whitespace, or a
> completely empty header?
>
>
>> If there's a better way of doing this, I would appreciate you providing
>> an example.
> Well, better way... One that does what you just described.
>
> Assuming you want to match "headers containing nothing at all", as per
> your previous paragraph. That would be nothing between the beginning and
> end.
> header __FOO Foo =~ /^$/
>
> Or, negated, not anything.
> header __FOO Foo !~ /./
>
> Now, since you specifically constrained this, you might want to check
> for the header's existence. Probably not worth it, though. The following
> is copied from stock 20_head_tests.cf, and documented in SA Conf.
> header __HAS_SUBJECT exists:Subject
>
>
> Anyway, in cases like these it's best to provide a *raw* sample, showing
> the headers in question completely un-munged and exactly as seen by SA.
> (Otherwise our help often is limited to guessing and an informal
> description.) This prohibits copy-n-paste from your MUA, which too often
> changes subtle but important details.
>
> One easy way to come to a conclusion whether you want to match
> whitespace or not, is the following ad-hoc header rule with spamassassin
> debug. The matching header's contents are shown in double quotes.
>
> spamassassin -D --cf="header FOO To =~ /^.*/"< msg 2>&1 | grep FOO
>
> And just for reference, 'grep' uses REs...
>
>
Thanks Karsten,
One of these days when I get some free time, I will be sitting down and
reading up on REs :)
Using your examples, and some hackery, I came up with this. It checks
for the existence of the To header as well, as SA doesn't seem to have a
rule for doing this on it's own (a grep -r "exists:To" * on the rules
pulled in from updates.spamassassin.org produced nothing).
# Message has empty To: and Subject: headers
# Likely spam
header __LW_HAS_TO exists:To
header __LW_EMPTY_TO To =~ /^$/
header __LW_EMPTY_SUBJECT Subject =~ /^$/
meta LW_EMPTY_SUBJECT_TO (__HAS_SUBJECT && __LW_HAS_TO &&
__LW_EMPTY_SUBJECT && __LW_EMPTY_TO)
describe LW_EMPTY_SUBJECT_TO Message has empty To and Subject headers
score LW_EMPTY_SUBJECT_TO 2.5
I added this to my custom .cf rules file and ran spamassassin --lint and
got no complaints. I ran it over a sample spam, and it hit. I took
another spam where both headers had information in them, and it didn't
hit. Guess it works as expected :)
Cheers,
Lawrence
Re: new headers rule
Posted by Karsten Bräckelmann <gu...@rudersport.de>.
Moving back on-list, since it doesn't appear to be personally directed
at me.
On Thu, 2010-11-04 at 19:22 -0230, Lawrence @ Rogers wrote:
> On 04/11/2010 7:13 PM, Karsten Bräckelmann wrote:
> > No, that requires the Subject to consist of exactly one whitespace.
> >
> > Read it out load. The ^ beginning of the string, followed by exactly one
> > whitespace char [2]. Followed by the $ end of the string.
>
> No offense, but I am a C and PHP programmer and Perl's documentation is
> lacking, to put it politely. Too much theory and far too few actual real
> world examples.
This is not about Perl, but Regular Expressions. The much more feature-
rich (and widely adopted) Perl flavor, out of all the existing variants.
But that's actually irrelevant in this case, cause you would need a very
limited sub-set only, pretty much available in any tool sporting REs.
Any introduction to REs would do, no need to tend to the Perl docs you
don't like. Though it sounds like you didn't even had a look at the docs
I pointed you to.
> That is exactly what I am trying to match, and according to my tests, it
> works as expected. When the To and Subject are empty, all that's there
> (before the newline) is one whitespace.
Are you referring to the whitespace delimiter between the Header: and
its content? It's not part of the content.
> What I am looking to check is a situation where both the To: and
> Subject: headers contain nothing at all, but are set (I've seen this in
> several spam e-mails recently)
Now you're confusing me. Do you want to match a single whitespace, or a
completely empty header?
> If there's a better way of doing this, I would appreciate you providing
> an example.
Well, better way... One that does what you just described.
Assuming you want to match "headers containing nothing at all", as per
your previous paragraph. That would be nothing between the beginning and
end.
header __FOO Foo =~ /^$/
Or, negated, not anything.
header __FOO Foo !~ /./
Now, since you specifically constrained this, you might want to check
for the header's existence. Probably not worth it, though. The following
is copied from stock 20_head_tests.cf, and documented in SA Conf.
header __HAS_SUBJECT exists:Subject
Anyway, in cases like these it's best to provide a *raw* sample, showing
the headers in question completely un-munged and exactly as seen by SA.
(Otherwise our help often is limited to guessing and an informal
description.) This prohibits copy-n-paste from your MUA, which too often
changes subtle but important details.
One easy way to come to a conclusion whether you want to match
whitespace or not, is the following ad-hoc header rule with spamassassin
debug. The matching header's contents are shown in double quotes.
spamassassin -D --cf="header FOO To =~ /^.*/" < msg 2>&1 | grep FOO
And just for reference, 'grep' uses REs...
--
char *t="\10pse\0r\0dtu\0.@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4";
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i<l;i++){ i%8? c<<=1:
(c=*++x); c&128 && (s+=h); if (!(h>>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}
Re: new headers rule
Posted by Karsten Bräckelmann <gu...@rudersport.de>.
On Thu, 2010-11-04 at 18:08 -0230, Lawrence @ Rogers wrote:
> On 04/11/2010 5:56 PM, Karsten Bräckelmann wrote:
> > > header __LW_EMPTY_SUBJECT Subject =~ /[[:space:]]$/
> > That rule does *not* do what you intend. It matches, if the last char of
> > the Subject happens to be a whitespace.
> >
> > By definition, that header is not empty. Moreover, it is not equivalent
> > to a header that has no printable chars, which seems to be what you
> > actually tried the RE to match.
>
> How's about this then
No offense meant, Lawrence, but I strongly suggest you should at least
read up on some (Perl) RE introduction before writing such rules.
perlre [1] is *the* reference. You'll find suitable introduction docs in
its Description section.
Also, actually test your rules against the samples.
> # Message has empty To: and Subject: headers
> header __LW_EMPTY_SUBJECT Subject =~ /^[[:space:]]$/
No, that requires the Subject to consist of exactly one whitespace.
Read it out load. The ^ beginning of the string, followed by exactly one
whitespace char [2]. Followed by the $ end of the string.
[1] http://perldoc.perl.org/perlre.html
[2] A char [class] is just a char class, an alternation. It still is a
single char in length, unless a quantifier is given.
--
char *t="\10pse\0r\0dtu\0.@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4";
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i<l;i++){ i%8? c<<=1:
(c=*++x); c&128 && (s+=h); if (!(h>>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}
Re: new headers rule
Posted by "Lawrence @ Rogers" <la...@nl.rogers.com>.
On 04/11/2010 5:56 PM, Karsten Bräckelmann wrote:
> On Thu, 2010-11-04 at 15:55 -0230, Lawrence @ Rogers wrote:
>> I've noticed a bunch of spams coming in recently that have no To: and
>> Subject: and have cobbled together the following rule to combat them.
>> Any feedback would be appreciated.
> Just as a side note, there is a difference between a missing and an
> empty header.
>
>> # Message has empty To: and Subject: headers
>> # Likely spam
>> header __LW_EMPTY_SUBJECT Subject =~ /[[:space:]]$/
> That rule does *not* do what you intend. It matches, if the last char of
> the Subject happens to be a whitespace.
>
> By definition, that header is not empty. Moreover, it is not equivalent
> to a header that has no printable chars, which seems to be what you
> actually tried the RE to match.
>
>
How's about this then
# Message has empty To: and Subject: headers
# Likely spam
header __LW_EMPTY_TO To =~ /^[[:space:]]$/
header __LW_EMPTY_SUBJECT Subject =~ /^[[:space:]]$/
meta LW_EMPTY_SUBJECT_TO (__LW_EMPTY_SUBJECT && __LW_EMPTY_TO)
describe LW_EMPTY_SUBJECT_TO Message has empty To and Subject headers
score LW_EMPTY_SUBJECT_TO 2.5
Re: new headers rule
Posted by Karsten Bräckelmann <gu...@rudersport.de>.
On Thu, 2010-11-04 at 15:55 -0230, Lawrence @ Rogers wrote:
> I've noticed a bunch of spams coming in recently that have no To: and
> Subject: and have cobbled together the following rule to combat them.
> Any feedback would be appreciated.
Just as a side note, there is a difference between a missing and an
empty header.
> # Message has empty To: and Subject: headers
> # Likely spam
> header __LW_EMPTY_SUBJECT Subject =~ /[[:space:]]$/
That rule does *not* do what you intend. It matches, if the last char of
the Subject happens to be a whitespace.
By definition, that header is not empty. Moreover, it is not equivalent
to a header that has no printable chars, which seems to be what you
actually tried the RE to match.
--
char *t="\10pse\0r\0dtu\0.@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4";
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i<l;i++){ i%8? c<<=1:
(c=*++x); c&128 && (s+=h); if (!(h>>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}