Re: FW: Bayes Tokens

[Justin Mason <jm...@jmason.org>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


> In addition to the actual words and phrases in a message and its headers,
> *any* aspect or property of the message -- whether intrinsic or derived --
> can be considered a Bayes token.  POPFile takes this into account to a
> limited extent with their "pseudowords"
> (http://popfile.sourceforge.net/cgi-bin/wiki.pl?Glossary/PseudoWord).

We do this extensively ;)  see the source to Bayes.pm for details.

> But a
> rule-based system can be enfolded into a Bayes solution, because the result
> of evaluating any rule can be considered a token.  Imagine if SpamAssassin
> had an optional mode where, instead of using manual scoring, all the active
> rules of its impressively comprehensive ruleset were tokenized?  I think
> this would result in most impressive accuracy.

This is under investigation.  some previous attempts failed pretty
badly, but we think there might be a way to do it correctly.

> I'm imagining examples such as testing for blacklist membership, which some
> people (mostly elsewhere, apparently) think is a bad idea.  I could let
> Bayes decide just exactly how relevant it is to *my* corpus.  How about the
> "Message is x% to y% HTML" rules, or "text to image ratio" rules.  This
> would develop a very precise profile of what I consider to be ham and spam.
> 
> I've emailed Paul Graham and Gary Robinson for their opinions, and both
> agree that this is a good idea.  Paul pointed out that he mentions the
> possibility in the appendix to A Plan for Spam, and Gary mentioned a
> commercial product (PureMessage) that apparently does some of this.  It sure
> would be nice to see it in SpamAssassin someday.

- --j.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (GNU/Linux)
Comment: Exmh CVS

iD8DBQFAJEfRQTcbUG5Y7woRApcIAJ0R45Y8fXC+p+XCCt3MmnLAFEWawACfVA9k
EgVQDJPoVcZGbprzJsPOZ5A=
=20Ub
-----END PGP SIGNATURE-----

Re: How can I catch this one

[Bill Landry <Bi...@Pointshare.com>]

----- Original Message ----- 
From: "Loren Wilton" <lw...@earthlink.net>
Cc: <sp...@incubator.apache.org>
Sent: Friday, February 06, 2004 7:39 PM
Subject: How can I catch this one


> The following seems like a real common way to end a spam, yet no rule
> catches it.  I've been trying to write such a rule, but I just don't know
> enough about regexps to get it to work rather than blow up in my face.  I
> assume you have to use rawbody to catch this, which is still
line-oriented,
> and the pattern has to span multiple lines to work.
>
> Sample junk:
>
> </BODY></HTML>
> <br>
> baboon poppy burnett lengthwise hoar deadline diophantine infra pius malay
> dire silica diffract alcestis beachcomb hammerhead scintillate pussy giant
> genii deformation bodice anvil chromic scatterbrain contingent diploid
> humphrey considerate globule beauteous drunkard bellini buoy palmyra
> revolutionary cranelike cried amelia govern <br>
> freeman fifty fragment gingham marks comptroller alfonso madmen rumford
> erbium cochran continuity bainite antiquarian maurine demonstrable bambi
> serf fickle denote arteriole janeiro cease drunk ifni chartroom antony
coma
> molten dupont allegory greengrocer airstrip drip crewel beriberi agone
> calico instantiate <br>

With some hints from other here on the list, here is what I've been using
successfully to catch these kinds of messages, as well a long strings of
garbage text:

body     RANDOMWORDS_7
/(?:\b(?!(?:from|even|more|were|with)\b)[a-zA-Z]{4,12}\s+){7}/
describe RANDOMWORDS_7        string of 7+ random words
score    RANDOMWORDS_7        1.0

body     RANDOMWORDS_10
/(?:\b(?!(?:from|even|more|were|with)\b)[a-zA-Z]{4,12}\s+){10}/
describe RANDOMWORDS_10       string of 10+ random words
score    RANDOMWORDS_10       2.0

body     RANDOMWORDS_15
/(?:\b(?!(?:from|even|more|were|with)\b)[a-zA-Z]{4,12}\s+){15}/
describe RANDOMWORDS_15       string of 15+ random words
score    RANDOMWORDS_15       3.0

body     RANDOMWORDS_20
/(?:\b(?!(?:from|even|more|were|with)\b)[a-zA-Z]{4,12}\s+){20}/
describe RANDOMWORDS_20       string of 20+ random words
score    RANDOMWORDS_20       4.0

body     RANDOMWORDS_25
/(?:\b(?!(?:from|even|more|were|with)\b)[a-zA-Z]{4,12}\s+){25}/
describe RANDOMWORDS_25       string of 25+ random words
score    RANDOMWORDS_25       5.0

body     LONGWORDS_15         /[a-zA-Z]{15}/
describe LONGWORDS_15         string of 15+ random letters
score    LONGWORDS_15         5.0

body     LONGWORDS_20         /[a-zA-Z]{20}/
describe LONGWORDS_20         string of 20+ random letters
score    LONGWORDS_20         5.0

body     LONGWORDS_25         /[a-zA-Z]{25}/
describe LONGWORDS_25         string of 25+ random letters
score    LONGWORDS_25         5.0

body     LONGWORDS_30         /[a-zA-Z]{30}/
describe LONGWORDS_30         string of 30+ random letters
score    LONGWORDS_30         5.0

body     LONGWORDS_35         /[a-zA-Z]{35}/
describe LONGWORDS_35         string of 35+ random letters
score    LONGWORDS_35         5.0

body     LONGWORDS_40         /[a-zA-Z]{40}/
describe LONGWORDS_40         string of 40+ random letters
score    LONGWORDS_40         5.0

body     LONGWORDS_45         /[a-zA-Z]{45}/
describe LONGWORDS_45         string of 45+ random letters
score    LONGWORDS_45         5.0

body     LONGWORDS_50         /[a-zA-Z]{50}/
describe LONGWORDS_50         string of 50+ random letters
score    LONGWORDS_50         5.0

Bill

How can I catch this one

[Loren Wilton <lw...@earthlink.net>]

The following seems like a real common way to end a spam, yet no rule
catches it.  I've been trying to write such a rule, but I just don't know
enough about regexps to get it to work rather than blow up in my face.  I
assume you have to use rawbody to catch this, which is still line-oriented,
and the pattern has to span multiple lines to work.

Sample junk:

</BODY></HTML>
<br>
baboon poppy burnett lengthwise hoar deadline diophantine infra pius malay
dire silica diffract alcestis beachcomb hammerhead scintillate pussy giant
genii deformation bodice anvil chromic scatterbrain contingent diploid
humphrey considerate globule beauteous drunkard bellini buoy palmyra
revolutionary cranelike cried amelia govern <br>
freeman fifty fragment gingham marks comptroller alfonso madmen rumford
erbium cochran continuity bainite antiquarian maurine demonstrable bambi
serf fickle denote arteriole janeiro cease drunk ifni chartroom antony coma
molten dupont allegory greengrocer airstrip drip crewel beriberi agone
calico instantiate <br>

---------------------------

This is in a valid html section of a mimed message.  There will be a mime
terminator later, so in general the message is not malformed except for the
tons of random words after the </HTML> tag.

I have seen this both with and without the <br> lines.  So a good rule has
to be able to skip an optional <br> on one or more lines after </HTML>, then
look for 20 or so lowercase-letters-only words, possibly mixed with more
<br> tags.

Suggestions on how to scan across multiple lines while parsing the rawbody?
I'd think the rule would be simple if I could just figure out how to write
it!  :-)

        Loren