possibly a dumb comment, apologies if I'm being a n00b

[R Michael Harman <rm...@auros.org>]

So, I've been looking over the types of messages that SA is missing, on my
mail stream...  It seems like most of them still contain trigger words
that would cause high scores, they're just slightly masked.  Has anyone
talked about applying some kind of fuzzy-matching techniques?  Taking the
trigger words, and generating a whole large set of patterns that match,
based on rules such as:

'a' => /a|A|@)/
'x' => /x|X|></

You might even be able to use a large corpus of spam to automatically
derive these rules.  (A corpus of parsed-out and "translated" tokens would
work better, obviously.)

You could also introduce some Hamming Distance effects to the match, so
that, say hamming_match( /hello/ , 1 ) would match "helldo", "helo", and
"hillo".  And then there's the possibility of doing phonetic matching,
like many spellcheckers.

Using any/all of this stuff would be pretty processor intensive --
probably much more practical for ISPs than for users -- but it seems like
it'd kill off almost all of the new crop of SA-evading spam.  Maybe
somebody could lure Larry Wall into building this kind of fuzzy-match
technology directly into the next major version of Perl? *g*

Just thought I'd throw that out there.  Aside from that, I'll probably
lurk for a while; if I end up feeling out of my depth (which is possible
-- my actual day job is as a linguist, and most of my coding skills, such
as they are, are aimed at that) I'll unsub.

Thanks,
Auros

------------------------------------------------------------------------
R Michael Harman / Auros Symtheos
rmharman@auros.org ............ http://www.auros.org/

Linguist and Eclectic Engineer, Lexicus, Motorola
rmharman@motorola.com ......... http://www.lexicus.mot.com/

Senior Reviews Editor, Strange Horizons Speculative Fiction Weekly
reviews@strangehorizons.com ... http://www.strangehorizons.com/

Re: possibly a dumb comment, apologies if I'm being a n00b

[R Michael Harman <rm...@auros.org>]

On Wed, 10 Mar 2004, Matt Kettler wrote:
> One thing that needs to be kept in mind is that SA is already quite
> effective against mis-spelling tactics... mis-spellings are VERY easy
> targets for any bayes system, you just have to be up-to-speed on your training.

Yeah, that sorta makes sense...  But fuzzy matching would eliminate the
need for keeping up on the Bayesian training, which apparently would make
a difference at some ISPs.

I've been trying to persuade my ISP to start using the Bayes stuff at the
site level, but they've been resistant to doing the extra work; or
possibly they're just totally clueless about the existance of the Bayesian
stuff -- when I emailed asking whether it was possible for individual
users to get access to sa-train, they suggested I add "score TEST_NAME"
lines to my user_prefs file.  SA runs on the mailserver, which doesn't
have shell access; to get mail onto the shell server, users run fetchmail.
But SA is not provided on the shell server, so I don't think I'd be able
to do Bayesian stuff for just my account.  Even if I could, having
individual users on the shell server all running their own Bayesian-tuned
SAs might be a processor issue...

I suggested that the admins send a note to all users asking for
contributions of spam and ham, or providing standardized folder names
where people could keep stuff for collation and use in training SA
sitewide, but they didn't seem to think it was worth it.

Maybe I just need a better ISP.  (Though finding one that'll provide a
*NIX shell environment _at all_ is pretty tough these days. *sigh*)

Thanks for the thoughtful reply,
Auros

------------------------------------------------------------------------
R Michael Harman / Auros Symtheos
rmharman@auros.org ............ http://www.auros.org/

Linguist and Eclectic Engineer, Lexicus, Motorola
rmharman@motorola.com ......... http://www.lexicus.mot.com/

Senior Reviews Editor, Strange Horizons Speculative Fiction Weekly
reviews@strangehorizons.com ... http://www.strangehorizons.com/

Re: possibly a dumb comment, apologies if I'm being a n00b

[Matt Kettler <mk...@EVI-INC.COM>]

At 12:28 PM 3/10/2004, R Michael Harman wrote:
>Taking the trigger words, and generating a whole large set of patterns 
>that match,
>based on rules such as:
>
>'a' => /a|A|@)/
>'x' => /x|X|></

Well, there's no reason to include a and A, since it's more efficient to 
declare the rule case-insenstive instead, but that aside, these kinds of 
obfuscation matches work well. I've been working with this stuff quite a 
bit in antidrug.cf.

There's also a wide variety of "gapping"  techniques used by spammers.

If for example, you look at my antidrug ruleset, you'll see my current 
regex for the v-word is:

body 
__DRUGS_MALEDYSFUNCTION1 
/(?:\b|\s)[_\W]{0,3}(?:\\\/|V)[_\W]{0,3}[i1|l][_\W]{0,3}[a4@][_\W]{0,3}g[_\W]{0,3}r[_\W]{0,3}[a4@][_\W]{0,3}x?[_\W]{0,3}(?:\b|\s)/i

This catches most of the common substitutions for A,O, V, and I. Allows for 
a number of gapping characters consisting of non-word and _ characters 
between letters (as few as 0 and as many as 3). It also allows an optional 
x that some spammers tack onto the end of the word.

However, there's a little bit of caution needed before comprehensively 
applying this technique everywhere. The large number of match combinations 
can lend itself to FPs. I've been slowly propagating these features 
throughout the antidrug ruleset.

>You could also introduce some Hamming Distance effects to the match, so
>that, say hamming_match( /hello/ , 1 ) would match "helldo", "helo", and
>"hillo".  And then there's the possibility of doing phonetic matching,
>like many spellcheckers.

There's been a fair amount of talk about doing things like this in the 
past. I've never seen anyone propose hamming distances or phonetic methods 
specifically, but many similar ideas have come by.

One thing that needs to be kept in mind is that SA is already quite 
effective against mis-spelling tactics... mis-spellings are VERY easy 
targets for any bayes system, you just have to be up-to-speed on your training.

Some similar proposals from the past:

Pure Spell checking has been shot-down after testing (not the same as what 
you proposed, but it is related)
         http://bugzilla.spamassassin.org/show_bug.cgi?id=2868

Soundex matching has been proposed, but is too unreliable, too slow, and 
offering little advantage that bayes doesn't provide
         http://bugzilla.spamassassin.org/show_bug.cgi?id=1407