You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by Kenneth Porter <sh...@sewingwitch.com> on 2005/02/27 16:35:18 UTC
Obfuscation (was: Millions and Billions)
--On Thursday, February 24, 2005 6:07 PM -0500 Phil Barnett
<ph...@philb.us> wrote:
> i or l = [|ííiil1]
>
> a = [aã@]
>
> e = [eé3]
>
> o = [o0]
It seems like this is getting overly-complicated. Are there any libraries
for doing fuzzy string matching and obfuscation detection that could be
used instead of Perl regex's?
Re: Obfuscation (was: Millions and Billions)
Posted by Robert Menschel <Ro...@Menschel.net>.
Hello Kenneth,
Sunday, February 27, 2005, 7:35:18 AM, you wrote:
KP> --On Thursday, February 24, 2005 6:07 PM -0500 Phil Barnett
KP> <ph...@philb.us> wrote:
>> i or l = [|ííiil1]
>> a = [aã@]
>> e = [eé3]
>> o = [o0]
KP> It seems like this is getting overly-complicated. Are there any libraries
KP> for doing fuzzy string matching and obfuscation detection that could be
KP> used instead of Perl regex's?
Check out http://www.exit0.us/index.php?pagename=ChrissMediocreObfuScript
Bob Menschel
Re: Obfuscation
Posted by Marc Perkel <ma...@perkel.com>.
I had a trick I was using in Exim that worked pretty well and cound be
recoded in perl.
First - I had a list of words spelled correctly that spammers often
deliberately misspell.
What I did was take the subject and the first 200 characters of the
body. Then I removed all the words matching the list that were spelled
correctly. Then I translated letters used to objuscate into the letters
that they were faking and I removed all characters that we junk spacing
characters. This process "corrected" the spelling of the ofuscated
words. I then compared the strings to my original list and if I found a
word it was because it was deliberately misspelled.
Kenneth Porter wrote:
> --On Thursday, February 24, 2005 6:07 PM -0500 Phil Barnett
> <ph...@philb.us> wrote:
>
>> i or l = [|ííiil1]
>>
>> a = [aã@]
>>
>> e = [eé3]
>>
>> o = [o0]
>
>
> It seems like this is getting overly-complicated. Are there any
> libraries for doing fuzzy string matching and obfuscation detection
> that could be used instead of Perl regex's?
>
--
Marc Perkel - marc@perkel.com
Spam Filter: http://www.junkemailfilter.com
My Blog: http://marc.perkel.com
My Religion: http://www.churchofreality.org
~ "If it's real - we believe in it!" ~
Re: Obfuscation
Posted by Marc Perkel <ma...@perkel.com>.
I had a trick I was using in Exim that worked pretty well and cound be
recoded in perl.
First - I had a list of words spelled correctly that spammers often
deliberately misspell.
What I did was take the subject and the first 200 characters of the
body. Then I removed all the words matching the list that were spelled
correctly. Then I translated letters used to objuscate into the letters
that they were faking and I removed all characters that we junk spacing
characters. This process "corrected" the spelling of the ofuscated
words. I then compared the strings to my original list and if I found a
word it was because it was deliberately misspelled.
Kenneth Porter wrote:
> --On Thursday, February 24, 2005 6:07 PM -0500 Phil Barnett
> <ph...@philb.us> wrote:
>
>> i or l = [|ííiil1]
>>
>> a = [aã@]
>>
>> e = [eé3]
>>
>> o = [o0]
>
>
> It seems like this is getting overly-complicated. Are there any
> libraries for doing fuzzy string matching and obfuscation detection
> that could be used instead of Perl regex's?
>
--
Marc Perkel - marc@perkel.com
Spam Filter: http://www.junkemailfilter.com
My Blog: http://marc.perkel.com
My Religion: http://www.churchofreality.org
~ "If it's real - we believe in it!" ~
Re: Obfuscation (was: Millions and Billions)
Posted by Kenneth Porter <sh...@sewingwitch.com>.
--On Sunday, February 27, 2005 7:46 PM -0800 Loren Wilton
<lw...@earthlink.net> wrote:
> He has a point. A complicated regex is complicated, and that can mean
> slow. It also by definition means "incomprehensible to humans", and so
> has to be generated by a tool, and then not touched or looked at.
Thankfully we have people like Loren to perform a fuzzy match on my
obfuscated question. ;)
I was thinking "maintenance complexity" more than "runtime complexity",
although simplifying the former would hopefully make the runtime more
efficient.
Re: Obfuscation (was: Millions and Billions)
Posted by Loren Wilton <lw...@earthlink.net>.
> > I just question whether regex's are the right "complicated solution".
> >
> > How does Google or one of the dictionary sites guess the correct
spelling
> > for a misspelled word?
>
> Great, why don't you go see if google can guess the correct spelling for
>
> c l @ L i @ s
He has a point. A complicated regex is complicated, and that can mean slow.
It also by definition means "incomprehensible to humans", and so has to be
generated by a tool, and then not touched or looked at.
Since a tool can generate the matching pattern and convert it to a re, it
seems that a tool could in theory generate a matching pattern and convert it
to something else that might be either more comprehensible or more
efficient. Or possibly a tool could be made that would do a direct fuzzy
match from the unobfuscated word. (However, I think this last possibility
would be slower than pre-obfuscating; but possibly it wouldn't be.)
The problem is that perl doesn't have any syntax to efficiently describe
this obfuscated match other than an incomprehensible regex.
Someone could invent such a tool, and it could either be a plugin to SA or a
part (or addon subroutine) called by perl itself. In fact I believe that at
least two fuzzy matching plugins have been added to SA in the last week.
Whether they are as efficient, or more efficient, than the current horrid
re's is an interesting question.
Loren
Re: Obfuscation (was: Millions and Billions)
Posted by Phil Barnett <ph...@philb.us>.
On Sunday 27 February 2005 06:31 pm, Kenneth Porter wrote:
> --On Sunday, February 27, 2005 11:48 AM -0500 Phil Barnett <ph...@philb.us>
>
> wrote:
> > All you have requested here is for someone else to do the complicated
> > stuff and make it easy for you. Someone has to get the code as complex
> > as it needs to be. If not you, then the guy that makes the library you
> > seek.
>
> I just question whether regex's are the right "complicated solution".
>
> How does Google or one of the dictionary sites guess the correct spelling
> for a misspelled word?
Great, why don't you go see if google can guess the correct spelling for
c l @ L i @ s
--
"The significant problems we face cannot be solved at the same level of
thinking we were at when we created them"
Albert Einstein
Re: Obfuscation (was: Millions and Billions)
Posted by Kenneth Porter <sh...@sewingwitch.com>.
--On Sunday, February 27, 2005 11:48 AM -0500 Phil Barnett <ph...@philb.us>
wrote:
> All you have requested here is for someone else to do the complicated
> stuff and make it easy for you. Someone has to get the code as complex
> as it needs to be. If not you, then the guy that makes the library you
> seek.
I just question whether regex's are the right "complicated solution".
How does Google or one of the dictionary sites guess the correct spelling
for a misspelled word?
Re: Obfuscation (was: Millions and Billions)
Posted by Phil Barnett <ph...@philb.us>.
On Sunday 27 February 2005 10:35 am, Kenneth Porter wrote:
> --On Thursday, February 24, 2005 6:07 PM -0500 Phil Barnett
>
> <ph...@philb.us> wrote:
> > i or l = [|ííiil1]
> >
> > a = [aã@]
> >
> > e = [eé3]
> >
> > o = [o0]
>
> It seems like this is getting overly-complicated. Are there any libraries
> for doing fuzzy string matching and obfuscation detection that could be
> used instead of Perl regex's?
I believe there are three premises here:
1: Never make software more complex than it needs to be.
2: Always make software as complex as is needed to get the job done.
3: Spammers aren't necessarily all stupid.
All you have requested here is for someone else to do the complicated stuff
and make it easy for you. Someone has to get the code as complex as it needs
to be. If not you, then the guy that makes the library you seek.
--
"The significant problems we face cannot be solved at the same level of
thinking we were at when we created them"
Albert Einstein