You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by Alex <my...@gmail.com> on 2010/03/18 23:05:34 UTC
Yahoo/URL spam
Hi,
I'm having a real problem with this persistent spam that contains just
a URL as the body, and is always from yahoo. I've got an example here:
http://pastebin.com/UqzhDHEu
'example.com' is my change. I'm using SA v3.2.5 with postfix/amavis.
I'm concerned that the bayes score is always low. I can't determine
any other patterns from this message to key on for other rules. Ideas
most welcome.
Thanks!
Best,
Alex
Re: Yahoo/URL spam
Posted by John Horne <jo...@plymouth.ac.uk>.
On Tue, 2010-03-23 at 13:18 -0400, Alex wrote:
> Hi Charles,
>
> >> /^[^a-z]{0,10}(http:\/\/|www\.)(\w+\.)+(com|net|org|biz|cn|ru)\/?[^
> >> ]{0,20}[^a-z]{0,10}$/msi
>
> This is what I have:
>
> /^[^a-z]{0,10}(http:\/\/|www\.)(\w+\.)+(com|net|org|biz|cn|ru)\/?[^
> ]{0,20}[a-z]{0,10}$/msi
^^^^^
The original had [^a-z]
John.
--
John Horne, University of Plymouth, UK
Tel: +44 (0)1752 587287 Fax: +44 (0)1752 587001
Re: [sa] Re: Yahoo/URL spam
Posted by Alex <my...@gmail.com>.
Hi,
>> This is what I have:
>> /^[^a-z]{0,10}(http:\/\/|www\.)(\w+\.)+(com|net|org|biz|cn|ru)\/?[^
>> ]{0,20}[a-z]{0,10}$/msi
>
> My bad. I got an option wrong. Please remove the 'm' above.
> I always get it backwards. According to 'man perlre' (the definitive
> resource for SA regexes!) the 'm' makes '^' match every newline!
> We want it to only match the beginning of the body.
Much better. Not sure how I introduced that typo. Thanks so much.
Best,
Alex
Re: [sa] Re: Yahoo/URL spam
Posted by Mike Grau <m....@kcc.state.ks.us>.
On 3/23/2010 2:49 PM the voices made Charles Gregory write:
> On Tue, 23 Mar 2010, Alex wrote:
>> This is what I have:
>> /^[^a-z]{0,10}(http:\/\/|www\.)(\w+\.)+(com|net|org|biz|cn|ru)\/?[^
>> ]{0,20}[a-z]{0,10}$/msi
>
> My bad. I got an option wrong. Please remove the 'm' above.
> I always get it backwards. According to 'man perlre' (the definitive
> resource for SA regexes!) the 'm' makes '^' match every newline!
> We want it to only match the beginning of the body.
>
> So just remove it, and, as noted by others, add the '^' that was
> missing... like so....
>
> ... ]{0,20}[^a-z]{0,10}$/si
Hello,
You might want to change (\w+\.)+ to ([\w-]+\.)+ to account for
domains like polster-jj.de
-- MG
Re: [sa] Re: Yahoo/URL spam
Posted by Charles Gregory <cg...@hwcn.org>.
On Tue, 23 Mar 2010, Alex wrote:
> This is what I have:
> /^[^a-z]{0,10}(http:\/\/|www\.)(\w+\.)+(com|net|org|biz|cn|ru)\/?[^
> ]{0,20}[a-z]{0,10}$/msi
My bad. I got an option wrong. Please remove the 'm' above.
I always get it backwards. According to 'man perlre' (the definitive
resource for SA regexes!) the 'm' makes '^' match every newline!
We want it to only match the beginning of the body.
So just remove it, and, as noted by others, add the '^' that was
missing... like so....
... ]{0,20}[^a-z]{0,10}$/si
- Charles
Re: Yahoo/URL spam
Posted by Alex <my...@gmail.com>.
Hi Charles,
>> /^[^a-z]{0,10}(http:\/\/|www\.)(\w+\.)+(com|net|org|biz|cn|ru)\/?[^
>> ]{0,20}[^a-z]{0,10}$/msi
>> This allows for some amount (up to ten chars?) of text before and
>> after the URI if I'm reading that right, correct?
>
> Nope. With the /ms flags ^ and $ at beginning and end match the *whole* body
> as a single 'string' and permit 'any character' (. or [^x]) matches to also
> match newlines. So the above regex translates to:
This was very helpful, thanks. I might be doing something wrong, or
there's a typo somewhere. It seems to catch situations where there's
more than just a URL in the body, such as just additional text.
This is what I have:
/^[^a-z]{0,10}(http:\/\/|www\.)(\w+\.)+(com|net|org|biz|cn|ru)\/?[^
]{0,20}[a-z]{0,10}$/msi
Thanks again,
Alex
Re: Yahoo/URL spam
Posted by Charles Gregory <cg...@hwcn.org>.
On Mon, 22 Mar 2010, Alex wrote:
>> rawbody __BODY_ONLY_URI
> /^[^a-z]{0,10}(http:\/\/|www\.)(\w+\.)+(com|net|org|biz|cn|ru)\/?[^ ]{0,20}[^a-z]{0,10}$/msi
> This allows for some amount (up to ten chars?) of text before and
> after the URI if I'm reading that right, correct?
Nope. With the /ms flags ^ and $ at beginning and end match the *whole*
body as a single 'string' and permit 'any character' (. or [^x]) matches
to also match newlines. So the above regex translates to:
/^ - Beginning of body
[^a-z]{0,10} - match 0-10 non-alpha characters *including* newlines
(http:\/\/|www\.) - match a uri beginning with http *or* www
(\w+\.)+ - match multiple occurences of word followed by .
(this will match 'domain.' *or* 'www.domain.')
(com|net|biz|org|cn|ru) - match TLD (adjust to fit your mail)
\/? - match a slash if there is one
[^ ]{0,20} - match 0-20 non-blank characters (page name, if given)
[^a-z]{0,10} - match 0-10 non-alpha chars including newlines
(did I TYPO in my OP and leave out the '^'?)
$ - match end of body
/msi
> Is it possible to determine the beginning of the line with a body rule?
Insert '\n' into the above regex where you want to match newline.....
> I didn't think that was possible. I believe this is also what this is
> trying to do?
It's possible, but NOT what this regex does. Essentially this regex
matches against a complete body that consists of nothing more than a
single URI on a line, with possible blank lines before or after.
Rather than test for newlines, I test for non-alpha so that a stray space
or tab or LF code does not fail to match.
This simple regex can also be 'dressed up' with elements of the form
(\<[^\>\<]+\> +)+ to match any HTML code inserted before or after the
URI. A regex could also check for a link consisting of text
enclosed by <a href=...> ... </a>
They key is to be sure that you don't use '*' or '+' in any context where
it could 'run away' and try to match large message bodies.... This way as
soon as the body exceeds 40 characters on either side of an unbroken
string of characters it stops the test. Relatively efficient for a rawbody
test....
- C
Re: Yahoo/URL spam
Posted by Alex <my...@gmail.com>.
Hi,
> Lots of ham may contain a URI, but how much ham contains ONLY a URI?
>
> Rough outline of rule, untested.....
>
> rawbody __BODY_ONLY_URI
> /^[^a-z]{0,10}(http:\/\/|www\.)(\w+\.)+(com|net|org|biz|cn|ru)\/?[^
> ]{0,20}[a-z]{0,10}$/msi
>
> Combine that with 'frequent abusers' like Yahoo, and you've got something
> you can give a few points....
This allows for some amount (up to ten chars?) of text before and
after the URI if I'm reading that right, correct? Is it possible to
determine the beginning of the line with a body rule? I didn't think
that was possible. I believe this is also what this is trying to do?
Thanks,
Alex
Re: Yahoo/URL spam
Posted by Alex <my...@gmail.com>.
Hi,
I'm having some additional difficulty with body URI rules and hoped
someone could help.
> rawbody __BODY_ONLY_URI
> /^[^a-z]{0,10}(http:\/\/|www\.)(\w+\.)+(com|net|org|biz|cn|ru)\/?[^
> ]{0,20}[a-z]{0,10}$/msi
This doesn't seem to catch a quoted-printable body and I can't figure
out how to adapt it to allow for the 'Content-Transfer-Encoding:" that
precedes the URL, if that's even the right approach. Here's an
example:
http://pastebin.com/NDR1n4sN
Ideas greatly appreciated.
Thanks,
Alex
Re: Yahoo/URL spam
Posted by Charles Gregory <cg...@hwcn.org>.
On Thu, 18 Mar 2010, Ned Slider wrote:
> If that's not an option, how about a meta rule for FROM_YAHOO and
> __HAS_ANY_URI (this rule exists in SA).
Lots of ham may contain a URI, but how much ham contains ONLY a URI?
Rough outline of rule, untested.....
rawbody __BODY_ONLY_URI
/^[^a-z]{0,10}(http:\/\/|www\.)(\w+\.)+(com|net|org|biz|cn|ru)\/?[^ ]{0,20}[a-z]{0,10}$/msi
Combine that with 'frequent abusers' like Yahoo, and you've got something
you can give a few points....
There will probably need to be a variant on this to account for HTML mail
and/or the 'standard' footers inserted by free mail agents. Which
incidentally, suprises me here. I thought Yahoo always added a tagline?
- C
Re: Yahoo/URL spam
Posted by Ned Slider <ne...@unixmail.co.uk>.
Alex wrote:
> Hi,
>
> I'm having a real problem with this persistent spam that contains just
> a URL as the body, and is always from yahoo. I've got an example here:
>
> http://pastebin.com/UqzhDHEu
>
> 'example.com' is my change. I'm using SA v3.2.5 with postfix/amavis.
> I'm concerned that the bayes score is always low. I can't determine
> any other patterns from this message to key on for other rules. Ideas
> most welcome.
>
I can see why those would be tricky if Bayes isn't catching them. Still,
keep training Bayes on them.
How much mail (ham) from yahoo do you see?
Personally, I've about had it with hotmail/yahoo etc and have given them
a score across the board so they now have far fewer points to play with
before being classified as spam - I then whitelist all known senders
from these troublesome domains. I'm small, so it's possible to do that
and it takes less time dealing with the occasional FP than it did trying
to stop their spam.
If that's not an option, how about a meta rule for FROM_YAHOO and
__HAS_ANY_URI (this rule exists in SA). Unlike hotmail that has a URI
embedded in the footer of every mail, this might work for the example
you post.