You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by Linda Walsh <sa...@tlinx.org> on 2005/09/16 08:47:41 UTC
SpmAssn 3.04 v. 2.6x false negative rate: Help???
Ever since I "upgraded" to the 3.x series I've had a major jump
in spams that are getting through.
Initially my upgrade was to 3.02 as distributed in SuSE 9.3 and
my problems were related to old configuration files/options where
NONE of my spam was being tagged into the spam folder (i.e. the SPAM header
wasn't set in the subject, as my filtering system makes use of).
I've gotten all of the "lint" out of my config files, ported my old
DB to the new format, and even ran the learning mechanism over several
old "SPAM" archives (~150Mb) and current "HAM" input folders ~100Mb.
About 100 spams a day are getting through and requiring manual
processing with about 100/day being correctly filtered into the spam
folder. That's a huge drop in detected spams. I've tried dialing
down the threshold from the default to my previous 5, then to 4.8...
not wanting to be overly aggressive. But I'm wondering if the default
weightings for various tests have been changed between the 2.6x and 3.0x
series.
I note a new 3.1.0 release, but noticed no improvement going from 3.02
to 3.04.
It _seems_ like, maybe, some of the weightings of the various tests
changed which is throwing off the classifier. I'll see multiple instances
of various, identical spams going to different email addresses on my
server -- most often with "Subject: Re[<x>]:", where x=[0-9]. They are
the most numerous offenders as they'll come in to multiple accounts
at nearly the same time (or a few seconds apart). One copy of those
messages will result in duplicate spam being sent to several accounts,
and my multiple personalities, er, um, "users" :-), are getting annoyed
with me.
Also of note: "sa-learn" is MUCH slower in 3.0.x than it was in 2.6.x though
with the compiled "spamc" client, I can see that the processing of incoming
spam is handled with a lower load on the server.
One voice in my head says, screw it, stop your whining and go back to what
worked (2.6x), but another part of me says "3.x" is where the future is, and
if there is a problem in my setup, I should take the time to figure out
what the problem is and try to make it work.
Looking at a partial header of one note:
X-Spam-Report:
* 0.0 HTML_MESSAGE BODY: HTML included in message
* 3.5 BAYES_99 BODY: Bayesian spam probability is 99 to 100%
* [score: 1.0000]
----
Content was multi-part message in MIME format, same messsage in plain and
HTML text:
"<content deleted due to spamassassin filtering>"
------
Content involved advertising product to increasing one's
chance of producing offspring via chance encounters with receptive
female partners. Is 5.0 too high a default in 3.x, though I would have
expected it to count a little bit more for an HTML message...
Ooops another batch of 80+ just came in....SA tastes great, less filling!
re: first posting attempt:
<<< 552 spam score (9.1) exceeded threshold
<BZZZT!!! NO discussion involving things that look too much like spam
on a list to designed to talk about a tool to detect such spam>
And ironically, the irony of this restriction may never be known if this
note never makes it to the list...;-/.
Sigh,
Linda
Re: SA 3.04: high fail rate; X-SA-no-reject?; more details.
Posted by Magnus Holmgren <ho...@lysator.liu.se>.
Linda Walsh skrev:
> The full list of SA hits? -- for that message, that was it, here's another
> passed. Note, there is a weird header "X-SA-Do-Not-Rej: Yes" which doesn't
> look normal:
X-SA-Do-Not-Rej is an internal header used between the Exim MTA and the
SA-Exim plugin for said MTA, commonly used to disable the SMTP-time
rejection otherwise applied to messages with sufficiently high scores.
It should obviously be removed from all incoming messages, but some
admins might miss that so the spammers see their chance. If you don't
use SA-Exim, or Exim for that matter, or use another header name or use
ACL variables instead, you can as well add a rule to give messages
containing that header some points. Beware however that legitimate mail
could contain it as well, if some admin has forgotten to remove it from
outgoing mail.
--
Magnus Holmgren
holmgren@lysator.liu.se
Re: SA 3.04: high fail rate; X-SA-no-reject?; more details.
Posted by jdow <jd...@earthlink.net>.
From: "Linda Walsh" <sa...@tlinx.org>
>
>
> Loren Wilton wrote:
>
>>If you are only correctly classifying 50% of the spam (you said 100 caught
>>to 100 missed, I htink) then you have SERIOUS problems of some sort.
> ----
> Yeah, well, I try not to be too reactionary on computer
> things like this -- especially when it could just be a
> matter of flipping a config switch somewhere and things get
> instantly better. While the number of spams getting through
> are significantly higher, probably 75-80% of them are duplicate
> emails sent to multiple email addresses -- including some
> blacklisting To-Addresses. Apparently, the spammer isn't being
> kind enough to send the spam to the black-listed To-Add'ies first
> and with the new spamc client, sendmail notices the lower load
> average and likely allows more parallel incoming instances to
> process incoming email before a given spam gets "locked out".
> I suppose this could be a "downside" of this efficiency, but
> previous to this I never saw multiple instances of these
> simple spams get through **undetected**. This makes me think
> it isn't just the increased efficiency causing problems as
> I would have expected at least one or two duplicate spams
> that wouldn't have been caught by "other means" (than being
> sent to a blacklisted To-addr).
Linda, looking at your score for Bayes 99 I think you can safely
raise it if you are running a very small mail service with well
known customers. I run with it at a full 5 here. I get very few
escaped spams, perhaps 0.1% within a factor of two either way.
I do have some slightly negative scoring rules when I can determine
a message is likely legitimate by specific for me rules. So if ham
marked as BAYES_99 only about 1 in 16000 got through recently. And
I very seldom have ham coming through as spam, except from the kernel
mailing list and the FC4 list when someone posts oddly formatted
messages, usually patches or debug logs. Perhaps I might get two of
those a week out of 5000 ham emails. So this works at this site with
two people.
A quick trick would be to go to the SARE site and get their version
of sa-stats.pl. Rename it to something distinguishable from the mostly
useless sa-stats.pl that comes with spamassassin itself. Then run it
something like this: /etc/mail/spamassassin/mysa-stats.pl -f maillog*
Once you've run it look at the BAYES_99 rule. It SHOULD sit right at
the top of your top spam rules ranking:
TOP SPAM RULES FIRED
------------------------------------------------------------
RANK RULE NAME COUNT %OFRULES %OFMAIL %OFSPAM
%OFHAM
------------------------------------------------------------
1 BAYES_99 6630 4.41 27.24 85.98
0.01
If it does not look something like this with numbers near 20000 or more
emails incoming over the duration of all your saved logs then BAYES may
need more careful training. Note how much ham was marked with BAYES_99.
I use that to "justify" setting the score for BAYES_99 up to a full 5
after the careful inclusion of some small negative scoring rules that
can off set it slightly in special cases. In any case you might find you
can justify a nearly perfect rule, high spam catch and very low or no
ham catch (this was a bad month for ham caught by BAYES_99), being scored
high enough to mark a message as spam all by itself or nearly high enough.
One other serious hint, do NOT run this list through SpamAssassin. That
may help protect your BAYES scores from subtle shifts such as might come
if you merely have it white listed.
{^_^}
Re: SA 3.04: high fail rate; X-SA-no-reject?; more details.
Posted by Linda Walsh <sa...@tlinx.org>.
Loren Wilton wrote:
>If you are only correctly classifying 50% of the spam (you said 100 caught
>to 100 missed, I htink) then you have SERIOUS problems of some sort.
>
----
Yeah, well, I try not to be too reactionary on computer
things like this -- especially when it could just be a
matter of flipping a config switch somewhere and things get
instantly better. While the number of spams getting through
are significantly higher, probably 75-80% of them are duplicate
emails sent to multiple email addresses -- including some
blacklisting To-Addresses. Apparently, the spammer isn't being
kind enough to send the spam to the black-listed To-Add'ies first
and with the new spamc client, sendmail notices the lower load
average and likely allows more parallel incoming instances to
process incoming email before a given spam gets "locked out".
I suppose this could be a "downside" of this efficiency, but
previous to this I never saw multiple instances of these
simple spams get through **undetected**. This makes me think
it isn't just the increased efficiency causing problems as
I would have expected at least one or two duplicate spams
that wouldn't have been caught by "other means" (than being
sent to a blacklisted To-addr).
> As a
>happy 2.63 user that upgraded to 3.04, it too a little minor fiddling, but
>by and large things are *much* better now, and they were good before.
>
>
-- *(oh the salt, the salt [in the wound]...:-) )* ---
>Also, you mentioned training with 'old spam' and 'new ham'. Presumably you
>were talking about bayes training. Really training with new spam,
>especially the stuff slipping through, would be the right thing to do. Spam
>has changed considerably in character in just the last 6 months.
>
>
----
Sorry, unclear: I archive current spams after "sa-learn"ing on
them, so "archives" contain anything older than whatever I
haven't processed "recently". With SA 2.63, I'd go through my
Junk email folder sometimes as infrequently as once/month and find
maybe 6-10 emails that should have gone to subscribed lists or
where from recent online vendors that sent me spammy looking
receipts (although those were rare). I'd drop them in my "despam"
folder for later "ham learning". But sifted folders of junk
email, I process(sa-learn-junk) in bulk and archive.
>Suggestion: let us see the full list of SA hits on some of the stuff
>slipping through.
>
>
The full list of SA hits? -- for that message, that was it, here's another
passed. Note, there is a weird header "X-SA-Do-Not-Rej: Yes" which doesn't
look normal:
---junk email that passed as ham; sent to multiple email accounts---
Received: (qmail 16547 invoked from network); 16 Sep 2005 18:08:51 -0000
Received: from unknown (HELO thaimail.org) ([202.150.81.42])
(envelope-sender <Ve...@ukrpost.net>)
by mail7.sea5.speakeasy.net (qmail-ldap-1.03) with SMTP
for <80...@tlinx.org>; 16 Sep 2005 18:08:49 -0000
From: "Molnar Chris" <Ve...@ukrpost.net>
To: "Siedler Clemens" <80...@tlinx.org>
Subject: Re[6]:
Date: Fri, 16 Sep 2005 18:09:04 +0000
Message-ID: <6d...@thaimail.org>
X-SA-Do-Not-Rej: Yes
MIME-Version: 1.0
Content-Type: multipart/alternative;
boundary="----=_NextPart_000_40CE_1F627A89.B53D40CE"
X-Priority: 3
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook Express 6.00.2900.2527
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2527
X-Spam-DCC: :
X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on
ishtar.sc.tlinx.org
X-Spam-Level: ***
X-Spam-Status: No, score=3.5 required=4.8 tests=BAYES_99,HTML_MESSAGE
autolearn=no version=3.0.4
X-Spam-Pyzor:
X-Spam-Report:
* 0.0 HTML_MESSAGE BODY: HTML included in message
* 3.5 BAYES_99 BODY: Bayesian spam probability is 99 to 100%
* [score: 1.0000]
Status:
X-Status:
X-Keywords: Junk
--------------------------------------------------------------------------------
In a rejected email, I see many more tests:
------------------junk email correctly
labeled----------------------------------
Subject: ***SPAM*** Athena, Electric-chair for little or no-cost
MIME-Version: 1.0
X-Mailid: 6977
Content-Type: multipart/alternative; boundary="==8aa9d3a4cb398b"
Date: Thu, 15 Sep 2005 14:56:00 -0700
X-Spam-Prev-Subject: Athena, Electric-chair for little or no-cost
X-Spam-DCC: :
X-Spam-Flag: YES
X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on
ishtar.sc.tlinx.org
X-Spam-Level: ******
X-Spam-Status: Yes, score=6.9 required=4.8 tests=BAYES_99,HTML_90_100,
HTML_IMAGE_ONLY_20,HTML_IMAGE_RATIO_02,HTML_MESSAGE,HTML_WEB_BUGS,
MIME_HTML_MOSTLY,MPART_ALT_DIFF,MSGID_FROM_MTA_HEADER,
MSGID_FROM_MTA_ID autolearn=no version=3.0.4
X-Spam-Pyzor:
X-Spam-Report:
* 1.7 MSGID_FROM_MTA_ID Message-Id for external message added
locally
* 0.4 HTML_IMAGE_ONLY_20 BODY: HTML: images with 1600-2000
bytes of wor
ds
* 0.0 HTML_IMAGE_RATIO_02 BODY: HTML has a low ratio of text to
image a
rea
* 0.0 HTML_90_100 BODY: Message is 90% to 100% HTML
* 0.0 HTML_WEB_BUGS BODY: Image tag intended to identify you
* 1.0 MIME_HTML_MOSTLY BODY: Multipart message mostly text/html
MIME
* 0.0 HTML_MESSAGE BODY: HTML included in message
* 0.1 MPART_ALT_DIFF BODY: HTML and text parts are different
* 3.5 BAYES_99 BODY: Bayesian spam probability is 99 to 100%
* [score: 1.0000]
* 0.1 MSGID_FROM_MTA_HEADER Message-Id was added by a relay
----------------------------------------------------------------------------------
It seems likely this is a "default" configuration problem, or my
slight tinkering with a default config that sometimes turns off most
of the useful tests.
About a 20% chance of being related to having installed 3.0.2 from the SuSE
package then upgrading components to 3.0.4 through CPAN and manual copying
over non-standard placement of 3.0.2 images (like location for spamd).
Example:
*Ignoring docs*
SuSE's SA rpm has RC scripts and:
/usr/bin/sa-learn
/usr/bin/spamassassin
/usr/bin/spamc
/usr/sbin/spamd -- *non standard loc*; but copied 3.0.4 spamd from /usr/bin
over 3.0.2 location and verfied start of correct version daemon.
-----
SuSE's perl-spamassassin RPM has more files that could provide for
some unstandard placement. The have:
/etc/mail/spamassassin/init.pre
/etc/mail/spamassassin/local.cf
Then the various module files under "vendor/<perl
version>/Mail/SpamAssassin".
All of the rule files are placed under /usr/share/spamassassin/
Note - the rule files under /usr/share/spamassassin are marked as
requiring
SA version 3.000004, so it seems this directory is the correct dir for
the rule-base.
====================
So ... I'm wondering...why were most of the local tests disabled? Is
it related to the "X-SA-no-reject"? That could be a "hole"....I note
that non of my "valid" email has that header marked in it -- but I
did notice one or more spam's that passed through with that header.
Could be coincidence?
Linda
p.p.s. -- This is truely an annoying problem if it occurs, and I don't
know if it will occur on this message with all the junk-marked headers
that are included, but when does the "SA" list reject
content about "Spm" because an email discussing it contains too much
content that looks like "Spm"? Seems there should be a way to include
or attach content (I don't think most modern emailers have built-in
rot13 enc/decoders built-in :-)) ...?
Re: SpmAssn 3.04 v. 2.6x false negative rate: Help???
Posted by Loren Wilton <lw...@earthlink.net>.
If you are only correctly classifying 50% of the spam (you said 100 caught
to 100 missed, I htink) then you have SERIOUS problems of some sort. As a
happy 2.63 user that upgraded to 3.04, it too a little minor fiddling, but
by and large things are *much* better now, and they were good before.
I used to have a 4.6 threshhold on 2.6x, but 5.0 is working fine on 3.04.
We did take bayes_99 up to somewhere around 5, and I would probably put it
at at least 4 - 4.5.
However, that can't be anywhere near your whole problem. But you didn't
show a complete list of hits for an FN, so we can only guess. My first
guesses are going to be that you don't have trusted networks set correctly
and so are getting ALL_TRUSTED firing when it shouldn't, or that you aren't
running network tests at all. Or worse, you are configured for network
tests but have a problem that is keeping them from working.
The particular spam you mention is one of the lower-scoring forms currently.
If you don't have net tests, and aren't running some of the SARE rulesets,
then you might be getting a few of these slipping through. Net tests should
be catching the vast majority though.
Also, you mentioned training with 'old spam' and 'new ham'. Presumably you
were talking about bayes training. Really training with new spam,
especially the stuff slipping through, would be the right thing to do. Spam
has changed considerably in character in just the last 6 months.
Suggestion: let us see the full list of SA hits on some of the stuff
slipping through.
Loren