You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by "L. A. Walsh" <sa...@tlinx.org> on 2005/08/29 03:41:40 UTC
V3.0.2 vs. 2.63 detection rate decrease (?)
I remember an email some time ago about an increase in non-ham messages
getting through when 3.0 first came out.
I recently upgraded my distro which included an upgrade to
SA-3.0.2.
I moved my old database, and imported the information using
sa-learn --import;
I then re-ran sa-learn on about 200M of archived "Junk" (non-ham) and
over all my current folders of "ham".
Unfortunately, I'm still noticing a significantly higher percentage of
SPAM getting through -- where before, I might get 0-3 SPAM emails
through per week, now I'm seeing about 15-20 through/day.
It's too early to tell on the false positive rate, but that ran...
between 0-4/week (if I count ads from vendors I did business with,
but hadn't bothered to unsub from their lists) to
0-4/month of real false negatives that were usually from businesses
I'd just done business with and hadn't whitelisted.
Is this new decreased detection rate a permanent "feature" or
trade-off of the newer-less memory intensive features of spamd/spamc,
or do I need to retune something?
Just shooting in the dark incase there's an FAQ type doc for
people upgrading from 2.6x to 3.0.x...
thanks in advance...
Linda
Re: SA globbing rules: thanks! (was Re: Basic Question: RE syntax?
)
Posted by Matt Kettler <mk...@evi-inc.com>.
Linda A. Walsh wrote:
> Matt Kettler wrote:
>
>> Linda A. Walsh wrote:
>>
>>> Actually, I know sed/[e]grep/bash{2,3} et al. RE's, just need to know
>>> which syntax is supported, i.e. old "sh", bash2 compatible, bash3
>>> (w/iterators),
>>> sed compat RE's, grep (w/POSIX support), egrep (w/alternation)...etc.
>>
>>
>> FWIW, perl regexes are based on, and compatible with posix extended
>> regular
>> expressions.
>>
> ---
> I thought some of the "(?..)" constructs were experimental
> and not compatible with the with strict POSIX usage?
I meant that Perl regexes are a superset of posix extended regexes.
So the compatibility is really one way. Perl will run a posix extended regex,
but a engine based extended regex will fail to run a full-blown perl regex.
>> To quote the Mail::SpamAssassin::Conf manpage:
>
> ---
> Yeay!! That's what I was looking for...Didn't know the right manpage. Was
> just looking at Spamassassin, Mail::Spamassassin, sa-learn, spamc, spamd...just
> didn't go far enough. Thanks!
Yes.. that's the important manpage..
SA globbing rules: thanks! (was Re: Basic Question: RE syntax? )
Posted by "Linda A. Walsh" <sa...@tlinx.org>.
Matt Kettler wrote:
> Linda A. Walsh wrote:
>> Actually, I know sed/[e]grep/bash{2,3} et al. RE's, just need to know
>>which syntax is supported, i.e. old "sh", bash2 compatible, bash3
>>(w/iterators),
>>sed compat RE's, grep (w/POSIX support), egrep (w/alternation)...etc.
>
> FWIW, perl regexes are based on, and compatible with posix extended regular
> expressions.
>
---
I thought some of the "(?..)" constructs were experimental
and not compatible with the with strict POSIX usage?
> To quote the Mail::SpamAssassin::Conf manpage:
---
Yeay!! That's what I was looking for...Didn't know the right manpage. Was
just looking at Spamassassin, Mail::Spamassassin, sa-learn, spamc, spamd...just
didn't go far enough. Thanks!
It's not exactly a posix or bash style fileglob though, but more the "easy-to
understand", "DOS-style" file-glob, since on *nix, something like
"*@*.[0-9]*.[0-9]*" would seem like it could match hosts with numeric-only
TLD's , but wouldn't seem to be be permitted metachars
under SA's globbing rules. _Not_ a complaint; only pointing out
what appears to be the the case from the manpage. ;^)
Re: Basic Question: RE syntax? (was Re: V3.0.2 vs. 2.63 detection
rate decrease (?))
Posted by Matt Kettler <mk...@evi-inc.com>.
Linda A. Walsh wrote:
> Matt Kettler wrote:
> Actually, I know sed/[e]grep/bash{2,3} et al. RE's, just need to know
> which syntax is supported, i.e. old "sh", bash2 compatible, bash3
> (w/iterators),
> sed compat RE's, grep (w/POSIX support), egrep (w/alternation)...etc.
FWIW, perl regexes are based on, and compatible with posix extended regular
expressions.
However, perl regexes are only used when creating rules in spamassassin. They
are not used in whitelists.
You
> say "fileglob" format, is that as in "sh" or "ash", or "bash3.x"? I'm
> still not real clear from the docs (sorry) ...?
file globbing is the format that sh/bash and MS dos use for file name expansions.
ie: *.txt matches joe.txt, whatever.txt but not bluetxt.
To quote the Mail::SpamAssassin::Conf manpage:
-----------------------------
Whitelist and blacklist addresses are now file-glob-style patterns, so
friend@somewhere.com, *@isp.com, or *.domain.net will all work. Specifically, *
and ? are allowed, but all other metacharacters are not. Regular expressions are
not used for security reasons.
Multiple addresses per line, separated by spaces, is OK. Multiple
whitelist_allows_relays lines is also OK.
The specified email address does not have to match exactly the address
previously used in a whitelist_from_rcvd line as it is compared to the address
in the header.
e.g.
whitelist_allows_relays joe@example.com fred@example.com
whitelist_allows_relays *@example.com
Re: Basic Question: RE syntax? (was Re: V3.0.2 vs. 2.63 detection
rate decrease (?))
Posted by "Linda A. Walsh" <sa...@tlinx.org>.
Matt Kettler wrote:
> (That said, if you're a sysadmin I'd strongly suggest learning basic regex
> syntax. "grep" and many other command-line sysadmin tools become so much more
> powerful if you know how to create a regex.)
----
Actually, I know sed/[e]grep/bash{2,3} et al. RE's, just need to know
which syntax is supported, i.e. old "sh", bash2 compatible, bash3 (w/iterators),
sed compat RE's, grep (w/POSIX support), egrep (w/alternation)...etc. You
say "fileglob" format, is that as in "sh" or "ash", or "bash3.x"? I'm still
not real clear from the docs (sorry) ...?
Thanks,
Linda
Re: Spamassassin scoring bad after years of service......
Posted by Matt Kettler <mk...@evi-inc.com>.
Lorin G. Tremblay wrote:
> Hi!
>
> Was wondering if anyone had any clue to why spamassassin would start to
> score spam badly and let almost any spam throught.
>
> Tehy was no change in the hardware of software, it just started to score
> spam really badly, but had workes for atleast a full year without any glitch!
>
> Any pointer would be great!
Could be almost anything causing your problem. A couple example X-Spam-Statuses
might be a start..
Beyond that, well, if you're running anything older than 2.64 you're probably so
outdated that modern obfuscation techniques are effective against your version
of SA.
Realistically, SA needs to be updated about once a year, unless you're putting a
lot of hand-tuning and custom-rule writing into it.
Re: Re(2): Spamassassin scoring bad after years of service......
Posted by Loren Wilton <lw...@earthlink.net>.
You really also need to post a spam, complete with rule hits, showing scores
you don't like, so we can get some idea of why suddenly you would be getting
lower scores than before.
However:
# SpamAssassin config file for version 2.5x
# generated by http://www.yrex.com/spam/spamconfig.php (version 1.01)
If you really are running 2.5x, I think most of us will suggest that that by
itself is probably the reason! Spammers change their tactics daily, and the
defense tool you are using is ancient! There are just any number of
commonly used techniques that 2.5 doesn't have a clue about these days.
Normally I'd suggest adding SARE rules if you don't already have them; but
2.5 is so old that they won't help you a lot.
Basic suggestion: move forward. At LEAST 2.64 if for some reason you don't
want to get a lot newer; but with 3.1 out any day now, even 2.6 isn't going
to be supported very well all that much longer. I would suggest 3.0.4 at
the moment (and not anything between 3.0 and 3.0.3 - they had problems).
Make sure that you have net tests enabled. Make sure that SURBL is working
correctly. Pick up a smattering of rules files from rulesemporium.com.
Make sure that you have trusted_hosts set correctly in your configuration,
and that 'spamassassin --lint' passes. That should pretty much do it for
you.
Loren
Re: Spamassassin scoring bad after years of service......
Posted by Matt Kettler <mk...@evi-inc.com>.
Lorin G. Tremblay wrote:
> You are right sorry about that.
Well, I can't help you diagnose your hit-problems based on a config file. As
said before, at absolute minimum I'd need to see a list of hits, if not a sample
mail.
That said, there's some things you should consider about your config file:
> Mac OS X -- Communigate - SA3.04 - AWL - AUTOLEARN
> # Use terse version of the spam report
> use_terse_report 1
Error: Ancient option, not supported by modern versions of SA, delete this line.
> # Bayes database path
> bayes_path /etc/mail/spamassassin/Bayes/spamassassin
Error: You need bayes_file_mode 0777 when you set a bayes path. Otherwise you
must guarantee that only one user will ever execute any SA tools on your system.
Warning: that path seems a little off. They pretty much always should end in
"bayes". Right now SA will attempt to use the following files:
/etc/mail/spamassassin/Bayes/spamassassin_toks
/etc/mail/spamassassin/Bayes/spamassassin_seen
But I suspect you wanted it to use the ending "spamassassin" as a directory
name, not a filename. The weirdness of "bayes_path" is it's not a path. It's a
path plus partial filename.
As long as you realize that your system uses "spamassassin_toks" instead of
"bayes_toks" you should be fine, but you should be aware of it.
> # Learning minimum value
> bayes_min_spam_num 140
Suggestion: I'd question the wisdom of lowering this below the default of 200,
but by now you should have enough training this is irrelevant. I'd kill the line.
>
> # Mail which scores outside this range will be fed back into SpamAssassin's
> # learning system automatically, to train the Bayesian scanner.
> bayes_auto_learn_threshold_nonspam 1.0
> bayes_auto_learn_threshold_spam 5.0
Warning: your learning threshold of 5.0 is rather low. It's so low SA won't even
honor it, as other hard-coded criteria add up to an absolute minimum of 6.0.
That setting really is asking for the bayes database to go awry unless you've
got manual training to fix mis-learning.
> # default: per-user whitelist:
> auto_whitelist_path ~/.spamassassin/auto-whitelist
> auto_whitelist_file_mode 0600
Error: the mode should be 0700 not 0600. SA requires the X bit on all the
"file_mode" options as they are sometimes used in directory creation.
Fortunately these circumstances are rare, but it's worth fixing.
> # use this for a system-wide whitelist:
> auto_whitelist_path /etc/mail/spamassassin/Autowhitelist/auto-whitelist
> auto_whitelist_file_mode 0666
Error: the mode should be 0777 not 0666.
Suggestion: comment out One or the other of these pairs of AWL declarations. You
can't be both site-wide and per-user. Currently you'll end up using the
site-wide settings, so you probably ought comment out the first set.
> score HTML_WEB_BUGS 12.0
Warning: This could cause considerable FP problems.
Compare that score to the default this rule gets:
score HTML_WEB_BUGS 0.166 0.013 0.311 0.035
And look at STATISTICS.txt, which says this rule has a S/O of 0.895. (ie: 10.5%
of all mail hitting this rule is nonspam) which is why it's scored so low.
Are you sure that rule should have such a high score?
Re(2): Spamassassin scoring bad after years of service......
Posted by "Lorin G. Tremblay" <lo...@compoart.com>.
You are right sorry about that.
I started to notice the problem a couple of weeks ago a
Mac OS X -- Communigate - SA3.04 - AWL - AUTOLEARN
Here are my settings:
# SpamAssassin config file for version 2.5x
# generated by http://www.yrex.com/spam/spamconfig.php (version 1.01)
# How many hits before a message is considered spam.
required_score 4.0
# Whether to change the subject of suspected spam
rewrite_header Subject *****SPAM*****
# Encapsulate spam in an attachment
report_safe 0
# Use terse version of the spam report
use_terse_report 1
###########################################################################
## Bayes settings
# Enable the Bayes system
use_bayes 1
# Bayes database path
bayes_path /etc/mail/spamassassin/Bayes/spamassassin
# Enable Bayes auto-learning
# Set this to 0 to turn off auto-learning.
bayes_auto_learn 1
# Learning minimum value
bayes_min_spam_num 140
# Mail which scores outside this range will be fed back into SpamAssassin's
# learning system automatically, to train the Bayesian scanner.
bayes_auto_learn_threshold_nonspam 1.0
bayes_auto_learn_threshold_spam 5.0
###########################################################################
###########################################################################
# Automatic-whitelist directory, for the default db-based whitelist
# backend. By default, each user has their own, in their ~/.spamassassin
# directory with mode 0600, but for system-wide SpamAssassin use, you may
# want to share this across all users; uncomment and customise the below
# lines. (Make sure the mode has --x bits set.)
auto_whitelist_factor 0.5
# default: per-user whitelist:
auto_whitelist_path ~/.spamassassin/auto-whitelist
auto_whitelist_file_mode 0600
# use this for a system-wide whitelist:
auto_whitelist_path /etc/mail/spamassassin/Autowhitelist/auto-whitelist
auto_whitelist_file_mode 0666
##########################################################################
##########################################################################
# Headers to be added to mail (can be overriden by users in
# ~/.spamassassin/user_prefs)
# Make sure the header stays entirely on one line here
clear_headers
#X-Spam-Checker-Version: SpamAssassin 2.60-cvs (1.188-2003-05-24-exp)
add_header all Checker-Version SpamAssassin _VERSION_ (_SUBVERSION_) on
mail.2smart4u.com
#X-Spam-Flag: YES
add_header spam Flag _YESNOCAPS_
#X-Spam-Level: *************
add_header all Level _STARS(*)_
#X-Spam-Status: Yes, hits=14.0 required=5.0 tests=BAYES_99,CALL_FREE
# DATE_IN_PAST_12_24,DCC_CHECK,DRASTIC_REDUCED,FROM_HAS_MIXED_NUMS
# FROM_HAS_MIXED_NUMS3,HOME_EMPLOYMENT,INVALID_DATE,INVALID_MSGID
# LINES_OF_YELLING,MSGID_HAS_NO_AT,NO_REAL_NAME,ONCE_IN_LIFETIME
# REMOVE_SUBJ,SMTPD_IN_RCVD,T_INVALID_DATE_2,T_INVALID_DATE_HMS
# UNDISC_RECIPS autolearn=spam version=2.60-cvs
add_header all Status "_YESNO_, hits=_HITS_ required=_REQD_
tests=_TESTS_ autolearn=_AUTOLEARN_ version=_VERSION_"
###########################################################################
# Enable or disable network checks
skip_rbl_checks 1
use_razor2 1
use_dcc 0
use_pyzor 1
dns_available yes
# Mail using languages used in these country codes will not be marked
# as being possibly spam in a foreign language.
ok_languages en fr
# Mail using locales used in these country codes will not be marked
# as being possibly spam in a foreign language.
ok_locales en
score HTML_WEB_BUGS 12.0
.....................................................................
Lorin G. Tremblay lorin@compoart.com
Compo Art L.G.T. Inc. www.compoart.com
1005A Wellington Tel: 514-398-9701
Montreal (Quebec) H3C 1V3 Fax: 514-398-9704
.....................................................................
The nature of change
"All things change. Yet nothing is extinguished... there is nothing in the
whole world which is permanent. Everything flows onwards and all things
are brought into being with a changing nature. The ages themselves glide
by in constant movement, for still waters will never reach the sea."
>Lorin G. Tremblay wrote:
>> Was wondering if anyone had any clue to why spamassassin would start to
>> score spam badly and let almost any spam throught.
>> Tehy was no change in the hardware of software, it just started to score
>> spam really badly, but had workes for atleast a full year without any
>glitch!
>
>Unfortunately, we are not psychic and cannot determine what the problem
>might be with the amount of information you gave us. What version are
>you running? Are you using AWL, Whitelisting, Autolearning, SQL Based
>Prefs? What platform are you on? What type of spam doesn't get caught
>anymore and what does the SA report say? We need more details to help you.
>
>Jay
>--
>Jay Lee
>Network / Systems Administrator
>Information Technology Dept.
>Philadelphia Biblical University
>--
Re: Spamassassin scoring bad after years of service......
Posted by Jay Lee <jl...@pbu.edu>.
Lorin G. Tremblay wrote:
> Was wondering if anyone had any clue to why spamassassin would start to
> score spam badly and let almost any spam throught.
> Tehy was no change in the hardware of software, it just started to score
> spam really badly, but had workes for atleast a full year without any glitch!
Unfortunately, we are not psychic and cannot determine what the problem
might be with the amount of information you gave us. What version are
you running? Are you using AWL, Whitelisting, Autolearning, SQL Based
Prefs? What platform are you on? What type of spam doesn't get caught
anymore and what does the SA report say? We need more details to help you.
Jay
--
Jay Lee
Network / Systems Administrator
Information Technology Dept.
Philadelphia Biblical University
--
Spamassassin scoring bad after years of service......
Posted by "Lorin G. Tremblay" <lo...@compoart.com>.
Hi!
Was wondering if anyone had any clue to why spamassassin would start to
score spam badly and let almost any spam throught.
Tehy was no change in the hardware of software, it just started to score
spam really badly, but had workes for atleast a full year without any glitch!
Any pointer would be great!
.....................................................................
Lorin G. Tremblay lorin@compoart.com
Compo Art L.G.T. Inc. www.compoart.com
1005A Wellington Tel: 514-398-9701
Montreal (Quebec) H3C 1V3 Fax: 514-398-9704
.....................................................................
The nature of leadership
"You will find true success in those efforts that captivate your heart and
soul. Belief fuels passion, and passion rarely fails."
Re: Basic Question: RE syntax? (was Re: V3.0.2 vs. 2.63 detection
rate decrease (?))
Posted by Matt Kettler <mk...@evi-inc.com>.
Linda A. Walsh wrote:
> Have a different question -- and I feel a bit embarrassed to ask this but
> it isn't obvious from the manpage.
>
> What type of Regular Expressions go into the the white/black list entries?
> filename type RE's (what I thought more people would be familiar with, but
> on further thought, didn't really make alot of sense for something written
> in perl), or perl RE's (which I, personally, prefer, but are much more
> "arcane" to the average, non-perl-literate, end-user)?
whitelist_from, whitelist_to, etc take file-glob format. (ie: *@evi-inc.com)
I assume that's what you mean by "filename type RE's" even though it is
definitively not a form of Regular Expression at all.
> Are they the powerful perl RE's, or did someone actually put in extra
> effort toward simplifying (dumbing down) user-design of "RE's".
Effort was put in to dumb it down and make it not based on Regular Expressions
of any sort, which are generally more powerful but foreign to many.
(That said, if you're a sysadmin I'd strongly suggest learning basic regex
syntax. "grep" and many other command-line sysadmin tools become so much more
powerful if you know how to create a regex.)
Basic Question: RE syntax? (was Re: V3.0.2 vs. 2.63 detection rate
decrease (?))
Posted by "Linda A. Walsh" <sa...@tlinx.org>.
Have a different question -- and I feel a bit embarrassed to ask this but
it isn't obvious from the manpage.
What type of Regular Expressions go into the the white/black list entries?
filename type RE's (what I thought more people would be familiar with, but
on further thought, didn't really make alot of sense for something written
in perl), or perl RE's (which I, personally, prefer, but are much more
"arcane" to the average, non-perl-literate, end-user)?
Are they the powerful perl RE's, or did someone actually put in extra
effort toward simplifying (dumbing down) user-design of "RE's". If not,
that's fine for my personal use, but it might be a "_footnote_" for
usability
on a "GUI", _someday_, to allow for WIN32 or POSIX style filename filename
syntax specifications that would be internally converted to perl RE's in the
userprefs file.
Thanks -- and, *ahem*, just example of usage or a pointer to the perlre man
page would be a useful cross reference.
Linda
Loren Wilton wrote:
>First off I'd make sure you ran lint to see if you still have any old 2.6x
>options in your 3.0 config. That seems to have happened to a lot of people,
>and can screw things up.
>
---
Yeppers....this was the main problem ... my headers weren't getting
rewritten/marked as before, so they weren't getting diverted into the
proper folder. Thanks...still have to upgrade to 3.0.4, but have the
'technology' (downloaded), still working out the 'bugs' from the upgrade
to SuSE 9.3 (various packages didn't upgrade well, and some didn't upgrade
at all...etc; have to go through .rpmnew/save/old changes...make sure my
DNS server is still updating (last time I upgraded, DNS failed after
some TLD's failed at around 1 week...*doh*!))...
Re: V3.0.2 vs. 2.63 detection rate decrease (?)
Posted by Loren Wilton <lw...@earthlink.net>.
First off I'd make sure you ran lint to see if you still have any old 2.6x
options in your 3.0 config. That seems to have happened to a lot of people,
and can screw things up.
In general 3.0.4 seems to be better than 2.64, which was the expected end
goal. The earlier 3.0 and 3.0.x releases had some assorted
problems/annoyances. Perhaps the worst of these was the low score assigned
to bayes_99 that made it near useless.
I would upgrade your new 3.0.2 installation to 3.0.4. I believe that 3.0.2
has a known DOS vunerability. Also, the bayes_99 score has been fixed in
3.0.4, as have several other things.
If you are using SARE rules, make sure to check them. Several rule files
need to change when going from 2.6x to 3.0.x.
With all of the above done, you really should be getting better detection on
3.0.4 than on 2.64.
The only thing that seems to be happening that causes some FNs for some
people is if you have user rules. Occasionally you will see an 'insecure
dependency' line in your log, and this will usually correspond to a spam
that slips through without getting scored. YOu should look at the FNs you
have been getting and see if they have SA scores, or if they made it through
unscored.
Loren