You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by Kārlis Repsons <ka...@gmail.com> on 2010/02/02 12:47:00 UTC
Avoid ham to be taken for spam
Hi,
in fact, all spam filters are normally designed with an intent to get rid of
spam, not ham, but anyway, I'm confused with my possible chances to miss some
mails for no really valid reason. I've seen that long list in [1], but that
doesn't say much of what should be avoided. For example, I remember somewhere
it was said, that SpamCop may be "too aggressive".
So, how should I understand better what to avoid in what cases? (I'd rather
leave out some dangerous tests and train Bayes filter)
[1] http://spamassassin.apache.org/full/3.0.x/dist/rules/STATISTICS.txt
Re: Avoid ham to be taken for spam
Posted by Marc Perkel <ma...@perkel.com>.
Kārlis Repsons wrote:
> Hi,
> in fact, all spam filters are normally designed with an intent to get rid of
> spam, not ham, but anyway, I'm confused with my possible chances to miss some
> mails for no really valid reason. I've seen that long list in [1], but that
> doesn't say much of what should be avoided. For example, I remember somewhere
> it was said, that SpamCop may be "too aggressive".
> So, how should I understand better what to avoid in what cases? (I'd rather
> leave out some dangerous tests and train Bayes filter)
>
> [1] http://spamassassin.apache.org/full/3.0.x/dist/rules/STATISTICS.txt
>
Different filtering systems have different philosophies relating to the
relative importance of blocking spam vs. passing ham. Some people are
fine with losing some good emails to get rid of spam. Other people are
fine will getting more spam to ensure that good email isn't blocked.
SpamAssassin is more designed to detect spam than to protect ham. I
personally focus on white rules and white lists to ensure good email
gets through. Probably because those customers scream louder and rightly so.
I do think that SA should focus more on ham detection and not just spam
detection. You have to reprogram your brain to get it into the mode of
detecting ham. Ham is often easy to detect. They aren't evading and
hiding like the spammers are.
Re: Avoid ham to be taken for spam
Posted by RW <rw...@googlemail.com>.
On Tue, 2 Feb 2010 11:47:00 +0000
Kārlis Repsons <ka...@gmail.com> wrote:
> Hi,
> in fact, all spam filters are normally designed with an intent to get
> rid of spam, not ham, but anyway, I'm confused with my possible
> chances to miss some mails for no really valid reason. I've seen that
> long list in [1], but that doesn't say much of what should be
> avoided. For example, I remember somewhere it was said, that SpamCop
> may be "too aggressive". So, how should I understand better what to
> avoid in what cases? (I'd rather leave out some dangerous tests and
> train Bayes filter)
Anything particularly dangerous is off or low-scored by default.
When people say that Spam cop is too aggressive (and it's actually much
more reliable than it used to be) they are talking about outright
rejection not scoring.
I suggest that even if you don't want to train Bayes you still use it
with autotraining. The negative scores from Bayes help protect against
FP's.
Re: Avoid ham to be taken for spam
Posted by Mike Cardwell <sp...@lists.grepular.com>.
On 02/02/2010 15:21, Kārlis Repsons wrote:
> By the way, I feel interested in scores. For example, I've set up an automatic
> sorting, which divides spam into three categories: gray, certain, heavy. I was
> looking at that STATISTICS.txt and my first impression about boundaries was:
> {4, 6.6, 8}, 4 being the first valid "spam score". However, currently I have
> {3, 4, 8}, which might be too drastic for later use, just now when I train
> filters and receive no letters, which could easily go as spam. But what would
> be your three scalar combination (given, you only wish to personally check
> "grey" folder)?
Decide how much spam you're willing to manually check, then parse your
logs to determine the largest threshold you could set which wouldn't
exceed that volume.
--
Mike Cardwell : UK based IT Consultant, Perl developer, Linux admin
Cardwell IT Ltd. : UK Company - http://cardwellit.com/ #06920226
Technical Blog : Tech Blog - https://secure.grepular.com/
Spamalyser : Spam Tool - http://spamalyser.com/
Re: Avoid ham to be taken for spam
Posted by jdow <jd...@earthlink.net>.
From: "Marc Perkel" <ma...@perkel.com>
Sent: Tuesday, 2010/February/02 14:48
>
> Mark Martinec wrote:
>> On Tuesday 02 February 2010 18:53:35 Marc Perkel wrote:
>>
>>> If you are worried about losing good email add this rule to your
>>> ruleset:
>>>
>>> header RCVD_IN_HOSTKARMA_W eval:check_rbl_sub('HOSTKARMA-lastexternal',
>>> '127.0.0.1') describe RCVD_IN_HOSTKARMA_W Sender listed in
>>> HOSTKARMA-WHITE
>>> tflags RCVD_IN_HOSTKARMA_W net nice
>>> score RCVD_IN_HOSTKARMA_W -5
>>>
>>> This is a white list and it is very extensive. It will protect a lot of
>>> your ham while blessing very little spam. So if you are ham oriented,
>>> not losing good email, this is a list that will work for you.
>>>
>>
>> We received 52 spam messages per day on the average in January
>> which got a hit on RCVD_IN_HOSTKARMA_W. Luckily we only score
>> it at -0.5 . Of these 52 spam messages, 3.6 messages per day
>> would have been treated as ham had we used a rule score -5
>> and required_hits 5.
>>
>> Examining messages that were blocked but whitelisted by
>> RCVD_IN_HOSTKARMA_W, I found four newsletters in the whole
>> January (not per day), each sent to avg three of our recipients,
>> which could potentially be worth saving. None of them were
>> worth rescuing from a quarantine.
>>
>> In summary, score of -5 for RCVD_IN_HOSTKARMA_W is grossly overrated.
>>
>> Mark
>>
>>
>
> Mark, I think you are missing the important point. This user is more
> interested in getting good emails than rejecting spam. It's a different
> value system than yours. You have to understand it in terms of getting
> ham rather than rejecting spam.
You have a good point there, Marc. An ISP that has many users few of
which even bother to check the spam mailbox which doesn't offer efficient
tools for review might find your whitelist more useful than a group of
people who get the opportunity to review a spam score ordered spam mailbox.
I can scan mine in moments. Scores over 10 are VERY seldom ham, one a year
perhaps. And as it happens few spams here score under 10 of late. So review
is about 2 or 3 minutes plus time to feed them to the Bayes training
repository I have for hams and spams in case I need to restart my Bayes
training processes.
That way I appear to miss nothing *I* consider ham by title or sender.
And yet I waste very little time a day even if I have a couple hams to
pull out of the spam box some days. (Even one is unusual of late.) So
that's a rate of about .1% mis-identified ham and .01% mis-identified
spam.
For those who wonder this is one of my user defined rules I use.
"rewrite_header Subject *****SPAM***** _SCORE(00)_ **"
I like the way that automatically alphabetically sorts spam by score in a
three digit range.
It seems the utility of your rule depends on how savvy those receiving
the mail might be. I'm lucky with only two users both of whom are
reasonably clever in this regard.
Mark, for you, perhaps -5 is overrated. It's adjustable. Given Marc's
rationale for the score I can see it's utility in some situations. As
with any score, when you identify it as a problem, readjust its score.
That is what your local .cf file(s) are for, ya know.
{^_^}
{^_^}
Re: Avoid ham to be taken for spam
Posted by Marc Perkel <ma...@perkel.com>.
Mark Martinec wrote:
> On Tuesday 02 February 2010 18:53:35 Marc Perkel wrote:
>
>> If you are worried about losing good email add this rule to your ruleset:
>>
>> header RCVD_IN_HOSTKARMA_W eval:check_rbl_sub('HOSTKARMA-lastexternal',
>> '127.0.0.1') describe RCVD_IN_HOSTKARMA_W Sender listed in HOSTKARMA-WHITE
>> tflags RCVD_IN_HOSTKARMA_W net nice
>> score RCVD_IN_HOSTKARMA_W -5
>>
>> This is a white list and it is very extensive. It will protect a lot of
>> your ham while blessing very little spam. So if you are ham oriented,
>> not losing good email, this is a list that will work for you.
>>
>
> We received 52 spam messages per day on the average in January
> which got a hit on RCVD_IN_HOSTKARMA_W. Luckily we only score
> it at -0.5 . Of these 52 spam messages, 3.6 messages per day
> would have been treated as ham had we used a rule score -5
> and required_hits 5.
>
> Examining messages that were blocked but whitelisted by
> RCVD_IN_HOSTKARMA_W, I found four newsletters in the whole
> January (not per day), each sent to avg three of our recipients,
> which could potentially be worth saving. None of them were
> worth rescuing from a quarantine.
>
> In summary, score of -5 for RCVD_IN_HOSTKARMA_W is grossly overrated.
>
> Mark
>
>
Mark, I think you are missing the important point. This user is more
interested in getting good emails than rejecting spam. It's a different
value system than yours. You have to understand it in terms of getting
ham rather than rejecting spam.
Re: Avoid ham to be taken for spam
Posted by Mark Martinec <Ma...@ijs.si>.
On Tuesday 02 February 2010 18:53:35 Marc Perkel wrote:
> If you are worried about losing good email add this rule to your ruleset:
>
> header RCVD_IN_HOSTKARMA_W eval:check_rbl_sub('HOSTKARMA-lastexternal',
> '127.0.0.1') describe RCVD_IN_HOSTKARMA_W Sender listed in HOSTKARMA-WHITE
> tflags RCVD_IN_HOSTKARMA_W net nice
> score RCVD_IN_HOSTKARMA_W -5
>
> This is a white list and it is very extensive. It will protect a lot of
> your ham while blessing very little spam. So if you are ham oriented,
> not losing good email, this is a list that will work for you.
We received 52 spam messages per day on the average in January
which got a hit on RCVD_IN_HOSTKARMA_W. Luckily we only score
it at -0.5 . Of these 52 spam messages, 3.6 messages per day
would have been treated as ham had we used a rule score -5
and required_hits 5.
Examining messages that were blocked but whitelisted by
RCVD_IN_HOSTKARMA_W, I found four newsletters in the whole
January (not per day), each sent to avg three of our recipients,
which could potentially be worth saving. None of them were
worth rescuing from a quarantine.
In summary, score of -5 for RCVD_IN_HOSTKARMA_W is grossly overrated.
Mark
Re: Avoid ham to be taken for spam
Posted by Marc Perkel <ma...@perkel.com>.
If you are worried about losing good email add this rule to your ruleset:
header RCVD_IN_HOSTKARMA_W eval:check_rbl_sub('HOSTKARMA-lastexternal', '127.0.0.1')
describe RCVD_IN_HOSTKARMA_W Sender listed in HOSTKARMA-WHITE
tflags RCVD_IN_HOSTKARMA_W net nice
score RCVD_IN_HOSTKARMA_W -5
This is a white list and it is very extensive. It will protect a lot of
your ham while blessing very little spam. So if you are ham oriented,
not losing good email, this is a list that will work for you.
Re: Avoid ham to be taken for spam
Posted by Kārlis Repsons <ka...@gmail.com>.
On Tuesday 02 February 2010 15:08:01 Matus UHLAR - fantomas wrote:
> On 02.02.10 11:47, Kārlis Repsons wrote:
> > in fact, all spam filters are normally designed with an intent to get rid
> > of spam, not ham, but anyway, I'm confused with my possible chances to
> > miss some mails for no really valid reason.
>
> _any_ spam filter can have false positives. If there was spam filter that
> would have no false positives (and would block all spam), there would be no
> spam anymore. If it would not hit any spam, all spammers would change their
> spam not to hit that one.
>
> > I've seen that long list in [1], but that doesn't say much of what should
> > be avoided. For example, I remember somewhere it was said, that SpamCop
> > may be "too aggressive".
>
> Spamcop is perfectly good, if properly used. and I think it's perfect even
> at SMTP time...
>
> > So, how should I understand better what to avoid in what cases? (I'd
> > rather leave out some dangerous tests and train Bayes filter)
>
> training bayes filter should help you much. If you notice any false
> positives, you can use whitelisting ot increasing the required score.
Well, thanks for replies. Sort of answer I expected, at least, didn't hear any
cries about SpamCop and will enable it.
By the way, I feel interested in scores. For example, I've set up an automatic
sorting, which divides spam into three categories: gray, certain, heavy. I was
looking at that STATISTICS.txt and my first impression about boundaries was:
{4, 6.6, 8}, 4 being the first valid "spam score". However, currently I have
{3, 4, 8}, which might be too drastic for later use, just now when I train
filters and receive no letters, which could easily go as spam. But what would
be your three scalar combination (given, you only wish to personally check
"grey" folder)?
Re: Avoid ham to be taken for spam
Posted by Matus UHLAR - fantomas <uh...@fantomas.sk>.
On 02.02.10 11:47, Kārlis Repsons wrote:
> in fact, all spam filters are normally designed with an intent to get rid
> of spam, not ham, but anyway, I'm confused with my possible chances to
> miss some mails for no really valid reason.
_any_ spam filter can have false positives. If there was spam filter that
would have no false positives (and would block all spam), there would be no
spam anymore. If it would not hit any spam, all spammers would change their
spam not to hit that one.
> I've seen that long list in [1], but that doesn't say much of what should
> be avoided. For example, I remember somewhere it was said, that SpamCop
> may be "too aggressive".
Spamcop is perfectly good, if properly used. and I think it's perfect even
at SMTP time...
> So, how should I understand better what to avoid in what cases? (I'd rather
> leave out some dangerous tests and train Bayes filter)
training bayes filter should help you much. If you notice any false
positives, you can use whitelisting ot increasing the required score.
--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Atheism is a non-prophet organization.