You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by Matt Kettler <mk...@evi-inc.com> on 2006/02/16 00:27:04 UTC
Over-scoring of SURBL lists...
All this hubub about not filtering the list has made me come to a realization.
The SURBL URIBLs are collectively massively over-scored in SA 3.1.0.
The problem lies in the SURBL lists, over time, become largely redundant with
one another. A URI may be first listed by one or another of the SURBL lists, as
each has their own feeds, but if it's really used in spam run it will quickly
get listed in 3 or more of them.
Take for example this ONE uri that was posted to the list:
checpri *MUNGED*.com
This is currently listed in SC, JP, and AB on SURBL.
score URIBL_AB_SURBL 0 3.306 0 3.812
score URIBL_JP_SURBL 0 3.360 0 4.087
score URIBL_SC_SURBL 0 3.600 0 4.498
In a set3 configuration that's 12.397 points, just for having ONE URI in the
message. I don't know about you, but it strikes me as rather excessive.
Compare this to the RBLs supported by 3.1.0. XBL is the highest scoring RBL and
it's only 3.8.. You'd have to be listed in at least 5 RBLs to break 12 points
with SA 3.1.0.. The highest four scoring RBLs are:
score RCVD_IN_XBL 0 3.114 0 3.897
score RCVD_IN_NJABL_SPAM 0 1.905 0 2.775
score RCVD_IN_DSBL 0 1.801 0 2.600
score RCVD_IN_WHOIS_BOGONS 0 1.811 0 2.430
----------
11.702
Also consider that these lists have highly diverse listing criteria, and merely
sourcing spam is not enough to get listed in all of these 4.
Yet a mere 3 URIBLs sails right past the 12 point mark with ease. And these 3
URIBLs (as well as OB and uribl.com's BLACK) all have highly similar listing
criteria. They all list on slightly different policies, but when you remove the
fine details they all list based on "domains reported as spam which don't appear
to be used legitimately". The differences exist in where they collect reports
from, and how much checking they do for legitimacy.
The other problem is that I've seen a repeated pattern where FP's get reported
to more than one list. In fact, I rarely see a FP that isn't at least
double-listed. (For example I had the download site for paid-registered upgrades
to a programmer's text editor get double-listed recently. It will cost you
$39.95 to get signed up for that "spam")
This makes me wonder if SA wouldn't be better off having some kind of meta rules
that simply count how many URIBLs the message is listed in, or at least some
kind of score-limiting feedback on multiple hits. This would allow lists to
score high individually, but prevent overlapping FPs from being driven into
astronomical score levels just for containing a single URI that someone
mis-reported to multiple sources.
Thoughts, concepts?
Re: Over-scoring of SURBL lists...
Posted by Rune Kristian Viken <ru...@basefarm.no>.
On Thursday 16 February 2006 21:05, jdow wrote:
> ...
>> The URL-lists are made in a different manner.
>>
>> Take for example - a fully legit message from one friend to another
>> that contains something like this:
>>
>>
>> Hi $name, god I'm getting tired of all the spam we're receiving about
>> http://uri-here/uri , I've noticed that they're actually located in the
>> neighbourhood - maybe we should pay them a visit and tell them what we
>> think about them?
>>
>> regards,
>> $name
>> .. that message is fully legit, and the only "wrong" thing about it is
>> the mentioning of the uri. It'll get blocked, while still violating
>> nothing more than one rule that in practice is repeated multiple times.
> Rune, there are two canonical means of solving that petty issue. If
> there is someone likely to send you such a message white list her. Or
> simply munge the name, for example http://uri-here-M/uri/.
Certainly, the only problem with your approach is that we're suddenly set
several years back with that apporach. The great thing with spamassassin
has always been that it's usually not "one single factor" that triggers
whether the message is categorized as spam or not. It's a combination of
multiple factors.
Furthermore, while both you and I fully understand your nice example, a lot
of users will not. We're back to users having to understand munging,
manually having to edit whitelists, and so forth.
This issue isn't very important to me, but I would think of it as sad if
spamsassassin became less usefull by default due to overzealousness. For
me it's easy enough just to edit the rules to suit my needs and remove the
over-board URI checks, but ..
--
Rune Kristian Viken
Basefarm AS
Tlf: (+47) 98 28 28 41
Re: Over-scoring of SURBL lists...
Posted by mouss <us...@free.fr>.
jdow a écrit :
> Rune, there are two canonical means of solving that petty issue. If
> there is someone likely to send you such a message white list her. Or
> simply munge the name, for example http://uri-here-M/uri/.
>
I would like to whitelist all legitimate senders. unfortunately, I don't
have their addresses:)
If I could whitelist non spammers (or blocklist spammers), I wouldn't
need heuristic/bayesian/blahblahmatic filters.
so the argument: "whitelist is the answer" that I've seen here more than
once is just useless here. once again, the FPs I get are from addresses
I can't whitelist before getting their mail, which gets tagged as spam
before I can whitelist it. ok?
1- friends/collegues/partners/... do change their email address.
2- lost friends can get my email address (from the web or from another
friend)
3- when opt-in a letter, I don't know what addresse they'll use
....
I understand that things may be easy for some people who will just
ignore email. but let me claim to be from the other part (not sure it's
a minority...).
My approach with SA so far was: "if this test generates an FP, look at
it, if it sound silly, disable it". just because some perceptron (or
call it whatever you prefer) gets inputs and religiously generates
scores doesn't mean I should religiously obeit it. if you give more
conditions to the preceptron, it'll generate other results. so let's
stop the "hey, the scores are the optimal result of the preceptron".
also, when someone's filter misses spam, the common answer is 'use
SARE". but it seems to me that SARE is not managed (they don't seem to
care about false positive reports). worst, all SA docs suggest using
external rules, without warning about the dangers....
the default SA rules use sorbs lists. This is now known to be too
aggressive (lists large ISPs for example, even their duhl isn't safe).
ahem...
Re: Over-scoring of SURBL lists...
Posted by jdow <jd...@earthlink.net>.
From: "Rune Kristian Viken" <ru...@basefarm.no>
> On Thursday 16 February 2006 02:22, Bill Landry wrote:
>
>>> This makes me wonder if SA wouldn't be better off having some kind of
>>> meta rules that simply count how many URIBLs the message is listed in, or
>>> at least some kind of score-limiting feedback on multiple hits. This
>>> would allow lists to score high individually, but prevent overlapping FPs
>>> from being driven into astronomical score levels just for containing a
>>> single URI that someone mis-reported to multiple sources.
>>> Thoughts, concepts?
>> So, Matt, why is this any different than an IP addresses that gets listed
>> in many different RBLs and reported by SA?
>
> Not Matt here, but I've got some input on my own.
...
> The URL-lists are made in a different manner.
>
> Take for example - a fully legit message from one friend to another that
> contains something like this:
>
> :::
> Hi $name, god I'm getting tired of all the spam we're receiving about
> http://uri-here/uri , I've noticed that they're actually located in the
> neighbourhood - maybe we should pay them a visit and tell them what we
> think about them?
>
> regards,
> $name
> :::
>
> .. that message is fully legit, and the only "wrong" thing about it is the
> mentioning of the uri. It'll get blocked, while still violating nothing
> more than one rule that in practice is repeated multiple times.
Rune, there are two canonical means of solving that petty issue. If
there is someone likely to send you such a message white list her. Or
simply munge the name, for example http://uri-here-M/uri/.
{o.o}
Re: Over-scoring of SURBL lists...
Posted by Jeff Chan <je...@surbl.org>.
On Thursday, February 16, 2006, 4:59:08 AM, Rune Viken wrote:
> The URL-lists are made in a different manner.
> Take for example - a fully legit message from one friend to another that
> contains something like this:
> :::
> Hi $name, god I'm getting tired of all the spam we're receiving about
> http://uri-here/uri , I've noticed that they're actually located in the
> neighbourhood - maybe we should pay them a visit and tell them what we
> think about them?
> regards,
> $name
> :::
> .. that message is fully legit, and the only "wrong" thing about it is the
> mentioning of the uri. It'll get blocked, while still violating nothing
> more than one rule that in practice is repeated multiple times.
Yes, don't filter your ISP's abuse address (if you are an ISP),
and don't filter your discussion list messages about security,
spam, etc.
QED
Jeff C.
--
Jeff Chan
mailto:jeffc@surbl.org
http://www.surbl.org/
Re: Over-scoring of SURBL lists...
Posted by Rune Kristian Viken <ru...@basefarm.no>.
On Thursday 16 February 2006 02:22, Bill Landry wrote:
>> This makes me wonder if SA wouldn't be better off having some kind of
>> meta rules that simply count how many URIBLs the message is listed in, or
>> at least some kind of score-limiting feedback on multiple hits. This
>> would allow lists to score high individually, but prevent overlapping FPs
>> from being driven into astronomical score levels just for containing a
>> single URI that someone mis-reported to multiple sources.
>> Thoughts, concepts?
> So, Matt, why is this any different than an IP addresses that gets listed
> in many different RBLs and reported by SA?
Not Matt here, but I've got some input on my own.
The great thing about spamassassin is that it uses multiple rules to score
one message as spam. Here one rule is essentially the same.
When we're talking about relays or spamsources that is listed in multiple
BLs - it's the originating host/network that is getting blocked,
furthermore - as other has pointed out - there is a big difference between
the various list.
The URL-lists are made in a different manner.
Take for example - a fully legit message from one friend to another that
contains something like this:
:::
Hi $name, god I'm getting tired of all the spam we're receiving about
http://uri-here/uri , I've noticed that they're actually located in the
neighbourhood - maybe we should pay them a visit and tell them what we
think about them?
regards,
$name
:::
.. that message is fully legit, and the only "wrong" thing about it is the
mentioning of the uri. It'll get blocked, while still violating nothing
more than one rule that in practice is repeated multiple times.
--
Rune Kristian Viken
Basefarm AS
Tlf: (+47) 98 28 28 41
Re: Over-scoring of SURBL lists...
Posted by Jeff Chan <je...@surbl.org>.
On Wednesday, February 15, 2006, 7:00:33 PM, Matt Kettler wrote:
> 2) diversity of criteria:
> SURBL - all lists have nearly identical listing criteria, except ph. All but PH
> are "spotted in spam, doesn't appear to have legit use" and nothing more. JP,
> AB, SC, WS and OB are all effectively the same list with different input points.
The various SURBL lists and URIBL.com may have similar listing
criteria, but their original data sources and processing
technologies (think listing rules/logic) are mostly very
different. That they happen to notice some of the same domains
could be taken as independent verification of spammyness. If so
I think there is a value in having the scores add as they do.
Perhaps a person to address this issue is someone like Henry
Stern whose Perceptron system is used to generate the scores.
OTOH you may have a valid point that most of the other non-URIBL
SA rules are mostly unrelated, whereas the URIBL rules are all
about the same topic: inclusion in URIBL lists. As such, perhaps
the score generation system should not treat them like the other
mostly unrelated rules.
OTOOH the Perceptron scoring system is literally results-driven
at least over the test corpora, and it's often hard to argue with
results.
> DNSBLs - lists have wildly different listing criteria. Some are identical to
> each other, but there are 4 different criteria in the top 5.
And wildly different FP rates. It's not too surprising that some
are scored quite low, while others like XBL are scored relatively
high. It's probably only the low scores that make most of them
slightly useful, unlike lists such as XBL, which is highly
useful.
BTW, if you or anyone finds any FPs on SURBLs, *****please*****
report them to whitelist at surbl dot org. We really need
everyone's help with this. If you use our data please help
improve our community with your feedback! If we could make
one condition of use, that would be it.
Jeff C.
--
Jeff Chan
mailto:jeffc@surbl.org
http://www.surbl.org/
Re: Over-scoring of SURBL lists...
Posted by Matt Kettler <mk...@evi-inc.com>.
Bill Landry wrote:
>>>
>>> So, Matt, why is this any different than an IP addresses that gets
>>> listed in many different RBLs and reported by SA?
>>
>> Read the middle of my message, I already pointed that out as being
>> different.
>
> I read the middle of your message, still don't see the difference.
>
There's a HUGE difference in score, and diversity.
1) Score:
SURBL - high scores on many rules, The highest 3 break the 12 point level.
DNSBLs - moderate scores or small scores on rules. It takes the highest-scoring
5 DNSBLs to exceed 12 points.
2) diversity of criteria:
SURBL - all lists have nearly identical listing criteria, except ph. All but PH
are "spotted in spam, doesn't appear to have legit use" and nothing more. JP,
AB, SC, WS and OB are all effectively the same list with different input points.
DNSBLs - lists have wildly different listing criteria. Some are identical to
each other, but there are 4 different criteria in the top 5.
-score RCVD_IN_XBL 0 3.114 0 3.897
XBL lists exploited machines. To be listed here, you need to be infected or
vulnerable somehow.
-score RCVD_IN_NJABL_SPAM 0 1.905 0 2.775
-score RCVD_IN_DSBL 0 1.801 0 2.600
These two list basically on "spam source" alone as a critera.
-score RCVD_IN_WHOIS_BOGONS 0 1.811 0 2.430
This lists based on having bogus IP address that's unallocated.
-score RCVD_IN_SORBS_SOCKS 0 1.823 0 2.159
This list is based on being an open socks proxy.
Re: Over-scoring of SURBL lists...
Posted by Bill Landry <bi...@pointshare.com>.
----- Original Message -----
From: "Matt Kettler" <mk...@evi-inc.com>
> Bill Landry wrote:
>> ---- Original Message ----- From: "Matt Kettler" <mk...@evi-inc.com>
>>
>>> This makes me wonder if SA wouldn't be better off having some kind of
>>> meta rules
>>> that simply count how many URIBLs the message is listed in, or at
>>> least some
>>> kind of score-limiting feedback on multiple hits. This would allow
>>> lists to
>>> score high individually, but prevent overlapping FPs from being driven
>>> into
>>> astronomical score levels just for containing a single URI that someone
>>> mis-reported to multiple sources.
>>>
>>> Thoughts, concepts?
>>
>> So, Matt, why is this any different than an IP addresses that gets
>> listed in many different RBLs and reported by SA?
>
> Read the middle of my message, I already pointed that out as being
> different.
I read the middle of your message, still don't see the difference.
Bill
Re: Over-scoring of SURBL lists...
Posted by Matt Kettler <mk...@evi-inc.com>.
Bill Landry wrote:
> ---- Original Message ----- From: "Matt Kettler" <mk...@evi-inc.com>
>
>> This makes me wonder if SA wouldn't be better off having some kind of
>> meta rules
>> that simply count how many URIBLs the message is listed in, or at
>> least some
>> kind of score-limiting feedback on multiple hits. This would allow
>> lists to
>> score high individually, but prevent overlapping FPs from being driven
>> into
>> astronomical score levels just for containing a single URI that someone
>> mis-reported to multiple sources.
>>
>> Thoughts, concepts?
>
> So, Matt, why is this any different than an IP addresses that gets
> listed in many different RBLs and reported by SA?
Read the middle of my message, I already pointed that out as being different.
Re: Over-scoring of SURBL lists...
Posted by Bill Landry <bi...@pointshare.com>.
---- Original Message -----
From: "Matt Kettler" <mk...@evi-inc.com>
> This makes me wonder if SA wouldn't be better off having some kind of meta
> rules
> that simply count how many URIBLs the message is listed in, or at least
> some
> kind of score-limiting feedback on multiple hits. This would allow lists
> to
> score high individually, but prevent overlapping FPs from being driven
> into
> astronomical score levels just for containing a single URI that someone
> mis-reported to multiple sources.
>
> Thoughts, concepts?
So, Matt, why is this any different than an IP addresses that gets listed in
many different RBLs and reported by SA? The more the merrier, I say. When
spam is detected by many different external sources and internal tests, this
lends credibility to the spam designation, in my view. Otherwise, you could
make the same meta rule distinction and apply it to RBLs & RHSBLs, as well -
and possibly Pyzor & Razor hits, too.
Just my two cents...
Bill