You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by Jeff Chan <je...@surbl.org> on 2004/04/17 00:46:46 UTC
Bill Stearns' sa-blacklist available as SURBL: ws.surbl.org
I probably should have introduced this second SURBL list
that can be used together with or in place of sc.surbl.org
before mentioning that its name was changing from sa.surbl.org
to ws.surbl.org. :-) Note that the two lists have different
data sources, so strictly speaking one is not a replacement for
the other. They're two different lists. sc uses URI domains
from SpamCop reports. The data source for ws data is described
below. Both lists have merits and we'd encourage you to consider
trying both.
Here's an announcement with the additional update that
we've changed the *sample rule names* for the ws list to use
"WS" instead of "SA":
__
http://www.surbl.org/ (with some live links)
More SURBL lists
In addition to the first SpamCop URI-derived SURBL sc.surbl.org, we
are pleased to host another RBL compatible with the SpamCopURI or
URIDNSBL SpamAssassin plugins, or any other software that can
check message body domains against a name-based RBL. Data for the
second SURBL ws.surbl.org comes from the domains in Bill Stearns'
SpamAssassin blacklist: sa-blacklist. This is a large list of
spam domains, including those found in spam message body URIs.
Both ws.surbl.org and sc.surbl.org SURBLs can be used in the same
SA installation by using two sets of rules.
An SA 2.63 rule and score using SpamCopURI (but not the SpamCop
data!) looks like this:
uri WS_URI_RBL eval:check_spamcop_uri_rbl('ws.surbl.org','127.0.0.2')
describe WS_URI_RBL URI's domain appears in spamcop database at ws.surbl.org
tflags WS_URI_RBL net
score WS_URI_RBL 3.0
An SA 3.0 rule and score using URIBL's urirhsbl looks like this:
urirhsbl URIBL_WS_SURBL ws.surbl.org. A
header URIBL_WS_SURBL eval:check_uridnsbl('URIBL_WS_SURBL')
describe URIBL_WS_SURBL Contains a URL listed in the WS SURBL blocklist
tflags URIBL_WS_SURBL net
score URIBL_WS_SURBL 3.0
More details about ws.surbl.org are available in the section
"Additional SURBLs for spam URI testing" (copied below).
Please note that the name of this list is being changed from
sa.surbl.org to ws.surbl.org. If you were using the old name in
your rules please update them to the new name.
...
Additional SURBLs for spam URI testing
Additional SURBLs that list domains occurring in spam message
bodies may be used with the same routines that use the
sc.surbl.org RBL.
sa-blacklist available as RBL: ws.surbl.org
In cooperation with Bill Stearns, SURBL is making his
sa-blacklist SpamAssassin blacklist available as the RBL
ws.surbl.org. It can be used in the same way as sc.surbl.org, for
example by adding urirhsbl and SpamCopURI rules as described in
the Quick Start section at the top of this document. Like sc,
ws.surbl.org is available through DNS and, for large-volume mail
servers, as rsynced BIND and rbldns zone files. Raymond
Dijkxhoorn has graciously agreed to host the ws.surbl.org zone
files from his rsync server along with sc.surbl.org's. Please
contact him at rsync@surbl.org for rsync access.
Both sc and ws RBLs can be used in the same installation. The
choice of using either or both or none is yours. Their data
differs somewhat, and we'll try to briefly describe and link some
of the differences here. Bill's list is rather large at about
9600 domains. It consists of domains found in spam message body
URIs and some spam sender and spam operator domains. Given that
the former are more relevant to isolate these days, most of the
recent additions to Bill's list have been URI domains. Those are
also the domains most relevant for use with the message body
checking approach which we propose throughout this site.
The data in sa-blacklist and therefore ws.surbl.org differ from
the SpamCop URI report data described above in that the list is
about ten times larger, more stable, and may have a slightly
higher false positive rate. Bill's policy for inclusion and
cleaning of the sa-blacklist is quite sound, however, so folks
should feel comfortable giving this list a try in addition to the
sc list. ws may currently detect some spam that sc misses, and
vice versa, but it's worth mentioning that the current sc is a
working prototype and that we expect the performance of sc to
improve as we tune the sc data engine further. sc just got out of
the gate, yet it already has some worthy competition in ws.
Thanks Bill!
Because ws is larger and more stable, the zone files for it gets
a six hour TTL compared to 10 minutes for sc. Due to the
differences between the time scales, sizes, and data sources of
ws and sc, we probably won't be offering a combined ws plus sc
list. For example it would be difficult to say what TTL a merged
list should get, and you probably would not want a megabyte plus
BIND zone file refreshing every 10 minutes. For those using
rsynced zone files that would probably not be an issue, but for
those using BIND, the DNS traffic quite well could be.
We encourage you to give ws.surbl.org a try.
Please note that the name of this list is being changed from
sa.surbl.org to ws.surbl.org. If you were using the old name in
your rules please update them to the new name.
Jeff C.
--
Jeff Chan
mailto:jeffc@surbl.org
http://www.surbl.org/
Re: Bill Stearns' sa-blacklist available as SURBL: ws.surbl.org
Posted by William Stearns <ws...@pobox.com>.
Good morning, Don,
On Tue, 20 Apr 2004, Don Newcomer wrote:
> Wow, I implemented this yesterday and, after about 18 hours, this rule was
> #4 in my list of most-used rules! Good job, guys!
Fantastic! How are the system cpu load and memory usage at the
moment? I assume your enthusiasm means they're both in acceptable ranges?
Cheers,
- Bill
---------------------------------------------------------------------------
"... It ... could ... work!"
-- Gene Wilder in Young Frankenstein
--------------------------------------------------------------------------
William Stearns (wstearns@pobox.com). Mason, Buildkernel, freedups, p0f,
rsync-backup, ssh-keyinstall, dns-check, more at: http://www.stearns.org
--------------------------------------------------------------------------
Re: Bill Stearns' sa-blacklist available as SURBL: ws.surbl.org
Posted by David B Funk <db...@engineering.uiowa.edu>.
On Wed, 21 Apr 2004, Simon Byrnand wrote:
> The other reason why its good to keep tests with different source data
> apart is that you can score them seperately based on your confidence of
> the data, and when both tests hit for the same message the scores ADD.
> That won't happen with a combined list, you get all or nothing.
>
> For example I score sc.surbl.org at 4.0 and ws.surbl.org at 2.5, and if
> BOTH tests hit for the same message it amounts to 6.5.
It is possible to structure a 'merged' zone such that one DNS lookup
reflects multiple independent lists. Check out how the MAPS RBL+
zone works. (see the SA routines 'check_rbl' and 'check_rbl_sub'
are used to "pick apart" a multi-valued zone).
--
Dave Funk University of Iowa
<dbfunk (at) engineering.uiowa.edu> College of Engineering
319/335-5751 FAX: 319/384-0549 1256 Seamans Center
Sys_admin/Postmaster/cell_admin Iowa City, IA 52242-1527
#include <std_disclaimer.h>
Better is not better, 'standard' is better. B{
Re: Bill Stearns' sa-blacklist available as SURBL: ws.surbl.org
Posted by Simon Byrnand <si...@igrin.co.nz>.
>
> Question (don't know if the right people will see this here on SA-Talk):
>
> Would it be possible to have 'surbl.org' run a *combined* blacklist, so
> that people who want to check both 'ws.surbl.org' *and* 'sc.surbl.org' can
> do it with ONE dns lookup request, instead of two?
I can think of a couple of reasons why thats not really necessary. (apart
from the line of reasoning in Jeff's reply)
DNS lookups aren't really that much overhead providing that the
nameservers aren't slow - don't forget that if you have RBL tests enabled
on SpamAssassin, which most people do, then it is already doing about 10
lookups (each different RBL list) *PER* ip address found in the received
headers, so one extra lookup per unique URL found would not be that
significant.
If the nameservers *are* slow, then its going to be slow whether you do
one or two lookups :)
DNS lookups can be done in parallel so that the latency is only that of
the slowest responding dns server rather than all of them one after
another. (I'm not sure whether SpamCopURI or SA 3.0 currently actually do
the DNS lookups in parallel like the main RBL code does, but even if they
don't, they probably will in the future, when the code is optimized a bit
more)
Also your local caching nameserver will remember recently looked up hits
and misses - and if you run a mailserver that handles lots of users (an
ISP for example) then you'll usually find that each different kind of spam
comes in runs - where you'll get the same spam delivered to dozens or even
hundreds of users in a short period of time - and your nameserver will be
able to respond very quickly with cached replies for all but the first one
of the group.
(Bill Stearns ws list has quite a long TTL, 6 hours I think, although the
main list sc has a fairly short one, about 5 minutes...)
The other reason why its good to keep tests with different source data
apart is that you can score them seperately based on your confidence of
the data, and when both tests hit for the same message the scores ADD.
That won't happen with a combined list, you get all or nothing.
For example I score sc.surbl.org at 4.0 and ws.surbl.org at 2.5, and if
BOTH tests hit for the same message it amounts to 6.5.
Regards,
Simon
Re: Bill Stearns' sa-blacklist available as SURBL: ws.surbl.org
Posted by Jeff Chan <je...@surbl.org>.
On Tuesday, April 20, 2004, 11:10:46 PM, Christian Quest wrote:
>>On Tuesday, April 20, 2004, 6:10:30 PM, Bill Landry wrote:
>>> Jeff, why not one DNS query that supports multiple result codes:
>>> 127.0.0.2 = Spamcop
>>> 127.0.0.3 = WS List
>>> 127.0.0.4 = BigEvil List
>>> 127.0.0.5 = etc...
>>
>>> Same thing multi-test RBLs like AHBL, Sorbs, Blars, FiveTen, NJABL and
>>> others do.
>>
>>Yes, we may end up doing that in a combined list.
> Or even:
> 127.0.0.2 = listed on list only
> 127.0.0.3 = listed in two lists
> 127.0.0.4 = listed in 3 lists...
> etc...
> So that it gives some scoring criteria as well...
An interesting idea. The nice thing about .2, .4, .8, etc
is that by setting bits, it can be clear what the particular
source of hits is, i.e. which lists are matching a given domain.
> It would also be great to have this available for other RBLs reducing
> the queries and providing at the same time some basic scoring.
Indeed, but there aren't too many message body domain lists to
turn into SURBLs. Most lists deal with sender domains, open
relays, etc. Those can't really be used where SURBLs are used,
i.e., to check message body URIs.
With mine, Bill's, Chris and Paul's I think we have most of the
publically known ones, though I'm interested in adding more
if there are any low-FP lists that anyone would like to publish
as an SURBL.
Jeff C.
--
Jeff Chan
mailto:jeffc@surbl.org
http://www.surbl.org/
Re: Bill Stearns' sa-blacklist available as SURBL: ws.surbl.org
Posted by Christian Quest <li...@cquest.org>.
>On Tuesday, April 20, 2004, 6:10:30 PM, Bill Landry wrote:
>> ----- Original Message -----
>> From: "Jeff Chan" <je...@surbl.org>
>
>>> So the quick answer is they'll probably not be combined.
>>>
>>> However we probably will offer a combined version of Bill's
>>> list and Chris' BigEvil list since they are more similar in
>>> character.
>
>> Jeff, why not one DNS query that supports multiple result codes:
>
>> test: somedomain.com.sc.surbl.org
>
>> results:
>
>> 127.0.0.2 = Spamcop
>> 127.0.0.3 = WS List
>> 127.0.0.4 = BigEvil List
>> 127.0.0.5 = etc...
>
>> Same thing multi-test RBLs like AHBL, Sorbs, Blars, FiveTen, NJABL and
>> others do.
>
>Yes, we may end up doing that in a combined list.
Or even:
127.0.0.2 = listed on list only
127.0.0.3 = listed in two lists
127.0.0.4 = listed in 3 lists...
etc...
So that it gives some scoring criteria as well...
It would also be great to have this available for other RBLs reducing
the queries and providing at the same time some basic scoring.
Best,
--
Christian Quest (cquest@cquest.org)
Les cotes - 74910 Seyssel (France) Mobile: +33 6 09 04 01 56
Web: http://www.cquest.org/cq/ iChat/AIM: cricrixquest
Re: Bill Stearns' sa-blacklist available as SURBL: ws.surbl.org
Posted by Jeff Chan <je...@surbl.org>.
On Tuesday, April 20, 2004, 6:10:30 PM, Bill Landry wrote:
> ----- Original Message -----
> From: "Jeff Chan" <je...@surbl.org>
>> So the quick answer is they'll probably not be combined.
>>
>> However we probably will offer a combined version of Bill's
>> list and Chris' BigEvil list since they are more similar in
>> character.
> Jeff, why not one DNS query that supports multiple result codes:
> test: somedomain.com.sc.surbl.org
> results:
> 127.0.0.2 = Spamcop
> 127.0.0.3 = WS List
> 127.0.0.4 = BigEvil List
> 127.0.0.5 = etc...
> Same thing multi-test RBLs like AHBL, Sorbs, Blars, FiveTen, NJABL and
> others do.
Yes, we may end up doing that in a combined list.
Jeff C.
Re: Bill Stearns' sa-blacklist available as SURBL: ws.surbl.org
Posted by Bill Landry <Bi...@Pointshare.com>.
----- Original Message -----
From: "Jeff Chan" <je...@surbl.org>
> Good question, which Matt asks also. Here's my response :-)
>
> > The data in sa-blacklist and therefore ws.surbl.org differ from
> > the sc.surbl.org SpamCop URI report data described in the Data
> > section in that the list is about twenty times larger, more
> > stable, and may have a slightly higher false positive rate.
> > Bill's policy for inclusion and cleaning of the sa-blacklist is
> > quite sound, however, so folks should feel comfortable giving
> > this list a try in addition to the sc list. ws may currently
> > detect some spam that sc misses, and vice versa, but it's worth
> > mentioning that the current sc is a working prototype and that
> > we expect the performance of sc to improve as we tune the sc
> > data engine further. sc just got out of the gate, yet it
> > already has some worthy competition in ws. Thanks Bill!
> >
> > Because ws is larger and more stable, the zone files for it
> > gets a six hour TTL compared to 10 minutes for sc. Due to the
> > differences between the time scales, sizes, and data sources of
> > ws and sc, we probably won't be offering a combined ws plus sc
> > list. For example it would be difficult to say what TTL a
> > merged list should get, and you probably would not want a
> > megabyte plus BIND zone file refreshing every 10 minutes. For
> > those using rsynced zone files that would probably not be an
> > issue, but for those using BIND, the DNS traffic quite well
> > could be.
>
> So the quick answer is they'll probably not be combined.
>
> However we probably will offer a combined version of Bill's
> list and Chris' BigEvil list since they are more similar in
> character.
Jeff, why not one DNS query that supports multiple result codes:
test: somedomain.com.sc.surbl.org
results:
127.0.0.2 = Spamcop
127.0.0.3 = WS List
127.0.0.4 = BigEvil List
127.0.0.5 = etc...
Same thing multi-test RBLs like AHBL, Sorbs, Blars, FiveTen, NJABL and
others do.
Bill
Re: Bill Stearns' sa-blacklist available as SURBL: ws.surbl.org
Posted by Jeff Chan <je...@surbl.org>.
On Tuesday, April 20, 2004, 2:19:29 PM, Matthew Trent wrote:
> On Tuesday 20 April 2004 01:59 pm, Jeff Chan wrote:
>> So the quick answer is they'll probably not be combined.
>>
>> However we probably will offer a combined version of Bill's
>> list and Chris' BigEvil list since they are more similar in
>> character.
>>
>> Jeff C.
> Bummer...
However Daniel Quinlan has asked Justin Mason if they can do
a combined lookup in URIDNSBL so that multiple lists are queried
with a single command. That may accomplish a similar effect
on the client side in SA 3.0.
Jeff C.
Re: Bill Stearns' sa-blacklist available as SURBL: ws.surbl.org
Posted by Matthew Trent <mt...@localaccess.com>.
On Tuesday 20 April 2004 01:59 pm, Jeff Chan wrote:
> So the quick answer is they'll probably not be combined.
>
> However we probably will offer a combined version of Bill's
> list and Chris' BigEvil list since they are more similar in
> character.
>
> Jeff C.
Bummer...
--
Matt
Systems Administrator
Local Access Communications
360.330.5535
Re: Bill Stearns' sa-blacklist available as SURBL: ws.surbl.org
Posted by Jeff Chan <je...@surbl.org>.
On Tuesday, April 20, 2004, 1:20:05 PM, Charles Gregory wrote:
> Would it be possible to have 'surbl.org' run a *combined* blacklist, so
> that people who want to check both 'ws.surbl.org' *and* 'sc.surbl.org' can
> do it with ONE dns lookup request, instead of two?
Good question, which Matt asks also. Here's my response :-)
> The data in sa-blacklist and therefore ws.surbl.org differ from
> the sc.surbl.org SpamCop URI report data described in the Data
> section in that the list is about twenty times larger, more
> stable, and may have a slightly higher false positive rate.
> Bill's policy for inclusion and cleaning of the sa-blacklist is
> quite sound, however, so folks should feel comfortable giving
> this list a try in addition to the sc list. ws may currently
> detect some spam that sc misses, and vice versa, but it's worth
> mentioning that the current sc is a working prototype and that
> we expect the performance of sc to improve as we tune the sc
> data engine further. sc just got out of the gate, yet it
> already has some worthy competition in ws. Thanks Bill!
>
> Because ws is larger and more stable, the zone files for it
> gets a six hour TTL compared to 10 minutes for sc. Due to the
> differences between the time scales, sizes, and data sources of
> ws and sc, we probably won't be offering a combined ws plus sc
> list. For example it would be difficult to say what TTL a
> merged list should get, and you probably would not want a
> megabyte plus BIND zone file refreshing every 10 minutes. For
> those using rsynced zone files that would probably not be an
> issue, but for those using BIND, the DNS traffic quite well
> could be.
So the quick answer is they'll probably not be combined.
However we probably will offer a combined version of Bill's
list and Chris' BigEvil list since they are more similar in
character.
Jeff C.
Re: Bill Stearns' sa-blacklist available as SURBL: ws.surbl.org
Posted by Matthew Trent <mt...@localaccess.com>.
On Tuesday 20 April 2004 01:20 pm, Charles Gregory wrote:
> Question (don't know if the right people will see this here on SA-Talk):
>
> Would it be possible to have 'surbl.org' run a *combined* blacklist, so
> that people who want to check both 'ws.surbl.org' *and* 'sc.surbl.org' can
> do it with ONE dns lookup request, instead of two?
>
> - Charles
Ahh, a man after my own heart. :)
--
Matt
Systems Administrator
Local Access Communications
360.330.5535
Re: Bill Stearns' sa-blacklist available as SURBL: ws.surbl.org
Posted by Charles Gregory <cg...@hwcn.org>.
Question (don't know if the right people will see this here on SA-Talk):
Would it be possible to have 'surbl.org' run a *combined* blacklist, so
that people who want to check both 'ws.surbl.org' *and* 'sc.surbl.org' can
do it with ONE dns lookup request, instead of two?
- Charles
Re: Bill Stearns' sa-blacklist available as SURBL: ws.surbl.org
Posted by Don Newcomer <ne...@dickinson.edu>.
Wow, I implemented this yesterday and, after about 18 hours, this rule was
#4 in my list of most-used rules! Good job, guys!
================================================================================
Don Newcomer Dickinson College
Senior Manager, Systems P.O. Box 1773
newcomer@dickinson.edu Carlisle, PA 17013
Phone: (717) 245-1256
FAX: (717) 245-1690
On Fri, 16 Apr 2004, Jeff Chan wrote:
> I probably should have introduced this second SURBL list
> that can be used together with or in place of sc.surbl.org
> before mentioning that its name was changing from sa.surbl.org
> to ws.surbl.org. :-) Note that the two lists have different
> data sources, so strictly speaking one is not a replacement for
> the other. They're two different lists. sc uses URI domains
> from SpamCop reports. The data source for ws data is described
> below. Both lists have merits and we'd encourage you to consider
> trying both.
>
> Here's an announcement with the additional update that
> we've changed the *sample rule names* for the ws list to use
> "WS" instead of "SA":
> __
>
> http://www.surbl.org/ (with some live links)
>
> More SURBL lists
>
> In addition to the first SpamCop URI-derived SURBL sc.surbl.org, we
> are pleased to host another RBL compatible with the SpamCopURI or
> URIDNSBL SpamAssassin plugins, or any other software that can
> check message body domains against a name-based RBL. Data for the
> second SURBL ws.surbl.org comes from the domains in Bill Stearns'
> SpamAssassin blacklist: sa-blacklist. This is a large list of
> spam domains, including those found in spam message body URIs.
> Both ws.surbl.org and sc.surbl.org SURBLs can be used in the same
> SA installation by using two sets of rules.
>
> An SA 2.63 rule and score using SpamCopURI (but not the SpamCop
> data!) looks like this:
>
> uri WS_URI_RBL eval:check_spamcop_uri_rbl('ws.surbl.org','127.0.0.2')
> describe WS_URI_RBL URI's domain appears in spamcop database at ws.surbl.org
> tflags WS_URI_RBL net
>
> score WS_URI_RBL 3.0
>
> An SA 3.0 rule and score using URIBL's urirhsbl looks like this:
>
> urirhsbl URIBL_WS_SURBL ws.surbl.org. A
> header URIBL_WS_SURBL eval:check_uridnsbl('URIBL_WS_SURBL')
> describe URIBL_WS_SURBL Contains a URL listed in the WS SURBL blocklist
> tflags URIBL_WS_SURBL net
>
> score URIBL_WS_SURBL 3.0
>
> More details about ws.surbl.org are available in the section
> "Additional SURBLs for spam URI testing" (copied below).
>
> Please note that the name of this list is being changed from
> sa.surbl.org to ws.surbl.org. If you were using the old name in
> your rules please update them to the new name.
>
> ...
>
> Additional SURBLs for spam URI testing
>
> Additional SURBLs that list domains occurring in spam message
> bodies may be used with the same routines that use the
> sc.surbl.org RBL.
>
> sa-blacklist available as RBL: ws.surbl.org
>
> In cooperation with Bill Stearns, SURBL is making his
> sa-blacklist SpamAssassin blacklist available as the RBL
> ws.surbl.org. It can be used in the same way as sc.surbl.org, for
> example by adding urirhsbl and SpamCopURI rules as described in
> the Quick Start section at the top of this document. Like sc,
> ws.surbl.org is available through DNS and, for large-volume mail
> servers, as rsynced BIND and rbldns zone files. Raymond
> Dijkxhoorn has graciously agreed to host the ws.surbl.org zone
> files from his rsync server along with sc.surbl.org's. Please
> contact him at rsync@surbl.org for rsync access.
>
> Both sc and ws RBLs can be used in the same installation. The
> choice of using either or both or none is yours. Their data
> differs somewhat, and we'll try to briefly describe and link some
> of the differences here. Bill's list is rather large at about
> 9600 domains. It consists of domains found in spam message body
> URIs and some spam sender and spam operator domains. Given that
> the former are more relevant to isolate these days, most of the
> recent additions to Bill's list have been URI domains. Those are
> also the domains most relevant for use with the message body
> checking approach which we propose throughout this site.
>
> The data in sa-blacklist and therefore ws.surbl.org differ from
> the SpamCop URI report data described above in that the list is
> about ten times larger, more stable, and may have a slightly
> higher false positive rate. Bill's policy for inclusion and
> cleaning of the sa-blacklist is quite sound, however, so folks
> should feel comfortable giving this list a try in addition to the
> sc list. ws may currently detect some spam that sc misses, and
> vice versa, but it's worth mentioning that the current sc is a
> working prototype and that we expect the performance of sc to
> improve as we tune the sc data engine further. sc just got out of
> the gate, yet it already has some worthy competition in ws.
> Thanks Bill!
>
> Because ws is larger and more stable, the zone files for it gets
> a six hour TTL compared to 10 minutes for sc. Due to the
> differences between the time scales, sizes, and data sources of
> ws and sc, we probably won't be offering a combined ws plus sc
> list. For example it would be difficult to say what TTL a merged
> list should get, and you probably would not want a megabyte plus
> BIND zone file refreshing every 10 minutes. For those using
> rsynced zone files that would probably not be an issue, but for
> those using BIND, the DNS traffic quite well could be.
>
> We encourage you to give ws.surbl.org a try.
>
> Please note that the name of this list is being changed from
> sa.surbl.org to ws.surbl.org. If you were using the old name in
> your rules please update them to the new name.
>
> Jeff C.
> --
> Jeff Chan
> mailto:jeffc@surbl.org
> http://www.surbl.org/
>