Re: [sa-list] Re: Auto-RBL was: Why did this not hit more? (SPF, DKIM, Ironport,

["Dan Mahoney, System Admin" <da...@prime.gushi.org>]

On Mon, 8 Oct 2007, Rob McEwen wrote:

> Therefore, I recommend that you re-think your choices here! Don't let your 
> quest for "guaranteed long-term perfection" keep you from making 
> **substantial** progress today!

Rob,

Then help rally the SA team to include those RBLs that you mentioned in 
the stock config.

Also, rally them to update the documentation on the wiki on how to 
configure SA for third-party DNSBL's, because it 
blows (and refers to years-old versions of SA).  Yes, I know the point of 
a wiki is that ANYONE can update it, but I'm not about to update it with 
information I don't understand for certain.

((Q: This documentation doesn't seem to cover how to configure 
dns-blocklists. It says "Support for these is built-in" but I can't 
believe that all free BL's is called each time a mail is beeing checked. 
There must be a way to configure which to use.

A: You're right. You might look at the [WWW] Mail::SpamAssassin::Conf 
documentation page which I admit doesn't really say how to configure which 
DNSBL to use, or the rules file [WWW] 20_dnsbl_tests.cf, for internal 
details, but no clear examples of how to configure the inclusion of 
various DNSBLs either. For the latest list of DNSBLs you want to be using 
SpamAssassin version 2.63 or 3.0.0-pre2, for the same reason that you 
wouldn't use an out-of-date virus scanner, but that also doesn't really 
have anything to do with the question.))

Finally, rally them to pay attention to the topic I'm proposing here, 
which is: allow users to run their own RBL + feeder so that they can 
auto-rbl and floodgate themselves (and yes, it allows me to combine your 
corpus, plus my corpus, plus HIS corpus) in a scoring config, which is 
FUN...or it lets you say, quite simply "SA said you sent too much spam, 
now sendmail won't listen for X hours per spam run".

<soapbox>

While I've had a long history of getting decent responses from the 
developers on this list some of the time -- nobody has managed to answer 
the questions I've asked in the previous thread:

* can we do something with the ironport headers

* can we do something with the SPF softfail which my MTA registered but SA 
didn't (and why didn't it?)

* can we do something with the X-Originating-IP: 127:1 (is it a legit 
header, or is it there to evade filters?)

* can we fix something about the DKIM_POLICY_SIGNSOME,

* and after I changed the topic: Can we get a plugin that lets us feed our 
own blocklists, currently I get dictionary floods that are enough to 
overload SA (even right now).

and many is the time I've just sent an email out to this list on a given 
topic, seen a lack of useful answer, and shrugged it off.

</soapbox>

--

"Check it out, it's just like Christmas.  Except it sucks."

-Jason Seguerra, 3/2/05

--------Dan Mahoney--------
Techie,  Sysadmin,  WebGeek
Gushi on efnet/undernet IRC
ICQ: 13735144   AIM: LarpGM
Site:  http://www.gushi.org
---------------------------

RE: [sa-list] Re: Auto-RBL was: Why did this not hit more? (SPF, DKIM, Ironport,

[Anthony Kamau <an...@anroet.com>]

> -----Original Message-----
> From: Dan Mahoney, System Admin [mailto:danm@prime.gushi.org]
> Sent: Tuesday, 9 October 2007 7:14 AM
> To: Rob McEwen
> Cc: users@spamassassin.apache.org
> Subject: Re: [sa-list] Re: Auto-RBL was: Why did this not hit more? (SPF,
> DKIM, Ironport,
> 
> On Mon, 8 Oct 2007, Rob McEwen wrote:
> 
> > Therefore, I recommend that you re-think your choices here! Don't let
> your
> > quest for "guaranteed long-term perfection" keep you from making
> > **substantial** progress today!
> 
> Rob,
> 
> Then help rally the SA team to include those RBLs that you mentioned in
> the stock config.
> 
> Also, rally them to update the documentation on the wiki on how to
> configure SA for third-party DNSBL's, because it
> blows (and refers to years-old versions of SA).  Yes, I know the point of
> a wiki is that ANYONE can update it, but I'm not about to update it with
> information I don't understand for certain.

You should update the Wiki nevertheless and append a disclaimer of sorts!
Choosing not to update in fear of appearing clueless is just lame!  If you
believe that what you are posting is halfway valid, then someone else can
update.  This is the sole function of a Wiki as otherwise there'd be no need
for an UPDATE function!!!
.
.
.

Re: [sa-list] Re: Auto-RBL was: Why did this not hit more? (SPF, DKIM, Ironport,

[Bill Landry <bi...@inetmsg.com>]

Dan Mahoney, System Admin wrote:
> On Mon, 8 Oct 2007, Rob McEwen wrote:
> 
>> Therefore, I recommend that you re-think your choices here! Don't let
>> your quest for "guaranteed long-term perfection" keep you from making
>> **substantial** progress today!
> 
> Rob,
> 
> Then help rally the SA team to include those RBLs that you mentioned in
> the stock config.
> 
> Also, rally them to update the documentation on the wiki on how to
> configure SA for third-party DNSBL's, because it blows (and refers to
> years-old versions of SA).  Yes, I know the point of a wiki is that
> ANYONE can update it, but I'm not about to update it with information I
> don't understand for certain.
> 
> ((Q: This documentation doesn't seem to cover how to configure
> dns-blocklists. It says "Support for these is built-in" but I can't
> believe that all free BL's is called each time a mail is beeing checked.
> There must be a way to configure which to use.
> 
> A: You're right. You might look at the [WWW] Mail::SpamAssassin::Conf
> documentation page which I admit doesn't really say how to configure
> which DNSBL to use, or the rules file [WWW] 20_dnsbl_tests.cf, for
> internal details, but no clear examples of how to configure the
> inclusion of various DNSBLs either. For the latest list of DNSBLs you
> want to be using SpamAssassin version 2.63 or 3.0.0-pre2, for the same
> reason that you wouldn't use an out-of-date virus scanner, but that also
> doesn't really have anything to do with the question.))
> 
> Finally, rally them to pay attention to the topic I'm proposing here,
> which is: allow users to run their own RBL + feeder so that they can
> auto-rbl and floodgate themselves (and yes, it allows me to combine your
> corpus, plus my corpus, plus HIS corpus) in a scoring config, which is
> FUN...or it lets you say, quite simply "SA said you sent too much spam,
> now sendmail won't listen for X hours per spam run".
> 
> <soapbox>
> 
> While I've had a long history of getting decent responses from the
> developers on this list some of the time -- nobody has managed to answer
> the questions I've asked in the previous thread:
> 
> * can we do something with the ironport headers
> 
> * can we do something with the SPF softfail which my MTA registered but
> SA didn't (and why didn't it?)
> 
> * can we do something with the X-Originating-IP: 127:1 (is it a legit
> header, or is it there to evade filters?)
> 
> * can we fix something about the DKIM_POLICY_SIGNSOME,
> 
> * and after I changed the topic: Can we get a plugin that lets us feed
> our own blocklists, currently I get dictionary floods that are enough to
> overload SA (even right now).

Why would you be accepting messages to non-existent users?  If you reject these
at the MTA, then SA would never see them and your MTA would not have to deal
with bounces to forged sender addresses (backscatter).

Bill

> and many is the time I've just sent an email out to this list on a given
> topic, seen a lack of useful answer, and shrugged it off.
> 
> </soapbox>
> 
> -- 
> 
> "Check it out, it's just like Christmas.  Except it sucks."
> 
> -Jason Seguerra, 3/2/05
> 
> --------Dan Mahoney--------
> Techie,  Sysadmin,  WebGeek
> Gushi on efnet/undernet IRC
> ICQ: 13735144   AIM: LarpGM
> Site:  http://www.gushi.org
> ---------------------------
> 

Re: [sa-list] Re: Auto-RBL was: Why did this not hit more? (SPF, DKIM, Ironport,

[Rob McEwen <ro...@invaluement.com>]

Dan,

 >Then help rally the SA team to include those RBLs
 >that you mentioned in the stock config.

My RBL (ivmSIP.com) wouldn't work as a default value in SA because it is 
only available via RSYNC or Zone Transfer to subscribers (or... 
currently... "testers" who have specifically requested access).

The other weird thing is that I use SA as a "helper app" in my spam 
filtering and I've custom written my own spam filter. Mostly, I still 
include SA in the mix for SARE rules (& other rules),  as well as 
checksum filtering like RAZAR, etc. But I've turned off all RBL & URIBL 
filtering in SA because I do those on my own and, most of the time, SA 
isn't even needed.

As a result, I pay very little attention to many of the implementation 
details of "RBLs" in SA since I don't personally use them in SA. I have 
enough to worry about without these extra details. However, I'll be 
happy to share some tips that might help others or the SA folks with 
possible improvements in future versions.

First, one thing that I did years ago (and continue to do) is that I'm 
always carefully reviewing lists that I might potentially use and/or am 
already using. For example, if I notice that a particular dnsbl is 
hitting on more and more messages which ultimately score under the spam 
threshold and, upon examination, I verify that most or all of these 
really are legit, them I'm at least going to lower the points assigned 
to hits on that dnsbl... and I might even remove that dnsbl from my spam 
filtering altogether.

If, on the other hand, I find that ALL such messages really were spam, I 
might start increasing the points given to that particular list, 
assuming that I'm not also seeing some FPs from that list.

Next, if I see a spam (that wasn't sent from a legit ISPs mailserver) 
and it scored rather low, I'll then take that IP and run it against a 
spam blacklist checker (dnsstuff, robtex, etc) to see if there are any 
RBLs that would have caught it, but that I'm not using yet. (Of course, 
I ignore various FP-ridden lists like APEWS in that search.) If I see a 
pattern whereby a particular list consistently hits on IPs that scored 
too low in my spam filtering, I might then add that dnsbl to my 
filtering... starting off with a low score... then double-checking for 
FPs... then bumping the score up depending on how little FPs there are. 
(in this case, I'm calling any "hits" on legit messages a "FP", but, at 
this stage, these will generate too low a score to outright block and 
this "FP" really did get delivered to the inbox.)

Doing this, over the years, I've added a good mix of RBLs with very fine 
tuned scoring (in my own spam filtering program, not referring to SA).

At one point, I noticed that many of the more aggressive dnsbls are 
really really good at catching new IPs, but have too many FPs. As a 
result, I have to keep their "score" low. But it seemed such a shame 
because these IPs were taking too long to get on the FP-safe dnsbls. 
Then I noticed that, many times, three or four of the more aggressive 
RBLs would quickly hit on the same spammer's IP, where that IP that 
wasn't yet on SpamHaus, etc... then... if a few lists hit on that new 
spammer' IP, chances were, it was worthy of blocking in comparison to if 
just one list hit on it... so much so that the score really needed to be 
higher than merely the sum of the FP-risky dnsbl's scores.

As a result, I changed my formula so that I took into account the number 
of dnsbls that hit on that IP as well as the score. (it was something 
like.. for every added dnsbld "hit" the overall RBL score would get 
increased by an additional 10% or 20%)... next, I adjusted down some of 
the "raw" scores so as to not allow the RBL scoring to get out of 
control. IOW... the whole really was worth more than the sum of its 
parts! Get it?

Of course, even then, I have extensive whitelisting of IPs that I have 
placed in front of this... both my own (that I've put literally 
thousands of hours into!) and third parties. Currently, my own IP 
blacklist isn't (yet) on dnsstuff or robtex... but if it something like 
it were there and produced by someone else... I would have spotted it in 
that systematic checking that I described and I'd have been thrilled at 
its results... IOW... I created a product that I myself would have 
greatly desired to have if it had been created and distributed by 
someone else. I probably would have been one the first subscribers.. had 
this been someone else's product. (Why? Because my RBL provides that 
same "fast reacting aggressiveness"... just without the FPs!)

Still, besides my own RBL's "subscription" barrier to inclusion... other 
lists which also require RSYNC access would not be able to come 
"preinstalled" in SA since they too need a little TLC to get up and 
running in one's spam filtering environment. These couldn't be used "out 
of the box" without some configuring of various programs on one's 
server. Something else to ponder.

I hope this is beneficial and helps future SA versions! Doing all of 
this, I believe I've taken the "RBL" portion of my spam filtering to a 
level that is beyond what many thought possible.

Rob McEwen