A new paradigm for DNS based lists

[Marc Perkel <su...@junkemailfilter.com>]

I'd like to suggest a new way of looking at DNS list lookups and I want 
to encourage other list providers to so something like what I'm doing 
with my Hostkarma list. If this were more standard it would greatly 
increase the accuracy of the lists and reduce the number of network 
calls SA has to make. Better accuracy - better performance.

We have a concept called a yellow list. Yellow means the the IP source 
is a mixture of spam and non-spam and that the IP address contains no 
information as to if the message is spam our not. Yellow is for Yahoo, 
Hotmail, Gmail, and other ISP/Freemail sources.

The idea is that once it is determined that the source is yellow there 
is no need to check any other lists. If someone else has it blacklisted 
then that would be an error.

White lists on my system means the IP only sends good email. That's a 
different definition than most lists which white means "do not 
blacklist". I have a "NOBL" status for IPs that I don't want to 
blacklist, but might be something I might whitelist in the future. White 
means the IP sends nothing but good email.

On my system if an IP is yellow I don't do any other DNS lookup calls. 
It passes on to content testing rules. Tht saves a lot of lookups. If 
the IP is white I pass the email wither any content scanning and that 
bypasses spamassassin entirely. If the IP is on several blacklists then 
the message is bounced without any further processing.

I use the NOBL list to avoid blacklist checks. If it's not white or 
yellow, a NOBL listing sends the message on to content scanning rules 
and bypasses all blacklist tests.

With the blacklists I start with my best blacklists first. About 3 of 
them. If they are on 2 of my 3 best I bounce it. Then I check the next 3 
best lists and if they are on 2 of the 6 then it's bounced. Other lists 
have lower scoring but because I do the good lists first I save the time 
often of having to check the less accurate lists.

Also - I don't include non performing lists or lists that are highly 
inaccurate like UCE-PROTECT, RCF-Ignorant, Backscatterer, APEWS. I'm not 
using Spamhaus because of the price, but it's a really good list. Also 
like barracuda, spamcop, mailspike, gbudb, manitu, Invalument was great 
too when I had it for free for a short period.

The point here is that accuracy and speed are greatly improved using 
this system and I'm leaving a lighter load on all the other DNS lists 
providers. I'm processing over 90% of incoming email without 
Spanassassin seeing it. If SA were to do what I'm doing then most email 
would never see any other rules than the DNS list rules. I'm doing the 
DNS lists in Exim and a single server allow me to process thousands of 
domains for tens of thousands of email accounts.

My 2 cents ...

-- 
Marc Perkel - Sales/Support
support@junkemailfilter.com
http://www.junkemailfilter.com
Junk Email Filter dot com
415-992-3400

Re: A new paradigm for DNS based lists

[Yet Another Ninja <sa...@alexb.ch>]

On 2010-12-29 20:50, Marc Perkel wrote:
>
>
> On 12/29/2010 11:10 AM, David F. Skoll wrote:
>> On Wed, 29 Dec 2010 09:33:25 -0800
>> Marc Perkel<su...@junkemailfilter.com> wrote:
>>
>>> Yes - there's no point in doing DNS blacklist lookups on yahoo,
>>> hotmail, and gmail as well as thousands of other mixed source
>>> providers.
>> I disagree. I have a strong feeling that some of those providers
>> route less-trustworthy mail through certain IP addresses and
>> more-trustworthy mail through others. For example, some of Yahoo's
>> servers are listed in our "good" list while others are listed in our
>> "bad" list. The difference in observed behaviour between the two sets of
>> Yahoo servers is very dramatic.
>>
>> We don't outright block hosts in the bad list, but we do add points.
>>
>> Regards,
>>
>> David.
>>
>
> Hi David,
>
> My idea doesn't preclude you from having a "bad yahoo" list and adding
> points. I'm just saying that when it comes to checking other blacklists
> to see if any yahoo server is listed it's a waste of resources. If it's
> a yahoo server of any flavore why look it up on the blacklists?

coz we can't be bothered to do otherwise?

Re: A new paradigm for DNS based lists

["David F. Skoll" <df...@roaringpenguin.com>]

On Wed, 29 Dec 2010 11:50:56 -0800
Marc Perkel <su...@junkemailfilter.com> wrote:

> My idea doesn't preclude you from having a "bad yahoo" list and
> adding points. I'm just saying that when it comes to checking other
> blacklists to see if any yahoo server is listed it's a waste of
> resources. If it's a yahoo server of any flavore why look it up on
> the blacklists?

Well, if you use our DNSBL, you'll find some Yahoo servers listed as
bad and some as good.  (Our DNSBL is not publicly available, but in
principle there could be a trustworthy publicly-available list that
uses the same listing criteria as ours.)

Giving Hotmail, Yahoo, etc. servers a free pass will simply shift spammer
economics in favour of CAPTCHA-breaking and/or phishing to obtain freemail
credentials.  That won't do anyone any good.

Regards,

David.

Re: A new paradigm for DNS based lists

[Marc Perkel <su...@junkemailfilter.com>]

On 12/29/2010 11:10 AM, David F. Skoll wrote:
> On Wed, 29 Dec 2010 09:33:25 -0800
> Marc Perkel<su...@junkemailfilter.com>  wrote:
>
>> Yes - there's no point in doing DNS blacklist lookups on yahoo,
>> hotmail, and gmail as well as thousands of other mixed source
>> providers.
> I disagree.  I have a strong feeling that some of those providers
> route less-trustworthy mail through certain IP addresses and
> more-trustworthy mail through others.  For example, some of Yahoo's
> servers are listed in our "good" list while others are listed in our
> "bad" list.  The difference in observed behaviour between the two sets of
> Yahoo servers is very dramatic.
>
> We don't outright block hosts in the bad list, but we do add points.
>
> Regards,
>
> David.
>

Hi David,

My idea doesn't preclude you from having a "bad yahoo" list and adding 
points. I'm just saying that when it comes to checking other blacklists 
to see if any yahoo server is listed it's a waste of resources. If it's 
a yahoo server of any flavore why look it up on the blacklists?

-- 
Marc Perkel - Sales/Support
support@junkemailfilter.com
http://www.junkemailfilter.com
Junk Email Filter dot com
415-992-3400

Re: A new paradigm for DNS based lists

["David F. Skoll" <df...@roaringpenguin.com>]

On Wed, 29 Dec 2010 09:33:25 -0800
Marc Perkel <su...@junkemailfilter.com> wrote:

> Yes - there's no point in doing DNS blacklist lookups on yahoo,
> hotmail, and gmail as well as thousands of other mixed source
> providers.

I disagree.  I have a strong feeling that some of those providers
route less-trustworthy mail through certain IP addresses and
more-trustworthy mail through others.  For example, some of Yahoo's
servers are listed in our "good" list while others are listed in our
"bad" list.  The difference in observed behaviour between the two sets of
Yahoo servers is very dramatic.

We don't outright block hosts in the bad list, but we do add points.

Regards,

David.

Re: A new paradigm for DNS based lists

[Benny Pedersen <me...@junc.org>]

On ons 29 dec 2010 18:33:25 CET, Marc Perkel wrote

> I would skip test if they have SPF because spammers often set their  
> SPF correctly.

stop this throlling, spammers dont add whitelist_from_spf into spamassassin

-- 
xpoint http://www.unicom.com/pw/reply-to-harmful.html

Re: A new paradigm for DNS based lists

[Daniel McDonald <da...@austinenergy.com>]

On 12/29/10 11:33 AM, "Marc Perkel" <su...@junkemailfilter.com> wrote:

> 
> 
> On 12/29/2010 9:24 AM, Matt wrote:
>> So any email from hotmail.com, gmail.com, yahoo.com, etc. if there SPF
>> or DKIM passes skip any further DNS tests?
>> 
>> 
> 
> Yes - there's no point in doing DNS blacklist lookups on yahoo, hotmail,
> and gmail as well as thousands of other mixed source providers. The IP
> tells you nothing. That's why I suggest the yellow listing.

There may be no reason to check the last-external address, but plenty of
reasons to do deep parsing and check the original source address or some
intermediate relay.

 
> I would skip test if they have SPF because spammers often set their SPF
> correctly.

Please stop talking about SPF until you understand the purpose for which it
is intended, which you obviously still don't based on this comment (despite
the flame war over SPF you started a few weeks ago.)


-- 
Daniel J McDonald, CCIE # 2495, CISSP # 78281

Re: A new paradigm for DNS based lists

[Marc Perkel <su...@junkemailfilter.com>]

On 12/29/2010 9:24 AM, Matt wrote:
> So any email from hotmail.com, gmail.com, yahoo.com, etc. if there SPF
> or DKIM passes skip any further DNS tests?
>
>

Yes - there's no point in doing DNS blacklist lookups on yahoo, hotmail, 
and gmail as well as thousands of other mixed source providers. The IP 
tells you nothing. That's why I suggest the yellow listing.

I would skip test if they have SPF because spammers often set their SPF 
correctly.

-- 
Marc Perkel - Sales/Support
support@junkemailfilter.com
http://www.junkemailfilter.com
Junk Email Filter dot com
415-992-3400

Re: A new paradigm for DNS based lists

[Benny Pedersen <me...@junc.org>]

On ons 29 dec 2010 18:24:00 CET, Matt wrote
> So any email from hotmail.com, gmail.com, yahoo.com, etc. if there SPF
> or DKIM passes skip any further DNS tests?

blind testing if sender is one of them, dont do more mta testing ?

if wanting to reduce load on sa then whitelist from spf or dkim, and  
based on that shortcicuit future sa testing, just dont whitelist with  
vildcards

-- 
xpoint http://www.unicom.com/pw/reply-to-harmful.html

Re: A new paradigm for DNS based lists

[Matt <lm...@gmail.com>]

Could a similiar thing be accomplished with a simple list of free
email provider etc. domains and checking there SPF or DKIM records and
if they pass bypassing any other DNS lists?

So any email from hotmail.com, gmail.com, yahoo.com, etc. if there SPF
or DKIM passes skip any further DNS tests?