You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by da...@chaosreigns.com on 2011/04/01 20:34:16 UTC
New DNS white/blacklist + spamassassin rules Re: Please report IPs
delivering ham and spam with this script
While I still plan for this to primarily be used via rsync and a
spamassassin plugin, I've loaded the data into DNS records and created
spamassassin rules so it can easily be tested now. It's updating
automatically once a day.
I'm hoping this will encourage people to contribute data. Because now you
should get an immediate improvement in your spam filtration, based on data
you've provided on what IPs send you ham and spam.
More info, including the script to submit data (either from spam/ham
folders, or individual emails piped to standard input) here:
http://www.chaosreigns.com/iprep/
The spamassassin rules:
ifplugin Mail::SpamAssassin::Plugin::DNSEval
header __RCVD_IN_IPREP eval:check_rbl('iprep-firsttrusted', 'iprep.chaosreigns.com.')
tflags __RCVD_IN_IPREP nice net
header RCVD_IN_IPREPDNS_100 eval:check_rbl_sub('iprep-firsttrusted', '127.\d+.\d+.100')
describe RCVD_IN_IPREPDNS_100 Sender listed at http://www.chaosreigns.com/iprep/, 100% ham
tflags RCVD_IN_IPREPDNS_100 nice net
header RCVD_IN_IPREPDNS_50 eval:check_rbl_sub('iprep-firsttrusted', '127.\d+.\d+.50')
describe RCVD_IN_IPREPDNS_50 Sender listed at http://www.chaosreigns.com/iprep/, 50% ham
tflags RCVD_IN_IPREPDNS_50 nice net
header RCVD_IN_IPREPDNS_0 eval:check_rbl_sub('iprep-firsttrusted', '127.\d+.\d+.0')
describe RCVD_IN_IPREPDNS_0 Sender listed at http://www.chaosreigns.com/iprep/, 0% ham
tflags RCVD_IN_IPREPDNS_0 net
meta RCVD_NOT_IN_IPREPDNS ( ! RCVD_IN_IPREPDNS_100 && ! RCVD_IN_IPREPDNS_50 && ! RCVD_IN_IPREPDNS_0 && ! NO_RELAYS )
describe RCVD_NOT_IN_IPREPDNS Sender not listed at http://www.chaosreigns.com/iprep/
tflags RCVD_NOT_IN_IPREPDNS net
score RCVD_IN_IPREPDNS_100 -0.1
score RCVD_IN_IPREPDNS_50 -0.0001
score RCVD_IN_IPREPDNS_0 0.1
score RCVD_NOT_IN_IPREPDNS 0.0001
endif
For people not contributing data, this is not likely to be useful yet.
Out of the 86,899 IPs I have data for, all but 38 are either 100% spam or
100% ham, so a great predictor of what the next email from known IPs will
be. This is why blacklists and whitelists, including spamassassin's AWL
(which is another combination of both) are nothing new.
The advantages I'm providing over SA's AWL are:
1) It's based on human verified ham and spam, not SA's previous opinions of
emails.
2) Shared knowledge from other people's email.
What I hope to be an advantage over dnswl.org, which I've been involved in,
is increased automation.
Here's a test I ran using only the last 500 of my own emails. All hand
categorized as spam or ham, and sorted by received data. One by one it
learns the IP as a ham source, spammer, or mix, and using what it has
learned, guesses what the next email is. Every 100 emails it reports its
success rate for the last 100 emails:
$ ./progress.pl
Rank 100, hit 51.7647058823529% of ham, hit 0% of spam.
Rank 50, hit 0% of ham, hit 0% of spam.
Rank 0, hit 0% of ham, hit 0% of spam.
Rank none, hit 48.2352941176471% of ham, hit 100% of spam.
Rank 100, hit 76% of ham, hit 0% of spam.
Rank 50, hit 0% of ham, hit 0% of spam.
Rank 0, hit 0% of ham, hit 28% of spam.
Rank none, hit 24% of ham, hit 72% of spam.
Rank 100, hit 72.3684210526316% of ham, hit 0% of spam.
Rank 50, hit 0% of ham, hit 0% of spam.
Rank 0, hit 0% of ham, hit 4.16666666666667% of spam.
Rank none, hit 27.6315789473684% of ham, hit 95.8333333333333% of spam.
Rank 100, hit 79.4520547945205% of ham, hit 0% of spam.
Rank 50, hit 0% of ham, hit 0% of spam.
Rank 0, hit 0% of ham, hit 48.1481481481481% of spam.
Rank none, hit 20.5479452054795% of ham, hit 51.8518518518519% of spam.
Rank 100, hit 79.2682926829268% of ham, hit 0% of spam.
Rank 50, hit 0% of ham, hit 0% of spam.
Rank 0, hit 0% of ham, hit 27.7777777777778% of spam.
Rank none, hit 20.7317073170732% of ham, hit 72.2222222222222% of spam.
So after 400 emails, RCVD_IN_IPREPDNS_100 is hitting 79% of ham and no
spam. I don't think anything else spamassassin uses can do this well.
But I have data from 184,335 emails. Using all that data, results for
the last 10,000 emails were:
Rank 100, hit 94.1176470588235% of ham, hit 0.0101553772722657% of spam.
Rank 50, hit 1.30718954248366% of ham, hit 0.0101553772722657% of spam.
Rank 0, hit 0% of ham, hit 64.2022951152635% of spam.
Rank none, hit 4.57516339869281% of ham, hit 35.7773941301919% of spam.
RCVD_IN_IPREPDNS_100 hits 94% of ham, and 0.01% of spam.
RCVD_IN_IPREPDNS_0 hits 64% of spam and no ham. Again, I don't think
anything else spamassassin uses can do this well.
But results this good can only be expected for people contributing data.
At least until we get more people contributing data.
--
"The price of freedom is the willingness to do sudden battle, anywhere,
at any time, and with utter recklessness." - Robert A. Heinlein
http://www.ChaosReigns.com
Re: Please report IPs delivering ham and spam with this script
Posted by da...@chaosreigns.com.
On 04/01, David F. Skoll wrote:
> o 536,596 (5.8%) sent _only_ ham
>
> o 7,821,574 (86%) sent _only_ spam
>
> o The remaining 744,705 (8.2%) sent a mixture. Most Yahoo! servers are in
> this category.
Sounds reasonable. It's nice to see the numbers, thanks.
> You saw less than 0.05% sending a mixture, which means you are probably
> not getting a good sample.
Yup. I don't have enough data. That's why I'm asking for more.
--
"Life is either a daring adventure or it is nothing at all."
- Helen Keller
http://www.ChaosReigns.com
Re: Please report IPs delivering ham and spam with this script
Posted by "David F. Skoll" <df...@roaringpenguin.com>.
On Fri, 1 Apr 2011 14:34:16 -0400
darxus@chaosreigns.com wrote:
> Out of the 86,899 IPs I have data for, all but 38 are either 100%
> spam or 100% ham,
That sounds a bit funny.
We have data on over 17 million IP addresses (collected using
http://mimedefang.org/reputation) Of those, about 9 million report at
least one ham or one spam -- the remainder either never made it past
greylisting or only tried emailing nonexistent recipient addresses.
Of those 9,102,875 hosts:
o 536,596 (5.8%) sent _only_ ham
o 7,821,574 (86%) sent _only_ spam
o The remaining 744,705 (8.2%) sent a mixture. Most Yahoo! servers are in
this category.
You saw less than 0.05% sending a mixture, which means you are probably
not getting a good sample.
Regards,
David.
PS: If anyone wants to contribute to and download *our* reputation
list, please see http://mimedefang.org/reputation and email me
off-list. Please be aware that unlike darxus' list, ours is not
freely-available, though we generally give free downloads to
organizations willing to feed us reputation data if they do a
statistically-useful amount of mail (>= 50K messages/day).
Re: New DNS white/blacklist + spamassassin rules Re: Please report
IPs delivering ham and spam with this script
Posted by da...@chaosreigns.com.
On 04/01, Michael Scheidell wrote:
> On 4/1/11 2:34 PM, darxus@chaosreigns.com wrote:
> >header RCVD_IN_IPREPDNS_0 eval:check_rbl_sub('iprep-firsttrusted', '127.\d+.\d+.0')
> >describe RCVD_IN_IPREPDNS_0 Sender listed athttp://www.chaosreigns.com/iprep/, 0% ham
> >tflags RCVD_IN_IPREPDNS_0 net
> >
> might actually need a quantity qualifier.
>
> (if this ip is 0 % ham... does that actually mean it is 100% spam?)
>
> or does that mean that I (so far) only saw one email hit it, and it is spam?
It means that all of the email seen from that IP so far has been spam.
Which may only have been one email.
> other than this is marking 'spam rates' and DCC commercial does the
> same thing for 'bulk' rates, what is the difference between this
> and DCC?
The "commercial" part.
> maybe 2nd or 3rd octet could contain 'confidence factor'.. eg:
It does, actually. A logarithm of the count of emails seen from that IP
(newer emails weighted more than old emails, and scaled up so small old
counts are greater than 0).
I haven't studied data enough to figure out what threshold is best for
what, and I don't think the existing rule definition language provides
a good way to specify a range.
Also, ignoring it is working quite well.
--
"I refuse to tip toe through life only to arrive safely at death."
http://www.ChaosReigns.com
Re: New DNS white/blacklist + spamassassin rules Re: Please report
IPs delivering ham and spam with this script
Posted by Michael Scheidell <mi...@secnap.com>.
On 4/1/11 2:34 PM, darxus@chaosreigns.com wrote:
> header RCVD_IN_IPREPDNS_0 eval:check_rbl_sub('iprep-firsttrusted', '127.\d+.\d+.0')
> describe RCVD_IN_IPREPDNS_0 Sender listed athttp://www.chaosreigns.com/iprep/, 0% ham
> tflags RCVD_IN_IPREPDNS_0 net
>
might actually need a quantity qualifier.
(if this ip is 0 % ham... does that actually mean it is 100% spam?)
or does that mean that I (so far) only saw one email hit it, and it is spam?
other than this is marking 'spam rates' and DCC commercial does the same
thing for 'bulk' rates, what is the difference between this and DCC?
note: dcc uses (for large installs) a local, VLDB that they 'sync'
(flood they call it) in real time. but it not only tells you the bulk
rate of the sender's ip, but the 'bulk hit rate' for the email you just got.
sounds similar, but bulk vs spam.
(and its inverse.. you collect percentages of HAM. the collect
percentages of BULK).
maybe 2nd or 3rd octet could contain 'confidence factor'.. eg:
some sliding scale of how many actual emails you have seen?
--
Michael Scheidell, CTO
o: 561-999-5000
d: 561-948-2259
ISN: 1259*1300
>*| *SECNAP Network Security Corporation
* Best Intrusion Prevention Product, Networks Product Guide
* Certified SNORT Integrator
* Hot Company Award, World Executive Alliance
* Best in Email Security, 2010 Network Products Guide
* King of Spam Filters, SC Magazine
______________________________________________________________________
This email has been scanned and certified safe by SpammerTrap(r).
For Information please see http://www.secnap.com/products/spammertrap/
______________________________________________________________________
Re: New DNS white/blacklist + spamassassin rules Re: Please report IPs delivering ham and spam with this script
Posted by Mark Martinec <Ma...@ijs.si>.
> > Do not forget to backslash-quote dots in a regular expression
> > if you mean a literal dot instead of 'any character'.
>
> Eep. That was copied from existing rules. I believe you're right, and
> there are a bunch of rules that need more escaping. Thanks.
True, there is a bunch of rules that need more escaping.
It is noted somewhere in the bug tracking (but not as a standalone ticket),
and needs a volunteer to do the cleaning :)
Mark
Re: New DNS white/blacklist + spamassassin rules Re: Please report
IPs delivering ham and spam with this script
Posted by da...@chaosreigns.com.
On 04/01, Mark Martinec wrote:
> > eval:check_rbl_sub('iprep-firsttrusted', '127.\d+.\d+.100') describe
>
> Do not forget to backslash-quote dots in a regular expression
> if you mean a literal dot instead of 'any character'.
Eep. That was copied from existing rules. I believe you're right, and
there are a bunch of rules that need more escaping. Thanks.
--
"Will I ever learn? I hope not, I'm having too much fun."
- Brent "Minime" Avis, motorcycle.com
http://www.ChaosReigns.com
Re: New DNS white/blacklist + spamassassin rules Re: Please report
IPs delivering ham and spam with this script
Posted by da...@chaosreigns.com.
On 04/01, Mark Martinec wrote:
> > eval:check_rbl_sub('iprep-firsttrusted', '127.\d+.\d+.100') describe
>
> Do not forget to backslash-quote dots in a regular expression
> if you mean a literal dot instead of 'any character'.
Updated rules (thanks again):
ifplugin Mail::SpamAssassin::Plugin::DNSEval
header __RCVD_IN_IPREPDNS eval:check_rbl('iprep-firsttrusted', 'iprep.chaosreigns.com.')
tflags __RCVD_IN_IPREPDNS nice net
header RCVD_IN_IPREPDNS_100 eval:check_rbl_sub('iprep-firsttrusted', '^127\.\d+\.\d+\.100$')
describe RCVD_IN_IPREPDNS_100 Sender listed at http://www.chaosreigns.com/iprep/, 100% ham
tflags RCVD_IN_IPREPDNS_100 nice net
header RCVD_IN_IPREPDNS_50 eval:check_rbl_sub('iprep-firsttrusted', '^127\.\d+\.\d+\.50$')
describe RCVD_IN_IPREPDNS_50 Sender listed at http://www.chaosreigns.com/iprep/, 50% ham
tflags RCVD_IN_IPREPDNS_50 nice net
header RCVD_IN_IPREPDNS_0 eval:check_rbl_sub('iprep-firsttrusted', '^127\.\d+\.\d+\.0$')
describe RCVD_IN_IPREPDNS_0 Sender listed at http://www.chaosreigns.com/iprep/, 0% ham
tflags RCVD_IN_IPREPDNS_0 net
meta RCVD_NOT_IN_IPREPDNS ( ! RCVD_IN_IPREPDNS_100 && ! RCVD_IN_IPREPDNS_50 && ! RCVD_IN_IPREPDNS_0 && ! NO_RELAYS )
describe RCVD_NOT_IN_IPREPDNS Sender not listed at http://www.chaosreigns.com/iprep/
tflags RCVD_NOT_IN_IPREPDNS net
score RCVD_IN_IPREPDNS_100 -0.1
score RCVD_IN_IPREPDNS_50 -0.0001
score RCVD_IN_IPREPDNS_0 0.1
score RCVD_NOT_IN_IPREPDNS 0.0001
endif
--
"Go forth, and be excellent to one another." - http://www.jhuger.com/fredski.php
http://www.ChaosReigns.com
Re: New DNS white/blacklist + spamassassin rules Re: Please report IPs delivering ham and spam with this script
Posted by Mark Martinec <Ma...@ijs.si>.
> eval:check_rbl_sub('iprep-firsttrusted', '127.\d+.\d+.100') describe
Do not forget to backslash-quote dots in a regular expression
if you mean a literal dot instead of 'any character'.
Mark