You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by Franck Martin <fm...@linkedin.com> on 2013/07/31 19:08:34 UTC
Creating new rules
Hi all,
I noticed there is no rules to check if the domain in various emails fields are on blocking lists like DBL at spamhaus. I'm willing to work on some of these rules, but I would appreciate any advice to bootstrap the process. If you can reference documents or say something like, look at this rule and this rule, this is close to what you need to do.
Thanks.
Re: Creating new rules
Posted by RW <rw...@googlemail.com>.
On Thu, 01 Aug 2013 19:08:12 +0200
Benny Pedersen wrote:
> RW skrev den 2013-08-01 18:00:
>
> > If you use /32 and the sender has a different IP address each time
> > there's no score averaging.
>
> servers changeing sender ip daily ?, its not a real problem clients
> does, there would be one static ip first
I think you have first and last the wrong way around. If the client has
a public IP address (as recorded by the submission server) then that's
the *first* routable address. With webmail the browser ip address is
used if it's recorded in a header.
Re: Creating new rules
Posted by Benny Pedersen <me...@junc.eu>.
RW skrev den 2013-08-01 18:00:
> If you use /32 and the sender has a different IP address each time
> there's no score averaging.
servers changeing sender ip daily ?, its not a real problem clients
does, there would be one static ip first
Re: Creating new rules
Posted by RW <rw...@googlemail.com>.
On Thu, 01 Aug 2013 16:36:22 +0200
Benny Pedersen wrote:
> RW skrev den 2013-08-01 14:39:
>
> > This would make sense if the IP address were the the first trusted
> > address or last external, but AWL uses the first routable address
> > which
> > is commonly dynamic.
>
> why is this in error ?
If you use /32 and the sender has a different IP address each time
there's no score averaging.
Re: Creating new rules
Posted by Benny Pedersen <me...@junc.eu>.
RW skrev den 2013-08-01 14:39:
> This would make sense if the IP address were the the first trusted
> address or last external, but AWL uses the first routable address
> which
> is commonly dynamic.
why is this in error ?
Re: Creating new rules
Posted by RW <rw...@googlemail.com>.
On Thu, 01 Aug 2013 12:34:26 +0200
Benny Pedersen wrote:
> Jari Fredriksson skrev den 2013-07-31 22:04:
> > AWL plugin does it anyway, if enabled. But it does not use any
> > external
> > backlists for it...
>
> if its runs with default /16 is just a joke
>
> change it to /24 or /32 then its more no joke
This would make sense if the IP address were the the first trusted
address or last external, but AWL uses the first routable address which
is commonly dynamic.
Re: Creating new rules
Posted by Benny Pedersen <me...@junc.eu>.
Jari Fredriksson skrev den 2013-07-31 22:04:
> 31.07.2013 21:05, Franck Martin kirjoitti:
>> Ah yes, I saw these rules, but this is to check the domains of urls
>> in
>> the messages, not to check for instance that the domain used in the
>> From: header is on the DBL.
> Address in From: is usually always forged in Spam nowadays. There is
> not
> much use for checking these.
http://blog.returnpath.com/blog/ken-takahashi/demystifying-spf-dkim-and-dmarc
> AWL plugin does it anyway, if enabled. But it does not use any
> external
> backlists for it...
if its runs with default /16 is just a joke
change it to /24 or /32 then its more no joke
when this is done, add sagrey plugin, stops one time senders
sa 3.4.0 should have history plugin but its missing in 3.4.0-rc2
Re: Creating new rules
Posted by Jari Fredriksson <ja...@iki.fi>.
31.07.2013 21:05, Franck Martin kirjoitti:
> Ah yes, I saw these rules, but this is to check the domains of urls in the messages, not to check for instance that the domain used in the From: header is on the DBL.
Address in From: is usually always forged in Spam nowadays. There is not
much use for checking these.
AWL plugin does it anyway, if enabled. But it does not use any external
backlists for it...
--
jarif.bit
Re: Creating new rules
Posted by Jari Fredriksson <ja...@iki.fi>.
31.07.2013 21:05, Franck Martin kirjoitti:
> On Jul 31, 2013, at 7:56 PM, Ralf Hildebrandt <Ra...@charite.de>
> wrote:
>
>> * Franck Martin <fm...@linkedin.com>:
>>
>>> I looked at http://spamassassin.apache.org/tests_3_3_x.html could not find any rule that do the above. Please help.
>> That's a bit odd. I found it being mentioned here:
>>
>> http://www.spamhaus.org/faq/section/Spamhaus%20DBL#287
>> http://spamassassin.1065346.n5.nabble.com/enabling-SpamHaus-DBL-td55862.html
>> http://svn.apache.org/repos/asf/spamassassin/trunk/rules/25_uribl.cf
>>
>> and by all means it should be enabled by default.
>>
> Ah yes, I saw these rules, but this is to check the domains of urls in the messages, not to check for instance that the domain used in the From: header is on the DBL.
>
http://svn.apache.org/repos/asf/spamassassin/trunk/rules/25_uribl.cf
--
jarif.bit
RE: Creating new rules
Posted by Kevin Miller <Ke...@ci.juneau.ak.us>.
Because some spammers are pretty dumb. Not all of course. Addresses are constantly being harvested. If you got a list of half a million addresses, are you going to vet all those? Oft times they'll just blast them out with a botnet and the ones that fail are just collateral damage. I think the goal is usually quantity over quality. Not being a spammer though, i could be wrong. <g>
Also, it may be that the domain wasn't in a blacklist when they botted it but gets put in pretty quickly via razor, pyzor, and various MTAs that report to RBLs. I've seen a dozen or so spam hit or server and w/in 15 - 20 minutes it'll be on someone's RBL. If it works for you, live it up. Those are just my thoughts - others here have a much more informed opinion I expect....
...Kevin
________________________________________
From: Franck Martin [fmartin@linkedin.com]
Sent: Wednesday, July 31, 2013 1:06 PM
To: Kevin Miller
Cc: Ralf Hildebrandt; <us...@spamassassin.apache.org>
Subject: Re: Creating new rules
On Jul 31, 2013, at 10:08 PM, Kevin Miller <Ke...@ci.juneau.ak.us> wrote:
> Problem is, the from adddress is often a "Joe job" - i.e., a forged address, so the domain mentioned there likely doesn't have anything to do with the actual source of the mail. It seems to me that if the domain isn't the actual source of he spam, it can be detrimental to be filtering on it, particularly if Bayes is learning from it or your MTA auto-reports it to RBLs.
>
Why would they use a forged domain which is on a blacklist? I think they would tend to use a domain which is well known with good reputation. As well known domains are getting protected, then they have to move to use their own domain, which happens to appear on blacklist...
Now as we move to IPv6, reputation will shift from an IP based type reputation, to a domain based type reputation. Unfortunately, spam assassin seems to be lacking some rules.
Nevertheless, it does not matter, if it is the right or wrong direction, my question remains: how do I create such a rule?
Re: Creating new rules
Posted by Franck Martin <fm...@linkedin.com>.
On Aug 1, 2013, at 12:44 PM, Benny Pedersen <me...@junc.eu> wrote:
> Franck Martin skrev den 2013-07-31 23:06:
>
>> Now as we move to IPv6, reputation will shift from an IP based type
>> reputation, to a domain based type reputation. Unfortunately, spam
>> assassin seems to be lacking some rules.
>
> still missing dmarc spamassassin plugin, there is a dkim_reput but i dont see much help there, it could be bootstrapped if one have own dkim_repution server and reporting based on opendkim
>
> and it failed for me with http://www.dkim-reputation.org/ it might work, but would work better if more used it
While interesting, I think this is a dead end... There is some IETF work to do some reputation system... not sure exactly what
>
>> Nevertheless, it does not matter, if it is the right or wrong
>> direction, my question remains: how do I create such a rule?
>
> rule for ?
grabbing a domain from some headers and checking it with a DNSBL.
Re: Creating new rules
Posted by Benny Pedersen <me...@junc.eu>.
Franck Martin skrev den 2013-07-31 23:06:
> Why would they use a forged domain which is on a blacklist? I think
> they would tend to use a domain which is well known with good
> reputation. As well known domains are getting protected, then they
> have to move to use their own domain, which happens to appear on
> blacklist...
agre with that, here i blacklist_from that have spf_pass and spamming
sender, and also just spamming domain that is not dkim signed or get spf
results, eg score on spf_none :)
> Now as we move to IPv6, reputation will shift from an IP based type
> reputation, to a domain based type reputation. Unfortunately, spam
> assassin seems to be lacking some rules.
still missing dmarc spamassassin plugin, there is a dkim_reput but i
dont see much help there, it could be bootstrapped if one have own
dkim_repution server and reporting based on opendkim
and it failed for me with http://www.dkim-reputation.org/ it might
work, but would work better if more used it
> Nevertheless, it does not matter, if it is the right or wrong
> direction, my question remains: how do I create such a rule?
rule for ?
Re: Creating new rules
Posted by Franck Martin <fm...@linkedin.com>.
On Jul 31, 2013, at 11:19 PM, RGB Camera <za...@gmail.com>>
wrote:
On Wed, Jul 31, 2013 at 2:06 PM, Franck Martin <fm...@linkedin.com>> wrote:
On Jul 31, 2013, at 10:08 PM, Kevin Miller <Ke...@ci.juneau.ak.us>> wrote:
> Problem is, the from adddress is often a "Joe job" - i.e., a forged address, so the domain mentioned there likely doesn't have anything to do with the actual source of the mail. It seems to me that if the domain isn't the actual source of he spam, it can be detrimental to be filtering on it, particularly if Bayes is learning from it or your MTA auto-reports it to RBLs.
>
Why would they use a forged domain which is on a blacklist?
Indeed, if someone uses a forged domain which is on a blacklist in the header of their mail, we want to block that email too.
Some smart B2B spammers know about this loophole in SpamAssassin and don't use their domain name in the message body, using it only in the header where the URI checks aren't done.
Let me give some background...
I'm part of the people that adopted DMARC (cf www.dmarc.org<http://www.dmarc.org>) this provide protection for the domains that are heavily spoofed.
Why I'm not keen in reproducing the DMARC checks in spamassassin, because it is better handled via a milter like opendmarc (because of reporting capabilities which is important), I would not make a fuss if I see something like DMARC in spamassassin.
During the development of DMARC, we realized that there are a few holes to plug for it to be more effective, as well as realizing, in general, domain reputation will become more and more important.
One of this rule, is to check how the From: header is formed cf http://tools.ietf.org/html/draft-ietf-appsawg-malformed-mail-07 and rate negatively when some headers are malformed
The other is to extract all the domains from the following fields: envelope from, from: sender, reply-to and helo/ehlo, and check them against DNSBL
There may be other rules, but this is what comes to mind, last one is suggested on spamhaus FAQ but does not seem to have made it in spamassassin.
While at the moment DMARC is for domains heavily spoofed, the above rules should benefit everyone
Re: Creating new rules
Posted by RGB Camera <za...@gmail.com>.
On Wed, Jul 31, 2013 at 2:06 PM, Franck Martin <fm...@linkedin.com> wrote:
>
> On Jul 31, 2013, at 10:08 PM, Kevin Miller <Ke...@ci.juneau.ak.us>
> wrote:
>
> > Problem is, the from adddress is often a "Joe job" - i.e., a forged
> address, so the domain mentioned there likely doesn't have anything to do
> with the actual source of the mail. It seems to me that if the domain
> isn't the actual source of he spam, it can be detrimental to be filtering
> on it, particularly if Bayes is learning from it or your MTA auto-reports
> it to RBLs.
> >
>
> Why would they use a forged domain which is on a blacklist?
>
Indeed, if someone uses a forged domain which is on a blacklist in the
header of their mail, we want to block that email too.
Some smart B2B spammers know about this loophole in SpamAssassin and don't
use their domain name in the message body, using it only in the header
where the URI checks aren't done.
Re: Creating new rules
Posted by Franck Martin <fm...@linkedin.com>.
On Jul 31, 2013, at 10:08 PM, Kevin Miller <Ke...@ci.juneau.ak.us> wrote:
> Problem is, the from adddress is often a "Joe job" - i.e., a forged address, so the domain mentioned there likely doesn't have anything to do with the actual source of the mail. It seems to me that if the domain isn't the actual source of he spam, it can be detrimental to be filtering on it, particularly if Bayes is learning from it or your MTA auto-reports it to RBLs.
>
Why would they use a forged domain which is on a blacklist? I think they would tend to use a domain which is well known with good reputation. As well known domains are getting protected, then they have to move to use their own domain, which happens to appear on blacklist...
Now as we move to IPv6, reputation will shift from an IP based type reputation, to a domain based type reputation. Unfortunately, spam assassin seems to be lacking some rules.
Nevertheless, it does not matter, if it is the right or wrong direction, my question remains: how do I create such a rule?
RE: Creating new rules
Posted by Kevin Miller <Ke...@ci.juneau.ak.us>.
Problem is, the from adddress is often a "Joe job" - i.e., a forged address, so the domain mentioned there likely doesn't have anything to do with the actual source of the mail. It seems to me that if the domain isn't the actual source of he spam, it can be detrimental to be filtering on it, particularly if Bayes is learning from it or your MTA auto-reports it to RBLs.
YMMV...
...Kevin
________________________________________
From: Franck Martin [fmartin@linkedin.com]
Sent: Wednesday, July 31, 2013 10:05 AM
To: Ralf Hildebrandt
Cc: <us...@spamassassin.apache.org>
Subject: Re: Creating new rules
On Jul 31, 2013, at 7:56 PM, Ralf Hildebrandt <Ra...@charite.de>
wrote:
> * Franck Martin <fm...@linkedin.com>:
>
>> I looked at http://spamassassin.apache.org/tests_3_3_x.html could not find any rule that do the above. Please help.
>
> That's a bit odd. I found it being mentioned here:
>
> http://www.spamhaus.org/faq/section/Spamhaus%20DBL#287
> http://spamassassin.1065346.n5.nabble.com/enabling-SpamHaus-DBL-td55862.html
> http://svn.apache.org/repos/asf/spamassassin/trunk/rules/25_uribl.cf
>
> and by all means it should be enabled by default.
>
Ah yes, I saw these rules, but this is to check the domains of urls in the messages, not to check for instance that the domain used in the From: header is on the DBL.
Re: Creating new rules
Posted by Franck Martin <fm...@linkedin.com>.
On Jul 31, 2013, at 7:56 PM, Ralf Hildebrandt <Ra...@charite.de>
wrote:
> * Franck Martin <fm...@linkedin.com>:
>
>> I looked at http://spamassassin.apache.org/tests_3_3_x.html could not find any rule that do the above. Please help.
>
> That's a bit odd. I found it being mentioned here:
>
> http://www.spamhaus.org/faq/section/Spamhaus%20DBL#287
> http://spamassassin.1065346.n5.nabble.com/enabling-SpamHaus-DBL-td55862.html
> http://svn.apache.org/repos/asf/spamassassin/trunk/rules/25_uribl.cf
>
> and by all means it should be enabled by default.
>
Ah yes, I saw these rules, but this is to check the domains of urls in the messages, not to check for instance that the domain used in the From: header is on the DBL.
Re: Creating new rules
Posted by Ralf Hildebrandt <Ra...@charite.de>.
* Franck Martin <fm...@linkedin.com>:
> I looked at http://spamassassin.apache.org/tests_3_3_x.html could not find any rule that do the above. Please help.
That's a bit odd. I found it being mentioned here:
http://www.spamhaus.org/faq/section/Spamhaus%20DBL#287
http://spamassassin.1065346.n5.nabble.com/enabling-SpamHaus-DBL-td55862.html
http://svn.apache.org/repos/asf/spamassassin/trunk/rules/25_uribl.cf
and by all means it should be enabled by default.
--
Ralf Hildebrandt Charite Universitätsmedizin Berlin
ralf.hildebrandt@charite.de Campus Benjamin Franklin
http://www.charite.de Hindenburgdamm 30, 12203 Berlin
Geschäftsbereich IT, Abt. Netzwerk fon: +49-30-450.570.155
Re: Creating new rules
Posted by Franck Martin <fm...@linkedin.com>.
On Jul 31, 2013, at 7:43 PM, Jari Fredriksson <ja...@iki.fi> wrote:
> 31.07.2013 20:08, Franck Martin kirjoitti:
>> Hi all,
>>
>> I noticed there is no rules to check if the domain in various emails fields are on blocking lists like DBL at spamhaus. I'm willing to work on some of these rules, but I would appreciate any advice to bootstrap the process. If you can reference documents or say something like, look at this rule and this rule, this is close to what you need to do.
>>
> SpamAssassin will and does check those RBL:s. They are NET rules, and
> not active when doing -D
>
Hmm, thanks
I looked at http://spamassassin.apache.org/tests_3_3_x.html could not find any rule that do the above. Please help.
Re: Creating new rules
Posted by Jari Fredriksson <ja...@iki.fi>.
31.07.2013 20:08, Franck Martin kirjoitti:
> Hi all,
>
> I noticed there is no rules to check if the domain in various emails fields are on blocking lists like DBL at spamhaus. I'm willing to work on some of these rules, but I would appreciate any advice to bootstrap the process. If you can reference documents or say something like, look at this rule and this rule, this is close to what you need to do.
>
SpamAssassin will and does check those RBL:s. They are NET rules, and
not active when doing -D
This may be the issue now, you have tried -D to a message and see what
triggers? When run normally the network checks will be executed.
--
jarif.bit