You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spamassassin.apache.org by Henrik Krohns <he...@hege.li> on 2018/09/23 13:04:48 UTC
def_whitelist_auth
On Sun, Sep 23, 2018 at 12:36:41PM -0000, davej@apache.org wrote:
>
> More trusted senders from system-generated subdomains that aren't user mailboxes that can be compromised.
>
> Modified:
> spamassassin/trunk/rules/60_whitelist_auth.cf
>
> ...
> +def_whitelist_auth *@*.bebe.com
> +def_whitelist_auth *@*.homeadvisor.com
> +def_whitelist_auth *@*.handsonaswegrow.com
> +def_whitelist_auth *@*.in.gov
> +def_whitelist_auth *@*.oldchicago.com
> ...
I'm curious, are there guidelines on what can be added here? How are these
lists generated? Who verifies and checks that old domains don't age and go
to some spammers etc? Most of the listed stuff seems pretty pointless for
general population. Paypal and other _globally_ known services make sense.
Should we encourage committers to add lists of say local banks and
government institutions? I would have plenty, but I don't know if it's
SpamAssassins purpose to be a global reputation service with all the
maintenance work it requires.
-hk
Re: def_whitelist_auth
Posted by Benny Pedersen <me...@junc.eu>.
Henrik Krohns skrev den 2018-09-24 13:50:
> They BOTH expand to so much stuff, that I have no idea how a common
> committer like me can conclude if they are "safe" and not common
> sources of
> hacked accounts and such. :-)
sender-id is depricated
https://app.dmarcanalyzer.com/dns/spf?simple=1 but tools still counts it
:/
sadly, if we should do something make spf plugin count number of ips
only on ipv4 and incl ipv6, but not track sender-id ips at all, when
number of ips is to high consider that mail as spf fails, for +all we
could consider it to be ?all eq equal, i just see +all as do not reject
please
Re: def_whitelist_auth
Posted by Henrik Krohns <he...@hege.li>.
On Mon, Sep 24, 2018 at 06:15:12AM -0500, Dave Jones wrote:
> On 9/23/18 1:42 PM, Henrik Krohns wrote:
> >On Sun, Sep 23, 2018 at 12:25:36PM -0500, Dave Jones wrote:
> >>
> >>Consider the difference between these two SPF records:
> >># dig email.chase.com txt +short
> >>"v=spf1 include:epsl1.com -all"
> >># dig chase.com txt +short
> >>"v=spf1 a:spf.jpmchase.com ip4:207.162.228.0/24 ip4:207.162.229.0/24
> >>ip4:207.162.225.0/24 ip4:196.37.232.50 ip4:159.53.46.0/24 ip4:159.53.36.0/24
> >>ip4:159.53.110.0/24 ip4:159.53.78.0/24 include:tpo.chase.com -all"
> >>
> >>Do this make sense that *@*.chase.com is safer to trust than *@chase.com?
> >
> >Honestly I have no idea. As I don't have any decend mail feed these days,
> >doesn't seem like I can help much. Some local domains I checked, subdomain
> >or not, point to many different companies and mailers. To judge them worthy
> >globally whitelisting does require data, experience and contacts, so I'm
> >happy to leave this stuff to you and others. :-)
> >
> >-hk
> >
>
> I was only referring to the SPF record difference since you pointed out how
> unsafe it would be to trust some SPF records with includes that expand out
> to a large number of IPs.
>
> The two SPF records above are VERY different which supports the logic I
> listed out.
I don't understand, how are they different?
$ dig txt epsl1.com +short
"spf2.0/pra ip4:142.54.244.0/23 ip4:142.54.247.0/24 ip4:159.127.162.0/23 ip4:159.127.178.0/24 ip4:173.203.61.92/32 include:bfi0.com -all"
"v=spf1 ip4:142.54.244.0/23 ip4:142.54.247.0/24 ip4:159.127.162.0/23 ip4:159.127.178.0/24 ip4:173.203.61.92/32 include:bfi0.com -all"
$ dig txt bfi0.com +short
"spf2.0/pra ip4:93.191.146.0/23 ip4:206.132.3.0/24 ip4:206.132.1.0/24 ip4:216.35.62.0/25 ip4:216.33.63.0/24 ip4:209.67.13.128/25 ip4:208.70.142.0/23 ip4:142.54.200.0/23 ip4:142.54.246.0/24 -all"
"v=spf1 ip4:93.191.146.0/23 ip4:206.132.3.0/24 ip4:206.132.1.0/24 ip4:216.35.62.0/25 ip4:216.33.63.0/24 ip4:209.67.13.128/25 ip4:208.70.142.0/23 ip4:142.54.200.0/23 ip4:142.54.246.0/24 -all"
$ dig txt tpo.chase.com +short
"v=spf1 ip4:68.233.76.14/32 ip4:63.150.74.35/32 ip4:198.64.159.0/24 ip4:198.104.137.206/32 ip4:161.58.88.0/24 -all"
They BOTH expand to so much stuff, that I have no idea how a common
committer like me can conclude if they are "safe" and not common sources of
hacked accounts and such. :-)
-hk
Re: def_whitelist_auth
Posted by Dave Jones <da...@apache.org>.
On 9/23/18 1:42 PM, Henrik Krohns wrote:
> On Sun, Sep 23, 2018 at 12:25:36PM -0500, Dave Jones wrote:
>>
>> Consider the difference between these two SPF records:
>> # dig email.chase.com txt +short
>> "v=spf1 include:epsl1.com -all"
>> # dig chase.com txt +short
>> "v=spf1 a:spf.jpmchase.com ip4:207.162.228.0/24 ip4:207.162.229.0/24
>> ip4:207.162.225.0/24 ip4:196.37.232.50 ip4:159.53.46.0/24 ip4:159.53.36.0/24
>> ip4:159.53.110.0/24 ip4:159.53.78.0/24 include:tpo.chase.com -all"
>>
>> Do this make sense that *@*.chase.com is safer to trust than *@chase.com?
>
> Honestly I have no idea. As I don't have any decend mail feed these days,
> doesn't seem like I can help much. Some local domains I checked, subdomain
> or not, point to many different companies and mailers. To judge them worthy
> globally whitelisting does require data, experience and contacts, so I'm
> happy to leave this stuff to you and others. :-)
>
> -hk
>
I was only referring to the SPF record difference since you pointed out
how unsafe it would be to trust some SPF records with includes that
expand out to a large number of IPs.
The two SPF records above are VERY different which supports the logic I
listed out.
Dave
Re: def_whitelist_auth
Posted by Henrik Krohns <he...@hege.li>.
On Sun, Sep 23, 2018 at 12:25:36PM -0500, Dave Jones wrote:
>
> Consider the difference between these two SPF records:
> # dig email.chase.com txt +short
> "v=spf1 include:epsl1.com -all"
> # dig chase.com txt +short
> "v=spf1 a:spf.jpmchase.com ip4:207.162.228.0/24 ip4:207.162.229.0/24
> ip4:207.162.225.0/24 ip4:196.37.232.50 ip4:159.53.46.0/24 ip4:159.53.36.0/24
> ip4:159.53.110.0/24 ip4:159.53.78.0/24 include:tpo.chase.com -all"
>
> Do this make sense that *@*.chase.com is safer to trust than *@chase.com?
Honestly I have no idea. As I don't have any decend mail feed these days,
doesn't seem like I can help much. Some local domains I checked, subdomain
or not, point to many different companies and mailers. To judge them worthy
globally whitelisting does require data, experience and contacts, so I'm
happy to leave this stuff to you and others. :-)
-hk
Re: def_whitelist_auth
Posted by Dave Jones <da...@apache.org>.
On 9/23/18 10:46 AM, Henrik Krohns wrote:
> On Sun, Sep 23, 2018 at 09:15:33AM -0500, Dave Jones wrote:
>>
>> Keep in mind that these entries are usually subdomains that will not be
>> user/human mailboxes that can be compromised. These entries are verified to
>> be system-generated and have other rule hits making them trustworthy senders
>> that honor opt-out requests without harvesting/validating the email
>> addresses and handle abuse reports of their rogue customers.
>
> I guess that makes sense since system-generated messages are more prone to
> FPs, perhaps being identical looking and mass sent.
>
> But lets say we take for example the 5 biggest banks, healthcare or
> insurance companies in some country (Finland? I trust them to be competent
> enough. :-D). What are the chances of someone hijacking a mailbox there
> and sending masses of spam? They can't even anonymize themselves. Or if
> they tried to impersonate someone, what difference would SA default
> whitelisting make? Very likely custom phishing would not be caught in
> any filters anyway, unless foolishly having some uribl links etc.
>
Domains with user mailboxes that can be compromised should not be added
to this list unless we can know for sure the sender properly filters
outbound email or uses 2FA to stop compromised accounts. That is why
you see entries like *@*.example.com and not *@example.com since
*@example.com will most likely be user mailboxes that can be compromised.
Phishing can be caught if you have local meta rules matching content
that is common to phishing. For example, I have local rules that add 4
to 8 points for emails with "Chase" in various locations then the
whitelist_auth *@*.chase.com entry will allow through real chase.com emails.
> If backgrounds are checked carefully, I don't think there is much difference
> in whitelisting system or user domains. Someone could as easily hack a
> mass-mailer account.
>
I have not seen a hacked mass-mailer account in 4+ years. I have seen
rogue customers of the mass-mailers. Most mass-mailers I have worked
with want to keep their domain reputation in good standing so they
welcome feedback via Spamcop or their abuse contacts to lock/disable
their rogue customer.
I would not put a mass-mailer domain in 60_whitelist_auth.cf but I have
put a few into my local whitelist_auth list.
I do have these in my local list:
whitelist_auth *@amazon.com
whitelist_auth *@*.amazon.com
but this is very different from:
whitelist_auth *@amazonses.com
whitelist_auth *@*.amazonses.com
which is a mass-mailler with some rogue customers.
> It would be nice to create to guidelines on what to check if considering
> adding something, and not having a private backend to utilize..
>
Here's my system's logic:
1. Must have at least 100 hits the past week
2. AND must not hit FREEMAIL, URIBL_IVMURI, or DMARC FAIL
3. AND must hit RCVD_IN_HOSTKARMA_W, RCVD_IN_MSPIKE_H4,
RCVD_IN_MSPIKE_H3, RCVD_IN_RP_CERTIFIED, RCVD_IN_RP_SAFE, or RCVD_IN_IADB_*
4. AND must hit SPF_PASS, DKIM_VALID_AU, or DMARC_PASS
5. AND have an average SA score < -2.0
6. AND then pass domain level requirements to be a subdomain (not the
primary domain where user mailboxes could be compromised)
I have a script run that query every Sunday morning and spit out entries
to add. It automatically adds these to my local lists.
> For example, if this is the SPF record:
>
> include:spf.messagelabs.com include:_spf.anpdm.com ip4:194.9.95.111 -all
>
> That does have quite many loopholes, even though you see reputable companies
> handling the mail. Would this be considered safe for whitelisting then?
>
Logic rule #6 above will exclude user mailboxes that I would not trust.
Consider the difference between these two SPF records:
# dig email.chase.com txt +short
"v=spf1 include:epsl1.com -all"
# dig chase.com txt +short
"v=spf1 a:spf.jpmchase.com ip4:207.162.228.0/24 ip4:207.162.229.0/24
ip4:207.162.225.0/24 ip4:196.37.232.50 ip4:159.53.46.0/24
ip4:159.53.36.0/24 ip4:159.53.110.0/24 ip4:159.53.78.0/24
include:tpo.chase.com -all"
Do this make sense that *@*.chase.com is safer to trust than
*@chase.com? Many company have started using subdomains properly to
send from so SPF, DKIM, and DMARC can have their own settings diferent
from their main user email. We want to encourage this all over the
Internet so we can separate out those system-generated emails from user
mail.
> Of course there is DNSWL etc which can be utilized.
>
> -hk
>
Re: def_whitelist_auth
Posted by Henrik Krohns <he...@hege.li>.
On Sun, Sep 23, 2018 at 09:15:33AM -0500, Dave Jones wrote:
>
> Keep in mind that these entries are usually subdomains that will not be
> user/human mailboxes that can be compromised. These entries are verified to
> be system-generated and have other rule hits making them trustworthy senders
> that honor opt-out requests without harvesting/validating the email
> addresses and handle abuse reports of their rogue customers.
I guess that makes sense since system-generated messages are more prone to
FPs, perhaps being identical looking and mass sent.
But lets say we take for example the 5 biggest banks, healthcare or
insurance companies in some country (Finland? I trust them to be competent
enough. :-D). What are the chances of someone hijacking a mailbox there
and sending masses of spam? They can't even anonymize themselves. Or if
they tried to impersonate someone, what difference would SA default
whitelisting make? Very likely custom phishing would not be caught in
any filters anyway, unless foolishly having some uribl links etc.
If backgrounds are checked carefully, I don't think there is much difference
in whitelisting system or user domains. Someone could as easily hack a
mass-mailer account.
It would be nice to create to guidelines on what to check if considering
adding something, and not having a private backend to utilize..
For example, if this is the SPF record:
include:spf.messagelabs.com include:_spf.anpdm.com ip4:194.9.95.111 -all
That does have quite many loopholes, even though you see reputable companies
handling the mail. Would this be considered safe for whitelisting then?
Of course there is DNSWL etc which can be utilized.
-hk
Re: def_whitelist_auth
Posted by "Kevin A. McGrail" <km...@apache.org>.
I love documentation :-)
--
Kevin A. McGrail
VP Fundraising, Apache Software Foundation
Chair Emeritus Apache SpamAssassin Project
https://www.linkedin.com/in/kmcgrail - 703.798.0171
On Sun, Sep 23, 2018 at 10:15 AM Dave Jones <da...@apache.org> wrote:
> On 9/23/18 8:31 AM, Kevin A. McGrail wrote:
> > On 9/23/2018 9:04 AM, Henrik Krohns wrote:
> >> I'm curious, are there guidelines on what can be added here? How are
> these
> >> lists generated? Who verifies and checks that old domains don't age
> and go
> >> to some spammers etc? Most of the listed stuff seems pretty pointless
> for
> >> general population. Paypal and other _globally_ known services make
> sense.
> >>
> >> Should we encourage committers to add lists of say local banks and
> >> government institutions? I would have plenty, but I don't know if it's
> >> SpamAssassins purpose to be a global reputation service with all the
> >> maintenance work it requires.
> >
> > I would say it's valuable and to add it. People can always choose not
> > to use our rulesets.
> >
>
> I have an automated method to find low-scoring trusted senders from a
> highly tuned SA instance. If these entries cause any problems, users
> can report it to the mailing list and they will be removed. So far
> there has been one entry reported that was too risky for the general
> population and it was reported and removed. Otherwise, I have feedback
> that it has helped improve FPs.
>
> Keep in mind that these entries are usually subdomains that will not be
> user/human mailboxes that can be compromised. These entries are
> verified to be system-generated and have other rule hits making them
> trustworthy senders that honor opt-out requests without
> harvesting/validating the email addresses and handle abuse reports of
> their rogue customers.
>
> My goal was to create low/zero risk entries that the mail filtering
> industry can see that promotes good SPF, DKIM, and DMARC settings to
> raise awareness all around the Internet.
>
> Another purpose of these entries is to allow local meta rules of certain
> email content to add points to block junk senders while allowing through
> those senders in this list that are known to be good and honor opt-out
> requests.
>
> Many of these entries are vetted by private RBLs and DBLs which
> indirectly helps those SA installations that aren't able to subscribe to
> those RBLs and DBLs or fine tune their SA rules and settings.
>
> I proposed the idea on the mailing list a couple of years ago about
> having a centralized clearinghouse of known good senders but no one
> stepped up with any ideas. Paul Stead's dkimwl.org is the closest thing
> to this that I have found and I think this has been added to 3.4.2
> commented out so some may enable it but most won't.
>
> I have local whitelist_auth entries that are several times longer than
> what I am putting into SA with zero customer complaints and I am
> filtering for about 90,000 mailboxes. I know there are larger SA
> environments out there but we all can't publish our local (secret) meta
> rules without the spammers abusing them. However, we can publish these
> safe senders in the SA ruleset to promote good sending to get on the list.
>
> If we want to document these guidelines for how these entries are
> vetted, I will be glad to do that and welcome others to help contribute
> to the entries to get input from all around the world since everyone has
> different mail flow seeing different trustworthy senders.
>
> Dave
>
Re: def_whitelist_auth
Posted by Matthias Leisi <ma...@leisi.net>.
> I proposed the idea on the mailing list a couple of years ago about having a centralized clearinghouse of known good senders but no one stepped up with any ideas. Paul Stead's dkimwl.org is the closest thing to this that I have found and I think this has been added to 3.4.2 commented out so some may enable it but most won’t.
dnswl.org <http://dnswl.org/> also has a domain-based list which may be used on (DKIM-) auth’ed domains.
> I have local whitelist_auth entries that are several times longer than what I am putting into SA with zero customer complaints and I am filtering for about 90,000 mailboxes. I know there are larger SA environments out there but we all can't publish our local (secret) meta rules without the spammers abusing them. However, we can publish these safe senders in the SA ruleset to promote good sending to get on the list.
dnswl.org <http://dnswl.org/> always welcomes new data sources for safe publishing :)
— Matthias
Re: def_whitelist_auth
Posted by Dave Jones <da...@apache.org>.
On 9/23/18 8:31 AM, Kevin A. McGrail wrote:
> On 9/23/2018 9:04 AM, Henrik Krohns wrote:
>> I'm curious, are there guidelines on what can be added here? How are these
>> lists generated? Who verifies and checks that old domains don't age and go
>> to some spammers etc? Most of the listed stuff seems pretty pointless for
>> general population. Paypal and other _globally_ known services make sense.
>>
>> Should we encourage committers to add lists of say local banks and
>> government institutions? I would have plenty, but I don't know if it's
>> SpamAssassins purpose to be a global reputation service with all the
>> maintenance work it requires.
>
> I would say it's valuable and to add it. People can always choose not
> to use our rulesets.
>
I have an automated method to find low-scoring trusted senders from a
highly tuned SA instance. If these entries cause any problems, users
can report it to the mailing list and they will be removed. So far
there has been one entry reported that was too risky for the general
population and it was reported and removed. Otherwise, I have feedback
that it has helped improve FPs.
Keep in mind that these entries are usually subdomains that will not be
user/human mailboxes that can be compromised. These entries are
verified to be system-generated and have other rule hits making them
trustworthy senders that honor opt-out requests without
harvesting/validating the email addresses and handle abuse reports of
their rogue customers.
My goal was to create low/zero risk entries that the mail filtering
industry can see that promotes good SPF, DKIM, and DMARC settings to
raise awareness all around the Internet.
Another purpose of these entries is to allow local meta rules of certain
email content to add points to block junk senders while allowing through
those senders in this list that are known to be good and honor opt-out
requests.
Many of these entries are vetted by private RBLs and DBLs which
indirectly helps those SA installations that aren't able to subscribe to
those RBLs and DBLs or fine tune their SA rules and settings.
I proposed the idea on the mailing list a couple of years ago about
having a centralized clearinghouse of known good senders but no one
stepped up with any ideas. Paul Stead's dkimwl.org is the closest thing
to this that I have found and I think this has been added to 3.4.2
commented out so some may enable it but most won't.
I have local whitelist_auth entries that are several times longer than
what I am putting into SA with zero customer complaints and I am
filtering for about 90,000 mailboxes. I know there are larger SA
environments out there but we all can't publish our local (secret) meta
rules without the spammers abusing them. However, we can publish these
safe senders in the SA ruleset to promote good sending to get on the list.
If we want to document these guidelines for how these entries are
vetted, I will be glad to do that and welcome others to help contribute
to the entries to get input from all around the world since everyone has
different mail flow seeing different trustworthy senders.
Dave
Re: def_whitelist_auth
Posted by "Kevin A. McGrail" <km...@apache.org>.
On 9/23/2018 9:04 AM, Henrik Krohns wrote:
> I'm curious, are there guidelines on what can be added here? How are these
> lists generated? Who verifies and checks that old domains don't age and go
> to some spammers etc? Most of the listed stuff seems pretty pointless for
> general population. Paypal and other _globally_ known services make sense.
>
> Should we encourage committers to add lists of say local banks and
> government institutions? I would have plenty, but I don't know if it's
> SpamAssassins purpose to be a global reputation service with all the
> maintenance work it requires.
I would say it's valuable and to add it. People can always choose not
to use our rulesets.
--
Kevin A. McGrail
VP Fundraising, Apache Software Foundation
Chair Emeritus Apache SpamAssassin Project
https://www.linkedin.com/in/kmcgrail - 703.798.0171