You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by David Jones <dj...@ena.com> on 2018/01/24 14:12:19 UTC
Penalty for no/bad SPF
Google Chrome and other browsers have been slowly penalizing sites not
using encryption to the point that soon they will be alerting users of
plain HTTP sites. This along with letsencrypt.org has been moving the
HTTPS bar forward to improve web security and privacy.
I think it's time for the SA community to help move the bar forward with
SPF. The problem is many sysadmins that don't understand SPF have been
implementing SPF incorrectly (thank you Microsoft Office 365) and
incompletely without understanding they are shooting themselves in the foot.
I decided about a month ago to start sending feedback emails to senders
with SPF PERMERR and SPF FAIL in an attempt to help them help themselves
improve _their_ mail delivery. If you setup your SPF record like
Microsoft recommends with a "-all" and it's not completely covering all
legit sources of email, it's completely useless for any MTAs and mail
filters to take SPF_FAIL hits seriously. We should have rejected the
email per that sending domain's own wishes but we all know they didn't
intend for us to really block it so what good is it?
What does everyone think about slowly increasing the score for SPF_NONE
and SPF_FAIL over time in the SA rulesets to force the awareness and
importance of proper SPF? This may need to have an official
announcement of what the plans/timelines would be so we could get the
word out.
These days with DMARC reporting, it's not impossible to figure out a
good SPF record like it was 10 years ago. The real problem with SMTP in
general is there is no reliable way to get feedback to mail admins
without sending confusing technical emails to regular users.
--
David Jones
Re: Penalty for no/bad SPF
Posted by Vincent Fox <vb...@ucdavis.edu>.
SPF is designed for authentication, not spam filtering. Using a crowbar as a hammer. We apply a small score mainly so we see the elements reported.
If the "majors" are using in their hygiene stack, for evalation like you are, I haven't seen much evidence of that. Of course it's hard to test, since we don't have log access or often intelligible header diagnostics.
But from years of blackbox practical experience working cases:
Cust: My Spam...err bulkmail is going to Junk, can you hook me up w/ SPF!
Tech : OK we hooked you up.
Cust: My Spam....err bulkmail is still going to Junk
Tech: OK now diagnosed your ACTUAL problem (URL, PTR, spammy content. etc). We need to do X, Y, and Z.
Cust: Great that fixed it!
I can't recall a case in the last 5 years where SPF, or DKIM for that matter, tipped anyone from Junk to not.
I apply an increased score for RDNS_NONE. Because I think its an ABOMINATION that so many operations think it OK to skip DNS plumbing. But I do recognize it seems a hopeless battle, trying to clean up the internet.
YMMV.
________________________________
From: Rupert Gallagher <ru...@protonmail.com>
Sent: Wednesday, January 24, 2018 3:00:37 PM
To: David Jones; 'users@spamassassin.apache.org'
Subject: Re: Penalty for no/bad SPF
If all those smarties who speak against spf would simply shut-up and implement it correctly, with dkim and dmarc, and read the dmarc reports, then they would know how valuable it is.
On raising the score: done, long ago, happy with the outcome, and strongly recommend it.
Re: Penalty for no/bad SPF
Posted by Rupert Gallagher <ru...@protonmail.com>.
If all those smarties who speak against spf would simply shut-up and implement it correctly, with dkim and dmarc, and read the dmarc reports, then they would know how valuable it is.
On raising the score: done, long ago, happy with the outcome, and strongly recommend it.
Re: Penalty for no/bad SPF
Posted by David Jones <dj...@ena.com>.
On 01/25/2018 02:13 AM, Matus UHLAR - fantomas wrote:
>> On 01/24/2018 03:45 PM, Joseph Brennan wrote:
>>> The New York Times nytimes.com has a SPF record with too many DNS
>>> lookups. Are you willing to block that? That one amazes me since SPF
>>> is the simplest of these ventures to implement correctly, and since
>>> the Times's frequent mailings of news updates evidently are not
>>> affected enough by SPF fail for the Times to go fix it.
>
> On 24.01.18 16:04, David Jones wrote:
>> The key point here is the bulk nytimes.com email that is system
>> generated, i.e. not humans with real mailboxes that could be
>> compromised, is from subdomains so this entry would be safe since they
>> do have good SPF records on subdomains:
>>
>> whitelist_auth *@*.nytimes.com
>
> this only applies when SPF succeeds so it won't fix their broken SPF :-)
>
But this encourages them to make sure their SPF is not broken if we do
this in the main SA ruleset for everyone running sa-update regularly.
--
David Jones
Re: Penalty for no/bad SPF
Posted by Matus UHLAR - fantomas <uh...@fantomas.sk>.
>On 01/24/2018 03:45 PM, Joseph Brennan wrote:
>>The New York Times nytimes.com has a SPF record with too many DNS
>>lookups. Are you willing to block that? That one amazes me since
>>SPF is the simplest of these ventures to implement correctly, and
>>since the Times's frequent mailings of news updates evidently are
>>not affected enough by SPF fail for the Times to go fix it.
On 24.01.18 16:04, David Jones wrote:
>The key point here is the bulk nytimes.com email that is system
>generated, i.e. not humans with real mailboxes that could be
>compromised, is from subdomains so this entry would be safe since
>they do have good SPF records on subdomains:
>
>whitelist_auth *@*.nytimes.com
this only applies when SPF succeeds so it won't fix their broken SPF :-)
--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Windows 2000: 640 MB ought to be enough for anybody
Re: Penalty for no/bad SPF
Posted by David Jones <dj...@ena.com>.
On 01/24/2018 03:45 PM, Joseph Brennan wrote:
>
> David Jones <dj...@ena.com> wrote:
>
>> SA could be the large force that helps improve the mail standards like
>> DMARC -- SPF + DKIM with a little extra on top.
>
> DMARC is not a standard according to RFC 7489, "Status of This Memo".
> It's just informational, for those who want to play the game. DMARC is
> destroying forwarding and mailing lists, and I'm sorry to see the
> elephants in the email room implementing it-- though Gmail still does
> not always reject based on DMARC reject, as if they use that plus some
> internal system to make the call.
>
> The New York Times nytimes.com has a SPF record with too many DNS
> lookups. Are you willing to block that? That one amazes me since SPF is
> the simplest of these ventures to implement correctly, and since the
> Times's frequent mailings of news updates evidently are not affected
> enough by SPF fail for the Times to go fix it.
>
> Joseph Brennan
> Columbia University IT
>
>
>
>
The key point here is the bulk nytimes.com email that is system
generated, i.e. not humans with real mailboxes that could be
compromised, is from subdomains so this entry would be safe since they
do have good SPF records on subdomains:
whitelist_auth *@*.nytimes.com
In fact, I have put this in the 60_whitelist_auth.cf:
def_whitelist_auth *@*.nytimes.com
... so bulk emails from them should be scoring pretty low for everyone
running sa-update.
This should prove my main point of this thread.
SA actually allows for more than 10 DNS lookups so it's more forgiving
than the actual spec and would likely hit SPF_PASS still as long as the
emails came from a covered source.
I have a recent email from a human-looking email address @nytimes.com
that SA shows as SPF_PASS from a Google mail server.
--
David Jones
Re: Penalty for no/bad SPF
Posted by David Jones <dj...@ena.com>.
On 01/25/2018 11:43 AM, Pedro David Marco wrote:
> Do not forget that accounts in valid servers are hacked oftenly...
>
>
> ---
> PedroD
I have stated multiple times, I am not recommending whitelist_* entries
for domains with user mailboxes that can be compromised unless there is
a very specific reason to.
For example, this would not be wise or recommended:
whitelist_auth *@nytimes.com
but this would be:
whitelist_auth *@*.nytimes.com
As more an more bulk senders use subdomains with good SPF, DKIM, and
DMARC, this allows us to safelist them if they have a good reputation,
handle abuse reports, have proper opt-out, etc.
--
David Jones
Re: Penalty for no/bad SPF
Posted by Pedro David Marco <pe...@yahoo.com>.
Do not forget that accounts in valid servers are hacked oftenly...
---PedroD
Re: Penalty for no/bad SPF
Posted by Karol Augustin <ka...@augustin.pl>.
On 2018-01-24 21:45, Joseph Brennan wrote:
> David Jones <dj...@ena.com> wrote:
>
>> SA could be the large force that helps improve the mail standards like
>> DMARC -- SPF + DKIM with a little extra on top.
>
> DMARC is not a standard according to RFC 7489, "Status of This Memo".
> It's just informational, for those who want to play the game. DMARC is
> destroying forwarding and mailing lists, and I'm sorry to see the
> elephants in the email room implementing it-- though Gmail still does
> not always reject based on DMARC reject, as if they use that plus some
> internal system to make the call.
>
DMARC is not destroying anything if forwarding and mailing lists are
configured properly (like this one). The whole point of DKIM/DMARC was
to authenticate forwarded e-mail, which is broken by design in SPF. If
we could make all mailing lists operators fix the DKIM breaking features
like title modification and adding footers we could just reject
literally everything that fails DKIM. Then the spoofing problem would be
fixed once and for all and SPF would be just a fail-safe in case
something went wrong.
Gmail implementing DMARC is probably the best anti phishing/spoofing
decision made in the last few years. I am sure you would agree if you
were administering paypal's or banks mail servers.
Karol
--
Karol Augustin
karol@augustin.pl
http://karolaugustin.pl/
+353 85 775 5312
Re: Penalty for no/bad SPF
Posted by Martin Gregorie <ma...@gregorie.org>.
On Wed, 2018-01-24 at 14:24 -0800, John Hardin wrote:
> I think he was referring to MTA-side forwarding, not forwarding an
> email you received (which forward comes *from you*).
>
I was wondering if this could be related to Joseph's comment that
"DMARC is destroying forwarding and mailing lists" as some sort of
circumvention, thats's all.
> > and the more so because this is not an optional feature.
>
> *that* is unjustified.
>
Agreed: I feel a Bugzilla request coming on.
Martin
Re: Penalty for no/bad SPF
Posted by John Hardin <jh...@impsec.org>.
On Wed, 24 Jan 2018, Martin Gregorie wrote:
> On Wed, 2018-01-24 at 16:45 -0500, Joseph Brennan wrote:
>> DMARC is not a standard according to RFC 7489, "Status of This Memo".
>> It's just informational, for those who want to play the game. DMARC
>> is destroying forwarding and mailing lists,
>>
> Could this be why recent releases of the Evolution MUA now send
> forwarded messages as attachments appended to a new message?
I think he was referring to MTA-side forwarding, not forwarding an email
you received (which forward comes *from you*).
> Its annoying, particularly if you're intending to intersperse comments
> in the original message as part of a detailed discussion,
Reply?
> and the more so because this is not an optional feature.
*that* is unjustified.
--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
Vista is at best mildly annoying and at worst makes you want to
rush to Redmond, Wash. and rip somebody's liver out. -- Forbes
-----------------------------------------------------------------------
3 days until Wolfgang Amadeus Mozart's 262nd Birthday
Re: Penalty for no/bad SPF
Posted by Martin Gregorie <ma...@gregorie.org>.
On Wed, 2018-01-24 at 16:45 -0500, Joseph Brennan wrote:
> DMARC is not a standard according to RFC 7489, "Status of This Memo".
> It's just informational, for those who want to play the game. DMARC
> is destroying forwarding and mailing lists,
>
Could this be why recent releases of the Evolution MUA now send
forwarded messages as attachments appended to a new message?
Its annoying, particularly if you're intending to intersperse comments
in the original message as part of a detailed discussion, and the more
so because this is not an optional feature.
Apologies if this is a bit OTT, but I don't understand why anybody
would think its a good way to handle forwarded messages.
Martin
Re: Penalty for no/bad SPF
Posted by Joseph Brennan <br...@columbia.edu>.
David Jones <dj...@ena.com> wrote:
> SA could be the large force that helps improve the mail standards like
> DMARC -- SPF + DKIM with a little extra on top.
DMARC is not a standard according to RFC 7489, "Status of This Memo". It's
just informational, for those who want to play the game. DMARC is
destroying forwarding and mailing lists, and I'm sorry to see the elephants
in the email room implementing it-- though Gmail still does not always
reject based on DMARC reject, as if they use that plus some internal system
to make the call.
The New York Times nytimes.com has a SPF record with too many DNS lookups.
Are you willing to block that? That one amazes me since SPF is the simplest
of these ventures to implement correctly, and since the Times's frequent
mailings of news updates evidently are not affected enough by SPF fail for
the Times to go fix it.
Joseph Brennan
Columbia University IT
Re: Penalty for no/bad SPF
Posted by John Hardin <jh...@impsec.org>.
On Wed, 24 Jan 2018, Dianne Skoll wrote:
> On Wed, 24 Jan 2018 14:20:57 -0800 (PST)
> John Hardin <jh...@impsec.org> wrote:
>
>>> At this point, I would be willing to penalize sites with bad SPF
>>> records (syntactically invalid; more than one different SPF record
>>> attached to the same domain, etc.) Those people really deserve
>>> penalties because they've messed up.
>
>> Does that include "+all" or authorizing more than a class-b space
>> through any method, which I'd characterize as "malicious" rather than
>> "messed up"?
>
> +all is malicious for sure. More than a Class-B might just be bad
> planning AKA Microsoft's outbound IP address list.
I was thinking more the case where subrange assignments were used to avoid
explicitly using "+all" as a way to avoid naive malice checks, e.g. doing
"+ip4:0.0.0.0/1 +ip4:255.0.0.0/1". For reliability the threshold size
might need to be larger than a class-B (that was off-the-cuff), or it
might need some explicit whitelisting for broken-yet-legit domains (AKA
msft).
> However, a malicious actor can use the "exists:" mechanism to simulate
> +all in a way that can't easily be proven by an SPF evaluator. :(
>
> I would like to see the exists: mechanism tossed.
So we add a point for using "exists:" and for defining multiple subranges
that add up to > class-B (or whatever threshold seems reasonable) and for
any other constructs that are abused by spammers to define "SPF pass
wherever my bots send from".
It's sounding like we need "SPF cluebat" and "SPF malice" scoring
plugins... :)
--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
...intellectuals have no interest in what _creates_ wealth, and
what _inhibits_ the creation of wealth. They are very concerned
about the _distribution_ of it, but they act as if wealth just
exists somehow. It's like manna from heaven, it's only a
question of how we split it up. -- Thomas Sowell
-----------------------------------------------------------------------
3 days until the 51st anniversary of the loss of Apollo 1
Re: Penalty for no/bad SPF
Posted by Dianne Skoll <df...@roaringpenguin.com>.
On Wed, 24 Jan 2018 14:20:57 -0800 (PST)
John Hardin <jh...@impsec.org> wrote:
> > At this point, I would be willing to penalize sites with bad SPF
> > records (syntactically invalid; more than one different SPF record
> > attached to the same domain, etc.) Those people really deserve
> > penalties because they've messed up.
> Does that include "+all" or authorizing more than a class-b space
> through any method, which I'd characterize as "malicious" rather than
> "messed up"?
+all is malicious for sure. More than a Class-B might just be bad
planning AKA Microsoft's outbound IP address list.
However, a malicious actor can use the "exists:" mechanism to simulate
+all in a way that can't easily be proven by an SPF evaluator. :(
I would like to see the exists: mechanism tossed.
Regards,
Dianne.
Re: Penalty for no/bad SPF
Posted by John Hardin <jh...@impsec.org>.
On Wed, 24 Jan 2018, Bill Cole wrote:
> On 24 Jan 2018, at 17:20 (-0500), John Hardin wrote:
>
>> On Wed, 24 Jan 2018, Dianne Skoll wrote:
>>
>>> At this point, I would be willing to penalize sites with bad SPF
>>> records (syntactically invalid; more than one different SPF record
>>> attached to the same domain, etc.) Those people really deserve
>>> penalties because they've messed up.
>>
>> Does that include "+all" or authorizing more than a class-b space through
>> any method, which I'd characterize as "malicious" rather than "messed up"?
>
> There are entities that still hold fast to their legacy Class A networks and
> expose them to some degree to the world. Those who have tried to change
> policy from inside such an organization might argue that a multiple-B SPF
> authorization is neither malicious nor messed up in itself, but rather merely
> an admission of a reality which i arguably messed up but not at all
> malicious.
Somebody legitimately that big could be whitelisted (SPF Malice score = 0).
--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
To be civilized is to restrain the ability to commit mayhem.
To be incapable of committing mayhem is not the mark of the
civilized, merely the domesticated. -- Trefor Thomas
-----------------------------------------------------------------------
3 days until Wolfgang Amadeus Mozart's 262nd Birthday
Re: Penalty for no/bad SPF
Posted by Bill Cole <sa...@billmail.scconsult.com>.
On 24 Jan 2018, at 17:20 (-0500), John Hardin wrote:
> On Wed, 24 Jan 2018, Dianne Skoll wrote:
>
>> At this point, I would be willing to penalize sites with bad SPF
>> records (syntactically invalid; more than one different SPF record
>> attached to the same domain, etc.) Those people really deserve
>> penalties because they've messed up.
>
> Does that include "+all" or authorizing more than a class-b space
> through any method, which I'd characterize as "malicious" rather than
> "messed up"?
There are entities that still hold fast to their legacy Class A networks
and expose them to some degree to the world. Those who have tried to
change policy from inside such an organization might argue that a
multiple-B SPF authorization is neither malicious nor messed up in
itself, but rather merely an admission of a reality which i arguably
messed up but not at all malicious.
--
Bill Cole
bill@scconsult.com or billcole@apache.org
(AKA @grumpybozo and many *@billmail.scconsult.com addresses)
Currently Seeking Steady Work: https://linkedin.com/in/billcole
Re: Penalty for no/bad SPF
Posted by John Hardin <jh...@impsec.org>.
On Wed, 24 Jan 2018, Dianne Skoll wrote:
> At this point, I would be willing to penalize sites with bad SPF
> records (syntactically invalid; more than one different SPF record
> attached to the same domain, etc.) Those people really deserve
> penalties because they've messed up.
Does that include "+all" or authorizing more than a class-b space through
any method, which I'd characterize as "malicious" rather than "messed up"?
--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
Vista is at best mildly annoying and at worst makes you want to
rush to Redmond, Wash. and rip somebody's liver out. -- Forbes
-----------------------------------------------------------------------
3 days until Wolfgang Amadeus Mozart's 262nd Birthday
Re: Penalty for no/bad SPF
Posted by Dianne Skoll <df...@roaringpenguin.com>.
At this point, I would be willing to penalize sites with bad SPF
records (syntactically invalid; more than one different SPF record
attached to the same domain, etc.) Those people really deserve
penalties because they've messed up.
I would not be willing to penalize sites with *no* SPF at all just
yet. Maybe eventually.
Regards,
Dianne.
Re: Penalty for no/bad SPF
Posted by Joseph Brennan <br...@columbia.edu>.
I've noticed anecdotally that many times when a legitimate source has no
PTR on the mail host, or has a PTR with no matching A record, the domain
does have a SPF record authorizing the host. This carelessness is probably
born from ignorance or overwork. The PTR pass could be used to balance out
the PTR fail. I have not had a chance yet to test this out in real mail
flow to see how close it comes to being something good enough to reject
mail.
Joseph Brennan
Re: Penalty for no/bad SPF
Posted by David Jones <dj...@ena.com>.
On 01/25/2018 02:19 PM, Reindl Harald wrote:
>
>
> Am 25.01.2018 um 20:45 schrieb David Jones:
>> On 01/25/2018 01:19 PM, David Jones wrote:
>>> On 01/25/2018 12:59 PM, Reindl Harald wrote:
>>>>
>>>>
>>>> Am 25.01.2018 um 19:48 schrieb David Jones:
>>>>> Since very few sites can reject on SPF fails because SPF failures
>>>>> are so prevalent on legit email, I don't think this is happening in
>>>>> the real world.
>>>>
>>>> says who?
>>>>
>>>> check_recipient_access proxy:hash:/etc/postfix/skip_spf_check.cf
>>>> permit_dnswl_client dnswl-aggregate.thelounge.net=127.0.0.5
>>>> permit_dnswl_client wl.mailspike.net=127.0.0.[19;20]
>>>> permit_dnswl_client list.dnswl.org=127.0.[0..255].[2;3]
>>>> check_policy_service unix:private/spf-policy
>>>
>>> You are excluding a ton of clients from SPF checks with that config.
>>> How many total IPs are covered in your local
>>> dnswl-aggregate.thelounge.net whitelist?
>>>
>>> My policyd-spf runs from the Postfix master.cf to add headers to all
>>> email for SA to examine.
>>
>> If you are excluding hundred's of thousands of IPs in those 4 Postfix
>> config lines, then that's not a legitimate claim to be rejecting SPF
>> fails.
>
> that's called a smart setup which prevents false-positives and using SPF
> in the mid-stage where the spf-polidcyd for SPF_PAS does a "permit" and
> hands over to the milters to save most of the generic stuff
>
>> You usually are very good with providing numbers so show us how many
>> rejects are happening from SPF failure out of the total volume of email
>
> not much, but that's not because the whitelists but because postscreen
> filters out the majority long before smptd even get a connection
>
> SPF_NONE which maskes it to the milters get a 0.5 penalty
>
> current month
>
> Connections: 215161
> Postscreen WL: 20214 (9.39 %)
> Delivered: 38684
> Blocked: 176477
> Invalid User: 858
> Disallowed User: 16
> Reject Postscreen: 90180
> Reject Postfix: 6403
> Reject Milter: 2188
> Reject Temporary: 497
> Greylisted: 1768
> Blacklist: 89565
> Pregreet: 75250
> Hangup: 124674
> Protocol Error: 118
> Illegal Syntax: 2
> SpamAssassin: 2184
> Virus (Milter): 4
> Virus (SA): 292
> Helo: 263
> Subject: 96
> From: 20
> Attachment: 1
> Header Length: 5
> Sender Regex: 1
> Sender Blocked: 163
> Sender Verify: 42
> Sender Invalid: 61
> Sender Spoofed: 46
> Sender Parked: 0
> Spam-TLD: 82
> PTR Missing: 142
> PTR Generic: 425
> SPF: 155
It sounds like to a good/solid setup but I am still thinking there are a
lot of potential SPF failures being excluded by those 4 Postfix lines.
How many mailboxes do you filter for? Those numbers seem rather low so
maybe a few hundred mailboxes?
--
David Jones
Re: Penalty for no/bad SPF
Posted by David Jones <dj...@ena.com>.
On 01/25/2018 01:19 PM, David Jones wrote:
> On 01/25/2018 12:59 PM, Reindl Harald wrote:
>>
>>
>> Am 25.01.2018 um 19:48 schrieb David Jones:
>>> Since very few sites can reject on SPF fails because SPF failures are
>>> so prevalent on legit email, I don't think this is happening in the
>>> real world.
>>
>> says who?
>>
>> check_recipient_access proxy:hash:/etc/postfix/skip_spf_check.cf
>> permit_dnswl_client dnswl-aggregate.thelounge.net=127.0.0.5
>> permit_dnswl_client wl.mailspike.net=127.0.0.[19;20]
>> permit_dnswl_client list.dnswl.org=127.0.[0..255].[2;3]
>> check_policy_service unix:private/spf-policy
>
> You are excluding a ton of clients from SPF checks with that config. How
> many total IPs are covered in your local dnswl-aggregate.thelounge.net
> whitelist?
>
> My policyd-spf runs from the Postfix master.cf to add headers to all
> email for SA to examine.
>
If you are excluding hundred's of thousands of IPs in those 4 Postfix
config lines, then that's not a legitimate claim to be rejecting SPF fails.
You usually are very good with providing numbers so show us how many
rejects are happening from SPF failure out of the total volume of email.
--
David Jones
Re: Penalty for no/bad SPF
Posted by David Jones <dj...@ena.com>.
On 01/25/2018 12:59 PM, Reindl Harald wrote:
>
>
> Am 25.01.2018 um 19:48 schrieb David Jones:
>> Since very few sites can reject on SPF fails because SPF failures are
>> so prevalent on legit email, I don't think this is happening in the
>> real world.
>
> says who?
>
> check_recipient_access proxy:hash:/etc/postfix/skip_spf_check.cf
> permit_dnswl_client dnswl-aggregate.thelounge.net=127.0.0.5
> permit_dnswl_client wl.mailspike.net=127.0.0.[19;20]
> permit_dnswl_client list.dnswl.org=127.0.[0..255].[2;3]
> check_policy_service unix:private/spf-policy
You are excluding a ton of clients from SPF checks with that config.
How many total IPs are covered in your local
dnswl-aggregate.thelounge.net whitelist?
My policyd-spf runs from the Postfix master.cf to add headers to all
email for SA to examine.
--
David Jones
Re: Penalty for no/bad SPF
Posted by David Jones <dj...@ena.com>.
On 01/25/2018 12:27 PM, RW wrote:
> On Thu, 25 Jan 2018 09:53:12 -0600
> David Jones wrote:
>
>> On 01/25/2018 09:34 AM, RW wrote:
>
>>>> There is nothing wrong with stopping a soft fail if that is what
>>>> they want to do. In fact, most people should stop at soft fail
>>>> unless they really know what they are doing or they are a major
>>>> brand with a high risk spoofing.
>>>
>>> There's more to it than that.
>>>
>>> All of the above use DMARC and if you use -all in combination with
>>> DMARC you are allowing the SPF result (which is only one component
>>> of DMARC) and SPF's legacy policy mechanism to overide both the
>>> DMARC result and the DMARC policy. The DMARC RFC has a warning
>>> about this.
>>
>> My understanding based on real world results and the link below says
>> that for DMARC to pass you have to have SPF pass and envelope-from
>> domain alignment _OR_ DKIM pass and header From: domain alignment.
>> If you have both then it's even better.
>>
>> https://blog.returnpath.com/how-to-explain-dmarc-in-plain-english/
>>
>> SPF_PASS can hit with either "~all" or "-all" so it doesn't make a
>> difference to DMARC pass.
>
> From RFC 7489
>
> .10.1. Issues Specific to SPF
>
> ...
>
> Some receiver architectures might implement SPF in advance of any
> DMARC operations. This means that a "-" prefix on a sender's SPF
> mechanism, such as "-all", could cause that rejection to go into
> effect early in handling, causing message rejection before any DMARC
> processing takes place. Operators choosing to use "-all" should be
> aware of this.
>
Since very few sites can reject on SPF fails because SPF failures are so
prevalent on legit email, I don't think this is happening in the real world.
This is my main point that some large force like Google or SA needs to
take steps to make SPF failures worth something. This would start with
Microsoft stop recommending "-all" to their Office 365 customers who
don't know what they are doing. Google correctly recommends "~all" to
their customers as a default. Then if you know what you are doing and
have run DMARC reporting for several months to get your SPF record
complete, then you switch to "-all".
--
David Jones
Re: Penalty for no/bad SPF
Posted by RW <rw...@googlemail.com>.
On Thu, 25 Jan 2018 09:53:12 -0600
David Jones wrote:
> On 01/25/2018 09:34 AM, RW wrote:
> >> There is nothing wrong with stopping a soft fail if that is what
> >> they want to do. In fact, most people should stop at soft fail
> >> unless they really know what they are doing or they are a major
> >> brand with a high risk spoofing.
> >
> > There's more to it than that.
> >
> > All of the above use DMARC and if you use -all in combination with
> > DMARC you are allowing the SPF result (which is only one component
> > of DMARC) and SPF's legacy policy mechanism to overide both the
> > DMARC result and the DMARC policy. The DMARC RFC has a warning
> > about this.
>
> My understanding based on real world results and the link below says
> that for DMARC to pass you have to have SPF pass and envelope-from
> domain alignment _OR_ DKIM pass and header From: domain alignment.
> If you have both then it's even better.
>
> https://blog.returnpath.com/how-to-explain-dmarc-in-plain-english/
>
> SPF_PASS can hit with either "~all" or "-all" so it doesn't make a
> difference to DMARC pass.
From RFC 7489
.10.1. Issues Specific to SPF
...
Some receiver architectures might implement SPF in advance of any
DMARC operations. This means that a "-" prefix on a sender's SPF
mechanism, such as "-all", could cause that rejection to go into
effect early in handling, causing message rejection before any DMARC
processing takes place. Operators choosing to use "-all" should be
aware of this.
Re: Penalty for no/bad SPF
Posted by David Jones <dj...@ena.com>.
On 01/25/2018 09:34 AM, RW wrote:
> On Wed, 24 Jan 2018 16:26:58 -0600
> David Jones wrote:
>
>> On 01/24/2018 04:00 PM, Vincent Fox wrote:
>
>>> However, look at all the major providers with messed up records and
>>> neutral or soft fail. They should have the most resources to
>>> accomplish this and the most incentives to list all their
>>> netblocks and set to hard fail.
>>>
>>>
>>> Google is soft fail.
>>> Hotmail is soft fail.
>
>
> And Yahoo has the strongest DMARC policy and the weakest SPF policy.
>
>> There is nothing wrong with stopping a soft fail if that is what they
>> want to do. In fact, most people should stop at soft fail unless
>> they really know what they are doing or they are a major brand with a
>> high risk spoofing.
>
> There's more to it than that.
>
> All of the above use DMARC and if you use -all in combination with
> DMARC you are allowing the SPF result (which is only one component of
> DMARC) and SPF's legacy policy mechanism to overide both the DMARC
> result and the DMARC policy. The DMARC RFC has a warning about this.
>
My understanding based on real world results and the link below says
that for DMARC to pass you have to have SPF pass and envelope-from
domain alignment _OR_ DKIM pass and header From: domain alignment. If
you have both then it's even better.
https://blog.returnpath.com/how-to-explain-dmarc-in-plain-english/
SPF_PASS can hit with either "~all" or "-all" so it doesn't make a
difference to DMARC pass.
--
David Jones
Re: Penalty for no/bad SPF
Posted by RW <rw...@googlemail.com>.
On Wed, 24 Jan 2018 16:26:58 -0600
David Jones wrote:
> On 01/24/2018 04:00 PM, Vincent Fox wrote:
> > However, look at all the major providers with messed up records and
> > neutral or soft fail. They should have the most resources to
> > accomplish this and the most incentives to list all their
> > netblocks and set to hard fail.
> >
> >
> > Google is soft fail.
> > Hotmail is soft fail.
And Yahoo has the strongest DMARC policy and the weakest SPF policy.
> There is nothing wrong with stopping a soft fail if that is what they
> want to do. In fact, most people should stop at soft fail unless
> they really know what they are doing or they are a major brand with a
> high risk spoofing.
There's more to it than that.
All of the above use DMARC and if you use -all in combination with
DMARC you are allowing the SPF result (which is only one component of
DMARC) and SPF's legacy policy mechanism to overide both the DMARC
result and the DMARC policy. The DMARC RFC has a warning about this.
Re: Penalty for no/bad SPF
Posted by David Jones <dj...@ena.com>.
On 01/24/2018 04:00 PM, Vincent Fox wrote:
> so there's this argument that goes:
>
>
> "well we won't really see the benefits until it's FULLY and RIGIDLY
> implemented."
>
>
> However, look at all the major providers with messed up records and
> neutral or soft fail. They should have the most resources to
> accomplish this and the most incentives to list all their netblocks and
> set to hard fail.
>
>
> Google is soft fail.
>
> Hotmail is soft fail.
>
> (etc etc ad nauseum)
>
>
> I rest my case.
>
>
There is nothing wrong with stopping a soft fail if that is what they
want to do. In fact, most people should stop at soft fail unless they
really know what they are doing or they are a major brand with a high
risk spoofing.
People are blindly following Microsoft's DNS entries for Office 365
setup with "-all" when they don't know what they are doing. Microsoft
should be telling people to do "~all" in their setup instructions. Then
Microsoft should be checking their customer's SPF records for them and
showing them when it's broken in the the Admin Center.
1. We need SPF_FAIL hits to mean something and they don't.
2. We can use subdomains with SPF_PASS to safelist trusted senders that
are targets of spoofing.
> After 14+ years we are still having this ridiculous argument about how
> in 14 MORE years when we finally fully implement this flawed technology,
> it'll do something useful. Meanwhile i see it as being more risk than
> benefit.
>
With a big force like SA or Google, we could do this in a couple of
years slowly and easily then start doing the same for DKIM.
>
> Frankly I'd rather these manhours be used on having correct A & PTR
> records, which seems to be beyond the pale for some bulkmail vendors.
>
We could do the same thing for RDNS_NONE hits. Good idea.
--
David Jones
Re: Penalty for no/bad SPF
Posted by Vincent Fox <vb...@ucdavis.edu>.
so there's this argument that goes:
"well we won't really see the benefits until it's FULLY and RIGIDLY implemented."
However, look at all the major providers with messed up records and neutral or soft fail. They should have the most resources to accomplish this and the most incentives to list all their netblocks and set to hard fail.
Google is soft fail.
Hotmail is soft fail.
(etc etc ad nauseum)
I rest my case.
After 14+ years we are still having this ridiculous argument about how in 14 MORE years when we finally fully implement this flawed technology, it'll do something useful. Meanwhile i see it as being more risk than benefit.
Frankly I'd rather these manhours be used on having correct A & PTR records, which seems to be beyond the pale for some bulkmail vendors.
________________________________
From: David Jones <dj...@ena.com>
Sent: Wednesday, January 24, 2018 12:12:56 PM
To: users@spamassassin.apache.org
Subject: Re: Penalty for no/bad SPF
On 01/24/2018 01:58 PM, Vincent Fox wrote:
> I'd rather not think about the manhours I've wasted this year on SPF.
>
>
> The guy at Evotec.com, among others, who thinks rejecting
>
> for SOFTFAIL is a perfectly valid anti-spoofing strategy and
>
> doesn't blink when pointed to RFC 4408 sec 2.5.5.
>
>
> Vendors who's first response is:
>
> "Our LEGIT spam....errr bulkmail is ending in your Junk. Response
>
> #1 in our binder is you MUST list us in your SPF record."
>
> Dig, dig, dig maillogs. All emails using Envelope From properly
>
> so SPF is a waste of everyone's time.
>
The Internet is very slow to change. It takes a large force like Google
to improve things slowly over time. They are doing good work in the TLS
and browser encryption area. SA could be the large force that helps
improve the mail standards like DMARC -- SPF + DKIM with a little extra
on top.
>
> Records we included to ours, where the vendor makes a typo in
>
> THEIR SPF record on a Friday night. Or decides to add 9 sub-includes.
>
> Either way our record suddenly returning PERMERROR and we
>
> have to get someone in, and boot vendor off the island on a Sunday.
>
I have a script that checks all of our customer's SPF records for syntax
problems and too many DNS lookups based on pyspf just like
http://www.kitterman.com/spf/validate.html does so I can correct it or
notify them immediately.
>
> Endless hours explaining to campus clients, what SPF is and why
>
If SA all around the world says the same thing you are telling them then
they will have to listen and fix their problem or remove their SPF
record which is better than having an incorrect one.
> it is not a good primary strategy to solve Junk mail issues
>
> The only good thing I have to say about SPF, is it seems to
>
> be a permanent employment program for people who are
>
> otherwise useless.
>
--
David Jones
Re: Penalty for no/bad SPF
Posted by David Jones <dj...@ena.com>.
On 01/24/2018 01:58 PM, Vincent Fox wrote:
> I'd rather not think about the manhours I've wasted this year on SPF.
>
>
> The guy at Evotec.com, among others, who thinks rejecting
>
> for SOFTFAIL is a perfectly valid anti-spoofing strategy and
>
> doesn't blink when pointed to RFC 4408 sec 2.5.5.
>
>
> Vendors who's first response is:
>
> "Our LEGIT spam....errr bulkmail is ending in your Junk. Response
>
> #1 in our binder is you MUST list us in your SPF record."
>
> Dig, dig, dig maillogs. All emails using Envelope From properly
>
> so SPF is a waste of everyone's time.
>
The Internet is very slow to change. It takes a large force like Google
to improve things slowly over time. They are doing good work in the TLS
and browser encryption area. SA could be the large force that helps
improve the mail standards like DMARC -- SPF + DKIM with a little extra
on top.
>
> Records we included to ours, where the vendor makes a typo in
>
> THEIR SPF record on a Friday night. Or decides to add 9 sub-includes.
>
> Either way our record suddenly returning PERMERROR and we
>
> have to get someone in, and boot vendor off the island on a Sunday.
>
I have a script that checks all of our customer's SPF records for syntax
problems and too many DNS lookups based on pyspf just like
http://www.kitterman.com/spf/validate.html does so I can correct it or
notify them immediately.
>
> Endless hours explaining to campus clients, what SPF is and why
>
If SA all around the world says the same thing you are telling them then
they will have to listen and fix their problem or remove their SPF
record which is better than having an incorrect one.
> it is not a good primary strategy to solve Junk mail issues
>
> The only good thing I have to say about SPF, is it seems to
>
> be a permanent employment program for people who are
>
> otherwise useless.
>
--
David Jones
Re: Penalty for no/bad SPF
Posted by Vincent Fox <vb...@ucdavis.edu>.
I'd rather not think about the manhours I've wasted this year on SPF.
The guy at Evotec.com, among others, who thinks rejecting
for SOFTFAIL is a perfectly valid anti-spoofing strategy and
doesn't blink when pointed to RFC 4408 sec 2.5.5.
Vendors who's first response is:
"Our LEGIT spam....errr bulkmail is ending in your Junk. Response
#1 in our binder is you MUST list us in your SPF record."
Dig, dig, dig maillogs. All emails using Envelope From properly
so SPF is a waste of everyone's time.
Records we included to ours, where the vendor makes a typo in
THEIR SPF record on a Friday night. Or decides to add 9 sub-includes.
Either way our record suddenly returning PERMERROR and we
have to get someone in, and boot vendor off the island on a Sunday.
Endless hours explaining to campus clients, what SPF is and why
it is not a good primary strategy to solve Junk mail issues.
The only good thing I have to say about SPF, is it seems to
be a permanent employment program for people who are
otherwise useless.
________________________________
From: Martin Gregorie <ma...@gregorie.org>
Sent: Wednesday, January 24, 2018 11:32:27 AM
To: users@spamassassin.apache.org
Subject: Re: Penalty for no/bad SPF
On Wed, 2018-01-24 at 19:01 +0000, Vincent Fox wrote:
> SPF is a zombie legacy that someone should shoot in
> the head.
>
SPF is still good for what I've always thought was its main use:
detecting spam delivered by backscatter. Given that its dirt cheap to
implement, and easy too verify now that there are good checking tools
there's no reason or necessity to kill it.
> Maybe then we could design something that is useful for what we all
> desire, which is properly authenticating senders.
>
... provided that you can persuade all legit senders to buy into using
it while preventing all spammers and bots from hijacking it.
Martin
Re: Penalty for no/bad SPF
Posted by Martin Gregorie <ma...@gregorie.org>.
On Wed, 2018-01-24 at 19:01 +0000, Vincent Fox wrote:
> SPF is a zombie legacy that someone should shoot in
> the head.
>
SPF is still good for what I've always thought was its main use:
detecting spam delivered by backscatter. Given that its dirt cheap to
implement, and easy too verify now that there are good checking tools
there's no reason or necessity to kill it.
> Maybe then we could design something that is useful for what we all
> desire, which is properly authenticating senders.
>
... provided that you can persuade all legit senders to buy into using
it while preventing all spammers and bots from hijacking it.
Martin
Re: Penalty for no/bad SPF
Posted by Dianne Skoll <df...@roaringpenguin.com>.
On Wed, 24 Jan 2018 19:01:28 +0000
Vincent Fox <vb...@ucdavis.edu> wrote:
> SPF is a zombie legacy that someone should shoot in
> the head.
+1
> Maybe then we could design something that
> is useful for what we all desire, which is properly
> authenticating senders.
We cannot authenticate senders and keep SMTP as useful as it is now.
The whole beauty of email is that some stranger can contact you out of the
blue. And that mailing lists can send mail on your behalf.
We can work on people not being able to spoof "well-known" senders; for
that, I'd much prefer DKIM+DMARC over SPF. SPF was a mistake and we should
all admit that.
Regards,
Dianne.
Re: Penalty for no/bad SPF
Posted by David Jones <dj...@ena.com>.
On 01/24/2018 01:01 PM, Vincent Fox wrote:
> SPF is designed for whitelisting, not blacklist.
>
Not true. We are supposed to reject email that doesn't pass SPF checks
if the domain has told us to with "-all". That is a form of
blacklisting controlled by the domain itself to protect it's own
brand/reputation from spoofing. If you don't know what you are doing
and create an SPF record with "-all", then that's your own fault.
Google has started putting many of these emails with SPF_FAIL in their
Spam folder which puts more importance to get your SPF record correct
and complete.
>
> Remember when "shields" appeared in mail
>
> clients, and how fast that feature disappeared?
>
> Far too many people clicking on phish that seemed
>
> "authentic". With the explosion of cheap domains
>
> and registrars, there's really no snowshoe Black Hat
>
> operation that can't comply. Compliance is 99.9999%
>
> in every phish I've investigated last year, the outlier
>
> I can recall was a simple typo in 1 server in the
>
> sender's network.
>
SPF is authorization not authentication. There is a difference and
complete security requires both. You use SPF to tell the Internet
authorized mail servers for your domain. That doesn't have a direct tie
to spam but it does allow a level of whitelisting (I like to call it
safelisting) for senders you trust.
>
> SPF is a zombie legacy that someone should shoot in
>
> the head. Maybe then we could design something that
>
> is useful for what we all desire, which is properly
>
> authenticating senders.
>
>
If keep in mind that SPF is authorization and DKIM is authentication,
you really need both. When a sender with DMARC p=reject aligns
perfectly in both SPF and DKIM then all you know is that email is legit
and not spoofed. It doesn't say anything about spam or ham in the
content. If you whitelist trusted senders then you segment them out of
the way which allows fine tuning on the rest of the mail flow.
>
> ------------------------------------------------------------------------
> *From:* David Jones <dj...@ena.com>
> *Sent:* Wednesday, January 24, 2018 6:12:19 AM
> *To:* 'users@spamassassin.apache.org'
> *Subject:* Penalty for no/bad SPF
> Google Chrome and other browsers have been slowly penalizing sites not
> using encryption to the point that soon they will be alerting users of
> plain HTTP sites. This along with letsencrypt.org has been moving the
> HTTPS bar forward to improve web security and privacy.
>
> I think it's time for the SA community to help move the bar forward with
> SPF. The problem is many sysadmins that don't understand SPF have been
> implementing SPF incorrectly (thank you Microsoft Office 365) and
> incompletely without understanding they are shooting themselves in the foot.
>
> I decided about a month ago to start sending feedback emails to senders
> with SPF PERMERR and SPF FAIL in an attempt to help them help themselves
> improve _their_ mail delivery. If you setup your SPF record like
> Microsoft recommends with a "-all" and it's not completely covering all
> legit sources of email, it's completely useless for any MTAs and mail
> filters to take SPF_FAIL hits seriously. We should have rejected the
> email per that sending domain's own wishes but we all know they didn't
> intend for us to really block it so what good is it?
>
> What does everyone think about slowly increasing the score for SPF_NONE
> and SPF_FAIL over time in the SA rulesets to force the awareness and
> importance of proper SPF? This may need to have an official
> announcement of what the plans/timelines would be so we could get the
> word out.
>
> These days with DMARC reporting, it's not impossible to figure out a
> good SPF record like it was 10 years ago. The real problem with SMTP in
> general is there is no reliable way to get feedback to mail admins
> without sending confusing technical emails to regular users.
>
> --
> David Jones
--
David Jones
Re: Penalty for no/bad SPF
Posted by Vincent Fox <vb...@ucdavis.edu>.
SPF is designed for whitelisting, not blacklist.
Remember when "shields" appeared in mail
clients, and how fast that feature disappeared?
Far too many people clicking on phish that seemed
"authentic". With the explosion of cheap domains
and registrars, there's really no snowshoe Black Hat
operation that can't comply. Compliance is 99.9999%
in every phish I've investigated last year, the outlier
I can recall was a simple typo in 1 server in the
sender's network.
SPF is a zombie legacy that someone should shoot in
the head. Maybe then we could design something that
is useful for what we all desire, which is properly
authenticating senders.
________________________________
From: David Jones <dj...@ena.com>
Sent: Wednesday, January 24, 2018 6:12:19 AM
To: 'users@spamassassin.apache.org'
Subject: Penalty for no/bad SPF
Google Chrome and other browsers have been slowly penalizing sites not
using encryption to the point that soon they will be alerting users of
plain HTTP sites. This along with letsencrypt.org has been moving the
HTTPS bar forward to improve web security and privacy.
I think it's time for the SA community to help move the bar forward with
SPF. The problem is many sysadmins that don't understand SPF have been
implementing SPF incorrectly (thank you Microsoft Office 365) and
incompletely without understanding they are shooting themselves in the foot.
I decided about a month ago to start sending feedback emails to senders
with SPF PERMERR and SPF FAIL in an attempt to help them help themselves
improve _their_ mail delivery. If you setup your SPF record like
Microsoft recommends with a "-all" and it's not completely covering all
legit sources of email, it's completely useless for any MTAs and mail
filters to take SPF_FAIL hits seriously. We should have rejected the
email per that sending domain's own wishes but we all know they didn't
intend for us to really block it so what good is it?
What does everyone think about slowly increasing the score for SPF_NONE
and SPF_FAIL over time in the SA rulesets to force the awareness and
importance of proper SPF? This may need to have an official
announcement of what the plans/timelines would be so we could get the
word out.
These days with DMARC reporting, it's not impossible to figure out a
good SPF record like it was 10 years ago. The real problem with SMTP in
general is there is no reliable way to get feedback to mail admins
without sending confusing technical emails to regular users.
--
David Jones
Re: Penalty for no/bad SPF
Posted by Giovanni Bechis <gi...@paclan.it>.
On Wed, Jan 24, 2018 at 08:12:19AM -0600, David Jones wrote:
> Google Chrome and other browsers have been slowly penalizing sites not
> using encryption to the point that soon they will be alerting users of
> plain HTTP sites. This along with letsencrypt.org has been moving the
> HTTPS bar forward to improve web security and privacy.
>
> I think it's time for the SA community to help move the bar forward with
> SPF. The problem is many sysadmins that don't understand SPF have been
> implementing SPF incorrectly (thank you Microsoft Office 365) and
> incompletely without understanding they are shooting themselves in the foot.
>
> I decided about a month ago to start sending feedback emails to senders
> with SPF PERMERR and SPF FAIL in an attempt to help them help themselves
> improve _their_ mail delivery. If you setup your SPF record like
> Microsoft recommends with a "-all" and it's not completely covering all
> legit sources of email, it's completely useless for any MTAs and mail
> filters to take SPF_FAIL hits seriously. We should have rejected the
> email per that sending domain's own wishes but we all know they didn't
> intend for us to really block it so what good is it?
>
> What does everyone think about slowly increasing the score for SPF_NONE
> and SPF_FAIL over time in the SA rulesets to force the awareness and
> importance of proper SPF? This may need to have an official
> announcement of what the plans/timelines would be so we could get the
> word out.
>
it sounds like a plan, I am all for that.
Giovanni
> These days with DMARC reporting, it's not impossible to figure out a
> good SPF record like it was 10 years ago. The real problem with SMTP in
> general is there is no reliable way to get feedback to mail admins
> without sending confusing technical emails to regular users.
>
> --
> David Jones
Re: Penalty for no/bad SPF
Posted by Dianne Skoll <df...@roaringpenguin.com>.
On Thu, 25 Jan 2018 05:19:38 -0500
Bill Shirley <bi...@philly.polymerindustries.biz> wrote:
> I'm all for tightening up standards compliance with email, but what I
> would see if this would happen is a request from my customers saying:
> Bob'semails (bob@bad-spf.com) are going to the spam folder; whitelist
> him please Bob's email admin won't ever know that his SPF is failing.
This matches our experience. We perform a very mild form of SPF enforcement:
If a message fails or softfails SPF, then we ignore any sender or domain
whitelist. One of our top support questions is how to allow the whitelist
anyway. :(
Regards,
Dianne.
Re: Penalty for no/bad SPF
Posted by David Jones <dj...@ena.com>.
On 01/25/2018 04:19 AM, Bill Shirley wrote:
> I'm all for tightening up standards compliance with email, but what I
> would see
> if this would happen is a request from my customers saying:
> Bob'semails (bob@bad-spf.com) are going to the spam folder; whitelist
> him please
> Bob's email admin won't ever know that his SPF is failing.
>
> Bill
>
I am trying to help with this problem by sending emails nor more than
per day like this one back to the sender:
https://pastebin.com/DC2EA34s
This way the sender knows that they just sent the email and gets this
response immediately. If they can read the first sentence and forward
it to their mail admins, then there's a small chance this will help.
I am sure that some users are setting up an Inbox rule to delete this
email. I have had a couple of responses wanting help with fixing their
SPF record so there is some good coming from this.
This email response is completely automated. I am using swatch to watch
for SPF_FAIL in my mail logs then it launches a script that generates
that form email filling some details. Swatch is limited to replying
only once per every 24 hours per sender address.
--
David Jones
Re: Penalty for no/bad SPF
Posted by David Jones <dj...@ena.com>.
On 01/25/2018 04:49 AM, Giovanni Bechis wrote:
> On 01/25/18 11:19, Bill Shirley wrote:
>> I'm all for tightening up standards compliance with email, but what I would see
>> if this would happen is a request from my customers saying:
>> Bob'semails (bob@bad-spf.com) are going to the spam folder; whitelist him please
>> Bob's email admin won't ever know that his SPF is failing.
>>
> IMHO when the majority (including some big providers) will put Bob's email into spam folders, his admin will take a look at his spf records; nobody knows if it will work or not but I think it is worth trying.
> Nobody cared about https till last year, now even very simple web sites uses it only because web browsers are giving warnings, not because they do care about security.
> Giovanni
>
Exactly. If we partnered with Google (maybe Google Summer of Code) to
add DMARC and ARC support to SA. Then we announce that we want to move
these email standards forward over the next two years.
This would help in the fight against spoofing and improve our options to
whitelist good senders to better target the spammers.
--
David Jones
Re: Penalty for no/bad SPF
Posted by Giovanni Bechis <gi...@paclan.it>.
On 01/25/18 11:19, Bill Shirley wrote:
> I'm all for tightening up standards compliance with email, but what I would see
> if this would happen is a request from my customers saying:
> Bob'semails (bob@bad-spf.com) are going to the spam folder; whitelist him please
> Bob's email admin won't ever know that his SPF is failing.
>
IMHO when the majority (including some big providers) will put Bob's email into spam folders, his admin will take a look at his spf records; nobody knows if it will work or not but I think it is worth trying.
Nobody cared about https till last year, now even very simple web sites uses it only because web browsers are giving warnings, not because they do care about security.
Giovanni
> Bill
>
> On 1/24/2018 2:59 PM, David Jones wrote:
>> On 01/24/2018 01:33 PM, Bill Cole wrote:
>>> On 24 Jan 2018, at 9:12, David Jones wrote:
>>>
>>>> What does everyone think about slowly increasing the score for SPF_NONE and SPF_FAIL over time in the SA rulesets to force the awareness and importance of proper SPF?
>>>
>>> -1
>>>
>>> In every real mailstream I've worked with in the lifetime of SPF, lack of SPF has *always* had a correlation with ham, not spam.
>>>
>>
>> I am not suggesting that SPF_PASS = ham and SPF_FAIL = spam.
>>
>>>
>>> SPF hard failures are a more complicated case because the sort of spam that hits SPF_FAIL tends to come from IPs that show up in good DNSBLs within a few minutes, making it hard for a site using DNSBLs to know how much of it there is. With that caveat, I see more ham hitting SPF_FAIL than I do spam where SPF_FAIL (which I have locally nailed at 2.0) is a decisive factor. Most SPF_FAIL spam scores well into double digits here.
>>
>> I am proposing that if SPF were more accurately deployed then SPF_FAIL would be worth something. We could whitelist_auth more trusted senders and then be able to turn up the scores for the rest of the mail flow.
>>
>> If the huge SA community around the world were to help push SPF adoption and accurate deployments, then we could move on to DKIM too. Right now, the best option we have is to get DMARC properly deployed as much as possible where p=reject actually rejects the message unlike SPF_FAIL that we can't trust.
>>
>
Re: Penalty for no/bad SPF
Posted by Bill Shirley <bi...@philly.polymerindustries.biz>.
I'm all for tightening up standards compliance with email, but what I would see
if this would happen is a request from my customers saying:
Bob'semails (bob@bad-spf.com) are going to the spam folder; whitelist him please
Bob's email admin won't ever know that his SPF is failing.
Bill
On 1/24/2018 2:59 PM, David Jones wrote:
> On 01/24/2018 01:33 PM, Bill Cole wrote:
>> On 24 Jan 2018, at 9:12, David Jones wrote:
>>
>>> What does everyone think about slowly increasing the score for SPF_NONE and SPF_FAIL over time in the SA rulesets to force
>>> the awareness and importance of proper SPF?
>>
>> -1
>>
>> In every real mailstream I've worked with in the lifetime of SPF, lack of SPF has *always* had a correlation with ham, not spam.
>>
>
> I am not suggesting that SPF_PASS = ham and SPF_FAIL = spam.
>
>>
>> SPF hard failures are a more complicated case because the sort of spam that hits SPF_FAIL tends to come from IPs that show up
>> in good DNSBLs within a few minutes, making it hard for a site using DNSBLs to know how much of it there is. With that
>> caveat, I see more ham hitting SPF_FAIL than I do spam where SPF_FAIL (which I have locally nailed at 2.0) is a decisive
>> factor. Most SPF_FAIL spam scores well into double digits here.
>
> I am proposing that if SPF were more accurately deployed then SPF_FAIL would be worth something. We could whitelist_auth more
> trusted senders and then be able to turn up the scores for the rest of the mail flow.
>
> If the huge SA community around the world were to help push SPF adoption and accurate deployments, then we could move on to
> DKIM too. Right now, the best option we have is to get DMARC properly deployed as much as possible where p=reject actually
> rejects the message unlike SPF_FAIL that we can't trust.
>
Re: Penalty for no/bad SPF
Posted by Ian Zimmerman <it...@very.loosely.org>.
On 2018-01-24 18:10, Bill Cole wrote:
> 1. Mail with an envelope sender domain that has no SPF record is more
> likely to be spam than the overall mail stream.
>
> 2. Mail whose envelope sender domain has a published SPF record which
> repudiates the sending IP is more likely to be spam than the overall
> mail stream.
>
> I don't see evidence that either of those are true now, that they have
> ever been true, or that they are becoming closer to true over time.
I am not taking sides in this dispute, but I'd like to point out that
there is a phenomenon called "self-fulfilling prophesy". As in all
affairs that involve human behavior and persuasion.
--
Please don't Cc: me privately on mailing lists and Usenet,
if you also post the followup to the list or newsgroup.
To reply privately _only_ on Usenet, fetch the TXT record for the domain.
Re: Penalty for no/bad SPF
Posted by Bill Cole <sa...@billmail.scconsult.com>.
On 24 Jan 2018, at 14:59 (-0500), David Jones wrote:
> On 01/24/2018 01:33 PM, Bill Cole wrote:
>> On 24 Jan 2018, at 9:12, David Jones wrote:
>>
>>> What does everyone think about slowly increasing the score for
>>> SPF_NONE and SPF_FAIL over time in the SA rulesets to force the
>>> awareness and importance of proper SPF?
>>
>> -1
>>
>> In every real mailstream I've worked with in the lifetime of SPF,
>> lack of SPF has *always* had a correlation with ham, not spam.
>>
>
> I am not suggesting that SPF_PASS = ham and SPF_FAIL = spam.
I did not think or say that you were.
You are proposing that SA as a project should *act as if* these
correlations exist and are steadily becoming stronger:
1. Mail with an envelope sender domain that has no SPF record is more
likely to be spam than the overall mail stream.
2. Mail whose envelope sender domain has a published SPF record which
repudiates the sending IP is more likely to be spam than the overall
mail stream.
I don't see evidence that either of those are true now, that they have
ever been true, or that they are becoming closer to true over time.
>> SPF hard failures are a more complicated case because the sort of
>> spam that hits SPF_FAIL tends to come from IPs that show up in good
>> DNSBLs within a few minutes, making it hard for a site using DNSBLs
>> to know how much of it there is. With that caveat, I see more ham
>> hitting SPF_FAIL than I do spam where SPF_FAIL (which I have locally
>> nailed at 2.0) is a decisive factor. Most SPF_FAIL spam scores well
>> into double digits here.
>
> I am proposing that if SPF were more accurately deployed then SPF_FAIL
> would be worth something.
That is a truism but what you are proposing is that SA should behave as
if SPF deployment is becoming broader and more accurate steadily over
time, as a sort of sympathetic magic to drive that improvement.
I am saying that history implies that this would not work as hoped.
Also, it is a bad precedent for project policy.Slowly rising SA scores
don't exert any pressure on a sender with broken SPF until they actually
cause mail to be dropped or rejected. We should not be intentionally
increasing false positives no matter how noble the end goal.
Individual sites obviously can make that choice on their own, now. For
example, on my own personal system I would very likely see some FPs due
to my unjustifiably high fixed score for SPF_FAIL, were it not for other
bespoke tweaks which make actually effective FPs extremely unlikely.
> We could whitelist_auth more trusted senders and then be able to turn
> up the scores for the rest of the mail flow.
>
> If the huge SA community around the world were to help push SPF
> adoption and accurate deployments, then we could move on to DKIM too.
> Right now, the best option we have is to get DMARC properly deployed
> as much as possible where p=reject actually rejects the message unlike
> SPF_FAIL that we can't trust.
SA is not the vehicle for such pressure. What has been driving careful
DMARC deployment is not the long tail of sites with <10k users using SA,
it is the handful of behemoth providers plus some key commercial and
government systems that honor p=reject. This was also the strongest
driver for SPF adoption, until those behemoths recognized how flawed SPF
& DKIM alone were and came up with DMARC as a "solution" with the
unsurprising side effect of damaging traditional discussion mailing
lists.
--
Bill Cole
bill@scconsult.com or billcole@apache.org
(AKA @grumpybozo and many *@billmail.scconsult.com addresses)
Currently Seeking Steady Work: https://linkedin.com/in/billcole
Re: Penalty for no/bad SPF
Posted by David Jones <dj...@ena.com>.
On 01/24/2018 01:33 PM, Bill Cole wrote:
> On 24 Jan 2018, at 9:12, David Jones wrote:
>
>> What does everyone think about slowly increasing the score for
>> SPF_NONE and SPF_FAIL over time in the SA rulesets to force the
>> awareness and importance of proper SPF?
>
> -1
>
> In every real mailstream I've worked with in the lifetime of SPF, lack
> of SPF has *always* had a correlation with ham, not spam.
>
I am not suggesting that SPF_PASS = ham and SPF_FAIL = spam.
>
> SPF hard failures are a more complicated case because the sort of spam
> that hits SPF_FAIL tends to come from IPs that show up in good DNSBLs
> within a few minutes, making it hard for a site using DNSBLs to know how
> much of it there is. With that caveat, I see more ham hitting SPF_FAIL
> than I do spam where SPF_FAIL (which I have locally nailed at 2.0) is a
> decisive factor. Most SPF_FAIL spam scores well into double digits here.
I am proposing that if SPF were more accurately deployed then SPF_FAIL
would be worth something. We could whitelist_auth more trusted senders
and then be able to turn up the scores for the rest of the mail flow.
If the huge SA community around the world were to help push SPF adoption
and accurate deployments, then we could move on to DKIM too. Right now,
the best option we have is to get DMARC properly deployed as much as
possible where p=reject actually rejects the message unlike SPF_FAIL
that we can't trust.
--
David Jones
Re: Penalty for no/bad SPF
Posted by Bill Cole <sa...@billmail.scconsult.com>.
On 24 Jan 2018, at 9:12, David Jones wrote:
> What does everyone think about slowly increasing the score for
> SPF_NONE and SPF_FAIL over time in the SA rulesets to force the
> awareness and importance of proper SPF?
-1
In every real mailstream I've worked with in the lifetime of SPF, lack
of SPF has *always* had a correlation with ham, not spam.
SPF hard failures are a more complicated case because the sort of spam
that hits SPF_FAIL tends to come from IPs that show up in good DNSBLs
within a few minutes, making it hard for a site using DNSBLs to know how
much of it there is. With that caveat, I see more ham hitting SPF_FAIL
than I do spam where SPF_FAIL (which I have locally nailed at 2.0) is a
decisive factor. Most SPF_FAIL spam scores well into double digits here.