You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by Robert Fitzpatrick <ro...@webtent.org> on 2016/06/09 15:23:04 UTC
SPF should always hit?
Excuse me if this is too lame a question, but I have the SPF plugin
enabled and it hits a lot. Should SPF_ something hit on every message if
the domain has an SPF record in DNS?
Furthermore, a message found as Google phishing did not get a hit on a
email address where the domain has SPF setup. Not sure if it would fail
anyway if the envelope from is the culprit?
--
Robert
Re: SPF should always hit?
Posted by Reindl Harald <h....@thelounge.net>.
Am 09.06.2016 um 17:23 schrieb Robert Fitzpatrick:
> Excuse me if this is too lame a question, but I have the SPF plugin
> enabled and it hits a lot. Should SPF_ something hit on every message if
> the domain has an SPF record in DNS?
and if it's SPF_NONE
> Furthermore, a message found as Google phishing did not get a hit on a
> email address where the domain has SPF setup. Not sure if it would fail
> anyway if the envelope from is the culprit?
SPF is always about envelopes (no trolling about sender-id needed this
time) and with the infos you provided what do you expect for answers?
just read https://en.wikipedia.org/wiki/Sender_Policy_Framework
SPF is more than just "fail"
T_SPF_PERMERROR
T_SPF_TEMPERROR
T_SPF_HELO_PERMERROR
T_SPF_HELO_TEMPERROR
SPF_FAIL
SPF_HELO_FAIL
SPF_HELO_NEUTRAL
SPF_HELO_NONE
SPF_HELO_PASS
SPF_HELO_SOFTFAIL
SPF_NEUTRAL
SPF_NONE
SPF_PASS
SPF_SOFTFAIL
Re: SPF should always hit? SOLVED
Posted by Reindl Harald <h....@thelounge.net>.
Am 11.07.2016 um 21:02 schrieb David B Funk:
> On Mon, 11 Jul 2016, Reindl Harald wrote:
>> SA has also a weakness or design mistake here
>>
>> "envelope_sender_header X-Local-Envelope-From" while that header comes
>> from postfix with customized configuration because we use it in own
>> rules has no fallback
>> __________________________________
>>
>> By default, various MTAs will use different headers, such as the
>> following:
>>
>> X-Envelope-From
>> Envelope-Sender
>> X-Sender
>> Return-Path
>> __________________________________
>>
>> well, in case of "envelope_sender_header" present in the configuration
>> and that header is missing for whatever reason there is *no fallback*
>> while for most cases it would be better to use
>> "envelope_sender_header" as prefered one instead the only one
>>
>> that it is not the case can you see when "add_header all Status
>> _YESNO_, score=_SCORE_, tag-level=_REQD_, block-level=8.0,
>> envelope=_SENDERDOMAIN_, from=_AUTHORDOMAIN_, _TOKENSUMMARY_" ends
>> with SENDERDOMAIN_ in your headers
>
> The SA Conf man page seems to indicate that it -should- fall back to its
> heuristic if the envelope_sender_header is missing:
>
> To avoid this heuristic failure, the "envelope_sender_header" setting
> may be helpful. Name
> the header that your MTA or MDA adds to messages containing the
> address used at the MAIL
> FROM step of the SMTP transaction.
>
> If the header in question contains "<" or ">" characters at the start
> and end of the email
> address in the right-hand side, as in the SMTP transaction, these
> will be stripped.
>
> If the header is not found in a message, or if it's value does not
> contain an "@" sign,
> SpamAssassin will issue a warning in the logs and fall back to its
> default heuristics.
>
> It doesn't look like that fall-back is working. If you completely omit
> the envelope_sender_header config setting, the heuristic works.
> Maybe you should file a bug-report.
looks so
> One additional question, if you're setting the envelope_sender_header
> configwhy aren't you actually supplying it?
because i have *no idea* from where it comes that postfix sometimes ignores
check_sender_access proxy:pcre:/etc/postfix/x_envelope_from.cf
check_recipient_access proxy:pcre:/etc/postfix/x_envelope_to.cf
> If you cannot depend upon your system to actually supply the header you
> list
> in your envelope_sender_header config, then don't set that parameter
well, the idea is to add a own heaer in the MTA instead rely on
heuristic which hopefully don't use a randm but wrong header (if that
would be impossible the other problem also won't exist)
Re: SPF should always hit? SOLVED
Posted by David B Funk <db...@engineering.uiowa.edu>.
On Mon, 11 Jul 2016, Reindl Harald wrote:
>
> Am 11.07.2016 um 19:30 schrieb RW:
[snip..]
>> It sounds like SA is not able to parse the envelope sender out of the
>> headers.
>>
>> See the description for envelope_sender_header in
>> man Mail::SpamAssassin::Conf
>
> SA has also a weakness or design mistake here
>
> "envelope_sender_header X-Local-Envelope-From" while that header comes from
> postfix with customized configuration because we use it in own rules has no
> fallback
> __________________________________
>
> By default, various MTAs will use different headers, such as the following:
>
> X-Envelope-From
> Envelope-Sender
> X-Sender
> Return-Path
> __________________________________
>
> well, in case of "envelope_sender_header" present in the configuration and
> that header is missing for whatever reason there is *no fallback* while for
> most cases it would be better to use "envelope_sender_header" as prefered one
> instead the only one
>
> that it is not the case can you see when "add_header all Status _YESNO_,
> score=_SCORE_, tag-level=_REQD_, block-level=8.0, envelope=_SENDERDOMAIN_,
> from=_AUTHORDOMAIN_, _TOKENSUMMARY_" ends with SENDERDOMAIN_ in your headers
The SA Conf man page seems to indicate that it -should- fall back to its
heuristic if the envelope_sender_header is missing:
To avoid this heuristic failure, the "envelope_sender_header" setting may be helpful. Name
the header that your MTA or MDA adds to messages containing the address used at the MAIL
FROM step of the SMTP transaction.
If the header in question contains "<" or ">" characters at the start and end of the email
address in the right-hand side, as in the SMTP transaction, these will be stripped.
If the header is not found in a message, or if it's value does not contain an "@" sign,
SpamAssassin will issue a warning in the logs and fall back to its default heuristics.
It doesn't look like that fall-back is working. If you completely omit the
envelope_sender_header config setting, the heuristic works.
Maybe you should file a bug-report.
One additional question, if you're setting the envelope_sender_header config
why aren't you actually supplying it?
If you cannot depend upon your system to actually supply the header you list
in your envelope_sender_header config, then don't set that parameter.
--
Dave Funk University of Iowa
<dbfunk (at) engineering.uiowa.edu> College of Engineering
319/335-5751 FAX: 319/384-0549 1256 Seamans Center
Sys_admin/Postmaster/cell_admin Iowa City, IA 52242-1527
#include <std_disclaimer.h>
Better is not better, 'standard' is better. B{
Re: SPF should always hit? SOLVED
Posted by Reindl Harald <h....@thelounge.net>.
Am 11.07.2016 um 19:30 schrieb RW:
> On Mon, 11 Jul 2016 12:49:04 -0400
> Robert Fitzpatrick wrote:
>
>> I finally was able to get SPF checks to be more reliable by making
>> sure Postfix SPF policies were in place. Here is a good read ....
>>
>> https://github.com/mail-in-a-box/mailinabox/issues/698
>> Excerpt: It's worth noting that lack of postfix's spf checker renders
>> spamassassin's flagging impaired because without it spamassassin in
>> my case is only adding helo_pass and that's all regarding spfs.
>
> It sounds like SA is not able to parse the envelope sender out of the
> headers.
>
> See the description for envelope_sender_header in
> man Mail::SpamAssassin::Conf
SA has also a weakness or design mistake here
"envelope_sender_header X-Local-Envelope-From" while that header comes
from postfix with customized configuration because we use it in own
rules has no fallback
__________________________________
By default, various MTAs will use different headers, such as the following:
X-Envelope-From
Envelope-Sender
X-Sender
Return-Path
__________________________________
well, in case of "envelope_sender_header" present in the configuration
and that header is missing for whatever reason there is *no fallback*
while for most cases it would be better to use "envelope_sender_header"
as prefered one instead the only one
that it is not the case can you see when "add_header all Status _YESNO_,
score=_SCORE_, tag-level=_REQD_, block-level=8.0,
envelope=_SENDERDOMAIN_, from=_AUTHORDOMAIN_, _TOKENSUMMARY_" ends with
SENDERDOMAIN_ in your headers
Re: SPF should always hit? SOLVED
Posted by RW <rw...@googlemail.com>.
On Mon, 11 Jul 2016 12:49:04 -0400
Robert Fitzpatrick wrote:
> I finally was able to get SPF checks to be more reliable by making
> sure Postfix SPF policies were in place. Here is a good read ....
>
> https://github.com/mail-in-a-box/mailinabox/issues/698
> Excerpt: It's worth noting that lack of postfix's spf checker renders
> spamassassin's flagging impaired because without it spamassassin in
> my case is only adding helo_pass and that's all regarding spfs.
It sounds like SA is not able to parse the envelope sender out of the
headers.
See the description for envelope_sender_header in
man Mail::SpamAssassin::Conf
Re: SPF should always hit? SOLVED
Posted by Robert Fitzpatrick <ro...@webtent.org>.
Robert Fitzpatrick wrote:
> Joe Quinn wrote:
>> On 6/9/2016 11:23 AM, Robert Fitzpatrick wrote:
>>> Excuse me if this is too lame a question, but I have the SPF plugin
>>> enabled and it hits a lot. Should SPF_ something hit on every message
>>> if the domain has an SPF record in DNS?
>>>
>>> Furthermore, a message found as Google phishing did not get a hit on a
>>> email address where the domain has SPF setup. Not sure if it would
>>> fail anyway if the envelope from is the culprit?
>>>
>> In a perfect world, every message you scan will hit one of the following:
>> SPF_HELO_NONE
>> SPF_HELO_NEUTRAL
>> SPF_HELO_PASS
>> SPF_HELO_FAIL
>> SPF_HELO_SOFTFAIL
>> T_SPF_HELO_PERMERROR
>> T_SPF_HELO_TEMPERROR
>>
>> And additionally one of the following:
>> SPF_NONE
>> SPF_NEUTRAL
>> SPF_PASS
>> SPF_FAIL
>> SPF_SOFTFAIL
>> T_SPF_PERMERROR
>> T_SPF_TEMPERROR
>>
I finally was able to get SPF checks to be more reliable by making sure
Postfix SPF policies were in place. Here is a good read ....
https://github.com/mail-in-a-box/mailinabox/issues/698
Excerpt: It's worth noting that lack of postfix's spf checker renders
spamassassin's flagging impaired because without it spamassassin in my
case is only adding helo_pass and that's all regarding spfs.
Once we got Postfix SPF checks setup using the Python version and
disabling rejects in the config, we now have headers we can be sure are
handled by our custom rules in addition to any SA checks.
--
Robert
Re: SPF should always hit?
Posted by Robert Fitzpatrick <ro...@webtent.org>.
Joe Quinn wrote:
> On 6/9/2016 11:23 AM, Robert Fitzpatrick wrote:
>> Excuse me if this is too lame a question, but I have the SPF plugin
>> enabled and it hits a lot. Should SPF_ something hit on every message
>> if the domain has an SPF record in DNS?
>>
>> Furthermore, a message found as Google phishing did not get a hit on a
>> email address where the domain has SPF setup. Not sure if it would
>> fail anyway if the envelope from is the culprit?
>>
> In a perfect world, every message you scan will hit one of the following:
> SPF_HELO_NONE
> SPF_HELO_NEUTRAL
> SPF_HELO_PASS
> SPF_HELO_FAIL
> SPF_HELO_SOFTFAIL
> T_SPF_HELO_PERMERROR
> T_SPF_HELO_TEMPERROR
>
> And additionally one of the following:
> SPF_NONE
> SPF_NEUTRAL
> SPF_PASS
> SPF_FAIL
> SPF_SOFTFAIL
> T_SPF_PERMERROR
> T_SPF_TEMPERROR
>
> In practice, there's almost certainly a few edge cases where messages
> can avoid getting one in either category. For purposes of writing your
> own metas against these, the rules that matter most for measuring
> spamminess are the none, pass, and fail/softfail results. The rest are
> for total coverage of the results that an SPF query can yield, for
> debugging and documentation purposes.
>
> Also, none of these will hit at all if you disable network tests.
Yes, network tests are on. I have lots of messages hitting, it is harder
to find one that doesn't have hits as you suggested. However, I can find
several out of our database of 280K messages cached which do not hit any
of these rules. So, what would be a reason they didn't hit?
The only custom rule I have with SPF_* is with SPF_FAIL combined without
DKIM to give higher score:
meta WT_FORGED_SENDER (SPF_FAIL && !DKIM_VALID)
describe WT_FORGED_SENDER To score high when SPF fails without valid DKIM
score WT_FORGED_SENDER 8.0
Here is the score for this particular example:
2.095 FREEMAIL_FORGED_REPLYTO Freemail in Reply-To, but not From
1.000 XPRIO_SHORT_SUBJ (No description provided)
0.250 FREEMAIL_REPLYTO_END_DIGIT Reply-To freemail username ends in digit
0.001 HTML_MESSAGE HTML included in message
0.001 HEADER_FROM_DIFFERENT_DOMAINS (No description provided)
0.000 RCVD_IN_DNSWL_NONE Sender listed at http://www.dnswl.org/, low trust
-1.900 BAYES_00 Bayesian spam probability is 0 to 1%
-5.000 RCVD_IN_JMF_W (No description provided)
--
Robert
Re: SPF should always hit?
Posted by Joe Quinn <jq...@pccc.com>.
On 6/9/2016 11:23 AM, Robert Fitzpatrick wrote:
> Excuse me if this is too lame a question, but I have the SPF plugin
> enabled and it hits a lot. Should SPF_ something hit on every message
> if the domain has an SPF record in DNS?
>
> Furthermore, a message found as Google phishing did not get a hit on a
> email address where the domain has SPF setup. Not sure if it would
> fail anyway if the envelope from is the culprit?
>
In a perfect world, every message you scan will hit one of the following:
SPF_HELO_NONE
SPF_HELO_NEUTRAL
SPF_HELO_PASS
SPF_HELO_FAIL
SPF_HELO_SOFTFAIL
T_SPF_HELO_PERMERROR
T_SPF_HELO_TEMPERROR
And additionally one of the following:
SPF_NONE
SPF_NEUTRAL
SPF_PASS
SPF_FAIL
SPF_SOFTFAIL
T_SPF_PERMERROR
T_SPF_TEMPERROR
In practice, there's almost certainly a few edge cases where messages
can avoid getting one in either category. For purposes of writing your
own metas against these, the rules that matter most for measuring
spamminess are the none, pass, and fail/softfail results. The rest are
for total coverage of the results that an SPF query can yield, for
debugging and documentation purposes.
Also, none of these will hit at all if you disable network tests.