You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spamassassin.apache.org by bu...@issues.apache.org on 2011/01/19 13:35:14 UTC
[Bug 6535] New: Unparseable received header for authenticated mail
sent through Yahoo SMTP servers
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6535
Summary: Unparseable received header for authenticated mail
sent through Yahoo SMTP servers
Product: Spamassassin
Version: 3.3.1
Platform: All
OS/Version: All
Status: NEW
Severity: enhancement
Priority: P2
Component: Libraries
AssignedTo: dev@spamassassin.apache.org
ReportedBy: steve.freegard@fsl.com
dbg: received-header: unparseable: from ammfk (schwertnerrddx@183.38.156.224
with login) by smtp143.mail.mud.yahoo.com with SMTP; 19 Jan 2011 03:31:12 -0800
PST
This prevents SpamAssassin from seeing and looking up the originating IP
address in any DNSBLs.
Patch to Received.pm against 3.3.1 to follow.
--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
[Bug 6535] Unparseable received header for authenticated mail sent
through Yahoo SMTP servers
Posted by bu...@issues.apache.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6535
--- Comment #5 from Kevin A. McGrail <km...@pccc.com> 2011-01-19 14:45:07 UTC ---
(In reply to comment #4)
> Created an attachment (id=4841)
--> (https://issues.apache.org/SpamAssassin/attachment.cgi?id=4841) [details]
> Sample e-mail
>
> This message currently hits on several RBLs with the patch present that are not
> hit otherwise:
>
> X-Spam-Status: No, score=3.5 required=5.0 tests=DCC_CHECK,DKIM_SIGNED,
> DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,RCVD_IN_BL_SPAMCOP_NET,
> RCVD_IN_DNSWL_NONE,RCVD_IN_RP_RNBL,T_TO_NO_BRKTS_FREEMAIL autolearn=no
> version=3.3.1
>
> Still not enough though ...
This is not necessarily a good thing.
For example, in my area of the country, I use my android-phone a lot and I'm
issued a DHCP address.
If that IP was to be scanned such as by your patch above, they often come up as
blacklisted because people have used the IPs so send Spam.
The whole reason to authenticate, IMO, is so that the last-received relay IP is
the only one that is checked and the other IPs would be ignored.
Make sense?
KAM
--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
[Bug 6535] Unparseable received header for authenticated mail sent
through Yahoo SMTP servers
Posted by bu...@issues.apache.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6535
Kevin A. McGrail <km...@pccc.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |kmcgrail@pccc.com
--- Comment #2 from Kevin A. McGrail <km...@pccc.com> 2011-01-19 13:52:04 UTC ---
(In reply to comment #0)
> dbg: received-header: unparseable: from ammfk (schwertnerrddx@183.38.156.224
> with login) by smtp143.mail.mud.yahoo.com with SMTP; 19 Jan 2011 03:31:12 -0800
> PST
>
> This prevents SpamAssassin from seeing and looking up the originating IP
> address in any DNSBLs.
>
> Patch to Received.pm against 3.3.1 to follow.
The patch looks ok although untested. My question is what DNSBLs are we using
that aren't using last-external? It seems difficult to maintain by writing all
these tests for received headers.
What tests are you hitting on that you are trying to fix with that patch?
regards,
KAM
--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
[Bug 6535] Unparseable received header for authenticated mail sent
through Yahoo SMTP servers
Posted by bu...@issues.apache.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6535
--- Comment #11 from Steve Freegard <st...@fsl.com> 2011-01-19 17:01:43 UTC ---
> It's also one of the reasons that I disagree with Barracuda's recommendation
to
> use "deep header parsing". It's the same thing as not using last-external
> relay and we know it has FPs especially due to DHCP pools.
Just wanted to make a further point here.
The Barracuda saga was caused by the deep-header parsing using RBLs that stated
they shouldn't be used for that purpose *and* they were doing this at SMTP time
(e.g. as a boolean accept/reject).
Here - we're scoring on it; and that's by design. The RCVD_IN_BL_SPAMCOP_NET
has gone through the scoring process as a deep-header parsing test and been
scored accordingly because of that.
Which also gives me the idea; why don't you add a -lastexternal variant of the
SpamCop lookup to your sandbox and see how it fares in the mass-checks compared
to the current version.
To continue Mark's point about non-parsing of a Received header as a bug;
consider that the patch I provided doesn't just supply the IP address of that
hop; but also the HELO used by that host (which *is* definitely useful in
rules).
--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
[Bug 6535] Unparseable received header for authenticated mail sent
through Yahoo SMTP servers
Posted by bu...@issues.apache.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6535
--- Comment #8 from Steve Freegard <st...@fsl.com> 2011-01-19 15:13:11 UTC ---
(In reply to comment #5)
>
> This is not necessarily a good thing.
>
> For example, in my area of the country, I use my android-phone a lot and I'm
> issued a DHCP address.
>
> If that IP was to be scanned such as by your patch above, they often come up as
> blacklisted because people have used the IPs so send Spam.
>
> The whole reason to authenticate, IMO, is so that the last-received relay IP is
> the only one that is checked and the other IPs would be ignored.
>
> Make sense?
Yeah - I hadn't thought about it that way.
However - this is how those particular DNSBLs (and SA) was designed; it needs
to be able to parse the relevant Received headers and add the necessary
metadata for rules to match on.
Without SA parsing this header; I *can't* write a rule or plug-in to get this
IP address out of the X-Spam-Relays-* metadata.
The other problem here is how to handle these then? They're likely phished or
compromised accounts.
--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
[Bug 6535] Unparseable received header for authenticated mail sent
through Yahoo SMTP servers
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6535
Mark Martinec <Ma...@ijs.si> changed:
What |Removed |Added
----------------------------------------------------------------------------
Target Milestone|Undefined |3.4.0
--- Comment #14 from Mark Martinec <Ma...@ijs.si> 2011-05-05 16:18:28 UTC ---
Folding in Steve's patch (for 3.4):
trunk:
Bug 6535: Unparseable received header for authenticated mail
sent through Yahoo SMTP servers'
Sending lib/Mail/SpamAssassin/Message/Metadata/Received.pm
Committed revision 1099859.
I believe the discussion here was distracted from the reported
problem, which was the inability to parse the syntactically
correct Received field (if the 'with login' in parenthesis is
considered a comment, which is permitted).
Whether the information form such trace header field is
useful or not is irrelevant here.
Whether deep parsing is useful or not is irrelevant here - this
is a topic for a separate bug report.
If a bug (a failure to parse) happens to avoid some other bug
or misconfiguration is not a reason for not fixing it IMO.
--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
[Bug 6535] Unparseable received header for authenticated mail sent
through Yahoo SMTP servers
Posted by bu...@issues.apache.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6535
--- Comment #9 from Kevin A. McGrail <km...@pccc.com> 2011-01-19 15:21:44 UTC ---
(In reply to comment #8)
> However - this is how those particular DNSBLs (and SA) was designed; it needs
> to be able to parse the relevant Received headers and add the necessary
> metadata for rules to match on.
Using IPs from anything but the last received header for RBLs should likely be
avoided. I might be convinced otherwise but while reading this bug, I haven't
come up with any good reasons to do so.
> Without SA parsing this header; I *can't* write a rule or plug-in to get this
> IP address out of the X-Spam-Relays-* metadata.
While I agree, I'm not sure how useful that IP will be to do anything with. We
know that IPs are compromised. And we know that's why people use
authentication.
It's also one of the reasons that I disagree with Barracuda's recommendation to
use "deep header parsing". It's the same thing as not using last-external
relay and we know it has FPs especially due to DHCP pools.
> The other problem here is how to handle these then? They're likely phished or
> compromised accounts.
Unfortunately, the only things that come to mind are to a) File a complaint
with Yahoo! abuse and b) focus on content filtering.
I hope you read what I wrote a comment ago that your patch sparked an issue
that is much larger than just your patch.
Regards,
KAM
--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
[Bug 6535] Unparseable received header for authenticated mail sent
through Yahoo SMTP servers
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6535
--- Comment #15 from Darxus <Da...@ChaosReigns.com> 2011-05-05 16:33:42 UTC ---
Adding code which increases complexity of spamassassin without any gain for
anyone is a reason for not fixing it. And Received.pm is already
embarrassingly huge.
--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
[Bug 6535] Unparseable received header for authenticated mail sent
through Yahoo SMTP servers
Posted by bu...@issues.apache.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6535
--- Comment #6 from Mark Martinec <Ma...@ijs.si> 2011-01-19 15:02:41 UTC ---
> This is not necessarily a good thing.
An unparseable Received header field on an otherwise syntactically correct
header field is never a good thing, it indicates a bug in the parser
(which is already convoluted thoroughly and would need to be rewritten;
it should always be able to parse syntactically correct fields without
a need for hundreds of exception cases).
Whether an unparseable header may happen to avoid some other problem
is unrelated. If it does, then there are two things to be fixed,
not just parsing.
--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
[Bug 6535] Unparseable received header for authenticated mail sent
through Yahoo SMTP servers
Posted by bu...@issues.apache.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6535
--- Comment #12 from Warren Togami <wt...@gmail.com> 2011-02-14 05:16:54 EST ---
> Which also gives me the idea; why don't you add a -lastexternal variant of the
> SpamCop lookup to your sandbox and see how it fares in the mass-checks compared
> to the current version.
Sadly, this isn't an apples to apples comparison with the deep parsing rule
because the original rule has the benefit of reuse. =(
--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
[Bug 6535] Unparseable received header for authenticated mail sent
through Yahoo SMTP servers
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6535
Darxus <Da...@ChaosReigns.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |Darxus@ChaosReigns.com
--- Comment #13 from Darxus <Da...@ChaosReigns.com> 2011-05-04 22:58:51 UTC ---
Close? Nothing to be gained from being able to parse this header.
Warren: Reuse could temporarily be disabled for RCVD_IN_BL_SPAMCOP_NET. I'm
certainly curious what how deep header parsing compares to just doing
last-external (or first-trusted).
--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
[Bug 6535] Unparseable received header for authenticated mail sent
through Yahoo SMTP servers
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6535
Kevin A. McGrail <km...@pccc.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |RESOLVED
Resolution| |FIXED
--- Comment #16 from Kevin A. McGrail <km...@pccc.com> 2011-05-05 19:36:58 UTC ---
Believe this is resolved now and in the trunk. Closing as resolved.
--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
[Bug 6535] Unparseable received header for authenticated mail sent
through Yahoo SMTP servers
Posted by bu...@issues.apache.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6535
Steve Freegard <st...@fsl.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |steve.freegard@fsl.com
--- Comment #3 from Steve Freegard <st...@fsl.com> 2011-01-19 14:39:55 UTC ---
(In reply to comment #2)
> The patch looks ok although untested. My question is what DNSBLs are we using
> that aren't using last-external? It seems difficult to maintain by writing all
> these tests for received headers.
>
> What tests are you hitting on that you are trying to fix with that patch?
I'm not having a problem with tests mis-firing or anything like that.
It was merely that I noticed one of these yesterday and after patching in
support for this received header was able to get it scored higher as the IP was
listed on Spamcop.
I'll attach an example.
--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
[Bug 6535] Unparseable received header for authenticated mail sent
through Yahoo SMTP servers
Posted by bu...@issues.apache.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6535
--- Comment #7 from Kevin A. McGrail <km...@pccc.com> 2011-01-19 15:12:39 UTC ---
(In reply to comment #6)
> > This is not necessarily a good thing.
>
> An unparseable Received header field on an otherwise syntactically correct
> header field is never a good thing, it indicates a bug in the parser
> (which is already convoluted thoroughly and would need to be rewritten;
> it should always be able to parse syntactically correct fields without
> a need for hundreds of exception cases).
>
> Whether an unparseable header may happen to avoid some other problem
> is unrelated. If it does, then there are two things to be fixed,
> not just parsing.
As anything but the last received header is subject to forgery, I'm not sure I
see it as something that needs to be parsed. I'm not blaming Steve for the
state of the parser and his patch looks good.
I'm just wondering WHY we are parsing it because I don't want to see the IP
being parsed from this header being used to check against RBLs.
I know that will cause FPs.
regards,
KAM
--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
[Bug 6535] Unparseable received header for authenticated mail sent
through Yahoo SMTP servers
Posted by bu...@issues.apache.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6535
--- Comment #4 from Steve Freegard <st...@fsl.com> 2011-01-19 14:42:19 UTC ---
Created an attachment (id=4841)
--> (https://issues.apache.org/SpamAssassin/attachment.cgi?id=4841)
Sample e-mail
This message currently hits on several RBLs with the patch present that are not
hit otherwise:
X-Spam-Status: No, score=3.5 required=5.0 tests=DCC_CHECK,DKIM_SIGNED,
DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,RCVD_IN_BL_SPAMCOP_NET,
RCVD_IN_DNSWL_NONE,RCVD_IN_RP_RNBL,T_TO_NO_BRKTS_FREEMAIL autolearn=no
version=3.3.1
Still not enough though ...
--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
[Bug 6535] Unparseable received header for authenticated mail sent
through Yahoo SMTP servers
Posted by bu...@issues.apache.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6535
--- Comment #10 from Steve Freegard <st...@fsl.com> 2011-01-19 15:45:21 UTC ---
(In reply to comment #9)
> Unfortunately, the only things that come to mind are to a) File a complaint
> with Yahoo! abuse and b) focus on content filtering.
a) Is going to be totally fruitless due to lack of clue on Yahoo's abuse desk.
All complaints now have to be in ARF format and the amoeba's that look at the
report will simply claim that the message didn't originate from Yahoo (despite
headers to the contrary).
b) Hmmm; other than score on the tripod URL (which BTW has a meta refresh to
redirect to a porn site) - there's very little to content filter on.
The problem here is that multiple services with poor filtering and controls are
being abused to inject this stuff.
One of the only tools available to us is the IP address that originated the
message. Sure - it might be a host on a dial-up pool; but it's one of the only
bits of useful information we have (and possibly the username of the
authenticated user).
The one good thing about Yahoo and the other freemailers is that you know for
sure that you can 'trust' the Received headers haven't been forged.
> I hope you read what I wrote a comment ago that your patch sparked an issue
> that is much larger than just your patch.
Sure - no problemo. Totally understand; it's a useful discussion to be having.
--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
[Bug 6535] Unparseable received header for authenticated mail sent
through Yahoo SMTP servers
Posted by bu...@issues.apache.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6535
--- Comment #1 from Steve Freegard <st...@fsl.com> 2011-01-19 07:46:31 UTC ---
Created an attachment (id=4840)
--> (https://issues.apache.org/SpamAssassin/attachment.cgi?id=4840)
Patch to parse Received header correctly
--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.