Re: SA rules & matching of private addresses

[Mabry Tyson <Ty...@AI.SRI.COM>]

On 10/2/12 8:30 PM, darxus@chaosreigns.com wrote:
> Run the email through "spamassassin -D received-header".  That'll tell you
> how and if the headers got parsed.  SA has certainly had bugs where it
> failed to parse received headers before, and IPv6 hasn't had a whole lot of
> use.
Thanks for pointing that out.  That does illuminate the situation.

The debug output shows that SA is (IMO, mis-) interpreting the 
"x-originating-ip" as a Received header.
That's a surprise to me because it is an uninterpreted optional (header) 
field (RFC5821, 3.6.8)
which could have any garbage in it and convey any semantics.
>
> There has also been a fair amount of work on IPv6 since the last release,
> so it's possible there was a bug, it got fixed, and you don't have the
> fix yet.

It turns out it isn't an IPv6 problem.  I had wrongly assumed that if a 
host with a private address
delivers to a trusted, local host, then that host would be considered 
trusted & local.

I removed the "x-originating-ip" header.  Then,  if I replace the IPv6 
link local addresses by
IPv4 10/8 addresses, I get the same behavior as with the IPv6 fe80::/10 
addresses.
If the private addresses are in the trusted & local addresses, then I 
get ALL_TRUSTED
and not RDNS_NONE.  If they are not in trusted & local,  then I get 
RDNS_NONE and not
ALL_TRUSTED.

So I have to add things like  10/8 and fe80::/10 to the trusted & local 
networks.

(When I add back in "x-originating-ip", I lose ALL_TRUSTED, which would 
be expected if you
treat it like a Received: header.)
>
> On 10/02, Mabry Tyson wrote:
>>     One user complained about a false positive.  When I examined the mail,
>>     there appeared to be at least two rules that didn't work as I thought they
>>     should because of a Received line in which IPv6 Link Local addresses were
>>     used.   It appears that a patch was previously put in that was thought to
>>     fix these kinds of things.
>>     The sender was apparently using AA.BB.CC.DD (a Comcast address, presumably
>>     his home address).
>>     He logged into the mail system of SRI.COM (independent of our mail system)
>>     and
>>     sent his mail from within it (which is why CCC.SRI.COM is the oldest
>>     Received line).
> That should result in a received header clearly indicating that the
> connection from comcast was authenticated, and SA should notice that
> and use it to skip the tests on that comcast IP.
>
> It mostly sounds like this is what's missing.  SRI.com not indicating
> the authentication in their received header in the standard way.
You say "standard way".  Can you point to a standard (eg, RFC) that 
indicates
how this is indicated?  Maybe it is my lack of knowledge, but I'm not 
aware of any
"standard" way.   That mail system is run by an independent business unit.
I have no influence on their operations of their Exchange server. But, 
if they're
not following a standard, I can suggest they follow a standard.  But they
probably won't listen to a "Well, SpamAssassin would recognize things
better if you did ....,"

RFC5322 (Message Format) doesn't mention "authenticate".
RFC5321 (SMTP) does mention authentication (p 73) but gives no 
indication this
should be reflected in any header nor any mechanism to do so.  It mentions
RFC4409 (Message Submission) as an example of a protocol that causes a 
message to be entered
onto the Internet.  However, RFC4409 gives no indication of how a conforming
service should (or even might) indicate authentication.  Nor does it 
indicate the
format of a Received header that it should add such that it indicates 
that this
protocol was used.

After reviewing these, I believe that the first host to receive this message
fails to add a Received header as required (which presumably would be 
where some
indication of authentication would be found).  The oldest Received 
header indicates
a transmission from that first host to another host.
(Ref: RFC5821, 3.7.2  " When forwarding a message into or out of the 
Internet
environment, a gateway MUST prepend a Received: line ...")

    4.
    [3]http://spamassassin.apache.org/tests_3_3_x.html has
    RCVD_IN_PBL = 3.6  (Spamhaus Policy black list)
    RCVD_IN_SBL = 2.6  (Spamhaus Spam black list)
    RCVD_IN_XBL = 0.7  (Spamhaus Botnet black list)
    which seems backward to me.  The 3.2 tests scoring seems more reasonable.

> Do not attempt to comprehend the depths of the mind of the re-scorer :P
>
> No seriously, it has no concept of "this rule means the email is more bad
> than another rule, therefore it should have a higher score".  Only "This
> score results in a better approximation of the 1 false positive in 2,500
> non-spams goal".  Which often results in unexpected things.  It comes
> up a lot.
>
> I very recently found a case where a rule that hit more non-spam than spam
> got a score of something like 3.  Which may have been suboptimal.
There may be many ways of assigning scores to the rules to
get nearly equivalent results (of correct assignments, false positives,
& misses) on any particular test set of messages.  Those different ways 
are not
necessarily nearly equivalent when applied to a different set of messages.

As I said, the 3.2 scores for these rules seem more reasonable, considering
what kind of hosts are on the black lists.  I would tend to believe those
are more likely to better judge other messages.

Re: SA rules & matching of private addresses

[SM <sm...@resistor.net>]

Hi Mabry,
At 03:46 04-10-2012, Mabry Tyson wrote:
>The debug output shows that SA is (IMO, mis-) interpreting the 
>"x-originating-ip" as a Received header.

The IP address from the X-Origination-IP header field, similarly to 
those in the Receiver header fields, is used for DNSBL lookups.

Regards,
-sm