You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by Russ Ringer <li...@avtcorp.com> on 2005/12/08 18:10:31 UTC
Re: false positive in RCVD_IN_SORBS_DUL test
On Thu, 8 Dec 2005 03:34:44 -0800, you wrote:
>score ALL_TRUSTED 0
>
>This is simply masking the problem, not setting trusted_networks correctly.
>And it is only masking the obvious problem - there are inobvious problems
>that will still score incorrectly.
>
>If you remove that line and start seeing ALL_TRUSTED hits where you don't
>think they should be, then you need to set trusted_networks correctly. If
>you remove that line and only see TRUSTED_NETWORKS hit where they should,
>you are better off. :-)
>
> Loren
Even with TRUSTED_NETWORKS set, the RCVD_IN_SORBS_DUL rule is
triggered. I don't see how this is correct, when the IP address that
triggered it was not the last hop. This rule should only be triggered
when "sent directly from dynamic IP address"
Re: false positive in RCVD_IN_SORBS_DUL test
Posted by Matt Kettler <mk...@evi-inc.com>.
Daryl C. W. O'Shea wrote:
> On 09/12/2005 6:30 PM, Matt Kettler wrote:
>
>>
>> Russ, Actually it looks like in SA 3.0.x and SA 3.1.0 the
>> trusted_networks
>> setting doesn't matter that much.
>
>
> Just so it's clear for anyone following along, Matt is referring to
> trusted_networks' affect on DUL rules. Regardless of how it affects DUL
> rules, it is still vital that trusted_networks be set correctly as it
> directly affects many other SpamAssassin functions.
>
True.. I did not mean to imply that trusted_networks on the general whole is
irrelevant.
Also, for completeness, trusted_networks can have some limited effect on the DUL
rules, but only to the extent it affects what internal_networks is. SA 3.1.0
will remove all leading internal hosts from the list of IPs before doing DUL checks.
But even that can't fix the particular case Russ is talking about, which is why
I said trusted_networks doesn't matter much. (in this case)
Re: false positive in RCVD_IN_SORBS_DUL test
Posted by "Daryl C. W. O'Shea" <sp...@dostech.ca>.
On 09/12/2005 6:30 PM, Matt Kettler wrote:
>
> Russ, Actually it looks like in SA 3.0.x and SA 3.1.0 the trusted_networks
> setting doesn't matter that much.
Just so it's clear for anyone following along, Matt is referring to
trusted_networks' affect on DUL rules. Regardless of how it affects DUL
rules, it is still vital that trusted_networks be set correctly as it
directly affects many other SpamAssassin functions.
Re: false positive in RCVD_IN_SORBS_DUL test
Posted by Matt Kettler <mk...@evi-inc.com>.
Russ Ringer wrote:
> On Thu, 8 Dec 2005 23:16:13 -0800, you wrote:
>
>
>>>Even with TRUSTED_NETWORKS set, the RCVD_IN_SORBS_DUL rule is
>>
>>triggered. I don't see how this is correct, when the IP address that
>>triggered it was not the last hop. This rule should only be triggered
>>when "sent directly from dynamic IP address"
>>
>>If someone hasn't suggested it already, post your trusted_* config lines
>>along with the headers for a message that you think hit wrong, and we can
>>probably help you figure out what is going wrong. The first guess would be
>>that you don't have trusted_networks set quite *right*, even though you have
>>it set to *something*.
>>
>> Loren
>
>
>
> TRUSTED_NETWORKS 10.0.0/24 198.135.234.36
Russ, Actually it looks like in SA 3.0.x and SA 3.1.0 the trusted_networks
setting doesn't matter that much.
For some reason the trust-path based code for DUL RBLs from 2.6x was reverted
back out and SA 3.0.x and higher use the classic "not first hop" algorithm from
SA 2.5x with some minor twists:
1) get a list of all external IPs in the Received: path
2) strip off any private IPs anywhere
3) remove the first hop from that list.
4) check these IPs.
In your example, SA 3.0.0 or higher will FP. SA 2.6x will not FP on this, since
it uses a trust-path based algorithm.
Really, as I see it, both algorithms are broken.
See http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4728
Re: false positive in RCVD_IN_SORBS_DUL test
Posted by "Daryl C. W. O'Shea" <sp...@dostech.ca>.
On 09/12/2005 6:13 PM, Russ Ringer wrote:
>
> This does look kind of fishy. I think I see why the rule was tripped.
> 209.30.176.199 is listed in SORBS DUL
> Looks like they are running proxy+ on a PPoX pool
> computer and relaying through it, so I guess it makes sense to trip
> the rule, or does it?
As I noted in my email to you yesterday at 3:32 PM EST this is exactly
the case.
At least for now... read the rest of the thread or bug 4728 for
discussion about this.
Re: false positive in RCVD_IN_SORBS_DUL test
Posted by Russ Ringer <li...@avtcorp.com>.
On Thu, 8 Dec 2005 23:16:13 -0800, you wrote:
>> Even with TRUSTED_NETWORKS set, the RCVD_IN_SORBS_DUL rule is
>triggered. I don't see how this is correct, when the IP address that
>triggered it was not the last hop. This rule should only be triggered
>when "sent directly from dynamic IP address"
>
>If someone hasn't suggested it already, post your trusted_* config lines
>along with the headers for a message that you think hit wrong, and we can
>probably help you figure out what is going wrong. The first guess would be
>that you don't have trusted_networks set quite *right*, even though you have
>it set to *something*.
>
> Loren
TRUSTED_NETWORKS 10.0.0/24 198.135.234.36
lookup from MX:
#host mail.avtcorp.com
mail.avtcorp.com has address 10.0.0.5
header that trip RCVD_IN_SORBS_DUL
Received: from smtp109.sbc.mail.mud.yahoo.com (68.142.198.208)
by mail.avtcorp.com with SMTP; 7 Dec 2005 21:03:26 -0000
Received: (qmail 42892 invoked from network); 7 Dec 2005 21:03:25
-0000
Received: from unknown (HELO proxyplus.universe)
(protected@swbell.net@209.30.176.199 with login)
by smtp109.sbc.mail.mud.yahoo.com with SMTP; 7 Dec 2005 21:03:25
-0000
Received: from cindy [156.56.61.27]
by Proxy+; Wed, 07 Dec 2005 15:17:34 -0600
for <pr...@avtcorp.com>
From: "cindy darling" <pr...@primomic.com>
To: "Judy Grecco" <pr...@avtcorp.com>
This does look kind of fishy. I think I see why the rule was tripped.
209.30.176.199 is listed in SORBS DUL
Looks like they are running proxy+ on a PPoX pool
computer and relaying through it, so I guess it makes sense to trip
the rule, or does it?
->Russ
Re: false positive in RCVD_IN_SORBS_DUL test
Posted by Loren Wilton <lw...@earthlink.net>.
> Even with TRUSTED_NETWORKS set, the RCVD_IN_SORBS_DUL rule is
triggered. I don't see how this is correct, when the IP address that
triggered it was not the last hop. This rule should only be triggered
when "sent directly from dynamic IP address"
If someone hasn't suggested it already, post your trusted_* config lines
along with the headers for a message that you think hit wrong, and we can
probably help you figure out what is going wrong. The first guess would be
that you don't have trusted_networks set quite *right*, even though you have
it set to *something*.
Loren
Re: false positive in RCVD_IN_SORBS_DUL test
Posted by "Daryl C. W. O'Shea" <sp...@dostech.ca>.
On 09/12/2005 5:30 PM, Matt Kettler wrote:
> Daryl C. W. O'Shea wrote:
>
>>Mail to internal users (from roaming users) isn't the problem though.
>>It's mail to external sites that see that my smart host is the second
>>"public IP hop" and look it up in DUL. Since my telco continues to
>>refuse to change my generic rDNS, my static IP has been listed in
>>SORBS-DUL and any of our mail not sent from the internal network gets
>>hit by SpamAssassin.
>
>
> Yeah, that falls under the "multi-hop behind a dynamic IP with legitimate
> relaying through a non-dynamic server" case.
>
> Really I think the use of notfirsthop in DUL testing is just plain broken. SA
> should only be checking the host that drops off to your MX against the DULs. It
> shouldn't be backtracking further.
Agreed. I should have some rules in my sandbox tonight/tomorrow to see
what would happen if -firstuntrusted was used.
I don't think people should be trusting other networks like the code
comments suggest... it opens up too many other problems, such as now the
sender (usually) didn't pass through an external relay and will now be
subject to DUL checks. So that whole we don't know who we're trusting
issue should be a non-issue IMHO.
> The current "external, nonprivate, notfirsthop" deals with most common FP cases,
> such as The "no private" fixes the "NATed co-op" case of:
> private IP -> public (dyn) -> ISP -> Recipient MX -> SA.
>
>
> but it is still broken for the case of:
>
> public IP -> public (dyn) -> ISP -> Recipient MX -> SA.
>
> Which is rare, but does exist.
Yeah, probably the only people who are doing it are those running a
personal mail server on their cable/dsl line and smart hosting to their
ISP. I actually do that for my personal mail just so I can use IMAP.
I think there are a lot of people affected by this that fall under the
"wrongly listed in DUL" case though.
> That said, if there's any way of doing so, I'd
> ditch your ISP ASAP. Since they can't set RDNS entries they are clearly not a
> business grade service, and are only suited to SOHO and home-user operations.
It's not that they can't, it's that I've yet to find 'the' person that
can or wants to. It's Bell Canada, the owner of one of the largest IP
networks in the world.
I've worked or done work for the other ISPs in town and would have no
problem with them if they could provide serivce but they can't.
Re: false positive in RCVD_IN_SORBS_DUL test
Posted by Matt Kettler <mk...@evi-inc.com>.
Daryl C. W. O'Shea wrote:
> Mail to internal users (from roaming users) isn't the problem though.
> It's mail to external sites that see that my smart host is the second
> "public IP hop" and look it up in DUL. Since my telco continues to
> refuse to change my generic rDNS, my static IP has been listed in
> SORBS-DUL and any of our mail not sent from the internal network gets
> hit by SpamAssassin.
Yeah, that falls under the "multi-hop behind a dynamic IP with legitimate
relaying through a non-dynamic server" case.
Really I think the use of notfirsthop in DUL testing is just plain broken. SA
should only be checking the host that drops off to your MX against the DULs. It
shouldn't be backtracking further.
The current "external, nonprivate, notfirsthop" deals with most common FP cases,
such as The "no private" fixes the "NATed co-op" case of:
private IP -> public (dyn) -> ISP -> Recipient MX -> SA.
but it is still broken for the case of:
public IP -> public (dyn) -> ISP -> Recipient MX -> SA.
Which is rare, but does exist. That said, if there's any way of doing so, I'd
ditch your ISP ASAP. Since they can't set RDNS entries they are clearly not a
business grade service, and are only suited to SOHO and home-user operations.
Re: false positive in RCVD_IN_SORBS_DUL test
Posted by "Daryl C. W. O'Shea" <sp...@dostech.ca>.
On 09/12/2005 4:42 PM, Matt Kettler wrote:
> Daryl C. W. O'Shea wrote:
>
>
>>The situation still sucks though. I can't have remote users use ESMTPSA
>>to send mail through our servers (without stripping received headers
>>before sending the message) since they'll have a public IP.
>
>
> Sure you can. At least, if you're using SA 3.1.0 it will automatically realize
> that any AUTH based transaction is coming from an "internal" user.
Yeah, I wrote that code. Support for that starts in 3.0.2 and is
expanded in each release after that.
Mail to internal users (from roaming users) isn't the problem though.
It's mail to external sites that see that my smart host is the second
"public IP hop" and look it up in DUL. Since my telco continues to
refuse to change my generic rDNS, my static IP has been listed in
SORBS-DUL and any of our mail not sent from the internal network gets
hit by SpamAssassin.
Re: false positive in RCVD_IN_SORBS_DUL test
Posted by Matt Kettler <mk...@evi-inc.com>.
Daryl C. W. O'Shea wrote:
>
> The situation still sucks though. I can't have remote users use ESMTPSA
> to send mail through our servers (without stripping received headers
> before sending the message) since they'll have a public IP.
Sure you can. At least, if you're using SA 3.1.0 it will automatically realize
that any AUTH based transaction is coming from an "internal" user.
Re: false positive in RCVD_IN_SORBS_DUL test
Posted by "Daryl C. W. O'Shea" <sp...@dostech.ca>.
On 09/12/2005 12:03 PM, Matt Kettler wrote:
> Daryl C. W. O'Shea wrote:
>
>>I suspect that the lack of affected mail in the scoring corpus is the
>>reason why it's gone unnoticed. I'd been meaning to run tests to
>>compare the hits between:
>>
>> -- notfirsthop and firstuntrusted
>
> I'd love to see that.
Just gotta get the tuits to do it. Going to be harder now that my lift
ticket is valid as of yesterday. :)
>> -- notfirsthop and "not private and not first hop"
>>
> Well, the current 'notfirsthop' in SA 3.1.0 is actually "notfirsthop,
> notinternal, notprivate".
>
> In sub check_rbl_backend they make use of "ip_list_uniq_and_strip_private" on
> the fullexternal set of IPs..
Ah... I missed that. When I generated my test message I didn't realize
I sent it across the 'net and not my vlan. While using that test
message I didn't notice that I was testing with a public IP before the
smart host and not the private IP I'd have on the vlan.
The situation still sucks though. I can't have remote users use ESMTPSA
to send mail through our servers (without stripping received headers
before sending the message) since they'll have a public IP.
Getting them to log in via the vlan is quite the battle, exasperated by
a low computer aptitude found in our manufacturing centric environment.
Oh well, that's what we get for being hours from the city with extremely
limited connectivity options from an uncooperative, and generally inept,
telco.
Daryl
Re: false positive in RCVD_IN_SORBS_DUL test
Posted by Matt Kettler <mk...@evi-inc.com>.
Daryl C. W. O'Shea wrote:
> I suspect that the lack of affected mail in the scoring corpus is the
> reason why it's gone unnoticed. I'd been meaning to run tests to
> compare the hits between:
>
> -- notfirsthop and firstuntrusted
I'd love to see that.
> -- notfirsthop and "not private and not first hop"
>
Well, the current 'notfirsthop' in SA 3.1.0 is actually "notfirsthop,
notinternal, notprivate".
In sub check_rbl_backend they make use of "ip_list_uniq_and_strip_private" on
the fullexternal set of IPs..
They also have an explanation as to why they stopped using firstuntrusted.
-----------------
if ($set =~ /-notfirsthop$/)
{
# use the external IP set, instead of the trusted set; the user may have
# specified some third-party relays as trusted. Also, don't use
# @originating; those headers are added by a phase of relaying through
# a server like Hotmail, which is not going to be in dialup lists anyway.
@ips = $self->ip_list_uniq_and_strip_private(@fullexternal);
if (scalar @ips > 1) { pop @ips; }
}
------------------
Which makes sense. I guess really what you want isn't "firsttrusted".. really,
"firstexternal" wouldn't work either. You really need to know which machine is
acting as the MX so you can do "hostdroppingmailtoMX"
Re: false positive in RCVD_IN_SORBS_DUL test
Posted by "Daryl C. W. O'Shea" <sp...@dostech.ca>.
On 08/12/2005 4:53 PM, Matt Kettler wrote:
> Daryl C. W. O'Shea wrote:
>
>>On 08/12/2005 3:52 PM, Matt Kettler wrote:
>
>
>>>Technically, the "notfirsthop" is a misnomer, and a carry over from
>>>really old
>>
>>3.x reverted to the old way. Try it out.
>>
>
>
> I see you are correct. But why on earth did the devels take a giant step
> backwards and do that?
I imagine the 2.6x code got changed after the 3.x code was written, but
that's just a guess.
> I can see putting the "notfirsthop" code back to really do that. It is correct
> and all. But I can't see why they didn't follow up and change all the rules to
> "firstuntrusted".
>
> The "notfirsthop" algorithm is completely broken for use with DUL RBLs. It can
> improperly penalize dialup users who have internal servers that correctly
> forward to the ISP relay SMTP server. (Think of "neighborhood co-op line share"
> type setups.)
I agree. I've had problems with it myself with servers on generic
static ranges that I can't get rDNS changed for and are in SORBS' DUL.
However, I had (apparently mistakenly) thought that everyone was aware
of it because it's come up at least three times in the last couple
months. Some people referred to 'it' being a good thing over 'limited
DUL checks by the MX'.
I suspect that the lack of affected mail in the scoring corpus is the
reason why it's gone unnoticed. I'd been meaning to run tests to
compare the hits between:
-- notfirsthop and firstuntrusted
-- notfirsthop and "not private and not first hop"
to see if there would be any improvements in the scoring corpora. Of
course even the "not private and not first hop" test wouldn't help
anyone with public IP sending mail to a smart host.
Daryl
Re: false positive in RCVD_IN_SORBS_DUL test
Posted by Matt Kettler <mk...@evi-inc.com>.
Daryl C. W. O'Shea wrote:
> On 08/12/2005 3:52 PM, Matt Kettler wrote:
>> Technically, the "notfirsthop" is a misnomer, and a carry over from
>> really old
>
> 3.x reverted to the old way. Try it out.
>
I see you are correct. But why on earth did the devels take a giant step
backwards and do that?
I can see putting the "notfirsthop" code back to really do that. It is correct
and all. But I can't see why they didn't follow up and change all the rules to
"firstuntrusted".
The "notfirsthop" algorithm is completely broken for use with DUL RBLs. It can
improperly penalize dialup users who have internal servers that correctly
forward to the ISP relay SMTP server. (Think of "neighborhood co-op line share"
type setups.)
Re: false positive in RCVD_IN_SORBS_DUL test
Posted by "Daryl C. W. O'Shea" <sp...@dostech.ca>.
On 08/12/2005 3:52 PM, Matt Kettler wrote:
> Daryl C. W. O'Shea wrote:
>>That's not what the rule is looking for (the last hop).
>>
>>The rule will lookup any hop that is NOT the FIRST hop. Since the mail
>>first passes through a proxy (the hop we don't check as long as there
>>are other external hops) and then passes through another hop (that we do
>>check) the rule is firing since that second hop is listed.
>>
>
>
> Comment for correctness:
>
> Technically, the "notfirsthop" is a misnomer, and a carry over from really old
> versions of spamassassin. In really old versions, it really worked this way. SA
> Checked every IP except the first hop.
>
> However, SA stopped doing that a long time ago. The implementation of this in SA
> 2.60 and higher is actually "first untrusted host delivering mail to a trusted
> host". The "notfirsthop" name remains in the rules, but it's an artifact, and is
> not really implemented this way.
3.x reverted to the old way. Try it out.
Re: false positive in RCVD_IN_SORBS_DUL test
Posted by Matt Kettler <mk...@evi-inc.com>.
Daryl C. W. O'Shea wrote:
> On 08/12/2005 12:10 PM, Russ Ringer wrote:
>
>> Even with TRUSTED_NETWORKS set, the RCVD_IN_SORBS_DUL rule is
>> triggered. I don't see how this is correct, when the IP address that
>> triggered it was not the last hop. This rule should only be triggered
>> when "sent directly from dynamic IP address"
>
>
> That's not what the rule is looking for (the last hop).
>
> The rule will lookup any hop that is NOT the FIRST hop. Since the mail
> first passes through a proxy (the hop we don't check as long as there
> are other external hops) and then passes through another hop (that we do
> check) the rule is firing since that second hop is listed.
>
Comment for correctness:
Technically, the "notfirsthop" is a misnomer, and a carry over from really old
versions of spamassassin. In really old versions, it really worked this way. SA
Checked every IP except the first hop.
However, SA stopped doing that a long time ago. The implementation of this in SA
2.60 and higher is actually "first untrusted host delivering mail to a trusted
host". The "notfirsthop" name remains in the rules, but it's an artifact, and is
not really implemented this way.
Re: false positive in RCVD_IN_SORBS_DUL test
Posted by "Daryl C. W. O'Shea" <sp...@dostech.ca>.
On 08/12/2005 12:10 PM, Russ Ringer wrote:
> Even with TRUSTED_NETWORKS set, the RCVD_IN_SORBS_DUL rule is
> triggered. I don't see how this is correct, when the IP address that
> triggered it was not the last hop. This rule should only be triggered
> when "sent directly from dynamic IP address"
That's not what the rule is looking for (the last hop).
The rule will lookup any hop that is NOT the FIRST hop. Since the mail
first passes through a proxy (the hop we don't check as long as there
are other external hops) and then passes through another hop (that we do
check) the rule is firing since that second hop is listed.
This is actually a problem I'm experiencing myself (internal LAN clients
smart hosting through a server on a static IP that I can't get out of
the dynamic lists) and intend on investigating if there's a better way
to do it without affecting spam hit rates too much.
Anyway, if you want the dynablocks to only take action on the connecting
host, run them on your MX instead and zero the scores for them in SA.
Daryl