You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spamassassin.apache.org by bu...@bugzilla.spamassassin.org on 2007/03/08 22:48:21 UTC
[Bug 5373] New: trusted networks isn't enough for people who use forwarding services
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=5373
Summary: trusted networks isn't enough for people who use
forwarding services
Product: Spamassassin
Version: 3.1.8
Platform: All
OS/Version: All
Status: NEW
Severity: normal
Priority: P5
Component: Rules
AssignedTo: dev@spamassassin.apache.org
ReportedBy: apache@robm.fastmail.fm
The trusted_networks allows you to define the IP addresses of networks you
trust. Consider this situation:
spam-zombie -> zoneedit-mail-forwarding -> fastmail.fm-customer
So fastmail.fm runs SpamAssassin. We set our known internal networks as
trusted_networks. This means the relay analysis algorithm will use the
zoneedit-mail-forwarding IP as the -lastexternal IP for some RBL tests such as:
header RCVD_IN_XBL eval:check_rbl('sblxbl-lastexternal', 'sbl-xbl.spamhaus.org.'
This isn't very useful, we really want to scan further back to test against the
source IP. Now the way to do this is to add all the known forwarding systems to
your trusted_networks. But to do that, you'd have to know the internal IP
arrangement of every place like gmail, yahoo, hotmail, zoneedit, pobox, etc, not
really feasible.
Instead a better idea is to add some trusted_hosts to the list. In this case, we
will "trust" a set of hosts if:
1. We think all their servers are secure and zombie free
2. We know they don't have any dialup/dsl customers with RDNS that contains
their hostname
So we have a new configuration option that looks like:
trusted_host pobox.com
If when scanning back through a Received header, we see an RDNS that ends with
pobox.com, we then do a forward lookup on the name, and if it matches the IP
address in the Received header, we assume the header isn't forged, and because
we trust pobox.com servers, we set that Received header as trusted.
This means we can add trust for whole providers with one configuration option
without having to know the internal IPs of that provider.
For the moment, I haven't created a patch for this, instead I've implemented it
as hooks that replace and override existing code (see attached).
Additional options
1. The trusted_hosts configuration option has an optional second parameter.
trusted_host pobox.com srs
When srs is added, it means we know the host supports SRS forwarding, we use
that in the Mail::SpamAssassin::Plugin::SPF to change how _get_relay works so it
doesn't scan back to the first trusted host based on the new above logic. Also
in attached code.
2. With this system, you can NOT had ISPs to your trusted_hosts set because most
ISPs have correct forward/reverse DNS for their dialup/dsl hosts, which are the
main things we're trying to check against.
A solution to this would be to try and detect dialup/dsl hosts based on their
rdns hostname. This is actually possible to do fairly accurately with a bit of
regexp hackery (see attached)
If this code were used as an exception to the is_trusted_relay test, then you
could also add ISPs to the trusted_hosts list.
3. Rather than each version/user of SA having to maintain a trusted_hosts list,
a better idea would be to have a URIWL that you could query against that would
maintain known good hostnames. Then that could be maintained separately to each
version of spamassassin. No code for that yet...
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 5373] trusted networks isn't enough for people who use forwarding services
Posted by bu...@bugzilla.spamassassin.org.
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=5373
------- Additional Comments From apache@robm.fastmail.fm 2007-03-08 13:58 -------
Created an attachment (id=3880)
--> (http://issues.apache.org/SpamAssassin/attachment.cgi?id=3880&action=view)
Code to determine if RDNS name is a dialup/dsl
We use this code on our systems to determine if a machine is a dialup/dsl host
(eg more dodgy and subject to more checks like greylisting, enumeration
detection, etc). Empirically, it seems to be pretty accurate these days
assuming that the machine has any rdns at all. If not, we assume it's on the
dodgy list.
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 5373] trusted networks isn't enough for people who use forwarding services
Posted by bu...@bugzilla.spamassassin.org.
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=5373
------- Additional Comments From jm@jmason.org 2007-03-22 03:43 -------
I don't think it's in time to get into 3.2.0; at this stage we should consider
that release in feature-freeze mode, IMO. 3.3.0, I'm fine with.
------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.
[Bug 5373] trusted networks isn't enough for people who use forwarding services
Posted by bu...@bugzilla.spamassassin.org.
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=5373
------- Additional Comments From jm@jmason.org 2007-03-20 08:14 -------
btw (In reply to comment #0)
> The trusted_networks allows you to define the IP addresses of networks you
> trust. Consider this situation:
>
> spam-zombie -> zoneedit-mail-forwarding -> fastmail.fm-customer
>
> So fastmail.fm runs SpamAssassin. We set our known internal networks as
> trusted_networks. This means the relay analysis algorithm will use the
> zoneedit-mail-forwarding IP as the -lastexternal IP for some RBL tests such as:
>
> header RCVD_IN_XBL eval:check_rbl('sblxbl-lastexternal', 'sbl-xbl.spamhaus.org.'
>
> This isn't very useful, we really want to scan further back to test against the
> source IP. Now the way to do this is to add all the known forwarding systems to
> your trusted_networks.
hmm, hold on. "trusted" != "internal" -- I think you're conflating
trusted_networks and internal_networks here.
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
Re: [Bug 5373] New: trusted networks isn't enough for people who use forwarding services
Posted by Michael Peddemors <mi...@linuxmagic.com>.
It might be worth some consideration for SA to start examining how it can
honour an external flag to indicate whether the relay client flag was set.
This allows systems with the ability to differentiate to examine messages
differently based on that flag. And would eliminate the need to maintain a
'trusted' network. As well, when the relay client is set via SMTP AUTH, often
the same logic would apply, as if the person was using an IP in a trusted
network. Of course, this is better treated as an enhancement request.
On March 8, 2007 01:48 pm, bugzilla-daemon@bugzilla.spamassassin.org wrote:
> http://issues.apache.org/SpamAssassin/show_bug.cgi?id=5373
>
> Summary: trusted networks isn't enough for people who use
> forwarding services
> Product: Spamassassin
> Version: 3.1.8
> Platform: All
> OS/Version: All
> Status: NEW
> Severity: normal
> Priority: P5
> Component: Rules
> AssignedTo: dev@spamassassin.apache.org
> ReportedBy: apache@robm.fastmail.fm
>
>
> The trusted_networks allows you to define the IP addresses of networks you
> trust. Consider this situation:
>
> spam-zombie -> zoneedit-mail-forwarding -> fastmail.fm-customer
>
> So fastmail.fm runs SpamAssassin. We set our known internal networks as
> trusted_networks. This means the relay analysis algorithm will use the
> zoneedit-mail-forwarding IP as the -lastexternal IP for some RBL tests such
> as:
>
> header RCVD_IN_XBL eval:check_rbl('sblxbl-lastexternal',
> 'sbl-xbl.spamhaus.org.'
>
> This isn't very useful, we really want to scan further back to test against
> the source IP. Now the way to do this is to add all the known forwarding
> systems to your trusted_networks. But to do that, you'd have to know the
> internal IP arrangement of every place like gmail, yahoo, hotmail,
> zoneedit, pobox, etc, not really feasible.
>
> Instead a better idea is to add some trusted_hosts to the list. In this
> case, we will "trust" a set of hosts if:
> 1. We think all their servers are secure and zombie free
> 2. We know they don't have any dialup/dsl customers with RDNS that contains
> their hostname
>
> So we have a new configuration option that looks like:
>
> trusted_host pobox.com
>
> If when scanning back through a Received header, we see an RDNS that ends
> with pobox.com, we then do a forward lookup on the name, and if it matches
> the IP address in the Received header, we assume the header isn't forged,
> and because we trust pobox.com servers, we set that Received header as
> trusted.
>
> This means we can add trust for whole providers with one configuration
> option without having to know the internal IPs of that provider.
>
> For the moment, I haven't created a patch for this, instead I've
> implemented it as hooks that replace and override existing code (see
> attached).
>
> Additional options
>
> 1. The trusted_hosts configuration option has an optional second parameter.
>
> trusted_host pobox.com srs
>
> When srs is added, it means we know the host supports SRS forwarding, we
> use that in the Mail::SpamAssassin::Plugin::SPF to change how _get_relay
> works so it doesn't scan back to the first trusted host based on the new
> above logic. Also in attached code.
>
> 2. With this system, you can NOT had ISPs to your trusted_hosts set because
> most ISPs have correct forward/reverse DNS for their dialup/dsl hosts,
> which are the main things we're trying to check against.
>
> A solution to this would be to try and detect dialup/dsl hosts based on
> their rdns hostname. This is actually possible to do fairly accurately with
> a bit of regexp hackery (see attached)
>
> If this code were used as an exception to the is_trusted_relay test, then
> you could also add ISPs to the trusted_hosts list.
>
> 3. Rather than each version/user of SA having to maintain a trusted_hosts
> list, a better idea would be to have a URIWL that you could query against
> that would maintain known good hostnames. Then that could be maintained
> separately to each version of spamassassin. No code for that yet...
>
>
>
> ------- You are receiving this mail because: -------
> You are the assignee for the bug, or are watching the assignee.
--
--
"Catch the Magic of Linux..."
------------------------------------------------------------------------
Michael Peddemors - President/CEO - LinuxMagic
Products, Services, Support and Development
Visit us at http://www.linuxmagic.com
------------------------------------------------------------------------
A Wizard IT Company - For More Info http://www.wizard.ca
"LinuxMagic" is a Registered TradeMark of Wizard Tower TechnoServices Ltd.
------------------------------------------------------------------------
604-589-0037 Beautiful British Columbia, Canada
This email and any electronic data contained are confidential and intended
solely for the use of the individual or entity to which they are addressed.
Please note that any views or opinions presented in this email are solely
those of the author and are not intended to represent those of the company.
[Bug 5373] trusted networks isn't enough for people who use forwarding services
Posted by bu...@bugzilla.spamassassin.org.
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=5373
------- Additional Comments From spamassassin@dostech.ca 2007-03-21 23:43 -------
I haven't looked at the code, but this is something I've had in mind for a
while. I wanted to pluginize most of Received.pm first though. I could see
this ending up in a 3.2.x release sometime in the summer though.
The SRS support part needs to end up in the SPF plugin though, it's not
something we should screw about with while determining trusted/internal though.
(In reply to comment #4)
> hmm, hold on. "trusted" != "internal" -- I think you're conflating
> trusted_networks and internal_networks here.
Maybe, maybe not. In his described case they'll be the same anyway. It really
has nothing to do with adding support for using rDNS to extend trust though.
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 5373] trusted networks isn't enough for people who use forwarding services
Posted by bu...@bugzilla.spamassassin.org.
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=5373
jm@jmason.org changed:
What |Removed |Added
----------------------------------------------------------------------------
Target Milestone|Undefined |3.3.0
------- Additional Comments From jm@jmason.org 2007-03-13 08:40 -------
I'm pretty keen on this. aiming at 3.3.0
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 5373] trusted networks isn't enough for people who use forwarding services
Posted by bu...@bugzilla.spamassassin.org.
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=5373
------- Additional Comments From apache@robm.fastmail.fm 2007-03-25 17:12 -------
Sorry, I'm pulling in this quote from
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=5374 because I think it's
more related to this and should respond here:
> FWIW, I think extending trusted_networks towards the sender further than
> internal_networks is pointless 99.99% of the time (your local DNS cache absorbs
> any penalty of a few extra DNS lookups in systems busy enough for the penalty to
> be an issue). I recommend that people don't do it. The only place I see
> differing trusted/internal configs is on the MSA side where you can't determine
> auth from the headers (or POPAuth). With msa_networks now implemented it's not
> even necessary in this case anymore.
In hindsight, I think I'm agreeing with this. I think I was confused about the
use of -lastexternal, and now I'm not convinced that my patch here actually adds
anything useful to DNSBL checks.
So I went back and checked some more to see what it was helping on, and really
it seemed to be 2 cases:
1. Forwarder hosts that for some reason were on an RBL. By extended trust into
them, their IPs aren't checked, and thus when some persons forwarding service
ends up being RBL listed for some reason, suddenly a lot of their email was
being marked as spam
2. SPF checks for forwarding services that don't use SRS because I have some
other code that overrides Mail::SpamAssassin::Plugin::SPF::_get_relay to check
each relay for ->{trust_type}.
Apart from that though, I'm not sure any of this is actually a help, and I may
have been leading things up the garden path. Still, things that reduce false
positives and overall accuracy for users is always a good thing.
Rob
------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.
[Bug 5373] trusted networks isn't enough for people who use forwarding services
Posted by bu...@bugzilla.spamassassin.org.
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=5373
------- Additional Comments From apache@robm.fastmail.fm 2007-03-08 13:54 -------
Created an attachment (id=3879)
--> (http://issues.apache.org/SpamAssassin/attachment.cgi?id=3879&action=view)
Add trusted_host option
The code we use that overwrites existing SA methods to add trusted_host
Unfortunately Mail::SpamAssassin::Message::Metadata::parse_received_headers
doesn't really provide a way to easily hook in a new trust system, so I had to
reproduce the whole function with really just a small change, these lines:
# trusted_networks matches?
if ($in_trusted && $did_user_specify_trust && !$self->is_trusted($relay,
$trusted))
{
$in_trusted = 0; # we're in deep water now
}
It would be nice to actually change parse_received_headers so there are more
"hook" places people could get into if they want to change the overall
algorithm in the future as well...
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 5373] trusted networks isn't enough for people who use forwarding services
Posted by bu...@bugzilla.spamassassin.org.
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=5373
------- Additional Comments From spamassassin@dostech.ca 2007-03-22 03:48 -------
Yeah, not now, but maybe a small patch to 3.2.x later...
(In reply to comment #5)
> I could see this ending up in a 3.2.x release sometime in the summer though.
------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.
[Bug 5373] trusted networks isn't enough for people who use forwarding services
Posted by bu...@bugzilla.spamassassin.org.
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=5373
spamassassin@dostech.ca changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |dev@spamassassin.apache.org
AssignedTo|dev@spamassassin.apache.org |spamassassin@dostech.ca
------- Additional Comments From spamassassin@dostech.ca 2007-03-21 23:48 -------
BTW... I meant I could see this implemented in Received.pm during 3.2,
pluginizing would be 3.3.
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
You are on the CC list for the bug, or are watching someone who is.
[Bug 5373] trusted networks isn't enough for people who use forwarding services
Posted by bu...@bugzilla.spamassassin.org.
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=5373
------- Additional Comments From jm@jmason.org 2007-03-22 04:02 -------
I was about to say "no", but then I realised that it doesn't affect _existing_
behaviour -- it's a new feature driven entirely by a new config keyword. so
maybe ;)
------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.