You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by Stefan `Sec` Zehl <se...@42.org> on 2008/02/26 13:44:10 UTC
AWL problem. Assigning very low scores to spam.
Hi,
I'm having trouble with the AWL of Spamassassin.
The AWL is acting seriously wrong. I get some spam with my own address
in the "From:" header, and the AWL assigns ridiculous scores to it.
I have quite a few cronjobs running which send mail with the same
"From"-address on my local machine but that shouldn't extend to external
Mails (at least, I hope).
To aid in debugging I have completely removed the auto-whitelist file in
my ~/.spamassassin. A few minutes later, the database looks like this:
| ice:~/.spamassassin>date
| Tue Feb 26 10:27:45 CET 2008
| ice:~/.spamassassin>dbedit -p auto-whitelist|grep 'sec@42.org'
| sec@42.org|ip=83.239 1
| sec@42.org|ip=none|totscore -99.133
| sec@42.org|ip=83.239|totscore 9.14
| sec@42.org|ip=none 2
which looks plausible to me.
But three hours later, the first spam gets through again. The Database
now look like this:
| ice:~/.spamassassin>date
| Tue Feb 26 13:35:05 CET 2008
| ice:~/.spamassassin>dbedit -p auto-whitelist|grep 'sec@42.org'
| sec@42.org|ip=117.47 5
| sec@42.org|ip=117.47|totscore -188.926
Why is this happening? Can This be fixed? Or do I have to turn AWL
completely off because it is broken by design?
CU,
Sec
--
A bureaucracy is like a computer program. Usually, the question is
how to arrange it so that what you want is composed of operations that
the bureaucracy supports. In addition, in any bureaucracy, there is
always *someone* whose job is to approve violations of the rules.
Re: SA trusts all hosts as soon as IPv6 is enabled? (was Re: AWL problem. Assigning very low scores to spam.)
Posted by Loren Wilton <lw...@earthlink.net>.
> Ok, here is a patch which fixes this specific (IPv6) problem until
> someone has time to make SA completely v6 aware:
>
> --- Mail/SpamAssassin/Message/Metadata/Received.pm.orig 2008-02-26
> 17:28:28.000000000 +0100
> +++ Mail/SpamAssassin/Message/Metadata/Received.pm 2008-02-26
> 17:28:52.000000000 +0100
> @@ -1208,7 +1208,8 @@
> $ip = Mail::SpamAssassin::Util::extract_ipv4_addr_from_string ($ip);
> if (!$ip) {
> dbg("received-header: could not parse IPv4 address, assuming IPv6");
> - return 0; # ignore IPv6 handovers
> +# return 0; # ignore IPv6 handovers
> + $ip="0.0.0.0";
> }
I'd suggest submitting this on the SA Bugzilla. It will get lost here on
the user's list.
I htink I'd submit a *second* bug about how any failed received line parse
causes all other headers to be trusted. That certainly wasn't how it worked
at at least one point in the past.
Loren
Re: SA trusts all hosts as soon as IPv6 is enabled? (was Re: AWL problem. Assigning very low scores to spam.)
Posted by Stefan `Sec` Zehl <se...@42.org>.
Hi,
On Tue, Feb 26, 2008 at 16:26 +0000, Justin Mason wrote:
> Stefan `Sec` Zehl writes:
> > Ok, so you're telling me that not only is this bug known, but it went
> > unfixed fot over a year?
>
> Unfortunately, nobody who's bothered by it, has bothered fixing it
> and sending us a patch. I'll omit any comments about IPv6 users ;)
[...]
> yes, we know that ;) If we had infinite time, it'd be fixed by now.
Ok, here is a patch which fixes this specific (IPv6) problem until
someone has time to make SA completely v6 aware:
--- Mail/SpamAssassin/Message/Metadata/Received.pm.orig 2008-02-26 17:28:28.000000000 +0100
+++ Mail/SpamAssassin/Message/Metadata/Received.pm 2008-02-26 17:28:52.000000000 +0100
@@ -1208,7 +1208,8 @@
$ip = Mail::SpamAssassin::Util::extract_ipv4_addr_from_string ($ip);
if (!$ip) {
dbg("received-header: could not parse IPv4 address, assuming IPv6");
- return 0; # ignore IPv6 handovers
+# return 0; # ignore IPv6 handovers
+ $ip="0.0.0.0";
}
# DISABLED: if we cut out localhost-to-localhost SMTP handovers,
> > But the bigger problem remains, and it is not the IPv6 stuff. The main
> > problem here is, that if the first Received header is (for what reason
> > ever) unparsable, all the other (spammer-controlled) headers are
> > trusted if they have an "auth" part. I would say the default here is
> > definitely the wrong way round.
>
> it's a bug. It needs fixing... the right way is to parse IPv6 headers.
> So far it hasn't been a significant problem, since I think yours is
> the first example I've seen of spam traversing IPv6 networks to arrive
> at a trusted network.
My point is. ANY reason to misparse a received-header leads to automatic
trusting of untrusted headers.
Do you trust SA to never misparse a Received-line? I have seen the
inside of that function and the tons of regexps there. I would not trust
it to be completely bugfree.
I may well be the first person to report a spam, but I am quite sure
there are more people out there with Spam mistakenly getting the
ALL_TRUSTED label. After all, who checks the headers of their
Spam-Mailbox regularely?
> > But then, I'm only a stupid user and who cares about those %)
> Hardly representative of our attitude.
I'll take your word for it. I was miffed realizing that after half a day
of debugging I found a year old bug -- which is still unfixed.
CU,
Sec
--
Hofstadter's Law: Everything takes longer than you expect,
even taking into account Hofstadter's Law.
Re: SA trusts all hosts as soon as IPv6 is enabled? (was Re: AWL problem. Assigning very low scores to spam.)
Posted by Stefan `Sec` Zehl <se...@42.org>.
On Tue, Feb 26, 2008 at 19:13 -0500, Daryl C. W. O'Shea wrote:
[...]
> If you or your company would like to fund the development of it, I'm
> willing to prioritize the work. Seriously. Otherwise, "should have by
> now" does not apply to free software. Especially free software that is
> easily monetized by its users. If the lack of a feature you want
> doesn't bother anyone else enough to implement it the only one you can
> expect to dedicate time or resources to the work is yourself.
As I said before, I'm only a "stupid user" and have no commercial
interest in SA. -- I did however produce a simple "workaround" patch and
sent it to this list already.
> In the case of IPv6 (in SA), none of
> us have had the need for it ourselves or perceived the need of it by
> enough users being greater than the need for other things we've spent
> our time on instead.
The problem here is not the missing of IPv6 support, but the fact that
it makes SA trust random headers.
CU,
Sec
--
perl -le 's,us(?=r),he,,print
if (($_=qq/(*_=*\047)=~y#!perl -e hk #rJust -`neocheat#,*_;
$^X hacker!/)=~s<.*;>;($_=$&)=~y~*~$~,$_;ee)'
Re: SA trusts all hosts as soon as IPv6 is enabled? (was Re: AWL
problem. Assigning very low scores to spam.)
Posted by "Daryl C. W. O'Shea" <sp...@dostech.ca>.
On 26/02/2008 11:07 AM, Stefan `Sec` Zehl wrote:
> Hi,
>
> On Tue, Feb 26, 2008 at 15:56 +0000, Justin Mason wrote:
>> The fix would be to implement support for IPv6 trust paths:
>>
>> http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4503
>> http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4964
>
> Ok, so you're telling me that not only is this bug known, but it went
> unfixed fot over a year?
Yeah -- although I consider it a feature enhancement, not a bug... SA
just doesn't support IPv6. Full blown IPv6 support has been on my list
of things I'd like to do for just over three years now. bug 4964
describes at most half of what needs to be done to implement full
support for IPv6.
> I must admit that I don't know much of SAs internals or how hard it is
> to fix this "the correct way".
>
> However a bug like that should have been fixed -- or at least worked
> around by now.
If you or your company would like to fund the development of it, I'm
willing to prioritize the work. Seriously. Otherwise, "should have by
now" does not apply to free software. Especially free software that is
easily monetized by its users. If the lack of a feature you want
doesn't bother anyone else enough to implement it the only one you can
expect to dedicate time or resources to the work is yourself.
> But then, I'm only a stupid user and who cares about those %)
That's absurd. If we didn't care about users we wouldn't expend the
effort to support the software (which is often as much or more than the
effort spent actually developing the software) or implementing anything
that doesn't benefit us directly. In the case of IPv6 (in SA), none of
us have had the need for it ourselves or perceived the need of it by
enough users being greater than the need for other things we've spent
our time on instead.
Daryl
Re: SA trusts all hosts as soon as IPv6 is enabled? (was Re: AWL problem. Assigning very low scores to spam.)
Posted by Stefan `Sec` Zehl <se...@42.org>.
Hi,
On Tue, Feb 26, 2008 at 15:56 +0000, Justin Mason wrote:
> The fix would be to implement support for IPv6 trust paths:
>
> http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4503
> http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4964
Ok, so you're telling me that not only is this bug known, but it went
unfixed fot over a year?
I must admit that I don't know much of SAs internals or how hard it is
to fix this "the correct way".
However a bug like that should have been fixed -- or at least worked
around by now.
A simple workaround would be to hardcode a fake IP (like "0.0.0.0") for
IPv6.
But the bigger problem remains, and it is not the IPv6 stuff. The main
problem here is, that if the first Received header is (for what reason
ever) unparsable, all the other (spammer-controlled) headers are
trusted if they have an "auth" part. I would say the default here is
definitely the wrong way round.
But then, I'm only a stupid user and who cares about those %)
CU,
Sec
--
Not a perfect solution, but far cheaper than one.
Re: SA trusts all hosts as soon as IPv6 is enabled? (was Re: AWL problem. Assigning very low scores to spam.)
Posted by Stefan `Sec` Zehl <se...@42.org>.
Hi,
Ok, I debugged this a bit more.
Problem is, these headers were marked as ALL_TRUSTED:
> > | Received: from mout4.freenet.de (mout4.freenet.de [IPv6:2001:748:100:40::2:6])
> > | (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
> > | (No client certificate requested)
> > | by ice.42.org (Postfix) with ESMTPS id D189AB85A
> > | for <se...@42.org>; Tue, 26 Feb 2008 11:51:08 +0100 (CET)
> > | Received: from [195.4.92.23] (helo=13.mx.freenet.de)
> > | by mout4.freenet.de with esmtpa (Exim 4.69)
> > | (envelope-from <ed...@yahoo.com>)
> > | id 1JTxOR-0002Vk-38; Tue, 26 Feb 2008 11:50:39 +0100
> > | Received: from [82.128.34.27] (port=1797 helo=User)
> > | by 13.mx.freenet.de with esmtpa (ID nkume1@freenet.de) (port 25) (Exim 4.69 #10)
> > | id 1JTxOO-0005uv-2T; Tue, 26 Feb 2008 11:50:38 +0100
The detailed problem is, the first header is completely ignored because
of its IPv6 content.
The second line contains "with esmtpa" which makes SpamAssassin
unconditionally trust this header. Case in Point:
SpamAssassin/Message/Metadata/Received.pm around line 192:
| # trusted_networks matches?
| if (!$relay->{auth} && !$trusted->contains_ip($relay->{ip})) {
| $in_trusted = 0;
It is completely irrelevant if the IP is in trusted_networks or not. If
the Received line contains "auth" which at this point contains "esmtpa"
it considers the Header good and trusted.
I fixed that particular problem for now by forcing "auth" to be empty
at the end of the "parse_received_line" function, but as $auth was
included for some reason, somebody should look closer at how to fix this
completely.
CU,
Sec
--
The problem with troubleshooting is that trouble shoots back.
SA trusts all hosts as soon as IPv6 is enabled? (was Re: AWL problem. Assigning very low scores to spam.)
Posted by Stefan `Sec` Zehl <se...@42.org>.
Hi,
On Tue, Feb 26, 2008 at 14:56 +0100, Stefan `Sec` Zehl wrote:
>
[... on producing ALL_TRUSTED with these header ...]
>
> | Received: from mout4.freenet.de (mout4.freenet.de [IPv6:2001:748:100:40::2:6])
> | (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
> | (No client certificate requested)
> | by ice.42.org (Postfix) with ESMTPS id D189AB85A
> | for <se...@42.org>; Tue, 26 Feb 2008 11:51:08 +0100 (CET)
> | Received: from [195.4.92.23] (helo=13.mx.freenet.de)
> | by mout4.freenet.de with esmtpa (Exim 4.69)
> | (envelope-from <ed...@yahoo.com>)
> | id 1JTxOR-0002Vk-38; Tue, 26 Feb 2008 11:50:39 +0100
> | Received: from [82.128.34.27] (port=1797 helo=User)
> | by 13.mx.freenet.de with esmtpa (ID nkume1@freenet.de) (port 25) (Exim 4.69 #10)
> | id 1JTxOO-0005uv-2T; Tue, 26 Feb 2008 11:50:38 +0100
I did some more Tests with these headers.
They are unconditionally marked as trusted. The problem is the following
line from "spamassasin -D -L -t":
| [52994] dbg: received-header: could not parse IPv4 address, assuming IPv6
As soon as this line appears, sa trusts everything. No matter what you
set in trusted_networks or anywhere else. It doesn't even parse that
header at all (notice that there are only two "parsed as" lines):
| [53147] dbg: received-header: parsed as [ ip=195.4.92.23 rdns= helo=13.mx.freenet.de by=mout4.freenet.de ident= envfrom=edward4gplottery@yahoo.com intl=0 id=1JTxOR-0002Vk-38 auth=esmtpa msa=0 ]
| [53147] dbg: received-header: relay 195.4.92.23 trusted? yes internal? yes msa? no
| [53147] dbg: received-header: parsed as [ ip=82.128.34.27 rdns= helo=User by=13.mx.freenet.de ident= envfrom= intl=0 id=1JTxOO-0005uv-2T auth=esmtpa msa=0 ]
| [53147] dbg: received-header: relay 82.128.34.27 trusted? yes internal? yes msa? no
Replacing the "[IPv6:2001:748:100:40::2:6]" with "[1.2.3.4]", everything
is back to normal:
| [53033] dbg: received-header: parsed as [ ip=1.2.3.4 rdns=mout4.freenet.de helo=mout4.freenet.de by=ice.42.org ident= envfrom= intl=0 id=D189AB85A auth= msa=0 ]
| [53033] dbg: received-header: relay 1.2.3.4 trusted? no internal? no msa? no
| [53033] dbg: received-header: parsed as [ ip=195.4.92.23 rdns= helo=13.mx.freenet.de by=mout4.freenet.de ident= envfrom=edward4gplottery@yahoo.com intl=0 id=1JTxOR-0002Vk-38 auth=esmtpa msa=0 ] [53033] dbg: received-header: relay 195.4.92.23 trusted? no internal? no msa? no
| [53033] dbg: received-header: parsed as [ ip=82.128.34.27 rdns= helo=User by=13.mx.freenet.de ident= envfrom= intl=0 id=1JTxOO-0005uv-2T auth=esmtpa msa=0 ]
| [53033] dbg: received-header: relay 82.128.34.27 trusted? no internal? no msa? no
So it appears that spamassassins v6 support is broken. -- Is there some config option i missed, or is the only solution to turn off IPv6 on my mailserver?
CU,
Sec
--
"The General who in a hundred battles is always victorious is not as
great as the one who achieves his objectives without fighting."
-- Sun Tzu
Re: AWL problem. Assigning very low scores to spam.
Posted by Stefan `Sec` Zehl <se...@42.org>.
Hi,
On Tue, Feb 26, 2008 at 08:38 -0500, Matt Kettler wrote:
> Stefan `Sec` Zehl wrote:
> >The AWL is acting seriously wrong. I get some spam with my own address
> >in the "From:" header, and the AWL assigns ridiculous scores to it.
> Any chance you have a broken trust path? (ie: does ALL_TRUSTED ever fire
> off on outside email?)
I'm not sure how I can check that...
Until a few days ago I had no "trusted_networks" in my config. After
googling around I set it "trusted_networks 194.77.85.2/27" in my
user_prefs. But that has not changed anything as far as I can tell
But you may be on to something. I found Mails in my spam-folder which
have ALL_TRUSTED set. Running such a message through
spamassassin -D -L -t produces this:
| [50155] dbg: conf: internal_networks not configured, using trusted_networks configuration for internal_networks; if you really want internal_networks to only contain the required 127/8 add 'internal_networks !0/0' to your configuration
| [50155] dbg: received-header: could not parse IPv4 address, assuming IPv6
| [50155] dbg: received-header: parsed as [ ip=195.4.92.23 rdns= helo=13.mx.freenet.de by=mout4.freenet.de ident= envfrom=edward4gplottery@yahoo.com intl=0 id=1JTxOR-0002Vk-38 auth=esmtpa msa=0 ]
| [50155] dbg: received-header: relay 195.4.92.23 trusted? yes internal? yes msa? no
| [50155] dbg: received-header: parsed as [ ip=82.128.34.27 rdns= helo=User by=13.mx.freenet.de ident= envfrom= intl=0 id=1JTxOO-0005uv-2T auth=esmtpa msa=0 ]
| [50155] dbg: received-header: relay 82.128.34.27 trusted? yes internal? yes msa? no
| [50155] dbg: metadata: X-Spam-Relays-Trusted: [ ip=195.4.92.23 rdns= helo=13.mx.freenet.de by=mout4.freenet.de ident= envfrom=edward4gplottery@yahoo.com intl=1 id=1JTxOR-0002Vk-38 auth=esmtpa msa=0 ] [ ip=82.128.34.27 rdns= helo=User by=13.mx.freenet.de ident= envfrom= intl=1 id=1JTxOO-0005uv-2T auth=esmtpa msa=0 ]
| [50155] dbg: metadata: X-Spam-Relays-Untrusted:
| [50155] dbg: metadata: X-Spam-Relays-Internal: [ ip=195.4.92.23 rdns= helo=13.mx.freenet.de by=mout4.freenet.de ident= envfrom=edward4gplottery@yahoo.com intl=1 id=1JTxOR-0002Vk-38 auth=esmtpa msa=0 ] [ ip=82.128.34.27 rdns= helo=User by=13.mx.freenet.de ident= envfrom= intl=1 id=1JTxOO-0005uv-2T auth=esmtpa msa=0 ]
This is clearly wrong. But Why?
The Received-Headers of this example Mail look like this:
| Received: from mout4.freenet.de (mout4.freenet.de [IPv6:2001:748:100:40::2:6])
| (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
| (No client certificate requested)
| by ice.42.org (Postfix) with ESMTPS id D189AB85A
| for <se...@42.org>; Tue, 26 Feb 2008 11:51:08 +0100 (CET)
| Received: from [195.4.92.23] (helo=13.mx.freenet.de)
| by mout4.freenet.de with esmtpa (Exim 4.69)
| (envelope-from <ed...@yahoo.com>)
| id 1JTxOR-0002Vk-38; Tue, 26 Feb 2008 11:50:39 +0100
| Received: from [82.128.34.27] (port=1797 helo=User)
| by 13.mx.freenet.de with esmtpa (ID nkume1@freenet.de) (port 25) (Exim 4.69 #10)
| id 1JTxOO-0005uv-2T; Tue, 26 Feb 2008 11:50:38 +0100
CU,
Sec
--
Procmail looks to me like an explosion at an ASCII factory.
Re: AWL problem. Assigning very low scores to spam.
Posted by Matt Kettler <mk...@verizon.net>.
Stefan `Sec` Zehl wrote:
> Hi,
>
> I'm having trouble with the AWL of Spamassassin.
>
> The AWL is acting seriously wrong. I get some spam with my own address
> in the "From:" header, and the AWL assigns ridiculous scores to it.
>
Any chance you have a broken trust path? (ie: does ALL_TRUSTED ever fire
off on outside email?)