You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by Quanah Gibson-Mount <qu...@zimbra.com> on 2013/10/22 21:01:46 UTC
Spam constantly being autolearned as ham
We have an issue where a lot of spam is being autolearned as HAM by SA. Do
people generally turn off autolearn? In looking at these cases, I'm not
seeing where it is particularly helpful, but it is particularly harmful.
Example:
X-Spam-Status: No, score=0.348 tagged_above=-10 required=3
tests=[BAYES_50=0.8, DKIM_SIGNED=0.1, DKIM_VALID=-0.1,
DKIM_VALID_AU=-0.1, HTML_IMAGE_RATIO_02=0.437, HTML_MESSAGE=0.001,
RP_MATCHES_RCVD=-0.8, T_HEADER_FROM_DIFFERENT_DOMAINS=0.01]
autolearn=ham
Authentication-Results: edge02-zcs.vmware.com (amavisd-new);
dkim=pass (1024-bit key) header.d=superwebmais.com;
domainkeys=fail (1024-bit key)
reason="fail (message has been altered)"
header.from=paula@superwebmais.com header.d=superwebmais.com
Received: from edge02-zcs.vmware.com ([127.0.0.1])
by localhost (edge02-zcs.vmware.com [127.0.0.1]) (amavisd-new, port 10024)
with ESMTP id UWg6H9T4tKVE; Tue, 22 Oct 2013 11:27:06 -0700 (PDT)
Received: from c115-smtp.pumpery.com (c115-smtp.pumpery.com [5.135.12.243])
by edge02-zcs.vmware.com (Postfix) with ESMTP id 76999784
for <>; Tue, 22 Oct 2013 11:27:05 -0700 (PDT)
Subject:
=?UTF-8?B?TmV0c2hvZXM6IFPDsyBIb2plIGF0w6kgNjAlIE9GRiBuYXMgbWVsaG9yZXMgbWFyY2FzIGUgQWRpZGFzIFNwcmluZ2JsYWRlIGVtIGF0ZSAxMnggc2VtIGp1cm9zLCBnYXJhbnRhIG8gc2V1IGFxdWk=?=
Message-ID: <6a...@pumpery.com>
Date: Tue, 22 Oct 2013 20:07:11 +0200
From: "Especial Esportes " <pa...@superwebmais.com>
Reply-To: paula@superwebmais.com
--Quanah
--
Quanah Gibson-Mount
Architect - Server
Zimbra, Inc.
--------------------
Zimbra :: the leader in open source messaging and collaboration
Re: Spam constantly being autolearned as ham
Posted by Axb <ax...@gmail.com>.
Correction:
...........
I hope, for your health, that you're *NOT* going to blacklist every from
in a missed spam
...........
Re: Spam constantly being autolearned as ham
Posted by Matus UHLAR - fantomas <uh...@fantomas.sk>.
>--On Tuesday, October 22, 2013 11:09 PM +0200 Axb
><ax...@gmail.com> wrote:
>
>>You've missed the point.
>>
>>mynetworks is not SA - it's Postfix and SA knows nothing about this
>>config option.
>>
>>as you have SA configured, RBL lookups are done against the vmware IPs
>>and I doubt those will be blacklisted, anywhere.
>>
>>If you add 208.91.0.0/22 to your SA trusted_networks (in local.cf)
On 22.10.13 14:28, Quanah Gibson-Mount wrote:
>My SA already has trusted_networks configured as well, but you are
>right, this range is missing, thanks. We push the mta network bits
>out to all portions of the mta (postfix, amavis, SA, dspam). It
>looks like VMW made some IP address changes w/o notifying me. Sigh.
you should apparently also configure internal_networks for all mailservers
that pass you mail, e.g. MX servers.
--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
You have the right to remain silent. Anything you say will be misquoted,
then used against you.
Re: Spam constantly being autolearned as ham
Posted by Matus UHLAR - fantomas <uh...@fantomas.sk>.
>--On Wednesday, October 23, 2013 1:35 AM +0200 Karsten Bräckelmann
><gu...@rudersport.de> wrote:
>> $ spamassassin --lint --cf="trusted_networks [::1]/128"
>> warn: netset: illegal network address given: '[::1]/128'
>>
>>Included by default as well. And even bad syntax.
On 22.10.13 16:48, Quanah Gibson-Mount wrote:
>However, it also does not cause harm to include the local addresses.
>Whether or not the syntax is bad sounds like an argument you can take
>to the postfix authors. Clearly their tool to generate it feels it
>is valid.
>
>The values themselves are generated by postfix, via postconf -d mynetworks
[...]
>However, it is a leftover from a bug in postfix a while back, I've
>fixed that.
does postfix have tool to generate SpamAssassin configs?
--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
I feel like I'm diagonally parked in a parallel universe.
Re: Spam constantly being autolearned as ham
Posted by Karsten Bräckelmann <gu...@rudersport.de>.
On Tue, 2013-10-22 at 16:48 -0700, Quanah Gibson-Mount wrote:
> --On Wednesday, October 23, 2013 1:35 AM +0200 Karsten Bräckelmann wrote:
> > $ spamassassin --lint --cf="trusted_networks 127.0.0.0/8"
> > warn: netset: cannot include 127.0.0.0/8 as it has already been included
> >
> > M::SA::Conf docs, section Network Test Options, option trusted_networks
> > states: "Note: 127/8 and ::1 are always included in trusted_networks,
> > regardless of your config."
> >
> > $ spamassassin --lint --cf="trusted_networks [::1]/128"
> > warn: netset: illegal network address given: '[::1]/128'
> >
> > Included by default as well. And even bad syntax.
>
> However, it also does not cause harm to include the local addresses.
> Whether or not the syntax is bad sounds like an argument you can take to
> the postfix authors. Clearly their tool to generate it feels it is valid.
I'm not arguing about the syntax being valid. Frankly, I couldn't care
less. ;)
Spotting bad configuration causing lint errors, however, immediately
triggers an alarm. Bad configuration usually means the OP failed to lint
check. Which means there may be more and much worse issues in the full
configuration than in the few lines posted...
> The values themselves are generated by postfix, via postconf -d mynetworks
Fair enough. Though I don't particularly care what other tools generate.
I do care what users feed SA and its configuration. I do care if people
tend here with issues. And in this case, there's some proper glue
missing to convert postconf output to SA trusted_networks arguments.
Breaking stuff or not -- I'd file a bug with the tool that automatically
generates config that fails lint. If I (wearing an admin hat) would spot
the lint issues.
--
char *t="\10pse\0r\0dtu\0.@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4";
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i<l;i++){ i%8? c<<=1:
(c=*++x); c&128 && (s+=h); if (!(h>>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}
Re: Spam constantly being autolearned as ham
Posted by Karsten Bräckelmann <gu...@rudersport.de>.
On Tue, 2013-10-22 at 16:59 -0700, Quanah Gibson-Mount wrote:
> --On Tuesday, October 22, 2013 4:48 PM -0700 Quanah Gibson-Mount
> <qu...@zimbra.com> wrote:
>
> >> $ spamassassin --lint --cf="trusted_networks [::1]/128"
> >> warn: netset: illegal network address given: '[::1]/128'
>
> Actually, it appears you are using an out of date spamassassin. ;)
>
> [zimbra@edge02-zcs ~]$ /opt/zimbra/zimbramon/bin/spamassassin --lint
> --cf="trusted_networks [::1]/128"
> Oct 22 16:58:40.587 [12363] warn: netset: cannot include
> 0:0:0:0:0:0:0:1/128 as it has already been included
Chapeau!
Well, that, or an old-ish NetAddr::IP. If we consider SpamAssassin 3.3.2
"outdated", rather than "not trunk". ;)
Blame lazy me for just testing on a Debian 7.2 system without diving
down trunk code.
--
char *t="\10pse\0r\0dtu\0.@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4";
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i<l;i++){ i%8? c<<=1:
(c=*++x); c&128 && (s+=h); if (!(h>>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}
Re: Spam constantly being autolearned as ham
Posted by Quanah Gibson-Mount <qu...@zimbra.com>.
--On Tuesday, October 22, 2013 4:48 PM -0700 Quanah Gibson-Mount
<qu...@zimbra.com> wrote:
>> $ spamassassin --lint --cf="trusted_networks [::1]/128"
>> warn: netset: illegal network address given: '[::1]/128'
Actually, it appears you are using an out of date spamassassin. ;)
[zimbra@edge02-zcs ~]$ /opt/zimbra/zimbramon/bin/spamassassin --lint
--cf="trusted_networks [::1]/128"
Oct 22 16:58:40.587 [12363] warn: netset: cannot include
0:0:0:0:0:0:0:1/128 as it has already been included
--Quanah
--
Quanah Gibson-Mount
Architect - Server
Zimbra, Inc.
--------------------
Zimbra :: the leader in open source messaging and collaboration
Re: Spam constantly being autolearned as ham
Posted by Benny Pedersen <me...@junc.eu>.
Quanah Gibson-Mount skrev den 2013-10-23 03:05:
> No, it was literally a bug in the early postfix 2.10 development
> releases. I reported it back to Wietse a few years ago, but never
> fixed my config. ;)
yes i remember it now, one should show diff in ifconfig vs postconf -d |
grep mynetwork
:-)
i still think main.cf as created default have to much defaults not
commented, well its an example congfig, but there is alot of distros
that use it as a must have :(
Re: Spam constantly being autolearned as ham
Posted by Quanah Gibson-Mount <qu...@zimbra.com>.
--On Wednesday, October 23, 2013 2:19 AM +0200 Benny Pedersen <me...@junc.eu>
wrote:
> Quanah Gibson-Mount skrev den 2013-10-23 01:48:
>
>> However, it is a leftover from a bug in postfix a while back, I've
>> fixed that.
>
> bah, its not in the output of ifconfig, is it ?, if it is dont blame
> postfix :)
No, it was literally a bug in the early postfix 2.10 development releases.
I reported it back to Wietse a few years ago, but never fixed my config. ;)
--Quanah
--
Quanah Gibson-Mount
Architect - Server
Zimbra, Inc.
--------------------
Zimbra :: the leader in open source messaging and collaboration
Re: Spam constantly being autolearned as ham
Posted by Benny Pedersen <me...@junc.eu>.
Quanah Gibson-Mount skrev den 2013-10-23 01:48:
> However, it is a leftover from a bug in postfix a while back, I've
> fixed that.
bah, its not in the output of ifconfig, is it ?, if it is dont blame
postfix :)
Re: Spam constantly being autolearned as ham
Posted by Quanah Gibson-Mount <qu...@zimbra.com>.
--On Wednesday, October 23, 2013 1:35 AM +0200 Karsten Bräckelmann
<gu...@rudersport.de> wrote:
^^^^^^^^^^^ ^^^^^^^^^
>> 204.14.232.64/28 204.14.234.64/28 202.129.242.65/32 96.43.144.64/32
>> 96.43.144.65/32 96.43.148.64/32 96.43.148.65/32 182.50.78.64/28
>> 208.91.2.22/31
>
> Excuse me for being blunt, but it appears you didn't lint check in quite
> a while. That is absolutely borked.
>
> $ spamassassin --lint --cf="trusted_networks 127.0.0.0/8"
> warn: netset: cannot include 127.0.0.0/8 as it has already been included
>
> M::SA::Conf docs, section Network Test Options, option trusted_networks
> states: "Note: 127/8 and ::1 are always included in trusted_networks,
> regardless of your config."
>
> $ spamassassin --lint --cf="trusted_networks [::1]/128"
> warn: netset: illegal network address given: '[::1]/128'
>
> Included by default as well. And even bad syntax.
However, it also does not cause harm to include the local addresses.
Whether or not the syntax is bad sounds like an argument you can take to
the postfix authors. Clearly their tool to generate it feels it is valid.
The values themselves are generated by postfix, via postconf -d mynetworks
> And that last address range [fe80::%eth0]/64 on the first line is just
> weird -- what's supposed to substitute that ethernet interface
> placeholder?
Generally it just gets dropped:
Oct 22 12:09:24 edge02-zcs amavis[27883]: SA info: netset: ignoring
interface scope '%eth0' in IP address [fe80::%eth0]/64
However, it is a leftover from a bug in postfix a while back, I've fixed
that.
--Quanah
--
Quanah Gibson-Mount
Architect - Server
Zimbra, Inc.
--------------------
Zimbra :: the leader in open source messaging and collaboration
Re: Spam constantly being autolearned as ham
Posted by Karsten Bräckelmann <gu...@rudersport.de>.
On Tue, 2013-10-22 at 21:50 -0500, David B Funk wrote:
> Actually that %thingy is the IPv6 zone_id (which functionally maps to the
> interface).
Good point, thanks. And appropriately named article link.
> Given that the LLA is the same bit range regardless of interface, on a
> multi-homed machine it is necessary to specify interface as well as address.
>
> See:
> RFC-4007 section 11
> http://blogs.gentoo.org/eva/2010/12/17/things-you-didnt-known-about-ipv6-link-local-address/
--
char *t="\10pse\0r\0dtu\0.@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4";
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i<l;i++){ i%8? c<<=1:
(c=*++x); c&128 && (s+=h); if (!(h>>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}
Re: Spam constantly being autolearned as ham
Posted by David B Funk <db...@engineering.uiowa.edu>.
On Wed, 23 Oct 2013, Karsten Bräckelmann wrote:
> On Wed, 2013-10-23 at 02:16 +0200, Benny Pedersen wrote:
>> Karsten Bräckelmann skrev den 2013-10-23 01:35:
>>
>>> And that last address range [fe80::%eth0]/64 on the first line is just
>>> weird -- what's supposed to substitute that ethernet interface
>>> placeholder?
>>
>> fe80::... is imho link local for ipv6,
> ^^^
> Invalid %thingy right there.
Actually that %thingy is the IPv6 zone_id (which functionally maps to the interface).
Given that the LLA is the same bit range regardless of interface, on a
multi-homed machine it is necessary to specify interface as well as address.
See:
RFC-4007 section 11
http://blogs.gentoo.org/eva/2010/12/17/things-you-didnt-known-about-ipv6-link-local-address/
--
Dave Funk University of Iowa
<dbfunk (at) engineering.uiowa.edu> College of Engineering
319/335-5751 FAX: 319/384-0549 1256 Seamans Center
Sys_admin/Postmaster/cell_admin Iowa City, IA 52242-1527
#include <std_disclaimer.h>
Better is not better, 'standard' is better. B{
Re: Spam constantly being autolearned as ham
Posted by Benny Pedersen <me...@junc.eu>.
Karsten Bräckelmann skrev den 2013-10-23 03:11:
>> fe80::... is imho link local for ipv6,
> ^^^
> Invalid %thingy right there.
was just another bad example not seen in ifconfig :)
>> in ipv4 192.168.0.0/16 :)
> IPv4 equivalent is 169.254/16.
yep, can i have more problems helping spamassassin team ?, now i mostly
have 8 cores to spare ?
how does others build the ham / spam corpus up in real life daily ?,
make a catch all domain and just wait ?
missing the old ninjas days, that was a time
Re: Spam constantly being autolearned as ham
Posted by Karsten Bräckelmann <gu...@rudersport.de>.
On Wed, 2013-10-23 at 02:16 +0200, Benny Pedersen wrote:
> Karsten Bräckelmann skrev den 2013-10-23 01:35:
>
> > And that last address range [fe80::%eth0]/64 on the first line is just
> > weird -- what's supposed to substitute that ethernet interface
> > placeholder?
>
> fe80::... is imho link local for ipv6,
^^^
Invalid %thingy right there.
> in ipv4 192.168.0.0/16 :)
IPv4 equivalent is 169.254/16.
--
char *t="\10pse\0r\0dtu\0.@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4";
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i<l;i++){ i%8? c<<=1:
(c=*++x); c&128 && (s+=h); if (!(h>>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}
Re: Spam constantly being autolearned as ham
Posted by Benny Pedersen <me...@junc.eu>.
Karsten Bräckelmann skrev den 2013-10-23 01:35:
> And that last address range [fe80::%eth0]/64 on the first line is just
> weird -- what's supposed to substitute that ethernet interface
> placeholder?
fe80::... is imho link local for ipv6, in ipv4 192.168.0.0/16 :)
why is it added anyway ?
Re: Spam constantly being autolearned as ham
Posted by Karsten Bräckelmann <gu...@rudersport.de>.
On Tue, 2013-10-22 at 15:59 -0700, Quanah Gibson-Mount wrote:
> > Quanah Gibson-Mount skrev den 2013-10-23 00:21:
> >
> > > Hm, actually, never mind. My trusted_networks has 10.0.0.0/8 which
> > > covers the IP address range these resolve to in their local DNS
> > > 10.113.208.x
> trusted_networks 127.0.0.0/8 10.0.0.0/8 [::1]/128 [fe80::%eth0]/64
^^^^^^^^^^^ ^^^^^^^^^
> 204.14.232.64/28 204.14.234.64/28 202.129.242.65/32 96.43.144.64/32
> 96.43.144.65/32 96.43.148.64/32 96.43.148.65/32 182.50.78.64/28
> 208.91.2.22/31
Excuse me for being blunt, but it appears you didn't lint check in quite
a while. That is absolutely borked.
$ spamassassin --lint --cf="trusted_networks 127.0.0.0/8"
warn: netset: cannot include 127.0.0.0/8 as it has already been included
M::SA::Conf docs, section Network Test Options, option trusted_networks
states: "Note: 127/8 and ::1 are always included in trusted_networks,
regardless of your config."
$ spamassassin --lint --cf="trusted_networks [::1]/128"
warn: netset: illegal network address given: '[::1]/128'
Included by default as well. And even bad syntax.
And that last address range [fe80::%eth0]/64 on the first line is just
weird -- what's supposed to substitute that ethernet interface
placeholder?
--
char *t="\10pse\0r\0dtu\0.@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4";
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i<l;i++){ i%8? c<<=1:
(c=*++x); c&128 && (s+=h); if (!(h>>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}
Re: Spam constantly being autolearned as ham
Posted by Quanah Gibson-Mount <qu...@zimbra.com>.
--On Wednesday, October 23, 2013 12:46 AM +0200 Benny Pedersen <me...@junc.eu>
wrote:
> Quanah Gibson-Mount skrev den 2013-10-23 00:21:
>
>> Hm, actually, never mind. My trusted_networks has 10.0.0.0/8 which
>> covers the IP address range these resolve to in their local DNS
>> 10.113.208.x
>>
>> I.e., if SA is acting off the hostname->IP mapping it gets from doing
>> a DNS lookup or from /etc/hosts, then trusted_networks already covers
>> the edge servers, so this shouldn't be an issue.
>
> trusted_networks have nothing to do with hostnames, see here for example
> localhost.junc.org :)
>
> you trust 127.0.0.1 right ?
Yes. ;)
trusted_networks 127.0.0.0/8 10.0.0.0/8 [::1]/128 [fe80::%eth0]/64
204.14.232.64/28 204.14.234.64/28 202.129.242.65/32 96.43.144.64/32
96.43.144.65/32 96.43.148.64/32 96.43.148.65/32 182.50.78.64/28
208.91.2.22/31
--Quanah
--
Quanah Gibson-Mount
Architect - Server
Zimbra, Inc.
--------------------
Zimbra :: the leader in open source messaging and collaboration
Re: Spam constantly being autolearned as ham
Posted by Benny Pedersen <me...@junc.eu>.
Quanah Gibson-Mount skrev den 2013-10-23 00:21:
> Hm, actually, never mind. My trusted_networks has 10.0.0.0/8 which
> covers the IP address range these resolve to in their local DNS
> 10.113.208.x
>
> I.e., if SA is acting off the hostname->IP mapping it gets from doing
> a DNS lookup or from /etc/hosts, then trusted_networks already covers
> the edge servers, so this shouldn't be an issue.
trusted_networks have nothing to do with hostnames, see here for example
localhost.junc.org :)
you trust 127.0.0.1 right ?
Re: Spam constantly being autolearned as ham
Posted by Quanah Gibson-Mount <qu...@zimbra.com>.
--On Tuesday, October 22, 2013 2:28 PM -0700 Quanah Gibson-Mount
<qu...@zimbra.com> wrote:
> --On Tuesday, October 22, 2013 11:09 PM +0200 Axb <ax...@gmail.com>
> wrote:
>
>> You've missed the point.
>>
>> mynetworks is not SA - it's Postfix and SA knows nothing about this
>> config option.
>>
>> as you have SA configured, RBL lookups are done against the vmware IPs
>> and I doubt those will be blacklisted, anywhere.
>>
>> If you add 208.91.0.0/22 to your SA trusted_networks (in local.cf)
>
> My SA already has trusted_networks configured as well, but you are right,
> this range is missing, thanks. We push the mta network bits out to all
> portions of the mta (postfix, amavis, SA, dspam). It looks like VMW made
> some IP address changes w/o notifying me. Sigh.
Hm, actually, never mind. My trusted_networks has 10.0.0.0/8 which covers
the IP address range these resolve to in their local DNS 10.113.208.x
I.e., if SA is acting off the hostname->IP mapping it gets from doing a DNS
lookup or from /etc/hosts, then trusted_networks already covers the edge
servers, so this shouldn't be an issue.
--Quanah
--
Quanah Gibson-Mount
Architect - Server
Zimbra, Inc.
--------------------
Zimbra :: the leader in open source messaging and collaboration
Re: Spam constantly being autolearned as ham
Posted by Quanah Gibson-Mount <qu...@zimbra.com>.
--On Tuesday, October 22, 2013 11:09 PM +0200 Axb <ax...@gmail.com>
wrote:
> You've missed the point.
>
> mynetworks is not SA - it's Postfix and SA knows nothing about this
> config option.
>
> as you have SA configured, RBL lookups are done against the vmware IPs
> and I doubt those will be blacklisted, anywhere.
>
> If you add 208.91.0.0/22 to your SA trusted_networks (in local.cf)
My SA already has trusted_networks configured as well, but you are right,
this range is missing, thanks. We push the mta network bits out to all
portions of the mta (postfix, amavis, SA, dspam). It looks like VMW made
some IP address changes w/o notifying me. Sigh.
--Quanah
--
Quanah Gibson-Mount
Architect - Server
Zimbra, Inc.
--------------------
Zimbra :: the leader in open source messaging and collaboration
Re: Spam constantly being autolearned as ham
Posted by Quanah Gibson-Mount <qu...@zimbra.com>.
--On Wednesday, October 23, 2013 10:57 PM +0200 Benny Pedersen <me...@junc.eu>
wrote:
> Quanah Gibson-Mount skrev den 2013-10-23 22:45:
>
>> Oct 23 09:05:53 edge01-zcs amavis[3250]: (03250-12) Checking:
>> 4vlUublDBL_R [162.213.112.166] <xx...@in.telligent.com> ->
>> <xx...@zimbra.com>
>> Oct 23 09:05:53 edge01-zcs amavis[3250]: (03250-12) Passed CLEAN
>> {RelayedInbound}, [162.213.112.166]:49611 [162.213.112.166]
>> <xx...@in.telligent.com> -> <xx...@zimbra.com>, Queue-ID: A39DD79F,
>> Message-ID: <DA...@DALOPS.corp.telligent.com>,
>> mail_id: 4vlUublDBL_R, Hits: -97.305, size: 7199, queued_as:
>> 7ACA71295, 484 ms
>
> where is uribl hits here ?
It's the only instance of "DBL" anywhere, is all. ;) No other hits for the
strings.
> is this mail gets -100 somewhere ?, too much whitelistning to not see the
> problem ?
in.telligent.com is our parent company, so yes, we whitelist anything they
send.
<http://blog.zimbra.com/blog/archives/2013/07/telligent-acquires-zimbra-from-vmware.html>
--Quanah
--
Quanah Gibson-Mount
Architect - Server
Zimbra, Inc.
--------------------
Zimbra :: the leader in open source messaging and collaboration
Re: Spam constantly being autolearned as ham
Posted by Benny Pedersen <me...@junc.eu>.
Quanah Gibson-Mount skrev den 2013-10-23 22:45:
> Oct 23 09:05:53 edge01-zcs amavis[3250]: (03250-12) Checking:
> 4vlUublDBL_R [162.213.112.166] <xx...@in.telligent.com> ->
> <xx...@zimbra.com>
> Oct 23 09:05:53 edge01-zcs amavis[3250]: (03250-12) Passed CLEAN
> {RelayedInbound}, [162.213.112.166]:49611 [162.213.112.166]
> <xx...@in.telligent.com> -> <xx...@zimbra.com>, Queue-ID: A39DD79F,
> Message-ID: <DA...@DALOPS.corp.telligent.com>,
> mail_id: 4vlUublDBL_R, Hits: -97.305, size: 7199, queued_as:
> 7ACA71295, 484 ms
where is uribl hits here ?
is this mail gets -100 somewhere ?, too much whitelistning to not see
the problem ?
Re: Spam constantly being autolearned as ham
Posted by Quanah Gibson-Mount <qu...@zimbra.com>.
--On Wednesday, October 23, 2013 10:34 PM +0200 Axb <ax...@gmail.com>
wrote:
> pls grep your logs for one of these: URIBL , SURBL , DBL (uppercase)
>
> Do you see any hits at all?
I see one:
Oct 23 09:05:53 edge01-zcs amavis[3250]: (03250-12) Checking: 4vlUublDBL_R
[162.213.112.166] <xx...@in.telligent.com> -> <xx...@zimbra.com>
Oct 23 09:05:53 edge01-zcs amavis[3250]: (03250-12) Passed CLEAN
{RelayedInbound}, [162.213.112.166]:49611 [162.213.112.166]
<xx...@in.telligent.com> -> <xx...@zimbra.com>, Queue-ID: A39DD79F, Message-ID:
<DA...@DALOPS.corp.telligent.com>, mail_id:
4vlUublDBL_R, Hits: -97.305, size: 7199, queued_as: 7ACA71295, 484 ms
> It is BCP to use a local resolver under your control for mail servers.
> Due to hammering public mirrors, an ISP/ASP's shared resolver may be
> tarpitted or blocked from doing queries to the BLs.
> If you run your own, you know when and what is happening and makes it
> easier to troubleshoot /monitor any potential issues.
Yeah, it's on my to-do list to add local dnscaching software to the Zimbra
product. ;)
--Quanah
--
Quanah Gibson-Mount
Architect - Server
Zimbra, Inc.
--------------------
Zimbra :: the leader in open source messaging and collaboration
Re: Spam constantly being autolearned as ham
Posted by Axb <ax...@gmail.com>.
On 10/23/2013 10:09 PM, Quanah Gibson-Mount wrote:
> --On Wednesday, October 23, 2013 10:06 PM +0200 Benny Pedersen
> <me...@junc.eu> wrote:
>
>>>> if you have own bind9 running on localhost
>>> bind9 is not installed on localhost.
>>
>> so resolv.conf is forwarding in wild ? :(
>
> resolve.conf uses VMWare's DNS servers which are not located on the MX
> servers.
>
pls grep your logs for one of these: URIBL , SURBL , DBL (uppercase)
Do you see any hits at all?
It is BCP to use a local resolver under your control for mail servers.
Due to hammering public mirrors, an ISP/ASP's shared resolver may be
tarpitted or blocked from doing queries to the BLs.
If you run your own, you know when and what is happening and makes it
easier to troubleshoot /monitor any potential issues.
Axb
Re: Spam constantly being autolearned as ham
Posted by Quanah Gibson-Mount <qu...@zimbra.com>.
--On Wednesday, October 23, 2013 10:06 PM +0200 Benny Pedersen <me...@junc.eu>
wrote:
>>> if you have own bind9 running on localhost
>> bind9 is not installed on localhost.
>
> so resolv.conf is forwarding in wild ? :(
resolve.conf uses VMWare's DNS servers which are not located on the MX
servers.
--Quanah
--
Quanah Gibson-Mount
Architect - Server
Zimbra, Inc.
--------------------
Zimbra :: the leader in open source messaging and collaboration
Re: Spam constantly being autolearned as ham
Posted by Benny Pedersen <me...@junc.eu>.
Quanah Gibson-Mount skrev den 2013-10-23 21:30:
> 50_scores.cf:score URIBL_DBL_SPAM 0 1.7 0 1.7
meta URIBL_DBL_SPAM (0.001) (0) (0.001) (0)
that will show if you get results or not if hits
but dont use it in production only on testing
>> if you have own bind9 running on localhost
> bind9 is not installed on localhost.
so resolv.conf is forwarding in wild ? :(
Re: Spam constantly being autolearned as ham
Posted by Quanah Gibson-Mount <qu...@zimbra.com>.
--On Wednesday, October 23, 2013 10:52 PM +0200 Benny Pedersen <me...@junc.eu>
wrote:
> Quanah Gibson-Mount skrev den 2013-10-23 22:09:
>
>> Ok, but the message body specifically has multiple links to
>> pumpery.com. So why didn't it get scored? That's what I don't
>> understand. ;)
>
> X-ASF-Spam-Status: No, hits=7.1 required=10.0
> tests=SPF_PASS,URIBL_BLACK,URIBL_DBL_SPAM,URIBL_JP_SURBL,URIBL_SC_SURBL
>
> error exists in localhost :=)
Right, but where is the error? ;) That's the whole question. ;)
In reading over <http://wiki.apache.org/spamassassin/DnsBlocklists> I came
across this statement:
A: Third, if your email gateway is behind a firewall make sure that
SpamAssassin is resolving the gateway to its external address. If
SpamAssassin resolves the gateway to an private IP or can't resolve the
name at all, it may mark the sending system as a trusted relay. As a
result, some or all of the spammer's systems will not be checked against
the DNSBL. (I'm not aware of anyway to specify 'last trusted relay' in SA).
and I wonder if that is the problem. The DNS that is used definitely
resolves the MX to its internal IP, and not its external IP.
--Quanah
--
Quanah Gibson-Mount
Architect - Server
Zimbra, Inc.
--------------------
Zimbra :: the leader in open source messaging and collaboration
Re: Spam constantly being autolearned as ham
Posted by Benny Pedersen <me...@junc.eu>.
Quanah Gibson-Mount skrev den 2013-10-23 22:09:
> Ok, but the message body specifically has multiple links to
> pumpery.com. So why didn't it get scored? That's what I don't
> understand. ;)
X-ASF-Spam-Status: No, hits=7.1 required=10.0
tests=SPF_PASS,URIBL_BLACK,URIBL_DBL_SPAM,URIBL_JP_SURBL,URIBL_SC_SURBL
error exists in localhost :=)
Re: Spam constantly being autolearned as ham
Posted by Benny Pedersen <me...@junc.eu>.
Quanah Gibson-Mount skrev den 2013-10-23 23:23:
> Oct 23 14:18:43.636 [24474] dbg: uridnsbl: domain "pumpery.com" listed
> (URIBL_BLOCKED): 127.0.0.1
this is not a url blacklistning, but a warning wmware dns is not paying
for dataservices
i wont say localhost one more time now, you get the point ? :)
Re: Spam constantly being autolearned as ham
Posted by John Hardin <jh...@impsec.org>.
On Wed, 23 Oct 2013, Quanah Gibson-Mount wrote:
> Still the spam score seems a bit low, I guess I may want to tweak the
> URIBL_DBL_SPAM and URIBL_BLOCKED scores.
URIBL_BLOCKED means the DNS server you're using to query the DNSBL has
exceeded the free access limits and is being blocked from getting actual
results.
You need to set up a DNS for mail only for URIBL lookups to work from your
current environment.
--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
The difference between ignorance and stupidity is that the stupid
desire to remain ignorant. -- Jim Bacon
-----------------------------------------------------------------------
510 days since the first successful private support mission to ISS (SpaceX)
Re: Spam constantly being autolearned as ham
Posted by Quanah Gibson-Mount <qu...@zimbra.com>.
--On Thursday, October 24, 2013 12:05 AM +0200 Benny Pedersen <me...@junc.eu>
wrote:
> Quanah Gibson-Mount skrev den 2013-10-23 23:51:
>> Is Amavis screwing with
>> things here, since SA is called via Amavis?
>
> if its is, try testing spampd so its showed its not that problem, running
> amavis and spampd nearly is equal to postfix setup, not much time to see
> if amavis is at fault for this
>
> note spampd is not spamd/spamc
I turned on debugging for SA at the amavis level, and I can see that
periodically RBL lookups do go through, but the majority of time, it looks
like VMW's dns servers are timing out our MX's. So apparently I need to go
talk to VMW for a bit (and deploy a local caching name server).
--Quanah
--
Quanah Gibson-Mount
Architect - Server
Zimbra, Inc.
--------------------
Zimbra :: the leader in open source messaging and collaboration
Re: Spam constantly being autolearned as ham
Posted by Benny Pedersen <me...@junc.eu>.
Quanah Gibson-Mount skrev den 2013-10-23 23:51:
> Is Amavis screwing with
> things here, since SA is called via Amavis?
if its is, try testing spampd so its showed its not that problem,
running amavis and spampd nearly is equal to postfix setup, not much
time to see if amavis is at fault for this
note spampd is not spamd/spamc
Re: Spam constantly being autolearned as ham
Posted by Benny Pedersen <me...@junc.eu>.
Quanah Gibson-Mount skrev den 2013-10-23 23:56:
> Yes, I see... Amavis turns off RBLs:
>
> $spamassasin_obj = Mail::SpamAssassin->new(
> { dont_copy_prefs => 1, local_tests_only => 1 } )
>
> That explains a lot. ;)
damm this happends only on precompiled problems distros like zimbra
i cant get this same result on freebsd and gentoo, whats my problem ?
:=)
Re: Spam constantly being autolearned as ham
Posted by Quanah Gibson-Mount <qu...@zimbra.com>.
--On Wednesday, October 23, 2013 2:56 PM -0700 Quanah Gibson-Mount
<qu...@zimbra.com> wrote:
>> Now, why don't I have "URIBL_BLOCKED" in *both*? It still seems to me
>> that URIBL lookups are not occurring when going through the MTA,
>> regardless of whether or not I'm blocked. Is Amavis screwing with things
>> here, since SA is called via Amavis?
>
> Yes, I see... Amavis turns off RBLs:
>
> $spamassasin_obj = Mail::SpamAssassin->new(
> { dont_copy_prefs => 1, local_tests_only => 1 } )
>
> That explains a lot. ;)
Or not... I have $sa_local_tests_only set to 0 in my amavisd.conf, so it
should be doing the URIBL tests.
--Quanah
--
Quanah Gibson-Mount
Architect - Server
Zimbra, Inc.
--------------------
Zimbra :: the leader in open source messaging and collaboration
Re: Spam constantly being autolearned as ham
Posted by Quanah Gibson-Mount <qu...@zimbra.com>.
--On Wednesday, October 23, 2013 2:51 PM -0700 Quanah Gibson-Mount
<qu...@zimbra.com> wrote:
> Now, why don't I have "URIBL_BLOCKED" in *both*? It still seems to me
> that URIBL lookups are not occurring when going through the MTA,
> regardless of whether or not I'm blocked. Is Amavis screwing with things
> here, since SA is called via Amavis?
Yes, I see... Amavis turns off RBLs:
$spamassasin_obj = Mail::SpamAssassin->new(
{ dont_copy_prefs => 1, local_tests_only => 1 } )
That explains a lot. ;)
--Quanah
--
Quanah Gibson-Mount
Architect - Server
Zimbra, Inc.
--------------------
Zimbra :: the leader in open source messaging and collaboration
Re: Spam constantly being autolearned as ham
Posted by Quanah Gibson-Mount <qu...@zimbra.com>.
--On Wednesday, October 23, 2013 11:32 PM +0200 Axb <ax...@gmail.com>
wrote:
> URIBL_BLOCKED is not good news .-)
> I wouldn't touch that score...
>
> http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block
Ok.
Here is something I don't understand -- Why I get utterly different values
from email that goes through the MTA, and the SA command line.
I *just* received an email, with the following scoring:
X-Spam-Flag: NO
X-Spam-Score: 2.717
X-Spam-Level: **
X-Spam-Status: No, score=2.717 tagged_above=-10 required=3
tests=[BAYES_50=0.8, HTML_EXTRA_CLOSE=0.001, HTML_MESSAGE=0.001,
RDNS_NONE=0.793, URI_HEX=1.122] autolearn=no
So I dumped it to a text file, and ran it through SA from the command line,
and I get:
X-Spam-Checker-Version: SpamAssassin 3.4.0-pre3-r1435395 (2013-01-18) on
edge02-zcs.vmware.com
X-Spam-Level: ***
X-Spam-Status: No, score=4.0 required=5.0
tests=RCVD_IN_MSPIKE_H2,RCVD_IN_PSBL,
RDNS_NONE,T_MIME_NO_TEXT,UNPARSEABLE_RELAY,URIBL_BLOCKED
autolearn=no
version=3.4.0-pre3-r1435395
Now, why don't I have "URIBL_BLOCKED" in *both*? It still seems to me that
URIBL lookups are not occurring when going through the MTA, regardless of
whether or not I'm blocked. Is Amavis screwing with things here, since SA
is called via Amavis?
--Quanah
--
Quanah Gibson-Mount
Architect - Server
Zimbra, Inc.
--------------------
Zimbra :: the leader in open source messaging and collaboration
Re: Spam constantly being autolearned as ham
Posted by John Hardin <jh...@impsec.org>.
On Wed, 23 Oct 2013, Axb wrote:
> On 10/23/2013 11:40 PM, John Hardin wrote:
>> On Wed, 23 Oct 2013, Axb wrote:
>>
>> > On 10/23/2013 11:23 PM, Quanah Gibson-Mount wrote:
>> > >
>> > > Still the spam score seems a bit low, I guess I may want to tweak the
>> > > URIBL_DBL_SPAM and URIBL_BLOCKED scores.
>> >
>> > URIBL_BLOCKED is not good news .-)
>> > I wouldn't touch that score...
>>
>> Maybe we need to rename that to URIBL_QUERY_BLOCKED for clarity?
>
> We have:
>
> describe URIBL_BLOCKED ADMINISTRATOR NOTICE: The query to URIBL was
> blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists\#dnsbl-block
> for more information.
>
> If ppl setup their reports not to show rule descriptions.... do we need to
> read them every description out loud?
[cynicism]Sometimes, yes.[/cynicism]
To be fair, few people include the hit descriptions for ham.
--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
Users mistake widespread adoption of Microsoft Office for the
development of a document format standard.
-----------------------------------------------------------------------
510 days since the first successful private support mission to ISS (SpaceX)
Re: Spam constantly being autolearned as ham
Posted by Benny Pedersen <me...@junc.eu>.
Axb skrev den 2013-10-23 23:47:
> If ppl setup their reports not to show rule descriptions.... do we
> need to read them every description out loud?
<ironical>
the more users that setup there dns servers to not forward, the more
datafeeds paying users is needed to handle the dns load
</ironical>
:)
Re: Spam constantly being autolearned as ham
Posted by Axb <ax...@gmail.com>.
On 10/23/2013 11:40 PM, John Hardin wrote:
> On Wed, 23 Oct 2013, Axb wrote:
>
>> On 10/23/2013 11:23 PM, Quanah Gibson-Mount wrote:
>>>
>>> Still the spam score seems a bit low, I guess I may want to tweak the
>>> URIBL_DBL_SPAM and URIBL_BLOCKED scores.
>>
>> URIBL_BLOCKED is not good news .-)
>> I wouldn't touch that score...
>
> Maybe we need to rename that to URIBL_QUERY_BLOCKED for clarity?
>
We have:
describe URIBL_BLOCKED ADMINISTRATOR NOTICE: The query to URIBL
was blocked. See
http://wiki.apache.org/spamassassin/DnsBlocklists\#dnsbl-block for more
information.
If ppl setup their reports not to show rule descriptions.... do we need
to read them every description out loud?
Re: Spam constantly being autolearned as ham
Posted by Benny Pedersen <me...@junc.eu>.
John Hardin skrev den 2013-10-23 23:40:
> Maybe we need to rename that to URIBL_QUERY_BLOCKED for clarity?
+1
Re: Spam constantly being autolearned as ham
Posted by John Hardin <jh...@impsec.org>.
On Wed, 23 Oct 2013, Axb wrote:
> On 10/23/2013 11:23 PM, Quanah Gibson-Mount wrote:
>>
>> Still the spam score seems a bit low, I guess I may want to tweak the
>> URIBL_DBL_SPAM and URIBL_BLOCKED scores.
>
> URIBL_BLOCKED is not good news .-)
> I wouldn't touch that score...
Maybe we need to rename that to URIBL_QUERY_BLOCKED for clarity?
--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
The difference between ignorance and stupidity is that the stupid
desire to remain ignorant. -- Jim Bacon
-----------------------------------------------------------------------
510 days since the first successful private support mission to ISS (SpaceX)
Re: Spam constantly being autolearned as ham
Posted by Axb <ax...@gmail.com>.
On 10/23/2013 11:23 PM, Quanah Gibson-Mount wrote:
> --On Wednesday, October 23, 2013 5:04 PM -0400 Kris Deugau
> <kd...@vianet.ca> wrote:
>
>> <g> Well, you didn't post the message body...
>>
>> *Usually* that indicates that the URI wasn't listed when the message was
>> originally processed, but checking again even 10-15 minutes later it is.
>> This is tricky to confirm unless you have enough access to the raw URI
>> lists to know when the URI was added.
>
> Ok, that makes sense. ;)
>
>> Post a complete example on pastebin - maybe there was something odd in
>> the message structure that caused the URIs to be skipped, but I can't
>> say I've ever seen one. SA goes to great lengths to mimic the idiocy
>> that many mail clients go to in picking URIs out of the message. Bad
>> grammar/typing with something like "... for dinner.It was ..." is enough
>> to cause "dinner.it" to get looked up, so it's much more likely the URI
>> simply wasn't listed when the message was first scanned.
>
> <http://ur1.ca/fxhkp>
>
>> Run the complete message through "spamassassin -D uridnsbl <message" -
>> you should get a line like:
>>
>> Oct 23 16:57:24.845 [12772] dbg: uridnsbl: domains to query:
>>
>> (hopefully with a list of URIs to actually query)
>
> Yeah, it definitely appears it is querying them correctly.
>
> The updated header even has:
>
> X-Spam-Checker-Version: SpamAssassin 3.4.0-pre3-r1435395 (2013-01-18) on
> edge02-zcs.vmware.com
> X-Spam-Level: **
> X-Spam-Status: No, score=2.3 required=5.0 tests=DKIM_SIGNED,
> HTML_IMAGE_RATIO_02,HTML_MESSAGE,RP_MATCHES_RCVD,T_DKIM_INVALID,
> T_HEADER_FROM_DIFFERENT_DOMAINS,UNPARSEABLE_RELAY,URIBL_BLOCKED,
> URIBL_DBL_SPAM autolearn=no version=3.4.0-pre3-r1435395
>
>
> Among the other bits, handy things like:
>
> Oct 23 14:18:43.636 [24474] dbg: uridnsbl: domain "pumpery.com" listed
> (URIBL_BLOCKED): 127.0.0.1
> Oct 23 14:18:43.638 [24474] dbg: uridnsbl: domain "pumpery.com" listed
> (URIBL_DBL_SPAM): 127.0.1.2
> Oct 23 14:18:43.739 [24474] dbg: uridnsbl: domain "nsports.com.br"
> listed (URIBL_BLOCKED): 127.0.0.1
>
> So I guess it wasn't listed at the time the message came in, as you noted.
>
> Still the spam score seems a bit low, I guess I may want to tweak the
> URIBL_DBL_SPAM and URIBL_BLOCKED scores.
URIBL_BLOCKED is not good news .-)
I wouldn't touch that score...
http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block
This why I suggested you run your own recursors....
Re: Spam constantly being autolearned as ham
Posted by Quanah Gibson-Mount <qu...@zimbra.com>.
--On Wednesday, October 23, 2013 5:04 PM -0400 Kris Deugau
<kd...@vianet.ca> wrote:
> <g> Well, you didn't post the message body...
>
> *Usually* that indicates that the URI wasn't listed when the message was
> originally processed, but checking again even 10-15 minutes later it is.
> This is tricky to confirm unless you have enough access to the raw URI
> lists to know when the URI was added.
Ok, that makes sense. ;)
> Post a complete example on pastebin - maybe there was something odd in
> the message structure that caused the URIs to be skipped, but I can't
> say I've ever seen one. SA goes to great lengths to mimic the idiocy
> that many mail clients go to in picking URIs out of the message. Bad
> grammar/typing with something like "... for dinner.It was ..." is enough
> to cause "dinner.it" to get looked up, so it's much more likely the URI
> simply wasn't listed when the message was first scanned.
<http://ur1.ca/fxhkp>
> Run the complete message through "spamassassin -D uridnsbl <message" -
> you should get a line like:
>
> Oct 23 16:57:24.845 [12772] dbg: uridnsbl: domains to query:
>
> (hopefully with a list of URIs to actually query)
Yeah, it definitely appears it is querying them correctly.
The updated header even has:
X-Spam-Checker-Version: SpamAssassin 3.4.0-pre3-r1435395 (2013-01-18) on
edge02-zcs.vmware.com
X-Spam-Level: **
X-Spam-Status: No, score=2.3 required=5.0 tests=DKIM_SIGNED,
HTML_IMAGE_RATIO_02,HTML_MESSAGE,RP_MATCHES_RCVD,T_DKIM_INVALID,
T_HEADER_FROM_DIFFERENT_DOMAINS,UNPARSEABLE_RELAY,URIBL_BLOCKED,
URIBL_DBL_SPAM autolearn=no version=3.4.0-pre3-r1435395
Among the other bits, handy things like:
Oct 23 14:18:43.636 [24474] dbg: uridnsbl: domain "pumpery.com" listed
(URIBL_BLOCKED): 127.0.0.1
Oct 23 14:18:43.638 [24474] dbg: uridnsbl: domain "pumpery.com" listed
(URIBL_DBL_SPAM): 127.0.1.2
Oct 23 14:18:43.739 [24474] dbg: uridnsbl: domain "nsports.com.br" listed
(URIBL_BLOCKED): 127.0.0.1
So I guess it wasn't listed at the time the message came in, as you noted.
Still the spam score seems a bit low, I guess I may want to tweak the
URIBL_DBL_SPAM and URIBL_BLOCKED scores.
--Quanah
--
Quanah Gibson-Mount
Architect - Server
Zimbra, Inc.
--------------------
Zimbra :: the leader in open source messaging and collaboration
Re: Spam constantly being autolearned as ham
Posted by Kris Deugau <kd...@vianet.ca>.
Quanah Gibson-Mount wrote:
> --On Wednesday, October 23, 2013 3:58 PM -0400 Kris Deugau
> <kd...@vianet.ca> wrote:
>
>> Only select headers have URIs extracted and passed to the DNS lookups;
>> I don't *think* Received: or Message-Id: are included. I've been
>> surprised now and then discovering a URI that *was* extracted from a
>> header. Otherwise all URI lookups are done on URIs found in the message
>> body.
>
> Ok, but the message body specifically has multiple links to pumpery.com.
> So why didn't it get scored? That's what I don't understand. ;)
<g> Well, you didn't post the message body...
*Usually* that indicates that the URI wasn't listed when the message was
originally processed, but checking again even 10-15 minutes later it is.
This is tricky to confirm unless you have enough access to the raw URI
lists to know when the URI was added.
Post a complete example on pastebin - maybe there was something odd in
the message structure that caused the URIs to be skipped, but I can't
say I've ever seen one. SA goes to great lengths to mimic the idiocy
that many mail clients go to in picking URIs out of the message. Bad
grammar/typing with something like "... for dinner.It was ..." is enough
to cause "dinner.it" to get looked up, so it's much more likely the URI
simply wasn't listed when the message was first scanned.
Run the complete message through "spamassassin -D uridnsbl <message" -
you should get a line like:
Oct 23 16:57:24.845 [12772] dbg: uridnsbl: domains to query:
(hopefully with a list of URIs to actually query)
-kgd
Re: Spam constantly being autolearned as ham
Posted by Quanah Gibson-Mount <qu...@zimbra.com>.
--On Wednesday, October 23, 2013 3:58 PM -0400 Kris Deugau
<kd...@vianet.ca> wrote:
> Only select headers have URIs extracted and passed to the DNS lookups;
> I don't *think* Received: or Message-Id: are included. I've been
> surprised now and then discovering a URI that *was* extracted from a
> header. Otherwise all URI lookups are done on URIs found in the message
> body.
Ok, but the message body specifically has multiple links to pumpery.com.
So why didn't it get scored? That's what I don't understand. ;)
--Quanah
--
Quanah Gibson-Mount
Architect - Server
Zimbra, Inc.
--------------------
Zimbra :: the leader in open source messaging and collaboration
Re: Spam constantly being autolearned as ham
Posted by Kris Deugau <kd...@vianet.ca>.
Quanah Gibson-Mount wrote:
> --On Wednesday, October 23, 2013 8:00 PM +0200 Benny Pedersen
> <me...@junc.eu> wrote:
>
>> Quanah Gibson-Mount skrev den 2013-10-23 19:37:
>>
>>>> pumpery.com listed on black.uribl.com
>>>> pumpery.com listed on jp.surbl.org
>>>> pumpery.com listed on sc.surbl.org
>>>> pumpery.com listed on dbl.spamhaus.org
>>
>> this is urlbl, nothing to do with trusted_networks
>
> Axb's point was that if trusted_networks is not configured correctly, SA
> will not do the URLBL checks correctly.
No, he had two sections to his reply that got mixed up with each other.
trusted_networks and the rest of the trust path settings don't affect
URI lookups, they only affect the IP lookups from the Received: headers.
Only select headers have URIs extracted and passed to the DNS lookups;
I don't *think* Received: or Message-Id: are included. I've been
surprised now and then discovering a URI that *was* extracted from a
header. Otherwise all URI lookups are done on URIs found in the message
body.
-kgd
Re: Spam constantly being autolearned as ham
Posted by Quanah Gibson-Mount <qu...@zimbra.com>.
--On Wednesday, October 23, 2013 8:00 PM +0200 Benny Pedersen <me...@junc.eu>
wrote:
> Quanah Gibson-Mount skrev den 2013-10-23 19:37:
>
>>> pumpery.com listed on black.uribl.com
>>> pumpery.com listed on jp.surbl.org
>>> pumpery.com listed on sc.surbl.org
>>> pumpery.com listed on dbl.spamhaus.org
>
> this is urlbl, nothing to do with trusted_networks
Axb's point was that if trusted_networks is not configured correctly, SA
will not do the URLBL checks correctly. I'm noting that trusted_networks
*is* configured correctly, and SA still does not appear to be doing the
checks correctly since emails with blacklisted web links are still flooding
my servers with spam.
I.e., there is no score anywhere from these blacklists being added into my
spam scores.
The module is loaded:
init.pre:loadplugin Mail::SpamAssassin::Plugin::URIDNSBL
So, for example, I believe I should have seen a score for URIBL_DBL_SPAM,
since the pumpery.com site is listed on dbl.spamhaus.org, and there were
multiple HTML links in the email for pumpery.com in the email.
50_scores.cf:score URIBL_DBL_SPAM 0 1.7 0 1.7
>> So, how do I determine why SA is failing to correctly query the RBLs?
>
> rndc querylog
>
> if you have own bind9 running on localhost
bind9 is not installed on localhost.
--Quanah
--
Quanah Gibson-Mount
Architect - Server
Zimbra, Inc.
--------------------
Zimbra :: the leader in open source messaging and collaboration
Re: Spam constantly being autolearned as ham
Posted by Benny Pedersen <me...@junc.eu>.
Quanah Gibson-Mount skrev den 2013-10-23 19:37:
>> pumpery.com listed on black.uribl.com
>> pumpery.com listed on jp.surbl.org
>> pumpery.com listed on sc.surbl.org
>> pumpery.com listed on dbl.spamhaus.org
this is urlbl, nothing to do with trusted_networks
> So, how do I determine why SA is failing to correctly query the RBLs?
rndc querylog
if you have own bind9 running on localhost
but dont mix problems with dnsbl and urlbl
with one is failing ?
Re: Spam constantly being autolearned as ham
Posted by Quanah Gibson-Mount <qu...@zimbra.com>.
--On Tuesday, October 22, 2013 11:09 PM +0200 Axb <ax...@gmail.com>
wrote:
> sent reply directly, sotrry - here's for the list
> On 10/22/2013 10:33 PM, Quanah Gibson-Mount wrote:
>> I don't get the concern about VMW. The vmw hosts are *my* MTAs and in
>> mynetworks.
>>
>> mail.zimbra.com -> load balanced name for edge01-zcs.vmware.com,
>> edge02-zcs.vmware.com
>>
>> The SPAM did not originate with my servers... It originated elsewhere.
>> This is rather clear:
>>
>> Received: from c115-smtp.pumpery.com (c115-smtp.pumpery.com
>> [5.135.12.243]) by edge02-zcs.vmware.com (Postfix) with ESMTP id 76999784
>> for <>; Tue, 22 Oct 2013 11:27:05 -0700 (PDT)
>>
>>
>> pumpery.com is the originator of this spam. I've blacklisted the from
>> in the meantime.
>
> If pumpery.com was in the msg's body, the URIBL plugin should have
> detected them
> yet another snowshoer on OVH (5.135.12.128/25)
> I hope, for your health, that you're going to blacklist every from in a
> missed spam
>
> pumpery.com listed on black.uribl.com
> pumpery.com listed on jp.surbl.org
> pumpery.com listed on sc.surbl.org
> pumpery.com listed on dbl.spamhaus.org
>
>
> You've missed the point.
>
> mynetworks is not SA - it's Postfix and SA knows nothing about this
> config option.
>
> as you have SA configured, RBL lookups are done against the vmware IPs
> and I doubt those will be blacklisted, anywhere.
So I've already confirmed this is *not* the case. My trusted_networks is
correct as configured -- Yet spam that should be blacklisted by the RBLs
continues to flow in.
So, how do I determine why SA is failing to correctly query the RBLs?
--Quanah
--
Quanah Gibson-Mount
Architect - Server
Zimbra, Inc.
--------------------
Zimbra :: the leader in open source messaging and collaboration
Re: Spam constantly being autolearned as ham
Posted by Axb <ax...@gmail.com>.
sent reply directly, sotrry - here's for the list
On 10/22/2013 10:33 PM, Quanah Gibson-Mount wrote:
> I don't get the concern about VMW. The vmw hosts are *my* MTAs and in
> mynetworks.
>
> mail.zimbra.com -> load balanced name for edge01-zcs.vmware.com,
> edge02-zcs.vmware.com
>
> The SPAM did not originate with my servers... It originated elsewhere.
> This is rather clear:
>
> Received: from c115-smtp.pumpery.com (c115-smtp.pumpery.com [5.135.12.243])
> by edge02-zcs.vmware.com (Postfix) with ESMTP id 76999784
> for <>; Tue, 22 Oct 2013 11:27:05 -0700 (PDT)
>
>
> pumpery.com is the originator of this spam. I've blacklisted the from
> in the meantime.
If pumpery.com was in the msg's body, the URIBL plugin should have
detected them
yet another snowshoer on OVH (5.135.12.128/25)
I hope, for your health, that you're going to blacklist every from in a
missed spam
pumpery.com listed on black.uribl.com
pumpery.com listed on jp.surbl.org
pumpery.com listed on sc.surbl.org
pumpery.com listed on dbl.spamhaus.org
You've missed the point.
mynetworks is not SA - it's Postfix and SA knows nothing about this
config option.
as you have SA configured, RBL lookups are done against the vmware IPs
and I doubt those will be blacklisted, anywhere.
If you add 208.91.0.0/22 to your SA trusted_networks (in local.cf)
SA will not lookup up those IPs but the ones before,
"http://spamassassin.apache.org/full/3.3.x/doc/Mail_SpamAssassin_Conf.txt"
NETWORK TEST OPTIONS
trusted_networks
internal_networks
This will increase detection accuracy
h2h
Re: Spam constantly being autolearned as ham
Posted by Quanah Gibson-Mount <qu...@zimbra.com>.
--On Tuesday, October 22, 2013 9:28 PM +0200 Axb <ax...@gmail.com>
wrote:
> On 10/22/2013 09:01 PM, Quanah Gibson-Mount wrote:
>> We have an issue where a lot of spam is being autolearned as HAM by SA.
>> Do people generally turn off autolearn?
>
> I only use autolearn - no drawbacks.
>
> assuming you are legitimately receiving this through vmware relays, add
> vmware's IPs to your trusted networks.
> That will help query BLs of IPs before the vmware hosts.
I don't get the concern about VMW. The vmw hosts are *my* MTAs and in
mynetworks.
mail.zimbra.com -> load balanced name for edge01-zcs.vmware.com,
edge02-zcs.vmware.com
The SPAM did not originate with my servers... It originated elsewhere.
This is rather clear:
Received: from c115-smtp.pumpery.com (c115-smtp.pumpery.com [5.135.12.243])
by edge02-zcs.vmware.com (Postfix) with ESMTP id 76999784
for <>; Tue, 22 Oct 2013 11:27:05 -0700 (PDT)
pumpery.com is the originator of this spam. I've blacklisted the from in
the meantime.
--Quanah
--
Quanah Gibson-Mount
Architect - Server
Zimbra, Inc.
--------------------
Zimbra :: the leader in open source messaging and collaboration
Re: Spam constantly being autolearned as ham
Posted by Axb <ax...@gmail.com>.
On 10/22/2013 09:01 PM, Quanah Gibson-Mount wrote:
> We have an issue where a lot of spam is being autolearned as HAM by SA.
> Do people generally turn off autolearn?
I only use autolearn - no drawbacks.
assuming you are legitimately receiving this through vmware relays, add
vmware's IPs to your trusted networks.
That will help query BLs of IPs before the vmware hosts.
h2h
Re: Spam constantly being autolearned as ham
Posted by Benny Pedersen <me...@junc.eu>.
Quanah Gibson-Mount skrev den 2013-10-22 21:21:
> header.d=superwebmais.com;
and it fails, not gething dkim_pass
that domain is not in your local.cf as whitelist_from_dkim .....
you still miss to tell me if you have autolearnthreshhold plugin
disabled or not
Re: Spam constantly being autolearned as ham
Posted by Quanah Gibson-Mount <qu...@zimbra.com>.
--On Tuesday, October 22, 2013 9:16 PM +0200 Benny Pedersen <me...@junc.eu>
wrote:
> Quanah Gibson-Mount skrev den 2013-10-22 21:11:
>
>> I'm not sure why you are talking about a mailing list?
>
> vmware sends dkim signed spams ?
>
> was it a bad example ?
I suggest re-reading the headers. The VMWare side was *validating* the
DKIM headers on the mail because the VMWare host is what is receiving the
email for delivery. The *spammer* DKIM signed their email.
header.d=superwebmais.com;
--Quanah
--
Quanah Gibson-Mount
Architect - Server
Zimbra, Inc.
--------------------
Zimbra :: the leader in open source messaging and collaboration
Re: Spam constantly being autolearned as ham
Posted by Benny Pedersen <me...@junc.eu>.
Quanah Gibson-Mount skrev den 2013-10-22 21:11:
> I'm not sure why you are talking about a mailing list?
vmware sends dkim signed spams ?
was it a bad example ?
Re: Spam constantly being autolearned as ham
Posted by Quanah Gibson-Mount <qu...@zimbra.com>.
--On Tuesday, October 22, 2013 9:09 PM +0200 Benny Pedersen <me...@junc.eu>
wrote:
> Quanah Gibson-Mount skrev den 2013-10-22 21:01:
>> We have an issue where a lot of spam is being autolearned as HAM by
>> SA. Do people generally turn off autolearn? In looking at these
>> cases, I'm not seeing where it is particularly helpful, but it is
>> particularly harmful.
>
> maillist is pr defination one thing all members wants, if thats not the
> case members would report spam to the owner of the maillist to resolve
> it, mostly its just disconnect to subscribed spamming user
I'm not sure why you are talking about a mailing list?
--Quanah
--
Quanah Gibson-Mount
Architect - Server
Zimbra, Inc.
--------------------
Zimbra :: the leader in open source messaging and collaboration
Re: Spam constantly being autolearned as ham
Posted by Benny Pedersen <me...@junc.eu>.
Quanah Gibson-Mount skrev den 2013-10-22 21:01:
> We have an issue where a lot of spam is being autolearned as HAM by
> SA. Do people generally turn off autolearn? In looking at these
> cases, I'm not seeing where it is particularly helpful, but it is
> particularly harmful.
maillist is pr defination one thing all members wants, if thats not the
case members would report spam to the owner of the maillist to resolve
it, mostly its just disconnect to subscribed spamming user
learning single users on maillist as spam needs more generic rules to
catch, i just like to know if you have disabled autolearnthreshol plugin
?
with adds a safety for not learning to much
Re: Spam constantly being autolearned as ham
Posted by Benny Pedersen <me...@junc.eu>.
Karsten Bräckelmann skrev den 2013-10-23 00:06:
>> and hope perl accept it, zero values cant be negative
> spamassassin --lint --cf="score RP_MATCHES_RCVD -0"
> Actually testing whether Perl / SA accepts it takes less time than your
> reply. ;)
my hp dl 585 gives 43000 bogomips, its more then my dell 1950 with 8
cores and ubuntu 12.04 lts :=)
just deleted a old citrix xenserver for this, license expired, so vms
could not be deleted, so much for opensources and rpm installs, worse
was it was 32bit xenserver on 64bit hardware, totaly waste resources
never mind its my problem now :)
Re: Spam constantly being autolearned as ham
Posted by Karsten Bräckelmann <gu...@rudersport.de>.
On Wed, 2013-10-23 at 00:02 +0200, Benny Pedersen wrote:
> Quanah Gibson-Mount skrev den 2013-10-22 23:43:
>
> > score RP_MATCHES_RCVD -0.8 -0.8 -0.8 -0.8
> >
> > so I'll update that to -0
>
> and hope perl accept it, zero values cant be negative
spamassassin --lint --cf="score RP_MATCHES_RCVD -0"
Actually testing whether Perl / SA accepts it takes less time than your
reply. ;)
--
char *t="\10pse\0r\0dtu\0.@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4";
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i<l;i++){ i%8? c<<=1:
(c=*++x); c&128 && (s+=h); if (!(h>>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}
Re: Spam constantly being autolearned as ham
Posted by Benny Pedersen <me...@junc.eu>.
Quanah Gibson-Mount skrev den 2013-10-22 23:43:
> score RP_MATCHES_RCVD -0.8 -0.8 -0.8 -0.8
>
> so I'll update that to -0
and hope perl accept it, zero values cant be negative
Re: Spam constantly being autolearned as ham
Posted by Karsten Bräckelmann <gu...@rudersport.de>.
On Tue, 2013-10-22 at 14:43 -0700, Quanah Gibson-Mount wrote:
> --On Tuesday, October 22, 2013 11:30 PM +0200 Karsten Bräckelmann wrote:
> > The usefulness of RP_MATCHES_RCVD is currently under discussion. I
> > suggest to zero out that rule, or assign it a negative zero.
>
> Ok, thanks. We'd already reduced its value recently after finding it
> mostly useless:
>
> score RP_MATCHES_RCVD -0.8 -0.8 -0.8 -0.8
>
> so I'll update that to -0
Interesting. Seems to be a valid score, passes linting.
I didn't mean "negative zero" literally, though, but a score of -0.001.
The default SA report shows 2 decimal places only -- a signed zero. The
amavis header includes the 3rd decimal place.
Unlike an actual zero score, this does not disable the rule altogether.
Needed e.g. in case of meta rules.
--
char *t="\10pse\0r\0dtu\0.@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4";
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i<l;i++){ i%8? c<<=1:
(c=*++x); c&128 && (s+=h); if (!(h>>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}
Re: Spam constantly being autolearned as ham
Posted by Quanah Gibson-Mount <qu...@zimbra.com>.
--On Tuesday, October 22, 2013 11:30 PM +0200 Karsten Bräckelmann
<gu...@rudersport.de> wrote:
> In other words: Non-Bayes ruleset scores may differ from the scores
> listed above. The score for BAYES_50 definitely needs to be subtracted.
> Which results in a negative score...
>
> The usefulness of RP_MATCHES_RCVD is currently under discussion. I
> suggest to zero out that rule, or assign it a negative zero.
Ok, thanks. We'd already reduced its value recently after finding it
mostly useless:
score RP_MATCHES_RCVD -0.8 -0.8 -0.8 -0.8
so I'll update that to -0
--Quanah
--
Quanah Gibson-Mount
Architect - Server
Zimbra, Inc.
--------------------
Zimbra :: the leader in open source messaging and collaboration
Re: Spam constantly being autolearned as ham
Posted by Karsten Bräckelmann <gu...@rudersport.de>.
On Tue, 2013-10-22 at 13:49 -0700, Quanah Gibson-Mount wrote:
> --On Tuesday, October 22, 2013 12:24 PM -0700 John Hardin
> <jh...@impsec.org> wrote:
>
> > On Tue, 22 Oct 2013, Quanah Gibson-Mount wrote:
> >> X-Spam-Status: No, score=0.348 tagged_above=-10 required=3
> >> tests=[BAYES_50=0.8, DKIM_SIGNED=0.1, DKIM_VALID=-0.1,
> >> DKIM_VALID_AU=-0.1, HTML_IMAGE_RATIO_02=0.437, HTML_MESSAGE=0.001,
> >> RP_MATCHES_RCVD=-0.8, T_HEADER_FROM_DIFFERENT_DOMAINS=0.01]
> >> autolearn=ham
> >
> > What are your thresholds set to? You might want to lower your ham
> > learning threshold and zero the RP_MATCHES_RCVD score.
> 10_default_prefs.cf:ifplugin Mail::SpamAssassin::Plugin::AutoLearnThreshold
> 10_default_prefs.cf:bayes_auto_learn_threshold_nonspam 0.1
> 10_default_prefs.cf:bayes_auto_learn_threshold_spam 12.0
>
> However, as I read the docs, the score is supposed to be lower for it to be
> autolearned. Last I checked, 0.348 > 0.1, so why was this autolearned as
> HAM if the cutoff is 0.1?
The Description section of the AutoLearnThreshold doc explains it:
Certain tests are ignored when determining whether a message should be
trained upon. Most notably that includes the BAYES_xx rules.
Moreover, auto-learning occurs using scores from either scoreset 0 or 1,
depending on what scoreset is used during message check. It is likely
that the message check and auto-learn scores will be different.
In other words: Non-Bayes ruleset scores may differ from the scores
listed above. The score for BAYES_50 definitely needs to be subtracted.
Which results in a negative score...
The usefulness of RP_MATCHES_RCVD is currently under discussion. I
suggest to zero out that rule, or assign it a negative zero.
--
char *t="\10pse\0r\0dtu\0.@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4";
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i<l;i++){ i%8? c<<=1:
(c=*++x); c&128 && (s+=h); if (!(h>>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}
Re: Spam constantly being autolearned as ham
Posted by Quanah Gibson-Mount <qu...@zimbra.com>.
--On Tuesday, October 22, 2013 12:24 PM -0700 John Hardin
<jh...@impsec.org> wrote:
> On Tue, 22 Oct 2013, Quanah Gibson-Mount wrote:
>
>> We have an issue where a lot of spam is being autolearned as HAM by SA.
>> Do people generally turn off autolearn? In looking at these cases, I'm
>> not seeing where it is particularly helpful, but it is particularly
>> harmful.
>>
>> Example:
>>
>> X-Spam-Status: No, score=0.348 tagged_above=-10 required=3
>> tests=[BAYES_50=0.8, DKIM_SIGNED=0.1, DKIM_VALID=-0.1,
>> DKIM_VALID_AU=-0.1, HTML_IMAGE_RATIO_02=0.437, HTML_MESSAGE=0.001,
>> RP_MATCHES_RCVD=-0.8, T_HEADER_FROM_DIFFERENT_DOMAINS=0.01]
>> autolearn=ham
>
> What are your thresholds set to? You might want to lower your ham
> learning threshold and zero the RP_MATCHES_RCVD score.
Thresholds are definitely enabled:
v310.pre:loadplugin Mail::SpamAssassin::Plugin::AutoLearnThreshold
And it looks like we use the defaults:
10_default_prefs.cf:ifplugin Mail::SpamAssassin::Plugin::AutoLearnThreshold
10_default_prefs.cf:bayes_auto_learn_threshold_nonspam 0.1
10_default_prefs.cf:bayes_auto_learn_threshold_spam 12.0
However, as I read the docs, the score is supposed to be lower for it to be
autolearned. Last I checked, 0.348 > 0.1, so why was this autolearned as
HAM if the cutoff is 0.1?
--Quanah
--
Quanah Gibson-Mount
Architect - Server
Zimbra, Inc.
--------------------
Zimbra :: the leader in open source messaging and collaboration
Re: Spam constantly being autolearned as ham
Posted by John Hardin <jh...@impsec.org>.
On Tue, 22 Oct 2013, Quanah Gibson-Mount wrote:
> We have an issue where a lot of spam is being autolearned as HAM by SA. Do
> people generally turn off autolearn? In looking at these cases, I'm not
> seeing where it is particularly helpful, but it is particularly harmful.
>
> Example:
>
> X-Spam-Status: No, score=0.348 tagged_above=-10 required=3
> tests=[BAYES_50=0.8, DKIM_SIGNED=0.1, DKIM_VALID=-0.1,
> DKIM_VALID_AU=-0.1, HTML_IMAGE_RATIO_02=0.437, HTML_MESSAGE=0.001,
> RP_MATCHES_RCVD=-0.8, T_HEADER_FROM_DIFFERENT_DOMAINS=0.01]
> autolearn=ham
What are your thresholds set to? You might want to lower your ham
learning threshold and zero the RP_MATCHES_RCVD score.
--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
Political Correctness is a doctrine which is based on the premise
that it is possible, through nothing more than a suitable choice
of words, to pick up a turd by the clean end.
-----------------------------------------------------------------------
509 days since the first successful private support mission to ISS (SpaceX)