You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by Peter Kelly <pe...@gmail.com> on 2015/09/11 12:12:14 UTC
Fwd: Large volume of 0.0 scores suddenly
Hi,
Starting on 3rd Sept, I have seen a huge number of 0.0 scores being
returned from spamassassin - see attached screenshot from my logs that show
I never once received a 0.0 score before 3rd Sept.
I use version 3.4.0 and process about 20k emails a day through it. I used
bayes and this has been regularly updated with 1000 ham and spam emails
(every months or so). Autolearning is on, at the default scores (0.1 and
12.0).
I have the cronjob enabled to update the rules nightly. Did anything change
on 3rd Sept that would explain this? Nothing has changed in my
configuration of spamassassin in months. I am now seeing a huge amount of
0.0 scores and TRUSTED_ALL rules. I have no trusted_networks set, never
have.
Any help greatly appreciated,
Peter
Re: Large volume of 0.0 scores suddenly
Posted by Peter Kelly <pe...@gmail.com>.
Sorry, yeah - all BAYES rules were gone after --clear. When I trained it
with another fresh 1000 spam and ham it started again with BAYES_00. I will
need to go through the spam and ham again
On 11 September 2015 at 23:26, RW <rw...@googlemail.com> wrote:
> On Fri, 11 Sep 2015 22:25:46 +0100
> Peter Kelly wrote:
>
>
> > I can actually see the 0.0 scores directly in the logs
>
>
> I tested one and it was out, but only by 0.04. I thought it was
> probably due to your cron job running a bit too early for this morning's
> update.
>
>
> > It must be the bayes is completely messed up. I actually did a
> > sa-learn --clear and ran a fresh 1000 spam and ham (verified) through
> > again, still seeing BAYES_00 an unusually high number of times.
>
> Are you sure you ran that on the right database? With an empty
> database you should see log entries without any BAYES_* rules.
>
> > I will look into my own DNS nameserver for the RELAY and TRUSTED
> > problem.
>
> ALL_TRUSTED doesn't have anything to do with DNS, it's determined from
> the received headers. spamassassin -D is useful for debugging it.
>
Re: Large volume of 0.0 scores suddenly
Posted by RW <rw...@googlemail.com>.
On Fri, 11 Sep 2015 22:25:46 +0100
Peter Kelly wrote:
> I can actually see the 0.0 scores directly in the logs
I tested one and it was out, but only by 0.04. I thought it was
probably due to your cron job running a bit too early for this morning's
update.
> It must be the bayes is completely messed up. I actually did a
> sa-learn --clear and ran a fresh 1000 spam and ham (verified) through
> again, still seeing BAYES_00 an unusually high number of times.
Are you sure you ran that on the right database? With an empty
database you should see log entries without any BAYES_* rules.
> I will look into my own DNS nameserver for the RELAY and TRUSTED
> problem.
ALL_TRUSTED doesn't have anything to do with DNS, it's determined from
the received headers. spamassassin -D is useful for debugging it.
Re: Large volume of 0.0 scores suddenly
Posted by Bill Cole <sa...@billmail.scconsult.com>.
On 11 Sep 2015, at 17:25, Peter Kelly wrote:
> Bill,
>
> I checked there first, I always assume it is something I am doing
> wrong
> first. Yes mailchecker (not that obsolete version) is the http service
> we
> use and it in turn uses this Golang lib for spamc -
> https://github.com/saintienn/go-spamc
>
> I can actually see the 0.0 scores directly in the logs at
> /var/log/mail.log
> on the spamassassin servers.
> E.g.
>
> Sep 11 21:07:19 ip-10-181-62-231 spamd[13929]: spamd: clean message
> (0.0/5.0) for (unknown):65534 in 0.4 seconds, 8128 bytes.
> Sep 11 21:07:19 ip-10-181-62-231 spamd[13929]: spamd: result: . 0 -
> BAYES_00,FREEMAIL_FROM,HTML_IMAGE_ONLY_32,HTML_MESSAGE,HTML_TAG_BALANCE_BODY,NO_RELAYS,SPF_NEUTRAL,T_DKIM_INVALID,URIBL_BLOCKED
Hmmm.... That reminds me: spamd only returns a score with a single
decimal place, so the fact that this one would add up to 0.039 doesn't
make the log line wrong in the way I thought your screenshot was wrong.
However, it DOES make any tool reporting greater precision (i.e. the
"0.00" values in your screenshot) misleading by design, so the problem
isn't (probably) misparsing but incorrect conversion of 0.0 to 0.00.
> It must be the bayes is completely messed up. I actually did a
> sa-learn
> --clear and ran a fresh 1000 spam and ham (verified) through again,
> still
> seeing BAYES_00 an unusually high number of times. I will look into my
> own
> DNS nameserver for the RELAY and TRUSTED problem.
That's not a DNS issue, it's a configuration issue and/or a data
preparation issue. Messages that match NO_RELAYS are either not properly
formatted, have had their headers mangled, or originated on the same
machine where they were delivered. ALL_TRUSTED can result from lesser
header damage and/or a mis-specified trusted_networks setting in
SpamAssassin.
Re: Large volume of 0.0 scores suddenly
Posted by Peter Kelly <pe...@gmail.com>.
Bill,
I checked there first, I always assume it is something I am doing wrong
first. Yes mailchecker (not that obsolete version) is the http service we
use and it in turn uses this Golang lib for spamc -
https://github.com/saintienn/go-spamc
I can actually see the 0.0 scores directly in the logs at /var/log/mail.log
on the spamassassin servers.
E.g.
Sep 11 21:07:19 ip-10-181-62-231 spamd[13929]: spamd: clean message
(0.0/5.0) for (unknown):65534 in 0.4 seconds, 8128 bytes.
Sep 11 21:07:19 ip-10-181-62-231 spamd[13929]: spamd: result: . 0 -
BAYES_00,FREEMAIL_FROM,HTML_IMAGE_ONLY_32,HTML_MESSAGE,HTML_TAG_BALANCE_BODY,NO_RELAYS,SPF_NEUTRAL,T_DKIM_INVALID,URIBL_BLOCKED
It must be the bayes is completely messed up. I actually did a sa-learn
--clear and ran a fresh 1000 spam and ham (verified) through again, still
seeing BAYES_00 an unusually high number of times. I will look into my own
DNS nameserver for the RELAY and TRUSTED problem.
On 11 September 2015 at 21:58, Bill Cole <
sausers-20150205@billmail.scconsult.com> wrote:
> On 11 Sep 2015, at 6:12, Peter Kelly wrote:
>
> Hi,
>>
>> Starting on 3rd Sept, I have seen a huge number of 0.0 scores being
>> returned from spamassassin - see attached screenshot from my logs that
>> show
>> I never once received a 0.0 score before 3rd Sept.
>>
>
> The default scores for the rules shown do not add up to 0.00 in the 2
> examples I did the arithmetic manually for.
>
> This indicates that whatever is generating those records (I'm guessing it
> is a version of this: https://github.com/peterkellyonline/mailchecker ?)
> is doing something wrong, I'd guess in parsing the spamc output.
>
Re: Large volume of 0.0 scores suddenly
Posted by Bill Cole <sa...@billmail.scconsult.com>.
On 11 Sep 2015, at 6:12, Peter Kelly wrote:
> Hi,
>
> Starting on 3rd Sept, I have seen a huge number of 0.0 scores being
> returned from spamassassin - see attached screenshot from my logs that
> show
> I never once received a 0.0 score before 3rd Sept.
The default scores for the rules shown do not add up to 0.00 in the 2
examples I did the arithmetic manually for.
This indicates that whatever is generating those records (I'm guessing
it is a version of this: https://github.com/peterkellyonline/mailchecker
?) is doing something wrong, I'd guess in parsing the spamc output.
Re: Fwd: Large volume of 0.0 scores suddenly
Posted by "Kevin A. McGrail" <KM...@PCCC.com>.
No, no changes. Run a manual check with -D and look for issues. Maybe your sql password changed or something that your install uses?
I would also look at the uribl blocked issue. Maybe that started on the 3rd for you? Perhaps your dns server is not working right and causing timeouts. See https://wiki.apache.org/spamassassin/DnsBlocklists under the first faq.
Regards,
KAM
On September 11, 2015 6:12:14 AM AST, Peter Kelly <pe...@gmail.com> wrote:
>Hi,
>
>Starting on 3rd Sept, I have seen a huge number of 0.0 scores being
>returned from spamassassin - see attached screenshot from my logs that
>show
>I never once received a 0.0 score before 3rd Sept.
>
>I use version 3.4.0 and process about 20k emails a day through it. I
>used
>bayes and this has been regularly updated with 1000 ham and spam emails
>(every months or so). Autolearning is on, at the default scores (0.1
>and
>12.0).
>
>I have the cronjob enabled to update the rules nightly. Did anything
>change
>on 3rd Sept that would explain this? Nothing has changed in my
>configuration of spamassassin in months. I am now seeing a huge amount
>of
>0.0 scores and TRUSTED_ALL rules. I have no trusted_networks set, never
>have.
>
>Any help greatly appreciated,
>
>Peter
Re: Large volume of 0.0 scores suddenly
Posted by Benny Pedersen <me...@junc.eu>.
Reindl Harald skrev den 2015-09-11 15:08:
> in other words: you don't need to pay for them until you not have your
> own recursion resolver because URIBL_BLOCKED won't go away in that
> case
its just badly marketing :=)
Re: Large volume of 0.0 scores suddenly
Posted by Reindl Harald <h....@thelounge.net>.
Am 11.09.2015 um 15:03 schrieb Peter Kelly:
> Why Antony? What would that do for me other than save hits against
> URIBL? I am signing up for their paid service so I will not have the
> URIBL_BLOCKED issue anymore. It does not explain the 0.0 issue I am
> having anyway.
what is so hard to understand in the fact that when you use a DNS
forwarder shared with other people any DNSBL/URIBL sees always the same
IP (not yours) and so the summary *of all users* using that DNS server
in other words: you don't need to pay for them until you not have your
own recursion resolver because URIBL_BLOCKED won't go away in that case
Re: Fwd: Large volume of 0.0 scores suddenly
Posted by Peter Kelly <pe...@gmail.com>.
Why Antony? What would that do for me other than save hits against URIBL? I
am signing up for their paid service so I will not have the URIBL_BLOCKED
issue anymore. It does not explain the 0.0 issue I am having anyway.
On 11 September 2015 at 13:42, Antony Stone <
Antony.Stone@spamassassin.open.source.it> wrote:
> > On 09/11/2015 01:17 PM, Peter Kelly wrote:
>
> > > - Are you using a local, non forwarding, DNS resolver/caching server ?
> > >
> > > No
>
> > > - Are you handling mail for a company, personal email, ISP, one domain,
> > > many domains, etc?
> > >
> > > Handling mail for thousands of different companies - we run a SaaS
> > > Helpdesk system like Zendesk. We see a huge range of emails and
> > > domains.
>
> Without a local DNS server I'm amazed you haven't had problems before now.
>
>
> Antony.
>
> --
> I want to build a machine that will be proud of me.
>
> - Danny Hillis, creator of The Connection Machine
>
> Please reply to the
> list;
> please *don't* CC
> me.
>
Re: Fwd: Large volume of 0.0 scores suddenly
Posted by Antony Stone <An...@spamassassin.open.source.it>.
> On 09/11/2015 01:17 PM, Peter Kelly wrote:
> > - Are you using a local, non forwarding, DNS resolver/caching server ?
> >
> > No
> > - Are you handling mail for a company, personal email, ISP, one domain,
> > many domains, etc?
> >
> > Handling mail for thousands of different companies - we run a SaaS
> > Helpdesk system like Zendesk. We see a huge range of emails and
> > domains.
Without a local DNS server I'm amazed you haven't had problems before now.
Antony.
--
I want to build a machine that will be proud of me.
- Danny Hillis, creator of The Connection Machine
Please reply to the list;
please *don't* CC me.
Re: Fwd: Large volume of 0.0 scores suddenly
Posted by Axb <ax...@gmail.com>.
please keep list mail on list...
On 09/11/2015 01:17 PM, Peter Kelly wrote:
> - Please post missed spam samples in pastebin.com - do not post
> samples to mailing lists
> I'll post example shortly
>
> - What SA version are you using? and on what operating system?
>
> 3.4.0 on Ubuntu 14.04
>
> - How are you using SA?
> (pls specify: amavis, MIMEDefang, a milter, Mailscanner, procmail,
> Fuglu, etc, etc)
>
> Just spamassassin on its own, calling the daemon from an app
>
> - Are you using SA in a PC/notebook? or on a server?
>
> Server
>
> - What plugins are you using?
> (pls specify: Razor, Pyzor, DCC, etc)
>
> Razor, Pyzor
>
> - Are you using RBLs?
> (specify: at SMTP level, only SA's lookups, etc)
>
> SA lookups
>
> - Are you using any additional rulesets?
>
> No
>
> - Are you using a local, non forwarding, DNS resolver/caching server ?
>
> No
>
> - Are you using per/user or site wide Bayes?
>
> site-wide
>
> - What Bayes backend are you using?
> (specify: default, SDBM, SQL, Redis, other)
>
> default, file
>
> - Are you handling mail for a company, personal email, ISP, one domain, many
> domains, etc?
>
> Handling mail for thousands of different companies - we run a SaaS
> Helpdesk system like Zendesk. We see a huge range of emails and
> domains.
>
>
> On 11 September 2015 at 11:22, Axb <ax...@gmail.com> wrote:
>
>> On 09/11/2015 12:12 PM, Peter Kelly wrote:
>>
>>> Hi,
>>>
>>> Starting on 3rd Sept, I have seen a huge number of 0.0 scores being
>>> returned from spamassassin - see attached screenshot from my logs that
>>> show
>>> I never once received a 0.0 score before 3rd Sept.
>>>
>>> I use version 3.4.0 and process about 20k emails a day through it. I used
>>> bayes and this has been regularly updated with 1000 ham and spam emails
>>> (every months or so). Autolearning is on, at the default scores (0.1 and
>>> 12.0).
>>>
>>> I have the cronjob enabled to update the rules nightly. Did anything
>>> change
>>> on 3rd Sept that would explain this? Nothing has changed in my
>>> configuration of spamassassin in months. I am now seeing a huge amount of
>>> 0.0 scores and TRUSTED_ALL rules. I have no trusted_networks set, never
>>> have.
>>>
>>>
>> You're not giving us much information to help you...
>>
>> pls see:
>>
>> https://svn.apache.org/repos/asf/spamassassin/trunk/rulesrc/sandbox/emailed/sa-list-template.txt
>>
>> and try to provide us with as much info as possible.
>>
>> iow, pls help *us* help *you*
>>
>
Re: Fwd: Large volume of 0.0 scores suddenly
Posted by Matus UHLAR - fantomas <uh...@fantomas.sk>.
On 11.09.15 14:13, Peter Kelly wrote:
>We have a SaaS app hosted in AWS that takes in 500k emails a month. We
>parse these emails and convert them into tickets for the customer - they
>see a Helpdesk system like Zendesk. Every incoming email gets run through
>spamassassin via the daemon.
does spamassassin see the whole e-mails, including all headers?
the NO_RELAYS test looks like you are not pushing the whole mails to SA,
which significantly decreases score.
Also, do you have way to train your spamassassin with spam? Since all spams
have BAYES_00, you should train them...
--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Your mouse has moved. Windows NT will now restart for changes to take
to take effect. [OK]
Re: Fwd: Large volume of 0.0 scores suddenly
Posted by Axb <ax...@gmail.com>.
On 09/11/2015 03:13 PM, Peter Kelly wrote:
> Axb,
>
> We have a SaaS app hosted in AWS that takes in 500k emails a month. We
> parse these emails and convert them into tickets for the customer - they
> see a Helpdesk system like Zendesk. Every incoming email gets run through
> spamassassin via the daemon.
>
> Here is a link to the output of --lint -D http://pastebin.com/8eM88hX2
is the app feeding spamd directly or are you using spamc ? or using the
API interface?
can you get hold of one of those pristine messages and test them
manually against spamassasssin ?
if the results look massively different, chances is that your app is not
doing the right thing and like Matus suspects, SA is not getting the
right thing.
Rules & scores do change via sa-update so depending on lots of stuff the
results may vary, possibly quite a lot.
As we don't have a sample msg of yours (pastebin) we can't compare with
any other setups...
ball over...
>
> On 11 September 2015 at 13:08, Axb <ax...@gmail.com> wrote:
>
>> On 09/11/2015 01:17 PM, Peter Kelly wrote:
>>
>>> - How are you using SA?
>>> (pls specify: amavis, MIMEDefang, a milter, Mailscanner, procmail,
>>> Fuglu, etc, etc)
>>>
>>> Just spamassassin on its own, calling the daemon from an app
>>>
>>
>> an "app"? Pls be more explicit.
>>
>> can you pastebin the output of
>>
>> spamassassin --lint -D
>>
>>
>> - Are you using a local, non forwarding, DNS resolver/caching server ?
>>>
>>> No
>>>
>>
>> you should, to avoid URIBL_BLOCKED
>> (http://uribl.com/refused.shtml)
>>
>>
>>
>>
>
Re: Fwd: Large volume of 0.0 scores suddenly
Posted by Axb <ax...@gmail.com>.
On 09/11/2015 03:13 PM, Peter Kelly wrote:
> Axb,
>
> We have a SaaS app hosted in AWS that takes in 500k emails a month. We
> parse these emails and convert them into tickets for the customer - they
> see a Helpdesk system like Zendesk. Every incoming email gets run through
> spamassassin via the daemon.
>
> Here is a link to the output of --lint -Dhttp://pastebin.com/8eM88hX2
btw:
Sep 11 13:14:04.305 [2812] dbg: diag: [...] module not installed:
Encode::Detect ('require' failed)
Sep 11 13:14:04.305 [2812] dbg: diag: [...] module not installed:
Digest::SHA1 ('require' failed)
etc...
suggest you install the "required" modules and see if spamassassin
--lint -D detects them.
missing modules may "disable" features...
Re: Fwd: Large volume of 0.0 scores suddenly
Posted by Peter Kelly <pe...@gmail.com>.
Axb,
We have a SaaS app hosted in AWS that takes in 500k emails a month. We
parse these emails and convert them into tickets for the customer - they
see a Helpdesk system like Zendesk. Every incoming email gets run through
spamassassin via the daemon.
Here is a link to the output of --lint -D http://pastebin.com/8eM88hX2
On 11 September 2015 at 13:08, Axb <ax...@gmail.com> wrote:
> On 09/11/2015 01:17 PM, Peter Kelly wrote:
>
>> - How are you using SA?
>> (pls specify: amavis, MIMEDefang, a milter, Mailscanner, procmail,
>> Fuglu, etc, etc)
>>
>> Just spamassassin on its own, calling the daemon from an app
>>
>
> an "app"? Pls be more explicit.
>
> can you pastebin the output of
>
> spamassassin --lint -D
>
>
> - Are you using a local, non forwarding, DNS resolver/caching server ?
>>
>> No
>>
>
> you should, to avoid URIBL_BLOCKED
> (http://uribl.com/refused.shtml)
>
>
>
>
Re: Fwd: Large volume of 0.0 scores suddenly
Posted by Axb <ax...@gmail.com>.
On 09/11/2015 01:17 PM, Peter Kelly wrote:
> - How are you using SA?
> (pls specify: amavis, MIMEDefang, a milter, Mailscanner, procmail,
> Fuglu, etc, etc)
>
> Just spamassassin on its own, calling the daemon from an app
an "app"? Pls be more explicit.
can you pastebin the output of
spamassassin --lint -D
> - Are you using a local, non forwarding, DNS resolver/caching server ?
>
> No
you should, to avoid URIBL_BLOCKED
(http://uribl.com/refused.shtml)
Re: Fwd: Large volume of 0.0 scores suddenly
Posted by Axb <ax...@gmail.com>.
On 09/11/2015 12:12 PM, Peter Kelly wrote:
> Hi,
>
> Starting on 3rd Sept, I have seen a huge number of 0.0 scores being
> returned from spamassassin - see attached screenshot from my logs that show
> I never once received a 0.0 score before 3rd Sept.
>
> I use version 3.4.0 and process about 20k emails a day through it. I used
> bayes and this has been regularly updated with 1000 ham and spam emails
> (every months or so). Autolearning is on, at the default scores (0.1 and
> 12.0).
>
> I have the cronjob enabled to update the rules nightly. Did anything change
> on 3rd Sept that would explain this? Nothing has changed in my
> configuration of spamassassin in months. I am now seeing a huge amount of
> 0.0 scores and TRUSTED_ALL rules. I have no trusted_networks set, never
> have.
>
You're not giving us much information to help you...
pls see:
https://svn.apache.org/repos/asf/spamassassin/trunk/rulesrc/sandbox/emailed/sa-list-template.txt
and try to provide us with as much info as possible.
iow, pls help *us* help *you*
Re: Fwd: Large volume of 0.0 scores suddenly
Posted by Benny Pedersen <me...@junc.eu>.
Peter Kelly skrev den 2015-09-11 15:01:
> This has nothing to do with URIBL. It has always been blocked for me.
> I am in the process of paying for their service. It has always been
> like that, yet the 0.0 scores only started last week. Been running for
> months before that.
so you already have a local dns server, and is now recieving so much
spam that you need to pay uribl for dataservice ?
wish it was me :=)
but my point is that a missing local dns could olso show other problems
Re: Fwd: Large volume of 0.0 scores suddenly
Posted by Peter Kelly <pe...@gmail.com>.
Hi Benny,
This has nothing to do with URIBL. It has always been blocked for me. I am
in the process of paying for their service. It has always been like that,
yet the 0.0 scores only started last week. Been running for months before
that.
Peter
On 11 September 2015 at 13:38, Benny Pedersen <me...@junc.eu> wrote:
> Peter Kelly skrev den 2015-09-11 12:12:
>
> Any help greatly appreciated,
>>
>
> google URIBL_BLOCKED
>
> https://www.google.dk/search?q=uribl_blcoked
>
> http://uribl.com/refused.shtml
>
> plenty of other links to see how and why
>
> do you miss a local dns resolver ?
>
> if yes you use shared problems and things like your questions come up
> randomly when more people dont read about it, its free to do nothing :=)
>
Re: Fwd: Large volume of 0.0 scores suddenly
Posted by Benny Pedersen <me...@junc.eu>.
Peter Kelly skrev den 2015-09-11 12:12:
> Any help greatly appreciated,
google URIBL_BLOCKED
https://www.google.dk/search?q=uribl_blcoked
http://uribl.com/refused.shtml
plenty of other links to see how and why
do you miss a local dns resolver ?
if yes you use shared problems and things like your questions come up
randomly when more people dont read about it, its free to do nothing :=)