You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by Ben Whyall <be...@whyall-systems.co.uk> on 2015/09/09 16:03:42 UTC

Problem with Bayes

Hi

I have a mail setup serving multiple domains that is running on ubuntu 
lts 14.04.03 lts.

Spamassassin 3.4.0
Perl 5.18.2
Exim 4.82

We have bayes stored in a mysql database

When spamassassin is run from exim, it is always returning a hit on rule 
Bayes_00 with a score of 0.0000, however if you run spamassassin < 
message, it will return a different bayes score.

in the /etc/defaults/spamassassin file we have the following

OPTIONS="-p 987 -s /var/log/spamd.log --nouser-config --helper-home-dir 
-H -u spamd"

the output of sa-learn -u spamd --dump magic is

0.000 0          3          0  non-token data: bayes db version
0.000          0      13794          0  non-token data: nspam
0.000          0       5619          0  non-token data: nham
0.000          0    1157620          0  non-token data: ntokens
0.000          0 1398925984          0  non-token data: oldest atime
0.000          0 1441801878          0  non-token data: newest atime
0.000          0          0          0  non-token data: last journal 
sync atime
0.000          0 1441797057          0  non-token data: last expiry atime
0.000          0    5529600          0  non-token data: last expire 
atime delta
0.000          0     783449          0  non-token data: last expire 
reduction count

Could anyone suggest how I might find out what is going awry ?

Ben

-- 
Sent from Postbox 
<https://www.postbox-inc.com/?utm_source=email&utm_medium=siglink&utm_campaign=reach>
-- 
Sent from Postbox 
<https://www.postbox-inc.com/?utm_source=email&utm_medium=siglink&utm_campaign=reach>

Re: Problem with Bayes

Posted by Reindl Harald <h....@thelounge.net>.

Am 09.09.2015 um 16:03 schrieb Ben Whyall:
> I have a mail setup serving multiple domains that is running on ubuntu
> lts 14.04.03 lts.
>
> Spamassassin 3.4.0
> Perl 5.18.2
> Exim 4.82
>
> We have bayes stored in a mysql database
>
> When spamassassin is run from exim, it is always returning a hit on rule
> Bayes_00 with a score of 0.0000, however if you run spamassassin <
> message, it will return a different bayes score.

sound like exim is running spamassassin as a different user and like 
using a different bayes

however, i observed that BAYES_xx for whatever reason *heavily* depends 
on Received-headers and so results are often differetn between spamc and 
running as milter for example

another example: in case there are no reveived headers at all bayes will 
never ever hit 999, put a purely random, valid received header on top 
and the same sample hits 999

observed that by stripa ton of headers out of our training-corpus with 
"formail" and rebuild bayes, hence now there is on single static 
Received-heaer on top of all samples and so they pass our masstests as 
expected


Re: Problem with Bayes

Posted by Ben Whyall <be...@whyall-systems.co.uk>.
Hi

Here is an explanation of how exim is talking to Spamassassin in my 
setup 
http://www.exim.org/exim-html-current/doc/html/spec_html/ch-content_scanning_at_acl_time.html

I suspect that as we have spam = nobody, SA was running the queries as 
nobody and as such not correctly using the right spam corpus.

Which the override statement is sorting.

Ben

> Matus UHLAR - fantomas <ma...@fantomas.sk>
> 09 September 2015 15:55via Postbox 
> <https://www.postbox-inc.com/?utm_source=email&utm_medium=sumlink&utm_campaign=reach>
>>> Ben Whyall <ma...@whyall-systems.co.uk>
>>> 09 September 2015 15:03via Postbox 
>>> <https://www.postbox-inc.com/?utm_source=email&utm_medium=sumlink&utm_campaign=reach> 
>>>
>
>>> When spamassassin is run from exim, it is always returning a hit on 
>>> rule Bayes_00 with a score of 0.0000, however if you run 
>>> spamassassin < message, it will return a different bayes score.
>
> that loooks like spamassassin called from exim uses different 
> username(s)...
>
>>> Axb <ma...@gmail.com>
>>> 09 September 2015 15:10via Postbox 
>>> <https://www.postbox-inc.com/?utm_source=email&utm_medium=sumlink&utm_campaign=reach> 
>>>
>>>
>>> No idea how Exim interfaces to SA but...
>
> ... so this is the most important question.
>
> On 09.09.15 15:21, Ben Whyall wrote:
>> I didnt have that option but I did have bayes_sql_username set
>>
>> I have added the sql_override as well.
>
> I don't think you should set such options without understsanding what 
> it may
> cause.
> it may work, but also may fsck up spam filtering.
>
>> I have restarted and will see what happens.
>>
>> I am using the ubuntu exim-daemon heavy and it calls spamd directly.
>
> that means how?
>
> Ben Whyall <ma...@whyall-systems.co.uk>
> 09 September 2015 15:21via Postbox 
> <https://www.postbox-inc.com/?utm_source=email&utm_medium=sumlink&utm_campaign=reach>
> Hi
>
> I didnt have that option but I did have bayes_sql_username set
>
> I have added the sql_override as well.
>
> I have restarted and will see what happens.
>
> I am using the ubuntu exim-daemon heavy and it calls spamd directly.
>
> Ben
>
>
> Ben Whyall <ma...@whyall-systems.co.uk>
> 09 September 2015 15:03via Postbox 
> <https://www.postbox-inc.com/?utm_source=email&utm_medium=sumlink&utm_campaign=reach>
> Hi
>
> I have a mail setup serving multiple domains that is running on ubuntu 
> lts 14.04.03 lts.
>
> Spamassassin 3.4.0
> Perl 5.18.2
> Exim 4.82
>
> We have bayes stored in a mysql database
>
> When spamassassin is run from exim, it is always returning a hit on 
> rule Bayes_00 with a score of 0.0000, however if you run spamassassin 
> < message, it will return a different bayes score.
>
> in the /etc/defaults/spamassassin file we have the following
>
> OPTIONS="-p 987 -s /var/log/spamd.log --nouser-config 
> --helper-home-dir -H -u spamd"
>
> the output of sa-learn -u spamd --dump magic is
>
> 0.000 0          3          0  non-token data: bayes db version
> 0.000          0      13794          0  non-token data: nspam
> 0.000          0       5619          0  non-token data: nham
> 0.000          0    1157620          0  non-token data: ntokens
> 0.000          0 1398925984          0  non-token data: oldest atime
> 0.000          0 1441801878          0  non-token data: newest atime
> 0.000          0          0          0  non-token data: last journal 
> sync atime
> 0.000          0 1441797057          0  non-token data: last expiry atime
> 0.000          0    5529600          0  non-token data: last expire 
> atime delta
> 0.000          0     783449          0  non-token data: last expire 
> reduction count
>
> Could anyone suggest how I might find out what is going awry ?
>
> Ben
>

-- 
Sent from Postbox 
<https://www.postbox-inc.com/?utm_source=email&utm_medium=siglink&utm_campaign=reach>

Re: Problem with Bayes

Posted by Matus UHLAR - fantomas <uh...@fantomas.sk>.
>On 09/09/2015 04:55 PM, Matus UHLAR - fantomas wrote:
>>On 09.09.15 15:21, Ben Whyall wrote:
>>>I didnt have that option but I did have bayes_sql_username set
>>>
>>>I have added the sql_override as well.
>>
>>I don't think you should set such options without understsanding what it
>>may
>>cause.
>>it may work, but also may fsck up spam filtering.

On 09.09.15 17:05, Axb wrote:
>Did I take it for granted that the op would check the docs
>(http://spamassassin.apache.org/full/3.4.x/doc/Mail_SpamAssassin_Conf.txt)
>before adding the option?

I would first search for answer to the question, how does calling SA from
exim work...

-- 
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
2B|!2B, that's a question!

Re: Problem with Bayes

Posted by Axb <ax...@gmail.com>.
On 09/09/2015 04:55 PM, Matus UHLAR - fantomas wrote:
> On 09.09.15 15:21, Ben Whyall wrote:
>> I didnt have that option but I did have bayes_sql_username set
>>
>> I have added the sql_override as well.
>
> I don't think you should set such options without understsanding what it
> may
> cause.
> it may work, but also may fsck up spam filtering.

Did I take it for granted that the op would check the docs 
(http://spamassassin.apache.org/full/3.4.x/doc/Mail_SpamAssassin_Conf.txt) 
before adding the option?

Re: Problem with Bayes

Posted by Matus UHLAR - fantomas <uh...@fantomas.sk>.
>>Ben Whyall <ma...@whyall-systems.co.uk>
>>09 September 2015 15:03via Postbox <https://www.postbox-inc.com/?utm_source=email&utm_medium=sumlink&utm_campaign=reach>

>>When spamassassin is run from exim, it is always returning a hit on 
>>rule Bayes_00 with a score of 0.0000, however if you run 
>>spamassassin < message, it will return a different bayes score.

that loooks like spamassassin called from exim uses different username(s)...

>>Axb <ma...@gmail.com>
>>09 September 2015 15:10via Postbox <https://www.postbox-inc.com/?utm_source=email&utm_medium=sumlink&utm_campaign=reach>
>>
>>No idea how Exim interfaces to SA but...

... so this is the most important question.

On 09.09.15 15:21, Ben Whyall wrote:
>I didnt have that option but I did have bayes_sql_username set
>
>I have added the sql_override as well.

I don't think you should set such options without understsanding what it may
cause.
it may work, but also may fsck up spam filtering.

>I have restarted and will see what happens.
>
>I am using the ubuntu exim-daemon heavy and it calls spamd directly.

that means how?

-- 
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Silvester Stallone: Father of the RISC concept.

Re: Problem with Bayes

Posted by Ben Whyall <be...@whyall-systems.co.uk>.
Hi

I didnt have that option but I did have bayes_sql_username set

I have added the sql_override as well.

I have restarted and will see what happens.

I am using the ubuntu exim-daemon heavy and it calls spamd directly.

Ben

> Axb <ma...@gmail.com>
> 09 September 2015 15:10via Postbox 
> <https://www.postbox-inc.com/?utm_source=email&utm_medium=sumlink&utm_campaign=reach>
>
>
> No idea how Exim interfaces to SA but...
>
> Do you have
> bayes_sql_override_username
> in local.cf ?
> if no, try adding that and restart spamd
>
> h2h
> Ben Whyall <ma...@whyall-systems.co.uk>
> 09 September 2015 15:03via Postbox 
> <https://www.postbox-inc.com/?utm_source=email&utm_medium=sumlink&utm_campaign=reach>
> Hi
>
> I have a mail setup serving multiple domains that is running on ubuntu 
> lts 14.04.03 lts.
>
> Spamassassin 3.4.0
> Perl 5.18.2
> Exim 4.82
>
> We have bayes stored in a mysql database
>
> When spamassassin is run from exim, it is always returning a hit on 
> rule Bayes_00 with a score of 0.0000, however if you run spamassassin 
> < message, it will return a different bayes score.
>
> in the /etc/defaults/spamassassin file we have the following
>
> OPTIONS="-p 987 -s /var/log/spamd.log --nouser-config 
> --helper-home-dir -H -u spamd"
>
> the output of sa-learn -u spamd --dump magic is
>
> 0.000 0          3          0  non-token data: bayes db version
> 0.000          0      13794          0  non-token data: nspam
> 0.000          0       5619          0  non-token data: nham
> 0.000          0    1157620          0  non-token data: ntokens
> 0.000          0 1398925984          0  non-token data: oldest atime
> 0.000          0 1441801878          0  non-token data: newest atime
> 0.000          0          0          0  non-token data: last journal 
> sync atime
> 0.000          0 1441797057          0  non-token data: last expiry atime
> 0.000          0    5529600          0  non-token data: last expire 
> atime delta
> 0.000          0     783449          0  non-token data: last expire 
> reduction count
>
> Could anyone suggest how I might find out what is going awry ?
>
> Ben
>

-- 
Sent from Postbox 
<https://www.postbox-inc.com/?utm_source=email&utm_medium=siglink&utm_campaign=reach>

Re: Problem with Bayes

Posted by Axb <ax...@gmail.com>.
On 09/09/2015 04:03 PM, Ben Whyall wrote:
> Hi
>
> I have a mail setup serving multiple domains that is running on ubuntu
> lts 14.04.03 lts.
>
> Spamassassin 3.4.0
> Perl 5.18.2
> Exim 4.82
>
> We have bayes stored in a mysql database
>
> When spamassassin is run from exim, it is always returning a hit on rule
> Bayes_00 with a score of 0.0000, however if you run spamassassin <
> message, it will return a different bayes score.
>
> in the /etc/defaults/spamassassin file we have the following
>
> OPTIONS="-p 987 -s /var/log/spamd.log --nouser-config --helper-home-dir
> -H -u spamd"
>
> the output of sa-learn -u spamd --dump magic is
>
> 0.000 0          3          0  non-token data: bayes db version
> 0.000          0      13794          0  non-token data: nspam
> 0.000          0       5619          0  non-token data: nham
> 0.000          0    1157620          0  non-token data: ntokens
> 0.000          0 1398925984          0  non-token data: oldest atime
> 0.000          0 1441801878          0  non-token data: newest atime
> 0.000          0          0          0  non-token data: last journal
> sync atime
> 0.000          0 1441797057          0  non-token data: last expiry atime
> 0.000          0    5529600          0  non-token data: last expire
> atime delta
> 0.000          0     783449          0  non-token data: last expire
> reduction count
>
> Could anyone suggest how I might find out what is going awry ?
>
> Ben
>

No idea how Exim interfaces to SA but...

Do you have
bayes_sql_override_username
in local.cf ?
if no, try adding that and restart spamd

h2h