sa-learn incapable of handling large amounts of messages?

[Caleb Cushing <xe...@gmail.com>]

I'm trying to run sa-learn --spam /spamdir/* on a directory with 2449 spam 
messages. But it doesn't seem to work. I'm not sure why.
-- 
Caleb Cushing

my blog http://xenoterracide.blogspot.com

Re: sa-learn incapable of handling large amounts of messages?

[D Hill <d....@yournetplus.com>]

On Tue, 15 Apr 2008 at 21:53 -0400, xenoterracide@gmail.com confabulated:

> I'm trying to run sa-learn --spam /spamdir/* on a directory with 2449 spam
> messages. But it doesn't seem to work. I'm not sure why.

What format are the messages in? mdir? mbox?

   sa-learn --spam --username=d.hill@example.com spam/

works from here as an mdir mailbox, and:

   sa-learn --spam --mbox --username=d.hill@example.com spam

works from here as an mbox format.

I have never had issues with learning messages using SA v3.2.4.

-d

Re: sa-learn incapable of handling large amounts of messages?

[Matt Kettler <mk...@verizon.net>]

Caleb Cushing wrote:
> ah... resolved. I forgot that these message hadn't been checked by spamd (and 
> I didn't realize they needed to) after running the whole directory through 
> spamassassin it works as expected.
>
> Learned tokens from 1741 message(s) (2469 message(s) examined)
>
> :D
>   
That shouldn't matter.. you should be able to take a completely new 
message, never touched by spamassassin or spamd, and feed it to sa-learn.


My guess is running it through spamassassin caused some weird corruption 
of the headers to be fixed by SA's parsers...

Re: sa-learn incapable of handling large amounts of messages?

[Caleb Cushing <xe...@gmail.com>]

ah... resolved. I forgot that these message hadn't been checked by spamd (and 
I didn't realize they needed to) after running the whole directory through 
spamassassin it works as expected.

Learned tokens from 1741 message(s) (2469 message(s) examined)

:D
-- 
Caleb Cushing

my blog http://xenoterracide.blogspot.com

Re: sa-learn incapable of handling large amounts of messages?

[Matt Kettler <mk...@verizon.net>]

Caleb Cushing wrote:
> I'm trying to run sa-learn --spam /spamdir/* on a directory with 2449 spam 
> messages. But it doesn't seem to work. I'm not sure why.
>   
That quantity should be no problem at all. The standard bayes benchmarks 
were based on 2000 message buckets.

http://wiki.apache.org/spamassassin/BayesBenchmarkResults

However, I would suggest ditching the * in that command line. Let 
spamassassin handle parsing all the files in the directory, rather than 
having your shell expand them and pass a gigantic command-line parameter 
to sa-learn.

(In the *nix world, *'s are expanded by the shell, prior to calling the 
application, with large directories this can result in a rather ugly 
large command line that is slow to process)

ie:
 sa-learn --spam /spamdir/

If it still isn't working, try it with debugging enabled and see where 
sa-learn is choking..

sa-learn -D --spam /spamdir/

Re: sa-learn incapable of handling large amounts of messages?

[Matt Kettler <mk...@verizon.net>]

Caleb Cushing wrote:
> On Tuesday 15 April 2008 11:27:55 pm Matt Kettler wrote:
>   
>> Really I'd just be wanting to see the start and end of it..
>>     
> head
> [7750] dbg: logger: adding facilities: all
> [7750] dbg: logger: logging level is DBG
> [7750] dbg: generic: SpamAssassin version 3.2.4
> [7750] dbg: config: score set 0 chosen.
> [7750] dbg: util: running in taint mode? no
> [7750] dbg: dns: no ipv6
> [7750] dbg: dns: is Net::DNS::Resolver available? yes
> [7750] dbg: dns: Net::DNS version: 0.61
> [7750] dbg: config: using "/etc/mail/spamassassin" for site rules pre files
> [7750] dbg: config: read file /etc/mail/spamassassin/init.pre
>
> tail
> [7750] dbg: bayes: tie-ing to DB file 
> R/W /home/xenoterracide/.spamassassin/bayes_toks
> [7750] dbg: bayes: tie-ing to DB file 
> R/W /home/xenoterracide/.spamassassin/bayes_seen
> [7750] dbg: bayes: found bayes db version 3
> [7750] dbg: locker: refresh_lock: 
> refresh /home/xenoterracide/.spamassassin/bayes.lock
> [7750] dbg: bayes: DB expiry: tokens in DB: 61967, Expiry max size: 150000, 
> Oldest atime: 1102006137, Newest atime: 1208252263, Last expire: 0, Current 
> time: 1208310434
> [7750] dbg: bayes: expiry completed
> [7750] dbg: bayes: untie-ing
> [7750] dbg: bayes: files locked, now unlocking lock
> [7750] dbg: locker: safe_unlock: 
> unlink /home/xenoterracide/.spamassassin/bayes.lock
> Learned tokens from 0 message(s) (0 message(s) examined)
>
>   
>> That said, what exactly do you mean in your original post by "doesn't
>> seem to work" ?
>>     
>
> by seems I mean doesn't. I'm assuming its something I'm doing wrong.
>   

Interesting.. that would tell me that none of the files looked like 
emails to spamassassin... Are you sure they're in ordinary format, with 
headers, etc?

Re: sa-learn incapable of handling large amounts of messages?

[Caleb Cushing <xe...@gmail.com>]

On Tuesday 15 April 2008 11:27:55 pm Matt Kettler wrote:
> Really I'd just be wanting to see the start and end of it..
head
[7750] dbg: logger: adding facilities: all
[7750] dbg: logger: logging level is DBG
[7750] dbg: generic: SpamAssassin version 3.2.4
[7750] dbg: config: score set 0 chosen.
[7750] dbg: util: running in taint mode? no
[7750] dbg: dns: no ipv6
[7750] dbg: dns: is Net::DNS::Resolver available? yes
[7750] dbg: dns: Net::DNS version: 0.61
[7750] dbg: config: using "/etc/mail/spamassassin" for site rules pre files
[7750] dbg: config: read file /etc/mail/spamassassin/init.pre

tail
[7750] dbg: bayes: tie-ing to DB file 
R/W /home/xenoterracide/.spamassassin/bayes_toks
[7750] dbg: bayes: tie-ing to DB file 
R/W /home/xenoterracide/.spamassassin/bayes_seen
[7750] dbg: bayes: found bayes db version 3
[7750] dbg: locker: refresh_lock: 
refresh /home/xenoterracide/.spamassassin/bayes.lock
[7750] dbg: bayes: DB expiry: tokens in DB: 61967, Expiry max size: 150000, 
Oldest atime: 1102006137, Newest atime: 1208252263, Last expire: 0, Current 
time: 1208310434
[7750] dbg: bayes: expiry completed
[7750] dbg: bayes: untie-ing
[7750] dbg: bayes: files locked, now unlocking lock
[7750] dbg: locker: safe_unlock: 
unlink /home/xenoterracide/.spamassassin/bayes.lock
Learned tokens from 0 message(s) (0 message(s) examined)

> That said, what exactly do you mean in your original post by "doesn't
> seem to work" ?

by seems I mean doesn't. I'm assuming its something I'm doing wrong.

-- 
Caleb Cushing

my blog http://xenoterracide.blogspot.com