You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by Jonas <jo...@vrt.dk> on 2011/12/16 13:30:01 UTC
sa-learn and modern spam sizes
Hi all
I found out today that the reason a spammer was giving me pain when trying to learn his spam mails as spam for bayes, was that they are too big?
There are a couple of hits on it on google with various people having the same problem, I didn't find much answers but it appears that the hardcoded limit is somewhere between 256-512kilobyte?
These spam mails are all between 1-2MB, so is there no way to learn this as spam with bayes?
On top of that he's sending via Gmail making it hard to use rbl's.
Looking forward to suggestions and clarification of the sa-learn limit (if its hardcoded I would strongly suggest it becomes configurable)
Med venlig hilsen / Best regards
Jonas Akrouh Larsen
TechBiz ApS
Laplandsgade 4, 2. sal
2300 København S
Office: 7020 0979
Direct: 3336 9974
Mobile: 5120 1096
Fax: 7020 0978
Web: www.techbiz.dk <http://www.techbiz.dk>
Re: sa-learn and modern spam sizes
Posted by RW <rw...@googlemail.com>.
On Fri, 16 Dec 2011 12:06:15 -0500
Kevin A. McGrail wrote:
>
> >>> There are a couple of hits on it on google with various people
> >>> having the same problem, I didn't find much answers but it
> >>> appears that the hardcoded limit is somewhere between
> >>> 256-512kilobyte?
> >> man spamc
> >>
> >> -s max_size, --max-size=max_size
> >> Set the maximum message size which will be sent to
> >> spamd -- any bigger than this threshold and the message will be
> >> returned unprocessed (default: 500 KB). If spamc gets handed a
> >> message bigger than this, it won't be passed to spamd. The
> >> maximum message size is 256 MB.
> >>
> >>
> > I do not use spamd/spamc. But the perl module of spamassassin.
> >
> > So is there no way to get around this?
> Hmm. I didn't think SA had a limit internally. Normally, you
> utilize a limit on spamc (-s/--max-size) or in procmail such as * <
> 524288.
>
> But if you call SA directly as an API, I don't think there is a
> limit. You might want to post this on the dev list.
It's an optional limit in ArchiveIterator.pm. It's turned-on in
sa-learn.
Re: sa-learn and modern spam sizes
Posted by Noel Butler <no...@ausics.net>.
On Fri, 2011-12-16 at 18:17 -0500, Joseph Brennan wrote:
> > The maximum message size is 256 MB.
>
>
> I've never seen spam larger than 3 MB.
>
About 3 years ago(?), remember all the pdf spam? SA caught some that
were about 5mb, but yes, on the whole it is rather rare to be more than
a few KB.
Re: sa-learn and modern spam sizes
Posted by "Kevin A. McGrail" <KM...@PCCC.com>.
On 12/21/2011 5:24 PM, Jonas wrote:
>> > Alright that sounds as a fix at least, but do you not agree that it should be a
>> user configurable setting? As in some sort of parameter or conf file?
>>
>> I would support a bug to make that a user configurable setting. Please open it
>> with more details about what you are trying to do.
>>
>> regards,
>> KAM
> I opened a bug since the problem is now clear and a tempfix is even provided, I hope there will be consensus to make this more user-friendly and simpler to configure.
>
> https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6735
>
> Thanks to all for comments/help
Thanks for the very well entered bug! I see no reason it can't be done.
Regards,
KAM
RE: sa-learn and modern spam sizes
Posted by Jonas <jo...@vrt.dk>.
> > Alright that sounds as a fix at least, but do you not agree that it should be a
> user configurable setting? As in some sort of parameter or conf file?
>
> I would support a bug to make that a user configurable setting. Please open it
> with more details about what you are trying to do.
>
> regards,
> KAM
I opened a bug since the problem is now clear and a tempfix is even provided, I hope there will be consensus to make this more user-friendly and simpler to configure.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6735
Thanks to all for comments/help
Med venlig hilsen / Best regards
Jonas Akrouh Larsen
TechBiz ApS
Laplandsgade 4, 2. sal
2300 København S
Office: 7020 0979
Direct: 3336 9974
Mobile: 5120 1096
Fax: 7020 0978
Web: www.techbiz.dk
Re: sa-learn and modern spam sizes
Posted by "Kevin A. McGrail" <KM...@PCCC.com>.
> Alright that sounds as a fix at least, but do you not agree that it
should be a user configurable setting? As in some sort of parameter or
conf file?
I would support a bug to make that a user configurable setting. Please
open it with more details about what you are trying to do.
regards,
KAM
RE: sa-learn and modern spam sizes
Posted by Jonas <jo...@vrt.dk>.
> -----Original Message-----
> From: RW [mailto:rwmaillists@googlemail.com]
> Sent: 20. december 2011 18:07
> To: users@spamassassin.apache.org
> Subject: Re: sa-learn and modern spam sizes
>
> On Tue, 20 Dec 2011 15:38:58 +0000
> Duane Hill wrote:
>
> > On Tuesday, December 20, 2011 at 15:26:06 UTC, Bowie_Bailey@BUC.com
> > confabulated:
> >
>
> > >> Should I open a bug about it?
> >
> > > If you're using mailscanner, why not ask on the mailscanner list?
> >
> > > http://www.mailscanner.info/support.html#mailing
> >
> > I don't believe this is a mailscanner issue. I also have
found
> > sa-learn not learning anything over a certain size.
>
> As I already said, the limit is in the ArchiveIterator module and is
turned-on in
> sa-learn. If you edit sa-learn and change opt_all from 0 to
> 1 in the code below the problem should go away
>
>
>
> my $iter = new Mail::SpamAssassin::ArchiveIterator(
> {
> 'opt_all' => 0, # skip messages over 250k
> 'opt_want_date' => 0,
> }
> );
>
Alright that sounds as a fix at least, but do you not agree that it
should be a user configurable setting? As in some sort of parameter or
conf file?
Re: sa-learn and modern spam sizes
Posted by Duane Hill <du...@gmail.com>.
On Tuesday, December 20, 2011 at 17:06:36 UTC, rwmaillists@googlemail.com confabulated:
> On Tue, 20 Dec 2011 15:38:58 +0000
> Duane Hill wrote:
>> On Tuesday, December 20, 2011 at 15:26:06 UTC, Bowie_Bailey@BUC.com
>> confabulated:
>>
>> >> Should I open a bug about it?
>>
>> > If you're using mailscanner, why not ask on the mailscanner list?
>>
>> > http://www.mailscanner.info/support.html#mailing
>>
>> I don't believe this is a mailscanner issue. I also have found
>> sa-learn not learning anything over a certain size.
> As I already said, the limit is in the ArchiveIterator module and is
> turned-on in sa-learn. If you edit sa-learn and change opt_all from 0 to
> 1 in the code below the problem should go away
> my $iter = new Mail::SpamAssassin::ArchiveIterator(
> {
> 'opt_all' => 0, # skip messages over 250k
> 'opt_want_date' => 0,
> }
> );
Sorry for the noise. I wasn't following the thread too close.
--
If at first you don't succeed, so much for skydiving.
Re: sa-learn and modern spam sizes
Posted by RW <rw...@googlemail.com>.
On Tue, 20 Dec 2011 15:38:58 +0000
Duane Hill wrote:
> On Tuesday, December 20, 2011 at 15:26:06 UTC, Bowie_Bailey@BUC.com
> confabulated:
>
> >> Should I open a bug about it?
>
> > If you're using mailscanner, why not ask on the mailscanner list?
>
> > http://www.mailscanner.info/support.html#mailing
>
> I don't believe this is a mailscanner issue. I also have found
> sa-learn not learning anything over a certain size.
As I already said, the limit is in the ArchiveIterator module and is
turned-on in sa-learn. If you edit sa-learn and change opt_all from 0 to
1 in the code below the problem should go away
my $iter = new Mail::SpamAssassin::ArchiveIterator(
{
'opt_all' => 0, # skip messages over 250k
'opt_want_date' => 0,
}
);
Re: sa-learn and modern spam sizes
Posted by Duane Hill <du...@gmail.com>.
On Tuesday, December 20, 2011 at 15:26:06 UTC, Bowie_Bailey@BUC.com confabulated:
> On 12/19/2011 4:03 AM, Jonas wrote:
>>>> I've never seen spam larger than 3 MB.
>>> which is much bigger than the 256 kB limit in sa-learn that the OP is having a
>>> problem with.
>> Indeed, of course I agree the avg. spam size is much much lower.
>>
>> But a lot of the "manual" spam, typically originating in asia where people send out spam through Hotmail/gmail can be 1-3MB in size. Most of these are electronics or textile oriented "business" offers.
>>
>> And my problem remains, our setup is based on MailScanner (a daemon like amavis-new) which doesn't use spamc/spamd so I'm unable to train my bayes on these 1MB+ size spams, which is a problem.
>>
>> So can I conclude that there's no real solution to this besides code change?
>>
>> Should I open a bug about it?
> If you're using mailscanner, why not ask on the mailscanner list?
> http://www.mailscanner.info/support.html#mailing
I don't believe this is a mailscanner issue. I also have found
sa-learn not learning anything over a certain size. It's been a while
(a month) since I've seen large spam so I never bothered.
mailhost# ls -l notspam.msg
-rw-r--r-- 1 duane duane 363516 Dec 20 15:35 notspam.msg
mailhost# sa-learn --ham --username=duane@virtualnetwork.com notspam.msg
Learned tokens from 0 message(s) (0 message(s) examined)
If I chop everything out except the message headers:
mailhost# sa-learn --ham --username=duane@virtualnetwork.com notspam.msg
Learned tokens from 1 message(s) (1 message(s) examined)
--
If at first you don't succeed...
...so much for skydiving.
Re: sa-learn and modern spam sizes
Posted by Bowie Bailey <Bo...@BUC.com>.
On 12/19/2011 4:03 AM, Jonas wrote:
>>> I've never seen spam larger than 3 MB.
>> which is much bigger than the 256 kB limit in sa-learn that the OP is having a
>> problem with.
> Indeed, of course I agree the avg. spam size is much much lower.
>
> But a lot of the "manual" spam, typically originating in asia where people send out spam through Hotmail/gmail can be 1-3MB in size. Most of these are electronics or textile oriented "business" offers.
>
> And my problem remains, our setup is based on MailScanner (a daemon like amavis-new) which doesn't use spamc/spamd so I'm unable to train my bayes on these 1MB+ size spams, which is a problem.
>
> So can I conclude that there's no real solution to this besides code change?
>
> Should I open a bug about it?
If you're using mailscanner, why not ask on the mailscanner list?
http://www.mailscanner.info/support.html#mailing
--
Bowie
RE: sa-learn and modern spam sizes
Posted by Jonas <jo...@vrt.dk>.
> > I've never seen spam larger than 3 MB.
>
> which is much bigger than the 256 kB limit in sa-learn that the OP is having a
> problem with.
Indeed, of course I agree the avg. spam size is much much lower.
But a lot of the "manual" spam, typically originating in asia where people send out spam through Hotmail/gmail can be 1-3MB in size. Most of these are electronics or textile oriented "business" offers.
And my problem remains, our setup is based on MailScanner (a daemon like amavis-new) which doesn't use spamc/spamd so I'm unable to train my bayes on these 1MB+ size spams, which is a problem.
So can I conclude that there's no real solution to this besides code change?
Should I open a bug about it?
Med venlig hilsen / Best regards
Jonas Akrouh Larsen
TechBiz ApS
Laplandsgade 4, 2. sal
2300 København S
Office: 7020 0979
Direct: 3336 9974
Mobile: 5120 1096
Fax: 7020 0978
Web: www.techbiz.dk
Re: sa-learn and modern spam sizes
Posted by RW <rw...@googlemail.com>.
On Fri, 16 Dec 2011 18:17:13 -0500
Joseph Brennan wrote:
>
> > The maximum message size is 256 MB.
>
>
> I've never seen spam larger than 3 MB.
which is much bigger than the 256 kB limit in sa-learn that the OP is
having a problem with.
Re: sa-learn and modern spam sizes
Posted by Joseph Brennan <br...@columbia.edu>.
> The maximum message size is 256 MB.
I've never seen spam larger than 3 MB.
Joseph Brennan
Lead Email Systems Engineer
Columbia University Information Technology
Re: sa-learn and modern spam sizes
Posted by "Kevin A. McGrail" <KM...@PCCC.com>.
>>> There are a couple of hits on it on google with various people having
>>> the same problem, I didn't find much answers but it appears that the
>>> hardcoded limit is somewhere between 256-512kilobyte?
>> man spamc
>>
>> -s max_size, --max-size=max_size
>> Set the maximum message size which will be sent to spamd -- any bigger
>> than this threshold and the message will be returned unprocessed (default: 500
>> KB). If spamc gets handed a message bigger
>> than this, it won't be passed to spamd. The maximum message size is
>> 256 MB.
>>
>>
> I do not use spamd/spamc. But the perl module of spamassassin.
>
> So is there no way to get around this?
Hmm. I didn't think SA had a limit internally. Normally, you utilize a
limit on spamc (-s/--max-size) or in procmail such as * < 524288.
But if you call SA directly as an API, I don't think there is a limit.
You might want to post this on the dev list.
A quick check and I couldn't find anything in the code or docs but I
didn't look exhaustively.
regards,
KAM
RE: sa-learn and modern spam sizes
Posted by Jonas <jo...@vrt.dk>.
> > There are a couple of hits on it on google with various people having
> > the same problem, I didn't find much answers but it appears that the
> > hardcoded limit is somewhere between 256-512kilobyte?
>
> man spamc
>
> -s max_size, --max-size=max_size
> Set the maximum message size which will be sent to spamd -- any bigger
> than this threshold and the message will be returned unprocessed (default: 500
> KB). If spamc gets handed a message bigger
> than this, it won't be passed to spamd. The maximum message size is
> 256 MB.
>
>
I do not use spamd/spamc. But the perl module of spamassassin.
So is there no way to get around this?
Med venlig hilsen / Best regards
Jonas Akrouh Larsen
TechBiz ApS
Laplandsgade 4, 2. sal
2300 København S
Office: 7020 0979
Direct: 3336 9974
Mobile: 5120 1096
Fax: 7020 0978
Web: www.techbiz.dk
Re: sa-learn and modern spam sizes
Posted by Robert Schetterer <ro...@schetterer.org>.
Am 16.12.2011 13:30, schrieb Jonas:
> Hi all
>
>
>
> I found out today that the reason a spammer was giving me pain when
> trying to learn his spam mails as spam for bayes, was that they are too big?
>
>
>
> There are a couple of hits on it on google with various people having
> the same problem, I didn’t find much answers but it appears that the
> hardcoded limit is somewhere between 256-512kilobyte?
man spamc
-s max_size, --max-size=max_size
Set the maximum message size which will be sent to spamd --
any bigger than this threshold and the message will be returned
unprocessed (default: 500 KB). If spamc gets handed a message bigger
than this, it won't be passed to spamd. The maximum message
size is 256 MB.
>
>
>
> These spam mails are all between 1-2MB, so is there no way to learn this
> as spam with bayes?
>
i guess this isnt a big problem, it may consum
machine power and time
but gurus should know more
>
>
> On top of that he’s sending via Gmail making it hard to use rbl’s.
>
>
>
> Looking forward to suggestions and clarification of the sa-learn limit
> (if its hardcoded I would strongly suggest it becomes configurable)
>
>
>
>
>
> Med venlig hilsen / Best regards
>
>
>
> Jonas Akrouh Larsen
>
>
>
> TechBiz ApS
>
> Laplandsgade 4, 2. sal
>
> 2300 København S
>
>
>
> Office: 7020 0979
>
> Direct: 3336 9974
>
> Mobile: 5120 1096
>
> Fax: 7020 0978
>
> Web: www.techbiz.dk <http://www.techbiz.dk>
>
>
>
>
>
--
Best Regards
MfG Robert Schetterer
Germany/Munich/Bavaria