You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by Asif Iqbal <iq...@qwestip.net> on 2004/07/27 02:54:07 UTC

SPAMD taking over 300 secs

Hi All

I am running spamd and seeing waay high process time. I am runnign spamd
in supervise mode like this

cat /service/spamd/run

#!/bin/sh
exec 2>&1
exec /usr/local/bin/spamd -s stderr --syslog-socket=inet -u qmailq

Notice I am not using -m 20 since I am getting hit max-children error
within few mins due to soo many spams

Any help/suggestion would be greatly appreciated

Thanks
-- 
Asif Iqbal
PGP Key: 0xE62693C5 KeyServer: pgp.mit.edu
There's no place like 127.0.0.1

Re: SPAMD taking over 300 secs

Posted by Bob McClure Jr <ro...@earthlink.net>.
On Mon, Jul 26, 2004 at 08:54:07PM -0400, Asif Iqbal wrote:
> Hi All
> 
> I am running spamd and seeing waay high process time. I am runnign spamd
> in supervise mode like this
> 
> cat /service/spamd/run
> 
> #!/bin/sh
> exec 2>&1
> exec /usr/local/bin/spamd -s stderr --syslog-socket=inet -u qmailq
> 
> Notice I am not using -m 20 since I am getting hit max-children error
> within few mins due to soo many spams
> 
> Any help/suggestion would be greatly appreciated
> 
> Thanks
> -- 
> Asif Iqbal
> PGP Key: 0xE62693C5 KeyServer: pgp.mit.edu
> There's no place like 127.0.0.1

What rulesets are you running?

You should not be running blacklist.cf or bigevil.cf which are huge
consumers of system memory, leading to heavy swap usage, which leads
to a server on its knees.  Ask me how I know.

What version of SA are you running?

If v2.63, here is your solution:

http://ws.surbl.org/

It will replace the user of blacklist.cf, blacklist_uri.cf, and
bigevil.cf.

Cheers,
-- 
Bob McClure, Jr.             Bobcat Open Systems, Inc.
robertmcclure@earthlink.net  http://www.bobcatos.com
Worry is the darkroom in which negatives are developed.

Re: SPAMD taking over 300 secs

Posted by Jim Maul <jm...@elih.org>.
Quoting Asif Iqbal <iq...@qwestip.net>:

> JGrotepass wrote:
>> Asif Iqbal wrote:
>>
>> >
>> >Is there a way I can limit the amount of memory the spamd can use and
>> >and the total size of mesg it scans?
>>
>> As I see that you're using qmail. I just found that the parm "-s xxxx"
>> for the "spamc" does limit the size of the mail that is scanned. The
>> default value (no -s Parm) is 250K.
>>
>> How are you feeding the mail from Qmail to Spamd?
>
> qmail-scanner-queue.pl calls spamc like that
>
> /usr/local/bin/spamc  -c -f
>
>
>>

I'd say its the network tests being slow.  Most of the time spamc/spamd on my
system take 1-2 seconds.  Ocassionally i see something like 20-30 seconds.
During these times if i run spamc without the network tests, the message
processes in 1-2 seconds again.  Network tests back in again and the scan time
goes back to 20-30 seconds.  Maybe there are dns problems..maybe you can try
other razor/pyzor/dcc servers.  Try it without the network tests and see what
the scan times are.

Jim

Re: SPAMD taking over 300 secs

Posted by Daniel Quinlan <qu...@pathname.com>.
Theo Van Dinter <fe...@kluge.net> writes:

> FYI: spamd and spamassassin should only ever wait 10 seconds for a bayes
> lock (it's considered "opportunistic").  sa-learn will wait up to 300
> seconds.  DNS-related tests should timeout after (default) 15 seconds.

15 seconds is the maximum default timeout.  Unless, DNS is completely
unresponsive, it's faster than that because it'll figure out that one or
two DNSBLs are down when others respond much more quickly.

> Add another (default) 10 seconds per Razor, DCC, Pyzor, if running.

Yeah, that has to get changed for 3.1...

Daniel

-- 
Daniel Quinlan
http://www.pathname.com/~quinlan/

Re: SPAMD taking over 300 secs

Posted by Theo Van Dinter <fe...@kluge.net>.
On Tue, Jul 27, 2004 at 05:54:29PM -0400, Asif Iqbal wrote:
> > you should disable the opportunistic expiry (aka auto-expiry), and run
> 
> How do I do that? Wiki page has the instruction?

"man Mail::SpamAssassin::Conf" is your friend. :)    IIRC, it's
"bayes_auto_expire 0".

> How should this cronjob look like?

"sa-learn --force-expire" should do it for a site-wide DB.

-- 
Randomly Generated Tagline:
if (instr(buf,sys_errlist[errno]))  /* you don't see this */
              -- Larry Wall in eval.c from the perl source code

Re: SPAMD taking over 300 secs

Posted by Theo Van Dinter <fe...@kluge.net>.
On Tue, Jul 27, 2004 at 03:12:49PM -0600, Johnny B. wrote:
> db for individual users over nfs.  if your mail message store for the 
> users is available over nfs you might have a bayes database locking/lock 
> file issue which in my case wound up extending delivery times to close 
> to 300 seconds.

FYI: spamd and spamassassin should only ever wait 10 seconds for a bayes
lock (it's considered "opportunistic").  sa-learn will wait up to 300
seconds.  DNS-related tests should timeout after (default) 15 seconds.
Add another (default) 10 seconds per Razor, DCC, Pyzor, if running.

However, spamd could be doing a bayes expire, which is not limited
by time.  Once it starts, it runs.  If you find this to be an issue,
you should disable the opportunistic expiry (aka auto-expiry), and run
it periodically via cron/etc.

-- 
Randomly Generated Tagline:
The Harvard Law states:  Under controlled conditions of light, temperature,
 humidity, and nutrition, the organism will do as it damn well pleases.
              -- Larry Wall in <19...@wall.org>

Re: SPAMD taking over 300 secs

Posted by "Johnny B." <jo...@peakpeak.com>.
i ran into something similar with spamassassin and its use of the bayes 
db for individual users over nfs.  if your mail message store for the 
users is available over nfs you might have a bayes database locking/lock 
file issue which in my case wound up extending delivery times to close 
to 300 seconds.  i also had network collisions which loaded the server 
unnecessarily so check that your network interface isn't experiencing 
issues as well otherwise load will be high and performance will really 
suffer.   use the command ifconig and look at the stats it'll show if 
there are errors.    if your users' mail message store isn't over nfs 
then you may also need a machine which has more processing power 
available, not sure what type of hardware you are trying to run this on 
with what additional software you are running and what youre trying to 
support more info is needed to determine root cause.

cheers

jb

Asif Iqbal wrote:

>JGrotepass wrote:
>  
>
>>Asif Iqbal wrote:
>>
>>    
>>
>>>Is there a way I can limit the amount of memory the spamd can use and
>>>and the total size of mesg it scans?
>>>      
>>>
>>As I see that you're using qmail. I just found that the parm "-s xxxx" 
>>for the "spamc" does limit the size of the mail that is scanned. The 
>>default value (no -s Parm) is 250K.
>>
>>How are you feeding the mail from Qmail to Spamd?
>>    
>>
>
>qmail-scanner-queue.pl calls spamc like that
>
>/usr/local/bin/spamc  -c -f 
>
>
>  
>
>>Regards,
>>Jochen
>>
>>    
>>
>
>  
>

Re: SPAMD taking over 300 secs

Posted by Asif Iqbal <iq...@qwestip.net>.
JGrotepass wrote:
> Asif Iqbal wrote:
> 
> >
> >Is there a way I can limit the amount of memory the spamd can use and
> >and the total size of mesg it scans?
> 
> As I see that you're using qmail. I just found that the parm "-s xxxx" 
> for the "spamc" does limit the size of the mail that is scanned. The 
> default value (no -s Parm) is 250K.
> 
> How are you feeding the mail from Qmail to Spamd?

qmail-scanner-queue.pl calls spamc like that

/usr/local/bin/spamc  -c -f 


> 
> Regards,
> Jochen
> 

-- 
Asif Iqbal
PGP Key: 0xE62693C5 KeyServer: pgp.mit.edu
There's no place like 127.0.0.1

Re: SPAMD taking over 300 secs

Posted by JGrotepass <jg...@arcor.de>.
Asif Iqbal wrote:

> 
> Is there a way I can limit the amount of memory the spamd can use and
> and the total size of mesg it scans?

As I see that you're using qmail. I just found that the parm "-s xxxx" 
for the "spamc" does limit the size of the mail that is scanned. The 
default value (no -s Parm) is 250K.

How are you feeding the mail from Qmail to Spamd?

Regards,
Jochen


Re: SPAMD taking over 300 secs

Posted by Asif Iqbal <iq...@qwestip.net>.
Asif Iqbal wrote:
> Hi All
> 
> I am running spamd and seeing waay high process time. I am runnign spamd
> in supervise mode like this
> 
> cat /service/spamd/run
> 
> #!/bin/sh
> exec 2>&1
> exec /usr/local/bin/spamd -s stderr --syslog-socket=inet -u qmailq

I had to put the -m 20 back since it was eating out all my memories and
CPUs

However I am noticing this now in the log very frequently and also my
qmail-smtp became slow due to that.

Jul 27 11:56:04 qmail spamc[11657]: connect(AF_INET) to spamd at
127.0.0.1 failed, retrying (#3 of 3): Connection timed out

Is there a way I can limit the amount of memory the spamd can use and
and the total size of mesg it scans?

Thanks

> 
> Notice I am not using -m 20 since I am getting hit max-children error
> within few mins due to soo many spams
> 
> Any help/suggestion would be greatly appreciated
> 
> Thanks
> -- 
> Asif Iqbal
> PGP Key: 0xE62693C5 KeyServer: pgp.mit.edu
> There's no place like 127.0.0.1

-- 
Asif Iqbal
PGP Key: 0xE62693C5 KeyServer: pgp.mit.edu
There's no place like 127.0.0.1

Re: SPAMD taking over 300 secs

Posted by xavier <li...@natch.dyndns.org>.
On Mon, Jul 26, 2004 at 08:54:07PM -0400, Asif Iqbal wrote:
> Hi All
> 
> I am running spamd and seeing waay high process time. I am runnign spamd
> in supervise mode like this

do you check the size before sending it to spamd ?

anything below 200kB will blow your ram, your swap, and bring your
server to its knees, eat your chair.

You're using qmail ; unfortunaly : I know only the exim (+exiscan) way of doing it :

add "condition = ${if <{$message_size}{200k}{1}{0}}"
every where your call spamd :

ex :

deny message = This message scored $spam_score spam points.
    condition = ${if <{$message_size}{200k}{1}{0}}
    spam = spamd:true
    condition = ${if >{$spam_score_int}{80}{1}{0}}



bye

-- 
xavier