You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spamassassin.apache.org by bu...@bugzilla.spamassassin.org on 2005/05/07 19:41:57 UTC
[Bug 4316] New: spamd loops spawning children that die immediately.
http://bugzilla.spamassassin.org/show_bug.cgi?id=4316
Summary: spamd loops spawning children that die immediately.
Product: Spamassassin
Version: 3.0.2
Platform: PC
OS/Version: Linux
Status: NEW
Severity: critical
Priority: P1
Component: spamc/spamd
AssignedTo: dev@spamassassin.apache.org
ReportedBy: gblack@smartertek.net
I'm running a Sendmail/procmail/spamassassin combo on Debian Linux using
spamc/spamd and recently noticed the following errors in my sendmail log:
sm-mta[30568]: j412QfpS030544: timeout waiting for input from local during
Draining Input
Checking the running processes showed things like:
17759 ? S 0:00 spamc
17812 ? S 0:00 spamc
17831 ? S 0:00 spamc
17980 ? S 0:00 spamc
18614 ? S 0:00 spamc
19870 ? S 0:00 spamc
20297 ? S 0:00 spamc
20517 ? S 0:00 spamc
21465 ? R 13:23 /usr/sbin/spamd --create-prefs --max-children 5
--helper-home-dir --max-conn-per-child 5 -d --pidfile=/var/run/spamd.pid
22280 ? Z 0:00 [spamd <defunct>]
Telnetting to spamd brought no response. Turning on logging would show
everything normal up until the problem occured - at which point there were NO
log entries from SA at all, and the last entry would show the finish of a
completely normal message processing. strace turned out to be more informative
and pointed me to the real culprit (I'll attach a trace in a minute). The end
result was failed communication between SA and syslog. A few days earlier I had
replaced my sysklogd for syslog-ng (1.5.15-1.1). It appears after several hours
(12-24 hours on the average with --max-conn-per-child set to 200) that the
communication between SA and syslog-ng would break at which point spamd would
start madly spawning children that would die immediately. It appears that spamd
is completely dependant upon syslog and cannot function correctly without it. I
stopped and started spamd and ran a strace on it. Then to test the theory, I
stopepd the syslog service. Once the syslog service was dead the process began.
The children died slowly at first as the parent tried to keep up by spawning
processes but eventually as more than one would die at a time, the parent
couldn't keep up and I was down to one parent madly spawning one child would
would die before the parent had a chance to spawn a second - naturally no spam
processing could be accomplished during this time. I've replaced syslog-ng with
my old sysklogd in hopes this will serve as a fix for the time being. I should
note that decreasing the value for --max-conn-per-child appears (have not done
enough testing to be certain) to also decrease the time beween failures of spamd
(or in other words, setting it to one may bring about the failure in 2-4 hours
instead of 12-24 hours). Stopping syslog seems to bring about the failure
immediately (it might not be immediately obvious from a ps, but a strace will
tell the story in a hurry).
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 4316] spamd loops spawning children that die immediately.
Posted by bu...@bugzilla.spamassassin.org.
http://bugzilla.spamassassin.org/show_bug.cgi?id=4316
------- Additional Comments From quinlan@pathname.com 2005-05-09 01:29 -------
Subject: Re: spamd loops spawning children that die immediately.
> I suppose I could get shot for asking this, but given the syslog
> issue, is there any support for throwing in an option to log to a file?
You'd only be shot if it was documented in the spamd manual.
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 4316] spamd loops spawning children that die immediately.
Posted by bu...@bugzilla.spamassassin.org.
http://bugzilla.spamassassin.org/show_bug.cgi?id=4316
------- Additional Comments From gblack@smartertek.net 2005-05-07 10:47 -------
Created an attachment (id=2855)
--> (http://bugzilla.spamassassin.org/attachment.cgi?id=2855&action=view)
strace of failing spamd
strace that shows spamd failing WHILE syslog WAS running. Syslog was syslog-ng
(1.5.15-1.1) on debian stable in this case. Probably did not surface with
sysklogd which had been used previously. Starting up a fresh run of spamd and
stopping syslog would produce the same results.
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 4316] spamd loops spawning children that die immediately.
Posted by bu...@bugzilla.spamassassin.org.
http://bugzilla.spamassassin.org/show_bug.cgi?id=4316
------- Additional Comments From quinlan@pathname.com 2005-05-09 00:41 -------
Subject: Re: spamd loops spawning children that die immediately.
> btw, yes, you need to restart (or SIGHUP) spamd after syslogd restart.
Is there any way to avoid that?
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 4316] spamd loops spawning children that die immediately.
Posted by bu...@bugzilla.spamassassin.org.
http://bugzilla.spamassassin.org/show_bug.cgi?id=4316
------- Additional Comments From gblack@smartertek.net 2005-05-07 10:56 -------
Doing some additional testing here, it appears that once the condition occurs,
there is no recovery. Any children who have not experienced the failure appear
to continue running normally until their time to terminate (max connections).
The parent process seems to be incapable of spawning any healthy children though
after the first initial failure even if the condition corrects itself (stop
syslog to cause failure, then restart it in an attempt to stablize things). At
the very least this appears to indicate that whenever syslog is stopped, SA
should be as well, and then not restarted until after syslog is restarted.
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 4316] spamd loops spawning children that die immediately.
Posted by bu...@bugzilla.spamassassin.org.
http://bugzilla.spamassassin.org/show_bug.cgi?id=4316
jm@jmason.org changed:
What |Removed |Added
----------------------------------------------------------------------------
Severity|critical |major
Status|NEW |RESOLVED
Priority|P1 |P3
Resolution| |DUPLICATE
------- Additional Comments From jm@jmason.org 2005-05-09 00:34 -------
pretty sure this is a dupe of bug 4237, fixed for 3.1.0.
summary: syslog-ng is seriously crappy. there's a patch in that bug.
btw, yes, you need to restart (or SIGHUP) spamd after syslogd restart.
*** This bug has been marked as a duplicate of 4237 ***
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 4316] spamd loops spawning children that die immediately.
Posted by bu...@bugzilla.spamassassin.org.
http://bugzilla.spamassassin.org/show_bug.cgi?id=4316
------- Additional Comments From gblack@smartertek.net 2005-05-09 01:01 -------
(In reply to comment #3)
> pretty sure this is a dupe of bug 4237, fixed for 3.1.0.
> summary: syslog-ng is seriously crappy. there's a patch in that bug.
>
> btw, yes, you need to restart (or SIGHUP) spamd after syslogd restart.
>
> *** This bug has been marked as a duplicate of 4237 ***
Crappy or no (I only used it because of another package wanting it) SA will need
to be able to handle breaks in communication with syslog without crashing
(seemingly random unplanned crashes aren't fun) and it will need to be able to
resume logging once the environment allows (otherwise logging is seriously
handicapped and basicly undependable). This does look like a dup of that bug
though. I did do some searching before opening this, but apparently still missed it.
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 4316] spamd loops spawning children that die immediately.
Posted by bu...@bugzilla.spamassassin.org.
http://bugzilla.spamassassin.org/show_bug.cgi?id=4316
------- Additional Comments From jm@jmason.org 2005-05-09 01:11 -------
'Is there any way to avoid that?'
not unless we find an alternative to Sys::syslog. what amavisd does (iirc) is
log via a pipe to a /usr/bin/logger process, which may be more reliable.
'SA will need
to be able to handle breaks in communication with syslog without crashing'
agreed...
'and it will need to be able to
resume logging once the environment allows'
unfortunately, we cannot do that reliably due to Sys::Syslog shortcomings.
hence the SIGHUP.
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
Re: [Bug 4316] spamd loops spawning children that die immediately.
Posted by Daniel Quinlan <qu...@pathname.com>.
> I suppose I could get shot for asking this, but given the syslog
> issue, is there any support for throwing in an option to log to a file?
You'd only be shot if it was documented in the spamd manual.
--
Daniel Quinlan
http://www.pathname.com/~quinlan/
[Bug 4316] spamd loops spawning children that die immediately.
Posted by bu...@bugzilla.spamassassin.org.
http://bugzilla.spamassassin.org/show_bug.cgi?id=4316
------- Additional Comments From gblack@smartertek.net 2005-05-09 01:18 -------
(In reply to comment #6)
> 'Is there any way to avoid that?'
>
> not unless we find an alternative to Sys::syslog. what amavisd does (iirc) is
> log via a pipe to a /usr/bin/logger process, which may be more reliable.
>
> 'SA will need
> to be able to handle breaks in communication with syslog without crashing'
>
> agreed...
>
> 'and it will need to be able to
> resume logging once the environment allows'
>
> unfortunately, we cannot do that reliably due to Sys::Syslog shortcomings.
> hence the SIGHUP.
I suppose I could get shot for asking this, but given the syslog issue, is there
any support for throwing in an option to log to a file?
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.