You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spamassassin.apache.org by bu...@bugzilla.spamassassin.org on 2005/05/05 05:14:57 UTC
[Bug 4304] New: Spamd child still running after shutdown
http://bugzilla.spamassassin.org/show_bug.cgi?id=4304
Summary: Spamd child still running after shutdown
Product: Spamassassin
Version: 3.0.3
Platform: PC
OS/Version: Linux
Status: NEW
Severity: normal
Priority: P2
Component: spamc/spamd
AssignedTo: dev@spamassassin.apache.org
ReportedBy: jregier@accesshub.com
After I shutdown spamd and the script reports success, I still have a spamd
process left running with a new PID. It's still listening to port 783. The
process goes away after running the shutdown script again. This seems to be
happening most of the time but not always.
I'll attach some debugging info.
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 4304] Spamd child still running after shutdown
Posted by bu...@bugzilla.spamassassin.org.
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4304
------- Additional Comments From jm@jmason.org 2006-10-04 04:36 -------
'I can sort of repro the condition by inserting an exit(0); right before the
server-side of the fork returns from spawn().'
ah, yes, well that would be expected -- the children won't exit if they're not
told to, and exit(0) at that point would never tell them to. I don't think
that's a valid test case, unfortunately, as a result.
I've written a test case here, and by adding a "sleep(5);" directly after
'$spamtest->call_plugins("spamd_child_init");', I can reproduce the problem you
describe in comment 9. It goes like this:
- external process starts up spamd
- spamd blocks SIGINT
- spamd forks off children
- external process kills spamd parent, it dies nearly immediately
- spamd children runs 'sleep(5)'
- external process starts new spamd parent (after short pause)
- new spamd parent chokes saying socket already in use
- spamd children finish loading up, unblock SIGINT, receive SIGINT and then die
that's pretty much what you describe... so that definitely *is* a bug.
If I move setup_child_sig_handlers() and sigprocmask( POSIX::SIG_UNBLOCK(),
$sigset ) to between "## CHILD" and
"$spamtest->call_plugins("spamd_child_init");", as the patch does, this is
fixed.
Since it's a confirmed bug, I've checked that into SVN trunk as r452849, anyway ;)
Some other things, though:
You say 'spamd children continue loading things up (this takes a while in our
environment)'. What is this exactly?
You note:
'if I know the parent is dead but I don't see a
"spamd: server killed by SIG$sig, shutting down" line in the logfile'
This can happen if a non-TERM or non-INT signal was sent -- like a SIGKILL
or (worse) a SIGBUS or SIGSEGV error. are you sure it's not something
more serious causing that?
also;
'Thinking of it that way I can't quite understand why the signal blocking is
there at all. I have a version of spamd here that removes all of it. Doesn't
help this particular case but may be worth putting in anyway?'
definitely not a good idea ;) the signal handling there is used
to avoid a known deadlock at fork() time -- the child pids cannot
use the same signal handler as the parent until after fork(), so the
only safe option is to keep signals masked until after fork and the
new child sig handlers are installed. This is pretty vital. The
bug however was that the masking was being kept on far too long.
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 4304] Spamd child still running after shutdown
Posted by bu...@bugzilla.spamassassin.org.
http://bugzilla.spamassassin.org/show_bug.cgi?id=4304
------- Additional Comments From jregier@accesshub.com 2005-05-04 20:18 -------
Created an attachment (id=2841)
--> (http://bugzilla.spamassassin.org/attachment.cgi?id=2841&action=view)
This is the procedure I'm using to reproduce and some logfile output.
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 4304] Spamd child still running after shutdown
Posted by bu...@bugzilla.spamassassin.org.
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4304
------- Additional Comments From jm@jmason.org 2006-10-04 04:38 -------
oh, for the record -- I'm quite sure bug 5081 (now fixed) is the original bug
reported by Jesse Regier here.
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 4304] Spamd child still running after shutdown
Posted by bu...@bugzilla.spamassassin.org.
http://bugzilla.spamassassin.org/show_bug.cgi?id=4304
------- Additional Comments From Bob@Menschel.net 2005-07-13 23:59 -------
Bug 4190 may be related.
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 4304] Spamd child still running after shutdown
Posted by bu...@bugzilla.spamassassin.org.
http://bugzilla.spamassassin.org/show_bug.cgi?id=4304
------- Additional Comments From jm@jmason.org 2005-05-10 23:37 -------
I think this may be fixed in 3.1.0 -- can't seem to reproduce it...
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 4304] Spamd child still running after shutdown
Posted by bu...@bugzilla.spamassassin.org.
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4304
------- Additional Comments From bob@norcalttora.com 2006-10-05 15:34 -------
Our code had some edge case coverage where when the parent process was deemed to be way out of
whack we would SIGKILL it. In the case I've describing here we were getting into this edge case every time.
Oops. I think my problem was a bug of my own.
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 4304] [review] Spamd child still running after shutdown
Posted by bu...@bugzilla.spamassassin.org.
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4304
felicity@apache.org changed:
What |Removed |Added
----------------------------------------------------------------------------
Status Whiteboard|needs 1 vote |go ahead
------- Additional Comments From felicity@apache.org 2006-12-08 11:53 -------
sure +1
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 4304] Spamd child still running after shutdown
Posted by bu...@bugzilla.spamassassin.org.
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4304
bob@norcalttora.com changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |bob@norcalttora.com
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 4304] Spamd child still running after shutdown
Posted by bu...@bugzilla.spamassassin.org.
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4304
------- Additional Comments From bob@norcalttora.com 2006-09-28 19:00 -------
Bug 5081 seems to apply to the prefork method right? We're round-robin so we shouldn't be, for instance,
scaling out more children to cope with the current load.
But looking at that patch I think I understand what may be going on and I may be able to address stuff on
our end. If I end up patching SA I'll bring the patch back, otherwise I'll just let everyone know what I've
stopped doing :-)
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 4304] Spamd child still running after shutdown
Posted by bu...@bugzilla.spamassassin.org.
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4304
------- Additional Comments From bob@norcalttora.com 2006-10-02 15:27 -------
I think I've identified my problem:
- external process starts up spamd
- spamd blocks SIGINT
- spamd forks off children
- external process kills spamd parent, it dies nearly immediately
- spamd children continue loading things up (this takes a while in our environment)
- external process starts new spamd parent (after short pause)
- external process chokes saying socket already in use
- spamd children finish loading up, unblock SIGINT, receive SIGINT and then die
Is there a reason to continue blocking SIGINT until the child is essentially fully ready to run? In my
simple test case moving the unblocking of SIGINT to immediately after the fork() within the child
process helps to make sure the child dies immediately and can't linger around for a while. I see that
certain things get done with Bayes there and Michael Parker has expressed concern over corrupting
databases.
Are there actual concerns or reasons for keeping those signals blocked for so long?
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 4304] [review] Spamd child still running after shutdown
Posted by bu...@bugzilla.spamassassin.org.
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4304
jm@jmason.org changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|RESOLVED |REOPENED
Keywords|triage |
Resolution|FIXED |
Summary|Spamd child still running |[review] Spamd child still
|after shutdown |running after shutdown
Status Whiteboard| |needs 2 votes
Target Milestone|Undefined |3.1.8
------- Additional Comments From jm@jmason.org 2006-10-06 07:36 -------
Tom -- good point. I'll reopen... patch applies cleanly to 3.1.x. aiming at
3.1.8, since 3.1.7 looks likely to be an immediate backout of another fix (see
bug 5119).
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 4304] Spamd child still running after shutdown
Posted by bu...@bugzilla.spamassassin.org.
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4304
------- Additional Comments From bob@norcalttora.com 2006-09-27 18:23 -------
I ran into this on FreeBSD 6.1 recently as well. My method for restarting SA is to locate the parent PID, kill
it and assume the child processes died.
Though I don't have good repro I've seen it more than once. We're on 3.1.0 and using --round-robin.
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 4304] [review] Spamd child still running after shutdown
Posted by bu...@bugzilla.spamassassin.org.
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4304
spamassassin@dostech.ca changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|REOPENED |RESOLVED
Resolution| |FIXED
------- Additional Comments From spamassassin@dostech.ca 2006-12-11 11:46 -------
[dos@FC5-VPC 3.1]$ svn ci -m "bug 4304: ..."
Sending spamd/spamd.raw
Transmitting file data .
Committed revision 485842.
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 4304] Spamd child still running after shutdown
Posted by bu...@bugzilla.spamassassin.org.
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4304
------- Additional Comments From jm@jmason.org 2006-10-03 02:12 -------
Created an attachment (id=3711)
--> (http://issues.apache.org/SpamAssassin/attachment.cgi?id=3711&action=view)
fix
Bob, does this work for your test case?
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 4304] Spamd child still running after shutdown
Posted by bu...@bugzilla.spamassassin.org.
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4304
------- Additional Comments From bob@norcalttora.com 2006-10-03 23:52 -------
So it turns out that that patch doesn't help in my case.
I can sort of repro the condition by inserting an exit(0); right before the server-side of the fork returns
from spawn(). The funny thing is that if you look at the code we die() if we can't unblock the signals. In
other words, if we can't unblock the signals we dump right into this condition :-)
Thinking of it that way I can't quite understand why the signal blocking is there at all. I have a version
of spamd here that removes all of it. Doesn't help this particular case but may be worth putting in
anyway?
Watching the logs as carefully as possible it appears that the bug occurs when the spamd parent is
killed but the kill_handler is not called. In other words, if I know the parent is dead but I don't see a
"spamd: server killed by SIG$sig, shutting down" line in the logfile I know with some certainty that I
have orphaned children. This can be reproed by setting the INT handler for the parent to default. A very
unlikely scenario is that the child signal setup is being called in the parent.
I have little idea what's causing this. Can we just rewrite spamd? :-)
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 4304] Spamd child still running after shutdown
Posted by bu...@bugzilla.spamassassin.org.
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4304
jm@jmason.org changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |RESOLVED
Resolution| |FIXED
------- Additional Comments From jm@jmason.org 2006-10-06 01:40 -------
fair enough!
well, we've found 2 bugs in this dept, and fixed them, and the third
was a PEBKAC error, sorry Bob ;). I think that means we can close this...
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 4304] [review] Spamd child still running after shutdown
Posted by bu...@bugzilla.spamassassin.org.
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4304
spamassassin@dostech.ca changed:
What |Removed |Added
----------------------------------------------------------------------------
Status Whiteboard|needs 2 votes |needs 1 vote
------- Additional Comments From spamassassin@dostech.ca 2006-12-08 10:00 -------
+1
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 4304] Spamd child still running after shutdown
Posted by bu...@bugzilla.spamassassin.org.
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4304
------- Additional Comments From jm@jmason.org 2006-09-27 21:00 -------
bug 5081 may also be related, since that fixes what Bob described ;)
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 4304] Spamd child still running after shutdown
Posted by bu...@bugzilla.spamassassin.org.
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4304
------- Additional Comments From jm@jmason.org 2006-09-28 19:38 -------
good point -- bug 5081 is indeed for the prefork case, not round-robin. my
mistake ;)
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 4304] Spamd child still running after shutdown
Posted by bu...@bugzilla.spamassassin.org.
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4304
------- Additional Comments From schulz@adi.com 2006-10-06 07:07 -------
Should the patch be applied to 3.1.7?
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 4304] Spamd child still running after shutdown
Posted by bu...@bugzilla.spamassassin.org.
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4304
------- Additional Comments From jm@jmason.org 2006-10-03 02:11 -------
thanks Bob! that looks like it, alright.
There's no good reason for it to keep the signals blocked. if plugins need to
perform lengthy initialisation that can't be interrupted, they need to take care
of signals themselves -- and that spamd_child_init hook is the wrong place, too,
I think.
I'll attach a patch that makes that change...
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 4304] Spamd child still running after shutdown
Posted by bu...@bugzilla.spamassassin.org.
http://bugzilla.spamassassin.org/show_bug.cgi?id=4304
Bob@Menschel.net changed:
What |Removed |Added
----------------------------------------------------------------------------
Keywords| |triage
------- Additional Comments From Bob@Menschel.net 2005-07-13 22:56 -------
Theory: The shutdown command is basically using
> killproc spamd
In what sequence does killproc work? If it kills parent and then children, this
should be OK. I suspect it kills sequentially by process number, or by
sequential position within the O/S procedure table. In either case, there could
be a child ahead of the parent spamd in the kill sequence. That allows for a
timing issue:
1) "killproc spamd" issued
2) child killed
3) spamd begins to respawn killed child
4) spamd killed, after child is already in process of starting, but before spamd
can record child in its internal process table
4a) spamd parent kills all children in its process table, missing that one
4b) spamd parent exits
leaving that child running.
Is this a viable theory? Should spamd parent perhaps be modified to defer kill
and HUP signals during the respawn process?
Or has this maybe already been done in 3.1.0?
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.