issues with spamd timeouts

[Kelsey Cummings <kg...@corp.sonic.net>]

Since upgrading to 3.4.0 from 3.3.2 we've seen problems with all of the
spamd children blocking (for a yet unknown reason) for long enough that
the clients all give up and retry another server.  Eventually, the spamd
child hits its internal timeout and then goes of and blocks indefinitely
while trying to write to the client (which is gone) that it timed out.
This particular situation can be avoided in part by ensuring that the
spamd timeout is lower than the spamc timeout.

This is quick patch to fix this so service_timeout() is guarded with an
alarm but I don't think it is the only write to a client that needs it.

diff -u /usr/bin/spamd.orig /usr/bin/spamd
--- /usr/bin/spamd.orig 2014-07-01 15:44:43.281790649 -0700
+++ /usr/bin/spamd      2014-07-01 15:50:15.183431230 -0700
@@ -2413,8 +2413,24 @@
 sub service_timeout {
   my ($err) = @_;
   my $resp = "EX_TIMEOUT";
-  print $client "SPAMD/1.0 $resphash{$resp} Timeout: $err\r\n";
   warn("spamd: timeout: $err\n");
+
+  eval {
+    Mail::SpamAssassin::Util::trap_sigalrm_fully(sub {
+                          die "Timeout sending EX_TIMEOUT $err to client";
+                        });
+    alarm 1;
+    print $client "SPAMD/1.0 $resphash{$resp} Timeout: $err\r\n";
+  };
+  alarm 0;
+
+  if ($@) {
+    die "spamd: $@";
+  }
+
+  $client->close;
+
+  return 1;
 }

Re: issues with spamd timeouts

[Kelsey Cummings <kg...@corp.sonic.net>]

On Thu, Jul 03, 2014 at 11:40:49AM +1200, Sidney Markowitz wrote:
> Kelsey Cummings wrote, On 3/07/14 11:21 am:
> > This is quick patch to fix this so service_timeout() is guarded with an
> > alarm but I don't think it is the only write to a client that needs it.
> 
> For any write to client other than the one that tries to tell the client that
> there was a timeout, wouldn't service_timeout() end up being called when the
> spamd times out while blocked waiting for that other write to client? What
> would be the situation with any other write to client that would need a
> similar patch?

Sidney, I only made a brief review of the code path so it is entirely
possible that you're correct and this the only write without a timeout.
Honestly, we're more concerned about what is causing spamd children to
block in the first place but that's proved to be harder to track down.

-- 
Kelsey Cummings - kgc@corp.sonic.net      sonic.net, inc.
System Architect                          2260 Apollo Way
707.522.1000                              Santa Rosa, CA 95407

Re: issues with spamd timeouts

[Sidney Markowitz <si...@sidney.com>]

Kelsey Cummings wrote, On 3/07/14 11:21 am:
> This is quick patch to fix this so service_timeout() is guarded with an
> alarm but I don't think it is the only write to a client that needs it.

For any write to client other than the one that tries to tell the client that
there was a timeout, wouldn't service_timeout() end up being called when the
spamd times out while blocked waiting for that other write to client? What
would be the situation with any other write to client that would need a
similar patch?