Net checks not running: no output; exit code 141 from mass-check

[Marc Andre Selig <a2...@sedacon.com>]

Hi all,

Yesterday was the second time the weekly --net check has not run for me.
No log files were created at all.

Trying to debug things, I just ran the mass-check command individually
(with the options as assembled by auto-mass-check.sh), but I got no output
at all, with no files being created either, and mass-check exited with
a return code of 141 after maybe two seconds of waiting.

I then tried to run the same mass-check command through the perl
debugger (just prepending perl -d), which seemed to work, so I exited and
re-started without the debugger.  This time, I got "status: starting scan
stage" and then things seemed to work as usual.  (I aborted, assuming
that the results would now be rather late.)  Five further invocations
of mass-check all seemed to work, i.e. none has aborted without any
output, but all have shown "status: starting scan stage" before I aborted
manually.  In short, I cannot now reproduce the errant behaviour described
in the previous paragraph, even though the file system appears unchanged
in comparison to 30 minutes ago when things did not work.

Is there anything else I can do to help debug the problem?

Regards
Marc

Re: Net checks not running: no output; exit code 141 from mass-check

[Axb <ax...@gmail.com>]

On 10/04/2012 12:22 PM, Marc Andre Selig wrote:

> I might be misinterpreting things anyway, as this is not my
> primary line of work.  I've put the strace output online at
> <http://www.sedacon.com/private/masscheck-121004.strace.bz2>, in case
> anybody else wants to have a look.

I admit that straces are usually above myknowledge or patience to google 
thru the results...

gut feeling tells me that possibly IDS, overzealous proxy or some other 
"intruder" in the net path is borking the Razor requests.

Re: Net checks not running: no output; exit code 141 from mass-check

[Marc Andre Selig <a2...@sedacon.com>]

On Mon, Sep 10, 2012 at 12:18:46PM -0400, Kevin A. McGrail wrote:
> On 9/8/2012 7:49 PM, Marc Andre Selig wrote:

> >Yesterday was the second time the weekly --net check has not run for me.
> >No log files were created at all.
> >
> >Trying to debug things, I just ran the mass-check command individually
> >(with the options as assembled by auto-mass-check.sh), but I got no output
> >at all, with no files being created either, and mass-check exited with
> >a return code of 141 after maybe two seconds of waiting.

I still don't receive any --net results.

A manual invocation of mass-check fails the first time around, then
succeeds when called again immediately.  Which makes the error kind of
difficult to debug. ;)

> I don't know what a return code of 141 means and I didn't see
> anything obvious.
> 
> Can you run an strace on the command next time and perhaps that will
> shed more light?

strace shows that mass-check dies after receiving a SIGPIPE signal (well,
141-128=13=SIGPIPE).  If I read things correctly, a cloudmark (razor)
server resets the connection, then mass-check still tries to write
to the corresponding fd, resulting in a broken pipe error and signal.
Sorry, but there's not much time left for me now, so I can't debug this
any further at the moment.

I might be misinterpreting things anyway, as this is not my
primary line of work.  I've put the strace output online at
<http://www.sedacon.com/private/masscheck-121004.strace.bz2>, in case
anybody else wants to have a look.

Regards,
Marc

Re: Net checks not running: no output; exit code 141 from mass-check

["Kevin A. McGrail" <KM...@PCCC.com>]

On 9/8/2012 7:49 PM, Marc Andre Selig wrote:
> Hi all,
>
> Yesterday was the second time the weekly --net check has not run for me.
> No log files were created at all.
>
> Trying to debug things, I just ran the mass-check command individually
> (with the options as assembled by auto-mass-check.sh), but I got no output
> at all, with no files being created either, and mass-check exited with
> a return code of 141 after maybe two seconds of waiting.
>
> I then tried to run the same mass-check command through the perl
> debugger (just prepending perl -d), which seemed to work, so I exited and
> re-started without the debugger.  This time, I got "status: starting scan
> stage" and then things seemed to work as usual.  (I aborted, assuming
> that the results would now be rather late.)  Five further invocations
> of mass-check all seemed to work, i.e. none has aborted without any
> output, but all have shown "status: starting scan stage" before I aborted
> manually.  In short, I cannot now reproduce the errant behaviour described
> in the previous paragraph, even though the file system appears unchanged
> in comparison to 30 minutes ago when things did not work.
>
> Is there anything else I can do to help debug the problem?

I don't know what a return code of 141 means and I didn't see anything 
obvious.

Can you run an strace on the command next time and perhaps that will 
shed more light?

regards,
KAM