turned off mass-checks, fyi

[Theo Van Dinter <fe...@apache.org>]

Just in case there's a question about where my results went...

This week I noticed that my usual run took around 23h to complete, which
is much (2x?) longer than usual.  Poking around, it seems that my second
machine starts running through it's message queue and then stops at some
point, leaving only the first machine to do the processing.

I'm not going to be able to deal with debugging it for a while, so I
decided to just turn off the cronjobs for now and take a look in a few
weeks when I get some time.

-- 
Randomly Selected Tagline:
"a) I want a pony b) My pony's broken! c) A co-worker got a newer and
 faster pony! I want one too!"
         - Person at work describing the standard trouble ticket types

256? (was: Re: turned off mass-checks, fyi)

[Karsten Bräckelmann <gu...@rudersport.de>]

Glad to hear your corpus is back, Theo. :)

> The problem is that ArchiveIterator is erroring out messages > 256k in
> size, regardless of the setting for opt_all.  From what I can tell,

256 kByte? Shouldn't this be like 512 [1] just like spamc, which raised
this limit quite a while ago -- any reason to use a different size here?


[1] err, 500 rather, according to the man pages

-- 
char *t="\10pse\0r\0dtu\0.@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4";
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i<l;i++){ i%8? c<<=1:
(c=*++x); c&128 && (s+=h); if (!(h>>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}

Re: turned off mass-checks, fyi

[Theo Van Dinter <fe...@apache.org>]

The problem is that ArchiveIterator is erroring out messages > 256k in
size, regardless of the setting for opt_all.  From what I can tell,
this was caused by r702559 which moved the size check into the file
read loop, where it doesn't check opt_all to figure out if it should
care about the size.

Fixed, I think, in r721962...   :)

On Sun, Nov 30, 2008 at 05:22:21PM -0500, Theo Van Dinter wrote:
> Now I need to go through and find out why the server has so many errors
> accessing a non-changing corpus. :(

-- 
Randomly Selected Tagline:
"1) Your kid mistakenly gets sent to the principal's office for asking
 the cafeteria ladies if they macerate."
         - From the Top 12 Signs You've Been Watching Too Much FoodTV

Re: turned off mass-checks, fyi

[Theo Van Dinter <fe...@apache.org>]

I finally had some time to look into this.  It appears the issue is that when
the server says it has X messages, but some of them are errors (couldn't be
read, etc,) the client doesn't realize that it actually has less than X
messages to process.  This causes the parent/child mass-check client to
deadlock when the parent reads off the end of the list, doesn't therefore send
a message to the child, and then waits for the child to return a result while
the child continues to wait for a message.

Looking through the code, there are several other potential ways that this
could also happen (lots of "next" and "last" entries while processing the
server's response) beyond just the msg-error entries from the server.  I
submitted r721907 to (hopefully) deal with the issue generically.

Now I need to go through and find out why the server has so many errors
accessing a non-changing corpus. :(


On Fri, Oct 31, 2008 at 12:06:35PM -0400, Theo Van Dinter wrote:
> This week I noticed that my usual run took around 23h to complete, which
> is much (2x?) longer than usual.  Poking around, it seems that my second
> machine starts running through it's message queue and then stops at some
> point, leaving only the first machine to do the processing.
> 
> I'm not going to be able to deal with debugging it for a while, so I
> decided to just turn off the cronjobs for now and take a look in a few
> weeks when I get some time.


-- 
Randomly Selected Tagline:
"Dad, are you okay?  I see food on your plate instead of blurry motions."
         - Lisa on the Simpsons, "Husbands and Knives"