Re: turned off mass-checks, fyi

[Theo Van Dinter <fe...@apache.org>]

I finally had some time to look into this.  It appears the issue is that when
the server says it has X messages, but some of them are errors (couldn't be
read, etc,) the client doesn't realize that it actually has less than X
messages to process.  This causes the parent/child mass-check client to
deadlock when the parent reads off the end of the list, doesn't therefore send
a message to the child, and then waits for the child to return a result while
the child continues to wait for a message.

Looking through the code, there are several other potential ways that this
could also happen (lots of "next" and "last" entries while processing the
server's response) beyond just the msg-error entries from the server.  I
submitted r721907 to (hopefully) deal with the issue generically.

Now I need to go through and find out why the server has so many errors
accessing a non-changing corpus. :(


On Fri, Oct 31, 2008 at 12:06:35PM -0400, Theo Van Dinter wrote:
> This week I noticed that my usual run took around 23h to complete, which
> is much (2x?) longer than usual.  Poking around, it seems that my second
> machine starts running through it's message queue and then stops at some
> point, leaving only the first machine to do the processing.
> 
> I'm not going to be able to deal with debugging it for a while, so I
> decided to just turn off the cronjobs for now and take a look in a few
> weeks when I get some time.


-- 
Randomly Selected Tagline:
"Dad, are you okay?  I see food on your plate instead of blurry motions."
         - Lisa on the Simpsons, "Husbands and Knives"

256? (was: Re: turned off mass-checks, fyi)

[Karsten Bräckelmann <gu...@rudersport.de>]

Glad to hear your corpus is back, Theo. :)

> The problem is that ArchiveIterator is erroring out messages > 256k in
> size, regardless of the setting for opt_all.  From what I can tell,

256 kByte? Shouldn't this be like 512 [1] just like spamc, which raised
this limit quite a while ago -- any reason to use a different size here?


[1] err, 500 rather, according to the man pages

-- 
char *t="\10pse\0r\0dtu\0.@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4";
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i<l;i++){ i%8? c<<=1:
(c=*++x); c&128 && (s+=h); if (!(h>>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}

Re: turned off mass-checks, fyi

[Theo Van Dinter <fe...@apache.org>]

The problem is that ArchiveIterator is erroring out messages > 256k in
size, regardless of the setting for opt_all.  From what I can tell,
this was caused by r702559 which moved the size check into the file
read loop, where it doesn't check opt_all to figure out if it should
care about the size.

Fixed, I think, in r721962...   :)

On Sun, Nov 30, 2008 at 05:22:21PM -0500, Theo Van Dinter wrote:
> Now I need to go through and find out why the server has so many errors
> accessing a non-changing corpus. :(

-- 
Randomly Selected Tagline:
"1) Your kid mistakenly gets sent to the principal's office for asking
 the cafeteria ladies if they macerate."
         - From the Top 12 Signs You've Been Watching Too Much FoodTV