Redundant Queue reads was Re: James performance

[Nitin Borwankar <ni...@borwankar.com>]

Hi,

I'm new to this list and to the James architecture but familiar with
these issues in other contexts.
Just thinking off the top off my head here and haven't thought through
the thread issues completely.

Would it make sense to save the indexes of the "retry" messages in an
index collection ?
A subset of the threads could work on the retry messages and the
collection would be synchronized,
threads would read/write/delete to/from the collection or set a done
flag on each retry message successfully
sent.

Also, separately,  a pointer to the first new message in the main queue
would be saved in a static variable that would be synchronized. Then
each thread would start searching at the index so identified, and when
done with its current message would read the static variable with an
intent to update it to point to the actual message it worked on ( note
that the actual message it worked on may be much farther in the queue as
other threads will also be working on the queue ).

Of course if some other thread had updated it to a larger integer then
this thread would do nothing, hence reducing contention on the static
synchronized integer.
The contention would shift to waiting to write to the static integer,
but this would happen far less often.

This would at least save the amount of redundant searching, unless I
have just moved the problem to some other part of the code.

Finally, is the problem an artifact of the Town way of doing things or
is the problems elsewhere ?

Does this make any sense at all in the context of the James architecture
or am I way off base ?

Nitin Borwankar,
nitin@borwankar.com

Serge Knystautas wrote:
> 
> ----- Original Message -----
> From: "John S. Gage" <jg...@epo.som.sunysb.edu>
> 
> > >As for locking, the reason for a lock is to prevent multiple threads from
> > >working on the same message simultaneously.  A thread obtains a lock on a
> > >message while it current works on it....  It doesn't make sense to write
> > >this locking to disk because if the JVM dies, the thread dies, so the
> lock
> > >is irrelevant.  If you had multiple JVMs working against a single queue,
> > >then locking on the disk (or database) would be required.
> >
> > I guess the confusion that I have (and it is undoubtedly my fault) is why
> > *all* the messages in the database have to be checked for their *locked*
> > status if that information is not in the database.  It seems inefficient.
> 
> Yes, it is very inefficient, but here's why (aside from not having enough
> time to optimize this)...
> 
> Say I'm a thread that wants to check the spool for a message, and try to get
> a lock on it if nobody else is accessing.  Here's what the spool does...
> 
> checks the oldest message... nope, some thread has a lock on it
> checks next oldest message... nope, that was a failed delivery attempt, and
> I'm not supposed to try for 1 more hour
> checks next... nope, some thread has a lock on it
> checks next... nope, some thread has a lock on it
> etc...
> 
> More than likely within some number (depending on your # of threads), there
> will be a message that the thread can lock onto.  However, it's entirely
> likely that this isn't the case (you could have 100 threads for instance in
> an extreme case, or a bunch of failed messages in the queue), so you need to
> be able to keep looking through the queue.  Also, it could be make the queue
> smarter about skipping over messages that are marked as failed and not ready
> to be retried.
> 
> Town can let you retrieve the first X records (say 10), but it can't really
> retrieve the next X records.  What I suppose you could do is retrieve the
> first X records (say 10), if that fails, requery the database and retrieve
> the first X times 2 records, etc...  This would require some double
> checking, but it's certainly better than retrieving 5000 records, just to
> guarantee finding an unlocked/ready message.  JDBC would be the best,
> because you could retrieve the first X records, and then use the same query
> set to retrieve the next X records, although some drivers may still be
> retrieving the whole record set anyway.  Any other ideas?
> 
> Serge Knystautas
> Loki Technologies
> http://www.lokitech.com/
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: james-user-help@jakarta.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-user-help@jakarta.apache.org