Modifying Cassandra's threadpool queues

[Waleed Reda <w....@gmail.com>]

0down votefavorite
<http://stackoverflow.com/questions/34763773/modifying-cassandras-threadpool-queues?noredirect=1#>

I've been meddling with Cassandra's (v 2.2.4) threadpool executors (namely
SEPExecutor.java module) and trying to change the queues used for storing
pending reads (that have no immediately available workers to serve). By
default, Cassandra uses a ConcurrentLinkedQueue (which is a non-blocking
queue variant). I'm currently trying to override this with a MultiQueue
setup in order to schedule requests in non-FIFO order.

Lets assume for simplicity that my MultiQueue implementation is an
extension of AbstractQueue that simply overrides the offer and poll
functions and randomly (de)queues requests to any of the enclosed
ConcurrentLinkedQueues. For polling, if one queue returns null, we
basically keep going through all the queues until we find a non-null
element (otherwise we return null). There's no locking mechanism in place
since my intention is to utilize the properties of the enclosed
ConcurrentLinkedQueues (which are non-blocking).

The main problem is that it seems I'm running into some sort of race
condition, where some of the assigned workers can't poll an item that
supposedly exists in the queue. In other words, the MultiQueue structure
appears to be non-linearizable. More specifically, I'm encountering a
NullPointerException on this line: SEPWorker.java [line 105]
<https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/concurrent/SEPWorker.java#L105>

Any clue as to what could be causing this, or how should I go about
maintaining the properties of a single ConcurrentLinkedQueue in a
MultiQueue setup?

Re: Modifying Cassandra's threadpool queues

[Benedict Elliott Smith <be...@datastax.com>]

The SEPWorker manages its own count of available items that have been
enqueued (but not yet dequeued), so it dequeues knowing there should be an
item available; if there is not, your queue is swallowing them, either
temporarily or permanently.

So, in all likelihood (as you suggest), you simply have a bug in your queue
and you should apply your regular techniques for figuring out where it is.
The problem isn't really Cassandra related, at least not yet.  My approach
is typically isolated randomised testing - you can see some simple examples
here:
https://github.com/belliottsmith/bes-utils/tree/master/test/bes/concurrent/collections



On Wed, Jan 13, 2016 at 12:41 PM, Waleed Reda <w....@gmail.com>
wrote:

> 0down votefavorite
> <
> http://stackoverflow.com/questions/34763773/modifying-cassandras-threadpool-queues?noredirect=1#
> >
>
> I've been meddling with Cassandra's (v 2.2.4) threadpool executors (namely
> SEPExecutor.java module) and trying to change the queues used for storing
> pending reads (that have no immediately available workers to serve). By
> default, Cassandra uses a ConcurrentLinkedQueue (which is a non-blocking
> queue variant). I'm currently trying to override this with a MultiQueue
> setup in order to schedule requests in non-FIFO order.
>
> Lets assume for simplicity that my MultiQueue implementation is an
> extension of AbstractQueue that simply overrides the offer and poll
> functions and randomly (de)queues requests to any of the enclosed
> ConcurrentLinkedQueues. For polling, if one queue returns null, we
> basically keep going through all the queues until we find a non-null
> element (otherwise we return null). There's no locking mechanism in place
> since my intention is to utilize the properties of the enclosed
> ConcurrentLinkedQueues (which are non-blocking).
>
> The main problem is that it seems I'm running into some sort of race
> condition, where some of the assigned workers can't poll an item that
> supposedly exists in the queue. In other words, the MultiQueue structure
> appears to be non-linearizable. More specifically, I'm encountering a
> NullPointerException on this line: SEPWorker.java [line 105]
> <
> https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/concurrent/SEPWorker.java#L105
> >
>
> Any clue as to what could be causing this, or how should I go about
> maintaining the properties of a single ConcurrentLinkedQueue in a
> MultiQueue setup?
>