Unavailable exception with CL.ANY

[Rajat Chopra <rc...@makara.com>]

Hi!
   I have a 16 node cluster with two keyspaces and several column families within each, all with RF=2.
All the reads/writes work with all column families but only one of them gives me an unavailable exception, even with CL.ANY consistency. The nodetool ring shows that three of the nodes of the cluster are down (they have their disk full because Cassandra runs out of memory and leaves humongous hprof files).

Anyhow, my question is this - is the unavailable exception being thrown because that particular column family (with its replica) happened to be stored in the three nodes that were down?
Is there a way to see what keys/cfs are stored in which node(s) of the cluster?

Thanks,
Rajat

Re: Unavailable exception with CL.ANY

[Robert Coli <rc...@digg.com>]

On Tue, Dec 14, 2010 at 1:08 AM, Rajat Chopra <rc...@makara.com> wrote:
> What I was keen on knowing was if there exists a way to know which keys
> reside on which node. Like a ‘nodetool column.path’ and it prints the
> nodelist that the column path resides on J. I have HH disabled for some
> other benevolent reason.


"get[Live]NaturalEndpoints" in the JMX interface does this.

http://wiki.apache.org/cassandra/JmxInterface#org.apache.cassandra.service.StorageService.Operations.getNaturalEndpoints

=Rob

RE: Unavailable exception with CL.ANY

[Rajat Chopra <rc...@makara.com>]

Okay. Got that. Thanks.
Bringing back just one of the nodes solved it.

What I was keen on knowing was if there exists a way to know which keys reside on which node. Like a 'nodetool column.path' and it prints the nodelist that the column path resides on :). I have HH disabled for some other benevolent reason.

Thanks for the help.
Rajat


From: aaron morton [mailto:aaron@thelastpickle.com]
Sent: Tuesday, December 14, 2010 12:46 AM
To: user@cassandra.apache.org
Subject: Re: Unavailable exception with CL.ANY

First thing you are going to need is some log messages. From the machine the client was connected to when it returned the UnavailableException.

At CL ANY even Hinted Handoff counts towards meeting the CL for a write http://wiki.apache.org/cassandra/HintedHandoff . So as long as the client can connect to a node in the cluster the write should be accepted. But there could still be reasons why the HH could not be recorded, the logs should help sort it out though.

For a read, with RF 2 and 3 nodes down there may be an entire section of the distributed hash table will be unavailable. So any read for a key that is replicated on 2 of the 3 down nodes will fail. I would advise getting those nodes up asap.

Keys in all CF's are distributed around the cluster according to the partitioner, replication strategy and initial tokens. If you are using the Random Partitioner, Simple Replication Strategy and set evenly spaced initial tokens then keys will be evenly spaced around the cluster.

Hope that helps
Aaron

On 14 Dec 2010, at 21:12, Rajat Chopra wrote:


Hi!
   I have a 16 node cluster with two keyspaces and several column families within each, all with RF=2.
All the reads/writes work with all column families but only one of them gives me an unavailable exception, even with CL.ANY consistency. The nodetool ring shows that three of the nodes of the cluster are down (they have their disk full because Cassandra runs out of memory and leaves humongous hprof files).

Anyhow, my question is this - is the unavailable exception being thrown because that particular column family (with its replica) happened to be stored in the three nodes that were down?
Is there a way to see what keys/cfs are stored in which node(s) of the cluster?

Thanks,
Rajat

Re: Unavailable exception with CL.ANY

[aaron morton <aa...@thelastpickle.com>]

First thing you are going to need is some log messages. From the machine the client was connected to when it returned the UnavailableException.

At CL ANY even Hinted Handoff counts towards meeting the CL for a write http://wiki.apache.org/cassandra/HintedHandoff . So as long as the client can connect to a node in the cluster the write should be accepted. But there could still be reasons why the HH could not be recorded, the logs should help sort it out though.

For a read, with RF 2 and 3 nodes down there may be an entire section of the distributed hash table will be unavailable. So any read for a key that is replicated on 2 of the 3 down nodes will fail. I would advise getting those nodes up asap.

Keys in all CF's are distributed around the cluster according to the partitioner, replication strategy and initial tokens. If you are using the Random Partitioner, Simple Replication Strategy and set evenly spaced initial tokens then keys will be evenly spaced around the cluster. 

Hope that helps
Aaron

On 14 Dec 2010, at 21:12, Rajat Chopra wrote:

> Hi!
>    I have a 16 node cluster with two keyspaces and several column families within each, all with RF=2.
> All the reads/writes work with all column families but only one of them gives me an unavailable exception, even with CL.ANY consistency. The nodetool ring shows that three of the nodes of the cluster are down (they have their disk full because Cassandra runs out of memory and leaves humongous hprof files).
>  
> Anyhow, my question is this – is the unavailable exception being thrown because that particular column family (with its replica) happened to be stored in the three nodes that were down?
> Is there a way to see what keys/cfs are stored in which node(s) of the cluster?
>  
> Thanks,
> Rajat
>