You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Oleg Proudnikov <ol...@cloudorange.com> on 2011/01/20 17:47:28 UTC
Lost MUTATIONS on several Cassandra nodes - no impact on the client
Hi All,
Could you please help me understand the impact of this behaviour?
I am running a 6 node 0.7-rc4 Cassandra cluster with RF=2
6 Hector clients (one per node) are performing single-threaded batch load
running on the same servers. CL=ONE.
Client performs one simple small query and an insert batch mutation. Each
mutation inserts several dozen columns into 7 column families. Total amount of
data is 10-20KB. It appears that this load is a little bit heavy for the cluster
to handle. I do get an occasional single node OOM.
ISSUE. I see periodic lost mutations on some nodes as shown below. The client
does not receive an exception and the nodes do not go down.
xxx.xxx.xxx.140 grep MUTA log/cassandra.log
xxx.xxx.xxx.141 grep MUTA log/cassandra.log
WARN [ScheduledTasks:1] 2011-01-18 13:19:03,918 MessagingService.java (line
545) Dropped 227 MUTATION messages in the last 5000ms
WARN [ScheduledTasks:1] 2011-01-18 13:19:08,924 MessagingService.java (line
545) Dropped 958 MUTATION messages in the last 5000ms
WARN [ScheduledTasks:1] 2011-01-18 13:52:37,616 MessagingService.java (line
545) Dropped 542 MUTATION messages in the last 5000ms
WARN [ScheduledTasks:1] 2011-01-18 16:02:27,787 MessagingService.java (line
545) Dropped 273 MUTATION messages in the last 5000ms
xxx.xxx.xxx.142 grep MUTA log/cassandra.log
WARN [ScheduledTasks:1] 2011-01-17 19:19:06,825 MessagingService.java (line
545) Dropped 699 MUTATION messages in the last 5000ms
WARN [ScheduledTasks:1] 2011-01-17 19:19:06,860 MessagingService.java (line
545) Dropped 10 READ messages in the last 5000ms
WARN [ScheduledTasks:1] 2011-01-18 04:01:05,464 MessagingService.java (line
545) Dropped 89 MUTATION messages in the last 5000ms
xxx.xxx.xxx.143 grep MUTA log/cassandra.log
xxx.xxx.xxx.144 grep MUTA log/cassandra.log
xxx.xxx.xxx.145 grep MUTA log/cassandra.log
Q1. Is it possible that Cassandra will drop both replicas for a given column
during these losses? Or does it guarantee that one replica is still written?
Q2. What does the lack of client exception mean? Does it tell me that at least
one replica is written?
Q3. If I were to use CL=ALL, would I get an exception(s) on the client(s) for
those losses?
Q2. Considering that I did not get an exception I will assume that one replica
is retained. Now, if the nodes stay up and the load on the cluster goes down,
will Cassandra attempt to create 2nd replica? Or will the 2nd replica be created
on a read? Is there a way to recreate lost replicas in batch mode?
Thank you very much,
Oleg
Re: Lost MUTATIONS on several Cassandra nodes - no impact on the client
Posted by ruslan usifov <ru...@gmail.com>.
2011/1/24 Jonathan Ellis <jb...@gmail.com>
> Currently it does not.
>
> And it is planned to make? And when? Or it not an important feature?
Re: Lost MUTATIONS on several Cassandra nodes - no impact on the client
Posted by Jonathan Ellis <jb...@gmail.com>.
Currently it does not.
On Mon, Jan 24, 2011 at 10:35 AM, ruslan usifov <ru...@gmail.com> wrote:
> Thanks for very useful explanations, i have a last question about
> batch_mutate method, does it guarantee that if any mutation in mutations
> list will throw UnavailableException, all batch will throw
> UnavailableException, and nothing is writing to cluster?
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com
Re: Lost MUTATIONS on several Cassandra nodes - no impact on the client
Posted by ruslan usifov <ru...@gmail.com>.
2011/1/23 Edward Capriolo <ed...@gmail.com>
> On Sun, Jan 23, 2011 at 11:23 AM, ruslan usifov <ru...@gmail.com>
> wrote:
> >
> >
> >
> >> On Sun, Jan 23, 2011 at 6:30 AM, ruslan usifov <ruslan.usifov@gmail.com
> >
> >> wrote:
> >>
> >> Right. The difference is that the gossip process builds a topology of
> >> UP/DOWN hosts so Unavailable is thrown quickly. If you need ALL and
> >> one replica is known down -> Unavailable.
> >>
> >
> > Is it possible to detect that write doesn't happen anywhere? Or it is
> only
> > possible detect consistency failure?
> >
>
> Regardless of what Exception is thrown you should retry the write from
> your client.
>
> If the method threw UnavailableException the write operation did not
> happen on any node. As the coordinator judged that it would have not
> succeeded.
>
> If the method threw TimedOutException the write could have succeeded
> on some nodes but it was not acknowledged on enough to meet the CL
> your requested.
>
> It would be nice if the exception could be populated more information such
> as
> TimedOutException:
> requested: 3
> succeeded: 1
> succeededList: 127.0.0.1
> requestedList: 127.0.0.1,127.0.0.2,127.0.0.3
>
> This would make the explanations more self explanatory, and would give
> more transparency to the clients.
>
> (knowing my luck thrift probably does not allow complex exception types)
>
Thanks for very useful explanations, i have a last question about
batch_mutate method, does it guarantee that if any mutation in mutations
list will throw UnavailableException, all batch will throw
UnavailableException, and nothing is writing to cluster?
Re: Lost MUTATIONS on several Cassandra nodes - no impact on the client
Posted by Edward Capriolo <ed...@gmail.com>.
On Sun, Jan 23, 2011 at 11:23 AM, ruslan usifov <ru...@gmail.com> wrote:
>
>
>
>> On Sun, Jan 23, 2011 at 6:30 AM, ruslan usifov <ru...@gmail.com>
>> wrote:
>>
>> Right. The difference is that the gossip process builds a topology of
>> UP/DOWN hosts so Unavailable is thrown quickly. If you need ALL and
>> one replica is known down -> Unavailable.
>>
>
> Is it possible to detect that write doesn't happen anywhere? Or it is only
> possible detect consistency failure?
>
Regardless of what Exception is thrown you should retry the write from
your client.
If the method threw UnavailableException the write operation did not
happen on any node. As the coordinator judged that it would have not
succeeded.
If the method threw TimedOutException the write could have succeeded
on some nodes but it was not acknowledged on enough to meet the CL
your requested.
It would be nice if the exception could be populated more information such as
TimedOutException:
requested: 3
succeeded: 1
succeededList: 127.0.0.1
requestedList: 127.0.0.1,127.0.0.2,127.0.0.3
This would make the explanations more self explanatory, and would give
more transparency to the clients.
(knowing my luck thrift probably does not allow complex exception types)
Re: Lost MUTATIONS on several Cassandra nodes - no impact on the client
Posted by ruslan usifov <ru...@gmail.com>.
On Sun, Jan 23, 2011 at 6:30 AM, ruslan usifov <ru...@gmail.com>
> wrote:
>
> Right. The difference is that the gossip process builds a topology of
> UP/DOWN hosts so Unavailable is thrown quickly. If you need ALL and
> one replica is known down -> Unavailable.
>
>
Is it possible to detect that write doesn't happen anywhere? Or it is only
possible detect consistency failure?
Re: Lost MUTATIONS on several Cassandra nodes - no impact on the client
Posted by Edward Capriolo <ed...@gmail.com>.
On Sun, Jan 23, 2011 at 6:30 AM, ruslan usifov <ru...@gmail.com> wrote:
>
>
> 2011/1/20 Jonathan Ellis <jb...@gmail.com>
>>
>> It guarantees that if the requested ConsistencyLevel is not achieved,
>> client will get a TimedOutException, which is a signal you need to add
>> capacity to handle what you are throwing at the cluster.
>>
> Sorry and when UnavailableException is thows? When data can't be saved
> anywhere?
>
Right. The difference is that the gossip process builds a topology of
UP/DOWN hosts so Unavailable is thrown quickly. If you need ALL and
one replica is known down -> Unavailable.
However if the coordinator believe the node was UP and the request
took longer then RCPTimeout (default 10,000,000 ms) ->
TimedOutException
Re: Lost MUTATIONS on several Cassandra nodes - no impact on the client
Posted by ruslan usifov <ru...@gmail.com>.
2011/1/20 Jonathan Ellis <jb...@gmail.com>
>
> It guarantees that if the requested ConsistencyLevel is not achieved,
> client will get a TimedOutException, which is a signal you need to add
> capacity to handle what you are throwing at the cluster.
>
> Sorry and when UnavailableException is thows? When data can't be saved
anywhere?
Re: Lost MUTATIONS on several Cassandra nodes - no impact on the client
Posted by Jonathan Ellis <jb...@gmail.com>.
On Thu, Jan 20, 2011 at 10:47 AM, Oleg Proudnikov <ol...@cloudorange.com> wrote:
> Q1. Is it possible that Cassandra will drop both replicas for a given column
> during these losses? Or does it guarantee that one replica is still written?
It guarantees that if the requested ConsistencyLevel is not achieved,
client will get a TimedOutException, which is a signal you need to add
capacity to handle what you are throwing at the cluster.
> Q2. What does the lack of client exception mean? Does it tell me that at least
> one replica is written?
As above.
> Q3. If I were to use CL=ALL, would I get an exception(s) on the client(s) for
> those losses?
Yes.
> Q2. Considering that I did not get an exception I will assume that one replica
> is retained. Now, if the nodes stay up and the load on the cluster goes down,
> will Cassandra attempt to create 2nd replica? Or will the 2nd replica be created
> on a read? Is there a way to recreate lost replicas in batch mode?
http://wiki.apache.org/cassandra/Operations#Repairing_missing_or_inconsistent_data
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com