You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Reverend Chip <re...@gmail.com> on 2010/12/07 01:58:58 UTC
Node goes AWOL briefly; failed replication does not report error
to client, though consistency=ALL
I'm running a big test -- ten nodes with 3T disk each. I'm using
0.7.0rc1. After some tuning help (thanks Tyler) lots of this is working
as it should. However a serious event occurred as well -- the server
froze up -- and though mutations were dropped, no error was reported to
the client. Here's what the log said on host X.19:
WARN [ScheduledTasks:1] 2010-12-06 14:04:11,125 MessagingService.java
(line 527) Dropped 76 MUTATION messages in the last 5000ms
Meanwhile, on the OTHER nodes, gossip decided the node was not available
for a while:
INFO [ScheduledTasks:1] 2010-12-06 14:04:02,396 Gossiper.java (line
195) InetAddress /X.19 is now dead.
INFO [GossipStage:1] 2010-12-06 14:04:06,127 Gossiper.java (line 569)
InetAddress /X.19 is now UP
And despite the fact that I was writing with consistency=ALL, none of my
clients reported any errors on their mutations.
Tyler has this information but I would like to know if anyone has seen
this before, and/or has a diagnosis.
Re: Node goes AWOL briefly; failed replication does not report error
to client, though consistency=ALL
Posted by Reverend Chip <re...@gmail.com>.
On 12/8/2010 7:30 AM, Jonathan Ellis wrote:
> On Tue, Dec 7, 2010 at 4:00 PM, Reverend Chip <re...@gmail.com> wrote:
>> Full DEBUG level logs would be a space problem; I'm loading at least 1T
>> per node (after 3x replication), and these events are rare. Can the
>> DEBUG logs be limited to the specific modules helpful for this diagnosis
>> of the gossip problem and, secondarily, the failure to report
>> replication failure?
> The gossip problem is almost certainly due to a GC pause. You can
> check that by enabling verbose GC logging (uncomment the lines in
> cassandra-env.sh).
Makes sense. I thought full GC events were already logged; I'll see to
the change. Meanwhile, what's the possible remediation if we're up
against GC? Maybe just making gossip more forgiving? Perhaps farming
gossip to a separate process, so frozen JVM can be distinguished from
dead JVM or offline machine.
I wonder whether we're alone in seeing this problem, or if no one else
is noticing, curious, or concerned.
> The replication failure is what we want DEBUG logs for, and
> restricting it to the right modules isn't going to help since when
> you're stress-testing writes, the write modules are going to be 99% of
> the log volume anyway.
>
> Maybe a script to constantly throw away all but the most recent log
> file until you see the WARN line would be sufficient workaround?
OK, that should be workable.
Re: Node goes AWOL briefly; failed replication does not report error
to client, though consistency=ALL
Posted by Jonathan Ellis <jb...@gmail.com>.
On Tue, Dec 7, 2010 at 4:00 PM, Reverend Chip <re...@gmail.com> wrote:
> On 12/7/2010 1:10 PM, Jonathan Ellis wrote:
>> I'm inclined to think there's a bug in your client, then.
>
> That doesn't pass the smell test. The very same client has logged
> timeout and unavailable exceptions on other occasions, e.g. when there
> are too many clients or (in a previous configuration) when the JVMs had
> insufficient memory. It's too much of a coincidence to believe that the
> client's exception reporting happens to fail only at the same time that
> a server experiences unexplained and problematic gossip failures.
You're probably right.
>> DEBUG-level
>> logs could confirm or refute this by logging for each insert how many
>> replicas are being blocked for, which nodes it got responses from, and
>> whether a TimedOutException from not getting ALL replies was returned
>> to the client.
>
> Full DEBUG level logs would be a space problem; I'm loading at least 1T
> per node (after 3x replication), and these events are rare. Can the
> DEBUG logs be limited to the specific modules helpful for this diagnosis
> of the gossip problem and, secondarily, the failure to report
> replication failure?
The gossip problem is almost certainly due to a GC pause. You can
check that by enabling verbose GC logging (uncomment the lines in
cassandra-env.sh).
The replication failure is what we want DEBUG logs for, and
restricting it to the right modules isn't going to help since when
you're stress-testing writes, the write modules are going to be 99% of
the log volume anyway.
Maybe a script to constantly throw away all but the most recent log
file until you see the WARN line would be sufficient workaround?
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com
Re: Node goes AWOL briefly; failed replication does not report error
to client, though consistency=ALL
Posted by Reverend Chip <re...@gmail.com>.
On 12/7/2010 1:10 PM, Jonathan Ellis wrote:
> I'm inclined to think there's a bug in your client, then.
That doesn't pass the smell test. The very same client has logged
timeout and unavailable exceptions on other occasions, e.g. when there
are too many clients or (in a previous configuration) when the JVMs had
insufficient memory. It's too much of a coincidence to believe that the
client's exception reporting happens to fail only at the same time that
a server experiences unexplained and problematic gossip failures.
> DEBUG-level
> logs could confirm or refute this by logging for each insert how many
> replicas are being blocked for, which nodes it got responses from, and
> whether a TimedOutException from not getting ALL replies was returned
> to the client.
Full DEBUG level logs would be a space problem; I'm loading at least 1T
per node (after 3x replication), and these events are rare. Can the
DEBUG logs be limited to the specific modules helpful for this diagnosis
of the gossip problem and, secondarily, the failure to report
replication failure?
> On Tue, Dec 7, 2010 at 2:37 PM, Reverend Chip <re...@gmail.com> wrote:
>> No, I'm afraid that's not it:
>> replica_placement_strategy: org.apache.cassandra.locator.SimpleStrategy
>> replication_factor: 3
>>
>> On 12/7/2010 6:37 AM, Jonathan Ellis wrote:
>>> If you are using NetworkTopologyStrategy you are probably hitting
>>> https://issues.apache.org/jira/browse/CASSANDRA-1804 which is fixed in
>>> rc2.
>>>
>>> On Mon, Dec 6, 2010 at 6:58 PM, Reverend Chip <re...@gmail.com> wrote:
>>>> I'm running a big test -- ten nodes with 3T disk each. I'm using
>>>> 0.7.0rc1. After some tuning help (thanks Tyler) lots of this is working
>>>> as it should. However a serious event occurred as well -- the server
>>>> froze up -- and though mutations were dropped, no error was reported to
>>>> the client. Here's what the log said on host X.19:
>>>>
>>>> WARN [ScheduledTasks:1] 2010-12-06 14:04:11,125 MessagingService.java
>>>> (line 527) Dropped 76 MUTATION messages in the last 5000ms
>>>>
>>>> Meanwhile, on the OTHER nodes, gossip decided the node was not available
>>>> for a while:
>>>>
>>>> INFO [ScheduledTasks:1] 2010-12-06 14:04:02,396 Gossiper.java (line
>>>> 195) InetAddress /X.19 is now dead.
>>>> INFO [GossipStage:1] 2010-12-06 14:04:06,127 Gossiper.java (line 569)
>>>> InetAddress /X.19 is now UP
>>>>
>>>> And despite the fact that I was writing with consistency=ALL, none of my
>>>> clients reported any errors on their mutations.
>>>>
>>>> Tyler has this information but I would like to know if anyone has seen
>>>> this before, and/or has a diagnosis.
>>>>
>>>>
>>>
>>
>
>
Re: Node goes AWOL briefly; failed replication does not report error
to client, though consistency=ALL
Posted by Jonathan Ellis <jb...@gmail.com>.
I'm inclined to think there's a bug in your client, then. DEBUG-level
logs could confirm or refute this by logging for each insert how many
replicas are being blocked for, which nodes it got responses from, and
whether a TimedOutException from not getting ALL replies was returned
to the client.
On Tue, Dec 7, 2010 at 2:37 PM, Reverend Chip <re...@gmail.com> wrote:
> No, I'm afraid that's not it:
> replica_placement_strategy: org.apache.cassandra.locator.SimpleStrategy
> replication_factor: 3
>
> On 12/7/2010 6:37 AM, Jonathan Ellis wrote:
>> If you are using NetworkTopologyStrategy you are probably hitting
>> https://issues.apache.org/jira/browse/CASSANDRA-1804 which is fixed in
>> rc2.
>>
>> On Mon, Dec 6, 2010 at 6:58 PM, Reverend Chip <re...@gmail.com> wrote:
>>> I'm running a big test -- ten nodes with 3T disk each. I'm using
>>> 0.7.0rc1. After some tuning help (thanks Tyler) lots of this is working
>>> as it should. However a serious event occurred as well -- the server
>>> froze up -- and though mutations were dropped, no error was reported to
>>> the client. Here's what the log said on host X.19:
>>>
>>> WARN [ScheduledTasks:1] 2010-12-06 14:04:11,125 MessagingService.java
>>> (line 527) Dropped 76 MUTATION messages in the last 5000ms
>>>
>>> Meanwhile, on the OTHER nodes, gossip decided the node was not available
>>> for a while:
>>>
>>> INFO [ScheduledTasks:1] 2010-12-06 14:04:02,396 Gossiper.java (line
>>> 195) InetAddress /X.19 is now dead.
>>> INFO [GossipStage:1] 2010-12-06 14:04:06,127 Gossiper.java (line 569)
>>> InetAddress /X.19 is now UP
>>>
>>> And despite the fact that I was writing with consistency=ALL, none of my
>>> clients reported any errors on their mutations.
>>>
>>> Tyler has this information but I would like to know if anyone has seen
>>> this before, and/or has a diagnosis.
>>>
>>>
>>
>>
>
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com
Re: Node goes AWOL briefly; failed replication does not report error
to client, though consistency=ALL
Posted by Reverend Chip <re...@gmail.com>.
No, I'm afraid that's not it:
replica_placement_strategy: org.apache.cassandra.locator.SimpleStrategy
replication_factor: 3
On 12/7/2010 6:37 AM, Jonathan Ellis wrote:
> If you are using NetworkTopologyStrategy you are probably hitting
> https://issues.apache.org/jira/browse/CASSANDRA-1804 which is fixed in
> rc2.
>
> On Mon, Dec 6, 2010 at 6:58 PM, Reverend Chip <re...@gmail.com> wrote:
>> I'm running a big test -- ten nodes with 3T disk each. I'm using
>> 0.7.0rc1. After some tuning help (thanks Tyler) lots of this is working
>> as it should. However a serious event occurred as well -- the server
>> froze up -- and though mutations were dropped, no error was reported to
>> the client. Here's what the log said on host X.19:
>>
>> WARN [ScheduledTasks:1] 2010-12-06 14:04:11,125 MessagingService.java
>> (line 527) Dropped 76 MUTATION messages in the last 5000ms
>>
>> Meanwhile, on the OTHER nodes, gossip decided the node was not available
>> for a while:
>>
>> INFO [ScheduledTasks:1] 2010-12-06 14:04:02,396 Gossiper.java (line
>> 195) InetAddress /X.19 is now dead.
>> INFO [GossipStage:1] 2010-12-06 14:04:06,127 Gossiper.java (line 569)
>> InetAddress /X.19 is now UP
>>
>> And despite the fact that I was writing with consistency=ALL, none of my
>> clients reported any errors on their mutations.
>>
>> Tyler has this information but I would like to know if anyone has seen
>> this before, and/or has a diagnosis.
>>
>>
>
>
Re: Can super column family use column_metadata?
Posted by Jonathan Ellis <jb...@gmail.com>.
On Tue, Dec 7, 2010 at 4:28 PM, Donal Zang <za...@ihep.ac.cn> wrote:
> Hi,
> I'm using 0.7.0-rc1,and when I use cassandra-cli to create a column family
> with metadata, I got "null",and no column family is created.
> The command I use is:
>
> create keyspace test;
> use test;
> create column family test1 with column_type = 'Super' and comparator =
> 'LongType' and column_metadata =[{column_name:a,validation_class:LongType}];
Looks like a bug in the CLI:
https://issues.apache.org/jira/browse/CASSANDRA-1835
> And also the examples gived by "help create column family;" won't work!
The first two work fine for me. The third has an column name that
conflicts with the comparator definition; I've fixed that in r1043274.
This works:
create column family Foo with comparator=UTF8Type and
column_metadata=[{ column_name:test, validation_class:IntegerType,
index_type:0, index_name:TextIdx}, { column_name:'other name',
validation_class:LongType }];
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com
Can super column family use column_metadata?
Posted by Donal Zang <za...@ihep.ac.cn>.
Hi,
I'm using 0.7.0-rc1,and when I use cassandra-cli to create a column
family with metadata, I got "null",and no column family is created.
The command I use is:
/create keyspace test;
use test;
create column family test1 with column_type = 'Super' and comparator =
'LongType' and column_metadata
=[{column_name:a,validation_class:LongType}];/
And also the examples gived by "help create column family;" won't work!
Any ideas?
Thanks!
Re: Node goes AWOL briefly; failed replication does not report error
to client, though consistency=ALL
Posted by Jonathan Ellis <jb...@gmail.com>.
If you are using NetworkTopologyStrategy you are probably hitting
https://issues.apache.org/jira/browse/CASSANDRA-1804 which is fixed in
rc2.
On Mon, Dec 6, 2010 at 6:58 PM, Reverend Chip <re...@gmail.com> wrote:
> I'm running a big test -- ten nodes with 3T disk each. I'm using
> 0.7.0rc1. After some tuning help (thanks Tyler) lots of this is working
> as it should. However a serious event occurred as well -- the server
> froze up -- and though mutations were dropped, no error was reported to
> the client. Here's what the log said on host X.19:
>
> WARN [ScheduledTasks:1] 2010-12-06 14:04:11,125 MessagingService.java
> (line 527) Dropped 76 MUTATION messages in the last 5000ms
>
> Meanwhile, on the OTHER nodes, gossip decided the node was not available
> for a while:
>
> INFO [ScheduledTasks:1] 2010-12-06 14:04:02,396 Gossiper.java (line
> 195) InetAddress /X.19 is now dead.
> INFO [GossipStage:1] 2010-12-06 14:04:06,127 Gossiper.java (line 569)
> InetAddress /X.19 is now UP
>
> And despite the fact that I was writing with consistency=ALL, none of my
> clients reported any errors on their mutations.
>
> Tyler has this information but I would like to know if anyone has seen
> this before, and/or has a diagnosis.
>
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com