You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by James Golick <ja...@gmail.com> on 2010/05/30 10:43:43 UTC
Inconsistency even after nodetool repair?
I was getting a ton of these in my logs:
INFO [pool-1-thread-77] 2010-05-30 03:39:18,263 StorageProxy.java (line
499) DigestMismatchException: Mismatch for key 142667/everything
(5546784193bbe4a4e066dc7fc142f589 vs 03edb87cd131a614d8256ddfdc50dd17)
The nodes were all getting really overloaded, presumably from the
read-repair, since everything else looked healthy. So, I ran nodetool repair
on both of our nodes, but even after running the repairs, I'm still seeing
the same thing in the logs and high load on the nodes.
Does that sound right?
Re: Inconsistency even after nodetool repair?
Posted by Stu Hood <st...@rackspace.com>.
The nodes only begin sending the data after the compactions have finished... https://issues.apache.org/jira/browse/CASSANDRA-579
-----Original Message-----
From: "James Golick" <ja...@gmail.com>
Sent: Monday, May 31, 2010 3:33pm
To: user@cassandra.apache.org
Subject: Re: Inconsistency even after nodetool repair?
It may not have actually finished at that point. Though, according to JMX,
both compactions of each CF had completed, so I assumed it was done.
On Mon, May 31, 2010 at 11:29 AM, Stu Hood <st...@rackspace.com> wrote:
> Did you watch in the logs to confirm that repair had actually finished? The
> `nodetool repair` call is not blocking before 0.6.3 (unreleased): see
> https://issues.apache.org/jira/browse/CASSANDRA-1090
>
> -----Original Message-----
> From: "James Golick" <ja...@gmail.com>
> Sent: Sunday, May 30, 2010 3:43am
> To: cassandra-user@incubator.apache.org
> Subject: Inconsistency even after nodetool repair?
>
> I was getting a ton of these in my logs:
>
> INFO [pool-1-thread-77] 2010-05-30 03:39:18,263 StorageProxy.java (line
> 499) DigestMismatchException: Mismatch for key 142667/everything
> (5546784193bbe4a4e066dc7fc142f589 vs 03edb87cd131a614d8256ddfdc50dd17)
>
> The nodes were all getting really overloaded, presumably from the
> read-repair, since everything else looked healthy. So, I ran nodetool
> repair
> on both of our nodes, but even after running the repairs, I'm still seeing
> the same thing in the logs and high load on the nodes.
>
> Does that sound right?
>
>
>
Re: Inconsistency even after nodetool repair?
Posted by James Golick <ja...@gmail.com>.
It may not have actually finished at that point. Though, according to JMX,
both compactions of each CF had completed, so I assumed it was done.
On Mon, May 31, 2010 at 11:29 AM, Stu Hood <st...@rackspace.com> wrote:
> Did you watch in the logs to confirm that repair had actually finished? The
> `nodetool repair` call is not blocking before 0.6.3 (unreleased): see
> https://issues.apache.org/jira/browse/CASSANDRA-1090
>
> -----Original Message-----
> From: "James Golick" <ja...@gmail.com>
> Sent: Sunday, May 30, 2010 3:43am
> To: cassandra-user@incubator.apache.org
> Subject: Inconsistency even after nodetool repair?
>
> I was getting a ton of these in my logs:
>
> INFO [pool-1-thread-77] 2010-05-30 03:39:18,263 StorageProxy.java (line
> 499) DigestMismatchException: Mismatch for key 142667/everything
> (5546784193bbe4a4e066dc7fc142f589 vs 03edb87cd131a614d8256ddfdc50dd17)
>
> The nodes were all getting really overloaded, presumably from the
> read-repair, since everything else looked healthy. So, I ran nodetool
> repair
> on both of our nodes, but even after running the repairs, I'm still seeing
> the same thing in the logs and high load on the nodes.
>
> Does that sound right?
>
>
>
RE: Inconsistency even after nodetool repair?
Posted by Stu Hood <st...@rackspace.com>.
Did you watch in the logs to confirm that repair had actually finished? The `nodetool repair` call is not blocking before 0.6.3 (unreleased): see https://issues.apache.org/jira/browse/CASSANDRA-1090
-----Original Message-----
From: "James Golick" <ja...@gmail.com>
Sent: Sunday, May 30, 2010 3:43am
To: cassandra-user@incubator.apache.org
Subject: Inconsistency even after nodetool repair?
I was getting a ton of these in my logs:
INFO [pool-1-thread-77] 2010-05-30 03:39:18,263 StorageProxy.java (line
499) DigestMismatchException: Mismatch for key 142667/everything
(5546784193bbe4a4e066dc7fc142f589 vs 03edb87cd131a614d8256ddfdc50dd17)
The nodes were all getting really overloaded, presumably from the
read-repair, since everything else looked healthy. So, I ran nodetool repair
on both of our nodes, but even after running the repairs, I'm still seeing
the same thing in the logs and high load on the nodes.
Does that sound right?