You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Mark <st...@gmail.com> on 2010/08/26 23:53:00 UTC
Repair help
I have a 2 node cluster (testing the waters) w/ a replication factor
of 2. One node got completed screwed up (see any of my previous messages
from today) so I deleted the commit log and data directory. I restarted
the node and rain nodetool repair as describe in
http://wiki.apache.org/cassandra/Operations. I waited for over an hour
and checked my ring only to find that nothing was repaired/replicated???
I only have a mere 7gigs of data so I would have thought this would have
been fairly quick?
Address Status State Load Token
129447565151094499156612104441060791022
x.x.x.x Up Normal 7.31 GB
12949228055906550350782255148181029323
x.x.x.x Up Normal 30.01 MB
129447565151094499156612104441060791022
I tried the alternative method of manually removing the token and then
bootstrapping however when I tried to remove the token via nodetool
removetoken an IllegalStateException was thrown... "replication factor
(2) exceeds number of endpoints (1)"
What should I do in this situation to get my node back up to where it
should be? Is there anywhere I can check that the repair is actually
running?
Thanks for any suggestions
ps I'm using 0.7.0 beta 1
Re: Repair help
Posted by Benjamin Black <b...@b3k.us>.
recommend "testing the waters" on release software (0.6.x), not beta.
On Thu, Aug 26, 2010 at 2:53 PM, Mark <st...@gmail.com> wrote:
> I have a 2 node cluster (testing the waters) w/ a replication factor of 2.
> One node got completed screwed up (see any of my previous messages from
> today) so I deleted the commit log and data directory. I restarted the node
> and rain nodetool repair as describe in
> http://wiki.apache.org/cassandra/Operations. I waited for over an hour and
> checked my ring only to find that nothing was repaired/replicated??? I only
> have a mere 7gigs of data so I would have thought this would have been
> fairly quick?
>
> Address Status State Load Token
>
> 129447565151094499156612104441060791022
> x.x.x.x Up Normal 7.31 GB
> 12949228055906550350782255148181029323
> x.x.x.x Up Normal 30.01 MB
> 129447565151094499156612104441060791022
>
> I tried the alternative method of manually removing the token and then
> bootstrapping however when I tried to remove the token via nodetool
> removetoken an IllegalStateException was thrown... "replication factor (2)
> exceeds number of endpoints (1)"
>
> What should I do in this situation to get my node back up to where it should
> be? Is there anywhere I can check that the repair is actually running?
>
> Thanks for any suggestions
>
> ps I'm using 0.7.0 beta 1
>
>
>
>
Re: Repair help
Posted by Jonathan Ellis <jb...@gmail.com>.
that's not a very useful log level
really you should set it debug when running beta software :)
On Thu, Aug 26, 2010 at 5:43 PM, Mark <st...@gmail.com> wrote:
> On 8/26/10 3:03 PM, Aaron Morton wrote:
>>
>> Check the logs for errors and run nodetool streams to see if it's moving
>> data around.
>>
>> Aaron
>>
>>
>> On 27 Aug, 2010,at 09:53 AM, Mark <st...@gmail.com> wrote:
>>
>>> I have a 2 node cluster (testing the waters) w/ a replication factor
>>> of 2. One node got completed screwed up (see any of my previous messages
>>> from today) so I deleted the commit log and data directory. I restarted
>>> the node and rain nodetool repair as describe in
>>> http://wiki.apache.org/cassandra/Operations. I waited for over an hour
>>> and checked my ring only to find that nothing was repaired/replicated???
>>> I only have a mere 7gigs of data so I would have thought this would have
>>> been fairly quick?
>>>
>>> Address Status State Load Token
>>>
>>> 129447565151094499156612104441060791022
>>> x.x.x.x Up Normal 7.31 GB
>>> 12949228055906550350782255148181029323
>>> x.x.x.x Up Normal 30.01 MB
>>> 129447565151094499156612104441060791022
>>>
>>> I tried the alternative method of manually removing the token and then
>>> bootstrapping however when I tried to remove the token via nodetool
>>> removetoken an IllegalStateException was thrown... "replication factor
>>> (2) exceeds number of endpoints (1)"
>>>
>>> What should I do in this situation to get my node back up to where it
>>> should be? Is there anywhere I can check that the repair is actually
>>> running?
>>>
>>> Thanks for any suggestions
>>>
>>> ps I'm using 0.7.0 beta 1
>>>
>>>
>>>
> Nothing in the logs (log level is set to ERROR though)
>
> Streams:
>
> bin/nodetool --host localhost --port 8080 streams
> Mode: Normal
> Not sending any streams.
> Not receiving any streams.
>
> Any reason in jconsole the forceTableFlush and forceTableRepair are grayed
> out?
>
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com
Re: Repair help
Posted by Mark <st...@gmail.com>.
On 8/26/10 3:03 PM, Aaron Morton wrote:
> Check the logs for errors and run nodetool streams to see if it's
> moving data around.
>
> Aaron
>
>
> On 27 Aug, 2010,at 09:53 AM, Mark <st...@gmail.com> wrote:
>
>> I have a 2 node cluster (testing the waters) w/ a replication factor
>> of 2. One node got completed screwed up (see any of my previous messages
>> from today) so I deleted the commit log and data directory. I restarted
>> the node and rain nodetool repair as describe in
>> http://wiki.apache.org/cassandra/Operations. I waited for over an hour
>> and checked my ring only to find that nothing was repaired/replicated???
>> I only have a mere 7gigs of data so I would have thought this would have
>> been fairly quick?
>>
>> Address Status State Load Token
>>
>> 129447565151094499156612104441060791022
>> x.x.x.x Up Normal 7.31 GB
>> 12949228055906550350782255148181029323
>> x.x.x.x Up Normal 30.01 MB
>> 129447565151094499156612104441060791022
>>
>> I tried the alternative method of manually removing the token and then
>> bootstrapping however when I tried to remove the token via nodetool
>> removetoken an IllegalStateException was thrown... "replication factor
>> (2) exceeds number of endpoints (1)"
>>
>> What should I do in this situation to get my node back up to where it
>> should be? Is there anywhere I can check that the repair is actually
>> running?
>>
>> Thanks for any suggestions
>>
>> ps I'm using 0.7.0 beta 1
>>
>>
>>
Nothing in the logs (log level is set to ERROR though)
Streams:
bin/nodetool --host localhost --port 8080 streams
Mode: Normal
Not sending any streams.
Not receiving any streams.
Any reason in jconsole the forceTableFlush and forceTableRepair are
grayed out?
Re: Repair help
Posted by Aaron Morton <aa...@thelastpickle.com>.
Check the logs for errors and run nodetool streams to see if it's moving data around.
Aaron
On 27 Aug, 2010,at 09:53 AM, Mark <st...@gmail.com> wrote:
I have a 2 node cluster (testing the waters) w/ a replication factor
of 2. One node got completed screwed up (see any of my previous messages
from today) so I deleted the commit log and data directory. I restarted
the node and rain nodetool repair as describe in
http://wiki.apache.org/cassandra/Operations. I waited for over an hour
and checked my ring only to find that nothing was repaired/replicated???
I only have a mere 7gigs of data so I would have thought this would have
been fairly quick?
Address Status State Load Token
129447565151094499156612104441060791022
x.x.x.x Up Normal 7.31 GB
12949228055906550350782255148181029323
x.x.x.x Up Normal 30.01 MB
129447565151094499156612104441060791022
I tried the alternative method of manually removing the token and then
bootstrapping however when I tried to remove the token via nodetool
removetoken an IllegalStateException was thrown... "replication factor
(2) exceeds number of endpoints (1)"
What should I do in this situation to get my node back up to where it
should be? Is there anywhere I can check that the repair is actually
running?
Thanks for any suggestions
ps I'm using 0.7.0 beta 1