You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Anshu Vajpayee <an...@gmail.com> on 2017/11/13 03:54:10 UTC
Node Failure Scenario
Hi All ,
There was a node failure in one of production cluster due to disk failure.
After h/w recovery that node is noew ready be part of cluster, but it
doesn't has any data due to disk crash.
I can think of following option :
1. replace the node with same. using replace_address
2. Set bootstrap=false , start the node and run the repair to stream the
data.
Please suggest if both option are good and which is best as per your
experience. This is live production cluster.
Thanks,
--
*C*heers,*
*Anshu V*
Re: Node Failure Scenario
Posted by Anshu Vajpayee <an...@gmail.com>.
Thank you Jonathan and all.
On Tue, Nov 14, 2017 at 10:53 PM, Jonathan Haddad <jo...@jonhaddad.com> wrote:
> Anthony’s suggestions using replace_address_first_boot lets you avoid that
> requirement, and it’s specifically why it was added in 2.2.
> On Tue, Nov 14, 2017 at 1:02 AM Anshu Vajpayee <an...@gmail.com>
> wrote:
>
>> Thanks guys ,
>>
>> I thikn better to pass replace_address on command line rather than update
>> the cassndra-env file so that there would not be requirement to remove it
>> later.
>>
>>
>> On Tue, Nov 14, 2017 at 6:32 AM, Anthony Grasso <anthony.grasso@gmail.com
>> > wrote:
>>
>>> Hi Anshu,
>>>
>>> To add to Erick's comment, remember to remove the *replace_address* method
>>> from the *cassandra-env.sh* file once the node has rejoined
>>> successfully. The node will fail the next restart otherwise.
>>>
>>> Alternatively, use the *replace_address_first_boot* method which works
>>> exactly the same way as *replace_address* the only difference is there
>>> is no need to remove it from the *cassandra-env.sh* file.
>>>
>>> Kind regards,
>>> Anthony
>>>
>>> On 13 November 2017 at 14:59, Erick Ramirez <fl...@gmail.com>
>>> wrote:
>>>
>>>> Use the replace_address method with its own IP address. Make sure you
>>>> delete the contents of the following directories:
>>>> - data/
>>>> - commitlog/
>>>> - saved_caches/
>>>>
>>>> Forget rejoining with repair -- it will just cause more problems.
>>>> Cheers!
>>>>
>>>> On Mon, Nov 13, 2017 at 2:54 PM, Anshu Vajpayee <
>>>> anshu.vajpayee@gmail.com> wrote:
>>>>
>>>>> Hi All ,
>>>>>
>>>>> There was a node failure in one of production cluster due to disk
>>>>> failure. After h/w recovery that node is noew ready be part of cluster,
>>>>> but it doesn't has any data due to disk crash.
>>>>>
>>>>>
>>>>>
>>>>> I can think of following option :
>>>>>
>>>>>
>>>>>
>>>>> 1. replace the node with same. using replace_address
>>>>>
>>>>> 2. Set bootstrap=false , start the node and run the repair to stream
>>>>> the data.
>>>>>
>>>>>
>>>>>
>>>>> Please suggest if both option are good and which is best as per your
>>>>> experience. This is live production cluster.
>>>>>
>>>>>
>>>>> Thanks,
>>>>>
>>>>>
>>>>> --
>>>>> *C*heers,*
>>>>> *Anshu V*
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>>
>> --
>> *C*heers,*
>> *Anshu V*
>>
>>
>>
--
*C*heers,*
*Anshu V*
Re: Node Failure Scenario
Posted by Jonathan Haddad <jo...@jonhaddad.com>.
Anthony’s suggestions using replace_address_first_boot lets you avoid that
requirement, and it’s specifically why it was added in 2.2.
On Tue, Nov 14, 2017 at 1:02 AM Anshu Vajpayee <an...@gmail.com>
wrote:
> Thanks guys ,
>
> I thikn better to pass replace_address on command line rather than update
> the cassndra-env file so that there would not be requirement to remove it
> later.
>
>
> On Tue, Nov 14, 2017 at 6:32 AM, Anthony Grasso <an...@gmail.com>
> wrote:
>
>> Hi Anshu,
>>
>> To add to Erick's comment, remember to remove the *replace_address* method
>> from the *cassandra-env.sh* file once the node has rejoined
>> successfully. The node will fail the next restart otherwise.
>>
>> Alternatively, use the *replace_address_first_boot* method which works
>> exactly the same way as *replace_address* the only difference is there
>> is no need to remove it from the *cassandra-env.sh* file.
>>
>> Kind regards,
>> Anthony
>>
>> On 13 November 2017 at 14:59, Erick Ramirez <fl...@gmail.com> wrote:
>>
>>> Use the replace_address method with its own IP address. Make sure you
>>> delete the contents of the following directories:
>>> - data/
>>> - commitlog/
>>> - saved_caches/
>>>
>>> Forget rejoining with repair -- it will just cause more problems. Cheers!
>>>
>>> On Mon, Nov 13, 2017 at 2:54 PM, Anshu Vajpayee <
>>> anshu.vajpayee@gmail.com> wrote:
>>>
>>>> Hi All ,
>>>>
>>>> There was a node failure in one of production cluster due to disk
>>>> failure. After h/w recovery that node is noew ready be part of cluster,
>>>> but it doesn't has any data due to disk crash.
>>>>
>>>>
>>>>
>>>> I can think of following option :
>>>>
>>>>
>>>>
>>>> 1. replace the node with same. using replace_address
>>>>
>>>> 2. Set bootstrap=false , start the node and run the repair to stream
>>>> the data.
>>>>
>>>>
>>>>
>>>> Please suggest if both option are good and which is best as per your
>>>> experience. This is live production cluster.
>>>>
>>>>
>>>> Thanks,
>>>>
>>>>
>>>> --
>>>> *C*heers,*
>>>> *Anshu V*
>>>>
>>>>
>>>>
>>>
>>
>
>
> --
> *C*heers,*
> *Anshu V*
>
>
>
Re: Node Failure Scenario
Posted by Anshu Vajpayee <an...@gmail.com>.
Thanks guys ,
I thikn better to pass replace_address on command line rather than update
the cassndra-env file so that there would not be requirement to remove it
later.
On Tue, Nov 14, 2017 at 6:32 AM, Anthony Grasso <an...@gmail.com>
wrote:
> Hi Anshu,
>
> To add to Erick's comment, remember to remove the *replace_address* method
> from the *cassandra-env.sh* file once the node has rejoined successfully.
> The node will fail the next restart otherwise.
>
> Alternatively, use the *replace_address_first_boot* method which works
> exactly the same way as *replace_address* the only difference is there is
> no need to remove it from the *cassandra-env.sh* file.
>
> Kind regards,
> Anthony
>
> On 13 November 2017 at 14:59, Erick Ramirez <fl...@gmail.com> wrote:
>
>> Use the replace_address method with its own IP address. Make sure you
>> delete the contents of the following directories:
>> - data/
>> - commitlog/
>> - saved_caches/
>>
>> Forget rejoining with repair -- it will just cause more problems. Cheers!
>>
>> On Mon, Nov 13, 2017 at 2:54 PM, Anshu Vajpayee <anshu.vajpayee@gmail.com
>> > wrote:
>>
>>> Hi All ,
>>>
>>> There was a node failure in one of production cluster due to disk
>>> failure. After h/w recovery that node is noew ready be part of cluster,
>>> but it doesn't has any data due to disk crash.
>>>
>>>
>>>
>>> I can think of following option :
>>>
>>>
>>>
>>> 1. replace the node with same. using replace_address
>>>
>>> 2. Set bootstrap=false , start the node and run the repair to stream the
>>> data.
>>>
>>>
>>>
>>> Please suggest if both option are good and which is best as per your
>>> experience. This is live production cluster.
>>>
>>>
>>> Thanks,
>>>
>>>
>>> --
>>> *C*heers,*
>>> *Anshu V*
>>>
>>>
>>>
>>
>
--
*C*heers,*
*Anshu V*
Re: Node Failure Scenario
Posted by Anthony Grasso <an...@gmail.com>.
Hi Anshu,
To add to Erick's comment, remember to remove the *replace_address* method
from the *cassandra-env.sh* file once the node has rejoined successfully.
The node will fail the next restart otherwise.
Alternatively, use the *replace_address_first_boot* method which works
exactly the same way as *replace_address* the only difference is there is
no need to remove it from the *cassandra-env.sh* file.
Kind regards,
Anthony
On 13 November 2017 at 14:59, Erick Ramirez <fl...@gmail.com> wrote:
> Use the replace_address method with its own IP address. Make sure you
> delete the contents of the following directories:
> - data/
> - commitlog/
> - saved_caches/
>
> Forget rejoining with repair -- it will just cause more problems. Cheers!
>
> On Mon, Nov 13, 2017 at 2:54 PM, Anshu Vajpayee <an...@gmail.com>
> wrote:
>
>> Hi All ,
>>
>> There was a node failure in one of production cluster due to disk
>> failure. After h/w recovery that node is noew ready be part of cluster,
>> but it doesn't has any data due to disk crash.
>>
>>
>>
>> I can think of following option :
>>
>>
>>
>> 1. replace the node with same. using replace_address
>>
>> 2. Set bootstrap=false , start the node and run the repair to stream the
>> data.
>>
>>
>>
>> Please suggest if both option are good and which is best as per your
>> experience. This is live production cluster.
>>
>>
>> Thanks,
>>
>>
>> --
>> *C*heers,*
>> *Anshu V*
>>
>>
>>
>
Re: Node Failure Scenario
Posted by Erick Ramirez <fl...@gmail.com>.
Use the replace_address method with its own IP address. Make sure you
delete the contents of the following directories:
- data/
- commitlog/
- saved_caches/
Forget rejoining with repair -- it will just cause more problems. Cheers!
On Mon, Nov 13, 2017 at 2:54 PM, Anshu Vajpayee <an...@gmail.com>
wrote:
> Hi All ,
>
> There was a node failure in one of production cluster due to disk
> failure. After h/w recovery that node is noew ready be part of cluster,
> but it doesn't has any data due to disk crash.
>
>
>
> I can think of following option :
>
>
>
> 1. replace the node with same. using replace_address
>
> 2. Set bootstrap=false , start the node and run the repair to stream the
> data.
>
>
>
> Please suggest if both option are good and which is best as per your
> experience. This is live production cluster.
>
>
> Thanks,
>
>
> --
> *C*heers,*
> *Anshu V*
>
>
>