Re: How to replace a dead *seed* node while keeping quorum

[Ron Siemens <rs...@greatergood.com>]

I have an update on this.  I witnessed this same split ring problem, this time while doing a rolling upgrade from 1.1.4 to 1.1.6.  I found an easier workaround than modifying configs and restarting.  I found that by explicitly specifying the same token on the commandline using "-Dcassandra.replace_token=" when bringing up the new node, this problem wasn't exhibited.  Everything worked smoothly.

Ron

On Oct 10, 2012, at 12:38 PM, Ron Siemens wrote:

> 
> I witnessed the same behavior as reported by Edward and James.
> 
> Removing the host from its own seed list does not solve the problem.  Removing it from config of all nodes and restarting each, then restarting the failed node worked.
> 
> Ron
> 
> On Sep 12, 2012, at 4:42 PM, Edward Sargisson wrote:
> 
>> I'm reposting my colleague's reply to Rob to the list (with James' permission) in case others are interested.
>> 
>> I'll add to James' post below to say I don't believe we saw the message that that slice of code would have printed.
>> 
>> "
>> Hey Rob,
>> 
>> Ed's AWOL right now and I'm not on u@c.a.o, but I can tell you that when 
>> I removed the downed seed node from its own list of seed nodes in 
>> cassandra.yaml that it didn't join the existing ring nor did it get any 
>> schemas or data from the existing ring; it felt like timeouts were 
>> happening. (IANA Cassandra wizard, so excuse my terminology impedance.)
>> 
>> Changing the machine's hostname and giving it a new IP, it behaved as 
>> expected; joining the ring, syncing both schema and associated data.
>> 
>> Downed node is 1.1.4, the rest of the ring is 1.1.2.
>> 
>> I'm in a situation where I can revert the IP/hostname change and retry 
>> the scenario as needed if you've got any ideas.
>> 
>> HTH,
>> 
>>    JAmes"
>> 
>> Cheers,
>> Edward
>> 
>> On 12-09-12 03:53 PM, Rob Coli wrote:
>>> On Tue, Sep 11, 2012 at 4:21 PM, Edward Sargisson
>>> <ed...@globalrelay.net> wrote:
>>>> If the downed node is a seed node then neither of the replace a dead node
>>>> procedures work (-Dcassandra.replace_token and taking initial_token-1). The
>>>> ring remains split.
>>>> [...]
>>>> In other words, if the host name is on the seeds list then it appears that
>>>> the rest of the ring refuses to bootstrap it.
>>> Close, but not exactly...
>>> 
>>> "./src/java/org/apache/cassandra/service/StorageService.java" line 559 of 3090
>>> "
>>> if (DatabaseDescriptor.isAutoBootstrap()
>>>                 &&
>>> DatabaseDescriptor.getSeeds().contains(FBUtilities.getBroadcastAddress())
>>>                 && !SystemTable.isBootstrapped())
>>>             logger_.info("This node will not auto bootstrap because it
>>> is configured to be a seed node.");
>>> "
>>> 
>>> getSeeds asks your seed provider for a list of seeds. If you are using
>>> the SimpleSeedProvider, this basically turns the list from "seeds" in
>>> cassandra.yaml on the local node into a list of hosts.
>>> 
>>> So it isn't that the other nodes have this node in their seed list..
>>> it's that the node you are replacing has itself in its own seed list,
>>> and shouldn't. I understand that it can be tricky in conf management
>>> tools to make seed nodes' seed lists not contain themselves, but I
>>> believe it is currently necessary in this case.
>>> 
>>> FWIW, it's unclear to me (and Aaron Morton, whose curiousity was
>>> apparently equally piqued and is looking into it further..) why
>>> exactly seed nodes shouldn't bootstrap. It's possible that they only
>>> shouldn't bootstrap without being in "hibernate" mode, and that the
>>> code just hasn't been re-written post replace_token/hibernate to say
>>> that it's ok for seed nodes to bootstrap as long as they hibernate...
>>> 
>>> =Rob
>>> 
>> 
>> -- 
>> Edward Sargisson
>> senior java developer
>> Global Relay
>> 
>> edward.sargisson@globalrelay.net
>> 
>> 
>> 866.484.6630 
>> New York | Chicago | Vancouver  |  London  (+44.0800.032.9829)  |  Singapore  (+65.3158.1301)
>> 
>> Global Relay Archive supports email, instant messaging, BlackBerry, Bloomberg, Thomson Reuters, Pivot, YellowJacket, LinkedIn, Twitter, Facebook and more. 
>> 
>> Ask about Global Relay Message — The Future of Collaboration in the Financial Services World
>> 
>> All email sent to or from this address will be retained by Global Relay’s email archiving system. This message is intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential, and exempt from disclosure under applicable law.  Global Relay will not be liable for any compliance or technical information provided herein.  All trademarks are the property of their respective owners.
>