You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Edward Sargisson <ed...@globalrelay.net> on 2012/09/12 01:21:34 UTC
How to replace a dead *seed* node while keeping quorum
Hi all,
We just ran into an interesting and unexpected situation with restarting
a downed node.
If the downed node is a seed node then neither of the replace a dead
node procedures work (-Dcassandra.replace_token and taking
initial_token-1). The ring remains split.
The host is listed as a seed in the config for the other members of the
ring. If we rename the host then it will rejoin the ring.
In other words, if the host name is on the seeds list then it appears
that the rest of the ring refuses to bootstrap it.
This leads to a problem: If the node needs to be taken out of the seeds
list on every working node then that requires a restart of each node -
which means that, for short periods, the ring is missing 2 nodes and a
quorum read or write (RF=3) will fail.
Are there any useful tricks for restarting the node with the same
hostname or are we expected to rename the node?
Cheers,
Edward
--
Edward Sargisson
senior java developer
Global Relay
edward.sargisson@globalrelay.net <ma...@globalrelay.net>
*866.484.6630*
New York | Chicago | Vancouver | London (+44.0800.032.9829) | Singapore
(+65.3158.1301)
Global Relay Archive supports email, instant messaging, BlackBerry,
Bloomberg, Thomson Reuters, Pivot, YellowJacket, LinkedIn, Twitter,
Facebook and more.
Ask about *Global Relay Message*
<http://www.globalrelay.com/services/message>*--- *The Future of
Collaboration in the Financial Services World
*
*All email sent to or from this address will be retained by Global
Relay's email archiving system. This message is intended only for the
use of the individual or entity to which it is addressed, and may
contain information that is privileged, confidential, and exempt from
disclosure under applicable law. Global Relay will not be liable for
any compliance or technical information provided herein. All trademarks
are the property of their respective owners.
Re: How to replace a dead *seed* node while keeping quorum
Posted by Ron Siemens <rs...@greatergood.com>.
I have an update on this. I witnessed this same split ring problem, this time while doing a rolling upgrade from 1.1.4 to 1.1.6. I found an easier workaround than modifying configs and restarting. I found that by explicitly specifying the same token on the commandline using "-Dcassandra.replace_token=" when bringing up the new node, this problem wasn't exhibited. Everything worked smoothly.
Ron
On Oct 10, 2012, at 12:38 PM, Ron Siemens wrote:
>
> I witnessed the same behavior as reported by Edward and James.
>
> Removing the host from its own seed list does not solve the problem. Removing it from config of all nodes and restarting each, then restarting the failed node worked.
>
> Ron
>
> On Sep 12, 2012, at 4:42 PM, Edward Sargisson wrote:
>
>> I'm reposting my colleague's reply to Rob to the list (with James' permission) in case others are interested.
>>
>> I'll add to James' post below to say I don't believe we saw the message that that slice of code would have printed.
>>
>> "
>> Hey Rob,
>>
>> Ed's AWOL right now and I'm not on u@c.a.o, but I can tell you that when
>> I removed the downed seed node from its own list of seed nodes in
>> cassandra.yaml that it didn't join the existing ring nor did it get any
>> schemas or data from the existing ring; it felt like timeouts were
>> happening. (IANA Cassandra wizard, so excuse my terminology impedance.)
>>
>> Changing the machine's hostname and giving it a new IP, it behaved as
>> expected; joining the ring, syncing both schema and associated data.
>>
>> Downed node is 1.1.4, the rest of the ring is 1.1.2.
>>
>> I'm in a situation where I can revert the IP/hostname change and retry
>> the scenario as needed if you've got any ideas.
>>
>> HTH,
>>
>> JAmes"
>>
>> Cheers,
>> Edward
>>
>> On 12-09-12 03:53 PM, Rob Coli wrote:
>>> On Tue, Sep 11, 2012 at 4:21 PM, Edward Sargisson
>>> <ed...@globalrelay.net> wrote:
>>>> If the downed node is a seed node then neither of the replace a dead node
>>>> procedures work (-Dcassandra.replace_token and taking initial_token-1). The
>>>> ring remains split.
>>>> [...]
>>>> In other words, if the host name is on the seeds list then it appears that
>>>> the rest of the ring refuses to bootstrap it.
>>> Close, but not exactly...
>>>
>>> "./src/java/org/apache/cassandra/service/StorageService.java" line 559 of 3090
>>> "
>>> if (DatabaseDescriptor.isAutoBootstrap()
>>> &&
>>> DatabaseDescriptor.getSeeds().contains(FBUtilities.getBroadcastAddress())
>>> && !SystemTable.isBootstrapped())
>>> logger_.info("This node will not auto bootstrap because it
>>> is configured to be a seed node.");
>>> "
>>>
>>> getSeeds asks your seed provider for a list of seeds. If you are using
>>> the SimpleSeedProvider, this basically turns the list from "seeds" in
>>> cassandra.yaml on the local node into a list of hosts.
>>>
>>> So it isn't that the other nodes have this node in their seed list..
>>> it's that the node you are replacing has itself in its own seed list,
>>> and shouldn't. I understand that it can be tricky in conf management
>>> tools to make seed nodes' seed lists not contain themselves, but I
>>> believe it is currently necessary in this case.
>>>
>>> FWIW, it's unclear to me (and Aaron Morton, whose curiousity was
>>> apparently equally piqued and is looking into it further..) why
>>> exactly seed nodes shouldn't bootstrap. It's possible that they only
>>> shouldn't bootstrap without being in "hibernate" mode, and that the
>>> code just hasn't been re-written post replace_token/hibernate to say
>>> that it's ok for seed nodes to bootstrap as long as they hibernate...
>>>
>>> =Rob
>>>
>>
>> --
>> Edward Sargisson
>> senior java developer
>> Global Relay
>>
>> edward.sargisson@globalrelay.net
>>
>>
>> 866.484.6630
>> New York | Chicago | Vancouver | London (+44.0800.032.9829) | Singapore (+65.3158.1301)
>>
>> Global Relay Archive supports email, instant messaging, BlackBerry, Bloomberg, Thomson Reuters, Pivot, YellowJacket, LinkedIn, Twitter, Facebook and more.
>>
>> Ask about Global Relay Message — The Future of Collaboration in the Financial Services World
>>
>> All email sent to or from this address will be retained by Global Relay’s email archiving system. This message is intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. Global Relay will not be liable for any compliance or technical information provided herein. All trademarks are the property of their respective owners.
>
Re: How to replace a dead *seed* node while keeping quorum
Posted by Ron Siemens <rs...@greatergood.com>.
I witnessed the same behavior as reported by Edward and James.
Removing the host from its own seed list does not solve the problem. Removing it from config of all nodes and restarting each, then restarting the failed node worked.
Ron
On Sep 12, 2012, at 4:42 PM, Edward Sargisson wrote:
> I'm reposting my colleague's reply to Rob to the list (with James' permission) in case others are interested.
>
> I'll add to James' post below to say I don't believe we saw the message that that slice of code would have printed.
>
> "
> Hey Rob,
>
> Ed's AWOL right now and I'm not on u@c.a.o, but I can tell you that when
> I removed the downed seed node from its own list of seed nodes in
> cassandra.yaml that it didn't join the existing ring nor did it get any
> schemas or data from the existing ring; it felt like timeouts were
> happening. (IANA Cassandra wizard, so excuse my terminology impedance.)
>
> Changing the machine's hostname and giving it a new IP, it behaved as
> expected; joining the ring, syncing both schema and associated data.
>
> Downed node is 1.1.4, the rest of the ring is 1.1.2.
>
> I'm in a situation where I can revert the IP/hostname change and retry
> the scenario as needed if you've got any ideas.
>
> HTH,
>
> JAmes"
>
> Cheers,
> Edward
>
> On 12-09-12 03:53 PM, Rob Coli wrote:
>> On Tue, Sep 11, 2012 at 4:21 PM, Edward Sargisson
>> <ed...@globalrelay.net> wrote:
>>> If the downed node is a seed node then neither of the replace a dead node
>>> procedures work (-Dcassandra.replace_token and taking initial_token-1). The
>>> ring remains split.
>>> [...]
>>> In other words, if the host name is on the seeds list then it appears that
>>> the rest of the ring refuses to bootstrap it.
>> Close, but not exactly...
>>
>> "./src/java/org/apache/cassandra/service/StorageService.java" line 559 of 3090
>> "
>> if (DatabaseDescriptor.isAutoBootstrap()
>> &&
>> DatabaseDescriptor.getSeeds().contains(FBUtilities.getBroadcastAddress())
>> && !SystemTable.isBootstrapped())
>> logger_.info("This node will not auto bootstrap because it
>> is configured to be a seed node.");
>> "
>>
>> getSeeds asks your seed provider for a list of seeds. If you are using
>> the SimpleSeedProvider, this basically turns the list from "seeds" in
>> cassandra.yaml on the local node into a list of hosts.
>>
>> So it isn't that the other nodes have this node in their seed list..
>> it's that the node you are replacing has itself in its own seed list,
>> and shouldn't. I understand that it can be tricky in conf management
>> tools to make seed nodes' seed lists not contain themselves, but I
>> believe it is currently necessary in this case.
>>
>> FWIW, it's unclear to me (and Aaron Morton, whose curiousity was
>> apparently equally piqued and is looking into it further..) why
>> exactly seed nodes shouldn't bootstrap. It's possible that they only
>> shouldn't bootstrap without being in "hibernate" mode, and that the
>> code just hasn't been re-written post replace_token/hibernate to say
>> that it's ok for seed nodes to bootstrap as long as they hibernate...
>>
>> =Rob
>>
>
> --
> Edward Sargisson
> senior java developer
> Global Relay
>
> edward.sargisson@globalrelay.net
>
>
> 866.484.6630
> New York | Chicago | Vancouver | London (+44.0800.032.9829) | Singapore (+65.3158.1301)
>
> Global Relay Archive supports email, instant messaging, BlackBerry, Bloomberg, Thomson Reuters, Pivot, YellowJacket, LinkedIn, Twitter, Facebook and more.
>
> Ask about Global Relay Message — The Future of Collaboration in the Financial Services World
>
> All email sent to or from this address will be retained by Global Relay’s email archiving system. This message is intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. Global Relay will not be liable for any compliance or technical information provided herein. All trademarks are the property of their respective owners.
Re: How to replace a dead *seed* node while keeping quorum
Posted by Edward Sargisson <ed...@globalrelay.net>.
I'm reposting my colleague's reply to Rob to the list (with James'
permission) in case others are interested.
I'll add to James' post below to say I don't believe we saw the message
that that slice of code would have printed.
"
Hey Rob,
Ed's AWOL right now and I'm not onu@c.a.o, but I can tell you that when
I removed the downed seed node from its own list of seed nodes in
cassandra.yaml that it didn't join the existing ring nor did it get any
schemas or data from the existing ring; it felt like timeouts were
happening. (IANA Cassandra wizard, so excuse my terminology impedance.)
Changing the machine's hostname and giving it a new IP, it behaved as
expected; joining the ring, syncing both schema and associated data.
Downed node is 1.1.4, the rest of the ring is 1.1.2.
I'm in a situation where I can revert the IP/hostname change and retry
the scenario as needed if you've got any ideas.
HTH,
JAmes"
Cheers,
Edward
On 12-09-12 03:53 PM, Rob Coli wrote:
> On Tue, Sep 11, 2012 at 4:21 PM, Edward Sargisson
> <ed...@globalrelay.net> wrote:
>> If the downed node is a seed node then neither of the replace a dead node
>> procedures work (-Dcassandra.replace_token and taking initial_token-1). The
>> ring remains split.
>> [...]
>> In other words, if the host name is on the seeds list then it appears that
>> the rest of the ring refuses to bootstrap it.
> Close, but not exactly...
>
> "./src/java/org/apache/cassandra/service/StorageService.java" line 559 of 3090
> "
> if (DatabaseDescriptor.isAutoBootstrap()
> &&
> DatabaseDescriptor.getSeeds().contains(FBUtilities.getBroadcastAddress())
> && !SystemTable.isBootstrapped())
> logger_.info("This node will not auto bootstrap because it
> is configured to be a seed node.");
> "
>
> getSeeds asks your seed provider for a list of seeds. If you are using
> the SimpleSeedProvider, this basically turns the list from "seeds" in
> cassandra.yaml on the local node into a list of hosts.
>
> So it isn't that the other nodes have this node in their seed list..
> it's that the node you are replacing has itself in its own seed list,
> and shouldn't. I understand that it can be tricky in conf management
> tools to make seed nodes' seed lists not contain themselves, but I
> believe it is currently necessary in this case.
>
> FWIW, it's unclear to me (and Aaron Morton, whose curiousity was
> apparently equally piqued and is looking into it further..) why
> exactly seed nodes shouldn't bootstrap. It's possible that they only
> shouldn't bootstrap without being in "hibernate" mode, and that the
> code just hasn't been re-written post replace_token/hibernate to say
> that it's ok for seed nodes to bootstrap as long as they hibernate...
>
> =Rob
>
--
Edward Sargisson
senior java developer
Global Relay
edward.sargisson@globalrelay.net <ma...@globalrelay.net>
*866.484.6630*
New York | Chicago | Vancouver | London (+44.0800.032.9829) | Singapore
(+65.3158.1301)
Global Relay Archive supports email, instant messaging, BlackBerry,
Bloomberg, Thomson Reuters, Pivot, YellowJacket, LinkedIn, Twitter,
Facebook and more.
Ask about *Global Relay Message*
<http://www.globalrelay.com/services/message>*--- *The Future of
Collaboration in the Financial Services World
*
*All email sent to or from this address will be retained by Global
Relay's email archiving system. This message is intended only for the
use of the individual or entity to which it is addressed, and may
contain information that is privileged, confidential, and exempt from
disclosure under applicable law. Global Relay will not be liable for
any compliance or technical information provided herein. All trademarks
are the property of their respective owners.
Re: How to replace a dead *seed* node while keeping quorum
Posted by Mohit Anchlia <mo...@gmail.com>.
How can this be resolved in this case?
On Wed, Sep 12, 2012 at 3:53 PM, Rob Coli <rc...@palominodb.com> wrote:
> On Tue, Sep 11, 2012 at 4:21 PM, Edward Sargisson
> <ed...@globalrelay.net> wrote:
> > If the downed node is a seed node then neither of the replace a dead node
> > procedures work (-Dcassandra.replace_token and taking initial_token-1).
> The
> > ring remains split.
> > [...]
> > In other words, if the host name is on the seeds list then it appears
> that
> > the rest of the ring refuses to bootstrap it.
>
> Close, but not exactly...
>
> "./src/java/org/apache/cassandra/service/StorageService.java" line 559 of
> 3090
> "
> if (DatabaseDescriptor.isAutoBootstrap()
> &&
> DatabaseDescriptor.getSeeds().contains(FBUtilities.getBroadcastAddress())
> && !SystemTable.isBootstrapped())
> logger_.info("This node will not auto bootstrap because it
> is configured to be a seed node.");
> "
>
> getSeeds asks your seed provider for a list of seeds. If you are using
> the SimpleSeedProvider, this basically turns the list from "seeds" in
> cassandra.yaml on the local node into a list of hosts.
>
> So it isn't that the other nodes have this node in their seed list..
> it's that the node you are replacing has itself in its own seed list,
> and shouldn't. I understand that it can be tricky in conf management
> tools to make seed nodes' seed lists not contain themselves, but I
> believe it is currently necessary in this case.
>
> FWIW, it's unclear to me (and Aaron Morton, whose curiousity was
> apparently equally piqued and is looking into it further..) why
> exactly seed nodes shouldn't bootstrap. It's possible that they only
> shouldn't bootstrap without being in "hibernate" mode, and that the
> code just hasn't been re-written post replace_token/hibernate to say
> that it's ok for seed nodes to bootstrap as long as they hibernate...
>
> =Rob
>
> --
> =Robert Coli
> AIM>ALK - rcoli@palominodb.com
> YAHOO - rcoli.palominob
> SKYPE - rcoli_palominodb
>
Re: How to replace a dead *seed* node while keeping quorum
Posted by Rob Coli <rc...@palominodb.com>.
On Tue, Sep 11, 2012 at 4:21 PM, Edward Sargisson
<ed...@globalrelay.net> wrote:
> If the downed node is a seed node then neither of the replace a dead node
> procedures work (-Dcassandra.replace_token and taking initial_token-1). The
> ring remains split.
> [...]
> In other words, if the host name is on the seeds list then it appears that
> the rest of the ring refuses to bootstrap it.
Close, but not exactly...
"./src/java/org/apache/cassandra/service/StorageService.java" line 559 of 3090
"
if (DatabaseDescriptor.isAutoBootstrap()
&&
DatabaseDescriptor.getSeeds().contains(FBUtilities.getBroadcastAddress())
&& !SystemTable.isBootstrapped())
logger_.info("This node will not auto bootstrap because it
is configured to be a seed node.");
"
getSeeds asks your seed provider for a list of seeds. If you are using
the SimpleSeedProvider, this basically turns the list from "seeds" in
cassandra.yaml on the local node into a list of hosts.
So it isn't that the other nodes have this node in their seed list..
it's that the node you are replacing has itself in its own seed list,
and shouldn't. I understand that it can be tricky in conf management
tools to make seed nodes' seed lists not contain themselves, but I
believe it is currently necessary in this case.
FWIW, it's unclear to me (and Aaron Morton, whose curiousity was
apparently equally piqued and is looking into it further..) why
exactly seed nodes shouldn't bootstrap. It's possible that they only
shouldn't bootstrap without being in "hibernate" mode, and that the
code just hasn't been re-written post replace_token/hibernate to say
that it's ok for seed nodes to bootstrap as long as they hibernate...
=Rob
--
=Robert Coli
AIM>ALK - rcoli@palominodb.com
YAHOO - rcoli.palominob
SKYPE - rcoli_palominodb