You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Michael Allen (JIRA)" <ji...@apache.org> on 2011/06/24 20:43:47 UTC
[jira] [Created] (CASSANDRA-2825) Auto bootstrapping the 4th node
in a 4 node cluster doesn't work, when no token explicitly assigned in
config.
Auto bootstrapping the 4th node in a 4 node cluster doesn't work, when no token explicitly assigned in config.
--------------------------------------------------------------------------------------------------------------
Key: CASSANDRA-2825
URL: https://issues.apache.org/jira/browse/CASSANDRA-2825
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 0.8.0, 0.8.1
Reporter: Michael Allen
Fix For: 0.8.2
This was done in sequence. A, B, C, and D. Node A with token 0 explicitly set in config. The rest with auto_bootstrap: true and no token explicitly assigned. B and C work as expected. D ends up stealing C's token.
from system.log on C:
INFO [GossipStage:1] 2011-06-24 16:40:41,947 Gossiper.java (line 638) Node /10.171.47.226 is now part of the cluster
INFO [GossipStage:1] 2011-06-24 16:40:41,947 Gossiper.java (line 606) InetAddress /10.171.47.226 is now UP
INFO [GossipStage:1] 2011-06-24 16:42:09,432 StorageService.java (line 769) Nodes /10.171.47.226 and /10.171.55.77 have the same token 61078635599166706937511052402724559481. /10.171.47.226 is the new owner
WARN [GossipStage:1] 2011-06-24 16:42:09,432 TokenMetadata.java (line 120) Token 61078635599166706937511052402724559481 changing ownership from /10.171.55.77 to /10.171.47.226
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2825) Auto bootstrapping the 4th node
in a 4 node cluster doesn't work, when no token explicitly assigned in
config.
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13068504#comment-13068504 ]
Jonathan Ellis commented on CASSANDRA-2825:
-------------------------------------------
let's compare w/ Table.SYSTEM_TABLE, otherwise +1
> Auto bootstrapping the 4th node in a 4 node cluster doesn't work, when no token explicitly assigned in config.
> --------------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-2825
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2825
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.0, 0.8.1
> Reporter: Michael Allen
> Assignee: Brandon Williams
> Fix For: 0.8.2
>
> Attachments: 2825-v2.txt, 2825.txt
>
>
> This was done in sequence. A, B, C, and D. Node A with token 0 explicitly set in config. The rest with auto_bootstrap: true and no token explicitly assigned. B and C work as expected. D ends up stealing C's token.
> from system.log on C:
> INFO [GossipStage:1] 2011-06-24 16:40:41,947 Gossiper.java (line 638) Node /10.171.47.226 is now part of the cluster
> INFO [GossipStage:1] 2011-06-24 16:40:41,947 Gossiper.java (line 606) InetAddress /10.171.47.226 is now UP
> INFO [GossipStage:1] 2011-06-24 16:42:09,432 StorageService.java (line 769) Nodes /10.171.47.226 and /10.171.55.77 have the same token 61078635599166706937511052402724559481. /10.171.47.226 is the new owner
> WARN [GossipStage:1] 2011-06-24 16:42:09,432 TokenMetadata.java (line 120) Token 61078635599166706937511052402724559481 changing ownership from /10.171.55.77 to /10.171.47.226
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2825) Auto bootstrapping the 4th node
in a 4 node cluster doesn't work, when no token explicitly assigned in
config.
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13073642#comment-13073642 ]
Hudson commented on CASSANDRA-2825:
-----------------------------------
Integrated in Cassandra-0.8 #249 (See [https://builds.apache.org/job/Cassandra-0.8/249/])
Don't sample the system table keys when choosing a bootstrap token.
Patch by brandonwilliams, reviewed by jbellis for CASSANDRA-2825
brandonwilliams : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1152876
Files :
* /cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/service/StorageService.java
> Auto bootstrapping the 4th node in a 4 node cluster doesn't work, when no token explicitly assigned in config.
> --------------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-2825
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2825
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.0, 0.8.1
> Reporter: Michael Allen
> Assignee: Brandon Williams
> Fix For: 0.8.3
>
> Attachments: 2825-v2.txt, 2825-v3.txt, 2825.txt
>
>
> This was done in sequence. A, B, C, and D. Node A with token 0 explicitly set in config. The rest with auto_bootstrap: true and no token explicitly assigned. B and C work as expected. D ends up stealing C's token.
> from system.log on C:
> INFO [GossipStage:1] 2011-06-24 16:40:41,947 Gossiper.java (line 638) Node /10.171.47.226 is now part of the cluster
> INFO [GossipStage:1] 2011-06-24 16:40:41,947 Gossiper.java (line 606) InetAddress /10.171.47.226 is now UP
> INFO [GossipStage:1] 2011-06-24 16:42:09,432 StorageService.java (line 769) Nodes /10.171.47.226 and /10.171.55.77 have the same token 61078635599166706937511052402724559481. /10.171.47.226 is the new owner
> WARN [GossipStage:1] 2011-06-24 16:42:09,432 TokenMetadata.java (line 120) Token 61078635599166706937511052402724559481 changing ownership from /10.171.55.77 to /10.171.47.226
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2825) Auto bootstrapping the 4th node
in a 4 node cluster doesn't work, when no token explicitly assigned in
config.
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13068974#comment-13068974 ]
Brandon Williams commented on CASSANDRA-2825:
---------------------------------------------
Impressive, I have no idea how this is breaking testTokenRoundtrip():
{noformat}
public void testTokenRoundtrip() throws Exception
{
StorageService.instance.initServer();
// fetch a bootstrap token from the local node
assert BootStrapper.getBootstrapTokenFrom(FBUtilities.getLocalAddress()) != null;
}
{noformat}
The log just shows a bunch of attempts to connect to the seed (127.0.0.2) which hasn't started yet.
> Auto bootstrapping the 4th node in a 4 node cluster doesn't work, when no token explicitly assigned in config.
> --------------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-2825
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2825
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.0, 0.8.1
> Reporter: Michael Allen
> Assignee: Brandon Williams
> Fix For: 0.8.2
>
> Attachments: 2825-v2.txt, 2825.txt
>
>
> This was done in sequence. A, B, C, and D. Node A with token 0 explicitly set in config. The rest with auto_bootstrap: true and no token explicitly assigned. B and C work as expected. D ends up stealing C's token.
> from system.log on C:
> INFO [GossipStage:1] 2011-06-24 16:40:41,947 Gossiper.java (line 638) Node /10.171.47.226 is now part of the cluster
> INFO [GossipStage:1] 2011-06-24 16:40:41,947 Gossiper.java (line 606) InetAddress /10.171.47.226 is now UP
> INFO [GossipStage:1] 2011-06-24 16:42:09,432 StorageService.java (line 769) Nodes /10.171.47.226 and /10.171.55.77 have the same token 61078635599166706937511052402724559481. /10.171.47.226 is the new owner
> WARN [GossipStage:1] 2011-06-24 16:42:09,432 TokenMetadata.java (line 120) Token 61078635599166706937511052402724559481 changing ownership from /10.171.55.77 to /10.171.47.226
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (CASSANDRA-2825) Auto bootstrapping the 4th node
in a 4 node cluster doesn't work, when no token explicitly assigned in
config.
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sylvain Lebresne reopened CASSANDRA-2825:
-----------------------------------------
Reopening because the patch broke BootStrapperTest (it somehow hangs forever until junit timeout)
> Auto bootstrapping the 4th node in a 4 node cluster doesn't work, when no token explicitly assigned in config.
> --------------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-2825
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2825
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.0, 0.8.1
> Reporter: Michael Allen
> Assignee: Brandon Williams
> Fix For: 0.8.2
>
> Attachments: 2825-v2.txt, 2825.txt
>
>
> This was done in sequence. A, B, C, and D. Node A with token 0 explicitly set in config. The rest with auto_bootstrap: true and no token explicitly assigned. B and C work as expected. D ends up stealing C's token.
> from system.log on C:
> INFO [GossipStage:1] 2011-06-24 16:40:41,947 Gossiper.java (line 638) Node /10.171.47.226 is now part of the cluster
> INFO [GossipStage:1] 2011-06-24 16:40:41,947 Gossiper.java (line 606) InetAddress /10.171.47.226 is now UP
> INFO [GossipStage:1] 2011-06-24 16:42:09,432 StorageService.java (line 769) Nodes /10.171.47.226 and /10.171.55.77 have the same token 61078635599166706937511052402724559481. /10.171.47.226 is the new owner
> WARN [GossipStage:1] 2011-06-24 16:42:09,432 TokenMetadata.java (line 120) Token 61078635599166706937511052402724559481 changing ownership from /10.171.55.77 to /10.171.47.226
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (CASSANDRA-2825) Auto bootstrapping the 4th node
in a 4 node cluster doesn't work, when no token explicitly assigned in
config.
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis reassigned CASSANDRA-2825:
-----------------------------------------
Assignee: Brandon Williams
> Auto bootstrapping the 4th node in a 4 node cluster doesn't work, when no token explicitly assigned in config.
> --------------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-2825
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2825
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.0, 0.8.1
> Reporter: Michael Allen
> Assignee: Brandon Williams
> Fix For: 0.8.2
>
>
> This was done in sequence. A, B, C, and D. Node A with token 0 explicitly set in config. The rest with auto_bootstrap: true and no token explicitly assigned. B and C work as expected. D ends up stealing C's token.
> from system.log on C:
> INFO [GossipStage:1] 2011-06-24 16:40:41,947 Gossiper.java (line 638) Node /10.171.47.226 is now part of the cluster
> INFO [GossipStage:1] 2011-06-24 16:40:41,947 Gossiper.java (line 606) InetAddress /10.171.47.226 is now UP
> INFO [GossipStage:1] 2011-06-24 16:42:09,432 StorageService.java (line 769) Nodes /10.171.47.226 and /10.171.55.77 have the same token 61078635599166706937511052402724559481. /10.171.47.226 is the new owner
> WARN [GossipStage:1] 2011-06-24 16:42:09,432 TokenMetadata.java (line 120) Token 61078635599166706937511052402724559481 changing ownership from /10.171.55.77 to /10.171.47.226
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2825) Auto bootstrapping the 4th node
in a 4 node cluster doesn't work, when no token explicitly assigned in
config.
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13068054#comment-13068054 ]
Brandon Williams commented on CASSANDRA-2825:
---------------------------------------------
Though 85070591730234615865843651857942052864 shows a load of 4.55KB when it is added, 90s later when another node checks it will have 13.4K because it will have updated. The reason it has more load than the source at this point is because it flushed LocationInfo three times, where the source only flushed it twice. That should be rare in practice, since typically if a cluster has no data you just start it up without bootstrapping, but still we should protect against handing out tokens that already are in use.
> Auto bootstrapping the 4th node in a 4 node cluster doesn't work, when no token explicitly assigned in config.
> --------------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-2825
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2825
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.0, 0.8.1
> Reporter: Michael Allen
> Assignee: Brandon Williams
> Fix For: 0.8.2
>
>
> This was done in sequence. A, B, C, and D. Node A with token 0 explicitly set in config. The rest with auto_bootstrap: true and no token explicitly assigned. B and C work as expected. D ends up stealing C's token.
> from system.log on C:
> INFO [GossipStage:1] 2011-06-24 16:40:41,947 Gossiper.java (line 638) Node /10.171.47.226 is now part of the cluster
> INFO [GossipStage:1] 2011-06-24 16:40:41,947 Gossiper.java (line 606) InetAddress /10.171.47.226 is now UP
> INFO [GossipStage:1] 2011-06-24 16:42:09,432 StorageService.java (line 769) Nodes /10.171.47.226 and /10.171.55.77 have the same token 61078635599166706937511052402724559481. /10.171.47.226 is the new owner
> WARN [GossipStage:1] 2011-06-24 16:42:09,432 TokenMetadata.java (line 120) Token 61078635599166706937511052402724559481 changing ownership from /10.171.55.77 to /10.171.47.226
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2825) Auto bootstrapping the 4th node
in a 4 node cluster doesn't work, when no token explicitly assigned in
config.
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13068531#comment-13068531 ]
Hudson commented on CASSANDRA-2825:
-----------------------------------
Integrated in Cassandra-0.8 #228 (See [https://builds.apache.org/job/Cassandra-0.8/228/])
Don't sample the system table keys when choosing a bootstrap token.
Patch by brandonwilliams, reviewed by jbellis for CASSANDRA-2825
brandonwilliams : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1148866
Files :
* /cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/service/StorageService.java
> Auto bootstrapping the 4th node in a 4 node cluster doesn't work, when no token explicitly assigned in config.
> --------------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-2825
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2825
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.0, 0.8.1
> Reporter: Michael Allen
> Assignee: Brandon Williams
> Fix For: 0.8.2
>
> Attachments: 2825-v2.txt, 2825.txt
>
>
> This was done in sequence. A, B, C, and D. Node A with token 0 explicitly set in config. The rest with auto_bootstrap: true and no token explicitly assigned. B and C work as expected. D ends up stealing C's token.
> from system.log on C:
> INFO [GossipStage:1] 2011-06-24 16:40:41,947 Gossiper.java (line 638) Node /10.171.47.226 is now part of the cluster
> INFO [GossipStage:1] 2011-06-24 16:40:41,947 Gossiper.java (line 606) InetAddress /10.171.47.226 is now UP
> INFO [GossipStage:1] 2011-06-24 16:42:09,432 StorageService.java (line 769) Nodes /10.171.47.226 and /10.171.55.77 have the same token 61078635599166706937511052402724559481. /10.171.47.226 is the new owner
> WARN [GossipStage:1] 2011-06-24 16:42:09,432 TokenMetadata.java (line 120) Token 61078635599166706937511052402724559481 changing ownership from /10.171.55.77 to /10.171.47.226
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2825) Auto bootstrapping the 4th node
in a 4 node cluster doesn't work, when no token explicitly assigned in
config.
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13057960#comment-13057960 ]
Brandon Williams commented on CASSANDRA-2825:
---------------------------------------------
I'm unable to reproduce. What I don't understand is how token 61078635599166706937511052402724559481 can be involved if the first node started at 0. My ring ends up looking like:
{noformat}
Address DC Rack Status State Load Owns Token
148820376661671484777635343879628161952
10.179.65.102 datacenter1 rack1 Up Normal 315.59 MB 12.53% 0
10.179.64.227 datacenter1 rack1 Up Normal 157.76 MB 49.97% 85015475407349018891169115382886066923
10.179.111.137 datacenter1 rack1 Up Normal 78.99 MB 25.01% 127563185643974719468523634380546988377
10.179.110.154 datacenter1 rack1 Up Normal 39.55 MB 12.49% 148820376661671484777635343879628161952
{noformat}
> Auto bootstrapping the 4th node in a 4 node cluster doesn't work, when no token explicitly assigned in config.
> --------------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-2825
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2825
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.0, 0.8.1
> Reporter: Michael Allen
> Assignee: Brandon Williams
> Fix For: 0.8.2
>
>
> This was done in sequence. A, B, C, and D. Node A with token 0 explicitly set in config. The rest with auto_bootstrap: true and no token explicitly assigned. B and C work as expected. D ends up stealing C's token.
> from system.log on C:
> INFO [GossipStage:1] 2011-06-24 16:40:41,947 Gossiper.java (line 638) Node /10.171.47.226 is now part of the cluster
> INFO [GossipStage:1] 2011-06-24 16:40:41,947 Gossiper.java (line 606) InetAddress /10.171.47.226 is now UP
> INFO [GossipStage:1] 2011-06-24 16:42:09,432 StorageService.java (line 769) Nodes /10.171.47.226 and /10.171.55.77 have the same token 61078635599166706937511052402724559481. /10.171.47.226 is the new owner
> WARN [GossipStage:1] 2011-06-24 16:42:09,432 TokenMetadata.java (line 120) Token 61078635599166706937511052402724559481 changing ownership from /10.171.55.77 to /10.171.47.226
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2825) Auto bootstrapping the 4th node
in a 4 node cluster doesn't work, when no token explicitly assigned in
config.
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13068039#comment-13068039 ]
Brandon Williams commented on CASSANDRA-2825:
---------------------------------------------
The difference here is that when I tried to repro, I started with data in the cluster. Since I never ran cleanup, token 0's range was always being split. But if you start without any data, for some reason it decides to split 85070591730234615865843651857942052864 twice. It's not clear why this is happening, since 0's load was still greater at the time, but at least one problem is that we do not assert that the token isn't in use when we determine it.
> Auto bootstrapping the 4th node in a 4 node cluster doesn't work, when no token explicitly assigned in config.
> --------------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-2825
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2825
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.0, 0.8.1
> Reporter: Michael Allen
> Assignee: Brandon Williams
> Fix For: 0.8.2
>
>
> This was done in sequence. A, B, C, and D. Node A with token 0 explicitly set in config. The rest with auto_bootstrap: true and no token explicitly assigned. B and C work as expected. D ends up stealing C's token.
> from system.log on C:
> INFO [GossipStage:1] 2011-06-24 16:40:41,947 Gossiper.java (line 638) Node /10.171.47.226 is now part of the cluster
> INFO [GossipStage:1] 2011-06-24 16:40:41,947 Gossiper.java (line 606) InetAddress /10.171.47.226 is now UP
> INFO [GossipStage:1] 2011-06-24 16:42:09,432 StorageService.java (line 769) Nodes /10.171.47.226 and /10.171.55.77 have the same token 61078635599166706937511052402724559481. /10.171.47.226 is the new owner
> WARN [GossipStage:1] 2011-06-24 16:42:09,432 TokenMetadata.java (line 120) Token 61078635599166706937511052402724559481 changing ownership from /10.171.55.77 to /10.171.47.226
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2825) Auto bootstrapping the 4th node
in a 4 node cluster doesn't work, when no token explicitly assigned in
config.
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13069017#comment-13069017 ]
Sylvain Lebresne commented on CASSANDRA-2825:
---------------------------------------------
I've reverted the patch too so we can do a release of 0.8.2 without having to wait on the unit test fix.
> Auto bootstrapping the 4th node in a 4 node cluster doesn't work, when no token explicitly assigned in config.
> --------------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-2825
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2825
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.0, 0.8.1
> Reporter: Michael Allen
> Assignee: Brandon Williams
> Fix For: 0.8.2
>
> Attachments: 2825-v2.txt, 2825.txt
>
>
> This was done in sequence. A, B, C, and D. Node A with token 0 explicitly set in config. The rest with auto_bootstrap: true and no token explicitly assigned. B and C work as expected. D ends up stealing C's token.
> from system.log on C:
> INFO [GossipStage:1] 2011-06-24 16:40:41,947 Gossiper.java (line 638) Node /10.171.47.226 is now part of the cluster
> INFO [GossipStage:1] 2011-06-24 16:40:41,947 Gossiper.java (line 606) InetAddress /10.171.47.226 is now UP
> INFO [GossipStage:1] 2011-06-24 16:42:09,432 StorageService.java (line 769) Nodes /10.171.47.226 and /10.171.55.77 have the same token 61078635599166706937511052402724559481. /10.171.47.226 is the new owner
> WARN [GossipStage:1] 2011-06-24 16:42:09,432 TokenMetadata.java (line 120) Token 61078635599166706937511052402724559481 changing ownership from /10.171.55.77 to /10.171.47.226
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2825) Auto bootstrapping the 4th node
in a 4 node cluster doesn't work, when no token explicitly assigned in
config.
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13068122#comment-13068122 ]
Jonathan Ellis commented on CASSANDRA-2825:
-------------------------------------------
can we generalize that to "check that it's not any known endpoint" and make it a "real" (non assert) check?
> Auto bootstrapping the 4th node in a 4 node cluster doesn't work, when no token explicitly assigned in config.
> --------------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-2825
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2825
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.0, 0.8.1
> Reporter: Michael Allen
> Assignee: Brandon Williams
> Fix For: 0.8.2
>
> Attachments: 2825.txt
>
>
> This was done in sequence. A, B, C, and D. Node A with token 0 explicitly set in config. The rest with auto_bootstrap: true and no token explicitly assigned. B and C work as expected. D ends up stealing C's token.
> from system.log on C:
> INFO [GossipStage:1] 2011-06-24 16:40:41,947 Gossiper.java (line 638) Node /10.171.47.226 is now part of the cluster
> INFO [GossipStage:1] 2011-06-24 16:40:41,947 Gossiper.java (line 606) InetAddress /10.171.47.226 is now UP
> INFO [GossipStage:1] 2011-06-24 16:42:09,432 StorageService.java (line 769) Nodes /10.171.47.226 and /10.171.55.77 have the same token 61078635599166706937511052402724559481. /10.171.47.226 is the new owner
> WARN [GossipStage:1] 2011-06-24 16:42:09,432 TokenMetadata.java (line 120) Token 61078635599166706937511052402724559481 changing ownership from /10.171.55.77 to /10.171.47.226
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2825) Auto bootstrapping the 4th node
in a 4 node cluster doesn't work, when no token explicitly assigned in
config.
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Williams updated CASSANDRA-2825:
----------------------------------------
Attachment: 2825.txt
Patch to exclude the system ks from key sampling when choosing a token, and assert that the token is not our own.
What was happening is that when 0's range was split, it didn't have more than 3 keys, so it used the midpoint. When 85070591730234615865843651857942052864 split, it did have enough keys (in the system ks) and it sampled 61078635599166706937511052402724559481. After this, 61078635599166706937511052402724559481 had the highest load and when it sampled it also arrived at 61078635599166706937511052402724559481 since it was using the system ks.
This patches forces using the midpoint method when there is no data other than the system ks, which is the right thing to do since the system ks data is never moved.
> Auto bootstrapping the 4th node in a 4 node cluster doesn't work, when no token explicitly assigned in config.
> --------------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-2825
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2825
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.0, 0.8.1
> Reporter: Michael Allen
> Assignee: Brandon Williams
> Fix For: 0.8.2
>
> Attachments: 2825.txt
>
>
> This was done in sequence. A, B, C, and D. Node A with token 0 explicitly set in config. The rest with auto_bootstrap: true and no token explicitly assigned. B and C work as expected. D ends up stealing C's token.
> from system.log on C:
> INFO [GossipStage:1] 2011-06-24 16:40:41,947 Gossiper.java (line 638) Node /10.171.47.226 is now part of the cluster
> INFO [GossipStage:1] 2011-06-24 16:40:41,947 Gossiper.java (line 606) InetAddress /10.171.47.226 is now UP
> INFO [GossipStage:1] 2011-06-24 16:42:09,432 StorageService.java (line 769) Nodes /10.171.47.226 and /10.171.55.77 have the same token 61078635599166706937511052402724559481. /10.171.47.226 is the new owner
> WARN [GossipStage:1] 2011-06-24 16:42:09,432 TokenMetadata.java (line 120) Token 61078635599166706937511052402724559481 changing ownership from /10.171.55.77 to /10.171.47.226
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2825) Auto bootstrapping the 4th node
in a 4 node cluster doesn't work, when no token explicitly assigned in
config.
Posted by "Michael Allen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13067348#comment-13067348 ]
Michael Allen commented on CASSANDRA-2825:
------------------------------------------
{noformat}
Address DC Rack Status State Load Owns Token
10.171.46.195 datacenter1 rack1 Up Normal 8.91 KB 100.00% 0
Address DC Rack Status State Load Owns Token
85070591730234615865843651857942052864
10.171.46.195 datacenter1 rack1 Up Normal 8.91 KB 50.00% 0
10.170.121.70 datacenter1 rack1 Up Normal 4.55 KB 50.00% 85070591730234615865843651857942052864
Address DC Rack Status State Load Owns Token
85070591730234615865843651857942052864
10.171.46.195 datacenter1 rack1 Up Normal 8.91 KB 50.00% 0
10.171.34.247 datacenter1 rack1 Up Normal 4.55 KB 35.90% 61078635599166706937511052402724559481
10.170.121.70 datacenter1 rack1 Up Normal 13.4 KB 14.10% 85070591730234615865843651857942052864
AddressDC Rack Status State Load Owns Token
85070591730234615865843651857942052864
10.171.46.195 datacenter1 rack1 Up Normal 8.91 KB 50.00% 0
10.171.23.65 datacenter1 rack1 Up Normal 13.47 KB 35.90% 61078635599166706937511052402724559481
10.170.121.70 datacenter1 rack1 Up Normal 13.4 KB 14.10% 85070591730234615865843651857942052864
{noformat}
> Auto bootstrapping the 4th node in a 4 node cluster doesn't work, when no token explicitly assigned in config.
> --------------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-2825
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2825
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.0, 0.8.1
> Reporter: Michael Allen
> Assignee: Brandon Williams
> Fix For: 0.8.2
>
>
> This was done in sequence. A, B, C, and D. Node A with token 0 explicitly set in config. The rest with auto_bootstrap: true and no token explicitly assigned. B and C work as expected. D ends up stealing C's token.
> from system.log on C:
> INFO [GossipStage:1] 2011-06-24 16:40:41,947 Gossiper.java (line 638) Node /10.171.47.226 is now part of the cluster
> INFO [GossipStage:1] 2011-06-24 16:40:41,947 Gossiper.java (line 606) InetAddress /10.171.47.226 is now UP
> INFO [GossipStage:1] 2011-06-24 16:42:09,432 StorageService.java (line 769) Nodes /10.171.47.226 and /10.171.55.77 have the same token 61078635599166706937511052402724559481. /10.171.47.226 is the new owner
> WARN [GossipStage:1] 2011-06-24 16:42:09,432 TokenMetadata.java (line 120) Token 61078635599166706937511052402724559481 changing ownership from /10.171.55.77 to /10.171.47.226
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2825) Auto bootstrapping the 4th node
in a 4 node cluster doesn't work, when no token explicitly assigned in
config.
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13066285#comment-13066285 ]
Brandon Williams commented on CASSANDRA-2825:
---------------------------------------------
Can you show the ring at each step when you repro?
> Auto bootstrapping the 4th node in a 4 node cluster doesn't work, when no token explicitly assigned in config.
> --------------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-2825
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2825
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.0, 0.8.1
> Reporter: Michael Allen
> Assignee: Brandon Williams
> Fix For: 0.8.2
>
>
> This was done in sequence. A, B, C, and D. Node A with token 0 explicitly set in config. The rest with auto_bootstrap: true and no token explicitly assigned. B and C work as expected. D ends up stealing C's token.
> from system.log on C:
> INFO [GossipStage:1] 2011-06-24 16:40:41,947 Gossiper.java (line 638) Node /10.171.47.226 is now part of the cluster
> INFO [GossipStage:1] 2011-06-24 16:40:41,947 Gossiper.java (line 606) InetAddress /10.171.47.226 is now UP
> INFO [GossipStage:1] 2011-06-24 16:42:09,432 StorageService.java (line 769) Nodes /10.171.47.226 and /10.171.55.77 have the same token 61078635599166706937511052402724559481. /10.171.47.226 is the new owner
> WARN [GossipStage:1] 2011-06-24 16:42:09,432 TokenMetadata.java (line 120) Token 61078635599166706937511052402724559481 changing ownership from /10.171.55.77 to /10.171.47.226
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2825) Auto bootstrapping the 4th node
in a 4 node cluster doesn't work, when no token explicitly assigned in
config.
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Williams updated CASSANDRA-2825:
----------------------------------------
Attachment: 2825-v3.txt
v3 adds a check that tokenMetadata_.getEndpoint(token) is not null when checking for existing membership. This fixes the test.
> Auto bootstrapping the 4th node in a 4 node cluster doesn't work, when no token explicitly assigned in config.
> --------------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-2825
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2825
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.0, 0.8.1
> Reporter: Michael Allen
> Assignee: Brandon Williams
> Fix For: 0.8.3
>
> Attachments: 2825-v2.txt, 2825-v3.txt, 2825.txt
>
>
> This was done in sequence. A, B, C, and D. Node A with token 0 explicitly set in config. The rest with auto_bootstrap: true and no token explicitly assigned. B and C work as expected. D ends up stealing C's token.
> from system.log on C:
> INFO [GossipStage:1] 2011-06-24 16:40:41,947 Gossiper.java (line 638) Node /10.171.47.226 is now part of the cluster
> INFO [GossipStage:1] 2011-06-24 16:40:41,947 Gossiper.java (line 606) InetAddress /10.171.47.226 is now UP
> INFO [GossipStage:1] 2011-06-24 16:42:09,432 StorageService.java (line 769) Nodes /10.171.47.226 and /10.171.55.77 have the same token 61078635599166706937511052402724559481. /10.171.47.226 is the new owner
> WARN [GossipStage:1] 2011-06-24 16:42:09,432 TokenMetadata.java (line 120) Token 61078635599166706937511052402724559481 changing ownership from /10.171.55.77 to /10.171.47.226
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2825) Auto bootstrapping the 4th node
in a 4 node cluster doesn't work, when no token explicitly assigned in
config.
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Williams updated CASSANDRA-2825:
----------------------------------------
Attachment: 2825-v2.txt
v2 detects any token conflicts and throws a proper exception when they occur.
> Auto bootstrapping the 4th node in a 4 node cluster doesn't work, when no token explicitly assigned in config.
> --------------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-2825
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2825
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.0, 0.8.1
> Reporter: Michael Allen
> Assignee: Brandon Williams
> Fix For: 0.8.2
>
> Attachments: 2825-v2.txt, 2825.txt
>
>
> This was done in sequence. A, B, C, and D. Node A with token 0 explicitly set in config. The rest with auto_bootstrap: true and no token explicitly assigned. B and C work as expected. D ends up stealing C's token.
> from system.log on C:
> INFO [GossipStage:1] 2011-06-24 16:40:41,947 Gossiper.java (line 638) Node /10.171.47.226 is now part of the cluster
> INFO [GossipStage:1] 2011-06-24 16:40:41,947 Gossiper.java (line 606) InetAddress /10.171.47.226 is now UP
> INFO [GossipStage:1] 2011-06-24 16:42:09,432 StorageService.java (line 769) Nodes /10.171.47.226 and /10.171.55.77 have the same token 61078635599166706937511052402724559481. /10.171.47.226 is the new owner
> WARN [GossipStage:1] 2011-06-24 16:42:09,432 TokenMetadata.java (line 120) Token 61078635599166706937511052402724559481 changing ownership from /10.171.55.77 to /10.171.47.226
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira