You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Wojciech Meler (JIRA)" <ji...@apache.org> on 2011/07/30 05:58:09 UTC
[jira] [Created] (CASSANDRA-2974) nodetool removetoken hang
nodetool removetoken hang
-------------------------
Key: CASSANDRA-2974
URL: https://issues.apache.org/jira/browse/CASSANDRA-2974
Project: Cassandra
Issue Type: Bug
Affects Versions: 0.8.1
Reporter: Wojciech Meler
one node died - i tried to remove it with removetoken but it hanged:
RemovalStatus: Removing token (9452287970026068429538183539771339207). Waiting for replication confirmation from [/10.0.3.78].
nodetool netstats doesn't show any streams:
# nodetool -h 10.0.3.65 -p 8080 netstats
Mode: Normal
Not sending any streams.
Not receiving any streams.
Pool Name Active Pending Completed
Commands n/a 0 332578
Responses n/a 0 646405
# nodetool -h 10.0.3.66 -p 8080 netstats
Mode: Normal
Nothing streaming to /10.0.3.78
Not receiving any streams.
Pool Name Active Pending Completed
Commands n/a 178 739797
Responses n/a 0 1294349
# nodetool -h 10.0.3.71 -p 8080 netstats
Mode: Normal
Not sending any streams.
Not receiving any streams.
Pool Name Active Pending Completed
Commands n/a 84 2031299
Responses n/a 0 357749
BTW. "Nothing streaming to /10.0.3.78" is quite funny :)
also nodetool ring show strange things - almost whole nodes report:
10.0.3.65 datacenter1 rack1 Up Normal 114.2 GB 5.56% 0
10.0.3.77 datacenter1 rack1 Down Leaving 158.09 GB 5.56% 9452287970026068429538183539771339207
10.0.3.71 datacenter1 rack1 Up Normal 196.76 GB 5.56% 18904575940052136859076367079542678414
10.0.3.66 datacenter1 rack1 Up Normal 178.95 GB 5.56% 28356863910078205288614550619314017621
10.0.3.78 datacenter1 rack1 Up Normal 227.05 GB 5.56% 37809151880104273718152734159085356828
10.0.3.72 datacenter1 rack1 Up Normal 110.83 GB 5.56% 47261439850130342147690917698856696035
10.0.3.67 datacenter1 rack1 Up Normal 117.45 GB 5.56% 56713727820156410577229101238628035242
10.0.3.79 datacenter1 rack1 Up Normal 138.62 GB 5.56% 66166015790182479006767284778399374449
10.0.3.73 datacenter1 rack1 Up Normal 110.49 GB 5.56% 75618303760208547436305468318170713656
10.0.3.68 datacenter1 rack1 Up Normal 114.82 GB 5.56% 85070591730234615865843651857942052863
10.0.3.80 datacenter1 rack1 Up Normal 145.51 GB 5.56% 94522879700260684295381835397713392070
10.0.3.74 datacenter1 rack1 Up Normal 113.63 GB 5.56% 103975167670286752724920018937484731277
10.0.3.69 datacenter1 rack1 Up Normal 111.24 GB 5.56% 113427455640312821154458202477256070484
10.0.3.81 datacenter1 rack1 Up Normal 142.12 GB 5.56% 122879743610338889583996386017027409691
10.0.3.75 datacenter1 rack1 Up Normal 110.87 GB 5.56% 132332031580364958013534569556798748898
10.0.3.70 datacenter1 rack1 Up Normal 113.21 GB 5.56% 141784319550391026443072753096570088105
10.0.3.82 datacenter1 rack1 Up Normal 163.29 GB 5.56% 151236607520417094872610936636341427312
10.0.3.76 datacenter1 rack1 Up Normal 112.68 GB 5.56% 160688895490443163302149120176112766519
but 2 nodes which should take responsibility for removed token says:
10.0.3.65 datacenter1 rack1 Up Normal 114.2 GB 5.56% 0
10.0.3.77 datacenter1 rack1 Up Leaving 158.09 GB 5.56% 9452287970026068429538183539771339207
10.0.3.71 datacenter1 rack1 Up Normal 196.76 GB 5.56% 18904575940052136859076367079542678414
10.0.3.66 datacenter1 rack1 Up Normal 178.95 GB 5.56% 28356863910078205288614550619314017621
10.0.3.78 datacenter1 rack1 Up Normal 227.05 GB 5.56% 37809151880104273718152734159085356828
10.0.3.72 datacenter1 rack1 Up Normal 110.83 GB 5.56% 47261439850130342147690917698856696035
10.0.3.67 datacenter1 rack1 Up Normal 117.45 GB 5.56% 56713727820156410577229101238628035242
10.0.3.79 datacenter1 rack1 Up Normal 138.62 GB 5.56% 66166015790182479006767284778399374449
10.0.3.73 datacenter1 rack1 Up Normal 110.49 GB 5.56% 75618303760208547436305468318170713656
10.0.3.68 datacenter1 rack1 Up Normal 114.82 GB 5.56% 85070591730234615865843651857942052863
10.0.3.80 datacenter1 rack1 Up Normal 145.51 GB 5.56% 94522879700260684295381835397713392070
10.0.3.74 datacenter1 rack1 Up Normal 113.63 GB 5.56% 103975167670286752724920018937484731277
10.0.3.69 datacenter1 rack1 Up Normal 111.24 GB 5.56% 113427455640312821154458202477256070484
10.0.3.81 datacenter1 rack1 Up Normal 142.12 GB 5.56% 122879743610338889583996386017027409691
10.0.3.75 datacenter1 rack1 Up Normal 110.87 GB 5.56% 132332031580364958013534569556798748898
10.0.3.70 datacenter1 rack1 Up Normal 113.21 GB 5.56% 141784319550391026443072753096570088105
10.0.3.82 datacenter1 rack1 Up Normal 163.29 GB 5.56% 151236607520417094872610936636341427312
10.0.3.76 datacenter1 rack1 Up Normal 112.68 GB 5.56% 160688895490443163302149120176112766519
they started to report it as UP after nodetool removetoken was called
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CASSANDRA-2974) nodetool removetoken hang
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis resolved CASSANDRA-2974.
---------------------------------------
Resolution: Duplicate
removetoken was fairly broken prior to CASSANDRA-2496 (fixed for 0.8.3).
> nodetool removetoken hang
> -------------------------
>
> Key: CASSANDRA-2974
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2974
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.8.1
> Reporter: Wojciech Meler
>
> one node died - i tried to remove it with removetoken but it hanged:
> RemovalStatus: Removing token (9452287970026068429538183539771339207). Waiting for replication confirmation from [/10.0.3.78].
> nodetool netstats doesn't show any streams:
> # nodetool -h 10.0.3.65 -p 8080 netstats
> Mode: Normal
> Not sending any streams.
> Not receiving any streams.
> Pool Name Active Pending Completed
> Commands n/a 0 332578
> Responses n/a 0 646405
> # nodetool -h 10.0.3.66 -p 8080 netstats
> Mode: Normal
> Nothing streaming to /10.0.3.78
> Not receiving any streams.
> Pool Name Active Pending Completed
> Commands n/a 178 739797
> Responses n/a 0 1294349
> # nodetool -h 10.0.3.71 -p 8080 netstats
> Mode: Normal
> Not sending any streams.
> Not receiving any streams.
> Pool Name Active Pending Completed
> Commands n/a 84 2031299
> Responses n/a 0 357749
> BTW. "Nothing streaming to /10.0.3.78" is quite funny :)
> also nodetool ring show strange things - almost whole nodes report:
> 10.0.3.65 datacenter1 rack1 Up Normal 114.2 GB 5.56% 0
> 10.0.3.77 datacenter1 rack1 Down Leaving 158.09 GB 5.56% 9452287970026068429538183539771339207
> 10.0.3.71 datacenter1 rack1 Up Normal 196.76 GB 5.56% 18904575940052136859076367079542678414
> 10.0.3.66 datacenter1 rack1 Up Normal 178.95 GB 5.56% 28356863910078205288614550619314017621
> 10.0.3.78 datacenter1 rack1 Up Normal 227.05 GB 5.56% 37809151880104273718152734159085356828
> 10.0.3.72 datacenter1 rack1 Up Normal 110.83 GB 5.56% 47261439850130342147690917698856696035
> 10.0.3.67 datacenter1 rack1 Up Normal 117.45 GB 5.56% 56713727820156410577229101238628035242
> 10.0.3.79 datacenter1 rack1 Up Normal 138.62 GB 5.56% 66166015790182479006767284778399374449
> 10.0.3.73 datacenter1 rack1 Up Normal 110.49 GB 5.56% 75618303760208547436305468318170713656
> 10.0.3.68 datacenter1 rack1 Up Normal 114.82 GB 5.56% 85070591730234615865843651857942052863
> 10.0.3.80 datacenter1 rack1 Up Normal 145.51 GB 5.56% 94522879700260684295381835397713392070
> 10.0.3.74 datacenter1 rack1 Up Normal 113.63 GB 5.56% 103975167670286752724920018937484731277
> 10.0.3.69 datacenter1 rack1 Up Normal 111.24 GB 5.56% 113427455640312821154458202477256070484
> 10.0.3.81 datacenter1 rack1 Up Normal 142.12 GB 5.56% 122879743610338889583996386017027409691
> 10.0.3.75 datacenter1 rack1 Up Normal 110.87 GB 5.56% 132332031580364958013534569556798748898
> 10.0.3.70 datacenter1 rack1 Up Normal 113.21 GB 5.56% 141784319550391026443072753096570088105
> 10.0.3.82 datacenter1 rack1 Up Normal 163.29 GB 5.56% 151236607520417094872610936636341427312
> 10.0.3.76 datacenter1 rack1 Up Normal 112.68 GB 5.56% 160688895490443163302149120176112766519
> but 2 nodes which should take responsibility for removed token says:
> 10.0.3.65 datacenter1 rack1 Up Normal 114.2 GB 5.56% 0
> 10.0.3.77 datacenter1 rack1 Up Leaving 158.09 GB 5.56% 9452287970026068429538183539771339207
> 10.0.3.71 datacenter1 rack1 Up Normal 196.76 GB 5.56% 18904575940052136859076367079542678414
> 10.0.3.66 datacenter1 rack1 Up Normal 178.95 GB 5.56% 28356863910078205288614550619314017621
> 10.0.3.78 datacenter1 rack1 Up Normal 227.05 GB 5.56% 37809151880104273718152734159085356828
> 10.0.3.72 datacenter1 rack1 Up Normal 110.83 GB 5.56% 47261439850130342147690917698856696035
> 10.0.3.67 datacenter1 rack1 Up Normal 117.45 GB 5.56% 56713727820156410577229101238628035242
> 10.0.3.79 datacenter1 rack1 Up Normal 138.62 GB 5.56% 66166015790182479006767284778399374449
> 10.0.3.73 datacenter1 rack1 Up Normal 110.49 GB 5.56% 75618303760208547436305468318170713656
> 10.0.3.68 datacenter1 rack1 Up Normal 114.82 GB 5.56% 85070591730234615865843651857942052863
> 10.0.3.80 datacenter1 rack1 Up Normal 145.51 GB 5.56% 94522879700260684295381835397713392070
> 10.0.3.74 datacenter1 rack1 Up Normal 113.63 GB 5.56% 103975167670286752724920018937484731277
> 10.0.3.69 datacenter1 rack1 Up Normal 111.24 GB 5.56% 113427455640312821154458202477256070484
> 10.0.3.81 datacenter1 rack1 Up Normal 142.12 GB 5.56% 122879743610338889583996386017027409691
> 10.0.3.75 datacenter1 rack1 Up Normal 110.87 GB 5.56% 132332031580364958013534569556798748898
> 10.0.3.70 datacenter1 rack1 Up Normal 113.21 GB 5.56% 141784319550391026443072753096570088105
> 10.0.3.82 datacenter1 rack1 Up Normal 163.29 GB 5.56% 151236607520417094872610936636341427312
> 10.0.3.76 datacenter1 rack1 Up Normal 112.68 GB 5.56% 160688895490443163302149120176112766519
> they started to report it as UP after nodetool removetoken was called
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira