You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Brandon Williams (JIRA)" <ji...@apache.org> on 2013/03/11 19:09:14 UTC
[jira] [Resolved] (CASSANDRA-5332) 1.2.2 Nodetool reports 1.1.6
nodes as down
[ https://issues.apache.org/jira/browse/CASSANDRA-5332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Williams resolved CASSANDRA-5332.
-----------------------------------------
Resolution: Duplicate
See CASSANDRA-5102. The minimum minor revision you can upgrade from is 1.1.7
> 1.2.2 Nodetool reports 1.1.6 nodes as down
> ------------------------------------------
>
> Key: CASSANDRA-5332
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5332
> Project: Cassandra
> Issue Type: Bug
> Environment: Sun Java 6_u39
> Ubuntu 10.04.1
> Cassandra 1.2.2 and 1.1.6
> Reporter: Arya Goudarzi
>
> During exercise of a rolling upgrade from 1.1.6 to 1.2.2, I upgraded one node from 1.1.6 to 1.2.2 today. It appears that node running 1.2.2 reports nodes in 1.1.6 as Down in nodetool. This means that the nodes running 1.1.6 see all other nodes including 1.2.2 as Up. Here is the ring and gossip from nodes with 1.1.6 for example. "231.121" is the upgraded node:
> Address DC Rack Status State Load Effective-Ownership Token
> 141784319550391026443072753098378663700
> XX.180.36 us-east 1b Up Normal 49.47 GB 25.00% 1808575600
> XX.231.121 us-east 1c Up Normal 47.08 GB 25.00% 7089215977519551322153637656637080005
> XX.177.177 us-east 1d Up Normal 33.64 GB 25.00% 14178431955039102644307275311465584410
> XX.7.148 us-east 1b Up Normal 41.27 GB 25.00% 42535295865117307932921825930779602030
> XX.20.9 us-east 1c Up Normal 38.51 GB 25.00% 49624511842636859255075463585608106435
> XX.86.255 us-east 1d Up Normal 34.78 GB 25.00% 56713727820156410577229101240436610840
> XX.63.230 us-east 1b Up Normal 38.11 GB 25.00% 85070591730234615865843651859750628460
> XX.163.36 us-east 1c Up Normal 44.25 GB 25.00% 92159807707754167187997289514579132865
> XX.31.234 us-east 1d Up Normal 44.66 GB 25.00% 99249023685273718510150927169407637270
> XX.132.169 us-east 1b Up Normal 44.2 GB 25.00% 127605887595351923798765477788721654890
> XX.71.63 us-east 1c Up Normal 38.74 GB 25.00% 134695103572871475120919115443550159295
> XX.197.209 us-east 1d Up Normal 41.5 GB 25.00% 141784319550391026443072753098378663700
> /XX.71.63
> RACK:1c
> SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
> LOAD:4.1598705272E10
> DC:us-east
> INTERNAL_IP:XX.194.92
> STATUS:NORMAL,134695103572871475120919115443550159295
> RPC_ADDRESS:XX.194.92
> RELEASE_VERSION:1.1.6
> /XX.86.255
> RACK:1d
> SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
> LOAD:3.734334162E10
> DC:us-east
> INTERNAL_IP:XX.6.195
> STATUS:NORMAL,56713727820156410577229101240436610840
> RPC_ADDRESS:XX.6.195
> RELEASE_VERSION:1.1.6
> /XX.7.148
> RACK:1b
> SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
> LOAD:4.4316975808E10
> DC:us-east
> INTERNAL_IP:XX.47.250
> STATUS:NORMAL,42535295865117307932921825930779602030
> RPC_ADDRESS:XX.47.250
> RELEASE_VERSION:1.1.6
> /XX.63.230
> RACK:1b
> SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
> LOAD:4.0918593305E10
> DC:us-east
> INTERNAL_IP:XX.89.127
> STATUS:NORMAL,85070591730234615865843651859750628460
> RPC_ADDRESS:XX.89.127
> RELEASE_VERSION:1.1.6
> /XX.132.169
> RACK:1b
> SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
> LOAD:4.745883458E10
> DC:us-east
> INTERNAL_IP:XX.94.161
> STATUS:NORMAL,127605887595351923798765477788721654890
> RPC_ADDRESS:XX.94.161
> RELEASE_VERSION:1.1.6
> /XX.180.36
> RACK:1b
> SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
> LOAD:5.311963027E10
> DC:us-east
> INTERNAL_IP:XX.123.112
> STATUS:NORMAL,1808575600
> RPC_ADDRESS:XX.123.112
> RELEASE_VERSION:1.1.6
> /XX.163.36
> RACK:1c
> SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
> LOAD:4.7516755022E10
> DC:us-east
> INTERNAL_IP:XX.163.180
> STATUS:NORMAL,92159807707754167187997289514579132865
> RPC_ADDRESS:XX.163.180
> RELEASE_VERSION:1.1.6
> /XX.31.234
> RACK:1d
> SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
> LOAD:4.7954372912E10
> DC:us-east
> INTERNAL_IP:XX.192.159
> STATUS:NORMAL,99249023685273718510150927169407637270
> RPC_ADDRESS:XX.192.159
> RELEASE_VERSION:1.1.6
> /XX.197.209
> RACK:1d
> SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
> LOAD:4.4558968005E10
> DC:us-east
> INTERNAL_IP:XX.66.205
> STATUS:NORMAL,141784319550391026443072753098378663700
> RPC_ADDRESS:XX.66.205
> RELEASE_VERSION:1.1.6
> /XX.177.177
> RACK:1d
> SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
> LOAD:3.6115572697E10
> DC:us-east
> INTERNAL_IP:XX.65.57
> STATUS:NORMAL,14178431955039102644307275311465584410
> RPC_ADDRESS:XX.65.57
> RELEASE_VERSION:1.1.6
> /XX.20.9
> RACK:1c
> SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
> LOAD:4.1352503882E10
> DC:us-east
> INTERNAL_IP:XX.33.229
> STATUS:NORMAL,49624511842636859255075463585608106435
> RPC_ADDRESS:XX.33.229
> RELEASE_VERSION:1.1.6
> /XX.231.121
> RACK:1c
> SCHEMA:09487aa5-3380-33ab-b9a5-bcc8476066b0
> X4:9c765678-d058-4d85-a588-638ce10ff984
> X3:7
> DC:us-east
> INTERNAL_IP:XX.223.241
> RPC_ADDRESS:XX.223.241
> RELEASE_VERSION:1.2.2
> Now the nodetool on the 1.2.2 node shows all nodes as Down but itself. Gossipinfo looks good though:
> Datacenter: us-east
> ==========
> Replicas: 3
> Address Rack Status State Load Owns Token
> 56713727820156410577229101240436610840
> XX.132.169 1b Down Normal 44.2 GB 25.00% 127605887595351923798765477788721654890
> XX.7.148 1b Down Normal 41.27 GB 25.00% 42535295865117307932921825930779602030
> XX.180.36 1b Down Normal 49.47 GB 25.00% 1808575600
> XX.63.230 1b Down Normal 38.11 GB 25.00% 85070591730234615865843651859750628460
> XX.231.121 1c Up Normal 47.25 GB 25.00% 7089215977519551322153637656637080005
> XX.71.63 1c Down Normal 38.74 GB 25.00% 134695103572871475120919115443550159295
> XX.177.177 1d Down Normal 33.64 GB 25.00% 14178431955039102644307275311465584410
> XX.31.234 1d Down Normal 44.66 GB 25.00% 99249023685273718510150927169407637270
> XX.20.9 1c Down Normal 38.51 GB 25.00% 49624511842636859255075463585608106435
> XX.163.36 1c Down Normal 44.25 GB 25.00% 92159807707754167187997289514579132865
> XX.197.209 1d Down Normal 41.5 GB 25.00% 141784319550391026443072753098378663700
> XX.86.255 1d Down Normal 34.78 GB 25.00% 56713727820156410577229101240436610840
> /XX.71.63
> RACK:1c
> RPC_ADDRESS:XX.194.92
> RELEASE_VERSION:1.1.6
> INTERNAL_IP:XX.194.92
> STATUS:NORMAL,134695103572871475120919115443550159295
> SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
> DC:us-east
> LOAD:4.1598705272E10
> /XX.86.255
> RACK:1d
> RPC_ADDRESS:XX.6.195
> RELEASE_VERSION:1.1.6
> INTERNAL_IP:XX.6.195
> STATUS:NORMAL,56713727820156410577229101240436610840
> SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
> DC:us-east
> LOAD:3.7343205002E10
> /XX.7.148
> RACK:1b
> RPC_ADDRESS:XX.47.250
> RELEASE_VERSION:1.1.6
> INTERNAL_IP:XX.47.250
> STATUS:NORMAL,42535295865117307932921825930779602030
> SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
> DC:us-east
> LOAD:4.4316975808E10
> /XX.63.230
> RACK:1b
> RPC_ADDRESS:XX.89.127
> RELEASE_VERSION:1.1.6
> INTERNAL_IP:XX.89.127
> STATUS:NORMAL,85070591730234615865843651859750628460
> SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
> DC:us-east
> LOAD:4.0918456687E10
> /XX.132.169
> RACK:1b
> RPC_ADDRESS:XX.94.161
> RELEASE_VERSION:1.1.6
> INTERNAL_IP:XX.94.161
> STATUS:NORMAL,127605887595351923798765477788721654890
> SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
> DC:us-east
> LOAD:4.745883458E10
> /XX.180.36
> RACK:1b
> RPC_ADDRESS:XX.123.112
> RELEASE_VERSION:1.1.6
> INTERNAL_IP:XX.123.112
> STATUS:NORMAL,1808575600
> SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
> DC:us-east
> LOAD:5.311963027E10
> /XX.163.36
> RACK:1c
> RPC_ADDRESS:XX.163.180
> RELEASE_VERSION:1.1.6
> INTERNAL_IP:XX.163.180
> STATUS:NORMAL,92159807707754167187997289514579132865
> SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
> DC:us-east
> LOAD:4.7516755022E10
> /XX.31.234
> RACK:1d
> RPC_ADDRESS:XX.192.159
> RELEASE_VERSION:1.1.6
> INTERNAL_IP:XX.192.159
> STATUS:NORMAL,99249023685273718510150927169407637270
> SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
> DC:us-east
> LOAD:4.7954372912E10
> /XX.197.209
> RACK:1d
> RPC_ADDRESS:XX.66.205
> RELEASE_VERSION:1.1.6
> INTERNAL_IP:XX.66.205
> STATUS:NORMAL,141784319550391026443072753098378663700
> SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
> DC:us-east
> LOAD:4.4559013211E10
> /XX.177.177
> RACK:1d
> RPC_ADDRESS:XX.65.57
> RELEASE_VERSION:1.1.6
> INTERNAL_IP:XX.65.57
> STATUS:NORMAL,14178431955039102644307275311465584410
> SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
> DC:us-east
> LOAD:3.6115572697E10
> /XX.20.9
> RACK:1c
> RPC_ADDRESS:XX.33.229
> RELEASE_VERSION:1.1.6
> INTERNAL_IP:XX.33.229
> STATUS:NORMAL,49624511842636859255075463585608106435
> SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
> DC:us-east
> LOAD:4.1352367264E10
> /XX.231.121
> HOST_ID:9c765678-d058-4d85-a588-638ce10ff984
> RACK:1c
> RPC_ADDRESS:XX.223.241
> RELEASE_VERSION:1.2.2
> INTERNAL_IP:XX.223.241
> STATUS:NORMAL,7089215977519551322153637656637080005
> NET_VERSION:7
> SCHEMA:8b8948f5-d56f-3a96-8005-b9452e42cd67
> SEVERITY:0.0
> DC:us-east
> LOAD:5.0710624207E10
> Is this just a display bug in nodetool or this upgraded node really sees the other ones as dead?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira