You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Michael Shuler (JIRA)" <ji...@apache.org> on 2014/01/07 18:09:50 UTC
[jira] [Commented] (CASSANDRA-6554) Cluster is read-only during
upgrade of nodes from 1.2 -> 2.0
[ https://issues.apache.org/jira/browse/CASSANDRA-6554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13864403#comment-13864403 ]
Michael Shuler commented on CASSANDRA-6554:
-------------------------------------------
So... this morning, while carefully running through my steps to reproduce, the upgraded node0 accepts writes via stress, as well as node1 and node2.
{code}
mshuler@hana:~$ ctool launch -i m1.medium -j 1.7_25 upgradetest 3
mshuler@hana:~$ ctool install -v 1.2.13 -i tar -t http://www.us.apache.org/dist/cassandra/1.2.13/apache-cassandra-1.2.13-bin.tar.gz upgradetest cassandra
Entering the API (tail -f /home/mshuler/.automaton/automaton.log for more info)
mshuler@hana:~$ ctool start upgradetest cassandra
----
mshuler@hana:~$ ctool ssh upgradetest 0
<...>
automaton@ip-10-139-1-113:~$ ./dsc-cassandra/tools/bin/cassandra-stress
Created keyspaces. Sleeping 1s for propagation.
total,interval_op_rate,interval_key_rate,latency/95th/99.9th,elapsed_time
3869,386,386,104.3,210.7,327.0,10
10832,696,696,84.9,187.9,292.6,20
19889,905,905,72.8,168.2,275.7,30
<...>
966878,2595,2595,24.1,31.1,195.2,435
994244,2736,2736,24.0,31.1,195.2,445
1000000,575,575,23.8,31.1,195.2,447
END
automaton@ip-10-139-1-113:~$ nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Owns (effective) Host ID Token Rack
UN 10.139.11.168 97.51 MB 33.3% ec97c163-8f2d-4019-a3d1-55df5e4037d4 -3074457345618258603 rack1
UN 10.139.1.113 97.44 MB 33.3% 33b1cd06-e17b-4332-8066-0c6c401e0cf3 -9223372036854775808 rack1
UN 10.238.221.115 97.34 MB 33.3% 73a76d3f-73ef-481d-b603-0833c0ff80c2 3074457345618258602 rack1
automaton@ip-10-139-1-113:~$ nodetool gossipinfo
/10.238.221.115
RELEASE_VERSION:1.2.13
DC:datacenter1
STATUS:NORMAL,3074457345618258602
RPC_ADDRESS:0.0.0.0
NET_VERSION:6
LOAD:1.02066255E8
SCHEMA:8b351435-81ef-3a14-adf7-8555e2f19ecd
RACK:rack1
SEVERITY:0.0
HOST_ID:73a76d3f-73ef-481d-b603-0833c0ff80c2
/10.139.1.113
RELEASE_VERSION:1.2.13
DC:datacenter1
STATUS:NORMAL,-9223372036854775808
RPC_ADDRESS:0.0.0.0
NET_VERSION:6
LOAD:9.2039687E7
SCHEMA:8b351435-81ef-3a14-adf7-8555e2f19ecd
RACK:rack1
SEVERITY:0.0
HOST_ID:33b1cd06-e17b-4332-8066-0c6c401e0cf3
/10.139.11.168
RELEASE_VERSION:1.2.13
DC:datacenter1
STATUS:NORMAL,-3074457345618258603
RPC_ADDRESS:0.0.0.0
NET_VERSION:6
LOAD:1.02245066E8
SCHEMA:8b351435-81ef-3a14-adf7-8555e2f19ecd
RACK:rack1
SEVERITY:0.0
HOST_ID:ec97c163-8f2d-4019-a3d1-55df5e4037d4
----
automaton@ip-10-139-1-113:~$ wget -q http://www.us.apache.org/dist/cassandra/2.0.4/apache-cassandra-2.0.4-bin.tar.gz
automaton@ip-10-139-1-113:~$ tar xzf apache-cassandra-2.0.4-bin.tar.gz
automaton@ip-10-139-1-113:~$ cp -p apache-cassandra-2.0.4/conf/cassandra.yaml apache-cassandra-2.0.4/conf/cassandra.yaml.dist
automaton@ip-10-139-1-113:~$ vi apache-cassandra-2.0.4/conf/cassandra.yaml
[transfer values from 1.2.13 config]
automaton@ip-10-139-1-113:~$ diff apache-cassandra-2.0.4/conf/cassandra.yaml.dist apache-cassandra-2.0.4/conf/cassandra.yaml
10c10
< cluster_name: 'Test Cluster'
---
> cluster_name: upgradetest
24c24
< num_tokens: 256
---
> #num_tokens: 256
30c30
< # initial_token:
---
> initial_token: -9223372036854775808
227c227
< - seeds: "127.0.0.1"
---
> - seeds: 10.139.1.113,10.139.11.168
297c297
< listen_address: localhost
---
> listen_address: 10.139.1.113
335c335
< rpc_address: localhost
---
> rpc_address: 0.0.0.0
----
automaton@ip-10-139-1-113:~$ nodetool drain
automaton@ip-10-139-1-113:~$ ps aux | grep [j]ava
automaton@ip-10-139-1-113:~$ mv dsc-cassandra/ dsc-cassandra_1.2.13
automaton@ip-10-139-1-113:~$ mv apache-cassandra-2.0.4 dsc-cassandra
----
mshuler@hana:~$ ctool start upgradetest -n 0 cassandra
----
automaton@ip-10-139-1-113:~$ nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Owns Host ID Token Rack
UN 10.139.1.113 98.94 MB 33.3% 33b1cd06-e17b-4332-8066-0c6c401e0cf3 -9223372036854775808 rack1
DN 10.139.11.168 97.51 MB 33.3% ec97c163-8f2d-4019-a3d1-55df5e4037d4 -3074457345618258603 rack1
DN 10.238.221.115 97.34 MB 33.3% 73a76d3f-73ef-481d-b603-0833c0ff80c2 3074457345618258602 rack1
automaton@ip-10-139-1-113:~$ nodetool gossipinfo
/10.238.221.115
SEVERITY:0.0
RPC_ADDRESS:0.0.0.0
DC:datacenter1
RELEASE_VERSION:1.2.13
LOAD:1.02066255E8
STATUS:NORMAL,3074457345618258602
SCHEMA:8b351435-81ef-3a14-adf7-8555e2f19ecd
NET_VERSION:6
RACK:rack1
HOST_ID:73a76d3f-73ef-481d-b603-0833c0ff80c2
/10.139.1.113
RPC_ADDRESS:0.0.0.0
SEVERITY:0.0
DC:datacenter1
RELEASE_VERSION:2.0.4
LOAD:1.03750451E8
STATUS:NORMAL,-9223372036854775808
SCHEMA:dfafb212-5b8f-31cb-a80b-2ba58fcef73d
NET_VERSION:7
RACK:rack1
HOST_ID:33b1cd06-e17b-4332-8066-0c6c401e0cf3
/10.139.11.168
SEVERITY:0.0
RPC_ADDRESS:0.0.0.0
DC:datacenter1
RELEASE_VERSION:1.2.13
LOAD:1.02245066E8
STATUS:NORMAL,-3074457345618258603
SCHEMA:8b351435-81ef-3a14-adf7-8555e2f19ecd
NET_VERSION:6
RACK:rack1
HOST_ID:ec97c163-8f2d-4019-a3d1-55df5e4037d4
automaton@ip-10-139-1-113:~$ ./dsc-cassandra/tools/bin/cassandra-stress
Unable to create stress keyspace: Keyspace names must be case-insensitively unique ("Keyspace1" conflicts with "Keyspace1")
total,interval_op_rate,interval_key_rate,latency,95th,99.9th,elapsed_time
3553,355,355,111.6,195.9,452.3,10
8296,474,474,119.8,199.8,449.7,21
13498,520,520,117.9,197.3,438.7,31
<...>
940610,2883,2883,22.3,29.2,159.4,395
971153,3054,3054,22.3,29.0,159.4,405
1000000,2884,2884,22.0,28.8,160.0,415
Averages from the middle 80% of values:
interval_op_rate : 2835
interval_key_rate : 2835
latency median : 24.2
latency 95th percentile : 46.5
latency 99.9th percentile : 385.9
Total operation time : 00:06:55
END
automaton@ip-10-139-1-113:~$
{code}
> Cluster is read-only during upgrade of nodes from 1.2 -> 2.0
> ------------------------------------------------------------
>
> Key: CASSANDRA-6554
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6554
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: EC2 Ubuntu Precise 12.04
> Oracle JRE 1.7_25
> C* 1.2.13 upgrade to 2.0.4
> Reporter: Michael Shuler
>
> During an upgrade from 1.2.13 to 2.0.3/2.0.4, the cluster is read-only and writes fail, until the entire cluster is fully upgraded.
> (I'm gathering complete repro steps, test results, and logs to try to help and will post those asap, as well as try other versions to see what happens)
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)