You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Cathy Daw (Created) (JIRA)" <ji...@apache.org> on 2011/10/01 02:56:45 UTC
[jira] [Created] (CASSANDRA-3290) Upgrade from 0.7.9 to 1.0, the
java stress tool gives "error reading key" on data created in 0.7.9
Upgrade from 0.7.9 to 1.0, the java stress tool gives "error reading key" on data created in 0.7.9
--------------------------------------------------------------------------------------------------
Key: CASSANDRA-3290
URL: https://issues.apache.org/jira/browse/CASSANDRA-3290
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.0.0
Reporter: Cathy Daw
*TEST CASE*
* Install three node cluster with C* 0.7.9 with node1 as the seed
* Run the 0.7.9 stress tool against node2,node3: ./stress -d node2,node3 -o insert
* Run `nodetool -h node1 drain`
* Upgrade binaries and edit cassandra.yaml for node1.
* Startup C* 1.0 on node
* Run the 0.7.9 stress tool against node2,node3: ./stress -d node2,node3 -o insert
*ISSUE*
* Run the 0.7.9 stress tool to read data: ./stress -d node2 -o read
{code}
cathy3:~/apache-cassandra-0.7.9-src/contrib/stress/bin# ./stress -d cathy3 -o read
total,interval_op_rate,interval_key_rate,avg_latency,elapsed_time
Operation [17] retried 10 times - error reading key 0521977
Operation [9] retried 10 times - error reading key 0595365
Operation [15] retried 10 times - error reading key 0513247
{code}
*Additional notes*
* There are no errors in the log files
* After upgrading the full ring, and running READ from both the old and new versions of stress tool, the error still exists
* Once all nodes are upgraded, if I run INSERT followed by READ, the new reads are fine
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3290) Upgrade from 0.7.9 to 1.0, the
java stress tool gives "error reading key" on data created in 0.7.9
Posted by "Jonathan Ellis (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-3290:
--------------------------------------
Fix Version/s: 1.0.0
Assignee: Brandon Williams
> Upgrade from 0.7.9 to 1.0, the java stress tool gives "error reading key" on data created in 0.7.9
> --------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-3290
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3290
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Reporter: Cathy Daw
> Assignee: Brandon Williams
> Fix For: 1.0.0
>
>
> *TEST CASE*
> * Install three node cluster with C* 0.7.9 with node1 as the seed
> * Run the 0.7.9 stress tool against node2,node3: ./stress -d node2,node3 -o insert
> * Run `nodetool -h node1 drain`
> * Upgrade binaries and edit cassandra.yaml for node1.
> * Startup C* 1.0 on node
> * Run the 0.7.9 stress tool against node2,node3: ./stress -d node2,node3 -o insert
> *ISSUE*
> * Run the 0.7.9 stress tool to read data: ./stress -d node2 -o read
> {code}
> cathy3:~/apache-cassandra-0.7.9-src/contrib/stress/bin# ./stress -d cathy3 -o read
> total,interval_op_rate,interval_key_rate,avg_latency,elapsed_time
> Operation [17] retried 10 times - error reading key 0521977
> Operation [9] retried 10 times - error reading key 0595365
> Operation [15] retried 10 times - error reading key 0513247
> {code}
> *Additional notes*
> * There are no errors in the log files
> * After upgrading the full ring, and running READ from both the old and new versions of stress tool, the error still exists
> * Once all nodes are upgraded, if I run INSERT followed by READ, the new reads are fine
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-3290) Upgrade from 0.7.9
to 1.0, the java stress tool gives "error reading key" on data created in
0.7.9
Posted by "Cathy Daw (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118677#comment-13118677 ]
Cathy Daw edited comment on CASSANDRA-3290 at 10/1/11 5:33 AM:
---------------------------------------------------------------
I see the same issue upgrading from 0.8.2, but now I see the following exception. In this case, one node was drained and upgraded to 1.0 and the other two nodes were still at 0.8.2.
This is a pretty straight forward procedure using a vanilla install and the out of the box stress scripts.
{code}
ERROR 05:27:33,751 Fatal exception in thread Thread[HintedHandoff:1,5,main]
java.lang.RuntimeException: java.lang.RuntimeException: Could not reach schema agreement with /50.57.111.189 in 60000ms
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.RuntimeException: Could not reach schema agreement with /50.57.111.189 in 60000ms
at org.apache.cassandra.db.HintedHandOffManager.waitForSchemaAgreement(HintedHandOffManager.java:220)
at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:231)
at org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager.java:81)
at org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:333)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
... 3 more
ERROR 05:27:33,753 Fatal exception in thread Thread[HintedHandoff:1,5,main]
java.lang.RuntimeException: java.lang.RuntimeException: Could not reach schema agreement with /50.57.111.189 in 60000ms
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.RuntimeException: Could not reach schema agreement with /50.57.111.189 in 60000ms
at org.apache.cassandra.db.HintedHandOffManager.waitForSchemaAgreement(HintedHandOffManager.java:220)
at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:231)
at org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager.java:81)
at org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:333)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
... 3 more
{code}
*Node Tool output for 1 node in 1.0.0 and the other two at 0.8.2*
{code}
cathy1:~/apache-cassandra-1.0.0-rc1-src/bin# ./nodetool -h cathy1 ring
Address DC Rack Status State Load Owns Token
113427455640312821154458202477256070484
50.57.111.111 datacenter1 rack1 Up Normal 181.56 MB 33.33% 0
50.57.111.126 datacenter1 rack1 Up Normal 178.84 MB 33.33% 56713727820156410577229101238628035242
50.57.111.189 datacenter1 rack1 Up Normal 180.2 MB 33.33% 113427455640312821154458202477256070484
{code}
was (Author: cdaw):
I see the same issue upgrading from 0.8.2
> Upgrade from 0.7.9 to 1.0, the java stress tool gives "error reading key" on data created in 0.7.9
> --------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-3290
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3290
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Reporter: Cathy Daw
> Assignee: Brandon Williams
> Fix For: 1.0.0
>
>
> *TEST CASE*
> * Install three node cluster with C* 0.7.9 with node1 as the seed
> * Run the 0.7.9 stress tool against node2,node3: ./stress -d node2,node3 -o insert
> * Run `nodetool -h node1 drain`
> * Upgrade binaries and edit cassandra.yaml for node1.
> * Startup C* 1.0 on node
> * Run the 0.7.9 stress tool against node2,node3: ./stress -d node2,node3 -o insert
> *ISSUE*
> * Run the 0.7.9 stress tool to read data: ./stress -d node2 -o read
> {code}
> cathy3:~/apache-cassandra-0.7.9-src/contrib/stress/bin# ./stress -d cathy3 -o read
> total,interval_op_rate,interval_key_rate,avg_latency,elapsed_time
> Operation [17] retried 10 times - error reading key 0521977
> Operation [9] retried 10 times - error reading key 0595365
> Operation [15] retried 10 times - error reading key 0513247
> {code}
> *Additional notes*
> * There are no errors in the log files
> * After upgrading the full ring, and running READ from both the old and new versions of stress tool, the error still exists
> * Once all nodes are upgraded, if I run INSERT followed by READ, the new reads are fine
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3290) Upgrade from 0.7.9 to 1.0, the
java stress tool gives "error reading key" on data created in 0.7.9
Posted by "Cathy Daw (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118677#comment-13118677 ]
Cathy Daw commented on CASSANDRA-3290:
--------------------------------------
I see the same issue upgrading from 0.8.2
> Upgrade from 0.7.9 to 1.0, the java stress tool gives "error reading key" on data created in 0.7.9
> --------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-3290
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3290
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Reporter: Cathy Daw
> Assignee: Brandon Williams
> Fix For: 1.0.0
>
>
> *TEST CASE*
> * Install three node cluster with C* 0.7.9 with node1 as the seed
> * Run the 0.7.9 stress tool against node2,node3: ./stress -d node2,node3 -o insert
> * Run `nodetool -h node1 drain`
> * Upgrade binaries and edit cassandra.yaml for node1.
> * Startup C* 1.0 on node
> * Run the 0.7.9 stress tool against node2,node3: ./stress -d node2,node3 -o insert
> *ISSUE*
> * Run the 0.7.9 stress tool to read data: ./stress -d node2 -o read
> {code}
> cathy3:~/apache-cassandra-0.7.9-src/contrib/stress/bin# ./stress -d cathy3 -o read
> total,interval_op_rate,interval_key_rate,avg_latency,elapsed_time
> Operation [17] retried 10 times - error reading key 0521977
> Operation [9] retried 10 times - error reading key 0595365
> Operation [15] retried 10 times - error reading key 0513247
> {code}
> *Additional notes*
> * There are no errors in the log files
> * After upgrading the full ring, and running READ from both the old and new versions of stress tool, the error still exists
> * Once all nodes are upgraded, if I run INSERT followed by READ, the new reads are fine
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3290) Upgrade from 0.7.9 to 1.0, the
java stress tool gives "error reading key" on data created in 0.7.9
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118675#comment-13118675 ]
Jonathan Ellis commented on CASSANDRA-3290:
-------------------------------------------
CASSANDRA-3289 would fit the description, except you should be seeing AssertionErrors in the log if that were the case. Are you running a custom script or did you otherwise turn off -ea, by chance?
> Upgrade from 0.7.9 to 1.0, the java stress tool gives "error reading key" on data created in 0.7.9
> --------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-3290
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3290
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Reporter: Cathy Daw
> Assignee: Brandon Williams
> Fix For: 1.0.0
>
>
> *TEST CASE*
> * Install three node cluster with C* 0.7.9 with node1 as the seed
> * Run the 0.7.9 stress tool against node2,node3: ./stress -d node2,node3 -o insert
> * Run `nodetool -h node1 drain`
> * Upgrade binaries and edit cassandra.yaml for node1.
> * Startup C* 1.0 on node
> * Run the 0.7.9 stress tool against node2,node3: ./stress -d node2,node3 -o insert
> *ISSUE*
> * Run the 0.7.9 stress tool to read data: ./stress -d node2 -o read
> {code}
> cathy3:~/apache-cassandra-0.7.9-src/contrib/stress/bin# ./stress -d cathy3 -o read
> total,interval_op_rate,interval_key_rate,avg_latency,elapsed_time
> Operation [17] retried 10 times - error reading key 0521977
> Operation [9] retried 10 times - error reading key 0595365
> Operation [15] retried 10 times - error reading key 0513247
> {code}
> *Additional notes*
> * There are no errors in the log files
> * After upgrading the full ring, and running READ from both the old and new versions of stress tool, the error still exists
> * Once all nodes are upgraded, if I run INSERT followed by READ, the new reads are fine
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3290) Upgrade from 0.7.9 to 1.0, the
java stress tool gives "error reading key" on data created in 0.7.9
Posted by "Cathy Daw (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118694#comment-13118694 ]
Cathy Daw commented on CASSANDRA-3290:
--------------------------------------
Brandon ... Before you jump on this, please let me look at the test case a little more in the morning.
What I am seeing is the following on a 3-node 0.8.6 without an upgrade when all three nodes are up:
{panel}
*OK:* ./stress --operation=INSERT --num-keys=20000 --columns=10 --consistency-level=QUORUM --average-size-values --replication-factor=2 --nodes=cathy2,cathy3
{code}
Created keyspaces. Sleeping 2s for propagation.
total,interval_op_rate,interval_key_rate,avg_latency,elapsed_time
20000,2000,2000,0.01193475,5
END
{code}
*ERROR:* ./stress --operation=READ --nodes=cathy2 --n 10
{code}
total,interval_op_rate,interval_key_rate,avg_latency,elapsed_time
Operation [0] retried 10 times - error reading key 03
0,0,0,NaN,0
END
{code}
{panel}
> Upgrade from 0.7.9 to 1.0, the java stress tool gives "error reading key" on data created in 0.7.9
> --------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-3290
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3290
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Reporter: Cathy Daw
> Assignee: Brandon Williams
> Fix For: 1.0.0
>
>
> *TEST CASE*
> * Install three node cluster with C* 0.7.9 with node1 as the seed
> * Run the 0.7.9 stress tool against node2,node3: ./stress -d node2,node3 -o insert
> * Run `nodetool -h node1 drain`
> * Upgrade binaries and edit cassandra.yaml for node1.
> * Startup C* 1.0 on node
> * Run the 0.7.9 stress tool against node2,node3: ./stress -d node2,node3 -o insert
> *ISSUE*
> * Run the 0.7.9 stress tool to read data: ./stress -d node2 -o read
> {code}
> cathy3:~/apache-cassandra-0.7.9-src/contrib/stress/bin# ./stress -d cathy3 -o read
> total,interval_op_rate,interval_key_rate,avg_latency,elapsed_time
> Operation [17] retried 10 times - error reading key 0521977
> Operation [9] retried 10 times - error reading key 0595365
> Operation [15] retried 10 times - error reading key 0513247
> {code}
> *Additional notes*
> * There are no errors in the log files
> * After upgrading the full ring, and running READ from both the old and new versions of stress tool, the error still exists
> * Once all nodes are upgraded, if I run INSERT followed by READ, the new reads are fine
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CASSANDRA-3290) Upgrade from 0.7.9 to 1.0, the
java stress tool gives "error reading key" on data created in 0.7.9
Posted by "Cathy Daw (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Cathy Daw resolved CASSANDRA-3290.
----------------------------------
Resolution: Not A Problem
stress tool retrieving more rows than inserted since I was not specifying number of keys
> Upgrade from 0.7.9 to 1.0, the java stress tool gives "error reading key" on data created in 0.7.9
> --------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-3290
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3290
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Reporter: Cathy Daw
> Assignee: Brandon Williams
> Fix For: 1.0.0
>
>
> *TEST CASE*
> * Install three node cluster with C* 0.7.9 with node1 as the seed
> * Run the 0.7.9 stress tool against node2,node3: ./stress -d node2,node3 -o insert
> * Run `nodetool -h node1 drain`
> * Upgrade binaries and edit cassandra.yaml for node1.
> * Startup C* 1.0 on node
> * Run the 0.7.9 stress tool against node2,node3: ./stress -d node2,node3 -o insert
> *ISSUE*
> * Run the 0.7.9 stress tool to read data: ./stress -d node2 -o read
> {code}
> cathy3:~/apache-cassandra-0.7.9-src/contrib/stress/bin# ./stress -d cathy3 -o read
> total,interval_op_rate,interval_key_rate,avg_latency,elapsed_time
> Operation [17] retried 10 times - error reading key 0521977
> Operation [9] retried 10 times - error reading key 0595365
> Operation [15] retried 10 times - error reading key 0513247
> {code}
> *Additional notes*
> * There are no errors in the log files
> * After upgrading the full ring, and running READ from both the old and new versions of stress tool, the error still exists
> * Once all nodes are upgraded, if I run INSERT followed by READ, the new reads are fine
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira