You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Richard Low (JIRA)" <ji...@apache.org> on 2011/07/25 18:40:09 UTC
[jira] [Created] (CASSANDRA-2946) HintedHandoff fails with could
not reach schema agreement
HintedHandoff fails with could not reach schema agreement
---------------------------------------------------------
Key: CASSANDRA-2946
URL: https://issues.apache.org/jira/browse/CASSANDRA-2946
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 0.8.2
Reporter: Richard Low
To reproduce, have two nodes A and B.
1. On node A, create a keyspace with replication factor 1 and add a column family
2. Ensure node B has created the keyspace and column family
3. Take down node B
4. Insert some keys to A at CL.ANY, ensuring some keys should be written to B
5. Bring up node B
6. When hints are delivered, I get the error:
ERROR [HintedHandoff:1] 2011-07-25 17:19:14,729 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[HintedHandoff:1,1,main]
java.lang.RuntimeException: java.lang.RuntimeException: Could not reach schema agreement with /10.2.129.9 in 60000ms
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
Caused by: java.lang.RuntimeException: Could not reach schema agreement with /10.2.129.9 in 60000ms
at org.apache.cassandra.db.HintedHandOffManager.waitForSchemaAgreement(HintedHandOffManager.java:290)
at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:301)
at org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager.java:89)
at org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:394)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
... 3 more
If I use DatabaseDescriptor.getDefsVersion() instead of gossiper.getEndpointStateForEndpoint(FBUtilities.getLocalAddress()).getApplicationState(ApplicationState.SCHEMA) then the error goes away, and the hints are correctly delivered.
This may be the same issue as Aaron saw here: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/ApplicationState-Schema-has-drifted-from-DatabaseDescriptor-td6006576.html, and may be related to CASSANDRA-2083.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2946) HintedHandoff fails with could
not reach schema agreement
Posted by "Richard Low (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13071024#comment-13071024 ]
Richard Low commented on CASSANDRA-2946:
----------------------------------------
describe cluster always shows agreement. Selected log messages:
DEBUG [HintedHandoff:1] 2011-07-26 11:22:35,526 HintedHandOffManager.java (line 300) Checking remote schema before delivering hints
...
DEBUG [pool-2-thread-1] 2011-07-26 11:22:44,965 CassandraServer.java (line 1123) checking schema agreement
...
DEBUG [pool-2-thread-1] 2011-07-26 11:22:44,969 StorageProxy.java (line 823) Schemas are in agreement.
...
ERROR [HintedHandoff:1] 2011-07-26 11:23:36,788 AbstractCassandraDaemon.java (line 138) Fatal exception in thread Thread[HintedHandoff:1,1,main]
java.lang.RuntimeException: java.lang.RuntimeException: Could not reach schema agreement with /10.2.129.9 in 60000ms
So StorageProxy thinks the schema agrees but HintedHandoffManager doesn't.
> HintedHandoff fails with could not reach schema agreement
> ---------------------------------------------------------
>
> Key: CASSANDRA-2946
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2946
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.2
> Reporter: Richard Low
>
> To reproduce, have two nodes A and B.
> 1. On node A, create a keyspace with replication factor 1 and add a column family
> 2. Ensure node B has created the keyspace and column family
> 3. Take down node B
> 4. Insert some keys to A at CL.ANY, ensuring some keys should be written to B
> 5. Bring up node B
> 6. When hints are delivered, I get the error:
> ERROR [HintedHandoff:1] 2011-07-25 17:19:14,729 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[HintedHandoff:1,1,main]
> java.lang.RuntimeException: java.lang.RuntimeException: Could not reach schema agreement with /10.2.129.9 in 60000ms
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
> Caused by: java.lang.RuntimeException: Could not reach schema agreement with /10.2.129.9 in 60000ms
> at org.apache.cassandra.db.HintedHandOffManager.waitForSchemaAgreement(HintedHandOffManager.java:290)
> at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:301)
> at org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager.java:89)
> at org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:394)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> ... 3 more
> If I use DatabaseDescriptor.getDefsVersion() instead of gossiper.getEndpointStateForEndpoint(FBUtilities.getLocalAddress()).getApplicationState(ApplicationState.SCHEMA) then the error goes away, and the hints are correctly delivered.
> This may be the same issue as Aaron saw here: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/ApplicationState-Schema-has-drifted-from-DatabaseDescriptor-td6006576.html, and may be related to CASSANDRA-2083.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2946) HintedHandoff fails with could
not reach schema agreement
Posted by "Richard Low (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13071609#comment-13071609 ]
Richard Low commented on CASSANDRA-2946:
----------------------------------------
Yes, that fix works. Thanks.
> HintedHandoff fails with could not reach schema agreement
> ---------------------------------------------------------
>
> Key: CASSANDRA-2946
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2946
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.1
> Reporter: Richard Low
> Assignee: Jonathan Ellis
> Fix For: 0.8.3
>
> Attachments: 2946.txt
>
>
> To reproduce, have two nodes A and B.
> 1. On node A, create a keyspace with replication factor 1 and add a column family
> 2. Ensure node B has created the keyspace and column family
> 3. Take down node B
> 4. Insert some keys to A at CL.ANY, ensuring some keys should be written to B
> 5. Bring up node B
> 6. When hints are delivered, I get the error:
> ERROR [HintedHandoff:1] 2011-07-25 17:19:14,729 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[HintedHandoff:1,1,main]
> java.lang.RuntimeException: java.lang.RuntimeException: Could not reach schema agreement with /10.2.129.9 in 60000ms
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
> Caused by: java.lang.RuntimeException: Could not reach schema agreement with /10.2.129.9 in 60000ms
> at org.apache.cassandra.db.HintedHandOffManager.waitForSchemaAgreement(HintedHandOffManager.java:290)
> at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:301)
> at org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager.java:89)
> at org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:394)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> ... 3 more
> If I use DatabaseDescriptor.getDefsVersion() instead of gossiper.getEndpointStateForEndpoint(FBUtilities.getLocalAddress()).getApplicationState(ApplicationState.SCHEMA) then the error goes away, and the hints are correctly delivered.
> This may be the same issue as Aaron saw here: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/ApplicationState-Schema-has-drifted-from-DatabaseDescriptor-td6006576.html, and may be related to CASSANDRA-2083.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2946) HintedHandoff fails with could
not reach schema agreement
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13071298#comment-13071298 ]
Brandon Williams commented on CASSANDRA-2946:
---------------------------------------------
+1, what happened was node A never updated it's gossip state, but the other node learned of its schema version by RPC. When node B restarted, it no longer knew the schema version for A, and tried to get it from gossip where it was still old.
bq. if it does we can also change the HH code to use DD.
I don't know about that, having this as a check to make sure this doesn't happen again is a far easier thing to catch than other "gossip is out of date" problems we might encounter.
> HintedHandoff fails with could not reach schema agreement
> ---------------------------------------------------------
>
> Key: CASSANDRA-2946
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2946
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.1
> Reporter: Richard Low
> Assignee: Jonathan Ellis
> Fix For: 0.8.3
>
> Attachments: 2946.txt
>
>
> To reproduce, have two nodes A and B.
> 1. On node A, create a keyspace with replication factor 1 and add a column family
> 2. Ensure node B has created the keyspace and column family
> 3. Take down node B
> 4. Insert some keys to A at CL.ANY, ensuring some keys should be written to B
> 5. Bring up node B
> 6. When hints are delivered, I get the error:
> ERROR [HintedHandoff:1] 2011-07-25 17:19:14,729 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[HintedHandoff:1,1,main]
> java.lang.RuntimeException: java.lang.RuntimeException: Could not reach schema agreement with /10.2.129.9 in 60000ms
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
> Caused by: java.lang.RuntimeException: Could not reach schema agreement with /10.2.129.9 in 60000ms
> at org.apache.cassandra.db.HintedHandOffManager.waitForSchemaAgreement(HintedHandOffManager.java:290)
> at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:301)
> at org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager.java:89)
> at org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:394)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> ... 3 more
> If I use DatabaseDescriptor.getDefsVersion() instead of gossiper.getEndpointStateForEndpoint(FBUtilities.getLocalAddress()).getApplicationState(ApplicationState.SCHEMA) then the error goes away, and the hints are correctly delivered.
> This may be the same issue as Aaron saw here: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/ApplicationState-Schema-has-drifted-from-DatabaseDescriptor-td6006576.html, and may be related to CASSANDRA-2083.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2946) HintedHandoff fails with could
not reach schema agreement
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13070921#comment-13070921 ]
Jonathan Ellis commented on CASSANDRA-2946:
-------------------------------------------
does the disagreement show up with "describe cluster" from the cli? if so, at what point do the schemas of A and B diverge?
> HintedHandoff fails with could not reach schema agreement
> ---------------------------------------------------------
>
> Key: CASSANDRA-2946
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2946
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.2
> Reporter: Richard Low
>
> To reproduce, have two nodes A and B.
> 1. On node A, create a keyspace with replication factor 1 and add a column family
> 2. Ensure node B has created the keyspace and column family
> 3. Take down node B
> 4. Insert some keys to A at CL.ANY, ensuring some keys should be written to B
> 5. Bring up node B
> 6. When hints are delivered, I get the error:
> ERROR [HintedHandoff:1] 2011-07-25 17:19:14,729 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[HintedHandoff:1,1,main]
> java.lang.RuntimeException: java.lang.RuntimeException: Could not reach schema agreement with /10.2.129.9 in 60000ms
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
> Caused by: java.lang.RuntimeException: Could not reach schema agreement with /10.2.129.9 in 60000ms
> at org.apache.cassandra.db.HintedHandOffManager.waitForSchemaAgreement(HintedHandOffManager.java:290)
> at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:301)
> at org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager.java:89)
> at org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:394)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> ... 3 more
> If I use DatabaseDescriptor.getDefsVersion() instead of gossiper.getEndpointStateForEndpoint(FBUtilities.getLocalAddress()).getApplicationState(ApplicationState.SCHEMA) then the error goes away, and the hints are correctly delivered.
> This may be the same issue as Aaron saw here: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/ApplicationState-Schema-has-drifted-from-DatabaseDescriptor-td6006576.html, and may be related to CASSANDRA-2083.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2946) HintedHandoff fails with could
not reach schema agreement
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13071382#comment-13071382 ]
Jonathan Ellis commented on CASSANDRA-2946:
-------------------------------------------
okay, then I'll add a comment to that effect, assuming Richard confirms this fixes the bug.
> HintedHandoff fails with could not reach schema agreement
> ---------------------------------------------------------
>
> Key: CASSANDRA-2946
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2946
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.1
> Reporter: Richard Low
> Assignee: Jonathan Ellis
> Fix For: 0.8.3
>
> Attachments: 2946.txt
>
>
> To reproduce, have two nodes A and B.
> 1. On node A, create a keyspace with replication factor 1 and add a column family
> 2. Ensure node B has created the keyspace and column family
> 3. Take down node B
> 4. Insert some keys to A at CL.ANY, ensuring some keys should be written to B
> 5. Bring up node B
> 6. When hints are delivered, I get the error:
> ERROR [HintedHandoff:1] 2011-07-25 17:19:14,729 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[HintedHandoff:1,1,main]
> java.lang.RuntimeException: java.lang.RuntimeException: Could not reach schema agreement with /10.2.129.9 in 60000ms
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
> Caused by: java.lang.RuntimeException: Could not reach schema agreement with /10.2.129.9 in 60000ms
> at org.apache.cassandra.db.HintedHandOffManager.waitForSchemaAgreement(HintedHandOffManager.java:290)
> at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:301)
> at org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager.java:89)
> at org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:394)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> ... 3 more
> If I use DatabaseDescriptor.getDefsVersion() instead of gossiper.getEndpointStateForEndpoint(FBUtilities.getLocalAddress()).getApplicationState(ApplicationState.SCHEMA) then the error goes away, and the hints are correctly delivered.
> This may be the same issue as Aaron saw here: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/ApplicationState-Schema-has-drifted-from-DatabaseDescriptor-td6006576.html, and may be related to CASSANDRA-2083.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2946) HintedHandoff fails with could
not reach schema agreement
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-2946:
--------------------------------------
Attachment: 2946.txt
this should fix the gossip-out-of-sync bug.
if it does we can also change the HH code to use DD.
> HintedHandoff fails with could not reach schema agreement
> ---------------------------------------------------------
>
> Key: CASSANDRA-2946
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2946
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.2
> Reporter: Richard Low
> Attachments: 2946.txt
>
>
> To reproduce, have two nodes A and B.
> 1. On node A, create a keyspace with replication factor 1 and add a column family
> 2. Ensure node B has created the keyspace and column family
> 3. Take down node B
> 4. Insert some keys to A at CL.ANY, ensuring some keys should be written to B
> 5. Bring up node B
> 6. When hints are delivered, I get the error:
> ERROR [HintedHandoff:1] 2011-07-25 17:19:14,729 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[HintedHandoff:1,1,main]
> java.lang.RuntimeException: java.lang.RuntimeException: Could not reach schema agreement with /10.2.129.9 in 60000ms
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
> Caused by: java.lang.RuntimeException: Could not reach schema agreement with /10.2.129.9 in 60000ms
> at org.apache.cassandra.db.HintedHandOffManager.waitForSchemaAgreement(HintedHandOffManager.java:290)
> at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:301)
> at org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager.java:89)
> at org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:394)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> ... 3 more
> If I use DatabaseDescriptor.getDefsVersion() instead of gossiper.getEndpointStateForEndpoint(FBUtilities.getLocalAddress()).getApplicationState(ApplicationState.SCHEMA) then the error goes away, and the hints are correctly delivered.
> This may be the same issue as Aaron saw here: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/ApplicationState-Schema-has-drifted-from-DatabaseDescriptor-td6006576.html, and may be related to CASSANDRA-2083.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2946) HintedHandoff fails with could
not reach schema agreement
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13071829#comment-13071829 ]
Hudson commented on CASSANDRA-2946:
-----------------------------------
Integrated in Cassandra-0.8 #240 (See [https://builds.apache.org/job/Cassandra-0.8/240/])
keep gossipped version in sync with actual on migration coordinator
patch by jbellis; reviewed by brandonwilliams and tested by Richard Low for CASSANDRA-2946
jbellis : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1151494
Files :
* /cassandra/branches/cassandra-0.8/CHANGES.txt
* /cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/db/migration/Migration.java
* /cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/db/HintedHandOffManager.java
> HintedHandoff fails with could not reach schema agreement
> ---------------------------------------------------------
>
> Key: CASSANDRA-2946
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2946
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.1
> Reporter: Richard Low
> Assignee: Jonathan Ellis
> Fix For: 0.8.3
>
> Attachments: 2946.txt
>
>
> To reproduce, have two nodes A and B.
> 1. On node A, create a keyspace with replication factor 1 and add a column family
> 2. Ensure node B has created the keyspace and column family
> 3. Take down node B
> 4. Insert some keys to A at CL.ANY, ensuring some keys should be written to B
> 5. Bring up node B
> 6. When hints are delivered, I get the error:
> ERROR [HintedHandoff:1] 2011-07-25 17:19:14,729 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[HintedHandoff:1,1,main]
> java.lang.RuntimeException: java.lang.RuntimeException: Could not reach schema agreement with /10.2.129.9 in 60000ms
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
> Caused by: java.lang.RuntimeException: Could not reach schema agreement with /10.2.129.9 in 60000ms
> at org.apache.cassandra.db.HintedHandOffManager.waitForSchemaAgreement(HintedHandOffManager.java:290)
> at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:301)
> at org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager.java:89)
> at org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:394)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> ... 3 more
> If I use DatabaseDescriptor.getDefsVersion() instead of gossiper.getEndpointStateForEndpoint(FBUtilities.getLocalAddress()).getApplicationState(ApplicationState.SCHEMA) then the error goes away, and the hints are correctly delivered.
> This may be the same issue as Aaron saw here: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/ApplicationState-Schema-has-drifted-from-DatabaseDescriptor-td6006576.html, and may be related to CASSANDRA-2083.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira