You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Gabriel Rosendorf (Created) (JIRA)" <ji...@apache.org> on 2011/11/22 16:56:39 UTC
[jira] [Created] (CASSANDRA-3522) Fatal exception in thread Thread
Fatal exception in thread Thread
---------------------------------
Key: CASSANDRA-3522
URL: https://issues.apache.org/jira/browse/CASSANDRA-3522
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.0.0
Environment: Ubuntu 10.04 LTS - 8GB mem/ 360 GB disk - Hosted by RS
Reporter: Gabriel Rosendorf
Priority: Minor
Seeing this recurring exception on all machines in a 5 node cluster. Recently upgraded to 1.0.0 from 0.7.2.
ERROR [MutationStage:20] 2011-11-22 15:25:55,758 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[MutationStage:20,5,main]
java.lang.AssertionError
at org.apache.cassandra.locator.TokenMetadata.getToken(TokenMetadata.java:273)
at org.apache.cassandra.service.StorageProxy$4.run(StorageProxy.java:350)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3522) Fatal exception in thread Thread
Posted by "Jonathan Ellis (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-3522:
--------------------------------------
Assignee: Brandon Williams
The HH assertion means you had some hint corruption from 1.0.0 (CASSANDRA-3466). You should remove your Hint column families from the system/ keyspace.
> Fatal exception in thread Thread
> ---------------------------------
>
> Key: CASSANDRA-3522
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3522
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Environment: Ubuntu 10.04 LTS - 8GB mem/ 360 GB disk - Hosted by RS
> Reporter: Gabriel Rosendorf
> Assignee: Brandon Williams
> Priority: Minor
>
> Seeing this recurring exception on all machines in a 5 node cluster. Recently upgraded to 1.0.0 from 0.7.2.
> ERROR [MutationStage:20] 2011-11-22 15:25:55,758 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[MutationStage:20,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.locator.TokenMetadata.getToken(TokenMetadata.java:273)
> at org.apache.cassandra.service.StorageProxy$4.run(StorageProxy.java:350)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3522) Fatal exception in thread
Thread
Posted by "Gabriel Rosendorf (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13155451#comment-13155451 ]
Gabriel Rosendorf commented on CASSANDRA-3522:
----------------------------------------------
Also, since the upgrade from 1.0.0 to 1.0.3 I'm getting Hinted Handoff Exceptions:
INFO [HintedHandoff:4] 2011-11-22 21:28:48,046 HintedHandOffManager.java (line 268) Started hinted handoff for token: 113427455640312814857969558651062452224 with IP: /10.179.64.134
ERROR [HintedHandoff:4] 2011-11-22 21:28:48,077 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[HintedHandoff:4,1,main]
java.lang.AssertionError
at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:301)
at org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager.java:81)
at org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:353)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
> Fatal exception in thread Thread
> ---------------------------------
>
> Key: CASSANDRA-3522
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3522
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Environment: Ubuntu 10.04 LTS - 8GB mem/ 360 GB disk - Hosted by RS
> Reporter: Gabriel Rosendorf
> Priority: Minor
>
> Seeing this recurring exception on all machines in a 5 node cluster. Recently upgraded to 1.0.0 from 0.7.2.
> ERROR [MutationStage:20] 2011-11-22 15:25:55,758 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[MutationStage:20,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.locator.TokenMetadata.getToken(TokenMetadata.java:273)
> at org.apache.cassandra.service.StorageProxy$4.run(StorageProxy.java:350)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3522) Fatal exception in thread
Thread
Posted by "Gabriel Rosendorf (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13155440#comment-13155440 ]
Gabriel Rosendorf commented on CASSANDRA-3522:
----------------------------------------------
Upgraded to 1.0.3, still getting the same exception on all machines. Disk usage is also going up quickly on all the servers.
> Fatal exception in thread Thread
> ---------------------------------
>
> Key: CASSANDRA-3522
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3522
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Environment: Ubuntu 10.04 LTS - 8GB mem/ 360 GB disk - Hosted by RS
> Reporter: Gabriel Rosendorf
> Priority: Minor
>
> Seeing this recurring exception on all machines in a 5 node cluster. Recently upgraded to 1.0.0 from 0.7.2.
> ERROR [MutationStage:20] 2011-11-22 15:25:55,758 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[MutationStage:20,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.locator.TokenMetadata.getToken(TokenMetadata.java:273)
> at org.apache.cassandra.service.StorageProxy$4.run(StorageProxy.java:350)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3522) Fatal exception in thread
Thread
Posted by "Gabriel Rosendorf (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13155450#comment-13155450 ]
Gabriel Rosendorf commented on CASSANDRA-3522:
----------------------------------------------
Looks like this on one node:
Address DC Rack Status State Load Owns Token
141784319550391032739561396922763706368
10.183.65.16 datacenter1 rack1 Up Normal 129.55 GB 16.67% 0
10.183.47.195 datacenter1 rack1 Up Normal 181.65 GB 33.33% 56713727820156407428984779325531226112
10.182.96.44 datacenter1 rack1 Up Normal 165.17 GB 16.67% 85070591730234615865843651857942052864
10.179.64.134 datacenter1 rack1 Up Normal 128.11 GB 16.67% 113427455640312814857969558651062452224
10.183.65.18 datacenter1 rack1 Up Normal 140.6 GB 16.67% 141784319550391032739561396922763706368
But on another node, it all of a sudden thinks that a node that I decommissioned yesterday is still part of the ring, just down:
Address DC Rack Status State Load Owns Token
141784319550391032739561396922763706368
10.183.65.16 datacenter1 rack1 Up Normal 129.55 GB 16.67% 0
10.183.47.195 datacenter1 rack1 Up Normal 181.65 GB 33.33% 56713727820156407428984779325531226112
10.182.96.44 datacenter1 rack1 Up Normal 165.17 GB 16.67% 85070591730234615865843651857942052864
10.179.64.134 datacenter1 rack1 Up Normal 128.11 GB 16.67% 113427455640312814857969558651062452224
10.179.77.102 datacenter1 rack1 Down Normal ? 7.61% 126376470034382387444230631795475767049
10.183.65.18 datacenter1 rack1 Up Normal 140.6 GB 9.06% 141784319550391032739561396922763706368
> Fatal exception in thread Thread
> ---------------------------------
>
> Key: CASSANDRA-3522
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3522
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Environment: Ubuntu 10.04 LTS - 8GB mem/ 360 GB disk - Hosted by RS
> Reporter: Gabriel Rosendorf
> Priority: Minor
>
> Seeing this recurring exception on all machines in a 5 node cluster. Recently upgraded to 1.0.0 from 0.7.2.
> ERROR [MutationStage:20] 2011-11-22 15:25:55,758 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[MutationStage:20,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.locator.TokenMetadata.getToken(TokenMetadata.java:273)
> at org.apache.cassandra.service.StorageProxy$4.run(StorageProxy.java:350)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CASSANDRA-3522) Fatal exception in thread Thread
Posted by "Gabriel Rosendorf (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gabriel Rosendorf resolved CASSANDRA-3522.
------------------------------------------
Resolution: Fixed
Fix Version/s: 1.0.3
> Fatal exception in thread Thread
> ---------------------------------
>
> Key: CASSANDRA-3522
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3522
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Environment: Ubuntu 10.04 LTS - 8GB mem/ 360 GB disk - Hosted by RS
> Reporter: Gabriel Rosendorf
> Assignee: Brandon Williams
> Priority: Minor
> Fix For: 1.0.3
>
>
> Seeing this recurring exception on all machines in a 5 node cluster. Recently upgraded to 1.0.0 from 0.7.2.
> ERROR [MutationStage:20] 2011-11-22 15:25:55,758 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[MutationStage:20,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.locator.TokenMetadata.getToken(TokenMetadata.java:273)
> at org.apache.cassandra.service.StorageProxy$4.run(StorageProxy.java:350)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3522) Fatal exception in thread
Thread
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13155446#comment-13155446 ]
Jonathan Ellis commented on CASSANDRA-3522:
-------------------------------------------
What does nodetool ring look like?
> Fatal exception in thread Thread
> ---------------------------------
>
> Key: CASSANDRA-3522
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3522
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Environment: Ubuntu 10.04 LTS - 8GB mem/ 360 GB disk - Hosted by RS
> Reporter: Gabriel Rosendorf
> Priority: Minor
>
> Seeing this recurring exception on all machines in a 5 node cluster. Recently upgraded to 1.0.0 from 0.7.2.
> ERROR [MutationStage:20] 2011-11-22 15:25:55,758 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[MutationStage:20,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.locator.TokenMetadata.getToken(TokenMetadata.java:273)
> at org.apache.cassandra.service.StorageProxy$4.run(StorageProxy.java:350)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3522) Fatal exception in thread
Thread
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13155221#comment-13155221 ]
Jonathan Ellis commented on CASSANDRA-3522:
-------------------------------------------
please upgrade to 1.0.3 to verify that this isn't an already-fixed bug like CASSANDRA-2961
> Fatal exception in thread Thread
> ---------------------------------
>
> Key: CASSANDRA-3522
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3522
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Environment: Ubuntu 10.04 LTS - 8GB mem/ 360 GB disk - Hosted by RS
> Reporter: Gabriel Rosendorf
> Priority: Minor
>
> Seeing this recurring exception on all machines in a 5 node cluster. Recently upgraded to 1.0.0 from 0.7.2.
> ERROR [MutationStage:20] 2011-11-22 15:25:55,758 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[MutationStage:20,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.locator.TokenMetadata.getToken(TokenMetadata.java:273)
> at org.apache.cassandra.service.StorageProxy$4.run(StorageProxy.java:350)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3522) Fatal exception in thread
Thread
Posted by "Gabriel Rosendorf (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13155465#comment-13155465 ]
Gabriel Rosendorf commented on CASSANDRA-3522:
----------------------------------------------
Ok, removing the Hint CFs took care of the Hinted Handoff exception. I ran nodetool removetoken on that errant token from the machine that I decommissioned yesterday, so nodetool ring is the same on all machines. I did a full cluster restart, and so far I'm not seeing any exceptions.
> Fatal exception in thread Thread
> ---------------------------------
>
> Key: CASSANDRA-3522
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3522
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Environment: Ubuntu 10.04 LTS - 8GB mem/ 360 GB disk - Hosted by RS
> Reporter: Gabriel Rosendorf
> Assignee: Brandon Williams
> Priority: Minor
>
> Seeing this recurring exception on all machines in a 5 node cluster. Recently upgraded to 1.0.0 from 0.7.2.
> ERROR [MutationStage:20] 2011-11-22 15:25:55,758 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[MutationStage:20,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.locator.TokenMetadata.getToken(TokenMetadata.java:273)
> at org.apache.cassandra.service.StorageProxy$4.run(StorageProxy.java:350)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira