You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Cathy Daw (JIRA)" <ji...@apache.org> on 2011/09/13 03:09:08 UTC
[jira] [Created] (CASSANDRA-3192) NPE in RowRepairResolver
NPE in RowRepairResolver
------------------------
Key: CASSANDRA-3192
URL: https://issues.apache.org/jira/browse/CASSANDRA-3192
Project: Cassandra
Issue Type: Bug
Reporter: Cathy Daw
On a 3 node brisk cluster (running against C* 1.0 branch), I was running the java stress tool and the terasort concurrently in two sessions. Eventually both jobs failed with TimedOutException.
>From this point forward most additional activity will fail with a TimedOutException.
* Java Stress Tool - 5 rows / 10 columns - Operation [0] retried 10 times - error inserting key 0 ((TimedOutException))
* Hive - show tables: FAILED: Error in metadata: com.datastax.bdp.hadoop.hive.metastore.CassandraHiveMetaStoreException: There was a problem with the Cassandra Hive MetaStore: Could not connect to Cassandra. Reason: Error connecting to node localhost
However, the Cassandra CLI appears to be happy
* Cassandra CLI: you can successfully insert and read using consistencylevel as ONE or ALL
The seed node has the following error repeatedly occurring in the logs. The other two nodes have no errors.
{code}
ERROR [ReadRepairStage:15] 2011-09-13 00:44:25,971 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[ReadRepairStage:15,5,main]
java.lang.RuntimeException: java.lang.NullPointerException
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.NullPointerException
at org.apache.cassandra.service.RowRepairResolver.resolve(RowRepairResolver.java:82)
at org.apache.cassandra.service.AsyncRepairCallback$1.runMayThrow(AsyncRepairCallback.java:54)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
... 3 more
{code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3192) NPE in RowRepairResolver
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-3192:
--------------------------------------
Attachment: 3192.txt
patch to fix regression caused by CASSANDRA-2643
> NPE in RowRepairResolver
> ------------------------
>
> Key: CASSANDRA-3192
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3192
> Project: Cassandra
> Issue Type: Bug
> Reporter: Cathy Daw
> Attachments: 3192.txt
>
>
> On a 3 node brisk cluster (running against C* 1.0 branch), I was running the java stress tool and the terasort concurrently in two sessions. Eventually both jobs failed with TimedOutException.
>
> From this point forward most additional activity will fail with a TimedOutException.
> * Java Stress Tool - 5 rows / 10 columns - Operation [0] retried 10 times - error inserting key 0 ((TimedOutException))
> * Hive - show tables: FAILED: Error in metadata: com.datastax.bdp.hadoop.hive.metastore.CassandraHiveMetaStoreException: There was a problem with the Cassandra Hive MetaStore: Could not connect to Cassandra. Reason: Error connecting to node localhost
> However, the Cassandra CLI appears to be happy
> * Cassandra CLI: you can successfully insert and read using consistencylevel as ONE or ALL
> The seed node has the following error repeatedly occurring in the logs. The other two nodes have no errors.
> {code}
> ERROR [ReadRepairStage:15] 2011-09-13 00:44:25,971 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[ReadRepairStage:15,5,main]
> java.lang.RuntimeException: java.lang.NullPointerException
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.lang.NullPointerException
> at org.apache.cassandra.service.RowRepairResolver.resolve(RowRepairResolver.java:82)
> at org.apache.cassandra.service.AsyncRepairCallback$1.runMayThrow(AsyncRepairCallback.java:54)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> ... 3 more
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3192) NPE in RowRepairResolver
Posted by "Cathy Daw (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13103267#comment-13103267 ]
Cathy Daw commented on CASSANDRA-3192:
--------------------------------------
I noticed that after starting up the server, this error message was in the log file of the second node (non-seed) after startup.
*node2*
{code}
ERROR 01:30:05,522 Fatal exception in thread Thread[HintedHandoff:1,5,main]
java.lang.AssertionError
at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:282)
at org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager.java:81)
at org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:333)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{code}
> NPE in RowRepairResolver
> ------------------------
>
> Key: CASSANDRA-3192
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3192
> Project: Cassandra
> Issue Type: Bug
> Reporter: Cathy Daw
>
> On a 3 node brisk cluster (running against C* 1.0 branch), I was running the java stress tool and the terasort concurrently in two sessions. Eventually both jobs failed with TimedOutException.
>
> From this point forward most additional activity will fail with a TimedOutException.
> * Java Stress Tool - 5 rows / 10 columns - Operation [0] retried 10 times - error inserting key 0 ((TimedOutException))
> * Hive - show tables: FAILED: Error in metadata: com.datastax.bdp.hadoop.hive.metastore.CassandraHiveMetaStoreException: There was a problem with the Cassandra Hive MetaStore: Could not connect to Cassandra. Reason: Error connecting to node localhost
> However, the Cassandra CLI appears to be happy
> * Cassandra CLI: you can successfully insert and read using consistencylevel as ONE or ALL
> The seed node has the following error repeatedly occurring in the logs. The other two nodes have no errors.
> {code}
> ERROR [ReadRepairStage:15] 2011-09-13 00:44:25,971 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[ReadRepairStage:15,5,main]
> java.lang.RuntimeException: java.lang.NullPointerException
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.lang.NullPointerException
> at org.apache.cassandra.service.RowRepairResolver.resolve(RowRepairResolver.java:82)
> at org.apache.cassandra.service.AsyncRepairCallback$1.runMayThrow(AsyncRepairCallback.java:54)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> ... 3 more
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CASSANDRA-3192) NPE in RowRepairResolver
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Williams resolved CASSANDRA-3192.
-----------------------------------------
Resolution: Duplicate
Solved in CASSANDRA-3156
> NPE in RowRepairResolver
> ------------------------
>
> Key: CASSANDRA-3192
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3192
> Project: Cassandra
> Issue Type: Bug
> Reporter: Cathy Daw
>
> On a 3 node brisk cluster (running against C* 1.0 branch), I was running the java stress tool and the terasort concurrently in two sessions. Eventually both jobs failed with TimedOutException.
>
> From this point forward most additional activity will fail with a TimedOutException.
> * Java Stress Tool - 5 rows / 10 columns - Operation [0] retried 10 times - error inserting key 0 ((TimedOutException))
> * Hive - show tables: FAILED: Error in metadata: com.datastax.bdp.hadoop.hive.metastore.CassandraHiveMetaStoreException: There was a problem with the Cassandra Hive MetaStore: Could not connect to Cassandra. Reason: Error connecting to node localhost
> However, the Cassandra CLI appears to be happy
> * Cassandra CLI: you can successfully insert and read using consistencylevel as ONE or ALL
> The seed node has the following error repeatedly occurring in the logs. The other two nodes have no errors.
> {code}
> ERROR [ReadRepairStage:15] 2011-09-13 00:44:25,971 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[ReadRepairStage:15,5,main]
> java.lang.RuntimeException: java.lang.NullPointerException
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.lang.NullPointerException
> at org.apache.cassandra.service.RowRepairResolver.resolve(RowRepairResolver.java:82)
> at org.apache.cassandra.service.AsyncRepairCallback$1.runMayThrow(AsyncRepairCallback.java:54)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> ... 3 more
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3192) NPE in RowRepairResolver
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13103575#comment-13103575 ]
Sylvain Lebresne commented on CASSANDRA-3192:
---------------------------------------------
+1
> NPE in RowRepairResolver
> ------------------------
>
> Key: CASSANDRA-3192
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3192
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.0
> Reporter: Cathy Daw
> Assignee: Jonathan Ellis
> Fix For: 1.0.0
>
> Attachments: 3192.txt
>
>
> On a 3 node brisk cluster (running against C* 1.0 branch), I was running the java stress tool and the terasort concurrently in two sessions. Eventually both jobs failed with TimedOutException.
>
> From this point forward most additional activity will fail with a TimedOutException.
> * Java Stress Tool - 5 rows / 10 columns - Operation [0] retried 10 times - error inserting key 0 ((TimedOutException))
> * Hive - show tables: FAILED: Error in metadata: com.datastax.bdp.hadoop.hive.metastore.CassandraHiveMetaStoreException: There was a problem with the Cassandra Hive MetaStore: Could not connect to Cassandra. Reason: Error connecting to node localhost
> However, the Cassandra CLI appears to be happy
> * Cassandra CLI: you can successfully insert and read using consistencylevel as ONE or ALL
> The seed node has the following error repeatedly occurring in the logs. The other two nodes have no errors.
> {code}
> ERROR [ReadRepairStage:15] 2011-09-13 00:44:25,971 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[ReadRepairStage:15,5,main]
> java.lang.RuntimeException: java.lang.NullPointerException
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.lang.NullPointerException
> at org.apache.cassandra.service.RowRepairResolver.resolve(RowRepairResolver.java:82)
> at org.apache.cassandra.service.AsyncRepairCallback$1.runMayThrow(AsyncRepairCallback.java:54)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> ... 3 more
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (CASSANDRA-3192) NPE in RowRepairResolver
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis reopened CASSANDRA-3192:
---------------------------------------
Assignee: Jonathan Ellis
> NPE in RowRepairResolver
> ------------------------
>
> Key: CASSANDRA-3192
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3192
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.0
> Reporter: Cathy Daw
> Assignee: Jonathan Ellis
> Fix For: 1.0.0
>
> Attachments: 3192.txt
>
>
> On a 3 node brisk cluster (running against C* 1.0 branch), I was running the java stress tool and the terasort concurrently in two sessions. Eventually both jobs failed with TimedOutException.
>
> From this point forward most additional activity will fail with a TimedOutException.
> * Java Stress Tool - 5 rows / 10 columns - Operation [0] retried 10 times - error inserting key 0 ((TimedOutException))
> * Hive - show tables: FAILED: Error in metadata: com.datastax.bdp.hadoop.hive.metastore.CassandraHiveMetaStoreException: There was a problem with the Cassandra Hive MetaStore: Could not connect to Cassandra. Reason: Error connecting to node localhost
> However, the Cassandra CLI appears to be happy
> * Cassandra CLI: you can successfully insert and read using consistencylevel as ONE or ALL
> The seed node has the following error repeatedly occurring in the logs. The other two nodes have no errors.
> {code}
> ERROR [ReadRepairStage:15] 2011-09-13 00:44:25,971 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[ReadRepairStage:15,5,main]
> java.lang.RuntimeException: java.lang.NullPointerException
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.lang.NullPointerException
> at org.apache.cassandra.service.RowRepairResolver.resolve(RowRepairResolver.java:82)
> at org.apache.cassandra.service.AsyncRepairCallback$1.runMayThrow(AsyncRepairCallback.java:54)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> ... 3 more
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira