You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Sanjay (JIRA)" <ji...@apache.org> on 2013/06/20 09:26:20 UTC
[jira] [Updated] (CASSANDRA-5673) NullPointerException on running
instances
[ https://issues.apache.org/jira/browse/CASSANDRA-5673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sanjay updated CASSANDRA-5673:
------------------------------
Description:
Hallo,
We are having sporadic NullPointerException in some of the cassandra nodes in cluster (See stacktrace).
We are having two Datacenter, each having 15 nodes with RF = 2, OS is SLES with java-1_6_0-ibm-1.6.0_sr12.0-0.5.1.
At present only workaround is to stop the application running on same node and run repair tool on cassandra. We are unable to identify the cause of error.
1)
INFO|ScheduledTasks:1|org.apache.cassandra.service.GCInspector|GC for MarkSweepCompact: 347 ms for 1 collections, 138398568 used; ma
x is 1051721728
2013-06-19T16:25:50:843|ERROR|ReplicateOnWriteStage:115|org.apache.cassandra.service.CassandraDaemon|Exception in thread Thread[ReplicateOnWriteStage:115,5,m
ain]
java.lang.RuntimeException: java.lang.NullPointerException
at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1582)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:897)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:919)
at java.lang.Thread.run(Thread.java:738)
Caused by: java.lang.NullPointerException
at java.util.TreeSet.iterator(TreeSet.java:230)
at org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:163)
at org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNamesIterator.java:64)
at org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:81)
at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:68)
at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:274)
at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65)
at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1357)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1214)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1126)
at org.apache.cassandra.db.Table.getRow(Table.java:347)
at org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:64)
at org.apache.cassandra.db.CounterMutation.makeReplicationMutation(CounterMutation.java:90)
at org.apache.cassandra.service.StorageProxy$7$1.runMayThrow(StorageProxy.java:796)
at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1578)
... 3 more
2013-06-19T16:26:01:001|ERROR|ReadStage:4833|org.apache.cassandra.service.CassandraDaemon|Exception in thread Thread[ReadStage:4833,5,main]
java.lang.RuntimeException: java.lang.NullPointerException
at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1582)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:897)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:919)
at java.lang.Thread.run(Thread.java:738)
Caused by: java.lang.NullPointerException
at java.util.TreeSet.iterator(TreeSet.java:230)
at org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:163)
at org.
2)
2013-06-19T08:38:23:436| INFO|Thread-2447|org.apache.cassandra.service.StorageService|Starting repair command #2, repairing 1 ranges for keyspace system_auth
2013-06-19T08:58:25:685|ERROR|ReadStage:9270|org.apache.cassandra.service.CassandraDaemon|Exception in thread Thread[ReadStage:9270,5,main]
java.lang.NullPointerException
at java.util.TreeSet.iterator(TreeSet.java:230)
at org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:163)
at org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNamesIterator.java:64)
at org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:81)
at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:68)
at org.apache.cassandra.db.CollationController.collectTimeOrderedData(CollationController.java:133)
at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65)
at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1357)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1214)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1126)
at org.apache.cassandra.db.Table.getRow(Table.java:347)
at org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:64)
at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:44)
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:56)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:908)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:931)
at java.lang.Thread.run(Thread.java:738)
Best Regards
Sanjay
was:
Hallo,
We are having sporadic NullPointerException in some of the cassandra nodes in cluster (See stacktrace).
We are having two Datacenter, each having 15 nodes with RF = 2, OS is SLES with java-1_6_0-ibm-1.6.0_sr12.0-0.5.1.
At present only workaround is to stop the application running on same node and run repair tool on cassandra. We are unable to identify the cause of error.
INFO|ScheduledTasks:1|org.apache.cassandra.service.GCInspector|GC for MarkSweepCompact: 347 ms for 1 collections, 138398568 used; ma
x is 1051721728
2013-06-19T16:25:50:843|ERROR|ReplicateOnWriteStage:115|org.apache.cassandra.service.CassandraDaemon|Exception in thread Thread[ReplicateOnWriteStage:115,5,m
ain]
java.lang.RuntimeException: java.lang.NullPointerException
at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1582)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:897)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:919)
at java.lang.Thread.run(Thread.java:738)
Caused by: java.lang.NullPointerException
at java.util.TreeSet.iterator(TreeSet.java:230)
at org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:163)
at org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNamesIterator.java:64)
at org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:81)
at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:68)
at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:274)
at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65)
at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1357)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1214)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1126)
at org.apache.cassandra.db.Table.getRow(Table.java:347)
at org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:64)
at org.apache.cassandra.db.CounterMutation.makeReplicationMutation(CounterMutation.java:90)
at org.apache.cassandra.service.StorageProxy$7$1.runMayThrow(StorageProxy.java:796)
at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1578)
... 3 more
2013-06-19T16:26:01:001|ERROR|ReadStage:4833|org.apache.cassandra.service.CassandraDaemon|Exception in thread Thread[ReadStage:4833,5,main]
java.lang.RuntimeException: java.lang.NullPointerException
at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1582)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:897)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:919)
at java.lang.Thread.run(Thread.java:738)
Caused by: java.lang.NullPointerException
at java.util.TreeSet.iterator(TreeSet.java:230)
at org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:163)
at org.
Best Regards
Sanjay
> NullPointerException on running instances
> -----------------------------------------
>
> Key: CASSANDRA-5673
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5673
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.2.4
> Reporter: Sanjay
>
> Hallo,
> We are having sporadic NullPointerException in some of the cassandra nodes in cluster (See stacktrace).
> We are having two Datacenter, each having 15 nodes with RF = 2, OS is SLES with java-1_6_0-ibm-1.6.0_sr12.0-0.5.1.
> At present only workaround is to stop the application running on same node and run repair tool on cassandra. We are unable to identify the cause of error.
> 1)
> INFO|ScheduledTasks:1|org.apache.cassandra.service.GCInspector|GC for MarkSweepCompact: 347 ms for 1 collections, 138398568 used; ma
> x is 1051721728
> 2013-06-19T16:25:50:843|ERROR|ReplicateOnWriteStage:115|org.apache.cassandra.service.CassandraDaemon|Exception in thread Thread[ReplicateOnWriteStage:115,5,m
> ain]
> java.lang.RuntimeException: java.lang.NullPointerException
> at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1582)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:897)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:919)
> at java.lang.Thread.run(Thread.java:738)
> Caused by: java.lang.NullPointerException
> at java.util.TreeSet.iterator(TreeSet.java:230)
> at org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:163)
> at org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNamesIterator.java:64)
> at org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:81)
> at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:68)
> at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:274)
> at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65)
> at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1357)
> at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1214)
> at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1126)
> at org.apache.cassandra.db.Table.getRow(Table.java:347)
> at org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:64)
> at org.apache.cassandra.db.CounterMutation.makeReplicationMutation(CounterMutation.java:90)
> at org.apache.cassandra.service.StorageProxy$7$1.runMayThrow(StorageProxy.java:796)
> at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1578)
> ... 3 more
> 2013-06-19T16:26:01:001|ERROR|ReadStage:4833|org.apache.cassandra.service.CassandraDaemon|Exception in thread Thread[ReadStage:4833,5,main]
> java.lang.RuntimeException: java.lang.NullPointerException
> at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1582)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:897)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:919)
> at java.lang.Thread.run(Thread.java:738)
> Caused by: java.lang.NullPointerException
> at java.util.TreeSet.iterator(TreeSet.java:230)
> at org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:163)
> at org.
> 2)
> 2013-06-19T08:38:23:436| INFO|Thread-2447|org.apache.cassandra.service.StorageService|Starting repair command #2, repairing 1 ranges for keyspace system_auth
> 2013-06-19T08:58:25:685|ERROR|ReadStage:9270|org.apache.cassandra.service.CassandraDaemon|Exception in thread Thread[ReadStage:9270,5,main]
> java.lang.NullPointerException
> at java.util.TreeSet.iterator(TreeSet.java:230)
> at org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:163)
> at org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNamesIterator.java:64)
> at org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:81)
> at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:68)
> at org.apache.cassandra.db.CollationController.collectTimeOrderedData(CollationController.java:133)
> at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65)
> at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1357)
> at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1214)
> at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1126)
> at org.apache.cassandra.db.Table.getRow(Table.java:347)
> at org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:64)
> at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:44)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:56)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:908)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:931)
> at java.lang.Thread.run(Thread.java:738)
> Best Regards
> Sanjay
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira