You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by 曹志富 <ca...@gmail.com> on 2015/05/29 03:39:57 UTC
what this error mean
I have a 25 noedes C* cluster with C* 2.1.3. These days a node occur split
brain many times。
check the log I found this:
INFO [MemtableFlushWriter:118] 2015-05-29 08:07:39,176
Memtable.java:378 - Completed flushing
/home/ant/apache-cassandra-2.1.3/bin/../data/data/system/sstable_activity-5a1ff2
67ace03f128563cfae6103c65e/system-sstable_activity-ka-4371-Data.db (8187
bytes) for commitlog position ReplayPosition(segmentId=1432775133526,
position=16684949)
ERROR [IndexSummaryManager:1] 2015-05-29 08:10:30,209
CassandraDaemon.java:167 - Exception in thread
Thread[IndexSummaryManager:1,1,main]
java.lang.AssertionError: null
at
org.apache.cassandra.io.sstable.SSTableReader.cloneWithNewSummarySamplingLevel(SSTableReader.java:921)
~[apache-cassandra-2.1.3.jar:2.1.3]
at
org.apache.cassandra.io.sstable.IndexSummaryManager.adjustSamplingLevels(IndexSummaryManager.java:410)
~[apache-cassandra-2.1.3.jar:2.1.3]
at
org.apache.cassandra.io.sstable.IndexSummaryManager.redistributeSummaries(IndexSummaryManager.java:288)
~[apache-cassandra-2.1.3.jar:2.1.3]
at
org.apache.cassandra.io.sstable.IndexSummaryManager.redistributeSummaries(IndexSummaryManager.java:238)
~[apache-cassandra-2.1.3.jar:2.1.3]
at
org.apache.cassandra.io.sstable.IndexSummaryManager$1.runMayThrow(IndexSummaryManager.java:139)
~[apache-cassandra-2.1.3.jar:2.1.3]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[apache-cassandra-2.1.3.jar:2.1.3]
at
org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:82)
~[apache-cassandra-2.
1.3.jar:2.1.3]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
[na:1.7.0_71]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
[na:1.7.0_71]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
[na:1.7.0_71]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[na:1.7.0_71]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_71]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_71]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]
I want to know why this and how to fix this
Thanks all
--------------------------------------
Ranger Tsao
Re: what this error mean
Posted by 曹志富 <ca...@gmail.com>.
seems the issue can clause brain split
--------------------------------------
Ranger Tsao
2015-05-29 14:57 GMT+08:00 Jason Wee <pe...@gmail.com>:
> why it happened? from the code, it looks like this condition is not null
> https://github.com/apache/cassandra/blob/cassandra-2.1.3/src/java/org/apache/cassandra/io/sstable/SSTableReader.java#L921
>
> or you can quickly fix this by upgrading to 2.1.5, i noticed there is code
> change for this class
> https://github.com/apache/cassandra/blob/cassandra-2.1.5/src/java/org/apache/cassandra/io/sstable/SSTableReader.java#L921
>
>
> hth
>
> jason
>
>
>
> On Fri, May 29, 2015 at 9:39 AM, 曹志富 <ca...@gmail.com> wrote:
>
>> I have a 25 noedes C* cluster with C* 2.1.3. These days a node occur
>> split brain many times。
>>
>> check the log I found this:
>>
>> INFO [MemtableFlushWriter:118] 2015-05-29 08:07:39,176
>> Memtable.java:378 - Completed flushing
>> /home/ant/apache-cassandra-2.1.3/bin/../data/data/system/sstable_activity-5a1ff2
>> 67ace03f128563cfae6103c65e/system-sstable_activity-ka-4371-Data.db (8187
>> bytes) for commitlog position ReplayPosition(segmentId=1432775133526,
>> position=16684949)
>> ERROR [IndexSummaryManager:1] 2015-05-29 08:10:30,209
>> CassandraDaemon.java:167 - Exception in thread
>> Thread[IndexSummaryManager:1,1,main]
>> java.lang.AssertionError: null
>> at
>> org.apache.cassandra.io.sstable.SSTableReader.cloneWithNewSummarySamplingLevel(SSTableReader.java:921)
>> ~[apache-cassandra-2.1.3.jar:2.1.3]
>> at
>> org.apache.cassandra.io.sstable.IndexSummaryManager.adjustSamplingLevels(IndexSummaryManager.java:410)
>> ~[apache-cassandra-2.1.3.jar:2.1.3]
>> at
>> org.apache.cassandra.io.sstable.IndexSummaryManager.redistributeSummaries(IndexSummaryManager.java:288)
>> ~[apache-cassandra-2.1.3.jar:2.1.3]
>> at
>> org.apache.cassandra.io.sstable.IndexSummaryManager.redistributeSummaries(IndexSummaryManager.java:238)
>> ~[apache-cassandra-2.1.3.jar:2.1.3]
>> at
>> org.apache.cassandra.io.sstable.IndexSummaryManager$1.runMayThrow(IndexSummaryManager.java:139)
>> ~[apache-cassandra-2.1.3.jar:2.1.3]
>> at
>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>> ~[apache-cassandra-2.1.3.jar:2.1.3]
>> at
>> org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:82)
>> ~[apache-cassandra-2.
>> 1.3.jar:2.1.3]
>> at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> [na:1.7.0_71]
>> at
>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>> [na:1.7.0_71]
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>> [na:1.7.0_71]
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>> [na:1.7.0_71]
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> [na:1.7.0_71]
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> [na:1.7.0_71]
>> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]
>>
>> I want to know why this and how to fix this
>>
>> Thanks all
>> --------------------------------------
>> Ranger Tsao
>>
>
>
Re: what this error mean
Posted by Jason Wee <pe...@gmail.com>.
why it happened? from the code, it looks like this condition is not null
https://github.com/apache/cassandra/blob/cassandra-2.1.3/src/java/org/apache/cassandra/io/sstable/SSTableReader.java#L921
or you can quickly fix this by upgrading to 2.1.5, i noticed there is code
change for this class
https://github.com/apache/cassandra/blob/cassandra-2.1.5/src/java/org/apache/cassandra/io/sstable/SSTableReader.java#L921
hth
jason
On Fri, May 29, 2015 at 9:39 AM, 曹志富 <ca...@gmail.com> wrote:
> I have a 25 noedes C* cluster with C* 2.1.3. These days a node occur split
> brain many times。
>
> check the log I found this:
>
> INFO [MemtableFlushWriter:118] 2015-05-29 08:07:39,176
> Memtable.java:378 - Completed flushing
> /home/ant/apache-cassandra-2.1.3/bin/../data/data/system/sstable_activity-5a1ff2
> 67ace03f128563cfae6103c65e/system-sstable_activity-ka-4371-Data.db (8187
> bytes) for commitlog position ReplayPosition(segmentId=1432775133526,
> position=16684949)
> ERROR [IndexSummaryManager:1] 2015-05-29 08:10:30,209
> CassandraDaemon.java:167 - Exception in thread
> Thread[IndexSummaryManager:1,1,main]
> java.lang.AssertionError: null
> at
> org.apache.cassandra.io.sstable.SSTableReader.cloneWithNewSummarySamplingLevel(SSTableReader.java:921)
> ~[apache-cassandra-2.1.3.jar:2.1.3]
> at
> org.apache.cassandra.io.sstable.IndexSummaryManager.adjustSamplingLevels(IndexSummaryManager.java:410)
> ~[apache-cassandra-2.1.3.jar:2.1.3]
> at
> org.apache.cassandra.io.sstable.IndexSummaryManager.redistributeSummaries(IndexSummaryManager.java:288)
> ~[apache-cassandra-2.1.3.jar:2.1.3]
> at
> org.apache.cassandra.io.sstable.IndexSummaryManager.redistributeSummaries(IndexSummaryManager.java:238)
> ~[apache-cassandra-2.1.3.jar:2.1.3]
> at
> org.apache.cassandra.io.sstable.IndexSummaryManager$1.runMayThrow(IndexSummaryManager.java:139)
> ~[apache-cassandra-2.1.3.jar:2.1.3]
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> ~[apache-cassandra-2.1.3.jar:2.1.3]
> at
> org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:82)
> ~[apache-cassandra-2.
> 1.3.jar:2.1.3]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> [na:1.7.0_71]
> at
> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> [na:1.7.0_71]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> [na:1.7.0_71]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> [na:1.7.0_71]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [na:1.7.0_71]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_71]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]
>
> I want to know why this and how to fix this
>
> Thanks all
> --------------------------------------
> Ranger Tsao
>