You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Serhat Rıfat Demircan (JIRA)" <ji...@apache.org> on 2016/07/04 13:32:11 UTC
[jira] [Commented] (CASSANDRA-12113) Cassandra 3.5.0 Repair Error
[ https://issues.apache.org/jira/browse/CASSANDRA-12113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15361317#comment-15361317 ]
Serhat Rıfat Demircan commented on CASSANDRA-12113:
---------------------------------------------------
On source node:
{noformat}
ERROR [RepairJobTask:18] 2016-06-16 02:56:47,502 RepairSession.java:290 - [repair #cc96d270-3352-11e6-b94d-af0e016acd9a] Session completed with the following error
org.apache.cassandra.streaming.StreamException: Stream failed
at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85) ~[apache-cassandra-3.5.0.jar:3.5.0]
at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202) ~[guava-18.0.jar:na]
at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:213) ~[apache-cassandra-3.5.0.jar:3.5.0]
at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:189) ~[apache-cassandra-3.5.0.jar:3.5.0]
at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:429) ~[apache-cassandra-3.5.0.jar:3.5.0]
at org.apache.cassandra.streaming.StreamSession.onError(StreamSession.java:524) ~[apache-cassandra-3.5.0.jar:3.5.0]
at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:217) ~[apache-cassandra-3.5.0.jar:3.5.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_91]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_91]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_91]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_91]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
{noformat}
{noformat}
ERROR [StreamReceiveTask:4] 2016-06-16 02:58:01,817 StreamSession.java:519 - [Stream #f1c8a720-3353-11e6-b94d-af0e016acd9a] Streaming error occurred
java.lang.RuntimeException: Outgoing stream handler has been closed
at org.apache.cassandra.streaming.ConnectionHandler.sendMessage(ConnectionHandler.java:142) ~[apache-cassandra-3.5.0.jar:3.5.0]
at org.apache.cassandra.streaming.StreamSession.maybeCompleted(StreamSession.java:711) ~[apache-cassandra-3.5.0.jar:3.5.0]
at org.apache.cassandra.streaming.StreamSession.taskCompleted(StreamSession.java:667) ~[apache-cassandra-3.5.0.jar:3.5.0]
at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:211) ~[apache-cassandra-3.5.0.jar:3.5.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_91]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_91]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_91]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_91]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
{noformat}
> Cassandra 3.5.0 Repair Error
> ----------------------------
>
> Key: CASSANDRA-12113
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12113
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
> Environment: Production
> Reporter: Serhat Rıfat Demircan
> Assignee: Paulo Motta
>
> I got the following error while repairing nodes with the "nodetool repair" command. Error occured on 2 nodes in the cluster which have 9 nodes.
>
> Interesting thing is corrupted sstable is no more exists one of 2 nodes. Copied existing one to test cluster and restored table from that sstable. No error occured on test cluster.
> {noformat}
> ERROR [StreamReceiveTask:6] 2016-06-16 02:56:47,480 StreamReceiveTask.java:215 - Error applying streamed data:
> java.lang.RuntimeException: java.util.concurrent.ExecutionException: org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: /var/lib/cassandra/data/keyspace/table/ma-1518-big-Data.db
> at org.apache.cassandra.utils.Throwables.maybeFail(Throwables.java:50) ~[apache-cassandra-3.5.0.jar:3.5.0]
> at org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:372) ~[apache-cassandra-3.5.0.jar:3.5.0]
> at org.apache.cassandra.index.SecondaryIndexManager.buildIndexesBlocking(SecondaryIndexManager.java:375) ~[apache-cassandra-3.5.0.jar:3.5.0]
> at org.apache.cassandra.index.SecondaryIndexManager.buildAllIndexesBlocking(SecondaryIndexManager.java:262) ~[apache-cassandra-3.5.0.jar:3.5.0]
> at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:182) ~[apache-cassandra-3.5.0.jar:3.5.0]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_91]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_91]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_91]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_91]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
> Caused by: java.util.concurrent.ExecutionException: org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: /var/lib/cassandra/data/keyspace/table/ma-1518-big-Data.db
> at java.util.concurrent.FutureTask.report(FutureTask.java:122) [na:1.8.0_91]
> at java.util.concurrent.FutureTask.get(FutureTask.java:192) [na:1.8.0_91]
> at org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:365) ~[apache-cassandra-3.5.0.jar:3.5.0]
> ... 8 common frames omitted
> Caused by: org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: /var/lib/cassandra/data/keyspace/table/ma-1518-big-Data.db
> at org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:367) ~[apache-cassandra-3.5.0.jar:3.5.0]
> at org.apache.cassandra.db.columniterator.AbstractSSTableIterator.hasNext(AbstractSSTableIterator.java:229) ~[apache-cassandra-3.5.0.jar:3.5.0]
> at org.apache.cassandra.db.columniterator.SSTableIterator.hasNext(SSTableIterator.java:32) ~[apache-cassandra-3.5.0.jar:3.5.0]
> at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:100) ~[apache-cassandra-3.5.0.jar:3.5.0]
> at org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.computeNext(UnfilteredRowIteratorWithLowerBound.java:93) ~[apache-cassandra-3.5.0.jar:3.5.0]
> at org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.computeNext(UnfilteredRowIteratorWithLowerBound.java:25) ~[apache-cassandra-3.5.0.jar:3.5.0]
> at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.5.0.jar:3.5.0]
> at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:374) ~[apache-cassandra-3.5.0.jar:3.5.0]
> at org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:186) ~[apache-cassandra-3.5.0.jar:3.5.0]
> at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:155) ~[apache-cassandra-3.5.0.jar:3.5.0]
> at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.5.0.jar:3.5.0]
> at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:419) ~[apache-cassandra-3.5.0.jar:3.5.0]
> at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:279) ~[apache-cassandra-3.5.0.jar:3.5.0]
> at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.5.0.jar:3.5.0]
> at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:112) ~[apache-cassandra-3.5.0.jar:3.5.0]
> at org.apache.cassandra.index.SecondaryIndexManager.indexPartition(SecondaryIndexManager.java:549) ~[apache-cassandra-3.5.0.jar:3.5.0]
> at org.apache.cassandra.db.Keyspace.indexPartition(Keyspace.java:570) ~[apache-cassandra-3.5.0.jar:3.5.0]
> at org.apache.cassandra.index.internal.CollatedViewIndexBuilder.build(CollatedViewIndexBuilder.java:70) ~[apache-cassandra-3.5.0.jar:3.5.0]
> at org.apache.cassandra.db.compaction.CompactionManager$12.run(CompactionManager.java:1472) ~[apache-cassandra-3.5.0.jar:3.5.0]
> ... 5 common frames omitted
> Caused by: java.io.IOException: Error building row with data deserialized from /var/lib/cassandra/data/keyspace/table/ma-1518-big-Data.db - chunk length 65536, data length 85398996.
> at org.apache.cassandra.db.rows.UnfilteredSerializer.deserializeRowBody(UnfilteredSerializer.java:446) ~[apache-cassandra-3.5.0.jar:3.5.0]
> at org.apache.cassandra.db.UnfilteredDeserializer$CurrentDeserializer.readNext(UnfilteredDeserializer.java:211) ~[apache-cassandra-3.5.0.jar:3.5.0]
> at org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.computeNext(SSTableIterator.java:125) ~[apache-cassandra-3.5.0.jar:3.5.0]
> at org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.hasNextInternal(SSTableIterator.java:149) ~[apache-cassandra-3.5.0.jar:3.5.0]
> at org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:354) ~[apache-cassandra-3.5.0.jar:3.5.0]
> ... 23 common frames omitted
> Caused by: java.lang.AssertionError: Memory was freed
> at org.apache.cassandra.io.util.SafeMemory.checkBounds(SafeMemory.java:103) ~[apache-cassandra-3.5.0.jar:3.5.0]
> at org.apache.cassandra.io.util.Memory.getLong(Memory.java:260) ~[apache-cassandra-3.5.0.jar:3.5.0]
> at org.apache.cassandra.io.compress.CompressionMetadata.chunkFor(CompressionMetadata.java:235) ~[apache-cassandra-3.5.0.jar:3.5.0]
> at org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBufferMmap(CompressedRandomAccessReader.java:170) ~[apache-cassandra-3.5.0.jar:3.5.0]
> at org.apache.cassandra.io.util.RandomAccessReader.reBuffer(RandomAccessReader.java:111) ~[apache-cassandra-3.5.0.jar:3.5.0]
> at org.apache.cassandra.io.util.RebufferingInputStream.readByte(RebufferingInputStream.java:144) ~[apache-cassandra-3.5.0.jar:3.5.0]
> at org.apache.cassandra.utils.vint.VIntCoding.readUnsignedVInt(VIntCoding.java:63) ~[apache-cassandra-3.5.0.jar:3.5.0]
> at org.apache.cassandra.io.util.RebufferingInputStream.readUnsignedVInt(RebufferingInputStream.java:209) ~[apache-cassandra-3.5.0.jar:3.5.0]
> at org.apache.cassandra.db.SerializationHeader.readLocalDeletionTime(SerializationHeader.java:210) ~[apache-cassandra-3.5.0.jar:3.5.0]
> at org.apache.cassandra.db.rows.Cell$Serializer.deserialize(Cell.java:224) ~[apache-cassandra-3.5.0.jar:3.5.0]
> at org.apache.cassandra.db.rows.UnfilteredSerializer.readSimpleColumn(UnfilteredSerializer.java:455) ~[apache-cassandra-3.5.0.jar:3.5.0]
> at org.apache.cassandra.db.rows.UnfilteredSerializer.deserializeRowBody(UnfilteredSerializer.java:433) ~[apache-cassandra-3.5.0.jar:3.5.0]
> ... 27 common frames omitted
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)