You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cassandra.apache.org by Micha <mi...@fantasymail.de> on 2017/11/13 10:55:26 UTC
exception during repair (3.11)
Hi,
I get the following exception during repair. After some of these are
thrown, the cassandra node shuts down. Now I don't know how to get
things working again.
A few days ago there were errors due to a column being much too big.
This is fixed but could this be the reason for this, some corrupt data
because of an insert with a much too big column?
Cheers
Michael
C* 3.11, 7 node cluster
Exception:
ERROR [MessagingService-Incoming-/192.168.0.3] 2017-11-13 09:27:42,263
CassandraDaemon.java:228 - Exception in thread
Thread[MessagingService-Incoming-/192.168.0.3,5,main]
java.io.IOError: java.io.EOFException: EOF after 261977 bytes out of 316099
at
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:227)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:215)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize30(PartitionUpdate.java:848)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize(PartitionUpdate.java:809)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:415)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:434)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.hints.Hint$Serializer.deserialize(Hint.java:148)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.hints.HintMessage$Serializer.deserialize(HintMessage.java:123)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.hints.HintMessage$Serializer.deserialize(HintMessage.java:82)
~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.net.MessageIn.read(MessageIn.java:123)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:192)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:180)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:94)
~[apache-cassandra-3.11.1.jar:3.11.1]
Caused by: java.io.EOFException: EOF after 261977 bytes out of 316099
at
org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:68)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:60)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.io.util.TrackedDataInputPlus.readFully(TrackedDataInputPlus.java:93)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:402)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.db.marshal.AbstractType.readValue(AbstractType.java:446)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.db.rows.Cell$Serializer.deserialize(Cell.java:245)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.db.rows.UnfilteredSerializer.readSimpleColumn(UnfilteredSerializer.java:644)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.db.rows.UnfilteredSerializer.lambda$deserializeRowBody$1(UnfilteredSerializer.java:609)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.utils.btree.BTree.applyForwards(BTree.java:1242)
~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.utils.btree.BTree.apply(BTree.java:1197)
~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.Columns.apply(Columns.java:377)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.db.rows.UnfilteredSerializer.deserializeRowBody(UnfilteredSerializer.java:605)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.db.rows.UnfilteredSerializer.deserializeOne(UnfilteredSerializer.java:480)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.db.rows.UnfilteredSerializer.deserialize(UnfilteredSerializer.java:436)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:222)
~[apache-cassandra-3.11.1.jar:3.11.1]
... 13 common frames omitted
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
For additional commands, e-mail: dev-help@cassandra.apache.org
Re: exception during repair (3.11)
Posted by Jeff Jirsa <jj...@gmail.com>.
Can you open a JIRA with your stack trace?
What exactly was the problem with the column being too big? How big was it? What did you do to “fix” it?
--
Jeff Jirsa
> On Nov 13, 2017, at 2:55 AM, Micha <mi...@fantasymail.de> wrote:
>
> Hi,
>
> I get the following exception during repair. After some of these are
> thrown, the cassandra node shuts down. Now I don't know how to get
> things working again.
>
> A few days ago there were errors due to a column being much too big.
> This is fixed but could this be the reason for this, some corrupt data
> because of an insert with a much too big column?
>
>
>
> Cheers
> Michael
>
> C* 3.11, 7 node cluster
>
>
> Exception:
>
> ERROR [MessagingService-Incoming-/192.168.0.3] 2017-11-13 09:27:42,263
> CassandraDaemon.java:228 - Exception in thread
> Thread[MessagingService-Incoming-/192.168.0.3,5,main]
> java.io.IOError: java.io.EOFException: EOF after 261977 bytes out of 316099
> at
> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:227)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:215)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize30(PartitionUpdate.java:848)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize(PartitionUpdate.java:809)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:415)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:434)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.hints.Hint$Serializer.deserialize(Hint.java:148)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.hints.HintMessage$Serializer.deserialize(HintMessage.java:123)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.hints.HintMessage$Serializer.deserialize(HintMessage.java:82)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at org.apache.cassandra.net.MessageIn.read(MessageIn.java:123)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:192)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:180)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:94)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> Caused by: java.io.EOFException: EOF after 261977 bytes out of 316099
> at
> org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:68)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:60)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.io.util.TrackedDataInputPlus.readFully(TrackedDataInputPlus.java:93)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:402)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.db.marshal.AbstractType.readValue(AbstractType.java:446)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.db.rows.Cell$Serializer.deserialize(Cell.java:245)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.db.rows.UnfilteredSerializer.readSimpleColumn(UnfilteredSerializer.java:644)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.db.rows.UnfilteredSerializer.lambda$deserializeRowBody$1(UnfilteredSerializer.java:609)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.utils.btree.BTree.applyForwards(BTree.java:1242)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at org.apache.cassandra.utils.btree.BTree.apply(BTree.java:1197)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at org.apache.cassandra.db.Columns.apply(Columns.java:377)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.db.rows.UnfilteredSerializer.deserializeRowBody(UnfilteredSerializer.java:605)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.db.rows.UnfilteredSerializer.deserializeOne(UnfilteredSerializer.java:480)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.db.rows.UnfilteredSerializer.deserialize(UnfilteredSerializer.java:436)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:222)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> ... 13 common frames omitted
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: dev-help@cassandra.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
For additional commands, e-mail: dev-help@cassandra.apache.org