You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Radim Kolar (JIRA)" <ji...@apache.org> on 2012/09/23 20:33:07 UTC
[jira] [Created] (CASSANDRA-4707) EOFException during HH delivery
Radim Kolar created CASSANDRA-4707:
--------------------------------------
Summary: EOFException during HH delivery
Key: CASSANDRA-4707
URL: https://issues.apache.org/jira/browse/CASSANDRA-4707
Project: Cassandra
Issue Type: Bug
Affects Versions: 1.1.5
Environment: windows
Reporter: Radim Kolar
i have got following error during HH every replay. I did nodetool scrub on system.hintedhandoff and sstable is not corrupted, it must have invalid data inside.
INFO [HintedHandoff:10] 2012-09-23 20:26:21,988 HintedHandOffManager.java (line 294) Started hinted handoff for token: 138224439286689469893643387845607487221 with IP: /10.0.0.9
ERROR [HintedHandoff:10] 2012-09-23 20:26:21,988 AbstractCassandraDaemon.java (line 135) Exception in thread Thread[HintedHandoff:10,1,main]
java.io.IOError: java.io.EOFException
at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:259)
at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:275)
at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:232)
at edu.stanford.ppl.concurrent.SnapTreeMap.<init>(SnapTreeMap.java:453)
at org.apache.cassandra.db.AtomicSortedColumns$Holder.<init>(AtomicSortedColumns.java:311)
at org.apache.cassandra.db.AtomicSortedColumns.<init>(AtomicSortedColumns.java:77)
at org.apache.cassandra.db.AtomicSortedColumns.<init>(AtomicSortedColumns.java:48)
at org.apache.cassandra.db.AtomicSortedColumns$1.fromSorted(AtomicSortedColumns.java:61)
at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:399)
at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:144)
at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:136)
at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:129)
at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:439)
at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:447)
at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:353)
at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:256)
at org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:84)
at org.apache.cassandra.db.HintedHandOffManager$3.runMayThrow(HintedHandOffManager.java:436)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.io.EOFException
at java.io.DataInputStream.readFully(DataInputStream.java:197)
at java.io.DataInputStream.readFully(DataInputStream.java:169)
at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:401)
at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:380)
at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:88)
at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:255)
... 21 more
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4707) EOFException during HH delivery
Posted by "Radim Kolar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13461659#comment-13461659 ]
Radim Kolar commented on CASSANDRA-4707:
----------------------------------------
Its related to schema changes. Hints were for nonexisting CF, i recreated that CF inserted some data and hints were replayed without error.
> EOFException during HH delivery
> -------------------------------
>
> Key: CASSANDRA-4707
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4707
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.1.5
> Environment: windows
> Reporter: Radim Kolar
>
> i have got following error during HH every replay. I did nodetool scrub on system.hintedhandoff and sstable is not corrupted, it must have invalid data inside.
> INFO [HintedHandoff:10] 2012-09-23 20:26:21,988 HintedHandOffManager.java (line 294) Started hinted handoff for token: 138224439286689469893643387845607487221 with IP: /10.0.0.9
> ERROR [HintedHandoff:10] 2012-09-23 20:26:21,988 AbstractCassandraDaemon.java (line 135) Exception in thread Thread[HintedHandoff:10,1,main]
> java.io.IOError: java.io.EOFException
> at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:259)
> at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:275)
> at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:232)
> at edu.stanford.ppl.concurrent.SnapTreeMap.<init>(SnapTreeMap.java:453)
> at org.apache.cassandra.db.AtomicSortedColumns$Holder.<init>(AtomicSortedColumns.java:311)
> at org.apache.cassandra.db.AtomicSortedColumns.<init>(AtomicSortedColumns.java:77)
> at org.apache.cassandra.db.AtomicSortedColumns.<init>(AtomicSortedColumns.java:48)
> at org.apache.cassandra.db.AtomicSortedColumns$1.fromSorted(AtomicSortedColumns.java:61)
> at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:399)
> at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:144)
> at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:136)
> at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:129)
> at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:439)
> at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:447)
> at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:353)
> at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:256)
> at org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:84)
> at org.apache.cassandra.db.HintedHandOffManager$3.runMayThrow(HintedHandOffManager.java:436)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:197)
> at java.io.DataInputStream.readFully(DataInputStream.java:169)
> at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:401)
> at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:380)
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:88)
> at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:255)
> ... 21 more
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4707) EOFException during HH delivery
Posted by "Radim Kolar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13461908#comment-13461908 ]
Radim Kolar commented on CASSANDRA-4707:
----------------------------------------
I did drop/create CF on node with hints stored. It was part of other unrelated benchmark. On hh target node i used nodetool resetlocalschema.
> EOFException during HH delivery
> -------------------------------
>
> Key: CASSANDRA-4707
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4707
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.1.5
> Environment: windows
> Reporter: Radim Kolar
>
> i have got following error during HH every replay. I did nodetool scrub on system.hintedhandoff and sstable is not corrupted, it must have invalid data inside.
> INFO [HintedHandoff:10] 2012-09-23 20:26:21,988 HintedHandOffManager.java (line 294) Started hinted handoff for token: 138224439286689469893643387845607487221 with IP: /10.0.0.9
> ERROR [HintedHandoff:10] 2012-09-23 20:26:21,988 AbstractCassandraDaemon.java (line 135) Exception in thread Thread[HintedHandoff:10,1,main]
> java.io.IOError: java.io.EOFException
> at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:259)
> at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:275)
> at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:232)
> at edu.stanford.ppl.concurrent.SnapTreeMap.<init>(SnapTreeMap.java:453)
> at org.apache.cassandra.db.AtomicSortedColumns$Holder.<init>(AtomicSortedColumns.java:311)
> at org.apache.cassandra.db.AtomicSortedColumns.<init>(AtomicSortedColumns.java:77)
> at org.apache.cassandra.db.AtomicSortedColumns.<init>(AtomicSortedColumns.java:48)
> at org.apache.cassandra.db.AtomicSortedColumns$1.fromSorted(AtomicSortedColumns.java:61)
> at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:399)
> at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:144)
> at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:136)
> at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:129)
> at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:439)
> at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:447)
> at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:353)
> at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:256)
> at org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:84)
> at org.apache.cassandra.db.HintedHandOffManager$3.runMayThrow(HintedHandOffManager.java:436)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:197)
> at java.io.DataInputStream.readFully(DataInputStream.java:169)
> at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:401)
> at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:380)
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:88)
> at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:255)
> ... 21 more
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4707) EOFException during HH delivery
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13463669#comment-13463669 ]
Pavel Yaskevich commented on CASSANDRA-4707:
--------------------------------------------
Radim also claims that exception from CASSANDRA-4701 is related to this one.
> EOFException during HH delivery
> -------------------------------
>
> Key: CASSANDRA-4707
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4707
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.1.5
> Environment: windows
> Reporter: Radim Kolar
>
> i have got following error during HH every replay. I did nodetool scrub on system.hintedhandoff and sstable is not corrupted, it must have invalid data inside.
> INFO [HintedHandoff:10] 2012-09-23 20:26:21,988 HintedHandOffManager.java (line 294) Started hinted handoff for token: 138224439286689469893643387845607487221 with IP: /10.0.0.9
> ERROR [HintedHandoff:10] 2012-09-23 20:26:21,988 AbstractCassandraDaemon.java (line 135) Exception in thread Thread[HintedHandoff:10,1,main]
> java.io.IOError: java.io.EOFException
> at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:259)
> at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:275)
> at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:232)
> at edu.stanford.ppl.concurrent.SnapTreeMap.<init>(SnapTreeMap.java:453)
> at org.apache.cassandra.db.AtomicSortedColumns$Holder.<init>(AtomicSortedColumns.java:311)
> at org.apache.cassandra.db.AtomicSortedColumns.<init>(AtomicSortedColumns.java:77)
> at org.apache.cassandra.db.AtomicSortedColumns.<init>(AtomicSortedColumns.java:48)
> at org.apache.cassandra.db.AtomicSortedColumns$1.fromSorted(AtomicSortedColumns.java:61)
> at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:399)
> at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:144)
> at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:136)
> at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:129)
> at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:439)
> at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:447)
> at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:353)
> at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:256)
> at org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:84)
> at org.apache.cassandra.db.HintedHandOffManager$3.runMayThrow(HintedHandOffManager.java:436)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:197)
> at java.io.DataInputStream.readFully(DataInputStream.java:169)
> at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:401)
> at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:380)
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:88)
> at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:255)
> ... 21 more
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4707) EOFException during HH delivery
Posted by "Ivan Velykorodnyy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484143#comment-13484143 ]
Ivan Velykorodnyy commented on CASSANDRA-4707:
----------------------------------------------
Looks like a similar error here:
# TimedOutException on the client
# EOFException on the single cassandra 1.1.5 node:
{code}
ERROR [ReadStage:3028] 2012-10-25 06:40:37,406 AbstractCassandraDaemon.java (line 135) Exception in thread Thread[ReadStage:3028,5,main]
java.io.IOError: java.io.EOFException
at org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SSTableSliceIterator.java:66)
at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.EOFException
at java.io.RandomAccessFile.readFully(RandomAccessFile.java:399)
at java.io.RandomAccessFile.readFully(RandomAccessFile.java:377)
at org.apache.cassandra.io.util.RandomAccessReader.readBytes(RandomAccessReader.java:324)
at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:398)
at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:380)
at org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SSTableSliceIterator.java:56)
... 14 more
{code}
# node restart resolved the issue
There was a scheduled test processing 500 msgs (some cql3 selects+single insert per a message) every 15 mins. Test failed in the middle and all consecutive runs failed as well. Cassandra node restart fixed test.
> EOFException during HH delivery
> -------------------------------
>
> Key: CASSANDRA-4707
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4707
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.1.5
> Environment: windows
> Reporter: Radim Kolar
>
> i have got following error during HH every replay. I did nodetool scrub on system.hintedhandoff and sstable is not corrupted, it must have invalid data inside.
> INFO [HintedHandoff:10] 2012-09-23 20:26:21,988 HintedHandOffManager.java (line 294) Started hinted handoff for token: 138224439286689469893643387845607487221 with IP: /10.0.0.9
> ERROR [HintedHandoff:10] 2012-09-23 20:26:21,988 AbstractCassandraDaemon.java (line 135) Exception in thread Thread[HintedHandoff:10,1,main]
> java.io.IOError: java.io.EOFException
> at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:259)
> at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:275)
> at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:232)
> at edu.stanford.ppl.concurrent.SnapTreeMap.<init>(SnapTreeMap.java:453)
> at org.apache.cassandra.db.AtomicSortedColumns$Holder.<init>(AtomicSortedColumns.java:311)
> at org.apache.cassandra.db.AtomicSortedColumns.<init>(AtomicSortedColumns.java:77)
> at org.apache.cassandra.db.AtomicSortedColumns.<init>(AtomicSortedColumns.java:48)
> at org.apache.cassandra.db.AtomicSortedColumns$1.fromSorted(AtomicSortedColumns.java:61)
> at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:399)
> at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:144)
> at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:136)
> at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:129)
> at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:439)
> at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:447)
> at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:353)
> at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:256)
> at org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:84)
> at org.apache.cassandra.db.HintedHandOffManager$3.runMayThrow(HintedHandOffManager.java:436)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:197)
> at java.io.DataInputStream.readFully(DataInputStream.java:169)
> at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:401)
> at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:380)
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:88)
> at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:255)
> ... 21 more
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4707) EOFException during HH delivery
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13461862#comment-13461862 ]
Jonathan Ellis commented on CASSANDRA-4707:
-------------------------------------------
I don't see how this ties into schema changes. Here's CFSerializer.deserialize:
{code}
public ColumnFamily deserialize(DataInput dis, IColumnSerializer.Flag flag, ISortedColumns.Factory factory) throws IOException
{
if (!dis.readBoolean())
return null;
// create a ColumnFamily based on the cf id
int cfId = dis.readInt();
if (Schema.instance.getCF(cfId) == null)
throw new UnknownColumnFamilyException("Couldn't find cfId=" + cfId, cfId);
ColumnFamily cf = ColumnFamily.create(cfId, factory);
deserializeFromSSTableNoColumns(cf, dis);
deserializeColumns(dis, cf, flag);
return cf;
}
{code}
So:
# serialization relies on the cfId, not cfname, so creating a new cf with the old name should have no effect (the new cf will get a different id)
# if the cf is dropped, it won't be in the Schema map anymore and UnknownColumnFamilyException should be thrown (which HHOM handles)
If you can give steps to reproduce that would help a lot.
> EOFException during HH delivery
> -------------------------------
>
> Key: CASSANDRA-4707
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4707
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.1.5
> Environment: windows
> Reporter: Radim Kolar
>
> i have got following error during HH every replay. I did nodetool scrub on system.hintedhandoff and sstable is not corrupted, it must have invalid data inside.
> INFO [HintedHandoff:10] 2012-09-23 20:26:21,988 HintedHandOffManager.java (line 294) Started hinted handoff for token: 138224439286689469893643387845607487221 with IP: /10.0.0.9
> ERROR [HintedHandoff:10] 2012-09-23 20:26:21,988 AbstractCassandraDaemon.java (line 135) Exception in thread Thread[HintedHandoff:10,1,main]
> java.io.IOError: java.io.EOFException
> at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:259)
> at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:275)
> at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:232)
> at edu.stanford.ppl.concurrent.SnapTreeMap.<init>(SnapTreeMap.java:453)
> at org.apache.cassandra.db.AtomicSortedColumns$Holder.<init>(AtomicSortedColumns.java:311)
> at org.apache.cassandra.db.AtomicSortedColumns.<init>(AtomicSortedColumns.java:77)
> at org.apache.cassandra.db.AtomicSortedColumns.<init>(AtomicSortedColumns.java:48)
> at org.apache.cassandra.db.AtomicSortedColumns$1.fromSorted(AtomicSortedColumns.java:61)
> at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:399)
> at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:144)
> at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:136)
> at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:129)
> at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:439)
> at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:447)
> at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:353)
> at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:256)
> at org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:84)
> at org.apache.cassandra.db.HintedHandOffManager$3.runMayThrow(HintedHandOffManager.java:436)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:197)
> at java.io.DataInputStream.readFully(DataInputStream.java:169)
> at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:401)
> at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:380)
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:88)
> at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:255)
> ... 21 more
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira