You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "ivan (JIRA)" <ji...@apache.org> on 2011/07/21 14:35:58 UTC
[jira] [Created] (CASSANDRA-2930) corrupt commitlog
corrupt commitlog
-----------------
Key: CASSANDRA-2930
URL: https://issues.apache.org/jira/browse/CASSANDRA-2930
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 0.8.1
Environment: Linux, amd64.
Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
Reporter: ivan
We get "Exception encountered during startup" error while Cassandra starts.
Error messages:
INFO 13:56:28,736 Finished reading /var/lib/cassandra/commitlog/CommitLog-1310637513214.log
ERROR 13:56:28,736 Exception encountered during startup.
java.io.IOError: java.io.EOFException
at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:265)
at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:281)
at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:236)
at java.util.concurrent.ConcurrentSkipListMap.buildFromSorted(ConcurrentSkipListMap.java:1493)
at java.util.concurrent.ConcurrentSkipListMap.<init>(ConcurrentSkipListMap.java:1443)
at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:419)
at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:139)
at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:127)
at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:382)
at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:278)
at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:158)
at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:175)
at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:368)
at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:80)
Caused by: java.io.EOFException
at java.io.DataInputStream.readFully(DataInputStream.java:180)
at java.io.DataInputStream.readFully(DataInputStream.java:152)
at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:394)
at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:368)
at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:87)
at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:261)
... 13 more
Exception encountered during startup.
java.io.IOError: java.io.EOFException
at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:265)
at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:281)
at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:236)
at java.util.concurrent.ConcurrentSkipListMap.buildFromSorted(ConcurrentSkipListMap.java:1493)
at java.util.concurrent.ConcurrentSkipListMap.<init>(ConcurrentSkipListMap.java:1443)
at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:419)
at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:139)
at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:127)
at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:382)
at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:278)
at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:158)
at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:175)
at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:368)
at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:80)
Caused by: java.io.EOFException
at java.io.DataInputStream.readFully(DataInputStream.java:180)
at java.io.DataInputStream.readFully(DataInputStream.java:152)
at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:394)
at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:368)
at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:87)
at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:261)
... 13 more
After some debugging I found that in some serialized supercolumns column counter is less than the number of serialized columns. Difference was always 1 in corrupt commitlogs. This error always appears with supercolumns with more than one column, but there are properly serialized supercolumns also in commitlog.
I have no clue yet why this error happens. I suspect it maybe a race condition.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2930) corrupt commitlog
Posted by "Cathy Daw (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13194247#comment-13194247 ]
Cathy Daw commented on CASSANDRA-2930:
--------------------------------------
I was able to reproduce this with C* 1.0.7 running the following script. Running the following script, I had a timeout error from Pycassa, and on the next server startup, I received the stack above.
{code}
#!/usr/bin/python
import time
import pycassa
from pycassa import system_manager
from pycassa.system_manager import *
from threading import Thread
def test_setup():
m = pycassa.system_manager.SystemManager('localhost:9160')
pool = pycassa.pool.ConnectionPool('testks',server_list=['localhost:9160', ], timeout=5, pool_size=16, max_overflow=0, prefill=False, pool_timeout=30, max_retries=8)
kspaces = m.list_keyspaces()
if 'testks' in kspaces:
m.drop_keyspace('testks')
m.create_keyspace('testks', SIMPLE_STRATEGY, {'replication_factor': '1'})
cfs = m.get_keyspace_column_families('testks')
if 'test_super_cf' not in cfs:
m.create_column_family('testks', 'test_super_cf', super=True, key_validation_class=system_manager.ASCII_TYPE,
comparator_type=system_manager.INT_TYPE, subcomparator_type=system_manager.INT_TYPE,
default_validation_class=system_manager.INT_TYPE)
global db
db = pycassa.ColumnFamily(pool, 'test_super_cf',read_consistency_level=pycassa.ConsistencyLevel.QUORUM,
write_consistency_level=pycassa.ConsistencyLevel.QUORUM)
def test_mutation():
global keepGoing
keepGoing=True
global currentTest
# Test #1
print "Thread 1: Start test1"
currentTest = 'test1'
for a in range(0, 1000):
db.insert('testrow', {a : dict([(i, i) for i in range(100000)])})
print "Thread 1: Finish test1"
# Test #2
print "Thread 1: Start test2"
currentTest = 'test2'
db.remove('testrow', super_column=400, columns=dict([(i, i) for i in range(100000)]))
print "Thread 1: Start test2"
# Test #3
print "Thread 1: Start test3"
currentTest = 'test3'
for a in range(501, 1000):
db.remove('testrow', super_column=a)
print "Thread 1: Finish test3"
time.sleep(3)
keepGoing=False
def check_rowcount():
while keepGoing:
test = currentTest
currentSuperColCount = db.get_count('testrow')
currentSub1ColCount = db.get_count('testrow', super_column=1)
currentSub400ColCount = db.get_count('testrow', super_column=400)
if currentTest != 'test1' and currentSuperColCount not in [0, 500, 1000]:
print "--- Thread 2: Mismatch Super Column Count. # Super Columns: " + str(currentSuperColCount) + ". Current Test: " + test
if currentSub1ColCount not in [0, 100000]:
print "--- Thread 2: Mismatch Sub Column Count. # for column 1: " + str(currentSub1ColCount) + ". Current Test: " + test
if currentSub400ColCount not in [0, 100000]:
print "--- Thread 2: Mismatch Sub Column Count. # for column 400: " + str(currentSub400ColCount) + ". Current Test: " + test
if __name__ == '__main__':
test_setup()
t1 = Thread(target = test_mutation)
t1.start()
time.sleep(1)
t2 = Thread(target = check_rowcount)
t2.start()
{code}
> corrupt commitlog
> -----------------
>
> Key: CASSANDRA-2930
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2930
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.1
> Environment: Linux, amd64.
> Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
> Reporter: ivan
> Assignee: Rick Branson
> Attachments: CommitLog-1310637513214.log
>
>
> We get "Exception encountered during startup" error while Cassandra starts.
> Error messages:
> INFO 13:56:28,736 Finished reading /var/lib/cassandra/commitlog/CommitLog-1310637513214.log
> ERROR 13:56:28,736 Exception encountered during startup.
> java.io.IOError: java.io.EOFException
> at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:265)
> at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:281)
> at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:236)
> at java.util.concurrent.ConcurrentSkipListMap.buildFromSorted(ConcurrentSkipListMap.java:1493)
> at java.util.concurrent.ConcurrentSkipListMap.<init>(ConcurrentSkipListMap.java:1443)
> at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:419)
> at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:139)
> at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:127)
> at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:382)
> at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:278)
> at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:158)
> at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:175)
> at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:368)
> at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:80)
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:180)
> at java.io.DataInputStream.readFully(DataInputStream.java:152)
> at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:394)
> at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:368)
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:87)
> at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:261)
> ... 13 more
> Exception encountered during startup.
> java.io.IOError: java.io.EOFException
> at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:265)
> at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:281)
> at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:236)
> at java.util.concurrent.ConcurrentSkipListMap.buildFromSorted(ConcurrentSkipListMap.java:1493)
> at java.util.concurrent.ConcurrentSkipListMap.<init>(ConcurrentSkipListMap.java:1443)
> at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:419)
> at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:139)
> at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:127)
> at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:382)
> at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:278)
> at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:158)
> at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:175)
> at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:368)
> at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:80)
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:180)
> at java.io.DataInputStream.readFully(DataInputStream.java:152)
> at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:394)
> at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:368)
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:87)
> at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:261)
> ... 13 more
> After some debugging I found that in some serialized supercolumns column counter is less than the number of serialized columns. Difference was always 1 in corrupt commitlogs. This error always appears with supercolumns with more than one column, but there are properly serialized supercolumns also in commitlog.
> I have no clue yet why this error happens. I suspect it maybe a race condition.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2930) corrupt commitlog
Posted by "Rick Branson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13195049#comment-13195049 ]
Rick Branson commented on CASSANDRA-2930:
-----------------------------------------
After analyzing the corrupt commit log posted, the issue is that the serialization code actually wrote out an incorrect supercolumn. The commit log code faithfully checksum'd and wrote the data out as if it was valid. It appears that the subcolumn count exceeded the actual subcolumn data written out when serialized by 1, so the deserialization code is expecting another subcolumn and runs out of buffer, bailing with EOFException.
I was unable to repro with Cathy's script after running it about a half dozen times.
> corrupt commitlog
> -----------------
>
> Key: CASSANDRA-2930
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2930
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.1
> Environment: Linux, amd64.
> Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
> Reporter: ivan
> Assignee: Rick Branson
> Attachments: CommitLog-1310637513214.log
>
>
> We get "Exception encountered during startup" error while Cassandra starts.
> Error messages:
> INFO 13:56:28,736 Finished reading /var/lib/cassandra/commitlog/CommitLog-1310637513214.log
> ERROR 13:56:28,736 Exception encountered during startup.
> java.io.IOError: java.io.EOFException
> at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:265)
> at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:281)
> at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:236)
> at java.util.concurrent.ConcurrentSkipListMap.buildFromSorted(ConcurrentSkipListMap.java:1493)
> at java.util.concurrent.ConcurrentSkipListMap.<init>(ConcurrentSkipListMap.java:1443)
> at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:419)
> at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:139)
> at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:127)
> at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:382)
> at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:278)
> at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:158)
> at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:175)
> at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:368)
> at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:80)
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:180)
> at java.io.DataInputStream.readFully(DataInputStream.java:152)
> at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:394)
> at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:368)
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:87)
> at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:261)
> ... 13 more
> Exception encountered during startup.
> java.io.IOError: java.io.EOFException
> at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:265)
> at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:281)
> at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:236)
> at java.util.concurrent.ConcurrentSkipListMap.buildFromSorted(ConcurrentSkipListMap.java:1493)
> at java.util.concurrent.ConcurrentSkipListMap.<init>(ConcurrentSkipListMap.java:1443)
> at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:419)
> at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:139)
> at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:127)
> at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:382)
> at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:278)
> at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:158)
> at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:175)
> at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:368)
> at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:80)
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:180)
> at java.io.DataInputStream.readFully(DataInputStream.java:152)
> at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:394)
> at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:368)
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:87)
> at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:261)
> ... 13 more
> After some debugging I found that in some serialized supercolumns column counter is less than the number of serialized columns. Difference was always 1 in corrupt commitlogs. This error always appears with supercolumns with more than one column, but there are properly serialized supercolumns also in commitlog.
> I have no clue yet why this error happens. I suspect it maybe a race condition.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2930) corrupt commitlog
Posted by "Rick Branson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13195361#comment-13195361 ]
Rick Branson commented on CASSANDRA-2930:
-----------------------------------------
+1 on CASSANDRA-3237. Polishing the supercolumn turd is futile. The heisenbug that's generating this one is so rare that few have even seen it, let alone found a way to reproduce it reliably.
> corrupt commitlog
> -----------------
>
> Key: CASSANDRA-2930
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2930
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.1
> Environment: Linux, amd64.
> Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
> Reporter: ivan
> Assignee: Rick Branson
> Attachments: CommitLog-1310637513214.log
>
>
> We get "Exception encountered during startup" error while Cassandra starts.
> Error messages:
> INFO 13:56:28,736 Finished reading /var/lib/cassandra/commitlog/CommitLog-1310637513214.log
> ERROR 13:56:28,736 Exception encountered during startup.
> java.io.IOError: java.io.EOFException
> at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:265)
> at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:281)
> at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:236)
> at java.util.concurrent.ConcurrentSkipListMap.buildFromSorted(ConcurrentSkipListMap.java:1493)
> at java.util.concurrent.ConcurrentSkipListMap.<init>(ConcurrentSkipListMap.java:1443)
> at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:419)
> at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:139)
> at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:127)
> at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:382)
> at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:278)
> at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:158)
> at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:175)
> at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:368)
> at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:80)
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:180)
> at java.io.DataInputStream.readFully(DataInputStream.java:152)
> at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:394)
> at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:368)
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:87)
> at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:261)
> ... 13 more
> Exception encountered during startup.
> java.io.IOError: java.io.EOFException
> at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:265)
> at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:281)
> at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:236)
> at java.util.concurrent.ConcurrentSkipListMap.buildFromSorted(ConcurrentSkipListMap.java:1493)
> at java.util.concurrent.ConcurrentSkipListMap.<init>(ConcurrentSkipListMap.java:1443)
> at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:419)
> at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:139)
> at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:127)
> at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:382)
> at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:278)
> at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:158)
> at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:175)
> at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:368)
> at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:80)
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:180)
> at java.io.DataInputStream.readFully(DataInputStream.java:152)
> at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:394)
> at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:368)
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:87)
> at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:261)
> ... 13 more
> After some debugging I found that in some serialized supercolumns column counter is less than the number of serialized columns. Difference was always 1 in corrupt commitlogs. This error always appears with supercolumns with more than one column, but there are properly serialized supercolumns also in commitlog.
> I have no clue yet why this error happens. I suspect it maybe a race condition.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2930) corrupt commitlog
Posted by "ivan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ivan updated CASSANDRA-2930:
----------------------------
Attachment: CommitLog-1310637513214.log
A corrupt serialized row from a corrupt commitlog.
> corrupt commitlog
> -----------------
>
> Key: CASSANDRA-2930
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2930
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.1
> Environment: Linux, amd64.
> Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
> Reporter: ivan
> Attachments: CommitLog-1310637513214.log
>
>
> We get "Exception encountered during startup" error while Cassandra starts.
> Error messages:
> INFO 13:56:28,736 Finished reading /var/lib/cassandra/commitlog/CommitLog-1310637513214.log
> ERROR 13:56:28,736 Exception encountered during startup.
> java.io.IOError: java.io.EOFException
> at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:265)
> at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:281)
> at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:236)
> at java.util.concurrent.ConcurrentSkipListMap.buildFromSorted(ConcurrentSkipListMap.java:1493)
> at java.util.concurrent.ConcurrentSkipListMap.<init>(ConcurrentSkipListMap.java:1443)
> at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:419)
> at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:139)
> at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:127)
> at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:382)
> at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:278)
> at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:158)
> at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:175)
> at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:368)
> at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:80)
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:180)
> at java.io.DataInputStream.readFully(DataInputStream.java:152)
> at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:394)
> at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:368)
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:87)
> at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:261)
> ... 13 more
> Exception encountered during startup.
> java.io.IOError: java.io.EOFException
> at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:265)
> at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:281)
> at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:236)
> at java.util.concurrent.ConcurrentSkipListMap.buildFromSorted(ConcurrentSkipListMap.java:1493)
> at java.util.concurrent.ConcurrentSkipListMap.<init>(ConcurrentSkipListMap.java:1443)
> at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:419)
> at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:139)
> at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:127)
> at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:382)
> at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:278)
> at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:158)
> at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:175)
> at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:368)
> at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:80)
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:180)
> at java.io.DataInputStream.readFully(DataInputStream.java:152)
> at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:394)
> at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:368)
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:87)
> at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:261)
> ... 13 more
> After some debugging I found that in some serialized supercolumns column counter is less than the number of serialized columns. Difference was always 1 in corrupt commitlogs. This error always appears with supercolumns with more than one column, but there are properly serialized supercolumns also in commitlog.
> I have no clue yet why this error happens. I suspect it maybe a race condition.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2930) corrupt commitlog
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13195069#comment-13195069 ]
Jonathan Ellis commented on CASSANDRA-2930:
-------------------------------------------
bq. the issue is that the serialization code actually wrote out an incorrect supercolumn
We've had supercolumn races in the past (CASSANDRA-2675) and it's possible we have more. It's fairly hairy. IMO the right fix is to simplify them internally to composites (CASSANDRA-3237).
> corrupt commitlog
> -----------------
>
> Key: CASSANDRA-2930
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2930
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.1
> Environment: Linux, amd64.
> Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
> Reporter: ivan
> Assignee: Rick Branson
> Attachments: CommitLog-1310637513214.log
>
>
> We get "Exception encountered during startup" error while Cassandra starts.
> Error messages:
> INFO 13:56:28,736 Finished reading /var/lib/cassandra/commitlog/CommitLog-1310637513214.log
> ERROR 13:56:28,736 Exception encountered during startup.
> java.io.IOError: java.io.EOFException
> at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:265)
> at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:281)
> at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:236)
> at java.util.concurrent.ConcurrentSkipListMap.buildFromSorted(ConcurrentSkipListMap.java:1493)
> at java.util.concurrent.ConcurrentSkipListMap.<init>(ConcurrentSkipListMap.java:1443)
> at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:419)
> at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:139)
> at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:127)
> at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:382)
> at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:278)
> at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:158)
> at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:175)
> at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:368)
> at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:80)
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:180)
> at java.io.DataInputStream.readFully(DataInputStream.java:152)
> at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:394)
> at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:368)
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:87)
> at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:261)
> ... 13 more
> Exception encountered during startup.
> java.io.IOError: java.io.EOFException
> at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:265)
> at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:281)
> at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:236)
> at java.util.concurrent.ConcurrentSkipListMap.buildFromSorted(ConcurrentSkipListMap.java:1493)
> at java.util.concurrent.ConcurrentSkipListMap.<init>(ConcurrentSkipListMap.java:1443)
> at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:419)
> at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:139)
> at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:127)
> at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:382)
> at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:278)
> at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:158)
> at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:175)
> at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:368)
> at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:80)
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:180)
> at java.io.DataInputStream.readFully(DataInputStream.java:152)
> at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:394)
> at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:368)
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:87)
> at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:261)
> ... 13 more
> After some debugging I found that in some serialized supercolumns column counter is less than the number of serialized columns. Difference was always 1 in corrupt commitlogs. This error always appears with supercolumns with more than one column, but there are properly serialized supercolumns also in commitlog.
> I have no clue yet why this error happens. I suspect it maybe a race condition.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2930) corrupt commitlog
Posted by "Jackson Chung (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jackson Chung updated CASSANDRA-2930:
-------------------------------------
Labels: datastax_qa (was: )
> corrupt commitlog
> -----------------
>
> Key: CASSANDRA-2930
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2930
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.1
> Environment: Linux, amd64.
> Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
> Reporter: ivan
> Assignee: Rick Branson
> Labels: datastax_qa
> Attachments: CommitLog-1310637513214.log
>
>
> We get "Exception encountered during startup" error while Cassandra starts.
> Error messages:
> INFO 13:56:28,736 Finished reading /var/lib/cassandra/commitlog/CommitLog-1310637513214.log
> ERROR 13:56:28,736 Exception encountered during startup.
> java.io.IOError: java.io.EOFException
> at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:265)
> at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:281)
> at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:236)
> at java.util.concurrent.ConcurrentSkipListMap.buildFromSorted(ConcurrentSkipListMap.java:1493)
> at java.util.concurrent.ConcurrentSkipListMap.<init>(ConcurrentSkipListMap.java:1443)
> at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:419)
> at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:139)
> at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:127)
> at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:382)
> at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:278)
> at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:158)
> at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:175)
> at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:368)
> at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:80)
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:180)
> at java.io.DataInputStream.readFully(DataInputStream.java:152)
> at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:394)
> at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:368)
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:87)
> at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:261)
> ... 13 more
> Exception encountered during startup.
> java.io.IOError: java.io.EOFException
> at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:265)
> at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:281)
> at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:236)
> at java.util.concurrent.ConcurrentSkipListMap.buildFromSorted(ConcurrentSkipListMap.java:1493)
> at java.util.concurrent.ConcurrentSkipListMap.<init>(ConcurrentSkipListMap.java:1443)
> at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:419)
> at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:139)
> at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:127)
> at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:382)
> at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:278)
> at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:158)
> at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:175)
> at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:368)
> at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:80)
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:180)
> at java.io.DataInputStream.readFully(DataInputStream.java:152)
> at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:394)
> at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:368)
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:87)
> at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:261)
> ... 13 more
> After some debugging I found that in some serialized supercolumns column counter is less than the number of serialized columns. Difference was always 1 in corrupt commitlogs. This error always appears with supercolumns with more than one column, but there are properly serialized supercolumns also in commitlog.
> I have no clue yet why this error happens. I suspect it maybe a race condition.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2930) corrupt commitlog
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13069197#comment-13069197 ]
Jonathan Ellis commented on CASSANDRA-2930:
-------------------------------------------
Sounds like https://issues.apache.org/jira/browse/CASSANDRA-2675.
Are you sure you're actually running 0.8.1? We've had a lot of 0.8.0 installs that people thought were 0.8.1 due to incorrect packages being published.
grep "Cassandra version" /var/log/cassandra/system.log should verify.
> corrupt commitlog
> -----------------
>
> Key: CASSANDRA-2930
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2930
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.1
> Environment: Linux, amd64.
> Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
> Reporter: ivan
> Attachments: CommitLog-1310637513214.log
>
>
> We get "Exception encountered during startup" error while Cassandra starts.
> Error messages:
> INFO 13:56:28,736 Finished reading /var/lib/cassandra/commitlog/CommitLog-1310637513214.log
> ERROR 13:56:28,736 Exception encountered during startup.
> java.io.IOError: java.io.EOFException
> at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:265)
> at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:281)
> at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:236)
> at java.util.concurrent.ConcurrentSkipListMap.buildFromSorted(ConcurrentSkipListMap.java:1493)
> at java.util.concurrent.ConcurrentSkipListMap.<init>(ConcurrentSkipListMap.java:1443)
> at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:419)
> at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:139)
> at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:127)
> at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:382)
> at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:278)
> at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:158)
> at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:175)
> at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:368)
> at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:80)
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:180)
> at java.io.DataInputStream.readFully(DataInputStream.java:152)
> at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:394)
> at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:368)
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:87)
> at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:261)
> ... 13 more
> Exception encountered during startup.
> java.io.IOError: java.io.EOFException
> at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:265)
> at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:281)
> at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:236)
> at java.util.concurrent.ConcurrentSkipListMap.buildFromSorted(ConcurrentSkipListMap.java:1493)
> at java.util.concurrent.ConcurrentSkipListMap.<init>(ConcurrentSkipListMap.java:1443)
> at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:419)
> at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:139)
> at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:127)
> at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:382)
> at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:278)
> at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:158)
> at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:175)
> at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:368)
> at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:80)
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:180)
> at java.io.DataInputStream.readFully(DataInputStream.java:152)
> at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:394)
> at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:368)
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:87)
> at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:261)
> ... 13 more
> After some debugging I found that in some serialized supercolumns column counter is less than the number of serialized columns. Difference was always 1 in corrupt commitlogs. This error always appears with supercolumns with more than one column, but there are properly serialized supercolumns also in commitlog.
> I have no clue yet why this error happens. I suspect it maybe a race condition.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CASSANDRA-2930) corrupt commitlog
Posted by "Jonathan Ellis (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis resolved CASSANDRA-2930.
---------------------------------------
Resolution: Cannot Reproduce
Fix Version/s: (was: 0.8.8)
Assignee: (was: Sylvain Lebresne)
Closing as can't reproduce until/unless someone can reproduce on a more recent C* version.
> corrupt commitlog
> -----------------
>
> Key: CASSANDRA-2930
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2930
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.1
> Environment: Linux, amd64.
> Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
> Reporter: ivan
> Attachments: CommitLog-1310637513214.log
>
>
> We get "Exception encountered during startup" error while Cassandra starts.
> Error messages:
> INFO 13:56:28,736 Finished reading /var/lib/cassandra/commitlog/CommitLog-1310637513214.log
> ERROR 13:56:28,736 Exception encountered during startup.
> java.io.IOError: java.io.EOFException
> at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:265)
> at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:281)
> at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:236)
> at java.util.concurrent.ConcurrentSkipListMap.buildFromSorted(ConcurrentSkipListMap.java:1493)
> at java.util.concurrent.ConcurrentSkipListMap.<init>(ConcurrentSkipListMap.java:1443)
> at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:419)
> at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:139)
> at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:127)
> at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:382)
> at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:278)
> at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:158)
> at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:175)
> at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:368)
> at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:80)
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:180)
> at java.io.DataInputStream.readFully(DataInputStream.java:152)
> at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:394)
> at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:368)
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:87)
> at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:261)
> ... 13 more
> Exception encountered during startup.
> java.io.IOError: java.io.EOFException
> at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:265)
> at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:281)
> at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:236)
> at java.util.concurrent.ConcurrentSkipListMap.buildFromSorted(ConcurrentSkipListMap.java:1493)
> at java.util.concurrent.ConcurrentSkipListMap.<init>(ConcurrentSkipListMap.java:1443)
> at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:419)
> at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:139)
> at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:127)
> at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:382)
> at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:278)
> at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:158)
> at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:175)
> at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:368)
> at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:80)
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:180)
> at java.io.DataInputStream.readFully(DataInputStream.java:152)
> at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:394)
> at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:368)
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:87)
> at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:261)
> ... 13 more
> After some debugging I found that in some serialized supercolumns column counter is less than the number of serialized columns. Difference was always 1 in corrupt commitlogs. This error always appears with supercolumns with more than one column, but there are properly serialized supercolumns also in commitlog.
> I have no clue yet why this error happens. I suspect it maybe a race condition.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2930) corrupt commitlog
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-2930:
--------------------------------------
Fix Version/s: 0.8.3
> corrupt commitlog
> -----------------
>
> Key: CASSANDRA-2930
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2930
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.1
> Environment: Linux, amd64.
> Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
> Reporter: ivan
> Assignee: Sylvain Lebresne
> Fix For: 0.8.3
>
> Attachments: CommitLog-1310637513214.log
>
>
> We get "Exception encountered during startup" error while Cassandra starts.
> Error messages:
> INFO 13:56:28,736 Finished reading /var/lib/cassandra/commitlog/CommitLog-1310637513214.log
> ERROR 13:56:28,736 Exception encountered during startup.
> java.io.IOError: java.io.EOFException
> at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:265)
> at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:281)
> at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:236)
> at java.util.concurrent.ConcurrentSkipListMap.buildFromSorted(ConcurrentSkipListMap.java:1493)
> at java.util.concurrent.ConcurrentSkipListMap.<init>(ConcurrentSkipListMap.java:1443)
> at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:419)
> at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:139)
> at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:127)
> at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:382)
> at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:278)
> at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:158)
> at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:175)
> at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:368)
> at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:80)
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:180)
> at java.io.DataInputStream.readFully(DataInputStream.java:152)
> at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:394)
> at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:368)
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:87)
> at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:261)
> ... 13 more
> Exception encountered during startup.
> java.io.IOError: java.io.EOFException
> at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:265)
> at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:281)
> at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:236)
> at java.util.concurrent.ConcurrentSkipListMap.buildFromSorted(ConcurrentSkipListMap.java:1493)
> at java.util.concurrent.ConcurrentSkipListMap.<init>(ConcurrentSkipListMap.java:1443)
> at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:419)
> at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:139)
> at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:127)
> at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:382)
> at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:278)
> at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:158)
> at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:175)
> at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:368)
> at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:80)
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:180)
> at java.io.DataInputStream.readFully(DataInputStream.java:152)
> at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:394)
> at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:368)
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:87)
> at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:261)
> ... 13 more
> After some debugging I found that in some serialized supercolumns column counter is less than the number of serialized columns. Difference was always 1 in corrupt commitlogs. This error always appears with supercolumns with more than one column, but there are properly serialized supercolumns also in commitlog.
> I have no clue yet why this error happens. I suspect it maybe a race condition.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2930) corrupt commitlog
Posted by "ivan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13069243#comment-13069243 ]
ivan commented on CASSANDRA-2930:
---------------------------------
Hi Jonathan,
we built our Cassandra package from git://git.apache.org/cassandra.git cassandra-0.8.1 branch.
As I see it's the same as official 0.8.1 Cassandra code.
grep "Cassandra version" /var/log/cassandra/system.log:
INFO [main] 2011-07-21 14:42:48,553 StorageService.java (line 378) Cassandra version: 0.8.1
I checked CASSANDRA-2675 report.
Patch 0002-Avoid-modifying-super-column-in-memtable-being-flush-v2.patch is in our code.
PAtch 0001-Don-t-remove-columns-from-super-columns-in-memtable.patch is not in our code, but as I see it's not in official package and trunk also.
This error happens rarely. I found it 2 to 3 times in a 128MB commitlog.
I suspect some race condition, but a RowMutation shouldn't change. (SuperColumn.java:371)
So I have no clue yet. It's welcomed any further test suggestion.
Regards,
ivan
> corrupt commitlog
> -----------------
>
> Key: CASSANDRA-2930
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2930
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.1
> Environment: Linux, amd64.
> Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
> Reporter: ivan
> Attachments: CommitLog-1310637513214.log
>
>
> We get "Exception encountered during startup" error while Cassandra starts.
> Error messages:
> INFO 13:56:28,736 Finished reading /var/lib/cassandra/commitlog/CommitLog-1310637513214.log
> ERROR 13:56:28,736 Exception encountered during startup.
> java.io.IOError: java.io.EOFException
> at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:265)
> at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:281)
> at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:236)
> at java.util.concurrent.ConcurrentSkipListMap.buildFromSorted(ConcurrentSkipListMap.java:1493)
> at java.util.concurrent.ConcurrentSkipListMap.<init>(ConcurrentSkipListMap.java:1443)
> at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:419)
> at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:139)
> at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:127)
> at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:382)
> at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:278)
> at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:158)
> at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:175)
> at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:368)
> at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:80)
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:180)
> at java.io.DataInputStream.readFully(DataInputStream.java:152)
> at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:394)
> at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:368)
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:87)
> at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:261)
> ... 13 more
> Exception encountered during startup.
> java.io.IOError: java.io.EOFException
> at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:265)
> at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:281)
> at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:236)
> at java.util.concurrent.ConcurrentSkipListMap.buildFromSorted(ConcurrentSkipListMap.java:1493)
> at java.util.concurrent.ConcurrentSkipListMap.<init>(ConcurrentSkipListMap.java:1443)
> at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:419)
> at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:139)
> at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:127)
> at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:382)
> at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:278)
> at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:158)
> at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:175)
> at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:368)
> at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:80)
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:180)
> at java.io.DataInputStream.readFully(DataInputStream.java:152)
> at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:394)
> at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:368)
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:87)
> at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:261)
> ... 13 more
> After some debugging I found that in some serialized supercolumns column counter is less than the number of serialized columns. Difference was always 1 in corrupt commitlogs. This error always appears with supercolumns with more than one column, but there are properly serialized supercolumns also in commitlog.
> I have no clue yet why this error happens. I suspect it maybe a race condition.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (CASSANDRA-2930) corrupt commitlog
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis reassigned CASSANDRA-2930:
-----------------------------------------
Assignee: Sylvain Lebresne
> corrupt commitlog
> -----------------
>
> Key: CASSANDRA-2930
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2930
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.1
> Environment: Linux, amd64.
> Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
> Reporter: ivan
> Assignee: Sylvain Lebresne
> Fix For: 0.8.3
>
> Attachments: CommitLog-1310637513214.log
>
>
> We get "Exception encountered during startup" error while Cassandra starts.
> Error messages:
> INFO 13:56:28,736 Finished reading /var/lib/cassandra/commitlog/CommitLog-1310637513214.log
> ERROR 13:56:28,736 Exception encountered during startup.
> java.io.IOError: java.io.EOFException
> at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:265)
> at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:281)
> at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:236)
> at java.util.concurrent.ConcurrentSkipListMap.buildFromSorted(ConcurrentSkipListMap.java:1493)
> at java.util.concurrent.ConcurrentSkipListMap.<init>(ConcurrentSkipListMap.java:1443)
> at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:419)
> at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:139)
> at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:127)
> at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:382)
> at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:278)
> at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:158)
> at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:175)
> at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:368)
> at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:80)
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:180)
> at java.io.DataInputStream.readFully(DataInputStream.java:152)
> at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:394)
> at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:368)
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:87)
> at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:261)
> ... 13 more
> Exception encountered during startup.
> java.io.IOError: java.io.EOFException
> at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:265)
> at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:281)
> at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:236)
> at java.util.concurrent.ConcurrentSkipListMap.buildFromSorted(ConcurrentSkipListMap.java:1493)
> at java.util.concurrent.ConcurrentSkipListMap.<init>(ConcurrentSkipListMap.java:1443)
> at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:419)
> at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:139)
> at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:127)
> at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:382)
> at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:278)
> at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:158)
> at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:175)
> at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:368)
> at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:80)
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:180)
> at java.io.DataInputStream.readFully(DataInputStream.java:152)
> at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:394)
> at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:368)
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:87)
> at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:261)
> ... 13 more
> After some debugging I found that in some serialized supercolumns column counter is less than the number of serialized columns. Difference was always 1 in corrupt commitlogs. This error always appears with supercolumns with more than one column, but there are properly serialized supercolumns also in commitlog.
> I have no clue yet why this error happens. I suspect it maybe a race condition.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (CASSANDRA-2930) corrupt commitlog
Posted by "Jackson Chung (Reopened) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jackson Chung reopened CASSANDRA-2930:
--------------------------------------
Assignee: Rick Branson
we have seen this again on 0.8.7, same stacktrace and reasoning (col count off by 1).
may deserve a 2nd look to see how we could end up at this stage.
> corrupt commitlog
> -----------------
>
> Key: CASSANDRA-2930
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2930
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.1
> Environment: Linux, amd64.
> Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
> Reporter: ivan
> Assignee: Rick Branson
> Attachments: CommitLog-1310637513214.log
>
>
> We get "Exception encountered during startup" error while Cassandra starts.
> Error messages:
> INFO 13:56:28,736 Finished reading /var/lib/cassandra/commitlog/CommitLog-1310637513214.log
> ERROR 13:56:28,736 Exception encountered during startup.
> java.io.IOError: java.io.EOFException
> at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:265)
> at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:281)
> at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:236)
> at java.util.concurrent.ConcurrentSkipListMap.buildFromSorted(ConcurrentSkipListMap.java:1493)
> at java.util.concurrent.ConcurrentSkipListMap.<init>(ConcurrentSkipListMap.java:1443)
> at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:419)
> at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:139)
> at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:127)
> at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:382)
> at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:278)
> at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:158)
> at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:175)
> at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:368)
> at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:80)
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:180)
> at java.io.DataInputStream.readFully(DataInputStream.java:152)
> at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:394)
> at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:368)
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:87)
> at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:261)
> ... 13 more
> Exception encountered during startup.
> java.io.IOError: java.io.EOFException
> at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:265)
> at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:281)
> at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:236)
> at java.util.concurrent.ConcurrentSkipListMap.buildFromSorted(ConcurrentSkipListMap.java:1493)
> at java.util.concurrent.ConcurrentSkipListMap.<init>(ConcurrentSkipListMap.java:1443)
> at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:419)
> at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:139)
> at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:127)
> at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:382)
> at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:278)
> at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:158)
> at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:175)
> at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:368)
> at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:80)
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:180)
> at java.io.DataInputStream.readFully(DataInputStream.java:152)
> at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:394)
> at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:368)
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:87)
> at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:261)
> ... 13 more
> After some debugging I found that in some serialized supercolumns column counter is less than the number of serialized columns. Difference was always 1 in corrupt commitlogs. This error always appears with supercolumns with more than one column, but there are properly serialized supercolumns also in commitlog.
> I have no clue yet why this error happens. I suspect it maybe a race condition.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CASSANDRA-2930) corrupt commitlog
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis resolved CASSANDRA-2930.
---------------------------------------
Resolution: Cannot Reproduce
Assignee: (was: Rick Branson)
> corrupt commitlog
> -----------------
>
> Key: CASSANDRA-2930
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2930
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.1
> Environment: Linux, amd64.
> Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
> Reporter: ivan
> Labels: datastax_qa
> Attachments: CommitLog-1310637513214.log
>
>
> We get "Exception encountered during startup" error while Cassandra starts.
> Error messages:
> INFO 13:56:28,736 Finished reading /var/lib/cassandra/commitlog/CommitLog-1310637513214.log
> ERROR 13:56:28,736 Exception encountered during startup.
> java.io.IOError: java.io.EOFException
> at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:265)
> at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:281)
> at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:236)
> at java.util.concurrent.ConcurrentSkipListMap.buildFromSorted(ConcurrentSkipListMap.java:1493)
> at java.util.concurrent.ConcurrentSkipListMap.<init>(ConcurrentSkipListMap.java:1443)
> at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:419)
> at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:139)
> at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:127)
> at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:382)
> at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:278)
> at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:158)
> at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:175)
> at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:368)
> at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:80)
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:180)
> at java.io.DataInputStream.readFully(DataInputStream.java:152)
> at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:394)
> at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:368)
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:87)
> at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:261)
> ... 13 more
> Exception encountered during startup.
> java.io.IOError: java.io.EOFException
> at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:265)
> at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:281)
> at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:236)
> at java.util.concurrent.ConcurrentSkipListMap.buildFromSorted(ConcurrentSkipListMap.java:1493)
> at java.util.concurrent.ConcurrentSkipListMap.<init>(ConcurrentSkipListMap.java:1443)
> at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:419)
> at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:139)
> at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:127)
> at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:382)
> at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:278)
> at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:158)
> at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:175)
> at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:368)
> at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:80)
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:180)
> at java.io.DataInputStream.readFully(DataInputStream.java:152)
> at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:394)
> at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:368)
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:87)
> at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:261)
> ... 13 more
> After some debugging I found that in some serialized supercolumns column counter is less than the number of serialized columns. Difference was always 1 in corrupt commitlogs. This error always appears with supercolumns with more than one column, but there are properly serialized supercolumns also in commitlog.
> I have no clue yet why this error happens. I suspect it maybe a race condition.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2930) corrupt commitlog
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13216112#comment-13216112 ]
Jonathan Ellis commented on CASSANDRA-2930:
-------------------------------------------
If this has the same cause as CASSANDRA-3957, then switching to batch commitlog sync mode should workaround it.
> corrupt commitlog
> -----------------
>
> Key: CASSANDRA-2930
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2930
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.1
> Environment: Linux, amd64.
> Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
> Reporter: ivan
> Assignee: Rick Branson
> Labels: datastax_qa
> Attachments: CommitLog-1310637513214.log
>
>
> We get "Exception encountered during startup" error while Cassandra starts.
> Error messages:
> INFO 13:56:28,736 Finished reading /var/lib/cassandra/commitlog/CommitLog-1310637513214.log
> ERROR 13:56:28,736 Exception encountered during startup.
> java.io.IOError: java.io.EOFException
> at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:265)
> at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:281)
> at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:236)
> at java.util.concurrent.ConcurrentSkipListMap.buildFromSorted(ConcurrentSkipListMap.java:1493)
> at java.util.concurrent.ConcurrentSkipListMap.<init>(ConcurrentSkipListMap.java:1443)
> at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:419)
> at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:139)
> at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:127)
> at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:382)
> at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:278)
> at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:158)
> at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:175)
> at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:368)
> at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:80)
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:180)
> at java.io.DataInputStream.readFully(DataInputStream.java:152)
> at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:394)
> at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:368)
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:87)
> at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:261)
> ... 13 more
> Exception encountered during startup.
> java.io.IOError: java.io.EOFException
> at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:265)
> at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:281)
> at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:236)
> at java.util.concurrent.ConcurrentSkipListMap.buildFromSorted(ConcurrentSkipListMap.java:1493)
> at java.util.concurrent.ConcurrentSkipListMap.<init>(ConcurrentSkipListMap.java:1443)
> at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:419)
> at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:139)
> at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:127)
> at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:382)
> at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:278)
> at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:158)
> at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:175)
> at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:368)
> at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:80)
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:180)
> at java.io.DataInputStream.readFully(DataInputStream.java:152)
> at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:394)
> at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:368)
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:87)
> at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:261)
> ... 13 more
> After some debugging I found that in some serialized supercolumns column counter is less than the number of serialized columns. Difference was always 1 in corrupt commitlogs. This error always appears with supercolumns with more than one column, but there are properly serialized supercolumns also in commitlog.
> I have no clue yet why this error happens. I suspect it maybe a race condition.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira