You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Andy Stec (Created) (JIRA)" <ji...@apache.org> on 2011/10/25 23:20:32 UTC
[jira] [Created] (CASSANDRA-3402) Runtime exception thrown
periodically under load
Runtime exception thrown periodically under load
------------------------------------------------
Key: CASSANDRA-3402
URL: https://issues.apache.org/jira/browse/CASSANDRA-3402
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.0.0
Environment: Cassandra 1.0
Red Hat Enterprise Linux Server 6.0
Reporter: Andy Stec
Fix For: 1.0.1
The exception listed below is thrown periodically. We're using thrift interface for C++. Jonathan Ellis requested that we open a bug for this.
ERROR [ReadStage:1761] 2011-10-25 12:17:16,088 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[ReadStage:1761,5,main]
java.lang.RuntimeException: error reading 5 of 5
at org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:83)
at org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:40)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
at org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:107)
at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:145)
at org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:124)
at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:98)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
at org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:116)
at org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:144)
at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:225)
at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:61)
at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1297)
at org.apache.cassandra.db.ColumnFamilyStore.cacheRow(ColumnFamilyStore.java:1128)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1157)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1114)
at org.apache.cassandra.db.Table.getRow(Table.java:388)
at org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:58)
at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:62)
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.cassandra.db.ColumnSerializer$CorruptColumnException: invalid column name length 0
at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:89)
at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:82)
at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:72)
at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:36)
at org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:79)
... 24 more
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3402) Runtime exception thrown
periodically under load
Posted by "Sylvain Lebresne (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13135833#comment-13135833 ]
Sylvain Lebresne commented on CASSANDRA-3402:
---------------------------------------------
A few questions to help narrow it down:
* Do you have assertion enables on the node(s) on which you're getting this exception? You should be able to find if you do in the cassandra-env.sh (search for assertions, by default assertions are enabled so if you haven't made modifications it should be). I'm asking because there is an assertion on the write path that checks we don't write columns with a name of length 0, so if assertion are enabled and you haven't seen any assertion error in the logs, this would narrow it down quite a bit.
* You said you got this exception periodically. Does the exception always says "error reading 5 of 5", or are the numbers different sometimes?
* I don't suppose you've find a way to reproduce this easily on a blank database?
> Runtime exception thrown periodically under load
> ------------------------------------------------
>
> Key: CASSANDRA-3402
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3402
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Environment: Cassandra 1.0
> Red Hat Enterprise Linux Server 6.0
> Reporter: Andy Stec
> Assignee: Sylvain Lebresne
> Fix For: 1.0.1
>
>
> The exception listed below is thrown periodically. We're using thrift interface for C++. Jonathan Ellis requested that we open a bug for this.
> ERROR [ReadStage:1761] 2011-10-25 12:17:16,088 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[ReadStage:1761,5,main]
> java.lang.RuntimeException: error reading 5 of 5
> at org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:83)
> at org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:40)
> at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
> at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
> at org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:107)
> at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:145)
> at org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:124)
> at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:98)
> at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
> at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
> at org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:116)
> at org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:144)
> at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:225)
> at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:61)
> at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1297)
> at org.apache.cassandra.db.ColumnFamilyStore.cacheRow(ColumnFamilyStore.java:1128)
> at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1157)
> at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1114)
> at org.apache.cassandra.db.Table.getRow(Table.java:388)
> at org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:58)
> at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:62)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: org.apache.cassandra.db.ColumnSerializer$CorruptColumnException: invalid column name length 0
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:89)
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:82)
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:72)
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:36)
> at org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:79)
> ... 24 more
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3402) Runtime exception thrown
periodically under load
Posted by "Andy Stec (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andy Stec updated CASSANDRA-3402:
---------------------------------
Attachment: system.log.gz
Hi Sylvain,
Looks like assertions are enabled:
# enable assertions. disabling this in production will give a modest
# performance benefit (around 5%).
JVM_OPTS="$JVM_OPTS -ea"
The exception always says "error reading 5 of 5". We don't have a way of
reproducing the problem on a blank database. The exception is thrown after
load runs for several hours. I attached the full system.log.
Thanks,
Andy
On Wed, Oct 26, 2011 at 4:47 AM, Sylvain Lebresne (Commented) (JIRA) <
> Runtime exception thrown periodically under load
> ------------------------------------------------
>
> Key: CASSANDRA-3402
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3402
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Environment: Cassandra 1.0
> Red Hat Enterprise Linux Server 6.0
> Reporter: Andy Stec
> Assignee: Sylvain Lebresne
> Fix For: 1.0.1
>
> Attachments: system.log.gz
>
>
> The exception listed below is thrown periodically. We're using thrift interface for C++. Jonathan Ellis requested that we open a bug for this.
> ERROR [ReadStage:1761] 2011-10-25 12:17:16,088 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[ReadStage:1761,5,main]
> java.lang.RuntimeException: error reading 5 of 5
> at org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:83)
> at org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:40)
> at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
> at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
> at org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:107)
> at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:145)
> at org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:124)
> at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:98)
> at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
> at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
> at org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:116)
> at org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:144)
> at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:225)
> at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:61)
> at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1297)
> at org.apache.cassandra.db.ColumnFamilyStore.cacheRow(ColumnFamilyStore.java:1128)
> at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1157)
> at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1114)
> at org.apache.cassandra.db.Table.getRow(Table.java:388)
> at org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:58)
> at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:62)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: org.apache.cassandra.db.ColumnSerializer$CorruptColumnException: invalid column name length 0
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:89)
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:82)
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:72)
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:36)
> at org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:79)
> ... 24 more
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (CASSANDRA-3402) Runtime exception thrown
periodically under load
Posted by "Jonathan Ellis (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis reassigned CASSANDRA-3402:
-----------------------------------------
Assignee: Sylvain Lebresne
> Runtime exception thrown periodically under load
> ------------------------------------------------
>
> Key: CASSANDRA-3402
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3402
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Environment: Cassandra 1.0
> Red Hat Enterprise Linux Server 6.0
> Reporter: Andy Stec
> Assignee: Sylvain Lebresne
> Fix For: 1.0.1
>
>
> The exception listed below is thrown periodically. We're using thrift interface for C++. Jonathan Ellis requested that we open a bug for this.
> ERROR [ReadStage:1761] 2011-10-25 12:17:16,088 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[ReadStage:1761,5,main]
> java.lang.RuntimeException: error reading 5 of 5
> at org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:83)
> at org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:40)
> at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
> at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
> at org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:107)
> at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:145)
> at org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:124)
> at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:98)
> at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
> at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
> at org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:116)
> at org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:144)
> at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:225)
> at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:61)
> at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1297)
> at org.apache.cassandra.db.ColumnFamilyStore.cacheRow(ColumnFamilyStore.java:1128)
> at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1157)
> at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1114)
> at org.apache.cassandra.db.Table.getRow(Table.java:388)
> at org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:58)
> at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:62)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: org.apache.cassandra.db.ColumnSerializer$CorruptColumnException: invalid column name length 0
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:89)
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:82)
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:72)
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:36)
> at org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:79)
> ... 24 more
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3402) Runtime exception thrown
periodically under load
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13175060#comment-13175060 ]
Jonathan Ellis commented on CASSANDRA-3402:
-------------------------------------------
Andy, have you been able to test a newer Cassandra release?
> Runtime exception thrown periodically under load
> ------------------------------------------------
>
> Key: CASSANDRA-3402
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3402
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Environment: Cassandra 1.0
> Red Hat Enterprise Linux Server 6.0
> Reporter: Andy Stec
> Assignee: Sylvain Lebresne
> Fix For: 1.0.7
>
> Attachments: system.log.gz
>
>
> The exception listed below is thrown periodically. We're using thrift interface for C++. Jonathan Ellis requested that we open a bug for this.
> ERROR [ReadStage:1761] 2011-10-25 12:17:16,088 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[ReadStage:1761,5,main]
> java.lang.RuntimeException: error reading 5 of 5
> at org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:83)
> at org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:40)
> at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
> at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
> at org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:107)
> at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:145)
> at org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:124)
> at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:98)
> at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
> at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
> at org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:116)
> at org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:144)
> at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:225)
> at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:61)
> at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1297)
> at org.apache.cassandra.db.ColumnFamilyStore.cacheRow(ColumnFamilyStore.java:1128)
> at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1157)
> at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1114)
> at org.apache.cassandra.db.Table.getRow(Table.java:388)
> at org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:58)
> at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:62)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: org.apache.cassandra.db.ColumnSerializer$CorruptColumnException: invalid column name length 0
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:89)
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:82)
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:72)
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:36)
> at org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:79)
> ... 24 more
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CASSANDRA-3402) Runtime exception thrown
periodically under load
Posted by "Jonathan Ellis (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis resolved CASSANDRA-3402.
---------------------------------------
Resolution: Cannot Reproduce
Fix Version/s: (was: 1.0.9)
Assignee: (was: Sylvain Lebresne)
I haven't seen anyone else hit this either. I'm going to guess it was fixed in one of the 1.0 maintenance releases. Please feel free to re-open if you hit this on 1.0.8+.
> Runtime exception thrown periodically under load
> ------------------------------------------------
>
> Key: CASSANDRA-3402
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3402
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Environment: Cassandra 1.0
> Red Hat Enterprise Linux Server 6.0
> Reporter: Andy Stec
> Attachments: system.log.gz
>
>
> The exception listed below is thrown periodically. We're using thrift interface for C++. Jonathan Ellis requested that we open a bug for this.
> ERROR [ReadStage:1761] 2011-10-25 12:17:16,088 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[ReadStage:1761,5,main]
> java.lang.RuntimeException: error reading 5 of 5
> at org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:83)
> at org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:40)
> at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
> at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
> at org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:107)
> at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:145)
> at org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:124)
> at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:98)
> at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
> at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
> at org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:116)
> at org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:144)
> at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:225)
> at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:61)
> at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1297)
> at org.apache.cassandra.db.ColumnFamilyStore.cacheRow(ColumnFamilyStore.java:1128)
> at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1157)
> at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1114)
> at org.apache.cassandra.db.Table.getRow(Table.java:388)
> at org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:58)
> at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:62)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: org.apache.cassandra.db.ColumnSerializer$CorruptColumnException: invalid column name length 0
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:89)
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:82)
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:72)
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:36)
> at org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:79)
> ... 24 more
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3402) Runtime exception thrown
periodically under load
Posted by "Sylvain Lebresne (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13156756#comment-13156756 ]
Sylvain Lebresne commented on CASSANDRA-3402:
---------------------------------------------
Hi Andy,
It would be interesting to see if you can reproduce after an upgrade to 1.0.3. If so, the exception (and log) should give a bit more info hopefully (we've added details). Also, 1.0.3 now include a sha1 of each sstable, which would help ruling out an external (hardware) corruption.
> Runtime exception thrown periodically under load
> ------------------------------------------------
>
> Key: CASSANDRA-3402
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3402
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Environment: Cassandra 1.0
> Red Hat Enterprise Linux Server 6.0
> Reporter: Andy Stec
> Assignee: Sylvain Lebresne
> Fix For: 1.0.4
>
> Attachments: system.log.gz
>
>
> The exception listed below is thrown periodically. We're using thrift interface for C++. Jonathan Ellis requested that we open a bug for this.
> ERROR [ReadStage:1761] 2011-10-25 12:17:16,088 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[ReadStage:1761,5,main]
> java.lang.RuntimeException: error reading 5 of 5
> at org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:83)
> at org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:40)
> at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
> at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
> at org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:107)
> at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:145)
> at org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:124)
> at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:98)
> at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
> at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
> at org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:116)
> at org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:144)
> at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:225)
> at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:61)
> at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1297)
> at org.apache.cassandra.db.ColumnFamilyStore.cacheRow(ColumnFamilyStore.java:1128)
> at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1157)
> at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1114)
> at org.apache.cassandra.db.Table.getRow(Table.java:388)
> at org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:58)
> at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:62)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: org.apache.cassandra.db.ColumnSerializer$CorruptColumnException: invalid column name length 0
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:89)
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:82)
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:72)
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:36)
> at org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:79)
> ... 24 more
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira