You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Andrew Klopper (JIRA)" <ji...@apache.org> on 2016/07/24 15:00:22 UTC
[jira] [Commented] (CASSANDRA-11726) IndexOutOfBoundsException when
selecting (distinct) row ids from counter table.
[ https://issues.apache.org/jira/browse/CASSANDRA-11726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15391081#comment-15391081 ]
Andrew Klopper commented on CASSANDRA-11726:
--------------------------------------------
We are also experiencing this issue since upgrading to Cassandra 3.7 from 2.x.
We make use of a lot of counter tables, and on 3.7 queries like "SELECT entity_id FROM entity_stats" fail immediately while "SELECT * FROM entity_stats" works fine.
We are seeing this on four different clusters, all separately upgraded. Dumping the data and recreating the keyspaces from scratch does not help.
The error reported by cqlsh is:
{code}ReadFailure: code=1300 [Replica(s) failed to execute read] message="Operation failed - received 0 responses and 1 failures" info={'failures': 1, 'received_responses': 0, 'required_responses': 1, 'consistency': 'ONE'}{code}
The error in the logs is:
{code}WARN [SharedPool-Worker-3] 2016-07-24 12:46:13,948 AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread Thread[SharedPool-Worker-3,5,main]: {}
java.lang.RuntimeException: java.lang.IndexOutOfBoundsException
at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2470) ~[apache-cassandra-3.7.jar:3.7]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_92]
at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136) [apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-3.7.jar:3.7]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_92]
Caused by: java.lang.IndexOutOfBoundsException: null
at java.nio.Buffer.checkIndex(Buffer.java:546) ~[na:1.8.0_92]
at java.nio.HeapByteBuffer.getShort(HeapByteBuffer.java:314) ~[na:1.8.0_92]
at org.apache.cassandra.db.context.CounterContext.headerLength(CounterContext.java:141) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.db.context.CounterContext.access$100(CounterContext.java:76) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.db.context.CounterContext$ContextState.<init>(CounterContext.java:758) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.db.context.CounterContext$ContextState.wrap(CounterContext.java:765) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.db.context.CounterContext.merge(CounterContext.java:271) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.db.Conflicts.mergeCounterValues(Conflicts.java:76) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.db.rows.Cells.reconcile(Cells.java:143) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.db.rows.Row$Merger$ColumnDataReducer.getReduced(Row.java:614) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.db.rows.Row$Merger$ColumnDataReducer.getReduced(Row.java:572) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:217) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:156) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.db.rows.Row$Merger.merge(Row.java:549) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator$MergeReducer.getReduced(UnfilteredRowIterators.java:473) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator$MergeReducer.getReduced(UnfilteredRowIterators.java:437) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:217) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:156) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:419) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:279) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:100) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:112) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.db.transform.UnfilteredRows.isEmpty(UnfilteredRows.java:38) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:64) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:24) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:76) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:289) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:134) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:127) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:123) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:65) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:292) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1799) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2466) ~[apache-cassandra-3.7.jar:3.7]
... 5 common frames omitted{code}
> IndexOutOfBoundsException when selecting (distinct) row ids from counter table.
> -------------------------------------------------------------------------------
>
> Key: CASSANDRA-11726
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11726
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: C* 3.5, cluster of 4 nodes.
> Reporter: Jaroslav Kamenik
>
> I have simple table containing counters:
> {code}
> CREATE TABLE tablename (
> object_id ascii,
> counter_id ascii,
> count counter,
> PRIMARY KEY (object_id, counter_id)
> ) WITH CLUSTERING ORDER BY (counter_id ASC)
> AND bloom_filter_fp_chance = 0.01
> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
> AND comment = ''
> AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
> AND compression = {'enabled': 'false'}
> AND crc_check_chance = 1.0
> AND dclocal_read_repair_chance = 0.1
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.0
> AND speculative_retry = '99PERCENTILE';
> {code}
> Counters are often inc/decreased, whole rows are queried, deleted sometimes.
> After some time I tried to query all object_ids, but it failed with:
> {code}
> cqlsh:woc> consistency quorum;
> cqlsh:woc> select object_id from tablename;
> ServerError: <ErrorMessage code=0000 [Server error] message="java.lang.IndexOutOfBoundsException">
> {code}
> select * from ..., select where .., updates works well..
> With consistency one it works sometimes, so it seems something is broken at one server, but I tried to repair table there and it did not help.
> Whole exception from server log:
> {code}
> java.lang.IndexOutOfBoundsException: null
> at java.nio.Buffer.checkIndex(Buffer.java:546) ~[na:1.8.0_73]
> at java.nio.HeapByteBuffer.getShort(HeapByteBuffer.java:314) ~[na:1.8.0_73]
> at org.apache.cassandra.db.context.CounterContext.headerLength(CounterContext.java:141) ~[apache-cassandra-3.5.jar:3.5]
> at org.apache.cassandra.db.context.CounterContext.access$100(CounterContext.java:76) ~[apache-cassandra-3.5.jar:3.5]
> at org.apache.cassandra.db.context.CounterContext$ContextState.<init>(CounterContext.java:758) ~[apache-cassandra-3.5.jar:3.5]
> at org.apache.cassandra.db.context.CounterContext$ContextState.wrap(CounterContext.java:765) ~[apache-cassandra-3.5.jar:3.5]
> at org.apache.cassandra.db.context.CounterContext.merge(CounterContext.java:271) ~[apache-cassandra-3.5.jar:3.5]
> at org.apache.cassandra.db.Conflicts.mergeCounterValues(Conflicts.java:76) ~[apache-cassandra-3.5.jar:3.5]
> at org.apache.cassandra.db.rows.Cells.reconcile(Cells.java:143) ~[apache-cassandra-3.5.jar:3.5]
> at org.apache.cassandra.db.rows.Row$Merger$ColumnDataReducer.getReduced(Row.java:591) ~[apache-cassandra-3.5.jar:3.5]
> at org.apache.cassandra.db.rows.Row$Merger$ColumnDataReducer.getReduced(Row.java:549) ~[apache-cassandra-3.5.jar:3.5]
> at org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:217) ~[apache-cassandra-3.5.jar:3.5]
> at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:156) ~[apache-cassandra-3.5.jar:3.5]
> at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.5.jar:3.5]
> at org.apache.cassandra.db.rows.Row$Merger.merge(Row.java:526) ~[apache-cassandra-3.5.jar:3.5]
> at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator$MergeReducer.getReduced(UnfilteredRowIterators.java:473) ~[apache-cassandra-3.5.jar:3.5]
> at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator$MergeReducer.getReduced(UnfilteredRowIterators.java:437) ~[apache-cassandra-3.5.jar:3.5]
> at org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:217) ~[apache-cassandra-3.5.jar:3.5]
> at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:156) ~[apache-cassandra-3.5.jar:3.5]
> at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.5.jar:3.5]
> at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:419) ~[apache-cassandra-3.5.jar:3.5]
> at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:279) ~[apache-cassandra-3.5.jar:3.5]
> at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.5.jar:3.5]
> at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:112) ~[apache-cassandra-3.5.jar:3.5]
> at org.apache.cassandra.db.transform.FilteredRows.isEmpty(FilteredRows.java:30) ~[apache-cassandra-3.5.jar:3.5]
> at org.apache.cassandra.db.transform.Filter.closeIfEmpty(Filter.java:49) ~[apache-cassandra-3.5.jar:3.5]
> at org.apache.cassandra.db.transform.Filter.applyToPartition(Filter.java:22) ~[apache-cassandra-3.5.jar:3.5]
> at org.apache.cassandra.db.transform.Filter.applyToPartition(Filter.java:6) ~[apache-cassandra-3.5.jar:3.5]
> at org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:76) ~[apache-cassandra-3.5.jar:3.5]
> at org.apache.cassandra.service.StorageProxy$SingleRangeResponse.computeNext(StorageProxy.java:2000) ~[apache-cassandra-3.5.jar:3.5]
> at org.apache.cassandra.service.StorageProxy$SingleRangeResponse.computeNext(StorageProxy.java:1972) ~[apache-cassandra-3.5.jar:3.5]
> at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.5.jar:3.5]
> at org.apache.cassandra.db.transform.BaseIterator.tryGetMoreContents(BaseIterator.java:113) ~[apache-cassandra-3.5.jar:3.5]
> at org.apache.cassandra.db.transform.BaseIterator.hasMoreContents(BaseIterator.java:81) ~[apache-cassandra-3.5.jar:3.5]
> at org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:85) ~[apache-cassandra-3.5.jar:3.5]
> at org.apache.cassandra.service.StorageProxy$RangeCommandIterator.computeNext(StorageProxy.java:2041) ~[apache-cassandra-3.5.jar:3.5]
> at org.apache.cassandra.service.StorageProxy$RangeCommandIterator.computeNext(StorageProxy.java:2010) ~[apache-cassandra-3.5.jar:3.5]
> at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.5.jar:3.5]
> at org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:72) ~[apache-cassandra-3.5.jar:3.5]
> at org.apache.cassandra.cql3.statements.SelectStatement.process(SelectStatement.java:655) ~[apache-cassandra-3.5.jar:3.5]
> at org.apache.cassandra.cql3.statements.SelectStatement.processResults(SelectStatement.java:382) ~[apache-cassandra-3.5.jar:3.5]
> at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:335) ~[apache-cassandra-3.5.jar:3.5]
> at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:209) ~[apache-cassandra-3.5.jar:3.5]
> at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:76) ~[apache-cassandra-3.5.jar:3.5]
> at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:206) ~[apache-cassandra-3.5.jar:3.5]
> at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:237) ~[apache-cassandra-3.5.jar:3.5]
> at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:222) ~[apache-cassandra-3.5.jar:3.5]
> at org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:115) ~[apache-cassandra-3.5.jar:3.5]
> at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:507) [apache-cassandra-3.5.jar:3.5]
> at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:401) [apache-cassandra-3.5.jar:3.5]
> at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.23.Final.jar:4.0.23.Final]
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) [netty-all-4.0.23.Final.jar:4.0.23.Final]
> at io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32) [netty-all-4.0.23.Final.jar:4.0.23.Final]
> at io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324) [netty-all-4.0.23.Final.jar:4.0.23.Final]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_73]
> at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) [apache-cassandra-3.5.jar:3.5]
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-3.5.jar:3.5]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_73]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)