You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Alex Petrov (Jira)" <ji...@apache.org> on 2020/02/03 16:00:05 UTC
[jira] [Commented] (CASSANDRA-15529)
AbstractLocalAwareExecutorService.java exceptions after upgrade from 2.1.16
to 3.11.4
[ https://issues.apache.org/jira/browse/CASSANDRA-15529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17029059#comment-17029059 ]
Alex Petrov commented on CASSANDRA-15529:
-----------------------------------------
[~Pooja_nair] is it possible to get a snapshot (possibly, anonymized if it's produdciton data) of what sstable dump would look like? This looks like it might be of a concern.
It would be also good to know what kind of insert/update/delete query patterns you roughly have for that schema, so I could try to reproduce locally.
> AbstractLocalAwareExecutorService.java exceptions after upgrade from 2.1.16 to 3.11.4
> -------------------------------------------------------------------------------------
>
> Key: CASSANDRA-15529
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15529
> Project: Cassandra
> Issue Type: Bug
> Components: Cluster/Schema
> Reporter: Pooja Nair
> Priority: Urgent
> Labels: 2.1.16, 3.11.4
>
> Hello Team,
> We have cluster running on cassandra 3.11.4
> Following is the table schema of the tables that is being used in our system.
> {code:java}
> cqlsh> desc KEYSPACE "SAL"
>
> CREATE KEYSPACE "SAL" WITH replication = {'class': 'NetworkTopologyStrategy', 'DC_EAST': '3', 'DC_WEST': '3'} AND durable_writes = true;
>
> CREATE TABLE "SAL".sal_purge (
> key text,
> column1 text,
> column2 text,
> value text,
> PRIMARY KEY (key, column1, column2)
> ) WITH COMPACT STORAGE
> AND CLUSTERING ORDER BY (column1 ASC, column2 ASC)
> AND bloom_filter_fp_chance = 0.1
> AND caching = '{"keys":"NONE", "rows_per_partition":"NONE"}'
> AND comment = 'Holds items to be removed as [shardid][salid][timestamp]. The table records SALIDs to be deleted along with their deletion times (which may be modified)'
> AND compaction = {'class': 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
> AND compression = {'chunk_length_kb': '64', 'sstable_compression': 'org.apache.cassandra.io.compress.SnappyCompressor'}
> AND dclocal_read_repair_chance = 0.0
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.1
> AND speculative_retry = '99.0PERCENTILE';
>
> CREATE TABLE "SAL".sal_ref (
> key text,
> column1 text,
> column2 text,
> value text,
> PRIMARY KEY (key, column1, column2)
> ) WITH COMPACT STORAGE
> AND CLUSTERING ORDER BY (column1 ASC, column2 ASC)
> AND bloom_filter_fp_chance = 0.025
> AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}'
> AND comment = 'Holds owner references to content as [salid][lcid/opid]'
> AND compaction = {'sstable_size_in_mb': '180', 'class': 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
> AND compression = {'chunk_length_kb': '64', 'sstable_compression': 'org.apache.cassandra.io.compress.SnappyCompressor'}
> AND dclocal_read_repair_chance = 0.0
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.0
> AND speculative_retry = '99.0PERCENTILE';
> {code}
> Things to note:
> # The column2 is always passed a null value during insertion
> # column2 is a part of primary key
> # Range select and Range delete is done through our app.
> Iniatally the cluster was on casssandra version 2.1.16 and have been recently upgraded to 3.11.4 post the upgrade, we see that the nodes are going down, and log the below exceptions during startup and even after node is up. This one node is causing the whole cluster to behave improperly.
> {code:java}
> WARN [Native-Transport-Requests-47] 2020-01-29 13:49:05,190 AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread Thread[Native-Transport-Requests-47,5,main]: {} java.lang.RuntimeException: java.lang.IllegalStateException: UnfilteredRowIterator for SAL.sal_purge has an open RT bound as its last item at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2588) ~[apache-cassandra-3.11.4.jar:3.11.4] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0-internal] at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162) [apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:134) [apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.concurrent.SEPExecutor.maybeExecuteImmediately(SEPExecutor.java:194) [apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.service.AbstractReadExecutor.makeRequests(AbstractReadExecutor.java:117) [apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.service.AbstractReadExecutor.makeDataRequests(AbstractReadExecutor.java:85) [apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.service.AbstractReadExecutor$SpeculatingReadExecutor.executeAsync(AbstractReadExecutor.java:271) [apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.service.StorageProxy$SinglePartitionReadLifecycle.doInitialQueries(StorageProxy.java:1778) [apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1731) [apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1671) [apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1586) [apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.db.SinglePartitionReadCommand$Group.execute(SinglePartitionReadCommand.java:1209) [apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:315) [apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:285) [apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:117) [apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:225) [apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:532) [apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:509) [apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:146) [apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:566) [apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:410) [apache-cassandra-3.11.4.jar:3.11.4] at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357) [netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35) [netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:348) [netty-all-4.0.44.Final.jar:4.0.44.Final] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0-internal] at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162) [apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:114) [apache-cassandra-3.11.4.jar:3.11.4] at java.lang.Thread.run(Thread.java:748) [na:1.8.0-internal] Caused by: java.lang.IllegalStateException: UnfilteredRowIterator for SAL.sal_purge has an open RT bound as its last item at org.apache.cassandra.db.transform.RTBoundCloser$RowsTransformation.moreContents(RTBoundCloser.java:109) ~[apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.db.transform.RTBoundCloser$RowsTransformation.moreContents(RTBoundCloser.java:63) ~[apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.db.transform.BaseIterator.tryGetMoreContents(BaseIterator.java:121) ~[apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.db.transform.BaseIterator.hasMoreContents(BaseIterator.java:111) ~[apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:159) ~[apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:136) ~[apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:92) ~[apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:79) ~[apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:308) ~[apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:187) ~[apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:180) ~[apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:176) ~[apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:76) ~[apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:353) ~[apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1876) ~[apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2584) ~[apache-cassandra-3.11.4.jar:3.11.4] ... 29 common frames omitted Suppressed: java.lang.IllegalStateException: PROCESSED UnfilteredRowIterator for SAL.sal_purge has an illegal RT bounds sequence: expected all RTs to be closed, but the last one is open at org.apache.cassandra.db.transform.RTBoundValidator$RowsTransformation.ise(RTBoundValidator.java:120) ~[apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.db.transform.RTBoundValidator$RowsTransformation.onPartitionClose(RTBoundValidator.java:113) ~[apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.db.transform.BaseRows.runOnClose(BaseRows.java:91) ~[apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.db.transform.BaseIterator.close(BaseIterator.java:86) ~[apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:309) ~[apache-cassandra-3.11.4.jar:3.11.4] ... 36 common frames omitted
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org