You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "David Capwell (Jira)" <ji...@apache.org> on 2020/02/24 21:17:00 UTC
[jira] [Commented] (CASSANDRA-15594) Uncaught exception on thread
Thread[MutationStage-1,5,main]
[ https://issues.apache.org/jira/browse/CASSANDRA-15594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17043867#comment-17043867 ]
David Capwell commented on CASSANDRA-15594:
-------------------------------------------
Based off your stack trace this is what I see
{code}
public NativeClustering(NativeAllocator allocator, OpOrder.Group writeOp, Clustering clustering)
{
int count = clustering.size();
int metadataSize = (count * 2) + 4;
int dataSize = clustering.dataSize();
int bitmapSize = ((count + 7) >>> 3);
assert count < 64 << 10;
assert dataSize < 64 << 10;
{code}
So "org.apache.cassandra.db.NativeClustering.<init>(NativeClustering.java:45) ~[apache-cassandra-3.11.4.jar:3.11.4]" would most likely be the dataSize assert.
This then calls
{code}
public int dataSize()
{
int size = 0;
for (int i = 0; i < size(); i++)
{
ByteBuffer bb = get(i);
size += bb == null ? 0 : bb.remaining();
}
return size;
}
{code}
Given this, the sum of bytes for column2 and column3 is greater than 65,536 bytes (64 kibibytes) so doesn't fit in a short; which is the current limitation.
[~ar0], if you look at your examples, do you see that the two columns have values which may be larger than that limit?
I was able to replicate this locally by setting memtable_allocation_type to be offheap_objects; this isn't the case with other allocations
> Uncaught exception on thread Thread[MutationStage-1,5,main]
> -----------------------------------------------------------
>
> Key: CASSANDRA-15594
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15594
> Project: Cassandra
> Issue Type: Bug
> Reporter: Aleksandr
> Priority: Normal
> Labels: 3.11.4
>
> Hi.
> We have a cluster of 3 machines running Cassandra 3.11.4 for several months, cpp-driver 2.12.0 is used to communicate with Cassandra.
> We have 13 tables in total.
> Recently we've noticed that one particular INSERT operation fails with the error "Write failure: Operation failed - received 0 responses and 2 failures" - we retry twice the same query with timeouts 1 and 5.1 seconds. The query is:
> {code:java}
> INSERT INTO keyspace1.table1 (column1, column2, column3, column4, column5) VALUES (?, ?, ?, ?, ?) IF NOT EXISTS{code}
> Every time this particular INSERT happens we see the error in the log:
> {code:java}
> WARN [MutationStage-1] 2020-02-21 12:13:46,435 AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread Thread[MutationStage-1,5,main]: {}
> java.lang.AssertionError: null
> at org.apache.cassandra.db.NativeClustering.<init>(NativeClustering.java:45) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.utils.memory.NativeAllocator$CloningBTreeRowBuilder.newRow(NativeAllocator.java:80) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.db.rows.Rows.copy(Rows.java:43) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.db.partitions.AtomicBTreePartition$RowUpdater.apply(AtomicBTreePartition.java:332) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.db.partitions.AtomicBTreePartition$RowUpdater.apply(AtomicBTreePartition.java:295) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.utils.btree.NodeBuilder.addNewKey(NodeBuilder.java:323) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.utils.btree.NodeBuilder.update(NodeBuilder.java:184) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.utils.btree.TreeBuilder.update(TreeBuilder.java:95) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.utils.btree.BTree.update(BTree.java:182) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.db.partitions.AtomicBTreePartition.addAllWithSizeDelta(AtomicBTreePartition.java:156) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.db.Memtable.put(Memtable.java:282) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:1352) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:626) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:470) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:450) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.service.paxos.PaxosState.commit(PaxosState.java:150) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.service.StorageProxy$6.runMayThrow(StorageProxy.java:571) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2637) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_232]
> at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:134) [apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:114) [apache-cassandra-3.11.4.jar:3.11.4]
> at java.lang.Thread.run(Thread.java:748) [na:1.8.0_232]
> WARN [MutationStage-4] 2020-02-21 12:13:46,449 AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread Thread[MutationStage-4,5,main]: {}
> java.lang.AssertionError: null
> at org.apache.cassandra.db.NativeClustering.<init>(NativeClustering.java:45) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.utils.memory.NativeAllocator$CloningBTreeRowBuilder.newRow(NativeAllocator.java:80) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.db.rows.Rows.copy(Rows.java:43) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.db.partitions.AtomicBTreePartition$RowUpdater.apply(AtomicBTreePartition.java:332) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.db.partitions.AtomicBTreePartition$RowUpdater.apply(AtomicBTreePartition.java:295) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.utils.btree.NodeBuilder.addNewKey(NodeBuilder.java:323) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.utils.btree.NodeBuilder.update(NodeBuilder.java:184) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.utils.btree.TreeBuilder.update(TreeBuilder.java:95) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.utils.btree.BTree.update(BTree.java:182) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.db.partitions.AtomicBTreePartition.addAllWithSizeDelta(AtomicBTreePartition.java:156) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.db.Memtable.put(Memtable.java:282) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:1352) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:626) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:470) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:450) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.service.paxos.PaxosState.commit(PaxosState.java:150) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.service.paxos.CommitVerbHandler.doVerb(CommitVerbHandler.java:33) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:66) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_232]
> at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:134) [apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:114) [apache-cassandra-3.11.4.jar:3.11.4]
> at java.lang.Thread.run(Thread.java:748) [na:1.8.0_232]
> WARN [MutationStage-5] 2020-02-21 12:13:46,461 AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread Thread[MutationStage-5,5,main]: {}
> java.lang.AssertionError: null
> at org.apache.cassandra.db.NativeClustering.<init>(NativeClustering.java:45) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.utils.memory.NativeAllocator$CloningBTreeRowBuilder.newRow(NativeAllocator.java:80) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.db.rows.Rows.copy(Rows.java:43) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.db.partitions.AtomicBTreePartition$RowUpdater.apply(AtomicBTreePartition.java:332) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.db.partitions.AtomicBTreePartition$RowUpdater.apply(AtomicBTreePartition.java:295) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.utils.btree.NodeBuilder.addNewKey(NodeBuilder.java:323) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.utils.btree.NodeBuilder.update(NodeBuilder.java:184) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.utils.btree.TreeBuilder.update(TreeBuilder.java:95) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.utils.btree.BTree.update(BTree.java:182) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.db.partitions.AtomicBTreePartition.addAllWithSizeDelta(AtomicBTreePartition.java:156) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.db.Memtable.put(Memtable.java:282) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:1352) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:626) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:470) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:450) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.service.paxos.PaxosState.commit(PaxosState.java:150) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.service.paxos.CommitVerbHandler.doVerb(CommitVerbHandler.java:33) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:66) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_232]
> at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:134) [apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:114) [apache-cassandra-3.11.4.jar:3.11.4]
> at java.lang.Thread.run(Thread.java:748) [na:1.8.0_232]{code}
>
> The following table schema is used for this table:
> {code:java}
> CREATE TABLE keyspace1.table1 (
> column1 text,
> column2 bigint,
> column3 text,
> column4 uuid,
> column5 bigint,
> PRIMARY KEY (column1, column2, column3)
> ) WITH CLUSTERING ORDER BY (column2 ASC, column3 ASC)
> AND bloom_filter_fp_chance = 0.01
> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
> AND comment = ''
> AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
> AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
> AND crc_check_chance = 1.0
> AND dclocal_read_repair_chance = 0.1
> AND default_time_to_live = 0
> AND gc_grace_seconds = 86400
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.0
> AND speculative_retry = '99PERCENTILE';{code}
>
> Two small additions:
> 1) We couldn't reproduce this issue on a different cluster with 3 machines - everything works fine. No issues when only machine is used either.
> 2) Only this one INSERT fails for this table. There are others similar INSERTS into the same table, but they don't fail.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org