[jira] [Comment Edited] (CASSANDRA-13973) IllegalArgumentException in upgradesstables compaction

["Jeff Jirsa (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-13973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16236109#comment-16236109 ] 

Jeff Jirsa edited comment on CASSANDRA-13973 at 11/2/17 4:52 PM:
-----------------------------------------------------------------

Thanks for the feedback [~slebresne]. I really appreciate you taking the time to respond. The risky feeling here is why I'm moving slow on this myself - it seems straightforward, but obviously the potential for unexpected surprises here is pretty high. 

Better messaging on that assert makes a lot of sense. Also good to see the second confirmation that changing {{column_index_size_in_kb}} is a good workaround, the drawback is that it's instance-wide, so if you have just a handful of wide rows (like this user, the histogram shows their 99% size is less than 1MB, but their max size is 394GB), you suffer a disk penalty on all keyspaces/tables/rows in order to not crash on the one bad row. 

[~dankinder] if you need to unblock yourself right now, changing {{column_index_size_in_kb}} on your instance to 256 (7G of index data needs to go under 2G in size, so multiplying factor of 4) PROBABLY works past this issue, but expect a bit more disk IO (particularly reads) after the change (+upgradesstables)






was (Author: jjirsa):
Thanks for the feedback [~slebresne]. I really appreciate you taking the time to respond. The risky feeling here is why I'm moving slow on this myself - it seems straightforward, but obviously the potential for unexpected surprises here is pretty high. 

Better messaging on that assert makes a lot of sense. Also good to see the second confirmation that changing {{column_index_size_in_kb}} is a good workaround, the drawback is that it's instance-wide, so if you have just a handful of wide rows (like this user, the histogram shows their 99% size is less than 1MB, but their max size is 394GB), you suffer a disk penalty on all keyspaces/tables/rows in order to not crash on the one bad row. 

[~dankinder] if you need to unblock yourself right now, changing {{column_index_size_in_kb}} on your instance to 256k (7G of index data needs to go under 2G in size, so multiplying factor of 4) PROBABLY works past this issue, but expect a bit more disk IO (particularly reads) after the change (+upgradesstables)





> IllegalArgumentException in upgradesstables compaction
> ------------------------------------------------------
>
>                 Key: CASSANDRA-13973
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-13973
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Compaction
>            Reporter: Dan Kinder
>            Assignee: Jeff Jirsa
>            Priority: Major
>             Fix For: 3.0.x, 3.11.x, 4.x
>
>
> After an upgrade from 2.2.6 to 3.0.15 (sstable version la to mc), when I try to run upgradesstables, most of them upgrade fine but I see the exception below on several nodes, and it doesn't complete.
> CASSANDRA-12717 looks similar but the stack trace is not the same, so I assumed it is not identical. The various nodes this happens on all give the same trace.
> Might be notable that this is an analytics cluster with some large partitions, in the GB size.
> {noformat}
> error: Out of range: 7316844981
> -- StackTrace --
> java.lang.IllegalArgumentException: Out of range: 7316844981
> at com.google.common.primitives.Ints.checkedCast(Ints.java:91)
> at org.apache.cassandra.db.RowIndexEntry$IndexedEntry.promotedSize(RowIndexEntry.java:329)
> at org.apache.cassandra.db.RowIndexEntry$Serializer.serialize(RowIndexEntry.java:133)
> at org.apache.cassandra.io.sstable.format.big.BigTableWriter$IndexWriter.append(BigTableWriter.java:409)
> at org.apache.cassandra.io.sstable.format.big.BigTableWriter.afterAppend(BigTableWriter.java:120)
> at org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:157)
> at org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:125)
> at org.apache.cassandra.db.compaction.writers.MaxSSTableSizeWriter.realAppend(MaxSSTableSizeWriter.java:88)
> at org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:109)
> at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:195)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:89)
> at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
> at org.apache.cassandra.db.compaction.CompactionManager$5.execute(CompactionManager.java:424)
> at org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:311)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org