You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Caleb Rackliffe (Jira)" <ji...@apache.org> on 2020/09/11 00:47:00 UTC
[jira] [Commented] (CASSANDRA-15164) Overflowed Partition Cell
Histograms Can Prevent Compactions from Executing
[ https://issues.apache.org/jira/browse/CASSANDRA-15164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17193934#comment-17193934 ]
Caleb Rackliffe commented on CASSANDRA-15164:
---------------------------------------------
Let's revisit the stack trace from CASSANDRA-15326, which is almost certainly the same here:
{noformat}
Exception in thread Thread[CompactionExecutor:113041,1,main]
java.lang.IllegalStateException: Unable to compute ceiling for max when histogram overflowed
at org.apache.cassandra.utils.EstimatedHistogram.rawMean(EstimatedHistogram.java:231) ~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.utils.EstimatedHistogram.mean(EstimatedHistogram.java:220) ~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.io.sstable.metadata.StatsMetadata.getEstimatedDroppableTombstoneRatio(StatsMetadata.java:115) ~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.io.sstable.format.SSTableReader.getEstimatedDroppableTombstoneRatio(SSTableReader.java:1926) ~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.worthDroppingTombstones(AbstractCompactionStrategy.java:424) ~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy.getNextBackgroundSSTables(SizeTieredCompactionStrategy.java:99) ~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy.getNextBackgroundTask(SizeTieredCompactionStrategy.java:183) ~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.db.compaction.CompactionStrategyManager.getNextBackgroundTask(CompactionStrategyManager.java:153) ~[apache-cassandra-3.11.4.jar:3.11.4]
{noformat}
All our compaction strategies, at some point, want to know what the ratio of droppable tombstones to cells is on average for the partitions in an SSTable. However, if we ever have more than about 1.9 billion cells in a partition, the {{EstimatedHistogram}} that tracks this will overflow. Then, when compaction attempts to get the mean number of cells per partition {{EstimatedHistogram}} throws an {{IllegalStateException}} that aborts the attempt at compaction. This can continue indefinitely.
In C* 4.0, full checksum validation for metadata components exists, but it's also possible that, in previous versions, the serialization/deserialization cycle for {{EstimatedHistogram}} could introduce corruption that breaks the mean calculation.
> Overflowed Partition Cell Histograms Can Prevent Compactions from Executing
> ---------------------------------------------------------------------------
>
> Key: CASSANDRA-15164
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15164
> Project: Cassandra
> Issue Type: Bug
> Components: CQL/Interpreter
> Reporter: Ankur Jha
> Assignee: Caleb Rackliffe
> Priority: Urgent
> Labels: compaction, partition
>
> Hi, we are running 6 node Cassandra cluster in production with 3 seed node but from last night one of our seed nodes is continuously throwing an error like this;-
> cassandra.protocol.ServerError: <Error from server: code=0000 [Server error] message="java.lang.IllegalStateException: Unable to compute ceiling for max when histogram overflowed">
> For a cluster to be up and running I Drained this node.
> Can somebody help me out with this?
>
> Any help or lead would be appreciated
>
> Note : We are using Cassandra version 3.7
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org