You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Alexey Zotov (Jira)" <ji...@apache.org> on 2021/04/17 18:46:00 UTC
[jira] [Commented] (CASSANDRA-15669) LeveledCompactionStrategy
compact last level throw an ArrayIndexOutOfBoundsException
[ https://issues.apache.org/jira/browse/CASSANDRA-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17324337#comment-17324337 ]
Alexey Zotov commented on CASSANDRA-15669:
------------------------------------------
I have checked this issue. I feel I have a kind of clear understanding of what is going on.
[~sunhaihong]
Just a point of my curiosity - what values for {{sstable_size_in_mb}} and {{fanout_size }}params do you use in prod and how much data do you have? I'm just wondering how you were able to face this issue.
[~marcuse]
Looks like you are the best person to discuss this issue (as I can see, you actively participated in LCS development).
First of all, I was able to reproduce this issue. I explored the code and probably I found a few issues.
# Wrong estimates calculation
There is the following comment in the code:
{code:java}
// allocate enough generations for a PB of data, with a 1-MB sstable size. (Note that if maxSSTableSize is
// updated, we will still have sstables of the older, potentially smaller size. So don't make this
// dependent on maxSSTableSize.)
static final int MAX_LEVEL_COUNT = (int) Math.log10(1000 * 1000 * 1000);
{code}
It states about a PB of data for 1-MB sstable size configuration, but it does not seem to be correct. It would be correct if 10 levels were supported. However, 9 levels are currently supported. Here are my calculations (1-MB sstable size and 10 fanout size):
{code:java}
L0: 4 * 1 MB = 4 MB
L1: 10^1 * 1 MB = 10 MB
L2: 10^2 * 1 MB = 100 MB
L3: 10^3 * 1 MB = 1000 MB
L4: 10^4 * 1 MB = 10000 MB = 9.76 GB
L5: 10^5 * 1 MB = 100000 MB = 97.65 GB
L6: 10^6 * 1 MB = 1000000 MB = 976.56 GB
L7: 10^7 * 1 MB = 10000000 MB = 9765.62 GB = 9.53 TB
L8: 10^8 * 1 MB = 100000000 MB = 97656.25 GB = 95.36 TB
L9: 10^9 * 1 MB = 1000000000 MB = 976562.50 GB = 953.67 TB <-- this level is not supported {code}
Here is the place where it is clearly shown that 9 levels (including L0) are supported at the moment:
{code:java}
// note that since l0 is broken out, levels[0] represents L1:
private final TreeSet<SSTableReader> [] levels = new TreeSet[MAX_LEVEL_COUNT - 1];
{code}
Either the comment needs to be fixed or the number of levels needs to be increased. I believe fixing the comment would be easier and amount of data still would be enough for a regular C* setup.
# L8 is not being handled properly
The issue happens when compaction for L8 is going to be started. Here is the flow: {{getCompactionCandidates}} --> {{getCandidatesFor(i)}} --> {{generations.get(level + 1)}} .So while checking the compaction candidates for the L8, it tries to see what's going on L9 level and immediately fails. Which means L8 is not ever being built/compacted. I think we need to handle the highest level (8th) separately in {{getCandidatesFor(i)}} method as we handle L0 (with a separate condition).
# There is no proper handling of a situation when there is more data than supported
As it is stated in the first point, we target to support a certain amount of data. I think a clear error should be thrown when there is an attempt to handle more data than expected on the highest level, smth like:
{code:java}
long bytesForLevel = SSTableReader.getTotalBytes(sstablesInLevel);
long maxBytesForLevel = maxBytesForLevel(i, maxSSTableSizeInBytes);
if (i == generations.levelCount() - 1 && bytesForLevel > maxBytesForLevel)
throw new RuntimeException("Current size (" + bytesForLevel + ") exceeds max size (" + maxBytesForLevel + ") for " + i + " level");
{code}
I'd be glad to hear you feedback on the points above. If you find the suggestions reasonable, I'd like to come up with a patch (I have a draft, but before polishing it I'd like to validate my understanding).
> LeveledCompactionStrategy compact last level throw an ArrayIndexOutOfBoundsException
> ------------------------------------------------------------------------------------
>
> Key: CASSANDRA-15669
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15669
> Project: Cassandra
> Issue Type: Bug
> Reporter: sunhaihong
> Assignee: sunhaihong
> Priority: Normal
> Attachments: cfs_compaction_info.png, error_info.png
>
>
> Cassandra will throw an ArrayIndexOutOfBoundsException when compact last level.
> My test is as follows：
> # Create a table with LeveledCompactionStrategy and its params are 'enabled': 'true', 'fanout_size': '2', 'max_threshold': '32', 'min_threshold': '4', 'sstable_size_in_mb': '2'（fanout_size and sstable_size_in_mb are too small just to make it easier to reproduce the problem）;
> # Insert data into the table by stress;
> # Cassandra throw an ArrayIndexOutOfBoundsException when compact level9 sstables(this level score bigger than 1.001)
> ERROR [CompactionExecutor:4] 2020-03-28 08:59:00,990 CassandraDaemon.java:442 - Exception in thread Thread[CompactionExecutor:4,1,main]
> java.lang.ArrayIndexOutOfBoundsException: 9
> at org.apache.cassandra.db.compaction.LeveledManifest.getLevel(LeveledManifest.java:814)
> at org.apache.cassandra.db.compaction.LeveledManifest.getCandidatesFor(LeveledManifest.java:746)
> at org.apache.cassandra.db.compaction.LeveledManifest.getCompactionCandidates(LeveledManifest.java:398)
> at org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getNextBackgroundTask(LeveledCompactionStrategy.java:131)
> at org.apache.cassandra.db.compaction.CompactionStrategyHolder.lambda$getBackgroundTaskSuppliers$0(CompactionStrategyHolder.java:109)
> at org.apache.cassandra.db.compaction.AbstractStrategyHolder$TaskSupplier.getTask(AbstractStrategyHolder.java:66)
> at org.apache.cassandra.db.compaction.CompactionStrategyManager.getNextBackgroundTask(CompactionStrategyManager.java:214)
> at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:289)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
> at java.util.concurrent.FutureTask.run(FutureTask.java)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.lang.Thread.run(Thread.java:748)
> I tested it on cassandra version 3.11.3 & 4.0-alpha3. The exception all happened.
> once it triggers, level1- leveln compaction no longer works, level0 is still valid
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org