You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Joey Lynch (Jira)" <ji...@apache.org> on 2021/09/20 14:15:00 UTC
[jira] [Comment Edited] (CASSANDRA-14898) Key cache loading is very
slow when there are many SSTables
[ https://issues.apache.org/jira/browse/CASSANDRA-14898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17417675#comment-17417675 ]
Joey Lynch edited comment on CASSANDRA-14898 at 9/20/21, 2:14 PM:
------------------------------------------------------------------
[~n.v.harikrishna] Thank you for the patch! Would you like to get this into 3.0/3.11/4.0 and trunk? If so I think we need patches against each branch (the metadata variable names have changed slightly so I think separate patches might be easiest rather than one patch that we merge up); I am happy to do the minor changes on each branch if you like as well.
Review of patch:
* I think introducing a little bit of state here is wise, if we want to be extra careful do you think we could use [AutoCloseable|https://docs.oracle.com/javase/tutorial/essential/exceptions/tryResourceClose.html] somehow to ensure the state is cleaned up? Seems reasonable for {{close}} to just cleanup the state used during load or dump.
* Did you mean to remove RowIndexEntry.Serializer.skipForCache(input); on line 453 ?
* Do you think we could improve the tests in AutoSavingCacheTest to have a case where a few sstables are created across different ks/table and we want to confirm the key cache is loaded properly with more than one keyspace/table?
* Regarding the deadline approach that is an interesting idea and I like it, I'd recommend you implement it in the while loop in AutoSavingCache.loadSaved (the while in.available loop) instead of by cancelling the future at the high level.
was (Author: jolynch):
[~n.v.harikrishna] Thank you for the patch. Would you like to get this into 3.0/3.11/4.0 and trunk? If so I think we need patches against each branch (the metadata variable names have changed slightly so I think separate patches might be easiest rather than one patch that we merge up); I am happy to do the minor changes on each branch if you like as well.
Review of patch:
* I think introducing a little bit of state here is wise, if we want to be extra careful do you think we could use [AutoCloseable|https://docs.oracle.com/javase/tutorial/essential/exceptions/tryResourceClose.html] somehow to ensure the state is cleaned up? Seems reasonable for {{close}} to just cleanup the state used during load or dump.
* Did you mean to remove RowIndexEntry.Serializer.skipForCache(input); on line 453 ?
* Do you think we could improve the tests in AutoSavingCacheTest to have a case where a few sstables are created across different ks/table and we want to confirm the key cache is loaded properly with more than one keyspace/table?
* Regarding the deadline approach that is an interesting idea and I like it, I'd recommend you implement it in the while loop in AutoSavingCache.loadSaved (the while in.available loop) instead of by cancelling the future at the high level.
> Key cache loading is very slow when there are many SSTables
> -----------------------------------------------------------
>
> Key: CASSANDRA-14898
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14898
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Local Write-Read Paths
> Environment: AWS i3.2xlarge, 4 physical cores (8 threads), 60GB of RAM, loading about 8MB of KeyCache with 10k keys in it.
> Reporter: Joey Lynch
> Assignee: Venkata Harikrishna Nukala
> Priority: Low
> Labels: Performance, low-hanging-fruit
> Attachments: key_cache_load_slow.svg
>
>
> While dealing with a production issue today where some 3.0.17 nodes had close to ~8k sstables on disk due to excessive write pressure, we had a few nodes crash due to OOM and then they took close to 17 minutes to load the key cache and recover. This excessive key cache load significantly increased the duration of the outage (to mitigate we just removed the saved key cache files). For example here is one example taking 17 minutes to load 10k keys, or about 10 keys per second (which is ... very slow):
> {noformat}
> INFO [pool-3-thread-1] 2018-11-15 21:50:21,885 AutoSavingCache.java:190 - reading saved cache /mnt/data/cassandra/saved_caches/KeyCache-d.db
> INFO [pool-3-thread-1] 2018-11-15 22:07:16,490 AutoSavingCache.java:166 - Completed loading (1014606 ms; 10103 keys) KeyCache cache
> {noformat}
> I've witnessed similar behavior in the past with large LCS clusters, and indeed it appears that any time the number of sstables is large, KeyCache loading takes a _really_ long time. Today I got a flame graph and I believe that I found the issue and I think it's reasonably easy to fix. From what I can tell the {{KeyCacheSerializer::deserialize}} [method |https://github.com/apache/cassandra/blob/06209037ea56b5a2a49615a99f1542d6ea1b2947/src/java/org/apache/cassandra/service/CacheService.java#L445] which is called for every key is linear in the number of sstables due to the [call|https://github.com/apache/cassandra/blob/06209037ea56b5a2a49615a99f1542d6ea1b2947/src/java/org/apache/cassandra/service/CacheService.java#L459] to {{ColumnFamilyStore::getSSTables}} which ends up calling {{View::select}} [here|https://github.com/apache/cassandra/blob/06209037ea56b5a2a49615a99f1542d6ea1b2947/src/java/org/apache/cassandra/db/lifecycle/View.java#L139]. The {{View::select}} call is linear in the number of sstables and causes a _lot_ of {{HashSet}} [resizing|https://github.com/apache/cassandra/blob/06209037ea56b5a2a49615a99f1542d6ea1b2947/src/java/org/apache/cassandra/db/lifecycle/View.java#L139] when the number of sstables is much greater than 16 (the default size of the backing {{HashMap}}).
> As we see in the attached flamegraph we spend 50% of our CPU time in these {{getSSTable}} calls, of which 36% is spent adding sstables to the HashSet in {{View::select}} and 17% is spent just iterating the sstables in the first place. A full 16% of CPU time is spent _just resizing the HashMap_. Then another 4% is spend calling {{CacheService::findDesc}} which does [a linear search|https://github.com/apache/cassandra/blob/06209037ea56b5a2a49615a99f1542d6ea1b2947/src/java/org/apache/cassandra/service/CacheService.java#L475] for the sstable generation.
> I believe that this affects at least Cassandra 3.0.17 and trunk, and could be pretty easily fixed by either caching the getSSTables call or at the very least pre-sizing the {{HashSet}} in {{View::select}} to be the size of the sstables map.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org