You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Marcus Eriksson (JIRA)" <ji...@apache.org> on 2015/12/17 15:53:46 UTC
[jira] [Comment Edited] (CASSANDRA-10829) cleanup + repair
generates a lot of logs
[ https://issues.apache.org/jira/browse/CASSANDRA-10829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15062136#comment-15062136 ]
Marcus Eriksson edited comment on CASSANDRA-10829 at 12/17/15 2:52 PM:
-----------------------------------------------------------------------
Problem is that we use the sstables marked as 'compacting' to create the canonical view (to get the original sstable instances) and when running cleanup, scrub, upgradesstables and anticompaction we mark each sstable as compacted as we finish it, but we keep it as compacting until the entire operation is done. This means that we have sstables which are marked as compacted in the CANONICAL_SSTABLES and we cannot reference them and it makes us log the 'spinning ... ' message.
There is also an issue with the log rate limiting, it was meant to only log every 100ms, but it logs every iteration.
Patch that unmarks compacting once the sstable finished and fixes the rate limiting:
https://github.com/krummas/cassandra/commits/marcuse/10829
and tests:
https://cassci.datastax.com/view/Dev/view/krummas/job/krummas-marcuse-10829-dtest/
https://cassci.datastax.com/view/Dev/view/krummas/job/krummas-marcuse-10829-testall/
seems only the rate limiting fix applies to 2.2+ (edit: the other issues are already fixed with LifecycleTransaction)
was (Author: krummas):
Problem is that we use the sstables marked as 'compacting' to create the canonical view (to get the original sstable instances) and when running cleanup, scrub, upgradesstables and anticompaction we mark each sstable as compacted as we finish it, but we keep it as compacting until the entire operation is done. This means that we have sstables which are marked as compacted in the CANONICAL_SSTABLES and we cannot reference them and it makes us log the 'spinning ... ' message.
There is also an issue with the log rate limiting, it was meant to only log every 100ms, but it logs every iteration.
Patch that unmarks compacting once the sstable finished and fixes the rate limiting:
https://github.com/krummas/cassandra/commits/marcuse/10829
and tests:
https://cassci.datastax.com/view/Dev/view/krummas/job/krummas-marcuse-10829-dtest/
https://cassci.datastax.com/view/Dev/view/krummas/job/krummas-marcuse-10829-testall/
seems only the rate limiting fix applies to 2.2+
> cleanup + repair generates a lot of logs
> ----------------------------------------
>
> Key: CASSANDRA-10829
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10829
> Project: Cassandra
> Issue Type: Bug
> Environment: 5 nodes on Cassandra 2.1.11 (on Debian)
> Reporter: Fabien Rousseau
> Assignee: Marcus Eriksson
>
> One of our node generates a lot of cassandra logs (int the 10 MB/s) and CPU usage has increased (by a factor 2-3).
> This was most probably triggered by a "nodetool snapshot" while a cleanup was already running on this node.
> An example of those logs:
> 2015-12-08 09:15:17,794 INFO [ValidationExecutor:689]ColumnFamilyStore.java:1923 Spinning trying to capture released readers [...]
> 2015-12-08 09:15:17,794 INFO [ValidationExecutor:689]ColumnFamilyStore.java:1924 Spinning trying to capture all readers [...]
> 2015-12-08 09:15:17,795 INFO [ValidationExecutor:689]ColumnFamilyStore.java:1923 Spinning trying to capture released readers [...]
> 2015-12-08 09:15:17,795 INFO [ValidationExecutor:689]ColumnFamilyStore.java:1924 Spinning trying to capture all readers [...]
> (I removed SSTableReader information because it's rather long... I can share it privately if needed)
> Note that the date has not been changed (only 1ms between logs)
> It should not generate that gigantic amount of logs :)
> This is probably linked to: https://issues.apache.org/jira/browse/CASSANDRA-9637
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)