You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Richard Hesse <rh...@gmail.com> on 2022/10/30 17:07:17 UTC
Help determining pending compactions
Hi, I'm hoping to get some help with a vexing issue with one of our
keyspaces. During Reaper repair sessions, one keyspace will end up with
hanging, non-started compactions. That is, the number of compactions as
reported by nodetool compactionstats stays flat and there are no running
compactions. Is there a way to determine which tables Cassandra is stuck on
here?
Looking at graphs of pending compactions during Reaper sessions, the number
of compactions will shoot up (as expected). The number will work its way
down, and sometimes it will stop and plateau at a fixed level. A full
compaction will get things going again, but we prefer to avoid those (even
with the -s option).
I realize there are various compaction tuning parameters regarding minimum
age, tombstone percentage, etc but I need to know which sstables to look at
first before blindly changing values. These are leveled compaction strategy
tables FWIW.
TIA!
-richard
Re: Help determining pending compactions
Posted by Richard Hesse <rh...@gmail.com>.
Sorry about that. 4.0.6
On Sun, Oct 30, 2022, 11:19 AM Dinesh Joshi <dj...@apache.org> wrote:
> It would be helpful if you could tell us what version of Cassandra you’re
> using?
>
> Dinesh
>
> > On Oct 30, 2022, at 10:07 AM, Richard Hesse <rh...@gmail.com> wrote:
> >
> >
> > Hi, I'm hoping to get some help with a vexing issue with one of our
> keyspaces. During Reaper repair sessions, one keyspace will end up with
> hanging, non-started compactions. That is, the number of compactions as
> reported by nodetool compactionstats stays flat and there are no running
> compactions. Is there a way to determine which tables Cassandra is stuck on
> here?
> >
> > Looking at graphs of pending compactions during Reaper sessions, the
> number of compactions will shoot up (as expected). The number will work its
> way down, and sometimes it will stop and plateau at a fixed level. A full
> compaction will get things going again, but we prefer to avoid those (even
> with the -s option).
> >
> > I realize there are various compaction tuning parameters regarding
> minimum age, tombstone percentage, etc but I need to know which sstables to
> look at first before blindly changing values. These are leveled compaction
> strategy tables FWIW.
> >
> > TIA!
> >
> > -richard
>
Re: Help determining pending compactions
Posted by Dinesh Joshi <dj...@apache.org>.
It would be helpful if you could tell us what version of Cassandra you’re using?
Dinesh
> On Oct 30, 2022, at 10:07 AM, Richard Hesse <rh...@gmail.com> wrote:
>
>
> Hi, I'm hoping to get some help with a vexing issue with one of our keyspaces. During Reaper repair sessions, one keyspace will end up with hanging, non-started compactions. That is, the number of compactions as reported by nodetool compactionstats stays flat and there are no running compactions. Is there a way to determine which tables Cassandra is stuck on here?
>
> Looking at graphs of pending compactions during Reaper sessions, the number of compactions will shoot up (as expected). The number will work its way down, and sometimes it will stop and plateau at a fixed level. A full compaction will get things going again, but we prefer to avoid those (even with the -s option).
>
> I realize there are various compaction tuning parameters regarding minimum age, tombstone percentage, etc but I need to know which sstables to look at first before blindly changing values. These are leveled compaction strategy tables FWIW.
>
> TIA!
>
> -richard
Re: Help determining pending compactions
Posted by Richard Hesse <rh...@gmail.com>.
Thanks for the tip Eric. We're actually on 3.2 and the issue isn't with the
Reaper. The issue is with Cassandra. It will report that a table has
pending compactions, but it will never actually start compacting. The
pending number stays at that level until we run a manual compaction.
-richard
On Mon, Nov 7, 2022 at 8:53 AM Eric Ferrenbach <
eric.ferrenbach@milliporesigma.com> wrote:
> We had issues where Reaper would never actually start some repairs. The
> GUI would say RUNNING but the progress would be 0/xxxx.
>
> Datastax support said there is a bug and recommended upgrading to 3.2.
>
> Upgrading Reaper to 3.2 resolved our issue.
>
>
>
> Hope this helps.
>
> Eric
>
>
>
>
>
>
>
> *From:* Richard Hesse <rh...@gmail.com>
> *Sent:* Sunday, October 30, 2022 12:07 PM
> *To:* user@cassandra.apache.org
> *Subject:* Help determining pending compactions
>
>
>
> [WARNING - EXTERNAL EMAIL] Do not open links or attachments unless you
> recognize the sender of this email. If you are unsure please click the
> button "Report suspicious email"
>
>
>
> Hi, I'm hoping to get some help with a vexing issue with one of our
> keyspaces. During Reaper repair sessions, one keyspace will end up with
> hanging, non-started compactions. That is, the number of compactions as
> reported by nodetool compactionstats stays flat and there are no running
> compactions. Is there a way to determine which tables Cassandra is stuck on
> here?
>
>
>
> Looking at graphs of pending compactions during Reaper sessions, the
> number of compactions will shoot up (as expected). The number will work its
> way down, and sometimes it will stop and plateau at a fixed level. A full
> compaction will get things going again, but we prefer to avoid those (even
> with the -s option).
>
>
>
> I realize there are various compaction tuning parameters regarding minimum
> age, tombstone percentage, etc but I need to know which sstables to look at
> first before blindly changing values. These are leveled compaction strategy
> tables FWIW.
>
>
>
> TIA!
>
>
>
> -richard
>
>
>
> This message and any attachment are confidential and may be privileged or
> otherwise protected from disclosure. If you are not the intended recipient,
> you must not copy this message or attachment or disclose the contents to
> any other person. If you have received this transmission in error, please
> notify the sender immediately and delete the message and any attachment
> from your system. Merck KGaA, Darmstadt, Germany and any of its
> subsidiaries do not accept liability for any omissions or errors in this
> message which may arise as a result of E-Mail-transmission or for damages
> resulting from any unauthorized changes of the content of this message and
> any attachment thereto. Merck KGaA, Darmstadt, Germany and any of its
> subsidiaries do not guarantee that this message is free of viruses and does
> not accept liability for any damages caused by any virus transmitted
> therewith.
>
>
>
> Click emdgroup.com/disclaimer
> <https://www.emdgroup.com/en/legal-disclaimer/mail-disclaimer.html> to
> access the German, French, Spanish, Portuguese, Turkish, Polish and Slovak
> versions of this disclaimer.
>
>
>
> Please find our Privacy Statement information by clicking here: emdgroup.com/privacy-statement
> (U.S.) <https://www.emdgroup.com/en/privacy-statement.html> or emdserono.com/privacy-statement
> (Canada) <https://www.emdserono.ca/ca-en/privacy-statement.html>
>
RE: Help determining pending compactions
Posted by Eric Ferrenbach <er...@milliporesigma.com>.
We had issues where Reaper would never actually start some repairs. The GUI would say RUNNING but the progress would be 0/xxxx.
Datastax support said there is a bug and recommended upgrading to 3.2.
Upgrading Reaper to 3.2 resolved our issue.
Hope this helps.
Eric
From: Richard Hesse <rh...@gmail.com>
Sent: Sunday, October 30, 2022 12:07 PM
To: user@cassandra.apache.org
Subject: Help determining pending compactions
[WARNING - EXTERNAL EMAIL] Do not open links or attachments unless you recognize the sender of this email. If you are unsure please click the button "Report suspicious email"
Hi, I'm hoping to get some help with a vexing issue with one of our keyspaces. During Reaper repair sessions, one keyspace will end up with hanging, non-started compactions. That is, the number of compactions as reported by nodetool compactionstats stays flat and there are no running compactions. Is there a way to determine which tables Cassandra is stuck on here?
Looking at graphs of pending compactions during Reaper sessions, the number of compactions will shoot up (as expected). The number will work its way down, and sometimes it will stop and plateau at a fixed level. A full compaction will get things going again, but we prefer to avoid those (even with the -s option).
I realize there are various compaction tuning parameters regarding minimum age, tombstone percentage, etc but I need to know which sstables to look at first before blindly changing values. These are leveled compaction strategy tables FWIW.
TIA!
-richard
This message and any attachment are confidential and may be privileged or otherwise protected from disclosure. If you are not the intended recipient, you must not copy this message or attachment or disclose the contents to any other person. If you have received this transmission in error, please notify the sender immediately and delete the message and any attachment from your system. Merck KGaA, Darmstadt, Germany and any of its subsidiaries do not accept liability for any omissions or errors in this message which may arise as a result of E-Mail-transmission or for damages resulting from any unauthorized changes of the content of this message and any attachment thereto. Merck KGaA, Darmstadt, Germany and any of its subsidiaries do not guarantee that this message is free of viruses and does not accept liability for any damages caused by any virus transmitted therewith.
Click emdgroup.com/disclaimer<https://www.emdgroup.com/en/legal-disclaimer/mail-disclaimer.html> to access the German, French, Spanish, Portuguese, Turkish, Polish and Slovak versions of this disclaimer.
Please find our Privacy Statement information by clicking here: emdgroup.com/privacy-statement (U.S.)<https://www.emdgroup.com/en/privacy-statement.html> or emdserono.com/privacy-statement (Canada)<https://www.emdserono.ca/ca-en/privacy-statement.html>