You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by kurt greaves <ku...@instaclustr.com> on 2017/05/02 05:59:46 UTC

Re: [Cassandra] nodetool compactionstats not showing pending task.

I believe this is a bug with the estimation of tasks, however not aware of
any JIRA that covers the issue.

On 28 April 2017 at 06:19, Abhishek Kumar Maheshwari <
Abhishek.Maheshwari@timesinternet.in> wrote:

> Hi ,
>
>
>
> I will try with JMX but I try with tpstats. In tpstats its showing pending
> compaction as 0 but in nodetool compactionstats its showing 3. So, for me
> its seems strange.
>
>
>
> *Thanks & Regards,*
> *Abhishek Kumar Maheshwari*
> *+91- 9999805591 <+91%2099998%2005591> (Mobile)*
>
> Times Internet Ltd. | A Times of India Group Company
>
> FC - 6, Sector 16A, Film City,  Noida,  U.P. 201301 | INDIA
>
> *P** Please do not print this email unless it is absolutely necessary.
> Spread environmental awareness.*
>
>
>
> *From:* Alain RODRIGUEZ [mailto:arodrime@gmail.com]
> *Sent:* Thursday, April 27, 2017 4:45 PM
> *To:* user@cassandra.apache.org
> *Subject:* Re: [Cassandra] nodetool compactionstats not showing pending
> task.
>
>
>
> Maybe try to monitor through JMX with 'org.apache.cassandra.db:type=CompactionManager',
> attribute 'Compactions' or 'CompactionsSummary'
>
>
>
> C*heers
>
> -----------------------
>
> Alain Rodriguez - @arodream - alain@thelastpickle.com
>
> France
>
>
>
> The Last Pickle - Apache Cassandra Consulting
>
> http://www.thelastpickle.com
>
>
>
> 2017-04-27 12:27 GMT+02:00 Alain RODRIGUEZ <ar...@gmail.com>:
>
> Hi,
>
>
>
> I am not sure about this one. It happened to me in the past as well. I
> never really wondered about it as it was gone after a while or a restart
> off the top of my head. To get rid of it, a restart might be enough.
>
>
>
> But if you feel like troubleshooting this, I think the first thing is to
> try to see if compactions are really happening. Maybe using JMX, I believe
> `org.apache.cassandra.metrics:type=Compaction,name=PendingTasks` is what
> is used by 'nodetool compactionstats' but they might be more info there.
> Actually I don't really know what the 'system.compactions_in_progress'
> was replaced by, but any way to double check you could think of would
> probably help understanding better what's happening.
>
>
>
> Does someone now the way to check pending compactions details in 3.0.9?
>
>
>
> C*heers,
>
> -----------------------
>
> Alain Rodriguez - @arodream - alain@thelastpickle.com
>
> France
>
>
>
> The Last Pickle - Apache Cassandra Consulting
>
> http://www.thelastpickle.com
>
>
>
> 2017-04-25 15:13 GMT+02:00 Abhishek Kumar Maheshwari <Abhishek.Maheshwari@
> timesinternet.in>:
>
> Hi All,
>
>
>
> In Production, I am using Cassandra 3.0.9.
>
>
>
> While I am running nodetool compactionstats command its just showing count
> not any other information like below:
>
>
>
> [mohit.kundra@AdtechAppXXXX bin]$ ./nodetool -h XXX.XX.XX.XX
> compactionstats
>
> pending tasks: 3
>
> [mohit.kundra@AdtechAppXXXXX bin]$
>
>
>
> So, this is some Cassandra bug or what? I am not able to understand.
>
>
>
>
>
> *Thanks & Regards,*
> *Abhishek Kumar Maheshwari*
> *+91- 9999805591 <+91%2099998%2005591> (Mobile)*
>
> Times Internet Ltd. | A Times of India Group Company
>
> FC - 6, Sector 16A, Film City,  Noida,  U.P. 201301 | INDIA
>
> *P** Please do not print this email unless it is absolutely necessary.
> Spread environmental awareness.*
>
>
>
> "Learn journalism at India's largest media house - The Times of India
> Group. Last Date 28 April, 2017. Visit www.tcms.in for details."
>
>
>
>
>

Re: [Cassandra] nodetool compactionstats not showing pending task.

Posted by Oleksandr Shulgin <ol...@zalando.de>.
On Fri, May 5, 2017 at 1:20 PM Alain RODRIGUEZ <ar...@gmail.com> wrote:

> Sorry to hear the restart did not help.
>

Hi,

We are hitting the same issue since a few weeks on version 3.0.16.
Normally, restarting an affected node helps, but this is something we would
like to avoid doing.

What makes it worse for us is that Cassandra Reaper stops scheduling new
repair jobs if it sees that the node have more than 20 pending compaction
tasks.  We could bump this threshold, but in general the estimate could be
more accurate (or the actual tasks should be started timely).

Maybe try to monitor through JMX with
'org.apache.cassandra.db:type=CompactionManager',
>> attribute 'Compactions' or 'CompactionsSummary'
>
>
> What is this attribute showing?
>

For example, I have right now a node showing "pending tasks: 16" and no
compaction running.  Here is the JMX output (well, via Jolokia):

            "Compactions": [],
            "CoreCompactorThreads": 1,
            "CompactionSummary": [],
            "MaximumCompactorThreads": 1,
            "CoreValidationThreads": 1,
            "MaximumValidatorThreads": 2147483647,

Here is the Apache Cassandra Jira:
> https://issues.apache.org/jira/browse/CASSANDRA. You search here
> <https://issues.apache.org/jira/browse/CASSANDRA-12529?jql=project%20%3D%20CASSANDRA%20AND%20text%20~%20%22pending%20compactions%22%20ORDER%20BY%20created%20DESC>
>  (
> https://issues.apache.org/jira/browse/CASSANDRA-12529?jql=project%20%3D%20CASSANDRA%20AND%20text%20~%20%22pending%20compactions%22%20ORDER%20BY%20created%20DESC),
> for example.
>

I believe this is a separate issue.  There some actual compaction tasks are
running, but not making progress.  And we never TRUNCATEd our tables, and
for sure not recently.

Any more pointers on how to debug this?

Regards,
--
Alex

Re: [Cassandra] nodetool compactionstats not showing pending task.

Posted by Alain RODRIGUEZ <ar...@gmail.com>.
Hi,

Sorry to hear the restart did not help.

Maybe try to monitor through JMX with
'org.apache.cassandra.db:type=CompactionManager',
> attribute 'Compactions' or 'CompactionsSummary'


What is this attribute showing?

Here is the Apache Cassandra Jira:
https://issues.apache.org/jira/browse/CASSANDRA. You search here
<https://issues.apache.org/jira/browse/CASSANDRA-12529?jql=project%20%3D%20CASSANDRA%20AND%20text%20~%20%22pending%20compactions%22%20ORDER%20BY%20created%20DESC>
 (
https://issues.apache.org/jira/browse/CASSANDRA-12529?jql=project%20%3D%20CASSANDRA%20AND%20text%20~%20%22pending%20compactions%22%20ORDER%20BY%20created%20DESC),
for example.

C*heers,
-----------------------
Alain Rodriguez - @arodream - alain@thelastpickle.com
France

The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com

2017-05-05 6:01 GMT+01:00 Abhishek Kumar Maheshwari <
Abhishek.Maheshwari@timesinternet.in>:

> I just restart the cluster but still facing same issue. Please let me know
> where I can search on JIRA or will raise new ticket for the same?
>
>
>
> *Thanks & Regards,*
> *Abhishek Kumar Maheshwari*
> *+91- 9999805591 <+91%2099998%2005591> (Mobile)*
>
> Times Internet Ltd. | A Times of India Group Company
>
> FC - 6, Sector 16A, Film City,  Noida,  U.P. 201301 | INDIA
>
> *P** Please do not print this email unless it is absolutely necessary.
> Spread environmental awareness.*
>
>
>
> *From:* kurt greaves [mailto:kurt@instaclustr.com]
> *Sent:* Tuesday, May 2, 2017 11:30 AM
> *To:* Abhishek Kumar Maheshwari <Ab...@timesinternet.in>
> *Cc:* Alain RODRIGUEZ <ar...@gmail.com>; user@cassandra.apache.org
>
> *Subject:* Re: [Cassandra] nodetool compactionstats not showing pending
> task.
>
>
>
> I believe this is a bug with the estimation of tasks, however not aware of
> any JIRA that covers the issue.
>
>
>
> On 28 April 2017 at 06:19, Abhishek Kumar Maheshwari <Abhishek.Maheshwari@
> timesinternet.in> wrote:
>
> Hi ,
>
>
>
> I will try with JMX but I try with tpstats. In tpstats its showing pending
> compaction as 0 but in nodetool compactionstats its showing 3. So, for me
> its seems strange.
>
>
>
> *Thanks & Regards,*
> *Abhishek Kumar Maheshwari*
> *+91- 9999805591 <+91%2099998%2005591> (Mobile)*
>
> Times Internet Ltd. | A Times of India Group Company
>
> FC - 6, Sector 16A, Film City,  Noida,  U.P. 201301 | INDIA
>
> *P** Please do not print this email unless it is absolutely necessary.
> Spread environmental awareness.*
>
>
>
> *From:* Alain RODRIGUEZ [mailto:arodrime@gmail.com]
> *Sent:* Thursday, April 27, 2017 4:45 PM
> *To:* user@cassandra.apache.org
> *Subject:* Re: [Cassandra] nodetool compactionstats not showing pending
> task.
>
>
>
> Maybe try to monitor through JMX with 'org.apache.cassandra.db:type=CompactionManager',
> attribute 'Compactions' or 'CompactionsSummary'
>
>
>
> C*heers
>
> -----------------------
>
> Alain Rodriguez - @arodream - alain@thelastpickle.com
>
> France
>
>
>
> The Last Pickle - Apache Cassandra Consulting
>
> http://www.thelastpickle.com
>
>
>
> 2017-04-27 12:27 GMT+02:00 Alain RODRIGUEZ <ar...@gmail.com>:
>
> Hi,
>
>
>
> I am not sure about this one. It happened to me in the past as well. I
> never really wondered about it as it was gone after a while or a restart
> off the top of my head. To get rid of it, a restart might be enough.
>
>
>
> But if you feel like troubleshooting this, I think the first thing is to
> try to see if compactions are really happening. Maybe using JMX, I believe
> `org.apache.cassandra.metrics:type=Compaction,name=PendingTasks` is what
> is used by 'nodetool compactionstats' but they might be more info there.
> Actually I don't really know what the 'system.compactions_in_progress'
> was replaced by, but any way to double check you could think of would
> probably help understanding better what's happening.
>
>
>
> Does someone now the way to check pending compactions details in 3.0.9?
>
>
>
> C*heers,
>
> -----------------------
>
> Alain Rodriguez - @arodream - alain@thelastpickle.com
>
> France
>
>
>
> The Last Pickle - Apache Cassandra Consulting
>
> http://www.thelastpickle.com
>
>
>
> 2017-04-25 15:13 GMT+02:00 Abhishek Kumar Maheshwari <Abhishek.Maheshwari@
> timesinternet.in>:
>
> Hi All,
>
>
>
> In Production, I am using Cassandra 3.0.9.
>
>
>
> While I am running nodetool compactionstats command its just showing count
> not any other information like below:
>
>
>
> [mohit.kundra@AdtechAppXXXX bin]$ ./nodetool -h XXX.XX.XX.XX
> compactionstats
>
> pending tasks: 3
>
> [mohit.kundra@AdtechAppXXXXX bin]$
>
>
>
> So, this is some Cassandra bug or what? I am not able to understand.
>
>
>
>
>
> *Thanks & Regards,*
> *Abhishek Kumar Maheshwari*
> *+91- 9999805591 <+91%2099998%2005591> (Mobile)*
>
> Times Internet Ltd. | A Times of India Group Company
>
> FC - 6, Sector 16A, Film City,  Noida,  U.P. 201301 | INDIA
>
> *P** Please do not print this email unless it is absolutely necessary.
> Spread environmental awareness.*
>
>
>
> "Learn journalism at India's largest media house - The Times of India
> Group. Last Date 28 April, 2017. Visit www.tcms.in for details."
>
>
>
>
>
>
>

RE: [Cassandra] nodetool compactionstats not showing pending task.

Posted by Abhishek Kumar Maheshwari <Ab...@timesinternet.in>.
I just restart the cluster but still facing same issue. Please let me know where I can search on JIRA or will raise new ticket for the same?

Thanks & Regards,
Abhishek Kumar Maheshwari
+91- 9999805591 (Mobile)
Times Internet Ltd. | A Times of India Group Company
FC - 6, Sector 16A, Film City,  Noida,  U.P. 201301 | INDIA
P Please do not print this email unless it is absolutely necessary. Spread environmental awareness.

From: kurt greaves [mailto:kurt@instaclustr.com]
Sent: Tuesday, May 2, 2017 11:30 AM
To: Abhishek Kumar Maheshwari <Ab...@timesinternet.in>
Cc: Alain RODRIGUEZ <ar...@gmail.com>; user@cassandra.apache.org
Subject: Re: [Cassandra] nodetool compactionstats not showing pending task.

I believe this is a bug with the estimation of tasks, however not aware of any JIRA that covers the issue.

On 28 April 2017 at 06:19, Abhishek Kumar Maheshwari <Ab...@timesinternet.in>> wrote:
Hi ,

I will try with JMX but I try with tpstats. In tpstats its showing pending compaction as 0 but in nodetool compactionstats its showing 3. So, for me its seems strange.

Thanks & Regards,
Abhishek Kumar Maheshwari
+91- 9999805591<tel:+91%2099998%2005591> (Mobile)
Times Internet Ltd. | A Times of India Group Company
FC - 6, Sector 16A, Film City,  Noida,  U.P. 201301 | INDIA
P Please do not print this email unless it is absolutely necessary. Spread environmental awareness.

From: Alain RODRIGUEZ [mailto:arodrime@gmail.com<ma...@gmail.com>]
Sent: Thursday, April 27, 2017 4:45 PM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: Re: [Cassandra] nodetool compactionstats not showing pending task.

Maybe try to monitor through JMX with 'org.apache.cassandra.db:type=CompactionManager', attribute 'Compactions' or 'CompactionsSummary'

C*heers
-----------------------
Alain Rodriguez - @arodream - alain@thelastpickle.com<ma...@thelastpickle.com>
France

The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com

2017-04-27 12:27 GMT+02:00 Alain RODRIGUEZ <ar...@gmail.com>>:
Hi,

I am not sure about this one. It happened to me in the past as well. I never really wondered about it as it was gone after a while or a restart off the top of my head. To get rid of it, a restart might be enough.

But if you feel like troubleshooting this, I think the first thing is to try to see if compactions are really happening. Maybe using JMX, I believe `org.apache.cassandra.metrics:type=Compaction,name=PendingTasks` is what is used by 'nodetool compactionstats' but they might be more info there. Actually I don't really know what the 'system.compactions_in_progress' was replaced by, but any way to double check you could think of would probably help understanding better what's happening.

Does someone now the way to check pending compactions details in 3.0.9?

C*heers,
-----------------------
Alain Rodriguez - @arodream - alain@thelastpickle.com<ma...@thelastpickle.com>
France

The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com

2017-04-25 15:13 GMT+02:00 Abhishek Kumar Maheshwari <Ab...@timesinternet.in>>:
Hi All,

In Production, I am using Cassandra 3.0.9.

While I am running nodetool compactionstats command its just showing count not any other information like below:

[mohit.kundra@AdtechAppXXXX bin]$ ./nodetool -h XXX.XX.XX.XX compactionstats
pending tasks: 3
[mohit.kundra@AdtechAppXXXXX bin]$

So, this is some Cassandra bug or what? I am not able to understand.


Thanks & Regards,
Abhishek Kumar Maheshwari
+91- 9999805591<tel:+91%2099998%2005591> (Mobile)
Times Internet Ltd. | A Times of India Group Company
FC - 6, Sector 16A, Film City,  Noida,  U.P. 201301 | INDIA
P Please do not print this email unless it is absolutely necessary. Spread environmental awareness.

"Learn journalism at India's largest media house - The Times of India Group. Last Date 28 April, 2017. Visit www.tcms.in<http://www.tcms.in> for details."