You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by John Wong <go...@gmail.com> on 2016/08/08 14:22:10 UTC
compactions stuck and restart doesn't kill it
Hi
We have a compaction stuck. No progress ever made.
nodetool compactionstats
pending tasks: 2
compaction type keyspace column family
completed total unit progress
Compaction my_columnfamily 0
410383527 bytes 0.00%
Active compaction remaining time : 0h00m06s
I tried nodetool stop compactions --COMPACTION but that didn't do anything.
We also tried multiple node restarts, but a few mins later compactions will
come back. What can we do?
We are still at version 1.2.19 unfortunately (we have plan to move
forward...)
Thanks.
John
Re: compactions stuck and restart doesn't kill it
Posted by Surbhi Gupta <su...@gmail.com>.
I have faced similar kind of issue while we were using LCS ...
We ran the scrub and it was helpful to clear the corruption and compaction
caught up well after that ...
On 8 August 2016 at 13:23, John Wong <go...@gmail.com> wrote:
> Hi.
>
> I don't see any error logs indicate corrupted sstable error. Is it safe to
> run this?
>
> Thanks.
>
> On Mon, Aug 8, 2016 at 1:18 PM, Surbhi Gupta <su...@gmail.com>
> wrote:
>
>> Can you see if any of the sstable is corrupt ?
>> I have seen with my past experience , if any of the sstable which is
>> part of the compaction is corrupt then compaction kind of hangs.
>> Try to run nodetool scrub ...
>>
>> On 8 August 2016 at 09:34, John Wong <go...@gmail.com> wrote:
>>
>>> On Mon, Aug 8, 2016 at 11:16 AM, Surbhi Gupta <su...@gmail.com>
>>> wrote:
>>>
>>>> Once you restart a node compaction will start automatically, if u dont
>>>> want to do so then do
>>>> nodetool disableautocompaction as soon as node is started .
>>>>
>>>>
>>> Thanks. I certainly can give that a try for the specific column family,
>>> but even so this would be a temporary solution, as the root issue is the
>>> compaction won't finish. I see nothing in the log. FYI, gathering more info
>>> from the developer, this CF stores large rows.
>>>
>>> If anyone ever encountered similar issue, please let me know.
>>>
>>> John
>>>
>>
>>
>
Re: compactions stuck and restart doesn't kill it
Posted by John Wong <go...@gmail.com>.
Hi.
I don't see any error logs indicate corrupted sstable error. Is it safe to
run this?
Thanks.
On Mon, Aug 8, 2016 at 1:18 PM, Surbhi Gupta <su...@gmail.com>
wrote:
> Can you see if any of the sstable is corrupt ?
> I have seen with my past experience , if any of the sstable which is part
> of the compaction is corrupt then compaction kind of hangs.
> Try to run nodetool scrub ...
>
> On 8 August 2016 at 09:34, John Wong <go...@gmail.com> wrote:
>
>> On Mon, Aug 8, 2016 at 11:16 AM, Surbhi Gupta <su...@gmail.com>
>> wrote:
>>
>>> Once you restart a node compaction will start automatically, if u dont
>>> want to do so then do
>>> nodetool disableautocompaction as soon as node is started .
>>>
>>>
>> Thanks. I certainly can give that a try for the specific column family,
>> but even so this would be a temporary solution, as the root issue is the
>> compaction won't finish. I see nothing in the log. FYI, gathering more info
>> from the developer, this CF stores large rows.
>>
>> If anyone ever encountered similar issue, please let me know.
>>
>> John
>>
>
>
Re: compactions stuck and restart doesn't kill it
Posted by Surbhi Gupta <su...@gmail.com>.
Can you see if any of the sstable is corrupt ?
I have seen with my past experience , if any of the sstable which is part
of the compaction is corrupt then compaction kind of hangs.
Try to run nodetool scrub ...
On 8 August 2016 at 09:34, John Wong <go...@gmail.com> wrote:
> On Mon, Aug 8, 2016 at 11:16 AM, Surbhi Gupta <su...@gmail.com>
> wrote:
>
>> Once you restart a node compaction will start automatically, if u dont
>> want to do so then do
>> nodetool disableautocompaction as soon as node is started .
>>
>>
> Thanks. I certainly can give that a try for the specific column family,
> but even so this would be a temporary solution, as the root issue is the
> compaction won't finish. I see nothing in the log. FYI, gathering more info
> from the developer, this CF stores large rows.
>
> If anyone ever encountered similar issue, please let me know.
>
> John
>
Re: compactions stuck and restart doesn't kill it
Posted by John Wong <go...@gmail.com>.
On Mon, Aug 8, 2016 at 11:16 AM, Surbhi Gupta <su...@gmail.com>
wrote:
> Once you restart a node compaction will start automatically, if u dont
> want to do so then do
> nodetool disableautocompaction as soon as node is started .
>
>
Thanks. I certainly can give that a try for the specific column family, but
even so this would be a temporary solution, as the root issue is the
compaction won't finish. I see nothing in the log. FYI, gathering more info
from the developer, this CF stores large rows.
If anyone ever encountered similar issue, please let me know.
John
Re: compactions stuck and restart doesn't kill it
Posted by Surbhi Gupta <su...@gmail.com>.
Once you restart a node compaction will start automatically, if u dont want
to do so then do
nodetool disableautocompaction as soon as node is started .
On 8 August 2016 at 07:22, John Wong <go...@gmail.com> wrote:
> Hi
>
> We have a compaction stuck. No progress ever made.
>
> nodetool compactionstats
> pending tasks: 2
> compaction type keyspace column family
> completed total unit progress
> Compaction my_columnfamily 0
> 410383527 bytes 0.00%
> Active compaction remaining time : 0h00m06s
>
> I tried nodetool stop compactions --COMPACTION but that didn't do anything.
>
> We also tried multiple node restarts, but a few mins later compactions
> will come back. What can we do?
>
> We are still at version 1.2.19 unfortunately (we have plan to move
> forward...)
>
> Thanks.
>
> John
>