You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Donald Smith <Do...@audiencescience.com> on 2014/09/24 23:35:43 UTC

Would warnings about overlapping SStables explain high pending compactions?

On one of our nodes we have lots of pending compactions (499).    In the past we've seen pending compactions go up to 2400 and all the way back down again.

Investigating, I saw warnings such as the following in the logs about overlapping SStables and about needing to run "nodetool scrub" on a table.  Would the overlapping SStables explain the pending compactions?

WARN [RMI TCP Connection(2)-10.5.50.30] 2014-09-24 09:14:11,207 LeveledManifest.java (line 154) At level 1, SSTableReader(path='/data/data/XYZ/ABC/XYZ-ABC-jb-388233-Data.db') [DecoratedKey(-6112875836465333229, 3366636664393031646263356234663832383264616561666430383739383738), DecoratedKey(-4509284829153070912, 3366336562386339376664376633353635333432636662373739626465393636)] overlaps SSTableReader(path='/data/data/XYZ/ABC/XYZ-ABC_blob-jb-388150-Data.db') [DecoratedKey(-4834684725563291584, 3366336233366664363664363632666365303664333936336337343566373838), DecoratedKey(-4136919579566299218, 3366613535646662343235336335633862666530316164323232643765323934)].  This could be caused by a bug in Cassandra 1.1.0 .. 1.1.3 or due to the fact that you have dropped sstables from another node into the data directory. Sending back to L0.  If you didn't drop in sstables, and have not yet run scrub, you should do so since you may also have rows out-of-order within an sstable

Thanks

Donald A. Smith | Senior Software Engineer
P: 425.201.3900 x 3866
C: (206) 819-5965
F: (646) 443-2333
donalds@AudienceScience.com<ma...@AudienceScience.com>

[AudienceScience]

RE: Would warnings about overlapping SStables explain high pending compactions?

Posted by Donald Smith <Do...@audiencescience.com>.
Version 2.0.9.   We have 11 ongoing compactions on that node.

From: Marcus Eriksson [mailto:krummas@gmail.com]
Sent: Thursday, September 25, 2014 12:45 AM
To: user@cassandra.apache.org
Subject: Re: Would warnings about overlapping SStables explain high pending compactions?

Not really

What version are you on? Do you have pending compactions and no ongoing compactions?

/Marcus

On Wed, Sep 24, 2014 at 11:35 PM, Donald Smith <Do...@audiencescience.com>> wrote:
On one of our nodes we have lots of pending compactions (499).    In the past we’ve seen pending compactions go up to 2400 and all the way back down again.

Investigating, I saw warnings such as the following in the logs about overlapping SStables and about needing to run “nodetool scrub” on a table.  Would the overlapping SStables explain the pending compactions?

WARN [RMI TCP Connection(2)-10.5.50.30] 2014-09-24 09:14:11,207 LeveledManifest.java (line 154) At level 1, SSTableReader(path='/data/data/XYZ/ABC/XYZ-ABC-jb-388233-Data.db') [DecoratedKey(-6112875836465333229, 3366636664393031646263356234663832383264616561666430383739383738), DecoratedKey(-4509284829153070912, 3366336562386339376664376633353635333432636662373739626465393636)] overlaps SSTableReader(path='/data/data/XYZ/ABC/XYZ-ABC_blob-jb-388150-Data.db') [DecoratedKey(-4834684725563291584, 3366336233366664363664363632666365303664333936336337343566373838), DecoratedKey(-4136919579566299218, 3366613535646662343235336335633862666530316164323232643765323934)].  This could be caused by a bug in Cassandra 1.1.0 .. 1.1.3 or due to the fact that you have dropped sstables from another node into the data directory. Sending back to L0.  If you didn't drop in sstables, and have not yet run scrub, you should do so since you may also have rows out-of-order within an sstable

Thanks

Donald A. Smith | Senior Software Engineer
P: 425.201.3900 x 3866<tel:425.201.3900%20x%203866>
C: (206) 819-5965<tel:%28206%29%20819-5965>
F: (646) 443-2333<tel:%28646%29%20443-2333>
donalds@AudienceScience.com<ma...@AudienceScience.com>

[AudienceScience]

Re: Would warnings about overlapping SStables explain high pending compactions?

Posted by Marcus Eriksson <kr...@gmail.com>.
Not really

What version are you on? Do you have pending compactions and no ongoing
compactions?

/Marcus

On Wed, Sep 24, 2014 at 11:35 PM, Donald Smith <
Donald.Smith@audiencescience.com> wrote:

>  On one of our nodes we have lots of pending compactions (499).    In the
> past we’ve seen pending compactions go up to 2400 and all the way back down
> again.
>
>
>
> Investigating, I saw warnings such as the following in the logs about
> overlapping SStables and about needing to run “nodetool scrub” on a table.
> Would the overlapping SStables explain the pending compactions?
>
>
>
> WARN [RMI TCP Connection(2)-10.5.50.30] 2014-09-24 09:14:11,207
> LeveledManifest.java (line 154) At level 1,
> SSTableReader(path='/data/data/XYZ/ABC/XYZ-ABC-jb-388233-Data.db')
> [DecoratedKey(-6112875836465333229,
> 3366636664393031646263356234663832383264616561666430383739383738),
> DecoratedKey(-4509284829153070912,
> 3366336562386339376664376633353635333432636662373739626465393636)]
> overlaps
> SSTableReader(path='/data/data/XYZ/ABC/XYZ-ABC_blob-jb-388150-Data.db')
> [DecoratedKey(-4834684725563291584,
> 3366336233366664363664363632666365303664333936336337343566373838),
> DecoratedKey(-4136919579566299218,
> 3366613535646662343235336335633862666530316164323232643765323934)].  This
> could be caused by a bug in Cassandra 1.1.0 .. 1.1.3 or due to the fact
> that you have dropped sstables from another node into the data directory.
> Sending back to L0.  If you didn't drop in sstables, and have not yet run
> scrub, you should do so since you may also have rows out-of-order within an
> sstable
>
>
>
> Thanks
>
>
>
> *Donald A. Smith* | Senior Software Engineer
> P: 425.201.3900 x 3866
> C: (206) 819-5965
> F: (646) 443-2333
> donalds@AudienceScience.com
>
>
> [image: AudienceScience]
>
>
>