You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Jeff Williams <je...@wherethebitsroam.com> on 2015/07/06 16:25:32 UTC
Compaction issues, 2.0.12
Hi,
I have a keyspace which is using about 19GB on most nodes in the cluster.
However on one node the data directory for the only table in that keyspace
now uses 52GB. It seems that old sstables are not removed after compaction.
Here is a view of the data directory Data files before compaction (I didn't
think to check sizes at the time):
$ ls /var/lib/cassandra/data/trackcontent/track_content/*Data.db
trackcontent-track_content-jb-30852-Data.db
trackcontent-track_content-jb-30866-Data.db
trackcontent-track_content-jb-37136-Data.db
trackcontent-track_content-jb-43371-Data.db
trackcontent-track_content-jb-44676-Data.db
trackcontent-track_content-jb-51845-Data.db
trackcontent-track_content-jb-55339-Data.db
trackcontent-track_content-jb-57353-Data.db
trackcontent-track_content-jb-57366-Data.db
trackcontent-track_content-jb-57367-Data.db
trackcontent-track_content-jb-57368-Data.db
Then when compaction was run I got the following in the logs:
INFO [CompactionExecutor:3546] 2015-07-06 12:01:04,246 CompactionTask.java
(line 120) Compacting [
SSTableReader(path='/var/lib/cassandra/data/trackcontent/track_content/trackcontent-track_content-jb-30852-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/trackcontent/track_content/trackcontent-track_content-jb-30866-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/trackcontent/track_content/trackcontent-track_content-jb-37136-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/trackcontent/track_content/trackcontent-track_content-jb-43371-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/trackcontent/track_content/trackcontent-track_content-jb-44676-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/trackcontent/track_content/trackcontent-track_content-jb-51845-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/trackcontent/track_content/trackcontent-track_content-jb-55339-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/trackcontent/track_content/trackcontent-track_content-jb-57353-Data.db')
SSTableReader(path='/var/lib/cassandra/data/trackcontent/track_content/trackcontent-track_content-jb-57366-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/trackcontent/track_content/trackcontent-track_content-jb-57371-Data.db'),
]
INFO [CompactionExecutor:3546] 2015-07-06 13:34:31,458 CompactionTask.java
(line 296) Compacted 10 sstables to
[/var/lib/cassandra/data/trackcontent/track_content/trackcontent-track_content-jb-57372,].
36,182,766,725 bytes to 16,916,868,412 (~46% of original) in 5,607,211ms =
2.877221MB/s. 84,600,535 total partitions merged to 40,835,912. Partition
merge counts were {1:864528, 2:36318841, 3:3523852, 4:124091, 5:6051, 6:25,
}
But after compaction completes, the data directory still looks like:
$ ls -l /var/lib/cassandra/data/trackcontent/track_content/*Data.db
-rw-r--r-- 1 cassandra cassandra 16642164891 Jun 29 06:55
/var/lib/cassandra/data/trackcontent/track_content/trackcontent-track_content-jb-30852-Data.db
-rw-r--r-- 1 cassandra cassandra 17216513377 Jun 30 08:36
/var/lib/cassandra/data/trackcontent/track_content/trackcontent-track_content-jb-30866-Data.db
-rw-r--r-- 1 cassandra cassandra 813683923 Jun 30 12:03
/var/lib/cassandra/data/trackcontent/track_content/trackcontent-track_content-jb-37136-Data.db
-rw-r--r-- 1 cassandra cassandra 855070477 Jun 30 13:15
/var/lib/cassandra/data/trackcontent/track_content/trackcontent-track_content-jb-43371-Data.db
-rw-r--r-- 1 cassandra cassandra 209921645 Jun 30 13:28
/var/lib/cassandra/data/trackcontent/track_content/trackcontent-track_content-jb-44676-Data.db
-rw-r--r-- 1 cassandra cassandra 213532360 Jul 2 03:16
/var/lib/cassandra/data/trackcontent/track_content/trackcontent-track_content-jb-51845-Data.db
-rw-r--r-- 1 cassandra cassandra 14763933 Jul 2 07:58
/var/lib/cassandra/data/trackcontent/track_content/trackcontent-track_content-jb-52921-Data.db
-rw-r--r-- 1 cassandra cassandra 16955005 Jul 2 08:09
/var/lib/cassandra/data/trackcontent/track_content/trackcontent-track_content-jb-53037-Data.db
-rw-r--r-- 1 cassandra cassandra 61322156 Jul 2 23:34
/var/lib/cassandra/data/trackcontent/track_content/trackcontent-track_content-jb-55339-Data.db
-rw-r--r-- 1 cassandra cassandra 61898096 Jul 4 01:02
/var/lib/cassandra/data/trackcontent/track_content/trackcontent-track_content-jb-57353-Data.db
-rw-r--r-- 1 cassandra cassandra 75912668 Jul 5 21:23
/var/lib/cassandra/data/trackcontent/track_content/trackcontent-track_content-jb-57366-Data.db
-rw-r--r-- 1 cassandra cassandra 32747132 Jul 6 11:01
/var/lib/cassandra/data/trackcontent/track_content/trackcontent-track_content-jb-57371-Data.db
-rw-r--r-- 1 cassandra cassandra 16916868412 Jul 6 13:34
/var/lib/cassandra/data/trackcontent/track_content/trackcontent-track_content-jb-57372-Data.db
-rw-r--r-- 1 cassandra cassandra 8392714 Jul 6 13:24
/var/lib/cassandra/data/trackcontent/track_content/trackcontent-track_content-jb-57373-Data.db
So it appears that the 10 sstables that were compacted
to /var/lib/cassandra/data/trackcontent/track_content/trackcontent-track_content-jb-57372
are still sitting around, with a couple of them being very large!
Any ideas for what I can check? Can I just delete the old sstables? (I'm
guessing not).
Jeff
Re: Compaction issues, 2.0.12
Posted by Jeff Williams <je...@wherethebitsroam.com>.
Thanks Rob, Jeff. I have updated the Jira issue with my information.
On 6 July 2015 at 23:46, Jeff Ferland <jb...@tubularlabs.com> wrote:
> I’ve seen the same thing:
> https://issues.apache.org/jira/browse/CASSANDRA-9577
>
> I’ve had cases where a restart clears the old tables, and I’ve had cases
> where a restart considers the old tables to be live.
>
> On Jul 6, 2015, at 1:51 PM, Robert Coli <rc...@eventbrite.com> wrote:
>
> On Mon, Jul 6, 2015 at 1:46 PM, Jeff Williams <je...@wherethebitsroam.com>
> wrote:
>
>> 1) Cassandra version is 2.0.12.
>>
>
>
>> 2) Interesting. Looking at JMX org.apache.cassandra.db -> ColumnFamilies
>> -> trackcontent -> track_content -> Attributes, I get:
>>
>> LiveDiskSpaceUsed: 17788740448, i.e. ~17GB
>> LiveSSTableCount: 3
>> TotalDiskSpaceUsed: 55714084629, i.e. ~55GB
>>
>> So it obviously knows about the extra disk space even though the "live"
>> space looks correct. I couldn't find anything to identify the actual files
>> though.
>>
>
> That's what I would expect.
>
>
>> 3) So that was even more interesting. After restarting the cassandra
>> daemon, the sstables were not deleted and now the same JMX attributes are:
>>
>> LiveDiskSpaceUsed: 55681040579, i.e. ~55GB
>> LiveSSTableCount: 8
>> TotalDiskSpaceUsed: 55681040579, i.e. ~55GB
>>
>> So some of my non-live tables are back live again, and obviously some of
>> the big ones!!
>>
>
> This is permanently fatal to consistency; sorry if I was not clear enough
> that if they were not live, there was some risk of Cassandra considering
> them live again upon restart.
>
> If I were you, I would either stop the node and remove the files you know
> shouldn't be live or do a major compaction ASAP.
>
> The behavior you just encountered sounds like a bug, and it is a rather
> serious one. SSTables which should be dead being marked live is very bad
> for consistency.
>
> Do you see any exceptions in your logs or anything? If you can repro, you
> should file a JIRA ticket with the apache project...
>
> =Rob
>
>
>
Re: Compaction issues, 2.0.12
Posted by Jeff Ferland <jb...@tubularlabs.com>.
I’ve seen the same thing: https://issues.apache.org/jira/browse/CASSANDRA-9577 <https://issues.apache.org/jira/browse/CASSANDRA-9577>
I’ve had cases where a restart clears the old tables, and I’ve had cases where a restart considers the old tables to be live.
> On Jul 6, 2015, at 1:51 PM, Robert Coli <rc...@eventbrite.com> wrote:
>
> On Mon, Jul 6, 2015 at 1:46 PM, Jeff Williams <jeffw@wherethebitsroam.com <ma...@wherethebitsroam.com>> wrote:
> 1) Cassandra version is 2.0.12.
>
> 2) Interesting. Looking at JMX org.apache.cassandra.db -> ColumnFamilies -> trackcontent -> track_content -> Attributes, I get:
>
> LiveDiskSpaceUsed: 17788740448 <tel:17788740448>, i.e. ~17GB
> LiveSSTableCount: 3
> TotalDiskSpaceUsed: 55714084629, i.e. ~55GB
>
> So it obviously knows about the extra disk space even though the "live" space looks correct. I couldn't find anything to identify the actual files though.
>
> That's what I would expect.
>
> 3) So that was even more interesting. After restarting the cassandra daemon, the sstables were not deleted and now the same JMX attributes are:
>
> LiveDiskSpaceUsed: 55681040579, i.e. ~55GB
> LiveSSTableCount: 8
> TotalDiskSpaceUsed: 55681040579, i.e. ~55GB
>
> So some of my non-live tables are back live again, and obviously some of the big ones!!
>
> This is permanently fatal to consistency; sorry if I was not clear enough that if they were not live, there was some risk of Cassandra considering them live again upon restart.
>
> If I were you, I would either stop the node and remove the files you know shouldn't be live or do a major compaction ASAP.
>
> The behavior you just encountered sounds like a bug, and it is a rather serious one. SSTables which should be dead being marked live is very bad for consistency.
>
> Do you see any exceptions in your logs or anything? If you can repro, you should file a JIRA ticket with the apache project...
>
> =Rob
>
Re: Compaction issues, 2.0.12
Posted by Robert Coli <rc...@eventbrite.com>.
On Mon, Jul 6, 2015 at 1:46 PM, Jeff Williams <je...@wherethebitsroam.com>
wrote:
> 1) Cassandra version is 2.0.12.
>
> 2) Interesting. Looking at JMX org.apache.cassandra.db -> ColumnFamilies
> -> trackcontent -> track_content -> Attributes, I get:
>
> LiveDiskSpaceUsed: 17788740448, i.e. ~17GB
> LiveSSTableCount: 3
> TotalDiskSpaceUsed: 55714084629, i.e. ~55GB
>
> So it obviously knows about the extra disk space even though the "live"
> space looks correct. I couldn't find anything to identify the actual files
> though.
>
That's what I would expect.
> 3) So that was even more interesting. After restarting the cassandra
> daemon, the sstables were not deleted and now the same JMX attributes are:
>
> LiveDiskSpaceUsed: 55681040579, i.e. ~55GB
> LiveSSTableCount: 8
> TotalDiskSpaceUsed: 55681040579, i.e. ~55GB
>
> So some of my non-live tables are back live again, and obviously some of
> the big ones!!
>
This is permanently fatal to consistency; sorry if I was not clear enough
that if they were not live, there was some risk of Cassandra considering
them live again upon restart.
If I were you, I would either stop the node and remove the files you know
shouldn't be live or do a major compaction ASAP.
The behavior you just encountered sounds like a bug, and it is a rather
serious one. SSTables which should be dead being marked live is very bad
for consistency.
Do you see any exceptions in your logs or anything? If you can repro, you
should file a JIRA ticket with the apache project...
=Rob
Re: Compaction issues, 2.0.12
Posted by Jeff Williams <je...@wherethebitsroam.com>.
Thanks Rob.
1) Cassandra version is 2.0.12.
2) Interesting. Looking at JMX org.apache.cassandra.db -> ColumnFamilies ->
trackcontent -> track_content -> Attributes, I get:
LiveDiskSpaceUsed: 17788740448, i.e. ~17GB
LiveSSTableCount: 3
TotalDiskSpaceUsed: 55714084629, i.e. ~55GB
So it obviously knows about the extra disk space even though the "live"
space looks correct. I couldn't find anything to identify the actual files
though.
3) So that was even more interesting. After restarting the cassandra
daemon, the sstables were not deleted and now the same JMX attributes are:
LiveDiskSpaceUsed: 55681040579, i.e. ~55GB
LiveSSTableCount: 8
TotalDiskSpaceUsed: 55681040579, i.e. ~55GB
So some of my non-live tables are back live again, and obviously some of
the big ones!!
Jef
On 6 July 2015 at 20:26, Robert Coli <rc...@eventbrite.com> wrote:
> On Mon, Jul 6, 2015 at 7:25 AM, Jeff Williams <je...@wherethebitsroam.com>
> wrote:
>
>> So it appears that the 10 sstables that were compacted
>> to /var/lib/cassandra/data/trackcontent/track_content/trackcontent-track_content-jb-57372
>> are still sitting around, with a couple of them being very large!
>> Any ideas for what I can check? Can I just delete the old sstables? (I'm
>> guessing not).
>>
>
> 1) What version of Cassandra?
>
> 2) There are JMX methods in some versions to identify which files are live
> in the data dir, you could try to use those to see if those files are live.
> Does "load" seem to show that they are live from the perspective of
> Cassandra?
>
> 3) If they are not live, restarting the Cassandra daemon on the node will
> likely result in them being deleted.
>
> =Rob
>
>
Re: Compaction issues, 2.0.12
Posted by Robert Coli <rc...@eventbrite.com>.
On Mon, Jul 6, 2015 at 7:25 AM, Jeff Williams <je...@wherethebitsroam.com>
wrote:
> So it appears that the 10 sstables that were compacted
> to /var/lib/cassandra/data/trackcontent/track_content/trackcontent-track_content-jb-57372
> are still sitting around, with a couple of them being very large!
> Any ideas for what I can check? Can I just delete the old sstables? (I'm
> guessing not).
>
1) What version of Cassandra?
2) There are JMX methods in some versions to identify which files are live
in the data dir, you could try to use those to see if those files are live.
Does "load" seem to show that they are live from the perspective of
Cassandra?
3) If they are not live, restarting the Cassandra daemon on the node will
likely result in them being deleted.
=Rob