You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Ben Chobot <be...@instructure.com> on 2013/03/28 16:12:28 UTC
lots of extra bytes on disk
Some of my cassandra nodes in my 1.1.5 cluster show a large discrepancy between what cassandra says the SSTables should sum up to, and what df and du claim exist. During repairs, this is almost always pretty bad, but post-repair compactions tend to bring those numbers to within a few percent of each other... usually. Sometimes they remain much further apart after compactions have finished - for instance, I'm looking at one node now that claims to have 205GB of SSTables, but actually has 450GB of files living in that CF's data directory. No pending compactions, and the most recent compaction for this CF finished just a few hours ago.
nodetool cleanup has no effect.
What could be causing these extra bytes, and how to get them to go away? I'm ok with a few extra GB of unexplained data, but an extra 245GB (more than all the data this node is supposed to have!) is a little extreme.
Re: lots of extra bytes on disk
Posted by Ben Chobot <be...@instructure.com>.
Actually, due to a misconfiguration, we weren't snapshotting at all on some of the nodes that are experiencing this problem. So while we've fixed that, snapshot don't explain the problem.
On Mar 28, 2013, at 10:54 AM, Hiller, Dean wrote:
> Have you cleaned up your snapshotsŠthose take extra space and don't just
> go away unless you delete them.
>
> Dean
>
> On 3/28/13 11:46 AM, "Ben Chobot" <be...@instructure.com> wrote:
>
>> Are you also running 1.1.5? I'm wondering (ok hoping) that this might be
>> fixed if I upgrade.
>>
>> On Mar 28, 2013, at 8:53 AM, Lanny Ripple wrote:
>>
>>> We occasionally (twice now on a 40 node cluster over the last 6-8
>>> months) see this. My best guess is that Cassandra can fail to mark an
>>> SSTable for cleanup somehow. Forced GC's or reboots don't clear them
>>> out. We disable thrift and gossip; drain; snapshot; shutdown; clear
>>> data/Keyspace/Table/*.db and restore (hard-linking back into place to
>>> avoid data transfer) from the just created snapshot; restart.
>>>
>>>
>>> On Mar 28, 2013, at 10:12 AM, Ben Chobot <be...@instructure.com> wrote:
>>>
>>>> Some of my cassandra nodes in my 1.1.5 cluster show a large
>>>> discrepancy between what cassandra says the SSTables should sum up to,
>>>> and what df and du claim exist. During repairs, this is almost always
>>>> pretty bad, but post-repair compactions tend to bring those numbers to
>>>> within a few percent of each other... usually. Sometimes they remain
>>>> much further apart after compactions have finished - for instance, I'm
>>>> looking at one node now that claims to have 205GB of SSTables, but
>>>> actually has 450GB of files living in that CF's data directory. No
>>>> pending compactions, and the most recent compaction for this CF
>>>> finished just a few hours ago.
>>>>
>>>> nodetool cleanup has no effect.
>>>>
>>>> What could be causing these extra bytes, and how to get them to go
>>>> away? I'm ok with a few extra GB of unexplained data, but an extra
>>>> 245GB (more than all the data this node is supposed to have!) is a
>>>> little extreme.
>>>
>>
>
Re: lots of extra bytes on disk
Posted by "Hiller, Dean" <De...@nrel.gov>.
Oh and since our LCS was 10MB per file it was easy to tell which files did
not convert yet. Also, we ended up blowing away a CF on node 5(of 6) and
running a full repair on that CF and after he was at a normal size again
as well.
Dean
On 3/28/13 12:35 PM, "Hiller, Dean" <De...@nrel.gov> wrote:
>We had a runaway STCS like this due to our own mistakes but were not sure
>how to clean it up. We went to LCS instead of STCS and that seemed to
>bring it way back down since the STCS had repeats and such between
>SSTables which LCS avoids mostly. I can't help much more than that info
>though.
>
>Dean
>
>On 3/28/13 12:31 PM, "Ben Chobot" <be...@instructure.com> wrote:
>
>>Sorry to make it confusing. I didn't have snapshots on some nodes; I just
>>made a snapshot on a node with this problem.
>>
>>So to be clear, on this one example node....
>> Cassandra reports ~250GB of space used
>> In a CF data directory (before snapshots existed), du -sh showed ~550GB
>> After the snapshot, du in the same directory still showed ~550GB
>>(they're hard links, so that's correct)
>> du in the snapshot directory for that CF shows ~250GB, and ls shows ~50
>>fewer files.
>>
>>
>>
>>On Mar 28, 2013, at 11:10 AM, Hiller, Dean wrote:
>>
>>> I am confused. I thought you said you don't have a snapshot. Df/du
>>> reports space used by existing data AND the snapshot. Cassandra only
>>> reports on space used by actual data........if you move the snapshots,
>>>does
>>> df/du match what cassandra says?
>>>
>>> Dean
>>>
>>> On 3/28/13 12:05 PM, "Ben Chobot" <be...@instructure.com> wrote:
>>>
>>>> .....though interestingly, the snapshot of these CFs have the "right"
>>>> amount of data in them (i.e. it agrees with the live SSTable size
>>>> reported by cassandra). Is it total insanity to remove the files from
>>>>the
>>>> data directory not included in the snapshot, so long as they were
>>>>created
>>>> before the snapshot?
>>>>
>>>> On Mar 28, 2013, at 10:54 AM, Hiller, Dean wrote:
>>>>
>>>>> Have you cleaned up your snapshotsŠthose take extra space and don't
>>>>>just
>>>>> go away unless you delete them.
>>>>>
>>>>> Dean
>>>>>
>>>>> On 3/28/13 11:46 AM, "Ben Chobot" <be...@instructure.com> wrote:
>>>>>
>>>>>> Are you also running 1.1.5? I'm wondering (ok hoping) that this
>>>>>>might
>>>>>> be
>>>>>> fixed if I upgrade.
>>>>>>
>>>>>> On Mar 28, 2013, at 8:53 AM, Lanny Ripple wrote:
>>>>>>
>>>>>>> We occasionally (twice now on a 40 node cluster over the last 6-8
>>>>>>> months) see this. My best guess is that Cassandra can fail to mark
>>>>>>>an
>>>>>>> SSTable for cleanup somehow. Forced GC's or reboots don't clear
>>>>>>>them
>>>>>>> out. We disable thrift and gossip; drain; snapshot; shutdown;
>>>>>>>clear
>>>>>>> data/Keyspace/Table/*.db and restore (hard-linking back into place
>>>>>>>to
>>>>>>> avoid data transfer) from the just created snapshot; restart.
>>>>>>>
>>>>>>>
>>>>>>> On Mar 28, 2013, at 10:12 AM, Ben Chobot <be...@instructure.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Some of my cassandra nodes in my 1.1.5 cluster show a large
>>>>>>>> discrepancy between what cassandra says the SSTables should sum up
>>>>>>>> to,
>>>>>>>> and what df and du claim exist. During repairs, this is almost
>>>>>>>>always
>>>>>>>> pretty bad, but post-repair compactions tend to bring those
>>>>>>>>numbers
>>>>>>>> to
>>>>>>>> within a few percent of each other... usually. Sometimes they
>>>>>>>>remain
>>>>>>>> much further apart after compactions have finished - for instance,
>>>>>>>> I'm
>>>>>>>> looking at one node now that claims to have 205GB of SSTables, but
>>>>>>>> actually has 450GB of files living in that CF's data directory. No
>>>>>>>> pending compactions, and the most recent compaction for this CF
>>>>>>>> finished just a few hours ago.
>>>>>>>>
>>>>>>>> nodetool cleanup has no effect.
>>>>>>>>
>>>>>>>> What could be causing these extra bytes, and how to get them to go
>>>>>>>> away? I'm ok with a few extra GB of unexplained data, but an extra
>>>>>>>> 245GB (more than all the data this node is supposed to have!) is a
>>>>>>>> little extreme.
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: lots of extra bytes on disk
Posted by "Hiller, Dean" <De...@nrel.gov>.
We had a runaway STCS like this due to our own mistakes but were not sure
how to clean it up. We went to LCS instead of STCS and that seemed to
bring it way back down since the STCS had repeats and such between
SSTables which LCS avoids mostly. I can't help much more than that info
though.
Dean
On 3/28/13 12:31 PM, "Ben Chobot" <be...@instructure.com> wrote:
>Sorry to make it confusing. I didn't have snapshots on some nodes; I just
>made a snapshot on a node with this problem.
>
>So to be clear, on this one example node....
> Cassandra reports ~250GB of space used
> In a CF data directory (before snapshots existed), du -sh showed ~550GB
> After the snapshot, du in the same directory still showed ~550GB
>(they're hard links, so that's correct)
> du in the snapshot directory for that CF shows ~250GB, and ls shows ~50
>fewer files.
>
>
>
>On Mar 28, 2013, at 11:10 AM, Hiller, Dean wrote:
>
>> I am confused. I thought you said you don't have a snapshot. Df/du
>> reports space used by existing data AND the snapshot. Cassandra only
>> reports on space used by actual data........if you move the snapshots,
>>does
>> df/du match what cassandra says?
>>
>> Dean
>>
>> On 3/28/13 12:05 PM, "Ben Chobot" <be...@instructure.com> wrote:
>>
>>> .....though interestingly, the snapshot of these CFs have the "right"
>>> amount of data in them (i.e. it agrees with the live SSTable size
>>> reported by cassandra). Is it total insanity to remove the files from
>>>the
>>> data directory not included in the snapshot, so long as they were
>>>created
>>> before the snapshot?
>>>
>>> On Mar 28, 2013, at 10:54 AM, Hiller, Dean wrote:
>>>
>>>> Have you cleaned up your snapshotsŠthose take extra space and don't
>>>>just
>>>> go away unless you delete them.
>>>>
>>>> Dean
>>>>
>>>> On 3/28/13 11:46 AM, "Ben Chobot" <be...@instructure.com> wrote:
>>>>
>>>>> Are you also running 1.1.5? I'm wondering (ok hoping) that this might
>>>>> be
>>>>> fixed if I upgrade.
>>>>>
>>>>> On Mar 28, 2013, at 8:53 AM, Lanny Ripple wrote:
>>>>>
>>>>>> We occasionally (twice now on a 40 node cluster over the last 6-8
>>>>>> months) see this. My best guess is that Cassandra can fail to mark
>>>>>>an
>>>>>> SSTable for cleanup somehow. Forced GC's or reboots don't clear
>>>>>>them
>>>>>> out. We disable thrift and gossip; drain; snapshot; shutdown; clear
>>>>>> data/Keyspace/Table/*.db and restore (hard-linking back into place
>>>>>>to
>>>>>> avoid data transfer) from the just created snapshot; restart.
>>>>>>
>>>>>>
>>>>>> On Mar 28, 2013, at 10:12 AM, Ben Chobot <be...@instructure.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Some of my cassandra nodes in my 1.1.5 cluster show a large
>>>>>>> discrepancy between what cassandra says the SSTables should sum up
>>>>>>> to,
>>>>>>> and what df and du claim exist. During repairs, this is almost
>>>>>>>always
>>>>>>> pretty bad, but post-repair compactions tend to bring those numbers
>>>>>>> to
>>>>>>> within a few percent of each other... usually. Sometimes they
>>>>>>>remain
>>>>>>> much further apart after compactions have finished - for instance,
>>>>>>> I'm
>>>>>>> looking at one node now that claims to have 205GB of SSTables, but
>>>>>>> actually has 450GB of files living in that CF's data directory. No
>>>>>>> pending compactions, and the most recent compaction for this CF
>>>>>>> finished just a few hours ago.
>>>>>>>
>>>>>>> nodetool cleanup has no effect.
>>>>>>>
>>>>>>> What could be causing these extra bytes, and how to get them to go
>>>>>>> away? I'm ok with a few extra GB of unexplained data, but an extra
>>>>>>> 245GB (more than all the data this node is supposed to have!) is a
>>>>>>> little extreme.
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: lots of extra bytes on disk
Posted by Ben Chobot <be...@instructure.com>.
Sorry to make it confusing. I didn't have snapshots on some nodes; I just made a snapshot on a node with this problem.
So to be clear, on this one example node....
Cassandra reports ~250GB of space used
In a CF data directory (before snapshots existed), du -sh showed ~550GB
After the snapshot, du in the same directory still showed ~550GB (they're hard links, so that's correct)
du in the snapshot directory for that CF shows ~250GB, and ls shows ~50 fewer files.
On Mar 28, 2013, at 11:10 AM, Hiller, Dean wrote:
> I am confused. I thought you said you don't have a snapshot. Df/du
> reports space used by existing data AND the snapshot. Cassandra only
> reports on space used by actual data........if you move the snapshots, does
> df/du match what cassandra says?
>
> Dean
>
> On 3/28/13 12:05 PM, "Ben Chobot" <be...@instructure.com> wrote:
>
>> .....though interestingly, the snapshot of these CFs have the "right"
>> amount of data in them (i.e. it agrees with the live SSTable size
>> reported by cassandra). Is it total insanity to remove the files from the
>> data directory not included in the snapshot, so long as they were created
>> before the snapshot?
>>
>> On Mar 28, 2013, at 10:54 AM, Hiller, Dean wrote:
>>
>>> Have you cleaned up your snapshotsŠthose take extra space and don't just
>>> go away unless you delete them.
>>>
>>> Dean
>>>
>>> On 3/28/13 11:46 AM, "Ben Chobot" <be...@instructure.com> wrote:
>>>
>>>> Are you also running 1.1.5? I'm wondering (ok hoping) that this might
>>>> be
>>>> fixed if I upgrade.
>>>>
>>>> On Mar 28, 2013, at 8:53 AM, Lanny Ripple wrote:
>>>>
>>>>> We occasionally (twice now on a 40 node cluster over the last 6-8
>>>>> months) see this. My best guess is that Cassandra can fail to mark an
>>>>> SSTable for cleanup somehow. Forced GC's or reboots don't clear them
>>>>> out. We disable thrift and gossip; drain; snapshot; shutdown; clear
>>>>> data/Keyspace/Table/*.db and restore (hard-linking back into place to
>>>>> avoid data transfer) from the just created snapshot; restart.
>>>>>
>>>>>
>>>>> On Mar 28, 2013, at 10:12 AM, Ben Chobot <be...@instructure.com>
>>>>> wrote:
>>>>>
>>>>>> Some of my cassandra nodes in my 1.1.5 cluster show a large
>>>>>> discrepancy between what cassandra says the SSTables should sum up
>>>>>> to,
>>>>>> and what df and du claim exist. During repairs, this is almost always
>>>>>> pretty bad, but post-repair compactions tend to bring those numbers
>>>>>> to
>>>>>> within a few percent of each other... usually. Sometimes they remain
>>>>>> much further apart after compactions have finished - for instance,
>>>>>> I'm
>>>>>> looking at one node now that claims to have 205GB of SSTables, but
>>>>>> actually has 450GB of files living in that CF's data directory. No
>>>>>> pending compactions, and the most recent compaction for this CF
>>>>>> finished just a few hours ago.
>>>>>>
>>>>>> nodetool cleanup has no effect.
>>>>>>
>>>>>> What could be causing these extra bytes, and how to get them to go
>>>>>> away? I'm ok with a few extra GB of unexplained data, but an extra
>>>>>> 245GB (more than all the data this node is supposed to have!) is a
>>>>>> little extreme.
>>>>>
>>>>
>>>
>>
>
Re: lots of extra bytes on disk
Posted by "Hiller, Dean" <De...@nrel.gov>.
I am confused. I thought you said you don't have a snapshot. Df/du
reports space used by existing data AND the snapshot. Cassandra only
reports on space used by actual data........if you move the snapshots, does
df/du match what cassandra says?
Dean
On 3/28/13 12:05 PM, "Ben Chobot" <be...@instructure.com> wrote:
>.....though interestingly, the snapshot of these CFs have the "right"
>amount of data in them (i.e. it agrees with the live SSTable size
>reported by cassandra). Is it total insanity to remove the files from the
>data directory not included in the snapshot, so long as they were created
>before the snapshot?
>
>On Mar 28, 2013, at 10:54 AM, Hiller, Dean wrote:
>
>> Have you cleaned up your snapshotsŠthose take extra space and don't just
>> go away unless you delete them.
>>
>> Dean
>>
>> On 3/28/13 11:46 AM, "Ben Chobot" <be...@instructure.com> wrote:
>>
>>> Are you also running 1.1.5? I'm wondering (ok hoping) that this might
>>>be
>>> fixed if I upgrade.
>>>
>>> On Mar 28, 2013, at 8:53 AM, Lanny Ripple wrote:
>>>
>>>> We occasionally (twice now on a 40 node cluster over the last 6-8
>>>> months) see this. My best guess is that Cassandra can fail to mark an
>>>> SSTable for cleanup somehow. Forced GC's or reboots don't clear them
>>>> out. We disable thrift and gossip; drain; snapshot; shutdown; clear
>>>> data/Keyspace/Table/*.db and restore (hard-linking back into place to
>>>> avoid data transfer) from the just created snapshot; restart.
>>>>
>>>>
>>>> On Mar 28, 2013, at 10:12 AM, Ben Chobot <be...@instructure.com>
>>>>wrote:
>>>>
>>>>> Some of my cassandra nodes in my 1.1.5 cluster show a large
>>>>> discrepancy between what cassandra says the SSTables should sum up
>>>>>to,
>>>>> and what df and du claim exist. During repairs, this is almost always
>>>>> pretty bad, but post-repair compactions tend to bring those numbers
>>>>>to
>>>>> within a few percent of each other... usually. Sometimes they remain
>>>>> much further apart after compactions have finished - for instance,
>>>>>I'm
>>>>> looking at one node now that claims to have 205GB of SSTables, but
>>>>> actually has 450GB of files living in that CF's data directory. No
>>>>> pending compactions, and the most recent compaction for this CF
>>>>> finished just a few hours ago.
>>>>>
>>>>> nodetool cleanup has no effect.
>>>>>
>>>>> What could be causing these extra bytes, and how to get them to go
>>>>> away? I'm ok with a few extra GB of unexplained data, but an extra
>>>>> 245GB (more than all the data this node is supposed to have!) is a
>>>>> little extreme.
>>>>
>>>
>>
>
Re: lots of extra bytes on disk
Posted by Ben Chobot <be...@instructure.com>.
.....though interestingly, the snapshot of these CFs have the "right" amount of data in them (i.e. it agrees with the live SSTable size reported by cassandra). Is it total insanity to remove the files from the data directory not included in the snapshot, so long as they were created before the snapshot?
On Mar 28, 2013, at 10:54 AM, Hiller, Dean wrote:
> Have you cleaned up your snapshotsŠthose take extra space and don't just
> go away unless you delete them.
>
> Dean
>
> On 3/28/13 11:46 AM, "Ben Chobot" <be...@instructure.com> wrote:
>
>> Are you also running 1.1.5? I'm wondering (ok hoping) that this might be
>> fixed if I upgrade.
>>
>> On Mar 28, 2013, at 8:53 AM, Lanny Ripple wrote:
>>
>>> We occasionally (twice now on a 40 node cluster over the last 6-8
>>> months) see this. My best guess is that Cassandra can fail to mark an
>>> SSTable for cleanup somehow. Forced GC's or reboots don't clear them
>>> out. We disable thrift and gossip; drain; snapshot; shutdown; clear
>>> data/Keyspace/Table/*.db and restore (hard-linking back into place to
>>> avoid data transfer) from the just created snapshot; restart.
>>>
>>>
>>> On Mar 28, 2013, at 10:12 AM, Ben Chobot <be...@instructure.com> wrote:
>>>
>>>> Some of my cassandra nodes in my 1.1.5 cluster show a large
>>>> discrepancy between what cassandra says the SSTables should sum up to,
>>>> and what df and du claim exist. During repairs, this is almost always
>>>> pretty bad, but post-repair compactions tend to bring those numbers to
>>>> within a few percent of each other... usually. Sometimes they remain
>>>> much further apart after compactions have finished - for instance, I'm
>>>> looking at one node now that claims to have 205GB of SSTables, but
>>>> actually has 450GB of files living in that CF's data directory. No
>>>> pending compactions, and the most recent compaction for this CF
>>>> finished just a few hours ago.
>>>>
>>>> nodetool cleanup has no effect.
>>>>
>>>> What could be causing these extra bytes, and how to get them to go
>>>> away? I'm ok with a few extra GB of unexplained data, but an extra
>>>> 245GB (more than all the data this node is supposed to have!) is a
>>>> little extreme.
>>>
>>
>
Re: lots of extra bytes on disk
Posted by "Hiller, Dean" <De...@nrel.gov>.
Have you cleaned up your snapshotsŠthose take extra space and don't just
go away unless you delete them.
Dean
On 3/28/13 11:46 AM, "Ben Chobot" <be...@instructure.com> wrote:
>Are you also running 1.1.5? I'm wondering (ok hoping) that this might be
>fixed if I upgrade.
>
>On Mar 28, 2013, at 8:53 AM, Lanny Ripple wrote:
>
>> We occasionally (twice now on a 40 node cluster over the last 6-8
>>months) see this. My best guess is that Cassandra can fail to mark an
>>SSTable for cleanup somehow. Forced GC's or reboots don't clear them
>>out. We disable thrift and gossip; drain; snapshot; shutdown; clear
>>data/Keyspace/Table/*.db and restore (hard-linking back into place to
>>avoid data transfer) from the just created snapshot; restart.
>>
>>
>> On Mar 28, 2013, at 10:12 AM, Ben Chobot <be...@instructure.com> wrote:
>>
>>> Some of my cassandra nodes in my 1.1.5 cluster show a large
>>>discrepancy between what cassandra says the SSTables should sum up to,
>>>and what df and du claim exist. During repairs, this is almost always
>>>pretty bad, but post-repair compactions tend to bring those numbers to
>>>within a few percent of each other... usually. Sometimes they remain
>>>much further apart after compactions have finished - for instance, I'm
>>>looking at one node now that claims to have 205GB of SSTables, but
>>>actually has 450GB of files living in that CF's data directory. No
>>>pending compactions, and the most recent compaction for this CF
>>>finished just a few hours ago.
>>>
>>> nodetool cleanup has no effect.
>>>
>>> What could be causing these extra bytes, and how to get them to go
>>>away? I'm ok with a few extra GB of unexplained data, but an extra
>>>245GB (more than all the data this node is supposed to have!) is a
>>>little extreme.
>>
>
Re: lots of extra bytes on disk
Posted by Ben Chobot <be...@instructure.com>.
Are you also running 1.1.5? I'm wondering (ok hoping) that this might be fixed if I upgrade.
On Mar 28, 2013, at 8:53 AM, Lanny Ripple wrote:
> We occasionally (twice now on a 40 node cluster over the last 6-8 months) see this. My best guess is that Cassandra can fail to mark an SSTable for cleanup somehow. Forced GC's or reboots don't clear them out. We disable thrift and gossip; drain; snapshot; shutdown; clear data/Keyspace/Table/*.db and restore (hard-linking back into place to avoid data transfer) from the just created snapshot; restart.
>
>
> On Mar 28, 2013, at 10:12 AM, Ben Chobot <be...@instructure.com> wrote:
>
>> Some of my cassandra nodes in my 1.1.5 cluster show a large discrepancy between what cassandra says the SSTables should sum up to, and what df and du claim exist. During repairs, this is almost always pretty bad, but post-repair compactions tend to bring those numbers to within a few percent of each other... usually. Sometimes they remain much further apart after compactions have finished - for instance, I'm looking at one node now that claims to have 205GB of SSTables, but actually has 450GB of files living in that CF's data directory. No pending compactions, and the most recent compaction for this CF finished just a few hours ago.
>>
>> nodetool cleanup has no effect.
>>
>> What could be causing these extra bytes, and how to get them to go away? I'm ok with a few extra GB of unexplained data, but an extra 245GB (more than all the data this node is supposed to have!) is a little extreme.
>
Re: lots of extra bytes on disk
Posted by Lanny Ripple <la...@spotright.com>.
We occasionally (twice now on a 40 node cluster over the last 6-8 months) see this. My best guess is that Cassandra can fail to mark an SSTable for cleanup somehow. Forced GC's or reboots don't clear them out. We disable thrift and gossip; drain; snapshot; shutdown; clear data/Keyspace/Table/*.db and restore (hard-linking back into place to avoid data transfer) from the just created snapshot; restart.
On Mar 28, 2013, at 10:12 AM, Ben Chobot <be...@instructure.com> wrote:
> Some of my cassandra nodes in my 1.1.5 cluster show a large discrepancy between what cassandra says the SSTables should sum up to, and what df and du claim exist. During repairs, this is almost always pretty bad, but post-repair compactions tend to bring those numbers to within a few percent of each other... usually. Sometimes they remain much further apart after compactions have finished - for instance, I'm looking at one node now that claims to have 205GB of SSTables, but actually has 450GB of files living in that CF's data directory. No pending compactions, and the most recent compaction for this CF finished just a few hours ago.
>
> nodetool cleanup has no effect.
>
> What could be causing these extra bytes, and how to get them to go away? I'm ok with a few extra GB of unexplained data, but an extra 245GB (more than all the data this node is supposed to have!) is a little extreme.