You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Huy Le <hu...@springpartners.com> on 2011/08/17 16:03:22 UTC
nodetool repair caused high disk space usage
Hi,
After upgrading to cass 0.8.4 from cass 0.6.11. I ran scrub. That worked
fine. Then I ran nodetool repair on one of the nodes. The disk usage on
data directory increased from 40GB to 480GB, and it's still growing.
The cluster has 4 nodes with replica factor 3. The ring shows:
Address DC Rack Status State Load
Owns Token
141784319550391026443072753096570088103
10.245.xxx.xxx datacenter1 rack1 Up Normal 39.89 GB
25.00% 14178431955039102644307275309657008807
10.242.xxx.xxx datacenter1 rack1 Up Normal 80.98 GB
25.00% 56713727820156410577229101238628035239
10.242.xxx.xxx datacenter1 rack1 Up Normal 80.08 GB
25.00% 99249023685273718510150927167599061671
10.244.xxx.xxx datacenter1 rack1 Up Normal 80.23 GB
25.00% 141784319550391026443072753096570088103
Does anyone know what's might be causing this issue?
Huy
--
Huy Le
Spring Partners, Inc.
http://springpadit.com
Re: nodetool repair caused high disk space usage
Posted by Huy Le <hu...@springpartners.com>.
Philippe,
Besides the system keyspace, we have only one user keyspace. However, tell
me that we can also try repairing one CF at a time.
We have two concurrent compactors configured. Will change that to one.
Huy
On Wed, Aug 17, 2011 at 6:10 PM, Philippe <wa...@gmail.com> wrote:
> Huy,
> Have you tried repairing one keyspace at a time and then giving it some
> breathing time to compact.
> My current observations is that the streams of repairs are triggering
> massive compactions which are filling up my disks too. Another idea I'd like
> to try is to limit the number of concurrent compactions because it appears
> that "Validation" are not constrained by it. That way, I would have fewer
> concurrent compactions are the disk would not fill up (as fast?)
>
>
> 2011/8/17 Huy Le <hu...@springpartners.com>
>
>> I restarted the cluster and kicked off repair on the same node again. It
>> only made the matter worse. It filled up the 830GB partition, and cassandra
>> on the node repair ran on crashed. I restarted it, and now I am running
>> compaction to reduce disk usage.
>>
>> Repair after upgrading to 0.8.4 is still a problem. Does anyone have any
>> further info on the issue? Thanks!
>>
>> Huy
>>
>>
>> On Wed, Aug 17, 2011 at 11:13 AM, Huy Le <hu...@springpartners.com>wrote:
>>
>>> Sorry for the duplicate thread. I saw the issue being referenced to
>>> https://issues.apache.org/jira/browse/CASSANDRA-2280. However, I am
>>> running version 0.8.4. I saw your comment in on of the threads that the
>>> issue is not reprocible, but multiple users have the same issue. This there
>>> anything that I should do to determine the cause of this issue for I do a
>>> rolling restart and try to run repair again? Thanks!
>>>
>>> Huy
>>>
>>>
>>> On Wed, Aug 17, 2011 at 11:03 AM, Philippe <wa...@gmail.com> wrote:
>>>
>>>> Look at my last two or three threads. I've encountered the same thing
>>>> and got some pointers/answers.
>>>> On Aug 17, 2011 4:03 PM, "Huy Le" <hu...@springpartners.com> wrote:
>>>> > Hi,
>>>> >
>>>> > After upgrading to cass 0.8.4 from cass 0.6.11. I ran scrub. That
>>>> worked
>>>> > fine. Then I ran nodetool repair on one of the nodes. The disk usage
>>>> on
>>>> > data directory increased from 40GB to 480GB, and it's still growing.
>>>> >
>>>> > The cluster has 4 nodes with replica factor 3. The ring shows:
>>>> >
>>>> > Address DC Rack Status State Load
>>>> > Owns Token
>>>> >
>>>> > 141784319550391026443072753096570088103
>>>> > 10.245.xxx.xxx datacenter1 rack1 Up Normal 39.89 GB
>>>> > 25.00% 14178431955039102644307275309657008807
>>>> > 10.242.xxx.xxx datacenter1 rack1 Up Normal 80.98 GB
>>>> > 25.00% 56713727820156410577229101238628035239
>>>> > 10.242.xxx.xxx datacenter1 rack1 Up Normal 80.08 GB
>>>> > 25.00% 99249023685273718510150927167599061671
>>>> > 10.244.xxx.xxx datacenter1 rack1 Up Normal 80.23 GB
>>>> > 25.00% 141784319550391026443072753096570088103
>>>> >
>>>> > Does anyone know what's might be causing this issue?
>>>> >
>>>> > Huy
>>>> >
>>>> > --
>>>> > Huy Le
>>>> > Spring Partners, Inc.
>>>> > http://springpadit.com
>>>>
>>>
>>>
>>>
>>> --
>>> Huy Le
>>> Spring Partners, Inc.
>>> http://springpadit.com
>>>
>>
>>
>>
>> --
>> Huy Le
>> Spring Partners, Inc.
>> http://springpadit.com
>>
>
>
--
Huy Le
Spring Partners, Inc.
http://springpadit.com
Re: nodetool repair caused high disk space usage
Posted by Philippe <wa...@gmail.com>.
Huy,
Have you tried repairing one keyspace at a time and then giving it some
breathing time to compact.
My current observations is that the streams of repairs are triggering
massive compactions which are filling up my disks too. Another idea I'd like
to try is to limit the number of concurrent compactions because it appears
that "Validation" are not constrained by it. That way, I would have fewer
concurrent compactions are the disk would not fill up (as fast?)
2011/8/17 Huy Le <hu...@springpartners.com>
> I restarted the cluster and kicked off repair on the same node again. It
> only made the matter worse. It filled up the 830GB partition, and cassandra
> on the node repair ran on crashed. I restarted it, and now I am running
> compaction to reduce disk usage.
>
> Repair after upgrading to 0.8.4 is still a problem. Does anyone have any
> further info on the issue? Thanks!
>
> Huy
>
>
> On Wed, Aug 17, 2011 at 11:13 AM, Huy Le <hu...@springpartners.com> wrote:
>
>> Sorry for the duplicate thread. I saw the issue being referenced to
>> https://issues.apache.org/jira/browse/CASSANDRA-2280. However, I am
>> running version 0.8.4. I saw your comment in on of the threads that the
>> issue is not reprocible, but multiple users have the same issue. This there
>> anything that I should do to determine the cause of this issue for I do a
>> rolling restart and try to run repair again? Thanks!
>>
>> Huy
>>
>>
>> On Wed, Aug 17, 2011 at 11:03 AM, Philippe <wa...@gmail.com> wrote:
>>
>>> Look at my last two or three threads. I've encountered the same thing and
>>> got some pointers/answers.
>>> On Aug 17, 2011 4:03 PM, "Huy Le" <hu...@springpartners.com> wrote:
>>> > Hi,
>>> >
>>> > After upgrading to cass 0.8.4 from cass 0.6.11. I ran scrub. That
>>> worked
>>> > fine. Then I ran nodetool repair on one of the nodes. The disk usage on
>>> > data directory increased from 40GB to 480GB, and it's still growing.
>>> >
>>> > The cluster has 4 nodes with replica factor 3. The ring shows:
>>> >
>>> > Address DC Rack Status State Load
>>> > Owns Token
>>> >
>>> > 141784319550391026443072753096570088103
>>> > 10.245.xxx.xxx datacenter1 rack1 Up Normal 39.89 GB
>>> > 25.00% 14178431955039102644307275309657008807
>>> > 10.242.xxx.xxx datacenter1 rack1 Up Normal 80.98 GB
>>> > 25.00% 56713727820156410577229101238628035239
>>> > 10.242.xxx.xxx datacenter1 rack1 Up Normal 80.08 GB
>>> > 25.00% 99249023685273718510150927167599061671
>>> > 10.244.xxx.xxx datacenter1 rack1 Up Normal 80.23 GB
>>> > 25.00% 141784319550391026443072753096570088103
>>> >
>>> > Does anyone know what's might be causing this issue?
>>> >
>>> > Huy
>>> >
>>> > --
>>> > Huy Le
>>> > Spring Partners, Inc.
>>> > http://springpadit.com
>>>
>>
>>
>>
>> --
>> Huy Le
>> Spring Partners, Inc.
>> http://springpadit.com
>>
>
>
>
> --
> Huy Le
> Spring Partners, Inc.
> http://springpadit.com
>
Re: nodetool repair caused high disk space usage
Posted by Huy Le <hu...@springpartners.com>.
I restarted the cluster and kicked off repair on the same node again. It
only made the matter worse. It filled up the 830GB partition, and cassandra
on the node repair ran on crashed. I restarted it, and now I am running
compaction to reduce disk usage.
Repair after upgrading to 0.8.4 is still a problem. Does anyone have any
further info on the issue? Thanks!
Huy
On Wed, Aug 17, 2011 at 11:13 AM, Huy Le <hu...@springpartners.com> wrote:
> Sorry for the duplicate thread. I saw the issue being referenced to
> https://issues.apache.org/jira/browse/CASSANDRA-2280. However, I am
> running version 0.8.4. I saw your comment in on of the threads that the
> issue is not reprocible, but multiple users have the same issue. This there
> anything that I should do to determine the cause of this issue for I do a
> rolling restart and try to run repair again? Thanks!
>
> Huy
>
>
> On Wed, Aug 17, 2011 at 11:03 AM, Philippe <wa...@gmail.com> wrote:
>
>> Look at my last two or three threads. I've encountered the same thing and
>> got some pointers/answers.
>> On Aug 17, 2011 4:03 PM, "Huy Le" <hu...@springpartners.com> wrote:
>> > Hi,
>> >
>> > After upgrading to cass 0.8.4 from cass 0.6.11. I ran scrub. That worked
>> > fine. Then I ran nodetool repair on one of the nodes. The disk usage on
>> > data directory increased from 40GB to 480GB, and it's still growing.
>> >
>> > The cluster has 4 nodes with replica factor 3. The ring shows:
>> >
>> > Address DC Rack Status State Load
>> > Owns Token
>> >
>> > 141784319550391026443072753096570088103
>> > 10.245.xxx.xxx datacenter1 rack1 Up Normal 39.89 GB
>> > 25.00% 14178431955039102644307275309657008807
>> > 10.242.xxx.xxx datacenter1 rack1 Up Normal 80.98 GB
>> > 25.00% 56713727820156410577229101238628035239
>> > 10.242.xxx.xxx datacenter1 rack1 Up Normal 80.08 GB
>> > 25.00% 99249023685273718510150927167599061671
>> > 10.244.xxx.xxx datacenter1 rack1 Up Normal 80.23 GB
>> > 25.00% 141784319550391026443072753096570088103
>> >
>> > Does anyone know what's might be causing this issue?
>> >
>> > Huy
>> >
>> > --
>> > Huy Le
>> > Spring Partners, Inc.
>> > http://springpadit.com
>>
>
>
>
> --
> Huy Le
> Spring Partners, Inc.
> http://springpadit.com
>
--
Huy Le
Spring Partners, Inc.
http://springpadit.com
Re: nodetool repair caused high disk space usage
Posted by Huy Le <hu...@springpartners.com>.
Sorry for the duplicate thread. I saw the issue being referenced to
https://issues.apache.org/jira/browse/CASSANDRA-2280. However, I am
running version 0.8.4. I saw your comment in on of the threads that the
issue is not reprocible, but multiple users have the same issue. This there
anything that I should do to determine the cause of this issue for I do a
rolling restart and try to run repair again? Thanks!
Huy
On Wed, Aug 17, 2011 at 11:03 AM, Philippe <wa...@gmail.com> wrote:
> Look at my last two or three threads. I've encountered the same thing and
> got some pointers/answers.
> On Aug 17, 2011 4:03 PM, "Huy Le" <hu...@springpartners.com> wrote:
> > Hi,
> >
> > After upgrading to cass 0.8.4 from cass 0.6.11. I ran scrub. That worked
> > fine. Then I ran nodetool repair on one of the nodes. The disk usage on
> > data directory increased from 40GB to 480GB, and it's still growing.
> >
> > The cluster has 4 nodes with replica factor 3. The ring shows:
> >
> > Address DC Rack Status State Load
> > Owns Token
> >
> > 141784319550391026443072753096570088103
> > 10.245.xxx.xxx datacenter1 rack1 Up Normal 39.89 GB
> > 25.00% 14178431955039102644307275309657008807
> > 10.242.xxx.xxx datacenter1 rack1 Up Normal 80.98 GB
> > 25.00% 56713727820156410577229101238628035239
> > 10.242.xxx.xxx datacenter1 rack1 Up Normal 80.08 GB
> > 25.00% 99249023685273718510150927167599061671
> > 10.244.xxx.xxx datacenter1 rack1 Up Normal 80.23 GB
> > 25.00% 141784319550391026443072753096570088103
> >
> > Does anyone know what's might be causing this issue?
> >
> > Huy
> >
> > --
> > Huy Le
> > Spring Partners, Inc.
> > http://springpadit.com
>
--
Huy Le
Spring Partners, Inc.
http://springpadit.com
Re: nodetool repair caused high disk space usage
Posted by Philippe <wa...@gmail.com>.
Look at my last two or three threads. I've encountered the same thing and
got some pointers/answers.
On Aug 17, 2011 4:03 PM, "Huy Le" <hu...@springpartners.com> wrote:
> Hi,
>
> After upgrading to cass 0.8.4 from cass 0.6.11. I ran scrub. That worked
> fine. Then I ran nodetool repair on one of the nodes. The disk usage on
> data directory increased from 40GB to 480GB, and it's still growing.
>
> The cluster has 4 nodes with replica factor 3. The ring shows:
>
> Address DC Rack Status State Load
> Owns Token
>
> 141784319550391026443072753096570088103
> 10.245.xxx.xxx datacenter1 rack1 Up Normal 39.89 GB
> 25.00% 14178431955039102644307275309657008807
> 10.242.xxx.xxx datacenter1 rack1 Up Normal 80.98 GB
> 25.00% 56713727820156410577229101238628035239
> 10.242.xxx.xxx datacenter1 rack1 Up Normal 80.08 GB
> 25.00% 99249023685273718510150927167599061671
> 10.244.xxx.xxx datacenter1 rack1 Up Normal 80.23 GB
> 25.00% 141784319550391026443072753096570088103
>
> Does anyone know what's might be causing this issue?
>
> Huy
>
> --
> Huy Le
> Spring Partners, Inc.
> http://springpadit.com
Re: nodetool repair caused high disk space usage
Posted by Philippe <wa...@gmail.com>.
>
> Do you have an indication that at least the disk space is in fact
> consistent with the amount of data being streamed between the nodes? I
> think you had 90 -> ~ 450 gig with RF=3, right? Still sounds like a
> lot assuming repairs are not running concurrently (and compactions are
> able to run after a repair before the next repair of a neighbor
> starts).
>
Hi Peter,
When a repair was running on the 40GB keyspace I'd usually see range repairs
for about up to a couple thousand ranges for each CF. If range = #keys then
that's a very small amount of data being moved around.
However, at the time, I hadn't noticed that there were multiple repairs
running concurrently on the same nodes and on the neighbors so I suppose my
experience is invalid for possibly finding a bug. But I suspect it will help
someone out along the way because they'll have multiple repairs going on too
and I have a much better understanding of what's going on myself.
I've reloaded all my data in my cluster now, the load is 140GB on each node
and I've been able to run a repair on each CF that comes out almost 100%
consistent. I'm now starting to run the daily repair crons again to see if
they go out of whack or not.
Re: nodetool repair caused high disk space usage
Posted by Peter Schuller <pe...@infidyne.com>.
> In our case they get created exclusively during repairs. Compactionstats
> showed a huge number of sstable build compactions
Do you have an indication that at least the disk space is in fact
consistent with the amount of data being streamed between the nodes? I
think you had 90 -> ~ 450 gig with RF=3, right? Still sounds like a
lot assuming repairs are not running concurrently (and compactions are
able to run after a repair before the next repair of a neighbor
starts).
--
/ Peter Schuller (@scode on twitter)
Re: nodetool repair caused high disk space usage
Posted by Philippe <wa...@gmail.com>.
Péter,
In our case they get created exclusively during repairs. Compactionstats
showed a huge number of sstable build compactions
On Aug 20, 2011 1:23 AM, "Peter Schuller" <pe...@infidyne.com>
wrote:
>> Is there any chance that the entire file from source node got streamed to
>> destination node even though only small amount of data in hte file from
>> source node is supposed to be streamed destination node?
>
> Yes, but the thing that's annoying me is that even if so - you should
> not be seeing a 40 gb -> hundreds of gig increase even if all
> neighbors sent all their data.
>
> Can you check system.log for references to these sstables to see when
> and under what circumstances they got written?
>
> --
> / Peter Schuller (@scode on twitter)
Re: nodetool repair caused high disk space usage
Posted by Huy Le <hu...@springpartners.com>.
After having done so many tries, I am not sure which log entries correspond
to what. However, there were many of this type:
WARN [CompactionExecutor:14] 2011-08-18 18:47:00,596 CompactionManager.java
(line 730) Index file contained a different key or row size; using key from
data file
And there were out of disk space errors (because hundreds of gigs were used
up).
Anyhow, disk space usage is under control now, at least for 3 nodes so far,
the repair the last node is still running . Here is what I did that led to
the disk space usage under control:
* Shutdown cass
* Restored data from backup for 0.6.11.
* Started up cass 0.6.11
* Ran repair on all nodes, one at a time. After repair, the node that
showed up in ring as having 40GB of data before, now have 120GB of data.
All other nodes showed the same amount of data. Not much disk space usage
increase compare to what seen before. Just some Compacted files.
* Ran compact on all nodes
* Drained all nodes
* Shutdown cass
* Started up cass with version 0.8.4
* Applied schema to 0.8.4
* Ran scrub on all nodes
* Ran repair on each of the 4 nodes, one at at time (repair on the last node
is still running). Data size show up in ring the same size as it was in
cass 0.6.11. No disk space usage increase.
So it seems like data was in inconsistent state before it was upgraded to
cass 0.8.4, and some how that triggered cass 0.8.4 repair to cause disk
usage out of control. Or may be data is already consistent across the
nodes, and now running repair does not do any kind of data transfer.
Once the repair for this last node is completed, I will start populating
some data by using our application. While using the app, I will randomly
restart few nodes, one at a time to cause data missing on some nodes. Then
I will run repair again to see if the disk usage still under control.
Huy
On Fri, Aug 19, 2011 at 7:22 PM, Peter Schuller <peter.schuller@infidyne.com
> wrote:
> > Is there any chance that theentire file from source node got streamed to
> > destination node even though only small amount of data in hte file from
> > source node is supposed to be streamed destination node?
>
> Yes, but the thing that's annoying me is that even if so - you should
> not be seeing a 40 gb -> hundreds of gig increase even if all
> neighbors sent all their data.
>
> Can you check system.log for references to these sstables to see when
> and under what circumstances they got written?
>
> --
> / Peter Schuller (@scode on twitter)
>
--
Huy Le
Spring Partners, Inc.
http://springpadit.com
Re: nodetool repair caused high disk space usage
Posted by Héctor Izquierdo Seliva <iz...@strands.com>.
El sáb, 20-08-2011 a las 01:22 +0200, Peter Schuller escribió:
> > Is there any chance that the entire file from source node got streamed to
> > destination node even though only small amount of data in hte file from
> > source node is supposed to be streamed destination node?
>
> Yes, but the thing that's annoying me is that even if so - you should
> not be seeing a 40 gb -> hundreds of gig increase even if all
> neighbors sent all their data.
>
I'm having the very same issue. Trying to repair a node with 90GB of
data fills up the 1.5TB drive, and it's still trying to send more data.
This is on 0.8.1.
Re: nodetool repair caused high disk space usage
Posted by Peter Schuller <pe...@infidyne.com>.
> Is there any chance that the entire file from source node got streamed to
> destination node even though only small amount of data in hte file from
> source node is supposed to be streamed destination node?
Yes, but the thing that's annoying me is that even if so - you should
not be seeing a 40 gb -> hundreds of gig increase even if all
neighbors sent all their data.
Can you check system.log for references to these sstables to see when
and under what circumstances they got written?
--
/ Peter Schuller (@scode on twitter)
Re: nodetool repair caused high disk space usage
Posted by Huy Le <hu...@springpartners.com>.
>
> To confirm - are you saying the data directory size is huge, but the
> live size as reported by nodetool ring and nodetool info does NOT
> reflect this inflated size?
>
That's correct.
> What files *do* you have in the data directory? Any left-over *tmp*
> files for example?
>
>
The files that consumed most do disk space were of the CF that was 23+GB.
There were multiple copies of 23GB files of that CF, and they did not have
any Compacted file associated with them nor were they of *tmp* files.
> Are you sure you're only running a single repair at a time? (Sorry if
> this was covered, I did a quick swipe through thread history because I
> was unsure whether I was confusing two different threads, and I don't
> think so.)
>
>
Absolutely sure I was running single repair. In fact, after few data reset
(restore from backup), and retries (reset data on each retry, except one
time I did not reset data for different test) on the same node, I thought
may be there was some kind of hardware issue but OS not reported; so I reset
data and try repair on diff node, I ran into the same issue.
This morning I restored data from backup and kicked off repair with version
0.6.11, and it worked fine.
> The question is what's taking the space. If it's sstables, they really
> should be either compacted onces that are marked for deletion but
> being retained, or "live" sstables in which case they should show up
> as load in nodetool.
>
>
I also tested compaction once, and it did reduce the size, but I could not
wait for it to finish as the size of the directory containing the sstables
grew over 700GB, and it was just taking too long to compact down to the
expected size.
> What else... maybe streams are being re-tried from the source nodes
> and the disk space is coming from a bunch of half-finished streams of
> the same data. But if so, those should be *tmp* files IIRC.
>
As mentioned above, most of disk space used were from multiple files of a
big CF.
Is there any chance that the entire file from source node got streamed to
destination node even though only small amount of data in hte file from
source node is supposed to be streamed destination node?
>
> I'm just wildly speculation, but it would be nice to get to the bottom of
> this.
>
> --
> / Peter Schuller (@scode on twitter)
>
--
Huy Le
Spring Partners, Inc.
http://springpadit.com
Re: nodetool repair caused high disk space usage
Posted by Peter Schuller <pe...@infidyne.com>.
> There were few Compacted files. I thought that might have been the cause,
> but it wasn't it. We have a CF that is 23GB, and while repair is running,
> there are multiple instances of that CF created along with other CFs.
To confirm - are you saying the data directory size is huge, but the
live size as reported by nodetool ring and nodetool info does NOT
reflect this inflated size?
What files *do* you have in the data directory? Any left-over *tmp*
files for example?
Are you sure you're only running a single repair at a time? (Sorry if
this was covered, I did a quick swipe through thread history because I
was unsure whether I was confusing two different threads, and I don't
think so.)
The question is what's taking the space. If it's sstables, they really
should be either compacted onces that are marked for deletion but
being retained, or "live" sstables in which case they should show up
as load in nodetool.
What else... maybe streams are being re-tried from the source nodes
and the disk space is coming from a bunch of half-finished streams of
the same data. But if so, those should be *tmp* files IIRC.
I'm just wildly speculation, but it would be nice to get to the bottom of this.
--
/ Peter Schuller (@scode on twitter)
Re: nodetool repair caused high disk space usage
Posted by Huy Le <hu...@springpartners.com>.
There were few Compacted files. I thought that might have been the cause,
but it wasn't it. We have a CF that is 23GB, and while repair is running,
there are multiple instances of that CF created along with other CFs.
I checked the stream directory across cluster of four nodes, but it was
empty.
I can not reproduce this issue in version 0.6.11 with a copy of data backed
up prior to 0.8.4 upgrade.
Currently repair is still running on two 0.6.11 nodes. My plan is to run
compact across the cluster running 0.6.11. When done, make another attempt
to upgrade it to 0.8.4.
Huy
On Fri, Aug 19, 2011 at 2:26 PM, Peter Schuller <peter.schuller@infidyne.com
> wrote:
> > After upgrading to cass 0.8.4 from cass 0.6.11. I ran scrub. That
> worked
> > fine. Then I ran nodetool repair on one of the nodes. The disk usage on
> > data directory increased from 40GB to 480GB, and it's still growing.
>
> If you check your data directory, does it contain a lot of
> "*Compacted" files? It sounds like you're churning sstables from a
> combination of compactions/flushes (including triggered by repair) and
> the old ones aren't being deleted. I wonder if there is still some
> issue causing sstable retention
>
> Since you're on 0.8.4, I'm a bit suspicious. I'd have to re-check each
> JIRA but I think the major known repair problems should be fixed
> except for CASSANDRA-2280 which is not your problem since you're going
> form a total load of 40 gig to hundreds of gigs (so even with all
> cf:s streaming, that's unexpected).
>
> Do you have any old left-over streams active on the nodes? "nodetool
> netstats". If there are "stuck" streams, they might be causing sstable
> retention beyond what you'd expect.
>
> --
> / Peter Schuller (@scode on twitter)
>
--
Huy Le
Spring Partners, Inc.
http://springpadit.com
Re: nodetool repair caused high disk space usage
Posted by Peter Schuller <pe...@infidyne.com>.
> After upgrading to cass 0.8.4 from cass 0.6.11. I ran scrub. That worked
> fine. Then I ran nodetool repair on one of the nodes. The disk usage on
> data directory increased from 40GB to 480GB, and it's still growing.
If you check your data directory, does it contain a lot of
"*Compacted" files? It sounds like you're churning sstables from a
combination of compactions/flushes (including triggered by repair) and
the old ones aren't being deleted. I wonder if there is still some
issue causing sstable retention
Since you're on 0.8.4, I'm a bit suspicious. I'd have to re-check each
JIRA but I think the major known repair problems should be fixed
except for CASSANDRA-2280 which is not your problem since you're going
form a total load of 40 gig to hundreds of gigs (so even with all
cf:s streaming, that's unexpected).
Do you have any old left-over streams active on the nodes? "nodetool
netstats". If there are "stuck" streams, they might be causing sstable
retention beyond what you'd expect.
--
/ Peter Schuller (@scode on twitter)