You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Mark <st...@gmail.com> on 2010/12/08 05:44:13 UTC
Ran out of space during cleanup.. HELP
3 Node cluster and I just ran a nodetool cleanup on node #3. 1 and 2 are
now at 100% disk space. What should I do?
Re: Ran out of space during cleanup.. HELP
Posted by Mark <st...@gmail.com>.
Does cassandra suffer from this same issue in 0.7? You would think it
would at least warn if not prevent anticompaction if it knows there is a
good chance of running out of space.
On 12/8/10 8:05 AM, Jake Maizel wrote:
> I was in a similar situation and luckily had snapshots to clear and
> gain space but you are correct. I would be careful of using the disk
> more than 50% as the anit-compaction during cleanup could fail.
>
> I don't have any experience with adding a data directory on the fly.
>
> On Wed, Dec 8, 2010 at 4:51 PM, Mark<st...@gmail.com> wrote:
>> Did both but didn't seem to help. I have another drive on that machine with
>> some free space. If I add another directory to the DataFileDirectory config
>> and restart, will it start using that directory?
>>
>> Anything else I can do?
>>
>> This actually leads me to an important question. Should I always make sure
>> that Cassandra doesn't get past 50% of the drives free space, otherwise an
>> anticompaction like this can just destroy the machine?
>>
>> On 12/8/10 1:12 AM, Jake Maizel wrote:
>>> Also, look for any snapshots that can be cleared with nodetool
>>> clearsnapshot or just run the command to remove any that exist.
>>>
>>> On Wed, Dec 8, 2010 at 9:04 AM, Oleg Anastasyev<ol...@gmail.com>
>>> wrote:
>>>> Mark<static.void.dev<at> gmail.com> writes:
>>>>
>>>>> Caused by: java.lang.RuntimeException: Insufficient disk space to flush
>>>>> at
>>>>> On 12/7/10 8:44 PM, Mark wrote:
>>>>>> 3 Node cluster and I just ran a nodetool cleanup on node #3. 1 and 2
>>>>>> are now at 100% disk space. What should I do?
>>>> Is there files with -tmp n their names ?
>>>> Try to remove them to free up disk space.
>>>>
>>>>
>>>>
>>>
>
>
Re: Ran out of space during cleanup.. HELP
Posted by Jake Maizel <ja...@soundcloud.com>.
I was in a similar situation and luckily had snapshots to clear and
gain space but you are correct. I would be careful of using the disk
more than 50% as the anit-compaction during cleanup could fail.
I don't have any experience with adding a data directory on the fly.
On Wed, Dec 8, 2010 at 4:51 PM, Mark <st...@gmail.com> wrote:
> Did both but didn't seem to help. I have another drive on that machine with
> some free space. If I add another directory to the DataFileDirectory config
> and restart, will it start using that directory?
>
> Anything else I can do?
>
> This actually leads me to an important question. Should I always make sure
> that Cassandra doesn't get past 50% of the drives free space, otherwise an
> anticompaction like this can just destroy the machine?
>
> On 12/8/10 1:12 AM, Jake Maizel wrote:
>>
>> Also, look for any snapshots that can be cleared with nodetool
>> clearsnapshot or just run the command to remove any that exist.
>>
>> On Wed, Dec 8, 2010 at 9:04 AM, Oleg Anastasyev<ol...@gmail.com>
>> wrote:
>>>
>>> Mark<static.void.dev<at> gmail.com> writes:
>>>
>>>> Caused by: java.lang.RuntimeException: Insufficient disk space to flush
>>>> at
>>>> On 12/7/10 8:44 PM, Mark wrote:
>>>>>
>>>>> 3 Node cluster and I just ran a nodetool cleanup on node #3. 1 and 2
>>>>> are now at 100% disk space. What should I do?
>>>>
>>>
>>> Is there files with -tmp n their names ?
>>> Try to remove them to free up disk space.
>>>
>>>
>>>
>>
>>
>
--
Jake Maizel
Network Operations
Soundcloud
Mail & GTalk: jake@soundcloud.com
Skype: jakecloud
Rosenthaler strasse 13, 101 19, Berlin, DE
Re: Ran out of space during cleanup.. HELP
Posted by Mark <st...@gmail.com>.
Did both but didn't seem to help. I have another drive on that machine
with some free space. If I add another directory to the
DataFileDirectory config and restart, will it start using that directory?
Anything else I can do?
This actually leads me to an important question. Should I always make
sure that Cassandra doesn't get past 50% of the drives free space,
otherwise an anticompaction like this can just destroy the machine?
On 12/8/10 1:12 AM, Jake Maizel wrote:
> Also, look for any snapshots that can be cleared with nodetool
> clearsnapshot or just run the command to remove any that exist.
>
> On Wed, Dec 8, 2010 at 9:04 AM, Oleg Anastasyev<ol...@gmail.com> wrote:
>> Mark<static.void.dev<at> gmail.com> writes:
>>
>>> Caused by: java.lang.RuntimeException: Insufficient disk space to flush
>>> at
>>> On 12/7/10 8:44 PM, Mark wrote:
>>>> 3 Node cluster and I just ran a nodetool cleanup on node #3. 1 and 2
>>>> are now at 100% disk space. What should I do?
>>>
>>
>> Is there files with -tmp n their names ?
>> Try to remove them to free up disk space.
>>
>>
>>
>
>
Re: Ran out of space during cleanup.. HELP
Posted by Jake Maizel <ja...@soundcloud.com>.
Also, look for any snapshots that can be cleared with nodetool
clearsnapshot or just run the command to remove any that exist.
On Wed, Dec 8, 2010 at 9:04 AM, Oleg Anastasyev <ol...@gmail.com> wrote:
>
> Mark <static.void.dev <at> gmail.com> writes:
>
>> Caused by: java.lang.RuntimeException: Insufficient disk space to flush
>> at
>> >
>> On 12/7/10 8:44 PM, Mark wrote:
>> > 3 Node cluster and I just ran a nodetool cleanup on node #3. 1 and 2
>> > are now at 100% disk space. What should I do?
>>
>>
>
>
> Is there files with -tmp n their names ?
> Try to remove them to free up disk space.
>
>
>
--
Jake Maizel
Network Operations
Soundcloud
Mail & GTalk: jake@soundcloud.com
Skype: jakecloud
Rosenthaler strasse 13, 101 19, Berlin, DE
Re: Ran out of space during cleanup.. HELP
Posted by Oleg Anastasyev <ol...@gmail.com>.
Mark <static.void.dev <at> gmail.com> writes:
> Caused by: java.lang.RuntimeException: Insufficient disk space to flush
> at
> >
> On 12/7/10 8:44 PM, Mark wrote:
> > 3 Node cluster and I just ran a nodetool cleanup on node #3. 1 and 2
> > are now at 100% disk space. What should I do?
>
>
Is there files with -tmp n their names ?
Try to remove them to free up disk space.
Re: Ran out of space during cleanup.. HELP
Posted by Mark <st...@gmail.com>.
Tried restarting node #1 and now received:
Caused by: java.lang.RuntimeException: Insufficient disk space to flush
at
org.apache.cassandra.db.ColumnFamilyStore.getFlushPath(ColumnFamilyStore.java:442)
at
org.apache.cassandra.db.Memtable.writeSortedContents(Memtable.java:149)
at org.apache.cassandra.db.Memtable.access$000(Memtable.java:42)
at org.apache.cassandra.db.Memtable$1.runMayThrow(Memtable.java:173)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
... 6 more
Any tips?
On 12/7/10 8:44 PM, Mark wrote:
> 3 Node cluster and I just ran a nodetool cleanup on node #3. 1 and 2
> are now at 100% disk space. What should I do?