You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Kevin Burton <bu...@spinn3r.com> on 2014/08/21 00:45:59 UTC
Disk failure policy should let you run some basic commands.
So , right now, I have a full cassandra cluster… all my nodes are down.
Fun!
And I have a table, which I could just issue a truncate command to. It's
just a log table so dropping the data is fine.
but instead, I can't do that because my cluster is completely offline.
Now, the disk failure policy of stop is still ideal, because I don't want
INSERTs to partially work, but having some type of basic data manipulation
CQL like DROP TABLE or TRUNCATE TABLE would be nice.
And realistically, how do you even recover from this situation? You can't
compact the table, and you can't delete data or truncate the table either.
So when your cluster fills up, it's just dead.
Kevin
--
Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
<http://spinn3r.com>
Re: Disk failure policy should let you run some basic commands.
Posted by Robert Coli <rc...@eventbrite.com>.
On Wed, Aug 20, 2014 at 4:24 PM, Kevin Burton <bu...@spinn3r.com> wrote:
>
>>> +1, though because you can't drop the snapshots those two commands
>> automatically create (if the snapshot-before-DROP even works with disk
>> full, which it probably doesn't...) you still need access to the machines
>> to reclaim your disk space.
>>
>>
>
> True.. I actually disabled the snapshot feature for this reason.
>
> A "without snapshot" feature to drop/truncate table would be nice.
>
https://issues.apache.org/jira/browse/CASSANDRA-3710
Provides this, doesn't it?
> And realistically, how do you even recover from this situation? You can't
>>> compact the table, and you can't delete data or truncate the table either.
>>>
>>
>> 1) Delete some SSTables on the OS level and restart Cassandra. This is
>> obviously bad for consistency.
>> OR
>>
>
> Yes… luckily our tables are partitioned by day so I just deleted the
> oldest day.
>
Yeah, lucky you. :D
> 2) Add disk space to your node. This may not be possible.
>>
>
> Yes… it's not possible because you can't compact the table without some
> minor amount of disk space :-/
>
No, I mean ADD DISK CAPACITY TO THE SYSTEM by f/e plugging in a USB drive
or mounting a network disk.
=Rob
Re: Disk failure policy should let you run some basic commands.
Posted by Kevin Burton <bu...@spinn3r.com>.
>
>
>> +1, though because you can't drop the snapshots those two commands
> automatically create (if the snapshot-before-DROP even works with disk
> full, which it probably doesn't...) you still need access to the machines
> to reclaim your disk space.
>
>
True.. I actually disabled the snapshot feature for this reason.
A "without snapshot" feature to drop/truncate table would be nice.
> And realistically, how do you even recover from this situation? You can't
>> compact the table, and you can't delete data or truncate the table either.
>>
>
> 1) Delete some SSTables on the OS level and restart Cassandra. This is
> obviously bad for consistency.
> OR
>
Yes… luckily our tables are partitioned by day so I just deleted the oldest
day.
> 2) Add disk space to your node. This may not be possible.
>
>
Yes… it's not possible because you can't compact the table without some
minor amount of disk space :-/
--
Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
<http://spinn3r.com>
Re: Disk failure policy should let you run some basic commands.
Posted by Robert Coli <rc...@eventbrite.com>.
On Wed, Aug 20, 2014 at 3:45 PM, Kevin Burton <bu...@spinn3r.com> wrote:
> So , right now, I have a full cassandra cluster… all my nodes are down.
>
When your cluster is down, #cassandra is probably a better outlet than
cassandra-user@.
> And I have a table, which I could just issue a truncate command to. It's
> just a log table so dropping the data is fine.
>
> but instead, I can't do that because my cluster is completely offline.
>
> Now, the disk failure policy of stop is still ideal, because I don't want
> INSERTs to partially work, but having some type of basic data manipulation
> CQL like DROP TABLE or TRUNCATE TABLE would be nice.
>
+1, though because you can't drop the snapshots those two commands
automatically create (if the snapshot-before-DROP even works with disk
full, which it probably doesn't...) you still need access to the machines
to reclaim your disk space.
> And realistically, how do you even recover from this situation? You can't
> compact the table, and you can't delete data or truncate the table either.
>
1) Delete some SSTables on the OS level and restart Cassandra. This is
obviously bad for consistency.
OR
2) Add disk space to your node. This may not be possible.
=Rob