You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Michał Łowicki <ml...@gmail.com> on 2016/02/28 15:33:07 UTC
Replacing disks
Hi,
I've two disks on single box (500GB + 200GB). data_file_directories in
cassandra.yaml has two entries. I would like to replace 200GB with 500GB as
it's running out of space and to align it with others we've in the cluster.
The plan is to stop C*, attach new disk, move data from 200GB to new one
and mount it at the same point in the hierarchy. When done start C*.
Additionally I would like to move some data from the old 500GB to the new
one to distribute used disk space equally. Probably all related files for
single SSTable should be moved i.e.
foo-bar-ka-1630184-CompressionInfo.db
foo-bar-ka-1630184-Data.db
foo-bar-ka-1630184-Digest.sha1
foo-bar-ka-1630184-Filter.db
foo-bar-ka-1630184-Index.db
foo-bar-ka-1630184-Statistics.db
foo-bar-ka-1630184-Summary.db
foo-bar-ka-1630184-TOC.txt
Is this something which should work or you see some obstacles? (C* 2.1.13).
--
BR,
Michał Łowicki
Re: Replacing disks
Posted by Michał Łowicki <ml...@gmail.com>.
On Mon, Feb 29, 2016 at 8:52 AM, Alain RODRIGUEZ <ar...@gmail.com> wrote:
> I wrote that a few days ago:
> http://thelastpickle.com/blog/2016/02/25/removing-a-disk-mapping-from-cassandra.html
>
> I believe this might help you.
>
Yes, looks promising. Thanks!
> C*heers,
> -----------------------
>
> Alain Rodriguez - alain@thelastpickle.com
> France
>
> The Last Pickle - Apache Cassandra Consulting
> http <http://www.thelastpickle.com/>:// <http://www.thelastpickle.com/>
> www.thelastpickle.com
> Le 28 févr. 2016 15:17, "Clint Martin" <
> clintlmartin@coolfiretechnologies.com> a écrit :
>
>> Code wise, I am not completely familiar with what accomplishes the
>> behavior. But my understanding and experience is that Cass 2.1 picks the
>> drive with the most free space when picking a destination for a compaction
>> operation.
>> (This is an overly simplistic description. Reality is always more
>> nuanced. datastax had a blog post that describes this better as well as
>> limitations to the algorithm in 2.1 which are addressed in the 3.x releases
>> )
>>
>> Clint
>> On Feb 28, 2016 10:11 AM, "Michał Łowicki" <ml...@gmail.com> wrote:
>>
>>>
>>>
>>> On Sun, Feb 28, 2016 at 4:00 PM, Clint Martin <
>>> clintlmartin@coolfiretechnologies.com> wrote:
>>>
>>>> Your plan for replacing your 200gb drive sounds good to me. Since you
>>>> are running jbod, I wouldn't worry about manually redistributing data from
>>>> your other disk to the new one. Cassandra will do that for you as it
>>>> performs compaction.
>>>>
>>>
>>> Is this done by pickWriteableDirectory
>>> <https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/Directories.java#L386>
>>> ?
>>>
>>>> While you're doing the drive change, you need to complete the swap and
>>>> restart of the node before the hinted handoff window expires on the other
>>>> nodes. If you do not complete in time, you'll want to perform a repair on
>>>> the node.
>>>>
>>>
>>> Yes. Thanks!
>>>
>>>
>>>>
>>>>
>>>> Clint
>>>> On Feb 28, 2016 9:33 AM, "Michał Łowicki" <ml...@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I've two disks on single box (500GB + 200GB). data_file_directories
>>>>> in cassandra.yaml has two entries. I would like to replace 200GB with 500GB
>>>>> as it's running out of space and to align it with others we've in the
>>>>> cluster. The plan is to stop C*, attach new disk, move data from 200GB to
>>>>> new one and mount it at the same point in the hierarchy. When done start C*.
>>>>>
>>>>> Additionally I would like to move some data from the old 500GB to the
>>>>> new one to distribute used disk space equally. Probably all related files
>>>>> for single SSTable should be moved i.e.
>>>>>
>>>>> foo-bar-ka-1630184-CompressionInfo.db
>>>>>
>>>>> foo-bar-ka-1630184-Data.db
>>>>>
>>>>> foo-bar-ka-1630184-Digest.sha1
>>>>>
>>>>> foo-bar-ka-1630184-Filter.db
>>>>>
>>>>> foo-bar-ka-1630184-Index.db
>>>>>
>>>>> foo-bar-ka-1630184-Statistics.db
>>>>>
>>>>> foo-bar-ka-1630184-Summary.db
>>>>>
>>>>> foo-bar-ka-1630184-TOC.txt
>>>>>
>>>>> Is this something which should work or you see some obstacles? (C*
>>>>> 2.1.13).
>>>>> --
>>>>> BR,
>>>>> Michał Łowicki
>>>>>
>>>>
>>>
>>>
>>> --
>>> BR,
>>> Michał Łowicki
>>>
>>
--
BR,
Michał Łowicki
Re: Replacing disks
Posted by Alain RODRIGUEZ <ar...@gmail.com>.
I wrote that a few days ago:
http://thelastpickle.com/blog/2016/02/25/removing-a-disk-mapping-from-cassandra.html
I believe this might help you.
C*heers,
-----------------------
Alain Rodriguez - alain@thelastpickle.com
France
The Last Pickle - Apache Cassandra Consulting
http <http://www.thelastpickle.com/>:// <http://www.thelastpickle.com/>
www.thelastpickle.com
Le 28 févr. 2016 15:17, "Clint Martin" <
clintlmartin@coolfiretechnologies.com> a écrit :
> Code wise, I am not completely familiar with what accomplishes the
> behavior. But my understanding and experience is that Cass 2.1 picks the
> drive with the most free space when picking a destination for a compaction
> operation.
> (This is an overly simplistic description. Reality is always more
> nuanced. datastax had a blog post that describes this better as well as
> limitations to the algorithm in 2.1 which are addressed in the 3.x releases
> )
>
> Clint
> On Feb 28, 2016 10:11 AM, "Michał Łowicki" <ml...@gmail.com> wrote:
>
>>
>>
>> On Sun, Feb 28, 2016 at 4:00 PM, Clint Martin <
>> clintlmartin@coolfiretechnologies.com> wrote:
>>
>>> Your plan for replacing your 200gb drive sounds good to me. Since you
>>> are running jbod, I wouldn't worry about manually redistributing data from
>>> your other disk to the new one. Cassandra will do that for you as it
>>> performs compaction.
>>>
>>
>> Is this done by pickWriteableDirectory
>> <https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/Directories.java#L386>
>> ?
>>
>>> While you're doing the drive change, you need to complete the swap and
>>> restart of the node before the hinted handoff window expires on the other
>>> nodes. If you do not complete in time, you'll want to perform a repair on
>>> the node.
>>>
>>
>> Yes. Thanks!
>>
>>
>>>
>>>
>>> Clint
>>> On Feb 28, 2016 9:33 AM, "Michał Łowicki" <ml...@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> I've two disks on single box (500GB + 200GB). data_file_directories in
>>>> cassandra.yaml has two entries. I would like to replace 200GB with 500GB as
>>>> it's running out of space and to align it with others we've in the cluster.
>>>> The plan is to stop C*, attach new disk, move data from 200GB to new one
>>>> and mount it at the same point in the hierarchy. When done start C*.
>>>>
>>>> Additionally I would like to move some data from the old 500GB to the
>>>> new one to distribute used disk space equally. Probably all related files
>>>> for single SSTable should be moved i.e.
>>>>
>>>> foo-bar-ka-1630184-CompressionInfo.db
>>>>
>>>> foo-bar-ka-1630184-Data.db
>>>>
>>>> foo-bar-ka-1630184-Digest.sha1
>>>>
>>>> foo-bar-ka-1630184-Filter.db
>>>>
>>>> foo-bar-ka-1630184-Index.db
>>>>
>>>> foo-bar-ka-1630184-Statistics.db
>>>>
>>>> foo-bar-ka-1630184-Summary.db
>>>>
>>>> foo-bar-ka-1630184-TOC.txt
>>>>
>>>> Is this something which should work or you see some obstacles? (C*
>>>> 2.1.13).
>>>> --
>>>> BR,
>>>> Michał Łowicki
>>>>
>>>
>>
>>
>> --
>> BR,
>> Michał Łowicki
>>
>
Re: Replacing disks
Posted by Clint Martin <cl...@coolfiretechnologies.com>.
Code wise, I am not completely familiar with what accomplishes the
behavior. But my understanding and experience is that Cass 2.1 picks the
drive with the most free space when picking a destination for a compaction
operation.
(This is an overly simplistic description. Reality is always more nuanced.
datastax had a blog post that describes this better as well as limitations
to the algorithm in 2.1 which are addressed in the 3.x releases )
Clint
On Feb 28, 2016 10:11 AM, "Michał Łowicki" <ml...@gmail.com> wrote:
>
>
> On Sun, Feb 28, 2016 at 4:00 PM, Clint Martin <
> clintlmartin@coolfiretechnologies.com> wrote:
>
>> Your plan for replacing your 200gb drive sounds good to me. Since you are
>> running jbod, I wouldn't worry about manually redistributing data from your
>> other disk to the new one. Cassandra will do that for you as it performs
>> compaction.
>>
>
> Is this done by pickWriteableDirectory
> <https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/Directories.java#L386>
> ?
>
>> While you're doing the drive change, you need to complete the swap and
>> restart of the node before the hinted handoff window expires on the other
>> nodes. If you do not complete in time, you'll want to perform a repair on
>> the node.
>>
>
> Yes. Thanks!
>
>
>>
>>
>> Clint
>> On Feb 28, 2016 9:33 AM, "Michał Łowicki" <ml...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I've two disks on single box (500GB + 200GB). data_file_directories in
>>> cassandra.yaml has two entries. I would like to replace 200GB with 500GB as
>>> it's running out of space and to align it with others we've in the cluster.
>>> The plan is to stop C*, attach new disk, move data from 200GB to new one
>>> and mount it at the same point in the hierarchy. When done start C*.
>>>
>>> Additionally I would like to move some data from the old 500GB to the
>>> new one to distribute used disk space equally. Probably all related files
>>> for single SSTable should be moved i.e.
>>>
>>> foo-bar-ka-1630184-CompressionInfo.db
>>>
>>> foo-bar-ka-1630184-Data.db
>>>
>>> foo-bar-ka-1630184-Digest.sha1
>>>
>>> foo-bar-ka-1630184-Filter.db
>>>
>>> foo-bar-ka-1630184-Index.db
>>>
>>> foo-bar-ka-1630184-Statistics.db
>>>
>>> foo-bar-ka-1630184-Summary.db
>>>
>>> foo-bar-ka-1630184-TOC.txt
>>>
>>> Is this something which should work or you see some obstacles? (C*
>>> 2.1.13).
>>> --
>>> BR,
>>> Michał Łowicki
>>>
>>
>
>
> --
> BR,
> Michał Łowicki
>
Re: Replacing disks
Posted by Michał Łowicki <ml...@gmail.com>.
On Sun, Feb 28, 2016 at 4:00 PM, Clint Martin <
clintlmartin@coolfiretechnologies.com> wrote:
> Your plan for replacing your 200gb drive sounds good to me. Since you are
> running jbod, I wouldn't worry about manually redistributing data from your
> other disk to the new one. Cassandra will do that for you as it performs
> compaction.
>
Is this done by pickWriteableDirectory
<https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/Directories.java#L386>
?
> While you're doing the drive change, you need to complete the swap and
> restart of the node before the hinted handoff window expires on the other
> nodes. If you do not complete in time, you'll want to perform a repair on
> the node.
>
Yes. Thanks!
>
>
> Clint
> On Feb 28, 2016 9:33 AM, "Michał Łowicki" <ml...@gmail.com> wrote:
>
>> Hi,
>>
>> I've two disks on single box (500GB + 200GB). data_file_directories in
>> cassandra.yaml has two entries. I would like to replace 200GB with 500GB as
>> it's running out of space and to align it with others we've in the cluster.
>> The plan is to stop C*, attach new disk, move data from 200GB to new one
>> and mount it at the same point in the hierarchy. When done start C*.
>>
>> Additionally I would like to move some data from the old 500GB to the new
>> one to distribute used disk space equally. Probably all related files for
>> single SSTable should be moved i.e.
>>
>> foo-bar-ka-1630184-CompressionInfo.db
>>
>> foo-bar-ka-1630184-Data.db
>>
>> foo-bar-ka-1630184-Digest.sha1
>>
>> foo-bar-ka-1630184-Filter.db
>>
>> foo-bar-ka-1630184-Index.db
>>
>> foo-bar-ka-1630184-Statistics.db
>>
>> foo-bar-ka-1630184-Summary.db
>>
>> foo-bar-ka-1630184-TOC.txt
>>
>> Is this something which should work or you see some obstacles? (C*
>> 2.1.13).
>> --
>> BR,
>> Michał Łowicki
>>
>
--
BR,
Michał Łowicki
Re: Replacing disks
Posted by Clint Martin <cl...@coolfiretechnologies.com>.
Your plan for replacing your 200gb drive sounds good to me. Since you are
running jbod, I wouldn't worry about manually redistributing data from your
other disk to the new one. Cassandra will do that for you as it performs
compaction.
While you're doing the drive change, you need to complete the swap and
restart of the node before the hinted handoff window expires on the other
nodes. If you do not complete in time, you'll want to perform a repair on
the node.
Clint
On Feb 28, 2016 9:33 AM, "Michał Łowicki" <ml...@gmail.com> wrote:
> Hi,
>
> I've two disks on single box (500GB + 200GB). data_file_directories in
> cassandra.yaml has two entries. I would like to replace 200GB with 500GB as
> it's running out of space and to align it with others we've in the cluster.
> The plan is to stop C*, attach new disk, move data from 200GB to new one
> and mount it at the same point in the hierarchy. When done start C*.
>
> Additionally I would like to move some data from the old 500GB to the new
> one to distribute used disk space equally. Probably all related files for
> single SSTable should be moved i.e.
>
> foo-bar-ka-1630184-CompressionInfo.db
>
> foo-bar-ka-1630184-Data.db
>
> foo-bar-ka-1630184-Digest.sha1
>
> foo-bar-ka-1630184-Filter.db
>
> foo-bar-ka-1630184-Index.db
>
> foo-bar-ka-1630184-Statistics.db
>
> foo-bar-ka-1630184-Summary.db
>
> foo-bar-ka-1630184-TOC.txt
>
> Is this something which should work or you see some obstacles? (C* 2.1.13).
> --
> BR,
> Michał Łowicki
>