You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Analia Lorenzatto <an...@gmail.com> on 2015/05/15 19:17:32 UTC

After running nodetool clean up, the used disk space was increased

Hello guys,

I have a cassandra cluster = 2.1.0-2 comprised of 3 nodes.  I successfully
added the third node last week.  After that, I ran nodetool cleanup on one
of the other two nodes, and it finished well but it increased the used disk
space.
Before running the clean up the node was 197 GB of used space, and after
that it is 329GB used.  It is my understanding that the clean up frees up
some space, but in this case it was highly increased.

I am running out of space, that's why I added a third node.  Do you have
any clue on how to proceed with that situation?

Thanks in advance!!

-- 
Saludos / Regards.

Analía Lorenzatto.

“It's possible to commit no errors and still lose. That is not weakness.
That is life".  By Captain Jean-Luc Picard.

Re: After running nodetool clean up, the used disk space was increased

Posted by Analia Lorenzatto <an...@gmail.com>.
The cluster is comprised of 3 nodes, with a RP=2:

# nodetool status Datacenter: us-east
===================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address     Load       Tokens  Owns (effective)  Host ID
                Rack
UN  10.x.x.a  420.05 GB  256     66.7%
eed9e9f5-f279-4b2f-b521-c056cbf65b52  1c
UN  10.x.x.b  415.68 GB  256     68.3%
19492c26-4458-4a0b-af04-72e0aab6598e  1c
UN  10.x.x.c  164.45 GB  256     64.9%
b8da952c-24b3-444a-a34e-7a1804eee6e6  1c

10.x.x.c is the last node added.

Thanks!

On Fri, May 15, 2015 at 4:02 PM, Kiran mk <co...@gmail.com> wrote:

> What is the data distribution status across nodes ? What is the RP ?
> On May 16, 2015 12:30 AM, "Analia Lorenzatto" <an...@gmail.com>
> wrote:
>
>> Thanks Kiran for answering!
>>
>> I already ran cleanup on just one node.  At this moment, I am running on
>> the second one, but it did not finish there.
>>
>>
>> On Fri, May 15, 2015 at 3:37 PM, Kiran mk <co...@gmail.com>
>> wrote:
>>
>>> Did you try running nodetool cleanup on all the nodes ?
>>> On May 15, 2015 10:47 PM, "Analia Lorenzatto" <
>>> analialorenzatto@gmail.com> wrote:
>>>
>>>> Hello guys,
>>>>
>>>> I have a cassandra cluster = 2.1.0-2 comprised of 3 nodes.  I
>>>> successfully added the third node last week.  After that, I ran nodetool
>>>> cleanup on one of the other two nodes, and it finished well but it
>>>> increased the used disk space.
>>>> Before running the clean up the node was 197 GB of used space, and
>>>> after that it is 329GB used.  It is my understanding that the clean up
>>>> frees up some space, but in this case it was highly increased.
>>>>
>>>> I am running out of space, that's why I added a third node.  Do you
>>>> have any clue on how to proceed with that situation?
>>>>
>>>> Thanks in advance!!
>>>>
>>>> --
>>>> Saludos / Regards.
>>>>
>>>> Analía Lorenzatto.
>>>>
>>>> “It's possible to commit no errors and still lose. That is not
>>>> weakness.  That is life".  By Captain Jean-Luc Picard.
>>>>
>>>
>>
>>
>> --
>> Saludos / Regards.
>>
>> Analía Lorenzatto.
>>
>> “It's possible to commit no errors and still lose. That is not weakness.
>> That is life".  By Captain Jean-Luc Picard.
>>
>


-- 
Saludos / Regards.

Analía Lorenzatto.

“It's possible to commit no errors and still lose. That is not weakness.
That is life".  By Captain Jean-Luc Picard.

Re: After running nodetool clean up, the used disk space was increased

Posted by Kiran mk <co...@gmail.com>.
What is the Replication Factor ?  What does ring status saying ?
On May 16, 2015 12:32 AM, "Kiran mk" <co...@gmail.com> wrote:

> What is the data distribution status across nodes ? What is the RP ?
> On May 16, 2015 12:30 AM, "Analia Lorenzatto" <an...@gmail.com>
> wrote:
>
>> Thanks Kiran for answering!
>>
>> I already ran cleanup on just one node.  At this moment, I am running on
>> the second one, but it did not finish there.
>>
>>
>> On Fri, May 15, 2015 at 3:37 PM, Kiran mk <co...@gmail.com>
>> wrote:
>>
>>> Did you try running nodetool cleanup on all the nodes ?
>>> On May 15, 2015 10:47 PM, "Analia Lorenzatto" <
>>> analialorenzatto@gmail.com> wrote:
>>>
>>>> Hello guys,
>>>>
>>>> I have a cassandra cluster = 2.1.0-2 comprised of 3 nodes.  I
>>>> successfully added the third node last week.  After that, I ran nodetool
>>>> cleanup on one of the other two nodes, and it finished well but it
>>>> increased the used disk space.
>>>> Before running the clean up the node was 197 GB of used space, and
>>>> after that it is 329GB used.  It is my understanding that the clean up
>>>> frees up some space, but in this case it was highly increased.
>>>>
>>>> I am running out of space, that's why I added a third node.  Do you
>>>> have any clue on how to proceed with that situation?
>>>>
>>>> Thanks in advance!!
>>>>
>>>> --
>>>> Saludos / Regards.
>>>>
>>>> Analía Lorenzatto.
>>>>
>>>> “It's possible to commit no errors and still lose. That is not
>>>> weakness.  That is life".  By Captain Jean-Luc Picard.
>>>>
>>>
>>
>>
>> --
>> Saludos / Regards.
>>
>> Analía Lorenzatto.
>>
>> “It's possible to commit no errors and still lose. That is not weakness.
>> That is life".  By Captain Jean-Luc Picard.
>>
>

Re: After running nodetool clean up, the used disk space was increased

Posted by Kiran mk <co...@gmail.com>.
What is the data distribution status across nodes ? What is the RP ?
On May 16, 2015 12:30 AM, "Analia Lorenzatto" <an...@gmail.com>
wrote:

> Thanks Kiran for answering!
>
> I already ran cleanup on just one node.  At this moment, I am running on
> the second one, but it did not finish there.
>
>
> On Fri, May 15, 2015 at 3:37 PM, Kiran mk <co...@gmail.com> wrote:
>
>> Did you try running nodetool cleanup on all the nodes ?
>> On May 15, 2015 10:47 PM, "Analia Lorenzatto" <an...@gmail.com>
>> wrote:
>>
>>> Hello guys,
>>>
>>> I have a cassandra cluster = 2.1.0-2 comprised of 3 nodes.  I
>>> successfully added the third node last week.  After that, I ran nodetool
>>> cleanup on one of the other two nodes, and it finished well but it
>>> increased the used disk space.
>>> Before running the clean up the node was 197 GB of used space, and after
>>> that it is 329GB used.  It is my understanding that the clean up frees up
>>> some space, but in this case it was highly increased.
>>>
>>> I am running out of space, that's why I added a third node.  Do you have
>>> any clue on how to proceed with that situation?
>>>
>>> Thanks in advance!!
>>>
>>> --
>>> Saludos / Regards.
>>>
>>> Analía Lorenzatto.
>>>
>>> “It's possible to commit no errors and still lose. That is not
>>> weakness.  That is life".  By Captain Jean-Luc Picard.
>>>
>>
>
>
> --
> Saludos / Regards.
>
> Analía Lorenzatto.
>
> “It's possible to commit no errors and still lose. That is not weakness.
> That is life".  By Captain Jean-Luc Picard.
>

Re: After running nodetool clean up, the used disk space was increased

Posted by Analia Lorenzatto <an...@gmail.com>.
Thanks Kiran for answering!

I already ran cleanup on just one node.  At this moment, I am running on
the second one, but it did not finish there.


On Fri, May 15, 2015 at 3:37 PM, Kiran mk <co...@gmail.com> wrote:

> Did you try running nodetool cleanup on all the nodes ?
> On May 15, 2015 10:47 PM, "Analia Lorenzatto" <an...@gmail.com>
> wrote:
>
>> Hello guys,
>>
>> I have a cassandra cluster = 2.1.0-2 comprised of 3 nodes.  I
>> successfully added the third node last week.  After that, I ran nodetool
>> cleanup on one of the other two nodes, and it finished well but it
>> increased the used disk space.
>> Before running the clean up the node was 197 GB of used space, and after
>> that it is 329GB used.  It is my understanding that the clean up frees up
>> some space, but in this case it was highly increased.
>>
>> I am running out of space, that's why I added a third node.  Do you have
>> any clue on how to proceed with that situation?
>>
>> Thanks in advance!!
>>
>> --
>> Saludos / Regards.
>>
>> Analía Lorenzatto.
>>
>> “It's possible to commit no errors and still lose. That is not weakness.
>> That is life".  By Captain Jean-Luc Picard.
>>
>


-- 
Saludos / Regards.

Analía Lorenzatto.

“It's possible to commit no errors and still lose. That is not weakness.
That is life".  By Captain Jean-Luc Picard.

Re: After running nodetool clean up, the used disk space was increased

Posted by Kiran mk <co...@gmail.com>.
Did you try running nodetool cleanup on all the nodes ?
On May 15, 2015 10:47 PM, "Analia Lorenzatto" <an...@gmail.com>
wrote:

> Hello guys,
>
> I have a cassandra cluster = 2.1.0-2 comprised of 3 nodes.  I successfully
> added the third node last week.  After that, I ran nodetool cleanup on one
> of the other two nodes, and it finished well but it increased the used disk
> space.
> Before running the clean up the node was 197 GB of used space, and after
> that it is 329GB used.  It is my understanding that the clean up frees up
> some space, but in this case it was highly increased.
>
> I am running out of space, that's why I added a third node.  Do you have
> any clue on how to proceed with that situation?
>
> Thanks in advance!!
>
> --
> Saludos / Regards.
>
> Analía Lorenzatto.
>
> “It's possible to commit no errors and still lose. That is not weakness.
> That is life".  By Captain Jean-Luc Picard.
>

Re: After running nodetool clean up, the used disk space was increased

Posted by Analia Lorenzatto <an...@gmail.com>.
Yes Robert, I already cleared the snapshots.  After that, the used disk
space is:

10.x.x.b@$ df -h /mnt/cassandra
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvdb1      745G  174G  572G  24% /mnt/cassandra

But, the cluster shows me a different thing:

$ nodetool status
Datacenter: us-east
===================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address     Load       Tokens  Owns (effective)  Host ID
                Rack
UN  10.x.x.a  420.74 GB  256     66.7%
eed9e9f5-f279-4b2f-b521-c056cbf65b52  1c
UN  10.x.x.b  416.42 GB  256     68.3%
19492c26-4458-4a0b-af04-72e0aab6598e  1c
UN  10.x.x.c  165.15 GB  256     64.9%
b8da952c-24b3-444a-a34e-7a1804eee6e6  1c

I do not understand why the cluster still sees even more data before adding
the third node.

Thanks a lot!!


On Fri, May 15, 2015 at 6:27 PM, Robert Wille <rw...@fold3.com> wrote:

>  Have you cleared snapshots?
>
>  On May 15, 2015, at 2:24 PM, Analia Lorenzatto <
> analialorenzatto@gmail.com> wrote:
>
>  The Replication Factor = 2.  The RP is the default, but not sure how to
> check it.
> I am attaching the output of: nodetool ring
>
>  Thanks a lot!
>
> On Fri, May 15, 2015 at 4:17 PM, Kiran mk <co...@gmail.com> wrote:
>
>> run cleanup on all the nodes and wait till it completes.
>> On May 15, 2015 10:47 PM, "Analia Lorenzatto" <an...@gmail.com>
>> wrote:
>>
>>> Hello guys,
>>>
>>>  I have a cassandra cluster = 2.1.0-2 comprised of 3 nodes.  I
>>> successfully added the third node last week.  After that, I ran nodetool
>>> cleanup on one of the other two nodes, and it finished well but it
>>> increased the used disk space.
>>> Before running the clean up the node was 197 GB of used space, and after
>>> that it is 329GB used.  It is my understanding that the clean up frees up
>>> some space, but in this case it was highly increased.
>>>
>>>  I am running out of space, that's why I added a third node.  Do you
>>> have any clue on how to proceed with that situation?
>>>
>>>  Thanks in advance!!
>>>
>>>  --
>>>  Saludos / Regards.
>>>
>>>  Analía Lorenzatto.
>>>
>>> “It's possible to commit no errors and still lose. That is not
>>> weakness.  That is life".  By Captain Jean-Luc Picard.
>>>
>>
>
>
>  --
>  Saludos / Regards.
>
>  Analía Lorenzatto.
>
> “It's possible to commit no errors and still lose. That is not weakness.
> That is life".  By Captain Jean-Luc Picard.
>   <list>
>
>
>


-- 
Saludos / Regards.

Analía Lorenzatto.

“It's possible to commit no errors and still lose. That is not weakness.
That is life".  By Captain Jean-Luc Picard.

Re: After running nodetool clean up, the used disk space was increased

Posted by Robert Wille <rw...@fold3.com>.
Have you cleared snapshots?

On May 15, 2015, at 2:24 PM, Analia Lorenzatto <an...@gmail.com>> wrote:

The Replication Factor = 2.  The RP is the default, but not sure how to check it.
I am attaching the output of: nodetool ring

Thanks a lot!

On Fri, May 15, 2015 at 4:17 PM, Kiran mk <co...@gmail.com>> wrote:

run cleanup on all the nodes and wait till it completes.

On May 15, 2015 10:47 PM, "Analia Lorenzatto" <an...@gmail.com>> wrote:
Hello guys,

I have a cassandra cluster = 2.1.0-2 comprised of 3 nodes.  I successfully added the third node last week.  After that, I ran nodetool cleanup on one of the other two nodes, and it finished well but it increased the used disk space.
Before running the clean up the node was 197 GB of used space, and after that it is 329GB used.  It is my understanding that the clean up frees up some space, but in this case it was highly increased.

I am running out of space, that's why I added a third node.  Do you have any clue on how to proceed with that situation?

Thanks in advance!!

--
Saludos / Regards.

Analía Lorenzatto.

“It's possible to commit no errors and still lose. That is not weakness.  That is life".  By Captain Jean-Luc Picard.



--
Saludos / Regards.

Analía Lorenzatto.

“It's possible to commit no errors and still lose. That is not weakness.  That is life".  By Captain Jean-Luc Picard.
<list>


Re: After running nodetool clean up, the used disk space was increased

Posted by Analia Lorenzatto <an...@gmail.com>.
The Replication Factor = 2.  The RP is the default, but not sure how to
check it.
I am attaching the output of: nodetool ring

Thanks a lot!

On Fri, May 15, 2015 at 4:17 PM, Kiran mk <co...@gmail.com> wrote:

> run cleanup on all the nodes and wait till it completes.
> On May 15, 2015 10:47 PM, "Analia Lorenzatto" <an...@gmail.com>
> wrote:
>
>> Hello guys,
>>
>> I have a cassandra cluster = 2.1.0-2 comprised of 3 nodes.  I
>> successfully added the third node last week.  After that, I ran nodetool
>> cleanup on one of the other two nodes, and it finished well but it
>> increased the used disk space.
>> Before running the clean up the node was 197 GB of used space, and after
>> that it is 329GB used.  It is my understanding that the clean up frees up
>> some space, but in this case it was highly increased.
>>
>> I am running out of space, that's why I added a third node.  Do you have
>> any clue on how to proceed with that situation?
>>
>> Thanks in advance!!
>>
>> --
>> Saludos / Regards.
>>
>> Analía Lorenzatto.
>>
>> “It's possible to commit no errors and still lose. That is not weakness.
>> That is life".  By Captain Jean-Luc Picard.
>>
>


-- 
Saludos / Regards.

Analía Lorenzatto.

“It's possible to commit no errors and still lose. That is not weakness.
That is life".  By Captain Jean-Luc Picard.

Re: After running nodetool clean up, the used disk space was increased

Posted by Kiran mk <co...@gmail.com>.
run cleanup on all the nodes and wait till it completes.
On May 15, 2015 10:47 PM, "Analia Lorenzatto" <an...@gmail.com>
wrote:

> Hello guys,
>
> I have a cassandra cluster = 2.1.0-2 comprised of 3 nodes.  I successfully
> added the third node last week.  After that, I ran nodetool cleanup on one
> of the other two nodes, and it finished well but it increased the used disk
> space.
> Before running the clean up the node was 197 GB of used space, and after
> that it is 329GB used.  It is my understanding that the clean up frees up
> some space, but in this case it was highly increased.
>
> I am running out of space, that's why I added a third node.  Do you have
> any clue on how to proceed with that situation?
>
> Thanks in advance!!
>
> --
> Saludos / Regards.
>
> Analía Lorenzatto.
>
> “It's possible to commit no errors and still lose. That is not weakness.
> That is life".  By Captain Jean-Luc Picard.
>