You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by rohit reddy <ro...@gmail.com> on 2012/09/14 15:24:05 UTC

Cassandra node going down

Hi,

I'm facing a problem in Cassandra cluster deployed on EC2 where the node is
going down under write load.

I have configured a cluster of 4 Large EC2 nodes with RF of 2.
All nodes are instance storage backed. DISK is RAID0 with 800GB

I'm pumping in write requests at about 4000 writes/sec. One of the node
went down under this load. The total data size in each node was not more
than 7GB
Got the following WARN messages in the LOG file...

1. setting live ratio to minimum of 1.0 instead of 0.9003153296009601
2. Heap is 0.7515559786053904 full.  You may need to reduce memtable and/or
cache sizes.  Cassandra will now flush up to the two largest memtables to
free up memory.  Adjust flush_largest_memtables_at threshold in
cassandra.yaml if you don't want Cassandra to do
this automatically
3. WARN [CompactionExecutor:570] 2012-09-14 11:45:12,024
CompactionTask.java (line 84) insufficient space to compact all requested
files

All cassandra settings are default settings.
Do i need to tune anything to support this write rate?

Thanks
Rohit

Re: Cassandra node going down

Posted by Tyler Hobbs <ty...@datastax.com>.
You will need to run nodetool removetoken with the old node's token to
permanently remove it from the cluster.

On Fri, Sep 14, 2012 at 3:06 PM, rohit reddy <ro...@gmail.com>wrote:

> Thanks for the inputs.
> The disk on the EC2 node failed. This led to the problem. Now i have
> created a new cassandra node and added it to the cluster.
>
> Do i need to do anything to delete the old node from the cluster, or will
> the cluster balance it self.
> Asking this since in Datastax ops center its still showing the old node.
>
> Thanks
> Rohit
>
>
> On Fri, Sep 14, 2012 at 7:42 PM, Robin Verlangen <ro...@us2.nl> wrote:
>
>> Cassandra writes to memtables, that will get flushed to disk when it's
>> time. That might be because of running out of memory (the log message you
>> just posted), on a shutdown, or at other times. That's why you're using
>> memory while writing.
>>
>> You seem to be running on AWS, are you sure your data location is on the
>> right disk? Default is /var/lib/cassandra/data
>>
>> Best regards,
>>
>> Robin Verlangen
>> *Software engineer*
>> *
>> *
>> W http://www.robinverlangen.nl
>> E robin@us2.nl
>>
>> Disclaimer: The information contained in this message and attachments is
>> intended solely for the attention and use of the named addressee and may be
>> confidential. If you are not the intended recipient, you are reminded that
>> the information remains the property of the sender. You must not use,
>> disclose, distribute, copy, print or rely on this e-mail. If you have
>> received this message in error, please contact the sender immediately and
>> irrevocably delete this message and any copies.
>>
>>
>>
>> 2012/9/14 rohit reddy <ro...@gmail.com>
>>
>>> Hi Robin,
>>>
>>> I had checked that. Our disk size is about 800GB, and the total data
>>> size is not more than 40GB. Even if all the data is stored in one node,
>>> this won't happen.
>>>
>>> I'll try to see if the disk failed.
>>>
>>> Is this anything to do with VM memory?.. cause this logs suggests that..
>>> Heap is 0.7515559786053904 full.  You may need to reduce memtable and/or
>>> cache sizes.  Cassandra will now flush up to the two largest memtables to
>>> free up memory.  Adjust flush_largest_memtables_at threshold in
>>> cassandra.yaml if you don't want Cassandra to do this automatically
>>>
>>> But, i'm only testing writes, there are no reads on the cluster. Will
>>> the writes require so much memory. A large instance has 7.5GB, so by
>>> default cassandra allocates about 3.75 GB for the VM.
>>>
>>>
>>>
>>> On Fri, Sep 14, 2012 at 6:58 PM, Robin Verlangen <ro...@us2.nl> wrote:
>>>
>>>> Hi Robbit,
>>>>
>>>> I think it's running out of disk space, please verify that (on Linux:
>>>> df -h ).
>>>>
>>>> Best regards,
>>>>
>>>> Robin Verlangen
>>>> *Software engineer*
>>>> *
>>>> *
>>>> W http://www.robinverlangen.nl
>>>> E robin@us2.nl
>>>>
>>>> Disclaimer: The information contained in this message and attachments
>>>> is intended solely for the attention and use of the named addressee and may
>>>> be confidential. If you are not the intended recipient, you are reminded
>>>> that the information remains the property of the sender. You must not use,
>>>> disclose, distribute, copy, print or rely on this e-mail. If you have
>>>> received this message in error, please contact the sender immediately and
>>>> irrevocably delete this message and any copies.
>>>>
>>>>
>>>>
>>>> 2012/9/14 rohit reddy <ro...@gmail.com>
>>>>
>>>>> Hi,
>>>>>
>>>>> I'm facing a problem in Cassandra cluster deployed on EC2 where the
>>>>> node is going down under write load.
>>>>>
>>>>> I have configured a cluster of 4 Large EC2 nodes with RF of 2.
>>>>> All nodes are instance storage backed. DISK is RAID0 with 800GB
>>>>>
>>>>> I'm pumping in write requests at about 4000 writes/sec. One of the
>>>>> node went down under this load. The total data size in each node was not
>>>>> more than 7GB
>>>>> Got the following WARN messages in the LOG file...
>>>>>
>>>>> 1. setting live ratio to minimum of 1.0 instead of 0.9003153296009601
>>>>> 2. Heap is 0.7515559786053904 full.  You may need to reduce memtable
>>>>> and/or cache sizes.  Cassandra will now flush up to the two largest
>>>>> memtables to free up memory.  Adjust flush_largest_memtables_at threshold
>>>>> in cassandra.yaml if you don't want Cassandra to do
>>>>> this automatically
>>>>> 3. WARN [CompactionExecutor:570] 2012-09-14 11:45:12,024
>>>>> CompactionTask.java (line 84) insufficient space to compact all requested
>>>>> files
>>>>>
>>>>> All cassandra settings are default settings.
>>>>> Do i need to tune anything to support this write rate?
>>>>>
>>>>> Thanks
>>>>> Rohit
>>>>>
>>>>>
>>>>
>>>
>>
>


-- 
Tyler Hobbs
DataStax <http://datastax.com/>

Re: Cassandra node going down

Posted by rohit reddy <ro...@gmail.com>.
Thanks for the inputs.
The disk on the EC2 node failed. This led to the problem. Now i have
created a new cassandra node and added it to the cluster.

Do i need to do anything to delete the old node from the cluster, or will
the cluster balance it self.
Asking this since in Datastax ops center its still showing the old node.

Thanks
Rohit

On Fri, Sep 14, 2012 at 7:42 PM, Robin Verlangen <ro...@us2.nl> wrote:

> Cassandra writes to memtables, that will get flushed to disk when it's
> time. That might be because of running out of memory (the log message you
> just posted), on a shutdown, or at other times. That's why you're using
> memory while writing.
>
> You seem to be running on AWS, are you sure your data location is on the
> right disk? Default is /var/lib/cassandra/data
>
> Best regards,
>
> Robin Verlangen
> *Software engineer*
> *
> *
> W http://www.robinverlangen.nl
> E robin@us2.nl
>
> Disclaimer: The information contained in this message and attachments is
> intended solely for the attention and use of the named addressee and may be
> confidential. If you are not the intended recipient, you are reminded that
> the information remains the property of the sender. You must not use,
> disclose, distribute, copy, print or rely on this e-mail. If you have
> received this message in error, please contact the sender immediately and
> irrevocably delete this message and any copies.
>
>
>
> 2012/9/14 rohit reddy <ro...@gmail.com>
>
>> Hi Robin,
>>
>> I had checked that. Our disk size is about 800GB, and the total data size
>> is not more than 40GB. Even if all the data is stored in one node, this
>> won't happen.
>>
>> I'll try to see if the disk failed.
>>
>> Is this anything to do with VM memory?.. cause this logs suggests that..
>> Heap is 0.7515559786053904 full.  You may need to reduce memtable and/or
>> cache sizes.  Cassandra will now flush up to the two largest memtables to
>> free up memory.  Adjust flush_largest_memtables_at threshold in
>> cassandra.yaml if you don't want Cassandra to do this automatically
>>
>> But, i'm only testing writes, there are no reads on the cluster. Will the
>> writes require so much memory. A large instance has 7.5GB, so by default
>> cassandra allocates about 3.75 GB for the VM.
>>
>>
>>
>> On Fri, Sep 14, 2012 at 6:58 PM, Robin Verlangen <ro...@us2.nl> wrote:
>>
>>> Hi Robbit,
>>>
>>> I think it's running out of disk space, please verify that (on Linux: df
>>> -h ).
>>>
>>> Best regards,
>>>
>>> Robin Verlangen
>>> *Software engineer*
>>> *
>>> *
>>> W http://www.robinverlangen.nl
>>> E robin@us2.nl
>>>
>>> Disclaimer: The information contained in this message and attachments is
>>> intended solely for the attention and use of the named addressee and may be
>>> confidential. If you are not the intended recipient, you are reminded that
>>> the information remains the property of the sender. You must not use,
>>> disclose, distribute, copy, print or rely on this e-mail. If you have
>>> received this message in error, please contact the sender immediately and
>>> irrevocably delete this message and any copies.
>>>
>>>
>>>
>>> 2012/9/14 rohit reddy <ro...@gmail.com>
>>>
>>>> Hi,
>>>>
>>>> I'm facing a problem in Cassandra cluster deployed on EC2 where the
>>>> node is going down under write load.
>>>>
>>>> I have configured a cluster of 4 Large EC2 nodes with RF of 2.
>>>> All nodes are instance storage backed. DISK is RAID0 with 800GB
>>>>
>>>> I'm pumping in write requests at about 4000 writes/sec. One of the node
>>>> went down under this load. The total data size in each node was not more
>>>> than 7GB
>>>> Got the following WARN messages in the LOG file...
>>>>
>>>> 1. setting live ratio to minimum of 1.0 instead of 0.9003153296009601
>>>> 2. Heap is 0.7515559786053904 full.  You may need to reduce memtable
>>>> and/or cache sizes.  Cassandra will now flush up to the two largest
>>>> memtables to free up memory.  Adjust flush_largest_memtables_at threshold
>>>> in cassandra.yaml if you don't want Cassandra to do
>>>> this automatically
>>>> 3. WARN [CompactionExecutor:570] 2012-09-14 11:45:12,024
>>>> CompactionTask.java (line 84) insufficient space to compact all requested
>>>> files
>>>>
>>>> All cassandra settings are default settings.
>>>> Do i need to tune anything to support this write rate?
>>>>
>>>> Thanks
>>>> Rohit
>>>>
>>>>
>>>
>>
>

Re: Cassandra node going down

Posted by Robin Verlangen <ro...@us2.nl>.
Cassandra writes to memtables, that will get flushed to disk when it's
time. That might be because of running out of memory (the log message you
just posted), on a shutdown, or at other times. That's why you're using
memory while writing.

You seem to be running on AWS, are you sure your data location is on the
right disk? Default is /var/lib/cassandra/data

Best regards,

Robin Verlangen
*Software engineer*
*
*
W http://www.robinverlangen.nl
E robin@us2.nl

Disclaimer: The information contained in this message and attachments is
intended solely for the attention and use of the named addressee and may be
confidential. If you are not the intended recipient, you are reminded that
the information remains the property of the sender. You must not use,
disclose, distribute, copy, print or rely on this e-mail. If you have
received this message in error, please contact the sender immediately and
irrevocably delete this message and any copies.



2012/9/14 rohit reddy <ro...@gmail.com>

> Hi Robin,
>
> I had checked that. Our disk size is about 800GB, and the total data size
> is not more than 40GB. Even if all the data is stored in one node, this
> won't happen.
>
> I'll try to see if the disk failed.
>
> Is this anything to do with VM memory?.. cause this logs suggests that..
> Heap is 0.7515559786053904 full.  You may need to reduce memtable and/or
> cache sizes.  Cassandra will now flush up to the two largest memtables to
> free up memory.  Adjust flush_largest_memtables_at threshold in
> cassandra.yaml if you don't want Cassandra to do this automatically
>
> But, i'm only testing writes, there are no reads on the cluster. Will the
> writes require so much memory. A large instance has 7.5GB, so by default
> cassandra allocates about 3.75 GB for the VM.
>
>
>
> On Fri, Sep 14, 2012 at 6:58 PM, Robin Verlangen <ro...@us2.nl> wrote:
>
>> Hi Robbit,
>>
>> I think it's running out of disk space, please verify that (on Linux: df
>> -h ).
>>
>> Best regards,
>>
>> Robin Verlangen
>> *Software engineer*
>> *
>> *
>> W http://www.robinverlangen.nl
>> E robin@us2.nl
>>
>> Disclaimer: The information contained in this message and attachments is
>> intended solely for the attention and use of the named addressee and may be
>> confidential. If you are not the intended recipient, you are reminded that
>> the information remains the property of the sender. You must not use,
>> disclose, distribute, copy, print or rely on this e-mail. If you have
>> received this message in error, please contact the sender immediately and
>> irrevocably delete this message and any copies.
>>
>>
>>
>> 2012/9/14 rohit reddy <ro...@gmail.com>
>>
>>> Hi,
>>>
>>> I'm facing a problem in Cassandra cluster deployed on EC2 where the node
>>> is going down under write load.
>>>
>>> I have configured a cluster of 4 Large EC2 nodes with RF of 2.
>>> All nodes are instance storage backed. DISK is RAID0 with 800GB
>>>
>>> I'm pumping in write requests at about 4000 writes/sec. One of the node
>>> went down under this load. The total data size in each node was not more
>>> than 7GB
>>> Got the following WARN messages in the LOG file...
>>>
>>> 1. setting live ratio to minimum of 1.0 instead of 0.9003153296009601
>>> 2. Heap is 0.7515559786053904 full.  You may need to reduce memtable
>>> and/or cache sizes.  Cassandra will now flush up to the two largest
>>> memtables to free up memory.  Adjust flush_largest_memtables_at threshold
>>> in cassandra.yaml if you don't want Cassandra to do
>>> this automatically
>>> 3. WARN [CompactionExecutor:570] 2012-09-14 11:45:12,024
>>> CompactionTask.java (line 84) insufficient space to compact all requested
>>> files
>>>
>>> All cassandra settings are default settings.
>>> Do i need to tune anything to support this write rate?
>>>
>>> Thanks
>>> Rohit
>>>
>>>
>>
>

Re: Cassandra node going down

Posted by rohit reddy <ro...@gmail.com>.
Hi Robin,

I had checked that. Our disk size is about 800GB, and the total data size
is not more than 40GB. Even if all the data is stored in one node, this
won't happen.

I'll try to see if the disk failed.

Is this anything to do with VM memory?.. cause this logs suggests that..
Heap is 0.7515559786053904 full.  You may need to reduce memtable and/or
cache sizes.  Cassandra will now flush up to the two largest memtables to
free up memory.  Adjust flush_largest_memtables_at threshold in
cassandra.yaml if you don't want Cassandra to do this automatically

But, i'm only testing writes, there are no reads on the cluster. Will the
writes require so much memory. A large instance has 7.5GB, so by default
cassandra allocates about 3.75 GB for the VM.



On Fri, Sep 14, 2012 at 6:58 PM, Robin Verlangen <ro...@us2.nl> wrote:

> Hi Robbit,
>
> I think it's running out of disk space, please verify that (on Linux: df
> -h ).
>
> Best regards,
>
> Robin Verlangen
> *Software engineer*
> *
> *
> W http://www.robinverlangen.nl
> E robin@us2.nl
>
> Disclaimer: The information contained in this message and attachments is
> intended solely for the attention and use of the named addressee and may be
> confidential. If you are not the intended recipient, you are reminded that
> the information remains the property of the sender. You must not use,
> disclose, distribute, copy, print or rely on this e-mail. If you have
> received this message in error, please contact the sender immediately and
> irrevocably delete this message and any copies.
>
>
>
> 2012/9/14 rohit reddy <ro...@gmail.com>
>
>> Hi,
>>
>> I'm facing a problem in Cassandra cluster deployed on EC2 where the node
>> is going down under write load.
>>
>> I have configured a cluster of 4 Large EC2 nodes with RF of 2.
>> All nodes are instance storage backed. DISK is RAID0 with 800GB
>>
>> I'm pumping in write requests at about 4000 writes/sec. One of the node
>> went down under this load. The total data size in each node was not more
>> than 7GB
>> Got the following WARN messages in the LOG file...
>>
>> 1. setting live ratio to minimum of 1.0 instead of 0.9003153296009601
>> 2. Heap is 0.7515559786053904 full.  You may need to reduce memtable
>> and/or cache sizes.  Cassandra will now flush up to the two largest
>> memtables to free up memory.  Adjust flush_largest_memtables_at threshold
>> in cassandra.yaml if you don't want Cassandra to do
>> this automatically
>> 3. WARN [CompactionExecutor:570] 2012-09-14 11:45:12,024
>> CompactionTask.java (line 84) insufficient space to compact all requested
>> files
>>
>> All cassandra settings are default settings.
>> Do i need to tune anything to support this write rate?
>>
>> Thanks
>> Rohit
>>
>>
>

Re: Cassandra node going down

Posted by Robin Verlangen <ro...@us2.nl>.
Robbit = Rohit of course, excuse me.

Best regards,

Robin Verlangen
*Software engineer*
*
*
W http://www.robinverlangen.nl
E robin@us2.nl

Disclaimer: The information contained in this message and attachments is
intended solely for the attention and use of the named addressee and may be
confidential. If you are not the intended recipient, you are reminded that
the information remains the property of the sender. You must not use,
disclose, distribute, copy, print or rely on this e-mail. If you have
received this message in error, please contact the sender immediately and
irrevocably delete this message and any copies.



2012/9/14 Robin Verlangen <ro...@us2.nl>

> Hi Robbit,
>
> I think it's running out of disk space, please verify that (on Linux: df
> -h ).
>
> Best regards,
>
> Robin Verlangen
> *Software engineer*
> *
> *
> W http://www.robinverlangen.nl
> E robin@us2.nl
>
> Disclaimer: The information contained in this message and attachments is
> intended solely for the attention and use of the named addressee and may be
> confidential. If you are not the intended recipient, you are reminded that
> the information remains the property of the sender. You must not use,
> disclose, distribute, copy, print or rely on this e-mail. If you have
> received this message in error, please contact the sender immediately and
> irrevocably delete this message and any copies.
>
>
>
> 2012/9/14 rohit reddy <ro...@gmail.com>
>
>> Hi,
>>
>> I'm facing a problem in Cassandra cluster deployed on EC2 where the node
>> is going down under write load.
>>
>> I have configured a cluster of 4 Large EC2 nodes with RF of 2.
>> All nodes are instance storage backed. DISK is RAID0 with 800GB
>>
>> I'm pumping in write requests at about 4000 writes/sec. One of the node
>> went down under this load. The total data size in each node was not more
>> than 7GB
>> Got the following WARN messages in the LOG file...
>>
>> 1. setting live ratio to minimum of 1.0 instead of 0.9003153296009601
>> 2. Heap is 0.7515559786053904 full.  You may need to reduce memtable
>> and/or cache sizes.  Cassandra will now flush up to the two largest
>> memtables to free up memory.  Adjust flush_largest_memtables_at threshold
>> in cassandra.yaml if you don't want Cassandra to do
>> this automatically
>> 3. WARN [CompactionExecutor:570] 2012-09-14 11:45:12,024
>> CompactionTask.java (line 84) insufficient space to compact all requested
>> files
>>
>> All cassandra settings are default settings.
>> Do i need to tune anything to support this write rate?
>>
>> Thanks
>> Rohit
>>
>>
>

Re: Cassandra node going down

Posted by Robin Verlangen <ro...@us2.nl>.
Hi Robbit,

I think it's running out of disk space, please verify that (on Linux: df -h
).

Best regards,

Robin Verlangen
*Software engineer*
*
*
W http://www.robinverlangen.nl
E robin@us2.nl

Disclaimer: The information contained in this message and attachments is
intended solely for the attention and use of the named addressee and may be
confidential. If you are not the intended recipient, you are reminded that
the information remains the property of the sender. You must not use,
disclose, distribute, copy, print or rely on this e-mail. If you have
received this message in error, please contact the sender immediately and
irrevocably delete this message and any copies.



2012/9/14 rohit reddy <ro...@gmail.com>

> Hi,
>
> I'm facing a problem in Cassandra cluster deployed on EC2 where the node
> is going down under write load.
>
> I have configured a cluster of 4 Large EC2 nodes with RF of 2.
> All nodes are instance storage backed. DISK is RAID0 with 800GB
>
> I'm pumping in write requests at about 4000 writes/sec. One of the node
> went down under this load. The total data size in each node was not more
> than 7GB
> Got the following WARN messages in the LOG file...
>
> 1. setting live ratio to minimum of 1.0 instead of 0.9003153296009601
> 2. Heap is 0.7515559786053904 full.  You may need to reduce memtable
> and/or cache sizes.  Cassandra will now flush up to the two largest
> memtables to free up memory.  Adjust flush_largest_memtables_at threshold
> in cassandra.yaml if you don't want Cassandra to do
> this automatically
> 3. WARN [CompactionExecutor:570] 2012-09-14 11:45:12,024
> CompactionTask.java (line 84) insufficient space to compact all requested
> files
>
> All cassandra settings are default settings.
> Do i need to tune anything to support this write rate?
>
> Thanks
> Rohit
>
>