RE: [EXTERNAL] Lot of hints piling up

[Amit Singh <de...@nic.in>]

Check if any warn sign stating “ Node is down” or “Node is coming up” in system.log and also see on that node using nodetool tpstats whether dropped mutations are there or not.

 

From: Jai Bheemsen Rao Dhanwada [mailto:jaibheemsen@gmail.com] 
Sent: Wednesday, October 25, 2017 3:37 AM
To: user@cassandra.apache.org
Subject: Re: [EXTERNAL] Lot of hints piling up

 

No OOM or HEAP errors

 

On Tue, Oct 24, 2017 at 2:51 PM, Mohapatra, Kishore <Ki...@nuance.com> wrote:

Check how many sstables are there for the table you are having issues with.

You might be having the Heap pressure. Check your system.log for any OOM error or HEAP error.

 

Thanks

 

Kishore Mohapatra

Principal Operations DBA

Seattle, WA

Email :  <ma...@nuance.com> kishore.mohapatra@nuance.com

 

 

From: Jai Bheemsen Rao Dhanwada [mailto:jaibheemsen@gmail.com] 
Sent: Monday, October 23, 2017 11:54 AM
To: user@cassandra.apache.org
Subject: Re: [EXTERNAL] Lot of hints piling up

 

Do not see any errors in the logs or OS and compactions are happening in the regular interval and good too.

 

 

Issue here is, this causing replication lag across the datacenters.

 

On Mon, Oct 23, 2017 at 10:23 AM, Mohapatra, Kishore <Ki...@nuance.com> wrote:

Do you see any error in the cassandra log ?

Check compactionstats ?

Also check the OS level log messages to see if you are getting hardware level error messages.

 

Thanks

 

Kishore Mohapatra

Principal Operations DBA

Seattle, WA

Ph : 425-691-6417 (cell)

Email :  <ma...@nuance.com> kishore.mohapatra@nuance.com

 

 

From: Jai Bheemsen Rao Dhanwada [mailto:jaibheemsen@gmail.com] 
Sent: Friday, October 20, 2017 9:44 AM
To: user@cassandra.apache.org
Subject: [EXTERNAL] Lot of hints piling up

 

Hello,

 

We have cassandra cluster in 3 regions with version 2.1.13, and all of a sudden we started seeing lot of hints accumulating on the nodes. We are pretty sure there is no issue with the network between the regions and all the nodes are up and running all the time.

 

Is there any other reason for the hints accumulation other than the n/w? eg: wide rows or bigger objects?

 

Any pointers here could be very helpful.

 

b/w the hints gets processed after some point of time.