You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@cassandra.apache.org by Osman Yozgatlıoğlu <os...@gmail.com> on 2020/04/04 12:53:27 UTC

one node down and cluster works better

Hello,

I manage one cluster with 2 dc, 7 nodes each and replication factor is 2:2
My insertion performance dropped somehow.
I restarted nodes one by one and found one node degrades performance.
Verified this node after problem occurs a couple of times.
How can I continue to investigate?

Regards,
Osman

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org

Re: one node down and cluster works better

Posted by Osman Yozgatlıoğlu <os...@gmail.com>.

Thanks Mehmet and Erick,

I don't have any monitoring other than nodetool but I manage to see
some disk errors cause exceptions.
I changed faulty disk and performance ok now.

Regards,
Osman

On Sun, 5 Apr 2020 at 03:17, Erick Ramirez <er...@datastax.com> wrote:
>
> With only 2 replicas per DC, it means you're likely writing with a consistency level of either ONE or LOCAL_ONE. Everytime you hit the problematic node, the write performance drops. All other configurations being equal, this indicates an issue with the commitlog disk on the node.
>
> Get your sysadmin to check for any issues with the disk, perhaps there's a problem with the hardware and failure is impending. As a side note, it's probably a good time to ensure that cluster repairs are in order in case the node fails. Cheers!
>
> GOT QUESTIONS? Apache Cassandra experts from the community and DataStax have answers! Share your expertise on https://community.datastax.com/.
>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org

Re: one node down and cluster works better

Posted by Erick Ramirez <er...@datastax.com>.

With only 2 replicas per DC, it means you're likely writing with a
consistency level of either ONE or LOCAL_ONE. Everytime you hit the
problematic node, the write performance drops. All other configurations
being equal, this indicates an issue with the commitlog disk on the node.

Get your sysadmin to check for any issues with the disk, perhaps there's a
problem with the hardware and failure is impending. As a side note, it's
probably a good time to ensure that cluster repairs are in order in case
the node fails. Cheers!

GOT QUESTIONS? Apache Cassandra experts from the community and DataStax
have answers! Share your expertise on https://community.datastax.com/.

Re: one node down and cluster works better

Posted by mehmet bursali <bu...@yahoo.com.INVALID>.

Hi Osman,Do you use any monitoring solution such as prometheus on your cluster? If yes, you should install and use cassandra exporter from the link below and examine some detailed metrics.https://github.com/criteo/cassandra_exporter  ndroid’de Yahoo Postadan gönderildi 
 
  15:53’’4e’ 4 Nis 2020 Cmt tarihinde, Osman Yozgatlıoğlu<os...@gmail.com> şunu yazdı:   Hello,

I manage one cluster with 2 dc, 7 nodes each and replication factor is 2:2
My insertion performance dropped somehow.
I restarted nodes one by one and found one node degrades performance.
Verified this node after problem occurs a couple of times.
How can I continue to investigate?

Regards,
Osman

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org