You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by R1 J1 <rj...@gmail.com> on 2018/07/26 02:32:01 UTC
cassandro nodes restarts
cassandro nodes restarts
we see errors typically like these
WARN [Native-Transport-Requests-3] 2018-07-25 20:51:38,520
BatchStatement.java:301 - Batch for "keyspace.table"
is of size 19.386KiB, exceeding specified threshold of 5.000KiB by
14.386KiB.
Regards
R1J1
Re: cassandro nodes restarts
Posted by Rahul Singh <ra...@gmail.com>.
Do the same nodes reboot or is it arbitrary? I’m wondering if it’s an isolated incident related to dat / traffic skew or could happen on any coordinator
Rahul
On Jul 26, 2018, 12:31 AM -0400, Jeff Jirsa <jj...@gmail.com>, wrote:
> It’s a warning, but probably not causing you problems
>
> A 20kB batch is a hint that your batches are larger than Cassandra expects, but the 5k limit for that logger was somewhat arbitrary, and I would be shocked if 20kB batches were a problem unless you were already close to tipping your cluster
>
> If I were you I’d disable that warning (or set it much higher).
>
> --
> Jeff Jirsa
>
>
> > On Jul 25, 2018, at 7:32 PM, R1 J1 <rj...@gmail.com> wrote:
> >
> > cassandro nodes restarts
> >
> >
> >
> > we see errors typically like these
> >
> >
> > WARN [Native-Transport-Requests-3] 2018-07-25 20:51:38,520 BatchStatement.java:301 - Batch for "keyspace.table"
> > is of size 19.386KiB, exceeding specified threshold of 5.000KiB by 14.386KiB.
> >
> >
> > Regards
> > R1J1
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: user-help@cassandra.apache.org
>
Re: cassandro nodes restarts
Posted by Jeff Jirsa <jj...@gmail.com>.
It’s a warning, but probably not causing you problems
A 20kB batch is a hint that your batches are larger than Cassandra expects, but the 5k limit for that logger was somewhat arbitrary, and I would be shocked if 20kB batches were a problem unless you were already close to tipping your cluster
If I were you I’d disable that warning (or set it much higher).
--
Jeff Jirsa
> On Jul 25, 2018, at 7:32 PM, R1 J1 <rj...@gmail.com> wrote:
>
> cassandro nodes restarts
>
>
>
> we see errors typically like these
>
>
> WARN [Native-Transport-Requests-3] 2018-07-25 20:51:38,520 BatchStatement.java:301 - Batch for "keyspace.table"
> is of size 19.386KiB, exceeding specified threshold of 5.000KiB by 14.386KiB.
>
>
> Regards
> R1J1
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org
Re: cassandro nodes restarts
Posted by Rahul Singh <ra...@gmail.com>.
Need to review java gc, system , network, disk, memory, node, and table statistics. A lot can be discerned from visually examining the charts. Eg. if the nodes with the most local reads is failing or is it the one with the most writes or is it completely unrelated.
Since it’s a distributed system you need to review the data points together for all nodes. Data is the only way to see what’s going on. Either connect Prometheus / Grafana , get Datadog , New Relic, or something else to see the patterns across the cluster.
https://blog.anant.us/resources-for-monitoring-datastax-cassandra-spark-solr-performance/
I assembled that list recently — I would even add that getting system logs into ELK or Splunk could also show some patterns otherwise not detected tailing and gripping.
Rahul
On Jul 26, 2018, 10:20 AM -0400, R1 J1 <rj...@gmail.com>, wrote:
> Thanks for your prompt replies. No the same node is not bouncing over. When you say it is about to tip over: What can we do to stop that ?
>
> Also about that error : you guys are correct: it is a warning and might not be contributing to the node bounce issue and it can be removed by changing batch_size_warn_threshold_in_kb: 5
>
> R1J1
>
> > On Wed, Jul 25, 2018 at 10:32 PM, R1 J1 <rj...@gmail.com> wrote:
> > > cassandro nodes restarts
> > >
> > >
> > >
> > > we see errors typically like these
> > >
> > >
> > > WARN [Native-Transport-Requests-3] 2018-07-25 20:51:38,520 BatchStatement.java:301 - Batch for "keyspace.table"
> > > is of size 19.386KiB, exceeding specified threshold of 5.000KiB by 14.386KiB.
> > >
> > >
> > > Regards
> > > R1J1
>
Re: cassandro nodes restarts
Posted by R1 J1 <rj...@gmail.com>.
Thanks for your prompt replies. No the same node is not bouncing over. When
you say it is about to tip over: What can we do to stop that ?
Also about that error : you guys are correct: it is a warning and might
not be contributing to the node bounce issue and it can be removed by
changing batch_size_warn_threshold_in_kb: 5
R1J1
On Wed, Jul 25, 2018 at 10:32 PM, R1 J1 <rj...@gmail.com> wrote:
> cassandro nodes restarts
>
>
>
> we see errors typically like these
>
>
> WARN [Native-Transport-Requests-3] 2018-07-25 20:51:38,520
> BatchStatement.java:301 - Batch for "keyspace.table"
> is of size 19.386KiB, exceeding specified threshold of 5.000KiB by
> 14.386KiB.
>
>
> Regards
> R1J1
>