You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by "ZAIDI, ASAD A" <az...@att.com> on 2017/07/20 15:27:19 UTC
MUTATION messages were dropped in last 5000 ms for cross node
timeout
Hello Folks -
I'm using apache-cassandra 2.2.8.
I see many messages like below in my system.log file. In Cassandra.yaml file [ cross_node_timeout: true] is set and NTP server is also running correcting clock drift on 16node cluster. I do not see pending or blocked HintedHandoff in tpstats output though there are bunch of MUTATIONS dropped observed.
<start timeout message >
INFO [ScheduledTasks:1] 2017-07-20 08:02:52,511 MessagingService.java:946 - MUTATION messages were dropped in last 5000 ms: 822 for internal timeout and 2152 for cross node timeout
<end timeout message >
I'm seeking help here if you please let me know what I need to check in order to address these cross node timeouts.
Thank you,
Asad
Re: MUTATION messages were dropped in last 5000 ms for cross node
timeout
Posted by Akhil Mehra <ak...@gmail.com>.
Glad I could be of help :)
Hopefully the partition size resize goes smoothly.
Regards,
Akhil
> On 4/08/2017, at 5:41 AM, ZAIDI, ASAD A <az...@att.com> wrote:
>
> Hi Akhil,
>
> Thank you for your reply.
>
> I kept testing different timeout numbers over last week and eventually settled at setting *_request_timeout_in_ms parameters at 1.5minutes for coordinator wait time. That is the number where I donot see any dropped mutations.
>
> Also asked developers to tweak data model where we saw bunch of tables with really large partition size , some are ranging Partition-key size around ~6.6GB.. we’re now working to reduce the partition size of the tables. I am hoping corrected data model will help reduce coordinator wait time (get back to default number!) again.
>
> Thank again/Asad
>
> From: Akhil Mehra [mailto:akhilmehra@gmail.com]
> Sent: Friday, July 21, 2017 4:24 PM
> To: user@cassandra.apache.org
> Subject: Re: MUTATION messages were dropped in last 5000 ms for cross node timeout
>
> Hi Asad,
>
> The 5000 ms is not configurable (https://github.com/apache/cassandra/blob/8b3a60b9a7dbefeecc06bace617279612ec7092d/src/java/org/apache/cassandra/net/MessagingService.java#L423 <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_cassandra_blob_8b3a60b9a7dbefeecc06bace617279612ec7092d_src_java_org_apache_cassandra_net_MessagingService.java-23L423&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=FsmDztdsVuIKml8IDhdHdg&m=dp_TvjXTbUtu3Iu43aZ83eHl1fgW6l4P4PSQglF855g&s=USbrEM6jaGFIRKSUhJBx3VAkSSrXzid0db6TDV1vrDs&e=>). This just the time after which the number of dropped messages are reported. Thus dropped messages are reported every 5000ms.
>
> If you are looking to tweak the number of ms after which a message is considered dropped then you need to use the write_request_timeout_in_ms. The write_request_timeout_in_ms (http://docs.datastax.com/en/cassandra/2.1/cassandra/configuration/configCassandra_yaml_r.html <https://urldefense.proofpoint.com/v2/url?u=http-3A__docs.datastax.com_en_cassandra_2.1_cassandra_configuration_configCassandra-5Fyaml-5Fr.html&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=FsmDztdsVuIKml8IDhdHdg&m=dp_TvjXTbUtu3Iu43aZ83eHl1fgW6l4P4PSQglF855g&s=ab1NW9WoAXIlxT2kWjsiYFVaVidEnC_MB770pwTtqLs&e=>) can be used to increase the mutation timeout. By default it is set to 2000ms.
>
> I hope that helps.
>
> Regards,
> Akhil
>
>
> On 22/07/2017, at 2:46 AM, ZAIDI, ASAD A <az192g@att.com <ma...@att.com>> wrote:
>
> Hi Akhil,
>
> Thank you for your reply. Previously, I did ‘tune’ various timeouts – basically increased them a bit but none of those parameter listed in the link matches with that “were dropped in last 5000 ms”.
> I was wondering from where that [5000ms] number is coming from when, like I mentioned before, none of any timeout parameter settings matches that #!
>
> Load is intermittently high but again cpu queue length never goes beyond medium depth. I wonder if there is some internal limit that I’m still not aware of.
>
> Thanks/Asad
>
>
> From: Akhil Mehra [mailto:akhilmehra@gmail.com <ma...@gmail.com>]
> Sent: Thursday, July 20, 2017 3:47 PM
> To: user@cassandra.apache.org <ma...@cassandra.apache.org>
> Subject: Re: MUTATION messages were dropped in last 5000 ms for cross node timeout
>
> Hi Asad,
>
> http://cassandra.apache.org/doc/latest/faq/index.html#why-message-dropped <https://urldefense.proofpoint.com/v2/url?u=http-3A__cassandra.apache.org_doc_latest_faq_index.html-23why-2Dmessage-2Ddropped&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=FsmDztdsVuIKml8IDhdHdg&m=WcHuHKcjg2YCsAbw2NR_0-CiHr9JNxtCzYikia16mpo&s=0_0pQfoOZLuswpQ_lE-AU2bTMFLgRbR4k4Kh8vEOZSk&e=>
>
> As mentioned in the link above this is a load shedding mechanism used by Cassandra.
>
> Is you cluster under heavy load?
>
> Regards,
> Akhil
>
>
> On 21/07/2017, at 3:27 AM, ZAIDI, ASAD A <az192g@att.com <ma...@att.com>> wrote:
>
> Hello Folks –
>
> I’m using apache-cassandra 2.2.8.
>
> I see many messages like below in my system.log file. In Cassandra.yaml file [ cross_node_timeout: true] is set and NTP server is also running correcting clock drift on 16node cluster. I do not see pending or blocked HintedHandoff in tpstats output though there are bunch of MUTATIONS dropped observed.
>
> <start timeout message >
> INFO [ScheduledTasks:1] 2017-07-20 08:02:52,511 MessagingService.java:946 - MUTATION messages were dropped in last 5000 ms: 822 for internal timeout and 2152 for cross node timeout
> <end timeout message >
>
> I’m seeking help here if you please let me know what I need to check in order to address these cross node timeouts.
>
> Thank you,
> Asad
RE: MUTATION messages were dropped in last 5000 ms for cross node
timeout
Posted by "ZAIDI, ASAD A" <az...@att.com>.
Hi Akhil,
Thank you for your reply.
I kept testing different timeout numbers over last week and eventually settled at setting *_request_timeout_in_ms parameters at 1.5minutes for coordinator wait time. That is the number where I donot see any dropped mutations.
Also asked developers to tweak data model where we saw bunch of tables with really large partition size , some are ranging Partition-key size around ~6.6GB.. we’re now working to reduce the partition size of the tables. I am hoping corrected data model will help reduce coordinator wait time (get back to default number!) again.
Thank again/Asad
From: Akhil Mehra [mailto:akhilmehra@gmail.com]
Sent: Friday, July 21, 2017 4:24 PM
To: user@cassandra.apache.org
Subject: Re: MUTATION messages were dropped in last 5000 ms for cross node timeout
Hi Asad,
The 5000 ms is not configurable (https://github.com/apache/cassandra/blob/8b3a60b9a7dbefeecc06bace617279612ec7092d/src/java/org/apache/cassandra/net/MessagingService.java#L423<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_cassandra_blob_8b3a60b9a7dbefeecc06bace617279612ec7092d_src_java_org_apache_cassandra_net_MessagingService.java-23L423&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=FsmDztdsVuIKml8IDhdHdg&m=dp_TvjXTbUtu3Iu43aZ83eHl1fgW6l4P4PSQglF855g&s=USbrEM6jaGFIRKSUhJBx3VAkSSrXzid0db6TDV1vrDs&e=>). This just the time after which the number of dropped messages are reported. Thus dropped messages are reported every 5000ms.
If you are looking to tweak the number of ms after which a message is considered dropped then you need to use the write_request_timeout_in_ms. The write_request_timeout_in_ms (http://docs.datastax.com/en/cassandra/2.1/cassandra/configuration/configCassandra_yaml_r.html<https://urldefense.proofpoint.com/v2/url?u=http-3A__docs.datastax.com_en_cassandra_2.1_cassandra_configuration_configCassandra-5Fyaml-5Fr.html&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=FsmDztdsVuIKml8IDhdHdg&m=dp_TvjXTbUtu3Iu43aZ83eHl1fgW6l4P4PSQglF855g&s=ab1NW9WoAXIlxT2kWjsiYFVaVidEnC_MB770pwTtqLs&e=>) can be used to increase the mutation timeout. By default it is set to 2000ms.
I hope that helps.
Regards,
Akhil
On 22/07/2017, at 2:46 AM, ZAIDI, ASAD A <az...@att.com>> wrote:
Hi Akhil,
Thank you for your reply. Previously, I did ‘tune’ various timeouts – basically increased them a bit but none of those parameter listed in the link matches with that “were dropped in last 5000 ms”.
I was wondering from where that [5000ms] number is coming from when, like I mentioned before, none of any timeout parameter settings matches that #!
Load is intermittently high but again cpu queue length never goes beyond medium depth. I wonder if there is some internal limit that I’m still not aware of.
Thanks/Asad
From: Akhil Mehra [mailto:akhilmehra@gmail.com]
Sent: Thursday, July 20, 2017 3:47 PM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: Re: MUTATION messages were dropped in last 5000 ms for cross node timeout
Hi Asad,
http://cassandra.apache.org/doc/latest/faq/index.html#why-message-dropped<https://urldefense.proofpoint.com/v2/url?u=http-3A__cassandra.apache.org_doc_latest_faq_index.html-23why-2Dmessage-2Ddropped&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=FsmDztdsVuIKml8IDhdHdg&m=WcHuHKcjg2YCsAbw2NR_0-CiHr9JNxtCzYikia16mpo&s=0_0pQfoOZLuswpQ_lE-AU2bTMFLgRbR4k4Kh8vEOZSk&e=>
As mentioned in the link above this is a load shedding mechanism used by Cassandra.
Is you cluster under heavy load?
Regards,
Akhil
On 21/07/2017, at 3:27 AM, ZAIDI, ASAD A <az...@att.com>> wrote:
Hello Folks –
I’m using apache-cassandra 2.2.8.
I see many messages like below in my system.log file. In Cassandra.yaml file [ cross_node_timeout: true] is set and NTP server is also running correcting clock drift on 16node cluster. I do not see pending or blocked HintedHandoff in tpstats output though there are bunch of MUTATIONS dropped observed.
<start timeout message >
INFO [ScheduledTasks:1] 2017-07-20 08:02:52,511 MessagingService.java:946 - MUTATION messages were dropped in last 5000 ms: 822 for internal timeout and 2152 for cross node timeout
<end timeout message >
I’m seeking help here if you please let me know what I need to check in order to address these cross node timeouts.
Thank you,
Asad
Re: MUTATION messages were dropped in last 5000 ms for cross node
timeout
Posted by Akhil Mehra <ak...@gmail.com>.
Hi Asad,
The 5000 ms is not configurable (https://github.com/apache/cassandra/blob/8b3a60b9a7dbefeecc06bace617279612ec7092d/src/java/org/apache/cassandra/net/MessagingService.java#L423 <https://github.com/apache/cassandra/blob/8b3a60b9a7dbefeecc06bace617279612ec7092d/src/java/org/apache/cassandra/net/MessagingService.java#L423>). This just the time after which the number of dropped messages are reported. Thus dropped messages are reported every 5000ms.
If you are looking to tweak the number of ms after which a message is considered dropped then you need to use the write_request_timeout_in_ms. The write_request_timeout_in_ms (http://docs.datastax.com/en/cassandra/2.1/cassandra/configuration/configCassandra_yaml_r.html <http://docs.datastax.com/en/cassandra/2.1/cassandra/configuration/configCassandra_yaml_r.html>) can be used to increase the mutation timeout. By default it is set to 2000ms.
I hope that helps.
Regards,
Akhil
> On 22/07/2017, at 2:46 AM, ZAIDI, ASAD A <az...@att.com> wrote:
>
> Hi Akhil,
>
> Thank you for your reply. Previously, I did ‘tune’ various timeouts – basically increased them a bit but none of those parameter listed in the link matches with that “were dropped in last 5000 ms”.
> I was wondering from where that [5000ms] number is coming from when, like I mentioned before, none of any timeout parameter settings matches that #!
>
> Load is intermittently high but again cpu queue length never goes beyond medium depth. I wonder if there is some internal limit that I’m still not aware of.
>
> Thanks/Asad
>
>
> From: Akhil Mehra [mailto:akhilmehra@gmail.com]
> Sent: Thursday, July 20, 2017 3:47 PM
> To: user@cassandra.apache.org
> Subject: Re: MUTATION messages were dropped in last 5000 ms for cross node timeout
>
> Hi Asad,
>
> http://cassandra.apache.org/doc/latest/faq/index.html#why-message-dropped <https://urldefense.proofpoint.com/v2/url?u=http-3A__cassandra.apache.org_doc_latest_faq_index.html-23why-2Dmessage-2Ddropped&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=FsmDztdsVuIKml8IDhdHdg&m=WcHuHKcjg2YCsAbw2NR_0-CiHr9JNxtCzYikia16mpo&s=0_0pQfoOZLuswpQ_lE-AU2bTMFLgRbR4k4Kh8vEOZSk&e=>
>
> As mentioned in the link above this is a load shedding mechanism used by Cassandra.
>
> Is you cluster under heavy load?
>
> Regards,
> Akhil
>
>
> On 21/07/2017, at 3:27 AM, ZAIDI, ASAD A <az192g@att.com <ma...@att.com>> wrote:
>
> Hello Folks –
>
> I’m using apache-cassandra 2.2.8.
>
> I see many messages like below in my system.log file. In Cassandra.yaml file [ cross_node_timeout: true] is set and NTP server is also running correcting clock drift on 16node cluster. I do not see pending or blocked HintedHandoff in tpstats output though there are bunch of MUTATIONS dropped observed.
>
> <start timeout message >
> INFO [ScheduledTasks:1] 2017-07-20 08:02:52,511 MessagingService.java:946 - MUTATION messages were dropped in last 5000 ms: 822 for internal timeout and 2152 for cross node timeout
> <end timeout message >
>
> I’m seeking help here if you please let me know what I need to check in order to address these cross node timeouts.
>
> Thank you,
> Asad
RE: MUTATION messages were dropped in last 5000 ms for cross node
timeout
Posted by "ZAIDI, ASAD A" <az...@att.com>.
Hi Akhil,
Thank you for your reply. Previously, I did ‘tune’ various timeouts – basically increased them a bit but none of those parameter listed in the link matches with that “were dropped in last 5000 ms”.
I was wondering from where that [5000ms] number is coming from when, like I mentioned before, none of any timeout parameter settings matches that #!
Load is intermittently high but again cpu queue length never goes beyond medium depth. I wonder if there is some internal limit that I’m still not aware of.
Thanks/Asad
From: Akhil Mehra [mailto:akhilmehra@gmail.com]
Sent: Thursday, July 20, 2017 3:47 PM
To: user@cassandra.apache.org
Subject: Re: MUTATION messages were dropped in last 5000 ms for cross node timeout
Hi Asad,
http://cassandra.apache.org/doc/latest/faq/index.html#why-message-dropped<https://urldefense.proofpoint.com/v2/url?u=http-3A__cassandra.apache.org_doc_latest_faq_index.html-23why-2Dmessage-2Ddropped&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=FsmDztdsVuIKml8IDhdHdg&m=WcHuHKcjg2YCsAbw2NR_0-CiHr9JNxtCzYikia16mpo&s=0_0pQfoOZLuswpQ_lE-AU2bTMFLgRbR4k4Kh8vEOZSk&e=>
As mentioned in the link above this is a load shedding mechanism used by Cassandra.
Is you cluster under heavy load?
Regards,
Akhil
On 21/07/2017, at 3:27 AM, ZAIDI, ASAD A <az...@att.com>> wrote:
Hello Folks –
I’m using apache-cassandra 2.2.8.
I see many messages like below in my system.log file. In Cassandra.yaml file [ cross_node_timeout: true] is set and NTP server is also running correcting clock drift on 16node cluster. I do not see pending or blocked HintedHandoff in tpstats output though there are bunch of MUTATIONS dropped observed.
<start timeout message >
INFO [ScheduledTasks:1] 2017-07-20 08:02:52,511 MessagingService.java:946 - MUTATION messages were dropped in last 5000 ms: 822 for internal timeout and 2152 for cross node timeout
<end timeout message >
I’m seeking help here if you please let me know what I need to check in order to address these cross node timeouts.
Thank you,
Asad
Re: MUTATION messages were dropped in last 5000 ms for cross node
timeout
Posted by Akhil Mehra <ak...@gmail.com>.
Hi Asad,
http://cassandra.apache.org/doc/latest/faq/index.html#why-message-dropped <http://cassandra.apache.org/doc/latest/faq/index.html#why-message-dropped>
As mentioned in the link above this is a load shedding mechanism used by Cassandra.
Is you cluster under heavy load?
Regards,
Akhil
> On 21/07/2017, at 3:27 AM, ZAIDI, ASAD A <az...@att.com> wrote:
>
> Hello Folks –
>
> I’m using apache-cassandra 2.2.8.
>
> I see many messages like below in my system.log file. In Cassandra.yaml file [ cross_node_timeout: true] is set and NTP server is also running correcting clock drift on 16node cluster. I do not see pending or blocked HintedHandoff in tpstats output though there are bunch of MUTATIONS dropped observed.
>
> <start timeout message >
> INFO [ScheduledTasks:1] 2017-07-20 08:02:52,511 MessagingService.java:946 - MUTATION messages were dropped in last 5000 ms: 822 for internal timeout and 2152 for cross node timeout
> <end timeout message >
>
> I’m seeking help here if you please let me know what I need to check in order to address these cross node timeouts.
>
> Thank you,
> Asad
Re: MUTATION messages were dropped in last 5000 ms for cross node timeout
Posted by Erick Ramirez <fl...@gmail.com>.
Mutations get dropped because a node can't keep up with writes. If you
understand the Cassandra write path, writes are ACKed when the mutation is
appended to the commitlog which is why it's very fast.
Knowing that, dropped mutations mean that the disk is not able to keep up
with the IO. Another word for it is "overloaded".
So what do you do? Use fast SSDs (HDDs and EBS won't do) and size your
cluster correctly to deal with the peak load, i.e. add more nodes. Cheers!
On Fri, Jul 21, 2017 at 1:27 AM, ZAIDI, ASAD A <az...@att.com> wrote:
> Hello Folks –
>
>
>
> I’m using apache-cassandra 2.2.8.
>
>
>
> I see many messages like below in my system.log file. In Cassandra.yaml
> file [ cross_node_timeout: true] is set and NTP server is also running
> correcting clock drift on 16node cluster. I do not see pending or blocked
> HintedHandoff in tpstats output though there are bunch of MUTATIONS
> dropped observed.
>
>
>
> <start timeout message >
>
> INFO [ScheduledTasks:1] 2017-07-20 08:02:52,511 MessagingService.java:946
> - MUTATION messages were dropped in last 5000 ms: 822 for internal timeout
> and 2152 for cross node timeout
>
> <end timeout message >
>
>
>
> I’m seeking help here if you please let me know what I need to check in
> order to address these cross node timeouts.
>
>
>
> Thank you,
>
> Asad
>
>
>
RE: MUTATION messages were dropped in last 5000 ms for cross node
timeout
Posted by "ZAIDI, ASAD A" <az...@att.com>.
Thanks for your reply Subroto – I’ll try you suggestions to see if this help. I’ll revert with results.
From: Subroto Barua [mailto:sbarua116@yahoo.com.INVALID]
Sent: Thursday, July 20, 2017 12:22 PM
To: user@cassandra.apache.org
Subject: Re: MUTATION messages were dropped in last 5000 ms for cross node timeout
In a cloud environment, cross_node_timeout = true can cause issues; we had this issue in our environment and it is set to false now.
Dropped messages is an another issue
Subroto
On Jul 20, 2017, at 8:27 AM, ZAIDI, ASAD A <az...@att.com>> wrote:
Hello Folks –
I’m using apache-cassandra 2.2.8.
I see many messages like below in my system.log file. In Cassandra.yaml file [ cross_node_timeout: true] is set and NTP server is also running correcting clock drift on 16node cluster. I do not see pending or blocked HintedHandoff in tpstats output though there are bunch of MUTATIONS dropped observed.
<start timeout message >
INFO [ScheduledTasks:1] 2017-07-20 08:02:52,511 MessagingService.java:946 - MUTATION messages were dropped in last 5000 ms: 822 for internal timeout and 2152 for cross node timeout
<end timeout message >
I’m seeking help here if you please let me know what I need to check in order to address these cross node timeouts.
Thank you,
Asad
Re: MUTATION messages were dropped in last 5000 ms for cross node timeout
Posted by Subroto Barua <sb...@yahoo.com.INVALID>.
In a cloud environment, cross_node_timeout = true can cause issues; we had this issue in our environment and it is set to false now.
Dropped messages is an another issue
Subroto
> On Jul 20, 2017, at 8:27 AM, ZAIDI, ASAD A <az...@att.com> wrote:
>
> Hello Folks –
>
> I’m using apache-cassandra 2.2.8.
>
> I see many messages like below in my system.log file. In Cassandra.yaml file [ cross_node_timeout: true] is set and NTP server is also running correcting clock drift on 16node cluster. I do not see pending or blocked HintedHandoff in tpstats output though there are bunch of MUTATIONS dropped observed.
>
> <start timeout message >
> INFO [ScheduledTasks:1] 2017-07-20 08:02:52,511 MessagingService.java:946 - MUTATION messages were dropped in last 5000 ms: 822 for internal timeout and 2152 for cross node timeout
> <end timeout message >
>
> I’m seeking help here if you please let me know what I need to check in order to address these cross node timeouts.
>
> Thank you,
> Asad
>
RE: MUTATION messages were dropped in last 5000 ms for cross node
timeout
Posted by Anuj Wadehra <an...@yahoo.co.in.INVALID>.
Hi Asad,
You can increase it by 2 at a time. For example if its currently 2, try increasing it to 4 and retest.
We flush 5-6 tables at a time and use 3 memtable_flush_writers. It works great!! There were dropped mutations when it was set to one. The idea is to make sure that writes are not blocked.
ThanksAnuj
Sent from Yahoo Mail on Android
On Fri, 21 Jul 2017 at 20:04, ZAIDI, ASAD A<az...@att.com> wrote: #yiv0831784205 #yiv0831784205 -- _filtered #yiv0831784205 {panose-1:2 4 5 3 5 4 6 3 2 4;} _filtered #yiv0831784205 {font-family:Calibri;panose-1:2 15 5 2 2 2 4 3 2 4;}#yiv0831784205 #yiv0831784205 p.yiv0831784205MsoNormal, #yiv0831784205 li.yiv0831784205MsoNormal, #yiv0831784205 div.yiv0831784205MsoNormal {margin:0in;margin-bottom:.0001pt;font-size:12.0pt;}#yiv0831784205 a:link, #yiv0831784205 span.yiv0831784205MsoHyperlink {color:blue;text-decoration:underline;}#yiv0831784205 a:visited, #yiv0831784205 span.yiv0831784205MsoHyperlinkFollowed {color:purple;text-decoration:underline;}#yiv0831784205 p.yiv0831784205msonormal, #yiv0831784205 li.yiv0831784205msonormal, #yiv0831784205 div.yiv0831784205msonormal {margin-right:0in;margin-left:0in;font-size:12.0pt;}#yiv0831784205 p.yiv0831784205msochpdefault, #yiv0831784205 li.yiv0831784205msochpdefault, #yiv0831784205 div.yiv0831784205msochpdefault {margin-right:0in;margin-left:0in;font-size:12.0pt;}#yiv0831784205 span.yiv0831784205msohyperlink {}#yiv0831784205 span.yiv0831784205msohyperlinkfollowed {}#yiv0831784205 span.yiv0831784205emailstyle17 {}#yiv0831784205 p.yiv0831784205msonormal1, #yiv0831784205 li.yiv0831784205msonormal1, #yiv0831784205 div.yiv0831784205msonormal1 {margin:0in;margin-bottom:.0001pt;font-size:11.0pt;}#yiv0831784205 span.yiv0831784205msohyperlink1 {color:#0563C1;text-decoration:underline;}#yiv0831784205 span.yiv0831784205msohyperlinkfollowed1 {color:#954F72;text-decoration:underline;}#yiv0831784205 span.yiv0831784205emailstyle171 {color:windowtext;}#yiv0831784205 p.yiv0831784205msochpdefault1, #yiv0831784205 li.yiv0831784205msochpdefault1, #yiv0831784205 div.yiv0831784205msochpdefault1 {margin-right:0in;margin-left:0in;font-size:12.0pt;}#yiv0831784205 span.yiv0831784205EmailStyle27 {color:#1F497D;}#yiv0831784205 .yiv0831784205MsoChpDefault {} _filtered #yiv0831784205 {margin:1.0in 1.0in 1.0in 1.0in;}#yiv0831784205 div.yiv0831784205WordSection1 {}#yiv0831784205
Thank you for your reply. I’ll increase memTable_flush_writes and report back if it helps.
Is there any formula that we can use to arrive at correct number of memTable_flush_writers ? or the exercise would windup be like “try and error” taking much time to arrive at some number that may not be optimal.
Thank you again.
From: Anuj Wadehra [mailto:anujw_2003@yahoo.co.in]
Sent: Thursday, July 20, 2017 12:17 PM
To: ZAIDI, ASAD A <az...@att.com>; user@cassandra.apache.org
Subject: Re: MUTATION messages were dropped in last 5000 ms for cross node timeout
Hi Asad,
You can do following things:
1.Increase memtable_flush_writers especially if you have a write heavy load.
2.Make sure there are no big gc pauses on your nodes. If yes, go for heap tuning.
Please let us know whether above measures fixed your problem or not.
Thanks
Anuj
Sent from Yahoo Mail on Android
On Thu, 20 Jul 2017 at 20:57, ZAIDI, ASAD A
<az...@att.com> wrote:
Hello Folks –
I’m using apache-cassandra 2.2.8.
I see many messages like below in my system.log file. In Cassandra.yaml file [ cross_node_timeout: true] is set and NTP server is also running correcting clock drift on 16node cluster. I do not see pending or blocked HintedHandoff in tpstats output though there are bunch of MUTATIONS dropped observed.
<start timeout message >
INFO [ScheduledTasks:1] 2017-07-20 08:02:52,511 MessagingService.java:946 - MUTATION messages were dropped in last 5000 ms: 822 for internal timeout and 2152 for cross node timeout
<end timeout message >
I’m seeking help here if you please let me know what I need to check in order to address these cross node timeouts.
Thank you,
Asad
RE: MUTATION messages were dropped in last 5000 ms for cross node
timeout
Posted by "ZAIDI, ASAD A" <az...@att.com>.
Thank you for your reply. I’ll increase memTable_flush_writes and report back if it helps.
Is there any formula that we can use to arrive at correct number of memTable_flush_writers ? or the exercise would windup be like “try and error” taking much time to arrive at some number that may not be optimal.
Thank you again.
From: Anuj Wadehra [mailto:anujw_2003@yahoo.co.in]
Sent: Thursday, July 20, 2017 12:17 PM
To: ZAIDI, ASAD A <az...@att.com>; user@cassandra.apache.org
Subject: Re: MUTATION messages were dropped in last 5000 ms for cross node timeout
Hi Asad,
You can do following things:
1.Increase memtable_flush_writers especially if you have a write heavy load.
2.Make sure there are no big gc pauses on your nodes. If yes, go for heap tuning.
Please let us know whether above measures fixed your problem or not.
Thanks
Anuj
Sent from Yahoo Mail on Android<https://urldefense.proofpoint.com/v2/url?u=https-3A__overview.mail.yahoo.com_mobile_-3F.src-3DAndroid&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=FsmDztdsVuIKml8IDhdHdg&m=wu-cVspapPKFg21Ul4vRDSw9hpc-s7H1OxPMTPTEApE&s=YCtcGLTbnUV3ZWX_ZFNGe-DCoXPuokN0NCCwS_ahD2I&e=>
On Thu, 20 Jul 2017 at 20:57, ZAIDI, ASAD A
<az...@att.com>> wrote:
Hello Folks –
I’m using apache-cassandra 2.2.8.
I see many messages like below in my system.log file. In Cassandra.yaml file [ cross_node_timeout: true] is set and NTP server is also running correcting clock drift on 16node cluster. I do not see pending or blocked HintedHandoff in tpstats output though there are bunch of MUTATIONS dropped observed.
<start timeout message >
INFO [ScheduledTasks:1] 2017-07-20 08:02:52,511 MessagingService.java:946 - MUTATION messages were dropped in last 5000 ms: 822 for internal timeout and 2152 for cross node timeout
<end timeout message >
I’m seeking help here if you please let me know what I need to check in order to address these cross node timeouts.
Thank you,
Asad
Re: MUTATION messages were dropped in last 5000 ms for cross node
timeout
Posted by Anuj Wadehra <an...@yahoo.co.in.INVALID>.
Hi Asad,
You can do following things:
1.Increase memtable_flush_writers especially if you have a write heavy load.
2.Make sure there are no big gc pauses on your nodes. If yes, go for heap tuning.
Please let us know whether above measures fixed your problem or not.
ThanksAnuj
Sent from Yahoo Mail on Android
On Thu, 20 Jul 2017 at 20:57, ZAIDI, ASAD A<az...@att.com> wrote: <!--#yiv1235039646 _filtered #yiv1235039646 {font-family:"Cambria Math";panose-1:2 4 5 3 5 4 6 3 2 4;} _filtered #yiv1235039646 {font-family:Calibri;panose-1:2 15 5 2 2 2 4 3 2 4;}#yiv1235039646 #yiv1235039646 p.yiv1235039646MsoNormal, #yiv1235039646 li.yiv1235039646MsoNormal, #yiv1235039646 div.yiv1235039646MsoNormal {margin:0in;margin-bottom:.0001pt;font-size:11.0pt;font-family:"Calibri", sans-serif;}#yiv1235039646 a:link, #yiv1235039646 span.yiv1235039646MsoHyperlink {color:#0563C1;text-decoration:underline;}#yiv1235039646 a:visited, #yiv1235039646 span.yiv1235039646MsoHyperlinkFollowed {color:#954F72;text-decoration:underline;}#yiv1235039646 span.yiv1235039646EmailStyle17 {font-family:"Calibri", sans-serif;color:windowtext;}#yiv1235039646 .yiv1235039646MsoChpDefault {font-family:"Calibri", sans-serif;} _filtered #yiv1235039646 {margin:1.0in 1.0in 1.0in 1.0in;}#yiv1235039646 div.yiv1235039646WordSection1 {}-->
Hello Folks –
I’m using apache-cassandra 2.2.8.
I see many messages like below in my system.log file. In Cassandra.yaml file [ cross_node_timeout: true] is set and NTP server is also running correcting clock drift on 16node cluster. I do not see pending or blocked HintedHandoff in tpstats output though there are bunch of MUTATIONS dropped observed.
<start timeout message >
INFO [ScheduledTasks:1] 2017-07-20 08:02:52,511 MessagingService.java:946 - MUTATION messages were dropped in last 5000 ms: 822 for internal timeout and 2152 for cross node timeout
<end timeout message >
I’m seeking help here if you please let me know what I need to check in order to address these cross node timeouts.
Thank you,
Asad