You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Anumod Mullachery <an...@gmail.com> on 2017/11/01 19:24:16 UTC
Cassandra 3.10 - Hints & Dropped messages logs Vs Cass 2.x version
Hi All,
In cassandra v 2.1.15 , I'm able to pull the hints drop and dropped
messages from cassandra.log as below-
dropped hints-->
"/opt/xcal/apps/cassandra/logs/cassandra.log
<https://splunk.ccp.cable.comcast.com/en-US/app/search/search?q=search%20ring%3A%3A*%20%20%20NOT%20ring%3A%3AXHOMEcls_Prod0%20%20%20NOT%20ring%3A%3AXBO_prod0%20source%3D%22*cassandra.log*%22%20%22dropped%20hints%22%20%20ring%3DCI_ParentalControlCQL%20%7C%20rex%20%22HintedHandoffMetrics%20%5C%2F(%3F%3Cdestination%3E.*)%5Cshas%20(%3F%3Cdropped%3E%5Cd%2B)%22%20%7C%20bucket%20span%3D15m%20_time%20&display.page.search.mode=smart&dispatch.sample_ratio=1&earliest=-4h&latest=now&display.page.search.tab=events&display.general.type=events&sid=1509561773.447850_874E67FA-980C-410D-A52B-FE91D5E0AAAE#>
--> HintedHandoffMetrics.java:79 - /96.115.91.69 has 5 dropped hints,
because node is down past configured hint window."
Dropped messages -->
"/opt/xcal/apps/cassandra/logs/cassandra.log
<https://splunk.ccp.cable.comcast.com/en-US/app/search/search?q=search%20ring%3A%3A*%20ring!%3D*xbo*%20(sourcetype%3Dcassandra%20OR%20sourcetype%3Ddse)%20%22messages%20dropped%22%20&display.page.search.mode=smart&dispatch.sample_ratio=1&earliest=-4h&latest=now&display.page.search.tab=events&display.general.type=events&sid=1509562391.448085_874E67FA-980C-410D-A52B-FE91D5E0AAAE#>
--> NFO MessagingService 1 MUTATION messages dropped in last 5000ms"
But in Cassandra *V 3.10* , I'm not able to get the cassandra logs as v
2.1.15.
I'm not able to get any info from the cassandra logs for dropped hints &
dropped messages. ( I was testing this on Test cluster by stopping the
cassandra services on one of the node , where the hints window is set to 05
Mins and kept the node down for 1 hour , and monitoring the cassandra logs,
but nothing generated on dropped hints on any other nodes in the cluster ).
The only message I found is on the down node is that - "Paused hints
dispatch".
Can some one put some light on this issue on 3.10 , is there any changes in
the hints & dropped messages , logs or the process.
thanks in advance,
regards,
Anumod.
Re: Cassandra 3.10 - Hints & Dropped messages logs Vs Cass 2.x version
Posted by Anumod Mullachery <an...@gmail.com>.
Okay ..
Thanks,will check ..
Anumod
PA,USA
Sent from my iPhone
> On Nov 2, 2017, at 5:40 PM, kurt greaves <ku...@instaclustr.com> wrote:
>
> Well, pretty sure they still are. at least the mutation one is. but you should really use the dedicated metrics for this.
>
>> On 3 Nov. 2017 01:38, "Anumod Mullachery" <an...@gmail.com> wrote:
>> thanks ..
>>
>> so the dropped hints & messages are not captured in cassandra logs, post 3.x vs 2.x.
>>
>> -Anumod
>>
>>
>>
>>> On Wed, Nov 1, 2017 at 4:50 PM, kurt greaves <ku...@instaclustr.com> wrote:
>>> You can get dropped message statistics over JMX. for example nodetool tpstats has a counter for dropped hints from startup. that would be the preferred method for tracking this info, rather than parsing logs
>>>
>>> On 2 Nov. 2017 6:24 am, "Anumod Mullachery" <an...@gmail.com> wrote:
>>>
>>> Hi All,
>>>
>>> In cassandra v 2.1.15 , I'm able to pull the hints drop and dropped messages from cassandra.log as below-
>>>
>>>
>>> dropped hints-->
>>>
>>> "/opt/xcal/apps/cassandra/logs/cassandra.log --> HintedHandoffMetrics.java:79 - /96.115.91.69 has 5 dropped hints, because node is down past configured hint window."
>>>
>>> Dropped messages -->
>>>
>>> "/opt/xcal/apps/cassandra/logs/cassandra.log --> NFO MessagingService 1 MUTATION messages dropped in last 5000ms"
>>>
>>>
>>> But in Cassandra V 3.10 , I'm not able to get the cassandra logs as v 2.1.15.
>>>
>>> I'm not able to get any info from the cassandra logs for dropped hints & dropped messages. ( I was testing this on Test cluster by stopping the cassandra services on one of the node , where the hints window is set to 05 Mins and kept the node down for 1 hour , and monitoring the cassandra logs, but nothing generated on dropped hints on any other nodes in the cluster ).
>>>
>>> The only message I found is on the down node is that - "Paused hints dispatch".
>>>
>>> Can some one put some light on this issue on 3.10 , is there any changes in the hints & dropped messages , logs or the process.
>>>
>>> thanks in advance,
>>>
>>> regards,
>>>
>>> Anumod.
>>>
>>>
>>
Re: Cassandra 3.10 - Hints & Dropped messages logs Vs Cass 2.x version
Posted by kurt greaves <ku...@instaclustr.com>.
Well, pretty sure they still are. at least the mutation one is. but you
should really use the dedicated metrics for this.
On 3 Nov. 2017 01:38, "Anumod Mullachery" <an...@gmail.com>
wrote:
> thanks ..
>
> so the dropped hints & messages are not captured in cassandra logs, post
> 3.x vs 2.x.
>
> -Anumod
>
>
>
> On Wed, Nov 1, 2017 at 4:50 PM, kurt greaves <ku...@instaclustr.com> wrote:
>
>> You can get dropped message statistics over JMX. for example nodetool
>> tpstats has a counter for dropped hints from startup. that would be the
>> preferred method for tracking this info, rather than parsing logs
>>
>> On 2 Nov. 2017 6:24 am, "Anumod Mullachery" <an...@gmail.com>
>> wrote:
>>
>>
>> Hi All,
>>
>> In cassandra v 2.1.15 , I'm able to pull the hints drop and dropped
>> messages from cassandra.log as below-
>>
>>
>> dropped hints-->
>>
>> "/opt/xcal/apps/cassandra/logs/cassandra.log
>> <https://splunk.ccp.cable.comcast.com/en-US/app/search/search?q=search%20ring%3A%3A*%20%20%20NOT%20ring%3A%3AXHOMEcls_Prod0%20%20%20NOT%20ring%3A%3AXBO_prod0%20source%3D%22*cassandra.log*%22%20%22dropped%20hints%22%20%20ring%3DCI_ParentalControlCQL%20%7C%20rex%20%22HintedHandoffMetrics%20%5C%2F(%3F%3Cdestination%3E.*)%5Cshas%20(%3F%3Cdropped%3E%5Cd%2B)%22%20%7C%20bucket%20span%3D15m%20_time%20&display.page.search.mode=smart&dispatch.sample_ratio=1&earliest=-4h&latest=now&display.page.search.tab=events&display.general.type=events&sid=1509561773.447850_874E67FA-980C-410D-A52B-FE91D5E0AAAE#>
>> --> HintedHandoffMetrics.java:79 - /96.115.91.69 has 5 dropped hints,
>> because node is down past configured hint window."
>>
>> Dropped messages -->
>>
>> "/opt/xcal/apps/cassandra/logs/cassandra.log
>> <https://splunk.ccp.cable.comcast.com/en-US/app/search/search?q=search%20ring%3A%3A*%20ring!%3D*xbo*%20(sourcetype%3Dcassandra%20OR%20sourcetype%3Ddse)%20%22messages%20dropped%22%20&display.page.search.mode=smart&dispatch.sample_ratio=1&earliest=-4h&latest=now&display.page.search.tab=events&display.general.type=events&sid=1509562391.448085_874E67FA-980C-410D-A52B-FE91D5E0AAAE#>
>> --> NFO MessagingService 1 MUTATION messages dropped in last 5000ms"
>>
>>
>> But in Cassandra *V 3.10* , I'm not able to get the cassandra logs as v
>> 2.1.15.
>>
>> I'm not able to get any info from the cassandra logs for dropped hints &
>> dropped messages. ( I was testing this on Test cluster by stopping the
>> cassandra services on one of the node , where the hints window is set to 05
>> Mins and kept the node down for 1 hour , and monitoring the cassandra logs,
>> but nothing generated on dropped hints on any other nodes in the cluster ).
>>
>> The only message I found is on the down node is that - "Paused hints
>> dispatch".
>>
>> Can some one put some light on this issue on 3.10 , is there any changes
>> in the hints & dropped messages , logs or the process.
>>
>> thanks in advance,
>>
>> regards,
>>
>> Anumod.
>>
>>
>>
>
Re: Cassandra 3.10 - Hints & Dropped messages logs Vs Cass 2.x version
Posted by Anumod Mullachery <an...@gmail.com>.
thanks ..
so the dropped hints & messages are not captured in cassandra logs, post
3.x vs 2.x.
-Anumod
On Wed, Nov 1, 2017 at 4:50 PM, kurt greaves <ku...@instaclustr.com> wrote:
> You can get dropped message statistics over JMX. for example nodetool
> tpstats has a counter for dropped hints from startup. that would be the
> preferred method for tracking this info, rather than parsing logs
>
> On 2 Nov. 2017 6:24 am, "Anumod Mullachery" <an...@gmail.com>
> wrote:
>
>
> Hi All,
>
> In cassandra v 2.1.15 , I'm able to pull the hints drop and dropped
> messages from cassandra.log as below-
>
>
> dropped hints-->
>
> "/opt/xcal/apps/cassandra/logs/cassandra.log
> <https://splunk.ccp.cable.comcast.com/en-US/app/search/search?q=search%20ring%3A%3A*%20%20%20NOT%20ring%3A%3AXHOMEcls_Prod0%20%20%20NOT%20ring%3A%3AXBO_prod0%20source%3D%22*cassandra.log*%22%20%22dropped%20hints%22%20%20ring%3DCI_ParentalControlCQL%20%7C%20rex%20%22HintedHandoffMetrics%20%5C%2F(%3F%3Cdestination%3E.*)%5Cshas%20(%3F%3Cdropped%3E%5Cd%2B)%22%20%7C%20bucket%20span%3D15m%20_time%20&display.page.search.mode=smart&dispatch.sample_ratio=1&earliest=-4h&latest=now&display.page.search.tab=events&display.general.type=events&sid=1509561773.447850_874E67FA-980C-410D-A52B-FE91D5E0AAAE#>
> --> HintedHandoffMetrics.java:79 - /96.115.91.69 has 5 dropped hints,
> because node is down past configured hint window."
>
> Dropped messages -->
>
> "/opt/xcal/apps/cassandra/logs/cassandra.log
> <https://splunk.ccp.cable.comcast.com/en-US/app/search/search?q=search%20ring%3A%3A*%20ring!%3D*xbo*%20(sourcetype%3Dcassandra%20OR%20sourcetype%3Ddse)%20%22messages%20dropped%22%20&display.page.search.mode=smart&dispatch.sample_ratio=1&earliest=-4h&latest=now&display.page.search.tab=events&display.general.type=events&sid=1509562391.448085_874E67FA-980C-410D-A52B-FE91D5E0AAAE#>
> --> NFO MessagingService 1 MUTATION messages dropped in last 5000ms"
>
>
> But in Cassandra *V 3.10* , I'm not able to get the cassandra logs as v
> 2.1.15.
>
> I'm not able to get any info from the cassandra logs for dropped hints &
> dropped messages. ( I was testing this on Test cluster by stopping the
> cassandra services on one of the node , where the hints window is set to 05
> Mins and kept the node down for 1 hour , and monitoring the cassandra logs,
> but nothing generated on dropped hints on any other nodes in the cluster ).
>
> The only message I found is on the down node is that - "Paused hints
> dispatch".
>
> Can some one put some light on this issue on 3.10 , is there any changes
> in the hints & dropped messages , logs or the process.
>
> thanks in advance,
>
> regards,
>
> Anumod.
>
>
>
Re: Cassandra 3.10 - Hints & Dropped messages logs Vs Cass 2.x version
Posted by kurt greaves <ku...@instaclustr.com>.
You can get dropped message statistics over JMX. for example nodetool
tpstats has a counter for dropped hints from startup. that would be the
preferred method for tracking this info, rather than parsing logs
On 2 Nov. 2017 6:24 am, "Anumod Mullachery" <an...@gmail.com>
wrote:
Hi All,
In cassandra v 2.1.15 , I'm able to pull the hints drop and dropped
messages from cassandra.log as below-
dropped hints-->
"/opt/xcal/apps/cassandra/logs/cassandra.log
<https://splunk.ccp.cable.comcast.com/en-US/app/search/search?q=search%20ring%3A%3A*%20%20%20NOT%20ring%3A%3AXHOMEcls_Prod0%20%20%20NOT%20ring%3A%3AXBO_prod0%20source%3D%22*cassandra.log*%22%20%22dropped%20hints%22%20%20ring%3DCI_ParentalControlCQL%20%7C%20rex%20%22HintedHandoffMetrics%20%5C%2F(%3F%3Cdestination%3E.*)%5Cshas%20(%3F%3Cdropped%3E%5Cd%2B)%22%20%7C%20bucket%20span%3D15m%20_time%20&display.page.search.mode=smart&dispatch.sample_ratio=1&earliest=-4h&latest=now&display.page.search.tab=events&display.general.type=events&sid=1509561773.447850_874E67FA-980C-410D-A52B-FE91D5E0AAAE#>
--> HintedHandoffMetrics.java:79 - /96.115.91.69 has 5 dropped hints,
because node is down past configured hint window."
Dropped messages -->
"/opt/xcal/apps/cassandra/logs/cassandra.log
<https://splunk.ccp.cable.comcast.com/en-US/app/search/search?q=search%20ring%3A%3A*%20ring!%3D*xbo*%20(sourcetype%3Dcassandra%20OR%20sourcetype%3Ddse)%20%22messages%20dropped%22%20&display.page.search.mode=smart&dispatch.sample_ratio=1&earliest=-4h&latest=now&display.page.search.tab=events&display.general.type=events&sid=1509562391.448085_874E67FA-980C-410D-A52B-FE91D5E0AAAE#>
--> NFO MessagingService 1 MUTATION messages dropped in last 5000ms"
But in Cassandra *V 3.10* , I'm not able to get the cassandra logs as v
2.1.15.
I'm not able to get any info from the cassandra logs for dropped hints &
dropped messages. ( I was testing this on Test cluster by stopping the
cassandra services on one of the node , where the hints window is set to 05
Mins and kept the node down for 1 hour , and monitoring the cassandra logs,
but nothing generated on dropped hints on any other nodes in the cluster ).
The only message I found is on the down node is that - "Paused hints
dispatch".
Can some one put some light on this issue on 3.10 , is there any changes in
the hints & dropped messages , logs or the process.
thanks in advance,
regards,
Anumod.