You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cassandra.apache.org by Cyril Scetbon <cy...@free.fr> on 2013/11/09 22:44:14 UTC
local truncate of hints
Hi,
I would like to add a feature request for local truncates of hints but I want to know your opinion before.
We faced an issue last week concerning 3 new nodes that where not seen by some other nodes in the ring. a week later we resolved the issues but it was taking days with a high CPU load (85%) to apply GBytes of Hints. I noticed that CF Hints is defined with LocalStategy and so thought the truncate was only local but that wasn't. We had to remove sstables, commitlog and cached to remove all that old hints stored but not removed after max_hint_window_in_ms as target nodes where not dead (they were only not accepting hints).
So, I suggest that the truncate of hints stay local (maybe you need it to stay global per default ?) or that we could do it locally (a truncate argument or a new command like the one used for CONSISTENCY)
thanks
--
Cyril SCETBON
Re: local truncate of hints
Posted by Boris Yen <yu...@gmail.com>.
I quick glimpse at the code. It seems that truncate only cares about if any
node goes down. It does not check the replication strategy.
Boris
On Thu, Nov 14, 2013 at 9:30 PM, Cyril Scetbon <cy...@free.fr>wrote:
> anyone ?
>
> --
> Cyril SCETBON
>
> On 10 Nov 2013, at 10:48, Cyril Scetbon <cy...@free.fr> wrote:
>
> > Are you really sure of that ? cause Tracing says the opposite when I do
> it on one node :
> >
> > cqlsh:system> TRUNCATE hints ;
> >
> > Tracing session: fe0dbc80-49dd-11e3-a39b-e367b713537d
> >
> > activity
> | timestamp | source | source_elapsed
> >
> ----------------------------------------------------------------------------------------+--------------+------------+----------------
> >
> execute_cql3_query | 07:59:15,019 | 10.0.0.37 | 0
> > Parsing
> TRUNCATE hints ; | 07:59:15,020 | 10.0.0.37 | 991
> >
> Preparing statement | 07:59:15,021 | 10.0.0.37 | 2009
> > Enqueuing truncate messages to hosts [/10.0.0.185, /10.0.0.33, /
> 10.0.0.17, /10.0.0.37] | 07:59:15,023 | 10.0.0.37 | 4459
> > Message received
> from /10.0.0.37 | 07:59:15,186 | 10.0.0.33 | 121
> > Applying truncation
> of system.hints | 07:59:15,200 | 10.0.0.33 | 11064
> > Sending message
> to /10.0.0.37 | 07:59:15,214 | 10.0.0.37 | 195660
> > Sending message
> to /10.0.0.33 | 07:59:15,217 | 10.0.0.37 | 198664
> > Message received
> from /10.0.0.37 | 07:59:15,230 | 10.0.0.185 | 149
> > Applying truncation
> of system.hints | 07:59:15,240 | 10.0.0.185 | 6248
> > Message received
> from /10.0.0.37 | 07:59:15,244 | 10.0.0.17 | 117
> > Sending message
> to /10.0.0.185 | 07:59:15,247 | 10.0.0.37 | 134087
> > Sending message
> to /10.0.0.17 | 07:59:15,248 | 10.0.0.37 | 134760
> > Message received
> from /10.0.0.37 | 07:59:15,251 | 10.0.0.37 | 232058
> > Applying truncation
> of system.hints | 07:59:15,254 | 10.0.0.37 | 234904
> > Applying truncation
> of system.hints | 07:59:15,259 | 10.0.0.17 | 8574
> > Enqueuing response to truncate operation
> to /10.0.0.37 | 07:59:15,388 | 10.0.0.17 | 144491
> > Sending message
> to /10.0.0.37 | 07:59:15,389 | 10.0.0.17 | 145341
> > Message received
> from /10.0.0.17 | 07:59:15,461 | 10.0.0.37 | 441863
> > Processing response
> from /10.0.0.17 | 07:59:15,502 | 10.0.0.37 | 482942
> > Enqueuing response to truncate operation
> to /10.0.0.37 | 07:59:15,578 | 10.0.0.33 | 391940
> > Sending message
> to /10.0.0.37 | 07:59:15,578 | 10.0.0.33 | 392364
> > Message received
> from /10.0.0.33 | 07:59:15,632 | 10.0.0.37 | 613172
> > Processing response
> from /10.0.0.33 | 07:59:15,632 | 10.0.0.37 | 613303
> > Enqueuing response to truncate operation
> to /10.0.0.37 | 07:59:15,736 | 10.0.0.37 | 717066
> > Sending message
> to /10.0.0.37 | 07:59:15,738 | 10.0.0.37 | 719165
> > Message received
> from /10.0.0.37 | 07:59:15,739 | 10.0.0.37 | null
> > Processing response
> from /10.0.0.37 | 07:59:15,739 | 10.0.0.37 | null
> > Enqueuing response to truncate operation
> to /10.0.0.37 | 07:59:15,915 | 10.0.0.185 | 685116
> > Sending message
> to /10.0.0.37 | 07:59:15,915 | 10.0.0.185 | 685519
> > Message received
> from /10.0.0.185 | 07:59:15,936 | 10.0.0.37 | null
> > Processing response
> from /10.0.0.185 | 07:59:15,936 | 10.0.0.37 | null
> >
> Request complete | 07:59:15,936 | 10.0.0.37 | 917846
> >
> > I also tried to generate different hints on different nodes, and
> truncating the table on one node removed data everywhere :
> >
> >
> > node 1
> >
> > at t1
> >
> > cqlsh> select * from system.hints ;
> >
> > target_id | hint_id
> | message_version | mutation
> >
> --------------------------------------+--------------------------------------+-----------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> > 748cb454-da16-4218-b6ac-728f0fe0cf41 |
> 9f152ea0-49e9-11e3-af88-ed7ea188ea44 | 6 |
> 0x0006706e735f667200086373636574626f6e00000001019cad5323f2a6389388a31004124bdf2d7fffffff800000000000000000000000000000020003000000000004eacf278eb3a0000000000006000376616c00000004eacf278eb3a0000000040000270f
> > 78078974-1f02-4da8-9902-cca56291a9dc |
> 474e4c30-49ec-11e3-af88-ed7ea188ea44 | 6 |
> 0x0006706e735f6672000868706172736f6e7300000001019cad5323f2a6389388a31004124bdf2d7fffffff800000000000000000000000000000020003000000000004eacf6b945e10000000000006000376616c00000004eacf6b945e1000000004000010f7
> > 78078974-1f02-4da8-9902-cca56291a9dc |
> 4e64df70-49ec-11e3-af88-ed7ea188ea44 | 6 |
> 0x0006706e735f667200086373636574626f6e00000001019cad5323f2a6389388a31004124bdf2d7fffffff800000000000000000000000000000020003000000000004eacf6c49f270000000000006000376616c00000004eacf6c49f2700000000400001618
> >
> > at t2
> >
> > cqlsh:pns_fr> TRUNCATE system.hints ;
> > cqlsh:pns_fr> select * from system.hints ;
> >
> > node 2
> >
> > at t1
> >
> > cqlsh:pns_fr> select * from system.hints ;
> >
> > target_id | hint_id
> | message_version | mutation
> >
> --------------------------------------+--------------------------------------+-----------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> > 78078974-1f02-4da8-9902-cca56291a9dc |
> aa8f4da0-49ea-11e3-a8dc-e367b713537d | 6 |
> 0x0006706e735f667200086373636574626f6e00000001019cad5323f2a6389388a31004124bdf2d7fffffff800000000000000000000000000000020003000000000004eacf424d88b0000000000006000376616c00000004eacf424d88b000000004000004bc
> >
> > at t2
> >
> > cqlsh:pns_fr> select * from system.hints ;
> >
> > Regards
> > --
> > Cyril SCETBON
> >
> > On 09 Nov 2013, at 22:48, Jonathan Ellis <jb...@gmail.com> wrote:
> >
> >> Hints are not replicated (technically: replicated with LocalStrategy),
> >> so truncating hints will always only affect the local node.
> >>
> >> On Sat, Nov 9, 2013 at 3:44 PM, Cyril Scetbon <cy...@free.fr>
> wrote:
> >>> Hi,
> >>>
> >>> I would like to add a feature request for local truncates of hints but
> I want to know your opinion before.
> >>>
> >>> We faced an issue last week concerning 3 new nodes that where not seen
> by some other nodes in the ring. a week later we resolved the issues but it
> was taking days with a high CPU load (85%) to apply GBytes of Hints. I
> noticed that CF Hints is defined with LocalStategy and so thought the
> truncate was only local but that wasn't. We had to remove sstables,
> commitlog and cached to remove all that old hints stored but not removed
> after max_hint_window_in_ms as target nodes where not dead (they were only
> not accepting hints).
> >>>
> >>> So, I suggest that the truncate of hints stay local (maybe you need it
> to stay global per default ?) or that we could do it locally (a truncate
> argument or a new command like the one used for CONSISTENCY)
> >>>
> >>> thanks
> >>> --
> >>> Cyril SCETBON
> >>>
> >>
> >>
> >>
> >> --
> >> Jonathan Ellis
> >> Project Chair, Apache Cassandra
> >> co-founder, http://www.datastax.com
> >> @spyced
> >
>
>
Re: local truncate of hints
Posted by Cyril Scetbon <cy...@free.fr>.
anyone ?
--
Cyril SCETBON
On 10 Nov 2013, at 10:48, Cyril Scetbon <cy...@free.fr> wrote:
> Are you really sure of that ? cause Tracing says the opposite when I do it on one node :
>
> cqlsh:system> TRUNCATE hints ;
>
> Tracing session: fe0dbc80-49dd-11e3-a39b-e367b713537d
>
> activity | timestamp | source | source_elapsed
> ----------------------------------------------------------------------------------------+--------------+------------+----------------
> execute_cql3_query | 07:59:15,019 | 10.0.0.37 | 0
> Parsing TRUNCATE hints ; | 07:59:15,020 | 10.0.0.37 | 991
> Preparing statement | 07:59:15,021 | 10.0.0.37 | 2009
> Enqueuing truncate messages to hosts [/10.0.0.185, /10.0.0.33, /10.0.0.17, /10.0.0.37] | 07:59:15,023 | 10.0.0.37 | 4459
> Message received from /10.0.0.37 | 07:59:15,186 | 10.0.0.33 | 121
> Applying truncation of system.hints | 07:59:15,200 | 10.0.0.33 | 11064
> Sending message to /10.0.0.37 | 07:59:15,214 | 10.0.0.37 | 195660
> Sending message to /10.0.0.33 | 07:59:15,217 | 10.0.0.37 | 198664
> Message received from /10.0.0.37 | 07:59:15,230 | 10.0.0.185 | 149
> Applying truncation of system.hints | 07:59:15,240 | 10.0.0.185 | 6248
> Message received from /10.0.0.37 | 07:59:15,244 | 10.0.0.17 | 117
> Sending message to /10.0.0.185 | 07:59:15,247 | 10.0.0.37 | 134087
> Sending message to /10.0.0.17 | 07:59:15,248 | 10.0.0.37 | 134760
> Message received from /10.0.0.37 | 07:59:15,251 | 10.0.0.37 | 232058
> Applying truncation of system.hints | 07:59:15,254 | 10.0.0.37 | 234904
> Applying truncation of system.hints | 07:59:15,259 | 10.0.0.17 | 8574
> Enqueuing response to truncate operation to /10.0.0.37 | 07:59:15,388 | 10.0.0.17 | 144491
> Sending message to /10.0.0.37 | 07:59:15,389 | 10.0.0.17 | 145341
> Message received from /10.0.0.17 | 07:59:15,461 | 10.0.0.37 | 441863
> Processing response from /10.0.0.17 | 07:59:15,502 | 10.0.0.37 | 482942
> Enqueuing response to truncate operation to /10.0.0.37 | 07:59:15,578 | 10.0.0.33 | 391940
> Sending message to /10.0.0.37 | 07:59:15,578 | 10.0.0.33 | 392364
> Message received from /10.0.0.33 | 07:59:15,632 | 10.0.0.37 | 613172
> Processing response from /10.0.0.33 | 07:59:15,632 | 10.0.0.37 | 613303
> Enqueuing response to truncate operation to /10.0.0.37 | 07:59:15,736 | 10.0.0.37 | 717066
> Sending message to /10.0.0.37 | 07:59:15,738 | 10.0.0.37 | 719165
> Message received from /10.0.0.37 | 07:59:15,739 | 10.0.0.37 | null
> Processing response from /10.0.0.37 | 07:59:15,739 | 10.0.0.37 | null
> Enqueuing response to truncate operation to /10.0.0.37 | 07:59:15,915 | 10.0.0.185 | 685116
> Sending message to /10.0.0.37 | 07:59:15,915 | 10.0.0.185 | 685519
> Message received from /10.0.0.185 | 07:59:15,936 | 10.0.0.37 | null
> Processing response from /10.0.0.185 | 07:59:15,936 | 10.0.0.37 | null
> Request complete | 07:59:15,936 | 10.0.0.37 | 917846
>
> I also tried to generate different hints on different nodes, and truncating the table on one node removed data everywhere :
>
>
> node 1
>
> at t1
>
> cqlsh> select * from system.hints ;
>
> target_id | hint_id | message_version | mutation
> --------------------------------------+--------------------------------------+-----------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> 748cb454-da16-4218-b6ac-728f0fe0cf41 | 9f152ea0-49e9-11e3-af88-ed7ea188ea44 | 6 | 0x0006706e735f667200086373636574626f6e00000001019cad5323f2a6389388a31004124bdf2d7fffffff800000000000000000000000000000020003000000000004eacf278eb3a0000000000006000376616c00000004eacf278eb3a0000000040000270f
> 78078974-1f02-4da8-9902-cca56291a9dc | 474e4c30-49ec-11e3-af88-ed7ea188ea44 | 6 | 0x0006706e735f6672000868706172736f6e7300000001019cad5323f2a6389388a31004124bdf2d7fffffff800000000000000000000000000000020003000000000004eacf6b945e10000000000006000376616c00000004eacf6b945e1000000004000010f7
> 78078974-1f02-4da8-9902-cca56291a9dc | 4e64df70-49ec-11e3-af88-ed7ea188ea44 | 6 | 0x0006706e735f667200086373636574626f6e00000001019cad5323f2a6389388a31004124bdf2d7fffffff800000000000000000000000000000020003000000000004eacf6c49f270000000000006000376616c00000004eacf6c49f2700000000400001618
>
> at t2
>
> cqlsh:pns_fr> TRUNCATE system.hints ;
> cqlsh:pns_fr> select * from system.hints ;
>
> node 2
>
> at t1
>
> cqlsh:pns_fr> select * from system.hints ;
>
> target_id | hint_id | message_version | mutation
> --------------------------------------+--------------------------------------+-----------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> 78078974-1f02-4da8-9902-cca56291a9dc | aa8f4da0-49ea-11e3-a8dc-e367b713537d | 6 | 0x0006706e735f667200086373636574626f6e00000001019cad5323f2a6389388a31004124bdf2d7fffffff800000000000000000000000000000020003000000000004eacf424d88b0000000000006000376616c00000004eacf424d88b000000004000004bc
>
> at t2
>
> cqlsh:pns_fr> select * from system.hints ;
>
> Regards
> --
> Cyril SCETBON
>
> On 09 Nov 2013, at 22:48, Jonathan Ellis <jb...@gmail.com> wrote:
>
>> Hints are not replicated (technically: replicated with LocalStrategy),
>> so truncating hints will always only affect the local node.
>>
>> On Sat, Nov 9, 2013 at 3:44 PM, Cyril Scetbon <cy...@free.fr> wrote:
>>> Hi,
>>>
>>> I would like to add a feature request for local truncates of hints but I want to know your opinion before.
>>>
>>> We faced an issue last week concerning 3 new nodes that where not seen by some other nodes in the ring. a week later we resolved the issues but it was taking days with a high CPU load (85%) to apply GBytes of Hints. I noticed that CF Hints is defined with LocalStategy and so thought the truncate was only local but that wasn't. We had to remove sstables, commitlog and cached to remove all that old hints stored but not removed after max_hint_window_in_ms as target nodes where not dead (they were only not accepting hints).
>>>
>>> So, I suggest that the truncate of hints stay local (maybe you need it to stay global per default ?) or that we could do it locally (a truncate argument or a new command like the one used for CONSISTENCY)
>>>
>>> thanks
>>> --
>>> Cyril SCETBON
>>>
>>
>>
>>
>> --
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder, http://www.datastax.com
>> @spyced
>
Re: local truncate of hints
Posted by Cyril Scetbon <cy...@free.fr>.
Thanks I've created JIRA CASSANDRA-6367 for that functionality.
Regards.
On 17 Nov 2013, at 20:33, Jonathan Ellis <jb...@gmail.com> wrote:
> Right.
> On Nov 17, 2013 11:00 AM, "Cyril Scetbon" <cy...@free.fr> wrote:
>
>> and/or a nodetool purgelocalhints command ?
>>
>> On 15 Nov 2013, at 05:56, Jonathan Ellis <jb...@gmail.com> wrote:
>>
>>> You're right. You'd want to expose the node-local truncate over JMX.
>>>
>>> On Sun, Nov 10, 2013 at 3:48 AM, Cyril Scetbon <cy...@free.fr>
>> wrote:
>>>> Are you really sure of that ? cause Tracing says the opposite when I do
>> it on one node :
>>>>
>>>> cqlsh:system> TRUNCATE hints ;
>>>>
>>>> Tracing session: fe0dbc80-49dd-11e3-a39b-e367b713537d
>>>>
>>>> activity
>> | timestamp | source | source_elapsed
>>>>
>> ----------------------------------------------------------------------------------------+--------------+------------+----------------
>>>>
>> execute_cql3_query | 07:59:15,019 | 10.0.0.37 | 0
>>>> Parsing
>> TRUNCATE hints ; | 07:59:15,020 | 10.0.0.37 | 991
>>>>
>> Preparing statement | 07:59:15,021 | 10.0.0.37 | 2009
>>>> Enqueuing truncate messages to hosts [/10.0.0.185, /10.0.0.33, /
>> 10.0.0.17, /10.0.0.37] | 07:59:15,023 | 10.0.0.37 | 4459
>>>> Message received
>> from /10.0.0.37 | 07:59:15,186 | 10.0.0.33 | 121
>>>> Applying truncation
>> of system.hints | 07:59:15,200 | 10.0.0.33 | 11064
>>>> Sending
>> message to /10.0.0.37 | 07:59:15,214 | 10.0.0.37 | 195660
>>>> Sending
>> message to /10.0.0.33 | 07:59:15,217 | 10.0.0.37 | 198664
>>>> Message received
>> from /10.0.0.37 | 07:59:15,230 | 10.0.0.185 | 149
>>>> Applying truncation
>> of system.hints | 07:59:15,240 | 10.0.0.185 | 6248
>>>> Message received
>> from /10.0.0.37 | 07:59:15,244 | 10.0.0.17 | 117
>>>> Sending message
>> to /10.0.0.185 | 07:59:15,247 | 10.0.0.37 | 134087
>>>> Sending
>> message to /10.0.0.17 | 07:59:15,248 | 10.0.0.37 | 134760
>>>> Message received
>> from /10.0.0.37 | 07:59:15,251 | 10.0.0.37 | 232058
>>>> Applying truncation
>> of system.hints | 07:59:15,254 | 10.0.0.37 | 234904
>>>> Applying truncation
>> of system.hints | 07:59:15,259 | 10.0.0.17 | 8574
>>>> Enqueuing response to truncate
>> operation to /10.0.0.37 | 07:59:15,388 | 10.0.0.17 | 144491
>>>> Sending
>> message to /10.0.0.37 | 07:59:15,389 | 10.0.0.17 | 145341
>>>> Message received
>> from /10.0.0.17 | 07:59:15,461 | 10.0.0.37 | 441863
>>>> Processing response
>> from /10.0.0.17 | 07:59:15,502 | 10.0.0.37 | 482942
>>>> Enqueuing response to truncate
>> operation to /10.0.0.37 | 07:59:15,578 | 10.0.0.33 | 391940
>>>> Sending
>> message to /10.0.0.37 | 07:59:15,578 | 10.0.0.33 | 392364
>>>> Message received
>> from /10.0.0.33 | 07:59:15,632 | 10.0.0.37 | 613172
>>>> Processing response
>> from /10.0.0.33 | 07:59:15,632 | 10.0.0.37 | 613303
>>>> Enqueuing response to truncate
>> operation to /10.0.0.37 | 07:59:15,736 | 10.0.0.37 | 717066
>>>> Sending
>> message to /10.0.0.37 | 07:59:15,738 | 10.0.0.37 | 719165
>>>> Message received
>> from /10.0.0.37 | 07:59:15,739 | 10.0.0.37 | null
>>>> Processing response
>> from /10.0.0.37 | 07:59:15,739 | 10.0.0.37 | null
>>>> Enqueuing response to truncate
>> operation to /10.0.0.37 | 07:59:15,915 | 10.0.0.185 | 685116
>>>> Sending
>> message to /10.0.0.37 | 07:59:15,915 | 10.0.0.185 | 685519
>>>> Message received
>> from /10.0.0.185 | 07:59:15,936 | 10.0.0.37 | null
>>>> Processing response
>> from /10.0.0.185 | 07:59:15,936 | 10.0.0.37 | null
>>>>
>> Request complete | 07:59:15,936 | 10.0.0.37 | 917846
>>>>
>>>> I also tried to generate different hints on different nodes, and
>> truncating the table on one node removed data everywhere :
>>>>
>>>>
>>>> node 1
>>>>
>>>> at t1
>>>>
>>>> cqlsh> select * from system.hints ;
>>>>
>>>> target_id | hint_id
>> | message_version | mutation
>>>>
>> --------------------------------------+--------------------------------------+-----------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>>> 748cb454-da16-4218-b6ac-728f0fe0cf41 |
>> 9f152ea0-49e9-11e3-af88-ed7ea188ea44 | 6 |
>> 0x0006706e735f667200086373636574626f6e00000001019cad5323f2a6389388a31004124bdf2d7fffffff800000000000000000000000000000020003000000000004eacf278eb3a0000000000006000376616c00000004eacf278eb3a0000000040000270f
>>>> 78078974-1f02-4da8-9902-cca56291a9dc |
>> 474e4c30-49ec-11e3-af88-ed7ea188ea44 | 6 |
>> 0x0006706e735f6672000868706172736f6e7300000001019cad5323f2a6389388a31004124bdf2d7fffffff800000000000000000000000000000020003000000000004eacf6b945e10000000000006000376616c00000004eacf6b945e1000000004000010f7
>>>> 78078974-1f02-4da8-9902-cca56291a9dc |
>> 4e64df70-49ec-11e3-af88-ed7ea188ea44 | 6 |
>> 0x0006706e735f667200086373636574626f6e00000001019cad5323f2a6389388a31004124bdf2d7fffffff800000000000000000000000000000020003000000000004eacf6c49f270000000000006000376616c00000004eacf6c49f2700000000400001618
>>>>
>>>> at t2
>>>>
>>>> cqlsh:pns_fr> TRUNCATE system.hints ;
>>>> cqlsh:pns_fr> select * from system.hints ;
>>>>
>>>> node 2
>>>>
>>>> at t1
>>>>
>>>> cqlsh:pns_fr> select * from system.hints ;
>>>>
>>>> target_id | hint_id
>> | message_version | mutation
>>>>
>> --------------------------------------+--------------------------------------+-----------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>>> 78078974-1f02-4da8-9902-cca56291a9dc |
>> aa8f4da0-49ea-11e3-a8dc-e367b713537d | 6 |
>> 0x0006706e735f667200086373636574626f6e00000001019cad5323f2a6389388a31004124bdf2d7fffffff800000000000000000000000000000020003000000000004eacf424d88b0000000000006000376616c00000004eacf424d88b000000004000004bc
>>>>
>>>> at t2
>>>>
>>>> cqlsh:pns_fr> select * from system.hints ;
>>>>
>>>> Regards
>>>> --
>>>> Cyril SCETBON
>>>>
>>>> On 09 Nov 2013, at 22:48, Jonathan Ellis <jb...@gmail.com> wrote:
>>>>
>>>>> Hints are not replicated (technically: replicated with LocalStrategy),
>>>>> so truncating hints will always only affect the local node.
>>>>>
>>>>> On Sat, Nov 9, 2013 at 3:44 PM, Cyril Scetbon <cy...@free.fr>
>> wrote:
>>>>>> Hi,
>>>>>>
>>>>>> I would like to add a feature request for local truncates of hints
>> but I want to know your opinion before.
>>>>>>
>>>>>> We faced an issue last week concerning 3 new nodes that where not
>> seen by some other nodes in the ring. a week later we resolved the issues
>> but it was taking days with a high CPU load (85%) to apply GBytes of Hints.
>> I noticed that CF Hints is defined with LocalStategy and so thought the
>> truncate was only local but that wasn't. We had to remove sstables,
>> commitlog and cached to remove all that old hints stored but not removed
>> after max_hint_window_in_ms as target nodes where not dead (they were only
>> not accepting hints).
>>>>>>
>>>>>> So, I suggest that the truncate of hints stay local (maybe you need
>> it to stay global per default ?) or that we could do it locally (a truncate
>> argument or a new command like the one used for CONSISTENCY)
>>>>>>
>>>>>> thanks
>>>>>> --
>>>>>> Cyril SCETBON
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Jonathan Ellis
>>>>> Project Chair, Apache Cassandra
>>>>> co-founder, http://www.datastax.com
>>>>> @spyced
>>>>
>>>
>>>
>>>
>>> --
>>> Jonathan Ellis
>>> Project Chair, Apache Cassandra
>>> co-founder, http://www.datastax.com
>>> @spyced
>>
>>
Re: local truncate of hints
Posted by Jonathan Ellis <jb...@gmail.com>.
Right.
On Nov 17, 2013 11:00 AM, "Cyril Scetbon" <cy...@free.fr> wrote:
> and/or a nodetool purgelocalhints command ?
>
> On 15 Nov 2013, at 05:56, Jonathan Ellis <jb...@gmail.com> wrote:
>
> > You're right. You'd want to expose the node-local truncate over JMX.
> >
> > On Sun, Nov 10, 2013 at 3:48 AM, Cyril Scetbon <cy...@free.fr>
> wrote:
> >> Are you really sure of that ? cause Tracing says the opposite when I do
> it on one node :
> >>
> >> cqlsh:system> TRUNCATE hints ;
> >>
> >> Tracing session: fe0dbc80-49dd-11e3-a39b-e367b713537d
> >>
> >> activity
> | timestamp | source | source_elapsed
> >>
> ----------------------------------------------------------------------------------------+--------------+------------+----------------
> >>
> execute_cql3_query | 07:59:15,019 | 10.0.0.37 | 0
> >> Parsing
> TRUNCATE hints ; | 07:59:15,020 | 10.0.0.37 | 991
> >>
> Preparing statement | 07:59:15,021 | 10.0.0.37 | 2009
> >> Enqueuing truncate messages to hosts [/10.0.0.185, /10.0.0.33, /
> 10.0.0.17, /10.0.0.37] | 07:59:15,023 | 10.0.0.37 | 4459
> >> Message received
> from /10.0.0.37 | 07:59:15,186 | 10.0.0.33 | 121
> >> Applying truncation
> of system.hints | 07:59:15,200 | 10.0.0.33 | 11064
> >> Sending
> message to /10.0.0.37 | 07:59:15,214 | 10.0.0.37 | 195660
> >> Sending
> message to /10.0.0.33 | 07:59:15,217 | 10.0.0.37 | 198664
> >> Message received
> from /10.0.0.37 | 07:59:15,230 | 10.0.0.185 | 149
> >> Applying truncation
> of system.hints | 07:59:15,240 | 10.0.0.185 | 6248
> >> Message received
> from /10.0.0.37 | 07:59:15,244 | 10.0.0.17 | 117
> >> Sending message
> to /10.0.0.185 | 07:59:15,247 | 10.0.0.37 | 134087
> >> Sending
> message to /10.0.0.17 | 07:59:15,248 | 10.0.0.37 | 134760
> >> Message received
> from /10.0.0.37 | 07:59:15,251 | 10.0.0.37 | 232058
> >> Applying truncation
> of system.hints | 07:59:15,254 | 10.0.0.37 | 234904
> >> Applying truncation
> of system.hints | 07:59:15,259 | 10.0.0.17 | 8574
> >> Enqueuing response to truncate
> operation to /10.0.0.37 | 07:59:15,388 | 10.0.0.17 | 144491
> >> Sending
> message to /10.0.0.37 | 07:59:15,389 | 10.0.0.17 | 145341
> >> Message received
> from /10.0.0.17 | 07:59:15,461 | 10.0.0.37 | 441863
> >> Processing response
> from /10.0.0.17 | 07:59:15,502 | 10.0.0.37 | 482942
> >> Enqueuing response to truncate
> operation to /10.0.0.37 | 07:59:15,578 | 10.0.0.33 | 391940
> >> Sending
> message to /10.0.0.37 | 07:59:15,578 | 10.0.0.33 | 392364
> >> Message received
> from /10.0.0.33 | 07:59:15,632 | 10.0.0.37 | 613172
> >> Processing response
> from /10.0.0.33 | 07:59:15,632 | 10.0.0.37 | 613303
> >> Enqueuing response to truncate
> operation to /10.0.0.37 | 07:59:15,736 | 10.0.0.37 | 717066
> >> Sending
> message to /10.0.0.37 | 07:59:15,738 | 10.0.0.37 | 719165
> >> Message received
> from /10.0.0.37 | 07:59:15,739 | 10.0.0.37 | null
> >> Processing response
> from /10.0.0.37 | 07:59:15,739 | 10.0.0.37 | null
> >> Enqueuing response to truncate
> operation to /10.0.0.37 | 07:59:15,915 | 10.0.0.185 | 685116
> >> Sending
> message to /10.0.0.37 | 07:59:15,915 | 10.0.0.185 | 685519
> >> Message received
> from /10.0.0.185 | 07:59:15,936 | 10.0.0.37 | null
> >> Processing response
> from /10.0.0.185 | 07:59:15,936 | 10.0.0.37 | null
> >>
> Request complete | 07:59:15,936 | 10.0.0.37 | 917846
> >>
> >> I also tried to generate different hints on different nodes, and
> truncating the table on one node removed data everywhere :
> >>
> >>
> >> node 1
> >>
> >> at t1
> >>
> >> cqlsh> select * from system.hints ;
> >>
> >> target_id | hint_id
> | message_version | mutation
> >>
> --------------------------------------+--------------------------------------+-----------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> >> 748cb454-da16-4218-b6ac-728f0fe0cf41 |
> 9f152ea0-49e9-11e3-af88-ed7ea188ea44 | 6 |
> 0x0006706e735f667200086373636574626f6e00000001019cad5323f2a6389388a31004124bdf2d7fffffff800000000000000000000000000000020003000000000004eacf278eb3a0000000000006000376616c00000004eacf278eb3a0000000040000270f
> >> 78078974-1f02-4da8-9902-cca56291a9dc |
> 474e4c30-49ec-11e3-af88-ed7ea188ea44 | 6 |
> 0x0006706e735f6672000868706172736f6e7300000001019cad5323f2a6389388a31004124bdf2d7fffffff800000000000000000000000000000020003000000000004eacf6b945e10000000000006000376616c00000004eacf6b945e1000000004000010f7
> >> 78078974-1f02-4da8-9902-cca56291a9dc |
> 4e64df70-49ec-11e3-af88-ed7ea188ea44 | 6 |
> 0x0006706e735f667200086373636574626f6e00000001019cad5323f2a6389388a31004124bdf2d7fffffff800000000000000000000000000000020003000000000004eacf6c49f270000000000006000376616c00000004eacf6c49f2700000000400001618
> >>
> >> at t2
> >>
> >> cqlsh:pns_fr> TRUNCATE system.hints ;
> >> cqlsh:pns_fr> select * from system.hints ;
> >>
> >> node 2
> >>
> >> at t1
> >>
> >> cqlsh:pns_fr> select * from system.hints ;
> >>
> >> target_id | hint_id
> | message_version | mutation
> >>
> --------------------------------------+--------------------------------------+-----------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> >> 78078974-1f02-4da8-9902-cca56291a9dc |
> aa8f4da0-49ea-11e3-a8dc-e367b713537d | 6 |
> 0x0006706e735f667200086373636574626f6e00000001019cad5323f2a6389388a31004124bdf2d7fffffff800000000000000000000000000000020003000000000004eacf424d88b0000000000006000376616c00000004eacf424d88b000000004000004bc
> >>
> >> at t2
> >>
> >> cqlsh:pns_fr> select * from system.hints ;
> >>
> >> Regards
> >> --
> >> Cyril SCETBON
> >>
> >> On 09 Nov 2013, at 22:48, Jonathan Ellis <jb...@gmail.com> wrote:
> >>
> >>> Hints are not replicated (technically: replicated with LocalStrategy),
> >>> so truncating hints will always only affect the local node.
> >>>
> >>> On Sat, Nov 9, 2013 at 3:44 PM, Cyril Scetbon <cy...@free.fr>
> wrote:
> >>>> Hi,
> >>>>
> >>>> I would like to add a feature request for local truncates of hints
> but I want to know your opinion before.
> >>>>
> >>>> We faced an issue last week concerning 3 new nodes that where not
> seen by some other nodes in the ring. a week later we resolved the issues
> but it was taking days with a high CPU load (85%) to apply GBytes of Hints.
> I noticed that CF Hints is defined with LocalStategy and so thought the
> truncate was only local but that wasn't. We had to remove sstables,
> commitlog and cached to remove all that old hints stored but not removed
> after max_hint_window_in_ms as target nodes where not dead (they were only
> not accepting hints).
> >>>>
> >>>> So, I suggest that the truncate of hints stay local (maybe you need
> it to stay global per default ?) or that we could do it locally (a truncate
> argument or a new command like the one used for CONSISTENCY)
> >>>>
> >>>> thanks
> >>>> --
> >>>> Cyril SCETBON
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Jonathan Ellis
> >>> Project Chair, Apache Cassandra
> >>> co-founder, http://www.datastax.com
> >>> @spyced
> >>
> >
> >
> >
> > --
> > Jonathan Ellis
> > Project Chair, Apache Cassandra
> > co-founder, http://www.datastax.com
> > @spyced
>
>
Re: local truncate of hints
Posted by Cyril Scetbon <cy...@free.fr>.
and/or a nodetool purgelocalhints command ?
On 15 Nov 2013, at 05:56, Jonathan Ellis <jb...@gmail.com> wrote:
> You're right. You'd want to expose the node-local truncate over JMX.
>
> On Sun, Nov 10, 2013 at 3:48 AM, Cyril Scetbon <cy...@free.fr> wrote:
>> Are you really sure of that ? cause Tracing says the opposite when I do it on one node :
>>
>> cqlsh:system> TRUNCATE hints ;
>>
>> Tracing session: fe0dbc80-49dd-11e3-a39b-e367b713537d
>>
>> activity | timestamp | source | source_elapsed
>> ----------------------------------------------------------------------------------------+--------------+------------+----------------
>> execute_cql3_query | 07:59:15,019 | 10.0.0.37 | 0
>> Parsing TRUNCATE hints ; | 07:59:15,020 | 10.0.0.37 | 991
>> Preparing statement | 07:59:15,021 | 10.0.0.37 | 2009
>> Enqueuing truncate messages to hosts [/10.0.0.185, /10.0.0.33, /10.0.0.17, /10.0.0.37] | 07:59:15,023 | 10.0.0.37 | 4459
>> Message received from /10.0.0.37 | 07:59:15,186 | 10.0.0.33 | 121
>> Applying truncation of system.hints | 07:59:15,200 | 10.0.0.33 | 11064
>> Sending message to /10.0.0.37 | 07:59:15,214 | 10.0.0.37 | 195660
>> Sending message to /10.0.0.33 | 07:59:15,217 | 10.0.0.37 | 198664
>> Message received from /10.0.0.37 | 07:59:15,230 | 10.0.0.185 | 149
>> Applying truncation of system.hints | 07:59:15,240 | 10.0.0.185 | 6248
>> Message received from /10.0.0.37 | 07:59:15,244 | 10.0.0.17 | 117
>> Sending message to /10.0.0.185 | 07:59:15,247 | 10.0.0.37 | 134087
>> Sending message to /10.0.0.17 | 07:59:15,248 | 10.0.0.37 | 134760
>> Message received from /10.0.0.37 | 07:59:15,251 | 10.0.0.37 | 232058
>> Applying truncation of system.hints | 07:59:15,254 | 10.0.0.37 | 234904
>> Applying truncation of system.hints | 07:59:15,259 | 10.0.0.17 | 8574
>> Enqueuing response to truncate operation to /10.0.0.37 | 07:59:15,388 | 10.0.0.17 | 144491
>> Sending message to /10.0.0.37 | 07:59:15,389 | 10.0.0.17 | 145341
>> Message received from /10.0.0.17 | 07:59:15,461 | 10.0.0.37 | 441863
>> Processing response from /10.0.0.17 | 07:59:15,502 | 10.0.0.37 | 482942
>> Enqueuing response to truncate operation to /10.0.0.37 | 07:59:15,578 | 10.0.0.33 | 391940
>> Sending message to /10.0.0.37 | 07:59:15,578 | 10.0.0.33 | 392364
>> Message received from /10.0.0.33 | 07:59:15,632 | 10.0.0.37 | 613172
>> Processing response from /10.0.0.33 | 07:59:15,632 | 10.0.0.37 | 613303
>> Enqueuing response to truncate operation to /10.0.0.37 | 07:59:15,736 | 10.0.0.37 | 717066
>> Sending message to /10.0.0.37 | 07:59:15,738 | 10.0.0.37 | 719165
>> Message received from /10.0.0.37 | 07:59:15,739 | 10.0.0.37 | null
>> Processing response from /10.0.0.37 | 07:59:15,739 | 10.0.0.37 | null
>> Enqueuing response to truncate operation to /10.0.0.37 | 07:59:15,915 | 10.0.0.185 | 685116
>> Sending message to /10.0.0.37 | 07:59:15,915 | 10.0.0.185 | 685519
>> Message received from /10.0.0.185 | 07:59:15,936 | 10.0.0.37 | null
>> Processing response from /10.0.0.185 | 07:59:15,936 | 10.0.0.37 | null
>> Request complete | 07:59:15,936 | 10.0.0.37 | 917846
>>
>> I also tried to generate different hints on different nodes, and truncating the table on one node removed data everywhere :
>>
>>
>> node 1
>>
>> at t1
>>
>> cqlsh> select * from system.hints ;
>>
>> target_id | hint_id | message_version | mutation
>> --------------------------------------+--------------------------------------+-----------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>> 748cb454-da16-4218-b6ac-728f0fe0cf41 | 9f152ea0-49e9-11e3-af88-ed7ea188ea44 | 6 | 0x0006706e735f667200086373636574626f6e00000001019cad5323f2a6389388a31004124bdf2d7fffffff800000000000000000000000000000020003000000000004eacf278eb3a0000000000006000376616c00000004eacf278eb3a0000000040000270f
>> 78078974-1f02-4da8-9902-cca56291a9dc | 474e4c30-49ec-11e3-af88-ed7ea188ea44 | 6 | 0x0006706e735f6672000868706172736f6e7300000001019cad5323f2a6389388a31004124bdf2d7fffffff800000000000000000000000000000020003000000000004eacf6b945e10000000000006000376616c00000004eacf6b945e1000000004000010f7
>> 78078974-1f02-4da8-9902-cca56291a9dc | 4e64df70-49ec-11e3-af88-ed7ea188ea44 | 6 | 0x0006706e735f667200086373636574626f6e00000001019cad5323f2a6389388a31004124bdf2d7fffffff800000000000000000000000000000020003000000000004eacf6c49f270000000000006000376616c00000004eacf6c49f2700000000400001618
>>
>> at t2
>>
>> cqlsh:pns_fr> TRUNCATE system.hints ;
>> cqlsh:pns_fr> select * from system.hints ;
>>
>> node 2
>>
>> at t1
>>
>> cqlsh:pns_fr> select * from system.hints ;
>>
>> target_id | hint_id | message_version | mutation
>> --------------------------------------+--------------------------------------+-----------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>> 78078974-1f02-4da8-9902-cca56291a9dc | aa8f4da0-49ea-11e3-a8dc-e367b713537d | 6 | 0x0006706e735f667200086373636574626f6e00000001019cad5323f2a6389388a31004124bdf2d7fffffff800000000000000000000000000000020003000000000004eacf424d88b0000000000006000376616c00000004eacf424d88b000000004000004bc
>>
>> at t2
>>
>> cqlsh:pns_fr> select * from system.hints ;
>>
>> Regards
>> --
>> Cyril SCETBON
>>
>> On 09 Nov 2013, at 22:48, Jonathan Ellis <jb...@gmail.com> wrote:
>>
>>> Hints are not replicated (technically: replicated with LocalStrategy),
>>> so truncating hints will always only affect the local node.
>>>
>>> On Sat, Nov 9, 2013 at 3:44 PM, Cyril Scetbon <cy...@free.fr> wrote:
>>>> Hi,
>>>>
>>>> I would like to add a feature request for local truncates of hints but I want to know your opinion before.
>>>>
>>>> We faced an issue last week concerning 3 new nodes that where not seen by some other nodes in the ring. a week later we resolved the issues but it was taking days with a high CPU load (85%) to apply GBytes of Hints. I noticed that CF Hints is defined with LocalStategy and so thought the truncate was only local but that wasn't. We had to remove sstables, commitlog and cached to remove all that old hints stored but not removed after max_hint_window_in_ms as target nodes where not dead (they were only not accepting hints).
>>>>
>>>> So, I suggest that the truncate of hints stay local (maybe you need it to stay global per default ?) or that we could do it locally (a truncate argument or a new command like the one used for CONSISTENCY)
>>>>
>>>> thanks
>>>> --
>>>> Cyril SCETBON
>>>>
>>>
>>>
>>>
>>> --
>>> Jonathan Ellis
>>> Project Chair, Apache Cassandra
>>> co-founder, http://www.datastax.com
>>> @spyced
>>
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder, http://www.datastax.com
> @spyced
Re: local truncate of hints
Posted by Jonathan Ellis <jb...@gmail.com>.
You're right. You'd want to expose the node-local truncate over JMX.
On Sun, Nov 10, 2013 at 3:48 AM, Cyril Scetbon <cy...@free.fr> wrote:
> Are you really sure of that ? cause Tracing says the opposite when I do it on one node :
>
> cqlsh:system> TRUNCATE hints ;
>
> Tracing session: fe0dbc80-49dd-11e3-a39b-e367b713537d
>
> activity | timestamp | source | source_elapsed
> ----------------------------------------------------------------------------------------+--------------+------------+----------------
> execute_cql3_query | 07:59:15,019 | 10.0.0.37 | 0
> Parsing TRUNCATE hints ; | 07:59:15,020 | 10.0.0.37 | 991
> Preparing statement | 07:59:15,021 | 10.0.0.37 | 2009
> Enqueuing truncate messages to hosts [/10.0.0.185, /10.0.0.33, /10.0.0.17, /10.0.0.37] | 07:59:15,023 | 10.0.0.37 | 4459
> Message received from /10.0.0.37 | 07:59:15,186 | 10.0.0.33 | 121
> Applying truncation of system.hints | 07:59:15,200 | 10.0.0.33 | 11064
> Sending message to /10.0.0.37 | 07:59:15,214 | 10.0.0.37 | 195660
> Sending message to /10.0.0.33 | 07:59:15,217 | 10.0.0.37 | 198664
> Message received from /10.0.0.37 | 07:59:15,230 | 10.0.0.185 | 149
> Applying truncation of system.hints | 07:59:15,240 | 10.0.0.185 | 6248
> Message received from /10.0.0.37 | 07:59:15,244 | 10.0.0.17 | 117
> Sending message to /10.0.0.185 | 07:59:15,247 | 10.0.0.37 | 134087
> Sending message to /10.0.0.17 | 07:59:15,248 | 10.0.0.37 | 134760
> Message received from /10.0.0.37 | 07:59:15,251 | 10.0.0.37 | 232058
> Applying truncation of system.hints | 07:59:15,254 | 10.0.0.37 | 234904
> Applying truncation of system.hints | 07:59:15,259 | 10.0.0.17 | 8574
> Enqueuing response to truncate operation to /10.0.0.37 | 07:59:15,388 | 10.0.0.17 | 144491
> Sending message to /10.0.0.37 | 07:59:15,389 | 10.0.0.17 | 145341
> Message received from /10.0.0.17 | 07:59:15,461 | 10.0.0.37 | 441863
> Processing response from /10.0.0.17 | 07:59:15,502 | 10.0.0.37 | 482942
> Enqueuing response to truncate operation to /10.0.0.37 | 07:59:15,578 | 10.0.0.33 | 391940
> Sending message to /10.0.0.37 | 07:59:15,578 | 10.0.0.33 | 392364
> Message received from /10.0.0.33 | 07:59:15,632 | 10.0.0.37 | 613172
> Processing response from /10.0.0.33 | 07:59:15,632 | 10.0.0.37 | 613303
> Enqueuing response to truncate operation to /10.0.0.37 | 07:59:15,736 | 10.0.0.37 | 717066
> Sending message to /10.0.0.37 | 07:59:15,738 | 10.0.0.37 | 719165
> Message received from /10.0.0.37 | 07:59:15,739 | 10.0.0.37 | null
> Processing response from /10.0.0.37 | 07:59:15,739 | 10.0.0.37 | null
> Enqueuing response to truncate operation to /10.0.0.37 | 07:59:15,915 | 10.0.0.185 | 685116
> Sending message to /10.0.0.37 | 07:59:15,915 | 10.0.0.185 | 685519
> Message received from /10.0.0.185 | 07:59:15,936 | 10.0.0.37 | null
> Processing response from /10.0.0.185 | 07:59:15,936 | 10.0.0.37 | null
> Request complete | 07:59:15,936 | 10.0.0.37 | 917846
>
> I also tried to generate different hints on different nodes, and truncating the table on one node removed data everywhere :
>
>
> node 1
>
> at t1
>
> cqlsh> select * from system.hints ;
>
> target_id | hint_id | message_version | mutation
> --------------------------------------+--------------------------------------+-----------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> 748cb454-da16-4218-b6ac-728f0fe0cf41 | 9f152ea0-49e9-11e3-af88-ed7ea188ea44 | 6 | 0x0006706e735f667200086373636574626f6e00000001019cad5323f2a6389388a31004124bdf2d7fffffff800000000000000000000000000000020003000000000004eacf278eb3a0000000000006000376616c00000004eacf278eb3a0000000040000270f
> 78078974-1f02-4da8-9902-cca56291a9dc | 474e4c30-49ec-11e3-af88-ed7ea188ea44 | 6 | 0x0006706e735f6672000868706172736f6e7300000001019cad5323f2a6389388a31004124bdf2d7fffffff800000000000000000000000000000020003000000000004eacf6b945e10000000000006000376616c00000004eacf6b945e1000000004000010f7
> 78078974-1f02-4da8-9902-cca56291a9dc | 4e64df70-49ec-11e3-af88-ed7ea188ea44 | 6 | 0x0006706e735f667200086373636574626f6e00000001019cad5323f2a6389388a31004124bdf2d7fffffff800000000000000000000000000000020003000000000004eacf6c49f270000000000006000376616c00000004eacf6c49f2700000000400001618
>
> at t2
>
> cqlsh:pns_fr> TRUNCATE system.hints ;
> cqlsh:pns_fr> select * from system.hints ;
>
> node 2
>
> at t1
>
> cqlsh:pns_fr> select * from system.hints ;
>
> target_id | hint_id | message_version | mutation
> --------------------------------------+--------------------------------------+-----------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> 78078974-1f02-4da8-9902-cca56291a9dc | aa8f4da0-49ea-11e3-a8dc-e367b713537d | 6 | 0x0006706e735f667200086373636574626f6e00000001019cad5323f2a6389388a31004124bdf2d7fffffff800000000000000000000000000000020003000000000004eacf424d88b0000000000006000376616c00000004eacf424d88b000000004000004bc
>
> at t2
>
> cqlsh:pns_fr> select * from system.hints ;
>
> Regards
> --
> Cyril SCETBON
>
> On 09 Nov 2013, at 22:48, Jonathan Ellis <jb...@gmail.com> wrote:
>
>> Hints are not replicated (technically: replicated with LocalStrategy),
>> so truncating hints will always only affect the local node.
>>
>> On Sat, Nov 9, 2013 at 3:44 PM, Cyril Scetbon <cy...@free.fr> wrote:
>>> Hi,
>>>
>>> I would like to add a feature request for local truncates of hints but I want to know your opinion before.
>>>
>>> We faced an issue last week concerning 3 new nodes that where not seen by some other nodes in the ring. a week later we resolved the issues but it was taking days with a high CPU load (85%) to apply GBytes of Hints. I noticed that CF Hints is defined with LocalStategy and so thought the truncate was only local but that wasn't. We had to remove sstables, commitlog and cached to remove all that old hints stored but not removed after max_hint_window_in_ms as target nodes where not dead (they were only not accepting hints).
>>>
>>> So, I suggest that the truncate of hints stay local (maybe you need it to stay global per default ?) or that we could do it locally (a truncate argument or a new command like the one used for CONSISTENCY)
>>>
>>> thanks
>>> --
>>> Cyril SCETBON
>>>
>>
>>
>>
>> --
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder, http://www.datastax.com
>> @spyced
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder, http://www.datastax.com
@spyced
Re: local truncate of hints
Posted by Cyril Scetbon <cy...@free.fr>.
Are you really sure of that ? cause Tracing says the opposite when I do it on one node :
cqlsh:system> TRUNCATE hints ;
Tracing session: fe0dbc80-49dd-11e3-a39b-e367b713537d
activity | timestamp | source | source_elapsed
----------------------------------------------------------------------------------------+--------------+------------+----------------
execute_cql3_query | 07:59:15,019 | 10.0.0.37 | 0
Parsing TRUNCATE hints ; | 07:59:15,020 | 10.0.0.37 | 991
Preparing statement | 07:59:15,021 | 10.0.0.37 | 2009
Enqueuing truncate messages to hosts [/10.0.0.185, /10.0.0.33, /10.0.0.17, /10.0.0.37] | 07:59:15,023 | 10.0.0.37 | 4459
Message received from /10.0.0.37 | 07:59:15,186 | 10.0.0.33 | 121
Applying truncation of system.hints | 07:59:15,200 | 10.0.0.33 | 11064
Sending message to /10.0.0.37 | 07:59:15,214 | 10.0.0.37 | 195660
Sending message to /10.0.0.33 | 07:59:15,217 | 10.0.0.37 | 198664
Message received from /10.0.0.37 | 07:59:15,230 | 10.0.0.185 | 149
Applying truncation of system.hints | 07:59:15,240 | 10.0.0.185 | 6248
Message received from /10.0.0.37 | 07:59:15,244 | 10.0.0.17 | 117
Sending message to /10.0.0.185 | 07:59:15,247 | 10.0.0.37 | 134087
Sending message to /10.0.0.17 | 07:59:15,248 | 10.0.0.37 | 134760
Message received from /10.0.0.37 | 07:59:15,251 | 10.0.0.37 | 232058
Applying truncation of system.hints | 07:59:15,254 | 10.0.0.37 | 234904
Applying truncation of system.hints | 07:59:15,259 | 10.0.0.17 | 8574
Enqueuing response to truncate operation to /10.0.0.37 | 07:59:15,388 | 10.0.0.17 | 144491
Sending message to /10.0.0.37 | 07:59:15,389 | 10.0.0.17 | 145341
Message received from /10.0.0.17 | 07:59:15,461 | 10.0.0.37 | 441863
Processing response from /10.0.0.17 | 07:59:15,502 | 10.0.0.37 | 482942
Enqueuing response to truncate operation to /10.0.0.37 | 07:59:15,578 | 10.0.0.33 | 391940
Sending message to /10.0.0.37 | 07:59:15,578 | 10.0.0.33 | 392364
Message received from /10.0.0.33 | 07:59:15,632 | 10.0.0.37 | 613172
Processing response from /10.0.0.33 | 07:59:15,632 | 10.0.0.37 | 613303
Enqueuing response to truncate operation to /10.0.0.37 | 07:59:15,736 | 10.0.0.37 | 717066
Sending message to /10.0.0.37 | 07:59:15,738 | 10.0.0.37 | 719165
Message received from /10.0.0.37 | 07:59:15,739 | 10.0.0.37 | null
Processing response from /10.0.0.37 | 07:59:15,739 | 10.0.0.37 | null
Enqueuing response to truncate operation to /10.0.0.37 | 07:59:15,915 | 10.0.0.185 | 685116
Sending message to /10.0.0.37 | 07:59:15,915 | 10.0.0.185 | 685519
Message received from /10.0.0.185 | 07:59:15,936 | 10.0.0.37 | null
Processing response from /10.0.0.185 | 07:59:15,936 | 10.0.0.37 | null
Request complete | 07:59:15,936 | 10.0.0.37 | 917846
I also tried to generate different hints on different nodes, and truncating the table on one node removed data everywhere :
node 1
at t1
cqlsh> select * from system.hints ;
target_id | hint_id | message_version | mutation
--------------------------------------+--------------------------------------+-----------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
748cb454-da16-4218-b6ac-728f0fe0cf41 | 9f152ea0-49e9-11e3-af88-ed7ea188ea44 | 6 | 0x0006706e735f667200086373636574626f6e00000001019cad5323f2a6389388a31004124bdf2d7fffffff800000000000000000000000000000020003000000000004eacf278eb3a0000000000006000376616c00000004eacf278eb3a0000000040000270f
78078974-1f02-4da8-9902-cca56291a9dc | 474e4c30-49ec-11e3-af88-ed7ea188ea44 | 6 | 0x0006706e735f6672000868706172736f6e7300000001019cad5323f2a6389388a31004124bdf2d7fffffff800000000000000000000000000000020003000000000004eacf6b945e10000000000006000376616c00000004eacf6b945e1000000004000010f7
78078974-1f02-4da8-9902-cca56291a9dc | 4e64df70-49ec-11e3-af88-ed7ea188ea44 | 6 | 0x0006706e735f667200086373636574626f6e00000001019cad5323f2a6389388a31004124bdf2d7fffffff800000000000000000000000000000020003000000000004eacf6c49f270000000000006000376616c00000004eacf6c49f2700000000400001618
at t2
cqlsh:pns_fr> TRUNCATE system.hints ;
cqlsh:pns_fr> select * from system.hints ;
node 2
at t1
cqlsh:pns_fr> select * from system.hints ;
target_id | hint_id | message_version | mutation
--------------------------------------+--------------------------------------+-----------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
78078974-1f02-4da8-9902-cca56291a9dc | aa8f4da0-49ea-11e3-a8dc-e367b713537d | 6 | 0x0006706e735f667200086373636574626f6e00000001019cad5323f2a6389388a31004124bdf2d7fffffff800000000000000000000000000000020003000000000004eacf424d88b0000000000006000376616c00000004eacf424d88b000000004000004bc
at t2
cqlsh:pns_fr> select * from system.hints ;
Regards
--
Cyril SCETBON
On 09 Nov 2013, at 22:48, Jonathan Ellis <jb...@gmail.com> wrote:
> Hints are not replicated (technically: replicated with LocalStrategy),
> so truncating hints will always only affect the local node.
>
> On Sat, Nov 9, 2013 at 3:44 PM, Cyril Scetbon <cy...@free.fr> wrote:
>> Hi,
>>
>> I would like to add a feature request for local truncates of hints but I want to know your opinion before.
>>
>> We faced an issue last week concerning 3 new nodes that where not seen by some other nodes in the ring. a week later we resolved the issues but it was taking days with a high CPU load (85%) to apply GBytes of Hints. I noticed that CF Hints is defined with LocalStategy and so thought the truncate was only local but that wasn't. We had to remove sstables, commitlog and cached to remove all that old hints stored but not removed after max_hint_window_in_ms as target nodes where not dead (they were only not accepting hints).
>>
>> So, I suggest that the truncate of hints stay local (maybe you need it to stay global per default ?) or that we could do it locally (a truncate argument or a new command like the one used for CONSISTENCY)
>>
>> thanks
>> --
>> Cyril SCETBON
>>
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder, http://www.datastax.com
> @spyced
Re: local truncate of hints
Posted by Jonathan Ellis <jb...@gmail.com>.
Hints are not replicated (technically: replicated with LocalStrategy),
so truncating hints will always only affect the local node.
On Sat, Nov 9, 2013 at 3:44 PM, Cyril Scetbon <cy...@free.fr> wrote:
> Hi,
>
> I would like to add a feature request for local truncates of hints but I want to know your opinion before.
>
> We faced an issue last week concerning 3 new nodes that where not seen by some other nodes in the ring. a week later we resolved the issues but it was taking days with a high CPU load (85%) to apply GBytes of Hints. I noticed that CF Hints is defined with LocalStategy and so thought the truncate was only local but that wasn't. We had to remove sstables, commitlog and cached to remove all that old hints stored but not removed after max_hint_window_in_ms as target nodes where not dead (they were only not accepting hints).
>
> So, I suggest that the truncate of hints stay local (maybe you need it to stay global per default ?) or that we could do it locally (a truncate argument or a new command like the one used for CONSISTENCY)
>
> thanks
> --
> Cyril SCETBON
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder, http://www.datastax.com
@spyced