Hints not being sent from 3.0 to 4.0?

["Morten A. Iversen via user" <us...@cassandra.apache.org>]

Hi,

We are currently in the process of upgrading our environment from
3.0.27 to 4.0.4. However I see some issues with hints not being sent
from v3 nodes to v4 nodes.

We have a test environment with 2DCs, we are currently writing to DC1
and DC2 have been upgraded from version 3.0.27 -> 4.0.4

After the upgrade I see that all the v3 nodes in DC1 have hint files
stored for all the upgraded nodes in DC2. These files where created
while the nodes in DC2 was upgraded, however they do not disappear and
there is nothing in the logs mentioning anything about these files.

Does this mean that data written while a node is being upgraded is not
replicated to that node?

Hint related configs:
   hinted_handoff_enabled: true
   max_hint_window_in_ms: 172800000 # 48
   hourshinted_handoff_throttle_in_kb: 10240
   max_hints_delivery_threads: 8
   hints_directory: /var/lib/cassandra/hints
   hints_flush_period_in_ms: 10000
   max_hints_file_size_in_mb: 12

Regards
Morten

Re: Hints not being sent from 3.0/3.11 to 4.0?

["Morten A. Iversen via user" <us...@cassandra.apache.org>]

Hi,

I have finally had time to test this a bit. And the issue seems to be
there with Cassandra 3.11.13 as well as 3.0.27. Hinted handoff from
these versions to version 4.0.6 does not seem to work. 

The way I tested this was by setting up a 2 node cluster, 1 node in
each of dc1 and dc2. Install version 3.11.13 (or 3.0.27) on both of
them and confirm that hinted handoff is working by taking one node down
and then write 100k lines with cassandra-stress. Take it back up and
see that the data shows up in the DC that was down, the hints files
disappear from the node that was written to and the system.log shows
that hinted handoff was completed.

After that I truncate the table that was written to and upgrade one
node to 4.0.6. Repeat the process above only to see that the node gets
no data when it gets back up, the hint files are not deleted on the
node that was written to and there is no mention of hinted handoff in
the logs.

Is this supposed to work? And if not, how is it recommended to upgrade
production clusters without significant data inconsistencies?

Regards
Morten Iversen

On Tue, 2022-08-23 at 11:35 -0400, Jim Shaw wrote:
> Is it over max hint window ?  if over, better to do a full repair.
> check table system.hints,  do you see rows ?
> 
> As I remember, during upgrade, transactions will store in hints until
> other cluster have done upgrade, so for safety, change default 3
> hours hint window to long time just before starting upgrade.  
> 
> Regards,
> 
> Jim
> 
> 
> On Tue, Aug 23, 2022 at 8:33 AM Morten A. Iversen via user
> <us...@cassandra.apache.org> wrote:
> > Hi,
> > 
> > We are currently in the process of upgrading our environment from
> > 3.0.27 to 4.0.4. However I see some issues with hints not being
> > sent
> > from v3 nodes to v4 nodes.
> > 
> > We have a test environment with 2DCs, we are currently writing to
> > DC1
> > and DC2 have been upgraded from version 3.0.27 -> 4.0.4
> > 
> > After the upgrade I see that all the v3 nodes in DC1 have hint
> > files
> > stored for all the upgraded nodes in DC2. These files where created
> > while the nodes in DC2 was upgraded, however they do not disappear
> > and
> > there is nothing in the logs mentioning anything about these files.
> > 
> > Does this mean that data written while a node is being upgraded is
> > not
> > replicated to that node?
> > 
> > Hint related configs:
> >    hinted_handoff_enabled: true
> >    max_hint_window_in_ms: 172800000 # 48
> >    hourshinted_handoff_throttle_in_kb: 10240
> >    max_hints_delivery_threads: 8
> >    hints_directory: /var/lib/cassandra/hints
> >    hints_flush_period_in_ms: 10000
> >    max_hints_file_size_in_mb: 12
> > 
> > Regards
> > Morten

Re: Hints not being sent from 3.0 to 4.0?

[Jim Shaw <jx...@gmail.com>]

Is it over max hint window ?  if over, better to do a full repair.
check table system.hints,  do you see rows ?

As I remember, during upgrade, transactions will store in hints until other
cluster have done upgrade, so for safety, change default 3 hours hint
window to long time just before starting upgrade.

Regards,

Jim


On Tue, Aug 23, 2022 at 8:33 AM Morten A. Iversen via user <
user@cassandra.apache.org> wrote:

> Hi,
>
> We are currently in the process of upgrading our environment from
> 3.0.27 to 4.0.4. However I see some issues with hints not being sent
> from v3 nodes to v4 nodes.
>
> We have a test environment with 2DCs, we are currently writing to DC1
> and DC2 have been upgraded from version 3.0.27 -> 4.0.4
>
> After the upgrade I see that all the v3 nodes in DC1 have hint files
> stored for all the upgraded nodes in DC2. These files where created
> while the nodes in DC2 was upgraded, however they do not disappear and
> there is nothing in the logs mentioning anything about these files.
>
> Does this mean that data written while a node is being upgraded is not
> replicated to that node?
>
> Hint related configs:
>    hinted_handoff_enabled: true
>    max_hint_window_in_ms: 172800000 # 48
>    hourshinted_handoff_throttle_in_kb: 10240
>    max_hints_delivery_threads: 8
>    hints_directory: /var/lib/cassandra/hints
>    hints_flush_period_in_ms: 10000
>    max_hints_file_size_in_mb: 12
>
> Regards
> Morten
>