You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ignite.apache.org by "Kirill Gusakov (Jira)" <ji...@apache.org> on 2024/12/02 20:00:00 UTC

[jira] [Updated] (IGNITE-23823) Reset algorithm as a cause of possible data inconsistency

     [ https://issues.apache.org/jira/browse/IGNITE-23823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kirill Gusakov updated IGNITE-23823:
------------------------------------
    Summary: Reset algorithm as a cause of possible data inconsistency  (was: Reset algorithm inconsistency)

> Reset algorithm as a cause of possible data inconsistency
> ---------------------------------------------------------
>
>                 Key: IGNITE-23823
>                 URL: https://issues.apache.org/jira/browse/IGNITE-23823
>             Project: Ignite
>          Issue Type: Improvement
>            Reporter: Kirill Gusakov
>            Priority: Major
>
> *Motivation*
> We have an issue with the choosing the right node which should be the target for reset, because of lack of the expected guarantees issues with the lastLogId check
> * stable.assignments = [A(10),B(10),C(10),D(5),E(4)]. A(10) means, that node A has the lastLogId=10 
> * Nodes A, B, C dies
> * HA reset timer exhausted
> * The node D(5) choosed as the target for reset from the [D(5),E(4)] list.
> * Reset process is initiated:
>   * stable.assignments = [A,B,C,D,E], pending.assignments=[D(5)] planned.assignments=[D,E].
> * Everything is looking good, *but the E node actually can have potential infinite queue of unprocessed messages in the local queue*, and when it will process them, the index will be increased, for example to 6' and then to 7': E[7']
> * So, after the first rebalance success we will have pending.assignmentes=[D(7),E(7')], stable.assignments=[D(6)]. The index of D increased by two because of 2 rebalance reconfigurations (joint configuration + target configuration entries). 
> At this point we have a hidden data inconsistency between the 2 raft nodes, at the index 7.
> So, the main problem - we think that after the lastLogId check on the broken group (no majority) we can't have lastLogId updates - but it is not true. Some messages still can be "in-flight" on the local raft node.
> *Implementation notes*
> The only possible solution, that we have at the moment - clean up all nodes from the planned key (in reset flow), which is not in the force pending. TBD: investigate how we should implement it.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)