[jira] [Updated] (IGNITE-29088) Resend partition maps after detecting lost partitions under the IGNORE policy

["Maksim Davydov (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/IGNITE-29088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Maksim Davydov updated IGNITE-29088:
------------------------------------
    Description: 
*Problem*

In an in-memory cluster with the IGNORE partition loss policy (baseline auto-adjust on, timeout 0), a node leaves and takes the only copy of some partitions with it (backups = 0). Each lost partition is recreated empty on its new primary and owned there. Sometimes the coordinator never learns that: its partition map keeps the partition MOVING until the next exchange.

*Impact*

Until the next topology change, which in a stable cluster can be hours away, the coordinator and every node that takes its full map see no owner of the partition:
- SQL queries from those nodes fail after the retry timeout:
  "Failed to map SQL query to topology during timeout: 30000ms". With a MOVING partition in the map, ReducePartitionMapper maps partitions by their owners and finds none. On the new primary, whose own map is right, the same query succeeds.
- awaitPartitionMapExchange() in tests times out. Key operations and scan queries still work.

This is not what IGNORE is meant to do: the policy resets a lost partition silently, no
partition is LOST, and there is nothing to reset by hand.

*Cause*

With the exchange merge protocol, the new primary creates the lost partition as MOVING when it gets the coordinator's full message, and owns it a moment later in detectLostPartitions. The coordinator, in its own detectLostPartitions, already marks the partition OWNING for that node.

If the new primary sends its partition map between these two steps, the map carries MOVING with a newer update sequence, and the coordinator takes it. A typical sender is the scheduled resend (scheduleResendPartitions(), 1.5 s after an eviction or a map change). Nothing sends the node's map again after it owns the partition: the result of detectLostPartitions is dropped in GridDhtPartitionsExchangeFuture#detectLostPartitions. 

Also, inside GridDhtPartitionTopologyImpl#detectLostPartitions the result reflects only the last lost partition.

*How it shows*

IgniteTopologyValidatorGridSplitCacheTest (32 nodes, 50 caches without backups) times out in awaitPartitionMapExchange() after stopping its configless node in 9 of 15 local runs. Small clusters rarely send a map inside that window, so IgniteCachePartitionLossPolicySelfTest (3 nodes) doesn't hit it.

*Fix*

When detectLostPartitions changes a local partition, a non-coordinator node sends its single map again. A new test, CachePartitionLossIgnorePolicyMapTest, makes the new primary send its map inside the window every time and checks that the coordinator's map matches the nodes.

The resend is not limited to IGNORE: it fires whenever detectLostPartitions changes a local partition. Under the SAFE policies the node marks the partition LOST instead of owning it, and the same order of messages can leave the coordinator's map at MOVING for that node instead of LOST until the next exchange; the resend corrects that as well. It also fires on cluster activation with lost partitions. Loss events, lostPartitions() and resetLostPartitions() work as before: a late map with LOST can't override the reset, since the reset is a new exchange with a newer topology version. The cost is one extra single map per affected non-coordinator node per such exchange; the coordinator answers with one delayed full-map resend that covers them all.

  was:
*Problem*

In an in-memory cluster with the IGNORE partition loss policy (baseline auto-adjust on, timeout 0), a node leaves and takes the only copy of some partitions with it (backups = 0). Each lost partition is recreated empty on its new primary and owned there. Sometimes the coordinator never learns that: its partition map keeps the partition MOVING until the next exchange.

*Impact*

Until the next topology change, which in a stable cluster can be hours away, the coordinator and every node that takes its full map see no owner of the partition:
- SQL queries from those nodes fail after the retry timeout:
  "Failed to map SQL query to topology during timeout: 30000ms". With a MOVING partition in the map, ReducePartitionMapper maps partitions by their owners and finds none. On the new primary, whose own map is right, the same query succeeds.
- awaitPartitionMapExchange() in tests times out. Key operations and scan queries still work.

This is not what IGNORE is meant to do: the policy resets a lost partition silently, no
partition is LOST, and there is nothing to reset by hand.

*Cause*

With the exchange merge protocol, the new primary creates the lost partition as MOVING when it gets the coordinator's full message, and owns it a moment later in detectLostPartitions. The coordinator, in its own detectLostPartitions, already marks the partition OWNING for that node.

If the new primary sends its partition map between these two steps, the map carries MOVING with a newer update sequence, and the coordinator takes it. A typical sender is the scheduled resend (scheduleResendPartitions(), 1.5 s after an eviction or a map change). Nothing sends the node's map again after it owns the partition: the result of detectLostPartitions is dropped in GridDhtPartitionsExchangeFuture#detectLostPartitions. 

Also, inside GridDhtPartitionTopologyImpl#detectLostPartitions the result reflects only the last lost partition.

*How it shows*

IgniteTopologyValidatorGridSplitCacheTest (32 nodes, 50 caches without backups) times out in awaitPartitionMapExchange() after stopping its configless node in 9 of 15 local runs. Small clusters rarely send a map inside that window, so IgniteCachePartitionLossPolicySelfTest (3 nodes) doesn't hit it.

*Fix*

When detectLostPartitions changes a local partition, a non-coordinator node sends its single map again. A new test, CachePartitionLossIgnorePolicyMapTest, makes the new primary send its map inside the window every time and checks that the coordinator's map matches the nodes.


> Resend partition maps after detecting lost partitions under the IGNORE policy
> -----------------------------------------------------------------------------
>
>                 Key: IGNITE-29088
>                 URL: https://issues.apache.org/jira/browse/IGNITE-29088
>             Project: Ignite
>          Issue Type: Bug
>            Reporter: Maksim Davydov
>            Assignee: Maksim Davydov
>            Priority: Major
>              Labels: ise
>          Time Spent: 20m
>  Remaining Estimate: 0h
>
> *Problem*
> In an in-memory cluster with the IGNORE partition loss policy (baseline auto-adjust on, timeout 0), a node leaves and takes the only copy of some partitions with it (backups = 0). Each lost partition is recreated empty on its new primary and owned there. Sometimes the coordinator never learns that: its partition map keeps the partition MOVING until the next exchange.
> *Impact*
> Until the next topology change, which in a stable cluster can be hours away, the coordinator and every node that takes its full map see no owner of the partition:
> - SQL queries from those nodes fail after the retry timeout:
>   "Failed to map SQL query to topology during timeout: 30000ms". With a MOVING partition in the map, ReducePartitionMapper maps partitions by their owners and finds none. On the new primary, whose own map is right, the same query succeeds.
> - awaitPartitionMapExchange() in tests times out. Key operations and scan queries still work.
> This is not what IGNORE is meant to do: the policy resets a lost partition silently, no
> partition is LOST, and there is nothing to reset by hand.
> *Cause*
> With the exchange merge protocol, the new primary creates the lost partition as MOVING when it gets the coordinator's full message, and owns it a moment later in detectLostPartitions. The coordinator, in its own detectLostPartitions, already marks the partition OWNING for that node.
> If the new primary sends its partition map between these two steps, the map carries MOVING with a newer update sequence, and the coordinator takes it. A typical sender is the scheduled resend (scheduleResendPartitions(), 1.5 s after an eviction or a map change). Nothing sends the node's map again after it owns the partition: the result of detectLostPartitions is dropped in GridDhtPartitionsExchangeFuture#detectLostPartitions. 
> Also, inside GridDhtPartitionTopologyImpl#detectLostPartitions the result reflects only the last lost partition.
> *How it shows*
> IgniteTopologyValidatorGridSplitCacheTest (32 nodes, 50 caches without backups) times out in awaitPartitionMapExchange() after stopping its configless node in 9 of 15 local runs. Small clusters rarely send a map inside that window, so IgniteCachePartitionLossPolicySelfTest (3 nodes) doesn't hit it.
> *Fix*
> When detectLostPartitions changes a local partition, a non-coordinator node sends its single map again. A new test, CachePartitionLossIgnorePolicyMapTest, makes the new primary send its map inside the window every time and checks that the coordinator's map matches the nodes.
> The resend is not limited to IGNORE: it fires whenever detectLostPartitions changes a local partition. Under the SAFE policies the node marks the partition LOST instead of owning it, and the same order of messages can leave the coordinator's map at MOVING for that node instead of LOST until the next exchange; the resend corrects that as well. It also fires on cluster activation with lost partitions. Loss events, lostPartitions() and resetLostPartitions() work as before: a late map with LOST can't override the reset, since the reset is a new exchange with a newer topology version. The cost is one extra single map per affected non-coordinator node per such exchange; the coordinator answers with one delayed full-map resend that covers them all.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)