You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@helix.apache.org by "zpinto (via GitHub)" <gi...@apache.org> on 2023/10/17 20:31:57 UTC
[I] Helix Node/Instance Swap [helix]
zpinto opened a new issue, #2662:
URL: https://github.com/apache/helix/issues/2662
# Helix Node/Instance Swap
**Implementation for N -> N + 1 -> N instance swap for all replicas on the instance**
1. Bootstrap new replicas on the SWAP_IN node: New replicas are created for all the partitions hosted on the node to be replaced. These new replicas are then assigned and bootstrapped on the SWAP_IN-node.
2. Remove replicas from the SWAP_OUT node: After successfully bootstrapping **all** the new replicas, the SWAP_OUT node will have its replicas dropped. Waiting until all replicas bootstrap on the SWAP_IN node will ensure we maintain all partitions desired replication factor.
3. Configurable option to not populate SWAP_IN replicas in ExternalView until swap is completed. (Avoid spectator serving traffic)
## Requirements
- Swapping of a node requires “transaction”. When bootstrapping new replicas on the SWAP_IN host, replica count of all partitions should be N + 1 (and should be in topState or secondTopState) before SWAP_OUT node can begin dropping replicas (which will return replica count to N)
## Assumptions
When a node is being swapped, the following criteria must be met for the SWAP_IN node:
1. The cluster must have `TOPOLOGY` with at least one top level key that corresponds to a globally unique value:
- `n-layers/FAULT_ZONE_TYPE/n-layers/key_for_globally_unique_value/n-layers/logical_id_key`
- All instances in the cluster need a unique `DOMAIN` field. It is possible to have a unique domain field across 2 instances which share the same logicalId(value for the last key in the `TOPOLOGY` config)
- This is necessary as logicalId will need to be used for deterministic placement when the SWAP_IN node is introduced as an assignable node.
2. This will be further explained in the topology section
3. The `DOMAIN` field of the SWAP_IN instance must match the SWAP_OUT node exactly except for the value for `key_for_globally_unique_value`
4. If WAGED is enabled in the cluster, the `INSTANCE_CAPACITY_MAP` should match exactly.
Without these assumptions, it is not possible to swap a node as this would not be considered the best possible assignment for the cluster. If these assumptions cannot be met, this functionality should not be utilized. Instead the old node should use the Node Evacuation flow. A new instance can be added in the ENABLED state either before or after to replenish the capacity of removing the old node.
## Executions
- [ ] HelixAdmin.setInstanceOperation and addInstance sanity checks for SWAP_IN & SWAP_OUT
- Can only set instance to SWAP_OUT
- Can only add instance with the same logicaId as another if the already existing has InstanceOperation set to SWAP_OUT and there is not more than 1 already.
- [ ] Create HelixAdmin.isSwapReadyToComplete and completeSwapIfReady(returns if swap finished)
- completeSwapIfFinished will check if the swap is ready to complete. If ready, mark the SWAP_OUT instance as `HELIX_ENABLED: false` and clear the `INSTANCE_OPERATION` for SWAP_IN node.
- [ ] Change WAGED assignment tie breaker logic to use logicalId when present
- May want to make this configurable, as it will lead to shuffling when this change is deployed
- [ ] Refactor DelayedAutoRebalancerUtil and DelayedAutoRebalancer.computeBestPossibleStateForPartition
to filter out SWAP_IN node because we will add SWAP_IN node to the end of preference list in computeBestPossibleStateForPartition until the swap is complete
- Adding it to the end of the preference list will ensure that the SWAP_IN node only carries secondary states
- When SWAP_IN node has identical Current State to SWAP_OUT, isSwapFinished completeSwapIfFinished will return true
- [ ] Refactor ExternalViewComputationStage to not include Current State for SWAP_IN nodes
- Possible make this configurable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
Re: [I] Helix Node/Instance Swap [helix]
Posted by "zpinto (via GitHub)" <gi...@apache.org>.
zpinto commented on issue #2662:
URL: https://github.com/apache/helix/issues/2662#issuecomment-1865215277
This issue has now been closed with the merging of Application Cluster Manager branch. #2714
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
Re: [I] Helix Node/Instance Swap [helix]
Posted by "zpinto (via GitHub)" <gi...@apache.org>.
zpinto closed issue #2662: Helix Node/Instance Swap
URL: https://github.com/apache/helix/issues/2662
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org