You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pegasus.apache.org by "lengyuexuexuan (via GitHub)" <gi...@apache.org> on 2025/03/14 06:40:39 UTC
[I] feat: introduce a way to deploy replicas across multiple Availability Zones [incubator-pegasus]
lengyuexuexuan opened a new issue, #2213:
URL: https://github.com/apache/incubator-pegasus/issues/2213
## Feature Request
**Is your feature request related to a problem? Please describe:**
<!-- A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] -->
To further enhance the stability of the Pegasus service and prevent Az-level failures, the current recommended solution is to deploy a backup cluster in another Az and use the duplication feature to synchronize data to the backup cluster in real time.
However, the duplication approach incurs high costs, which do not meet the needs of some businesses. Therefore, an alternative approach is to deploy the three replicas of Pegasus across different Azs. While this may introduce some additional read and write latency, it improves disaster recovery capabilities at Az level without increasing costs.
**Describe the feature you'd like:**
<!-- A clear and concise description of what you want to happen. -->
1. Each shard's three replicas must be distributed across different availability zones (AZs), ensuring at least one replica in each AZ.
2. After operations such as creating new tables, cure, and balance, the above rule should still be maintained.
3. This feature can be dynamically enabled or disabled and persistently stored in ZooKeeper (ZK).
4. The system should support placing all primary shards in the primary AZ, with a dynamic toggle to enable or disable this behavior, and the ability to change the primary AZ.
5. The multi-AZ distribution can be automatically disabled based on the node ratio between the primary and backup AZs.
**The specific implementation of this method at Xiaomi**
1. Methods for Identifying the Availability Zone (AZ) of Each Node
1. Modify the startup script so that when a replica starts, the script retrieves the AZand writes it to the <run_dir>/replica directory with the filename "az.info"
```
# az.info
{"az":"ak"}
```
2. The replica sends the AZ information to the meta via the RPC_CM_CONFIG_SYNC request.
2. Relevant Configuration
1. All configurations can be dynamically modified and persisted to ZooKeeper.
2. ```
enable_cross_az = true
az_list = c3,ak
priamry_az = c3
enable_primary_not_cross_az = false
az_nodes_ratio = 2
```
3. Here is the purpose of these parameters
1. enable_cross_az is the global switch.
2. When cross-AZ is enabled, only nodes that belong to the AZs listed in the az_list can perform cure and balance operations.
3. primary_az specifies the primary AZ.
4. When enable_primary_not_cross_az is set to true, the primary replica will only be distributed on nodes within the primary AZ.
5. az_nodes_ratio is used to prevent a significant disparity in the number of nodes between different AZs. Continuing with the above rule could lead to uneven distribution, causing excessive pressure on some nodes. If there are two AZs, a ratio of 2 is recommended. If there are three AZs, a ratio of 1.5 is recommended.
3. cure
1. missing_primary
1. enable_primary_not_cross_az == true
1. When the secondary is empty(create table), meta finds the target node in the primary AZ that has the fewest primary nodes.
2. When the secondary isn't empty, meta should prioritize selecting the secondary node within the primary AZ as the new primary.
3. When the last drops isn't empty, the behavior should remain the same as before.
2. enable_primary_not_cross_az == false
1. The behavior should remain the same as before.
2. missing_secondary
1. First, determine which AZ the new secondary replica should be placed in.
2. Continue to prioritize selecting the node in last_drops, but add a check to ensure that it matches the expected AZ.
3. If no node in last_drop meets the requirements, then search for the node with the fewest partitions in the target AZ to serve as the target node.
4. balance
1. If enable_primary_not_cross_az == true
1. First, move all the primary shards to the primary AZ (using move_primary / copy_primary).
2. Perform primary balance within the primary AZ (using Ford-Fulkerson and copy_primary).
3. Perform secondary balance
1. Check all replicas. If they do not meet the multi-AZ distribution requirements, perform copy_secondary. During the copy_secondary process, the original secondary balance conditions must also be satisfied.
2. After all replicas meet the multi-AZ distribution requirements, perform the previous secondary balance. However, apply a restriction: if copying the secondary of a replica would change the multi-AZ distribution state of the replica, then skip that operation.
2. If enable_primary_not_cross_az == false
1. The primary balance process remains the same as before.
2. The secondary balance process is the same as described above.
5. test
1. if enable_primary_not_cross_az == true, all primary replicas are in primary az(C3), and all replicas' replication are present in both C3 and C4.
2. if enable_primary_not_cross_az == false, all replicas' replication are present in both C3 and C4.
3. The impact on read and write latency
1. read: If reading from a primary across AZs, the read latency will incur an additional cross-AZ RTT
2. write
1. If the client and primary are in the same AZ, then the write latency will incur an additional cross-AZ RTT.
2. If the client and primary are in different AZs, then the write latency will incur two additional cross-AZ RTTs. (client->primary one rtt, primary->secondary one rtt)
**The feature will soon be launched at Xiaomi.
If anyone needs more detailed design plans or test records, feel free to contact me.
Thanks.**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org