You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pegasus.apache.org by "lengyuexuexuan (via GitHub)" <gi...@apache.org> on 2025/03/13 11:58:55 UTC
[I] BUG(split): When a child copies mutations, the order of these mutations may be inconsistent. [incubator-pegasus]
lengyuexuexuan opened a new issue, #2210:
URL: https://github.com/apache/incubator-pegasus/issues/2210
## Bug Report
1. What did you do?
There are two places that may cause this issus.
The first is copying prepare_list.
1. copy prepare_list
1. https://github.com/apache/incubator-pegasus/blob/a875cb4261730a5eb5408bb79688ff447cb47135/src/replica/split/replica_split_manager.cpp#L305-L308
2. https://github.com/apache/incubator-pegasus/blob/a875cb4261730a5eb5408bb79688ff447cb47135/src/replica/prepare_list.cpp#L52-L57
3. During the process of copying the prepare_list, write operations were not stopped, so mutation_cache and last_commit_decree will continue to grow.
4. Since last_commit_decree is copied later, it is possible for last_committed_decree to be greater than the actual committed_decree in the prepare_list.
5. Therefore, this may lead to data loss.
2. plog append
1. https://github.com/apache/incubator-pegasus/blob/a875cb4261730a5eb5408bb79688ff447cb47135/src/replica/split/replica_split_manager.cpp#L370-L378
2. https://github.com/apache/incubator-pegasus/blob/a875cb4261730a5eb5408bb79688ff447cb47135/src/replica/split/replica_split_manager.cpp#L1359-L1366
3. Currently, when the child replicates mutations, there are also asynchronous writes coming from the client, and both operations will perform plog append.
4. The plog does not have a method to distinguish the order.
2. soulution
1. For the first issue, it can be resolved by copying the mutation_cache again after copying the last_commited_decree.
2. For the second issue, there is currently no good solution in mind.
3. Stopping client writes before the copy_prepare_list process seems like a good approach to prevent conflicts and ensure consistency.
4. By stopping client writes before the copy_prepare_list process, you can address both of the issues.
3. Here is the test for the write-stop duration
数据量 1M
| 分片数量 | 单分片阻写时间(min) | 单分片阻写时间(max) | 单分片阻写时间(avg) | 总体阻写时间 |
|--------------|---------------------|---------------------|---------------------|--------------|
| 32->64 | 1.1s | 2.1s | 1.9s | 85s |
| 64->128 | 1.5s | 61s | 4.2s | 83s |
| 128->256 | 1.3s | 5.9s | 3.2s | 85s |
| 256->512 | 1.3s | 32s | 3.1s | 89s |
| 512->1024 | 1.3s | 60s | 7.1s | 110s |
数据量 172G
| 分片数量 | 单分片阻写时间(min) | 单分片阻写时间(max) | 单分片阻写时间(avg) | 总体阻写时间 |
|--------------|---------------------|---------------------|---------------------|--------------|
| 64->128 | 1.2s | 121.5s | 20.9s | 141s |
| 128->256 | 1.8s | 81.5s | 14.9s | 122s |
| 256->512 | 2s | 66.1s | 11s | 144s |
| 512->1024 | (内存被打满 OOM) | | | |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org