You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pegasus.apache.org by "ninsmiracle (via GitHub)" <gi...@apache.org> on 2024/01/09 03:58:43 UTC
[I] Bug(duplication):some nodes can not send duplication data in master when big difference in data size [incubator-pegasus]
ninsmiracle opened a new issue, #1840:
URL: https://github.com/apache/incubator-pegasus/issues/1840
## Bug Report
Please answer these questions before submitting your issue. Thanks!
1. What did you do?
In online production situation. Some nodes in matser cluster can not send dupcalition data to backup cluster.
2. What version of Pegasus are you using?
[pegasus2.4](https://github.com/apache/incubator-pegasus/tree/v2.4)
3.Why?
The root cause is that the master cluster sent a write RPC to the backup cluster, and the request body size exceeded the maximum allowed write size set by the backup cluster. The reason why the master cluster sends this illegal data is due to the mechanism in the process of packaging and sending mutations:
the master cluster traverses the writes received in a time period, checks whether all writes have been traversed or whether the current batch bytes are already greater than the duplicate_log_batch_bytes set in the cluster hot standby configuration (config.ini). If it is not greater than the batch size, then two mutations are combined into one batch.
This can happen if the data length distribution of the table is too large.
For example:
The length of the first mutation A is 200byte. This naturally is to be combined with the next piece.
But the length of the next mutation B is 1048376(1048576-200) .
At this point, **A and B are already in a batch**, RPC will be sent out smoothly, but the standby cluster can not accept such a large write, and the standby cluster throws an ERR error. The master cluster is delayed recovery, throwing an ERR error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org
Re: [I] Bug(duplication):some nodes can not send duplication data in master when big difference in data size [incubator-pegasus]
Posted by "ruojieranyishen (via GitHub)" <gi...@apache.org>.
ruojieranyishen commented on issue #1840:
URL: https://github.com/apache/incubator-pegasus/issues/1840#issuecomment-2102079281
`max_copy_rate_megabytes_per_disk` causes a large number of threads in the `replica.default` thread pool to sleep. The `nfs_service_impl::on_copy` working on `replica.default` can also cause this issue.
In this thread pool:
1. affecting the timely processing of remote_command. For example:
```
>>> server_info -r
COMMAND: server-info
CALL [meta-server] [xxxx:45001] succeed: Pegasus Server 2.4.3-without-slog (84632317ee48b2c6c013f41d0e6f73ad4955b6bb) Release, Started at 2024-03-22 14:10:53
CALL [meta-server] [xxxx:45001] succeed: Pegasus Server 2.4.3-without-slog (84632317ee48b2c6c013f41d0e6f73ad4955b6bb) Release, Started at 2024-03-22 14:11:01
CALL [replica-server] [xxxx:45101] succeed: Pegasus Server 2.4.3-without-slog (84632317ee48b2c6c013f41d0e6f73ad4955b6bb) Release, Started at 2024-03-22 14:11:15
CALL [replica-server] [xxxx:45101] failed: ERR_TIMEOUT
CALL [replica-server] [xxxx:45101] succeed: Pegasus Server 2.4.3-without-slog (84632317ee48b2c6c013f41d0e6f73ad4955b6bb) Release, Started at 2024-04-15 17:50:28
CALL [replica-server] [xxxx:45101] failed: ERR_TIMEOUT
CALL [replica-server] [xxxx:45101] succeed: Pegasus Server 2.4.3-without-slog (84632317ee48b2c6c013f41d0e6f73ad4955b6bb) Release, Started at 2024-03-22 14:11:22
Succeed count: 5
Failed count: 2
```
2. Some RPCs are being delayed for transmission. For example:
```
RPC_CM_CONFIG_SYNC
RPC_CM_DUPLICATION_SYNC
RPC_NFS_COPY
```
May be related： https://github.com/apache/incubator-pegasus/issues/1840
3. Indirectly makes the Group Check ERR_TIMEOUT in `replica.replica` and causes ballot increase.
```
[general]
app_name : lpf_test
app_id : 46
partition_count : 100
max_replica_count : 3
[replicas]
pidx ballot replica_count primary secondaries
0 6 3/3 x.x.x.x:55101 [x.x.x.x:55101,x.x.x.x:55101]
1 7 3/3 x.x.x.x:55101 [x.x.x.x:55101,x.x.x.x:55101]
2 5 3/3 x.x.x.x:55101 [x.x.x.x:55101,x.x.x.x:55101]
3 9 3/3 x.x.x.x:55101 [x.x.x.x:55101,x.x.x.x:55101]
4 5 3/3 x.x.x.x:55101 [x.x.x.x:55101,x.x.x.x:55101]
5 6 3/3 x.x.x.x:55101 [x.x.x.x:55101,x.x.x.x:55101]
6 7 3/3 x.x.x.x:55101 [x.x.x.x:55101,x.x.x.x:55101]
```
# Solution
1. If the `max_copy_rate_megabytes_per_disk` is not set too low and not frequently adjusted, it is unlikely to trigger the problem.
2. If replacing the default RPC thread pool for NFS-related operations in `nfs_server_impl.cpp` can resolves this issue. For example:
```
-DEFINE_TASK_CODE_RPC(RPC_NFS_COPY, TASK_PRIORITY_COMMON, ::dsn::THREAD_POOL_DEFAULT)
+DEFINE_TASK_CODE_RPC(RPC_NFS_COPY, TASK_PRIORITY_COMMON, ::dsn::THREAD_POOL_LEARN)
DEFINE_TASK_CODE_RPC(RPC_NFS_GET_FILE_SIZE, TASK_PRIORITY_COMMON, ::dsn::THREAD_POOL_DEFAULT)
// test timer task code
DEFINE_TASK_CODE(LPC_NFS_REQUEST_TIMER, TASK_PRIORITY_COMMON, ::dsn::THREAD_POOL_DEFAULT)
-DEFINE_TASK_CODE_AIO(LPC_NFS_READ, TASK_PRIORITY_COMMON, THREAD_POOL_DEFAULT)
+DEFINE_TASK_CODE_AIO(LPC_NFS_READ, TASK_PRIORITY_COMMON, ::dsn::THREAD_POOL_LEARN)
-DEFINE_TASK_CODE_AIO(LPC_NFS_WRITE, TASK_PRIORITY_COMMON, THREAD_POOL_DEFAULT)
+DEFINE_TASK_CODE_AIO(LPC_NFS_WRITE, TASK_PRIORITY_COMMON, ::dsn::THREAD_POOL_LEARN)
-DEFINE_TASK_CODE_AIO(LPC_NFS_COPY_FILE, TASK_PRIORITY_COMMON, THREAD_POOL_DEFAULT)
+DEFINE_TASK_CODE_AIO(LPC_NFS_COPY_FILE, TASK_PRIORITY_COMMON, ::dsn::THREAD_POOL_LEARN)
```
**Changing the thread pool solve the above problem, but it is unclear whether there are other risks.**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org