You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@kvrocks.apache.org by "zhixinwen (via GitHub)" <gi...@apache.org> on 2025/10/31 20:20:41 UTC

[PR] fix(replication): fix stuck replication due to wrong watermark [kvrocks]

zhixinwen opened a new pull request, #3243:
URL: https://github.com/apache/kvrocks/pull/3243

   In order to reduce unnecessary callbacks, we set read watermark based on data size:
   https://github.com/apache/kvrocks/blob/2b7e69a5acd9e1490b96aafc84746d1baef6579e/src/cluster/replication.cc#L700
   
   However, this watermark is not reset after reading the data, which could result in stuck replication. For example:
   1. A batch of size 16kb is received by replica and watermark is set to 16kb
   2. A PING was received after the batch by the replica and the callback returns without resetting the watermark. Replica would continue to process data only after master sends at least 16kb data.
   3. Master only sent 1kb data, and that data is not processed by replica until master sends enough PING to replica to trigger the 16kb watermark. This causes the replication to get stuck.
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] fix(replication): fix stuck replication due to wrong watermark [kvrocks]

Posted by "git-hulk (via GitHub)" <gi...@apache.org>.

git-hulk commented on PR #3243:
URL: https://github.com/apache/kvrocks/pull/3243#issuecomment-3475725641

   @zhixinwen Thanks for your fix. Have you ever tested how much the performance will be affected if we don't set the low watermark?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] fix(replication): fix stuck replication due to wrong watermark [kvrocks]

Posted by "zhixinwen (via GitHub)" <gi...@apache.org>.

zhixinwen commented on PR #3243:
URL: https://github.com/apache/kvrocks/pull/3243#issuecomment-3475766324

   > @zhixinwen Thanks for your fix. Have you ever tested how much the performance will be affected if we don't set the low watermark?
   
   I don't remeber the exact details TBH, especially after I have done several other optimizations.
   My gut feeling is it may not matter a lot since replica is IO bound not CPU bound.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] fix(replication): fix stuck replication due to wrong watermark [kvrocks]

Posted by "PragmaTwice (via GitHub)" <gi...@apache.org>.

PragmaTwice merged PR #3243:
URL: https://github.com/apache/kvrocks/pull/3243


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] fix(replication): fix stuck replication due to wrong watermark [kvrocks]

Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.

sonarqubecloud[bot] commented on PR #3243:
URL: https://github.com/apache/kvrocks/pull/3243#issuecomment-3475801492

   ## [![Quality Gate Failed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-failed-20px.png 'Quality Gate Failed')](https://sonarcloud.io/dashboard?id=apache_kvrocks&pullRequest=3243) **Quality Gate failed**  
   Failed conditions  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/failed-16px.png '') [33.3% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_kvrocks&pullRequest=3243&metric=new_coverage&view=list) (required ≥ 50%)  
     
   [See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_kvrocks&pullRequest=3243)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org