You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@rocketmq.apache.org by GitBox <gi...@apache.org> on 2022/09/14 03:40:53 UTC

[GitHub] [rocketmq] guyinyou opened a new pull request, #5067: [ISSUE #5066] Turn off flow control

guyinyou opened a new pull request, #5067:
URL: https://github.com/apache/rocketmq/pull/5067

**Make sure set the target branch to `develop`**

## What is the purpose of the change

#5066

## Brief changelog

XX

## Verifying this change

XXXX

Follow this checklist to help us incorporate your contribution quickly and easily. Notice, `it would be helpful if you could finish the following 5 checklist(the last one is not necessary)before request the community to review your PR`.

- [x] Make sure there is a [Github issue](https://github.com/apache/rocketmq/issues) filed for the change (usually before you start working on it). Trivial changes like typos do not require a Github issue. Your pull request should address just this issue, without pulling in other changes - one PR resolves one issue.
- [x] Format the pull request title like `[ISSUE #123] Fix UnknownException when host config not exist`. Each commit in the pull request should have a meaningful subject line and body.
- [x] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
- [x] Write necessary unit-test(over 80% coverage) to verify your logic correction, more mock a little better when cross module dependency exist. If the new feature or significant change is committed, please remember to add integration-test in [test module](https://github.com/apache/rocketmq/tree/master/test).
- [x] Run `mvn -B clean apache-rat:check findbugs:findbugs checkstyle:checkstyle` to make sure basic checks pass. Run `mvn clean install -DskipITs` to make sure unit-test pass. Run `mvn clean test-compile failsafe:integration-test` to make sure integration-test pass.
- [ ] If this contribution is large, please file an [Apache Individual Contributor License Agreement](http://www.apache.org/licenses/#clas).

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] zhouxinyu commented on pull request #5067: [ISSUE #5066] Fix only individual queues are consumed in high tps scenarios

Posted by GitBox <gi...@apache.org>.

zhouxinyu commented on PR #5067:
URL: https://github.com/apache/rocketmq/pull/5067#issuecomment-1247517356

   Hi, It's not recommended to disable this flag, we can adjust the netty watermark to mitigate this problem.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] guyinyou commented on pull request #5067: [ISSUE #5066] Turn off flow control

Posted by GitBox <gi...@apache.org>.

guyinyou commented on PR #5067:
URL: https://github.com/apache/rocketmq/pull/5067#issuecomment-1246194497

   At high tps (and no performance bottleneck), the consumption rate is much lower than the production rate
   
   ![image](https://user-images.githubusercontent.com/36399867/190054453-b505da3e-0e5f-48b6-877e-a531968a2f99.png)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] zhouxinyu commented on pull request #5067: [ISSUE #5066] Fix only individual queues are consumed in high tps scenarios

Posted by GitBox <gi...@apache.org>.

zhouxinyu commented on PR #5067:
URL: https://github.com/apache/rocketmq/pull/5067#issuecomment-1247517924

   ```
   When the broker ignores the pullRequest, the client can only wait for 30s to time out, and the data of the queue cannot be consumed within 30s.
   ```
   
   Actually, we can add a more intelligent backoff policy to resolve this issue.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] RongtongJin merged pull request #5067: [ISSUE #5066] Fix only individual queues are consumed in high tps scenarios

Posted by GitBox <gi...@apache.org>.

RongtongJin merged PR #5067:
URL: https://github.com/apache/rocketmq/pull/5067


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] codecov-commenter commented on pull request #5067: [ISSUE #5066] Fix only individual queues are consumed in high tps scenarios

Posted by GitBox <gi...@apache.org>.

codecov-commenter commented on PR #5067:
URL: https://github.com/apache/rocketmq/pull/5067#issuecomment-1246315026

   # [Codecov](https://codecov.io/gh/apache/rocketmq/pull/5067?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#5067](https://codecov.io/gh/apache/rocketmq/pull/5067?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (91d73c1) into [develop](https://codecov.io/gh/apache/rocketmq/commit/9442628e42b4e8eb5115b39a7b0613f88f0853b1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (9442628) will **decrease** coverage by `0.01%`.
   > The diff coverage is `25.00%`.
   
   ```diff
   @@              Coverage Diff              @@
   ##             develop    #5067      +/-   ##
   =============================================
   - Coverage      43.18%   43.17%   -0.02%     
   - Complexity      7754     7758       +4     
   =============================================
     Files            995      995              
     Lines          69126    69170      +44     
     Branches        9152     9154       +2     
   =============================================
   + Hits           29852    29864      +12     
   - Misses         35519    35546      +27     
   - Partials        3755     3760       +5     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/rocketmq/pull/5067?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...ker/processor/DefaultPullMessageResultHandler.java](https://codecov.io/gh/apache/rocketmq/pull/5067/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9icm9rZXIvcHJvY2Vzc29yL0RlZmF1bHRQdWxsTWVzc2FnZVJlc3VsdEhhbmRsZXIuamF2YQ==) | `35.13% <0.00%> (ø)` | |
   | [.../java/org/apache/rocketmq/common/BrokerConfig.java](https://codecov.io/gh/apache/rocketmq/pull/5067/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9jb21tb24vQnJva2VyQ29uZmlnLmphdmE=) | `28.26% <33.33%> (ø)` | |
   | [.../org/apache/rocketmq/store/ha/DefaultHAClient.java](https://codecov.io/gh/apache/rocketmq/pull/5067/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3JvY2tldG1xL3N0b3JlL2hhL0RlZmF1bHRIQUNsaWVudC5qYXZh) | `61.02% <0.00%> (-6.16%)` | :arrow_down: |
   | [...etmq/broker/processor/EndTransactionProcessor.java](https://codecov.io/gh/apache/rocketmq/pull/5067/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9icm9rZXIvcHJvY2Vzc29yL0VuZFRyYW5zYWN0aW9uUHJvY2Vzc29yLmphdmE=) | `45.91% <0.00%> (-5.50%)` | :arrow_down: |
   | [...ache/rocketmq/common/stats/MomentStatsItemSet.java](https://codecov.io/gh/apache/rocketmq/pull/5067/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9jb21tb24vc3RhdHMvTW9tZW50U3RhdHNJdGVtU2V0LmphdmE=) | `39.13% <0.00%> (-4.35%)` | :arrow_down: |
   | [...ketmq/proxy/grpc/v2/channel/GrpcClientChannel.java](https://codecov.io/gh/apache/rocketmq/pull/5067/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHJveHkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3JvY2tldG1xL3Byb3h5L2dycGMvdjIvY2hhbm5lbC9HcnBjQ2xpZW50Q2hhbm5lbC5qYXZh) | `56.04% <0.00%> (-3.30%)` | :arrow_down: |
   | [.../apache/rocketmq/store/ha/DefaultHAConnection.java](https://codecov.io/gh/apache/rocketmq/pull/5067/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3JvY2tldG1xL3N0b3JlL2hhL0RlZmF1bHRIQUNvbm5lY3Rpb24uamF2YQ==) | `65.86% <0.00%> (-2.41%)` | :arrow_down: |
   | [...rocketmq/common/message/MessageClientIDSetter.java](https://codecov.io/gh/apache/rocketmq/pull/5067/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9jb21tb24vbWVzc2FnZS9NZXNzYWdlQ2xpZW50SURTZXR0ZXIuamF2YQ==) | `71.42% <0.00%> (-2.39%)` | :arrow_down: |
   | [...mq/client/impl/consumer/RebalanceLitePullImpl.java](https://codecov.io/gh/apache/rocketmq/pull/5067/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y2xpZW50L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9jbGllbnQvaW1wbC9jb25zdW1lci9SZWJhbGFuY2VMaXRlUHVsbEltcGwuamF2YQ==) | `69.86% <0.00%> (-1.37%)` | :arrow_down: |
   | [.../org/apache/rocketmq/store/timer/TimerMetrics.java](https://codecov.io/gh/apache/rocketmq/pull/5067/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3JvY2tldG1xL3N0b3JlL3RpbWVyL1RpbWVyTWV0cmljcy5qYXZh) | `67.42% <0.00%> (-1.33%)` | :arrow_down: |
   | ... and [18 more](https://codecov.io/gh/apache/rocketmq/pull/5067/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   :mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] lizhanhui commented on pull request #5067: [ISSUE #5066] Fix only individual queues are consumed in high tps scenarios

Posted by GitBox <gi...@apache.org>.

lizhanhui commented on PR #5067:
URL: https://github.com/apache/rocketmq/pull/5067#issuecomment-1251751252

   @RongtongJin Pull request should NOT be merged till all concerns are clarified. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] RongtongJin commented on a diff in pull request #5067: [ISSUE #5066] Fix only individual queues are consumed in high tps scenarios

Posted by GitBox <gi...@apache.org>.

RongtongJin commented on code in PR #5067:
URL: https://github.com/apache/rocketmq/pull/5067#discussion_r970279527


##########
common/src/main/java/org/apache/rocketmq/common/BrokerConfig.java:
##########
@@ -191,7 +191,7 @@ public class BrokerConfig extends BrokerIdentity {
      */
     private long brokerNotActiveTimeoutMillis = 10 * 1000;
 
-    private boolean netWorkFlowController = true;
+    private boolean netWorkFlowController = false;

Review Comment:
   How about renaming enableNetWorkFlowControl？



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] guyinyou commented on a diff in pull request #5067: [ISSUE #5066] Fix only individual queues are consumed in high tps scenarios

Posted by GitBox <gi...@apache.org>.

guyinyou commented on code in PR #5067:
URL: https://github.com/apache/rocketmq/pull/5067#discussion_r970281259


##########
common/src/main/java/org/apache/rocketmq/common/BrokerConfig.java:
##########
@@ -191,7 +191,7 @@ public class BrokerConfig extends BrokerIdentity {
      */
     private long brokerNotActiveTimeoutMillis = 10 * 1000;
 
-    private boolean netWorkFlowController = true;
+    private boolean netWorkFlowController = false;

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] fuyou001 commented on pull request #5067: [ISSUE #5066] Fix only individual queues are consumed in high tps scenarios

Posted by GitBox <gi...@apache.org>.

fuyou001 commented on PR #5067:
URL: https://github.com/apache/rocketmq/pull/5067#issuecomment-1247477193

   RocketMQ netty highwatermark default values is small.so we can adjustment


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] guyinyou commented on pull request #5067: [ISSUE #5066] Fix only individual queues are consumed in high tps scenarios

Posted by GitBox <gi...@apache.org>.

guyinyou commented on PR #5067:
URL: https://github.com/apache/rocketmq/pull/5067#issuecomment-1246195849

   broker.log as shown below
   
   ![image](https://user-images.githubusercontent.com/36399867/190054870-82ee49c1-6fde-4d15-9542-163d65feb519.png)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org