You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@rocketmq.apache.org by GitBox <gi...@apache.org> on 2020/12/22 01:07:14 UTC

[GitHub] [rocketmq] crazywen opened a new pull request #2487: [ISSUE #2476] Fix HaClient Infinite loop

crazywen opened a new pull request #2487:
URL: https://github.com/apache/rocketmq/pull/2487


   try to fix epollwait no block, trash selector, regen it!
   
   ## What is the purpose of the change
   
   fix epollwait no block
   [https://github.com/apache/rocketmq/issues/2487](url)
   
   Follow this checklist to help us incorporate your contribution quickly and easily. Notice, `it would be helpful if you could finish the following 5 checklist(the last one is not necessary)before request the community to review your PR`.
   
   - [x] Make sure there is a [Github issue](https://github.com/apache/rocketmq/issues) filed for the change (usually before you start working on it). Trivial changes like typos do not require a Github issue. Your pull request should address just this issue, without pulling in other changes - one PR resolves one issue. 
   - [x] Format the pull request title like `[ISSUE #123] Fix UnknownException when host config not exist`. Each commit in the pull request should have a meaningful subject line and body.
   - [x] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
   - [x] Write necessary unit-test(over 80% coverage) to verify your logic correction, more mock a little better when cross module dependency exist. If the new feature or significant change is committed, please remember to add integration-test in [test module](https://github.com/apache/rocketmq/tree/master/test).
   - [x] Run `mvn -B clean apache-rat:check findbugs:findbugs checkstyle:checkstyle` to make sure basic checks pass. Run `mvn clean install -DskipITs` to make sure unit-test pass. Run `mvn clean test-compile failsafe:integration-test`  to make sure integration-test pass.
   - [ ] If this contribution is large, please file an [Apache Individual Contributor License Agreement](http://www.apache.org/licenses/#clas).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] crazywen commented on pull request #2487: [ISSUE #2476] Fix HaClient Infinite loop

Posted by GitBox <gi...@apache.org>.
crazywen commented on pull request #2487:
URL: https://github.com/apache/rocketmq/pull/2487#issuecomment-749279094


   > @crazywen The original reason is that the HAClient.READ_MAX_BUFFER_SIZE muse be equal or larger than the haTransferBatchSize.
   > 
   > Otherwise, the slave HAClient cannot read all the data to byteBufferRead as it has been full already, and the reallocation will take no effect, the new buffer will still be full.
   > 
   > ![image](https://user-images.githubusercontent.com/8434428/102320144-df6d1d80-3fb6-11eb-8f06-6f018c0e056a.png)
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] dongeforever commented on pull request #2487: [ISSUE #2476] Fix HaClient Infinite loop

Posted by GitBox <gi...@apache.org>.
dongeforever commented on pull request #2487:
URL: https://github.com/apache/rocketmq/pull/2487#issuecomment-745866852


   @crazywen It is not a good way to fix the BUFFER SIZE.  You could introduce another config(for example haBufferReadSize) to control the HAClient read buffer size, which is now fixed to 4M by READ_MAX_BUFFER_SIZE.
   And then make sure the haTransferBatchSize is equal or smaller than haBufferReadSize.
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] dongeforever commented on pull request #2487: [ISSUE #2476] Fix HaClient Infinite loop

Posted by GitBox <gi...@apache.org>.
dongeforever commented on pull request #2487:
URL: https://github.com/apache/rocketmq/pull/2487#issuecomment-745862563


   @crazywen  The original reason is that the HAClient.READ_MAX_BUFFER_SIZE muse be equal or larger than the haTransferBatchSize.
   
   Otherwise, the slave HAClient cannot read all the data to byteBufferRead as it has been full already, and the reallocation will take no effect, the new buffer will still be full.
   
   ![image](https://user-images.githubusercontent.com/8434428/102320144-df6d1d80-3fb6-11eb-8f06-6f018c0e056a.png)
   
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] dongeforever commented on pull request #2487: [ISSUE #2476] Fix HaClient Infinite loop

Posted by GitBox <gi...@apache.org>.
dongeforever commented on pull request #2487:
URL: https://github.com/apache/rocketmq/pull/2487#issuecomment-751190597


   @crazywen Usually, the config of each node should keep the same in the same cluster(or at least for pair of master-slave which has the same broker name).
   If so, just make sure haTransferBatchSize is equal or smaller than haReceiveBatchSize in the config file.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] crazywen commented on pull request #2487: [ISSUE #2476] Fix HaClient Infinite loop

Posted by GitBox <gi...@apache.org>.
crazywen commented on pull request #2487:
URL: https://github.com/apache/rocketmq/pull/2487#issuecomment-749280725


   > @crazywen It is not a good way to fix the BUFFER SIZE. You could introduce another config(for example haReceiveBatchSize) to control the HAClient read buffer size, which is now fixed to 4M by READ_MAX_BUFFER_SIZE.
   > And then make sure the haTransferBatchSize is equal or smaller than haReceiveBatchSize.
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] crazywen commented on pull request #2487: [ISSUE #2476] Fix HaClient Infinite loop

Posted by GitBox <gi...@apache.org>.
crazywen commented on pull request #2487:
URL: https://github.com/apache/rocketmq/pull/2487#issuecomment-749282825


   
   
   
   > @crazywen It is not a good way to fix the BUFFER SIZE. You could introduce another config(for example haReceiveBatchSize) to control the HAClient read buffer size, which is now fixed to 4M by READ_MAX_BUFFER_SIZE.
   > And then make sure the haTransferBatchSize is equal or smaller than haReceiveBatchSize.
   
   haReceiveBatchSize also should have the max size, and the property is valid in slave, and haTransferBatchSize is valid in master, if like what you suggest, the master property should depend on the salve ?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] codecov-io commented on pull request #2487: [ISSUE #2476] Fix HaClient Infinite loop

Posted by GitBox <gi...@apache.org>.
codecov-io commented on pull request #2487:
URL: https://github.com/apache/rocketmq/pull/2487#issuecomment-749287610


   # [Codecov](https://codecov.io/gh/apache/rocketmq/pull/2487?src=pr&el=h1) Report
   > Merging [#2487](https://codecov.io/gh/apache/rocketmq/pull/2487?src=pr&el=desc) (cc68e7d) into [master](https://codecov.io/gh/apache/rocketmq/commit/9f95a972e10e0681bc3f2d00e9957aa212e897b5?el=desc) (9f95a97) will **increase** coverage by `0.76%`.
   > The diff coverage is `25.00%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/rocketmq/pull/2487/graphs/tree.svg?width=650&height=150&src=pr&token=4w0sxP1wZv)](https://codecov.io/gh/apache/rocketmq/pull/2487?src=pr&el=tree)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #2487      +/-   ##
   ============================================
   + Coverage     45.69%   46.46%   +0.76%     
   - Complexity     4271     4551     +280     
   ============================================
     Files           546      547       +1     
     Lines         35860    38475    +2615     
     Branches       4763     5551     +788     
   ============================================
   + Hits          16388    17876    +1488     
   - Misses        17429    18432    +1003     
   - Partials       2043     2167     +124     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/rocketmq/pull/2487?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | [...ache/rocketmq/store/config/MessageStoreConfig.java](https://codecov.io/gh/apache/rocketmq/pull/2487/diff?src=pr&el=tree#diff-c3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3JvY2tldG1xL3N0b3JlL2NvbmZpZy9NZXNzYWdlU3RvcmVDb25maWcuamF2YQ==) | `63.68% <25.00%> (+3.68%)` | `117.00 <1.00> (+40.00)` | |
   | [...e/rocketmq/client/exception/MQBrokerException.java](https://codecov.io/gh/apache/rocketmq/pull/2487/diff?src=pr&el=tree#diff-Y2xpZW50L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9jbGllbnQvZXhjZXB0aW9uL01RQnJva2VyRXhjZXB0aW9uLmphdmE=) | `56.25% <0.00%> (-10.42%)` | `3.00% <0.00%> (+2.00%)` | :arrow_down: |
   | [...ketmq/client/impl/consumer/PullMessageService.java](https://codecov.io/gh/apache/rocketmq/pull/2487/diff?src=pr&el=tree#diff-Y2xpZW50L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9jbGllbnQvaW1wbC9jb25zdW1lci9QdWxsTWVzc2FnZVNlcnZpY2UuamF2YQ==) | `71.11% <0.00%> (-8.89%)` | `8.00% <0.00%> (-2.00%)` | |
   | [...in/java/org/apache/rocketmq/test/util/MQAdmin.java](https://codecov.io/gh/apache/rocketmq/pull/2487/diff?src=pr&el=tree#diff-dGVzdC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcm9ja2V0bXEvdGVzdC91dGlsL01RQWRtaW4uamF2YQ==) | `38.88% <0.00%> (-5.56%)` | `7.00% <0.00%> (-1.00%)` | |
   | [...ache/rocketmq/common/stats/MomentStatsItemSet.java](https://codecov.io/gh/apache/rocketmq/pull/2487/diff?src=pr&el=tree#diff-Y29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9jb21tb24vc3RhdHMvTW9tZW50U3RhdHNJdGVtU2V0LmphdmE=) | `39.13% <0.00%> (-4.35%)` | `5.00% <0.00%> (-1.00%)` | |
   | [...ketmq/common/protocol/body/ConsumerConnection.java](https://codecov.io/gh/apache/rocketmq/pull/2487/diff?src=pr&el=tree#diff-Y29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9jb21tb24vcHJvdG9jb2wvYm9keS9Db25zdW1lckNvbm5lY3Rpb24uamF2YQ==) | `95.83% <0.00%> (-4.17%)` | `13.00% <0.00%> (-1.00%)` | |
   | [...mq/client/trace/hook/SendMessageTraceHookImpl.java](https://codecov.io/gh/apache/rocketmq/pull/2487/diff?src=pr&el=tree#diff-Y2xpZW50L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9jbGllbnQvdHJhY2UvaG9vay9TZW5kTWVzc2FnZVRyYWNlSG9va0ltcGwuamF2YQ==) | `67.34% <0.00%> (-3.39%)` | `4.00% <0.00%> (ø%)` | |
   | [...ava/org/apache/rocketmq/test/util/VerifyUtils.java](https://codecov.io/gh/apache/rocketmq/pull/2487/diff?src=pr&el=tree#diff-dGVzdC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcm9ja2V0bXEvdGVzdC91dGlsL1ZlcmlmeVV0aWxzLmphdmE=) | `46.26% <0.00%> (-2.99%)` | `14.00% <0.00%> (-1.00%)` | |
   | [...nt/impl/consumer/ConsumeMessageOrderlyService.java](https://codecov.io/gh/apache/rocketmq/pull/2487/diff?src=pr&el=tree#diff-Y2xpZW50L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9jbGllbnQvaW1wbC9jb25zdW1lci9Db25zdW1lTWVzc2FnZU9yZGVybHlTZXJ2aWNlLmphdmE=) | `42.59% <0.00%> (-2.89%)` | `16.00% <0.00%> (-3.00%)` | |
   | [...ava/org/apache/rocketmq/filter/util/BitsArray.java](https://codecov.io/gh/apache/rocketmq/pull/2487/diff?src=pr&el=tree#diff-ZmlsdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9maWx0ZXIvdXRpbC9CaXRzQXJyYXkuamF2YQ==) | `59.82% <0.00%> (-2.57%)` | `30.00% <0.00%> (-1.00%)` | |
   | ... and [39 more](https://codecov.io/gh/apache/rocketmq/pull/2487/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/rocketmq/pull/2487?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/rocketmq/pull/2487?src=pr&el=footer). Last update [9f95a97...cc68e7d](https://codecov.io/gh/apache/rocketmq/pull/2487?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] coveralls edited a comment on pull request #2487: [ISSUE #2476] Fix HaClient Infinite loop

Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #2487:
URL: https://github.com/apache/rocketmq/pull/2487#issuecomment-740564128


   
   [![Coverage Status](https://coveralls.io/builds/35534284/badge)](https://coveralls.io/builds/35534284)
   
   Coverage increased (+0.02%) to 51.319% when pulling **cc68e7d62cc65bb2707c6064166141295382f144 on crazywen:master** into **9f95a972e10e0681bc3f2d00e9957aa212e897b5 on apache:master**.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] crazywen removed a comment on pull request #2487: [ISSUE #2476] Fix HaClient Infinite loop

Posted by GitBox <gi...@apache.org>.
crazywen removed a comment on pull request #2487:
URL: https://github.com/apache/rocketmq/pull/2487#issuecomment-749279094


   > @crazywen The original reason is that the HAClient.READ_MAX_BUFFER_SIZE muse be equal or larger than the haTransferBatchSize.
   > 
   > Otherwise, the slave HAClient cannot read all the data to byteBufferRead as it has been full already, and the reallocation will take no effect, the new buffer will still be full.
   > 
   > ![image](https://user-images.githubusercontent.com/8434428/102320144-df6d1d80-3fb6-11eb-8f06-6f018c0e056a.png)
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] coveralls edited a comment on pull request #2487: [ISSUE #2476] Fix HaClient Infinite loop

Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #2487:
URL: https://github.com/apache/rocketmq/pull/2487#issuecomment-740564128


   
   [![Coverage Status](https://coveralls.io/builds/35878659/badge)](https://coveralls.io/builds/35878659)
   
   Coverage increased (+0.5%) to 51.832% when pulling **cc68e7d62cc65bb2707c6064166141295382f144 on crazywen:master** into **9f95a972e10e0681bc3f2d00e9957aa212e897b5 on apache:master**.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] crazywen closed pull request #2487: [ISSUE #2476] Fix HaClient Infinite loop

Posted by GitBox <gi...@apache.org>.
crazywen closed pull request #2487:
URL: https://github.com/apache/rocketmq/pull/2487


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] coveralls commented on pull request #2487: [ISSUE #2476] Fix HaClient Infinite loop

Posted by GitBox <gi...@apache.org>.
coveralls commented on pull request #2487:
URL: https://github.com/apache/rocketmq/pull/2487#issuecomment-740564128


   
   [![Coverage Status](https://coveralls.io/builds/35534113/badge)](https://coveralls.io/builds/35534113)
   
   Coverage increased (+0.04%) to 51.336% when pulling **cc68e7d62cc65bb2707c6064166141295382f144 on crazywen:master** into **9f95a972e10e0681bc3f2d00e9957aa212e897b5 on apache:master**.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] dongeforever edited a comment on pull request #2487: [ISSUE #2476] Fix HaClient Infinite loop

Posted by GitBox <gi...@apache.org>.
dongeforever edited a comment on pull request #2487:
URL: https://github.com/apache/rocketmq/pull/2487#issuecomment-745866852


   @crazywen It is not a good way to fix the BUFFER SIZE.  You could introduce another config(for example haReceiveBatchSize) to control the HAClient read buffer size, which is now fixed to 4M by READ_MAX_BUFFER_SIZE.
   And then make sure the haTransferBatchSize is equal or smaller than haReceiveBatchSize.
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org