You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@rocketmq.apache.org by GitBox <gi...@apache.org> on 2022/05/24 10:40:59 UTC
[GitHub] [rocketmq] hzh0425 opened a new pull request, #4367: [Summer of code] Support async leanrer in controller mode
hzh0425 opened a new pull request, #4367:
URL: https://github.com/apache/rocketmq/pull/4367
## What is the purpose of the change
tracking issue: https://github.com/apache/rocketmq/issues/4330
## Brief changelog
Add new broker role -- Async leanrer in controller mode
## Verifying this change
XXXX
Follow this checklist to help us incorporate your contribution quickly and easily. Notice, `it would be helpful if you could finish the following 5 checklist(the last one is not necessary)before request the community to review your PR`.
- [x] Make sure there is a [Github issue](https://github.com/apache/rocketmq/issues) filed for the change (usually before you start working on it). Trivial changes like typos do not require a Github issue. Your pull request should address just this issue, without pulling in other changes - one PR resolves one issue.
- [x] Format the pull request title like `[ISSUE #123] Fix UnknownException when host config not exist`. Each commit in the pull request should have a meaningful subject line and body.
- [x] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
- [x] Write necessary unit-test(over 80% coverage) to verify your logic correction, more mock a little better when cross module dependency exist. If the new feature or significant change is committed, please remember to add integration-test in [test module](https://github.com/apache/rocketmq/tree/master/test).
- [x] Run `mvn -B clean apache-rat:check findbugs:findbugs checkstyle:checkstyle` to make sure basic checks pass. Run `mvn clean install -DskipITs` to make sure unit-test pass. Run `mvn clean test-compile failsafe:integration-test` to make sure integration-test pass.
- [ ] If this contribution is large, please file an [Apache Individual Contributor License Agreement](http://www.apache.org/licenses/#clas).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] RongtongJin commented on a diff in pull request #4367: [Summer of code] Support async learner in controller mode
Posted by GitBox <gi...@apache.org>.
RongtongJin commented on code in PR #4367:
URL: https://github.com/apache/rocketmq/pull/4367#discussion_r880387878
##########
store/src/main/java/org/apache/rocketmq/store/ha/autoswitch/AutoSwitchHAConnection.java:
##########
@@ -278,16 +287,20 @@ protected boolean processReadResult(ByteBuffer byteBufferRead) {
case HANDSHAKE:
if (diff >= AutoSwitchHAClient.HANDSHAKE_HEADER_SIZE) {
isSlaveSendHandshake = true;
- // Flag(SyncFromLastFile)
- long syncFromLastFileFlag = byteBufferRead.getInt(readPosition + 4);
- if (syncFromLastFileFlag == AutoSwitchHAClient.SYNC_FROM_LAST_FILE) {
- AutoSwitchHAConnection.this.isSyncFromLastFile = true;
- LOGGER.info("Slave request sync from lastFile");
- }
// SlaveId
- AutoSwitchHAConnection.this.slaveId = byteBufferRead.getLong(readPosition + 8);
+ AutoSwitchHAConnection.this.slaveId = byteBufferRead.getLong(readPosition + 4);
// AddressLength
- int addressLength = byteBufferRead.getInt(readPosition + 16);
+ int addressLength = byteBufferRead.getInt(readPosition + 12);
+ // Flag(isSyncFromLastFile)
+ short syncFromLastFileFlag = byteBufferRead.getShort(readPosition + 16);
+ if (syncFromLastFileFlag == 1) {
+ AutoSwitchHAConnection.this.isSyncFromLastFile = true;
+ }
+ // Flag(isAsyncLearner role)
+ short isAsyncLearner = byteBufferRead.getShort(readPosition + 18);
+ if (isAsyncLearner == 1) {
+ AutoSwitchHAConnection.this.isAsyncLearner = true;
+ }
Review Comment:
当出现半包的时候，diff < AutoSwitchHAClient.HANDSHAKE_HEADER_SIZE + addressLength，也就是ip地址body传输了一半，这个时候直接复制slaveAddress=“”，并且继续往下走会得到错误的slaveAddress
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] RongtongJin commented on a diff in pull request #4367: [Summer of code] Support async learner in controller mode
Posted by GitBox <gi...@apache.org>.
RongtongJin commented on code in PR #4367:
URL: https://github.com/apache/rocketmq/pull/4367#discussion_r881571351
##########
store/src/main/java/org/apache/rocketmq/store/ha/autoswitch/AutoSwitchHAConnection.java:
##########
@@ -277,28 +286,32 @@ protected boolean processReadResult(ByteBuffer byteBufferRead) {
switch (slaveState) {
case HANDSHAKE:
if (diff >= AutoSwitchHAClient.HANDSHAKE_HEADER_SIZE) {
+ final int headerSize = AutoSwitchHAClient.HANDSHAKE_HEADER_SIZE;
Review Comment:
好像没必要定义一个本地变量
##########
store/src/main/java/org/apache/rocketmq/store/ha/autoswitch/AutoSwitchHAConnection.java:
##########
@@ -277,28 +286,32 @@ protected boolean processReadResult(ByteBuffer byteBufferRead) {
switch (slaveState) {
case HANDSHAKE:
if (diff >= AutoSwitchHAClient.HANDSHAKE_HEADER_SIZE) {
+ final int headerSize = AutoSwitchHAClient.HANDSHAKE_HEADER_SIZE;
isSlaveSendHandshake = true;
- // Flag(SyncFromLastFile)
- long syncFromLastFileFlag = byteBufferRead.getInt(readPosition + 4);
- if (syncFromLastFileFlag == AutoSwitchHAClient.SYNC_FROM_LAST_FILE) {
+ // Flag(isSyncFromLastFile)
+ short syncFromLastFileFlag = byteBufferRead.getShort(readPosition + headerSize - 8);
+ if (syncFromLastFileFlag == 1) {
AutoSwitchHAConnection.this.isSyncFromLastFile = true;
- LOGGER.info("Slave request sync from lastFile");
}
- // SlaveId
- AutoSwitchHAConnection.this.slaveId = byteBufferRead.getLong(readPosition + 8);
+ // Flag(isAsyncLearner role)
+ short isAsyncLearner = byteBufferRead.getShort(readPosition + headerSize - 6);
+ if (isAsyncLearner == 1) {
+ AutoSwitchHAConnection.this.isAsyncLearner = true;
+ }
// AddressLength
- int addressLength = byteBufferRead.getInt(readPosition + 16);
- // Address
- if (diff >= AutoSwitchHAClient.HANDSHAKE_HEADER_SIZE + addressLength) {
- final byte[] addressData = new byte[addressLength];
- byteBufferRead.position(readPosition + 20);
- byteBufferRead.get(addressData);
- AutoSwitchHAConnection.this.slaveAddress = new String(addressData);
- } else {
- AutoSwitchHAConnection.this.slaveAddress = "";
+ int addressLength = byteBufferRead.getInt(readPosition + headerSize - 4);
+ if (diff < AutoSwitchHAClient.HANDSHAKE_HEADER_SIZE + addressLength) {
+ break;
}
Review Comment:
这个应该在最上面判断。不然上面白计算了
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] coveralls commented on pull request #4367: [Summer of code] Support async learner in controller mode
Posted by GitBox <gi...@apache.org>.
coveralls commented on PR #4367:
URL: https://github.com/apache/rocketmq/pull/4367#issuecomment-1135995515
[](https://coveralls.io/builds/49406681)
Coverage increased (+0.01%) to 47.697% when pulling **c92e0c476db45c598f4fd71c8cd4591a9db678ed on hzh0425:feature/async_learner** into **f5c5f5c4f0ac9fe4bfbea0cf7a94c7037abdf716 on apache:5.0.0-beta-dledger-controller**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] codecov-commenter commented on pull request #4367: [Summer of code] Support async learner in controller mode
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #4367:
URL: https://github.com/apache/rocketmq/pull/4367#issuecomment-1137098500
# [Codecov](https://codecov.io/gh/apache/rocketmq/pull/4367?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#4367](https://codecov.io/gh/apache/rocketmq/pull/4367?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (46ddc3c) into [5.0.0-beta-dledger-controller](https://codecov.io/gh/apache/rocketmq/commit/f5c5f5c4f0ac9fe4bfbea0cf7a94c7037abdf716?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f5c5f5c) will **increase** coverage by `0.05%`.
> The diff coverage is `88.57%`.
```diff
@@ Coverage Diff @@
## 5.0.0-beta-dledger-controller #4367 +/- ##
===================================================================
+ Coverage 43.47% 43.53% +0.05%
- Complexity 6330 6344 +14
===================================================================
Files 835 835
Lines 59359 59369 +10
Branches 8092 8094 +2
===================================================================
+ Hits 25807 25845 +38
+ Misses 30210 30185 -25
+ Partials 3342 3339 -3
```
| [Impacted Files](https://codecov.io/gh/apache/rocketmq/pull/4367?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...mq/store/ha/autoswitch/AutoSwitchHAConnection.java](https://codecov.io/gh/apache/rocketmq/pull/4367/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3JvY2tldG1xL3N0b3JlL2hhL2F1dG9zd2l0Y2gvQXV0b1N3aXRjaEhBQ29ubmVjdGlvbi5qYXZh) | `73.29% <80.00%> (-0.46%)` | :arrow_down: |
| [...ache/rocketmq/store/config/MessageStoreConfig.java](https://codecov.io/gh/apache/rocketmq/pull/4367/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3JvY2tldG1xL3N0b3JlL2NvbmZpZy9NZXNzYWdlU3RvcmVDb25maWcuamF2YQ==) | `56.27% <100.00%> (+0.34%)` | :arrow_up: |
| [...cketmq/store/ha/autoswitch/AutoSwitchHAClient.java](https://codecov.io/gh/apache/rocketmq/pull/4367/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3JvY2tldG1xL3N0b3JlL2hhL2F1dG9zd2l0Y2gvQXV0b1N3aXRjaEhBQ2xpZW50LmphdmE=) | `75.65% <100.00%> (+0.09%)` | :arrow_up: |
| [...etmq/namesrv/routeinfo/BatchUnRegisterService.java](https://codecov.io/gh/apache/rocketmq/pull/4367/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-bmFtZXNydi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcm9ja2V0bXEvbmFtZXNydi9yb3V0ZWluZm8vQmF0Y2hVblJlZ2lzdGVyU2VydmljZS5qYXZh) | `94.73% <0.00%> (-5.27%)` | :arrow_down: |
| [...ache/rocketmq/common/stats/MomentStatsItemSet.java](https://codecov.io/gh/apache/rocketmq/pull/4367/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9jb21tb24vc3RhdHMvTW9tZW50U3RhdHNJdGVtU2V0LmphdmE=) | `39.13% <0.00%> (-4.35%)` | :arrow_down: |
| [.../org/apache/rocketmq/store/ha/DefaultHAClient.java](https://codecov.io/gh/apache/rocketmq/pull/4367/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3JvY2tldG1xL3N0b3JlL2hhL0RlZmF1bHRIQUNsaWVudC5qYXZh) | `58.46% <0.00%> (-0.52%)` | :arrow_down: |
| [.../apache/rocketmq/logging/inner/LoggingBuilder.java](https://codecov.io/gh/apache/rocketmq/pull/4367/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-bG9nZ2luZy9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcm9ja2V0bXEvbG9nZ2luZy9pbm5lci9Mb2dnaW5nQnVpbGRlci5qYXZh) | `64.71% <0.00%> (+0.31%)` | :arrow_up: |
| [...he/rocketmq/client/impl/consumer/ProcessQueue.java](https://codecov.io/gh/apache/rocketmq/pull/4367/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y2xpZW50L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9jbGllbnQvaW1wbC9jb25zdW1lci9Qcm9jZXNzUXVldWUuamF2YQ==) | `61.00% <0.00%> (+0.45%)` | :arrow_up: |
| [...rocketmq/broker/processor/PopMessageProcessor.java](https://codecov.io/gh/apache/rocketmq/pull/4367/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9icm9rZXIvcHJvY2Vzc29yL1BvcE1lc3NhZ2VQcm9jZXNzb3IuamF2YQ==) | `39.57% <0.00%> (+0.53%)` | :arrow_up: |
| ... and [8 more](https://codecov.io/gh/apache/rocketmq/pull/4367/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/rocketmq/pull/4367?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/rocketmq/pull/4367?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [f5c5f5c...46ddc3c](https://codecov.io/gh/apache/rocketmq/pull/4367?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] RongtongJin merged pull request #4367: [Summer of code] Support async learner in controller mode
Posted by GitBox <gi...@apache.org>.
RongtongJin merged PR #4367:
URL: https://github.com/apache/rocketmq/pull/4367
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org