You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@rocketmq.apache.org by "RongtongJin (via GitHub)" <gi...@apache.org> on 2023/04/24 03:53:12 UTC
[GitHub] [rocketmq] RongtongJin opened a new pull request, #6641: [ISSUE #6570] Fix the issue that expectLogicOffset is greater than currentLogicOffset in consumeQueue build when the message is illegal
RongtongJin opened a new pull request, #6641:
URL: https://github.com/apache/rocketmq/pull/6641
**Make sure set the target branch to `develop`**
## What is the purpose of the change
<!--
If this PR fixes a GitHub issue, please add `fixes #<XXX>` or `closes #<XXX>`. Please refer to the documentation for more information:
https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue
-->
fix #6570
## Brief changelog
Fix the issue that expectLogicOffset is greater than currentLogicOffset in consumeQueue build
## Verifying this change
XXXX
Follow this checklist to help us incorporate your contribution quickly and easily. Notice, `it would be helpful if you could finish the following 5 checklist(the last one is not necessary)before request the community to review your PR`.
- [x] Make sure there is a [Github issue](https://github.com/apache/rocketmq/issues) filed for the change (usually before you start working on it). Trivial changes like typos do not require a Github issue. Your pull request should address just this issue, without pulling in other changes - one PR resolves one issue.
- [x] Format the pull request title like `[ISSUE #123] Fix UnknownException when host config not exist`. Each commit in the pull request should have a meaningful subject line and body.
- [x] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
- [x] Write necessary unit-test(over 80% coverage) to verify your logic correction, more mock a little better when cross module dependency exist. If the new feature or significant change is committed, please remember to add integration-test in [test module](https://github.com/apache/rocketmq/tree/master/test).
- [x] Run `mvn -B clean apache-rat:check findbugs:findbugs checkstyle:checkstyle` to make sure basic checks pass. Run `mvn clean install -DskipITs` to make sure unit-test pass. Run `mvn clean test-compile failsafe:integration-test` to make sure integration-test pass.
- [ ] If this contribution is large, please file an [Apache Individual Contributor License Agreement](http://www.apache.org/licenses/#clas).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] caigy commented on a diff in pull request #6641: [ISSUE #6570] Fix the issue that expectLogicOffset is greater than currentLogicOffset in consumeQueue build when the message is illegal
Posted by "caigy (via GitHub)" <gi...@apache.org>.
caigy commented on code in PR #6641:
URL: https://github.com/apache/rocketmq/pull/6641#discussion_r1175944960
##########
store/src/main/java/org/apache/rocketmq/store/queue/QueueOffsetOperator.java:
##########
@@ -28,39 +28,49 @@
import org.apache.rocketmq.logging.org.slf4j.LoggerFactory;
/**
- * QueueOffsetAssigner is a component for assigning offsets for queues.
+ * QueueOffsetOperator is a component for operating offsets for queues.
*/
-public class QueueOffsetAssigner {
+public class QueueOffsetOperator {
private static final Logger log = LoggerFactory.getLogger(LoggerName.STORE_LOGGER_NAME);
private ConcurrentMap<String, Long> topicQueueTable = new ConcurrentHashMap<>(1024);
private ConcurrentMap<String, Long> batchTopicQueueTable = new ConcurrentHashMap<>(1024);
private ConcurrentMap<String/* topic-queueid */, Long/* offset */> lmqTopicQueueTable = new ConcurrentHashMap<>(1024);
- public long assignQueueOffset(String topicQueueKey, short messageNum) {
+ public long getQueueOffset(String topicQueueKey) {
+ return ConcurrentHashMapUtils.computeIfAbsent(this.topicQueueTable, topicQueueKey, k -> 0L);
+ }
+
+ public void increaseQueueOffset(String topicQueueKey, short messageNum) {
Long queueOffset = ConcurrentHashMapUtils.computeIfAbsent(this.topicQueueTable, topicQueueKey, k -> 0L);
- this.topicQueueTable.put(topicQueueKey, queueOffset + messageNum);
- return queueOffset;
+ topicQueueTable.put(topicQueueKey, queueOffset + messageNum);
Review Comment:
Should we use `replace(key, oldVal, newVal)` for atomicity?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] zhouxinyu commented on a diff in pull request #6641: [ISSUE #6570] Fix the issue that expectLogicOffset is greater than currentLogicOffset in consumeQueue build when the message is illegal
Posted by "zhouxinyu (via GitHub)" <gi...@apache.org>.
zhouxinyu commented on code in PR #6641:
URL: https://github.com/apache/rocketmq/pull/6641#discussion_r1176052975
##########
store/src/main/java/org/apache/rocketmq/store/ConsumeQueue.java:
##########
@@ -803,14 +801,37 @@ public void assignQueueOffset(QueueOffsetAssigner queueOffsetAssigner, MessageEx
for (int i = 0; i < queues.length; i++) {
String key = queueKey(queues[i], msg);
if (messageStore.getMessageStoreConfig().isEnableLmq() && MixAll.isLmq(key)) {
- queueOffsets[i] = queueOffsetAssigner.assignLmqOffset(key, (short) 1);
+ queueOffsets[i] = queueOffsetOperator.getLmqOffset(key);
}
}
MessageAccessor.putProperty(msg, MessageConst.PROPERTY_INNER_MULTI_QUEUE_OFFSET,
StringUtils.join(queueOffsets, MixAll.MULTI_DISPATCH_QUEUE_SPLITTER));
removeWaitStorePropertyString(msg);
}
+ @Override
+ public void increaseQueueOffset(QueueOffsetOperator queueOffsetOperator, MessageExtBrokerInner msg,
+ short messageNum) {
+ String topicQueueKey = getTopic() + "-" + getQueueId();
+ queueOffsetOperator.increaseQueueOffset(topicQueueKey, messageNum);
+
+ // For LMQ
+ if (!messageStore.getMessageStoreConfig().isEnableMultiDispatch() || msg.getTopic().startsWith(MixAll.RETRY_GROUP_TOPIC_PREFIX)) {
Review Comment:
Could you please write more comments to explain why three are some other branches?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] RongtongJin commented on a diff in pull request #6641: [ISSUE #6570] Fix the issue that expectLogicOffset is greater than currentLogicOffset in consumeQueue build when the message is illegal
Posted by "RongtongJin (via GitHub)" <gi...@apache.org>.
RongtongJin commented on code in PR #6641:
URL: https://github.com/apache/rocketmq/pull/6641#discussion_r1176231728
##########
store/src/main/java/org/apache/rocketmq/store/ConsumeQueue.java:
##########
@@ -803,14 +801,37 @@ public void assignQueueOffset(QueueOffsetAssigner queueOffsetAssigner, MessageEx
for (int i = 0; i < queues.length; i++) {
String key = queueKey(queues[i], msg);
if (messageStore.getMessageStoreConfig().isEnableLmq() && MixAll.isLmq(key)) {
- queueOffsets[i] = queueOffsetAssigner.assignLmqOffset(key, (short) 1);
+ queueOffsets[i] = queueOffsetOperator.getLmqOffset(key);
}
}
MessageAccessor.putProperty(msg, MessageConst.PROPERTY_INNER_MULTI_QUEUE_OFFSET,
StringUtils.join(queueOffsets, MixAll.MULTI_DISPATCH_QUEUE_SPLITTER));
removeWaitStorePropertyString(msg);
}
+ @Override
+ public void increaseQueueOffset(QueueOffsetOperator queueOffsetOperator, MessageExtBrokerInner msg,
+ short messageNum) {
+ String topicQueueKey = getTopic() + "-" + getQueueId();
+ queueOffsetOperator.increaseQueueOffset(topicQueueKey, messageNum);
+
+ // For LMQ
+ if (!messageStore.getMessageStoreConfig().isEnableMultiDispatch() || msg.getTopic().startsWith(MixAll.RETRY_GROUP_TOPIC_PREFIX)) {
Review Comment:
> Could you please write more comments to explain why three are some other branches?
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] odbozhou merged pull request #6641: [ISSUE #6570] Fix the issue that expectLogicOffset is greater than currentLogicOffset in consumeQueue build when the message is illegal
Posted by "odbozhou (via GitHub)" <gi...@apache.org>.
odbozhou merged PR #6641:
URL: https://github.com/apache/rocketmq/pull/6641
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] ni-ze commented on a diff in pull request #6641: [ISSUE #6570] Fix the issue that expectLogicOffset is greater than currentLogicOffset in consumeQueue build when the message is illegal
Posted by "ni-ze (via GitHub)" <gi...@apache.org>.
ni-ze commented on code in PR #6641:
URL: https://github.com/apache/rocketmq/pull/6641#discussion_r1177279061
##########
store/src/main/java/org/apache/rocketmq/store/queue/QueueOffsetOperator.java:
##########
@@ -28,47 +28,49 @@
import org.apache.rocketmq.logging.org.slf4j.LoggerFactory;
/**
- * QueueOffsetAssigner is a component for assigning offsets for queues.
+ * QueueOffsetOperator is a component for operating offsets for queues.
*/
-public class QueueOffsetAssigner {
+public class QueueOffsetOperator {
private static final Logger log = LoggerFactory.getLogger(LoggerName.STORE_LOGGER_NAME);
private ConcurrentMap<String, Long> topicQueueTable = new ConcurrentHashMap<>(1024);
private ConcurrentMap<String, Long> batchTopicQueueTable = new ConcurrentHashMap<>(1024);
private ConcurrentMap<String/* topic-queueid */, Long/* offset */> lmqTopicQueueTable = new ConcurrentHashMap<>(1024);
- public long assignQueueOffset(String topicQueueKey, short messageNum) {
+ public long getQueueOffset(String topicQueueKey) {
+ return ConcurrentHashMapUtils.computeIfAbsent(this.topicQueueTable, topicQueueKey, k -> 0L);
+ }
+
+ public void increaseQueueOffset(String topicQueueKey, short messageNum) {
Long queueOffset = ConcurrentHashMapUtils.computeIfAbsent(this.topicQueueTable, topicQueueKey, k -> 0L);
- this.topicQueueTable.put(topicQueueKey, queueOffset + messageNum);
- return queueOffset;
+ topicQueueTable.put(topicQueueKey, queueOffset + messageNum);
}
public void updateQueueOffset(String topicQueueKey, long offset) {
this.topicQueueTable.put(topicQueueKey, offset);
}
Review Comment:
updateQueueOffset seems not be used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] RongtongJin commented on a diff in pull request #6641: [ISSUE #6570] Fix the issue that expectLogicOffset is greater than currentLogicOffset in consumeQueue build when the message is illegal
Posted by "RongtongJin (via GitHub)" <gi...@apache.org>.
RongtongJin commented on code in PR #6641:
URL: https://github.com/apache/rocketmq/pull/6641#discussion_r1176124697
##########
store/src/main/java/org/apache/rocketmq/store/queue/QueueOffsetOperator.java:
##########
@@ -28,39 +28,49 @@
import org.apache.rocketmq.logging.org.slf4j.LoggerFactory;
/**
- * QueueOffsetAssigner is a component for assigning offsets for queues.
+ * QueueOffsetOperator is a component for operating offsets for queues.
*/
-public class QueueOffsetAssigner {
+public class QueueOffsetOperator {
private static final Logger log = LoggerFactory.getLogger(LoggerName.STORE_LOGGER_NAME);
private ConcurrentMap<String, Long> topicQueueTable = new ConcurrentHashMap<>(1024);
private ConcurrentMap<String, Long> batchTopicQueueTable = new ConcurrentHashMap<>(1024);
private ConcurrentMap<String/* topic-queueid */, Long/* offset */> lmqTopicQueueTable = new ConcurrentHashMap<>(1024);
- public long assignQueueOffset(String topicQueueKey, short messageNum) {
+ public long getQueueOffset(String topicQueueKey) {
+ return ConcurrentHashMapUtils.computeIfAbsent(this.topicQueueTable, topicQueueKey, k -> 0L);
+ }
+
+ public void increaseQueueOffset(String topicQueueKey, short messageNum) {
Long queueOffset = ConcurrentHashMapUtils.computeIfAbsent(this.topicQueueTable, topicQueueKey, k -> 0L);
- this.topicQueueTable.put(topicQueueKey, queueOffset + messageNum);
- return queueOffset;
+ topicQueueTable.put(topicQueueKey, queueOffset + messageNum);
Review Comment:
The current operation is all in a lock
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] codecov-commenter commented on pull request #6641: [ISSUE #6570] Fix the issue that expectLogicOffset is greater than currentLogicOffset in consumeQueue build when the message is illegal
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #6641:
URL: https://github.com/apache/rocketmq/pull/6641#issuecomment-1519361500
## [Codecov](https://codecov.io/gh/apache/rocketmq/pull/6641?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#6641](https://codecov.io/gh/apache/rocketmq/pull/6641?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (4ed3faf) into [develop](https://codecov.io/gh/apache/rocketmq/commit/14ad16b0d5e5ac770cf00660d8570d74a0a8a150?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (14ad16b) will **decrease** coverage by `0.04%`.
> The diff coverage is `77.46%`.
```diff
@@ Coverage Diff @@
## develop #6641 +/- ##
=============================================
- Coverage 43.13% 43.10% -0.04%
+ Complexity 9005 8994 -11
=============================================
Files 1107 1107
Lines 78297 78332 +35
Branches 10204 10212 +8
=============================================
- Hits 33776 33765 -11
- Misses 40302 40329 +27
- Partials 4219 4238 +19
```
| [Impacted Files](https://codecov.io/gh/apache/rocketmq/pull/6641?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...n/java/org/apache/rocketmq/store/MessageStore.java](https://codecov.io/gh/apache/rocketmq/pull/6641?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3JvY2tldG1xL3N0b3JlL01lc3NhZ2VTdG9yZS5qYXZh) | `0.00% <ø> (ø)` | |
| [...ketmq/store/plugin/AbstractPluginMessageStore.java](https://codecov.io/gh/apache/rocketmq/pull/6641?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3JvY2tldG1xL3N0b3JlL3BsdWdpbi9BYnN0cmFjdFBsdWdpbk1lc3NhZ2VTdG9yZS5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...main/java/org/apache/rocketmq/store/CommitLog.java](https://codecov.io/gh/apache/rocketmq/pull/6641?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3JvY2tldG1xL3N0b3JlL0NvbW1pdExvZy5qYXZh) | `64.97% <66.66%> (-0.27%)` | :arrow_down: |
| [...apache/rocketmq/store/queue/BatchConsumeQueue.java](https://codecov.io/gh/apache/rocketmq/pull/6641?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3JvY2tldG1xL3N0b3JlL3F1ZXVlL0JhdGNoQ29uc3VtZVF1ZXVlLmphdmE=) | `70.37% <75.00%> (+0.16%)` | :arrow_up: |
| [...n/java/org/apache/rocketmq/store/ConsumeQueue.java](https://codecov.io/gh/apache/rocketmq/pull/6641?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3JvY2tldG1xL3N0b3JlL0NvbnN1bWVRdWV1ZS5qYXZh) | `66.29% <76.19%> (+0.06%)` | :arrow_up: |
| [...org/apache/rocketmq/store/DefaultMessageStore.java](https://codecov.io/gh/apache/rocketmq/pull/6641?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3JvY2tldG1xL3N0b3JlL0RlZmF1bHRNZXNzYWdlU3RvcmUuamF2YQ==) | `46.80% <80.00%> (-0.06%)` | :arrow_down: |
| [...apache/rocketmq/store/queue/ConsumeQueueStore.java](https://codecov.io/gh/apache/rocketmq/pull/6641?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3JvY2tldG1xL3N0b3JlL3F1ZXVlL0NvbnN1bWVRdWV1ZVN0b3JlLmphdmE=) | `47.63% <83.33%> (+0.57%)` | :arrow_up: |
| [...ache/rocketmq/store/queue/QueueOffsetOperator.java](https://codecov.io/gh/apache/rocketmq/pull/6641?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3JvY2tldG1xL3N0b3JlL3F1ZXVlL1F1ZXVlT2Zmc2V0T3BlcmF0b3IuamF2YQ==) | `80.48% <91.66%> (ø)` | |
| [...pache/rocketmq/store/dledger/DLedgerCommitLog.java](https://codecov.io/gh/apache/rocketmq/pull/6641?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3JvY2tldG1xL3N0b3JlL2RsZWRnZXIvRExlZGdlckNvbW1pdExvZy5qYXZh) | `76.90% <100.00%> (+0.09%)` | :arrow_up: |
... and [22 files with indirect coverage changes](https://codecov.io/gh/apache/rocketmq/pull/6641/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org