You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@rocketmq.apache.org by "Koado (via GitHub)" <gi...@apache.org> on 2023/07/25 13:03:08 UTC
[GitHub] [rocketmq] Koado opened a new pull request, #7082: [Feature] Allow a BoundaryType to be specified when retrieving offset based on the timestamp
Koado opened a new pull request, #7082:
URL: https://github.com/apache/rocketmq/pull/7082
<!-- Please make sure the target branch is right. In most case, the target branch should be `develop`. -->
### Which Issue(s) This PR Fixes
<!-- Please ensure that the related issue has already been created, and [link this pull request to that issue using keywords](<https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword>) to ensure automatic closure. -->
[Feature] Allow a BoundaryType to be specified when retrieving offset based on the timestamp #7074
### Brief Description
<!-- Write a brief description for your pull request to help the maintainer understand the reasons behind your changes. -->
Add two interfaces to the DefaultMQAdminExt class for the reason that messages will miss if there are multiple messages with the same storeTime(endTime) when retrieving messages over a period of time.
1. searchLowerBoundaryOffset
2. searchUpperBoundaryOffset
### How Did You Test This Change?
<!-- In order to ensure the code quality of Apache RocketMQ, we expect every pull request to have undergone thorough testing. -->
I wrote a test case to determine whether the maximum offset of the queue is equal to searchUpperBoundaryOffset(queue's maxTimestamp), and the minimum offset of the queue is equal to searchLowerBoundaryOffset(queue's minTimestamp). The codes are below:
```java
String namesrvAddr = "127.0.0.1:9876";
String topic = "test-topic";
DefaultMQAdminExt mqAdminExt = new DefaultMQAdminExt();
mqAdminExt.setInstanceName(UUID.randomUUID().toString());
mqAdminExt.setNamesrvAddr(namesrvAddr);
mqAdminExt.start();
List<QueueTimeSpan> timeSpanList = mqAdminExt.queryConsumeTimeSpan(topic, null);
if (timeSpanList != null && timeSpanList.size() > 0) {
for (QueueTimeSpan timeSpan: timeSpanList) {
MessageQueue mq = timeSpan.getMessageQueue();
long maxOffset = mqAdminExt.maxOffset(mq);
long minOffset = mqAdminExt.minOffset(mq);
// if there is at least one message in queue, the maxOffset returns the queue's latest offset + 1
assertThat((maxOffset == 0 ? 0 : maxOffset - 1) == mqAdminExt.searchUpperBoundaryOffset(mq, timeSpan.getMaxTimeStamp())).isTrue();
assertThat(minOffset == mqAdminExt.searchLowerBoundaryOffset(mq, timeSpan.getMinTimeStamp())).isTrue();
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] caigy commented on pull request #7082: [ISSUE #7074] Allow a BoundaryType to be specified when retrieving offset based on the timestamp
Posted by "caigy (via GitHub)" <gi...@apache.org>.
caigy commented on PR #7082:
URL: https://github.com/apache/rocketmq/pull/7082#issuecomment-1653425394
Please fix failed tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] Koado commented on a diff in pull request #7082: [ISSUE #7074] Allow a BoundaryType to be specified when retrieving offset based on the timestamp
Posted by "Koado (via GitHub)" <gi...@apache.org>.
Koado commented on code in PR #7082:
URL: https://github.com/apache/rocketmq/pull/7082#discussion_r1278309596
##########
broker/src/main/java/org/apache/rocketmq/broker/processor/AdminBrokerProcessor.java:
##########
@@ -988,7 +988,15 @@ private RemotingCommand rewriteRequestForStaticTopic(SearchOffsetRequestHeader r
continue;
}
if (mappingDetail.getBname().equals(item.getBname())) {
- offset = this.brokerController.getMessageStore().getOffsetInQueueByTime(mappingContext.getTopic(), item.getQueueId(), timestamp);
+ MessageStore messageStore = this.brokerController.getMessageStore();
+ if (messageStore instanceof DefaultMessageStore) {
+ // get offset with specific boundary type
+ offset = ((DefaultMessageStore) messageStore).getOffsetInQueueByTime(requestHeader.getTopic(),
+ requestHeader.getQueueId(), requestHeader.getTimestamp(), requestHeader.getBoundaryType());
+ } else {
+ offset = messageStore.getOffsetInQueueByTime(requestHeader.getTopic(), requestHeader.getQueueId(),
+ requestHeader.getTimestamp());
+ }
Review Comment:
Yes, I also wanted to add an interface directly at the beginning. But later I found that the **BoundaryType** used in **TieredMessageStore** belongs to package _org.apache.rocketmq.tieredstore.common_ and the **BoundaryType** used in **DefaultMessageStore** belongs to package _org.apache.rocketmq.common_. So if I wanna to abstract an interface, the argument of the interface might be boundaryTypeName(String), which means that the enum class is converted to a string, or the string is converted to the enum class, each time the argument is passed and the argument is fetched.
Or maybe I should unify the two enum classes?What do you think I should do in this situation?
##########
broker/src/main/java/org/apache/rocketmq/broker/processor/AdminBrokerProcessor.java:
##########
@@ -988,7 +988,15 @@ private RemotingCommand rewriteRequestForStaticTopic(SearchOffsetRequestHeader r
continue;
}
if (mappingDetail.getBname().equals(item.getBname())) {
- offset = this.brokerController.getMessageStore().getOffsetInQueueByTime(mappingContext.getTopic(), item.getQueueId(), timestamp);
+ MessageStore messageStore = this.brokerController.getMessageStore();
+ if (messageStore instanceof DefaultMessageStore) {
+ // get offset with specific boundary type
+ offset = ((DefaultMessageStore) messageStore).getOffsetInQueueByTime(requestHeader.getTopic(),
+ requestHeader.getQueueId(), requestHeader.getTimestamp(), requestHeader.getBoundaryType());
+ } else {
+ offset = messageStore.getOffsetInQueueByTime(requestHeader.getTopic(), requestHeader.getQueueId(),
+ requestHeader.getTimestamp());
+ }
Review Comment:
Yes, I also wanted to add an interface directly at the beginning. But later I found that the **BoundaryType** used in **TieredMessageStore** belongs to package _org.apache.rocketmq.tieredstore.common_ and the **BoundaryType** used in **DefaultMessageStore** belongs to package _org.apache.rocketmq.common_. So if I wanna to abstract an interface, the argument of the interface might be boundaryTypeName(String), which means that the enum class is converted to a string, or the string is converted to the enum class, each time the argument is passed and the argument is fetched.
Or maybe I should unify the two enum classes?What do you think I should do in this situation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] Koado commented on a diff in pull request #7082: [ISSUE #7074] Allow a BoundaryType to be specified when retrieving offset based on the timestamp
Posted by "Koado (via GitHub)" <gi...@apache.org>.
Koado commented on code in PR #7082:
URL: https://github.com/apache/rocketmq/pull/7082#discussion_r1276990031
##########
tools/src/test/java/org/apache/rocketmq/tools/admin/DefaultMQAdminExtTest.java:
##########
@@ -512,6 +513,30 @@ public void testSearchOffset() throws Exception {
assertThat(defaultMQAdminExt.searchOffset(new MessageQueue(TOPIC1, BROKER1_NAME, 0), System.currentTimeMillis())).isEqualTo(101L);
}
+ @Test
+ public void testSearchOffsetWithSpecificBoundaryType() throws Exception {
+
+ String namesrvAddr = "127.0.0.1:9876";
+ String topic = "test-topic";
+
+ DefaultMQAdminExt mqAdminExt = new DefaultMQAdminExt();
+ mqAdminExt.setInstanceName(UUID.randomUUID().toString());
+ mqAdminExt.setNamesrvAddr(namesrvAddr);
+
+ mqAdminExt.start();
+ List<QueueTimeSpan> timeSpanList = mqAdminExt.queryConsumeTimeSpan(topic, null);
+ if (timeSpanList != null && timeSpanList.size() > 0) {
+ for (QueueTimeSpan timeSpan: timeSpanList) {
+ MessageQueue mq = timeSpan.getMessageQueue();
+ long maxOffset = mqAdminExt.maxOffset(mq);
+ long minOffset = mqAdminExt.minOffset(mq);
+ // if there is at least one message in queue, the maxOffset returns the queue's latest offset + 1
+ assertThat((maxOffset == 0 ? 0 : maxOffset - 1) == mqAdminExt.searchUpperBoundaryOffset(mq, timeSpan.getMaxTimeStamp())).isTrue();
+ assertThat(minOffset == mqAdminExt.searchLowerBoundaryOffset(mq, timeSpan.getMinTimeStamp())).isTrue();
+ }
+ }
+ }
Review Comment:
already replace it with mock, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] RongtongJin commented on a diff in pull request #7082: [ISSUE #7074] Allow a BoundaryType to be specified when retrieving offset based on the timestamp
Posted by "RongtongJin (via GitHub)" <gi...@apache.org>.
RongtongJin commented on code in PR #7082:
URL: https://github.com/apache/rocketmq/pull/7082#discussion_r1278280501
##########
broker/src/main/java/org/apache/rocketmq/broker/processor/AdminBrokerProcessor.java:
##########
@@ -988,7 +988,15 @@ private RemotingCommand rewriteRequestForStaticTopic(SearchOffsetRequestHeader r
continue;
}
if (mappingDetail.getBname().equals(item.getBname())) {
- offset = this.brokerController.getMessageStore().getOffsetInQueueByTime(mappingContext.getTopic(), item.getQueueId(), timestamp);
+ MessageStore messageStore = this.brokerController.getMessageStore();
+ if (messageStore instanceof DefaultMessageStore) {
+ // get offset with specific boundary type
+ offset = ((DefaultMessageStore) messageStore).getOffsetInQueueByTime(requestHeader.getTopic(),
+ requestHeader.getQueueId(), requestHeader.getTimestamp(), requestHeader.getBoundaryType());
+ } else {
+ offset = messageStore.getOffsetInQueueByTime(requestHeader.getTopic(), requestHeader.getQueueId(),
+ requestHeader.getTimestamp());
+ }
Review Comment:
I found that getOffsetInQueueByTime also exists in TieredMessageStore. Would it be better to abstract an interface instead of checking the type every time?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] RongtongJin commented on a diff in pull request #7082: [ISSUE #7074] Allow a BoundaryType to be specified when retrieving offset based on the timestamp
Posted by "RongtongJin (via GitHub)" <gi...@apache.org>.
RongtongJin commented on code in PR #7082:
URL: https://github.com/apache/rocketmq/pull/7082#discussion_r1274491231
##########
tools/src/test/java/org/apache/rocketmq/tools/admin/DefaultMQAdminExtTest.java:
##########
@@ -512,6 +513,30 @@ public void testSearchOffset() throws Exception {
assertThat(defaultMQAdminExt.searchOffset(new MessageQueue(TOPIC1, BROKER1_NAME, 0), System.currentTimeMillis())).isEqualTo(101L);
}
+ @Test
+ public void testSearchOffsetWithSpecificBoundaryType() throws Exception {
+
+ String namesrvAddr = "127.0.0.1:9876";
+ String topic = "test-topic";
+
+ DefaultMQAdminExt mqAdminExt = new DefaultMQAdminExt();
+ mqAdminExt.setInstanceName(UUID.randomUUID().toString());
+ mqAdminExt.setNamesrvAddr(namesrvAddr);
+
+ mqAdminExt.start();
+ List<QueueTimeSpan> timeSpanList = mqAdminExt.queryConsumeTimeSpan(topic, null);
+ if (timeSpanList != null && timeSpanList.size() > 0) {
+ for (QueueTimeSpan timeSpan: timeSpanList) {
+ MessageQueue mq = timeSpan.getMessageQueue();
+ long maxOffset = mqAdminExt.maxOffset(mq);
+ long minOffset = mqAdminExt.minOffset(mq);
+ // if there is at least one message in queue, the maxOffset returns the queue's latest offset + 1
+ assertThat((maxOffset == 0 ? 0 : maxOffset - 1) == mqAdminExt.searchUpperBoundaryOffset(mq, timeSpan.getMaxTimeStamp())).isTrue();
+ assertThat(minOffset == mqAdminExt.searchLowerBoundaryOffset(mq, timeSpan.getMinTimeStamp())).isTrue();
+ }
+ }
+ }
Review Comment:
Network connections or database connections in unit testing need to be replaced with mock or spy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] RongtongJin commented on a diff in pull request #7082: [ISSUE #7074] Allow a BoundaryType to be specified when retrieving offset based on the timestamp
Posted by "RongtongJin (via GitHub)" <gi...@apache.org>.
RongtongJin commented on code in PR #7082:
URL: https://github.com/apache/rocketmq/pull/7082#discussion_r1278516420
##########
broker/src/main/java/org/apache/rocketmq/broker/processor/AdminBrokerProcessor.java:
##########
@@ -988,7 +988,15 @@ private RemotingCommand rewriteRequestForStaticTopic(SearchOffsetRequestHeader r
continue;
}
if (mappingDetail.getBname().equals(item.getBname())) {
- offset = this.brokerController.getMessageStore().getOffsetInQueueByTime(mappingContext.getTopic(), item.getQueueId(), timestamp);
+ MessageStore messageStore = this.brokerController.getMessageStore();
+ if (messageStore instanceof DefaultMessageStore) {
+ // get offset with specific boundary type
+ offset = ((DefaultMessageStore) messageStore).getOffsetInQueueByTime(requestHeader.getTopic(),
+ requestHeader.getQueueId(), requestHeader.getTimestamp(), requestHeader.getBoundaryType());
+ } else {
+ offset = messageStore.getOffsetInQueueByTime(requestHeader.getTopic(), requestHeader.getQueueId(),
+ requestHeader.getTimestamp());
+ }
Review Comment:
I think it would be better to unify the two enumeration classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] codecov-commenter commented on pull request #7082: [ISSUE #7074] Allow a BoundaryType to be specified when retrieving offset based on the timestamp
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #7082:
URL: https://github.com/apache/rocketmq/pull/7082#issuecomment-1654872012
## [Codecov](https://app.codecov.io/gh/apache/rocketmq/pull/7082?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
> Merging [#7082](https://app.codecov.io/gh/apache/rocketmq/pull/7082?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (8e6230f) into [develop](https://app.codecov.io/gh/apache/rocketmq/commit/90c5382aee07879a80309f257f04114201ccaac6?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (90c5382) will **decrease** coverage by `0.03%`.
> Report is 7 commits behind head on develop.
> The diff coverage is `28.12%`.
```diff
@@ Coverage Diff @@
## develop #7082 +/- ##
=============================================
- Coverage 42.72% 42.69% -0.03%
- Complexity 9282 9290 +8
=============================================
Files 1138 1138
Lines 81146 81282 +136
Branches 10619 10641 +22
=============================================
+ Hits 34672 34707 +35
- Misses 42137 42228 +91
- Partials 4337 4347 +10
```
| [Files Changed](https://app.codecov.io/gh/apache/rocketmq/pull/7082?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...a/org/apache/rocketmq/client/impl/MQAdminImpl.java](https://app.codecov.io/gh/apache/rocketmq/pull/7082?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y2xpZW50L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9jbGllbnQvaW1wbC9NUUFkbWluSW1wbC5qYXZh) | `5.02% <0.00%> (-0.03%)` | :arrow_down: |
| [...g/apache/rocketmq/client/impl/MQClientAPIImpl.java](https://app.codecov.io/gh/apache/rocketmq/pull/7082?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y2xpZW50L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9jbGllbnQvaW1wbC9NUUNsaWVudEFQSUltcGwuamF2YQ==) | `23.52% <0.00%> (-0.04%)` | :arrow_down: |
| [...ing/protocol/header/SearchOffsetRequestHeader.java](https://app.codecov.io/gh/apache/rocketmq/pull/7082?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cmVtb3Rpbmcvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3JvY2tldG1xL3JlbW90aW5nL3Byb3RvY29sL2hlYWRlci9TZWFyY2hPZmZzZXRSZXF1ZXN0SGVhZGVyLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...apache/rocketmq/tools/admin/DefaultMQAdminExt.java](https://app.codecov.io/gh/apache/rocketmq/pull/7082?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-dG9vbHMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3JvY2tldG1xL3Rvb2xzL2FkbWluL0RlZmF1bHRNUUFkbWluRXh0LmphdmE=) | `28.96% <0.00%> (-0.33%)` | :arrow_down: |
| [...he/rocketmq/tools/admin/DefaultMQAdminExtImpl.java](https://app.codecov.io/gh/apache/rocketmq/pull/7082?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-dG9vbHMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3JvY2tldG1xL3Rvb2xzL2FkbWluL0RlZmF1bHRNUUFkbWluRXh0SW1wbC5qYXZh) | `24.91% <0.00%> (-0.03%)` | :arrow_down: |
| [...ocketmq/broker/processor/AdminBrokerProcessor.java](https://app.codecov.io/gh/apache/rocketmq/pull/7082?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-YnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9icm9rZXIvcHJvY2Vzc29yL0FkbWluQnJva2VyUHJvY2Vzc29yLmphdmE=) | `24.10% <23.07%> (-0.07%)` | :arrow_down: |
| [...he/rocketmq/remoting/protocol/RemotingCommand.java](https://app.codecov.io/gh/apache/rocketmq/pull/7082?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cmVtb3Rpbmcvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3JvY2tldG1xL3JlbW90aW5nL3Byb3RvY29sL1JlbW90aW5nQ29tbWFuZC5qYXZh) | `70.64% <33.33%> (-0.39%)` | :arrow_down: |
| [...org/apache/rocketmq/store/DefaultMessageStore.java](https://app.codecov.io/gh/apache/rocketmq/pull/7082?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3JvY2tldG1xL3N0b3JlL0RlZmF1bHRNZXNzYWdlU3RvcmUuamF2YQ==) | `47.04% <100.00%> (+0.13%)` | :arrow_up: |
... and [23 files with indirect coverage changes](https://app.codecov.io/gh/apache/rocketmq/pull/7082/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] RongtongJin merged pull request #7082: [ISSUE #7074] Allow a BoundaryType to be specified when retrieving offset based on the timestamp
Posted by "RongtongJin (via GitHub)" <gi...@apache.org>.
RongtongJin merged PR #7082:
URL: https://github.com/apache/rocketmq/pull/7082
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org