You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@rocketmq.apache.org by "bxfjb (via GitHub)" <gi...@apache.org> on 2023/12/01 03:12:54 UTC
[PR] fix dledger recover abnormally may lost consume queue of tail [rocketmq]
bxfjb opened a new pull request, #7599:
URL: https://github.com/apache/rocketmq/pull/7599
<!-- Please make sure the target branch is right. In most case, the target branch should be `develop`. -->
### Which Issue(s) This PR Fixes
<!-- Please ensure that the related issue has already been created, and [link this pull request to that issue using keywords](<https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword>) to ensure automatic closure. -->
Fixes #7355
### Brief Description
<!-- Write a brief description for your pull request to help the maintainer understand the reasons behind your changes. -->
rocketmq in dledger mode will recover correctly from an abnormal downtime
### How Did You Test This Change?
<!-- In order to ensure the code quality of Apache RocketMQ, we expect every pull request to have undergone thorough testing. -->
Run DLedgerCommitlogTest.testDLedgerAbnormallyRecover()
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix dledger recover abnormally may lost consume queue of tail [rocketmq]
Posted by "yuz10 (via GitHub)" <gi...@apache.org>.
yuz10 commented on code in PR #7599:
URL: https://github.com/apache/rocketmq/pull/7599#discussion_r1453601842
##########
store/src/main/java/org/apache/rocketmq/store/dledger/DLedgerCommitLog.java:
##########
@@ -343,14 +427,48 @@ private void recover(long maxPhyOffsetOfConsumeQueue) throws RocksDBException {
log.info("Will set the initial commitlog offset={} for dledger", dividedCommitlogOffset);
}
+ private boolean isMmapFileMatchedRecover(final MmapFile mmapFile) {
+ ByteBuffer byteBuffer = mmapFile.sliceByteBuffer();
+
+ int magicCode = byteBuffer.getInt(DLedgerEntry.BODY_OFFSET + MessageDecoder.MESSAGE_MAGIC_CODE_POSITION);
+ if (magicCode != MESSAGE_MAGIC_CODE) {
+ return false;
+ }
+
+ long storeTimestamp = byteBuffer.getLong(DLedgerEntry.BODY_OFFSET + MessageDecoder.MESSAGE_STORE_TIMESTAMP_POSITION);
Review Comment:
if bortHost is ipv6, the offset should be 12 bytes larger
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix dledger recover abnormally may lost consume queue of tail [rocketmq]
Posted by "yuz10 (via GitHub)" <gi...@apache.org>.
yuz10 commented on code in PR #7599:
URL: https://github.com/apache/rocketmq/pull/7599#discussion_r1456981076
##########
store/src/main/java/org/apache/rocketmq/store/dledger/DLedgerCommitLog.java:
##########
@@ -343,14 +427,57 @@ private void recover(long maxPhyOffsetOfConsumeQueue) throws RocksDBException {
log.info("Will set the initial commitlog offset={} for dledger", dividedCommitlogOffset);
}
+ private boolean isMmapFileMatchedRecover(final MmapFile mmapFile) {
+ ByteBuffer byteBuffer = mmapFile.sliceByteBuffer();
+
+ int magicCode = byteBuffer.getInt(DLedgerEntry.BODY_OFFSET + MessageDecoder.MESSAGE_MAGIC_CODE_POSITION);
+ if (magicCode != MESSAGE_MAGIC_CODE) {
+ return false;
+ }
+
+ int storeTimestampPosition;
+ int sysFlag = byteBuffer.getInt(MessageDecoder.SYSFLAG_POSITION);
Review Comment:
DLedgerEntry.BODY_OFFSET + MessageDecoder.SYSFLAG_POSITIO
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix dledger recover abnormally may lost consume queue of tail [rocketmq]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #7599:
URL: https://github.com/apache/rocketmq/pull/7599#issuecomment-1897937916
## [Codecov](https://app.codecov.io/gh/apache/rocketmq/pull/7599?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
Attention: `27 lines` in your changes are missing coverage. Please review.
> Comparison is base [(`65faea2`)](https://app.codecov.io/gh/apache/rocketmq/commit/65faea22fd54fd9875f2ca9d3088b4dc46d31cce?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 43.19% compared to head [(`72522ad`)](https://app.codecov.io/gh/apache/rocketmq/pull/7599?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 43.18%.
> Report is 38 commits behind head on develop.
| [Files](https://app.codecov.io/gh/apache/rocketmq/pull/7599?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...pache/rocketmq/store/dledger/DLedgerCommitLog.java](https://app.codecov.io/gh/apache/rocketmq/pull/7599?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3JvY2tldG1xL3N0b3JlL2RsZWRnZXIvRExlZGdlckNvbW1pdExvZy5qYXZh) | 67.46% | [16 Missing and 11 partials :warning: ](https://app.codecov.io/gh/apache/rocketmq/pull/7599?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## develop #7599 +/- ##
=============================================
- Coverage 43.19% 43.18% -0.02%
- Complexity 9782 9828 +46
=============================================
Files 1162 1167 +5
Lines 84378 84690 +312
Branches 10955 10993 +38
=============================================
+ Hits 36451 36572 +121
- Misses 43397 43570 +173
- Partials 4530 4548 +18
```
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/rocketmq/pull/7599?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix dledger recover abnormally may lost consume queue of tail [rocketmq]
Posted by "bxfjb (via GitHub)" <gi...@apache.org>.
bxfjb commented on code in PR #7599:
URL: https://github.com/apache/rocketmq/pull/7599#discussion_r1456989735
##########
store/src/main/java/org/apache/rocketmq/store/dledger/DLedgerCommitLog.java:
##########
@@ -343,14 +427,57 @@ private void recover(long maxPhyOffsetOfConsumeQueue) throws RocksDBException {
log.info("Will set the initial commitlog offset={} for dledger", dividedCommitlogOffset);
}
+ private boolean isMmapFileMatchedRecover(final MmapFile mmapFile) {
+ ByteBuffer byteBuffer = mmapFile.sliceByteBuffer();
+
+ int magicCode = byteBuffer.getInt(DLedgerEntry.BODY_OFFSET + MessageDecoder.MESSAGE_MAGIC_CODE_POSITION);
+ if (magicCode != MESSAGE_MAGIC_CODE) {
+ return false;
+ }
+
+ int storeTimestampPosition;
+ int sysFlag = byteBuffer.getInt(MessageDecoder.SYSFLAG_POSITION);
Review Comment:
fixed and thx
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix dledger recover abnormally may lost consume queue of tail [rocketmq]
Posted by "yuz10 (via GitHub)" <gi...@apache.org>.
yuz10 commented on code in PR #7599:
URL: https://github.com/apache/rocketmq/pull/7599#discussion_r1453601842
##########
store/src/main/java/org/apache/rocketmq/store/dledger/DLedgerCommitLog.java:
##########
@@ -343,14 +427,48 @@ private void recover(long maxPhyOffsetOfConsumeQueue) throws RocksDBException {
log.info("Will set the initial commitlog offset={} for dledger", dividedCommitlogOffset);
}
+ private boolean isMmapFileMatchedRecover(final MmapFile mmapFile) {
+ ByteBuffer byteBuffer = mmapFile.sliceByteBuffer();
+
+ int magicCode = byteBuffer.getInt(DLedgerEntry.BODY_OFFSET + MessageDecoder.MESSAGE_MAGIC_CODE_POSITION);
+ if (magicCode != MESSAGE_MAGIC_CODE) {
+ return false;
+ }
+
+ long storeTimestamp = byteBuffer.getLong(DLedgerEntry.BODY_OFFSET + MessageDecoder.MESSAGE_STORE_TIMESTAMP_POSITION);
Review Comment:
if bortHost is ipv6, the offset should be 12 bytes larger
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix dledger recover abnormally may lost consume queue of tail [rocketmq]
Posted by "bxfjb (via GitHub)" <gi...@apache.org>.
bxfjb commented on PR #7599:
URL: https://github.com/apache/rocketmq/pull/7599#issuecomment-1835389559
@cserwen I opened this pr for #7355 , could you please review it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix dledger recover abnormally may lost consume queue of tail [rocketmq]
Posted by "bxfjb (via GitHub)" <gi...@apache.org>.
bxfjb commented on code in PR #7599:
URL: https://github.com/apache/rocketmq/pull/7599#discussion_r1454496926
##########
store/src/main/java/org/apache/rocketmq/store/dledger/DLedgerCommitLog.java:
##########
@@ -343,14 +427,48 @@ private void recover(long maxPhyOffsetOfConsumeQueue) throws RocksDBException {
log.info("Will set the initial commitlog offset={} for dledger", dividedCommitlogOffset);
}
+ private boolean isMmapFileMatchedRecover(final MmapFile mmapFile) {
+ ByteBuffer byteBuffer = mmapFile.sliceByteBuffer();
+
+ int magicCode = byteBuffer.getInt(DLedgerEntry.BODY_OFFSET + MessageDecoder.MESSAGE_MAGIC_CODE_POSITION);
+ if (magicCode != MESSAGE_MAGIC_CODE) {
+ return false;
+ }
+
+ long storeTimestamp = byteBuffer.getLong(DLedgerEntry.BODY_OFFSET + MessageDecoder.MESSAGE_STORE_TIMESTAMP_POSITION);
Review Comment:
thanks a lot and fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix dledger recover abnormally may lost consume queue of tail [rocketmq]
Posted by "yuz10 (via GitHub)" <gi...@apache.org>.
yuz10 merged PR #7599:
URL: https://github.com/apache/rocketmq/pull/7599
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org