You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@rocketmq.apache.org by "RongtongJin (via GitHub)" <gi...@apache.org> on 2026/08/16 01:51:30 UTC

[PR] [ISSUE #10651] Adapt RocketMQ to latest DLedger and remove fastjson1 [rocketmq]

RongtongJin opened a new pull request, #10947:
URL: https://github.com/apache/rocketmq/pull/10947

   ### Which Issue(s) This PR Fixes
   
   - Fixes #10651
   
   ### Brief Description
   
   This PR reworks the DLedger 0.3.3 adaptation on top of the latest DLedger code and [openmessaging/dledger#336](https://github.com/openmessaging/dledger/pull/336). It is intended to supersede the incomplete adaptation in #10650.
   
   The change:
   
   - adapts RocketMQ broker, store, and controller code to the latest DLedger APIs;
   - restores immediate append positions required by RocketMQ through DLedger PR #336;
   - derives the committed physical boundary from DLedger index metadata and prevents reads from exposing an uncommitted tail;
   - enables fast commit-index advancement and handles DLedger NOOP/control entries safely during append, reput, and abnormal recovery;
   - handles latest DLedger's ordinary `AppendFuture` result for a single-message batch and `BatchAppendFuture` for multi-message batches;
   - removes RocketMQ's fastjson1 dependency and aligns Maven/Bazel on fastjson2 2.0.64;
   - adds JDK 8 cold-start compatibility fixtures, focused store regressions, and a real three-broker failover/full-restart integration test.
   
   The validated compatibility boundary is a coordinated full-stop upgrade. Mixed-version rolling upgrade and downgrade are not covered by this PR.
   
   > [!IMPORTANT]
   > This is intentionally a draft. The current DLedger coordinate, `0.3.3-pr336-f2-64-SNAPSHOT`, is a locally built artifact from DLedger master + PR #336 + fastjson2 2.0.64. Before this PR can be merged, DLedger #336 must be merged and a publicly resolvable DLedger artifact must be published; this PR must then switch to that coordinate and rerun the matrix.
   
   Related work: #10650, #10928, [openmessaging/dledger#336](https://github.com/openmessaging/dledger/pull/336).
   
   ### How Did You Test This Change?
   
   All Java verification used Amazon Corretto JDK 8.
   
   - DLedger PR #336 regression: 6/6 tests passed; full DLedger reactor passed; RPC cold-start/network harness 3/3 passed.
   - RocketMQ focused Maven tests:
     - remoting compatibility: 5/5;
     - DLedger store regression suite: 8/8;
     - controller: 7/7;
     - broker DLedger integration tests: 2/2;
     - broker container DLedger lifecycle: 1/1.
   - RocketMQ full Maven `clean test`: 19/19 modules, 3,124 tests, 0 failures, 0 errors, 42 skipped.
   - Bazel affected build: 5 targets, 203 actions.
   - Bazel tests: remoting 5/5, store 8/8, controller 1/1, single-broker IT 1/1, three-broker IT 1/1.
   - Full-stop data compatibility fixture: DLedger 0.3.2 wrote 10 messages; latest code read all 10 before the first new append, then wrote/read message 11.
   - Final dependency/bytecode audit: 7 packaged JARs and 1,360 classes scanned; no fastjson1 dependency, JAR entry, or class constant remained.
   - `git diff --check` passed.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [ISSUE #10651] Adapt RocketMQ to latest DLedger and remove fastjson1 [rocketmq]

Posted by "RockteMQ-AI (via GitHub)" <gi...@apache.org>.
RockteMQ-AI commented on PR #10947:
URL: https://github.com/apache/rocketmq/pull/10947#issuecomment-5339589176

   ⚠️ **Merge conflict detected**
   
   This PR has conflicts with the `develop` branch and cannot be merged. Please rebase or merge `develop` into your branch and resolve the conflicts:
   
   ```bash
   git fetch origin
   git checkout codex/dledger-latest-pr336-adapter
   git rebase origin/develop
   # resolve conflicts, then:
   git push --force-with-lease
   ```
   
   This is a one-time reminder. Feel free to @mention me for a re-review after conflicts are resolved.
   
   ---
   *Automated notification by "github-manager-bot"*


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [ISSUE #10651] Adapt RocketMQ to latest DLedger and remove fastjson1 [rocketmq]

Posted by "RockteMQ-AI (via GitHub)" <gi...@apache.org>.
RockteMQ-AI commented on code in PR #10947:
URL: https://github.com/apache/rocketmq/pull/10947#discussion_r3817790719


##########
store/src/main/java/org/apache/rocketmq/store/DefaultMessageStore.java:
##########
@@ -2741,16 +2741,19 @@ public void doReput() {
 
                         if (dispatchRequest.isSuccess()) {
                             if (size > 0) {
-                                currentReputTimestamp = dispatchRequest.getStoreTimestamp();
-                                DefaultMessageStore.this.doDispatch(dispatchRequest);
+                                if (dispatchRequest.getMsgSize() > 0) {

Review Comment:
   The msgSize > 0 guard in doReput correctly filters out NOOP entries from dispatch. This is important for the DLedger 0.4.x compatibility where NOOP entries are used for heartbeats. Good fix.



##########
store/src/main/java/org/apache/rocketmq/store/dledger/DLedgerCommitLog.java:
##########
@@ -69,6 +72,8 @@ public class DLedgerCommitLog extends CommitLog {
     private final DLedgerConfig dLedgerConfig;
     private final DLedgerMmapFileStore dLedgerFileStore;
     private final MmapFileList dLedgerFileList;
+    private volatile long cachedCommittedIndex = Long.MIN_VALUE;

Review Comment:
   The volatile cachedCommittedIndex/cachedCommittedPos fields are a good optimization to avoid repeated DLedger state lookups. Consider documenting the cache invalidation contract (when these are updated vs when they become stale).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org