You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@rocketmq.apache.org by "btlqql (via GitHub)" <gi...@apache.org> on 2026/10/02 04:41:49 UTC

[PR] fix(proxy): retry the proxy remoting client after a failed start [rocketmq-dashboard]

btlqql opened a new pull request, #5352:
URL: https://github.com/apache/rocketmq-dashboard/pull/5352

   ### Which Issue(s) This PR Fixes
   
   - Fixes #5057
   
   ### Problem
   
   `ProxyConsumerResolver.remotingClient()` marked the shared Netty remoting client as started
   *before* calling `start()`, and it published the client object before the start had run at all:
   
   ```java
   private NettyRemotingClient remotingClient() {
       NettyRemotingClient client = remotingClient;
       if (client == null) {
           synchronized (this) {
               if (remotingClient == null) {
                   remotingClient = new NettyRemotingClient(new NettyClientConfig());
               }
               client = remotingClient;
           }
       }
       if (clientStarted.compareAndSet(false, true)) {
           client.start();
       }
       return client;
   }
   ```
   
   Two consequences follow from that order:
   
   - A first `start()` that throws leaves both the client object and `clientStarted=true` behind, so
     every later proxy query skips `start()` and keeps using the client that never finished
     initialising. Proxy consumer diagnostics stay "unavailable" for the whole process lifetime, even
     after the transient startup problem is gone.
   - A second caller can observe `clientStarted=true` while the first caller's `start()` is still
     running and pass that half-initialised client to `invokeSync`.
   
   `shutdownRemotingClient()` also keyed off the same flag, so a candidate whose start failed was
   neither published correctly nor disposed of.
   
   ### Changes
   
   `server/src/main/java/org/apache/rocketmq/studio/provider/apache/ProxyConsumerResolver.java`
   
   - The client is now published only after `start()` returned. The `AtomicBoolean clientStarted` flag
     is gone: being published *is* the started state, so there is no window in which a client is
     visible while its startup is still running.
   - Initialisation happens under the resolver monitor (lock-free read on the published field first),
     so concurrent first-use callers wait for the startup instead of racing it.
   - A `start()` that throws disposes of the candidate (`shutdownFailedCandidateQuietly`) and rethrows;
     nothing is published, so the next request creates and starts a fresh client.
   - `shutdownRemotingClient()` takes the same monitor, so shutdown cannot run while a startup is in
     flight, and it only ever shuts down a client that was actually started.
   
   `server/src/test/java/org/apache/rocketmq/studio/provider/apache/ProxyConsumerResolverTest.java`
   
   - `firstStartFailureShouldNotBeCachedAndTheNextQueryShouldStartAFreshClientTest` — the first client's
     `start()` throws, the failed candidate must be shut down and must not be reused, and the next
     query must create and start a replacement.
   - `concurrentFirstUseShouldWaitUntilTheRemotingClientStartupFinishedTest` — the second caller arrives
     while the first caller is still inside `start()`; it must not reach `invokeSync` before the
     startup finished, and only one client may be constructed.
   
   ### How Did You Test This Change?
   
   Red (test file only, production source unmodified at `6a68042f`):
   
   ```
   cd server && mvn -B -ntp test -Dtest=ProxyConsumerResolverTest
   [ERROR] Tests run: 8, Failures: 2, Errors: 0, Skipped: 0
   [ERROR]   ProxyConsumerResolverTest.firstStartFailureShouldNotBeCachedAndTheNextQueryShouldStartAFreshClientTest:214
             Wanted but not invoked: nettyRemotingClient.shutdown();
   [ERROR]   ProxyConsumerResolverTest.concurrentFirstUseShouldWaitUntilTheRemotingClientStartupFinishedTest:271
             expected: null but was: java.lang.IllegalStateException: proxy query reached the client before startup finished
   ```
   
   Green (same worktree with the fix applied):
   
   ```
   cd server && mvn -B -ntp test -Dtest=ProxyConsumerResolverTest,RocketMQConsumerDiagnosticsProviderTest,RocketMQMetadataProviderTest,RocketMQAdminClientImplTest
   [INFO] Tests run: 8,  Failures: 0, Errors: 0, Skipped: 0 -- in ...ProxyConsumerResolverTest
   [INFO] Tests run: 80, Failures: 0, Errors: 0, Skipped: 0 -- in ...RocketMQAdminClientImplTest
   [INFO] Tests run: 11, Failures: 0, Errors: 0, Skipped: 0 -- in ...RocketMQConsumerDiagnosticsProviderTest
   [INFO] Tests run: 46, Failures: 0, Errors: 0, Skipped: 0 -- in ...RocketMQMetadataProviderTest
   [INFO] Tests run: 145, Failures: 0, Errors: 0, Skipped: 0
   [INFO] BUILD SUCCESS
   [INFO] You have 0 Checkstyle violations.
   ```
   
   Both runs execute Checkstyle at `validate` with 0 violations. This is a focused backend run, not the
   full suite (the suite has pre-existing failures on this branch that need MySQL 8 / external tooling).
   
   ### Overlap disclosure
   
   An independent PR for the same issue already exists: #5255 (its predecessor #5058 was closed by the
   inactivity bot). This change is an independent implementation written from the issue and the current
   source; the overlapping part is the intent both share (publish only after a successful start and
   retry with a fresh client). The differences are: the started state here is the publication itself
   rather than a separate boolean flag, `shutdownRemotingClient()` is serialised with initialisation,
   and the two regressions drive the interleavings through Mockito construction mocking instead of a
   change to the production surface. The issue is still open and the fix is still absent from
   `rocketmq-studio`, which is what the red run above shows.
   
   ### Checklist
   
   - [x] One coherent bug fix; no unrelated changes
   - [x] Conventional Commit subject
   - [x] Startup-failure/retry and concurrent-first-use regressions
   - [x] No API contract change, no new UI text, no new files
   - [ ] Full CI is green
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(proxy): retry the proxy remoting client after a failed start [rocketmq-dashboard]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #5352: fix(proxy): retry the proxy remoting client after a failed start
URL: https://github.com/apache/rocketmq-dashboard/pull/5352


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(proxy): retry the proxy remoting client after a failed start [rocketmq-dashboard]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5352:
URL: https://github.com/apache/rocketmq-dashboard/pull/5352#issuecomment-6091539095

   This pull request has been automatically closed because there has been no activity for 7 days. If you would like to continue working on this, please reopen the PR with updated changes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(proxy): retry the proxy remoting client after a failed start [rocketmq-dashboard]

Posted by "lizhimins (via GitHub)" <gi...@apache.org>.
lizhimins commented on PR #5352:
URL: https://github.com/apache/rocketmq-dashboard/pull/5352#issuecomment-6094445658

   Confirmed against trunk: ProxyConsumerResolver.java:227-231 publishes the client inside the monitor
   and :233-235 CASes + start()s outside it, so a first start() that throws leaves a permanently
   unstarted client behind, and a concurrent first-use caller can hand that half-initialised client to
   invokeSync. Both regressions are genuinely mutation-sensitive (I could name the reverting line for
   each assertion), and every symbol you reference exists — including mockConstruction, which already
   has precedent here in MqClientPoolTest:37,80. Four items before this can land:
   1. shutdownRemotingClient() now contends on the same monitor that remotingClient() holds *across*
    NettyRemotingClient.start(). A hung startup will block Spring's @PreDestroy indefinitely (there
    is no timeout), so the context close stalls until SIGKILL. Use a bounded ReentrantLock.tryLock,
    or keep shutdown off the initialization lock.
   2. Failure amplification: resolveConsumerConnectionStatus:96-110 calls remotingClient() once per
    proxy address, and RocketMQMetadataProvider:359 calls it once per consumer group. With a
    persistent start() failure, one group-list request now performs (groups x proxies) full
    construct/start/shutdown Netty lifecycles, on every request, with no backoff or negative cache.
    The old code was broken but cheap. Please add a short backoff or one attempt per resolve.
   3. Narrow shutdownFailedCandidateQuietly's `catch (Exception)` to RuntimeException and chain the
    shutdown failure with addSuppressed on the rethrown cause instead of only logging it.
   4. Wrap the latch awaits in assertThat(...).isTrue() — as written, a latch that never fires does
    not fail fast and the test dies on an unrelated assertion later.
   Two smaller notes: `import org.mockito.MockedConstruction;` sits before `org.mockito.Mock;`, which
   checkstyle does not enforce here but does break this file's alphabetical style; and in the
   concurrency test the `events` assertion only holds because both invokeSync labels are the same
   string — the concurrent caller can legitimately win the monitor first, so please say so in a
   comment before someone differentiates those labels and turns it flaky.
   Also, one factual correction: on trunk :241 (`clientStarted.get() && remotingClient != null`) both
   are true after a failed start, so @PreDestroy does shut the candidate down; what is missing is
   *prompt* disposal, i.e. the Netty event loops stay allocated for the rest of the process lifetime.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org