You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@rocketmq.apache.org by "RongtongJin (via GitHub)" <gi...@apache.org> on 2026/06/13 01:58:40 UTC

[I] HATest.testSemiSyncReplica can fail before HA slave ack is visible on master [rocketmq]

RongtongJin opened a new issue, #10494:
URL: https://github.com/apache/rocketmq/issues/10494

   ### Problem
   `HATest.testSemiSyncReplica` can be flaky with:
   
   ```text
   expected:<PUT_OK> but was:<FLUSH_SLAVE_TIMEOUT>
   ```
   
   The test setup waits until the slave-side HA client enters `TRANSFER`, then immediately starts semi-sync writes. Entering `TRANSFER` only proves the slave has connected locally. The master-side `HAConnection` may not have received the slave's initial offset report yet, leaving `slaveAckOffset` at `-1` during the first synchronous replication request.
   
   ### Impact
   On slower or busy CI machines, the first `asyncPutMessage` can race the initial slave ack report and time out even though the HA connection is otherwise healthy.
   
   ### Proposed fix
   Make the test wait for the actual readiness condition needed by semi-sync replication: the master-side HA connection is in `TRANSFER` and its `slaveAckOffset` has caught up to the slave's current max physical offset before sending messages.
   
   ### Validation
   Ran locally with Maven 3.9.9:
   
   ```bash
   mvn -pl store -am -Dtest=HATest#testSemiSyncReplica -DskipITs -DfailIfNoTests=false test
   mvn -pl store -am -Dtest=HATest -DskipITs -DfailIfNoTests=false test
   ```
   
   The full `HATest` run reported `Tests run: 4, Failures: 0, Errors: 0, Skipped: 1`.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] HATest.testSemiSyncReplica can fail before HA slave ack is visible on master [rocketmq]

Posted by "RockteMQ-AI (via GitHub)" <gi...@apache.org>.
RockteMQ-AI commented on issue #10494:
URL: https://github.com/apache/rocketmq/issues/10494#issuecomment-4709868251

   **Issue Evaluation**
   
   Category: `type/bug` | Status: **Confirmed**
   
   Verified against the current codebase. The race condition is real:
   
   1. `setUp()` (line 106) waits only for the **slave-side** `HAClient` to enter `TRANSFER` state.
   2. `DefaultHAConnection.slaveAckOffset` (line 58) and `AutoSwitchHAConnection.slaveAckOffset` (line 84) are both initialized to `-1`.
   3. The master-side `slaveAckOffset` is only updated when the master receives the slave's first offset report (`DefaultHAConnection` line 230 / `AutoSwitchHAConnection` line 341).
   4. `testSemiSyncReplica()` immediately starts semi-sync writes (`totalReplicas=2, inSyncReplicas=2`), which require the slave to have acked up to the current offset.
   
   On slower CI machines, the first `asyncPutMessage` can race the initial slave ack report and return `FLUSH_SLAVE_TIMEOUT` instead of `PUT_OK`.
   
   **Root Cause:** Missing wait condition — test should verify master-side `slaveAckOffset >= 0` (or the slave's current max physical offset) before sending messages.
   **Impact:** Flaky CI failures in `HATest.testSemiSyncReplica`.
   **Severity:** Low — test-only issue, no production impact.
   
   The proposed fix is correct. A PR would be welcome.
   
   ---
   *Automated evaluation by RockteMQ-AI*


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] HATest.testSemiSyncReplica can fail before HA slave ack is visible on master [rocketmq]

Posted by "RockteMQ-AI (via GitHub)" <gi...@apache.org>.
RockteMQ-AI commented on issue #10494:
URL: https://github.com/apache/rocketmq/issues/10494#issuecomment-4711391263

   **Issue Evaluation**
   
   Category: `type/bug` | Status: **Confirmed**
   
   `HATest.testSemiSyncReplica` is a flaky test due to a race condition between the test setup and the HA connection readiness.
   
   **Root Cause:** The test waits for the slave-side HA client to enter `TRANSFER` state, but this only confirms the slave has connected locally. The master-side `HAConnection` may not have received the slave's initial offset report yet, leaving `slaveAckOffset` at `-1`. When the first semi-sync write arrives, it times out because the master thinks the slave is behind.
   
   **Impact:** Intermittent CI failures (`FLUSH_SLAVE_TIMEOUT`), particularly on slower CI machines. The test is correct in intent but has an insufficient readiness condition.
   
   **Severity:** Medium — test reliability issue, not a production bug.
   
   **Proposed fix:** Wait for master-side `slaveAckOffset >= slave's current max physical offset` before sending messages. This ensures the readiness condition matches what semi-sync replication actually requires.
   
   ---
   *Automated evaluation by RockteMQ-AI*


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] HATest.testSemiSyncReplica can fail before HA slave ack is visible on master [rocketmq]

Posted by "RockteMQ-AI (via GitHub)" <gi...@apache.org>.
RockteMQ-AI commented on issue #10494:
URL: https://github.com/apache/rocketmq/issues/10494#issuecomment-4704441951

   **Issue Evaluation**
   
   Category: `type/bug` | Status: **Confirmed**
   
   Verified against the current test code. In `HATest.testSemiSyncReplica()` (line 134), after `asyncPutMessage` returns `PUT_OK` in semi-sync mode, the test immediately reads from the slave store via `lookMessageByOffset`. However, `PUT_OK` only confirms that the data has been replicated to the slave's commitLog — the slave's `ConsumeQueue`/`Index` dispatch may not have completed yet.
   
   **Root Cause:** Race condition between semi-sync replication ack and slave dispatch lifecycle. The test lacks synchronization to ensure the slave has fully dispatched the message before reading.
   **Impact:** Intermittent CI failures on `HATest.testSemiSyncReplica`.
   **Severity:** medium — test reliability, not production behavior.
   
   An automated fix proposal will be generated. Reply `/approve` to proceed with PR generation.
   
   ---
   *Automated evaluation by github-manager-bot*


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] HATest.testSemiSyncReplica can fail before HA slave ack is visible on master [rocketmq]

Posted by "yx9o (via GitHub)" <gi...@apache.org>.
yx9o closed issue #10494: HATest.testSemiSyncReplica can fail before HA slave ack is visible on master
URL: https://github.com/apache/rocketmq/issues/10494


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org