You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "davidzollo (via GitHub)" <gi...@apache.org> on 2026/07/14 14:15:41 UTC

[PR] [Fix][Connector-V2] Prevent silent StarRocks batch loss [seatunnel]

davidzollo opened a new pull request, #11431:
URL: https://github.com/apache/seatunnel/pull/11431

   ### Purpose of this pull request
   
   StarRocks Stream Load failures could previously exhaust retries and still clear the in-memory batch. The sink also treated a reused label as a reason to skip the batch, even though StarRocks may have committed the original transaction. Both paths can silently lose records.
   
   This patch makes the sink fail closed:
   
   - Retain the pending rows and their Stream Load label across failed `flush()` calls.
   - Release buffered rows only after `Success`, `Publish Timeout`, `VISIBLE`, or `COMMITTED`.
   - Query the existing label before deciding the outcome of `Label Already Exists`.
   - Create a replacement label only when StarRocks explicitly reports `ABORTED`.
   - Bound `PREPARE` polling to three minutes, apply the remaining deadline to HTTP I/O, and preserve thread interruption.
   - Reject false, null, failed, and unknown responses without clearing the batch.
   
   Related to #10680. Thanks to @xyueji for identifying the label-reuse failure mode. This PR keeps the pending batch and original label until the transaction state is known; immediately retrying an unknown outcome with a new label can duplicate data that StarRocks already committed.
   
   ### Does this PR introduce _any_ user-facing change?
   
   Yes.
   
   Previously, a StarRocks Stream Load failure could be logged while the buffered records were cleared, allowing the job to continue with missing rows. A reused label was also skipped without confirming whether its transaction was committed.
   
   After this change, unresolved outcomes fail the write and retain the pending batch and label. The sink does not claim exactly-once across process restarts; it preserves the existing at-least-once contract while preventing silent batch loss in one writer lifecycle.
   
   No configuration, default value, public API, SPI, serialization format, or connector registration changes are introduced.
   
   ### How was this patch tested?
   
   Unit and module verification on JDK 8:
   
   ```bash
   mvn -T 3C -f seatunnel-connectors-v2/connector-starrocks/pom.xml spotless:apply -nsu -Dmaven.gitcommitid.skip=true
   mvn -T 3C -f seatunnel-connectors-v2/connector-starrocks/pom.xml test -nsu -Dmaven.gitcommitid.skip=true
   mvn -T 3C -f seatunnel-connectors-v2/connector-starrocks/pom.xml -q -DskipTests verify -nsu -Dmaven.gitcommitid.skip=true
   ```
   
   Result: 88 tests, 0 failures, 0 errors, 0 skipped. The new regression matrix covers false/null results, retry exhaustion, pending-batch and label retention, new records arriving after a failed flush, interrupted retries, final-attempt label handling, `Success`, `Publish Timeout`, reused-label `VISIBLE`/`COMMITTED`/`ABORTED`/`UNKNOWN`, ordinary failures, unexpected statuses, bounded `PREPARE` polling, and interruption.
   
   The Stream Load protocol was also verified against a real `starrocks/allin1-ubuntu:3.3.4` FE/BE instance:
   
   1. The first request with a fixed label returned `Success`.
   2. A second request with the same label returned `Label Already Exists`.
   3. `get_load_state` returned `VISIBLE`.
   4. The target table contained only the first row, proving that the reused label did not insert a duplicate row.
   5. An invalid load returned `Fail`; `get_load_state` returned `ABORTED`; a later valid retry succeeded.
   
   A full MySQL CDC to StarRocks E2E job was not run. This patch does not modify the MySQL source, CDC deserialization, checkpoint state, schema handling, connector registration, or packaging; the real validation targets the Stream Load transaction protocol where this defect occurs. `seatunnel-engine-ui` was skipped because it is outside the affected module.
   
   ### Check list
   
   * [x] No new Jar binary package is added.
   * [x] Documentation is not required because no configuration or public contract changes.
   * [x] `incompatible-changes.md` is not required because this change is backward compatible.
   * [x] Connector registration and packaging files are not changed because this is an internal StarRocks sink correctness fix.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-V2] Prevent silent StarRocks batch data loss [seatunnel]

Posted by "corgy-w (via GitHub)" <gi...@apache.org>.
corgy-w merged PR #11431:
URL: https://github.com/apache/seatunnel/pull/11431


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org