You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "zhangshenghang (via GitHub)" <gi...@apache.org> on 2026/07/26 14:09:02 UTC
[PR] [Fix][Connector-V2] Wait for Doris data visibility [seatunnel]
zhangshenghang opened a new pull request, #11562:
URL: https://github.com/apache/seatunnel/pull/11562
## What does this PR do?
Fixes #11433.
The Doris 2PC committer now carries the stream-load label from the writer and polls the Doris FE load-state endpoint after the transaction commit succeeds. SeaTunnel reports the checkpoint as committed only after the label reaches `VISIBLE`; `ABORTED` and `CANCELLED` remain terminal failures.
Existing serialized commit information remains readable. When restoring an older checkpoint that does not contain a label, the existing commit behavior is retained.
## Tests
- Added a unit test for `COMMITTED` to `VISIBLE` polling.
- Added serializer round-trip and legacy-data compatibility tests.
- `./mvnw -q -DskipTests verify`
- `./mvnw -o -f seatunnel-connectors-v2/connector-doris/pom.xml -Dtest=DorisCommitterTest,DorisCommitInfoSerializerTest test`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Connector-V2] Wait for Doris data visibility [seatunnel]
Posted by "zhangshenghang (via GitHub)" <gi...@apache.org>.
zhangshenghang commented on PR #11562:
URL: https://github.com/apache/seatunnel/pull/11562#issuecomment-5862863796
Thanks @DanielLeens — both blockers are addressed in f8abb17f5f (on top of a fresh `upstream/dev` merge).
- **Issue 1 (unbounded HTTP call defeating the deadline)**: `HttpUtil`'s control client — the one used for `get_load_state` polling and the commit/abort control requests — now carries a `RequestConfig` with 30s connect and socket timeouts. The values match the connector's own source-side defaults (`DorisSourceOptions`: 30s/30s) and sit well below the default 300s `sink.visibility-timeout-ms`, so a single hung request is retried and the outer deadline stays meaningful. The stream-load *upload* client deliberately keeps no socket timeout so large body transfers are not cut off mid-flight.
- **Issue 2 (docs)**: `sink.visibility-timeout-ms` is now in the sink options table of `docs/en/connectors/sink/Doris.md` and `docs/zh/...`, with a short "Data Visibility Wait (2PC)" section in both languages explaining the COMMITTED-before-VISIBLE window, the `get_load_state` polling, ABORTED/CANCELLED as terminal failures, and that both the loop and the underlying control requests are bounded.
Verified locally: `connector-doris` module tests 137/137 green (incl. `DorisCommitterTest`). Issue 3 (regression test for repeated `IOException` until deadline exhaustion) is still open as the agreed non-blocking follow-up — happy to add it in a follow-up commit if you want it in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Connector-V2] Wait for Doris data visibility [seatunnel]
Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo merged PR #11562:
URL: https://github.com/apache/seatunnel/pull/11562
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org