You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "fatmanverse (via GitHub)" <gi...@apache.org> on 2026/07/23 10:02:35 UTC

[PR] [Fix][Connector-V2] Fix Kafka exactly-once sink losing first record on checkpoint [seatunnel]

fatmanverse opened a new pull request, #11541:
URL: https://github.com/apache/seatunnel/pull/11541

   ## Purpose of this pull request
   
   Fixes #11534.
   
   Under `semantics = EXACTLY_ONCE`, the Kafka sink could intermittently lose
   the first record of a transaction during checkpointing (e.g.
   `KafkaIT#testKafkaToKafkaExactlyOnceOnStreaming` receiving only 9 of 10
   records).
   
   ### Root cause
   
   `KafkaSinkWriter` sends records via the producer's asynchronous `send()`,
   which returns immediately. Kafka only marks a transaction as *started*
   (`transactionStarted = true`) once the `AddPartitionsToTxn` request — issued
   asynchronously by the producer's sender thread — is acknowledged by the broker.
   
   `KafkaTransactionSender#prepareCommit()` is called by the engine before
   `snapshotState()`. If a checkpoint reached `prepareCommit()` before the first
   record's transaction registration completed, `isTxnStarted()` returned a stale
   `false`, which was then persisted into `KafkaCommitInfo`.
   
   On checkpoint completion, `KafkaSinkCommitter` resumed the transaction with
   `txnStarted = false`, and Kafka's `TransactionManager` skips `EndTxn` in that
   state. The pending broker-side transaction was never committed and eventually
   timed out / was aborted, leaving that record permanently invisible to
   `read_committed` consumers.
   
   ### Fix
   
   In `prepareCommit()`:
   
   1. `flush()` pending sends **before** capturing the transaction state, so the
      `AddPartitionsToTxn` registration is reflected in `isTxnStarted()`.
   2. Fail fast (`TRANSACTION_NOT_STARTED`, `KAFKA-08`) when the transaction has
      records but is still reported as not started after flushing — this means the
      registration never completed or a send failed asynchronously. Committing
      with `txnStarted = false` would silently drop those records, so we abort the
      transaction via the checkpoint instead of producing a lossy commit info.
   
   The empty-transaction path (`recordNumInTransaction == 0`) is unchanged and
   still legitimately reports `txnStarted = false`.
   
   ## Does this PR introduce _any_ user-facing change?
   
   No. No config option, public API, or SPI contract is changed. Under a slow
   broker, checkpoints may block slightly longer on `flush()`, which is the
   correct trade-off for exactly-once (a checkpoint timeout is preferable to
   silent data loss).
   
   ## How was this patch tested?
   
   - **Unit** — new `KafkaTransactionSenderTest` (3 cases):
     - flush happens before the transaction state is captured (race modeled by
       flipping `isTxnStarted()` inside the `flush()` stub, not by invocation
       counting);
     - fail-fast when records were sent but the transaction is not started after
       flushing;
     - empty transaction still allowed with `txnStarted = false`.
   - **E2E** — `KafkaIT#testKafkaToKafkaExactlyOnceOnStreaming` is the existing
     reproducer for this race; its `checkData` assertion already fails on both a
     lost record (matched < 10) and a duplicate (matched > 10). Added a
     regression-guard javadoc documenting that intent.
   - `spotless:apply` clean; module unit tests pass.
   
   ## Check list
   
   - [x] Code changed are covered with tests, or it does not need tests for reason in the PR description.
   - [x] If any new Jar binary package adding in your PR, please add License Notice according [New License Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/contribution/new-license.md)
   - [x] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-V2] Fix Kafka exactly-once sink losing first record on checkpoint [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11541:
URL: https://github.com/apache/seatunnel/pull/11541#issuecomment-5279185807

   @goutamadwant Thanks for the extra pass and the doc feedback — you're right, and it's worth tightening.
   
   I checked the actual abort semantics: `SinkWriter#abortPrepare()` is documented as **only invoked by the Spark engine** on a `prepareCommit()` failure (`seatunnel-api/.../sink/SinkWriter.java`, javadoc on `abortPrepare()`). `KafkaSinkWriter#abortPrepare()` does call `kafkaProducerSender.abortTransaction()`, but for Zeta/Flink that path isn't triggered when `KAFKA-08`/`KAFKA-09` is thrown — the checkpoint just fails, and the in-flight transaction is only cleaned up later, when the writer restarts from the last successful checkpoint and `KafkaSinkWriter`'s constructor calls `abortTransaction(checkpointId + 1)` on the stale transaction (or the broker times it out via `transaction.timeout.ms` in the meantime).
   
   So "both errors abort the current transaction" is accurate for Spark but overstates it for Zeta/Flink. Your proposed wording ("fail the checkpoint and prevent the transaction from being committed, then records are replayed from the last completed checkpoint after recovery") matches the code for all three engines and is the better description.
   
   This is doc-precision only, no behavior change, so I agree it doesn't block merge — just flagging so the wording can be nudged if there's another doc pass.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-V2] Fix Kafka exactly-once sink losing first record on checkpoint [seatunnel]

Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo commented on PR #11541:
URL: https://github.com/apache/seatunnel/pull/11541#issuecomment-5139575804

   <img width="1268" height="326" alt="image" src="https://github.com/user-attachments/assets/33780d0b-5fe3-431b-bd78-2bff64b06b23" />
   Please solve conflicts. thx


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-V2] Fix Kafka exactly-once sink losing first record on checkpoint [seatunnel]

Posted by "fatmanverse (via GitHub)" <gi...@apache.org>.
fatmanverse commented on PR #11541:
URL: https://github.com/apache/seatunnel/pull/11541#issuecomment-5163661857

   @SEZ9 The blocking  lifecycle issue from your changes-requested review has been addressed in the current head , together with the requested regression coverage and error documentation. The source has since been re-reviewed without a new code blocker; the CI rerun is still in progress. Please re-review the current head or dismiss the stale changes-requested review when convenient.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-V2] Fix Kafka exactly-once sink losing first record on checkpoint [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11541:
URL: https://github.com/apache/seatunnel/pull/11541#issuecomment-5141522518

   Thanks @davidzollo for catching the current merge gate.
   
   From Daniel's source review side, I still do not see a reopened Kafka exactly-once blocker on this unchanged head, but GitHub is now reporting merge conflicts. Could the contributor please sync with the latest `dev` and resolve the conflicts first? After that, the checks should be rerun against the resolved head before merge.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-V2] Fix Kafka exactly-once sink losing first record on checkpoint [seatunnel]

Posted by "fatmanverse (via GitHub)" <gi...@apache.org>.
fatmanverse commented on PR #11541:
URL: https://github.com/apache/seatunnel/pull/11541#issuecomment-5089996470

   Thanks for the detailed re-review — the blocking finding was accurate and is fixed in `fb7ab542`, along with all four follow-ups. Two of them I implemented a little differently than suggested, so let me walk through the reasoning in case you see it another way.
   
   ### Issue 1 (blocking): async failure is now scoped to its transaction
   
   Confirmed — nothing ever cleared `asyncSendException`, so a transient send error would have turned into a permanent checkpoint-failure loop. It is now reset in `beginTransaction(String)`.
   
   Of the two options you offered I went with the reset rather than `getAndSet(null)` in `checkAsyncSendException()`, because of how it combines with Issue 3. Once `send()` also inspects the same reference, a consume-once read means the first `send()` throws and clears the flag, and the following `prepareCommit()` then sees `null`, treats the transaction as healthy, and commits a transaction that already lost a record. Keeping the failure until the next transaction begins avoids that interaction.
   
   For the same reason I left `abortTransaction()` untouched: every `beginTransaction()` builds a fresh producer through `getTransactionProducer()`, so callbacks from the previous producer can't reach the new transaction's state. With `beginTransaction()` as the single reset point, the failure's lifetime is exactly one transaction, which is what lets `send()` and `prepareCommit()` share the reference safely.
   
   The recovery path you asked for is covered by `senderRecoversAfterFailedTransaction`: begin → async failure → prepareCommit throws → abort → begin → successful send → commit succeeds.
   
   ### Issue 3: fail fast on the write path
   
   Good call. `send()` now calls `checkAsyncSendException()` before handing the record to the producer, so the error surfaces on the write path instead of a checkpoint interval later. `sendFailsFastAfterAsyncSendFailureIsRecorded` also asserts that no further record reaches the producer.
   
   ### Issue 4: reflection removed from the tests
   
   Added a package-private `@VisibleForTesting` constructor taking a `TransactionProducerFactory`, using the project's shaded Guava annotation as `KafkaSourceSplitEnumerator` does. The suite now drives the real `beginTransaction()` → `send()` → `prepareCommit()` lifecycle with no reflection, so the `recordNumInTransaction` reset semantics the new guard relies on are exercised rather than injected.
   
   ### Issue 5: secondary failures are logged
   
   The callback is now a named `onSendCompleted` method that logs at WARN with the transaction id when the CAS loses, so a multi-partition incident stays diagnosable.
   
   I went with logging rather than `Throwable#addSuppressed` here: the callback runs on the producer's sender thread while the task thread may already be reading that exception in `prepareCommit()`, so mutating it would introduce a data race.
   
   ### Issue 2: docs
   
   One thing worth flagging on the paths: I couldn't find `docs/en/connector-v2/Error-Quick-Reference.md` in the tree, and there doesn't seem to be an error-code quick reference anywhere under `docs/` — the Kafka sink doc sits at `docs/en/connectors/sink/Kafka.md`. The existing KAFKA-01 … KAFKA-07 codes are undocumented too, so building that reference looks like a larger piece of work than this PR should carry. Happy to open a separate PR for it if maintainers agree it's worth having, though it would need to cover well beyond Kafka.
   
   The operational note itself is in: the Semantics section of both `docs/en/connectors/sink/Kafka.md` and `docs/zh/connectors/sink/Kafka.md` now explains that under `EXACTLY_ONCE` an async send failure fails the checkpoint — naming both codes — instead of silently dropping records.
   
   ### Verification
   
   `./mvnw test -pl seatunnel-connectors-v2/connector-kafka` → 65 tests, 0 failures (`KafkaTransactionSenderTest` 4 → 6). `spotless:apply` clean.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-V2] Fix Kafka exactly-once sink losing first record on checkpoint [seatunnel]

Posted by "nzw921rx (via GitHub)" <gi...@apache.org>.
nzw921rx commented on PR #11541:
URL: https://github.com/apache/seatunnel/pull/11541#issuecomment-5275973619

   @davidzollo If you have time, please review this PR. thanks very much🚀


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-V2] Fix Kafka exactly-once sink losing first record on checkpoint [seatunnel]

Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on PR #11541:
URL: https://github.com/apache/seatunnel/pull/11541#issuecomment-5276310586

   @fatmanverse reviewed the latest changes against the Kafka transaction and checkpoint lifecycle.
   
   The flush before reading txnStarted addresses the reported first record race. Async send failures are also retained per transaction and checked before commit, while the empty transaction path remains unchanged.
   
   One small documentation point suggestion: saying both errors abort the transaction is not fully accurate for every engine. It may be better to say they fail the checkpoint and prevent the transaction from being committed, then records are replayed from the last completed checkpoint after recovery.
   
   No code correctness blocker from my review. Thanks for the changes and contribution! 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-V2] Fix Kafka exactly-once sink losing first record on checkpoint [seatunnel]

Posted by "fatmanverse (via GitHub)" <gi...@apache.org>.
fatmanverse commented on PR #11541:
URL: https://github.com/apache/seatunnel/pull/11541#issuecomment-5086639677

   Addressed the non-blocking async-send follow-up in `9360cde36`:
   
   - register a Kafka send callback and retain the first asynchronous failure;
   - check that failure immediately after `flush()`, before reading `txnStarted`;
   - propagate it as `PRODUCE_DATA_FAILED` (`KAFKA-09`) with the original cause;
   - add a regression test for `txnStarted = true` with a failed callback;
   - narrow the `TRANSACTION_NOT_STARTED` wording to transaction registration only.
   
   The branch was also rebased onto the latest `dev`, which includes the upstream fixes for the unrelated dead-link and `MetricsApiTest` CI failures.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-V2] Fix Kafka exactly-once sink losing first record on checkpoint [seatunnel]

Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo merged PR #11541:
URL: https://github.com/apache/seatunnel/pull/11541


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org