You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "zhang-arvin (via GitHub)" <gi...@apache.org> on 2026/08/20 04:21:06 UTC
[PR] [Fix][Connector-V2] Fix MongoDB CDC snapshot stuck in non-exactly-once mode [seatunnel]
zhang-arvin opened a new pull request, #11896:
URL: https://github.com/apache/seatunnel/pull/11896
## Summary
Fixes #11889 - MongoDB CDC non-exactly-once snapshot extraction stuck issue
## Root Cause
In non-exactly-once mode, the consumer (`pollSplitRecordsIfNotExactlyOnce`) stops reading from the queue after receiving the HIGH watermark event. However, the producer (`MongodbScanFetchTask`) continues to synchronously execute the backfill task which writes backfill events to the same queue. Since the consumer has stopped consuming, the queue fills up and the producer blocks forever, causing a deadlock.
### Deadlock chain:
1. Thread 306 (debezium-snapshot-reader) blocks on `queue.enqueue()` - queue full
2. Thread 284 (Source Data Fetcher) blocks on `elementsQueue.offer()` - downstream queue full
3. Thread 261 (BlockingWorker) `poll()` returns null, enters sleep loop
4. All downstream sink threads starved
### Root cause in code:
```
IncrementalSourceScanFetcher.pollSplitRecordsIfNotExactlyOnce():
reads HIGH watermark -> sets hasNextElement = false -> stops consuming
MongodbScanFetchTask.execute():
sends HIGH watermark
synchronously executes backfill (MongodbStreamFetchTask) -> writes to same queue
QUEUE FULL -> DEADLOCK
```
## Fix
Skip the backfill step when `isExactlyOnce` is false, as backfill is only needed for the exactly-once merge logic (merging overlapping snapshot and change records). In non-exactly-once mode, an END watermark is sent directly after the HIGH watermark to allow the task to complete normally.
## Verification
- The fix eliminates the queue deadlock by ensuring the producer does not write unconsumed backfill events in non-exactly-once mode
- The task completes normally by sending END watermark directly after HIGH watermark
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Connector-V2] Fix MongoDB CDC snapshot stuck in non-exactly-once mode [seatunnel]
Posted by "nzw921rx (via GitHub)" <gi...@apache.org>.
nzw921rx commented on PR #11896:
URL: https://github.com/apache/seatunnel/pull/11896#issuecomment-5351511562
Thanks for your contribution🚀
please run: `./mvnw spotless:apply` then commit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Connector-V2] Fix MongoDB CDC snapshot stuck in non-exactly-once mode [seatunnel]
Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on PR #11896:
URL: https://github.com/apache/seatunnel/pull/11896#issuecomment-5351603609
@zhang-arvin The behavior change is correct, but please add a regression test for the non-exactly-once snapshot path. A focused test can use a bounded `ChangeEventQueue`, consume through the HIGH watermark, and assert that no `MongodbStreamFetchTask` is created. I reproduced it this way: the previous code created one backfill task after HIGH, while the current head created none and completed. This directly protects the deadlock scenario from being reintroduced.
I also ran the clean MongoDB CDC module suite on this head: 27 tests passed.
+1 to @nzw921rx 's suggestion too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Connector-V2] Fix MongoDB CDC snapshot stuck in non-exactly-once mode [seatunnel]
Posted by "zhang-arvin (via GitHub)" <gi...@apache.org>.
zhang-arvin commented on PR #11896:
URL: https://github.com/apache/seatunnel/pull/11896#issuecomment-6051959528
CI-fact update and a genuine new commit. I rebased this branch onto the current `dev` tip, and unlike the earlier rounds the head has actually moved (previously unchanged through repeated rebases):
- old head: `49b46c4c4c45ac4b0ca3962dacdc12e6e95afae1`
- new head: `4dc263f1a6e91bf0deb41cac9d27f54d3fdf27b0`
Content check: the PR's own files are byte-for-byte identical — `git diff 49b46c4c..4dc263f1a` over the PR-scoped files is empty, and the diff vs `dev` is still the same 2 commits. So your approval on the substance still stands; this just clears the ~120-commit base drift and re-triggers `Build` on a fresh SHA. The rebase also carries in `#12639` (ignore deepwiki.com in the dead-link check), which removes the `Dead links` 429 noise affecting this batch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Connector-V2] Fix MongoDB CDC snapshot stuck in non-exactly-once mode [seatunnel]
Posted by "zhang-arvin (via GitHub)" <gi...@apache.org>.
zhang-arvin commented on PR #11896:
URL: https://github.com/apache/seatunnel/pull/11896#issuecomment-5354309307
> please run: `./mvnw spotless:apply` then commit
@nzw921rx Done! Spotless has been applied and committed. Thanks for the review! 🚀
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Connector-V2] Fix MongoDB CDC snapshot stuck in non-exactly-once mode [seatunnel]
Posted by "nzw921rx (via GitHub)" <gi...@apache.org>.
nzw921rx commented on PR #11896:
URL: https://github.com/apache/seatunnel/pull/11896#issuecomment-6461252665
https://github.com/apache/seatunnel/pull/13018
This PR has solved the same problem and has been merged. Thank you for providing ideas and such precise fixes. We look forward to your continued participation and contribution in the future!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Connector-V2] Fix MongoDB CDC snapshot stuck in non-exactly-once mode [seatunnel]
Posted by "nzw921rx (via GitHub)" <gi...@apache.org>.
nzw921rx commented on code in PR #11896:
URL: https://github.com/apache/seatunnel/pull/11896#discussion_r4356221570
##########
seatunnel-connectors-v2/connector-cdc/connector-cdc-mongodb/pom.xml:
##########
@@ -93,5 +93,11 @@
<version>${junit4.version}</version>
<scope>test</scope>
</dependency>
+ <dependency>
Review Comment:
Correctly fixed, please handle dependencies and remove them. Thank you, this will help clarify the dependencies of the project.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Connector-V2] Fix MongoDB CDC snapshot stuck in non-exactly-once mode [seatunnel]
Posted by "zhang-arvin (via GitHub)" <gi...@apache.org>.
zhang-arvin commented on PR #11896:
URL: https://github.com/apache/seatunnel/pull/11896#issuecomment-5372425066
@DanielLeens thanks for the thorough review! I have addressed Issue 1:
- Removed the unused END watermark enqueue in the non-exactly-once branch — now it just returns after logging, consistent with sibling connectors (MySQL/Oracle/Postgres/SqlServer)
- Updated the regression test to assert that the END watermark is **not** present in non-exactly-once mode, and the last event is the HIGH watermark
- Added a comment explaining why the END watermark is not needed (the reader only watches for HIGH, and the stream split already starts at the low watermark)
Ready for another look. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Connector-V2] Fix MongoDB CDC snapshot stuck in non-exactly-once mode [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11896:
URL: https://github.com/apache/seatunnel/pull/11896#issuecomment-5480623008
CI/merge-state update, following up on my 2026-08-25 review of this head (`e6a03652d`). At the time I wrote that review, `gh pr view` reported `mergeable: true, mergeable_state: blocked` (i.e. only branch-protection/review gating, no textual conflict with `dev`), and I diagnosed the three failing CI jobs (`Dead links`, an unrelated `connector-edge-socket` Windows flake, and the previously-identified `PostgresCDCIT` timeout) as unrelated to this diff, just needing a rerun.
Re-checking live just now, that fact has changed: `dev` has moved 91 commits ahead since this head was cut (`behind_by=91`), and GitHub currently reports this PR as `mergeable: CONFLICTING` / `mergeStateStatus: DIRTY` — an actual textual merge conflict against current `dev`, not just review/CI gating. Since there is now a real conflict, rerunning the same jobs will not resolve this on its own: the branch needs to be synced with the latest `dev` (merge or rebase), the conflict resolved, and then CI re-run on the new head. My source-level conclusion on `MongodbScanFetchTask.java`/`MongodbScanFetchTaskTest.java` from that review is otherwise unchanged and still stands.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Connector-V2] Fix MongoDB CDC snapshot stuck in non-exactly-once mode [seatunnel]
Posted by "zhang-arvin (via GitHub)" <gi...@apache.org>.
zhang-arvin commented on PR #11896:
URL: https://github.com/apache/seatunnel/pull/11896#issuecomment-5573957492
Thanks for the review! Rebased onto latest dev (resolved the MongodbScanFetchTaskTest conflict by keeping dev's `shouldPreserveSnapshotReadFailureCause` alongside this PR's new exactly-once test). The remaining CI failures are in unrelated connectors (Dead links / windows unit-test / updated-modules-it), classic flaky jobs. Ready for re-review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Connector-V2] Fix MongoDB CDC snapshot stuck in non-exactly-once mode [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on code in PR #11896:
URL: https://github.com/apache/seatunnel/pull/11896#discussion_r4357153467
##########
seatunnel-connectors-v2/connector-cdc/connector-cdc-mongodb/pom.xml:
##########
@@ -93,5 +93,11 @@
<version>${junit4.version}</version>
<scope>test</scope>
</dependency>
+ <dependency>
Review Comment:
+1 to @nzw921rx, and thank you for flagging this. I raised the same point in my earlier rounds (Low severity, non-blocking) and I re-checked it on the current head `4dc263f1a6e9`: the root `pom.xml` already declares `mockito-inline:${mockito.version}` with test scope in its direct `<dependencies>` block (about lines 712-717), so every child module inherits it. The pre-existing `MongodbRecordUtilsHeartbeatTest` in this same module already uses `Mockito.mockStatic(...)` without any module-level declaration, which is the evidence that the new block here adds nothing.
Concrete next step for @zhang-arvin: delete this `<dependency>` block (`org.mockito:mockito-inline`) from `connector-cdc-mongodb/pom.xml`, which also leaves this PR with no `pom.xml` change at all, and push. The production change in `MongodbScanFetchTask.java` and the tests stay as they are. I only did a static read here; nothing was run locally.
On CI: the latest `Build` run for this head ended as cancelled (not a failure caused by the diff), so the push that removes the pom block will also give you a fresh run on a new head. I will recheck the new head once it is pushed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Connector-V2] Fix MongoDB CDC snapshot stuck in non-exactly-once mode [seatunnel]
Posted by "zhang-arvin (via GitHub)" <gi...@apache.org>.
zhang-arvin commented on PR #11896:
URL: https://github.com/apache/seatunnel/pull/11896#issuecomment-5372740601
Thanks @goutamadwant, good suggestion. I'll add a regression test for the non-exactly-once snapshot path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Connector-V2] Fix MongoDB CDC snapshot stuck in non-exactly-once mode [seatunnel]
Posted by "zhang-arvin (via GitHub)" <gi...@apache.org>.
zhang-arvin commented on PR #11896:
URL: https://github.com/apache/seatunnel/pull/11896#issuecomment-5354310913
@goutamadwant Regression test added in `MongodbScanFetchTaskTest`. It uses a bounded `ChangeEventQueue`, consumes through the HIGH watermark, and verifies no `MongodbStreamFetchTask` is created — directly protecting the deadlock scenario. Spotless also applied per @nzw921rxs suggestion. Thanks for the review! 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Connector-V2] Fix MongoDB CDC snapshot stuck in non-exactly-once mode [seatunnel]
Posted by "zhang-arvin (via GitHub)" <gi...@apache.org>.
zhang-arvin commented on PR #11896:
URL: https://github.com/apache/seatunnel/pull/11896#issuecomment-5572630139
Rebased onto dev (including resolving the MongodbScanFetchTaskTest conflict: kept the dev version of shouldPreserveSnapshotReadFailureCause plus this PR's new tests). The remaining 3 CI failures are unrelated to Mongo CDC.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Connector-V2] Fix MongoDB CDC snapshot stuck in non-exactly-once mode [seatunnel]
Posted by "nzw921rx (via GitHub)" <gi...@apache.org>.
nzw921rx commented on PR #11896:
URL: https://github.com/apache/seatunnel/pull/11896#issuecomment-6415332914
![Uploading image.png…]()
please enable CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Connector-V2] Fix MongoDB CDC snapshot stuck in non-exactly-once mode [seatunnel]
Posted by "nzw921rx (via GitHub)" <gi...@apache.org>.
nzw921rx closed pull request #11896: [Fix][Connector-V2] Fix MongoDB CDC snapshot stuck in non-exactly-once mode
URL: https://github.com/apache/seatunnel/pull/11896
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org